We can upload or sync our files with the external directory using the rclone sync command. We will outline the syntax of this command on its own so that you can better understand how it works - but note that rclone sync is a very slow operation and could jam up the login node, so it should always be called with slurm, as described in the example below.
⚠️ Sync operations should always be run with slurm. ⚠️ ️ HURCS reserves the right to kill any sync operation running from a login node. ⚠️ This could result in data loss. ⚠️
The syntax for the command is: rclone sync <args> $sourcedir $destdir , where $source_dir and $dest_dir should be formatted depending on whether you are syncing from or to your drive location.
The following arguments can be used with this command:
--cache-rps <limit> & --tpslimit <limit> - limit the number of requests (to enforce compliance with remote drive's API's limits)
--dry-run - can be used to show what will be uploaded without executing
--copy-links - follow symbolic links
--drive-use-trash=$X (true\false) - deleted files can be either permanently deleted (false) or moved to gdrive’s trash (true - default)
⚠️ Warning! Regardless of whether you'll be syncing from or to your drive, the first path argument (i.e. $source_dir ) is always the one rclone will copy from, and the second one ($dest_dir ) will be overwritten. Running rclone sync with an empty source dir will erase the files in the destination folder! ⚠️
If you accidentally overwrite your external storage, hopefully the external provider will have provided some sort of snapshot or backup (You may want to check that it does this, before you attempt the overwrite.) If you accidentally overwrite your local storage, you can retrieve your files from the most recent snapshot.
We will want to use slurm's sbatch command to run rload sync, so as not to clog up the login node. Click here to see an example of this in action.