Reminder: The syntax for the rclone sync command with arguments may look like this:
rclone sync \
--cache-rps 50 --tpslimit 10 \
$source_dir $dest_dirWhere $source_dir and $dest_dir should be formatted depending on whether you are syncing from or to your drive location.
In general, both directory paths are written in standard linux-style (with forward slashes), but the remote path needs to be preceded by the name of the remote drive and a colon. Using the name of the remote drive that we set up in the previous example, an example of a setup of these variables could be:
# source directory or directory to SYNC:
# "A regular file path"
source_dir=/sci/nosnap/yaronw/zivben/HURCS
# remote location to sync to:
# "$NAME_OF_REMOTE:$PATH_AT_DRIVE"
dest_dir=gdrive:HURCS
Remember that regardless of whether you'll be syncing from or to your drive, the first path argument (i.e. $source_dir ) is always the one rclone will copy from, and the second one ($dest_dir ) will be overwritten. If $source_dir is empty then all files in $dest_dir will be erased.
⚠️ Note that all calls to rclone sync - including the above example - should always be run as the argument to a slurm command (srun, sbatch or salloc). Not doing so could result in the job being killed and files being lost.⚠️
An example of proper use of the rclone sync command within a slurm command would be like this :
srun rclone sync \
--cache-rps 50 --tpslimit 10 \
$source_dir $dest_dir