¶ Introduction
Data-Transfer may be either direct or indirect:
- Direct: A user actively performs either an upload or download process. This is usually the case when a user needs to copy data to the cluster or copy result files from the cluster.
- Indirect: A user indirectly requires some open channel of communication for other software to function properly. For example:
- Running an interactive process like Jupyter notebook- The Jupyter engine running on the cluster needs to freely communicate with the browser running on the user's machine.
- Need random access to a resource such as a database server running on the HURCS network. The user shouldn't need to enter their credentials for every DB query.
To perform a single direct data transfer, the easiest ways would be using a file transfer protocol like SCP/FTP. Many file transfer applications exist for all operating systems that support these protocols, we will provide examples using FileZilla, but the process should be similar for any other FTP/SCP client.
For interactive or random access types of data transfer, it's usually required to open a tunnel to the remote machine. This tunnel is the open communication channel that will be used by any other application that needs access to secure HURCS resources.
All the above assumes transfer from a local machine to the HURCS cluster storage. See our ‘rclone’ tutorial for information on syncing to and from cloud services, with emphasis on the Google-Drive service, available to all HUJI members.
¶ Linux And Mac
¶ Create A Tunnel
In order to create a tunnel to the cluster you need to use L option in your ssh command: -L LocalPort:RemoteHost:RemotePort
You need to use a local port in your machine that is not used by the operating system.
In this example we used port number 12345 for the local port.
The command to open a tunnel:
ssh -J username@bava.cs.huji.ac.il -L 12345:moriah-gw:22 username@moriah-gw.cs.huji.ac.il
Don't forger to change username in the command above to your username in the cluster!
After you run this command enter your OTP and IDng passwords when requested.
If you saved the host details for moriah cluster in file ~/.ssh/config as explained here then you can enter this command:
ssh -L 12345:moriah-gw:22 moriah
You can also enter in the config file a host for tunneling by adding these lines to ~/.ssh/config file:
Host moriah-tunnel
User username
HostName moriah-gw.cs.huji.ac.il
Port 22
ProxyJump hurcs-proxy
LocalForward 12345 moriah-gw:22Now all you need is to enter this command:
ssh moriah-tunnel
¶ Transferring Data
Now that you have a tunnel to the cluster you can use commands like scp and rsync to copy files to/from the cluster or use a software called FileZilla.
See instructions bellow for each option.
Don't close the terminal window you used for creating the tunnel or disconnect from that session.
- Open a new terminal window.
- Use same port number that you used when creating the tunnel (12345 in this example).
- Use localhost as in the command examples.
- Change the paths of the files or folders and your username.
- After you run the command you will be prompt only for your IDng (Unix) password on the cluster.
¶ RSYNC
rsync can sync files and folders between source and destination.
If you get disconnected from the cluster rsync can continue from where it stopped.
Use this command when you need to transfer many files and folders or big files.
To rsync files and folders TO the cluster:
rsync -ave "ssh -p 12345" /path/on/local/machine username@localhost:/path/on/cluster
To rsync files and folders FROM the cluster:
rsync -ave "ssh -p 12345" username@localhost:/path/on/cluster /path/on/local/machine
¶ SCP
scp can copy files and folders between source and destination.
This command is recommended when you need to transfer a few small files or folders.
To scp files and folders TO the cluster:
scp -P 12345 /path/on/local/machine username@localhost:/path/on/cluster
To scp files and folders FROM the cluster:
scp -P 12345 username@localhost:/path/on/cluster /path/on/local/machine
¶ FileZilla
FileZilla is a free FTP client solution that supports FTP, FTP over TLS (FTPS) and SFTP.
Download and install FileZilla from here.
Leave the terminal session you used for creating the tunnel in the previous stage open and launch FileZilla.
Follow these steps to create a new site in for the cluster:
- Go to “File” and select “Site Manager” or press Site Manager icon.
- Press “New Site” to create a new site.
- Select the General tab.
- In “Host” field enter: localhost
- In “Port” field enter the port you used to create the tunnel: in this example it's 12345
- Set “Protocol” to SFTP.
- Set “Logon Type” to "Ask for password".
Press connect and enter your username and IDng (Unix) password (not OTP).
On the left pane in FileZilla you will see the folders on your local PC and on the right pane folders on the remote host (cluster).
You can now transfer files to and from the cluster.
¶ Windows
In Windows you can use the following software to transfer data:
- WinScp - has built-in tunneling and all configuration can be done in one place.
- FileZilla - need to configure a tunnel with Mobaxterm and then use as explained for Linux and Mac above.
¶ WinScp
WinSCP is an SFTP client and FTP client for Microsoft Windows to copy files between a local computer and remote servers using FTP, FTPS, SCP, SFTP, WebDAV or S3 file transfer protocols.
With WinSCP you don't need to open a tunnel with Mobaxterm. You can use the program built-in Tunnel feature.
Download the software from here and install with default options.
Launch WinScp and press on "New Session" or select “Session” and then “New Session”.
Select “New Site” in left pane:
- File protocol: SFTP
- Host name: moriah-gw.cs.huji.ac.il
- Port number: 22
- User name: your username
- Press “Advanced” button.
In advanced site settings window:
- Select “Tunnel” in left pane
- Select “Connect through SSH tunnel”
- Host name: bava.cs.huji.ac.il
- Port number: 22
- User name: your username
- Press OK button
Press “Save” button under “User name” field. Enter any site name you want. You can also select to create a desktop shortcut.
Click OK button when done.
Press “Login” button. The first time you will need to accept the host key.
After that you will need to enter OTP and IDng password.
In the left pane you will see folders on your local computer and in the right pane folders in the cluster.
¶ Configure a Tunnel in Mobaxterm
Open Mobaxterm and press “Tunneling” icon. The following window will open. Press “New SSH tunnel” button.
Configure the tunnel
- Select “Local port forwarding”
- Enter local port number - in this example 12345.
- Enter SSH server details:
- SSH server: bava.cs.huji.ac.il
- SSH login: your username
- SSH port: 22
- Enter remote server details:
- Remote server: moriah-gw.cs.huji.ac.il
- Remote port: 22
Click save.
Now you have a new line for the tunnel to moriah cluster.
Name it and press start button.
You will be prompt for your OTP and you will need to allow Mobaxterm to communicate with private networks in the Windows security alert window.
