File transferring with SCP and Rclone
Learn how to transfer files from your local computer to your remote computer using SCP and Rsync
SCP (secure copy) and rsync are popular ways to transfer files between different computers, such as from your local computer to your Tensordock instance or vice versa.
Method 1: SCP
SCP (secure copy) is a command-line utility that allows you to securely copy files and directories between two locations.
With scp, you can copy a file or directory:
From your local system to a remote system.
From a remote system to your local system.
Between two remote systems from your local system.
When transferring data with scp, both the files and password are encrypted so that anyone snooping on the traffic doesn’t get anything sensitive.
SCP General Syntax
Before going into how to use the scp command, let’s start by reviewing the basic syntax.
The scp command syntax take the following form:
Copy
OPTION- scp options such as cipher, ssh configuration, ssh port, limit, recursive copy …etc.[user@]SRC_HOST:]file1- Source file.[user@]DEST_HOST:]file2- Destination file
Local files should be specified using an absolute or relative path, while remote file names should include a user and host specification.
scp provides a number of options that control every aspect of its behavior. The most widely used options are:
-P- Specifies the remote host ssh port.-p- Preserves files' modification and access times.-q- Use this option if you want to suppress the progress meter and non-error messages.-C- This option forcesscpto compresses the data as it is sent to the destination machine.-r- This option tellsscpto copy directories recursively.
Before you begin
The scp command relies on ssh for data transfer, so it requires an ssh key or password to authenticate on the remote systems.
The colon (:) is how scp distinguish between local and remote locations.
To be able to copy files, you must have at least read permissions on the source file and write permissions on the target system.
Warning: Be careful when copying files that share the same name and location on both systems, scp will overwrite files without warning.
When transferring large files, it is recommended to run the scp command inside a screen or tmux session.
Copy a Local File to a TensorDock instance
To copy a file from a local to a TensorDock instance run the following command:
Where file.txt is the name of the file we want to copy, remote_username is the user on the remote server (likely user), 10.10.0.2 is the server IP address. The /remote/directory is the path to the directory you want to copy the file to. If you don’t specify a remote directory, the file will be copied to the remote user home directory.
Omitting the filename from the destination location copies the file with the original name. If you want to save the file under a different name, you need to specify the new file name:
Because TesnorDock's SSH is listening on a forwarded port (rather than the default 22) we will have to specify the port using the -P argument. You can see in Figure 1 that port 20496 forwards to 22, so we will include -P 20496 in our scp command, as seen in Figure 2.
You will be prompted to enter the user password, and the transfer process will start.
The command to copy a directory is similar to copying files. The only difference is that you need to use the -r flag for recursive.
To copy a directory from a local to remote system, use the -r option:
Copy a TensorDock instance file to your local system
To copy a file from a TensorDock instance to a local system, use the remote location as the source and local location as the destination.
For example to copy a file named file.txt from a remote server with IP 10.10.0.2 run the following command:
Don't forget to include the -P option for port forwarding like in the previous section
You will again be asked to enter your password.
Method 2: Rclone
Rclone is a command-line program to manage files on cloud storage. It is a feature-rich alternative to cloud vendors' web storage interfaces. Over 70 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.
In this tutorial we will teach you how to use rclone on your Google Drive to transfer/sync files to your TensorDock instance.
Installing Rclone
Install rclone on your local computer.
To install rclone on Linux/macOS/BSD systems, run:
For beta installation, run:
Note that this script checks the version of rclone installed first and won't re-download if not needed.
If this does not work for you, refer to https://rclone.org/install/ for more information on installation.
Configuring rclone
To configure rclone, we will first configure it on our local computer and then transfer the configuration file to our TensorDock instance.
Once installed, enter rclone configto configure a new remote. Enter "n" to create a new remote and follow the prompts.
name: Name the remote something informative, e.g., "google-drive".
Storage: After choosing a name, the next prompt should be a long numbered menu with different storage solutions. Look for Google Drive and enter the associated number.
client_idand client_secret: The next prompts are Google Application Client ID client_idand Google Application Client Secret client_secret. Leave both prompts blank by pressing "Enter" to accept the default values. Rclone's default client ID is shared by all users of rclone, possibly resulting in slow performance. It is recommended to create your own client ID instead. Instructions for this can be found here.
scope: Choose "1" to allow access to all files.
service_account_file: Leave this blank by pressing "Enter" to use interactive login.
Edit advanced config? (y\n): Enter "n".
Use auto config?: Enter "y".
Your browser should then open a new page. Log in to your Google Account if needed and authorize the rclone app for access.
Configure this as a Shared Drive (Team Drive)?: Enter "n" if it is a personal drive. Enter "y" if it is a shared drive.
You should then see a configuration complete message and a copy of the configuration settings.
Keep this "google-drive" remote?: Enter "y" if all the information was entered correctly.
Enter "q" to exit the main rclone configmenu.
Copying config file to TensorDock instance
First, make sure you have rclone installed on your TensorDock instance, and if it is not, use the above instructions to install rclone on your TensorDock instance
Find the config file by running rclone config file, for example
Run the rclone config file on the TensorDock instance to find out where to transfer your config.
Now transfer it to the remote box (scp, cut paste, ftp, sftp, etc.) and place it in the correct place (use rclone config file on the TensorDock instance to find out where). For example:
Once copied to the correct location, you can then test the connection by listing files in the remote. For example:
Be sure to include the : in the directory name, as this indicates that you are using a remote box.
Once copied to the correct location, you can then test the connection by listing files in the remote. For example:
Using Rclone
You have now set up rclone on your TensorDock instance!
You can now treat your remote box similar to how you would a virtual machine.
You can also copy files very similar to scp, with instructions here: https://rclone.org/commands/rclone_copy/
Last updated
