SSH (part 5): Copying files using scp and sftp

5 April 2023 16 minutes Author: Endpool

Copying from and to a remote computer (scp and sftp)

Remote computer control is the ability to control a computer and use its functions from a distance, using another computer, smartphone or remote control. This distance can range from several meters to thousands of kilometers. Remote control of a computer is impossible without the use of special software – remote control programs. These are programs or functions of operating systems that allow remote access to a computer and control and administration of a remote computer in real time. Remote administration programs provide almost complete control over a remote computer: they provide the ability to remotely control the computer’s desktop, the ability to copy or delete files, launch applications, and more. It is useful for many scenarios, for example, for remote access to a computer from anywhere in the world, for training, for assisting technically challenged users, or for working with a computer that is located in another location.

The OpenSSH package includes two programs that use an encrypted SSH tunnel to copy files over the network. The first program – scp (“secure copy”) is similar to the cp program for copying files. The second program for copying files over SSH is sftp. As its name suggests, it is a secure replacement for ftp programs. sftp works like the original ftp program. However, instead of sending data in plain text, it uses an encrypted SSH tunnel. An important advantage of sftp over ftp is that it does not require a running FTP server on the remote host. It only requires an SSH server. This means that any remote machine running an SSH server can also be used as an FTP-like server.

General:

  • work through an SSH connection
  • transmit data through encrypted channels
  • do not require a special server or programs on the remote machine except the SSH server
  • use all the features of SSH, such as public key authentication and compression of the data being transmitted.

Team differences:

  • popular file managers understand the sftp protocol and can integrate with this command
  • sftp supports batch mode – files in which the sequence of actions is determined by the sequence of sftp commands
  • when recursively downloading or uploading folders, by default scp follows symbolic links and sftp does not
  • sftp is able to add partially transferred files if an interruption occurred during the transfer of a large file (configurable with an option)
  • scp does not work in interactive mode, while sftp can work in interactive mode and in auto-interactive mode
  • sftp supports the syntax of FTP commands for various actions in the file system (copying and moving files, creating and deleting files and folders, etc.)

How to use the scp utility?

The scp utility is very similar to the cp program, which copies files within the local computer. It is important to understand the key gist of scp syntax:

That is, there are only three elements in the team. The source file can be both a file on the local system and on a remote one. Similarly, the destination can be both a file on the local system and on a remote one. The path to the local file is specified in the usual way – the same as with the cp program.

A file on the remote system can be specified as follows:

Or as a URI in the form:

So, if we copy a file from a remote system to a local one, the general command looks like this:

As ./PATH/IN/LOCAL/SYSTEM you can specify simply . (dot) and then the file will be copied with the current name to the current working directory.

If we are copying a file from a local system to a remote one, the general view of the command is as follows:

The syntax of accessing a remote host is similar to SSH: USER@HOST, the only difference is that the path to the file (or just the file name) is indicated after the colon. When using a URI, the colon is followed by the port (if it is different from the default), and the file path is followed by the slash. Consider an example of executing a command on a remote system and downloading this file to the local system:

Download the file.

In the scp command, the OUTPUT-FILE is marked as [email protected]:dirlist.txt – here the colon is followed by the username on the remote system and the host address – and the information that is necessary for connecting via SSH. Then the file to be downloaded is separated by a colon. DESTINATION in this command is marked as . (dot) – which means the current working directory (on the local system).

Now consider an example of copying a file from a local machine to a remote one:

Copying a file.

In the scp send command:

  • nfile.txt – file name,
  • [email protected] – username and remote host (as in the SSH connection command),
  • . (точка) means that the file should be copied to the current working directory on the remote server, leaving the file name unchanged, i.e. nfile.txt

Options and settings

Uploading to the server or downloading a folder from the server

For recursive copying of entire directories, use the -r option. Remember that scp follows symlinks encountered while traversing the directory tree.

Saving the timestamp when transferring via scp

Using the -p option (not to be confused with the -P option, which changes the connection port), you can preserve the modification time, last access time of the file, and modes of the original file.

Data transfer rate limitation in scp

By default, the file is transferred at the maximum possible speed. If for some reason you need to reduce the speed limit, use the -l option, followed by the value in Kbit/s.

Specify port for scp

You can specify another port using the -P PORT option. Note that this option is capitalized ‘P‘ because -p is already used for another setting (preserving time and file modes).

We can also copy a file from one remote host to another remote host. The scp program can copy between two remote hosts. If the URI format is used, the port can only be specified for the target if the -3 option is used. By default, data is copied directly between two remote hosts. If the -3 option is specified, data between the two hosts will be transferred via the local host. This option disables the progress bar.

We can also use another configuration file for scp and specify the authentication keys file. The scp program works together with ssh, that is, the connection is made using ssh and all the settings of this program are applied. ssh has its own default settings, and it also reads information from its configuration files. If desired, you can specify a different file to use, this is done with the -F option – it specifies an alternative user configuration file for ssh. This option is passed directly to ssh. If passwordless (by key) login is configured, ssh will read the private key according to its settings or from the default path. The -i option can be used to select a different identifier (private key) file to authenticate against the public key on the server. This option is also passed directly to ssh.

Disabling strict filename checking can be done with the -T option. By default, when copying files from a remote host to a local directory, scp checks that the corresponding filenames match those requested on the command line to prevent the remote host from sending unexpected or unwanted files. Due to differences in how different operating systems and shells interpret file name wildcards, these checks can cause the correct files to be rejected. This option disables these checks at the cost of fully trusting that the server will not send unexpected filenames.

The -q option enables quiet mode: disables the progress bar, as well as warning and diagnostic messages from ssh.

How to use sftp?

The destination can be specified as [USER@]HOST[:PATH] or as a URI of the form sftp://[USER@]HOST[:PORT][/PATH] If the destination contains a PATH that is a directory, sftp will automatically receive the files , if a non-interactive authentication method is used; otherwise, it will be done after successful interactive authentication. If no path is specified or the path is a directory, sftp will log on to the specified host and enter interactive command mode, navigating to the remote directory if one was specified. An optional trailing slash can be used to force a path to be interpreted as a directory.

Interactive sftp commands

When entering interactive mode, sftp understands a set of commands similar to those for ftp(1). Commands are not case sensitive. Paths and file names that contain spaces must be enclosed in quotation marks. Any special characters contained within pathnames recognized by glob(3) must be escaped with a backslash (”).

  • bye – Exit sftp.
  • cd [WAY] – Change the remote directory to PATH. If PATH is not specified, the directory will be changed to the one from which the current session started.
  • chgrp [-h] PATH GROUP – Change the group of files PATH to GROUP. If the -h flag is specified, symbolic links will not be traversed. Paths can contain wildcard characters (glob) and can correspond to multiple files. GROUP must be a numeric GID.
  • chmod [-h] PATH MODE – Change access rights PATH to MODE. If the -h flag is specified, symbolic links will not be traversed, paths may contain wildcard characters (glob) and may match multiple files.
  • chown [-h] OWNER OF THE WAY– Changes the owner of the file PATH to OWNER. If the -h flag is specified, symbolic links will not be traversed, paths may contain wildcard characters (glob) and may match multiple files. The owner must be specified by a numeric UID.
  • df [-hi] [path] – Show usage information for the file system containing the current directory (or path, if specified). If the -h flag is specified, capacity information will be displayed using “mining” suffixes. The -i flag requests that site information be displayed in addition to capacity information. This command is only supported on servers that implement the [email protected] extension.
  • exit – Exit sftp.
  • get [-afPpr] REMOTE-PATH [LOCAL-PATH] -Gets the DELETED-PATH and stores this file on the local machine. If the name LOCAL-PATH is not specified, the file is saved with its previous name that it had on the remote machine. DELETED-PATH may contain wildcard characters (glob) and may match multiple files. If this is the case and LOCAL PATH is specified, then LOCAL PATH must be a directory. If the -a flag is specified, an attempt is made to download existing partially transferred files. Remember that the update assumes that any partial copy of the local file corresponds to the remote copy. If the extracted file contains differences from the partial local copy, the resulting file is likely to be corrupted. If the -f flag is specified, then after the file transfer is complete, fsync(2) will be called to flush the file to disk – this means that the file will not just be placed in the cache and queued for writing, but will actually be written to the physical disk at this point. If the -P flag or the -p flag is specified, then the file permissions and timestamps will also be fully copied. If the -r flag is specified, the directories will be copied recursively. Note that sftp does not follow symbolic links when performing recursive file transfers.
  • help – Displays help text for use by the program.
  • lcd [WAY] – Changes the working directory on the local machine (not the remote one!) to PATH. If PATH is not specified, changes directory to the home folder of the local user.
  • lls [OPTIONS-ls [WAY] – Lists the local directory specified as PATH or the current working directory if PATH is not specified. OPTIONS -ls can contain any flags supported by the ls command on the local system. PATH may contain wildcard characters (glob) and may correspond to multiple files.
  • lmkdir WAY – Creates the PATH directory on the local system.
  • ln [-s] OLD WAY NEW WAY – Creates a link from OLD-PATH to NEW-PATH. If the -s flag is specified, the link will be symbolic, otherwise it will be a hard link.
  • lpwd – Prints the current working directory on the local machine.
  • ls [-1afhlnrSt] [WAY] -Lists the files of the remote directory of the specified PATH or the current directory if PATH is not set. PATH may contain wildcard characters (glob) and may match multiple files.

The following flags are recognized by the ls command and change its behavior accordingly:

  • -1 – Information is displayed in one column.
  • -a – Adds files that start with a dot (‘.’) to the list.
  • -f – By default, the lexicographic sort order is used.
  • -h – When used with the long format option, uses the unit suffixes: Byte, Kilobyte, Megabyte, Gigabyte, Terabyte, Petabyte, and Exabyte to reduce the number of digits to four or fewer, using steps of 2 for sizes (K=1024, M=1048576, and . . ).
  • -l – Shows additional details, including permissions and owner information.
  • -n – Creates a long list with information about users and groups in numeric format.
  • -r – Reverse the sort order of the list.
  • -S – Sorts the list by file size.
  • -t – Sorts the list by the time of the last modification.
  • lumask umask – Sets the local umask to umask.
  • mkdir WAY – Creates a remote directory specified as PATH.
  • progress – Toggles the display of the progress bar.
  • put [-afPpr] LOCAL-PATH [REMOTE-PATH] – Unloads LOCAL PATH saves on remote machine. If the name REMOTE-PATH is not specified, the files are given the same name as on the local machine. A LOCAL PATH can contain wildcard characters (glob) and can match multiple files. If this is the case and DELETED-PATH is specified, then DELETED-PATH must be a directory. If the -a flag is specified, an attempt is made to unload existing partially transferred files. Remember that the update assumes that any partial copy of the local file corresponds to the remote copy. If the extracted file contains differences from the partial local copy, the resulting file is likely to be corrupted. If the -f flag is specified, then after the file transfer is complete, fsync(2) will be called to flush the file to disk – this means that the file will not just be placed in the cache and queued for writing, but will actually be written to the physical disk at this point. Note that this is only supported by servers that implement the [email protected] extension. If the -P flag or the -p flag is specified, then the file permissions and timestamps will also be fully copied. If the -r flag is specified, the directories will be copied recursively. Note that sftp does not follow symbolic links when performing recursive file transfers.
  • pwd – Show remote working directory.
  • quit – Exit sftp.
  • reget [-Ppr] REMOTE-PATH [LOCAL-PATH] – Resume download DELETED-PATH. Equivalent to the get command with the -a flag set.
  • reput [-Ppr] [LOCAL-PATH] REMOTE-PATH] – Resume unloading [LOCAL-PATH]. Equivalent to the put command with the -a flag set.
  • rename OLD WAY NEW WAY – Rename the remote file from OLD PATH to NEW PATH.
  • rm WAY– Delete the deleted file specified as PATH.
  • rmdir WAY– Delete the remote directory specified as PATH.
  • symlink OLD WAY NEW WAY – Create a symbolic link from OLD-PATH to NEW-PATH.
  • version – Show sftp protocol version.
  • !TEAM – Run the COMMAND in the local shell.
  • ! – Symbol for shielding transfer to the local shell.
  • ? – Synonym for help.

sftp options

The following options are specified on the command line to start the sftp client. They affect both running sftp in an interactive session and without interactive mode.

  • -a – An attempt is made to resume interrupted transfers rather than overwriting existing partial or completed file copies. If the partial content is different from what will be transmitted, the resulting files are likely to be corrupted.
  • -B BUFFER SIZE – Sets the size of the buffer that sftp uses for file transfers. Large buffers require fewer processing cycles due to higher memory consumption. The default value is 32,768 bytes.
  • -b BATCH_FILE – In batch mode, a series of commands is read from an input batch file rather than standard input. Since there is no user interaction, this mode should be used with non-interactive authentication to avoid having to enter a password during connection. BATCH_FILE in the form ‘-‘ can be used to indicate standard input. sftp will terminate if any of the following commands fail: get, put, reget, reput, rename, ln, rm, mkdir, chdir, ls, lchdir, chmod, chown, chgrp, lpwd, df, symlink, and lmkdir . Error termination can be suppressed on a per-command basis by prefixing the command with a “-” character (eg -rm /tmp/blah*). Command output can be suppressed by prefixing it with the ‘@’ symbol. These two prefixes can be combined in any order, for example -@ls /bsd.
  • -C – Enable compression (via the -C flag in ssh).
  • -D sftp_server_path – Connect directly to the local sftp server (not via ssh). This option can be useful when debugging the client and server.
  • -F CONFIGURATION_SSH – Specifies an alternate configuration file for ssh. This option is sent directly to ssh.
  • -f – Files must be written to disk immediately after transfer. When uploading files, this feature is enabled only if the [email protected] extension is implemented on the server.
  • -i IDENTIFICATION FILE – Selects a file from which to read an identifier (private key) for public key authentication. This option is passed directly to ssh.
  • -l LIMIT – Limits the bandwidth used, specified in Kbit/s.
  • -P PORT – Specifies the port to connect to on the remote host.
  • -p – Stores the modification time, access time, and modes of the original transferred files.
  • -q – Quiet mode: Disables the progress bar, as well as warning and diagnostic messages from ssh.
  • -R NUMBER OF REQUESTS – Specifies the maximum number of pending requests. Increasing this value may slightly improve file transfer speed, but will increase memory usage. Default is 64 outstanding requests.
  • -r – Recursively copy entire directories on upload and download. Note that sftp does not follow symbolic links encountered in tree traversal.
  • -v – Raise the level of verbality. This option is also passed to ssh.

SCP and SFTP are not only utilities, but also protocols. That is, other programs can support work with them and be used as a convenient graphical interface. Thanks to this, you will be able to manage files on the server, for example, through FileZilla or the usual file managers.

Other related articles
Found an error?
If you find an error, take a screenshot and send it to the bot.