Using rsync for backups on Linux/Unix systems

The rsync utility is a remarkably efficient file synchronization utility that can help you transfer data locally or over the network securely and reliably.

Despite having other tools available for Linux and Unix systems, rsync maintains its leading role in performing website migrations and saving backups on the same server or sending them to a remote system. In this article we will learn what rsync is and how using rsync for backups can help you keep your system protected from data loss.

What is rsync?

The name rsync stands for remote sync, and it is a utility for syncing data between systems. After it was released to the world over two decades ago, it has since become a file transfer and synchronization standard for Linux and Unix. It is included by default in all popular Linux/Unix distributions.

How does rsync work?

When using rsync for backups, it provides incremental file transfer, which only transfers new or updated files, resulting in faster data transfer times and lower bandwidth usage. The rsync utility determines what files differ between the source and destination systems by checking their modification dates and file sizes. Also, rsync will replace a file if it is newer on the destination system unless you use the update option.

Another great feature the utility provides is file compression — rsync can compress the data stream, reducing the size of the data sent even more. You can skip using this option when transferring data locally.

The rsync utility can be used both to transfer files and directories locally or to remote systems over the network. This is a nice feature when using rsync for backups of remote Linux/Unix systems.

Just like other popular file transfer utilities, including SSH File Transfer Protocol (SFTP) and Secure Copy Protocol (SCP), rsync goes over Secure Shell (SSH) for transmitting data to remote systems. By doing so, the transfer process adds a great level of security to the file synchronization process. But what sets rsync apart from other tools is how it efficiently performs the data transfer operations.

When to use rsync — the use cases

The rsync utility can help you accomplish different website administration tasks. Here are the main three use cases where the utility can be extremely useful:

• Website Migration — You can use rsync to migrate your website to a new server.
• Website Cloning — You can transfer your website files to create a staging version of it or move your website to a new domain name. The process is similar for all content management systems, including WordPress and Magento.
• Website Backups — You can save backups of your website by transferring data to a remote system or another directory on the same machine.
  

The rsync utility vs. SCP vs. SFTP — five factors to consider

SCP, which stands for Secure Copy or Secure Copy Protocol, also allows you to transfer data locally and remotely while keeping the process safe by employing encryption mechanisms. The SFTP tool is used for the same purpose.

However, the rsync utility provides significant benefits compared to other file transfer utilities like SCP or SFTP. Let’s see why rsync has become a universal file transfer utility by comparing the available options.

Below we will compare the three most common file transfer utilities for Linux and Unix systems, taking the five most essential aspects of the data transfer process into consideration:

As you can see, all three utilities can be used for remote file transfer, supporting data encryption by going over SSH. In addition, both rsync and SCP allow you to transfer data locally, while SFTP only works when copying data to remote systems.

The critical difference is the support for incremental file transfer that rsync provides. The incremental file transfer feature made rsync the utility of choice for creating backups. However, before we can dive into the process of using rsync for backups, we need to understand what incremental backups are and how to use them.

Incremental backups and rsync

An incremental backup saves only new data or "delta data points" that have been changed since the previous backup ran as opposed to a full backup. Full backups copy all data regardless of whether any parts of it were changed. Incremental backups save time and disk space, especially if your data does not undergo frequent changes.

The rsync utility can help you create both full and incremental backups. You can make full backups regularly and then run incremental backups by having rsync only save what was changed since the previous full backup ran.

Using rsync for backups

You can back up your website using rsync by transferring files to a special backup directory on the same server or to a remote system, for example, using cloud storage services. The rsync utility can be used to make your own custom backup scripts. This way, you can control how backups are saved and create your own backup retention policy.

Two easy ways to backup using rsync

Site admins should be aware that rsync as a utility does not provide any way to automate backup retention or create new incremental backups unless you use a custom script which can be difficult if you are not familiar enough with the Linux/Unix command line interface. But there are a few ways you can still use rsync for backups without having any coding skills.

The first approach includes file archiving and allows for saving multiple full backups. At the same time, the other focuses on having one complete backup of your data that can be easily updated as you apply changes to the files on the source system.

1. Using file archiving and compression

Using file archiving is especially useful if you want to save full website backups. File compression allows you to reduce disk usage while still being able to extract all files or directories from an archive once the need arises. For example, you can use Tar and Gzip to create and unzip Tar.Gz files.

One way to create backups using rsync is to archive and compress data on the source system and transfer it as one file to the destination machine to have a full backup of your website. The disadvantage of this approach is that as your full backups are compressed file archives in their nature, rsync will not be able to update files inside the existing compressed archive, but you can have multiple backups saved at once.

2. Using rsync to update the backup you saved

You may also have other backup solutions in place, including control panel backups, plugin backups, or off-server backup services, so you do not need to save multiple backups somewhere else. To be safe and ensure data redundancy, you can use rsync to transfer your website files and a database backup to a remote system and regularly update the contents of the backup as your website data changes. This way, you will have one full website backup elsewhere in case you need to restore your website when there are no other newer backups or backups from a particular day available.

Creating local backups using rsync

You can use rsync to back up your website locally by saving backups to another directory. In the example below, we use rsync to back up our website files from the html folder. We have created a new directory named Backups and will transfer the contents of the html folder to the Full_backup_May10 directory:

[root@host wordpresstest.com]# rsync -avzHP --dry-run html/ Backups/May/Full_backup_May10/
**
Here you will see the list of the files Rsync will transfer
**
sent 397,070 bytes  received 39,178 bytes  872,496.00 bytes/sec
total size is 140,486,085  speedup is 322.03 (DRY RUN)

The first thing we must pay attention to is the trailing slashes after the directory names. Here we are transferring all contents of the html directory to Full_backup_May10, which means we are only transferring its files and subdirectories. If we remove the slash after the html folder, we will have the whole directory copied over.

We are using the following options to specify the rsync behavior:

• a — Archive mode used to copy files recursively while preserving symbolic links, file permissions and ownership, and timestamps.
• v — Verbose mode to get more detailed information on the file transfer process.
• z — Enable file compression. This option can be omitted when transferring files locally but is incredibly useful when sending data over the network to reduce the amount of it transmitted.
• H — Preserve hard links.
• P — Show the progress of the data transfer.
• --dry-run or -n - Perform a trial run without making any changes.
  

We can transfer a file archive the same way:

[root@host wordpresstest.com]# rsync -avzHP --dry-run website_backup.tar.gz backups/may/full_backup_May10/
sending incremental file list
website_backup.tar.gz

sent 64 bytes  received 19 bytes  166.00 bytes/sec
total size is 41,196,947  speedup is 496,348.76 (DRY RUN)

Using rsync to backup data remotely

Transferring data over the network requires rsync to go over SSH to establish a secure connection and ensure encryption of the transmitted data. As we are making an SSH connection, we need to know the authentication credentials that will allow us to connect to the remote system. We can also connect to the remote system first and use rsync to transfer data from the source system. Both options can work.

1. Making a connection from the destination server

Here we are logged in to the destination machine and transferring a .tar.gz website backup file from the source server. The source server has a different SSH port, so we need to use the -e flag to specify it.

We are connecting to root, but any SSH user with the correct permissions can be used. The /var/www/wordpresstest.com/website_backup.tar.gz file is what we are transferring from the source system, and /home/wpsite is the directory on the destination machine:

[root@host public_html]# rsync -azvHP -e "ssh -p 2254" root@IP-Address-Of-The-Source-Server-or-Its-Hostname:/var/www/wordpresstest.com/website_backup.tar.gz /home/wpsite/
root@source_server's user password: 
receiving incremental file list
website_backup.tar.gz
     38,485,093 100%  104.27MB/s    0:00:00 (xfr#1, to-chk=0/1)

sent 43 bytes  received 38,494,588 bytes  8,554,362.44 bytes/sec
total size is 38,485,093  speedup is 1.00

2. Making a connection from the source server

Here we are connecting to the destination machine from the source server. You can see that we specify the file we will transfer right after listing our rsync options, but before the address of the destination system:

[root@host wordpresstest.com]# rsync -avzHP website_backup.tar.gz root@IP-Address-Of-The-Destination-Server:/home/wpsite/
root@67.43.11.226's password:
sending incremental file list
website_backup.tar.gz
     41,196,947 100%   11.90MB/s    0:00:03 (xfr#1, to-chk=0/1)

sent 36,676,064 bytes  received 43,491 bytes  2,532,383.10 bytes/sec
total size is 41,196,947  speedup is 1.12

Conclusion

In summarizing this article, rsync is a great file transfer utility for Linux and Unix systems that allows you to sync data between two systems over the network or copy files to another directory on the same machine. The rsync utility can be used for several different tasks, including website migration, cloning, or saving backups. Using the rsync utility for backups, you can create another backup storage depot to protect your data.

Nexcess offers managed hosting services optimized for the platform of choice. Regardless your Content Management System (CMS), we have created the perfect environment for your website to grow and develop, attracting more visitors and expanding your reach. The best solutions from the technology world paired with years of experience providing outstanding support — all included in our Managed Hosting Plans for a reasonable price.

Note about the original publication date

This article was originally published in September 2019. It has since been updated for accuracy and comprehensiveness.