How To Use BitTorrent Sync to Synchronize Directories in Ubuntu 14.04

 

Syncing files and directories between servers and local machines is a very common requirement when dealing with networked computers. One method of automatically syncing the contents of directories is with a technology called BitTorrent Sync. This software leverages the BitTorrent protocol that is commonly used for file sharing as a synchronization tool.

Communication through BitTorrent Sync is encrypted end-to-end based on a unique shared secret that is auto-generated. While BitTorrent as a file sharing mechanism is a public service, the way that BitTorrent Sync uses the protocol is private, meaning that files can be transferred securely.

In this guide, we will demonstrate how to install and use BitTorrent Sync on two Ubuntu 14.04 servers. We will show you how to set up your shared directories, and how to set up SSL encryption for the web interface to securely administer your servers.

Install BitTorrent Sync

The first step that we need to get started is to install the BitTorrent Sync software on both of our server instances. Many of the procedures in this guide will be mirrored across both machines, so make sure you duplicate your commands for each machine.

There is no official BitTorrent Sync package available in Ubuntu's default repositories. However, there is a well-maintained PPA (personal package archive) created by Leo Moll (known as tuxpoldo) that we can use to get up-to-date packages.

On both of your servers, add this PPA so that our systems can pull down the packages:

sudo add-apt-repository ppa:tuxpoldo/btsync

Now, we need to update our local package index so that our systems know about the newly available software. We'll then install BitTorrent Sync, as well as nginx to add SSL encryption to our web interface later on:

sudo apt-get update
sudo apt-get install btsync nginx

You will be asked quite a few questions in prompts when you attempt to install. For now, press ENTER through all of the prompts. We will be reconfiguring our services momentarily in a more in-depth manner.

Configure BitTorrent Sync

Now that the software is installed, we're actually going to run the configuration script that prompts us for values a second time. This time, however, we will have access to additional options that we require for our purposes.

To run the script again, this time choosing our settings, type this on each server:

sudo dpkg-reconfigure btsync

This will run you through even more prompts than during the initial installation. For the most part, we will be going with the default values and you can just press ENTER.

Below, I've outlined the values that you need to configure:

• Web Interface Bind IP Address: 127.0.0.1
• The username for accessing the web interface: [Choose whatever you would like. We will keep the admin account in this example.]
• The password for accessing the web interface: [Choose whatever you would like. We will be using password for demonstration purposes.]
• Umask value to set for the daemon: 002

As you can see, for the vast majority of settings, we can accept the defaults. The above choices though are very important. If you mis-configure these, run the command again to correct your selections.

Configure SSL Front-end to the BitTorrent Sync Web Interface

Now, we have BitTorrent Sync set up for the most part. We will set up our sync directories in a bit. But for now, we need to set up our nginx web server with SSL.

You may have noticed that we configured our web interface to only be available on the local loopback interface (127.0.0.1). This would normally mean that we would not have access to this when running BitTorrent Sync on a remote server.

We restricted access like this because, although the BitTorrent Sync traffic itself is encrypted, the traffic to the web interface is transmitted in plain text. This could allow anyone watching traffic between our server and local computer to see any communication sent between our machines.

We are going to set up nginx with SSL to proxy connections through SSL to our BitTorrent web interface. This will allow us to securely administer our BitTorrent Sync instance remotely.

Again, we will need to do all of these steps on both of our hosts.

Generate the SSL Certificate and Key

The first step towards getting this set up is to create a directory to hold our SSL certificate and key. We'll do this under the nginx configuration directory hierarchy:

sudo mkdir /etc/nginx/ssl

Now, we can create our SSL certificate and key in a single motion by issuing this command: 

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/nginx.key -out /etc/nginx/ssl/nginx.crt

You will be asked to fill out some information for your certificate. Fill out the fields as best as you can. The only one that really matters is this one:

Common Name (e.g. server FQDN or YOUR name) []:

In this field, enter your server's domain name or public IP address.

Configure Nginx to Encrypt Traffic with SSL and Pass to BitTorrent Sync

Now, we can configure our nginx server blocks to use our SSL certificates when communicating with remote clients. It will then the information to our BitTorrent Sync web interface listening on the local interface.

We will leave the default nginx server block file intact in case you need to use this in the future. Since BitTorrent Sync operates on port "8888" by default, we will use this as the front-end SSL port as well.

Create a new server block file by opening a new file with sudo privileges in your editor:

sudo nano /etc/nginx/sites-available/btsync

Inside, we need the to add the following lines:

server {
    listen server_domain_or_IP:8888 ssl;
    server_name server_domain_or_IP;

    access_log /var/log/nginx/access.log;

    ssl_certificate /etc/nginx/ssl/nginx.crt;
    ssl_certificate_key /etc/nginx/ssl/nginx.key;

    location / {
        proxy_pass http://127.0.0.1:8888;
    }
}

Make sure you change the red text to your server's domain name or public IP address. This will tell nginx to bind to the same port that the BitTorrent Sync web interface is using on the local interface. The difference is that nginx will use the public address and require SSL.

It will use the SSL certificate that we created to encrypt the traffic to the client. It will then pass it to the BitTorrent Sync interface. In this way, the traffic between the server and the client will be encrypted, but the BitTorrent Sync interface will operate as if we were accessing it from the server itself.

When you are finished, save and close the file.

Now, we just need to link the file so that it will be enabled:

sudo ln -s /etc/nginx/sites-available/btsync /etc/nginx/sites-enabled/

We can now restart the service to implement our changes:

sudo service nginx restart

Make sure you go through these procedures on each of your two servers.

Create a Shared Directory

We now have BitTorrent Sync configured, and have set up SSL and nginx to encrypt our sessions with the web interface. Before we begin to use the web interface, we should set up the directories that we want to sync. Because of the way that BitTorrent Sync creates files that it has mirrored from a remote host, our configuration for this portion is pretty important.

First, in this guide, we will be syncing directories located at /shared on both servers. Let's create these directories now:

sudo mkdir /shared

Once you have the directory, we are going to give our root account user ownership over the directory. At the same time, we will give the "btsync" group (this was created during the installation) group ownership of the directory:

sudo chown root:btsync /shared

There are many different ways you can configure this access, each with implications. We are demonstrating a fairly flexible system here that will minimize the permissions and ownership conflicts.

After we assign ownership, we should adjust permissions. We will set the setgid bit on the directory so that the btsync group will be given group ownership to any files created in the directory. To make this work correctly, we'll also need to give the group write permissions:

sudo chmod 2775 /shared

Finally, since our regular system account is not the user owner or group owner of the directory, we will need to add our regular account to the btsync group. This will allow us to access and interact with the content in this directory as our regular user:

sudo usermod -a -G btsync your_user

Note: At this point, you must log out and log back in for these changes to register in your current environment. Exit by typing:

exit

Now log back in.

Access the BitTorrent Sync Web Interface

Now that we have everything set up, we can begin taking a look at the administrative web interface to pull the pieces together.

To begin, you will need to access both servers in a web browser on port "8888" using the "https" protocol. This should look something like this:

https://server_domain_or_IP:8888

You will most likely see a warning displayed that looks like this:

This is only a warning telling you that your browser does not recognize the party that signed your SSL certificate. Since we generated self-signed SSL certificates, this makes perfect sense and is expected, and we can safely click "Proceed anyways" or whatever similar button your browser gives you.

You will be prompted for the username and password that you selected while configuring BitTorrent Sync. In our example, the credentials were admin and password, but yours (especially the password) may be different.

Once you authenticate, you should see the main BitTorrent Sync Web interface:

Add the Shared Directory to your First Server

We can not begin to add the directory we configured to the web interface.
Click on the "Add Folder" button in the upper-right corner. You will be given a dialog box for adding a directory to the BitTorrent Sync interface:

Scroll to the /shared directory that we created and click on it. It should populate the "Path" field with the correct value. Next to the "Secret" field, click on the "Generate" button to create a secret key for the directory:

Click on the "Add" button in the lower-right corner. Your directory will be added to the BitTorrent Sync web UI. Now, we have a new button available. Click on the "Secret/QR" button associated with the /shared directory that you just added:

You will be presented with a dialog box that gives you the secret for this directory. This is the way to sync this directory with another instance of BitTorrent Sync. The software allows you to set up full access to the directory (read and write access), or read-only access. For our guide, we will be configuring full access to allow two-way syncing, but this is simply a preference.

You will need to copy the "Full access" secret from this interface to set up the syncing with your second server.

Add the Shared Directory and Secret to the Second Server

Now that we have the first server configured to share its directory, we need to set up our second server.
We will go through most of the same steps, with some slight variations.

Once again, sign into the web interface, this time, using the second server's domain name or IP address. Remember to use "https" and port "8888":

https://second_server_domain_or_IP:8888

You will see the SSL warning again, and you will need to authenticate. You will come to the same empty interface that we saw before.

Click on the "Add Folder" button, as we did before. Select the /shared directory that we created.
At this point, instead of generating a new secret, we want to use the secret that was generated on the first server. This will allow these two instances to communicate, as each secret is unique and randomly generated. Enter the secret from the first server:

Click on the "Add" button in the lower right corner when you are finished.
In a few moments, the "Connected devices and status" column in the main interface will populate with the information about the companion server:

This means that your servers are communicating with each other and can sync content.

 

Test BitTorrent Syncing

Let's test our current setup.
On either of your servers (it does not matter which one if you configured full access), move into the /shared directory:

cd /shared

We can will create 10 sample files by typing:

touch file{1..10}

After a moment, on your other server, you should be able to see the files you created:

# On the second server
cd /shared
ls -l

total 0
-rw-rw-r-- 1 btsync btsync 0 May 19 17:07 file1
-rw-rw-r-- 1 btsync btsync 0 May 19 17:07 file10
-rw-rw-r-- 1 btsync btsync 0 May 19 17:07 file2
-rw-rw-r-- 1 btsync btsync 0 May 19 17:07 file3
. . .

As you can see, our files were synced over. If you look at the web interface though, this sync has not registered. This is because these files don't contain any actual data.

We will test whether it can detect when we transfer files with content by writing data to those files from our second server. This will also allow us to test that we can sync changes back to the first server.

On the second server, you can write the phrase "some content" to each of the files you created by typing:

for item in /shared/file{1..10}; do echo "some content" > $item; done

After a few seconds, the files on the first server should show the content you added:

# On first server
cat /shared/file1

some content

You should also see that the web interface has also been updated to reflect the number of files and the amount of space that has been synced across the servers:

If this is working, you have successfully configured BitTorrent Sync to mirror your changes between servers.

Conclusion

You should now have a flexible setup that allows you to securely transfer files between remote servers. Furthermore, this configuration allows you to administer the service through a secure connection by leveraging SSL.

The application itself is quite flexible and can be used in a variety of ways. Some useful features are the ability to scan secrets as QR codes on your mobile device, the ability to configure read-only access to content, and the ability to provide clients with one-time use secrets. You can also configure your servers to only communicate with certain hosts.

The BitTorrent Sync service also provides a simple version control system, which utilizes a hidden ./SyncArchive directory in shared directory to keep old versions of files. You can also implement restrictions like rate limiting if you want to make sure that your files are synced without affecting other services.