How to transfer large amounts of data via network.

The elegant, open source kdirstat (and it’s ports to MacOSX Disk Inventory X and Windows Windirstat) are quick ways to visualize what’s taking up space on your disk so you can either exclude the unwanted data that needs to be copied or delete it to make more space. All of these are fully native GUI applications that show disk space utilization by file type and directory structure.

rsync, from the fertile mind of Andrew (samba) Tridgell, is an application that will synchronize 2 directory trees, transferring only blocks which are different.

For example, if you had recently added some songs to your 120 GB MP3 collection and you wanted to refresh the collection to your backup machine, instead of sending the entire collection over the network, rsync would detect and send only the new songs.

For example, the first time rsync is used to transfer a directory tree, there will be no speedup.

but a few minutes later after adding danish_wind_industry.html to the FF directory

So the synchronization has a speedup of 5600-fold relative to the initial transfer.

Even more efficiently, if you had a huge database to back up and you had recently modified it so that most of the bits were identical, rsync would send only the blocks that contained the differences.

Here’s a modest example using a small binary database file:

After the transfer, I update the database and rsync it again:

There are many utilities based on rsync that are used to synchronize data on 2 sides of a connection by only transmitting the differences. The backup utility BackupPC is one.

The open source rsync is included by default with almost all Linux distributions. Versions of rsync exist for Windows as well, via Cygwin and DeltaCopy

Unison is a slightly different take on transmitting only changes. It uses a bi-directional sync algorithm to unify filesystems across a network. Native versions exist for Windows as well as Linux/Unix and it is usually available from the standard Linux repositories.

From a Ubuntu or Debian machine, to install it would require:

bbcp seems to be a very similar utility to bbftp below, with the exception that it does not require a remote server running. In this behavior, it’s much more like scp in that data transfer requires only user-executable copies on both sides of the connection. Short of access to a GridFTP site, this appears to be the fastest, most convenient single-node method for transferring data.

The code compiled & installed easily with one manual intervention

bbcp can act very much like scp for simple usage:

The file transferred in under 10s for a 633MB file, giving >63MB/s on a Gb net. Note that this is over our very fast internal campus backbone. That’s pretty good, but the transfer rate is sensitive to a number of things and can be tuned considerably. If you look at all the bbcp options, it’s obvious that bbcp was written to handle lots of exceptions.

If you increase the number of streams (-s) from the default 4 (as above), you can squeeze a bit more bandwidth from it as well:

or almost 85MB/s for 4.2GB which is very good sustained transfer.

Even traversing the CENIC net from UCI to SDSC is fairly good:

or almost 30MB/s.

When making the above test, I noticed the disks to and from which the data was being written can have a large effect on the transfer rate. If the data is not (or cannot be) cached in RAM, the transfer will eventually require the data to be read from or written to the disk. Depending on the storage system, this may slow the eventual transfer if the disk I/O cannot keep up with the the network. On the systems that I used in the example above, I saw this effect when I transferred the data to the /home partition (on a slow IDE disk - see below) rather than the higher performance RAID system that I used above.

You can see how the transfer rate decays as it approaches the write capacity of the /home disk.

bbcp can recursively copy directories with the -r flag. Like rsync, it first has to build a file list to send to the receiver, but unlike rsync, it doesn’t tell you that it’s doing that, so unless you use the '-D (debug) flag, it looks like it has just hung. The time required to build the file list is of course proportional to the complexity of the recursive directory scan. It can also do incremental copies like rsync with the -a -k flags.

Note that bbcp is very slow at copying deep directory trees of small files. If you need to copy such trees, you should first tar up the trees and use bbcp to copy the tarball. Such an approach will increase the transfer speed enormously. The most recent version of bbcp can use the -N named pipes option to use external programs or pipes to feed the network stream.

bbftp is a modification of the FTP protocol that enables you to open multiple simultaneous TCP streams to transfer data. It therefore allows you to sometimes bypass per-TCP restrictions that result from badly configured intervening machines.

In order to use it, you 'll need a bbftp client and server. Most places that recieve large amounts of data (SDSC, NCAR, other supercomputer centers, teragrid nodes) will already have a bbftp server running, but you can also compile and run the server yourself.

The more usual case is to run only the client. It builds very easily on Linux with just the typical curl/untar, cd, ./configure, make, make install dance:

Using bbftp is more complicated than the usual ftp client because it has its own syntax:

To send data to a server:

The data was sent at 21MB/s to SDSC thru 10 parallel TCP streams (but well below the peak bandwidth of about 90MB/s on a Gb network)

To get data from a server:

I was able to get the data at 45MB/s, about half of the theoretical maximum.

As a comparison, because the remote reciever is running an old (2.4) kernel which does not handle dynamic TCP window scaling, scp is only able to manage 2.2MB/s to this server:

Fast Data Transfer is an application for moving data quickly writ in Java so it can theoretically run on any platform. The performance results on the web page are very impressive, but in local tests, it was slower than bbcp and the startup time for Java (as well as its failure to work in scp mode (couldn’t find the fdt.jar, even tho it was in the CLASSPATH, required you to explicitly start the receiving FDT server (not hard - see below, but another step)) argue somewhat against it.

Starting the server is easy; it starts by default in server mode:

The client uses the same jarfile but a different syntax:

The speed is certainly impressive. Much more than scp:

But fdt is slower than bbcp. The following test was done at about the same time between the same hosts:

If you and your colleagues have to transfer data in the range of multiple GBs and you have to do it regularly, it’s probably worth setting up a GridFTP site. Because it allows multipoint, multi-stream TCP connections, it can transfer data at mulitple GB/s. However, it’s beyond the scope of this simple doc to describe its setup and use, so if this sounds useful, bother your local network guru/sysadmin.

Globus Online is grid technology that has been wrapped in a web interface to enable mere humans to use the capailities of the GRIDFTP system to transfer very large amounts of data very quickly between nodes that are part of this system. The tagline is Move files fast No IT required. The advantages are that when it’s set up, it works very well. The disadvatnages are that the No IT required part is at this point, fairly optimistic and that it will only work between registered sites (which you can add with the Globus Connect process, so it’s not very useful for ad hoc file transfers.

Snarky Point of Contention: The documentation overuses the word seamlessly which all computer users realizes is a contraction for seamlessly if nothing goes wrong and your setup is exactly like mine and monkeys fly out my butt. YMMV.

However, if you have a set of endpoints that frequently need to transfer large amounts of data, this approach would be very useful. Especially if you run a cluster or other multi-user system, there is an associated utility called Globus Connect Multi-User which will allow all users of registered endpoint to use the Globus transfer capabilities.

I presume it will be getting progressively easier to use as time passes.

netcat (aka nc) is installed by default on most Linux and MacOSX systems. It provides a way of opening TCP or UDP network connections between nodes, acting as an open pipe thru which you can send any data as fast as the connection will allow, imposing no additional protocol load on the transfer. Because of its widespread availability and it’s speed, it can be used to transmit data between 2 points relatively quickly, especially if the data doesn’t need to be encrypted or compressed (or if it already is).

However, to use netcat, you have to have login privs on both ends of the connection and you need to explicitly set up a listener that waits for a connection request on a specific port from the receiver. This is less convenient to do than simply initiating an scp or rsync connection from one end, but may be worth the effort if the size of the data transfer is very large. To monitor the transfer, you also have to use something like pv (pipeviewer); netcat itself is quite laconic.

How it works: On one end (the sending end, in this case), you need to set up a listening port:

This sends the honkin.big.file thru pv -pet which will display progress, ETA, and time taken. The command will hang, listening (-l) for a connection from the other end. The -q 1 option tells the sender to wait 1s after getting the EOF and then quit.

On the receiving end, you connect to the nc listener

(note: no -p to indicate port on the receiving side). The -b option to pv shows only bytes received.

Once the receive_host command is inititated, the transfer starts, as can be seen by the pv output on the sending side and the bytecount on the receiving side. When it finishes, both sides terminate the connection 1s after getting the EOF.

This arrangement is slightly arcane, but supports the unix tools philosophy which allows you to chain various small tools together to perform a task. While the above example shows the case for a single large file, it can also be modified only slightly to do recursive transfers, using tar, shown here recursively copying the local sge directory to the remote host.