How to transfer large amounts of data via network.

The elegant, Qt-only, open source qdirstat (latest iteration of the original, beautiful, but dependency-ridden kdirstat) and its ports to MacOSX Disk Inventory X and Windows Windirstat) are quick ways to visualize what’s taking up space on your disk so you can either exclude the unwanted data that needs to be copied or delete it to make more space. All of these are fully native GUI applications that show disk space utilization by file type and directory structure.

Unlike qdirstat above which requires graphics mode, gt5 (Linux only, altho Win10 now supports Ubuntu Linux utils) is a very slick, simple, fast terminal app which allows you to recursively identify large dirs and cursor your way thru them.

ping times are a measure of the roundtrip time RTT it takes for a packet to reach a destination and have an acknowledgement return. For example, I’m writing this on a laptop in Irvine CA. The ping time to my home wireless router is about 1ms. Because my ISP is Cox, a ping to a computer down the hill at UCI (moo) is more than 10x that at about 15ms, since according to traceroute, that ping has to travel thru 15 devices to LA and back.

And a ping across the country to Nova Scotia (7200 roundtrip miles by road) takes more than 100x as long, about 108ms, across at least 20 devices. Since that time would allow an unimpeded photon to travel about 16,000 miles, the rest of the delay is due to cable and device delays, which effectively slow the communications to about 40% of the speed of light.

The RTT above is important because the TCP protocol works by verifying the arrival of each packet, which requires a network round trip for each packet sent. When the ping times increase, obviously the number of packets that can be verified per time period decreases so TCP works slower over greater distances. This is bad. Not to belabor the point, but if you send a serial stream of TCP packets (FTP, rsync or almost any of the protocols mentioned in this doc), the rate at which you can send them, receive verification, and send another decreases as the ping time increases. This constraint is known as the Bandwidth-delay product and is a major component of why that shiny new 100Gb network switch yields such lousy performance over long distances.

There are a few ways to bypass or improve on this problem. You can compress your data before sending it, in effect sending more data in the same packet. You can try to increasing the packet size known as the (the Maximum Transmission Unit (MTU)). The problem with the latter is that many commodity Internets set MTUs fairly low (1500 bytes), altho high speed devices or academic networks often allow the use of jumbo frames which are MTUs with a payload of up to 9000 bytes. LAN MTUs can be as high as 64KB, allowing much more efficient transfer.

You can also skip the TCP protocol entirely and use the UDP protocol. This is a much less reliable mechanism for data transmission since it does not verify packet delivery or order. However, the underlying hardware for modern networks have gotten so reliable that UDP is again gaining use by wrapping a unreliable protocol with sideband TCP integrity checking so that very large (64K and larger) packets can be sent very quickly. GridFTP and its Globus variants Connect/MultiConnect , Aspera, Signiant, RBUDP, Tsunami, and other data transfer mechanisms use (or can use) this approach. However none of those are very easy to set up and use on an ad hoc basis (and Aspera & Signiant are not free). As a side note, Google has released its QUIC (Quick UDP Internet Connections) code for testing here, altho it currently seems alpha stage.

The last way of addressing the inherent limitation of long distance data transfer is to send multiple streams of TCP packets simultaneously. This parallel transfer of data is increasingly being used by a variety of applications or wrappers of existing applications. Google’s parallel composite transfer using gsutil, the much easier rclone (see also Using rclone to push data to your Google Drive).

NB: gsutil and rclone operate only with relatively blobby object filesystems on both ends; the rsync family works with POSIX filesystems and can thus do real syncing operations. Here’s a more extensive description of the differences.

bbcp and parsyncfp, and fpsync (part of the fpart pkg) all use this mechanism. The best one depends on your network, your endpoints, and what control you have over those endpoints. rclone was designed to transfer data to the Amazon, Google, and other clouds and supports many of those authentication protocols; parsyncfp and fpsync are essentially data-balancing parallel rsyncs, which assume a shell account on both ends of the network and the ability to set up ssh keys. bbcp can adjust number of TCP streams and packet window sizes to increase bandwidth considerably.

Note that both rclone and gsutil (and its parent toolkit boto) can handle cloud authentication protocols, whereas both rsync/parsyncfp and bbcp use ssh to authenticate connections. Also, bbcp does not compress or encrypt its data stream(s) unless requested via flag or a pipe to an external program.

There are a few parallel wrappers for rsync which can trmendously increase the speed at which large, deep directory trees are transferred, especially over WANs. I’ll describe them separately below.

Command to rsync data from UCI’s HPC cluster to a remote backup server.

Where we will transfer the dir tacg-4.6.0-src to user happy’s account on the server circus.tent.uci.edu in the dir ~/HPC-backups. In the example below, we have to enter a password. In the 2nd example, we’ve set up passwordless ssh.

bbcp seems to be a very similar utility to bbftp below, with the exception that it does not require a remote server running. In this behavior, it’s much more like scp in that data transfer requires only user-executable copies (preferably the same version) on both sides of the connection. Short of access to a GridFTP site, bbcp appears to be the fastest, most convenient single-node method for transferring data.

The author, Andrew Hanushevsky has made a number of precompiled binaries available as well as access to the bbcp git tree: git clone http://www.slac.stanford.edu/~abh/bbcp/bbcp.git Somebody at Caltech has written up a very nice bbcp HOWTO.

The code compiled & installed easily with one manual intervention

bbcp can act very much like scp for simple usage:

The file transferred in under 10s for a 633MB file, giving >63MB/s on a Gb net. Note that this is over our very fast internal campus backbone. That’s pretty good, but the transfer rate is sensitive to a number of things and can be tuned considerably. If you look at all the bbcp options, it’s obvious that bbcp was written to handle lots of exceptions.

If you increase the number of streams (-s) from the default 4 (as above), you can squeeze a bit more bandwidth from it as well:

or almost 85MB/s for 4.2GB which is very good sustained transfer.

Even traversing the CENIC net from UCI to SDSC is fairly good:

or almost 30MB/s.

When making the above test, I noticed the disks to and from which the data was being written can have a large effect on the transfer rate. If the data is not (or cannot be) cached in RAM, the transfer will eventually require the data to be read from or written to the disk. Depending on the storage system, this may slow the eventual transfer if the disk I/O cannot keep up with the the network. On the systems that I used in the example above, I saw this effect when I transferred the data to the /home partition (on a slow IDE disk - see below) rather than the higher performance RAID system that I used above.

You can see how the transfer rate decays as it approaches the write capacity of the /home disk.

bbcp can recursively copy directories with the -r flag. Like rsync, it first has to build a file list to send to the receiver, but unlike rsync, it doesn’t tell you that it’s doing that, so unless you use the -D (debug) flag, it looks like it has just hung. The time required to build the file list is of course proportional to the complexity of the recursive directory scan. It can also do incremental copies like rsync with the -a -k flags, which also allow it to recover from failed transfers.

Note that bbcp is very slow at copying deep directory trees of small files. If you need to copy such trees, you should first tar up the trees and use bbcp to copy the tarball. Such an approach will increase the transfer speed enormously.

The most recent version of bbcp can use the -N named pipes option to use external programs or pipes to feed the network stream. This allows you to specify an external program such as tar to provide the data stream for bbcp. Like this:

The above command uses bbcp’s named pipe option for both input and output (-N io) to take tar’s output from STDOUT (tar’s -O option), and using the above-described options to stream the tar’s output to bbcp to the remotehost where tar is invoked to decompose the bytestream and write it to the new location (-C /nffs/w2)

NB: the original bbcp help page on this option has (as of May09,2013) a typo or 2. The above example is correct and works.

NB: I have occasionally seen this error when using bbcp:

If you get this error, add the "-z" option to your command line (right after bbcp). ie"

bbftp is a modification of the FTP protocol that enables you to open multiple simultaneous TCP streams to transfer data. It therefore allows you to sometimes bypass per-TCP restrictions that result from badly configured intervening machines.

In order to use it, you 'll need a bbftp client and server. Most places that receive large amounts of data (SDSC, NCAR, other supercomputer centers, Teragrid nodes) will already have a bbftp server running, but you can also compile and run the server yourself.

The more usual case is to run only the client. It builds very easily on Linux with just the typical curl/untar, cd, ./configure, make, make install dance:

Using bbftp is more complicated than the usual ftp client because it has its own syntax:

To send data to a server:

The data was sent at 21MB/s to SDSC thru 10 parallel TCP streams (but well below the peak bandwidth of about 120MB/s on a Gb network)

To get data from a server:

I was able to get the data at 45MB/s, about half of the theoretical maximum.

As a comparison, because the remote reciever is running an old (2.4) kernel which does not handle dynamic TCP window scaling, scp is only able to manage 2.2MB/s to this server:

lftp is a simple but capable FTP replacement that can use multiple TCP streams like bbcp, resulting in better performance than vanilla FTP or other single stream mechanisms like scp. One restriction is that the multi-stream approach only works in get mode, so if you’re trying to upload data (put mode), it works only as well as a single stream approach. It will also do mirroring so if you’re trying to mirror an entire website or file tree, it can do that, much like the wget -m -p <website_head>.

In my testing over a 1Gb connection, lftp was about 5%-10% slower than bbcp on getting data (same number of streams with cache cleared each time) and noticeably slower on sending data. Both bbcp and lftp appear to be transferring to local cache and on transferring files smaller than the free RAM, will spend several seconds after the transfer is supposedly complete in syncing the data to disk.

Fast Data Transfer is an application for moving data quickly writ in Java so it can theoretically run on any platform. The performance results on the web page are very impressive, but in local tests, it was slower than bbcp and the startup time for Java (as well as its failure to work in scp mode (couldn’t find the fdt.jar, even tho it was in the CLASSPATH, required you to explicitly start the receiving FDT server (not hard - see below, but another step)) argue somewhat against it.

Starting the server is easy; it starts by default in server mode:

The client uses the same jarfile but a different syntax:

The speed is certainly impressive. Much more than scp:

But fdt is slower than bbcp. The following test was done at about the same time between the same hosts:

These are fairly new (mid-2011) approaches that claim to provide easy access to GridFTP-like speeds, reliable transfers, and No IT required, using the Globus Tookit infrastructure, which is an enormous and enormously complex set of APIs for authenticating users and distributing data aorund the world. Globus Connect and its more ambitious Globus Connect MultiUser sibling are attempts to make using the Globus mechanicals less horrific for users. In this it largely succeeds from the users' POV - those who are already part of a Globus/Grid node and who have specific requirements to transfer TBs of data on a regular basis and who have the endpoints set up for them. Otherwise it’s somewhat clunky since you have to explicitly set up endpoints beforehand and too complicated to set up unless you’re Linux-enhanced (ie. you do ssh public key exchange, and globus MyProxy configs in your sleep).

The latest iteration of this technology is a web interface that once set up allows you to initiate and monitor large data transfers between defined endpoints fairly easily. The process to install the software to your own system and add yourself to the system is fairly straightforward. Just follow the instructions for the different platforms.

The problem with this approach is that it’s a large amount of work for a small amount of advantage relative to bbcp. However, the Multiuser version allows all the users of a server or cluster to take advantage of this protocol with no additional effort, a better tradeoff between effort expended and advantages conferred.

The instructions for installing the Multiuser version are a little more elaborate. Herewith, their own devilish details for a sysadmin setting up the Globus Connect MultiUser (Linux-only so far).

The process for setting it up on your endpoint is described on the site, but it may be worthwhile describing the general overview which can be confusing. UCLA’s IDRE also has a setup description (Thanks, Prakashan.)

Snarky Point of Contention: The documentation overuses the word seamlessly which all computer users realize is a contraction for seamlessly if nothing goes wrong and your setup is exactly like mine and monkeys fly out my butt. YMMV.

Using the Globus Connect system requires you to:

I’ve gotten 40-50MB/s between UCI and the Broad Institute depending on time of day, system load, and phase of moon.

If you and your colleagues have to transfer data in the range of multiple GBs and you have to do it regularly, it’s probably worth setting up a GridFTP site. GridFTP is also based on the Globus toolkit and as such shares many of its advantages and frustrations. However, most of the frustrations are on the admin side, so once it’s set up, it becomes fairly easy for users. Because it allows multipoint, multi-stream TCP connections, it can transfer data at multiple GB/s. However, it’s beyond the scope of this simple doc to describe its setup and use, so if this sounds useful, bother your local network guru/sysadmin.

netcat (aka nc) is installed by default on most Linux and MacOSX systems. It provides a way of opening TCP or UDP network connections between nodes, acting as an open pipe thru which you can send any data as fast as the connection will allow, imposing no additional protocol load on the transfer. Because of its widespread availability and its speed, it can be used to transmit data between 2 points relatively quickly, especially if the data doesn’t need to be encrypted or compressed (or if it already is).

However, to use netcat, you have to have login privs on both ends of the connection and you need to explicitly set up a listener that waits for a connection request on a specific port from the receiver. This is less convenient to do than simply initiating an scp or rsync connection from one end, but may be worth the effort if the size of the data transfer is very large. To monitor the transfer, you also have to use something like pv (pipeviewer); netcat itself is quite laconic.

How it works: On one end (the sending end, in this case), you need to set up a listening port:

This sends the honkin.big.file thru pv -pet which will display progress, ETA, and time taken. The command will hang, listening (-l) for a connection from the other end. The -q 1 option tells the sender to wait 1s after getting the EOF and then quit.

On the receiving end, you connect to the nc listener

(note: no -p to indicate port on the receiving side). The -b option to pv shows only bytes received.

Once the receive_host command is initiated, the transfer starts, as can be seen by the pv output on the sending side and the bytecount on the receiving side. When it finishes, both sides terminate the connection 1s after getting the EOF.

This arrangement is slightly arcane, but supports the unix tools philosophy which allows you to chain various small tools together to perform a task. While the above example shows the case for a single large file, it can also be modified only slightly to do recursive transfers, using tar, shown here recursively copying the local sge directory to the remote host.

Updated Jan 06, 2015 After a licensing problem with the existing Broad Institute Globus system stopped us from using the browser-based Globus system to finish the transfer, we still had to transfer about 30TB of data from the Broad Inst to UCI.

We tried using the above mentioned bbcp from there to here, but since I controlled only one side of the connection, I couldn’t tune the transfer to provide more than about 1-3MB/s. The Broad people mentioned that the data was available via Aspera, and I grouchily agreed to try it again (see first review below) via the Linux commandline client ascp that had proved so trying previously.

This time, tho (I had full docs for the client and some experience) it started up and immediately provided a consistent 30-40MB/s over the same network path that I was using with bbcp. A tremendous, startling improvement over the default bbcp parameters, especially since it was the same network path over which bbcp was providing about 1/10th the bandwidth. Whether it’s due to autotuning of window size and streams, or some other internal magic, I can’t say. And I also can’t say whether (with control of both endpoints) I could have coerced bbcp to attain a similar bandwidth since I have seen bbcp do longhaul bandwidths of this magnitude. But seeing ascp immediately provide this magnitude of bandwidth, with no user-side tuning is very impressive indeed.

I’ll leave the previous notation in place to show how much ascp has improved.

first post, 2013 Aspera is a commercial company (recently bought by IBM) whose aim is to monetize large scale data transfer across networks. I have no experience with their Windows and Mac clients which may be very good, but their default Linux client, starting about 3years ago is not. Or to be more specific: It can work well, but it may well require a lot of tweaking and adjustment to work well. God knows, I had to. However, I was eventually able to transfer 15TB across the UC with the Linux client, which after the aforementioned tweaking worked ok.

Compared to the above-mentioned bbcp, or the consumer-skinned and smoothed Globus Online, the Aspera Linux client is still crude, difficult to use, poorly documented, and fails repeatedly. I posted some comments to the blog linked above, but here are my suggestions about its use when you can’t use an alternative approach.

When I say that the Linux client is poorly documented, it doesn’t mean that there is no documentation. It means that it is hard to find (no helpful links returned by the search service on their support page), the Documentation (click ascp Usage) is no better than most free software, and that the Examples (click ascp General Examples) are fairly sparse. Additionally, the customer support databases are behind an firewall and are therefore beyond the reach of google. This is certainly Aspera’s right, but it means self-help is essentially impossible.

I will say that opening a support ticket brought rapid (<1 hour from filing the ticket to a human response), and knowledgeable assistance. (Thanks, Bill!). ascp will fill your syslog with a ton of event logs which, if sent to Aspera, will probably allow them to debug the problem.

A non-Aspera employee advised me to use ascp’s parallel copy to speed up the transfer but that was apparently a mistake since that option (tho it appears to work) is usually only useful when the client node is CPU-bound (mine wasn’t) and neither increases overall copy speeds, nor allows you to restart failed transfers (and one of the parallel copies would fail every few minutes, possibly due to the timing issue noted below.) Since I was not able to get the parallel approach to work reliably for more than 30 min at a time, the best approach was to start a single serial process with the Linux ascp client after carefully reading the above blog post and correcting the command as per your needs.

Also, ascp can be very sensitive to timing issues so a modification may have to be made to the configuration file:

So, once that issue was settled, and I stopped trying to parallel copy, the following command worked reliably, maintaining the copy for at least a couple of days until the transfer finished. The transfer was not magically faster than what I would have seen via bbcp, and was considerably slower than a GridFTP transfer, but it did work.

where:

Again, read the docs carefully since the error messages are unhelpful.