parsyncfp - a parallel rsync wrapper for large data transfers

[i] = integer number                      [s] = "quoted string"
[f] = floating point number               ( ) = the default if any

--NP|np [i] (sqrt(#CPUs)) ..............  number of rsync processes to start
       optimal NP depends on many vars.   Try the default and incr as needed
--altcache|ac (~/.parsyncfp) ..... alternative cache dir for placing it on a
                another FS or for running multiple parsyncfps simultaneously
--startdir|sd [s] (`pwd`)  ..................  the directory it starts at(*)
--maxbw [i] (unlimited) ...........  in KB/s max bandwidth to use (--bwlimit
       passthru to rsync).  maxbw is the total BW to be used, NOT per rsync.
--maxload|ml [f] (NP+2)  ..........  max system load - if loadavg > maxload,
                                            an rsync proc will sleep for 10s
--chunksize|cs [s] (10G) .... aggregate size of files allocated to one rsync
                      process.  Can specify in 'human' terms [100M, 50K, 1T]
                                                   as well as integer bytes.
--rsyncopts|ro [s]  ...  options passed to rsync as quoted string (CAREFUL!)
         this opt triggers a pause before executing to verify the command(+)
--interface|i [s]  ......  network interface to monitor (not use; see above)
                                              Only SENT bytes are displayed.
--checkperiod|cp [i] (3) ........ sets the period in seconds between updates
--verbose|v [0-3] (2) ....sets chattiness. 3=debug; 2=normal; 1=less; 0=none
                     This only affects verbosity post-start; warning & error
                                             messages will still be printed.
--dispose|d [s] (l) .... what to do with the cache files. (l)eave untouched,
                                          (c)ompress to a tarball, (d)elete.
--email [s]  .....................  email address to send completion message
--nowait  .............  for scripting, sleep for a few s instead of pausing
--version  .................................  dumps version string and exits
--help  .........................................................  this help

(*) you can use globs/regexes with --startdir, but only if you're at that
point in the dir tree. ie: if you're not in the dir where the globs can be
expanded, then the glob will fail.  However, explicit dirs can be set from
anywhere if given an existing startdir.

(+) the '--rsyncopts' string can pass any rsync option to all the rsyncs that
will be started.  This allows options like '-z' (compression) or '--exclude-from'
to filter out unwanted files. Use any 'delete' options carefully tho.

IMPORTANT: rsync '--delete' options will not work with '--rsyncopts' bc the
multiple parallel rsyncs that parsyncfp launches are independent and therefore
don't know about each other (and so cannot exchange info about what should
be deleted or not.  Use a final 'rsync --delete' to clean up the transfer
if that's your need.


If you see an error related to "sh: /usr/bin/ls: Argument list too long",
it usually means that fpart has generated a huge list of chunkfiles (10s
of 1000s) and 'ls' has trouble processing that many.  This is usually
due to pointing parsyncfp at a huge filesystem, with millions of files,
with a chunksize that's too small (resulting in the above-noted too many
chunkfiles). You can either increase the chunksize ('--chunksize=100G)
which will result in a smaller number of chunk files to process, or split
up the source dirs among multiple parsyncfps (which can be done using the
'--altcache' option above).