Academic File Storage, Backup, and Sync Services

Most of the Cloud services mentioned here use network paths to/thru the commodity Internet with the result that streaming bandwidth to these services tops out at 1-2 MB/s, some considerably less. Streaming transfers thru Academic networks (Internet2, CENIC, CalREN, etc) can be considerably faster, especially with tuned and Grid-enabled applications (30-100MB/s, coast to coast), but few of the commercial resources have access to these networks unless requested by academic clients. When dealing with local-to-UCI machines, we can often get close to wire speeds for bandwith (10MB/s on 100MbE, 100MB/s on GbE. Similarly, the latency (the time for a signal to get from here to there) is much less for local services. Hence remote services will always seem very slow compared to local services. Caching proxies and other tricks can make this somewhat easier to live with, but this is a basic characteristic of networking, especially TCP-based networking.

This is made worse with multiple small file operations since each one requires some connection setup and teardown . For example, extracting the Linux kernel (626MB in ~60K files) from the distribution tarball takes about 11 sec on a local disk. The same operation via NFS to UCLA’s CASS storage resource takes about 77 min, a difference of about 400X. For networking speeds, local is better.

Further, many Local Area Network (LAN) protocols are blocked over Wide Area Networks (WANs). For example, the UCLA/CASS storage (below) can supply UCLA’s campus with SMB/CIFS file storage as the protocol is allowed inside the campus network, but is blocked external to their network. Hence UCI’s Mac and Windows clients cannot use the UCLA/CASS storage as a fileserver. UCI would have to run a local resource to provide that service.

Raw disk costs continue their long-term characteristic of halving about every 14 mo.

This makes comparison of commercial services costs difficult except as snapshots, or over time to get normalized trends. However it may be worthwhile to describe the current costs of raw and usable storage on a local basis. Due to the offerings of the commercial services in chunks of 100GB, I’ll normalize the costs of raw storage to 100GB as well at the time of this writing (04/2015)

This file storage is most often associated with LAN-based file servers where a Windows, Mac, or Linux client remote-mounts a SMB/CIFS or NFS filesystem and interacts with it as if it was a local filesystem. Current cloud services such as Google Drive or DropBox emulate this functionality when using their native clients with the added ability to sync and/or share those files as well. This kind of interaction is particularly sensitive to network latency when the type of work involves moving large chunks of data such as nonlinear editing of multimedia files, slicing and dicing of satellite data, and querying relational databases.

It should be noted that these cloud services are almost always object file systems which only support the http put, get, and head commands. As such, when you ask for a file, you get the whole file and you have to write the whole file back. - no partial reads, no appends, no striding to offsets, etc. POSIX filesystems allow for much finer-grain file manipulation and many applications, especially in the research domain require such file access. Most native clients for such cloud services will read the whole file, cache it locally on the client’s POSIX FS, allow it to be manipulated with POSIX semantics, and then when finished, write it back to the cloud in toto, sometimes with some server side magic that emulates an rsync operation.

Object FSs are often used as buckets to dump large chunks of infrequently accessed data into because they are represented as simpler, cheaper, and more robust than POSIX FSs. That can be true, but there is a base cost of storage and often the cheap gets overwhelmed by the cost of the rest of the system.

This functionality is chiefly for collaboration where one party wants to share files with others. DropBox is a front-runner here, chiefly for the ease of its interface, but UCI has dabbled with this kind of service by providing the commercial Webfiles which provides a limited amount of storage via a creaky, cranky interface. Users can share files via the native application in most cases or via the web via URLs to the files (like DropBox, Google Drive, Box). Problems with both commercial and local implementations involve limitations on space and opaque interfaces so that it is difficult to intuit which users or groups can do what with files.

This area is somewhat fuzzy - it usually means that changes in one file are propagated to all files, with only the bits that have changed moving across the network. However, it could mean that the changes are realtime (as would be critical when co-editing a document) or delayed until a timed service activates (as is the case when a backup job starts). Different services interpret this to mean different things. Bittorent/Resilio Sync, DropBox, and others support variants of this approach, tho there’s considerable overlap with the apps in the next section.

This area covers short term backup such as incremental backups or snapshots (like Apple’s Time Machine) and personal disaster recovery. This often includes syncing files across devices and easy client-side-initiated recovery. Examples of this kind of service are the Open Source Software (OSS) packages BackupPC and Zmanda, and the commercial services CrashPlan and BackBlaze.

This addresses long term file archiving. Files go in, stay in until required for disaster recovery and include services like Amazon Glacier, BackBlaze, and CrashPlan. Often, the cost for these types of plans are fairly low but with the expectation (often codified in the Service Level Agreement (SLA)) that there will be little file retrieval without incurring additional costs. Especially in the case of Amazon Glacier, there are few client-side applications available to interact with the service directly, partly because it has no external interfaces - you currently have to go thru Amazon’s S3 storage system to access Glacier, so it is essentially an API that you have to provide code to support. There are some applications that provide Graphical User Interfaces to it, typically as part of S3-specific File Transfer Managers like CyberDuck, Dragondisk, and the like. It is unlikely that any self-hosted system could match the cost of such commercial long term archival services.

Wikipedia has a good Comparison of Cloud Backup Services in terms of features. Note that Glacier is not mentioned since it is an API, not a client-side service currently.

There is considerable overlap among these services - for example, Google Drive allows syncing, and file sharing as well as file storage and co-editing of documents, as does Microsoft’s SharePoint, which aims to be the file-everything for all things Microsoft. Druva is another that claims to be all-devices Enterprise-ready for sharing, syncing, & backup.

and how some of the services fit into it. Small text in Blue indicates how users might have performed such functions in the past. Large Blue text in the Venn diagram indicates Open Source packages or services that can be run locally. (LibreOffice diagram file here)

The most popular local sharing service is called OwnCloud and it is available as source code, as a Bitnami installation package, and as an Amazon service. It has Windows, Mac, and Linux desktop clients, as well as iOS and Android clients. It uses Web Distributed Authoring and Versioning (aka WebDAV)for most of the file transport protocol, allowing connectionless access to files. This is a fairly low performance protocol, but since the service operates at near-LAN speeds, it can still be quite useful. Both network traffic and storage is encrypted. The file-level encryption is useful if you then share files outside of OwnCLoud - the files stay encrypted until the reader provides the decryption key.

It also allows you to add storage from various cloud services (Dropbox, SWIFT, FTPs, Google Docs, S3), tho that seems a strange attraction for a service whose main feature is that it’s locally based).

OwnCloud is old enough that it has developed apps that make it possible to co-author (in RichText) documents with up to 5 authors, much like Google Docs.

A recent trade rag article (free reg req) and a Linux-oriented podcast that spends a good amount of time on it (starts ~1m:36s into the podcast).

Pydio is very similar to OwnCloud with an arguably better user interface and reportedly better performance on large numbers of files. However, it lags OwnCloud in development time and therefore installations, and therefore up to date reviews. We will be testing it as well as the ongoing OwnCloud.

UCLA has developed a large campus storage system that provides local interactive file storage via NFS and SMB/CIFS as well as a Globus endpoint and a CrashPlan server (see below) that requires extra cost licenses to use it. They require a minimum purchase of 1TB (large relative to other services) for $171/yr, which is cheap for the space ($17/100GB/yr), but it’s a lot of space. UCLA is networkologically close to UCI and while we can attain very fast streaming transfer (up to ~90MB/s via Globus Connect, ~55MB/s via NFS). However, while Globus can be used by all clients, NFS is generally used only by Linux/Unix clients. The preferred connection protocol for Mac and Win clients (SMB/CIFS) is blocked across most Wide Area Networks and so UCI clients can not currently use UCLA’s storage for interactively used files. Additionally, while streaming large files works well, manipulation of large numbers of small files is very slow due to the file and IO overhead.

As noted above, Glacier is not currently a client-facing service; it was developed as the backup system for Amazon’s S3 storage system and therefore is usually accessed programmatically internal to the EC2 system. More recently, external developers have provided Hierarchical Management Software that can push data directly to Glacier.

Glacier is designed with the expectation that retrievals are infrequent, that data will go into it and stay there. You can retrieve up to 5% of your average monthly storage for free each month. More on the fee structure here.

The cost of Glacier storage is almost infinitesimal for individual docs - for a 100GB home dir, only a $1. However, altho you get free transfer to AWS in same region, retrieval costs are pro-rated and complex and while they can be reduced by spreading them over time, retrieval will generally be on the order of $10s, not $100s. However, when compared to the loss of 100GB of personal data, this is dirt cheap, if partially offset by the difficulty and time to retrieve it.

Backblaze has an interesting business model - they build their own infrastructure and are remarkably open about how they carry out their business. They also bill on a statistical model which allows unlimited backup for $5/mo (currently) and while they only support Macs and Windows, their client is very easy to configure and use, with much clearer documentation than CrashPlan. The client is very snappy and does not give misleading estimates of time to complete, just what’s been done. BackBlaze also allows you to back up local USB disks and (for extra $) will also ship your files back to you on a disk .

The CrashPlan combined client/server is writ in Java and runs smoothly on all platforms. It’s easy to install, tho impossible to select the free for 30 day trial; like many such free trials you have to have to 1st buy the month-by-month plan.

The GUI is somewhat confusing since when you select Backup, doesn’t execute directly but waits for idle time, configurable via settings but not clearly explained. The backup time to the CrashPlan cloud, was ~3hr for 14K files (9.1GB). CrashPlan tries to estimate how long a backup will take, but since it depends on many variables, it’s almost a random guess.

A notable point about the CrashPlan system is that the free client is also a server. You can install the same software to your own server and back up multiple clients to that server simply by providing the invitation key, using the same interface that the commercial service provides. This allows you to use a local server, speeding up backups by a factor of 10-20 fold. This in itself is a very useful feature for a small to medium sized research group.

Mozy (now owned by EMC), provides simple free, tiny account, hoping that you upgrade to a more expensive one. A basic personal plan costs $100/100GB/yr). It has a pretty good interface (Mac, Win, iOS, Android) for de/selecting files and backup in progress. Like a few others, Mozy lets you use their interface to back up to local USB disks with the same interface.

DropBox puts the emphasis on Sharing rather than backups. However it is possibly the best known, widest multi-platform interface and is in widespread use at UCI and elsewhere due to both its business plan and the ease of use and integration of its clients. It also has gone further than most in integrating with other large cloud platforms such as FaceBook and Twitter.

SugarSync is also very easy to use, with the emphasis on syncing rather than backups. It has no restrictions on directory structure (not necessary to put all the files you want to share in one dir) which helps to differentiate it from most other services. It has no maximum file size which is good for sharing large data files, but is fairly expensive ($100/yr/100GB/person). No Linux support.

Google Drive provides Windows and Mac clients, but only limited Linux utility via 2nd sources. The native clients integrate fairly well with the native filesystem and allow you to use Drive as an additional storage if the files are small. If using gmail, you can search across mail and uploaded files. Also available for chromebook access. Has some included apps including spreadsheets, WP, etc. ($24/100GB/yr)

The Linux clients are commandline only: grive (a commandline Github project last updated in 2013), and drive. The actual bandwidth to Google is extremely Internet-dependent. On the day I was using it, it could move data at ~160-500KB/s ; downloads were much faster, but only up to about 1-2MB/s.

It’s possible to attach Google Drive to large Linux systems as described [here], but bandwidth is still quite slow, about 3-5MB/s for large files.

However, note that via a fast network, such as UCI’s 10/40 Gbps Lightpath DMZ, it’s possible to send data to Google Drive at close to wire speeds with rclone. See how here.

Druva is much more business-oriented than most such services, with emphasis on encryption, multiple users, reliability, support for business-oriented devices, and speed. It uses in-line deduplication to speed backups and reduce their size. These features would have limited advantages for research data.

Amazon Cloud Drive is the consumer version of their S3 Cloud storage with various add-ons like auto-ripping CDs that you’ve bought from them (~$50/100GB/yr). And Amazon Zocalo is the business version ($30/100GB/yr).

iCloud is designed primarily for buyers of Apple’s iDevices, so their offerings are less price-sensitive than others and emphasize integration with Apple iProducts and social media sharing. Their current pricing is \~$60/100GB/yr, almost 3X Google’s cost.

Box is more business oriented, with plans including multiple user access ($60/100GB/yrw/ for 3-10 users, with 2GB maximum file size). Regardless of plan, it has a 5GB maximum file size.

Copy.com provides MacWinLin clients, sharing among multiple users, with the cost starting at a normalized ~$40/100GB/yr

An overview of some Cloud Backup Services