HPC Protected Storage Proposal

OIT acknowledges that research data is the fuel that drives the university. While recent increases in data growth have outstripped our ability to protect ALL the research data on the HPC cluster, we want to be able to protect that tranche that is deemed most valuable to the HPC users, especially code, reduced data, notes, documents, MD5 checksums, etc. To this end, we will be protecting (less loaded a term than backup) a user-selectable set of data with a minimum quota of 50GigaBytes (GB) for all users' HOME directories and up to 1TeraByte (TB) for those users whose data requirements are larger. Groups will be able to purchase large quotas from OIT to protect more data at a low cost, about 2x the price of raw disk.

This data protection is NOT like your TimeMachine or CrashPlan backups in that we will not protect some files (see below) that are too difficult to transport unless you repackage them. Files are not protected continuously, nor are they guaranteed to be protected at a particular time, nor can you save multiple versions of the files. You will be able to recover your own files in the event of an accidental deletion, but you cannot write to the Protected Storage yourself.

This service is meant to provide an economical, useful level of disaster recovery if a filesystem fails on HPC and to protect against accidental deletion.

The HPC Cluster hosts about 1PB of storage in various pools, chiefly in the 2 BeeGFS distributed filesystems (/dfs1, /dfs2) and the /pub filesystem. Users have been warned continuously and repeatedly that these are scratch filesystems and the data should be considered ephemeral - they could disappear at any time. This has happened - a storage server on the /dfs1 filesystem was disabled due to a disk controller malfunction (altho the data was recovered), a misconfigured script I wrote accidentally deleted a user’s data, and the /pub filesystem was accidentally deleted.

These fairly normal fragilities in the process of day-to-day sysadmin are becoming increasingly dangerous due to:

To address these issues, we’ve decided to provide secondary storage to protect at least some of the files stored on HPC. It should not be considered true Backup since due to the size of the system, we cannot afford to back up everything and we cannot guarantee to do it on a daily or even bi-daily basis. We describe the details below, but essentially, we will be protect a quota of storage for every user, then provide rented Protected Storage (ProtStor) for additional files on a $/GB basis.

Since we are very sensitive to price and we are proposing a secondary storage system, not a single primary, we can (and must) use generic storage hardware rather than first-tier products from vendors like EMC and NetApp. These generic but good-quality products can be combined with free softare into a very fast, fairly robust system described below.

Based on the approximate budget of $50K, we have decided to place the ProtStor in the ICS Data Center (ICSDC) to isolate it from the OIT Data Center (OITDC) in case of catastrophe and ICS has already agreed to this.

The transport from HPC to the ProtStor will be over the UCI LightPath 10Gb/s Science DMZ. There are already Lightpath endpoints in the OITDC and ICSDC, so no more building switches nor fiber have to be purchased/installed, except that fiber will have to be pulled internally at ICS to connect the building switch to the servers in the ICSDC.

The ProtStor will consist of a single BeeGFS filesystem of approximately 300-500TB, depending on final costs and requirements. The reason for using a single BeeGFS system instead of multiple stand-alone NFS mounts for the following reasons:

However, BeeGFS is more susceptible to hardware failure since a single controller or even a single array failure can essentially destroy the integrity of most of the data on the BeeGFS system. In a system where each storage server is independent, a similar failure can only affect the arrays on that system.

For example, if we start with ProtStor composed of 2 isolated systems of 3 arrays each: In the case of multiple ZFS volumes, if the controller went bad (as in the case with the infamous LSI Nitro), we would lose all 3 of those arrays, but the other isolated ZFS system would survive. In the less catastrophic case, if 3 disks simultaneously failed in a single array (but the controller continued to work normally), we would lose only that one array’s worth of data. (I have never heard of losing 3 disks simultaneously and only once seen a dual disk failure in my life.)

So while using individual ZFS volumes is more reliable, for us to functionally lose data in that scenario, we would have to lose BOTH the HPC filesystems AND the ProtStor filesystem simultaneously. The probability of that happening are remote (less than Donald Trump being elected).

We propose the optimal system will consist of the following components:

This ProtStor is not meant to be a complete backup; certainly not an archiving system, so rules have to be set to determine what gets backed up.

Here are some rules that we’ve discussed with some limits that I’ve calculated based on current HPC storage use.

Based on some quick tests of the system while it was under load (see Figure 2 below), we will see about 400-600MB/s writing to the ProtStor. This is considerably less than the max observed when I was transferring data from Gluster to BeeGFS since the Gluster nodes were writing directly, and the BeeGFS was otherwise idle. So the bandwidth that we can expect to see is almost 10x less than that max.

Due to that bottleneck, it will take about 4-6 days to initially populate the ProtStor, after which the incremental speed will be fairly fast. We probably could do daily backups if we needed to, but since we only RobinHood-scan the HPC filesystems once a week, a 1x week backup might be OK. We could also charge for more frequent backups, launching specific parsyncs at user/group dirs.

Garr is querying the RHDB to see what the file change churn is so we can get an idea of how fast the incremental time is.