1. A Reminder

Here are our primary storage systems:

    Mon Dec 14 11:19:05 [1.73 1.82 1.80]  root@hpc-s:/root
        1024 $ df -h /data /dfs1 /dfs2 /pub

        Filesystem            Size  Used Avail Use% Mounted on
        nas-7-7.local:/data    15T  4.7T  9.9T  33% /data
        fhgfs_dfs1            464T  364T  101T  79% /dfs1
        fhgfs_dfs2            191T   72T  120T  38% /dfs2
        nas-7-2.local:/pub     55T   47T  7.9T  86% /share/pub
    --------------------------------
    total used disk space:   487T

2. What to back up?

  • everything? All 500TB?

  • only designated dirs?

    • ie each user gets to backup _TB from an include file?

  • how often?

    • 1x/day? 1x/week? As often as possible?

  • most recent _ months?

    • All recent data? Only the 1st _TB from the recent file?

  • what kind of files?

    • ie: backing up ZOT files will take forever.

    • if there are >X000 files/dir, we skip?

3. Where to backup TO?

  • OITDC (IB is fastest network but puts backup data in same physical room as main storage)

  • OIT → LabC (same Building, but different room)

  • CalIT2 (different bldg, but Server room is sketchy, no 10GE)

  • ICS (monitored, 10GE).

4. Cost

We have 3 scenarios ranging from almost cost-of-raw-disk to significantly more expensive.

4.1. Cheapest

Sun Thumper top view

Re-provision Sun Thumpers (see above) and a spare metadata server to hold up to about 90x6TB disks (about 450TB usable) in a Backup BeeGFS. About $30K for a 500TB system (disks, networking cards, small switch)

  • cheap, expandable, we know how things work.

  • uses older hardware, so there’s more danger of hardware failure.

  • can build it as a LEGO brick NFS system or a BeeGFS // FS.

  • Pros LEGO

    • if a FS fails, it’s the only one; FS are single arrays.

    • probably more reliable. NFS & ZFS are both well debugged, would be rare to have the software fail. -

  • Pros BeeGFS

    • much faster and the bigger it is, the faster it is.

    • more scalable; just add arrays for more space.

    • we feel pretty comfortable with the software but it’s more complex than LEGO.

4.2. More Expensive

AHPC front & back views

Buy new white-box 500TB system from AHPC.

  • separate NFS filesystems (each one is not expandable, tho we can add filesystems - the LEGO brick system above.)

  • 5yr warranty.

  • slower than a BeeGFS // FS.

  • cost will be about $45K.

4.3. Even More Expensive

DDN GS7K

Buy a complete name-brand storage system from DDN or Dell/EMC

  • 5 year warranty

  • we don’t need to know how it works

  • considerably more expensive (about $120K for 500TB)

  • depending on optional software it can probably be as fast as BeeGFS.

  • much more $$ to expand (disks have DDN-specific firmware).