= Storage and Backup by Harry Mangalam v1.0 - Dec 14th, 2015 :icons: // HD="/home/hjm/nacs/"; // fileroot="HPC_users_meeting"; // asciidoc -a icons -a toc2 -b html5 -a numbered ${fileroot}.txt; // cd $HD; scp ${fileroot}.html ${fileroot}.txt x4500_top.jpg \ // Supermicro-36bay-storage.png GS7K-Open-Bay-small.png \ // moo:~/public_html/hpc; == A Reminder *Here are our primary storage systems:* ----------------------------------------------------------- Mon Dec 14 11:19:05 [1.73 1.82 1.80] root@hpc-s:/root 1024 $ df -h /data /dfs1 /dfs2 /pub Filesystem Size Used Avail Use% Mounted on nas-7-7.local:/data 15T 4.7T 9.9T 33% /data fhgfs_dfs1 464T 364T 101T 79% /dfs1 fhgfs_dfs2 191T 72T 120T 38% /dfs2 nas-7-2.local:/pub 55T 47T 7.9T 86% /share/pub -------------------------------- total used disk space: 487T ----------------------------------------------------------- == What to back up? - everything? All 500TB? - only designated dirs? * ie each user gets to backup _TB from an 'include' file? - how often? * 1x/day? 1x/week? As often as possible? - most recent _ months? * All recent data? Only the 1st _TB from the recent file? - what kind of files? * ie: backing up ZOT files will take forever. * if there are >X000 files/dir, we skip? == Where to backup TO? - OITDC (IB is fastest network but puts backup data in same physical room as main storage) - OIT -> LabC (same Building, but different room) - CalIT2 (different bldg, but Server room is sketchy, no 10GE) - ICS (monitored, 10GE). == Cost We have 3 scenarios ranging from almost cost-of-raw-disk to significantly more expensive. === Cheapest image:x4500_top.jpg[Sun Thumper top view] Re-provision Sun Thumpers (see above) and a spare metadata server to hold up to about 90x6TB disks (about 450TB usable) in a Backup BeeGFS. About $30K for a 500TB system (disks, networking cards, small switch) * cheap, expandable, we know how things work. * uses older hardware, so there's more danger of hardware failure. * can build it as a LEGO brick NFS system or a BeeGFS // FS. * Pros LEGO - if a FS fails, it's the only one; FS are single arrays. - probably more reliable. NFS & ZFS are both well debugged, would be rare to have the software fail. - * Pros BeeGFS - much faster and the bigger it is, the faster it is. - more scalable; just add arrays for more space. - we feel pretty comfortable with the software but it's more complex than LEGO. === More Expensive image:Supermicro-36bay-storage.png[AHPC front & back views] Buy new 'white-box' 500TB system from AHPC. * separate NFS filesystems (each one is not expandable, tho we can add filesystems - the LEGO brick system above.) * 5yr warranty. * slower than a BeeGFS // FS. * cost will be about $45K. === Even More Expensive image:GS7K-Open-Bay-small.png[DDN GS7K] Buy a complete 'name-brand' storage system from DDN or Dell/EMC * 5 year warranty * we don't need to know how it works * considerably more expensive (about $120K for 500TB) * depending on optional software it can probably be as fast as BeeGFS. * much more $$ to expand (disks have DDN-specific firmware).