= BigData-ASAP Configs & Costs Harry Mangalam v1.4 - October 30th, 2013 :icons: // fileroot="/home/hjm/nacs/BigData-ASAP-configs+costs-2"; asciidoc -a icons -a toc2 -b html5 -a numbered ${fileroot}.txt; scp ${fileroot}.html ${fileroot}.txt ${fileroot}-SS*.png moo:~/public_html; == Introduction In the following, 'Primary' or '1°'; refers to the system in the OITCD; 'Secondary' or '2°' refers to the backup system in either ICS or CalIT2. .Range of Costs for 80TB systems with 'No', 'Half', or 'Full' Backup The 'Option' labels (x-axis) refers to the the Option letters below. The bars are colored by whether the configuration is one with a 'Full Backup' (Blue), No Backup (Green), or Half Backup (Red). The Y-axis shows the cost of each option in thousands of dollars. image:BigData-ASAP-configs+costs-2.png[Range of Configurations and Costs] .In Tabular form, with # of servers [width="100%",cols="^e,^,^,^,^,^,^e",options="header"] |======================================================================================== |Backup Option |1° Servers | 1° TB | 2° Servers | 2° TB | Cost | Spreadsheet | A - None | 2 | 80 | 0 | 0 | $50.5K |http://moo.nac.uci.edu/~hjm/BigData-ASAP-configs+costs-2-SS-OptA.png[Opt A] | B - Half | 2 | 80 | 1 | 40 | $108.2K |http://moo.nac.uci.edu/~hjm/BigData-ASAP-configs+costs-2-SS-OptB.png[Opt B] | C - Full | 2 | 80 | 2 | 80 | $129.9K |http://moo.nac.uci.edu/~hjm/BigData-ASAP-configs+costs-2-SS-OptC.png[Opt C] |======================================================================================== Listed in increasing cost. Any Backup (aka replication) of the primary storage cluster incurs the cost of adding Single Mode Optical Fiber to the remote site and 2 x 10GbE switches to allow the 2 storage clusters to talk to each other on a private high-speed network, as well as the duplicated storage hardware. == *Option A:* 80TB Primary Storage, no Secondary Storage; ($50.5K) - 2 storage nodes; w/ 80TB usable (2 bricks of 40TB each) ($29K) - 2X bandwidth of a single chassis. - both storage nodes partially populated with disks (can expand to 240TB by adding only disks) - 1 MetaData server ($6K) - 2 I/O nodes ($10K) - 5 GPFS server licenses and support ($5.5K) - FDR IB rack switch ($8K) == *Option B:* 80 Primary Storage, 40TB Secondary Storage; ($108.2K) - 3 storage nodes; w/ 80TB usable (1° site: 2x 40TB ea; 2° site: 1x 40TB ea) ($43.5K) - 2X bandwidth of a single chassis in 1° site. - storage nodes partially populated with disks (can expand to 240TB (1°) / 120TB (2°) by adding only disks) - 1 MetaData server ($6K) - 2 I/O nodes ($15K) - 6 GPFS server licenses and support ($6.6K) - 1 FDR IB rack switches ($8K) - 2 10GbE switches ($14K) - install Single Mode Optical Fiber between OITDC and ICSDC == *Option C:* 80 Primary Storage, 80TB Secondary Storage; ($129.9K) - 4 storage nodes; w/ 80TB usable (each site has 2 bricks of 40TB each) ($58K) - 2X bandwidth of a single chassis. - both storage nodes partially populated with disks (can expand to 240TB by adding only disks) - 2 MetaData servers ($12K) - 3 I/O nodes ($15K) - 9 GPFS server licenses and support ($9.9K) - 2 FDR IB rack switches ($16K) - 2 10GbE switches ($14K) - install Single Mode Optical Fiber between OITDC and ICSDC == My recommendation In terms of progression, the best way of doing this is to install a 'Primary' storage stack in the OITDC ('Option A' above), get the interfaces up and running to make sure it's popular. Once the system has been shown to be useful and popular 'and' there is a demand for recharge backup, 'then' offer the 'Secondary/backup' as an option. At that point there might be Departmental pressure to both provide it and to pay for it. However, if we do provide a simultaneous 1° and 2° and there does not seem to be the institutional support for the 2° geo-replicate, we can easily merge the hardware back into the 1° storage system. == Abbreviations and Explanations: *GPFS* = IBM's General Parallel File System + *FHGFS* = Fraunhofer File System + *MDS* = MetaData Server, required for Fhgfs + *IB* = Infiniband + // *LHIB* = Long haul Infiniband + *QDR* = Quad data rate IB (41Gbs) + *FDR* = Fourteen data rate IB (54Gbs) + *10GbE* = 10Gb/s ethernet + *SMF* = Single Mode optical Fiber + *MMF* = Multi-mode optical Fiber + *I/O* = Input / Output + === Generic Storage Servers A 'Storage Server' is a generic storage chassis with room for 36 disks each in this case. They can be incrementally populated in chunks of 12 disks each (a 'brick') to provide 40TB for each 12-disk chunk. Since they are generic, they can be substituted with other similar or larger storage chassis' as price fluctuates. 'We are not bound to any vendor or supplier.' === Distributed Filesystems Distributed filesystems can be increased in size by providing more storage bricks. Their aggregate bandwidth can be increased by adding more chassis'. ie adding another chassis allows I/O at 2X the max I/O to one, 3 chassis allows 3X the I/O of one chassis. Hence the reasoning to initially provide 2 underpopulated chassis' in 'Option A' instead of filling the 1st. === I/O nodes These are servers that are placed between the end clients (Macs and PC) and the main filesystem to provide I/O buffering and security isolation from clients. They export the main filesystem to the clients while maintaining some security to it. By manipulating the CIFS/Samba rules and NFS exports, we should be able to prevent end-users from wandering the filesystem beyond what they should.