Big Data, ASAP

With the increased number and ability of digital sensors and channels to acquire data, UCI needs a network infrastructure that is able to Acquire, Store, Analyze, and Present (ASAP) that data in a timely, secure, and easy-to-use manner.

Data is getting big. From ubiquitous smartphones sensors, to genomic sequencers, confocal microscopes, PET/CAT scanners, to national-scale image and data collectors such as satellite, census, and other government data sources, and even commercial social networking data (Google, Facebook, Twitter), research data is getting larger. With that increase in data size comes the requirement to be able to store it and process it in reasonable times. We already are changing the mechanics and protocols for high speed data within our Data Centers, but in order to support research widely, we will have to expand those mechanisms, both for storing data and moving it into and out of our Data Centers. This proposal addresses that need by directly proposing a large campus storage system, backed by the very high-speed transport needed for data replication and back-end movement of large data sets to the analysis centers and back again.

UCI is embarking on a project to develop a campus-wide storage system that will provide redundant, geographically dispersed storage system for researchers to use for the ASAP operations noted above. The system has 2 faces; one for general researcher access that will be based on the existing, fairly slow (100Mbs-1 Gbs) campus network, and another higher speed 10-40Gbs network to allow the same underlying storage system to allow compute clusters and other specialized machinery to transfer bulk data to/from it with maximum speed.

We consider 2 types of data; the first is large, raw, unprocessed, cheap-to-reproduce data; the second is analyzed, reduced data that is worth considerably more in terms of the amount of labor used to produce it. That is: cheap, big, raw data (¢data) and expensive, small, processed data ($data). The point of research is to transform the former into the latter, with useful information extracted, concentrated, and hopefully published in the process. Publishing in this case refers to peer-reviewed journals, making the raw data available as a supplement to the journal publication, and/or archiving it for longer periods for public access. The last reference is is especially relevant since it is a state directive to the university and a requirement of many funding agencies.

In order to supply researchers with storage to process the ¢data, we have to consider very cheap, fairly reliable storage as opposed to very expensive, very reliable storage (from EMC, NetApp, DDN), since the price differential between these 2 is easily 10X or more. Since it is not the primary point of this submission, we do not describe the storage system in detail here, but it is based on the Fraunhofer Distributed file system or a similar technology and will be used to provide:

While there are ways to avoid & improve some network data transfer, eventually bytes need to move, and the larger the pipe, the easier the transfer. We are now seeing unprecedented amounts of data flow in and out of UCI, especially genomic, satellite, and CERN/LHC data. In order to move that data to/from the primary analysis engines in our Data Center, we minimally need 10Gb ethernet, and preferably higher speed, lower latency mechanisms such as Long-Haul Infiniband (LHIB).

Data traversing UCI’s primary internet connection (CENIC) tends to be packaged in ways that do not require short latencies (ie they are in archives which can be streamed in large chunks). However, internal to UCI where the data tends to be unpacked into separate (often small) files, and analyzed by applications that use MPI communication, low latency is almost as important as high bandwidth in terms of moving data or files around. In this scenario, it would be better to use a technology based on LHIB instead of 10Gb ethernet.

Intra-campus, these are the bottlenecks with our present network:

Figure 1 shows a block layout of the storage and connectivity that the UCI Research/ASAP Network would provide. As noted in the legend at the bottom of the diagram, the existing 1Gb campus ethernet backbone is shown in yellow lines. The new Long Haul Infiniband (LHIB) network is green, and the 10Gb backbone tree in blue. The actual network topologies are not shown; the 10G network is a star rather than the bus shown, for example. It’s the connections and their speeds that are important here.

The very low latency LHIB network would provide the rsync’ed replication service between the Primary storage in the OIT Data Center and the Backup storage in the ICS Data Center as well as the transport from that storage to the HPC and Green Planet compute clusters in the Data Center. Faculty clients in remote buildings (using MacOSX, Windows, Linux) would connect to the ASAP storage via the normal protocols (CIFS/SMB, NFS) via specialized IO nodes that have multiple 1GbE connections, but could also connect via Linux native clients directly. The IO nodes could also be Web servers to serve out this data to collaborators via the WWW.

The storage icons overlaid by the blue shading represent multi-chassis distributed filesystems at each point, with the Primary storage rsync’ed to the Backup one or more times a day. In the event of the Primary failing, the Backup can be reconfigured as Primary.

The 10Gb transport would also extend a large-bandwidth pipe to the campus border router to the Internet via CENIC and to the remote Medical facilities in Orange, where substantial Imaging and Genomic data is generated.

A campus research storage system such as proposed above is different than an administrative or financial storage system. We have planned for cheap, fast, multi-protocol, fairly reliable, and replicated storage but it is not High Availability nor is it formally backed up. It will require scheduled downtimes of up to a day and may lose some data. The data loss scenario is described below and could be best described as unlikely, but not impossible. Over the course of its lifetime (easily a decade, since it is meant to be upgradable on the fly), it will almost certainly lose some data.

(to be completed)