= Security Standards on HPC Harry Mangalam v1.00, July 9, 2018 :icons: // fileroot="/home/hjm/nacs/hpc/HPC_dbGaPSecurity"; asciidoc -a icons -a toc2 -a toclevels=3 -b html5 -a numbered ${fileroot}.txt; scp ${fileroot}.html ${fileroot}.txt moo:~/public_html == Introduction Increasing interest in analyzing annotated genomes on the HPC Cluster and elsewhere on campus has caused RCIC to increase the security protocols on HPC and while some remain inferior to the dbGaP requirements, we will be tightening the remaining problem areas so that they will be congruent wth the dbGaP requirements. == Restricted Data on HPC === dbGaP "The Database of Genotypes and Phenotypes (dbGaP) was developed to archive and distribute the data and results from studies that have investigated the interaction of genotype and phenotype in Humans." By definition therefore, it contains human subject data that, while somewhat anonymized, is still considered sensitive and therefore requires additional electronic safeguards for use. These requirements are described in the https://osp.od.nih.gov/wp-content/uploads/NIH_Best_Practices_for_Controlled-Access_Data_Subject_to_the_NIH_GDS_Policy.pdf[NIH Security Best Practices for Controlled-Access Data Subject to the NIH Genomic Data Sharing (GDS) Policy] paper. The requirements listed therein have been re-worked into a http://moo.nac.uci.edu/~hjm/Denune_Security_Summary[summary mapping paper] by John Denune of the UCI Security Team. Many of the proposed security enhancements in it are described in more detail below. (NB: John has to to provide the actual doc). === TCGA https://cancergenome.nih.gov/[The Cancer Genome Atlas] is a data collection similar to dbGap in terms of scope and complexity, but focussed on cancer specifically. As such, it also contains much human data, also similarly anonymized, but also unavoidably contains personally identifiable traits. Since it is also an NIH project, the release and analysis of that data is covered by the same security restrictions as dbGaP (see above). == Other Restricted Data The following types of restricted data are included for reference, but we 'explicitly' and *emphatically* do not host such data on HPC. We are planning for the next generation cluster and we may include such data on that resource if we can address their security requirements. === HIPAA https://en.wikipedia.org/wiki/Health_Insurance_Portability_and_Accountability_Act[The Health Insurance Portability and Accountability Act of 1996] ( aka HIPAA). The part that concerns us is 'Title II', (unironically called 'Administrative Simplification provisions') which describes the computer protocols, security that involve the transfer of medical records and especially the 'penalties' for their breach. The information concerns Personal Health Information (PHI) which is defined as "any information that is held by a 'covered entity' regarding health status, provision of health care, or health care payment that can be linked to any individual". While originally, the 'covered entity' referred to Health Insurance and Billing companies and Health Care Providers, the scope has been expanded to any organization that holds PHI data. And since PHI often covers genomic information, this is of concern to UC Irvine. There are levels of anonymization that can take place with such data and PHI is generally attributed to information relating to Electronic Medical Records (EMR) as opposed to raw data such as the aggregated (tho 'de-identified') genomic information in 'dbGaP' and 'TGCA'. There is an inherent problem with this dichotomy since genomic information, when coupled with the kind of metadata provided by 'dbGaP' and 'TGCA' provides a statistical narrowing of possibilities and when cross-matched with other information, can be used to identify the supposedly anonymous data. So, while it is not as sensitive as actual EMR, it cannot be defined as Open Source Data and must therefore be protected. === FISMA Federal Information Security Management Act (FISMA) refers to the regulatory act that created several levels of computer security. FISMA is not a single set of security guidelines, but a requirment that each Federal government agency define security regulations that must be implemented to host the various data types under its control. As such, it is complex, contradictory, expensive to implement and audit. It calls for not only a https://en.wikipedia.org/wiki/FIPS_199[census of the hardware and software] that make up computer systems and networks under its control, but also the classifications and protections of all the data types. Some of the documentation describing this process https://goo.gl/jc68x2[is listed here]. The 2 data types below are examples of how various Federal and State goverments have addressed the FISMA requirements. ==== Census Data While much of the Census data is presented by region in ways that cannot be exploited to the personal level and can be https://www.census.gov/data.html[directly downloaded] and used as Open Source Data, any part of it that contains personally identifiable traits is restricted to specific https://www.census.gov/fsrdc[Federal Statistical Research Data Centers], where such data can be studied under the proper computer security. ==== Corrections Similar to the Census information, some data on incarceration/corrections/prisons is available directly from the agencies involved, such as the https://www.bop.gov/about/statistics/[Federal Bureau of Prisons] and https://sites.cdcr.ca.gov/research/[California Department of Corrections and Rehabilitation], but access to any personally identifiable information is severely restricted. == HPC Current Controls The HPC cluster is a general purpose research cluster composed of about 10K 64 bit cores, with 3 large parallel filesystems totally about 2 PB, and about 40 NFS mounts of various sizes. It is nominally an open cluster, with no restricted domains , except that some proprietary software packages are restricted by group. We use the following controls to restrict access to the HPC cluster. Security techniques under consideration but not yet implemented cluster-wide are described but will be marked as such. === Login & Direct Access - ssh (passwords, shared key, ssh agent, X11). https://en.wikipedia.org/wiki/Secure_Shell[ssh] is the de facto standard for secure communication to Unix-like (*nix) Operating Systems. We currently allow the use of passwords synced by https://en.wikipedia.org/wiki/Kerberos_(protocol)[Kerberos] to our campus https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol[LDAP] server, so that even sysadmins do not have access to the password hashes. We encourage the use of https://en.wikipedia.org/wiki/Secure_Shell#Key_management[shared ssh] keys for convenience and security, but do not insist on it. If using ssh keys, we encourage the use of https://en.wikipedia.org/wiki/Ssh-agent[ssh-agent] to prevent unauthorized logins in case of theft. - https://en.wikipedia.org/wiki/Multi-factor_authentication[Two Factor Authentication]. The sysadmins can use a commercial two-factor authentication system embedded in a smart phone app called https://duo.com/product/trusted-users/two-factor-authentication[Duo] , however this technology has not been licensed for the entire campus. We are also experimenting with https://www.yubico.com/[Yubikey], a USB-based cryptographic hardware key, however this has not been rolled out to the user population.) - https://en.wikipedia.org/wiki/GNU_Screen[screen] / http://byobu.co/[byobu] are mechanisms to maintain multiple secure terminal sessions to HPC that survive a client shutdown (such as sleeping or hibernating a laptop to physically change location). Once an internet connection has been re-established, as long as a connection can be made to the HPC cluster (or to the screen-server, if different), all the sessions can be instantly recalled. - https://en.wikipedia.org/wiki/X2Go[x2go] is an X11 graphics compression utility that allows a functionality very similar to the 'screen' utility above, except that it allows disconnection and re-connection to graphics screens (including entire Desktop GUIs) as opposed to plain-text terminal sessions. === Kernel - https://meltdownattack.com/[Meltdown & Spectre] We have updated the Linux Kernel on our public-facing nodes to a patched version (https://goo.gl/hytAEG[4.4.115], at the time of writing. This provides increased protection at the cost of slower execution. Since our internal storage servers are not accessible to users other than root, we have not patched their kernels so as to maintain performance. - *Browser bugs*. Following the announcement of the Meltdown/Spectre bugs, we disabled all our Javascript-capable browsers until we could provide one which did not allow Javascript (https://www.dillo.org/[dillo]), mostly used to download data and view images generated on the cluster. We have since wrapped a patched version of https://goo.gl/9vpzn4[firefox] in a Singularity container that allows reasonably safe execution of the browser. ==== Firewall The HPC cluster is protected on the UCI Network side by the Campus https://en.wikipedia.org/wiki/Stateful_firewall[Stateful Firewall], run by the https://www.oit.uci.edu/help/security/[UCI Security Team]. All HPC users have to connect thru UCI's VPN or via an open host inside of the UCI network before logging into HPC. There is no direct inbound connection to HPC from the larger Internet. === LightPath DMZ UCI's https://www.oit.uci.edu/network/lightpath/[Lightpath 10GbE data network] was funded by the NSF, starting in 2016 and one of the criteria for funding was that firewalls were forbidden as constricting for the high speed data transfer. This leaves only https://en.wikipedia.org/wiki/Access_control_list[Access Control Lists] (ACLs) at the border router as constraining ingres. However, while HPC does have direct connection to the LightPath network, no such nodes allow inbound connection on the LightPath DMZ; only outbound connections are allowed. RCIC is investigating high-speed firewalls which could sustain full 10Gbs wirespeed and are testing some. Implementation will depend on agreement and synchronzation with the UCI Security Team. === Data Encryption We can create specific encrypted partitions on the HPC filesystems if required, but entire filesystems are not Encrypted At Rest (EAR) or by default, since theft of a multi-rack, multi-chassis distributed parallel filesystem is a fairly rare event. Usually the encrypted partitions are created on private filesystems on the owners' servers where they are invisible to other-groups. We can provide ecryptfs partitions so that the owner can decrypt the data on login with a passphrase and not even the sysadmins can view the data. If data requires backup, it is backed up encrypted, so that clear-text data is never visible. === Separation by Virtualization - *Virtual Machines* While HPC uses VMs for cluster services such as web and database servers, we do not support user-available VMs like those available on some XSEDE machines like https://portal.xsede.org/sdsc-comet[SDSC's Comet] or https://www.psc.edu/bridges[PSC's Bridges]. - *Containers* We do support https://en.wikipedia.org/wiki/Linux_containers[containers] via https://singularity.lbl.gov/[Singularity] and so can provide more isolated execution than normal, if required. - *VLANs / Networking* We are investigating the use of VLANs to provide additional isolation of applications since the Meltdown & Spectre bugs in all modern CPUs can be exploited to access previously protected memory contents of both 'VMs' and 'Containers'. Such an approach would use small blade servers to run secured applications with sensitive data. (This is only at the proposal stage; not working at present). === Web services HPC does not provide sophisticated web services such as https://usegalaxy.org/[Galaxy], PacBio's https://www.pacb.com/support/software-downloads/[SmrtAnalysis] web interface, or http://www.partek.com/partekflow[Partek's FLOW] system. We do provide a single containerized instance of the https://www.thevirtualbrain.org/[The Virtual Brain] to a research group with a reverse-proxied connection. HPC currently hosts a https://hpc.oit.uci.edu/[single web server] that provides access to static documentation and to files that users want to share with collaborators . The latter service is provided to all users, as part of a web-available but non-browseable part of the filesystem. Ie, the user has to send the collaborator an explicit link to the file being shared; users cannot 'look around' the filesystem, although they can download the files without a password. This system is in the process of being moved from an HPC-associated server to one on another network to isolate the HPC system. It will provide the same services, but not directly from an HPC-associated IP address. The required hardware has been ordered and we anticipate separation of the service will be complete in September, 2018. === Intrusion Detection We are testing a number of systems that monitor changes to specific files by comparing checksums to a database of previously verified files (the archetype of such systems is https://www.tripwire.com/[tripwire], which has become a commercial product and therefore quite expensive. This is in progress. == Timeline The timeline is still under discussion with the RCIC team. === Web server migration === Intrusion Detection === Two Factor Authentication