Distributed Filesystems: BeeGFS vs Gluster

Distributed filesystems are those where the files are distributed in whole or in part on different block devices, with direct interaction with client computers to allow IO scaling proportional to the number of servers. Acording to this definition, a network-shared NFS server would not be a distributed filesystem, whereas Lustre, Gluster, Ceph, PVFS2 (aka Orange), and Fraunhofer are distributed filesystems, altho they differ considerably on implementation details. While NFS is a well-debugged protocol and has been designed to cache files aggressively for both reads and writes, the single point bottleneck for IO, is a real weakness in a compute cluster environment.

We did not test Lustre, due to the widely held perception (and 2 bad past experiences) that it is very fragile to software stack & driver changes. Our users have said loudly that they value stability over speed. However, once Lustre produces a stable 2.4 series that supports Lustre-on-ZFS that doesn’t require an entire FTE to maintain it, we’ll try it. We tried Ceph about a year ago and it wasn’t nearly ready for production use. OrangeFS was interesting but there were so few mentions of it on the Internet and the overall traffic was so low that we decided to skip it. IBM’s GPFS is a very mature and capable filesystem that interfaces with other IBM products such as Tivoli and is supported by the full faith and credit of IBM, but as you might guess, it’s also quite expensive for a cluster filesystem (about $8K per server). We were looking for a free alternative. Via some web-browsing and direct recommendations, we decided to try Fraunhofer as an alternative, so this document will only discuss the Fraunhofer and Gluster filesystems.

Since we’re in production with Gluster, we have a vested interest in keeping it. It does work pretty well, with the few disadvantages mentioned above. Since that is the case, we will base our decision on these points:

See the Conclusions for the answers to these questions.

Can an SSD caching controller provide enough of a speed increase to address the current problems?

The disk controller on the Glfs2 test system uses on-board SSDs as part of the caching system, so rather than using expensive DRAM to provide a couple of GB that previous controllers might, the LSI Nytro 8100 4i embeds 90GB of SSD (specs say 100GB, but only 90 is usable) on the controller board and splits it to allow 45GB for write and read caching. It’s not clear that it’s dynamically balanced (ie uses more cache for reads if the job is read-heavy and vice versa). This architecture seems to be heavily influenced by the success that FusionIO has had with their very high performance (and very expensive) caching accelerators and now we’re starting to see these SSD-based accelerators integrated into prosumer grade controllers. While these are more expensive than their non-SSD products, it’s a small part of a large filesystem and their performance is quite impressive on single nodes; we are testing them to see what kind of performance benefit they give to distributed filesystems.

See the Conclusions for the answer to this questions.

We ran 3 versions of FhGFS - one on old hardware (FhGFS1) and another (FhGFS2) on new hardware, using the LSI Nytro controller mentioned above, and a 3rd (FhGFS3) which used split metadata servers from FhGFS1 with the newer hardware from FhGFS2. The dual metadata servers were connected via DDR IB to the same QDR switch as the storage servers and the clients. RDMA was used across all machines. MTU was set to 65536.

FhGFS1: This system ran on 4 storage servers, each of which has 4x64b Opteron cores, 16GB RAM (1 has 8GB) and 4-8 7K SATA disks in RAID5 config. The bricks used default XFS format. The metadata was split over 2x4core/16GB machines with single 120GB SATA ext4-formatted disks (default settings; no additional tuning).

FhGFS2: This system ran on 2 recent servers from Advanced HPC. They both used the same chassis, but differed slightly internally. One used an 8core AMD CPU while the other uses a 12core Intel CPU. The metadata server was a single 8core/64GB machine using an 87GB ext4 partition (7K SATA disk). It is NOT recommended to use such small metadata server storage for such a large filesystem. At the end of our testing, with only a small fraction of the main filesystem used, we ran into inode starvation on the metadata server and had to do some re-jiggering to allow the test to continue. This is mentioned in the FhGFS documentation, but I managed to miss it.

Both had 2x RAID6 arrays on the same controller, one with 18 disks; one with 17 disks. They used the same disks, the Hitachi 3TB SAS disks Model HUS723030ALS640.

This was strictly a test system, otherwise idle.

FhGFS3: This system used the same 2 storage servers that FhGFS2 used, but used the same 2 metadata servers (one also ran the Management Daemon) as FhGFS1, with slightly different storage. The metadata was placed on a RAID1 system that was tuned specifically to the FhGFS metadata recommendations. The storage server RAID systems were also tuned to the FhGFS storage recommendations for XFS formatting and mount options, IO scheduler, and Virtual Memory settings. See Configuration below.

Glfs1: This system is running on 4 identical 36-slot SuperMicro chassis' from Advanced HPC similar to the ones described above, differing mostly in the disk controller and RAM. Each of the 4 systems consisted of:

The 3ware Glfs1 above is in production; glusterfsd’s are running at a steady load of 1.5 - 2 continuously, supporting an academic compute cluster of ~2500 cores with an average aggregate load of about 43% over the last month.

Glfs2: This system was identical to the one noted above for FhGFS2.

A few tests were run on a Direct-Attached 6-spindle RAID5 (3TB, 7K SAS disks), hosted on a 64core, 512GB compute node. This fs is a testbed which was essentially idle.

The write test was a simple script that started 10 simultaneous dd sessions on each node, each one pushing 2GB from /dev/zero to files on the target filesystem for a total of 9nodes*2GB*10instances/node=180GB of data.

I acknowledge there are a number of problems with using dd and /dev/zero: the underlying filesystem may not match the blocking of dd, the controller may do inline compression (which would make short writes of zeros highly compressible), etc, but it’s often used and since we’re measuring the time to sync to disk, it’s a convenient shorthand for streaming writes.

The results indicate that for such writes, while the old hardware used for FhGFS1 restrained its ability to push data thru to the disks, FhGFS2 (on new hardware) was about as fast as the Glfs1 test system, but could not match the Glfs2 write speed (which used identical hardware). Gluster was almost 2X as fast with streaming writes. Note that on 2 ocassions, I tried to write this same amount to the local filesystem (DASR5) and both times it locked up.

This test writes 20GB of data in one of 2 ways: either as multiple individual Perl print statements of 100 chars each to STDOUT or as the same amount of data stacked in memory and written with a single print statement. This demonstrates the different ways the 2 filesystems handle single files - FhGFS distributes the files over the available bricks, while Gluster writes single files to individual bricks.

The results of this test, in which 20GB of data is written in 2 formats

* This test was mistakenly skipped on the Glfs2 system and the servers are no longer available.

The write profile partially explains the difference observed between Glfs1 and FhGFS2.

Note that only one server (4-W) gets all the data during the write.

Note that one server (biostor5) gets half the total written bytes

Note the total bytes being written.

While the Glfs1 filesytem was much slower than the FhGFS2 in this case, it was surprising that the big write took longer than the small writes. This has not been the case for other instances and both myself and another researcher at UCI found the same thing previously: that accumulating data into larger chunks and then writing it yielded much better performance. I cannot explain why this is not the case here.

The core command for this test was:

The read test was a simple recursive, case-insensitive search for peter thru the entire (compiled) linux kernel 3.7.9 tree. This tests the ability of the filesystem to stream multiple reads to competing nodes. In this case, the DASR5 was NFS-mounted to all the other nodes and the aggressive caching allowed all the client to hit the cache fairly well. All but the FhGFS1 system did fairly well, and based on the scatter of the data, no distributed system seemed to be better than the other under load.

Run with clusterfork via the multi-grep.sh script

This involved a single node recursing thru a directory tree of 1814 dirs and 28619 files. This tests the simple single user doing recusive queries of the data, something that interactive use often involves. Here the Direct Attached Storage was enormously faster than any of the distributed filesystems, but among the distributed ones, FhGFS was >2x faster, even with FhGFS1 running on extrmely inferior hardware. Here we can credit the metadata server-mediated layout as opposed to the DHT organization of Gluster. The Nytro controller seemed to help decrease the query time to 1/2 to 1/3 compared to the 3ware 9750-based filesystem.

executed on 64core node that had load of 2.

The mixed IO test is one which does a make clean and then a make -j8 bzImage of the linux kernel 3.7.9 to completion. It is also executed on 9 nodes and tests the ability of the filesystem to respond to a diverse mix of small reads and writes of various sizes. Surprisingly, FhGFS1 comes out fairly well in this test, beating Glfs1 despite much inferior hardware. Also surprising is that when running on equally fast hardware, Fgfs2 and Glfs2 are surprisingly close.

Given the speed at which FhGFS1 can complete this test, I would have expected FhGFS2 to have run much faster, however perhaps what we’re seeing is the asymptotic approach to the consumption limit. ie make -j8 cannot consume data as fast as the data can be supplied. To test this, I modified the test to use make -j32 to see if that increased the speed of the test. For the FhGFS2 test, it did, significantly, reducing the time to 1/2. For the Glfs1, it did not, increasing the time to completion.

For a baseline, make -j8 on a single node using the DASR5 takes 126s. and make -j32 takes 65.5s

Command for this test was:

Both Gluster and Fhgfs use standard native Linux filesystems for block-level storage. In both cases, the recommended filesystem is the XFS filesystem from SGI. Like many such filesystems, it has many tunables and these can have a significant impact on performance. This can be seen the results above, especially between the Fhgfs2 and Fhgfs3, the chief differences between which are changes in tuning.

While the man pages for XFS is readily available, they are quite opaque when it comes to actually figuring out what they do and especially how the parameters affect various kinds of IO performance. There are some fairly good non-official guides for configuring XFS, as it applies to various domains. The OSS video recording system MythTV has a very good page on tuning XFS, especially as to the mkfs tunables for video streaming which shares many features with general large file streaming IO. Notably, they include a script for generating tunable variables given your particular hardware.

The mount options that we used for Fhgfs3 largely follow the Fraunhofer storage recommendations. Below is the output from one of our storage server mount commands for the storage RAIDs.

Similar to the XFS tunables, the ext4 filesystem can also be tuned for better metadata. Again, our metadata server filesystems were tuned according to the Fraunhfer metadata reommendations. Below is the output from one of our metadata server mount commands for the metadata RAIDs.