File System Shootout

I haven't written about the kernel mailing list before, but something with direct correlation to clusters recently came across the list. On October 24, 2003, Mike Benoit posted an email to the kernel mailing list announcing some updates to his file system shootout. Mike used two file system benchmarks, Bonnie++ and IOZONE, that are designed to test hard drive and file system performance. Mike had previously posted to the kernel mailing list when he first had file system performance results. There were several suggestions for tuning specific file systems, and a request for using better hardware. He made the suggested changes and retested. He now used an Opteron 240 system with 512 Mb of RAM and a PII/450 with 512 Mb of RAM for the shootout. He tested EXT2, EXT3 with several options, XFS, JFS, ReiserFS v3 with two options, and Reiser4 with two options, all using several recent versions of the 2.6 test kernel. He ran the tests 3 times each on both a SCSI disk and an IDE disk and presents all of the results in a nice tabular form with some results highlighted.

Why are file systems so important to clusters? There are many clusters applications that read and write data to a file system. This file system can be local (i.e. in the node itself) or part of a central file system (e.g. NFS). Also, applications could be using a distributed file system like Lustre or a high-speed parallel file system like PVFS (Parallel Virtual File System) or GPFS (General Parallel File System). For all of these configurations, applications that are I/O (Input/Output) bound, spend a great deal of time writing to file systems. Hence, file system performance is important to them.

Mike has several interesting observations. First, in his opinion, based on his benchmark results, XFS and JFS give the best bang for the buck. That is, they are close to EXT2 in performance with a small amount of CPU usage. It's interesting to note that journaled file systems are slower than the the non-journaled file system EXT2. So if you don't mind an occasional, potentially long file system check (fsck), then EXT2 is still pretty fast. It might be very useful for relatively small read-only file systems.

For applications that are bound by I/O performance, he recommends Reiser4, or XFS, or ReiserFS v3. Remember though, that Reiser4 is still experimental and has not yet made it into the 2.6 kernel. However, according to Mike, the results are very encouraging. He also mentioned that if your file system uses lots of small files, then ReiserFS v3 is the way to go. However, if your file system has medium to large size files, then he recommends XFS. Mike goes on to mention that if you are CPU limited, he recommends JFS.

Finally, Mike makes some observations comparing SCSI disks to IDE disks. He ran the tests on a SCSI disk that was running at 10,000 RPM and an IDE disk running at 7,200 RPM. He found that Reiser4 had about a 50% boost in speed using SCSI disks compared to IDE disks. Both JFS and EXT3 gained the least speed by moving to the SCSI disks, only gaining about 5-20%. He also mentions that in one case JFS actually ran slower on a SCSI disk than on an IDE disk. He finally suggests that a 5 times cost difference for SCSI drives may not be worth the cost if an average improvement of 20% is all that is is seen over IDE drives.

This article was originally published in ClusterWorld Magazine. It has been updated and formated for the web. If you want to read more about HPC clusters and Linux you may wish to visit Linux Magazine.

Jeff Layton has been a cluster enthusiast since 1997 and spends far too much time reading mailing lists. He occasionally finds time to perform experiments on clusters in his basement.