Living the terabyte lifestyle
Recently Henry Newman and Jeff Layton (distinguished ClusterMonkey) had dinner at a Cuban restaurant. While dinning, they discussed the disparity between Linux file system sizes and the growth rate of hard drives. They note:
"With 3TB, drives ext 3/4 maxes out at five disk drives. Jeff and I thought that was just insane, given you can buy five 3TB drives at Fry's and put them in your desktop. XFS maxes out at 33.3TB disk drives, and even that is far too small in our opinion. Clearly, supported file system sizes have not scaled with disk drives sizes or the demand for big data."
The full discussion, The State of File Systems Technology, Problem Statement, is at the Enterprise Storage Forum. It seems to be one of those, "wait a minute, wow, I never looked at that" moments. This issue is not just a concern for HPC or the high end issue, it really may hit home in some smaller installations. Note that the issue is not just file system sizes, but is also about performance. Henry and Jeff have a plan to explore this issue (see the end of the article).
While we are at it, read Joe Landman's What are xfsâs real limits? blog. And, check out the ensuing discussion. I respect Joe's opinion. Like me, he likes to make arguments based on hard data and benchmarks. (Like there is any other way...) We'll touch base on this again as the project continues.