Cluster file systems are hot. What good is 1000 processors if you can't write to a file without clogging your network or server. Learn about the issues and experiences of parallel file systems with Distinguished Cluster Monkey Jeff Layton as your guide.

How to sleep well when running PVFS

In the last article we looked at performance improvements for PVFS1 and PVFS2. In this installment, we'll examine improving the resilience or redundancy of PVFS as well as putting some flexibility into the configuration.

Redundancy or Resiliency is the ability to tolerate errors or failures without failure of the entire system. For PVFS (Parallel Virtual File Systems) this means the ability to tolerate individual failures without all of the PVFS being unavailable.

I feel the need for PVFS speed

In a previous column I looked at the various ways to code for PVFS1 and PVFS2. However, I never really discussed how to architect a PVFS system for your application. A correct PVFS design can improve the I/O performance of your application, ease administrative burden, improve flexibility, and improve the fault-tolerance. In this column I'll examine how to enhance and tune performance.

Coding for PVFS1 and PVFS2

Arguably, one of the most popular parallel file systems is PVFS (Parallel Virtual File System). Work on PVFS began around 1993 at Clemson University. It has since grown through partnerships with the academic, government, and industrial community. PVFS also enjoys a licensed under the LGPL (Lesser General Public License). It has been deployed in production at a number of sites for several years. There are currently two versions of PVFS: PVFS1, the original version; and PVFS2, the new version under development and very close to production release. While PVFS is a great concept, most efficient use comes from recoding your application to take true advantage of either PVFS1 or PVFS2 (unless your code already uses MPI-IO). However, rewriting the I/O portion of the your code to use MPI-IO is really not a difficult task. In this column I'll be examining the various ways one can code to use PVFS1 or PVFS2.

Plenty of Options for Plenty of Files

Welcome to Cluster Money (aka' "The Monkey") and the Cluster File Systems Column. This column will be taking a look at these new file systems, how they work, how they fit into the HPC world, and how you can deploy them for maximum effect. It will also explore some of the details of file systems to help you understand such things as the difference between metadata and inodes. It will also discuss some of the underlying hardware that is important to file systems such as networking and the data storage devices itself.

This first column will present a brief overview of many of the file systems that are available today for clusters as well as some storage options for the file systems. This includes true parallel file systems, new network file systems, and even a storage option for high performance storage for clusters. The list isn't extensive but is intended to wet your appetite for more information about the explosion of storage that is happening.


Login And Newsletter

Create an account to access exclusive content, comment on articles, and receive our newsletters.


This work is licensed under CC BY-NC-SA 4.0

©2005-2023 Copyright Seagrove LLC, Some rights reserved. Except where otherwise noted, this site is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International. The Cluster Monkey Logo and Monkey Character are Trademarks of Seagrove LLC.