Best file system strategy

Phil Carns pcarns at
Fri Nov 10 18:41:04 EST 2000

> We just received our 8-node cluster and I am trying to set everything up. 
> 	Each node has:
> 			2-550 MHz Pentium II
> 			256Mb Ram
> 			1-18Gb U2 SCSI 
> Node 1 serves as the head node. 
> I want to get the group's knowledgable advice on how to set up the 
> user space (/home). There are nice size drives on each node, but how do 
> I access them? I already have a roughly 12Gb /home partition on each 
> Here are the options that I am looking at...
> 	1) NFS mount node#/home to each node as /home#. Users 
> 	   would be assigned different base home directories.
> 	2) Use pvfs on the /home space. I am not sure how to set 
> 	   up /home here.
> 	3) If I can find the $$, purchase a new /home disk for the 
> 	   head node, use pvfs on the old /home on each node to 
> 	   create a large scratch space for running computations
> Option 1 should entail quite a heavy communication load, I would think.
> As to option 2, I am concerned with the redundancy in pvfs. If I 
> understand correctly, were the system to lose a node, I have lost the 
> file system. I understand I can reboot the node, but what if I lose a 
> drive on a node. Is all lost in this case? 
> Option 3 looks the best, but it means purchasing a new disk. 

Hi Carey,

>From the PVFS perspective, 3) may be your best choice.  PVFS works great
as a "scratch space" for parallel computation, and will give you a big
increase in performance for those applications (shameless plug :).  If
you are concerned about redundancy (this is actually a good idea from the
management perspective even if you don't care about redundancy), then I
would recommend NFS for the actual /home directory.  NFS survives PC
failure no better than PVFS, but is at least easier to back up because it
resides on one disk.  It is also easier to make use of RAID for disk level
protection if you use NFS because you only have to buy one extra disk to
be able to use software raid mirroring.  Mirroring every drive in a PVFS
cluster could get expensive quickly.  

You don't necessarily need to purchase a new disk for home unless you need
more than 18G for that purpose.  It is perfectly acceptable to use
the existing disk on your head node as an NFS exported home directory and
then use your other 7 node disks as PVFS storage.  This is how we
typically setup clusters at our site and other people seem to have 
success with this configuration as well.

hope this helps,

Phil Carns
Parallel Architecture Research Lab
Clemson University

Beowulf mailing list
Beowulf at

More information about the Beowulf mailing list