data storage location

Guy Coates gmpc at
Wed Sep 10 15:01:55 EDT 2003

As always, it depends on what  the data access pattern like, and how many
nodes you have.

If the data access is read-only, *and* your node number is small (<10),
*and* you have a fast network then you *might* be able to get away with

However, for read/write access or more than a few nodes, you need to be a
bit more creative.

Fibre channel + switch may look good on paper but have their problems (not
least of which is cost).  If you have a moderate amount of fibre clients
trying to access the same disk controller you can easily saturate the
fibre ISLs or disk controllers. For example, 4 clients doing 50 Mbytes/S
sustained data access are easily going to saturate the fibre to your disk
controller, whether switched or on FC-AL.  If your data access is bursty
then you might do better. Furthermore, if you want all your clients to
access the same filesystem on the your fibre channel disks then you are
also going to need a clustered filesystem.

The alternative approach is to keep copies of the data on local disk on
each node. This gives you good IO rates, but you then have a substantial
data management problem; how to you copy 100Gb to each node in your
cluster in a sensible amount of time, and how do you update the data and
make sure it is kept consistent?

The commonest approach to data distribution is to do some sort of
cascading rsync/rcp which follows the topology of your network.

If your dataset is larger than the amount of local disk on your nodes, you
then have to partition your data up, and integrate that with your queuing
system, so that jobs which need a certain bit of the data end up on a node
which actually holds a copy.

Recently we've had some success using multicast methods to distribute data
to large numbers of nodes. udpcast is one of
the better multicast codes, but you'll need to put some wrappers around it
to make it useful. The multicast method is substantially faster than
rsyncing data on large clusters.


Guy Coates

Guy Coates,  Informatics System Group
The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1SA, UK
Tel: +44 (0)1223 834244 ex 7199

Beowulf mailing list, Beowulf at
To change your subscription (digest mode or unsubscribe) visit

More information about the Beowulf mailing list