[Beowulf] Semi-philosophical Question

Jag agrajag at dragaera.net
Thu Dec 18 17:35:36 EST 2003


On Thu, 2003-12-18 at 16:19, Leigh wrote:
> I was talking over Beowulf clusters with a coworker (as I have been working 
> on learning to build one for my company) and he came up with an interesting 
> question that I was unsure of.
> 
> As most of the data is saved upon the "gateway" and the other machines 

Not really in answer to your question, but some general info..
That is one configuration, but is not always the case.  As an example, I
currently have a cluster that has a node dedicated to sharing out a
terrabyte of space over NFS.  There's one node that's dedicated to doing
the scheduling (using SGE), and three other nodes allow user logins for
them to submit jobs from.  Jobs aren't executed on any of these nodes.

There's also something called pvfs that'll let you use the harddrives on
all your slave nodes and combine them into one shared filesystem that
they can all use.

> simply access it to use the data, what happens when multiple machines are 
> making use of the same data and they all try to save at once? Do they all 
> work as one system and save it only once, or can multiple nodes 
> theoretically be using one file and both try to save to it?

This is really an application specific question.  A lot of MPI jobs
shuffle all the data back to one of the processes and let that process
write out the output files, so you won't have a problem.  There are also
other programs that may have a seperate output file for every slave node
the job is run on.  What is your cluster going to be used for?  The best
way to answer the question is to determine what apps will be used and
see how they handle output.  If its an inhouse program, you may want to
make sure your programmers are aware they'll be writing to a shared
filesystem so that they don't accidently write the code in such a way
that the results get corrupted by having them all use the same output
file.

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



More information about the Beowulf mailing list