[Beowulf] Re: scratch File system for small cluster
mathog at caltech.edu
Thu Sep 25 15:00:37 EDT 2008
Joe Landman <landman at scalableinformatics.com> wrote
> Glen Beane wrote:
> > I am considering adding a small parallel file system ~(5-10TB) my small
> > cluster (~32 2x dual core Opteron nodes) that is used mostly by a
> > regular users. Currently the only storage accessible to all nodes
> > directory space which is provided by the Lab's IT department (this
is a SAN
> > volume connected to the head node by 2x FC links, and NFS exported
> > compute nodes). I don't have to "worry" about the IT provided SAN
> > they back it up, provide redundant hardware, etc. The parallel file
> > would be scratch space (and not backed up by IT). We have a mix of home
> > grown apps doing a pretty wide range of things (some do a lot of
> > don't), and things like BLAST and BLAT.
> Hi Glen:
> BLAST uses mmap'ed IO. This has some interesting ... interactions
> ... with parallel file systems.
Right, and it isn't just the mapping of the databases and input file.
One must also be careful with how BLAST output is directed. Sending it
all to the same NFS mounted file system as "node01.out", "node02.out",
etc. will do very unpleasant things to both your network and the file
server. Far better to write those locally to /tmp/nodeXX.out, and then
take some care in moving them back to the central file system later, so
that the data transfer can proceed without interference.
This doesn't mean you have to wait until the end of the run and send
each node's entire output file back at once. It can be more efficient,
but more complicated, to write the output files on each node in
reasonable sized chunks and then interleave the transfer of those to the
central store with the ongoing run. Whether this is worth the extra
effort depends mostly on the number of queries in the input file and
the verbosity of the output file.
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
More information about the Beowulf