[Beowulf] copying big files

Jan Heichler jan.heichler at gmx.net
Fri Aug 8 11:52:40 EDT 2008

Hallo Henning,

Freitag, 8. August 2008, meintest Du:

HF> Hi everybody,

HF> One needs basically a daemon which handles copying requests and establishes
HF> the connection to next node in the chain.

Why a daemon? Just MPI that starts up the processes on the remote nodes during programm startup. Advantage is that you can use any high-speed-interconnect which you have an MPI for. 

HF> Has somebody written such a tool?

I wrote a tool in Java in 2004 or 2005 during an internship for IBM at the IPK in Gatersleben that implemented your strategy. Worked good but had some flaws since i was using some Java specific remote procedure calls and daemons. Implementing it with C/C++ and MPI can't take more than a couple of hours and should be easy to do. 

You could use tar and pipe the datastream to copy whole directories without worrying to much about file attributes. I always wanted to implement it myself but i can't find time with my current job :-(

Could be a good project for a student to do.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.clustermonkey.net/pipermail/beowulf/attachments/20080808/f862522b/attachment-0001.html>
-------------- next part --------------
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list