[Beowulf] copying big files (Henning Fehrmann)

David Mathog mathog at caltech.edu
Fri Aug 8 13:33:41 EDT 2008

Perry E. Metzger wrote:
> Is there a reason bittorrent isn't suited to this application?

That would probably work too.  

I also recall reading about "broadcast"
methods for doing this sort of distribution.  I originally wrote nettee,
which is a derivative of "dolly", because Ghost (which uses a broadcast 
method) was dreadfully slow.  This was many moons ago, and there is
probably much better broadcast software around today, even in Ghost

nettee has a few whistles and bells for this "data push" application. 
bittorrent is more of a "data pull" application.  So on the top node,
nettee can easily tell you when all of the clients are in the chain (or
not), monitor the progress of the download, and indicate if anything
went wrong.  To do that with the bittorrent clients I assume you would
have to wrap them in scripts to report the desired information back to a
central point.  It looks to me like it would be easiest to have each
client report its own completion status, perhaps through a wget with a
carefully formed URL to a cgi script on a central web server.  Because
torrents are "pull", determining from the sending side(s) if a
particular torrent client has successfully completed a file download
looks like a nightmare.  It would seem to require examining the log
information on (potentially all) of the other clients.


David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list