mulitcast copy or snowball copy
rauch at inf.ethz.ch
Mon Aug 18 09:12:52 EDT 2003
On Mon, 18 Aug 2003, Rene Storm wrote:
> I want to distribute large files over a cluster.
> To raise performance I decided to copy the file to the local HD of
> any node in the cluster.
Longer description: I once wrote a tool called "Dolly" to clone whole
hard-disk drives, partitions, or large files to many nodes in a
cluster. It does so by sending the files concurrently around the
cluster in a "TCP chain". In a switched network, this solution is
often faster then IP multicast becauce Dolly can use the proven TCP
congestion control and error correction, whereas high-speed reliable
multicast is something difficult.
> Till now I've take a look at msync (multicast rsync).
Another tool is "udpcast".
> What happens if one node (in the middle) is down.
Dolly, can't handle that (it's a working prototype), but Atsushi
Manabe extended Dolly into Dolly++, which supposedly can handle node
failures (see link in ).
We use Dolly regularly to clone our small 16-node cluster and the
local support group uses Dolly to clone the larger 128-node
cluster. Because that cluster has two Fast Ethernet networks, we can
clone whole disks with about 20 MByte/s to all nodes in the cluster.
If you want to clone files instead of partitions, just specify your
file name in the config file instead of the device file.
Felix Rauch | Email: rauch at inf.ethz.ch
Institute for Computer Systems | Homepage: http://www.cs.inf.ethz.ch/~rauch/
ETH Zentrum / RZ H18 | Phone: +41 1 632 7489
CH - 8092 Zuerich / Switzerland | Fax: +41 1 632 1307
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
More information about the Beowulf