mulitcast copy or snowball copy

[Felix Rauch]

On Mon, 18 Aug 2003, Rene Storm wrote:
> Problem:
> I want to distribute large files over a cluster.
> To raise performance I decided to copy the file to the local HD of
> any node in the cluster.

Quick solution:
Dolly [1]
;-)

Longer description: I once wrote a tool called "Dolly" to clone whole
hard-disk drives, partitions, or large files to many nodes in a
cluster. It does so by sending the files concurrently around the
cluster in a "TCP chain". In a switched network, this solution is
often faster then IP multicast becauce Dolly can use the proven TCP
congestion control and error correction, whereas high-speed reliable
multicast is something difficult.

> Till now I've take a look at msync (multicast rsync).

Another tool is "udpcast".

> What happens if one node (in the middle) is down.

Dolly, can't handle that (it's a working prototype), but Atsushi
Manabe extended Dolly into Dolly++, which supposedly can handle node
failures (see link in [1]).

We use Dolly regularly to clone our small 16-node cluster and the
local support group uses Dolly to clone the larger 128-node
cluster. Because that cluster has two Fast Ethernet networks, we can
clone whole disks with about 20 MByte/s to all nodes in the cluster.

If you want to clone files instead of partitions, just specify your
file name in the config file instead of the device file.

- Felix

[1] http://www.cs.inf.ethz.ch/CoPs/patagonia/#dolly

-- 
Felix Rauch                      | Email: rauch at inf.ethz.ch
Institute for Computer Systems   | Homepage: http://www.cs.inf.ethz.ch/~rauch/
ETH Zentrum / RZ H18             | Phone: +41 1 632 7489
CH - 8092 Zuerich / Switzerland  | Fax:   +41 1 632 1307

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf