data storage location

[Felix Rauch]

On Wed, 10 Sep 2003, Guy Coates wrote:
[...]
> Recently we've had some success using multicast methods to distribute data
> to large numbers of nodes. udpcast http://www.udpcast.linux.lu/ is one of
> the better multicast codes, but you'll need to put some wrappers around it
> to make it useful. The multicast method is substantially faster than
> rsyncing data on large clusters.

I had two problems when testing UDPcast:

- A cheaper ATI CentreCom Fast-Ethernet switch multicasted data only
  with the speed of the slowest connected machine, even if that
  machine was *not* part of the multicast group. I had to unplug all
  10 Mbps links to speed up UDPcast above 1 MByte/s.

- With Gigabit Ethernet and a smaller cluster of 1 GHz PentiumIII
  machines I had to set a rate limitation on the sender. Otherwise the
  sender and the receivers lost synchronization and the transmission
  didn't work.

However, if you forgive me a short bit of advertising, we use the
"Dolly" programm to replicate large amounts of data in our clusters:
http://www.cs.inf.ethz.ch/CoPs/patagonia/#dolly

It replicates data with a multi-drop chain over TCP and scales quite
nicely. We only had performance problems on a switch with a rather
limited backplane, but otherwise we use it regularly in our 16- and
128-node clusters.

- Felix

-- 
Felix Rauch                      | Email: rauch at inf.ethz.ch
Institute for Computer Systems   | Homepage: http://www.cs.inf.ethz.ch/~rauch/
ETH Zentrum / RZ H18             | Phone: +41 1 632 7489
CH - 8092 Zuerich / Switzerland  | Fax:   +41 1 632 1307

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf