[Beowulf] good 24 port gige switch

Bill Broadley bill at cse.ucdavis.edu
Tue Mar 9 17:49:14 EST 2004


On Mon, Mar 08, 2004 at 10:40:01PM -0600, Russell Nordquist wrote:
> 
> thanks for all the good info. it got me to thinking....i have resources
> for comparing most components of a cluster excepts network switches. it
> would be nice to have a source of information for this as well.
> something like:
> 
> *bandwidth/latency between 2 hosts
> *bandwidth/latency at 25%/50%/75%/100% port usage
> *short vs long message comparisons

I use nrelay.c a small simple program I wrote that will MPI_Send
MPI_send very size packets between sets of nodes.

So I do something like the following to find best base latency
and bandwidth:
mpirun -np 2 ./nrelay 1  # then run with 10 100 1000 10000
size = 1, 2 nodes in  2.97 sec (  5.7 us/hop)    690 KB/sec
size=    10, 524288 hops,  2 nodes in  3.06 sec (  5.8 us/hop)   6688 KB/sec
size=   100, 524288 hops,  2 nodes in  4.19 sec (  8.0 us/hop)  48868 KB/sec
size=  1000, 524288 hops,  2 nodes in 15.37 sec ( 29.3 us/hop) 133267 KB/sec
size= 10000, 524288 hops,  2 nodes in 40.72 sec ( 77.7 us/hop) 502908 KB/sec

So we have an interconnect that manages 5.8 us for small messages and
500 MB/sec or so for large (10000 MPI_INTs).

Then I run:
mpirun -np 2,4,8,16,32,64 ./nrelay 10000  
size= 10000, 524288 hops,  2 nodes in 40.72 sec ( 77.7 us/hop) 502908 KB/sec
size= 10000, 524288 hops,  4 nodes in 39.79 sec ( 75.9 us/hop) 514698 KB/sec
size= 10000, 524288 hops,  8 nodes in 39.21 sec ( 74.8 us/hop) 522253 KB/sec
size= 10000, 524288 hops, 16 nodes in 45.53 sec ( 86.8 us/hop) 449772 KB/sec
size= 10000, 524288 hops, 32 nodes in 49.25 sec ( 93.9 us/hop) 415876 KB/sec
size= 10000, 524288 hops, 64 nodes in 52.90 sec (100.9 us/hop) 387111 KB/sec

So in this case it looks like the switch is becoming saturated.

The source is at:
http://cse.ucdavis.edu/bill/nrelay.c

I'd love to see numbers posted for various GigE, Myrinet, Dolphin and IB 
configurations

-- 
Bill Broadley
Computational Science and Engineering
UC Davis
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



More information about the Beowulf mailing list