Interesting read - Canada's fastest computer...

Mark Hahn hahn at
Wed Jul 30 19:25:27 EDT 2003

> Anyone have any idea what range of latencies and bandwidths are
> observed on that machine (as visible to MPI)?

see the bottom of

the machine is build for very latency-tolerant
aggregate-bandwidth-intensive codes.  you can see from the histograms
that their topology does a pretty good job of producing fast links,
but the 40-ish MB/s is going to be significantly affected by 
other traffic on the machine.  I guess the amount of interference
would depend largely on how efficient is the kernel's routing code.
for instance, is routing zero-copy?  I believe these are all Intel
7500CW boards, so their NICs probably have checksum-offloading
(or is that only done at endpoints?)

latency is not going to be great, if you're thinking in terms of 
myrinet or even flat 1000bT nets, since most routes will wind up
going through a small number of nodes.  it would be very interesting
to see similar histograms of latency or even just hop-count.  if 

I understand the topology correctly, you ascend into the express-cube
for 7/8ths of all possible random routes, and the weighted average
of CDCC hops is 0*(1/8)+1*(4/8)+2*(3/8)=1.25 hops.  without diagonals, 
the avg would be 1:3:3:1=1.5 hops, which isn't all that much worse.
but I think bisection cuts 8 4x1000bT links: 4 GB/s; without express
links, bisection would be half as much!

I think I'm missing something about the eth1 (point-to-point) links...

Beowulf mailing list, Beowulf at
To change your subscription (digest mode or unsubscribe) visit

More information about the Beowulf mailing list