[Beowulf] IB Cluster Configuration Options

Mark Hahn hahn at physics.mcmaster.ca
Fri Jul 29 11:08:43 EDT 2005

> As a means to reduce our cost the suggestion was raised to cascade 4 9024 
> switches, or reducing the node count, Opteron clock rate, and memory. Just 
> wondering how severe the hop penalty might be in the case of cascading 
> switches?

IB switch vendors claim quite low per-hop latencies.  of course, the real 
problem is that the base of a non-fat switching tree is highly contended.
(the voltaire 9024 spec says 140 ns, for instance).

the question is: how much are you willing to pay for good
bisection-bandwidth?  personally, I cannot imagine buying IB except for 
extremely bandwidth-intensive apps, in which case, you probably want 
a non-blocking (full bisection) switching network.  the 9024 doesn't 
seem to be offered with all 12x ports, which is kind of odd, since 
a classic fat tree would be most natural with 8 12x ports at the top.
but a set of 5 9024-12 switches would get you a pretty well-connected 
set of 60 nodes.  a hypercube with express links would use 8 switches
and have great bandwidth and at most 3 switches on any path (I think!).

maybe you should consider alternatives (Myri 10G seems extremely attractive,
for instance).

regards, mark hahn.

Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list