[Beowulf] 1.2 us IB latency?

Mark Hahn hahn at mcmaster.ca
Tue Mar 27 21:36:42 EDT 2007

>> a dramatic breakthrough: 1.2 us, 25M msg/s.  since we just happend to
>> or is this an example of message aggregation?  heck, from the url
>> above, it might even be counting intra-box messages.
> Nope, this is "normal" ping-pong for the new generation cards (connectx).

so if I do this:

start timer 
send(other,small-message)		recv(first,small-message)
recv(other,small-message)		send(first,small-message)
stop timer

I'll actually see 2.4 us between the timer calls?  if I understand,
aggregation would only help on a streaming test.  in fact, this kind 
of isolated RPC-like exchange is what I see most commonly.

> Maybe a bit optimistic though, I'd expect closer to 1.5 in a back-to-back
> config.

so for a small switch (24pt, say), how many hops to the internal fabric,
and they're, what, .2 us each?

also, does back-to-back work well?  I can imagine some cases where 
putting two dual-port cards in each node and creating a mesh might 
work well.

>> also, I'm sorta amazed people keep selling (and presumably buying)
>> dual-port IB cards.  doesn't that get quite expensive, switch-wise?
> Not defending them but, It could possibly maybe be useful if you have a
> stand-alone IB net for, say, storage or something else not mpi. Also, it's
> not like they're that much more expensive than single port ones...

yeah, I can see PHB's buying redundant fabrics.  I'd be more interested in
using the higher port-count for FNN or related topologies (assuming switches 
are cheap, at least at some size...)
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf


More information about the Beowulf mailing list