[Beowulf] Q: IB message rate & large core counts (per node)?

Gilad Shainer Shainer at mellanox.com
Tue Feb 23 17:51:48 EST 2010

PCIe Gen2 at 5GT, and 8/10 bit encoding and current chipsets
efficiencies gives you around 3.3GB/s per direction, so one IB QDR port
can handle that. For more BW out of the host, you can use more adapters
(single port ones are the cost effective solution for that). 




From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org]
On Behalf Of Brian Dobbins
Sent: Tuesday, February 23, 2010 2:42 PM
To: Patrick Geoffray
Cc: beowulf at beowulf.org
Subject: Re: [Beowulf] Q: IB message rate & large core counts (per



	  I thought PCIe Gen2 x 8 @ 500 Mhz gives 8GB/s?  I know there
are 250 and 500 Mhz variants in addition to the lane sizes, so while a
250 Mhz x8 link wouldn't provide enough bandwidth to a dual-port card,
the 500 Mhz one should.  But I'm woefully out of date on my hardware
knowledge, it seems.  Of course, EDR (eight data-rate) IB is on the
roadmap for 2011, so if we're in no rush that could help, too.

Er,.. I thought about this a half-second after I sent it.  First, that's
5.0 Ghz, not 500 Mhz, and second, that's probably aggregate
bi-directional throughout at 8 GB/s, isn't it?  So, I got it, 4 GB/s per
direction, which means dual-port won't get us much in terms of
unidirectional throughput.

  - Brian

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.clustermonkey.net/pipermail/beowulf/attachments/20100223/8881a7ea/attachment-0001.html>
-------------- next part --------------
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list