[Beowulf] Q: IB message rate & large core counts (per node)?
tom.ammon at utah.edu
Mon Mar 15 19:45:44 EDT 2010
If I understand correctly, 40GbE is 64/66 encoded.
On 3/15/2010 5:30 PM, Patrick Geoffray wrote:
> On 3/15/2010 5:24 PM, richard.walsh at comcast.net wrote:
>> to best and worst case). It would be good to add Ethernet to the mix
>> (1Gb, 10Gb, and 40Gb) as well.
> 10 Gb Ethernet uses 8b/10b with a signal rate of 12.5 Gb/s, for a raw
> bandwidth of 10 Gb/s. I don't know how 1Gb is encoded and 40 Gb/s is
> still in draft. Last time I looked at 40 Gb/s, it was pretty much four
> 10 Gb links put together, so I would say 8b/10b with 50 Gb/s signal rate.
>> >There is a similar protocol efficiency at the IB or Ethernet level, but
>> >the MTU is large enough that it's much smaller compared to PCIe.
>> Would you estimate less than 1%, 2%, 4% ... ??
> It depends on the packet size. For example, 14 Bytes Ethernet header on
> 1500 Bytes MTU, that's 1%. For Jumbo frames at 9000B MTU, it's much less
> than that. I don't know the header size in IB, but with an MTU of 2K or
> 4K, it's negligible.
> However, things are different for tiny packets. The minimum packet size
> on Ethernet is 60 Bytes. The maximum packet rate (not coalesced !) is
> 14.88 Mpps on a 10GE link, counting everything (inter-packet gap, CRC,
> etc). If you do the math, that's 14.88*60 = 892 MB/s on the link, or 684
> MB/s if you remove the 14B Ethernet header (54% efficiency).
> I don't think you can put all that on an Excel sheet :-)
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
Center for High Performance Computing
University of Utah
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
More information about the Beowulf