<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8">
<META NAME="Generator" CONTENT="MS Exchange Server version 6.5.7638.1">
<TITLE>Re: [Beowulf] Q: IB message rate & large core counts (per node)?</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/plain format -->
<P><FONT SIZE=2>That's correct<BR>
<BR>
<BR>
----- Original Message -----<BR>
From: beowulf-bounces@beowulf.org <beowulf-bounces@beowulf.org><BR>
To: Patrick Geoffray <patrick@myri.com><BR>
Cc: beowulf@beowulf.org <beowulf@beowulf.org><BR>
Sent: Mon Mar 15 16:45:44 2010<BR>
Subject: Re: [Beowulf] Q: IB message rate & large core counts (per node)?<BR>
<BR>
If I understand correctly, 40GbE is 64/66 encoded.<BR>
<BR>
Tom<BR>
<BR>
<BR>
On 3/15/2010 5:30 PM, Patrick Geoffray wrote:<BR>
> On 3/15/2010 5:24 PM, richard.walsh@comcast.net wrote:<BR>
> <BR>
>> to best and worst case). It would be good to add Ethernet to the mix<BR>
>> (1Gb, 10Gb, and 40Gb) as well.<BR>
>> <BR>
> 10 Gb Ethernet uses 8b/10b with a signal rate of 12.5 Gb/s, for a raw<BR>
> bandwidth of 10 Gb/s. I don't know how 1Gb is encoded and 40 Gb/s is<BR>
> still in draft. Last time I looked at 40 Gb/s, it was pretty much four<BR>
> 10 Gb links put together, so I would say 8b/10b with 50 Gb/s signal rate.<BR>
><BR>
> <BR>
>> >There is a similar protocol efficiency at the IB or Ethernet level, but<BR>
>> >the MTU is large enough that it's much smaller compared to PCIe.<BR>
>><BR>
>> Would you estimate less than 1%, 2%, 4% ... ??<BR>
>> <BR>
> It depends on the packet size. For example, 14 Bytes Ethernet header on<BR>
> 1500 Bytes MTU, that's 1%. For Jumbo frames at 9000B MTU, it's much less<BR>
> than that. I don't know the header size in IB, but with an MTU of 2K or<BR>
> 4K, it's negligible.<BR>
><BR>
> However, things are different for tiny packets. The minimum packet size<BR>
> on Ethernet is 60 Bytes. The maximum packet rate (not coalesced !) is<BR>
> 14.88 Mpps on a 10GE link, counting everything (inter-packet gap, CRC,<BR>
> etc). If you do the math, that's 14.88*60 = 892 MB/s on the link, or 684<BR>
> MB/s if you remove the 14B Ethernet header (54% efficiency).<BR>
><BR>
> I don't think you can put all that on an Excel sheet :-)<BR>
><BR>
> Patrick<BR>
> _______________________________________________<BR>
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing<BR>
> To change your subscription (digest mode or unsubscribe) visit <A HREF="http://www.beowulf.org/mailman/listinfo/beowulf">http://www.beowulf.org/mailman/listinfo/beowulf</A><BR>
> <BR>
<BR>
<BR>
--<BR>
--------------------------------------------------------------------<BR>
Tom Ammon<BR>
Network Engineer<BR>
Office: 801.587.0976<BR>
Mobile: 801.674.9273<BR>
<BR>
Center for High Performance Computing<BR>
University of Utah<BR>
<A HREF="http://www.chpc.utah.edu">http://www.chpc.utah.edu</A><BR>
<BR>
_______________________________________________<BR>
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing<BR>
To change your subscription (digest mode or unsubscribe) visit <A HREF="http://www.beowulf.org/mailman/listinfo/beowulf">http://www.beowulf.org/mailman/listinfo/beowulf</A><BR>
</FONT>
</P>
</BODY>
</HTML>