[Beowulf] SC|05: will 10Ge beat IB?
Jeffrey B. Layton
laytonjb at charter.net
Wed Nov 23 15:25:14 EST 2005
Greg Lindahl wrote:
>On Wed, Nov 23, 2005 at 10:04:46AM -0500, Douglas Eadline wrote:
>>Maybe not ultra low, but I have recently played with GAMMA
>>(http://www.disi.unige.it/project/gamma/) on some Intel
>>"workstation" GigE NICS. (The current version of GAMMA will only work on
>>Intel GigE chipsets)
>Scali and ParTec also have commercial products that are similar. In
>the ParTec case, they interface at a slightly higher level than the
>low-level driver, so it works with any ethernet device, not just a
>few. Scali is probably similar in this respect.
>This approach does give a better 0-byte ping-pong latency than normal
>TCP over ethernet.
I think Scali's MPI trick is a kernel bypass and their
own packets (not TCP). I just checked their website and
here's what they say:
DET (Direct Ethernet DAPL Support)
DET is a Scali DAT module (DAPL) that runs over any
communication interface with a MAC address (all standard
Ethernet devices). The semantics of DAT enables use of Remote
Direct Memory Access over legacy networks. Scali's DET
module alos enables channel aggregation, i.e. the traffic can
be multiplexed between several legacy adapters (Gbe) for
I'm not sure they are still developing this module since
not too many people use it since they priced it so high
(something like $400+ a node!!). It should increase bandwidth
a bit since the CPU usage is reduce a bit and the latency is
reduced as well.
However, just reducing latency doesn't always help.
As Pathscale has shown the world (well those of us who didn't
know), things like N/2 and message rate are very important
for many codes. It would be interesting to see a Netpip curve
for the Scali stuff, GAMMA, and ParTec vs. a "normal"
GigE curve. Doug - the holiday is coming up :)
More information about the Beowulf