[Beowulf] TOE on Linux?

Mark Hahn hahn at mcmaster.ca
Wed May 14 17:20:25 EDT 2008

> application environments.  Contrary to the detractors of the technologies 
> comments, the TOE/RDMA card *did* provide fairly significant performance 
> delta for real apps running MPI over gigabit ethernet.

well, let's not conflate these things: TOE and RDMA are not intrinsically

> I won't talk on the business side of them.  We did see 4x better (wallclock) 
> time on customers StarCD calculations being run over the TOE/RDMA engine than 
> over the pure gigabit path.

do you have a sense for whether this was due to RDMA giving you lower latency
by bypassing the stack (either host or the nic-toe stack)?  I don't think
I've ever heard of TOE itself providing any latency benefit.

also the more appropriate comparison is not a normal mpi-tcp-ip stack, but
rather something like GAMMA or OpenMX.  I don't suppose you had a chance 
to test those, did you?

> Customer was pleased until they realized that 
> the design of this card effectively killed their maximum memory size (3.1 GB 
> vs the 4 GB installed).  This was the iWarp bit.

I'm guessing that techniques from the IB world would ameliorate this...
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf


More information about the Beowulf mailing list