Solaris Fire Engine.

Donald Becker becker at
Mon Oct 20 20:08:31 EDT 2003

On Tue, 21 Oct 2003, Bogdan Costescu wrote:
> On Mon, 20 Oct 2003, Donald Becker wrote:
> > The only protocol stack changes that might benefit cluster use are
> > sendfile/zero-copy, and that doesn't apply to most current hardware or
> > typical cluster message passing.
> Has actually somebody tried to use sendfile in MPICH or LAM-MPI ? I 
> planned to do it, but this is somewhere in the middle of my always growing 
> TODO queue... Recipes for how to use it were posted a few times at least 
> on netdev list, so those interested can find them easily.

The trick is to
   memory map a file
   use that memory region as message buffers
   send the message buffers using sendfile()

My belief is that the page locking involved with sendfile() would be too
costly for anything smaller than about 32KB.  While I'm certain that
there are a few MPI applications that use messages that large, they
don't seem to be typical.

> But there are many projects that live outside the official kernel, the 
> Scyld network drivers being one good example. What's wrong with replacing 
> the IP stack with one maintained separately with performance in mind ?
> I agree though that this would mean somebody to take care of it and make 
> sure that it works with newer kernels...

>From my experience trying to keep the network driver interface stable, I
very much doubt that it would be possible to separately maintain a
network protocol stack.  Especially since it would be perceived as
competition with the in-kernel version, which brings out the worst

As a specific example, a few years ago we had cluster performance
patches for the 2.2 kernel.  Even while the 2.3.99 development was going
on, the 2.2 kernel changed too quickly to keep those patches up to date
and tested.

Donald Becker				becker at
Scyld Computing Corporation
914 Bay Ridge Road, Suite 220		Scyld Beowulf cluster system
Annapolis MD 21403			410-990-9993

Beowulf mailing list, Beowulf at
To change your subscription (digest mode or unsubscribe) visit

More information about the Beowulf mailing list