Solaris Fire Engine.

Bogdan Costescu bogdan.costescu at
Tue Oct 21 08:46:57 EDT 2003

On Mon, 20 Oct 2003, Donald Becker wrote:

> My belief is that the page locking involved with sendfile() would be too
> costly for anything smaller than about 32KB.

IIRC, both MPICH and LAM-MPI make the distinction between small and large
messages with the default cutoff being 64KB. So large messages could be
sent this way... I don't know what you meant with "too costly", but small
messsages are not too costly to copy in the stack (normal behaviour)  
especially with increasing cache sizes of today CPUs, while the large ones
(where copying time would be significant) could be sent without the extra
copy in the stack.

> While I'm certain that there are a few MPI applications that use
> messages that large, they don't seem to be typical.

... or might not care that much about the speedup.

> >From my experience trying to keep the network driver interface stable, I
> very much doubt that it would be possible to separately maintain a
> network protocol stack.

Well, it was late last night and probably I haven't chosen the most 
appropriate example... the Scyld network drivers are maintained by one 
person, while my suggestion was more going toward a community project.

> Especially since it would be perceived as competition with the in-kernel
> version, which brings out the worst behavior...

Yeah, political issues - I think that making the intent clear would solve
the problem: there is no competition, it serves a completely different
purpose. And given what you wrote in the previous e-mail about
"feature-driven", who would use it on normal computers when it misses
several "high-profile features" like iptables ? Even more, if it's clear
that it should only be used on local fast networks, several aspects of the
stack can be optimized without fear of breaking very high latency
(satellite) or very low bandwidth (phone modems) connections. But I guess 
that I should stop dreaming :-)

> As a specific example, a few years ago we had cluster performance
> patches for the 2.2 kernel.

Those maintained by Josip Loncaric ? Again it was a one-man show. 

I think that this is exactly the problem: there are small projects
maintained by one person but which depend on the free time or interest of
this person. Given that the clustering had moved from research-only into a
lucrative bussiness and that the software (Linux kernel, MPI libraries,
etc.) evolved quite a lot and the entry barrier into let's say kernel
programing is quite high, it's normal that not many people want to make 
the step. I already expressed about a year ago my oppinion that such 
projects can only be carried forward by companies that benefit from them 
or universities where work from students comes for free. But it seems that 
there are no companies thinking that they can benefit or universities 
where students' work is for free...

Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: Bogdan.Costescu at IWR.Uni-Heidelberg.De

Beowulf mailing list, Beowulf at
To change your subscription (digest mode or unsubscribe) visit

More information about the Beowulf mailing list