dealing with lots of sockets (was Re: [Beowulf] automount on high ports)

Greg Lindahl lindahl at
Wed Jul 2 21:05:37 EDT 2008

On Wed, Jul 02, 2008 at 08:48:32PM -0400, Lawrence Stewart wrote:

> Back in 1994, with 90 MHz pentiums, NCSA's httpd was the leading
> webserver with a design that forked a new process for every request.

Apache eventually moved to a model where forked processes handled
several requests serially before eventually dying and being re-forked.
This reduces the fork overhead per request to something reasonable.

Recently there's a threaded version, but that's not the default.

Our web crawler at Blekko is event-driven: the work is divided up into
short subroutines which do non-blocking things, and when blocking is
needed, you return to the "system" indicating what code to execute
when the answer you're waiting for comes back.

This is just event-driven programming inside-out. Works great, too,
because the code is prettier than your typical event-driven code.

Now Legion had pretty code, but the fact that all of the contexts
shared a single stack meant that only the guy at the top of the stack
could execute. But I digress.

-- greg

Beowulf mailing list, Beowulf at
To change your subscription (digest mode or unsubscribe) visit

This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

More information about the Beowulf mailing list