[Beowulf] Newbie Question: Batching Versus Custom RPC

Guy Coates gmpc at sanger.ac.uk
Fri Feb 20 13:39:49 EST 2004

> My question is basically this: is 2-5 seconds too small of a job to
> justify a batching system like *PBS or Gridengine?

That workload is do-able with the right queuing system. LSF (don't know
about gridengine off hand) has a concept of "job chunking" for dealing
with short running jobs.

The queuing system batches up a number of jobs (eg 10 or 20) and then
submits them all on one go to the work host where they run sequentially.
This cuts down on the scheduling overhead.

We've just had a user push 250,000 short running jobs though our cluster
this-afternoon using this approach.


Guy Coates

Guy Coates,  Informatics System Group
The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1SA, UK
Tel: +44 (0)1223 834244 ex 7199

Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list