[Beowulf] Newbie Question: Batching Versus Custom RPC

[Jakob Oestergaard]

On Thu, Feb 19, 2004 at 02:03:38PM -0500, Ryan Adams wrote:
...
> I have a problem that divides nicely (embarrassingly?) into
> parallelizable chunks.  Each chunk takes approximately 2 to 5 seconds to
> complete and requires no communication during that time.  Essentially
> there is a piece of data, around 500KB that must be processed and a
> result returned.  I'd like to process as many of these pieces of data as
> possible.  I am considering building a small heterogeneous cluster to do
> this (at home, basically), and am trying to decide exactly how to
> architect the task distribution.  

I had the following problem; lots and lots of compile jobs. They take
from a few seconds to a few minutes each.

No batch scheduling system that I tried, was up to the task (simply
waaay too long latency in the scheduling).

...
> I've been considering rolling my own scheduling system using some kind
> of RPC, but I've been around software development long enough to know
> that it is better to use something off-the-shelf if at all possible.

Maybe you would want to take a quick look at ANTS
 http://unthought.net/antsd/

ANTS was the solution I developed for the problem I had, and from the
sound of it, I think your problem may be a good fit for ANTS as well.

I've been updating it as of lately, but haven't put new releases on the
web site.  If you're interested, I can provide you with the new releases
(featuring krellm2 applet! ;) - but the basic functionality is unchanged
from the old release on the web site.

ANTS specifically schedules jobs very quickly - but it lacks the
advanced features of "real" batch systems (like accounting, gang
scheduling, job restart, etc. etc.).

 / jakob

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf