[Beowulf] Newbie Question: Batching Versus Custom RPC
jakob at unthought.net
Tue Feb 24 05:56:45 EST 2004
On Thu, Feb 19, 2004 at 02:03:38PM -0500, Ryan Adams wrote:
> I have a problem that divides nicely (embarrassingly?) into
> parallelizable chunks. Each chunk takes approximately 2 to 5 seconds to
> complete and requires no communication during that time. Essentially
> there is a piece of data, around 500KB that must be processed and a
> result returned. I'd like to process as many of these pieces of data as
> possible. I am considering building a small heterogeneous cluster to do
> this (at home, basically), and am trying to decide exactly how to
> architect the task distribution.
I had the following problem; lots and lots of compile jobs. They take
from a few seconds to a few minutes each.
No batch scheduling system that I tried, was up to the task (simply
waaay too long latency in the scheduling).
> I've been considering rolling my own scheduling system using some kind
> of RPC, but I've been around software development long enough to know
> that it is better to use something off-the-shelf if at all possible.
Maybe you would want to take a quick look at ANTS
ANTS was the solution I developed for the problem I had, and from the
sound of it, I think your problem may be a good fit for ANTS as well.
I've been updating it as of lately, but haven't put new releases on the
web site. If you're interested, I can provide you with the new releases
(featuring krellm2 applet! ;) - but the basic functionality is unchanged
from the old release on the web site.
ANTS specifically schedules jobs very quickly - but it lacks the
advanced features of "real" batch systems (like accounting, gang
scheduling, job restart, etc. etc.).
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
More information about the Beowulf