Running perl scripts and non-mpi programs on scyld

hanzl at hanzl at
Fri Apr 18 07:35:04 EDT 2003

> > SGE is quite nice and opensource batch spooling. Using it with
> > scyld-like cluster for this type of jobs is a bit tricky but quite
> > easy. We just create one 'queue' for every slave node and use node
> > number as a queue name. Then we use 'starter method' script like this:
> >
> >   #/bin/sh
> >   bpsh $QUEUE $*
> I didn't realize it was so easy to use SGE on top of a bproc based system.  

It was super-easy to meet our particular requirements, there might be
little more work if somebody needs more.

> I suppose it would take quite a bit more work to get it integrated to
> the point where you only needed one queue for the entire cluster.

Term "queue" in SGE is rather misleading. In fact there is no queue -
there is just a global set of jobs to be done (this set behaves as
queue if there is nothing else to order jobs) and set of places where
to execute jobs (these places are called "queues" in SGE). In our
solution, we have "one queue for the entire cluster" in the sense of
set of jobs to be done. We have however N queues in the latter sense
but all these "queues" have identical definition and one can just copy
them using script or qmon GUI.

There is however just one sge_execd daemon running (and one 'execution
host' to install - the head host). 

ps -auxf on head node gives something like this:

  \_ sge_shepherd-6889 -bg
  |   \_ /bin/sh sge-bproc-starter-method job_scripts/6889
  |       \_ bpsh 3 job_scripts/6889
  |           \_ [6889]
  |               \_ [te]
  |                   \_ [sh]
  |                       \_ [HERest]
  \_ sge_shepherd-6888 -bg
  |   \_ /bin/sh sge-bproc-starter-method job_scripts/6888
  |       \_ bpsh 8 job_scripts/6888
  |           \_ [6888]
  |               \_ [Ser]
  |                   \_ [sh]
  |                       \_ [HVite]

> So does SGE see the load on the slave node queues?

We blindly schedule fixed number of jobs (1 or 2) per node. It is
however possible to write so called "load sensor script" and see and
use real loads. Creating load sensors is well documented and these
script can use any commands like "bpsh -ap w" or "supermon".

> Does it show the number of processors and total memory in qhost?

No. Maybe one could trick it somehow, we did not care. We use qstat to
get rough per-box information what is going on. Load sensors could
make this more exact and could probably also provide memory
information. But I am happy with just "bpsh -ap free".

In general, SGE is a bit confused because we used just one sge_execd
instead of N. Information collected by sge_execd is incorrect as it
does not run on real execution box. Rest of SGE setup could compensate
for this if somebody cares.

(Or better yet, SGE can be changed to work even better with bproc. I think
bproc systems are important enough and SGE team is nice and responsive
enough for this to happen if we can demonstrate interest and propose sensible
ways of improvements.)


Beowulf mailing list, Beowulf at
To change your subscription (digest mode or unsubscribe) visit

More information about the Beowulf mailing list