[Beowulf] Selection of Beowulf cluster type

Robert G. Brown rgb at phy.duke.edu
Tue Jan 20 09:06:55 EST 2004

On Tue, 20 Jan 2004, Per Lindstrom wrote:

> Until I have learned to manage Debian I will use the 
> "slack-rpm-transfer-tool" and it will be very interesting to benchmark 
> the two distros with each other.
> Thanks a lot for all help and support in his case.

Just to be clear, the issue is not "benchmarking two distros" for
performance.  It is strictly manageability and scalability.

Suppose I want to (re)install a system upstairs in my house.  I walk up
to the system and reboot it (power cycling is OK).  When it boots, a PXE
prompt pops up.  I tell it to boot "linux-install".  The system issues a
query to determine who it is (gets an IP number and various other data
from a DHCP server), retrieves a kernel, and boots into a network-based
install.  The install is fully scripted -- it will even build/rebuild an
MD RAID for me.  Every package I have listed for the host is installed,
its keyboard, mouse, monitor, time base, and more is set up, and in the
post install routine all required account support (/etc/passwd, NFS
mounts, printers, whatever) are set up.  At the end the system reboots
itself into production.  

On a fast network connection and with a fast repository server, the time
required for me to fully (re)install a system can be as little as two or
three minutes, at most perhaps thirty minutes (where I need be in
attendance only for the first thirty seconds).  In fact, I don't NEED to
be in attendance.  By running a script, I can tell grub on an installed
system to boot linux-install by default and reboot; linux-install will
end by putting a new grub in place to go back to the normal default

Once installed, provided that I installed from a yummified server and
installed the yum package (appropriately configured for my network and
repository) the system will wake up sometime in the night, query the
server over anonymous http or ftp to see if there are any packages in
the repository that are more recent (have a more advanced revision
number) than what is installed.  If there are, it grabs the RPMs and
retrieves them (plus any needed dependencies) and updates the system's
installation.  If I'm managing five systems or five thousand systems, to
keep ALL OF THEM CURRENT within 24 hours to the latest patches and
package updates requires only that I drop an RPM onto the repository and
do nothing.  I don't even need root privileges on all those derived
clients -- yum is a client-pull tool, not a root-push tool.

So if RH announces a security bug in apache, what do I care?  I take the
patched RPM (generally released at the same time as the bug announcement
-- security bugs are not tolerated for long in linux) and drop it into
my repository and by the next morning every webserver in an INSTITUTION
scale linux network is once again secure, even if some of the
individuals running linux-based webservers are complete idiots who don't
have the foggiest clue about the need to keep the institution's network
secure by means of regular system maintenance.  The institution still
needs a genuine clueful expert, but it only needs ONE.

yum has lots of other lovely features.  yum will list all packages in
all repositories a system is configured to use, broken down into
"installed" and "available" lists.  yum will install anything that is
installed, and uninstall anything that you want to remove, AND will
augment or prune the dependency tree just the right amount in either
case.  yum always asks before doing things (unless you tell it not to in
e.g. a script).  yum was designed by systems adminstrators for systems
administrators, so it is feature rich precisely where sysadmins need it
to be and conservative in its design throughout.  Some sites that
maintain repositories for particular GPL tools have started to "yummify"
those repositories so that you can literally get updates hot off the
primary site from any network-connected system.

Note that if you replace the "kickstart" step above with whatever
primary installation mode an RPM-based distro favors, the other
RPM-based distros are equally yummifiable.  I happen to like kickstart
because I've used it enough to learn it and because it is sufficiently
feature rich that 95% or better of all systems can be templated to be
installed without human hands (99% or better if you sensibly choose the
hardware and put sane restrictions on the software) but yum actually
CREATES space for new installation methodology.  

For example, one hassle with kickstart is that if you run it over a slow
connection, e.g. DSL, it takes forever to install several GB worth of
software and if you interrupt it at any point it has to start over.
With yum, you can install an ABSOLUTELY MINIMAL system that takes only a
few minutes to install even over DSL, and then "finish" the system with
yum.  Unlike kickstart, yum is perfectly happy to be interrupted and
will just start where it left off until it finishes installing a given
list of packages.

THIS is where slackware failed -- it is package based, but its packages
tended to lag, in some cases significantly, the more aggressively
maintained commercial distros (as did/do Debian's packages, actually).
The thing that originally recommended it was that it could be installed
from floppy disks in "sets", which is precisely why I started with it
back in the mid-90's.  It is still distributed that way, for reasons
that are now somewhat difficult to fathom.  Its tgz packages don't
manage dependencies and dependency issues as well as .deb or .rpm
packages, and it is very, very easy for systems on a slackware-based LAN
to "drift" from their installed configuration and become heterogeneous
and difficult to update and maintain.  Package rebuilding is also much
more of an issue.  It just takes more human TIME to install and maintain
EACH SYSTEM on a LAN using slackware than it does using an RPM-based
distro or Debian.

As far as performance goes, well, performance is generally all but
identical across linuces because they all tend to use the same code base
EXCEPT that the most aggressively maintained distros will typically have
a small edge because their underlying packages are more up to date and
better maintained.  A distro with an openly out of date gcc/glibc might
underperform one with gcc/glibc up to date (with e.g. the newer SSE
instuctions supported), for example.  So for security AND performance,
one typically wants an aggressively maintained distribution, yet one
that doesn't advance so rapidly that it thereby becomes unstable because
it lives at the bleeding edge.  The major commercial linuces try to
tread precisely this line, and that is in fact the primary added value
in what they sell.  

The free linuces have to face the same issue.  Debian recognizes this
internally with stable and unstable versions, where unstable likely has
better performance but requires a certain degree of expertise to be
comfortable with.  I suspect that the fedora project is now RH's "RH
unstable" version, as they will no longer be able to tolerate any
significant degree of instability in their moderately expensive
commercial release.  Caosity and a few others will likely closely track
Fedora in level of relative (in)stability.  I don't know what slackware
does currently -- because of its relatively high degree of aggregation
and relatively weak dependency resolution, I'm guessing that it tends to
be very slowly maintained and advanced -- each release is stable and is
advanced only very incrementally with bugfixes only until the next
release, because of the difficulty in rebuilding just a single node and
its dependencies without unexpected side effects.  In the old days this
was a plague -- every time I'd add something new to a slackware box and
build it, if it required newer libraries than slackware had installed it
was a nightmare to rebuild everything required to make the new package

So the REAL issue is not performance.  It is human time, specifically
your time.  If you do your job well, you'll end up with a network that
damn near takes care of itself, where you do work once and it is
transparently replicated many times throughout your LAN.  Therefore pick
a distro that you are reasonably certain:

  a) Has an aggressive package update schedule, especially wrt security
updates but also performance.

  b) Has intelligent package dependency resolution (in my opinion, this
is rpm-based or debian).  

  c) Similarly, makes it easy to rebuild packages from source.  

  d) Has intelligent package management and maintenance tools, e.g. apt,
yum, urpmi.

  e) Supports the notion of a "package repository", ideally local
repositories that you can maintain yourself and that can provide high
bandwidth controlled load services to your clients WITHIN your LAN.

  f) Supports PXE-driven fully automated installation of one sort or

  g) Has some sort of bug-feedback-resolution mechanism, e.g. bugzilla.

  h) Is aggressively open source and charges FAIR PRICES per client, if
it charges at all.  Alas, IMHO a number of the commercial distros are
smelling large profits and are starting to charge like Microsoft for a
product they don't even own.  You get what you pay for, sure, but be
sure you pay for the VALUE of what you get relative to the absolutely
free linux distros if you go for any of the distros that seek to charge
you per seat.

  i) Is used by enough people that there exists a good online support
forum.  Good "enough" for your level of expertise, really -- I don't
mean to discourage people, especially competent system programmer
people, from choosing caosity or fedora just because they are so new
they don't have as many adopters yet as commercial distros or debian.
Linux is the quintessential "community" product; we all own it, and we
all are responsible for making it work.  Adopt one tiny bit of it and
help out, if you can.

There are probably more features I should add, but I've got to move on.
Busy list today, and also a busy teaching day for rgb (starting in one


Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu

Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list