Q: Building a small machine room? Materials/costs/etc.

Robert G. Brown rgb at phy.duke.edu
Wed Sep 17 14:06:16 EDT 2003

On Wed, 17 Sep 2003, Michael Stein wrote:

> Just some additions, good coverage..

Some good additions to my additions, and they reminded me of still

> Make sure the electricians know the power you intend to use -- they
> need to size the circuits *larger* than this by some factor (from
> code?) -- you can't put a 20 A load on a 20 A breaker.

Also make sure that they put a distribution panel (or two) IN THE SPACE.
Long runs between panel and receptacle carrying near-peak currents eat
voltage, which in turn requires still more current at the lower
voltages.  Overwiring (using a higher gauge wire than strictly necessary
for the run length) is better than marginal or underwiring.

> > Oh, and I'd strongly suggest getting a professional architect (one with
> > experience doing computer machine rooms) to do your plans.  And avoid
> > bozo electricians who try e.g. wiring three phase circuits with a single
> > common neutral.
> I'd strongly suggest that (in addition) you also run the numbers yourself:
> - power
> - heat flow (and power for it?)
> just to make sure (these numbers are large for more than a small
> single digit size cluster and get absolutely huge for 1000 nodes).
> Discovering a miscalculation here early is no big deal, late it's
> a disaster...

Very good advice.

I also forgot to mention two other important, even critical issues and a
Useful Rule of Thumb.

  a) Thermal kill switch.  You may well want the room to be equipped
with one.  This is a thermostatted switch that kills all room power if
the ambient temperature exceeds some threshold, e.g. 90F.  The idea is
that if AC fails and you don't get down there to shut down nodes fast
enough, the kill switch kills power all at once rather than permit the
nodes to overheat to where they physically break (as they will if they
get hot enough).

Remember, at 10 KW/rack, four racks is 40 KW all being released in your
itty bitty 15x25' room.  The room will go from ambient cool to meltdown
in a matter of minutes if AC fails, and (Murphy's law being what it is)
it WILL FAIL sooner or later.

  b) When working out the AC, remember that in many locations,
especially ones where it gets cold in the winter, physical plant people
often engage in the unhealthy practice of turning off their chillers or
putting them in a winter standby mode.  After all, its bloody cold
outside!  Who needs a chiller!

You do.  Believe me.  If the chilled water (or whatever) entering your
machine room stops being (say) 42F and starts being the standby
temperature of (say) 70F, the ambient air going into the front of your
nodes will rapidly go up to (say) 85F or even 90F, trip your kill
switch or break your nodes because it ALMOST trips and stays hot for
days, and make your life a living hell as you try to convince the
physical plant people to turn the air conditioning back on and they tell
you that they aren't budgeted for year round operation and besides it
will ice up their coils.  Been there, done that.

So be sure to get your physical plant people (or whoever is to run your
actual air conditioning chiller/blower system) to sign a contract
written in blood and promising you their firstborn child if they do a
silly little thing like turn the air conditioning off the first time the
outdoor temperature drops to ten below zero or something.  Then be
prepared to firebomb their houses when they do it anyway (just kidding,
just kidding...:-).

  c) Don't forget to budget in the COST OF OPERATION for all of this
infrastructure when you go with your hat in your hands looking for
money.  The remodelling will be a lot more than you think it will be by
the time you get the 16 or so tons of AC you need, the huge blower, the
chiller lines, and all that harmonically mitigated, line conditioned
power.  But THEN you get to buy the electricity and cooling.  A
reasonable rule of thumb estimate for electrical power is:

  $1 per watt per year of 24x7 operation.

This includes both the power itself and the cost of the AC required to
remove that power continuously.  Note that it is only an estimate, and
reality could be lower by a bit or higher by a factor of two, depending
on local utility costs.  For your presumed 40 KW cluster, though, you're
looking at $40K/year just to keep the thing plugged in and running,
AFTER paying off the renovation costs and buying all the hardware.

I didn't address the raised floor issue here or before, by the way, but
yes there are some good things about a raised floor design.  For
example, it is relatively easy to deliver AC and power directly to the
racks from underneath.  On the other hand, the raised floor has to
support the racks and they are (as noted) likely to be HEAVY and to
TORQUE on their floor support as well as just press down on it.  Also,
the hidden wiring is an advantage when it doesn't need to be worked on,
then it becomes a small disadvantage relative to more open and
accessible trays.  I suspect it costs more too.

Somebody who has tried both kinds of floor and prefers raised may be
able to do better than this.


Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu

Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list