Q: Building a small machine room? Materials/costs/etc.
johnb at quadrics.com
Thu Sep 18 09:39:22 EDT 2003
Just a couple of quick points on raised versus, erm, not-raised floors.
A raised floor is far and away the better idea for a nice, stable system,
IMHO. It's easier to dress off your cables nicely and neatly if you've got a
nice bit of under-floor space to store any excess. If the machine room will
house only clusters that form single rows, then you'd be wasting cash
getting a raised floor, to an extent. If you're going to have multiple,
interconnected rows, though, then the cables'd have to go overhead without
that space. Overheads are at least as difficult to sort out as under-floor
(YMMV - I'm a bit of a shorta**e :) So you're left with one major
difference: PRICE. A raised floor to support that kind of weight is not
inexpensive, I would imagine.
Just my tuppence.
> -----Original Message-----
> From: Robert G. Brown [mailto:rgb at phy.duke.edu]
> Sent: 17 September 2003 19:06
> To: Michael Stein
> Cc: Brian Dobbins; beowulf at beowulf.org
> Subject: Re: Q: Building a small machine room? Materials/costs/etc.
> On Wed, 17 Sep 2003, Michael Stein wrote:
> > Just some additions, good coverage..
> Some good additions to my additions, and they reminded me of still
> > Make sure the electricians know the power you intend to use -- they
> > need to size the circuits *larger* than this by some factor (from
> > code?) -- you can't put a 20 A load on a 20 A breaker.
> Also make sure that they put a distribution panel (or two) IN
> THE SPACE.
> Long runs between panel and receptacle carrying near-peak currents eat
> voltage, which in turn requires still more current at the lower
> voltages. Overwiring (using a higher gauge wire than
> strictly necessary
> for the run length) is better than marginal or underwiring.
> > > Oh, and I'd strongly suggest getting a professional
> architect (one with
> > > experience doing computer machine rooms) to do your
> plans. And avoid
> > > bozo electricians who try e.g. wiring three phase
> circuits with a single
> > > common neutral.
> > I'd strongly suggest that (in addition) you also run the
> numbers yourself:
> > - power
> > - heat flow (and power for it?)
> > just to make sure (these numbers are large for more than a small
> > single digit size cluster and get absolutely huge for 1000 nodes).
> > Discovering a miscalculation here early is no big deal, late it's
> > a disaster...
> Very good advice.
> I also forgot to mention two other important, even critical
> issues and a
> Useful Rule of Thumb.
> a) Thermal kill switch. You may well want the room to be equipped
> with one. This is a thermostatted switch that kills all room power if
> the ambient temperature exceeds some threshold, e.g. 90F. The idea is
> that if AC fails and you don't get down there to shut down nodes fast
> enough, the kill switch kills power all at once rather than permit the
> nodes to overheat to where they physically break (as they will if they
> get hot enough).
> Remember, at 10 KW/rack, four racks is 40 KW all being
> released in your
> itty bitty 15x25' room. The room will go from ambient cool
> to meltdown
> in a matter of minutes if AC fails, and (Murphy's law being
> what it is)
> it WILL FAIL sooner or later.
> b) When working out the AC, remember that in many locations,
> especially ones where it gets cold in the winter, physical
> plant people
> often engage in the unhealthy practice of turning off their
> chillers or
> putting them in a winter standby mode. After all, its bloody cold
> outside! Who needs a chiller!
> You do. Believe me. If the chilled water (or whatever) entering your
> machine room stops being (say) 42F and starts being the standby
> temperature of (say) 70F, the ambient air going into the front of your
> nodes will rapidly go up to (say) 85F or even 90F, trip your kill
> switch or break your nodes because it ALMOST trips and stays hot for
> days, and make your life a living hell as you try to convince the
> physical plant people to turn the air conditioning back on
> and they tell
> you that they aren't budgeted for year round operation and besides it
> will ice up their coils. Been there, done that.
> So be sure to get your physical plant people (or whoever is
> to run your
> actual air conditioning chiller/blower system) to sign a contract
> written in blood and promising you their firstborn child if they do a
> silly little thing like turn the air conditioning off the
> first time the
> outdoor temperature drops to ten below zero or something. Then be
> prepared to firebomb their houses when they do it anyway
> (just kidding,
> just kidding...:-).
> c) Don't forget to budget in the COST OF OPERATION for all of this
> infrastructure when you go with your hat in your hands looking for
> money. The remodelling will be a lot more than you think it
> will be by
> the time you get the 16 or so tons of AC you need, the huge
> blower, the
> chiller lines, and all that harmonically mitigated, line conditioned
> power. But THEN you get to buy the electricity and cooling. A
> reasonable rule of thumb estimate for electrical power is:
> $1 per watt per year of 24x7 operation.
> This includes both the power itself and the cost of the AC required to
> remove that power continuously. Note that it is only an estimate, and
> reality could be lower by a bit or higher by a factor of two,
> on local utility costs. For your presumed 40 KW cluster,
> though, you're
> looking at $40K/year just to keep the thing plugged in and running,
> AFTER paying off the renovation costs and buying all the hardware.
> I didn't address the raised floor issue here or before, by
> the way, but
> yes there are some good things about a raised floor design. For
> example, it is relatively easy to deliver AC and power directly to the
> racks from underneath. On the other hand, the raised floor has to
> support the racks and they are (as noted) likely to be HEAVY and to
> TORQUE on their floor support as well as just press down on it. Also,
> the hidden wiring is an advantage when it doesn't need to be
> worked on,
> then it becomes a small disadvantage relative to more open and
> accessible trays. I suspect it costs more too.
> Somebody who has tried both kinds of floor and prefers raised may be
> able to do better than this.
> Robert G. Brown
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
More information about the Beowulf