cooling systems

Robert G. Brown rgb at phy.duke.edu
Fri Apr 11 15:52:03 EDT 2003


On Fri, 11 Apr 2003, jbassett wrote:

> This is precisely the point that I am getting at. It seems indirect to me to 
> cool the ambient atmosphere in a room using air conditioners, then expect the 
> heat to distribute itself so that the temperature is at equilibrium throughout 
> the system. It seems entirely more sensible to have a system such that cold 
> air would be directed more precisely at the cpus, then have the exiting flow 
> directed through some sort of exhaust system which would take it to some place 
> that would act as a heat resovoir. In that way you could use the existing 
> airconditioning infrastructure at a facility by distributing the hot exhaust 
> from a cluster into the building. You could even use a venturi on an air duct 
> pipe to keep a vacuum going and redistribute the hot air.

I think that if you worked it all out you'd find that you CAN do this,
AND that it would provide you more precise control of temperatures, AND
that it would cost you a lot MORE than a standard AC/chiller/heat
exchanger with relatively simple but suitable ductwork and fans with the
ability to balance and redirect airflow.  Although I could easily be
wrong, since I don't really know what kind of cluster we're talking
about, or how big, or what kind of space you're trying to put it in.

First of all, I think you're almost certainly mistaken when you say that
you can use existing A/C infrastructure to cool a cluster, at least if
that cluster has more than 16 nodes or so (small clusters you often can,
and many do).  Orindary building AC that is servicing "offices" isn't
really generally engineered to remove more than a couple or three of KW
in a single room-sized space and deliver it back to its heat exchanger
-- the ductwork and delivery/return systems simply aren't adequate to do
more.  Sometimes cool air is delivered in one office and only
exhausted/returned several offices away, passing through several
interoffice vents in between!  

My office gets air through a square foot or two of ductwork.  That air
would have to howl in at ten below zero to cool a big cluster, and the
hot air would have to howl out just as fast.  Worse still, as we found
out the hard way, most physical plants will shut A/C chillers down
altogether in the winter time, or run them on a standby/intermittant
basis.  After all, who needs AC in the winter?  It's COLD outside, isn't
it?  So don't count on building air to be adequate OR reliable, unless
reengineered to be both for your particular needs.

Those needs can be quite variable.  A good sized cluster can consume as
much power as all the offices in a good sized building put together --
tens of KW -- and needs A/C just as much in the winter as in the summer,
unless you figure out a clever way of using the cluster room as a
"furnace" to warm all the offices while cooling the cluster.  To put it
in perspective, the A/C heat exchanger/blower alone in our server room
sits in a unit about two meters cubed in size and sounds like a 747 at
cruising altitude in operation, which is all the time.  Then there is
the actual chiller, which is far away and (fractionally speaking, as it
is shared) just as big.  Its air delivery and return are about a square
meter or more each in cross section before it starts splitting down.

At the moment it is removing a few tens of KW continuously, day and
night, and the ambient room air (delivered in a balanced way from
overhead down to the general fronts of the racks but not ducted right
down into them) hovers between 60 and 70, except in the air columns
right BEHIND the node racks where it is more like 70-80 (about a 15
degree difference between incoming and exiting air).  To work in the
room you need a jacket, unless you're standing behind the racks where
you could work comfortably in shorts and a tee shirt.  This is still a
pretty "small" cluster, too -- around 150 dual CPU nodes, plus sundry
single processor nodes and some servers -- with infrastructure capacity
that might support about twice as many nodes eventually as the room
fills.

How could this ever be managed by an ordinary office AC duct?

Second, look at the costs.  Putting active, directed coolers in each
chassis costs more than just fans (and generates more total heat).
Also, there will ALREADY be a warm air return in the room if it is A/C'd
at all -- your "some sort of exhaust system".  Air, like energy, is
conserved and whatever comes out of the blowers into the room must go
out of the room into the blowers.

In many cases simply directing cold incoming air down the case fronts
(intakes) and permitting the rising warm air off the case backs
(outflows) to get to the ceiling and into the return is sufficient -- a
reasonably stable airflow will set up to balance cool air out against
warm air in.  Note that this is NOT equilibrium -- the air going in at
the front is cool, out at the back warm, and they do NOT mix before the
warm air is exhausted -- it is just a stable pattern of circulation.

This may or may not be good enough -- for us it seems to be working, but
for you it might not.  If you want to do better, all it costs you is
more ductwork, more fans and control systems to ensure that the ducting
does its job of dumping cool air in a balanced way (so it all doesn't
come out of the ducts closest to the blower, leaving none for the back
part of the room) and picking up the outflowing warm air ditto, maybe a
raised floor (since without a raised floor the ductwork will interfere
with access to the nodes, front and back).  The capture of the exhaust
will be particularly tricky as you will no longer be exhausting the hot
air trapped on the ceiling as actively and will need to make sure that
spillover doesn't build up there and anomalously heat the upper part of
the room so that it DOES eventually mix with the ambient air.  

You can also build closed or open racks on raised floors and deliver
cool air directly in at the bottom and remove it at the top in what
amounts to a dedicated cooling chamber or airflow pattern, per rack.
However, as Joel said, folks do all of this -- lots of really big
clusters or server farms have very carefully engineered cooling systems,
and you can find websites where they discuss and illustrate particular
patterns of cool/warm airflow for various designs.

They're just expensive. All of this just costs more money, and you
seemed to be complaining about the (lesser) cost of ordinary A/C or
using A/C at all, not looking for ways of increasing costs still further
with complicated ductwork on top of ordinary A/C.

So the only point I was (and am) making is that NOTHING you do at the
node level gets you around the fundamental problem that leads you to A/C
in the first place -- ensuring that power in = heat out such that input
cooling air (or ambient air at the fan intakes) stays at or below may
75F, ideally much cooler.  There are lots of ways to make this happen
with or without fancy ductwork depending on cluster size and design, but
you MUST make it happen.  If your space DOES have adequate capacity in
building AC and your cluster isn't too huge, then you are lucky -- some
fans and maybe some ductwork and you'll likely be able to fly.  If
you're building a cluster that will draw, say, 4 KW and up, and don't
"happen" to have a space with lots of power and surplus AC capacity and
ductwork that can deliver and return air in a balanced way, (like a
former server room) you're almost certainly going to be looking at some
cluster-specific renovation to provide the required power and cooling.

   rgb

> 
> Joseph Bassett
> 
> 

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu



_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



More information about the Beowulf mailing list