Getting Serious: Cluster Infrastructure

Article Index

However, from my own direct experience, this is often not the case. Wiring a computer room differs from ordinary commercial or residential wiring in a number of important ways. Let's review some of those differences, so you can "keep an eye on" the folks who design and install your wiring to make sure that they do NOT "work you over" by installing wiring in ways that will prove to be inadequate -- and then charge you again to come in and make it right.

Note that you may not understand all of the rules below, but your electrical contractor should.

  • Keep all wiring runs back to the distribution panel short, and be sure that the circuits have properly arranged ground and neutral connections. Computers do not like ground loops (significant voltages between the neutral line and local ground). One of the references below is devoted just to the correct rules for grounding circuits used in information technology applications.
  • When wiring power poles with multiple phases, do not share a neutral wire. Each circuit should have its own neutral wire. This requirement is needed because switching power supplies only draw current in the middle third of each half-phase, so that currents on a shared neutral wire of three balanced phases add instead of cancel. The power factor of switching power supplies is not particularly close to unity and they generate significant harmonics at multiples of the basic line frequency, e.g. 180 Hz
  • Similarly be aware that peak currents will be higher than expected based on wattage of the load alone, resulting in more neutral wire heating and lowered nominal circuit capacity. Design for a healthy margin of capacity on each circuit.
  • Consider installing a thermal cut-off so that if the room temperature ever exceeds (say) 90F, all power to the nodes shuts down. Better shut down than burned out.
  • Consider using a Harmonic Distortion Correcting room transformer, especially for new construction. Such a transformer protects your main building supply transformer and dynamically corrects for the distorting properties of switching power supplies. They can save money in several ways (by protecting equipment and making equipment run more efficiently).
  • Depending on your budget and the cost of downtime, consider further power conditioning, surge protection, and uninterruptible power. Be sure uninterruptible power is wired according to codes and marked as such so that fire department personnel can readily shut it down in the event of a fire.

There are guidelines for wiring computer spaces available online, as you might expect, but they aren't horribly easy to find. See the Resources sidebar for the links. Remember, you are your own best advocate here. The beowulf mailing list abounds with stories of people who trusted their electricians to do a proper job of wiring a new computer space, only to learn the hard way that the electricians did it wrong.

Cooling

Computers "like" to breath in air for cooling that is less than 20C (68F). In fact, they really like to breath in ambient air for cooling that is less than 15C/60F, and don't mind if the air is down as cool as 50F/10C as long as it is fairly dry so condensation isn't a problem.

Power used by the cluster is released into the cluster room as heat. All of it. All of the time. The amount of heat can be quite staggering. A loaded rack with 80 CPUs can easily draw 8000 to 10000 watts. That is like running eight to ten room-capacity space heaters all at once in a single room -- the full power capacity of five to seven 20 ampere circuits (probably spread out on eight or more circuits).

If you release this heat into a small room with well-insulated walls, the temperature in that room will go up -- fast! If you release this heat into a large room with poorly insulated walls, the temperature of that room will still go up fast (just not as fast). To prevent the temperature in the room from zipping right on past the "happy zone" below 68F/20C into the "about to cause several hundred thousand dollars worth of damage zone" around 104F/40C (which might take as little as ten or twenty minutes depending on the room size and just how much power is going into it) one has to remove all of power you send in as fast as the computers spit it out as heat.

This task requires air conditioning and cooling capacity -- lots of it. The capacity computation is just like your power computation. You need enough air conditioning capacity to remove all of your power capacity, with a fairly hefty margin of surplus capacity so you can not only keep up with the heat, you can get ahead and keep the space cool.

This isn't all, however. Air conditioning is again a complex issue. You don't want air cold enough to create a block of ice in one end of the room and a pocket of trapped air in the other end of the room (where all of the systems are) hot enough to roast an egg or fry a system. This condition can easily happen if your cold air distribution and warm air returns aren't sensibly arranged. You also don't want your racks arranged so warm exhaust air from one row is used as intake cooling air by the second row and so forth -- each computer cools its CPUs relative to ambient air at the intake side, and feeding it already warm air can cause it to overheat.

This situation again suggests that you have your cluster room air conditioning designed and installed professionally, with special care taken to ensure adequate capacity and proper delivery of cool air where you need it and removal of warm air where you don't. This requirement may well involve extensive ducting either overhead, around the walls, or up through a raised floor if you go that way.

A few warnings: Be sure to guard against condensation -- in a humid environment, the room and ductwork will be cold enough to drip water down into expensive machines if the ductwork isn't appropriately wrapped. Also, having some sort of thermal monitoring and alarm system is very definitely called for. You don't want to discover that your A/C failed by discovering a room full of expensive baked hardware the next day (see also the suggestion for a thermal kill switch on your power above). Finally, beware the tendency in many institutions to turn off chillers and air conditioners in the wintertime. After all, it's cold outside!

Not in the cluster room. In there it is hot hot hot, and if the chiller goes off it will fry in no time, in the middle of winter. Be sure to make arrangements for the cluster room air conditioning system to be online 24x7 all year long!

Conclusion

This initial article is hardly exhaustive. There are lots of elements of cluster room design that we only touched upon, such as locks and security, monitoring, amenities, wiring trays and so forth. There are also style issues that we've avoided -- why it is a good idea to keep things neat and tidy, with cables tied up and in suitable channels, even if it costs a bit more that way. Hopefully we'll address some of these issues in future columns when we get down into the nitty-gritty of building serious clusters.

Hope to see you there! {mosgoogle right}

Sidebar: Resources

Computer Room Wiring

Three Phase Power Source Overloading

ITIC Grounding Guidelines (pdf)

Mirus International: Harmonics and Harmonic Mitigating Transformers

The Mirus International FAQ is very informative on the subject of harmonic distortion by switching power supplies and power for computer rooms in general.

This article was originally published in ClusterWorld Magazine. It has been updated and formatted for the web. If you want to read more about HPC clusters and Linux, you may wish to visit Linux Magazine.

Robert Brown, Ph.D, is has written extensively about Linux clusters. You can find his work and much more on his home page

    Search

    Feedburner

    Login Form

    Share The Bananas


    Creative Commons License
    ©2005-2012 Copyright Seagrove LLC, Some rights reserved. Except where otherwise noted, this site is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 License. The Cluster Monkey Logo and Monkey Character are Trademarks of Seagrove LLC.