Economics of clusters was Re: top500 list (was: opteron VS Itanium 2)

Jim Lux James.P.Lux at
Mon Nov 17 17:02:49 EST 2003

rgb wrote:

>With 1100 dual CPU nodes drawing perhaps 250 Watts apiece, the room
>needs some 275 KW of capacity, maybe 180 20 amp circuits (assuming one
>can drive roughly six nodes per circuit).  This costs ballpark estimate
>of $275,000/year just to feed and cool the nodes, more than the racks
>themselves.  The capital cost of the circuits, transformers, space
>renovation, and the chillers required to drive this cluster would likely
>add another seven digit number to your estimate and is a lot less
>ignorable than the cost of the racks or network;-)

Just the AC receptacles, boxes, and conduit (along with electricians to 
install it) alone will be a significant cost.. For comparison, when my 
tract house was built, they charged a flat fee of $50 to add a receptacle; 
for putting in conduit, installing a duplex receptacle, pulling the wire, 
and attaching it to the distribution panel in an industrial environment, 
you could figure about $30-50 in materials and a couple hours in labor (@ 
$50/hr fully burdened).  Just to do some quick back of the enveloping, lets 
assume $150/receptacle.

Say 200 circuits (based on rgb's calculation above), so you're at $30K, 
just for the end of the wire.  A typical 50 kVA pad mount single phase 
transformer runs about $1500-2000, plus about $700 to install it, and you'd 
need at least 6, probably more like 9, so that's another $20K. There's also 
panels, overcurrent protection, grounding, etc., getting the P.E. to design 
the system and sign and seal plans (and we licensed engineers don't come 
cheap<grin>). The infrastructure for a job like this would be many hundreds 
of thousands of dollars, before you rolled in the first rack of computers.

>Small nuclear power plant optional...
>Now the pizza cost, that can be ignored.

Unless it's a government funded facility, where OMB guidelines (and, more 
importantly, instiutional interpretation) say that provision of meals (in 
distinction to snacks at a meeting) is verboten (donuts: OK, bagels: NO; 
because bagels are food and doughnuts are not)<grin>

>The other thing that always amuses me about clusters like this is the
>Moore's Law effect.  They buy it this year, after spending a year
>(easily) preparing the site and building the requisite infrastructure.
>They operate it for three years (spending $2.25 million, say).  In the
>meantime, node power at constant cost has increased by a factor of 4.
>If they invested their capital in bonds for those three years (including
>the operating budget), and bought that 4x faster node hardware, they
>would BREAK EVEN on the amount of work they get done by year four, and
>have saved three years operating expenses plus interest in addition to
>the interest on the entire capital amount for three years -- an easy $3+

Unless one gets partial results early on that make the later years of 
analysis and computing more efficient. Difficult to quantify, but an 
important factor.  Also, there is a certain fixed amount of labor for 
"fiddling around to get it all to work" that will apply at the beginning of 
the computation, and earlier is better, because you're paying with 
non-inflated dollars.

In fact, here is a great argument for scalable clusters.  You can invest in 
all the infrastructure up front (because it's generally cheaper to buy 
things like buildings all at once) and implement a smaller cluster to get 
through the teething pains, and then, as the performance of the hardware 
improves, upgrade the cluster along the way.
If you haven't tied the computation inextricably to the particular 
implementation, then this may provide a more efficient/optimum use of a 
fixed amount of capital.

James Lux, P.E.
Spacecraft Telecommunications Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
tel: (818)354-2075
fax: (818)393-6875

Beowulf mailing list, Beowulf at
To change your subscription (digest mode or unsubscribe) visit

More information about the Beowulf mailing list