Economics of clusters was Re: top500 list (was: opteron VS Itanium 2)
James.P.Lux at jpl.nasa.gov
Mon Nov 17 17:02:49 EST 2003
>With 1100 dual CPU nodes drawing perhaps 250 Watts apiece, the room
>needs some 275 KW of capacity, maybe 180 20 amp circuits (assuming one
>can drive roughly six nodes per circuit). This costs ballpark estimate
>of $275,000/year just to feed and cool the nodes, more than the racks
>themselves. The capital cost of the circuits, transformers, space
>renovation, and the chillers required to drive this cluster would likely
>add another seven digit number to your estimate and is a lot less
>ignorable than the cost of the racks or network;-)
Just the AC receptacles, boxes, and conduit (along with electricians to
install it) alone will be a significant cost.. For comparison, when my
tract house was built, they charged a flat fee of $50 to add a receptacle;
for putting in conduit, installing a duplex receptacle, pulling the wire,
and attaching it to the distribution panel in an industrial environment,
you could figure about $30-50 in materials and a couple hours in labor (@
$50/hr fully burdened). Just to do some quick back of the enveloping, lets
Say 200 circuits (based on rgb's calculation above), so you're at $30K,
just for the end of the wire. A typical 50 kVA pad mount single phase
transformer runs about $1500-2000, plus about $700 to install it, and you'd
need at least 6, probably more like 9, so that's another $20K. There's also
panels, overcurrent protection, grounding, etc., getting the P.E. to design
the system and sign and seal plans (and we licensed engineers don't come
cheap<grin>). The infrastructure for a job like this would be many hundreds
of thousands of dollars, before you rolled in the first rack of computers.
>Small nuclear power plant optional...
>Now the pizza cost, that can be ignored.
Unless it's a government funded facility, where OMB guidelines (and, more
importantly, instiutional interpretation) say that provision of meals (in
distinction to snacks at a meeting) is verboten (donuts: OK, bagels: NO;
because bagels are food and doughnuts are not)<grin>
>The other thing that always amuses me about clusters like this is the
>Moore's Law effect. They buy it this year, after spending a year
>(easily) preparing the site and building the requisite infrastructure.
>They operate it for three years (spending $2.25 million, say). In the
>meantime, node power at constant cost has increased by a factor of 4.
>If they invested their capital in bonds for those three years (including
>the operating budget), and bought that 4x faster node hardware, they
>would BREAK EVEN on the amount of work they get done by year four, and
>have saved three years operating expenses plus interest in addition to
>the interest on the entire capital amount for three years -- an easy $3+
Unless one gets partial results early on that make the later years of
analysis and computing more efficient. Difficult to quantify, but an
important factor. Also, there is a certain fixed amount of labor for
"fiddling around to get it all to work" that will apply at the beginning of
the computation, and earlier is better, because you're paying with
In fact, here is a great argument for scalable clusters. You can invest in
all the infrastructure up front (because it's generally cheaper to buy
things like buildings all at once) and implement a smaller cluster to get
through the teething pains, and then, as the performance of the hardware
improves, upgrade the cluster along the way.
If you haven't tied the computation inextricably to the particular
implementation, then this may provide a more efficient/optimum use of a
fixed amount of capital.
James Lux, P.E.
Spacecraft Telecommunications Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
More information about the Beowulf