Economics of clusters was Re: top500 list (was: opteron VS Itanium 2)

Robert G. Brown rgb at
Mon Nov 17 19:56:46 EST 2003

On Mon, 17 Nov 2003, Jim Lux wrote:

> Say 200 circuits (based on rgb's calculation above), so you're at $30K, 
> just for the end of the wire.  A typical 50 kVA pad mount single phase 
> transformer runs about $1500-2000, plus about $700 to install it, and you'd 
> need at least 6, probably more like 9, so that's another $20K. There's also 
> panels, overcurrent protection, grounding, etc., getting the P.E. to design 
> the system and sign and seal plans (and we licensed engineers don't come 
> cheap<grin>). The infrastructure for a job like this would be many hundreds 
> of thousands of dollars, before you rolled in the first rack of computers.

Ya.  And the AC might well cost several times the electrical circuits.
And don't forget all the network wiring (as opposed to NICs and
switches).  Lots of pulls, cable trays, maybe raised floor action (this
looks like a high rent cluster likely to have a raised floor design and
custom cabinets).  Infrastructure and renovation costs pretty much
EQUALLED the costs of the first 100+ nodes we moved into our new cluster
space, and their capacity looks like it is many times ours.

> >The other thing that always amuses me about clusters like this is the
> >Moore's Law effect.  They buy it this year, after spending a year
> >(easily) preparing the site and building the requisite infrastructure.
> >They operate it for three years (spending $2.25 million, say).  In the
> >meantime, node power at constant cost has increased by a factor of 4.
> >If they invested their capital in bonds for those three years (including
> >the operating budget), and bought that 4x faster node hardware, they
> >would BREAK EVEN on the amount of work they get done by year four, and
> >have saved three years operating expenses plus interest in addition to
> >the interest on the entire capital amount for three years -- an easy $3+
> >million.
> Unless one gets partial results early on that make the later years of 
> analysis and computing more efficient. Difficult to quantify, but an 
> important factor.  Also, there is a certain fixed amount of labor for 
> "fiddling around to get it all to work" that will apply at the beginning of 
> the computation, and earlier is better, because you're paying with 
> non-inflated dollars.

Oh, yeah, you and Bill are right.  My argument was simplistic and won't
apply in all cases (especially as Bill noted if you're trying to scale a
real parallel computation across all N nodes all at once, and not just
divvying up compute cycles amongst a large number of users none of whom
are running computations that can scale to more than N/4 nodes anyway).

> In fact, here is a great argument for scalable clusters.  You can invest in 
> all the infrastructure up front (because it's generally cheaper to buy 
> things like buildings all at once) and implement a smaller cluster to get 
> through the teething pains, and then, as the performance of the hardware 
> improves, upgrade the cluster along the way.
> If you haven't tied the computation inextricably to the particular 
> implementation, then this may provide a more efficient/optimum use of a 
> fixed amount of capital.

This is my general feeling.  The other point is that for VTech to get
funding for a "supercluster" like this once is a strike of lightning --
$10 million dollar projects don't fall in your lap every day.  However,
in 4-5 years tops, the hardware is going to be aged out (in six years
contemporary computers will have a LOT more memory per node, processors
that are estimatable to be 16x as fast, we might be up to REALLY fast
networks or fast networks might be really cheap -- who knows?).  Some
joker like me will be able to build a cluster in their basement for
$100K and equal its throughput, especially when scaling penalties on
1100 nodes are taken into account.  So they'll have to go BACK to the
well early and often, just like a real supercomputer center, or be
obsoleted out of relevance by Moore's Law.  And if they go back to the
well every year, well, they're adopting a scalable cluster model.

This is the killer -- what exactly will they DO with the cluster that is
worth $7 million, plus the better part of a million a year just to run
it?  Not a whole lot of projects out there that are worth the up-front

It's really a matter of mindset.  I've seen or heard of lots of very
very expensive computers designed and assembled to accomplish some
"really important" computation "really fast" that have been funded by
all sorts of deep pocketed government agencies.  In some of those cases,
building the computer was so difficult that it didn't even get finished
before Moore's Law overtook it at 1/10th the cost using commodity
hardware (anything that takes years to build is at real risk of this).
Worse, a lot of the research funded this way isn't really burning issue
stuff in that the outcome won't change people's lives.  Worth doing,
sure, but not worth spending millions on to get a year or two earlier.
Moore's Law just trundles right along, and now we're spending huge
amounts to reach for teraflops, where a decade ago we were spending huge
amounts to reach for gigaflops and a decade before THAT a megaflop was
awesomely expensive.

Well hell, I do gigaflops at home these days, for a few thousand dollars
total.  In ten more years, Inshallah, I'll be doing teraflops on my
desktop and my personal digital assistant in my shirt pocket will be
doing gigaflops:-).  It really is a matter of waiting or not waiting to
accomplish particular tasks.  The REALLY big iron guys (or REALLY big
cluster guys:-) hate to hear that -- they make a living from their
really big supercomputers that live out on the bleeding edge.  So I'm
not surprised to hear that four out of four reject a scalable approach
in favor of the big project model.  The big science guys hate it too.

Doesn't stop it from being least for some projects.  YMMV,
and I'm not trying to break anybody's dolly;-)


P.S. -- anybody remember the good old days, when you'd have been
arrested and put in jail as a traitor to the American Way if you'd sold
a Russian or Chinese person a Gigaflop-capable computer because they
could use it to Simulate Nuclear Devices?

Developing GHz CPUs sort of put a squeeze on THAT idea, ay?  Especially
with the beowulf model to pursue.  Now beowulfs are being built that
follow the big iron model.  We have met the enemy and it is us...

> James Lux, P.E.
> Spacecraft Telecommunications Section
> Jet Propulsion Laboratory, Mail Stop 161-213
> 4800 Oak Grove Drive
> Pasadena CA 91109
> tel: (818)354-2075
> fax: (818)393-6875

Robert G. Brown	             
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at

Beowulf mailing list, Beowulf at
To change your subscription (digest mode or unsubscribe) visit

More information about the Beowulf mailing list