Power Usage and GigE switches - Sounds like Lunch to me!

The Beowulf mailing list provides detailed discussions about issues concerning Linux HPC clusters. In this article I review some postings to the Beowulf list on power usage (always a good topic) from two discussion threads on the beowulf list. I'll also review some discussions about 24-port GigE switches from 2004. Plus a bonus update to the age old question Where can I get a cheap GigE switch?

Max FLOPS/WATTS

Producing the most performance for a given power level is becoming an increasingly important issue. Given that Intel has currently stopped development of more powerful CPU cores because of the huge power consumption (the new Prescott dissipates about 103W of power), this question has taken on new meaning. On February 16 of 2004, Camm Maguire wanted to find the maximum flops (floating point operations per second) to watts ratio for a cluster. Joshua Baker-LePain was the first to reply. He thought the Dell Optiplex SX270 provided a good ratio. These boxes consist of an Intel P4 with Hyperthreading up to 3.2 GHz with on-board GigE, a laptop type hard drive and a simple 150 Watt power supply.

At this point, the discussion broke into two very interesting threads. The first thread began with a reply from Jim Lux, the "space" cluster expert. Jim pointed out that this very topic is very near and dear to his heart because he's designing power constrained systems for space based computations. Jim's first reply began by asking some very good questions. Was Camm interested in the power consumption at the wall plug per node? Or is it the power consumption for the CPU only? Should memory be included? How about networking power consumption? Jim pointed out that the networking part of the power equation is very interesting because you could have a number of low power nodes with a high-speed interconnect that itself uses a fair amount of power. In this case Jim thought it might be better to use a few high power nodes to save on the power for the networking. Jim also pointed out that you could forgo the networking switch and connect the nodes in a torus or cube fashion but this will cost some in CPU performance since the nodes have to route the data.

Andrew Carter responded to Jim's posting with a quick comment. He mentioned that something like the Via Eden/Nehemiah running at 1 GHz uses about 7W of power. He mentioned that the Mars Rovers are using about 5W of power to transmit back to Earth. Jim Lux responded that the power number was pretty accurate but it required a nice cyrogenically cooled receiver on Earth and hundreds of kilowatts to transmit from Earth to Mars. If makes you wonder that if we colonize planets and other bodies in space could we make an interplanetary Grid.

The second thread started with some postings from William Dieter who pointed out a cluster design tool at Aggregate.org that allows you to design based on power. It allows you to input cluster parameters such as memory, hard disk space, etc. and weight the resulting designs based on power consumption. The designs are based on a set of rules for individual parts. Bogdan Costescu responded that the rules could be somewhat simplistic in that reducing the power on the NIC, for example, would drive you to a NIC that increases the load on the CPU. William responded that the rules aren't perfect and there is still some development to be done (volunteers anyone?).

This type of discussion, power vs. performance, is going to become more common as processor power consumption increases dramatically.

Power Consumption for Opterons

The previous discussion made some points about power consumption. Another discussion centering on the Opteron also developed as well. As with many narrow topics, the discussion became much more general providing some interesting ideas and interesting comments.

On March 9, 2004, Trent Piepho asked about power consumption for dual Opterons measured with a "kill-a-watt" type device (you can plug the device into a wall outlet and then plug your node into the device to measure power usage, etc.). There were a number of replies to this posting with some very good numbers. Bill Broadley posted some numbers for a Sun V20z Sunfire (dual 2.2 Ghz Opterons) which had dual SCSI drives and 4 Gigs of memory. Idling the box was measured using 237-249 watts. When running a code called pstream which is related to the Stream benchmark, Bill measured 260-277 watts for one instance of pstream and 265-280 watts for two instances of pstream. The unsinkable Robert Brown posted some numbers for a dual Opteron machine he was testing (dual 242 Opteron) under a load average of 2. He measured 150 watts using his kill-a-watt device. He thought this was quite good and remarked that it was better than a dual Athlon that was just idling. Andrew Wang thought that both sets of results were very good and mentioned that an idling Itanium2 used about 120 watts just for the CPU alone. Mark Hahn also posted that his dual Opteron (dual 240's) used about 250 watts maximum while running two copies of Stream and one copy of Bonnie++. Robert Brown then posted that he tested his dual Opteron again and with a load factor of 3 he measured about 182 watts.

There was some discussion about Roberts numbers because they were much lower than either Bill's or Mark's. Jim Lux jumped in to say that the number sounded reasonable (Jim was taking a break from building his space based Death Star cluster). The 180 watts converts to about 140 watts DC which allows for about 100 watts total for the CPUs, 10 watts or so for the fans, and 30 watts for the board logic and RAM.

Trent Piepho also posted some numbers for various Intel Pentium III and Xeon machines under various types of loads. The dual 2.4 GHz Xeon box was a pretty hefty box with a total of 16 SATA drives, a single IDE drive, 1 GB of memory, CD-ROM, and 6 high-speed case fans (measured at 4.4 watts each). When running RAID and Bonnie++, the box was using about 534 watts (Ouch).