Jeff takes a look at what type of cluster $2500 will buy with today's technology. Back in 2005, 2.5x10E5 pennies bought eight Semprons and 14.5 GFLOPS. How many GFLOPS has two years brought us?

It's always fun to play what-if. What if I looked like Brad Pitt? What if I had lots of money? What if I lost a bunch of weight? What if I looked like Brad Pitt, lost of a bunch of weight, and had lots of money? What if I could spend that money on a new home cluster and build it myself? While worthy of an episode of Jerry Springer I think the last what-if is more appropriate for Cluster Monkey. [Men who spend their kids lunch money on Linux clusters on the next Jerry Springer Show - Ed.]

If you recall, our protagonists, Jeff and Doug, had built an 8 node cluster named Kronos for about $2,500. That was almost 18 months ago. What could we do now with $2,500? While we haven't built anything at this time, let's have a little fun and take a look at what we could build. (and what we might be building real soon now).

What could I do today?

A while ago (about 18 months), Doug Eadline and I wrote a series of articles about building a simple cluster for about $2,500. We chose to name the cluster Kronos for no real particular reason, but basically because we like old Sci-Fi movies that the younger generation can't appreciate. The first article talked about the "parts" list and the construction of the cluster. The second article talked about the software for the cluster. The third article talked about installing some MPI libraries and a job scheduler (SGE). In the final article we talked about some optimization we did to improve our HPL scores. Then Doug followed up with an article about trying to further improve the HPL scores.

The cluster, while a great source of cluster horsepower for us, is getting a bit long in tooth. Now we have dual-core and quad-core CPUs, memory has dropped in price, GigE has dropped in price, and the software is more mature. So I thought I would take a few moments and look at what we could do if we chose to build "Son of Kronos." While I don't have the funds to build such a beast, I think it's worthwhile to see what a good home built cluster would cost these days. (I do have a proposal in to the house management. It is still under consideration.)

Before I launch into showing you the prices let me start with some guidelines that I used in the pricing.

• I'm using AMD Athlon 64 X2 processors. Why? No particular reason. I'm sure I'll get lots of email saying "Intel Core 2 Duo is faster." While that may be true, I just chose to look at AMD for this article (and I have no desire to do a AMD vs. Intel pricing comparison. But if someone wants to fund me to build both systems and then run benchmarks on them I'll be glad to do that).
• I chose to use a disk-less booting system (Warewulf or Percus). Also, I decided not to put any disks in the compute nodes and put all of the storage in the master node using NFS.
• I decided to go with two GigE networks that are jumbo packet capable. One is for computational traffic and the second is for booting the nodes and for storage traffic.
• I wanted to have a decent amount of memory per core so I'm trying to get at least 1 GB of memory per core (2GB per node in the case of dual-core).
• I'm putting a couple of hard drives in the master node to provide storage for the cluster. You can run it as RAID-1 to get a little resiliency in the system or as RAID-0 to get some speed. It's up to you.
• To make my life easier, I chose to just use Newegg for the pricing. While I'm not advertising for them I have found them to have great prices, a wide selection, and good customer service.
• Lastly, the system will have at least one head node and one compute node. But I'll explore how many nodes I can get for various amounts of money.

So with these ground rules, let's start by looking at the head node.

Head Node

The table below lists the items I chose for my head node. I have included links to newegg for the items. The prices are current as of 1/25/2007 but are likely to change. Plus don't be surprised if some of the items disappear from the newegg site (welcome to the wild world of commodity parts!). In this initial configuration, I've chosen to use the 2.2 GHz AMD Athlon 64 X2 4200+ CPU. I thought it had a good price/performance so I decided to use it. Later on, I'll look at other CPUs and the affect on the total pricing.

I looked at various options for the parts and selected them for various reasons. I chose a motherboard with built-in graphics so I didn't have to buy a separate graphics card. Plus the motherboard has a built-in GigE NIC that I can use for booting the nodes and for the storage network. I also added an Intel GigE NIC for the computational traffic (I plan on using GAMMA and it works best with Intel NICs). The second GigE NIC is to connect the head node to the outside world. I chose Seagate hard drives since I've had good luck with them and the performance of the new ones (7200.10) is particularly good. I also put in a DVD burner, because you are likely to need something to load the OS on the machine and you can always use to archive data from the head node. Also notice that I put the OS on a separate disk. I did this so it's easier to RAID the data hard drives.

The hardest decision I had to make was on the case. There were some cheaper cases, but they looked to be more difficult to work with. I chose a case that had at least one 120mm fan in the front and one in the back. My desktop case has 120mm fans in it and they are deadly quiet. Also, I wanted a case that had slots for hard drives that are sideways. That is you can pop off the side of the case and just slide the hard drives in without having to take off the front of the case. The case I chose is larger than the compute node case (in Kronos we used the same case for all of the nodes) and it doesn't look the same, but I value convenience, price, cooling, and the skin on my knuckles more than aesthetics.

Compute Node

Next I want to define the compute nodes. Remember I made these nodes disk-less so I don't need any hard drives in them. To make things easier I will use the same motherboard, CPU, and memory that I chose for the head node. I also put the same Intel NIC in this node for the computational network.

Notice that the case I chose is a Micro-ATX case. I chose to use a Micro-ATX motherboard since I don't need a lot of components on the motherboard and it saves a little bit of money. The case is pretty small: 15" x 14" x 5.3"(HxDxW). It's maybe a little bit bigger than the compute cases we used in Kronos, but that's fine.

Networking

As I mentioned before, I wanted to use two GigE networks. One is for the computational traffic and one is for the booting/storage traffic. The reason that I want two separate networks is that GigE has gotten cheap enough that it's easy to have multiple GigE networks and that I want to minimize any interference on the computational traffic so that I can get the best possible performance. Also, as we showed in Kronos articles, jumbo packets can give you a performance boost. This fact is true for computational traffic and for NFS traffic. So I wanted to find GigE switches that are jumbo packet capable.

I chose 8-port GigE switches for two reasons. They are not much more expensive than 5-port switches but they are much cheaper than 16-port switches. Also, they allow room to grow the system configurations to a total of 8 nodes.

Configurations

Now let's start looking at pricing for various configurations. Let's start with the basic of a single head node and a single compute node (not much fun there).

Just to recap, the system has 4 cores (if you count the head node), and a total of 4 GB of memory. This price doesn't look too swift though. It's about $364.08 per core. Plus we have all of these unused GigE ports hanging around. But the configuration does give you an idea of what you could do as a starting point.

Let's see what we can get for about $2,500 (the original price for Kronos).

So you for about $2,500, which is what we originally spent on Kronos, results in 8 cores and 8 GB of memory. So this is about $310.06 per core (recall that the 2 node system was about $364.08 per core). Here's a quick comparison of the this system versus Kronos:

• CPUs: Current: 8x2.2GHz Athlon 64 cores (64-bit), Kronos: AMD Sempron 2500+ (32-bit)
• Memory: Current: 8GB total (2GB per node), Kronos: 2.3 GB total (512MB head node, 256MB per compute node)
• Interconnect: Current: 2xGigE, Kronos: GigE, FastE

So you can now get a machine with 8x2.2GHz (17.6 GHz total) 64-bit cores with 512KB of L2 cache per core versus 8x1.7GHz (13.6 GHz total) 32-bit cores with 128KB of L2 cache per core. From another perspective you get almost four times as much memory and a better storage network.

Let's keep going and examine a 6 node system (1 head node and 5 compute nodes).

So for the configuration you get 12 cores and 12GB of memory. This works out to be about $292.05 per core.

Finally, let's go all the way and have 8 total nodes: 1 head node and 7 compute nodes. This will use all of the GigE ports on the system. So unless we upgrade the network switches, this is the largest machine in terms of the number of nodes.

So for under $4,500 you get 16 cores and 16GB of memory. Not bad, not bad at all. This works out to about $285.23 per core.