Details, Details

Some other details need attention. It is important that you have motherboards that can boot without keyboards. Also, they should have the ability to reboot from a power loss without hitting the power button. This feature is important if you want to control the power from a plug strip. In addition, Wake-on-Lan will be a very helpful feature allowing the head node to powerdown and power up the worker nodes automatically. Another feature that you may never use, but may come in handy for diagnostics is the ability to boot from a USB floppy.

We would also like the motherboards to have some on-board features like networking and video. Finally, we need a small case has that has front to back airflow. Some cases have side airflow which limits the ability to stack systems close together.

Low Cost DIY Hardware

We have chosen the option to of building our own hardware. This choice has two advantages. First, we will get more for our money because we are providing the sweat equity of assembling the systems. Second, we can pick exactly what we want in the systems. Since the cluster we are designing has only 8 nodes, and the worker nodes are quite sparse the time for building them isn't too great. As any true geek would attest, touching the components, putting them together correctly, seeing the fruits of your labor are things that we relish about building our own systems.

There are many mail order companies that offer very good systems and parts backed with good warranties. There are also a number of websites that track prices of various products across mail order websites. Sites such as pricewatch.com, pricegrabber.com, froogle.com, provide good sources of prices and mail order companies. To compliment these websites, are other websites that allow people to "rate" the mail order companies they buy from. Sites such as resellerratings.com are useful for reading the inputs of users and getting an idea about the quality of the company.

For this article series we have chosen (and tested) the exact components we need. You can of course substitute you own and provided you stick close to our recipe, be successful as well. If you are using your own hardware, you may want to test one or two systems before you hit the buy button for eight nodes.

Our Choices and Rationale

Sidebar Three is detailed list of our hardware choices along with approximate prices. Please note that if you are on a budget, you can easily do a lesser number of nodes. We are proud to say that after some searching and twiddling with the design, we managed to stay within our budget and our design specifications.

Arguably our parts list in the sidebar represents one of many solutions. We realize the design may be optimized further by choosing other hardware, but we had to stop cutting bait and start fishing at some point. To show that we did put some thought into the hardware, we will provide some insights into our design process. Keep in mind, cost, availability, and heat were some of the most important factors guiding our decisions.

The choice for processor was between Sempron and Celeron D. We decided to go with Semprons as they are low cost reincarnations of the Athlon XP processor. We also chose to buy "retail" version of the processor that included an AMD approved cooler.

We chose the ASRock Micro-ATX motherboard based largely on price. As we understand, ASRock motherboards are made by ASUS. The one feature we liked is the ability of the motherboard to shut down if it senses CPU overheating. Clearly intended for the over-clocking crowd, this feature is good to have in the event that a CPU fan decides to stop working. The motherboard also has the ability to do Wake-on-Lan, PXE boot, and supports USB floppy booting. The only drawback is the use of 64MB of main memory for video RAM (The BIOS reserves 64MB of RAM for video). This technique is used on almost all "value" motherboards, so we consider a standard feature. It does have its advantages, because, when you have problems, video output is nice. If you use an AGP video card, as we did in the head node, the BIOS gives back the RAM. One of the other things we give up at this hardware level is option to use ECC memory. There is nothing like living on the edge.

Surprisingly, the case was one of the hardest things to find. The Super Flower model SF-101 was one of the best bargains we have found. The case is small (made for Micro-ATX boards), includes a power supply, can hold two hard drives, a DVD/CD drive, and a floppy. It also has bays for two micro hard drives (the kind used in laptops). The case is well made and is one of the few low cost cases that does not have sharp edges.

Since we are eliminating hard drives from the nodes, we decide to splurge a bit on the head node. We are using two 160 GB low power drives from Hitachi. We will probably use these as a RAID-1 mirror, but you could optionally configure them as a 320 GB RAID-0 device.

As far as the GigE NIC, we would have liked to used the Netgear GA302T, but it seems that these are near their "end-of-life" as they are hard to find. If you recall, in a previous article Eadline benchmarked these NICs and at $32, the performance was quite exceptional (35 us latency, 800 Mbits/second). Because availably of parts is important (See rule three above), we chose the Intel PRO/1000 MT. We'll have more on the performance of this NIC next time.

We chose a low cost AGP card for the head node and invested money in a good DVD reader/burner. The really cool thing about the DVD is that it supports vertical loading so we can orient the head node either vertically or horizontally. We also selected a low cost ($9) NIC for the local network (Remember when FastE NICs cost hundreds of dollars!)

Finally, we chose the FastE Ethernet switch based on cost. For the GigE switch we made sure the SMC 8505T had jumbo frame support. Most low cost switches use the same internal chipsets, so we assume performance will be about the same in this price range.

If we tally up the cost, we find that we are at $2490. There are some extras you may wish to purchase. If you do not have shelf space available, We highly recommend using the wire rack with wheels shown in Figure One. While the bottom row of cases is a tight fit, the rack is an otherwise perfect fit. The wheels are very important for two reasons. First, when running the wires, being able to turn the cluster makes the job much easier. Second, if you are going to want to move the cluster, it is really too heavy to lift on your own.

We also left the power distribution as an option as this will be determined by how you physically locate the systems. We do provide the wiring diagram we used in Figure Four.

Another option that can be useful is a USB floppy. Many people have these already, so we considered it optional. If the USB floppy is plugged in the front or the back of the case, the BIOS will list it in the Boot Menu (press F11 when booting). If you need to run something like Memtest-86 on any of the nodes, this can be real handy.

Lastly, in sticking with rule number two (use only screwdrivers), get some zip strip wire ties. In addition to collecting wires with these, we also use them to fasten things to the shelves.

Building the Head Node

Here is where you roll up you sleeves. If you have built PC before, you will find this process easy. It is recommend that you read though the steps below in any case. If you are a novice, you may want to get a an experience friend involved and consult some of the web sites mentioned in the Resources sidebar. We will provide an overview of the steps needed to build the nodes, but we will not cover things like how to set a jumper on the motherboard or insert DIMMs.

Always make sure you ground yourself before working with electronic equipment. Also, if you new to building PCs, here are the golden rules; don't force anything, read the manuals (like the little book that came with the motherboard), and when working on the hardware always remove power from the case by unplugging the plug at the power supply (do not just use the front panel switch). Finally, proceed at your own risk. We will not take responsibility if you hurt yourself or damage hardware. If you are unsure of the proper way to build a PC, consult a professional.

To assemble head node follow these steps:

1. Push out the back plate that comes with case and insert the one supplied with the ASRock motherboard.
2. Insert the memory and processor before putting the motherboard in the case. The retail processor includes a cooler that has a layer of heat sink paste already applied, make sure you line this up properly with the processor socket. Make sure you connect the processor fan. You will need to set the FSB jumper to 333Mhz. This jumper is item 27 on the motherboard layout pictured on page 3 of manual. Jumper pins 1-2 together on both posts. See page 11 of the manual for more information.
3. Place the gold standoffs in the case so that they line up with the motherboard and screw down the motherboard. Make sure you use the right screws for the standoffs.
4. Put the hard drive(s) and DVD drive in their respective bays. The bays are easily removed by taking out a few screws. If you are using two hard drives place each on a separate controller interface. You may need to buy an extra cable for this, make sure it is an ATA 133 cable. You may want to invest in round IDE cables as well. Obviously, the DVD will need to share a cable with one of the hard drives. You will need to set the drive master and slave settings appropriately.
5. Place the motherboard in the case and line it up in the standoffs and back panel. Once it is lined up, you can screw it down. Connect the large power supply connector to the motherboard. The power supply should not be plugged in at this point.
6. Connect all the cables. There are four types, power, system panel, data cables, and USB/sound front access cables. The power cables are needed for hard drives, the DVD drive, and the case fans (there are three case fans). See the manual (page 14) for the location and pin-outs of the system panel jumper. Connect the Reset, Power, Power LED, HDD LED to this jumper. You can also connect the chassis speaker as well to its jumper. If you plan on using the USB front access panel connect the black USB extension cables to the USB 2.0 connector (page 13). You will also need to connect the data cables. Ignore the audio connectors.
7. Depending on where and when you buy the motherboards you may need to upgrade the BIOS. The USB floppy will come in handy here. You can download the current BIOS from the ASRock Website. If you need a boot disk, you can down load one from the FreeDos site (see Resources Sidebar).
8. When everything is complete, double check your connections, and plug in the power supply. The CPU fan should start when you press the power button on the front of the case. Unplug the power supply and attach a monitor (to the motherboard video connector) and keyboard to see that the system is booting properly (you can enter the BIOS by pressing F2). If the system does not start up, recheck everything, and try again.
9. Add the AGP video and network cards. Once you install the video card, the motherboard video will no longer work. Make sure your connect the monitor to the video card or you will be wondering what happened to the video signal. (Of course we never did that!)

We will talk more about the BIOS next time. Figure Three (large version here shows a picture of the worker and master nodes. As you can see, the worker node has very little in it. Once you have the head node powered up, you may want to attach the USB floppy and run Memtest-86. This program will test the memory and detect any problems with your system. We recommend running MemTest86 for at least an hour on each machine (several hours is better). This step is very important because, it helps assure us that the memory subsystem is working correctly. We have found that mixing DIMMs from various manufactures can cause problems.

Figure Three: Master and worker nodes

Building the Worker Nodes

Once you have completed the Head node, the worker nodes should be easy. Do everything you did for the head node except don't install the DVD, video card, extra FastE NIC, or hard drive (which should be easy because you should not have the parts!). Repeat this process seven times.