Article Index

Step 4 - Technical Specifications

So we understand the code(s) that will be used on the cluster and we understand our user base. Hopefully you are testing the codes on cluster with various interconnects, processors, etc. Plus we are investigating what vendors we might select. The next step is the fun one - defining the technical specifications!!

While this sounds like the fun section of this article, I'm afraid I can't write too many details because the exact specifications depend upon the specific code(s), the results of your testing, and your specific company or specific needs. But, I will discuss various aspects of clusters and things to consider. OK, it will be like a laundry list, but at least they will give you some ideas to consider as well as some thing to think about.

Aspects of clusters

  • Processor
    • Integer Computations
    • Floating-point Computations
    • Memory bandwidth
    • Cache (L1 and L2)
    • Power (and heat)
  • Interconnect
  • Storage
    • Disks
    • Storage Network
    • File System
  • Software
    • Cluster Management
    • Compilers
    • MPI
  • Infrastructure
    • Power requirements
    • Weight
    • Size of racks
    • Cooling requirements
  • Support
  • Warranty
and so on. Let's talk about some of these aspects.

Processors

Processors are what people typically fixate upon. Who can blame them? Clusters are for computing and processors do the computing so it's logical to focus on them. However, they are but one part of clusters. But we are all human, or rather geek, so let's talk about processors!

Currently there are two main processor sources - Intel and AMD. Both make processors for the server market. I'm not going to review them in any detail in this article, but rather I'll just make a few comments about them, focusing on setting requirements.

Integer Computations

You may have some codes that are primarily driven by integer computations. These code don't have to be life science codes that do a great deal of pattern searching. I have seen many engineering codes that are heavily driven by integer computations. One way to specify an integer computation capability is by requiring a certain level of integer capability. A good way to do this is by specifying a certain SpecInt2000 level of performance. However, these results are for single core chips. If you have more than one socket or more than one core on a chip (pretty soon this will be the standard) then a better way to specify integer performance is by SpecInt2000_rate. It basically measures the integer performance as you add cores.

Floating-Point Computations

In technical computing people tend to focus on floating-point computations. How do you measure the floating-point performance so you can specify it? One way is by requiring a certain level of floating-point capability. A good way to do this is by specifying a certain SpecFP2000 level of performance. As with the integer performance, this measure of performance is for a single core. For multi-core chips and multi-socket systems, a better way to specify floating-point performance is by, SpecFP2000_rate.

Memory Bandwidth

In my opinion, memory bandwidth is one the most underrated performance measures for clusters. It measures how fast one can move data in and out of memory. Many computational codes need as much memory bandwidth as possible. Fortunately there is a simple benchmark for measuring this performance. The Stream benchmarks measures sustainable memory bandwidth for a simple vector kernel.

However, the Stream benchmark is not perfect. There are other memory bandwidth benchmarks available that will predict worst case memory performance. If possible, you should ask for these benchmarks as well.

Cache (L1 and L2)

Cache is generally important, particularly if the code is what is called "cache-friendly". Some codes that are very dependent on cache size and cache design. For example, structural analysis codes depend heavily on the L1 and L2 cache size and design or performance. You can specify L1 and L2 cache sizes if your codes are cache-friendly, but be careful not to specify cache sizes that aren't realistic.

Power and Heat

Processor power and heat are becoming two of the biggest concerns for cluster users. The reason is that current processors use a great deal of power for computing. The resulting heat has to be cooled in some fashion so the nodes don't overheat. Cooling requires even more power.

If you feel it is appropriate to specify the required power for the CPU and/or the heat produced by the CPU, you certainly can do this. I would caution you to talk to the vendors to specify realistic values.

Interconnect

There are a huge range of choices for cluster interconnects. Everything from simple Fast Ethernet in a single unmanaged switch to complicated network topologies with layers of Infiniband switches are available for clusters. When you right an Technical RFP, what should you do?

I recommend one of two options. The first option is or you to specify exactly what interconnect you want. I don't recommend you specify the hardware in detail, unless you are constrained to use that hardware for some reason. The second option is to ask for RFP responses for several interconnects. However, with each interconnect you should ask for benchmark results with your codes. Then you can do a price/performance analysis of the various interconnects.

One thing I don't recommend is asking for all possible interconnect options. Usually cluster vendors will only recommend a few interconnects, but I have seen customers who have asked for benchmark results for just about all possible interconnects for a wide variety of codes. This causes problem for the vendors forcing them to spend a great deal of time benchmarking and developing quotes.

A reasonable alternative to specifying interconnects is to specify interconnect characteristics that you know work well for your code. For example, you could specify a lower bound on bandwidth, and an upper bound on latency and N/2.

Storage

Storage is such a broad topic that I can't really discuss all of the details and options in the article. One suggestion I can make is to characterize storage by three items; disks, storage networks, and file systems. Then you can either request information on these aspects in the vendor response or you can specify bounds on these aspects.

Disks

You can specify certain types of disks if you like. What I think is a better idea is to specify characteristics you think you need. For example, you could ask for disk speeds, access times, number of platters, or power requirements.

Alternatively, rather than specify disk specific disk characteristics you could also specify which disk manufacturers you do not want. I have used this approach on technical RFPs when I knew of bad disks and did not want to have those disks show up in a cluster response from a vendor.

Storage Network

You are likely to need some sort of single name space file system. Something as simple as NFS or as complicated as a high-speed parallel file system. This file system will have to use a network to allow the nodes access to the data. Which network should it use? Can it share the network with computational (MPI) traffic? If it needs it's own network, what characteristics do you need?

File System

As with interconnects there are a number of options for file systems for cluster. They range from the simple such as NFS to the more complex such as high-speed parallel file systems. You can specify a single file system, but I think it's better to specify the IO requirements you need. What kind of IO speed do you need simultaneously on how many nodes? What kind of metadata access do you need? What size files do you typically work with.

By specifying the IO requirements you allow the vendor to choose a file system that they believe meets your requirements and one that they think will be stable and robust. One important aspect to consider is asking if the vendor has experience with the file system.

Another thing to consider is why do you need one file system? Disks and networks are fairly cheap so why have just one file system? For example, you could use a more resilient file system for your home directories and then something like PVFS2 for a high-speed parallel file system for running your codes.

Software

Software is a bit more nebulous to specify because there are so many options. It's better to specify functionality rather than specifics if you can help it. However, there may be a time when you have to specify a certain software because you have standardized on it or have found that it gives you the best performance.

Cluster Management

Cluster management is a topic likely to invoke lots of argument. Which one is better? Which one is faster? Which one is more stable? Which one does blank? While there are a huge number of cluster management packages available I think it's better to specify functionality rather than a specific package.

If you specify functionality, you need to be specific as possible. For example, you want to state that from the cluster master node you want to be able to shut down the power to a single node, group of nodes, or all nodes with a single command. If you can, be specific as possible to help the vendors develop the best solution for your situation.

Compilers

Compilers, fortunately, are a bit simpler than cluster management software. In the RFP you just need to specify which compiler or compilers you want. Which one(s) you should specify is dependent on your test results. The only advice I can offer is to be sure you specify exactly what you want (just saying I want the Portland Group compilers can lead to problems).

MPI

I think MPI libraries are one of the most important aspects to good performance. Consequently, specifying the correct MPI library or libraries is very important for the RFP.

On the other hand, you can try the opposite approach and let the vendor select the MPI library based on benchmarks. You would specify the benchmarks you want to see run, preferably your own codes, and then let the vendor pick the best one in terms of performance or price/performance. This approach will also help tell you if the vendor is a cluster architect company or just a box company. If they are a true "cluster architect" company, then they will test multiple MPI libraries and show you which one is best in their response. Others will just reach for the standard MPICH1 MPI library and give you the results.

Infrastructure

This is an often neglected aspect in a technical RFP. However, as I will mention, these aspects can often make or break a successful cluster.

Power Requirements

One requirement you can add to your RFP are the power requirements. You can other set an upper bound on the power or you can simply ask the vendor to tell you the power requirements for their proposed system.

I would go beyond a simple number for the power requirements to include the number of type of power connectors. Many times I have seen confusion between the vendor and customer on this issue. You should explicitly ask for a certain number of type of connectors or you should ask the vendor to specify what they propose. All too often I have seen customers not read the proposal and vendors not explicitly tell the customer what kind of connectors they are proposing and how many.

Weight

I've heard stories of customers who have received fully loaded racks and put them on raised floors only to watch them tip over because the supports collapse. I don't know if they are true, but you need to pay attention to the total weight of the system and the weight of each rack. As with the other aspects, you can set a weight limit or you can ask for the total weight of the system to be reported.

One aspect that tends to be ignored in regard to weight is the route to the final location of the cluster. I highly recommend that you walk the delivery route of the cluster to it's final location. Then check with the facilities people as to the weight bearing limit of the floors, elevators, ramps, lifts, etc. along the route.

Size of Racks

In addition to weight, the dimensions of the racks is also important. If the cluster is to be in a raised floor area, then be sure to know the distance from the rack to the ceiling. Many times there are restrictions on the height of racks relative to the ceiling. Also, more fundamentally, be sure the system will fit in the area you intend.

Similar to the discussion on weight, you also need to pay attention to route from the delivery dock to the final cluster location. Be sure to find out the shortest height along this route as well as the minimum width and depth of any turns along the way. I have seen several times where this is no way to get the cluster to the final location without the cluster being totally dismantled. My favorite story was an end user who order a rack, but didn't tell the vendor that it was to go on the second floor, in an office cubical, and there was no elevator.

Cooling Requirements

Along with weight, size, and power, the overall cooling requirements are very important. However, I would not stop with just asking for or specifying the specific total amount of cooling. I would also ask or specify the pressure from any vented tiles (if it's a raised floor machine room). I've seen cases where a machine room has enough cooling but not enough pressure to effectively cool the nodes at the top of a rack.

You have no rights to post comments

Search

Login And Newsletter

Create an account to access exclusive content, comment on articles, and receive our newsletters.

Feedburner


This work is licensed under CC BY-NC-SA 4.0

©2005-2023 Copyright Seagrove LLC, Some rights reserved. Except where otherwise noted, this site is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International. The Cluster Monkey Logo and Monkey Character are Trademarks of Seagrove LLC.