Cluster Documentation Project - User contributions [en]

Cluster Design

2006-02-11T00:35:08Z

Jleidel:

Clusters are all about design. Indeed, clusters are about designing machines
around problems. When designing a cluster the following '''basic''' questions
may be helpful. If you need help with answers, try asking on the [http://www.beowulf.org/mailman/listinfo/beowulf Beowulf Mailing List] or submit a question to the [[Cluster Questions]] section.

'''Consider your budget'''
* Do you need help with software and support? If so, reserve some funds for this part of the project. While clusters can be easy to set up initially, continued operation can be time consuming.
* Do you plan on assembling the cluster yourself? If yes, at what level? Starting from motherboards, cases, memory, processors, etc. can be very time consuming. Buy quality parts.
* If you plan on using a vendor, provide them with a detailed specification and don't forget to run benchmarks.

'''How many and what type of processors do I need?'''
* This depends on your application. How well does you application scale. Have you tested it on a real cluster? If it is a common application, have other run it on a cluster?
*It also depends on how you want to arrange your processors. The introduction dual-core CPUs make this a more interesting question.

'''What Type of Interconnect?'''
*Again, this depends on your application. If you can use Gigabit Ethernet, then you can probably buy more nodes. If you need a faster interconnect, then you may have to sacrifice nodes for interconnects to stay in your budget. Benchmarking your code(s) is important.
* Remember Gigabit Ethernet performance will vary by chip-set, some are better than others.

'''What type of software?'''
* This depends on your needs. There are several preconfigured distributions, but which one is best depends upon your needs.

'''Do I need disk drives on my nodes?'''
* This decision also depends on on your needs. In some cases nodes can be run without hard drives, in others local swap or scratch space is needed.

'''What type of storage?'''
* do you need a parallel file system?
* Will a basic NFS setup be enough? Will it scale to the nodes you need?
* How do I construct a diskless cluster?
** Can I circumvent the local swap dilemma?

There are many other questions that need to be answered, but these should give you a start. The book below (freely available) is also a good place to start.

Books

* [http://www.phy.duke.edu/resources/computing/brahma/Resources/beowulf_book.php Engineering a Beowulf-Style Compute Cluster by Robert G. Brown ] - A freely available book on building/designing Beowulf style clusters.

Cluster Design

2006-02-11T00:33:14Z

Jleidel:

Clusters are all about design. Indeed, clusters are about designing machines
around problems. When designing a cluster the following '''basic''' questions
may be helpful. If you need help with answers, try asking on the [http://www.beowulf.org/mailman/listinfo/beowulf Beowulf Mailing List] or submit a question to the [[Cluster Questions]] section.

'''Consider your budget'''
* Do you need help with software and support? If so, reserve some funds for this part of the project. While clusters can be easy to set up initially, continued operation can be time consuming.
* Do you plan on assembling the cluster yourself? If yes, at what level? Starting from motherboards, cases, memory, processors, etc. can be very time consuming. Buy quality parts.
* If you plan on using a vendor, provide them with a detailed specification and don't forget to run benchmarks.

'''How many and what type of processors do I need?'''
* This depends on your application. How well does you application scale. Have you tested it on a real cluster? If it is a common application, have other run it on a cluster?
*It also depends on how you want to arrange your processors. The introduction dual-core CPUs make this a more interesting question.

'''What Type of Interconnect?'''
*Again, this depends on your application. If you can use Gigabit Ethernet, then you can probably buy more nodes. If you need a faster interconnect, then you may have to sacrifice nodes for interconnects to stay in your budget. Benchmarking your code(s) is important.
* Remember Gigabit Ethernet performance will vary by chip-set, some are better than others.

'''What type of software?'''
* This depends on your needs. There are several preconfigured distributions, but which one is best depends upon your needs.

'''Do I need disk drives on my nodes?'''
* This decision also depends on on your needs. In some cases nodes can be run without
hard drives, in others local swap or scratch space is needed.

'''What type of storage?'''
* do you need a parallel file system?
* Will a basic NFS setup be enough? Will it scale to the nodes you need?
* How do I construct a cluster?

There are many other questions that need to be answered, but these should give you a start. The book below (freely available) is also a good place to start.

Books

* [http://www.phy.duke.edu/resources/computing/brahma/Resources/beowulf_book.php Engineering a Beowulf-Style Compute Cluster by Robert G. Brown ] - A freely available book on building/designing Beowulf style clusters.

Processors

2006-02-11T00:28:39Z

Jleidel:

*[http://www.amd.com AMD]
**Athlon Series
**Opteron Series
*[http://www.intel.com Intel]
**Pentium Series
**Xeon Series
**Itanium Series
*[http://www.openpowerproject.org Power Series (IBM)]

Oil and Gas

2006-02-10T02:23:42Z

Jleidel:

*[http://www.slb.com/content/services/software/reseng/index.asp? Schlumberger ECLIPSE]
*[http://www.lgc.com/landmark/index.htm Landmark Graphics]

Other

2006-02-10T02:19:42Z

Jleidel:

Monitoring
*[http://ganglia.sourceforge.net/ Ganglia]

Processors

2006-02-10T02:17:22Z

Jleidel:

*[http://www.amd.com AMD]
*[http://www.intel.com Intel]
*[http://www.openpowerproject.org Power Series]

Power Supplies

2006-02-10T02:14:56Z

Jleidel:

*[http://www.apc.com APC]

Cluster Benchmarking Packages

2006-02-02T02:19:01Z

Jleidel:

Here are some links to benchmarks:

Benchmark Suites
* [http://icl.cs.utk.edu/hpcc/ HPCC Challange Benchmark] - uses seven tests to measure several different performance parameters.
* [http://www.clustermonkey.net//content/view/38/27/ Beowulf Perforamnce Suite (BPS)] - Example results are [http://clustermonkey.net/download/kronos/bps-logs/ here].
* [http://cmbp.clustermonkey.net/ Cluster Monkey Benchmark Project] - just started. It is based on the BPS.
* [http://www.intel.com/cd/software/products/asmo-na/eng/cluster/clustertoolkit/219848.htm Intel MPI Benchmarks] - freely available.
* [http://perfbase.tigris.org/ perfbase] - a set of front end tools using a PostgreSQL database as backend, which together form a system for the management and analysis of the output of tests and experiments.
* [http://liinwww.ira.uka.de/~skampi/ SKaMPI] - a suite of tests designed to measure the performance of MPI.

File Systems I/O

* [http://www.nus.edu.sg/comcen/svu/publications/hpc_nus/sep_2005/Performance.pdf GPFS vs. NFS evaluation]
* EXT3, EXT3, Reiser, JFS, XFS benchmarks [http://linuxgazette.net/102/piszcz.html Part 1] and [http://linuxgazette.net/122/piszcz.html Part 2]
* [http://www.clustermonkey.net//content/view/62/28/ Benchmarking Parallel File Systems]
* [http://dcs.nac.uci.edu/~strombrg/nfs-test.html nfs-test] is a program that tries a bunch of different rsizes, wsizes, protocols (tcp vs udp) and NFS versions to optimize performance

*[http://www.iozone.org IOZONE] Filesystem Benchmark

Cluster Benchmarking Packages

2006-02-02T02:18:30Z

Jleidel:

Cluster Benchmarking Packages

2006-02-02T02:17:59Z

Jleidel:

Storage

2006-01-28T16:07:30Z

Jleidel:

RAID Definitions
*Level 0 - Striped Disk Array without Fault Tolerance
*Level 1 - Mirroring and Duplexing
*Level 2 - Error-Correcting Coding
*Level 3 - Bit-Interleaved Parity
*Level 4 - Dedicated Parity Drive
*Level 5 - Block Interleaved Distributed Parity
*Level 6 - Independent Data Disks with Double Parity
*Level 0+1 - Mirror of Stripes
*Level 10 - Stripe of Mirrors
Storage Interconnects
*Fibre Channel 2/4
*Infiniband
*Ethernet
**Gigabit
**10 Gigabit
*SCSI
Storage Vendors
*SAN
**[http://www.datadirectnetworks.com DataDirect]
**[http://www.engenio.com/ Engenio]
**[http://www.storagetek.com/ StorageTek]
**[http://www.emc.com EMC]
*NAS
**[http://www.netapp.com NetApp]
*Tape
*Software/Hardware Hybrid
**[http://www.panasas.com/ Panasas]
**[http://www.terrascale.com/ Terrascale]
**[http://www.ibrix.com/ Ibrix]

User:Jleidel

2006-01-28T15:49:07Z

Jleidel:

feel free to email me...

Interconnects

2006-01-28T15:46:55Z

Jleidel:

Ethernet
*Topologies
**[http://aggregate.org/FNN/ Flat Neighborhood Networks]
*Vendors
**[http://www.cisco.com Cisco]
**[http://www.extremenetworks.com/ Extreme Networks]

Myrinet
*[http://www.myricom.com/ Myricom]

Infiniband
*[http://www.openib.org/ OpenIB Alliance]
*[http://www.mellanox.com/ Mellanox]
*[http://www.voltair.com Voltaire]
*[http://www.silverstorm.com/ SilverStorm]
*[http://www.pathscale.com Pathscale]

Quadrics
*[http://www.quadrics.com Quadrics]

Dolphinics
*[http://www.dolphinics.com Dolphinics]

Interconnects

2006-01-28T15:45:19Z

Jleidel:

Ethernet
-Topologies
*[http://aggregate.org/FNN/ Flat Neighborhood Networks]
-Vendors
*[http://www.cisco.com Cisco]
*[http://www.extremenetworks.com/ Extreme Networks]

Myrinet
*[http://www.myricom.com/ Myricom]

Infiniband
*[http://www.openib.org/ OpenIB Alliance]
*[http://www.mellanox.com/ Mellanox]
*[http://www.voltair.com Voltaire]
*[http://www.silverstorm.com/ SilverStorm]
*[http://www.pathscale.com Pathscale]

Quadrics
*[http://www.quadrics.com Quadrics]

Dolphinics
*[http://www.dolphinics.com Dolphinics]

MPI Middleware

2006-01-28T15:36:04Z

Jleidel:

Further information on MPI can be found at the [http://www-unix.mcs.anl.gov/mpi/index.htm MPI] page and at the [http://www.mpi-forum.org/ MPI Forum].

Freely Available MPI versions:

* [http://www.lam-mpi.org/ LAM/MPI]
* [http://www.open-mpi.org/ Open MPI]
* [http://www-unix.mcs.anl.gov/mpi/mpich/ MPICH]
* [http://nowlab.cse.ohio-state.edu/projects/mpi-iba/ MVAPICH - MPI over Infiniband]

Commercial MPI versions:
*[http://www.scali.com/ Scali MPI-SCAMPI]
*[http://www.hp.com/go/mpi HP-MPI]
*[http://www.intel.com/cd/software/products/asmo-na/eng/cluster/mpi/index.htm Intel MPI]

MPI Middleware

2006-01-28T15:33:42Z

Jleidel:

Further information on MPI can be found at the [http://www-unix.mcs.anl.gov/mpi/index.htm MPI] page and at the [http://www.mpi-forum.org/ MPI Forum].

Freely Available MPI versions:

* [http://www.lam-mpi.org/ LAM/MPI]
* [http://www.open-mpi.org/ Open MPI]
* [http://www-unix.mcs.anl.gov/mpi/mpich/ MPICH]

Commercial MPI versions:
*[http://www.scali.com/ Scali MPI-SCAMPI]
*[http://www.hp.com/go/mpi HP-MPI]
*[http://www.intel.com/cd/software/products/asmo-na/eng/cluster/mpi/index.htm Intel MPI]

Libraries

2006-01-28T15:32:58Z

Jleidel:

Open Source Math Libraries:
*[http://math-atlas.sourceforge.net/ ATLAS - Automatically Tuned Linear Algebra System]
*[http://www.netlib.org/blas/ BLAS - Basic Linear Algebra Subprograms]

Commercial Math Libraries:
*[http://www.intel.com/cd/software/products/asmo-na/eng/perflib/219769.htm Intel Math Kernel Library]

MPI Middleware

2006-01-28T15:27:44Z

Jleidel:

Further information on MPI can be found at the [http://www-unix.mcs.anl.gov/mpi/index.htm MPI] page and at the [http://www.mpi-forum.org/ MPI Forum].

Freely Available MPI versions:

* [http://www.lam-mpi.org/ LAM/MPI]
* [http://www.open-mpi.org/ Open MPI]
* [http://www-unix.mcs.anl.gov/mpi/mpich/ MPICH]

Commercial MPI versions:
*[http://www.scali.com/ Scali MPI-SCAMPI]
*[http://www.hp.com/go/mpi HP-MPI]

MPI Middleware

2006-01-28T15:27:30Z

Jleidel:

Compilers

2006-01-28T15:25:26Z

Jleidel:

Standard (non parallel compilers)
*[http://www.intel.com/cd/software/products/asmo-na/eng/compilers/index.htm Intel Compilers]
*[http://www.pgroup.com/ Portland Group Compilers]
*[http://gcc.gnu.org/ GNU Compilers]

Compilers

2006-01-28T15:25:10Z

Jleidel:

Grid Computing

2006-01-28T15:24:50Z

Jleidel:

*[http://www.globus.org Globus Toolkit]

File Systems

2006-01-27T16:08:24Z

Jleidel:

*[http://www.lustre.org/ Lustre]
*[http://www.pvfs.org/ PVFS]
*[http://www.redhat.com/en_us/USA/home/solutions/gfs/ GFS]
*[http://www.panasas.com/ Panasas]
*[http://www.terrascale.com/ Terrascale]
*[http://nfs.sourceforge.net/ NFS]

Distributions

2006-01-27T16:01:59Z

Jleidel:

*[http://www.rocksclusters.org ROCKS]
*[http://www.scyld.com/ Scyld]
*[http://bofh.be/clusterknoppix/ Cluster Knoppix]

Compilers

2006-01-27T15:56:44Z

Jleidel: