Cluster Documentation Project - User contributions [en]

Talk:Cluster Benchmarking Packages

2006-02-27T06:43:32Z

Shewmaker:

It seems HINT has disappeared?!

It used to be at:
http://www.scl.ameslab.gov/HINT/

Cheers
Steve

I always liked HINT.

A company called [http://www.scl.ameslab.gov/Projects/old/ClusterCookbook/hintrun.html Technology Labs] tried to spin off the Analytical HINT technology, but they went out of business.

It later popped up on at hint.byu.edu, but it's no longer there.

The only version I could find to download off of the web is this [http://www.ima.umn.edu/~coult/hint.tar.gz serial UNIX HINT]

Paul A. Farrell has [http://discov.cs.kent.edu/resources/perf/hint/ serial, threaded, and PVM versions of HINT] on his web site.

I might have a copy of the MPI version somewhere. I'll look.

Andrew Shewmaker

Talk:Cluster Benchmarking Packages

2006-02-24T21:53:26Z

Shewmaker:

It seems HINT has disappeared?!

It used to be at:
http://www.scl.ameslab.gov/HINT/

Cheers
Steve

I always liked HINT.

A company called [http://www.scl.ameslab.gov/Projects/old/ClusterCookbook/hintrun.html Technology Labs] tried to spin off the Analytical HINT technology, but they went out of business.

It later popped up on at hint.byu.edu, but it's no longer there.

The only version I could find to download off of the web is this [http://www.ima.umn.edu/~coult/hint.tar.gz serial UNIX HINT]

I might have a copy of the MPI version somewhere. I'll look.

Andrew Shewmaker

Passwordless SSH (and RSH) Logins

2006-02-23T04:00:10Z

Shewmaker:

'''DISCLAIMER:''' Implementing any of these techniques will allow you to login to the target system without being prompted for a password. In all but one set of circumstances this is a very, very, very BAD thing.

'''OpenSSH Public Key Authentication'''

The info below comes from Robert G Brown (aka RGB) at Duke University email on the Beowulf list.

Now, let's arrange it so that we can login to a remote host (also running sshd) without a password. Let's start by seeing if we can login to the remote host at all, I<with> a password:

<pre>
rgb@lucifer|T:151>ssh lilith
The authenticity of host 'lilith (192.168.1.131)' can't be established.
RSA key fingerprint is 8d:55:10:15:8b:6c:64:65:17:00:a7:84:a3:35:9f:f6.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lilith,192.168.1.131' (RSA) to the list of known hosts.
rgb@lilith's password:
rgb@lilith|T:101>
</pre>

>>

So far, so good. Note that the FIRST time we remotely login, ssh will ask you to verify that the host you are connecting to is really that host. When you answer yes it will save its key fingerprint and use it thereafter to automatically verify that the host is who you think it is. This is one small part of the ssh security benefit. However, we had to enter a password to login. This is no big deal for a single host, but is a BIG deal if you have to do it 1024 times on a big cluster just to get pvm started up!

To avoid this, we use the ssh-keygen command to generate a public/private ssh key pair of our very own:

<pre>
rgb@lucifer|T:104>ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/rgb/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/rgb/.ssh/id_rsa.
Your public key has been saved in /home/rgb/.ssh/id_rsa.pub.
The key fingerprint is: c3:aa:6b:ba:35:57:95:aa:7b:45:48:94:c3:83:81:11
</pre>

>>

This generates a default 1024 bit RSA key; alternatively we could have made a DSA key or increased or decreased the number of bits in the key (decreasing being a Bad Idea). Note that we used a blank passphrase; this will keep ssh from prompting us for a passphrase when we connect.

A more secure option is to use a non-blank passphrase. In this case, you will have to use a couple more ssh tools (once per session).

<pre>
guest@localhost$ ssh-agent $SHELL
guest@localhost$ ssh-add
Enter passphrase for /home/guest/.ssh/id_rsa:
Identity added: /home/guest/.ssh/id_rsa (/home/guest/.ssh/id_rsa)
</pre>

If entering the passphrase once per session is annoying, then you should try [http://www.gentoo.org/proj/en/keychain/index.xml keychain], which will reuse ssh-agents across all of your sessions. The associated IBM developerWorks articles are very nice introductions to openssh public key authentication.

The last step is to create an authorized keys file in your ~/.ssh directory. If your home directory is NFS exported to all the nodes, then you are done; otherwise you'll also need to copy the I<entire .ssh directory> to all the hosts that don't already have it mounted. The following illustrates the steps and a test.

<pre>
rgb@lucifer|T:113>cd .ssh
rgb@lucifer|T:114>ls
id_rsa id_rsa.pub known_hosts
rgb@lucifer|T:115>cp id_rsa.pub authorized_keys
rgb@lucifer|T:116>cd ..
rgb@lucifer|T:118>scp -r .ssh lilith:
rgb@lilith's password:
known_hosts 100% |*****************************| 231 00:00
id_rsa 100% |*****************************| 883 00:00
id_rsa.pub 100% |*****************************| 220 00:00
authorized_keys 100% |*****************************| 220 00:00
rgb@lucifer|T:120>ssh lilith
rgb@lilith|T:101>
</pre>

>>

Note that with the last ssh we logged into lilith with no password!

ssh is really pretty easy to set up this way; if you read the man page(s) you can learn how to generate and add additional authorized keys and do fancier things with it, but many users will need no more than what we've done so far. A warning - it is a good idea to log into each host in your cluster one time after setting it up before proceeding further, to build up the known_hosts file so that you aren't prompted for each host the first time PVM starts up a virtual machine.

[end]

A closing note or two:

When compiling / running MPI 1.2x remember either the "setenv P4_RSHCOMMAND ssh" or run configure with "-rsh=ssh" (it'll whinge that you should use the command line option - ignore it)

Most Linux distro's are setup with some sensible default firewall settings. Remember to modify them so SSH is allowed in '''both''' directions!

'''OpenSSH hostbased authentication'''

A cluster adminstrator may want to save his users the trouble of setting up public keys themselves by enabling [http://www.omega.telia.net/vici/openssh/ hostbased authentication]. Keep in mind that if someone compromises your trusted host(s), then they will have comprimised your entire cluster.

'''RLOGIN and RSH'''

There's a fair degree of implementation difference between distro's. The RedHat flavours and derivatives all seem to be vaguely similar. You always want the non-Kerberos version of the "R" utilites. Rather than producing the whole thing, here's a link to an articles that worked for the author:

http://howto.x-tend.be/openMosixWiki/index.php/Enabling%20RSH/RLOGIN%20without%20a%20password%20%28even%20for%20root%21%29

Here's the upstream URL too: http://howto.x-tend.be/openMosixWiki/index.php/ -> Stick "RSH" in the search box.

In some respects the R utilites are quicker than their chatty SSH cousins. However, this becomes irrelevant after MPI starts up (that by design handles its own comms after the link is established). Also a lot of utilites leave their SSH connection open rather than rsh which they rerun on every request (eg. MOODSS).

Seriously think about *why* you want RSH/RLOGIN. As you can see above, SSH isn't hard and (I'd argue) a bit easier!

Passwordless SSH (and RSH) Logins

2006-02-23T03:48:58Z

Shewmaker:

'''DISCLAIMER:''' Implementing any of these techniques will allow you to login to the target system without being prompted for a password. In all but one set of circumstances this is a very, very, very BAD thing.

'''SSH'''

The info below comes from Robert G Brown (aka RGB) at Duke University email on the Beowulf list.

Now, let's arrange it so that we can login to a remote host (also running sshd) without a password. Let's start by seeing if we can login to the remote host at all, I<with> a password:

rgb@lucifer|T:151>ssh lilith

The authenticity of host 'lilith (192.168.1.131)' can't be established.

RSA key fingerprint is 8d:55:10:15:8b:6c:64:65:17:00:a7:84:a3:35:9f:f6.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'lilith,192.168.1.131' (RSA) to the list of known hosts.

rgb@lilith's password:

rgb@lilith|T:101>

>>

So far, so good. Note that the FIRST time we remotely login, ssh will ask you to verify that the host you are connecting to is really that host. When you answer yes it will save its key fingerprint and use it thereafter to automatically verify that the host is who you think it is. This is one small part of the ssh security benefit. However, we had to enter a password to login. This is no big deal for a single host, but is a BIG deal if you have to do it 1024 times on a big cluster just to get pvm started up!

To avoid this, we use the ssh-keygen command to generate a public/private ssh key pair of our very own:

rgb@lucifer|T:104>ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/rgb/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/rgb/.ssh/id_rsa.

Your public key has been saved in /home/rgb/.ssh/id_rsa.pub.

The key fingerprint is: c3:aa:6b:ba:35:57:95:aa:7b:45:48:94:c3:83:81:11

>>

This generates a default 1024 bit RSA key; alternatively we could have made a DSA key or increased or decreased the number of bits in the key (decreasing being a Bad Idea). Note that we used a blank passphrase; this will keep ssh from prompting us for a passphrase when we connect.

A more secure option is to use a non-blank passphrase. In this case, you will have to use a couple more ssh tools (once per session).

guest@localhost$ ssh-agent $SHELL
guest@localhost$ ssh-add
Enter passphrase for /home/guest/.ssh/id_rsa:
Identity added: /home/guest/.ssh/id_rsa (/home/guest/.ssh/id_rsa)

If entering the passphrase once per session is annoying, then you should try [http://www.gentoo.org/proj/en/keychain/index.xml keychain], which will reuse ssh-agents across all of your sessions. The associated IBM developerWorks articles are very nice introductions to openssh public key authentication.

The last step is to create an authorized keys file in your ~/.ssh directory. If your home directory is NFS exported to all the nodes, then you are done; otherwise you'll also need to copy the I<entire .ssh directory> to all the hosts that don't already have it mounted. The following illustrates the steps and a test.

rgb@lucifer|T:113>cd .ssh

rgb@lucifer|T:114>ls

id_rsa id_rsa.pub known_hosts

rgb@lucifer|T:115>cp id_rsa.pub authorized_keys

rgb@lucifer|T:116>cd ..

rgb@lucifer|T:118>scp -r .ssh lilith:

rgb@lilith's password:

known_hosts 100% |*****************************| 231 00:00

id_rsa 100% |*****************************| 883 00:00

id_rsa.pub 100% |*****************************| 220 00:00

authorized_keys 100% |*****************************| 220 00:00

rgb@lucifer|T:120>ssh lilith

rgb@lilith|T:101>

>>

Note that with the last ssh we logged into lilith with no password!

ssh is really pretty easy to set up this way; if you read the man page(s) you can learn how to generate and add additional authorized keys and do fancier things with it, but many users will need no more than what we've done so far. A warning - it is a good idea to log into each host in your cluster one time after setting it up before proceeding further, to build up the known_hosts file so that you aren't prompted for each host the first time PVM starts up a virtual machine.

[end]

A closing note or two:

When compiling / running MPI 1.2x remember either the "setenv P4_RSHCOMMAND ssh" or run configure with "-rsh=ssh" (it'll whinge that you should use the command line option - ignore it)

Most Linux distro's are setup with some sensible default firewall settings. Remember to modify them so SSH is allowed in '''both''' directions!

'''RLOGIN and RSH'''

There's a fair degree of implementation difference between distro's. The RedHat flavours and derivatives all seem to be vaguely similar. You always want the non-Kerberos version of the "R" utilites. Rather than producing the whole thing, here's a link to an articles that worked for the author:

http://howto.x-tend.be/openMosixWiki/index.php/Enabling%20RSH/RLOGIN%20without%20a%20password%20%28even%20for%20root%21%29

Here's the upstream URL too: http://howto.x-tend.be/openMosixWiki/index.php/ -> Stick "RSH" in the search box.

In some respects the R utilites are quicker than their chatty SSH cousins. However, this becomes irrelevant after MPI starts up (that by design handles its own comms after the link is established). Also a lot of utilites leave their SSH connection open rather than rsh which they rerun on every request (eg. MOODSS).

Seriously think about *why* you want RSH/RLOGIN. As you can see above, SSH isn't hard and (I'd argue) a bit easier!

Care and Feeding of Government Labs: What's Next?

2006-02-23T03:23:15Z

Shewmaker:

==List Of DOE Linux/Cluster Projects==

The US Department Of Energy labs sponsor or are members of many Linux and Cluster projects. Note that there is cooperation as well as competition among the labs.

'''Umbrella Projects'''
* [http://www.clustermatic.org Clustermatic]
* [http://www.llnl.gov/linux/ Linux @ Livermore]
* [http://www.scidac.org SciDAC]
* [http://www.scl.ameslab.gov/projects.html Scalable Computing Laboratory Projects] (Ames and Iowa State University)
* [http://www.csm.ornl.gov/torc/ TORC]

'''Firmware and Kernel level projects'''
* [http://bproc.sf.net BProc]
* [http://www.linuxbios.org LinuxBIOS]
* [http://www.lustre.org Lustre]
* [http://www.openib.org OpenIB]
* [http://www.pvfs.org PVFS]
* [http://swik.net/v9fs v9fs]
* [http://www-unix.mcs.anl.gov/zeptoos/ ZeptoOS]

'''Operating Systems, Distributions, and Configuration Management Tools'''
* [http://www.llnl.gov/linux/conman/conman.html ConMan]
* [http://www.llnl.gov/linux/genders/genders.html Genders]
* [http://www.onesis.org oneSIS]
* [http://oscar.openclustergroup.org OSCAR]
* [http://www.llnl.gov/linux/powerman/powerman.html PowerMan]
* [https://www.scientificlinux.org Scientific Linux]
* [http://www.yaci.org Yet Another Cluster Installer]

'''Programming Tools and Libraries'''
* [http://www-unix.mcs.anl.gov/mpi/mpich/ MPICH]
* [http://www.open-mpi.org Open MPI]
* [http://www.eclipse.org/photran/ Photran] - a Fortran IDE based on Eclipse
* [http://upc.nersc.gov Berkeley Unified Parallel C]

'''Visualization'''
* [http://www.llnl.gov/visit/ VisIt]
* [http://www.paraview.org ParaView]
* [http://www-unix.mcs.anl.gov/perfvis/ Performance Visualization]

Care and Feeding of Government Labs: What's Next?

2006-02-23T03:00:06Z

Shewmaker:

==List Of DOE Linux/Cluster Projects==

The US Department Of Energy labs sponsor or are members of many Linux and Cluster projects.

'''Umbrella Projects'''
* [http://www.clustermatic.org Clustermatic]
* [http://www.llnl.gov/linux/ Linux @ Livermore]
* [http://www.scidac.org SciDAC]
* [http://www.scl.ameslab.gov/projects.html Scalable Computing Laboratory Projects] (Ames Laboratory and Iowa State University)
* [http://www.csm.ornl.gov/torc/ TORC]

'''Firmware and Kernel level projects'''
* [http://bproc.sf.net BProc]
* [http://www.linuxbios.org LinuxBIOS]
* [http://www.lustre.org Lustre]
* [http://www.openib.org OpenIB]
* [http://www.pvfs.org PVFS]
* [http://swik.net/v9fs v9fs]
* [http://www-unix.mcs.anl.gov/zeptoos/ ZeptoOS]

'''Operating Systems, Distributions, and Configuration Management Tools'''
* [http://www.onesis.org oneSIS]
* [http://oscar.openclustergroup.org OSCAR]
* [https://www.scientificlinux.org Scientific Linux]
* [http://www.yaci.org Yet Another Cluster Installer]

'''Programming Tools and Libraries'''
* [http://www-unix.mcs.anl.gov/mpi/mpich/ MPICH]
* [http://www.open-mpi.org Open MPI]
* [http://www.eclipse.org/photran/ Photran] - a Fortran IDE based on Eclipse
* [http://upc.nersc.gov Berkeley Unified Parallel C]

'''Visualization'''
* [http://www.llnl.gov/visit/ VisIt]
* [http://www.paraview.org ParaView]
* [http://www-unix.mcs.anl.gov/perfvis/ Performance Visualization]

Memory

2006-02-21T06:48:01Z

Shewmaker:

==Error Detection And Correction==

A cluster of systems with large amounts of RAM provides system integrators and administrators with an opportunity to become familiar with [http://en.wikipedia.org/wiki/Soft_error Soft Errors].

According to arch/*/kernel/mce.c and arch/*/kernel/traps.c, Linux kernels older than 2.6.16 will either see an uncorrectable bit error as a Machine Check Exception (MCE), print out a message with the DIMM bank, and panic; or as a Non-Maskable Interrupt (NMI) and continue on with a "Dazed" message. An NMI would be seen if MCE panic was disabled with the mce=off kernel boot parameter.

There are new capabilites beginning with the 2.6.16 kernel. The code from the [http://bluesmoke.sourceforge.net EDAC] project was merged into the kernel as optional modules. The modules provide counters for correctable and uncorrectable errors, the ability to reset counters through sysfs, a reset counter - seconds since last reset, etc.

* [http://lwn.net/Articles/168972/ short LWN EDAC writeup]
* [http://lwn.net/Articles/168975/ edac.txt from the 2.6.16 kernel docs]

The Linux EDAC modules support the following memory controllers:

* AMD 76x
* Intel e752x
* Intel e7xxx
* Intel 82860
* Intel D82875P
* Radisys 82600

==I/O on 64-bit Systems With Large Amounts of Memory==

Part of the transition from 32-bit x86 to x86_64 with large amounts of memory involves handling I/O devices that only support 32-bit memory addresses. AMD products include a hardware IOMMU that makes everything work transparently, for the most part. Intel EM64T and IA64 products do not include an IOMMU, so the Linux kernel implements a "software I/O translation buffer". The memory allocated to the swiotlb is made unavailable to normal processes, and some device drivers (such as the proprietary [http://download.nvidia.com/XFree86/Linux-x86_64/1.0-8178/README/appendix-l.html NVIDIA graphics driver]) may require more memory to be reserved in order to operate reliably. See the Linux kernel's documentation for information about the swiotlb and iommu boot parameters. Much of the information summarized in this paragraph was learned from an [http://lwn.net/Articles/91870/ LWN DMA article] by Jonathan Corbet.

Memory

2006-02-21T02:57:10Z

Shewmaker: swiotlb

'''Error Detection And Correction'''

A cluster of systems with large amounts of RAM provides system integrators and administrators with an opportunity to become familiar with [http://en.wikipedia.org/wiki/Soft_error Soft Errors].

According to arch/*/kernel/mce.c and arch/*/kernel/traps.c, Linux kernels older than 2.6.16 will either see an uncorrectable bit error as a Machine Check Exception (MCE), print out a message with the DIMM bank, and panic; or as an NMI and continue on with a "Dazed" message. An NMI would be seen if MCE panic was disabled with the mce=off boot parameter.

There are new capabilites beginning with the 2.6.16 kernel. The code from the [http://bluesmoke.sourceforge.net EDAC] project was merged into the kernel as optional modules. The modules provide counters for correctable and uncorrectable errors, the ability to reset counters through sysfs, a reset counter - seconds since last reset, etc.

* [http://lwn.net/Articles/168972/ short LWN EDAC writeup]
* [http://lwn.net/Articles/168975/ edac.txt from the 2.6.16 kernel docs]

The Linux EDAC modules support the following memory controllers:

* AMD 76x
* Intel e752x
* Intel e7xxx
* Intel 82860
* Intel D82875P
* Radisys 82600

'''I/O on 64-bit Systems With Large Amount of Memory'''

Part of the transition from 32-bit x86 to x86_64 with large amounts of memory involves handling I/O devices that only support 32-bit memory addresses. The AMD chipsets include a hardware IOMMU that makes everything work transparently. Intel EM64T (and IA64) chipsets do not include an IOMMU, so the Linux kernel implements a "software I/O translation buffer". The memory allocated to the swiotlb is made unavailable to normal processes, and some devices (such a NVIDIA graphic cards) may require more memory to be reserved in order to operate reliably. See the Linux kernel's documentation for information about the swiotlb boot parameter. This paragraph is a short summary of part of an [http://lwn.net/Articles/91870/ LWN DMA article] by Jonathan Corbet.

Memory

2006-02-21T02:18:36Z

Shewmaker: