Cluster Documentation Project - User contributions [en]

Passwordless SSH (and RSH) Logins

2006-12-14T23:20:26Z

Steve vmwx:

'''DISCLAIMER:''' Implementing any of these techniques will allow you to login to the target system without being prompted for a password. In all but one set of circumstances this is a very, very, very BAD thing.

'''OpenSSH Public Key Authentication'''

The info below comes from Robert G Brown (aka RGB) at Duke University email on the Beowulf list.

Now, let's arrange it so that we can login to a remote host (also running sshd) without a password. Let's start by seeing if we can login to the remote host at all, I<with> a password:

<pre>
rgb@lucifer|T:151>ssh lilith
The authenticity of host 'lilith (192.168.1.131)' can't be established.
RSA key fingerprint is 8d:55:10:15:8b:6c:64:65:17:00:a7:84:a3:35:9f:f6.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lilith,192.168.1.131' (RSA) to the list of known hosts.
rgb@lilith's password:
rgb@lilith|T:101>
</pre>

>>

So far, so good. Note that the FIRST time we remotely login, ssh will ask you to verify that the host you are connecting to is really that host. When you answer yes it will save its key fingerprint and use it thereafter to automatically verify that the host is who you think it is. This is one small part of the ssh security benefit. However, we had to enter a password to login. This is no big deal for a single host, but is a BIG deal if you have to do it 1024 times on a big cluster just to get pvm started up!

To avoid this, we use the ssh-keygen command to generate a public/private ssh key pair of our very own:

<pre>
rgb@lucifer|T:104>ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/rgb/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/rgb/.ssh/id_rsa.
Your public key has been saved in /home/rgb/.ssh/id_rsa.pub.
The key fingerprint is: c3:aa:6b:ba:35:57:95:aa:7b:45:48:94:c3:83:81:11
</pre>

>>

This generates a default 1024 bit RSA key; alternatively we could have made a DSA key or increased or decreased the number of bits in the key (decreasing being a Bad Idea). Note that we used a blank passphrase; this will keep ssh from prompting us for a passphrase when we connect.

A more secure option is to use a non-blank passphrase. In this case, you will have to use a couple more ssh tools (once per session).

<pre>
guest@localhost$ ssh-agent $SHELL
guest@localhost$ ssh-add
Enter passphrase for /home/guest/.ssh/id_rsa:
Identity added: /home/guest/.ssh/id_rsa (/home/guest/.ssh/id_rsa)
</pre>

If entering the passphrase once per session is annoying, then you should try [http://www.gentoo.org/proj/en/keychain/index.xml keychain], which will reuse ssh-agents across all of your sessions. The associated IBM developerWorks articles are very nice introductions to openssh public key authentication.

The last step is to create an authorized keys file in your ~/.ssh directory. If your home directory is NFS exported to all the nodes, then you are done; otherwise you'll also need to copy the I<entire .ssh directory> to all the hosts that don't already have it mounted. The following illustrates the steps and a test.

<pre>
rgb@lucifer|T:113>cd .ssh
rgb@lucifer|T:114>ls
id_rsa id_rsa.pub known_hosts
rgb@lucifer|T:115>cp id_rsa.pub authorized_keys
rgb@lucifer|T:116>cd ..
rgb@lucifer|T:118>scp -r .ssh lilith:
rgb@lilith's password:
known_hosts 100% |*****************************| 231 00:00
id_rsa 100% |*****************************| 883 00:00
id_rsa.pub 100% |*****************************| 220 00:00
authorized_keys 100% |*****************************| 220 00:00
rgb@lucifer|T:120>ssh lilith
rgb@lilith|T:101>
</pre>

>>

Note that with the last ssh we logged into lilith with no password!

ssh is really pretty easy to set up this way; if you read the man page(s) you can learn how to generate and add additional authorized keys and do fancier things with it, but many users will need no more than what we've done so far. A warning - it is a good idea to log into each host in your cluster one time after setting it up before proceeding further, to build up the known_hosts file so that you aren't prompted for each host the first time PVM starts up a virtual machine.

[end]

A closing note or two:

For RedHat and derivatives, the permissions are quite fussy:

chmod 644 ~/.ssh/auth*

chmod 755 ~/.ssh

When compiling / running MPI 1.2x remember either the "setenv P4_RSHCOMMAND ssh" or run configure with "-rsh=ssh" (it'll whinge that you should use the command line option - ignore it)

Most Linux distro's are setup with some sensible default firewall settings. Remember to modify them so SSH is allowed in '''both''' directions!

'''OpenSSH hostbased authentication'''

A cluster adminstrator may want to save his users the trouble of setting up public keys themselves by enabling [http://www.omega.telia.net/vici/openssh/ hostbased authentication]. Keep in mind that if someone compromises your trusted host(s), then they will have comprimised your entire cluster.

'''RLOGIN and RSH'''

There's a fair degree of implementation difference between distro's. The RedHat flavours and derivatives all seem to be vaguely similar. You always want the non-Kerberos version of the "R" utilites. Rather than producing the whole thing, here's a link to an articles that worked for the author:

http://howto.x-tend.be/openMosixWiki/index.php/Enabling%20RSH/RLOGIN%20without%20a%20password%20%28even%20for%20root%21%29

Here's the upstream URL too: http://howto.x-tend.be/openMosixWiki/index.php/ -> Stick "RSH" in the search box.

In some respects the R utilites are quicker than their chatty SSH cousins. However, this becomes irrelevant after MPI starts up (that by design handles its own comms after the link is established). Also a lot of utilites leave their SSH connection open rather than rsh which they rerun on every request (eg. MOODSS).

Seriously think about *why* you want RSH/RLOGIN. As you can see above, SSH isn't hard and (I'd argue) a bit easier!

Cluster Questions

2006-02-22T23:51:44Z

Steve vmwx:

===Enter Your Questions Here===

===When it Gets Answered It Will Go Here===

'''How do I enter a question?'''

[http://agenda.clustermonkey.net/index.php/Special:Userlogin Register] and then return to this page. Click "Edit This Page" at the bottom of the page and enter you question like the one above.

Passwordless SSH (and RSH) Logins

2006-02-22T03:49:45Z

Steve vmwx:

'''DISCLAIMER:''' Implemeting any of these techniques will allow you to login to the target system without being prompted for a password. In all but one set of circumstances this is a very, very, very BAD thing.

'''SSH'''

The info below comes from Robert G Brown (aka RGB) at Duke University email on the Beowulf list.

Now, let's arrange it so that we can login to a remote host (also running sshd) without a password. Let's start by seeing if we can login to the remote host at all, I<with> a password:

rgb@lucifer|T:151>ssh lilith

The authenticity of host 'lilith (192.168.1.131)' can't be established.

RSA key fingerprint is 8d:55:10:15:8b:6c:64:65:17:00:a7:84:a3:35:9f:f6.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'lilith,192.168.1.131' (RSA) to the list of known hosts.

rgb@lilith's password:

rgb@lilith|T:101>

>>

So far, so good. Note that the FIRST time we remotely login, ssh will ask you to verify that the host you are connecting to is really that host. When you answer yes it will save its key fingerprint and use it thereafter to automatically verify that the host is who you think it is. This is one small part of the ssh security benefit. However, we had to enter a password to login. This is no big deal for a single host, but is a BIG deal if you have to do it 1024 times on a big cluster just to get pvm started up!

To avoid this, we use the ssh-keygen command to generate a public/private ssh key pair of our very own:

rgb@lucifer|T:104>ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/rgb/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/rgb/.ssh/id_rsa.

Your public key has been saved in /home/rgb/.ssh/id_rsa.pub.

The key fingerprint is: c3:aa:6b:ba:35:57:95:aa:7b:45:48:94:c3:83:81:11

>>

This generates a default 1024 bit RSA key; alternatively we could have made a DSA key or increased or decreased the number of bits in the key (decreasing being a Bad Idea). Note that we used a blank passphrase; this will keep ssh from prompting us for a passphrase when we connect.

The last step is to create an authorized keys file in your ~/.ssh directory. If your home directory is NFS exported to all the nodes, then you are done; otherwise you'll also need to copy the I<entire .ssh directory> to all the hosts that don't already have it mounted. The following illustrates the steps and a test.

rgb@lucifer|T:113>cd .ssh

rgb@lucifer|T:114>ls

id_rsa id_rsa.pub known_hosts

rgb@lucifer|T:115>cp id_rsa.pub authorized_keys

rgb@lucifer|T:116>cd ..

rgb@lucifer|T:118>scp -r .ssh lilith:

rgb@lilith's password:

known_hosts 100% |*****************************| 231 00:00

id_rsa 100% |*****************************| 883 00:00

id_rsa.pub 100% |*****************************| 220 00:00

authorized_keys 100% |*****************************| 220 00:00

rgb@lucifer|T:120>ssh lilith

rgb@lilith|T:101>

>>

Note that with the last ssh we logged into lilith with no password!

ssh is really pretty easy to set up this way; if you read the man page(s) you can learn how to generate and add additional authorized keys and do fancier things with it, but many users will need no more than what we've done so far. A warning - it is a good idea to log into each host in your cluster one time after setting it up before proceeding further, to build up the known_hosts file so that you aren't prompted for each host the first time PVM starts up a virtual machine.

[end]

A closing note or two:

When compiling / running MPI 1.2x remember either the "setenv P4_RSHCOMMAND ssh" or run configure with "-rsh=ssh" (it'll whinge that you should use the command line option - ignore it)

Most Linux distro's are setup with some sensible default firewall settings. Remember to modify them so SSH is allowed in '''both''' directions!

'''RLOGIN and RSH'''

There's a fair degree of implementation difference between distro's. The RedHat flavours and derivatives all seem to be vaguely similar. You always want the non-Kerberos version of the "R" utilites. Rather than producing the whole thing, here's a link to an articles that worked for the author:

http://howto.x-tend.be/openMosixWiki/index.php/Enabling%20RSH/RLOGIN%20without%20a%20password%20%28even%20for%20root%21%29

Here's the upstream URL too: http://howto.x-tend.be/openMosixWiki/index.php/ -> Stick "RSH" in the search box.

In some respects the R utilites are quicker than their chatty SSH cousins. However, this becomes irrelevant after MPI starts up (that by design handles its own comms after the link is established). Also a lot of utilites leave their SSH connection open rather than rsh which they rerun on every request (eg. MOODSS).

Seriously think about *why* you want RSH/RLOGIN. As you can see above, SSH isn't hard and (I'd argue) a bit easier!

Passwordless SSH (and RSH) Logins

2006-02-22T03:45:44Z

Steve vmwx:

Limiting MPI (1.2x) traffic to one NIC

2006-02-22T03:36:54Z

Steve vmwx:

As an example, say you want to run your nodes with a FastEther port for the admin and reporting functions but leave the GigaEther for the MPI traffic.

IP addresses are setup accordingly and they have their own host names. eg. borg1 is the GigaEther (eth1) while borg1_m (eth0) is the "management" interface. Now, if you don't tell MPI that there's a difference then it's going reach out and touch the machines with eth0.

To limit the MPI traffic to a specific interface you need to tell it which interface to use. A bit if string work in your mpirun.args file does the trick.

In this case:

MPI_HOST=`hostname | sed "s/_m//"`

This slices the "_m" off the end of the hostname.

(thanks to Reuti at the Uni of Marburg, Germany from the Beowulf email list)

Limiting MPI (1.2x) traffic to one NIC

2006-02-22T03:35:22Z

Steve vmwx:

As an example, say you run your nodes with a FastEther port for the admin and reporting functions and leave the GigaEther for the MPI traffic.

IP addresses are setup accordingly and they have their own host names. eg. borg1 is the GigaEther (eth1) while borg1_m (eth0) is the "management" interface. Now, if you don't tell MPI that there's a difference then it's going reach out and touch the machines with eth0.

To limit the MPI traffic to a specific interface you need to tell it which interface to use. A bit if string work in your mpirun.args file does the trick.

In this case:

MPI_HOST=`hostname | sed "s/_m//"`

This slices the "_m" off the end of the hostname.

(thanks to Reuti at the Uni of Marburg, Germany from the Beowulf email list)

Tuning Intel e1000 NICs

2006-02-22T03:30:02Z

Steve vmwx:

Switch off Interupt Throttle Rate (ITR) when loading the module. Try adding the following line to your modules.conf (or modprobe.conf if 2.6):

options e1000 InterruptThrottleRate=0,0

Use 0,0,0,0 of 2x NICs

(Thanks to Peter Kjellstrom at the Nation Computing Centre, Sweden - from the Beowulf email list)

All the magic options and generally good support info from Intel here:
http://support.intel.com/support/network/sb/CS-009209.htm

If you have e1000 cards you might also check out the GAMMA project for a non-TCP/IP stack (recently updated with 2.6.x kernel and amd64 x86-64 support):
http://www.disi.unige.it/project/gamma/

Tuning Intel e1000 NICs

2006-02-22T03:26:34Z

Steve vmwx:

Switch off Interupt Throttle Rate (ITR) when loading the module. Try adding the following line to your modules.conf (or modprobe.conf if 2.6):

options e1000 InterruptThrottleRate=0,0

Use 0,0,0,0 of 2x NICs

(Thanks to Peter Kjellstrom at the Nation Computing Centre, Sweden - from the Beowulf email list)

All the magic options and generally good support info from Intel here:
http://support.intel.com/support/network/sb/CS-009209.htm

If you want to go exotic, check out the Gamma project for a non-TCP/IP stack (recently updated with 2.6.x kernel and amd64 x86-64 support):
http://www.disi.unige.it/project/gamma/

Cluster How-To's

2006-02-22T03:21:45Z

Steve vmwx:

* [[Passwordless SSH (and RSH) Logins]]
* [[Tuning Intel e1000 NICs]]
* [[Limiting MPI (1.2x) traffic to one NIC]]

Cluster How-To's

2006-02-22T03:20:10Z

Steve vmwx:

* [[Passwordless SSH (and RSH) Logins]]

Passwordless SSH Logins

2006-02-22T03:17:23Z

Steve vmwx:

'''DISCLAIMER''': Implemeting any of these techniques will allow you to login to the target system without being prompted for a password. In all but one set of circumstances this is a very, very, very '''BAD''' thing.

'''SSH'''

The info below comes from an RGB (Robert G Brown) at Duke Uni, USA, posting on the Beowulf list.

Now, let's arrange it so that we can login to a remote host (also running sshd) without a password. Let's start by seeing if we can login to the remote host at all, I<with> a password:

rgb@lucifer|T:151>ssh lilith

The authenticity of host 'lilith (192.168.1.131)' can't be established.

RSA key fingerprint is 8d:55:10:15:8b:6c:64:65:17:00:a7:84:a3:35:9f:f6.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'lilith,192.168.1.131' (RSA) to the list of known hosts.

rgb@lilith's password:

rgb@lilith|T:101>

>>

So far, so good. Note that the FIRST time we remotely login, ssh will ask you to verify that the host you are connecting to is really that host. When you answer yes it will save its key fingerprint and use it thereafter to automatically verify that the host is who you think it is. This is one small part of the ssh security benefit. However, we had to enter a password to login. This is no big deal for a single host, but is a BIG deal if you have to do it 1024 times on a big cluster just to get pvm started up!

To avoid this, we use the ssh-keygen command to generate a public/private ssh key pair of our very own:

rgb@lucifer|T:104>ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/rgb/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/rgb/.ssh/id_rsa.

Your public key has been saved in /home/rgb/.ssh/id_rsa.pub.

The key fingerprint is: c3:aa:6b:ba:35:57:95:aa:7b:45:48:94:c3:83:81:11

>>

This generates a default 1024 bit RSA key; alternatively we could have made a DSA key or increased or decreased the number of bits in the key (decreasing being a Bad Idea). Note that we used a blank passphrase; this will keep ssh from prompting us for a passphrase when we connect.

The last step is to create an authorized keys file in your ~/.ssh directory. If your home directory is NFS exported to all the nodes, then you are done; otherwise you'll also need to copy the I<entire .ssh directory> to all the hosts that don't already have it mounted. The following illustrates the steps and a test.

rgb@lucifer|T:113>cd .ssh

rgb@lucifer|T:114>ls

id_rsa id_rsa.pub known_hosts

rgb@lucifer|T:115>cp id_rsa.pub authorized_keys

rgb@lucifer|T:116>cd ..

rgb@lucifer|T:118>scp -r .ssh lilith:

rgb@lilith's password:

known_hosts 100% |*****************************| 231 00:00

id_rsa 100% |*****************************| 883 00:00

id_rsa.pub 100% |*****************************| 220 00:00

authorized_keys 100% |*****************************| 220 00:00

rgb@lucifer|T:120>ssh lilith

rgb@lilith|T:101>

>>

Note that with the last ssh we logged into lilith with no password!

ssh is really pretty easy to set up this way; if you read the man page(s) you can learn how to generate and add additional authorized keys and do fancier things with it, but many users will need no more than what we've done so far. A warning - it is a good idea to log into each host in your cluster one time after setting it up before proceeding further, to build up the known_hosts file so that you aren't prompted for each host the first time PVM starts up a virtual machine.

[end]

A closing note or two:

When compiling / running MPI 1.2x remember either the "setenv P4_RSHCOMMAND ssh" or run configure with "-rsh=ssh" This means you don't have to worry about it from then on.

Most Linux distro's are setup with some sensible default firewall settings. Remember to modify them so SSH is allowed in ''both'' directions!

'''RLOGIN and RSH'''

For what is an insecure process, using the "R" untilites is truly evil. There's a fair degree of implementation difference between distro's too. The RedHat flavours and derivatives all seem to be vaguely similar. Regardless, you're aiming for the non-Kerberos version of the "R" utilites.

Rather than repeast the whole process, this procedure has worked for the author:
http://howto.x-tend.be/openMosixWiki/index.php/Enabling%20RSH/RLOGIN%20without%20a%20password%20%28even%20for%20root%21%29
From here:
http://howto.x-tend.be/openMosixWiki/index.php/

A couple of comments... In some respects the R utilites are quicker than their chatty SSH cousins. However, this becomes irrelevant after MPI starts up (that by design handles its own comms after the link is established). Also a lot of utilites leave their SSH connection open rather than rsh which they rerun on every request (eg. MOODSS).

Seriously think about *why* you want RSH/RLOGIN. As you can see above, SSH isn't hard and (I'd argue) a bit easier!

Administration

2006-02-21T01:57:19Z

Steve vmwx:

*[http://www.csm.ornl.gov/torc/C3/index.html C3 (Cluster Command and Control)] A set of command line tools for typical cluster tasks.

Administration

2006-02-21T01:54:34Z

Steve vmwx:

*[http://www.csm.ornl.gov/torc/C3/index.html C3 (Cluster Command and Control)]

Parallel Tools

2006-02-21T01:52:10Z

Steve vmwx:

*[http://www.csm.ornl.gov/torc/C3/index.html C3 (Cluster Command and Control)]

Other

2006-02-21T01:45:41Z

Steve vmwx:

Monitoring
*[http://ganglia.sourceforge.net/ Ganglia]

*[http://http://moodss.sourceforge.net/ MOODSS]

Talk:Cluster Benchmarking Packages

2006-02-21T01:29:11Z

Steve vmwx: Observation: It seems HINT has disappeared?!

It seems HINT has disappeared?!

It used to be at:
http://www.scl.ameslab.gov/HINT/

Cheers
Steve