3.1 UDP vs. TCP

Background

The original NFS used UDP UDP for transmitting data for NFS. UDP was chosen because is very simple, you are just transmitting packets and if they get lost everything is sent again. In addition, there is no load on the server when the connection to the network (or really the client) is not active. But UDP does have the problem that if the network has any congestion and packets are lost, all of the packets are retransmitted, increasing the load on the server and the network.

NFSv2, the original NFS that had widespread use, used UDP. A few companies put TCP into NFSv2, but it was not part of the NFS standard. With the advent of NFSv3 which was released around the mid-1990's, TCP was added to the protocol. Today, we can set our Linux NFS clients to accept data via UDP 9if we really want to) or to use TCP. The advantage in using TCP is that if a packet is lost, which can happen under reasonably heavy network congestion (or a bad switch), then only the lost packet is retransmitted, limiting the amount of data that has to be retransmitted.

Recommendations

Getting the best performance using either UDP or TCP, really depends upon the situation. In general, people just use TCP because it's the default in most distributions (pretty much all distributions using a 2.6.x kernel).

Steps for Testing UDP and TCP Performance

But if you want to experiment, it is recommended that you start with UDP, run the cluster under load (i.e. run applications), and watch the load on the server as well as time the performance of your applications.

To test UDP, you have to modify the file, /etc/fstab on all of your clients (compute nodes). It is beyond the scope of this wiki to tell you how to modify the file on all of the compute nodes. Please consult the manual for whatever tool you are using. But the nfs mount points on the client should look like

mount udp

Don't forget to keep any options on the nfs mount points that you had before.

To watch the load, just use top (man top if you need help). Open a separate terminal on the node that is the NFS server and enter the command top. Then in another terminal window, run your application(s) on the cluster. Watch the load as the applications start - you can even record the peak load during the run.

When you run your applications, be sure you time how long they take to run. You can use the time command in front of your application. If you run your application as ./foo you would run it as time ./foo. When the application is done, it will print out the real time, the user time, and the system time. You are interested in the real time.

After you test your cluster with UDP, test it with TCP.