Gigabit performance issues and NFS

Jakob Oestergaard jakob at
Mon Jun 2 06:15:58 EDT 2003

On Thu, May 29, 2003 at 09:35:54AM -0400, Doug Farley wrote:
> Fellow Wulfers,
> I know this isnt 100% wulf related, although it is part of my wulfs setup, 
> but this is the best forum where everyone has alot of good experience.

NFS is 'wulf related, whether we like it or not   :)

> Well Heres the deal, I have a nice 2TB Linux file server with an Intel 
> e1000 based nic in it.  And I have an SGI O3 (master node) that is dumping 
> to it with a tigon series gigabit card.  I've tuned both, and my ttcp and 
> netpipe performance average ~ 80-95MB/s which is more than reasonable for 
> me.  Both the fibre channel on my SGI and the raid (3ware) on my Linux box 
> can write at 40MB/s sustained, read is a little faster for both maybe ~ 
> 50MB/s sustained.  I can get ftp/http transfers between the two to go at 
> 39-40MB/s, which again i'm reasonably happy with.  BUT, the part that is 
> killing me is nfs and scp.  Both crawl in at around 8-11MB/s with no other 
> devices on the network.

11MB/sec with scp is quite good - considering everything is encrypted
and what not...

With NFS that's pretty poor though, I'd agree.

>  Any exports from the SGI i've exported with the 
> 32bitclients flag, and i've pumped my r&wsize windows up to 32K, and forced 
> nfs v3 on both Linux and Irix.  After spending a week scouring the web I've 
> found nothing that has worked, and SGI support thinks its a Linux nfs 
> problem, which could be, but i'd like to get the opinion of this crowd in 
> hopes of some light!

What does top and vmstat on your NFS server tell you?

How many nfsd threads are busy (in R or D state), during the writes ?

The default number of nfsd threads is 8, which may be a little low. I
run 16 threads here, on a somewhat smaller NFS server (also with a Gbit
NIC).   If you only see one or two nfsd threads in R or D state,
anywhere near the top of your "top", then this should not be the

Try specifying the "async" parameter for the given mount in your exports
file on the NFS server.  Just to see if this helps.  There are some
considerations you need to make here - if the client does a sync() and
you use the async option on the server, you are not guaranteed that the
data has reached the disk platters by the time the client sync() call
returns.  This may or may not matter for you.

What does vmstat say during such a big write?  Is the CPU idle or busy,
is it spending all it's time in the kernel?

How's the ping between the machines, when doing the write and when the
network is more idle?   You may have  a switch in between that does
store-and-forward instead of cut-through, when the network gets loaded.
Latency hurts NFS.

:   jakob at   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
Beowulf mailing list, Beowulf at
To change your subscription (digest mode or unsubscribe) visit

More information about the Beowulf mailing list