Gigabit performance issues and NFS

Doug Farley d.l.farley at larc.nasa.gov
Mon Jun 2 10:17:53 EDT 2003


Jakob,

For the switch I have a hp procurve 2708, from the netpipe results i'd say 
its a cut-through switch, however I may have not applied enough load to it 
to find out, and I have seen no data either way to determine otherwise.  On 
writing from the SGI to my Linux NFS server I have 4 nfsd's running on the 
Linux box, with all 4 in either a R or D state.  I've included copies of 
the top results for reference.   In regards to the use of async instead of 
sync in the exports, the speed was 10-14MB/s, there were again 2 nfsd's in 
R state and 2 in D state as before.  with a load average of ~ 4, each of 
the nfsd consuming ~ 3% cpu, and each of the biod's consuming about 
6%.    I hope that maybe some of these numbers/figures (while possibly 
excessive) might help the group shed some light onto my problem.

Thank you all again,

Doug Farley

For SGI Writing to Linux:

ping before:
         ----Linux-g.localdomain PING Statistics----
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max = 0.219/0.286/0.427 ms

----Linux-g.localdomain PING Statistics----
22889 packets transmitted, 22888 packets received, 0.0% packet loss
round-trip min/avg/max = 0.202/0.476/2.307 ms
   2072.5 packets/sec sent, 2072.5 packets/sec received

--- SGI-g.localdomain ping statistics ---
13 packets transmitted, 13 received, 0% loss, time 11993ms
rtt min/avg/max/mdev = 0.108/0.178/0.230/0.036 ms
/ping before

ping during:

----Linux-g.localdomain PING Statistics----
6 packets transmitted, 6 packets received, 0.0% packet loss
round-trip min/avg/max = 0.240/0.333/0.380 ms

----Linux-g.localdomain PING Statistics----
33140 packets transmitted, 33140 packets received, 0.0% packet loss
round-trip min/avg/max = 0.149/0.264/2.670 ms
   3687.1 packets/sec sent, 3687.1 packets/sec received

--- SGI-g.localdomain ping statistics ---
9 packets transmitted, 9 received, 0% loss, time 7999ms
rtt min/avg/max/mdev = 0.130/0.230/0.316/0.050 ms

--- SGI-g.localdomain ping statistics ---
25450 packets transmitted, 25449 received, 0% loss, time 11371ms
rtt min/avg/max/mdev = 0.076/0.243/2.221/0.059 ms, ipg/ewma 0.446/0.249 ms

/ping during

SGI:
top header before:
----
IRIX64 SGI 6.5 IP35          load averages: 0.02 0.00 0.00
57 processes:  56 sleeping, 1 running
8 CPUs: 99.4% idle,  0.0% usr,  0.4% ker,  0.0% wait,  0.0% xbrk,  0.2% intr
Memory: 8192M max, 7313M avail, 7264M free, 4096M swap, 4096M free swap
/top header before

top header during:
IRIX64 SGI 6.5 IP35          load averages: 1.28 0.48 0.18
61 processes:  59 sleeping, 2 running
8 CPUs: 83.6% idle,  0.1% usr,  7.3% ker,  6.2% wait,  0.0% xbrk,  2.9% intr
Memory: 8192M max, 7455M avail, 5703M free, 4096M swap, 4096M free swap

        PID       PGRP USERNAME PRI  SIZE   RES STATE    TIME WCPU% CPU% 
COMMAND
  511570519  511570519 dfarley   20  416K  304K sleep    0:29 22.2 30.16 ln
  511571530  511571530 root      21    0K    0K sleep    0:08  6.5  6.63 bio3d
    8975393    8975393 root      21    0K    0K sleep    0:07  6.1  6.53 bio3d
  511549378  511549378 root      21    0K    0K run/1   12:24  6.0  6.15 bio3d
  511563810  511563810 root      21    0K    0K sleep    0:08  6.5  5.74 bio3d
  511543776  511541950 root      20    0K    0K sleep   49:18  2.2  2.88 nfsd
  511568500  511568500 dfarley   20 2208K 1536K run/3    0:00  0.2  0.24 top
  511567378  511549203 dfarley   20 4208K 3104K sleep    0:00  0.0  0.02 sshd
       8928       8928 root      20 1808K 1088K sleep    1:02  0.0  0.01 prngd
        410        410 root      20 2512K 2512K sleep    1:34  0.0  0.01 ntpd
        381          0 root      20 2816K 2064K sleep    0:32  0.0  0.00 ipmon
/top header during
/SGI

Linux Box:
vmstat before:
    procs                      memory    swap          io     system 
  cpu
  r  b  w   swpd   free          buff    cache   si  so    bi     bo   in 
   cs  us  sy  id
  0  0  0  18980  18012  516412  418384  0   0     22    40   44    41   0 
  1   47
/vmstat before

vmstat during:
    procs                      memory    swap          io     system 
  cpu
  r  b  w   swpd   free          buff    cache   si  so    bi    bo   in 
      cs  us  sy  id
  0  0  5  18980   8552  518828 
445648   0   0    22    43   45          44   0   1  47
/vmstat during

top header before:
up 9 days, 17:03,  2 users,  load average: 0.33, 0.06, 0.00
72 processes: 70 sleeping, 2 running, 0 zombie, 0 stopped
CPU states:  0.0% user,  1.0% system,  0.0% nice, 99.0% idle
Mem:  1031204K av, 1015316K used,   15888K free,       0K shrd,  532128K buff
Swap: 2048276K av,   18980K used, 2029296K free                  422944K cached


/top header before

top header during:
up 9 days, 17:27,  2 users,  load average: 2.60, 0.76, 0.26
70 processes: 66 sleeping, 4 running, 0 zombie, 0 stopped
CPU states:  0.0% user, 25.6% system,  0.0% nice, 74.4% idle
Mem:  1031204K av, 1022200K used,    9004K free,       0K shrd,  517376K buff
Swap: 2048276K av,   18980K used, 2029296K free                  446512K cached

   PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
14732 root      15   0     0    0     0 RW    4.2  0.0   1:21 nfsd
14725 root      15   0     0    0     0 DW    3.8  0.0   0:55 nfsd
14726 root      14   0     0    0     0 RW    3.8  0.0   2:08 nfsd
14729 root      14   0     0    0     0 DW    3.6  0.0   2:52 nfsd
     5 root       9   0     0    0     0 SW    0.6  0.0   1:12 kswapd
     6 root       9   0     0    0     0 SW    0.6  0.0  32:40 kscand
14730 root       9   0     0    0     0 SW    0.2  0.0   1:23 nfsd
     1 root       9   0   156  128    96 S     0.0  0.0   0:04 init
     2 root       9   0     0    0     0 SW    0.0  0.0   0:00 keventd
     3 root       9   0     0    0     0 SW    0.0  0.0   0:00 kapmd
/top header during

/Linux Box

==============================
Doug Farley

Data Analysis and Imaging Branch
Systems Engineering Competency
NASA Langley Research Center

< D.L.FARLEY at LaRC.NASA.GOV >
< Phone +1 757 864-8141 >



At 12:15 PM 6/2/2003 +0200, Jakob Oestergaard wrote:
>On Thu, May 29, 2003 at 09:35:54AM -0400, Doug Farley wrote:
> > Fellow Wulfers,
> >
> > I know this isnt 100% wulf related, although it is part of my wulfs setup,
> > but this is the best forum where everyone has alot of good experience.
>
>NFS is 'wulf related, whether we like it or not   :)
>
> >
> > Well Heres the deal, I have a nice 2TB Linux file server with an Intel
> > e1000 based nic in it.  And I have an SGI O3 (master node) that is dumping
> > to it with a tigon series gigabit card.  I've tuned both, and my ttcp and
> > netpipe performance average ~ 80-95MB/s which is more than reasonable for
> > me.  Both the fibre channel on my SGI and the raid (3ware) on my Linux box
> > can write at 40MB/s sustained, read is a little faster for both maybe ~
> > 50MB/s sustained.  I can get ftp/http transfers between the two to go at
> > 39-40MB/s, which again i'm reasonably happy with.  BUT, the part that is
> > killing me is nfs and scp.  Both crawl in at around 8-11MB/s with no other
> > devices on the network.
>
>11MB/sec with scp is quite good - considering everything is encrypted
>and what not...
>
>With NFS that's pretty poor though, I'd agree.
>
> >  Any exports from the SGI i've exported with the
> > 32bitclients flag, and i've pumped my r&wsize windows up to 32K, and 
> forced
> > nfs v3 on both Linux and Irix.  After spending a week scouring the web 
> I've
> > found nothing that has worked, and SGI support thinks its a Linux nfs
> > problem, which could be, but i'd like to get the opinion of this crowd in
> > hopes of some light!
>
>What does top and vmstat on your NFS server tell you?
>
>How many nfsd threads are busy (in R or D state), during the writes ?
>
>The default number of nfsd threads is 8, which may be a little low. I
>run 16 threads here, on a somewhat smaller NFS server (also with a Gbit
>NIC).   If you only see one or two nfsd threads in R or D state,
>anywhere near the top of your "top", then this should not be the
>problem.
>
>Try specifying the "async" parameter for the given mount in your exports
>file on the NFS server.  Just to see if this helps.  There are some
>considerations you need to make here - if the client does a sync() and
>you use the async option on the server, you are not guaranteed that the
>data has reached the disk platters by the time the client sync() call
>returns.  This may or may not matter for you.
>
>What does vmstat say during such a big write?  Is the CPU idle or busy,
>is it spending all it's time in the kernel?
>
>How's the ping between the machines, when doing the write and when the
>network is more idle?   You may have  a switch in between that does
>store-and-forward instead of cut-through, when the network gets loaded.
>Latency hurts NFS.
>
>--
>................................................................
>:   jakob at unthought.net   : And I see the elder races,         :
>:.........................: putrid forms of man                :
>:   Jakob Østergaard      : See him rise and claim the earth,  :
>:        OZ9ABN           : his downfall is at hand.           :
>:.........................:............{Konkhra}...............:

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



More information about the Beowulf mailing list