<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2650.12">
<TITLE>RE: NFS 100:1 performance loss</TITLE>
</HEAD>
<BODY>
<P><FONT SIZE=2>Slow NFS because of UDP jives with the spec behind UDP: the broadcasts are not guaranteed to arrive in a particular order, and are not guaranteed to arrive at all. </FONT></P>
<P><FONT SIZE=2>TCP is definitely the way to go for fast and reliable. That's what FTP uses.</FONT>
</P>
<P><FONT SIZE=2>Richard Schilling</FONT>
<BR><FONT SIZE=2>Web Integration Programmer</FONT>
<BR><FONT SIZE=2>Affiliated Health Services</FONT>
<BR><FONT SIZE=2>Mount Vernon, WA </FONT>
</P>
<BR>
<P><FONT SIZE=2>> -----Original Message-----</FONT>
<BR><FONT SIZE=2>> From: Mel Jones [<A HREF="mailto:mello@msg.ucsf.edu">mailto:mello@msg.ucsf.edu</A>]</FONT>
<BR><FONT SIZE=2>> Sent: Friday, August 18, 2000 3:50 PM</FONT>
<BR><FONT SIZE=2>> To: josip@icase.edu</FONT>
<BR><FONT SIZE=2>> Cc: Beowulf mailing list</FONT>
<BR><FONT SIZE=2>> Subject: Re: NFS 100:1 performance loss</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Josip Loncaric wrote:</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> > Has anyone seen this NFS performance problem?</FONT>
<BR><FONT SIZE=2>> ></FONT>
<BR><FONT SIZE=2>> > Our Beowulf has two servers w/RAID-0 arrays which deliver 45MB/s,</FONT>
<BR><FONT SIZE=2>> > connected via Gigabit Ethernet. The machines are dual </FONT>
<BR><FONT SIZE=2>> PIII/500 systems</FONT>
<BR><FONT SIZE=2>> > with 512MB of RAM each. We are running Red Hat 6.2 updated </FONT>
<BR><FONT SIZE=2>> to kernel</FONT>
<BR><FONT SIZE=2>> > 2.2.16-3 and recently we updated to nfs-utils-0.1.9.1-1. </FONT>
<BR><FONT SIZE=2>> We are running</FONT>
<BR><FONT SIZE=2>> > 16 kernel nfsd threads on each machine. Here is how long </FONT>
<BR><FONT SIZE=2>> it takes to</FONT>
<BR><FONT SIZE=2>> > copy a 28,955,860 byte file from machine 1 to machine 2:</FONT>
<BR><FONT SIZE=2>> ></FONT>
<BR><FONT SIZE=2>> > rcp: 1.04 seconds (27.8 MB/s, where 1MB=10^6B)</FONT>
<BR><FONT SIZE=2>> > ftp: 1.12 seconds (25.8 MB/s)</FONT>
<BR><FONT SIZE=2>> > NFS 1KB: 12.35 seconds ( 2.3 MB/s, using rsize=wsize=1024)</FONT>
<BR><FONT SIZE=2>> > NFS 8KB: 129.42 seconds ( 0.2 MB/s, using rsize=wsize=8192)</FONT>
<BR><FONT SIZE=2>> ></FONT>
<BR><FONT SIZE=2>> > Clearly, there is something very wrong with NFS, </FONT>
<BR><FONT SIZE=2>> particularly with 8KB</FONT>
<BR><FONT SIZE=2>> > rsize/wsize (which should have improved performance!). Our system</FONT>
<BR><FONT SIZE=2>> > manager tells me that reduced Linux NFS performance with </FONT>
<BR><FONT SIZE=2>> 8KB rsize/wsize</FONT>
<BR><FONT SIZE=2>> > is a known problem, but even at 1KB our NFS is getting less </FONT>
<BR><FONT SIZE=2>> than 10% of</FONT>
<BR><FONT SIZE=2>> > the rcp or ftp performance.</FONT>
<BR><FONT SIZE=2>> ></FONT>
<BR><FONT SIZE=2>> > Any ideas?</FONT>
<BR><FONT SIZE=2>> > Josip</FONT>
<BR><FONT SIZE=2>> ></FONT>
<BR><FONT SIZE=2>> > --</FONT>
<BR><FONT SIZE=2>> > Dr. Josip Loncaric, Senior Staff Scientist </FONT>
<BR><FONT SIZE=2>> <A HREF="mailto:josip@icase.edu">mailto:josip@icase.edu</A></FONT>
<BR><FONT SIZE=2>> > ICASE, Mail Stop 132C PGP </FONT>
<BR><FONT SIZE=2>> key at <A HREF="http://www.icase.edu./~josip/" TARGET="_blank">http://www.icase.edu./~josip/</A></FONT>
<BR><FONT SIZE=2>> > NASA Langley Research Center </FONT>
<BR><FONT SIZE=2>> <A HREF="mailto:j.loncaric@larc.nasa.gov">mailto:j.loncaric@larc.nasa.gov</A></FONT>
<BR><FONT SIZE=2>> > Hampton, VA 23681-2199, USA </FONT>
<BR><FONT SIZE=2>> Tel. +1 757 864-2192 Fax +1 757 864-6134</FONT>
<BR><FONT SIZE=2>> ></FONT>
<BR><FONT SIZE=2>> > _______________________________________________</FONT>
<BR><FONT SIZE=2>> > Beowulf mailing list</FONT>
<BR><FONT SIZE=2>> > Beowulf@beowulf.org</FONT>
<BR><FONT SIZE=2>> > <A HREF="http://www.beowulf.org/mailman/listinfo/beowulf" TARGET="_blank">http://www.beowulf.org/mailman/listinfo/beowulf</A></FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> I've seen a similar but nowhere near as bad result on our </FONT>
<BR><FONT SIZE=2>> network. This may</FONT>
<BR><FONT SIZE=2>> be completely irrelevant but I found NFS traffic between my </FONT>
<BR><FONT SIZE=2>> SGI machines</FONT>
<BR><FONT SIZE=2>> about 4x slower than FTP. It got much better when I set the </FONT>
<BR><FONT SIZE=2>> transport to</FONT>
<BR><FONT SIZE=2>> TCP rather than UDP. The problem turned out to be that my </FONT>
<BR><FONT SIZE=2>> switches weren't</FONT>
<BR><FONT SIZE=2>> necessarily forwarding the packets in order of receipt and </FONT>
<BR><FONT SIZE=2>> this was taken</FONT>
<BR><FONT SIZE=2>> care of much more efficiently on the receiving machine with TCP.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> ... mello</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> _______________________________________________</FONT>
<BR><FONT SIZE=2>> Beowulf mailing list</FONT>
<BR><FONT SIZE=2>> Beowulf@beowulf.org</FONT>
<BR><FONT SIZE=2>> <A HREF="http://www.beowulf.org/mailman/listinfo/beowulf" TARGET="_blank">http://www.beowulf.org/mailman/listinfo/beowulf</A></FONT>
<BR><FONT SIZE=2>> </FONT>
</P>
</BODY>
</HTML>