Gigabit performance issues and NFS

[Josip Loncaric]

Doug,

In addition to NFS tuning suggested by Jakob, you may want to check your 
gigabit ethernet driver parameters.  This was the key problem we 
identified on Coral, our cluster at ICASE (my old LaRC home :-), and the 
fix was to increase the interrupt mitigation parameters for the acenic 
device driver.

The reason is this: NFS in Linux uses UDP, and it takes 6 normal UDP 
frames (unless you are using jumbo frames) to transfer one 8KB NFS block 
(a typical NFS tuning choice).  If the NFS server gets interrupted more 
than every 6th frame, chances are that some frames will be lost.   If 
any of the 6 frames is lost, all 6 have to be re-sent, leading to 
further losses, etc.  This can collapse NFS performance by 100:1.

We found that the following line in /etc/modules.conf restored 
reasonable (but not great) NFS performance:

options acenic tx_coal_tick=75 rx_coal_tick=75 max_tx_desc=32 max_rx_desc=6

The key bit is that last parameter "max_rx_desc=6" but the others help 
as well.  Optimal choices may be hardware dependent, but the above is a 
good starting point for the acenic device driver.

On the other hand, your NFS performance results are fairly typical.  We 
found that acenic-based Gigabit Ethernet can deliver ~72% of its rated 
performance only under rare circumstances and with special effort. 
Normally, we got FTP performance of about 28 MB/s and NFS performance of 
about 13 MB/s.

Sincerely,
Josip

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf