Gigabit performance issues and NFS
josip at lanl.gov
Mon Jun 2 13:21:02 EDT 2003
In addition to NFS tuning suggested by Jakob, you may want to check your
gigabit ethernet driver parameters. This was the key problem we
identified on Coral, our cluster at ICASE (my old LaRC home :-), and the
fix was to increase the interrupt mitigation parameters for the acenic
The reason is this: NFS in Linux uses UDP, and it takes 6 normal UDP
frames (unless you are using jumbo frames) to transfer one 8KB NFS block
(a typical NFS tuning choice). If the NFS server gets interrupted more
than every 6th frame, chances are that some frames will be lost. If
any of the 6 frames is lost, all 6 have to be re-sent, leading to
further losses, etc. This can collapse NFS performance by 100:1.
We found that the following line in /etc/modules.conf restored
reasonable (but not great) NFS performance:
options acenic tx_coal_tick=75 rx_coal_tick=75 max_tx_desc=32 max_rx_desc=6
The key bit is that last parameter "max_rx_desc=6" but the others help
as well. Optimal choices may be hardware dependent, but the above is a
good starting point for the acenic device driver.
On the other hand, your NFS performance results are fairly typical. We
found that acenic-based Gigabit Ethernet can deliver ~72% of its rated
performance only under rare circumstances and with special effort.
Normally, we got FTP performance of about 28 MB/s and NFS performance of
about 13 MB/s.
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
More information about the Beowulf