[Beowulf] Re: typical latencies for gigabit ethernet

Scott Atchley atchley at myri.com
Mon Jun 29 10:26:05 EDT 2009

On Jun 29, 2009, at 5:59 AM, Dave Love wrote:

>> Can you say something about any tuning you did to get decent results?
> To get the lowest latency, turn off rx interrupt coalescence, either
> with ethtool or module parameters, depending on the driver.  Of  
> course,
> you may not want to turn it off completely, depending on how much load
> the extra interrupts cause.

When I test Open-MX, I turn interrupt coalescing off. I run  
omx_pingpong to determine the lowest latency (LL). If the NIC's driver  
allows one to specify the interrupt value, I set it to LL-1. If the  
driver does not allow specifying the actual rate (i.e. it only has  
predetermined values), then I leave it off.

The downside is lower throughput for large messages on 10G Ethernet. I  
don't think it matters on gigabit.

Brice and Nathalie have a paper which implements an adaptive interrupt  
coalescing so that you do not have to manually tune anything:

"Finding a Tradeoff between Host Interrupt Load and MPI Latency over  
Ethernet" available at:


I do not know if it has been included in a release yet.

Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list