[Beowulf] Latency issues with Itanium2 (Tiger2) and e1000 gigE

Lars Henriksen lars at meshtechnologies.com
Sat Jan 10 05:38:32 EST 2004

On Sat, 2004-01-10 at 00:07, Craig Tierney wrote:
> I am having some problems with some new servers I just
> got.  I have several dual Itanium2 (Intel Tiger2 board) 
> connected via gigE through a Dell Powerconnect 5224 gigE
> switch.  
> Using mpptest to test latency and bandwidth the results
> are not so good.  Latency varies between 60-200 mu/s for
> small packets, and the bandwidth tops out at about 45 MB/s.
> - This happens with both a Debian and White Box (rebuilt
> from RHAS 3.0 sources).

> Does anyone have any suggestions on what else to look for?

Yes. I suppose the networking chip is an Intel model - 
something like a 82546 or 82547 running with the e1000 driver?

If this is the case, and the kernel module is inserted w/o
any special options, "Interrupt Throttling"  kicks in.

In the default case Intel has implemented a heuristical 
approach, which limits the number of interrupts per sec
the NIC produces. This is done to lower the CPU overhead
in high volume communications. Excellent for large transfers
but really bad for low-latency communication :-(

What we usually do is setting the InterruptThrottleRate to 
something other than the default value, 1.

Look here for information:

This does not solve the problem fully though:
Setting the ITR too high (50000-75000) makes the system 
unresponsive under certain network loads. 

Something like 25000 allows for most communication, like NFS
and SSH but still software like LAM-MPI has problems.

Setting the value arround 10000 makes most software run 
smoothly, but latency then is arround 50 us (i think) 
on our hardware.. But the latency is constant (no 60 to 250us)

We have not found an optimal solution (other than using
different NIC's)

If anyone has any ideas i would be grateful.

best regards
Lars Henriksen                  | MESH-Technologies A/S
Systems Manager & Consultant    | Forskerparken 10
www.meshtechnologies.com        | DK-5260 Odense M, Denmark
lars at meshtechnologies.com       | mobile: +45 2291 2904
direct: +45 6315 7310           | fax:    +45 6315 7314

Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list