Small GigE Switches - Revision history

Deadline at 18:55, 26 December 2008

2008-12-26T18:55:47Z

Deadline at 18:17, 26 December 2008

2008-12-26T18:17:19Z

New page

Initially I started posting results for [http://open-mx.gforge.inria.fr/ Open-MX] over GigE on my [http://limulus.basement-supercomputing.com/wiki/NorbertLimulus Limulus Cluster]. I used Netpipe MPI/TCP (2.4) for most of the tests. As Open MX requires Jumbo packets, I noticed that using Jumbo packets actually reduced the throughput! I'm still in the process of collecting data, so I cannot make any definite conclusions. I just noticed that the latest version of OpenMX (0.9.1) can run over standard frame sizes (1500). More results coming soon.

Kernel: 2.6.23 
CPU: Intel(R) Core(TM)2 Duo CPU E6550 @ 2.33GHz 
Interconnect: Intel 82572EI Gigabit Ethernet PCIe 1X 
Switches: [http://www.smc.com/index.cfm?event=viewProduct&localeCode=EN_USA&cid=6&scid=24&pid=1138 SMC 8505T] (5 port), [http://www.smc.com/index.cfm?event=viewProduct&localeCode=EN_USA&cid=6&scid=24&pid=1147 SMC 8508T] (8 port), [http://www.smc.com/index.cfm?event=viewProduct&localeCode=EN_USA&cid=7&scid=39&pid=1413 SMC GS16] (16 port), and a Cross-over Cable (Note: I tested a 5 port 3com and it would not only negotiate 100BT so it went back. I also tested an 8 port ProCurve and it work similarly to the SMC switch, more tests are needed.) 
MPI: LAM, MPICH-MX (Open MX 0.6)

==Summary Data ==
The following data use the 8505T, 8505T swtiches and a X-over cable:

[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-latency.mx.png|center|thumb|400px|Signature Graph]]
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-signature.mx.png|center|thumb|400px|Latency Graph]]
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize.mx.png|center|thumb|400px|Throughput Graph]]

==Interesting Comparisons Frame Size ==
(8505T, 8505T, X-over)
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-mpich-mx.png|center|thumb|400px|LAM/1500, LAM/9000, MPICH-MX/9000]]
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-MTU-size.png|center|thumb|400px|LAM run over range of MTU size]]
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-mx-switches.png|center|thumb|400px|Switch Effect on MPICH-MX]

* <a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-mpich-mx.png">LAM/1500, LAM/9000, MPICH-MX/9000</a>
* <a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-MTU-size.png">LAM run over range of MTU size (1500-9000)</a>
* <a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-mx-switches.png"></a>

SMC GS16 Switch LAM MTU (note Jumbo frames still reduces throughput, except 3000!)

*<a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-MTU-size-gs16.png">LAM run over range of MTU size </a>

==Mystery Is Solved==
Turn off Flow Control! I turned off Flow Control using Ethtool and the jumbo packets at the high end got much better, but the variability got much worse (see results below). Also, the kernel is now 2.6.26.2 (Using Fedora 8 now) More tests are needed as well.

* <a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-tcp-flowctrl-off-ITR1.png">Netpipe TCP run over range of MTU size with Flow Control Off </a>

Here is the Ethtool sequence I used to turn off Flow Control (check the man page for Ethtool for a full description of options).

# ethtool -a eth2
Pause parameters for eth2:
Autonegotiate: on
RX: on
TX: on

# ethtool -A eth2 autoneg off rx off tx off

# ethtool -a eth2
Pause parameters for eth2:
Autonegotiate: off
RX: off
TX: off

I also set the InterruptThrottleRate to "dynamic". See <TT>~/Documentation/networking/e1000.txt</TT>
in the kernel source directory for an an explanation of this. From the <TT>e1000.txt</TT> file:

''For situations where low latency is vital such as cluster or grid computing, the algorithm can reduce latency even more when InterruptThrottleRate is set to mode 1. In this mode, which operates the same as mode 3, the interruptThrottleRate will be increased stepwise to 70000 for traffic in class "Lowest latency".''

The sequence below sets the InterruptThrottleRate (which is <TT>rx-usecs</TT> to Ethtool) to 1 (dynamic)

# ethtool -c eth2
Coalesce parameters for eth2:
Adaptive RX: off TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 3
rx-frames: 0
rx-usecs-irq: 0
rx-frames-irq: 0

tx-usecs: 0
tx-frames: 0
tx-usecs-irq: 0
tx-frames-irq: 0

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

# ethtool -C eth2 rx-usecs 1

# ethtool -c eth2
Coalesce parameters for eth2:
Adaptive RX: off TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 1
rx-frames: 0
rx-usecs-irq: 0
rx-frames-irq: 0

tx-usecs: 0
tx-frames: 0
tx-usecs-irq: 0
tx-frames-irq: 0

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

@@ Line 8: / Line 8: @@
 ==Summary Data ==
-The following data use the 8505T, 8505T swtiches and a X-over cable:
+The following data use the 8505T, 8508T switches and a X-over cable. Netpipe MPI using MPICH-MX and LAM(TCP) are shown. In addition Netpipe TCP results are given for comparison. Click on the images to get high resolution
-[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-latency.mx.png|center|thumb|400px|Signature Graph]]
+[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-latency.mx.png|center|thumb|500px|Signature Graph]]
-[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-signature.mx.png|center|thumb|400px|Latency Graph]]
+[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-signature.mx.png|center|thumb|500px|Latency Graph]]
-[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize.mx.png|center|thumb|400px|Throughput  Graph]]
+[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize.mx.png|center|thumb|500px|Throughput  Graph]]
 ==Interesting Comparisons Frame Size ==
- (8505T, 8505T, X-over)
+If some specific data are compared we find some surprising results. First, if you compare LAM using a 1500 byte MTU (frame), LAM using a 9000 byte MTU, and  MPICH-MX using a 9000 byte MTU, you find the that best through put comes from smaller frame size! This result is opposite of what you would expect -- bigger frames better thoughput!
-* <a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-mpich-mx.png">LAM/1500, LAM/9000, MPICH-MX/9000</a>
+[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-mpich-mx.png|center|thumb|500px|LAM/1500, LAM/9000, MPICH-MX/9000]]
-SMC GS16 Switch LAM MTU (note Jumbo frames still reduces throughput, except 3000!)
+[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-MTU-size.png|center|thumb|500px|LAM run over range of MTU size]]
-*<a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-MTU-size-gs16.png">LAM run over range of MTU size </a>
+Perhaps it is the switch. The following results show the difference
 ==Mystery Is Solved==
-Turn off Flow Control! I turned off Flow Control using Ethtool and the jumbo packets at the high end got much better, but the variability got much worse (see results below). Also, the kernel is now 2.6.26.2 (Using Fedora 8 now) More tests are needed as well.
+Turn off Flow Control! I turned off Flow Control using Ethtool and the jumbo packets at the high end got much better, but the variability got much worse (see results below). Also, the kernel is now 2.6.26.2 (Using Fedora 8 now) More tests are needed, but we see that flow control was the problem.
-* <a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-tcp-flowctrl-off-ITR1.png">Netpipe TCP run over range of MTU size with Flow Control Off </a>
+==Using Ethtool==
 Here is the Ethtool sequence I used to turn off Flow Control (check the man page for Ethtool for a full description of options).