Small GigE Switches - Revision history

Deadline at 18:55, 26 December 2008

2008-12-26T18:55:47Z

← Older revision		Revision as of 14:55, 26 December 2008
Line 8:		Line 8:

	==Summary Data ==		==Summary Data ==
	The following data use the 8505T, ~~8505T swtiches~~ and a X-over cable:		The following data use the 8505T, 8508T switches and a X-over cable. Netpipe MPI using MPICH-MX and LAM(TCP) are shown. In addition Netpipe TCP results are given for comparison. Click on the images to get high resolution
			versions.

	[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-latency.mx.png\|center\|thumb\|~~400px~~\|Signature Graph]]		[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-latency.mx.png\|center\|thumb\|500px\|Signature Graph]]
	[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-signature.mx.png\|center\|thumb\|~~400px~~\|Latency Graph]]		[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-signature.mx.png\|center\|thumb\|500px\|Latency Graph]]
	[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize.mx.png\|center\|thumb\|~~400px~~\|Throughput Graph]]		[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize.mx.png\|center\|thumb\|500px\|Throughput Graph]]



	==Interesting Comparisons Frame Size ==		==Interesting Comparisons Frame Size ==
	~~(8505T~~, ~~8505T, X-over)~~		If some specific data are compared we find some surprising results. First, if you compare LAM using a 1500 byte MTU (frame), LAM using a 9000 byte MTU, and MPICH-MX using a 9000 byte MTU, you find the that best through put comes from smaller frame size! This result is opposite of what you would expect -- bigger frames better thoughput!
	~~[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-mpich-mx.png\|center\|thumb\|400px\|~~LAM/1500, LAM/9000, MPICH-MX/9000]]
	~~[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-~~MTU-size~~.png\|center\|thumb\|400px\|LAM run over range~~ of ~~MTU size]]~~
	~~[[Image:Small_GigE_Switches_netpipe.Intel~~-PT-~~PCIe-throughput_vs_blocksize-mx-switches.png\|center\|thumb\|400px\|Switch Effect on MPICH-MX]~~

	* <a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-mpich-mx.png">LAM/1500, LAM/9000, MPICH-MX/9000~~</a>~~		[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-mpich-mx.png\|center\|thumb\|500px\|LAM/1500, LAM/9000, MPICH-MX/9000]]
	* <a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-MTU-size.png">LAM run over range of MTU size (1500-9000)</a>
	* <a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-mx-switches.png"></a>

			The above result suggested another experiment. In the following graph LAM MPi was run over a range for MTU (frame) sizes. As the frame gets bigger, the
			throughput gets less! Something is not right here1

	~~SMC GS16 Switch~~ LAM MTU ~~(note Jumbo frames still reduces throughput, except 3000!)~~		[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-MTU-size.png\|center\|thumb\|500px\|LAM run over range of MTU size]]

	*<a ~~href="netpipe~~.~~Intel-PT-PCIe-throughput_vs_blocksize-lam-MTU-size-gs16.png">LAM run over range of MTU size </a>~~		Perhaps it is the switch. The following results show the difference
			between the 8505T, 8508T and a cross over cable. These results would
			indicate the switches are not working well will large frames (although
			they are advertised to work with large frames)

			[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-mx-switches.png\|center\|thumb\|500px\|Switch Effect on MPICH-MX]]

			Next a better swtich was used, an SMC GS16 Switch. The same tests were run and the variation of MTU (frame) for LAM was recorded Note Jumbo frames still reduces throughput, except at 3000! This switch is of better quality than
			the small switches, maybe it is not a switch issue.

			[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-MTU-size-gs16.png\|center\|thumb\|500px\|LAM run over range of MTU size]]

	==Mystery Is Solved==		==Mystery Is Solved==
	Turn off Flow Control! I turned off Flow Control using Ethtool and the jumbo packets at the high end got much better, but the variability got much worse (see results below). Also, the kernel is now 2.6.26.2 (Using Fedora 8 now) More tests are needed ~~as well~~.		Turn off Flow Control! I turned off Flow Control using Ethtool and the jumbo packets at the high end got much better, but the variability got much worse (see results below). Also, the kernel is now 2.6.26.2 (Using Fedora 8 now) More tests are needed, but we see that flow control was the problem.

			[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-tcp-flowctrl-off-ITR1.png\|center\|thumb\|500px\|Netpipe TCP run over range of MTU size with Flow Control Off ]]

			There are still some issues to resolve and more tests are needed. (coming soon)

	* <a href=~~"netpipe.Intel-PT-PCIe-throughput_vs_blocksize-tcp-flowctrl-off-ITR1.png">Netpipe TCP run over range of MTU size with Flow Control Off </a>~~		==Using Ethtool==

	Here is the Ethtool sequence I used to turn off Flow Control (check the man page for Ethtool for a full description of options).		Here is the Ethtool sequence I used to turn off Flow Control (check the man page for Ethtool for a full description of options).

Deadline at 18:17, 26 December 2008

2008-12-26T18:17:19Z

New page

Initially I started posting results for [http://open-mx.gforge.inria.fr/ Open-MX] over GigE on my [http://limulus.basement-supercomputing.com/wiki/NorbertLimulus Limulus Cluster]. I used Netpipe MPI/TCP (2.4) for most of the tests. As Open MX requires Jumbo packets, I noticed that using Jumbo packets actually reduced the throughput! I'm still in the process of collecting data, so I cannot make any definite conclusions. I just noticed that the latest version of OpenMX (0.9.1) can run over standard frame sizes (1500). More results coming soon.

Kernel: 2.6.23 
CPU: Intel(R) Core(TM)2 Duo CPU E6550 @ 2.33GHz 
Interconnect: Intel 82572EI Gigabit Ethernet PCIe 1X 
Switches: [http://www.smc.com/index.cfm?event=viewProduct&localeCode=EN_USA&cid=6&scid=24&pid=1138 SMC 8505T] (5 port), [http://www.smc.com/index.cfm?event=viewProduct&localeCode=EN_USA&cid=6&scid=24&pid=1147 SMC 8508T] (8 port), [http://www.smc.com/index.cfm?event=viewProduct&localeCode=EN_USA&cid=7&scid=39&pid=1413 SMC GS16] (16 port), and a Cross-over Cable (Note: I tested a 5 port 3com and it would not only negotiate 100BT so it went back. I also tested an 8 port ProCurve and it work similarly to the SMC switch, more tests are needed.) 
MPI: LAM, MPICH-MX (Open MX 0.6)

==Summary Data ==
The following data use the 8505T, 8505T swtiches and a X-over cable:

[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-latency.mx.png|center|thumb|400px|Signature Graph]]
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-signature.mx.png|center|thumb|400px|Latency Graph]]
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize.mx.png|center|thumb|400px|Throughput Graph]]

==Interesting Comparisons Frame Size ==
(8505T, 8505T, X-over)
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-mpich-mx.png|center|thumb|400px|LAM/1500, LAM/9000, MPICH-MX/9000]]
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-MTU-size.png|center|thumb|400px|LAM run over range of MTU size]]
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-mx-switches.png|center|thumb|400px|Switch Effect on MPICH-MX]

* <a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-mpich-mx.png">LAM/1500, LAM/9000, MPICH-MX/9000</a>
* <a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-MTU-size.png">LAM run over range of MTU size (1500-9000)</a>
* <a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-mx-switches.png"></a>

SMC GS16 Switch LAM MTU (note Jumbo frames still reduces throughput, except 3000!)

*<a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-MTU-size-gs16.png">LAM run over range of MTU size </a>

==Mystery Is Solved==
Turn off Flow Control! I turned off Flow Control using Ethtool and the jumbo packets at the high end got much better, but the variability got much worse (see results below). Also, the kernel is now 2.6.26.2 (Using Fedora 8 now) More tests are needed as well.

* <a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-tcp-flowctrl-off-ITR1.png">Netpipe TCP run over range of MTU size with Flow Control Off </a>

Here is the Ethtool sequence I used to turn off Flow Control (check the man page for Ethtool for a full description of options).

# ethtool -a eth2
Pause parameters for eth2:
Autonegotiate: on
RX: on
TX: on

# ethtool -A eth2 autoneg off rx off tx off

# ethtool -a eth2
Pause parameters for eth2:
Autonegotiate: off
RX: off
TX: off

I also set the InterruptThrottleRate to "dynamic". See <TT>~/Documentation/networking/e1000.txt</TT>
in the kernel source directory for an an explanation of this. From the <TT>e1000.txt</TT> file:

''For situations where low latency is vital such as cluster or grid computing, the algorithm can reduce latency even more when InterruptThrottleRate is set to mode 1. In this mode, which operates the same as mode 3, the interruptThrottleRate will be increased stepwise to 70000 for traffic in class "Lowest latency".''

The sequence below sets the InterruptThrottleRate (which is <TT>rx-usecs</TT> to Ethtool) to 1 (dynamic)

# ethtool -c eth2
Coalesce parameters for eth2:
Adaptive RX: off TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 3
rx-frames: 0
rx-usecs-irq: 0
rx-frames-irq: 0

tx-usecs: 0
tx-frames: 0
tx-usecs-irq: 0
tx-frames-irq: 0

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

# ethtool -C eth2 rx-usecs 1

# ethtool -c eth2
Coalesce parameters for eth2:
Adaptive RX: off TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 1
rx-frames: 0
rx-usecs-irq: 0
rx-frames-irq: 0

tx-usecs: 0
tx-frames: 0
tx-usecs-irq: 0
tx-frames-irq: 0

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0