Editing Small GigE Switches

Jump to: navigation, search

Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.
Latest revision Your text
Line 8: Line 8:
  
 
==Summary Data ==
 
==Summary Data ==
−
The following data use the 8505T, 8508T switches and a X-over cable. Netpipe MPI using MPICH-MX and LAM(TCP) are shown. In addition Netpipe TCP results are given for comparison. Click on the images to get high resolution
+
The following data use the 8505T, 8505T swtiches and a X-over cable:
−
versions.
 
  
−
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-latency.mx.png|center|thumb|500px|Signature Graph]]
+
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-latency.mx.png|center|thumb|400px|Signature Graph]]
−
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-signature.mx.png|center|thumb|500px|Latency Graph]]
+
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-signature.mx.png|center|thumb|400px|Latency Graph]]
−
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize.mx.png|center|thumb|500px|Throughput  Graph]]
+
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize.mx.png|center|thumb|400px|Throughput  Graph]]
  
  
  
 
==Interesting Comparisons Frame Size ==
 
==Interesting Comparisons Frame Size ==
−
If some specific data are compared we find some surprising results. First, if you compare LAM using a 1500 byte MTU (frame), LAM using a 9000 byte MTU, and  MPICH-MX using a 9000 byte MTU, you find the that best through put comes from smaller frame size! This result is opposite of what you would expect -- bigger frames better thoughput!
+
(8505T, 8505T, X-over)
 +
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-mpich-mx.png|center|thumb|400px|LAM/1500, LAM/9000, MPICH-MX/9000]]
 +
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-MTU-size.png|center|thumb|400px|LAM run over range of MTU size]]
 +
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-mx-switches.png|center|thumb|400px|Switch Effect on MPICH-MX]
  
−
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-mpich-mx.png|center|thumb|500px|LAM/1500, LAM/9000, MPICH-MX/9000]]
+
* <a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-mpich-mx.png">LAM/1500, LAM/9000, MPICH-MX/9000</a>
 +
* <a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-MTU-size.png">LAM run over range of MTU size (1500-9000)</a>
 +
* <a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-mx-switches.png"></a>
  
−
The above result suggested another experiment. In the following graph LAM MPi was run over a range for MTU (frame) sizes. As the frame gets bigger, the
 
−
throughput gets less! Something is not right here1
 
  
−
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-MTU-size.png|center|thumb|500px|LAM run over range of MTU size]]
+
SMC GS16 Switch LAM MTU (note Jumbo frames still reduces throughput, except 3000!)
  
−
Perhaps it is the switch. The following results show the difference
+
*<a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-MTU-size-gs16.png">LAM run over range of MTU size </a>
−
between the 8505T, 8508T and a cross over cable. These results would
 
−
indicate the switches are not working well will large frames (although
 
−
they are advertised to work with large frames)
 
  
−
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-mx-switches.png|center|thumb|500px|Switch Effect on MPICH-MX]]
 
  
−
Next a better swtich was used, an SMC GS16 Switch. The same tests were run and the variation of MTU (frame) for LAM was recorded Note Jumbo frames still reduces throughput, except at 3000! This switch is of better quality than
 
−
the small switches, maybe it is not a switch issue.
 
−
 
−
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-lam-MTU-size-gs16.png|center|thumb|500px|LAM run over range of MTU size]]
 
  
 
==Mystery Is Solved==
 
==Mystery Is Solved==
−
Turn off Flow Control! I turned off Flow Control using Ethtool and the jumbo packets at the high end got much better, but the variability got much worse (see results below). Also, the kernel is now 2.6.26.2 (Using Fedora 8 now) More tests are needed, but we see that flow control was the problem.
+
Turn off Flow Control! I turned off Flow Control using Ethtool and the jumbo packets at the high end got much better, but the variability got much worse (see results below). Also, the kernel is now 2.6.26.2 (Using Fedora 8 now) More tests are needed as well.
−
 
 
−
[[Image:Small_GigE_Switches_netpipe.Intel-PT-PCIe-throughput_vs_blocksize-tcp-flowctrl-off-ITR1.png|center|thumb|500px|Netpipe TCP run over range of MTU size with Flow Control Off ]]
 
−
 
 
−
There are still some issues to resolve and more tests are needed. (coming soon)
 
  
−
==Using Ethtool==
+
* <a href="netpipe.Intel-PT-PCIe-throughput_vs_blocksize-tcp-flowctrl-off-ITR1.png">Netpipe TCP run over range of MTU size with Flow Control Off </a>
  
 
Here is the Ethtool sequence I used to turn off Flow Control (check the man page for Ethtool for a full description of options).
 
Here is the Ethtool sequence I used to turn off Flow Control (check the man page for Ethtool for a full description of options).

Please note that all contributions to Cluster Documentation Project are considered to be released under the Attribution-NonCommercial-ShareAlike 2.5 (see Cluster Documentation Project:Copyrights for details). If you do not want your writing to be edited mercilessly and redistributed at will, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource. Do not submit copyrighted work without permission!

To protect the wiki against automated edit spam, we kindly ask you to solve the following CAPTCHA:

Cancel | Editing help (opens in new window)
Retrieved from "https://www.clustermonkey.net/cdp/index.php/Small_GigE_Switches"