[Beowulf] gigabit ethernet: horrible performance for 0 byte messages
Douglas Eadline, Cluster World Magazine
deadline at linux-mag.com
Fri Feb 13 16:54:28 EST 2004
I wondered about your low cost switch statement. I had done this test
before, but I thought I would redo it anyway. I have an SMC 8 port GigE
EasySwitch 8508T (PriceGrabber $140 to my door). I should say that the
switch is not loaded, so it may fall down if the load were higher.
This is just two nodes running netpipe through the switch.
Now starting main loop
0: 1 bytes 7287 times --> 0.22 Mbps in 0.000034 sec
1: 2 bytes 7338 times --> 0.46 Mbps in 0.000033 sec
2: 3 bytes 7469 times --> 0.68 Mbps in 0.000034 sec
3: 4 bytes 4923 times --> 0.90 Mbps in 0.000034 sec
4: 6 bytes 5545 times --> 1.36 Mbps in 0.000034 sec
5: 8 bytes 3711 times --> 1.81 Mbps in 0.000034 sec
6: 12 bytes 4637 times --> 2.67 Mbps in 0.000034 sec
My opinion: If you get a switch that can not "switch" then it
is broken by design. The original poster noted that his results seem to go
from OK to "really bad" for basic MPI tests. If a switch does this it is
"really broken". Of course it may not be the switch.
BTW, the results were for a $30 NIC (netgear GA302T) running in
a 66MHz slot. Top throughput was 800 Mbits/sec.
On Wed, 11 Feb 2004, Gerry Creager N5JXS wrote:
> Realize that not all switches are created equal when working with small
> (and, overall, 0-byte == small) packets. A number of otherwise decent
> network switches are less than stellar performers with small packets.
> We've evaluated this in my lab with an Anritsu MD-1230 Ethernet test
> system running under the RFC-2544 testing suite...
> There are switches that perform well with small packets, but it's been
> our experience that most switches, especially your lower cost switches
> (Cisco 2900/2950/3500, 4000/4500; Allied Telesyn *; Cabletron *; some
> others I can't recall right now) didn't perform well with smaller
> packets but did fine when the packet size was about 1500 bytes.
> Going with cheap switches is usually not a good way to improve performance.
> Douglas Eadline, Cluster World Magazine wrote:
> > On Wed, 11 Feb 2004, Bernhard Wegner wrote:
> >>I have a really small "cluster" of 4 PC's which are connected by a normal
> >>Ethernet 100 Mbit switch. Because the motherboards have Gigabit-LAN on board
> >>I thought I might be able to improve performance by connecting the machines
> >>via a Gigabit switch (which are really cheap nowadays).
> >>Everything seemed to work fine. The switch indicates 1000Mbit connections to
> >>the PC's and transfer rate for scp-ing large files is significantly higher
> >>now, but my software unsing mpich RUNS about a factor of 4-5 SLOWER NOW than
> >>with the 100 Mbit switch.
> >>I wasn't able to actually track down the problem, but it seems that there is
> >>a problem with small messages. When I run the performance test provided with
> >>mpich, it reports (bshort2/bshort4) extremely long times (e.g. 1500 us) for 0
> >>byte message length, while for larger messages everything looks fine (linear
> >>dependancy of transfer time on message length, everything below 300 us). I
> >>have also tried mpich2 which shows exactly the same behavior.
> >>Does anyone have any idea?
> > First, I assume you were running the 100BT through the same
> > onboard NICs and got reasonable performance. So some possible
> > things:
> > - the switch is a dog or it is broken
> > - your cables may be old or bad (but worked fine for 100BT)
> > - negotiation problem
> > Some things to try:
> > Use a cross over cable (cat5e) and see if you get the same problem.
> > You might try using a lower level benchmark (of the micro variety)
> > like netperf and netpipe.
> > The Beowulf Performance Suite:
> > http://www.clusterworld.com/article.pl?sid=03/03/17/1838236
> > has these tests. Also, the December and January issues of ClusterWorld
> > show how to test a network connection using netpipe. At some point this
> > content will be showing up on the web-page.
> > Also, the MPI Link-checker from Microway (www.microway.com)
> > http://www.clusterworld.com/article.pl?sid=04/02/09/1952250
> > May help.
> > Doug
> >>Here are the details of my system:
> >> - Suse Linux 9.0 (kernel 2.4.21)
> >> - mpich-126.96.36.199
> >> - motherboard ASUS P4P800
> >> - LAN (10/100/1000) on board (3COM 3C940 chipset)
> >> - LevelOne 10/100/1000 8-port Fast Ethernet Switch (chipset: TC9208M
> > +
> >> 8x88E1111-BAB, AT89C2051-24PI)
Editor-in-chief ClusterWorld Magazine
Cell: 610.390.7765 Redefining High Performance Computing
Fax: 610.865.6618 www.clusterworld.com
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
More information about the Beowulf