[Beowulf] gigabit ethernet: horrible performance for 0 byte messages
Brent M. Clements
bclem at rice.edu
Thu Feb 12 22:22:02 EST 2004
The best switch that we have found both in price and speed are the GigE
Switches from Dell. We use them in a few of our test clusters and smaller
clusters. They are actually pretty good performers and top even some of
the cisco switches.
Linux Technology Specialist
On Wed, 11 Feb 2004, Gerry Creager N5JXS wrote:
> Realize that not all switches are created equal when working with small
> (and, overall, 0-byte == small) packets. A number of otherwise decent
> network switches are less than stellar performers with small packets.
> We've evaluated this in my lab with an Anritsu MD-1230 Ethernet test
> system running under the RFC-2544 testing suite...
> There are switches that perform well with small packets, but it's been
> our experience that most switches, especially your lower cost switches
> (Cisco 2900/2950/3500, 4000/4500; Allied Telesyn *; Cabletron *; some
> others I can't recall right now) didn't perform well with smaller
> packets but did fine when the packet size was about 1500 bytes.
> Going with cheap switches is usually not a good way to improve performance.
> Douglas Eadline, Cluster World Magazine wrote:
> > On Wed, 11 Feb 2004, Bernhard Wegner wrote:
> >>I have a really small "cluster" of 4 PC's which are connected by a normal
> >>Ethernet 100 Mbit switch. Because the motherboards have Gigabit-LAN on board
> >>I thought I might be able to improve performance by connecting the machines
> >>via a Gigabit switch (which are really cheap nowadays).
> >>Everything seemed to work fine. The switch indicates 1000Mbit connections to
> >>the PC's and transfer rate for scp-ing large files is significantly higher
> >>now, but my software unsing mpich RUNS about a factor of 4-5 SLOWER NOW than
> >>with the 100 Mbit switch.
> >>I wasn't able to actually track down the problem, but it seems that there is
> >>a problem with small messages. When I run the performance test provided with
> >>mpich, it reports (bshort2/bshort4) extremely long times (e.g. 1500 us) for 0
> >>byte message length, while for larger messages everything looks fine (linear
> >>dependancy of transfer time on message length, everything below 300 us). I
> >>have also tried mpich2 which shows exactly the same behavior.
> >>Does anyone have any idea?
> > First, I assume you were running the 100BT through the same
> > onboard NICs and got reasonable performance. So some possible
> > things:
> > - the switch is a dog or it is broken
> > - your cables may be old or bad (but worked fine for 100BT)
> > - negotiation problem
> > Some things to try:
> > Use a cross over cable (cat5e) and see if you get the same problem.
> > You might try using a lower level benchmark (of the micro variety)
> > like netperf and netpipe.
> > The Beowulf Performance Suite:
> > http://www.clusterworld.com/article.pl?sid=03/03/17/1838236
> > has these tests. Also, the December and January issues of ClusterWorld
> > show how to test a network connection using netpipe. At some point this
> > content will be showing up on the web-page.
> > Also, the MPI Link-checker from Microway (www.microway.com)
> > http://www.clusterworld.com/article.pl?sid=04/02/09/1952250
> > May help.
> > Doug
> >>Here are the details of my system:
> >> - Suse Linux 9.0 (kernel 2.4.21)
> >> - mpich-18.104.22.168
> >> - motherboard ASUS P4P800
> >> - LAN (10/100/1000) on board (3COM 3C940 chipset)
> >> - LevelOne 10/100/1000 8-port Fast Ethernet Switch (chipset: TC9208M
> > +
> >> 8x88E1111-BAB, AT89C2051-24PI)
> Gerry Creager -- gerry.creager at tamu.edu
> Network Engineering -- AATLT, Texas A&M University
> Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578
> Page: 979.228.0173
> Office: 903A Eller Bldg, TAMU, College Station, TX 77843
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
More information about the Beowulf