[Beowulf] gigabit ethernet: horrible performance for 0 byte messages

Joel Jaeggli joelja at darkwing.uoregon.edu
Thu Feb 12 23:17:26 EST 2004


on varius revs of their code I've had regular (once a week) managment
stack crash on our dell switches which doesn't make it easy to collect
statistics, but they continue to forward packets just fine... the switches
are actually made by accton and they are also sold by smc...  depending
one who has better deals the dell 5212/5224 or smc 8612t/8624t may be
cheaper at any given time... the cisco cat-ios style cli and ssh support 
are a plus.

On Thu, 12 Feb 2004, Brent M. Clements wrote:

> The best switch that we have found both in price and speed are the GigE
> Switches from Dell. We use them in a few of our test clusters and smaller
> clusters. They are actually pretty good performers and top even some of
> the cisco switches.
> 
> -Brent
> 
> Brent Clements
> Linux Technology Specialist
> Information Technology
> Rice University
> 
> 
> On Wed, 11 Feb 2004, Gerry Creager N5JXS wrote:
> 
> > Realize that not all switches are created equal when working with small
> > (and, overall, 0-byte == small) packets.  A number of otherwise decent
> > network switches are less than stellar performers with small packets.
> > We've evaluated this in my lab with an Anritsu MD-1230 Ethernet test
> > system running under the RFC-2544 testing suite...
> >
> > There are switches that perform well with small packets, but it's been
> > our experience that most switches, especially your lower cost switches
> > (Cisco 2900/2950/3500, 4000/4500; Allied Telesyn *; Cabletron *; some
> > others I can't recall right now) didn't perform well with smaller
> > packets but did fine when the packet size was about 1500 bytes.
> >
> > Going with cheap switches is usually not a good way to improve performance.
> >
> > gerry
> >
> > Douglas Eadline, Cluster World Magazine wrote:
> > > On Wed, 11 Feb 2004, Bernhard Wegner wrote:
> > >
> > >
> > >>Hello,
> > >>
> > >>I have a really small "cluster" of 4 PC's which are connected by a normal
> > >>Ethernet 100 Mbit switch. Because the motherboards have Gigabit-LAN on board
> > >>I thought I might be able to improve performance by connecting the machines
> > >>via a Gigabit switch (which are really cheap nowadays).
> > >>
> > >>Everything seemed to work fine. The switch indicates 1000Mbit connections to
> > >>the PC's and transfer rate for scp-ing large files is significantly higher
> > >>now, but my software unsing mpich RUNS about a factor of 4-5 SLOWER NOW than
> > >>with the 100 Mbit switch.
> > >>
> > >>I wasn't able to actually track down the problem, but it seems that there is
> > >>a problem with small messages. When I run the performance test provided with
> > >>mpich, it reports (bshort2/bshort4) extremely long times (e.g. 1500 us) for 0
> > >>byte message length, while for larger messages everything looks fine (linear
> > >>dependancy of transfer time on message length, everything below 300 us). I
> > >>have also tried mpich2 which shows exactly the same behavior.
> > >>
> > >>Does anyone have any idea?
> > >
> > >
> > > First, I assume you were running the 100BT through the same
> > > onboard NICs and got reasonable performance. So some possible
> > > things:
> > >
> > > - the switch is a dog or it is broken
> > > - your cables may be old or bad (but worked fine for 100BT)
> > > - negotiation problem
> > >
> > > Some things to try:
> > >
> > > Use a cross over cable (cat5e) and see if you get the same problem.
> > > You might try using a lower level benchmark (of the micro variety)
> > > like netperf and netpipe.
> > >
> > > The Beowulf Performance Suite:
> > > http://www.clusterworld.com/article.pl?sid=03/03/17/1838236
> > >
> > > has these tests. Also, the December and January issues of ClusterWorld
> > > show how to test a network connection using netpipe. At some point this
> > > content will be showing up on the web-page.
> > >
> > > Also, the MPI Link-checker from Microway (www.microway.com)
> > >
> > > http://www.clusterworld.com/article.pl?sid=04/02/09/1952250
> > >
> > > May help.
> > >
> > >
> > > Doug
> > >
> > >
> > >>Here are the details of my system:
> > >> - Suse Linux 9.0 (kernel 2.4.21)
> > >> - mpich-1.2.5.2
> > >> - motherboard ASUS P4P800
> > >> - LAN (10/100/1000) on board (3COM 3C940 chipset)
> > >> - LevelOne 10/100/1000 8-port Fast Ethernet Switch (chipset: TC9208M
> > >
> > > +
> > >
> > >>   8x88E1111-BAB, AT89C2051-24PI)
> > >>
> > >>
> > >
> > >
> >
> > --
> > Gerry Creager -- gerry.creager at tamu.edu
> > Network Engineering -- AATLT, Texas A&M University
> > Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578
> > Page: 979.228.0173
> > Office: 903A Eller Bldg, TAMU, College Station, TX 77843
> >
> > _______________________________________________
> > Beowulf mailing list, Beowulf at beowulf.org
> > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> >
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> 

-- 
-------------------------------------------------------------------------- 
Joel Jaeggli  	       Unix Consulting 	       joelja at darkwing.uoregon.edu    
GPG Key Fingerprint:     5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2


_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



More information about the Beowulf mailing list