[Beowulf] gigabit ethernet: horrible performance for 0 byte messages

Joel Jaeggli joelja at darkwing.uoregon.edu
Thu Feb 12 23:19:16 EST 2004


Also they support jumbo (9k) frames which is a plus for us since we do nfs 
over them.

joelja

On Thu, 12 Feb 2004, Joel Jaeggli wrote:

> on varius revs of their code I've had regular (once a week) managment
> stack crash on our dell switches which doesn't make it easy to collect
> statistics, but they continue to forward packets just fine... the switches
> are actually made by accton and they are also sold by smc...  depending
> one who has better deals the dell 5212/5224 or smc 8612t/8624t may be
> cheaper at any given time... the cisco cat-ios style cli and ssh support 
> are a plus.
> 
> On Thu, 12 Feb 2004, Brent M. Clements wrote:
> 
> > The best switch that we have found both in price and speed are the GigE
> > Switches from Dell. We use them in a few of our test clusters and smaller
> > clusters. They are actually pretty good performers and top even some of
> > the cisco switches.
> > 
> > -Brent
> > 
> > Brent Clements
> > Linux Technology Specialist
> > Information Technology
> > Rice University
> > 
> > 
> > On Wed, 11 Feb 2004, Gerry Creager N5JXS wrote:
> > 
> > > Realize that not all switches are created equal when working with small
> > > (and, overall, 0-byte == small) packets.  A number of otherwise decent
> > > network switches are less than stellar performers with small packets.
> > > We've evaluated this in my lab with an Anritsu MD-1230 Ethernet test
> > > system running under the RFC-2544 testing suite...
> > >
> > > There are switches that perform well with small packets, but it's been
> > > our experience that most switches, especially your lower cost switches
> > > (Cisco 2900/2950/3500, 4000/4500; Allied Telesyn *; Cabletron *; some
> > > others I can't recall right now) didn't perform well with smaller
> > > packets but did fine when the packet size was about 1500 bytes.
> > >
> > > Going with cheap switches is usually not a good way to improve performance.
> > >
> > > gerry
> > >
> > > Douglas Eadline, Cluster World Magazine wrote:
> > > > On Wed, 11 Feb 2004, Bernhard Wegner wrote:
> > > >
> > > >
> > > >>Hello,
> > > >>
> > > >>I have a really small "cluster" of 4 PC's which are connected by a normal
> > > >>Ethernet 100 Mbit switch. Because the motherboards have Gigabit-LAN on board
> > > >>I thought I might be able to improve performance by connecting the machines
> > > >>via a Gigabit switch (which are really cheap nowadays).
> > > >>
> > > >>Everything seemed to work fine. The switch indicates 1000Mbit connections to
> > > >>the PC's and transfer rate for scp-ing large files is significantly higher
> > > >>now, but my software unsing mpich RUNS about a factor of 4-5 SLOWER NOW than
> > > >>with the 100 Mbit switch.
> > > >>
> > > >>I wasn't able to actually track down the problem, but it seems that there is
> > > >>a problem with small messages. When I run the performance test provided with
> > > >>mpich, it reports (bshort2/bshort4) extremely long times (e.g. 1500 us) for 0
> > > >>byte message length, while for larger messages everything looks fine (linear
> > > >>dependancy of transfer time on message length, everything below 300 us). I
> > > >>have also tried mpich2 which shows exactly the same behavior.
> > > >>
> > > >>Does anyone have any idea?
> > > >
> > > >
> > > > First, I assume you were running the 100BT through the same
> > > > onboard NICs and got reasonable performance. So some possible
> > > > things:
> > > >
> > > > - the switch is a dog or it is broken
> > > > - your cables may be old or bad (but worked fine for 100BT)
> > > > - negotiation problem
> > > >
> > > > Some things to try:
> > > >
> > > > Use a cross over cable (cat5e) and see if you get the same problem.
> > > > You might try using a lower level benchmark (of the micro variety)
> > > > like netperf and netpipe.
> > > >
> > > > The Beowulf Performance Suite:
> > > > http://www.clusterworld.com/article.pl?sid=03/03/17/1838236
> > > >
> > > > has these tests. Also, the December and January issues of ClusterWorld
> > > > show how to test a network connection using netpipe. At some point this
> > > > content will be showing up on the web-page.
> > > >
> > > > Also, the MPI Link-checker from Microway (www.microway.com)
> > > >
> > > > http://www.clusterworld.com/article.pl?sid=04/02/09/1952250
> > > >
> > > > May help.
> > > >
> > > >
> > > > Doug
> > > >
> > > >
> > > >>Here are the details of my system:
> > > >> - Suse Linux 9.0 (kernel 2.4.21)
> > > >> - mpich-1.2.5.2
> > > >> - motherboard ASUS P4P800
> > > >> - LAN (10/100/1000) on board (3COM 3C940 chipset)
> > > >> - LevelOne 10/100/1000 8-port Fast Ethernet Switch (chipset: TC9208M
> > > >
> > > > +
> > > >
> > > >>   8x88E1111-BAB, AT89C2051-24PI)
> > > >>
> > > >>
> > > >
> > > >
> > >
> > > --
> > > Gerry Creager -- gerry.creager at tamu.edu
> > > Network Engineering -- AATLT, Texas A&M University
> > > Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578
> > > Page: 979.228.0173
> > > Office: 903A Eller Bldg, TAMU, College Station, TX 77843
> > >
> > > _______________________________________________
> > > Beowulf mailing list, Beowulf at beowulf.org
> > > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> > >
> > _______________________________________________
> > Beowulf mailing list, Beowulf at beowulf.org
> > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> > 
> 
> 

-- 
-------------------------------------------------------------------------- 
Joel Jaeggli  	       Unix Consulting 	       joelja at darkwing.uoregon.edu    
GPG Key Fingerprint:     5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2


_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



More information about the Beowulf mailing list