Channel-bonding 2.2 vs 2.4?

Mike Weller weller at zyvex.com
Mon Aug 6 17:27:55 EDT 2001


Hello, I started a thread on the list about a month ago, and I've
finally found some time to play with channel-bonding again.  I'm
still  having problems.

Here's a memory-refresher:

My switch (HP Procurve 4000) supports load-balancing/trunking and
VLANs.  I was taking the wrong approach, apparently.

1) At first, I was attempted to do load-balancing with the switch. I'd
   create individual trunk groups for the channel-bonded ports.  This
   had moderate success, but wasn't what I wanted since the switch
   would distribute traffic based on the source address.  I could
   never get 200Mbps this way.

2) Someone on the beowulf at beowulf.org mailing list suggested setting
   up a VLAN.  I created 2 VLANs to isolate the networks.  To make a
   long story short, my switch supports VLANs, but duplicate MAC
   addresses confuse it.

3) I followed up on this suggestion provided by Jakob:

  From: <jakob at unthought.net> 
  >You can buy separate switches, unmanaged dirt cheap ones.  
  >Trunking N nics per node, requires N switches.
  >This works, is simple, and it's cheap.

   I tried dirt cheap linksys switches... put all eth0's on 1 switch
   and all eth1's on the other switch.  The RX and TX's are even on
   both sides.  I can communicate with it.  Unfortunately, I only
   achieve 50Mbps as opposed to 200Mbps :-( Using the same hardware
   and OS, I achieve 100Mbps in un-channel-bonded mode.

4) Someone else on the list wrote:

From: R C <zarquon at zarq.dhs.org>
  >As you found out, the HPs don't support duplicate NATs on seperate 
  >VLANs (even states this in the manual, but I had to dig to find it).
  >At that point I borrowed the lab switch and hooked it in.  Using
  >2.4.x, and 2 channels, I got about 160-170 Mbps with an Intel eepro100
  >(onboard) and an Adaptec duralan (starfire?) card.  The adaptec
  >tended to hiccup and give "Something Wicked Happened!" during full
  >speed tests, but rarely pulled down below 150 Mbps.
  >
  >Tests were done using ttcp, linux 2.4.4?, and Dell server boxes
  >(dual 500, 512MB).

   I noticed that he was using the 2.4 kernel and Scyld was still
   using 2.2.  I started to wonder if there were bonding issues with
   the 2.2 kernel.  I decided to take Scyld out of the equation and
   using 2 linux boxes with the 2.4.7 kernel (with a mandrake base):

[root at n0 /tmp]# uname -a
Linux n0 2.4.7mw2 #2 Tue Jul 24 19:54:20 CDT 2001 i686 unknown

>From two identical machines that both use the 2.4.7 kernel, and
connected with a linksys switch:
1 NIC (unchannel-boneded): 10000 Kbytes/sec 
2 NICs (channel-bonded): 6000 Kbytes/sec

1 NIC transfer with FTP: (PUT and GET):
150 Opening BINARY mode data connection for kern.tar.gz.
8186400 bytes sent in 0.765 secs (1e+04 Kbytes/sec)

150 Opening BINARY mode data connection for kern.tar.gz (8186400 bytes).
8186400 bytes received in 0.779 secs (1e+04 Kbytes/sec)

2 NICs:

[root at n0 /tmp]# insmod bonding
Using /lib/modules/2.4.7mw2/kernel/drivers/net/bonding.o
[root at n0 /tmp]# ifconfig bond0 inet 10.0.0.2 netmask 255.255.255.0
[root at n0 /tmp]# ifenslave bond0 eth0
master has no hw address assigned; getting one from slave!
The interface eth0 is up, shutting it down it to enslave it.
[root at n0 /tmp]# ifenslave bond0 eth1

FTP transfer:
150 Opening BINARY mode data connection for kern.tar.gz.
8186400 bytes sent in 1.44 secs (5.6e+03 Kbytes/sec)

150 Opening BINARY mode data connection for kern.tar.gz (8186400 bytes).
8186400 bytes received in 1.28 secs (6.2e+03 Kbytes/sec)

So, it's not Scyld's 2.2 kernel causing this.  The switches are
capable of performing 10000 Kbytes/sec as proven by the single-NIC
transfer rates, so one would think that it's completely OS-dependent.


To further verify that the BONDING is setup properly, I monitored
the RX and TX packets of the bond0, eth0 and eth1 interfaces on
both ends, to make sure that BOTH NICs were operational:

One bug I noticed was that the RX packets for bond0 is 0. You can
add up TX for eth0 and eth1 to give you the TX for bond0. I don't
know if this is relevant since it's functional:

[root at n0 /root]# ifconfig -a
bond0     Link encap:Ethernet  HWaddr 00:01:02:ED:C4:C8
          inet addr:10.0.0.2  Bcast:10.255.255.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:42282 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:36870337 (35.1 Mb)

eth0      Link encap:Ethernet  HWaddr 00:01:02:ED:C4:C8
          inet addr:10.0.0.2  Bcast:10.255.255.255  Mask:255.255.255.0
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:31673 errors:0 dropped:0 overruns:488 frame:0
          TX packets:39068 errors:0 dropped:0 overruns:0 carrier:6
          collisions:0 txqueuelen:100
          RX bytes:24773173 (23.6 Mb)  TX bytes:40110808 (38.2 Mb)
          Interrupt:11 Base address:0xe000

eth1      Link encap:Ethernet  HWaddr 00:01:02:ED:C4:C8
          inet addr:10.0.0.2  Bcast:10.255.255.255  Mask:255.255.255.0
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:17075 errors:0 dropped:0 overruns:458 frame:0
          TX packets:17475 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:13483933 (12.8 Mb)  TX bytes:14076285 (13.4 Mb)
          Interrupt:9 Base address:0xe400 


Using my "measure" script which takes differences, here's a before
and after:
root at n0 /root]# ./measure
bond0 rx 0 tx 0
eth0 rx 0 tx 0
eth1 rx 0 tx 0

<do the FTP transfer from 10.0.0.2 to 10.0.0.3>

[root at n0 /root]# ./measure
bond0 rx 0 tx 6532
eth0 rx 3269 tx 3266
eth1 rx 3264 tx 3266

The rx's and tx's matched on the other end as well.  the RX for bond0
is always 0 (bug?)


Any idea on what I should try next? 
Thanks!

-- 
Michael J. Weller, M.Sc.               office: (972) 235-7881 x.242
weller at zyvex.com                         cell: (214) 616-6340
Zyvex Corp., 1321 N Plano           facsimile: (972) 235-7882    
Richardson, TX 75081                      icq: 6180540


_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



More information about the Beowulf mailing list