Channel-bonding 2.2 vs 2.4?
Mike Weller
weller at zyvex.com
Mon Aug 6 17:27:55 EDT 2001
Hello, I started a thread on the list about a month ago, and I've
finally found some time to play with channel-bonding again. I'm
still having problems.
Here's a memory-refresher:
My switch (HP Procurve 4000) supports load-balancing/trunking and
VLANs. I was taking the wrong approach, apparently.
1) At first, I was attempted to do load-balancing with the switch. I'd
create individual trunk groups for the channel-bonded ports. This
had moderate success, but wasn't what I wanted since the switch
would distribute traffic based on the source address. I could
never get 200Mbps this way.
2) Someone on the beowulf at beowulf.org mailing list suggested setting
up a VLAN. I created 2 VLANs to isolate the networks. To make a
long story short, my switch supports VLANs, but duplicate MAC
addresses confuse it.
3) I followed up on this suggestion provided by Jakob:
From: <jakob at unthought.net>
>You can buy separate switches, unmanaged dirt cheap ones.
>Trunking N nics per node, requires N switches.
>This works, is simple, and it's cheap.
I tried dirt cheap linksys switches... put all eth0's on 1 switch
and all eth1's on the other switch. The RX and TX's are even on
both sides. I can communicate with it. Unfortunately, I only
achieve 50Mbps as opposed to 200Mbps :-( Using the same hardware
and OS, I achieve 100Mbps in un-channel-bonded mode.
4) Someone else on the list wrote:
From: R C <zarquon at zarq.dhs.org>
>As you found out, the HPs don't support duplicate NATs on seperate
>VLANs (even states this in the manual, but I had to dig to find it).
>At that point I borrowed the lab switch and hooked it in. Using
>2.4.x, and 2 channels, I got about 160-170 Mbps with an Intel eepro100
>(onboard) and an Adaptec duralan (starfire?) card. The adaptec
>tended to hiccup and give "Something Wicked Happened!" during full
>speed tests, but rarely pulled down below 150 Mbps.
>
>Tests were done using ttcp, linux 2.4.4?, and Dell server boxes
>(dual 500, 512MB).
I noticed that he was using the 2.4 kernel and Scyld was still
using 2.2. I started to wonder if there were bonding issues with
the 2.2 kernel. I decided to take Scyld out of the equation and
using 2 linux boxes with the 2.4.7 kernel (with a mandrake base):
[root at n0 /tmp]# uname -a
Linux n0 2.4.7mw2 #2 Tue Jul 24 19:54:20 CDT 2001 i686 unknown
>From two identical machines that both use the 2.4.7 kernel, and
connected with a linksys switch:
1 NIC (unchannel-boneded): 10000 Kbytes/sec
2 NICs (channel-bonded): 6000 Kbytes/sec
1 NIC transfer with FTP: (PUT and GET):
150 Opening BINARY mode data connection for kern.tar.gz.
8186400 bytes sent in 0.765 secs (1e+04 Kbytes/sec)
150 Opening BINARY mode data connection for kern.tar.gz (8186400 bytes).
8186400 bytes received in 0.779 secs (1e+04 Kbytes/sec)
2 NICs:
[root at n0 /tmp]# insmod bonding
Using /lib/modules/2.4.7mw2/kernel/drivers/net/bonding.o
[root at n0 /tmp]# ifconfig bond0 inet 10.0.0.2 netmask 255.255.255.0
[root at n0 /tmp]# ifenslave bond0 eth0
master has no hw address assigned; getting one from slave!
The interface eth0 is up, shutting it down it to enslave it.
[root at n0 /tmp]# ifenslave bond0 eth1
FTP transfer:
150 Opening BINARY mode data connection for kern.tar.gz.
8186400 bytes sent in 1.44 secs (5.6e+03 Kbytes/sec)
150 Opening BINARY mode data connection for kern.tar.gz (8186400 bytes).
8186400 bytes received in 1.28 secs (6.2e+03 Kbytes/sec)
So, it's not Scyld's 2.2 kernel causing this. The switches are
capable of performing 10000 Kbytes/sec as proven by the single-NIC
transfer rates, so one would think that it's completely OS-dependent.
To further verify that the BONDING is setup properly, I monitored
the RX and TX packets of the bond0, eth0 and eth1 interfaces on
both ends, to make sure that BOTH NICs were operational:
One bug I noticed was that the RX packets for bond0 is 0. You can
add up TX for eth0 and eth1 to give you the TX for bond0. I don't
know if this is relevant since it's functional:
[root at n0 /root]# ifconfig -a
bond0 Link encap:Ethernet HWaddr 00:01:02:ED:C4:C8
inet addr:10.0.0.2 Bcast:10.255.255.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:42282 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:36870337 (35.1 Mb)
eth0 Link encap:Ethernet HWaddr 00:01:02:ED:C4:C8
inet addr:10.0.0.2 Bcast:10.255.255.255 Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:31673 errors:0 dropped:0 overruns:488 frame:0
TX packets:39068 errors:0 dropped:0 overruns:0 carrier:6
collisions:0 txqueuelen:100
RX bytes:24773173 (23.6 Mb) TX bytes:40110808 (38.2 Mb)
Interrupt:11 Base address:0xe000
eth1 Link encap:Ethernet HWaddr 00:01:02:ED:C4:C8
inet addr:10.0.0.2 Bcast:10.255.255.255 Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:17075 errors:0 dropped:0 overruns:458 frame:0
TX packets:17475 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:13483933 (12.8 Mb) TX bytes:14076285 (13.4 Mb)
Interrupt:9 Base address:0xe400
Using my "measure" script which takes differences, here's a before
and after:
root at n0 /root]# ./measure
bond0 rx 0 tx 0
eth0 rx 0 tx 0
eth1 rx 0 tx 0
<do the FTP transfer from 10.0.0.2 to 10.0.0.3>
[root at n0 /root]# ./measure
bond0 rx 0 tx 6532
eth0 rx 3269 tx 3266
eth1 rx 3264 tx 3266
The rx's and tx's matched on the other end as well. the RX for bond0
is always 0 (bug?)
Any idea on what I should try next?
Thanks!
--
Michael J. Weller, M.Sc. office: (972) 235-7881 x.242
weller at zyvex.com cell: (214) 616-6340
Zyvex Corp., 1321 N Plano facsimile: (972) 235-7882
Richardson, TX 75081 icq: 6180540
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
More information about the Beowulf
mailing list