[Beowulf] Bonding Gbe using tg3 drivers on an HP cluster

Steffen Persvold sp at scali.com
Mon Nov 14 03:02:16 EST 2005


Walid,

To achieve your objective you need two switches (or back-to-back cables).
"Trunking" mode on a switch is not the same as "round-robin", i.e it will
not alternate ports when sending data. Trunking mode in a switch usually
implies a hashing algorithm where the sender and receiver MAC address is put
through a hash to find which port to use. Thus, you can't achieve more than
1Gps with two servers only. However, if you had a large network (with a lot
of different MAC addresses), you would get more than 1Gps from one server,
_if_ it was communicating with more than 1 peer at the same time.

Kind regards,

Steffen Persvold
Technical Director Americas
tel. 508-281-7100 x401
fax. 508-281-7171

http://www.scali.com/
Scaling the Linux datacenter
 

> -----Original Message-----
> From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On
> Behalf Of Walid
> Sent: Thursday, November 10, 2005 2:40 PM
> To: beowulf at beowulf.org
> Subject: [Beowulf] Bonding Gbe using tg3 drivers on an HP cluster
> 
> Dear All,
> 
> Reading through the bonding.txt document, I am failing to achive more
> than 1Gbps on a two  bonded  builtin 1 Gbe network interfaces on a
> proliant server DL-380,  what are we doing wrong, please find below
> our configuration
> 
> Walid.
> 
> 
> 
> Here is the test configuration we have
> 
>  +----------+                 +---------------+
>   +--------+
>   |           |eth0  port-E3|               +--------------------------+
> host2
>   |  host1 +-------------+  switch    | port-E4                    |
>   |          +-----trk13-----+  Procurve |              trk14       |
>   | RHEL 3.0 |eth1  port-F3|   5308 +--------------------------+ RHEL3.0
>  +----------+                  +-----------+ port-F4                  +---
> -----+
> 
> Objective:
>       Aggregate the bandwidth to achieve a 2 GB link, and active fail
> over capabilities
> 
> Requirements:
>       Configure the bonding from both sides of the switch, and the
> host to achieve the above objectives without causing network
> degradation or instability.
> 
> Configuration :
> 
> As above in the diagram
> The Operating system is RHEL3 Update 2, the kernel currently is
> 2.4.21-37.EL from Update 5 on both machines host1, and host2
> 
> The trunking option used in the switch was fec to trunk E3,F3 that are
> connected to host1, and E4, and F4 to host2
> 
> /etc/modules.conf have the following :
> 
> alias eth0 tg3
> alias eth1 tg3
> alias scsi_hostadapter cciss
> alias usb-controller usb-uhci
> alias usb-controller1 ehci-hcd
> alias bond0 bonding
> 
>  [root at xrdbm3 root]# cat /proc/net/bonding/bond0
> 
> Ethernet Channel Bonding Driver: v2.6.0 (January 14, 2004)
> Bonding Mode: load balancing (round-robin)
> MII Status: up
> MII Polling Interval (ms): 0
> Up Delay (ms): 0
> Down Delay (ms): 0
> 
> 
> Slave Interface: eth0
> MII Status: up
> Link Failure Count: 0
> Permanent HW addr: 00:14:38:c3:ee:1f
> 
> Slave Interface: eth1
> MII Status: up
> Link Failure Count: 0
> Permanent HW addr: 00:14:38:c3:ee:1e
> Ifconfig -a gives the following :
> bond0     Link encap:Ethernet  HWaddr 00:14:38:C3:EE:1F
> 
>           inet addr:10.8.194.203  Bcast:10.8.194.255  Mask:255.255.255.0
>           UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
>           RX packets:782876 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:78540 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:64669079 (61.6 Mb)  TX bytes:12222551 (11.6 Mb)
> 
> eth0      Link encap:Ethernet  HWaddr 00:14:38:C3:EE:1F
>           inet addr:10.8.194.203  Bcast:10.8.194.255  Mask:255.255.255.0
>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
>           RX packets:120113 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:39272 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:22234287 (21.2 Mb)  TX bytes:6111991 (5.8 Mb)
>           Interrupt:25
> 
>  eth1      Link encap:Ethernet  HWaddr 00:14:38:C3:EE:1F
>           inet addr:10.8.194.203  Bcast:10.8.194.255  Mask:255.255.255.0
>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
>           RX packets:662763 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:39268 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:42434792 (40.4 Mb)  TX bytes:6110560 (5.8 Mb)
>           Interrupt:26
> lo        Link encap:Local Loopback
>           inet addr:127.0.0.1  Mask:255.0.0.0
>           UP LOOPBACK RUNNING  MTU:16436  Metric:1
>           RX packets:2666075 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:2666075 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
> 
>           RX bytes:191362627 (182.4 Mb)  TX bytes:191362627 (182.4 Mb)
> 
> And ethtool gives the following
> 
> [root at xrdbm3 root]# ethtool eth0
> Settings for eth0:
>         Supported ports: [ MII ]
>         Supported link modes:   10baseT/Half 10baseT/Full
>                                 100baseT/Half 100baseT/Full
>                                 1000baseT/Half 1000baseT/Full
>         Supports auto-negotiation: Yes
>         Advertised link modes:  10baseT/Half 10baseT/Full
>                                 100baseT/Half 100baseT/Full
>                                 1000baseT/Half 1000baseT/Full
>         Advertised auto-negotiation: Yes
>         Speed: 1000Mb/s
>         Duplex: Full
>         Port: Twisted Pair
>         PHYAD: 1
>         Transceiver: internal
>         Auto-negotiation: on
>         Supports Wake-on: g
>         Wake-on: d
> 
>         Current message level: 0x000000ff (255)
>         Link detected: yes
> [root at xrdbm3 root]# ethtool eth1
> Settings for eth1:
>         Supported ports: [ MII ]
>         Supported link modes:   10baseT/Half 10baseT/Full
>                                 100baseT/Half 100baseT/Full
>                                 1000baseT/Half 1000baseT/Full
>         Supports auto-negotiation: Yes
>         Advertised link modes:  10baseT/Half 10baseT/Full
>                                 100baseT/Half 100baseT/Full
>                                 1000baseT/Half 1000baseT/Full
>         Advertised auto-negotiation: Yes
>         Speed: 1000Mb/s
>         Duplex: Full
>         Port: Twisted Pair
>         PHYAD: 1
>         Transceiver: internal
>         Auto-negotiation: on
>         Supports Wake-on: g
>         Wake-on: d
>         Current message level: 0x000000ff (255)
>         Link detected: yes
> --------------------------------------------------------------------------
> -------------------------------------
> 
> [root at xrdbm3 root]# cat /etc/sysconfig/network
> 
> NETWORKING=yes
> HOSTNAME=xrdbm3
> GATEWAYDEV=bond0
> GATEWAY=10.8.194.250
> 
> cat /etc/sysconfig/network-scripts/ifcfg-eth0
> DEVICE=eth0
> BOOTPROTO=none
> ONBOOT=yes
> USERCTL=no
> MASTER=bond0
> SLAVE=yes
> 
> [root at xrdbm3 root]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
> 
> DEVICE=eth1
> BOOTPROTO=none
> ONBOOT=yes
> USERCTL=no
> MASTER=bond0
> SLAVE=yes
> 
> [root at xrdbm3 root]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
> 
> DEVICE=bond0
> BOOTPROTO=none
> ONBOOT=yes
> USERCTL=no
> IPADDR=10.8.194.203
> NETMASK=255.255.255.0
> NETWORK=10.8.194.0
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



More information about the Beowulf mailing list