Per-Packet Load Balancing with Linux

The ISP I use at home and at work, Andrews and Arnold, support bonding of multiple lines with per-packet load balancing. Incoming traffic is handled by them using custom hardware and software - the control panel lets me select which lines should be used for each block of IP addresses and they then handle balancing the traffic over those lines on a per-packet basis.

At work we have four ADSL lines and to handle the load outgoing balancing we use a set of four ordinary DSL routers connected by ethernet to a four port D-Link ethernet card in a linux server which then does per-packet load balancing for outgoing traffic using the teql traffic scheduler. This post describes how we configure the load balancing.

The first step is to give each of the routers (we use Zyxel P660 routers) two different LAN addresses - one is in the 172.16 private address range and is unique to each router and used to address specific routers for management and configuration purposes; and the other is a common address from the block allocated to use by our ISP and is shared by all the routers.

The ability to configure multiple addresses for the LAN side interface is a requirement for this technique to work, and not all routers may support it…

Once that is done, we start the linux configuration (this is a RedHat/Fedora style system) by defining the bonded interface, teql0, by creating a teql.conf file in /etc/modprobe.d which defines an appropriate alias:

alias teql0 sch-teql

We then have to define the configuration for each interface in the normal way, with ifcfg files in /etc/sysconfig/network-scripts, starting with an ifcfg-ethN file for each of the bonded ethernet interfaces that looks something like:

DEVICE=eth{2,3,4,5}
ONBOOT=yes
BOOTPROTO=static
IPADDR=172.16.8.{1,5,9,13}
NETMASK=255.255.255.252
HWADDR=...

This places each ethernet interface on a local /30 network that is simply used for management purposes to allow each router to be connected to for configuration. An ifcfg-teql0 file is then created to define the bonded interface:

DEVICE=teql0
BOOTPROTO=static
IPADDR=...
NETMASK=...
ONBOOT=yes

The network details here are those for the IP range allocated by the ISP to us as this is the public interface where traffic sent to and from the ISP will be handled.

Next we have to make sure that reverse path filtering for the physical ethernet interfaces we are using is set to loose mode as they will be receiving packets that look like they should arrive on the traffic equalizer interface. To do this we edit /etc/sysctl.conf and add a line for each interface that looks like this:

net.ipv4.conf.eth{2,3,4,5}.rp_filter = 2

In order to bond the individual ethernet interfaces into the traffic equalizer we first create an ifup-pre-local script in /sbin which makes sure the teql0 interface is created before the ethernet interfaces are configured:

#!/bin/sh

case "$1" in
 ifcfg-eth2|ifcfg-eth3|ifcfg-eth4|ifcfg-eth5) modprobe teql0;;
esac

We then create an ifup-local script which adds the ethernet interfaces to the traffic equalizer:

#!/bin/sh

case "$1" in
 eth2|eth3|eth4|eth5) tc qdisc add dev $1 root teql0;;
esac

For good measure, an ifdown-local script removes the interfaces again:

#!/bin/sh

case "$1" in
 eth2|eth3|eth4|eth5) tc qdisc del dev $1 root;;
esac

A default route that uses the new bonded interface can be added in the normal by adding a line to /etc/sysconfig/network:

GATEWAY=...

The gateway address here is the common IP address allocated to each router.

For firewalling and packet tracing purposes, you should note that outgoing packets will be seen going into teql0 rather than the individual interfaces, but incoming packets will appear from each of the four ethernet interfaces, depending on which line they arrive from the ISP over.

That’s really about all there is to it - we do also have a small daemon process that monitors the lines and removes them from the traffic equalizer if they go down and adds them back when they come up again.

UPDATE: The traffic equalizer is completely broken in kernels 2.6.31 through 2.6.34…