Setup Gateway Routing On Multiple Network Interfaces

Introduction
Generally speaking, it's rare that you will need to have multiple network interfaces on a system and need to reach all the IP addresses on the system via external subnets. Most of the time, any additional IP addresses on a system will either reside on the same subnet or you will have some sort of heartbeat network in a cluster.

In the first case, all IP addresses can utilize the default gateway to get out of the system. You could even place these common-subnet IP addresses on interface aliases of a single NIC.

In the second case, you would have a NIC designated for transmission of data that would reach other subnets and, therefore, go through the default gateway. The other NICs would be for heartbeat traffic and communicate with other systems within the same subnet. Thereby, not needing to route through the default gateway.

However, there is the rare occassion that you would need to be able to reach each of the IP addresses in separate subnets on a single host from external subnets. Actually, it's not the reaching them part that is the issue; it's their response. As you can see in this diagram, With HOST-A having a default route of 10.200.5.203, you can receive a reply from the IP address within that subnet. However, with the other IP addresses on the host, you will not. The reason for this is because those interfaces do not know how to route back to you. The IP addresses on those interfaces are only aware of their own subnet, and cannot even find the subnet containing the default route out of the system. They are also unaware of the router specified for their own subnet.



So to counter act this issue, with Linux, you can setup additional routing tables, so when we setup a routing table for each interface we have, we can allow the IP address on that interface to have its own default route. This will allow the IP address attempting a reply to route the reply through a router that can get the response back to the host initiating the connection.

Creating Routes & Rules
To accomplish this, we will need to setup the routing tables on each additional interface excluding the interface managing the default gateway for the system.

First we will create an entry in the rt_tables file. This will allow us to use that table name with our further policy routes.
 * 1) echo "1 eth1" >>/etc/iproute2/rt_tables

Next we add two routes. One for the subnet so that the IP address can find its gateway. Another for specifying the default gateway for that interface.
 * 1) ip route add 10.200.6.0/24 dev eth1 src 10.200.6.55 table eth1
 * 2) ip route add default via 10.200.6.203 dev eth1 table eth1

Finally, we add policy route rules for ONLY that IP address that we want to use that table. This will not only ensure that the IP address we're trying to communicate with on that one interface can respond properly, but it will also ensure that we don't route information between subnets. The goal of this task is to communicate with individual IP addresses on other interfaces, not turn the system into another router allowing communication between subnets.
 * 1) ip rule add from 10.200.6.55/32 table eth1
 * 2) ip rule add to 10.200.6.55/32 table eth1

Automating on Startup
These 5 commands will need to be run for each interface we have on the system. To accomplish this task efficiently, I've added a "ROUTER=" line to the corresponding /etc/sysconfig/network-scripts/ifcfg-ethX file and placed a script in /etc/rc.local on the system. By deriving the table name and number after the interface, I've eliminated the need to keep this information somewhere additional.

/etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1 BOOTPROTO=static ONBOOT=yes HWADDR=00:50:56:98:01:55 NETWORK=10.200.6.0 BROADCAST=10.200.6.255 NETMASK=255.255.255.0 ROUTER=10.200.6.203 IPADDR=10.200.6.55
 * 1) Advanced Micro Devices [AMD] 79c970 [PCnet32 LANCE]

/etc/rc.d/rc.local

 * 1) !/bin/sh
 * 2) This script will be executed *after* all the other init scripts.
 * 3) You can put your own initialization stuff in here if you don't
 * 4) want to do the full Sys V style init stuff.
 * 1) want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

for interface_file in $(ls /etc/sysconfig/network-scripts/ifcfg-eth* | grep -v ifcfg-eth0) ;do . ${interface_file} prefix=$(ipcalc -p ${IPADDR} ${NETMASK} | awk -F= '{print $2}') tablenum=$(echo ${DEVICE} | sed 's/eth//g') if [ ${ONBOOT} != 'yes' ] ;then continue fi if ! grep "^${tablenum} ${DEVICE}$" /etc/iproute2/rt_tables >/dev/null ;then echo "${tablenum} ${DEVICE}" >>/etc/iproute2/rt_tables fi ip route add ${NETWORK}/${prefix} dev ${DEVICE} src ${IPADDR} table ${DEVICE} ip route add default via ${ROUTER} dev ${DEVICE} table ${DEVICE} ip rule add from ${IPADDR}/32 table ${DEVICE} ip rule add to ${IPADDR}/32 table ${DEVICE} done
 * 1) mbm - 04/16/2012 - add additional routes

Displaying Routes & Rules
These additional routes and rules will not show up in the normal routing table via "netstat -rn". You will need to use the "ip" command to see them. For reference, I've included the additions to the rt_tables file, the commands to get the routing tables and their expected output below:

Route Tables List
255    local 254    main 253    default 0      unspec 1 eth1 2 eth2
 * 1) cat /etc/iproute2/rt_tables
 * 2) reserved values
 * 1) reserved values
 * 1) local
 * 2) 1     inr.ruhep
 * 1) 1     inr.ruhep
 * 1) 1     inr.ruhep

Routing Tables
10.200.6.0/24 dev eth1 table eth1  scope link  src 10.200.6.55 default via 10.200.6.203 dev eth1 table eth1 10.200.64.0/24 dev eth2 table eth2  scope link  src 10.200.64.55 default via 10.200.64.203 dev eth2 table eth2 10.200.6.0/24 dev eth1 proto kernel  scope link  src 10.200.6.55 10.200.5.0/24 dev eth0 proto kernel  scope link  src 10.200.5.55 10.200.64.0/24 dev eth2 proto kernel  scope link  src 10.200.64.55 default via 10.200.5.203 dev eth0 broadcast 10.200.6.255 dev eth1 table 255  proto kernel  scope link  src 10.200.6.55 broadcast 10.200.5.0 dev eth0 table 255  proto kernel  scope link  src 10.200.5.55 broadcast 10.200.64.255 dev eth2 table 255  proto kernel  scope link  src 10.200.64.55 broadcast 127.255.255.255 dev lo table 255  proto kernel  scope link  src 127.0.0.1 local 10.200.5.55 dev eth0 table 255  proto kernel  scope host  src 10.200.5.55 broadcast 10.200.6.0 dev eth1 table 255  proto kernel  scope link  src 10.200.6.55 broadcast 10.200.5.255 dev eth0 table 255  proto kernel  scope link  src 10.200.5.55 broadcast 10.200.64.0 dev eth2 table 255  proto kernel  scope link  src 10.200.64.55 local 10.200.6.55 dev eth1 table 255  proto kernel  scope host  src 10.200.6.55 broadcast 127.0.0.0 dev lo table 255  proto kernel  scope link  src 127.0.0.1 local 10.200.64.55 dev eth2 table 255  proto kernel  scope host  src 10.200.64.55 local 127.0.0.1 dev lo table 255  proto kernel  scope host  src 127.0.0.1 local 127.0.0.0/8 dev lo table 255  proto kernel  scope host  src 127.0.0.1
 * 1) ip route show table all

IPRoute Rules
0:     from all lookup 255 32762: from all to 10.200.64.55 lookup eth2 32763: from 10.200.64.55 lookup eth2 32764: from all to 10.200.6.55 lookup eth1 32765: from 10.200.6.55 lookup eth1 32766: from all lookup main 32767: from all lookup default
 * 1) ip rule show

Conclusion
After these routes and rules have been added, you should be able to fully ping each IP address on the system. $ fping 10.200.5.55 10.200.6.55 10.200.64.55 10.200.5.55 is alive 10.200.6.55 is alive 10.200.64.55 is alive



Information gleamed from: Darien Kindlund