scenario #1

en1 tr0

.6 .6

10.5.2.0

Router
IP Multipath Routing
10.5.6.0
10.5.2.1

Router
scenario #2 10.5.6.111 and
ent1 Dead Gateway Detection
10.90.5.0

.67
ent2
.68 in AIX V 5.3

Cindy K Young
IBM Advanced Technical Support, Dallas
youngck@us.ibm.com
Dallas, TX June, 2005

©IBM 2005 - 1
IP Multipath Routing / Dead Gateway Detection
scenario #1
significant IP routing subsystem change

allows multiple routes to the same

en1 tr0
destination, including multiple defaults
.6 .6
uses route cost value to manipulate local
10.5.2.0
host's IP routing table to load balance
10.5.6.0 Router and/or react to changing network
10.5.2.1 conditions (failover)
Router
10.5.6.111
best suited for multi-homed hosts; may be
scenario #2 combined with other AIX networking
solutions
ent1
.67
10.90.5.0

ent2 bandwidth potential due to multiple

.68
adapters

AIX v5 minimum, key enhancements in v5.3

function transparent to the network (but

assumes IP routing protocols)

©IBM 2005 - 2
 IP Multipath Routing and Route Cost
IP Multipath Routing
allows more than one route to the same destination - including multiple defaults
for all IP interfaces - regardless of network type
works with or without AIX routing protocols
enabled when a new route to the same destination is added - previously, flagged
as a duplicate route but only one route used)
for load balancing - or -
for failover to backup routes (in conjunction with Dead Gateway Detection)
implemented first in AIX V5.0, enhanced in AIX V 5.3

Route Cost attribute

used for route selection - lowest value always selected; round robin selection for
duplicate routes with lowest cost (equivalent routes); higher value routes never selected
assign to individual routes - via mktcpip -C command, -hopcount flag of route
command, or in SMIT menu. Flag existed previously, but didn't impact route selection
view route costs via netstat -Cn command
value adjusted by system when a gateway is determined to be down

©IBM 2005 - 3
SMIT Configuration:
fastpath smitty mkroute
Add a Static Route

Type or select values in entry fields.

Press Enter AFTER making all desired changes.
[Entry Fields]
Destination TYPE net
* DESTINATION Address [default]
(dotted decimal or symbolic name)
* GATEWAY Address [10.5.2.1]
(dotted decimal or symbolic name)
COST [0]
Network MASK (hex or dotted decimal) [] New AIX
Network Interface [tr0] V5 SMIT
fields
(interface to associate route with)
Enable Active Dead Gateway Detection? no
Is this a Local (Interface) Route? no

Note: The Network Mask field should be left blank for host and default routes. Use only
for non-default, network routes.
©IBM 2005 - 4
IP Multipath Routing for load balancing
# netstat -Cn
Routing tables
Destination Gateway Flags Refs Use If Cost Config_Cost
scenario #1 Route tree for Protocol Family 2 (Internet):
default 10.5.6.1 UG 0 752 en1 0 0=>
default 10.5.2.1 UG 0 0 tr0 0 0
10.5.2/24 10.5.2.6 U 6 1820 tr0 0 0
10.5.6/24 10.5.6.6 U 3 2115 en1 0 0
10.90.5/24 10.5.2.1 UG 0 0 tr0 0 0=>
10.90.5/24 10.5.6.1 UG 0 0 en1 0 0
10.5.2.6 127.0.0.1 UGHS 0 58 lo0 0 0
en1 tr0
10.5.6.6 127.0.0.1 UGHS 0 137 lo0 0 0
127/8 127.0.0.1 U 0 542 lo0 0 0
.6 .6

10.5.2.0 equivalent routes permitted (same

destination, netmask, group id, cost)
Router scenario #2
10.5.6.0
10.5.2.1 applies to two or more IP interfaces
on different or same network
Router
10.5.6.1 ent1 cost determines route to use --
67
10.90.5.0

ent2 lowest cost always selected; round

68 robin between equivalent cost;
higher cost never selected
Note: Path MTU discovery was changed in AIX V 5.3 -- cloned host routes can no longer
interfere with round-robin selection!
©IBM 2005 - 5
AIX V5.3 IP Multipath Routing Enhancements
Additional route selection choices within the "multipath
routing set" in IP routing table
WRR weighted round-robin -- Default!
RND random
WRND weighted random
LUT lowest utilization (lowest reference count)
HSH hash-based (destination IP address)

Default behavior (WRR, weights of 1) matches previous AIX v5 round robin

Avoid HSH if set is comprised of host routes -or- if significant traffic to same IP
addr

Terminology change: Multipath routing set replaces equivalent routes. Set of

routes with same destination, cost, subnetmask and group values

policy is stored with the first route in set so a single change affects entire set

the mpr_policy (no option) sets system-wide behavior for all multipath routing
sets - individual route values override ©IBM 2005 - 6
Configurable IP Multipath Routing Example
# netstat -Cn two default routes
Routing tables are a multipath
Destination Gateway Flags Wt Policy If Cost Config_Cost routing set
Route Tree for Protocol Family 2 (Internet):
default 10.3.1.161 UG 3 WRR en0 0 0 => verify route usage
default 10.10.10.2 UG 2 -"- en1 0 0 via "Ref" and "Use"
10.3.1.160 10.3.1.172 UHSb 1 - en0 0 0 =>
10.3.1.160/27 10.3.1.172 U 1 - en0 0 0
columns in
10.3.1.172 127.0.0.1 UGHS 1 - lo0 0 0 netstat -rn
10.3.1.191 10.3.1.172 UHSb 1 - en0 0 0
10.10.10.0 10.10.10.4 UHSb 1 - en1 0 0 =>
10.10.10/24 10.10.10.4 U 1 - en1 0 0 fastpath smitty mkroute
10.10.10.4 127.0.0.1 UGHS 1 - lo0 Add
0 Static
0 Route
10.10.10.255 10.10.10.4 UHSb 1 - en1 0 0
127/8 127.0.0.1 U Type or
1 select
- values
lo0 in
0 entry 0fields.
[Entry Fields]
Destination TYPE net
DESTINATION Address [default]
Default GATEWAY Address [10.10.10.2]
COST [0]
first route in the set will be Network MASK (hexadecimal or dotted decimal) []
selected three times, then Network Interface []
the second route will be (interface to associate route with)
Enable Active Dead Gateway Detection? no
used twice before cycling Is this a Local (Interface) Route? no
back around Policy (for Multipath Routing Only) Weighted RR
Weight (for Weighted Multipath Routing Policy) [3]
Apply change to DATABASE only no

©IBM 2005 - 7
 Dead Gateway Detection (DGD)
Functional Overview
allows a host to detect and adjust its IP routing tables when a local (first hop)
gateway goes down - previously, the host continually tried the dead gateway
assumes network gateways (IP routers) run routing protocols and can react to
other router failures
do not run routing protocols (e.g., routed or gated) and DGD on the same host
two discovery alternatives - both active and passive manipulate "route cost" value but
differ in the amount of overhead incurred

Passive mode Dead Gateway Detection (Recommended!)

"best effort" service with little overhead
for those needing simple detection and failover - but don't want much overhead
enabled or disabled for the whole system - passive_dgd option of the no command

Active mode Dead Gateway Detection

quick detection - within seconds
for those requiring very high availability and no disruption in service visible
to users - ping traffic from several hosts could degrade network performance
enabled or disabled on a route-by-route basis in SMIT or commands - "A" flag
©IBM 2005 - 8
IP Multipath Routing w/DGD for failover to backup routes

Default passive mode behavior Default active mode behavior

enabled for the entire enabled on a route-by-route
system (disabled by basis
default) tr0 en1 pings every 5 seconds to all
ARP and TCP/IP .6 gateways in routing table
.6
recognize lack of with "A" flag
response from 10.5.2.0
gateway considered dead
a gateway if three ARP, TCP, or ping
10.5.6.0
Router responses are missed
10.5.2.1

Router
10.5.6.1

Failure/recovery
behavior (both modes)
ARP entry is deleted and route cost set to "max" (2147483647) on all routes
to dead gateway
if available, backup routes are used
routes rechecked every five minutes and, when the gateway is restored, the
route cost value is returned to user configured values
©IBM 2005 - 9
AIX V5.3 Routing Table
# netstat -in
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
en0 1500 link#2 0.d.60.a.56.82 394952 0 68600 0 0
en0 1500 9.19.126 9.19.126.84 394952 0 68600 0 0
lo0 16896 link#1 33917 0 33946 0 0
lo0 16896 127 127.0.0.1 33917 0 33946 0 0
# netstat -rn
lo0 16896 ::1 33917 0 33946 0 0
Routing tables
Destination Gateway Flags Refs Use If PMTU Exp Groups

Route tree for Protocol Family 2 (Internet):

default 9.19.126.1 UG 0 2495 en0 - -
9.19.126.0 9.19.126.84 UHSb 0 0 en0 - - =>
9.19.126/24 9.19.126.84 U 3 50467 en0 - -
9.19.126.84 127.0.0.1 UGHS 6 113 lo0 - -
9.19.126.255 9.19.126.84 UHSb 0 1 en0 - -
127/8 127.0.0.1 U 5 33806 lo0 - -
# pmtu display
dst gw If pmtu refcnt redisc_t exp
-------------------------------------------------------------------
9.19.98.112 9.19.126.1 en0 1500 0 3 2
9.19.126.84 127.0.0.1 lo0 16896 6 29 0
9.19.126.114 9.19.126.84 en0 1500 1 2 0
127.0.0.1 127.0.0.1 lo0 16896 2 29 0

New pmtu command manages PMTU table, "C" flag is gone!

PMTU and Exp columns unused and may be deleted in the future
©IBM 2005 - 10
 References

AIX V5.3 Product documentation (Information Center)

http://publib.boulder.ibm.com/infocenter/pseries/index.jsp

AIX V5.3 Sys Mgmt Guide: Communications and Networks

http://publib.boulder.ibm.com/infocenter/pseries/
topic/com.ibm.aix.doc/aixbman/commadmn/commadmn.pdf

AIX pSeries Advanced Tech Support website (IBM internal only)

http://w3.ibm.com/support/americas/pseries/network.html

©IBM 2005 - 11