Technote: AIX EtherChannel Load Balancing Options

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/T...

AIX EtherChannel Load Balancing Options

Cindy K Young Jorge R Nogueras Advanced Technical Support AIX; Ethernet; pSeries; RS/6000; TCP/IP
Document ID:

Document Author: Additional Author(s): Doc. Organization: Product(s) covered:

TD101260

Document Revised: 10/28/2003

Abstract: This article is an overall view of the user-configurable AIX V5 EtherChannel load balancing behavior options, including the new Hash Mode parameter. AIX EtherChannel Load Balancing Options This article is an overall view of the user-configurable AIX EtherChannel load balancing behavior options, including the new Hash Mode parameter. Introduction. A new SMIT menu option called "Hash Mode" appears when AIX users install software APARs available in November, 2003. The menu gives standard mode EtherChannel users the option of distributing outgoing traffic based upon TCP or UDP port values. Software support is provided in APAR IY48331 for AIX V 5.1 and APAR IY48170 for AIX V 5.2 Load balancing behavior. Two SMIT-selectable load balancing methods for outgoing traffic are offered -round robin, which spreads the outgoing traffic evenly across all the adapters in the EtherChannel; and standard, which selects the adapter by algorithm. The Hash Mode parameter, shown in the following SMIT configuration menu, determines the numerical value used in the algorithm. Change / Show Characteristics of an EtherChannel / Link Aggregation Type or select values in entry fields. Press Enter AFTER making all desired changes. EtherChannel / Link Aggregation EtherChannel / Link Aggregation Adapters Backup Adapter Delete Backup Adapter Add Main Adapter Delete Main Adapter Add Backup Adapter Enable Alternate Address Alternate Address Enable Gigabit Ethernet Jumbo Frames Mode Hash Mode Internet Address to Ping Number of Retries Retry Timeout (sec) Apply change to DATABASE only ent6 ent1,ent2,ent4,ent5 NONE

no [0x000000000000] no standard default [] [3] [1] no

The following chart summarizes the valid load balancing option combinations offered in the SMIT screen.

1 of 5

12/10/04 10:27 AM

Technote: AIX EtherChannel Load Balancing Options

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/T...

MODE standard or 8023ad

HASH MODE default

OUTGOING TRAFFIC DISTRIBUTION (across adapter ports within the EtherChannel) The traditional AIX behavior. The adapter selection algorithm uses the last byte of the destination IP address (for TCP/IP traffic) or MAC address (for ARP and other non-IP traffic). This mode is typically the best initial choice for a server with a large number of clients. The outgoing adapter path is selected via algorithm using the combined source and destination TCP or UDP port values. Average the TCP/IP address suffix values in the "Local" and "Foreign" columns shown by netstat -an command. Since each connection has a unique TCP or UDP port, the three port-based hash modes provide additional adapter distribution flexibility when there are several, separate TCP or UDP connections between an IP address pair. The adapter selection algorithm uses the source TCP or UDP port value. In netstat -an command output, the port is the TCP/IP address suffix value in the "Local" column. The outgoing adapter path is selected via algorithm using the destination system port value. In netstat -an command output, the TCP/IP address suffix in the "Foreign" column is the TCP or UDP destination port value. Outgoing traffic is spread evenly across all the adapter ports in the EtherChannel. This mode is the typical choice for two hosts connected back-to-back (i.e. without an intervening switch).

standard or 8023ad

src_dst_port

standard or 8023ad standard or 8023ad round robin

src_port

dst_port

default

Round Robin. All outgoing traffic is spread evenly across all of the adapters in the EtherChannel. It provides the highest bandwidth optimization for the AIX server system. While round robin distribution is the ideal way to utilize all the links equally, AIX administrators should consider that it also introduces the potential for out-of-order packets at the receiving system. Standard algorithm. The standard algorithm is used for both standard and IEEE 802.3ad-style link aggregations. AIX divides the last byte of the "numerical value" by the number of adapters in the EtherChannel and uses the remainder to identify the outgoing link. If the remainder is zero, the first adapter in the EtherChannel is selected; one means second adapter, and so on. (The adapters are selected in the order they are listed in the SMIT menu.) The "Hash Mode" selection determines the numerical value used in the calculation. By default, the last byte of the destination IP or MAC address is used in the calculation, but the source and/or destination TCP or UDP port values may also be utilized. These alternatives allow the AIX administrator to fine-tune the distribution of outgoing traffic across the real adapters in the EtherChannel. In default standard mode, the adapter selection algorithm is applied to the last byte of the destination IP address for IP traffic. For ARP and other non-IP traffic, the same formula is applied on the last byte of the destination MAC address. Unless there is an adapter failure which causes a failover, all traffic between a host pair in default standard mode goes out over the same adapter. For IP traffic, the port selected could be changed by selecting a different hash mode value or by implementing a different IP address (perhaps via IP aliases) at the target system. In src_dst_port hash mode, the source and destination port values are added, then divided by two. The resultant whole number (no decimals) is plugged into the standard algorithm. TCP or UDP traffic is sent on the adapter selected by the standard algorithm and selected hash mode value. Non-TCP or UDP traffic will fall back to the default algorithm, meaning the last byte of either the destination IP or MAC address. Incoming traffic distribution. Remember that the AIX adapter selection applies to outgoing packets only. The packets incoming to the AIX system are governed by the algorithm configured at the switch. The traffic distribution schemes used by the two components need not match -- the algorithms can be (and often are) different. For example, the switch might direct packets based on the XOR of source and destination MAC addresses. In this case, the destination MAC address is the same for packets going to the AIX system, but the different source address causes the packets to be sent over different adapters. Understanding packet distribution. Curious about how packets are being distributed over the ports on your system? Use the entstat or netstat command to view aggregate statistics of all the adapters in the EtherChannel. For example, netstat -v ent6 reports the key information about EtherChannel device ent6. Statistics for outgoing packets are a result of AIX's configuration. If round robin is configured, for example,

2 of 5

12/10/04 10:27 AM

Technote: AIX EtherChannel Load Balancing Options

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/T...

statistics will be nearly identical for each underlying adapter. The incoming packet statistics reflect the network switch configuration. Use the netstat -an command to view the TCP or UDP source and destination port address values. Predicting adapter utilization -- Example Environment. The effects of different hash mode values on the standard adapter selection algorithm are best illustrated in an example. In this fictitious environment, the EtherChannel (ent6) is configured as IP address 10.9.8.1 (the en6 interface) and is comprised of individual adapters ent1, ent2, ent4, ent5. To demonstrate the adapter selection, the example system communicates with six systems with sequential IP addresses ranging from 192.26.91.100 through 192.26.91.105. The outgoing adapter port is selected via the mathematical remainder of the calculation hash mode numerical value / the number of real adapters in the EtherChannel. There are four adapters in this EtherChannel, so the numerical value is always divided by four. The numerical value changes from the last byte of the destination IP address to the source port, then to the destination port, and then to the average of the two port values in the four Hash Modes. It is this change in numerical value that causes the variation in the adapter selection. The result of the calculation is the mathematical remainder. If the remainder is zero, the first adapter in the SMIT list is selected. In the example, this is adapter ent1. If the remainder is three, the fourth adapter (ent5 in our sample) is selected, and so forth. The complete matchup of the remainders, the selection order, and the actual adapters in the example are shown in the following figure. Remainder 0 1 2 3 Adapter in list 1st 2nd 3rd 4th Actual Adapter ent1 ent2 ent4 ent5

Figure 1: Adapter selection in the sample environment. The following partial netstat -an command output shows IP addresses and TCP/UDP ports for nine active connections the local system might have with the remote systems 192.26.91.100 through 192.26.91.105. In this case, the "Local Address" column reflects the EtherChannel's 10.0.8.1 IP address and the "Foreign Address" column reflects the IP address of the remote system. The numerical values appended to these local and foreign IP addresses are the source and destination ports, respectively, used in three of the four hash modes. # netstat -an Active Internet connections (including Proto Recv-Q Send-Q Local Address udp4 0 0 10.9.8.1.36156 tcp4 0 0 10.9.8.1.23 tcp4 0 0 10.9.8.1.23 tcp4 0 0 10.9.8.1.23 tcp4 0 0 10.9.8.1.23 tcp4 0 0 10.9.8.1.23 tcp4 0 0 10.9.8.1.23 tcp4 0 3 10.9.8.1.23 tcp4 0 0 10.9.8.1.23 servers) Foreign Address 192.26.91.105.6000 192.26.91.102.44070 192.26.91.103.44075 192.26.91.105.44078 192.26.91.101.44078 192.26.91.104.44089 192.26.91.100.32776 192.26.91.100.32777 192.26.91.104.44411

The following chart shows the mathematical result (the remainder) of applying the four hash modes to the standard algorithm. Check the remainder value in the Figure 1 chart to identify the real outgoing adapter selected. Default src_port dst_port src_dst_port Line Address Address Address Address # remainder remainder remainder remainder value value value value* 1 2 105 102 1 2 36156 23 0 3 6000 44070 0 2 42156 / 2 44093 / 2 2 2

3 of 5

12/10/04 10:27 AM

Technote: AIX EtherChannel Load Balancing Options

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/T...

3 4 5 6 7 8 9

103 105 101 104 100 100 104

3 1 1 0 0 0 0

23 23 23 23 23 23 23

3 3 3 3 3 3 3

44075 44078 44078 44089 32776 32777 44411

3 2 2 1 0 1 3

44098 / 2 44101 / 2 44101 / 2 44112 / 2 32799 / 2 32800 / 2 44434 / 2

1 2 2 0 3 0 1

* here the address values are first averaged (added, then divided by two). The resultant whole number value (excluding decimals) is divided by the four adapters in the example environment. Figure 2. Comparison of results: Standard adapter selection algorithm with various hash mode options. Observations. In the default hash mode, the numerical value used in the calculation is the last byte in the destination system IP address. Because this number is the same for all connections to the same destination IP address, all traffic between a host pair in standard mode goes out over the same adapter. Using this hash mode, traffic will be fairly well spread over the adapters if the random nature of the IP addresses result in an even selection and if all of the remote systems do generally the same things (meaning, for instance, one IP address does not have a huge download). In this example, the traffic is unbalanced when using src_port hash mode. Traffic shown on line 1 is sent out adapter ent1 and traffic shown on lines 2-9 will be sent over adapter ent5. Adapters ent2 and ent4 are not used. Why? The source port address value is 23 (the well-known port for telnet) on all of these connections, hinting at the way the connection was established. This indicates that for the connections on lines 2-9 the remote system telnet'ed to the local system. In dst_port hash mode, the outgoing traffic is fairly well balanced across the adapters in the EtherChannel. If, however, the local system established the connections, you might see the same results skew seen in the src_port hash. In src_dst_port hash mode, the averaging of the fields seems to balance the outgoing traffic fairly well. The skew in values disappears. So, regardless of the direction in which the connection is initiated, the resulting values distribute across the adapter ports. Considerations. The single most important consideration when choosing a load balancing method is nature of the traffic that is being sent across the EtherChannel. Yet, the final choice will probably be a trade-off that provides the best distribution for a subset of this traffic. Depending upon the environment, one administrator may choose to accommodate the majority of the traffic while another opts to design for the most important form of traffic. There is no single choice that will yield a perfect distribution all the time because the type of traffic being handled by the typical EtherChannels varies. It may, for example, handle mixtures of streaming TCP traffic one moment, some ARP traffic the next, then a huge file transfer followed by short-lived UDP exchanges. Administrators can use these key questions begin the process of selecting the mode of operation and hash mode best suited for an environment. Look for patterns and predictability. Network attachment. Is the local system connected to an Ethernet switch? Is it connected directly (back-to-back) with another host? Direction of connection establishment. Do remote systems typically connect to the local host? Is the local system a server for client systems? Does the local host establish connections to other hosts? IP addresses. Look for IP address patterns. Does the local host have a few, but long-lived TCP connections with the same remote hosts? Conversely, does it have many, short-lived connections to a variety of remote hosts? Traffic type. The main traffic load may be TCP/IP, but ARP and other non-IP traffic is distributed differently in standard mode. Is there any other significant type of traffic? Is there a key application that generates a predictable load? Conclusions. The results of any mode can be skewed if the numeric values are skewed. In fact, the load balancing options have been added since the initial AIX V4.3.3 EtherChannel implementation for just this reason. There is no one-size-fits-all solution to unique environments. The trick is to choose a mode that results in varied numeric values so that the load is distributed across all of the adapters in the EtherChannel. Generally speaking, round robin is ideal for back-to-back connections running jumbo frames. In this environment, there is no intervening switch, so there is no chance that processing at the switch could alter

4 of 5

12/10/04 10:27 AM

Technote: AIX EtherChannel Load Balancing Options

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/T...

the packet delivery time, order, or adapter path. On this direct cable network path, the receiver sees packets exactly as sent. Jumbo frames (9000 byte MTU) always yield better file transfer performance than traditional 1500 byte MTUs. In this case, however, they add another benefit. These larger packets take longer to send so it is less likely that the receiving host would be continuously interrupted with out-of-order packets. Round robin mode can be implemented in other environments but at increased risk of out-of-order packets at the receiving system. This risk is particularly high when there are few, long-lived, streaming TCP connections. When there are many such connections between a host pair, packets from different connections could be intermingled, thereby decreasing the chance of packets for the same connection arriving out-of-order. Check for out-of-order packet statistics in the "tcp" section of the netstat -s command output. A steadily-increasing value indicates a potential problem in traffic sent from an EtherChannel. If out-of-order packets are a problem on a system that must use traditional Ethernet MTUs and must be switch attached, try the various hash modes offered in standard mode operation. If no particular hash mode value is obvious, administrators may try each of them, testing and analyzing the results before moving to the next. Each mode has a particular strength but the "default" and "src_dst_port" modes are the logical starting points as they are more widely applicable. The default hash mode works well when the local host establishes connections to many different IP addresses. It is a good first choice for most busy systems because the random nature of IP address traffic tends to distribute the traffic in an acceptable manner. If the local host establishes lengthy connections to few IP addresses, look at one of the TCP or UDP hashing options. The "src_dst_port" hash mode option is probably the best initial choice because considers both the source and the destination TCP or UDP port values. In this mode, all of the packets in one TCP or UDP connection are sent over a single adapter so they are guaranteed to arrive in order, but the traffic is still spread because connections (even to the same host) may be sent over different adapters. The results of this hash mode aren't skewed by the connection establishment direction because it uses both the source and destination TCP or UDP port values. Use the "src_port" or "dst_port" hash mode options if port values change from one connection to another and the "src_dst_port" option is not yielding a desirable distribution. For instance, if the local host is an FTP server, the source TCP or UDP port will be more or less constant across connections. The netstat -s command would reflect 21, which is the FTP well-known port value. In this case, the "dst_port" hash mode option would be a better choice. If, on the other hand, the local host primarily establishes connections to other machines, the source TCP or UDP port values will be random, making the "src_port" hash mode a better option.

Classification: Platform(s): O/S:

Software RS/6000 or IBM eServer pSeries AIX

Category:

Planning and Design AIX EtherChannel, NIB, network interface backup, load, balancing, traffic, SMIT, configuration, methods, options, hash mode, standard, algorithm

Keywords:

5 of 5

12/10/04 10:27 AM