TCP Tuning Guide - Linux TCP Tuning http://www-didc.lbl.gov/TCP-tuning/linux.

html

TCP Tuning Guide

Version 1.0 - Last published Sep 12, 2007

System Tuning Linux TCP Tuning

TCP Background
Linux
There are a lot of differences between Linux version 2.4 and 2.6, so first
FreeBSD we'll cover the tuning issues that are the same in both 2.4 and 2.6. To
Solaris change TCP settings in, you add the entries below to the file
Windows XP
Windows Vista /etc/sysctl.conf, and then run "sysctl -p".
Mac OSX
Other Like all operating systems, the default maximum Linux TCP buffer sizes
More Info are way too small. I suggest changing them to the following settings:

Application Tuning # increase TCP max buffer size

TroubleShooting net.core.rmem_max = 16777216
Duplex Issues net.core.wmem_max = 16777216
Tools # increase Linux autotuning TCP buffer limits
# min, default, and max number of bytes to use
Tutorial net.ipv4.tcp_rmem = 4096 87380 16777216
More information net.ipv4.tcp_wmem = 4096 65536 16777216

Publications Note: you should leave tcp_mem alone. The defaults are fine.
O'Reilly Article
Usenix Article Another thing you can try that may help increase TCP throughput is to
Search increase the size of the interface queue. To do this, do the following:

ifconfig eth0 txqueuelen 1000

I've seen increases in bandwidth of up to 8x by doing this on some long,

fast paths. This is only a good idea for Gigabit Ethernet connected hosts,
and may have other side effects such as uneven sharing between multiple
streams.

Linux 2.4

Starting with Linux 2.4, Linux has implemented a sender-side

autotuning mechanism, so that setting the opitimal buffer size on the
sender is not needed. This assumes you have set large buffers on the
recieve side, as the sending buffer will not grow beyond the size of the
recieve buffer.

However, Linux 2.4 has some other strange behavior that one needs to
be aware of. For example: The value for ssthresh for a given path is
cached in the routing table. This means that if a connection has has a
retransmition and reduces its window, then all connections to that host
for the next 10 minutes will use a reduced window size, and not even try
to increase its window. The only way to disable this behavior is to do the
following before all new connections (you must be root):

1 of 3 09/26/2007 09:26 AM
TCP Tuning Guide - Linux TCP Tuning http://www-didc.lbl.gov/TCP-tuning/linux.html

sysctl -w net.ipv4.route.flush=1

More information on various tuning parameters for Linux 2.4 are

available in the Ipsysctl tutorial .

Linux 2.6

Starting in Linux 2.6.7 (and back-ported to 2.4.27), BIC TCP is part of

the kernel, and enabled by default. BIC TCP helps recover quickly from
packet loss on high-speed WANs, and appears to work quite well. A BIC
implementation bug was discovered, but this was fixed in Linux 2.6.11,
so you should upgrade to this version or higher.

Linux 2.6 also includes and both send and receiver-side automatic buffer
tuning (up to the maximum sizes specified above). There is also a setting
to fix the ssthresh caching weirdness described above.

There are a couple additional sysctl settings for 2.6:

# don't cache ssthresh from previous connection

net.ipv4.tcp_no_metrics_save = 1
# recommended to increase this for 1000 BT or higher
net.core.netdev_max_backlog = 2500
# for 10 GigE, use this
# net.core.netdev_max_backlog = 30000

Starting with version 2.6.13, Linux supports pluggable congestion

control algorithms . The congestion control algorithm used is set using
the sysctl variable net.ipv4.tcp_congestion_control, which is set to Reno
by default. (Apparently they decided that BIC was not quite ready for
prime time.) The current set of congestion control options are:

reno: Traditional TCP used by almost all other OSes. (default)

bic: BIC-TCP
highspeed: HighSpeed TCP: Sally Floyd's suggested algorithm
htcp: Hamilton TCP
hybla: For satellite links
scalable: Scalable TCP
vegas: TCP Vegas
westwood: optimized for lossy networks

For very long fast paths, I suggest trying HTCP or BIC-TCP if Reno is
not is not performing as desired. To set this, do the following:

sysctl -w net.ipv4.tcp_congestion_control=htcp

More information on each of these algorithms and some results can be

found here .

Note: Linux 2.6.11 and under has a serious problem with certain Gigabit
and 10 Gig ethernet drivers and NICs that support "tcp segmentation
offload", such as the Intel e1000 and ixgb drivers, the Broadcom tg3,

2 of 3 09/26/2007 09:26 AM
TCP Tuning Guide - Linux TCP Tuning http://www-didc.lbl.gov/TCP-tuning/linux.html

and the s2io 10 GigE drivers. This problem was fixed in version 2.6.12.
A workaround for this problem is to use ethtool to disable segmentation
offload:

ethtool -K eth0 tso off

This will reduce your overall performance, but will make TCP over
LFNs far more stable.

More information on tuning parameters and defaults for Linux 2.6 are
available in the file ip-sysctl.txt, which is part of the 2.6 source
distribution.

And finally a warning for both 2.4 and 2.6: for very large BDP paths
where the TCP window is > 20 MB, you are likely to hit the Linux
SACK implementation problem. If Linux has too many packets in flight
when it gets a SACK event, it takes too long to located the SACKed
packet, and you get a TCP timeout and CWND goes back to 1 packet.
Restricting the TCP buffer size to about 12 MB seems to avoid this
problem, but clearly limits your total throughput. Another solution is to
disable SACK.

Linux 2.2

If you are still running Linux 2.2, upgrade! If this is not possible, add the
following to /etc/rc.d/rc.local

echo 8388608 > /proc/sys/net/core/wmem_max

echo 8388608 > /proc/sys/net/core/rmem_max
echo 65536 > /proc/sys/net/core/rmem_default
echo 65536 > /proc/sys/net/core/wmem_default

© 2003-2004, Lawrence Berkeley

National Laboratory

3 of 3 09/26/2007 09:26 AM