L7-filter Userspace Version HOWTO

Last update 10 July 2008

If you have not already, please read the README.

Please note that this version of l7-filter is still in the early stages of development.
This documentation may be out-of-date. You should read the documentation
included in the package itself also, particularly the man page and the BUGS,
Changelog and TODO files.

This version of l7-filter has a number of compatibility problems with newer kernels.
We hope to eventually get it all sorted out, but for now the kernel version is the
development priority. If you can improve this version, please send your patches to
our mailing list!

Table of Contents

What you need to get

Installation
Getting it going
What it does
Making it useful
More information
Important links on this page:

Download page
Packet flow diagram
Pattern Writing HOWTO
Shaping script
What you need to get

Download the "l7-filter userspace version" and "Protocol definitions" packages from
our Sourceforge project page.

Installation

Compile the program:

Untar l7-filter-userspace-X.Y.tar.gz
Run ./configure
Run make
Run make install as root
./configure will probably find that you need to install some libraries
(libnetfilter_conntrack and libnetfilter_queue) from ftp.netfilter.org. Do so if
prompted.

Now set up the protocol definitions (a.k.a. pattern files). These files tell l7-filter how
protocol names correspond to regular expressions, for example "ftp" means
"^220[\x09-\x0d -~]*ftp". Uncompress the "Protocol definitions" package and make
the resulting directory /etc/l7-protocols.

Getting it going

First, a reminder: Just because you're using l7-filter, you don't need to do all of your
packet classification using it. It's likely that what you want to accomplish can be at
least partially done with less demanding classifiers, such as port matching. For
instance, you can probably assume that traffic on TCP port 80 that isn't matched by
any P2P patterns is HTTP; you don't need to actually use the HTTP pattern.

Kernel

For Linux 2.6.14 through 2.6.19.7, you simply need to have connection tracking and
the connection tracking netlink interface enabled. I think that this is the default in
most cases. Older versions of Linux don't have this interface and so will not work.

For Linux 2.6.20 and newer, Netfilter has new "Layer 3 Independent Connection
tracking" which l7-filter is not yet compatible with (mostly due to lack of library
support from libnetfilter_conntrack). While the old layer 3 dependent connection
tracking is still available, it is not selected by default, so you will probably need to
recompile your kernel with it. In the Linux kernel config, go to:

Networking ? Networking options ? Network packet filtering framework (Netfilter) ?

Core Netfilter Configuration

Under "Netfilter connection tracking support", select "Layer 3 Dependent

Connection tracking (OBSOLETE)". Then go to:

Networking ? Networking options ? Network packet filtering framework ? IP:

Netfilter Configuration

and enable "Connection tracking netlink interface" (and probably most of the rest of
the stuff on that page). Warning: the kernel guys like to rename these configuration
options frequently, so they may not be called exactly what we show here.

Either way, you need either the module ip_conntrack_netlink or nf_conntrack_netlink

or the same code compiled into your kernel.

l7-filter

Set up your l7-filter configuration file. It consists of pairs of protocol names and
Netfilter mark numbers. l7-filter will mark packets that match each given protocol
with the corresponding mark number. There is a sample in the l7-filter-userspace
source tarball. For a list of valid protocols, see the protocols page. You can also add
your own protocols. Marks are 32 bit integers, but l7-filter assigns special meaning
to 0, 1 and 2, so yours must be at least 3.

Load the ip_conntrack_netlink module with modprobe ip_conntrack_netlink, or make

sure it is compiled into your kernel.

Send traffic to l7-filter using one of:

iptables [specify table and chain] -j QUEUE

iptables [specify table and chain] -j NFQUEUE --queue-num [queue number]

NFQUEUE defaults to queue number 0 and so does l7-filter. So to send all traffic
passing through your machine to l7-filter on queue number 0:

iptables -A FORWARD -j (NF)QUEUE

Now run l7-filter:

l7-filter -f [configuration file] -q [queue number]

For example, if you're using queue number 0 and your configuration file is l7filter.conf:

l7-filter -f l7-filter.conf

In order to do its classification, l7-filter must be able to see all of the relevant traffic,
so make sure you do not use a chain like OUTPUT which only gets traffic going in
one direction. See this packet flow diagram for details. In some cases, l7-filter can
sucessfully match even if it can only see one side of the connection, but in general,
this won't work.

See man l7-filter for various ways of tweaking l7-filter's behavior.

What it does

l7-filter examines the application data in connections to determine what protocol is

being used and sets Netfilter marks accordingly. It expects that packets it receives
have no mark yet (that is, their mark is 0). It generally takes a few packets before
l7-filter can identify what protocol is being used, so it gives packets from new,
unidentified connections the special mark of 1. Eventually if l7-filter can't identify a
connection, it will give up. In this case, it marks them with the special value 2.
Otherwise, it marks packets with the marks specified in its configuration file.

l7-filter ACCEPTs all packets sent to it. This means that an iptables rule that sends
packets to l7-filter always terminates the chain and sends the packet on. (This is not
how we would like it to work, but the Netfilter libraries currently make any other
approach very difficult.) So in order to use the marks that l7-filter sets, you need to
use a chain that is downstream of it. See the next section of this document.

Making it useful

Just marking packets isn't very interesting. What you actually want is probably one
of these three things: (1) accounting (2) controlling bandwidth use (3) blocking. To
do these, you must set up rules to act on the marks that l7-filter applies to the
packets. These can either be iptables (firewall) or tc (QoS) rules.

1. Accouting

If you just want to keep track of what's in use on your network, simply match on the
mark in some chain that is downstream of where you are QUEUEing to l7-filter and
don't use any -j option. For instance, if l7-filter acts in FORWARD, you can do, for
instance:

iptables -t mangle -A POSTROUTING -m mark --mark 3

and then get statistics by using iptables -L. (See man iptables for details.)

2. Bandwidth Restriction

If l7-filter has marked, say, IMAP packets with 3, you can use tc ("traffic control", the
userspace tool for Linux QoS, part of the iproute2 package) to control the bandwidth
used by IMAP using commands like this:

tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 3 fw flowid 1:3

Did you understand that? You can try reading The Linux Advanced Routing and
Traffic Control HOWTO for enlightenment. You should do this so that you have some
idea what you're doing, but unfortunately, tc is incredibly obtuse and you're likely to
wish you just had a canned script such as this one to work from. This will need to be
modified if your setup is significantly different than mine, but it should provide a
good starting point.

Be prudent when choosing the amount of bandwidth you allow each protocol.
Restricting a protocol to an unusably low bandwidth can have similar consequences
to blocking it (see next section).

3. Blocking

Don't. Here's why:

l7-filter matching isn't foolproof: there may be both false positives (one protocol can
look like another) and false negatives (applications can do obscure things that we
didn't count on). Patterns that are known to regularly generate false positives are
marked "overmatching" on the protocols page, but others may also do so
occasionally.
Almost every type of Internet traffic has legitimate uses. For instance, P2P protocols,
while widely used to violate copyright, are also an efficient way to distribute open
source software and legally free music.
Programs can respond to being blocked by port-hopping, switching between TCP
and UDP, opening a new connection for every trivial operation, using encryption, or
employing other evasion tactics. Trying to block such protocols has consequences
on two levels:
In the case of port/protocol-hopping, you make it harder for yourself to identify
protocols that already act this way.
You encourage programmers to include these "features" in new programs, making it
harder for everyone in the future. For example: In early 2006, Bittorrent started
moving towards end-to-end encryption because many networks were either
blocking it or severely restricting its bandwidth.
l7-filter patterns are not generally designed with blocking in mind. We consider a
protocol to be well identified if the identification is useful for controlling its
bandwidth. This means, for instance, that for P2P applications, we do not focus on
catching connections that are not downloads.

Blocking with l7-filter provides no security, since any reasonably determined person
can easily circumvent it.
Instead of dropping packets you don't like, we recommend using Linux QoS to
restrict their bandwidth usage. If you insist on using l7-filter to drop packets, make
sure you have investigated other options first, such as the features of your HTTP
proxy (useful for worms).

More information

Dealing with FTP, IRC, etc.

Some protocols open child connections to transfer data. This version of l7-filter
currently cannot handle this by itself. It will only correctly match the parent
connection.

Other things that use Netfilter marks

If a packet already has a non-zero Netfilter mark when l7-filter gets it, it will pass it
on with that same mark without trying to identify it. You can modify this behaviour
using the -m and -c options. See man l7-filter.

Upgrading the protocol definitions

The protocol definitions are simple text files with a format described in the PatternHOWTO. They can be updated as a package or individually.

L7-filter only reads the definitions when it starts up, so if you update the protocol
definitions, restart it.

Other things to know

By default l7-filter looks at the first 10 packets or 12000 Bytes of each connection,
whichever is smaller. This can be modified with command line switches. See man l7filter.

It's possible (although rare) for a connection to be matchable by more than one
pattern. The patterns are tested in the order you specified in the configuration file.
After a match is made, l7-filter does not continue testing that connection, so
changing the order of your rules may change what happens.
Please see the FAQ for more information.

2) Install L7-filter

Open a terminal and do

tar -xvf linux-2.6.26.tar.bz2

tar -xvf netfilter-layer7-v2.19.tar.gz

2.1) Apply patch to Linux kernel source

cd linux-2.6.26
patch -p1 < ../netfilter-layer7-v2.19/kernel-2.6.25-layer7-2.19.patch

2.2) Apply patch & install iptables 1.4.0

tar -xvf iptables-1.4.0.tar.bz2

cd iptables-1.4.0
patch -p1 < ../netfilter-layer7-v2.19/iptables-1.4-for-kernel-2.6.20forward-layer72.19.patch
chmod +x extensions/.layer7-test
make KERNEL_DIR=~/linux-2.6.26

make install KERNEL_DIR=~/linux-2.6.26

3) Installing protocol definitions

tar -xvf l7-protocols-2008-04-23.tar.gz

cd l7-protocols-2008-04-23
mkdir /etc/l7-protocols
cp protocols/* /etc/l7-protocols

4) Compiling & installing new linux kernel

cd linux-2.6.26
make menuconfig
make all
make modules_install
make install

Important Note:- You must enable the following options (these are correct for Linux
2.6.26, but they tend to move around a lot, so you may have to go hunting if you
have a different kernel version):
* "Network packet filtering framework(Netfilter)" (Networking ? Networking option)
* "Netfilter connection tracking support" (... ? Network packet filtering
framework(Netfilter) ? Core Netfilter Configuration)
* "Connection tracking flow accounting" (on the same screen)
* Finally, "Layer 7 match support"
* Optional but highly recommended: Lots of other Netfilter options, notably "FTP
support" and other matches. If you don't know what you're doing, go ahead and
enable all of them.

5) Check GRUB setting

vim /etc/grub.conf

It must look like this:

default=0
timeout=3
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.26)
root (hd0,0)
kernel /vmlinuz-2.6.26 ro root=/dev/VolGroup00/LogVol00 rhgb quiet
initrd /initrd-2.6.26.img
title CentOS (2.6.18-53.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-53.el5 ro root=/dev/VolGroup00/LogVol00 rhgb quiet
initrd /initrd-2.6.18-53.el5.img

6) Finally

Well we are done, restart the system and enjoy.

restart

7) Test l7-filter

iptables -m layer7 --help

Have fun!