Research Report about IPsec VPN

Twan
talitwan@os3.nl

Jonel
jspellen@os3.nl

Fangbin,
fliu@os3.nl
2005-12-13

Abstract
IPsec VPN supplies a secure transport medium for the private net-
work in a public environment. In this case research, different aspects
of IPsec VPN has been investigated, such as the implementation of
IPsec VPN, scalability and security. Although, IPsec supplies a secure
transfer method over the internet, it is still vulnerable for some certain
sort of attack such as sniffing and so forth. Also, the scalability of
IPsec VPN is a big problem for its success, although it achieves a low
cost through applying the pubic network medium.

1
Contents
1 Introduction 3
1.1 What is VPN ? . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 What is IPsec . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Why IPsec . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Components of IPsec VPN 4

2.1 IPsec Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 IKE Management . . . . . . . . . . . . . . . . . . . . . . . . . 5

3 Building an IPsec connection 5

3.1 Transport or tunnel-mode . . . . . . . . . . . . . . . . . . . . 6
3.2 NAT traversal . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.3 IPsec between two hosts or networks . . . . . . . . . . . . . . 7
3.3.1 Linux to Linux . . . . . . . . . . . . . . . . . . . . . . 7
3.3.2 Windows to Windows . . . . . . . . . . . . . . . . . . 7
3.3.3 Linux to Windows . . . . . . . . . . . . . . . . . . . . 8

4 Scalability of IPsec VPN 9

5 Security of IPsec VPN 10

5.1 ISAKMP Vulnerability Id:20051114-01013 . . . . . . . . . . . 10
5.1.1 Recommendations . . . . . . . . . . . . . . . . . . . . 10
5.2 Vulnerability Advisory IPSEC id: 20050509-00386 . . . . . . 11
5.2.1 Solution . . . . . . . . . . . . . . . . . . . . . . . . . . 11
5.3 Weak Encryption . . . . . . . . . . . . . . . . . . . . . . . . . 12
5.3.1 The algorithm . . . . . . . . . . . . . . . . . . . . . . 12
5.4 Authentication cracking tool . . . . . . . . . . . . . . . . . . . 12
5.4.1 How IKECrack works . . . . . . . . . . . . . . . . . . 13
5.5 Microsoft PPTP protocol used with VPN . . . . . . . . . . . 13

6 Conclusion 13

2
1 Introduction
1.1 What is VPN ?
VPN is an abbreviation for Virtual Private Network. VPN is built up on
the basis of the existing network. Through transferring the private data
over public domain, the cost for it is reduced significantly. Since data is
transferred over a public medium, the confidentiality must be protected.

Various kinds of encryption can be used by VPN. There are two main
ways of encryptions, Symmetric Cryptography and Asymmetric Cryptogra-
phy . With symmetric cryptography, the same key is used for both of en-
crypting and decrypting the messages. On the other hand, with asymmetric
cryptography, two keys are used for encryption and decryption. In most of
cases, the asymmetric encryption is used to authenticating each other, while
the symmetric encryption is applied to supply the confidentiality of the data.
Some popular symmetric encryption algorithms include DES, AES, 3DES
and so forth. Some famous asymmetric algorithm consists of such as RSA,
DSA, and so forth.

1.2 What is IPsec

IPsec is a collection of some special internet protocols to supply a secure
data transfer over the network layer. The standard IP (Internet Protocol)
supports no security mechanism when it was designated firstly. With the
increasing demands of the internet security, some new protocol have been
developed for the network layer, such as AH, ESP, and so forth. IPsec be-
comes the name of the collection of all these protocols.

Therefore, IPsec can be utilized in multiple application for security pur-

pose. For example, the confidentiality and integrity of data transferred over
the network can be guaranteed, the authentication of data sender can be
realized, the relay and the analysis of the data transferred over the public
internet can be prevented, and so forth.

1.3 Why IPsec

IPsec has became a much more popular VPN security technology than many
other concurrent such as PPTP, L2TP, and so forth because of some utility

3
advantages supplied by it.

Firstly, many VPN product vendors support these protocol, since these
protocol is compatible with many other protocols. In this way, the vendor’s
products will be quite compatible with other vendor’s product.

Further, the automatic key exchange mechanism make the integrating

of new equipment into the VPN network much easier.

Another significant advantage IPsec supplies is that, it works in the

network layer, which means that the user application will not need to apply
this protocol, they can just enjoy a secure network transfer transparently.
All the way, the IPsec has been a standard method for the VPN technologies.

2 Components of IPsec VPN

In this section, the main working theory of IPsec by VPN will be introduced.
IPsec uses various protocols to process different kinds of security tasks, such
as origin authentication, data confidentiality keeping, access control and so
forth. Further, two main transfer modes are used within IPsec which are
transport mode and tunnel mode. These two modes are used for various
links over the transfer parties. Also, to establish a secure data transfer,
a confidential key exchange mechanism is used by IPsec. Finally, various
implementation modes of IPsec are supplied for various purposes.

2.1 IPsec Protocols

There are two main protocols to support the security on IPsec. The first one
is AH standing for Authentication Header, and the other is ESP for Encap-
sulated Security Payload. AH is used for the transport origin authentication,
further the access control and anti-replay protection and so forth.

The ESP can be used to supply a secure link to transfer confidentiality

data, and to achieve a limited traffic flow confidentiality. These two pro-
tocols can be used individually as well as be used together. When used
individually, a secure communication link can be built up for the transfer
parties. When used with each other, a combination of security service can
be supplied.

4
 For the authentication, a key exchange algorithm must be implemented
so that the confidentiality and integrity can be kept. The method for key
exchange will be discussed in the coming chapters.

2.2 IKE Management

As mentioned in the previous section, both of the transfer parties need to
authenticate each other before a confidentiality transfer can be initiated.
For this purpose, a special protocol called ”Internet Key Exchange” is used
for communication parties to negotiate, create and process the security as-
sociations used for their transfer. With Security Association, a number of
security policies is defined, for the connections.

There are two phases needed to build up a secure communication link.

In the first phase, the goal is to establish a secure transfer channel so that
in the second phase, the IPsec security associations can be exchanged safely.
In the second phase, the real IPsec security associations are negotiated and
built up. In each of these two phases, various mode is used such as main
mode or passive mode. Also various protection algorithm is used for each
mode, such as AES, DES, Deffie-Hellman for various goals.

3 Building an IPsec connection

Before encrypted data can travel from one side to the other, a number of
key exchanges has to be done. These exchanges, necessary to negotiate a
session key, are called phases. The first, main phase is mainly responsible
for the encryption negotiation. The second, quick phase initializes the SA’s,
Security Association, with the pre-shared keys or certificates. The SA’s are
the IPsec end-points and encrypt data with a session key. When a IPsec
connection expires, only the quick phase is needed to negotiate a new session
key and rebuild the connection. This expiring is critical, because excessive
use of the same session key will weaken the encryption.

The key exchange in the first, main phase can be done in main-mode
or aggressive-mode. The aggressive-mode skips the encryption mechanism
negotiation, thus it is recommended to use main-mode. Main-mode also
makes eaves-dropping more difficult (see chapter 5.1.1). This alone makes
its use preferred.

5
 There are several ways to define keys. Keys can be pre-shared and used
as a shared secret or another way is the use of the public/private key mech-
anism. The last mechanism is also known as certificates and is the most
recommended method while it reveals the least about the cipher. That way
it’s more difficult to crack the connection.

Thus to establish a VPN connection, the following procedures are exe-

cuted:

• IKE: Phase 1: main-mode or aggressive-mode (encryption negotiation)

• IKE: Phase 2: quick-mode (setup of SA’s)

• IPsec: starting tunnel (network data traffic)

3.1 Transport or tunnel-mode

IPsec can be utilized for multiple sorts of transfer endpoints. This can be
either encryption in transport-mode or tunnel-mode.

Tunnel-mode is useful when encryption is only needed between two fire-

walls (site to site). An example is a network connection between two remote
sites. The traffic between these to sites has to be secure. In this mode, the
protection of data is not fully provided, on the other hand, a host does not
need to know about IPsec, which makes security easy and invisible.

Transport-mode encrypts the data stream completely and adds an new

IP header in front of the ESP packet. Transport-mode is recommended
when users work off-site on a foreign network. This method is also referred
as end to end encryption.
The data stream that leaves a host is yet encrypted and can not be sniffed.
This can lead to some error prone situations, especially when the network’s
firewall or router is blocking certain ports or uses NAT. To work around the
problem with NAT, NAT traversal, also known as NAT-T was invented.

3.2 NAT traversal

Many experts believe that NAT is an bad solution to the IP shortage due
to IPv4. IETF designed IPsec that way that it shouldn’t work over a NAT
router. That way they thought that people would move to IPv6 - designed
by the IETF as well - sooner. Unfortunately IPv4 was propagated that much

6
that people circumvented this problem instead of moving to IPv6.

To solve issues with NAT routers, ESP has to travel through like other
TCP or UDP packets. So instead of being used as a protocol equally like IP,
ESP is encapsulated in an UDP packet. This way it is possible to connect
VPN’s over a NAT setup.

The use of NAT-T is not recommended. It makes things more compli-

cated than they already are. When a situation arises where a VPN must be
build over a NAT router, a better solution is to let the NAT router forward
all the incoming traffic to a default host. This host then can handle the ESP
traffic and do firewalling. There are NAT routers available on the market
that are IPsec aware. These routers can handle ESP traffic in a more del-
icate way without bludging it into UDP streams. They can even build the
transport-mode tunnel with the other side. [6]

3.3 IPsec between two hosts or networks

A SA, Security Authority, that connects a VPN to a LAN is also known as a
VPN gateway. This is because of its gateway-ing nature. The gateway can
connect one LAN to another. Multiple hosts can by this gateway connect
to ’the other side’.

3.3.1 Linux to Linux

.or unix to unix. To build a VPN connection between to Linux hosts is sim-
ple. Only IPsec is needed with a pre-shared key as minimum. When using
Kernel 2.6 the setup of IPsec is even more convenient because of the build-in
encryption capabilities. Kernel 2.4 needs to be patched against the userland
sources to fully support IPsec. Independent of kernel version, Openswan or
FreeSwan is necessary to build the meant connection. Although both kernels
work, version 2.6 is recommended. [18]

3.3.2 Windows to Windows

VPN technology was made available to Windows 2000 and XP out-of-the-
box. It’s fairly easy to interconnect modern Windows machines. With some
extra software it is also possible to connect Windows 98 and ME as well.
There’s a lot of third-party software on the market that does the same thing
as the already available implementation. This software on the other hand

7
can come in handy when creating certificates (discussed later).

The authentication of the underlaying IPsec connection is done primar-

ily by PKCS12-certificates, although XP (not 2000) supports pre-shared
keys. The creation of certificates is the hardest part of setting up a VPN
connection. Openssl on a Linux box can be useful when creating certificates.

The Microsoft implementation of VPN differs somewhat from the stan-

dard implementation. Microsoft uses an additional tunnel over IPsec to
establish a connection. This extra tunnel is created by PPTP or L2TP. The
reason why Microsoft had choosen to do things this way, is that they believe
that certificates can only be used to authenticate hosts, not humans. PAP
or CHAP are used to check user credentials and therefor an extra layer,
PPTP or L2TP is needed. [16]

Thus to establish a VPN connection, the following procedures are exe-

cuted:
• IKE: Phase 1: main-mode or aggressive-mode (negotiation)

• IKE: Phase 2: quick-mode (SA setup, host authentication)

• IPsec: starting tunnel (underlaying tunnel)

• L2TP: starting additional tunnel (network data transfer. user authen-

tication)

3.3.3 Linux to Windows

As said before, Windows uses an extra tunnel, authenticated with PAP or
CHAP, over the IPsec connection. PPTP is known to be insecure (see chap-
ter 5.5), which makes L2TP the preferred standard. Thus, when it comes
to connecting Windows machines to Linux, extra software is needed. This
software establishes the PPP connection over this tunnel, so that Linux can
talk to the Windows host.

The PAP or CHAP authentication can be done by the very available

PPPd, bundled with every linux or unix distribution. Further L2TPd is
needed, which is available as a RPM package. When PAP authentication is
used, one can use a single daemon, L2TPNSd, to do both creating a L2TP
tunnel and do the PPP authentication. One drawback of PAP is that it
isn’t encrypted. Although this isn’t a big issue, because of the underlaying

8
IPsec tunnel, it is not recommended. Instead, CHAP or MS-CHAP is the
preferred standard. Microsoft tends to use own products or standards and
therefor MS-CHAP has somewhat better support.[16]

It is also recommended to setup the IPsec connection with certificates.

The only difference is that on Linux, PEM certificates can be used. Even
DER format is supported on Linux. To make the IPsec connection work,
one has to be sure that on both ends, the same root certificate is used to
sign the individual ones. To eliminate problems, it is possible to use the
same certificate on both Linux and Windows. The Windows variant still
has to be in PKCS12 format.[17]

The IPsec tunnel across Internet must be setup with routable Internet
addresses, but the PPTP or L2TP tunnel can be established with private
or non-routable addresses. When the latter is the case, it has the advantage
that the connection is assured. It’s certain that the non-routable traffic will
not get to the other side without the L2TP tunnel. When routable addresses
are used, traffic can still reach another host without the use of the tunnel.
So it is recommended that PPTP or L2TP tunnels are made with private,
non-routable Internet addresses.

4 Scalability of IPsec VPN

Although the IPsec supplies a good protection for the confidentiality data
transferred over the public network, it has also raised some drawback for
its application. One of them is the limitation on the scalability of IPsec
over VPN. Since IPsec VPN is implemented by the tunnel transfer mode,
the tunnel server for the package processing will be extremely loaded when
multiple packages need to be sent to local endpoints.

Another drawback is that IPsec VPN does not support broadcast. IPsec
VPN is designated for the point-to-point communication which is secured
with tunnel mode. With this mode, the message sent over the internet are
all the unicast. This characteristic also make the relay operation for the
message impossible since the whole body of the original data is encrypted
and packaged with the IPsec package as explained in the section 3.1. Also
the bandwidth will be affected by the multiple unicast package.

9
5 Security of IPsec VPN
5.1 ISAKMP Vulnerability Id:20051114-01013
A group at the University of Oulu (Finland)[7] developed a test suite called
“OUSPG PROTOS ISAKMP” to generate abnormal ISAKMP traffic. As
they used this test suite against various IPSEC implementations, they found
them to be vulnerable[8].

The severity of these vulnerabilities varies by vendor. These flaws may

expose denial-of-service conditions, formatproven very susceptible to attack
string vulnerabilities, and buffer overflows. All these could shut down de-
vices and slow transmission of data across the Internet. In some cases, they
could also allow hackers to execute code and hijack a device.

Cisco and Juniper, two of the largest networking technology vendors,

acknowledged that some of their products are at risk. OpenSWAN an open
source software, may be used in many Linux and BSD based appliances is
also vulnerable.

The OUSPG PROTOS ISAKMP Test Suite does not test Internet Key
Exchange version 2 (IKEv2), it is based on IKEv1. ISAKMP consists of
two phases. In phase 1, the two parties negotiate a SA to agree on how to
protect the traffic in the next phase. In phase 2 keying material is derived
and the policy to share it is negotiated. In this way, security associations
for other security protocols are established.

Multiple ISAKMP implementations behave in anomalous way when they

receive and handle ISAKMP Phase 1 packets with invalid and/or abnormal
contents. By applying the OUSPG PROTOS ISAKMP Test Suite to a
variety of products, several vulnerabilities can be revealed that can have
varying effects.

5.1.1 Recommendations
These are the suggestions recommended by the NISCC[9] when mitigate to
IPSec against the issues discussed in this advisory:

• If possible, use packet filters and accepts ISAKMP negotiations only

from trusted IP-addresses

10
 • Avoid using “aggressive mode*” in phase 1

[*In “aggressive mode”, fewer exchanges are made and with fewer pack-
ets during the negotiation stage. The weakness of using this mode is that
both sides have exchanged information before there is a secure channel.]

The solution to this situation is by looking at the Vendor Information.

They came out with patches for platform specific remediation.

5.2 Vulnerability Advisory IPSEC id: 20050509-00386

In May 2005 a vulnerable[10] was identified about three attacks that apply
to certain configurations of IPsec. These configurations use Encapsulating
Security Payload (ESP) in tunnel mode with confidentiality only, or with
integrity protection being provided by a higher layer protocol. Some con-
figurations using AH to provide integrity protection are also vulnerable. In
these configurations, an attacker can modify sections of the IPsec packet,
causing either the clear text inner packet to be redirected or a network host
to generate an error message. In the latter case, these errors are relayed
via the Internet Control Message Protocol (ICMP); because of the design
of ICMP, these messages directly reveal segments of the header and pay-
load of the inner datagram in clear text. An attacker who can intercept the
ICMP messages can then retrieve plaintext data. The attacks have been
implemented and demonstrated to work under realistic conditions.

5.2.1 Solution
Any of the following methods[11] can be used to rectify this issue:

1. Configure ESP to use both confidentiality and integrity protection.

This is the recommended solution.

2. Use the AH protocol alongside ESP to provide integrity protection.

However, this must be done carefully: for example, the configuration
where AH in transport mode is applied end-to-end and tunneled inside
ESP is still vulnerable.

3. Remove the error reporting by restricting the generation of ICMP

messages or by filtering these messages at a firewall or security gateway.

11
5.3 Weak Encryption
In some cases an IPSec connection is made with the use of client software.
The user and group password are mostly store in the local user profile file.
If a weak encryption is used it can be revealed with the knowledge of a good
cryptographer. In this article some describe that he has found a way how
to reveal the password of a Cisco VPN Client. He describes[12]:
The main problem of the method used to encrypt the passwords is, that the
whole procedure is deterministically and no user input is used. This effec-
tively means that the encryption keys the Cisco Client calculates can also
be calculated by any other program whensoever this program knows the algo-
rithm. This algorithm was now reversed.

5.3.1 The algorithm

The algorithm which is used to encrypt a given user/group password is
shown below

• The current date as a string is retrieved (e.g. Mon Sep 19 20:00:00

2005) Then a SHA-1 Hash h1 is computed (20 Bytes)

• h1 is modified and a new Hash h2 is calculated

• h1 is again modified and h3 is calculated

• The 3DES key is made of h2 and the first 4 bytes of h3

• The password is encrypted using 3DES in CBC Mode. The IV consists

of the first 8 bytes from h1.

• The algorithm computes a last hash h4 from the encrypted password

• The key “enc UserPassword” in our profile file now looks like this:
h1|h4|encrypted password

5.4 Authentication cracking tool

A tool who can crack the IKE/IPSec authentication is IKECrack. It is an
open source tools which is design to brute force or dictionary attack the
password key/password used with Pre-Shared Key [PKE] IKE authentica-
tion. This tool was built to demonstrate proof-of-concept ad will work with
RFC 2409 based aggressive mode PSK authentication.

12
5.4.1 How IKECrack works
IKE Aggressive Mode BruteForce Summary Aggressive Mode IKE authen-
tication is composed of the following steps[13]:
1. Initiating client sends encryption options proposal, DH public key,
random number [nonce i], and an ID in an un-encrypted packet to the
gateway/responder.
2. Responder creates a DH public value, another random number [nonce r],
and calculates a HASH that is sent back to the initiator in an un-
encrypted packet. This hash is used to authenticate the parties to
each other, and is based on the exchange nonces, DH public values,
the initiator ID, other values from the initiator packet, and the Pre-
Shared-Key [PSK].
3. The Initiating client sends a reply packet also containing a HASH, but
this response is normally sent in an encrypted packet.

5.5 Microsoft PPTP protocol used with VPN

PPTP (Point-to-Point Tunneling Protocol) is a Microsoft VPN protocol
published as an RFC in 1999 for secure remote access. This protocol has
en is been used in many Microsoft based networks, firewall appliances, and
even pure Linux and Open Source environments[14].

In 2003 Joshua Wright created ASLEAP[15] tools to prove that a pass-

word based authentication system like Cisco LEAP is not a secure because
of one glaring weakness, it relies on humans to memorize strong passwords.
ASLEAP just happens to make that point abundantly clear since it had
the ability to scan through a 4 GB pre-computed password hash table at a
rate of 45 million passwords a second using a common desktop computer for
cracking passwords.

Better Solution is to use L2TP (Layer 2 Tunneling Protocol) protocol

with VPN.

6 Conclusion
IPsec is very useful, if used the right way. Use main-mode, not aggressive-
mode. When connecting Windows to Windows or Windows to Linux, use

13
L2TP instead of PPTP. Last but not least, use CHAP or MS-CHAP instead
of PAP.

IPsec is most proper to be applied in the communication of point-to-

point type. As a result, the scalability of IPsec has been limited with more
number of nodes added to the network. Also, since IPsec supply the con-
fidentiality and integrity of the original data through encrypt them and
adding its new header, many process options in the original header can not
be accessed when the package is transferred in the network.

IPsec provide a nice way to secure the data when it is transferred through
the public network through building up a secure link between sender and
receiver. But, on the other hand, it is still possible to be attacked by some
kind of sniffing attack, or man-in-the-middle attack for example on the local
network of the each end point before data is sent by the gateway over the
network.

Using the well known encryption algorithms is better than making your
own encryption algorithm. Because those well known algorithms have been
and are tried to be cracked by thousand of people everyday. This is why
it’s better using known encryption algorithms. Second in many cases peo-
ple make mistakes in the implementation of those encryptions algorithms in
their product.

Now a day MD5 hash algorithm is considered cracked. For the imple-
mentation for IPSec (HMAC) it would be better to use SHA-1 or other
strong hash algorithms. The same for DES encryption, it is also consider
cracked. AES and RSA are stronger encryption.

Despite of the complexity, IPsec has been able to work together with
many other services supplied by multiple network infrastructures, such as.
Therefore, IPsec has became almost a standard secure communication ser-
vices.

14
References
[1] Wipul Jayawickrama: Demystifying IPSec, Information Security Man-
agement System, 2003

[2] Sheila Frankel, Karen Kent, Ryan Lewkowski, Angela D. Orebaugh,

Ronald W. Ritchey, Steven R. Sharma: Guide to IPsec VPNs, Com-
puter Security Division, Information Technology Laboratory, National
Institute of Standards and Technology, Gaithersburg, MD 20899-8930,
January 2005

[3] The Illusion of Security: Using IPsec VPNs to Secure the Air, Trapeze
Networks,

[4] George Hadjichristofi: IPSec Interoperability and Scalability, Computer

Engineering, Virginia Tech, 2003

[5] Prakash Iyer, Victor Lortz, Ylian Saint-Hilaire: Scalable Deployment of

IPsec in Corporate Intranets, Intel Architecture Labs Internet Building
Blocks Initiative, 2000

[6] Charlie Kaufman, Radia Perlman, Mike Speciner: Network Security,

Private Communication in a PUBLIC World, second edition 2002

[7] PROTOS Test-Suite: c09-isakmp, University of Oulu, November 2005,

[8] IPSEC / ISAKMP Vulnerability wrapup, SANS, November 2005,

[9] Vulnerability Advisory 273756/NISCC/ISAKMP, NISCC, November

2005,

[10] Bill Brenner, News Writer: High-severity vulnerability in IPsec, Search-

Security.com, May 2005,

[11] NISCC Vulnerability Advisory IPSEC - 004033, NISCC, May 2005

[12] Geschrieben von HAL: Cisco Password Encryption reversed, EvilScien-

tists, October 2005

[13] Anton T. Rager: IKECrack, http://ikecrack.sourceforge.net/, 2001-

2002

[14] George Ou: PPTP VPN authentication protocol proven very susceptible
to attack, ZDnet.com, December 2004

15
[15] Joshua Wright: Asleap behind the wheel, http://asleap.sourceforge.net/,
sourceforge.net, 2004,

[16] Jacco de Leeuw: Using a Linux L2TP/IPsec VPN server,

http://www.jacco2.dds.nl/networking/freeswan-l2tp.html

[17] Nate Carlson: Configuring an ipsec tunnel between openswan

and windows 2000 / xp, http://www.natecarlson.com/linux/ipsec-
x509.php#casetup

[18] Xelerance Corporation: Openswan, http://www.openswan.org

16