WHITE PAPER
Table of Contents
Introduction
Transport Requirements
IP/MPLS Transport
Route Analytics
10
12
13
15
Concluding Remarks
17
Introduction
With the advent of the smart phones, tablets and other connected devices, traffic has grown
exponentially and created congestion in mobile networks. Circuit switching 2G and 3G mobile networks
allocate bandwidth statically (as they require TDM circuits) to each cell site and do not take advantage of
the statistical multiplexing found in packet switching networks. Wired service providers, however, have
adopted packet-switching IP/MPLS-based network architectures to take advantage of the bandwidth
efficiency and higher resiliency, while choosing Ethernet for framing due to significant cost per port
reductions.
To meet ever-increasing user traffic demands, mobile orators have embraced Long Term Evolution (LTE)
radio access. LTE Advanced can provide up to 1 Gbps bandwidth to each user. If unchecked, this LTE traffic
would further burden the mobile network. To address these challenges, the Third Generation Partnership
Project (3GPP) has defined System Architecture Evolution (SAE), the core network architecture for the
non-radio access part of the network. The main component of the SAE is a new IP-based Evolved Packet
Core (EPC), shown in Figure 1. Together, SAE and LTE form the Evolved Packet System (EPS). Mobile
operators have been deploying EPS networks, often marketed simply as LTE networks.
IP
eNodeB: This is the LTE evolved base station. eNodeBs are the radio towers to which user
equipment (UE), such as cell phones and tablets, connect.
Serving Gateway (S-GW): The S-GW is a data plane element. S-GWs are typically placed at the
demarcation point between the radio access network (RAN) and core network. The S-GWs main
purpose is to track the users mobility and to send its traffic to the appropriate eNodeB as the
user moves. All user packets are carried inside bearers logical pipelines connecting two or
more points. The S-GW redirects these bearers as a UE moves from one eNodeB to the next. It also
maintains state for the bearers when the UE enters low power mode and un-associates its bearers.
Packet Data Network (PDN) Gateway (P-GW): The P-GW is also a data plane element. P-GWs are
placed at the demarcation of the PDN. The P-GW assigns IP addresses to UEs, enforces QoS, filters
packets, collects charging information, and forwards UE IP packets to/from the PDN, including the
Internet.
Mobility Management Entity (MME): The MME is a control plane element. It manages the UE,
including access to the network, assignment of resources, and management of mobility (i.e.,
tracking, paging, roaming and hand-over). For example, when new packets arrive at an S-GW for a
UE that is in low power mode, MME pages that UE so that it can reestablish its bearers and receive
the packets S-GW had been buffering.
Policy and Charging Rules Function (PCRF). The PCRF is a control plane element. As the name
implies, it is the policy and charging brain for the network. However, enforcement is done at the
P-GW. The PCRF tells the P-GW how to handle packets. For example, for a user who has exceeded
their quota, the PCTF may instruct the P-GW to rate limit the users packets.
Home Subscriber Server (HSS): The HSS is a control plane element. It contains a users subscribed
services, such as whether or not the user is allowed to roam and the QoS treatments to which they
have subscribed.
The above description is overly simplified, but sufficient to serve this white paper. Since this white paper is
about providing an IP/MPLS transport to the EPS, the most relevant elements are eNodeBs, S-GWs, P-GWs
and MMEs. Also, we purposefully illustrated them as logical elements. They may be appliances on their
own, may be running in a blade in some other appliance such as an IP/MPLS router, or may be part of a
combo-device, for example, one with IP/MPLS routing and forwarding functionality.
Transport Requirements
There are two main kinds of packets the network carries: data (including voice) and control packets. UE
data packets are IP packets. At this IP layer, UE is one hop away from the P-GW. That is, the next IP hop
from the UE is the P-GW. This is because these packets are relayed through the eNodeB and S-GW using
GPRS Tunneling Protocol (GTP). GTP itself rides on UDP that rides on the networks true IP layer. The
control packets between the MME and UE are also relayed over IP (but use a different set of protocols).
eNodeBs communicate with S-GWs and MMEs. S-GWs also communicate with P-GWs. In addition, as a UE
moves between neighboring eNodeBs, the hand over is done directly between the involved eNodeBs.
Hence, eNodeBs also communicate with neighboring eNodeBs. This is best illustrated by Figure 2. The
GTP connections between eNodeBs are referred to as X2 connections and the GTP connection between
an eNodeB and an S-GW or an MME is referred to as an S1 connection. eNodeBs, S-GWs, MMEs and the S1
and X2 connections between them forms the RAN (E-UTRAN). The network that transports these S1 and
X2 connections is often called the backhaul network. The network that connects S-GWs to P-GWs and
other EPS elements and the Internet is called the backbone network.
MME/S-GW
MME/S-GW
HSS
PCRF
MME
UE
eNodeB
S-GW
P-GW
Operators
IP services (e.g. IMS)
IP/MPLS Transport
A typical mobile operator will have a national backbone network and for each of its regions it will have a
backhaul network. Already with 3G, backbone networks transported mobile traffic using IP/MPLS control
plane, more specifically using IP/MPLS BGP VPNs (often referred to as L3VPNs). Because now the payload
on the backhaul network is also IP, IP/MPLS can be used in the backhaul network as well.
There are many possible IP/MPLS transport architectures for backhaul and backbone networks [See Cisco
UMMT at https://communities.cisco.com/docs/DOC-30621]. In this paper, we focus on two scenarios that
are most widely deployed by the mobile operators. In both scenarios, L3VPNs are used in the backbone
network, just like with 3G networks. In the first scenario, L3VPNs are extended to the backhaul networks.
In the second scenario, L2VPNs, specifically Virtual Private Wire Service (VPWS), are used in the backhaul
network. Virtual Private LAN Service (VPLS) can also be used in the backhaul network. However, it is
difficult to manage and often avoided.
For both scenarios, at each cell, a router referred to as the cell site gateway (CSG) is paired with an eNodeB.
This CSG has an interface connected to the eNodeB and other interfaces connecting the cell site to the
rest of the IP/MPLS backhaul infrastructure.
Similarly S-GWs, P-GWs, MMEs, PCRFs and HHS reside in mobility sites. Each of these sites also has an
IP/MPLS router, referred to as a Mobility Site Gateway (MSG) or Mobile Transport Gateway (MTG). The MTG
has interfaces connected to the EPS elements, as well as interfaces connected to the backhaul and/or the
backbone infrastructure. This is pictorially depicted in Figure 3.
S1
eNodeB
SGW
MTG
MTG
MME
S1
MPLS VPN
CSG
MTG
SGW
Aggregation Node
(AGN)
CSG
Core Node
RAN
IP/MPLS
domain
Aggregation Network
IP/MPLS Domain
Pre-Aggregation
Node (AGN)
CSG
CSG
Aggregation Node
(AGN)
Core Network
IP/MPLS Domain
Core Node
Aggregation Node
(AGN)
CSG
Core Node
Aggregation Network
IP/MPLS
Domain
Pre-Aggregation
Node (AGN)
Core Node
Aggregation Node
(AGN)
RAN
IP/MPLS
domain
CSG
CSG
Backbone
Provider Edge
(PE) Router
PE
PE
P-GW
PE
Route Reflector
RR
PE
IS-IS
PE
PE
PE
PE
PE
L2VPN Backhaul
PE
S-GW
PE
OSPF PE
PE
Backbone
Provider Edge
(PE) Router
PE
PE
P-GW
PE
Route Reflector
RR
PE
IS-IS
PE
PE
PE
PE
S-GW
PE
L3VPN Backhaul
PE
PE
Inline RR
OSPF PE
PE
Route Analytics
Route analytics taps into the routing protocols the source of intelligence that determines how
IP/MPLS networks deliver traffic and uses these protocols to provide very accurate network topology
and routing visibility, including the IGP protocols, MP-BGP, Layer 2 and Layer 3 VPNs, RSVP-TE tunnels, and
the interactions among them.
Route analytics technology uses a collector that acts like a passive router. The collector peers with
selected routers across the network, and using the routing protocols OSPF, IS-IS and MP-BGPit
collects and records the control messages that routers exchange in order to calculate how traffic will be
sent across the network (see Figure 6). By processing this information just the way routers do albeit in
a more comprehensive fashion every Layer 3 routed path in the network can be calculated, from every
EPS element to every other element. Thus, a routing topology of the entire network can be created and
maintained for operational and engineering analysis. Since routing protocols report changes to the
topology in real time, this topology map is continuously updated in real time and always reflects exactly
the way network is operating.
Mobility Site
MME
Backbone
Provider Edge
(PE) Router
P-GW
PE
PE
PE
Route Reflector
PE
IS-IS
PE
PE
PE
L2VPN Backhaul
S-GW
PE
PE
PE
VLL
OSPF PE
Figure 6. Route analytics technology passively peers with, listens to and analyzes routing protocols to provide a
real-time, network-wide understanding of all IP/MPLS VPN topology changes
For protocols that can not be collected directly (e.g. RSVP-TE tunnels, static routes, pseudo-wires) route
analytics technology queries the routers using a combination of SNMP, NETCONF, and CLI. To catch the
changes in real time, it taps into SNMP traps, syslog messages and changes in IGP. For example, when a
link fails, IGP will communicate this to the collector and the collector will query the head-end routers of
the tunnels that were going over this link. This is both lighter weight than a polling system (only relevant
routers are queried and only when needed) and real time.
Figure 13. RSVP-TE Tunnels and missing node and link protection for the tunnel
Figure 13 shows the tunnels in the lab network. Some of these tunnels have secondary alternates that
are pre-signalled, as shown in the paths column. Most tunnels in this table have requested node and link
protection. However, as can be seen in the FRR column, most of the tunnels only have partial protection.
As new links and routers are added to the network, it is easy to forget to create these FRR tunnels. For the
highlighted tunnel, its path is shown underneath the table. Routers and links with red crosses show where
the protection is missing. It is also possible to see the protection path by clicking on protected elements.
The FRR tunnel protecting R3s failure is shown in Figure 14. Notice that this is a really long path (and
hence can violate some EPS performance requirements). And also notice that the FRR tunnel goes over
the tunnel tail, then backwards to R1, then forwards to the tunnel tail again. This is not unusual though
not efficient. This FRR tunnel may be protecting many other tunnels and it may be efficient for others.
Mobility
Site
Flow 1
Flow 2
Flow 3
Flow 4
eNodeB
Src
Src
Src
Src
Dest
Dest
Dest
Dest
...
...
...
...
NetFlow Data
Internet
User
Figure 15. Route analytics integrates NetFlow traffic statistics into the IP/MPLS VPN routing topology by
mapping traffic flows onto their routed paths. The result is a real-time, integrated routing and traffic topology.
Concluding Remarks
In order to handle the explosion of mobile traffic, mobile operators are deploying IP/MPLS in their
backhaul networks. The dynamic nature of the IP/MPLS control plane requires a new network
management paradigm as SNMP polling-based OSS systems can not keep up with the scale and rate of
changes in these environments.
Route analytics technology addresses this challenge very well as we have demonstrated above with a few,
certainly not exhaustive examples. Additional benefits of route analytics technology that are applicable to
mobile operators include:
Email us at info@packetdesign.com
Visit Packet Designs web site at www.packetdesign.com
Call us at +1.408.490.1000