Anda di halaman 1dari 58

Troubleshooting MPLS VPN Networks

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

Agenda
Control Plane Troubleshooting
Forwarding Plane Troubleshooting
Conclusion

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

Agenda
Control Plane
Control Plane Troubleshooting Tips
Real-life Examples
Summary of Helpful Cisco IOS Commands

Forwarding Plane
Dissecting LFIB
Load sharing in MPLS VPN Networks
Forwarding Plane Troubleshooting Tips
Real-life Examples
Summary of Helpful Cisco IOS Commands

Conclusion
MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

MPLS VPN - Troubleshooting Tips


Symptom

Tip

Cisco IOS
Command

VPNv4 Prefix Is Not Received at the


Remote (Receiving) PE

Make Sure that export RT <X> at


the Advertising PE Matches with
import RT <X> at the Receiving
PE

Sh ip vrf detail <vrf> | inc


Export|import|RT

VPNv4 Prefix Is Not Received at the


Remote (Receiving) PE

Validate the Match/Set Clause


within the Export-Map or ImportMap (if Any)

sh ip vrf de <vrf> | inc


route-map;

VPNv4 Prefix Is Not Received at the


Remote (Receiving) PE

If BGP Is Not the Chosen PE-CE


Protocol, then Validate BGP->IGP
Redistribution

sh run | b router <igp>

VPNv4 Prefix Is Not Received at the


Remote (Receiving) PE

Check whether the Remote PE Is


Configured as the rr-client within
VPNv4 af at the Route-Reflectors

Sh run | b address-family
vpnv4

4
MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

sh route-map <map>

MPLS VPN - Troubleshooting Tips (Cont.)


Cisco IOS
Command

Symptom

Tip

VPNv4 Prefix Is Not Received


at the Remote (Receiving) PE

Make Sure that the RouteReflectors and PEs Are


Configured to Send ExtCommunity towards the
iBGP Peers within the
VPNv4 af

sh run | b addressfamily vpnv4

VPNv4 Traffic Is Not Getting


Forwarded End-to-End

Check the Label


Information in BGP and
LFIB at the Advertising PE
Router

sh ip bgp vpn vrf <vrf>


label | inc <prefix>

Check the Label


Information in BGP and
FIB at the Receiving PE
Router

sh ip bgp vpn vrf <vrf>


label | inc <prefix>

6
VPNv4 Traffic Is Not Getting
Forwarded End-to-End

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

sh mpls for vrf <vrf> |


inc <prefix>

sh ip cef vrf <vrf>


<prefix>

MPLS VPN Label Stack

Outer (or IGP) label in the label stack provides a


LSP from ingress PE to egress PE via MPLS cloud

Inner (or BGP) label refers to the VPNv4 prefix at


the egress PE

tag rewrite with Se2/0, point2point, tags imposed: {2003 20}

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

Agenda
Control Plane
Control Plane Troubleshooting Tips
Real-life Examples
Summary of Helpful Cisco IOS Commands

Forwarding Plane
Dissecting LFIB
Load sharing in MPLS VPN Networks
Forwarding Plane Troubleshooting Tips
Real-life Examples
Summary of Helpful Cisco IOS Commands

Conclusion
MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

MPLS VPN Real Life Examples


Lets do some MPLS VPN trouble(shooting)

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

MPLS VPN Ctrl PlaneTrouble #1


#1: VPN prefix doesnt have any label in the LFIB on
the local PE

PE1
Ser2/0
200.1.61.4/30

CE1

Loop0:10.13.1.61/32

PE1#sh mpls forwarding vrf v1 | i 200.1.61.4


PE1#
PE1#sh ip bgp vpn vrf v1 label | i 200.1.61.4
AS#1
PE1#
PE1#sh ip bgp vpn vrf v1 200.1.61.4
MPLS Backbone
%Network not in the table
PE1#

TIP: Label allocation is done by BGP. So make sure the prefix is in the BGP
VRF table. Hintredistribute connected

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

MPLS VPN Ctrl PlaneTrouble #1 (Cont.)


PE1(conf)#router bgp 1
PE1(conf-router)#address-family ipv4 vrf v1
PE1(conf-router-af)#redistribute connected
PE1(conf-router-af)#end

PE1
Ser2/0
200.1.61.4/30

CE1

MPLS VPN Trouble

PE1#sh ip bgp vpn vrf v1 label | i 200.1.61.4


200.1.61.4/30
0.0.0.0
30/nolabel
PE1#
AS#1PE1#sh mpls forwarding vrf v1 | i 200.1.61.4
30
Aggregate
200.1.61.4/30[V] 0
MPLS Backbone
PE1#

Loop0:10.13.1.61/32

As soon as BGP gets the VPN prefix, it allocates the local label, and
installs the prefix+label in both BGP and LFIB

2004 Cisco Systems, Inc. All rights reserved.

10

MPLS VPN Ctrl PlaneTrouble #2


#2: LFIB doesnt have any label for the VPNv4 prefix
at the local PE, though BGP now does.
TIP: clear ip route vrf <vrf> <prefix>
If the above doesnt fix, then (soft) reset the BGP session

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

11

MPLS VPN Ctrl PlaneTrouble #3


#3: Remote PE (PE2) doesnt get the VPNv4 prefix from
PE1
!
RR1

PE1

AS#1
MPLS Backbone

Ser2/0
200.1.61.4/30

ip vrf v1
rd 1:1
route-target import 1:1

PE2
Loop0:10.13.1.62/32

CE-2

Loop0:10.13.1.61/32

PE2#sh ip bgp vpn vrf v1 200.1.61.4


% Network not in the table
PE2#
PE2#sh ip vrf de v1 | beg Import
No Import VPN route-target communities
No import route-map
No export route-map
PE2#

CE1

TIP: Validate route-target import config at PE2. If not present, then


configure it; Check for import-map as well

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

12

MPLS VPN Ctrl PlaneTrouble #4


#4: Remote PE (PE2) still doesnt get the VPNv4 prefix
from PE1
RR1

PE1

AS#1
MPLS Backbone

Ser2/0
200.1.61.4/30

!
ip vrf v1
rd 1:1
route-target import 1:1

PE2
Loop0:10.13.1.62/32

CE-2

Loop0:10.13.1.61/32

CE1

MPLS VPN Trouble

PE2#sh ip bgp vpn vrf v1 200.1.61.4


% Network not in the table
PE2#

We already fixed PE2; so lets go to PE1

Validate Route-target export in the VRF at the PE1

2004 Cisco Systems, Inc. All rights reserved.

13

MPLS VPN Ctrl PlaneTrouble #4 (Cont.)


PE1(conf)#ip vrf v1
PE1#sh ip bgp vpnv4 vpn vrf v1 200.1.61.4
PE1(conf-vrf)#route-target export 1:1 BGP routing table entry for 1:1:200.1.61.4/30, version 10
Paths: (2 available, best #2, table v1)
PE1(conf-vrf)#end

PE1

Ooops..RT Is Missing
Ser2/0

Advertised to non peer-group peers:


10.13.1.21 200.1.61.6
Local
0.0.0.0 from 0.0.0.0 (10.13.1.61)
Origin incomplete,
metric 0, localpref 100, weight
RR1
32768, valid, sourced, best
PE2
PE1#
AS#1

200.1.61.4/30

MPLS Backbone

Loop0:10.13.1.62/32

CE-2

Loop0:10.13.1.61/32

CE1

TIP: Configure Route-target export in the VRF on the local PE i.e. PE1

Lets make sure that RT is getting tagged to the VPNv4 prefix

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

14

MPLS VPN Ctrl PlaneTrouble #4 (Cont.)


PE1#sh ip bgp vpnv4
RR1 vpn vrf v1 200.1.61.4
BGP routing table entry for 1:1:200.1.61.4/30,
version 10
PE1
PE2
Paths: (2 available, best #2, table v1)
AS#1
Advertised to non peer-group peers:
Ser2/0
10.13.1.21
200.1.61.6
MPLS
Backbone
200.1.61.4/30
Loop0:10.13.1.61/32 Local
0.0.0.0 from 0.0.0.0 (10.13.1.61)
Origin incomplete, metric 0, localpref 100, weight
CE1
32768, valid, sourced, best
Extended Community: RT:1:1
RT is getting tagged
PE1#

Extra-TIP
If export or import map are also configured, then check the RT in set
clause, along with the match clause

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

15

MPLS VPN Ctrl PlaneTrouble#5


#5: Remote PE (PE2) STILL doesnt get the
VPNv4 prefix from PE1
RR1#sh ip bgp vpnv4 rd 1:1 200.1.61.4
BGP routing table entry for 1:1:200.1.61.4/30, version 14
Paths: (1 available, best
PE1
PE2#1, no table)
Advertised to non peer-group peers:
AS#1 10.13.1.62
CE-2
30
Local, (Received from a RR-client)
MPLS Backbone
.4/
1
Loop0:10.13.1.62/32
6
.
10.13.1.61 (metric 75) from 10.13.1.61 (10.13.1.61)
0.1
Loop0:10.13.1.61/32
20
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
RR1#

RR1

CE1

Looks Good on RR1

RR1 is indeed receiving the prefix from PE1


Make sure that RR is configured with neighbor <PE2> send-community
extended under vpnv4 address-family

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

16

MPLS VPN Ctrl PlaneTrouble #5 (Cont.)


Ooops. PE2 i.e
10.13.1.62 Is Missing
RR1(conf)#router bgp 1
RR1(conf-router)#address-family vpnv4
RR1(conf-router-af)#neighbor 10.13.1.62
send-community extended
RR1(conf-router-af)#neighbor 10.13.1.62
route-reflector-client
PE1
RR1(conf-router-af)#end
Ser2/0
200.1.61.4/30

RR1#sh run | inc send-community ext


neighbor 10.13.1.61 send-community extended
PE1#
RR1#sh run | inc send-community ext
neighbor 10.13.1.61 send-community extended
neighbor 10.13.1.62 send-community extended
PE1#
RR1

AS#1
MPLS Backbone

PE2

Loop0:10.13.1.62/32

CE-2

Loop0:10.13.1.61/32

CE1

MPLS VPN Trouble

TIP: All the MP-BGP peers must be configured with


send-community extended|both

Also make sure that PE1 and PE2 are configured as


route-reflector-client under vpnv4 af at the RR1

2004 Cisco Systems, Inc. All rights reserved.

17

MPLS VPN Control PlaneTrouble #6


#6: Remote PE (PE2) STILL doesnt get
the VPNv4 prefix from PE1

PE2#sh ip vrf detail v1 | i Import


Import route-map: raj-import
PE2#
PE2#sh route-map raj-import
RR1
route-map raj-import, permit,
PE1 sequence 10
PE2
Match clauses:
AS#1
extcommunity (extcommunity-list
filter):1
Ser2/0
Set clauses:
MPLS Backbone Loop0:10.13.1.62/32
Policy 200.1.61.4/30
routing matches: 0 packets, 0 bytes
Loop0:10.13.1.61/32
PE2#
PE2#sh
ip extcommunity-list 1
CE1
Extended community standard listPE2#sh
1
ip bgp vpn vrf v1 200.1.61.4
deny RT:1:1
% Network not in theOhtable
no.who did that
deny RT:2:2
PE2#
&^%@#%@^%
PE2#

CE-2

Thats ok. Lets


Remove RT 1:1
from the Filter.

Hmm we have already verified PE1 and RR1; something must be missing
on PE2 then
Lets check for any import-map at PE2 again

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

18

MPLS VPN Control PlaneTrouble #6 (Cont.)


PE2(conf)#no ip extcommunity-list 1 deny rt 1:1
RR1
PE2(conf)#end

PE1

Ser2/0
200.1.61.4/30

AS#1
MPLS Backbone

PE2
Loop0:10.13.1.62/32

CE-2

Loop0:10.13.1.61/32

CE1

PE#clear ip bgp * vpnv4 unicast in


PE2#sh ip bgp vpnv4 vrf v1 200.1.61.4
BGP routing table entry for 1:1:200.1.61.4/30, version 180
Paths: (1 available, best #1, table v1)
Advertised to non peer-group peers:
200.1.62.6
Local
10.13.1.61 (metric 75) from 10.13.1.21 (10.13.1.21)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
Originator: 10.13.1.61, Cluster list: 10.13.1.21
PE2#

TIP: If import-map is configured within the VRF, then import route-target


<rt> must be configured within the VRF for the relevant RT

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

19

MPLS VPN Control PlaneTrouble #7


#7: Label mismatch between BGP and FIB
PE2#sh ip bgp vpnv4 vrf v1 labels | i 200.1.61.4
200.1.61.4/30 10.13.1.61
nolabel/25
PE2#
RR1
PE2#sh ip cef vrf v1 200.1.61.4 PE1
200.1.61.4/30, version 64, epoch 0, cached adjacency toAS#1
Serial2/0
0 packets, 0 bytes
Ser2/0
tag information
set
MPLS Backbone
200.1.61.4/30
local tag: VPN-route-headLoop0:10.13.1.61/32
fast tag rewrite with Se2/0, point2point, tags imposed: {2003 20}
CE1
via 10.13.1.61,
0 dependencies, recursive
next hop 10.13.2.5, Serial2/0 via 10.13.1.61/32
valid cached adjacency
tag rewrite with Se2/0, point2point, tags imposed: {2003 20}
PE2#

PE2
Loop0:10.13.1.62/32

CE-2

Fix: clear ip route vrf <vrf> <prefix>.


If the mismatch doesnt go away, then debug ip bgp vpn and debug
mpls lfib cef to dig in.

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

20

MPLS VPN Control PlaneTrouble #8


#8: Remote PE receives the route, but
remote CE doesnt
PE1

AS#65000

Ser2/0

CE1

AS#1
MPLS Backbone

Loop0:10.13.1.61/32

PE2

router bgp 1
!
address-family ipv4 vrf v1
neighbor 200.1.62.6 as-override
exit-address-family
!

CE-2

AS#65000

Loop0:10.13.1.62/32

Loop0:5.5.5.5/32

TIP: If eBGP on PE-CE and VPN sites use the same ASN, then configure
as-override on the BGP VRF af on both PEs
If IGP on PE-CE, then validate BGP->IGP redistribution (within IGP VRF) on
the PE

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

21

MPLS VPN Control Plane


Show Commands on PE
1.

sh ip bgp vpn all summary


Analogous to sh ip bgp summary; Lists all the MPBGP and CE
peers

2.

sh ip bgp vpn all


Lists all the VPN prefixes advertised/rcvd by the router

3.

sh ip bgp vpn vrf <vrf> summary


Similar to the first one, but for a specific VRF

4.

sh ip bgp vpn vrf <vrf>


Lists all the VPN prefixes received in a specific VRF

5.

sh ip bgp vpn vrf <vrf> labels


List labels for the VPN prefixes in a VRF

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

22

MPLS VPN Control Plane


Show Commands on PE
If OSPF on PE-CE sh ip ospf neighbors
Lists both VPN(s) and non-VPN(s) OSPF neighbors

sh ip ospf <process-id>
Select the VRF associated process-id to see relevant OSPF info (a
lot of info)

sh ip ospf <process-id> database


Select the VRF associated process-id to see the OSPF database for
that VRF

clear ip ospf <process-id>


Clear OSPF neighbors in the VRF if VRF associated process-id is
chosen

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

23

MPLS VPN Control Plane


Show Commands on PE
If EIGRP on PE-CE
sh ip eigrp vrf <vrf> topology
Lists VRF specific EIGRP topology

sh ip eigrp vrf <vrf> neighbor|interface


Lists EIGRP neighbors or interfaces in the VRF

sh ip eigrp vrf <vrf> events


Shows VRF specific EIGRP events

clear ip eigrp vrf <vrf> neighbors


Clears VRF specific EIGRP neighbors

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

24

MPLS VPN Control Plane


Clear Commands on PE
Relevant towards RR (or remote PE) peers:
clear ip bgp * vpnv4 unicast in
Route-refresh request is sent to all the
MP-BGP peers

clear ip bgp <MP-BGP peer> vpnv4


unicast in
Route-refresh request is sent to a specific
MP-BGP peer

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

25

MPLS VPN Control Plane


Clear Commands on PE
Relevant towards CEs:
clear ip bgp * vrf < vrf >
Clear all PE-CE eBGP sessions in that vrf

clear ip bgp * vrf <vrf> in


Route-refresh message is sent to all the CEs in that vrf

clear ip bgp * vrf < vrf > out


Send respective VPN routes to all the CEs in that vrf

clear ip bgp <CE> vrf < vrf > soft in|out


soft reset of BGP session

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

26

MPLS VPN Control Plane


Show Commands on RR

Route-reflector know nothing about VRF


Following commands come quite handy (especially on RR)

1.

sh ip bgp vpn all

2.

sh ip bgp vpn rd <RD>


Lists all VPNv4 prefixes that have RD in them

3.

sh ip bgp vpn rd <RD> label


Lists labels for VPNv4 prefixes that have RD

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

27

MPLS VPN Control Plane


Debugs on PE
Be Careful on the Production Routers
1.

debug ip bgp vpnv4

Useful while troubleshooting label related


problems in BGP (could spit a lot of output)

2.

debug mpls lfib cef [acl]


Useful troubleshooting label mismatch in FIB/LFIB

3.

debug ip bgp vpnv4 import


Useful when VPN prefixes dont get imported in the VRF table
(could spit a lot of output)

4.

debug ip routing vrf <vrf> [acl]


Useful when VPN prefixes dont get installed in
the VRF routing table

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

28

Agenda
Control Plane
Control Plane Troubleshooting Tips
Real-life Examples
Summary of Helpful Cisco IOS Commands

Forwarding Plane
Dissecting LFIB
Loadsharing in MPLS VPN Networks
Forwarding Plane Troubleshooting Tips
Real-life Examples
Summary of Helpful Cisco IOS Commands

Conclusion
MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

29

MPLS VPN Forwarding Plane


Dissecting LFIB: show mpls forward
IP (or IGP) Prefix in the LFIB
RSP-PE-WEST-4#sh mpls forward 10.13.1.11 detail
Local Outgoing
Prefix
Bytes tag Outgoing
tag
tag or VC
or Tunnel Id
switched
interface
45
51
10.13.1.11/32
0
Fa1/1/1
MAC/Encaps=14/18, MRU=1500, Tag Stack{51}
0003FD1C828100044E7548298847 00033000
No output feature configured
Per-packet load-sharing
RSP-PE-WEST-4#

MRUMax Receivable Unit; The


Received Packet Will Be Transmitted
Unfragmented on Fa1/1/1, If Received
Packets Size Is Not More Than 1500B

Next Hop
10.13.7.33

Only One
Outgoing Label in
the Label Stack

MAC header = 0003FD1C828100044E754829


MPLS Ethertype = 0x8847
Label
= 0x00033000 = 51
0x00033000 = EXP+S
0x00033000 = MPLS TTL

Although MAC Header Is of 14 Bytes, Actual Encapsulation


I.E. MAC+MPLS Header Is of 18 Bytes (One Label Is 4 Bytes)
MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

30

MPLS VPN Forwarding Plane


Dissecting LFIB: show mpls forward (Cont.)
PE1

VPN Prefix in the LFIB

P1

PE2

CE1
5.5.5.5/32

PE1#sh
Local
tag
27

mpls for vrf v1 5.5.5.5 detail


Outgoing
Prefix
Bytes tag
tag or VC
or Tunnel Id
switched
Untagged
5.5.5.5/32[V]
0
MAC/Encaps=0/0, MRU=1504, Tag Stack{}
VPN route: v1
No output feature configured
Per-packet load-sharing
PE1#

Outgoing
interface
Se2/0

Next Hop
point2point

Se2/0 Is a PE-CE
Interface which
Is under VRF v1

Only 1504 Byte Size Packet Can Be Received because


15044 (for One Label 27) = 1500 Is the MTU Size of Se2/0
MAC/Encaps Field Corresponds to the tag adj, and
because the VRF Interface Doesnt Typically Have MPLS
Enabled, tag adj Is 0; hence, 0/0 Output

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

31

Agenda
Control Plane
Control Plane Troubleshooting Tips
Real-life Examples
Summary of Helpful Cisco IOS Commands

Forwarding Plane
Dissecting LFIB
Load sharing in MPLS VPN Networks
Forwarding Plane Troubleshooting Tips
Real-life Examples
Summary of Helpful Cisco IOS Commands

Conclusion
MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

32

MPLS VPN Fwd PlaneLoadsharing


Loadsharing in MPLS VPN network is same as that of
the IP network
i.e. FIB per-source-destination loadsharing

IP src and dest addresses inside the MPLS packet are


hashed to find the right LSP

Lets Go through PE-P and P-P Loadsharing


MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

33

MPLS VPN Fwd PlaneLoadsharing (I)


PE-P Loadsharing (Cont.)
PE1#sh ip cef vrf v1 200.1.62.4
200.1.62.4/30, version 13, epoch 0, per-destination sharing
0 packets, 0 bytes
tag information set
local tag: VPN-route-head
fast tag rewrite with
Recursive rewrite via 10.13.1.62/32, tags imposed {25}
via 10.13.1.62, 0 dependencies, recursive
next hop 10.13.1.9, Ethernet1/0 via 10.13.1.62/32
valid adjacency
tag rewrite with
Recursive rewrite via 10.13.1.62/32, tags imposed {25}
Recursive load sharing using 10.13.1.62/32.
PE1#

Because There Are Loadshared Paths


to the Egress PE i.e. 10.13.1.62/32

PE1

E0/0

E1/0

P1
Se2/0
Loop0:10.13.1.62/32
PE2

Only VPN Label Is Shown

Dont panicIGP label is chosen during the


forwarding (depending on the hash-bucket)
MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

34

MPLS VPN Fwd PlaneLoadsharing (I)


PE-P Loadsharing (Cont.)
PE1#sh ip cef 10.13.1.62
10.13.1.62/32, version 30, epoch 0, per-destination sharing
0 packets, 0 bytes
tag information set, shared
local tag: 18
via 10.13.1.5, Ethernet0/0, 1 dependency
traffic share 1
next hop 10.13.1.5, Ethernet0/0
valid adjacency
tag rewrite with Et0/0, 10.13.1.5, tags imposed: {2001}
via 10.13.1.9, Ethernet1/0, 1 dependency
traffic share 1
next hop 10.13.1.9, Ethernet1/0
valid adjacency
tag rewrite with Et1/0, 10.13.1.9, tags imposed: {2001}
0 packets, 0 bytes switched through the prefix
tmstats: external 0 packets, 0 bytes
internal 0 packets, 0 bytes
PE1#

CE1

30.1.61.4/30

PE1

E0/0

E1/0

P1
Se2/0
Loop0:10.13.1.62/32
CE2

PE2
200.1.61.4/30

IGP Label Is Right Here

IGP Label and the outgoing interface are derived


after the hash-bucket is decided
MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

35

MPLS VPN Fwd PlaneLoadsharing (I)


PE-P Loadsharing (cont.)

CE1

30.1.61.4/30

PE1

PE1#sh ip cef vrf v1 exact-route 30.1.61.4 200.1.62.4 internal


30.1.61.4
-> 200.1.62.4
: Ethernet1/0 (next hop 10.13.1.9)
Bucket 7 from 16, total 2 paths

E0/0

E1/0

P1
Se2/0
Loop0:10.13.1.62/32
CE2

PE2
200.1.61.4/30

In summary, the show-output in load-sharing case


gets bit tricky; but the fundamental is the same

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

36

MPLS Fwd PlaneLoadsharing (II)


PE1

P-P Loadsharing
P1#sh mpls for 10.13.1.62
Local Outgoing
Prefix
tag
tag or VC
or Tunnel Id
52
21
10.13.1.62/32
27
10.13.1.62/32
P1#

Bytes tag
switched
0
0

Outgoing
interface
Eth0/0
Eth1/0

Next Hop
point2point
point2point

For VPN traffic, P router hashes the IP


src+dest to apply the packet to the correct
hash bucket

P1
E0/0

E1/0

P2

P3

Se2/0
Loop0:10.13.1.62/32
PE2

sh ip cef exact-route command cant be used on the P


router since it doesnt know the VPN addresses
Hence, rely on (LFIB) counters to make sure the traffic is getting
loadshared

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

37

Agenda MPLSVPN Troubleshooting


Control Plane
Control Plane Troubleshooting Tips
Real-life Examples
Summary of Helpful Cisco IOS Commands

Forwarding Plane
Dissecting LFIB
Loadsharing in MPLS VPN Networks
Forwarding Plane Troubleshooting Tips
Real-life Examples
Summary of Helpful Cisco IOS Commands

Conclusion
MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

38

MPLS VPN Forwarding Plane


Troubleshooting Tips
Symptom

Tip

Cisco IOS
Command

CE CE Traffic Fails

Verify That the


PE PE VPN Traffic Can Pass Using
vrf Pings (Assuming the Control Plane
Information Has Already Been Verified*)

PE PE MPLS
Traffic Fails

Validate the PE->PE IP Connectivity;


and then Check
for the LSP

PE#ping <remotePE>

PE PE IP Traffic Passes, but


MPLS
Traffic Fails

Find out where Exactly the LSP Is


Broken

PE#ping mpls ipv4


<remotePE> **

Incoming MPLS Traffic


Is Dropped at the
Egress PE

Check the LFIB Entries on Both RP, LC


(and Relevant HW Engines, if Present)

PE#sh mpls for | i


<prefix>

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

PE#ping vrf <vrf>


<remote prefix>

39

MPLS VPN Forwarding Plane


Troubleshooting Tips (Cont.)
Symptom

Tip

CE CE Traffic Fails for


Certain MTU Sizes

Check the MPLS MTU Size of the MPLS


Enabled Interfaces and Make Sure It Is
More than the Reported Failed MTU Size

CE CE Traffic Fails for


Certain MTU Sizes

Verify that the Ethernet Switch Ports


inside the MPLS Core Is Enabled with
Baby Giant Support

Cisco IOS
Command
Router#sh mpls int
de | in MTU

Switch#sh port jumbo

*Please See the Troubleshooting Control Plane TechTalk;


**12.0(26)S Onwards
MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

40

MPLS VPN Forwarding Plane


Troubleshooting Steps
The VPN Traffic Outage Has Been Reported to You:
Step 1:
Step 2:

Step 3:
Step 4:
Step 5:
Step 6:
Step 7:

Step 8:

MPLS VPN Trouble

First, verify VRF ping from PE1 to PE2 (interface addresses)


If passed, then either CE PE or PE CE may be the problem =>
not a MPLS core problem; STOP and Check whether the packets
are getting dropped by ingress LC on PE
If failed, then MPLS core may be the problem; PROCEED
Ping ingress PE to egress PE to verify the IP reachability
If failed, then STOP and verify egress PEs route hop-by-hop
If passed, then traceroute PE1 PE2 and PE2 PE1 to ensure
the PE-to-PE LSP setup
Also check for the labels in each line of the traceroute output; if
traceroute fails for some reason, then STOP and verify the label at
each hop
If good, then the problem may be specific to the HW on either PE
or P routers; find out that whether the HW is dropping the packets
by looking at the interface counters by sh int <int>
2004 Cisco Systems, Inc. All rights reserved.

41

MPLS VPN Fwd PlaneTroubleshotting


Lets do some more trouble(shooting)

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

42

MPLS VPN Fwd Plane - Troubleshooting


Problem: A VPN Site Cant Reach Other Sites
Cause: The CE CE Traffic Is Getting Dropped Somewhere
PE1
200.1.61.4/30

P1
E0/0
E1/0

Ser2/0

MPLS Backbone

Loop0:10.13.1.61/32

200.1.62.4/30

Loop0:10.13.1.62/32

CE1

CE2

Tip 1: Check the control plane information first

FIB

PE2

LFIB

PE1#sh ip cef vrf v1 200.1.62.4;

PE1#sh mpls for vrf v1 | inc 200.1.61.4

PE2#sh ip cef vrf v1 200.1.61.4;

PE2#sh mpls for vrf v1 | inc 200.1.62.4

Make sure that the label information is correct

Turn on deb ip icmp on both PEs


Step 1: Issue ping vrf v1 <remote_PE-CE_address> on both PEs
Step 2: If they pass, then we have verified that the problem is not in the
MPLS core
2004 Cisco Systems, Inc. All rights reserved.

MPLS VPN Fwd Plane - Troubleshooting


200.1.61.4/30
Ser2/0

CE1
5.5.5.5/32

PE1

P1

E0/0
E1/0

Ser2/0

PE2

200.1.62.4/30

MPLS Backbone

Loop0:10.13.1.61/32

PE1#sh ip cef vrf v1 200.1.62.4


200.1.62.4/30, version 10, epoch 0, per-destination
sharing
0 packets, 0 bytes
tag information set
local tag: VPN-route-head
fast tag rewrite with
Recursive rewrite via 10.13.1.62/32, tags imposed
{25}
via 10.13.1.62, 0 dependencies, recursive
next hop 10.13.1.9, Ethernet1/0 via 10.13.1.62/32
valid adjacency
tag rewrite with
Recursive rewrite via 10.13.1.62/32, tags imposed
{25}
Recursive load sharing using 10.13.1.62/32.
PE1#

Validated the Labels


in PE1->PE2 Direction
2004 Cisco Systems, Inc. All rights reserved.

Loop0:10.13.1.62/32

CE-2
6.6.6.6/32

PE2#sh mpls for vrf v1 | inc 200.1.62.4


25 Aggregate 200.1.62.4/30[V] 0
PE2#
PE1#sh ip cef 10.13.1.62
10.13.1.62/32, version 56, epoch 0, per-destination sharing
0 packets, 0 bytes
tag information set
local tag: 18
via 10.13.1.5, Ethernet0/0, 1 dependency
traffic share 1
next hop 10.13.1.5, Ethernet0/0
valid adjacency
tag rewrite with Et0/0, 10.13.1.5, tags imposed: {2001}
via 10.13.1.9, Ethernet1/0, 2 dependencies
traffic share 1
next hop 10.13.1.9, Ethernet1/0
valid adjacency
tag rewrite with Et1/0, 10.13.1.9, tags imposed: {2001}
0 packets, 0 bytes switched through the prefix
PE1#

MPLS VPN Fwd Plane - Troubleshooting


200.1.61.4/30
Ser2/0

CE1
5.5.5.5/32

PE1

P1

E0/0
E1/0

Ser2/0

PE2

200.1.62.4/30

MPLS Backbone

Loop0:10.13.1.61/32

PE1#sh mpls for vrf v1 | i 200.1.61.4


28 Aggregate 200.1.61.4/30[V] 0
PE1#

Loop0:10.13.1.62/32

6.6.6.6/32

PE2#sh ip cef vrf v1 200.1.61.4


200.1.61.4/30, version 73, epoch 0, cached adjacency to
Serial2/0
0 packets, 0 bytes
tag information set
local tag: VPN-route-head
fast tag rewrite with Se2/0, point2point, tags imposed:
{2003 28}
via 10.13.1.61, 0 dependencies, recursive
next hop 10.13.2.5, Serial2/0 via 10.13.1.61/32
valid cached adjacency
tag rewrite with Se2/0, point2point, tags imposed:
{2003 28}
PE2#

Validated the Labels in PE2 PE1 Direction


2004 Cisco Systems, Inc. All rights reserved.

CE-2

MPLS VPN Fwd Plane - Troubleshooting


200.1.61.4/30
Ser2/0

CE1
5.5.5.5/32

PE1

P1

E0/0
E1/0

Ser2/0

PE2

200.1.62.4/30

MPLS Backbone

Loop0:10.13.1.61/32

Loop0:10.13.1.62/32

PE1#deb ip icmp
ICMP packet debugging is on
PE1#
Step 1 PE1#ping vrf v1 200.1.62.5
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.1.61.5, timeout is
2 seconds:
.....
Success rate is 0 percent (0/5)
PE1#

CE-2
6.6.6.6/32

PE2#deb ip icmp
ICMP packet debugging is on
PE2#
PE2#
*May 11 00:42:16.353: ICMP: echo reply sent, src
200.1.62.5, dst 200.1.61.5
*May 11 00:42:16.473: ICMP: echo reply sent, src
200.1.62.5, dst 200.1.61.5
*May 11 00:42:16.581: ICMP: echo reply sent, src
200.1.62.5, dst 200.1.61.5
*May 11 00:42:16.701: ICMP: echo reply sent, src
200.1.62.5, dst 200.1.61.5
*May 11 00:42:16.813: ICMP: echo reply sent, src
200.1.62.5, dst 200.1.61.5
PE2#

Step 3: Okalthough the vrf pings failed at PE1, ICMP debugs at PE2
confirms that PE1->PE2 LSP is error free
Lets ping in the opposite direction to check the PE2 PE1 LSP
2004 Cisco Systems, Inc. All rights reserved.

MPLS VPN Fwd Plane - Troubleshooting


200.1.61.4/30
Ser2/0

CE1
5.5.5.5/32

PE1

P1

E0/0
E1/0

Ser2/0

Loop0:10.13.1.61/32

Loop0:10.13.1.62/32

Since PE1 didnt get/show any


ICMP echos for the vrf pings
Either PE2 PE1 LSP is broken

Or PE1 is dropping the received MPLS packets for


some reason

Okso lets troubleshoot for (a) first


2004 Cisco Systems, Inc. All rights reserved.

CE-2
6.6.6.6/32

PE2#deb ip icmp
ICMP packet debugging is on
PE2#
PE2#ping vrf v1 200.1.61.5
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.1.61.5, timeout is
2 seconds:
.....
Success rate is 0 percent (0/5)
PE2#

b) Or PE1 doesnt have the LFIB entry for 200.1.61.5


c)

200.1.62.4/30

MPLS Backbone

PE1#deb ip icmp
ICMP packet debugging is on
PE1#
PE1#
PE1#

a)

PE2

We Already
Verified This
Earlier

MPLS VPN Fwd Plane - Troubleshooting


200.1.61.4/30
Ser2/0

CE1
5.5.5.5/32

PE1

P1

E0/0
E1/0

Ser2/0

PE2

200.1.62.4/30

MPLS Backbone

Loop0:10.13.1.61/32

Loop0:10.13.1.62/32

Step 4 PE1#ping 10.13.1.62


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.13.1.62,
timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip
min/avg/max = 40/57/92 ms
PE1#
P1#sh mpls forward
Local Outgoing
tag
tag or VC
2003
Untagged
Untagged
P1#

CE-2
6.6.6.6/32

PE2#ping 10.13.1.61
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.13.1.61,
timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip
min/avg/max = 28/52/72 ms
PE2#

10.13.1.61
Prefix
or Tunnel Id
10.13.1.61/32
10.13.1.61/32

Bytes tag
switched
0
0

Outgoing
interface
Et0/0
Et1/0

Step 7: IP reachability is confirmed between PE1 and PE2 (steps 1 and 2);
GOOD; but that doesnt validate the LSP in both directions
Step 7: Per P1s LFIB, it doesnt have the right label to reach PE1 (untagged vs. Pop).
2004 Cisco Systems, Inc. All rights reserved.

Next Hop
10.13.1.6
10.13.1.10

MPLS VPN Fwd Plane - Troubleshooting


Rememberuntagged outgoing label means that
get rid of the label stack; hence, the VPN label would
be lost at P1
untagged label for /32 routes inside the MPLS core
is almost always bad
To fix this untagged problem in LFIB,
Check whether LIB and LFIB are in-sync about this entry.
If not, then clear ip route 10.13.1.61 on P1
If yes, then flap the LDP neighbor by clear mpls ldp
neighbor 10.13.1.61 on P1 to relearn the correct binding
** See more Debugs at the Show commands section
2004 Cisco Systems, Inc. All rights reserved.

MPLS VPN Fwd Plane - Troubleshooting


Till now, there wasnt any clean way of DETECTING
THE BROKEN LSPS
LSP pings* can DETECT the broken LSPs
LSP traceroute* can PIN-POINT the culprit router
where the LSP could be broken
But we will still have to fix the LSP
LSP ping would simply replace steps 6 and 7
in the troubleshooting steps

*12.0(26)S Onwards
2004 Cisco Systems, Inc. All rights reserved.

MPLS VPN Forwarding Plane -Trouble #1 (Cont.)


PE1#ping mpls ipv4 10.13.1.62/32
Sending 5, 100-byte MPLS Echos to 10.13.1.62/32,
timeout is 2 seconds, send interval is 0 msec:
Codes: '!' - success, 'Q' - request not transmitted,
'.' - timeout, 'U' - unreachable,
'R' - downstream router but not target

LSP Ping Failed

Type escape sequence to abort.


RRRRR
Success rate is 0 percent (0/5)
PE1#

PE1#ping mpls ipv4 10.13.1.62/32


Sending 5, 100-byte MPLS Echos to 10.13.1.62/32,
timeout is 2 seconds, send interval is 0 msec:
Codes: '!' - success, 'Q' - request not transmitted,
'.' - timeout, 'U' - unreachable,
'R' - downstream router but not target

LSP Ping
Succeeded

Type escape sequence to abort.


!!!!!
Success rate is 0 percent (0/5)
PE1#

2004 Cisco Systems, Inc. All rights reserved.

MPLS VPN Forwarding Plane -Trouble #1 (Cont.)


LSP traceroute* is capable of differentiating between
untagged and pop/null
PE1#trace mpls ipv4 10.13.1.62/32
Tracing MPLS Label Switched Path to 10.13.1.62/32, timeout is 2 seconds
Codes: '!' - success, 'Q' - request not transmitted,
'.' - timeout, 'U' - unreachable,
'R' - downstream router but not target
Type escape sequence to abort.
0 10.13.1.10 MRU 1500 [Labels: 2002 Exp: 0]
R 1 10.13.2.14 MRU 1204 [No Label] 52 ms
! 2 10.13.2.14 52 ms
PE1#
PE1#trace mpls ipv4 10.13.1.62/32
Tracing MPLS Label Switched Path to 10.13.1.62/32, timeout is 2 seconds
Codes: '!' - success, 'Q' - request not transmitted,
'.' - timeout, 'U' - unreachable,
'R' - downstream router but not target
Type escape sequence to abort.
0 10.13.1.10 MRU 1500 [Labels: 2002 Exp: 0]
R 1 10.13.2.13 MRU 1512 [implicit-null] 40 ms
! 2 10.13.2.14 68 ms
PE1#

*12.0(26)S Onwards
2004 Cisco Systems, Inc. All rights reserved.

Agenda MPLSVPN Troubleshooting


Control Plane
Control Plane Troubleshooting Tips
Real-life Examples
Summary of Helpful Cisco IOS Commands

Forwarding Plane
Dissecting LFIB
Loadsharing in MPLS VPN Networks
Forwarding Plane Troubleshooting Tips
Real-life Examples
Summary of Helpful Cisco IOS Commands

Conclusion
MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

53

MPLS VPN Fwd PlaneShow Commands


sh mpls forwarding
Shows all LFIB entries (vpn, non-vpn, TE etc.)

sh mpls forwarding | inc <prefix>


Whether the prefix is present in the LFIB or not

sh mpls forwarding vrf <vrf> <prefix>


LFIB lookup based on a VPN prefix

sh mpls forwarding label <label>


LFIB lookup based on an incoming label

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

54

MPLS VPN Fwd PlaneShow Commands

sh ip arp vrf <vrf>


Lists ARP entries relevant to the <vrf> only

sh ip cef vrf <vrf > <prefix>


Displays the label stack, outgoing interface etc

sh mpls forwarding vrf <vrf>


Lists labels for the VPN prefixes learned from the CE(s)

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

55

MPLS VPN Fwd PlaneDebugs


Be Careful on the Production Routers
debug arp
Useful for VPN prefixes as well

debug mpls lfib cef [acl]


Useful when VPN prefixes have label mismatch among
BGP, FIB and LFIB.

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

56

Conclusion
MPLS seems cryptic, but it is not
Whether to look at FIB or LFIB?
Whether it is a BGP or MPLS problem?
Whether the problem is within the core or outside
the core?
Ongoing MPLS OAM work .

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

57

RST-3061
8186_05_2003_c1

2003, Cisco Systems, Inc. All rights reserved.

58