Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 1/89
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 2/89
Table of Contents
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 3/89
Section 2.3 Logging a Packet with a Filter ............................................................ 2-28
Objective: ...................................................................................................................... 2-28
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 4/89
Exercise: ........................................................................................................................ 7-44
Section 7.3 ................................................................................................................. 7-44
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 5/89
A.8.1. Commands common to any RSVP troubleshooting ......................................... 9-74
A.8.2. RSVP MPLS Tools Perform Commands ......................................................... 9-77
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 6/89
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Table of Figures
Table of Tables
Table 1-1: Router remote access addresses ..................................................................... 1-22
Table 1-2: Lab 1 command list ........................................................................................ 1-22
Table 2-1: Lab 2 command list ........................................................................................ 2-27
Table 3-1: Lab 3 configuration commands ...................................................................... 3-31
Table 5-1: IS-IS Commands ............................................................................................ 5-37
Table 8-1: Command List ................................................................................................ 8-50
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 7/89
Troubleshooting Methodology
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
An effective troubleshooting methodology will use layered network architecture as leverage towards
finding a problem, or at least isolating the problem. Walking up through the protocol stack in an organized,
methodological approach is key to fault determination and isolation.
Services Context
Customer Notified / Testing of
Resolution / Signoff
Qualitative issues are somewhat more difficult to troubleshoot and
isolate since they often deal with, at least from the customers
Ticket Closed perspective, intangiblesthings that are often subject and hard to
quantify. One of the major strengths of the Alcatel-Lucent Services
Routers and Ethernet Services Switches is that they have a rich suite
of tools that can be used to apply metrics to quality of service issues.
Because of the Alcatel-Lucent model of building services, (or service
Stop
tunnels) across an IP/MPLS core and the subsequent ability to
segregate customer traffic and manage it in discrete flows, it is
relatively straightforward to troubleshoot QoS issues. SAA, OA&M,
show, and debug commands can be used both in real-time, as well as
for statistical reporting afterwards.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 8/89
switches. Troubleshooting up the protocol stack is the preferred approach but it should be
viewed in the context of not only the underlying switched and IP/MPLS infrastructure but
equally importantly in the context of services.
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
A good troubleshooting approach understands that fault isolation is the single most important first step
towards fault determination then eventually fault resolution. The Alcatel-Lucent services model formalizes
the natural demarcations in a network with network and access ports.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 9/89
The OSI Model A Layered Approach to Network
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 10/89
The Physical Layer
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
General
Troubleshooting Layer 1 can be relatively straightforward. This is the Is it plugged in? layer. In
summary, the physical layer is responsible for bit-encoding and physical cabling topology. It defines how
binary 1s and 0s are represented on the wire and defines the physical interface that devices have with the
cabling topology. For example, Ethernet uses Manchester binary encoding at layer 1 and uses protocols
such as the nWay protocol to negotiate speed and duplex of nodes connecting to the cabling plant.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 11/89
The Data Link Layer
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
General
The Data Link layer frames / organizes the binary 1s and 0s into frames, labeling each frame with an
address field. For multi-access links such as Ethernet and FDDI, the frame labels the data with both a
source and destination address since the underlying physical topology might be shared media. In point-to-
point, non-shared data link configurations such as serial links, a single address field is required since the
data link itself is essentially being identified. These point-to-point data link configurations, often called
virtual circuits, are typical of packet-switched technologies such as ATM, Frame Relay and X.25.
Services that create a pseudo layer 2 are called VLL (Virtual Link Layer) services. ePipe, aPipe and fPipe
create pseudo point-to-point data link configurations. VPLS creates a pseudo multi-access data link
configuration. Troubleshooting these services is discussed briefly in their separate contexts below.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 12/89
Alcatel-Lucent Services Model Context
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
QoS solutions will often use information in the layer 2 PDU for classification and marking decisions. For
example source/destination SAPs (IEEE service access points), 801.q VLAN ID, 802.1p priority bits and
source/destination MAC addresses might be used to either segregate traffic into different forwarding
classes or to immediately dispatch high-priority traffic to LLQs. Problems often arise if the QoS solution
doesnt properly classify the customers traffic or else properly respect the layer 2 markings which indicate
priority.
VLL Services
PE B
PE A PE C
IP / MPLS
ePipe Network
service
PE D
Some Alcatel-Lucent services mimic or emulate layer 2 of the OSI model. For example, VLLs such as
aPipe, ePipe, and fPipe services create logical vs. physical point-to-point data links between peer devices.
The service appears as a single data link or wire between the peer devices. Customer devices (represented
as buildings in the diagram above) that use the service must be on the same IP subnet as a result. Many
problems with VLL type services can be rectified by visualizing the logical layout of the network.
Knowledge of framing, basic organization of the layer 2 PDU including 802.1p, Q-in-Q, 802.1q, as well as
STP are crucial.
The aPipe service uses ATM for transport. Customers traffic is mapped to ATM VCs, creating a VLL
mapped directly to an ATM VC.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 13/89
VPLS Service
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
PE B
VPLS Service
IP/LSP Full-
Mesh
PE A PE C
IP / MPLS
Network
PE D
The VPLS service extends the customers switched network across the IP/MPLS core. It provides for a
layer 2 class of VPN solution. The IP/MPLS core appears as a logical switch (pictured as a solid rectangle
in the above picture) to the customers equipment. As with VLL services, a thorough knowledge of
trunking, STP and layer 2 prioritization would be useful. For example, if IEEE 802.2 LLC frames are
switched across the network, the network may experience a higher load that Ethernet II frames since the
former is connection-oriented. As such retransmission and basic sequencing and acknowledgement may
create timing issues for the customers traffic if a proper QoS solution where this traffic is not given higher
priority than other flows is not implemented.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 14/89
The Network Layer
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
General
According to RFC 791, The Internet Protocol is responsible for providing addressing of the logical data
link between TCP/IP end systems. While it does not, itself, create or manage the logical data link (this is
the responsibility of the Transport Layer), it is responsible for packaging and fragmenting/reassembling
customer data for delivery by intermediate systems (routers) and providing enough information such that
routers can make informed decisions as to where next to transmit the data on a hop-by-hop basis. QoS
solutions can modify this PHB (per-hop behaviour) by respecting the markings in the ToS byte of the IP
header which indicate the requirement for differentiated services in the IP network. Traffic can be marked,
classified and forwarded based on other information found in the IP header including IP version number,
source/destination address, protocol number, etc. Knowledge of the organization of an IP packet, IP
addressing in general and IP address planning / subnetting in general is crucial. Furthermore, access lists
can be created to filter and block traffic based on just about any piece of information in the IP packet
header.
Dynamic routing protocols, both IGPs and EGPs, can take advantage of good subnetting and route
summarization techniquescareful IP address planning to reduce routing table sizes and the complexity
of the forwarding of IP datagrams throughout an IP network. Basic, common sense principles of 1 subnet =
1 wire (pseudo or physical) must be adhered to. Many troubleshooting exercises begin and end with
careful analysis of the per-hop behaviour of traffic through an IP network. Given the network layers
responsibility to address the logical endpoints for TCP and UDP sessions (as well as stateless, tunnelled
traffic such as GRE and IPSec) and routers responsibility to find the best path between the endpoints, it is
hard to overstate its importance.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 15/89
Alcatel-Lucent Services Model Context
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
PE A PE C
RI-1 RI-1
RI-2 RI-2
IP / MPLS
Network
VPRN
Service
RI-1 Green
PE D
RI-2
VPRN is a class of VPN that allows the connection of multiple sites in a routed domain over a provider
managed IP/MPLS network. From the customers perspective it looks like all sites are connected to a
private routed network administered by the service provider for that customer only. The Service provider,
however, can reuse the IP/MPLS infrastructure to offer multiple services.
Each VPRN appears like an additional routing instance, routes for a service between the various PEs are
exchanged using MP-BGP. From a troubleshooting perspective this is critical because each service
contains a separate instance of routed traffic, segregating the customers routed traffic and allowing traffic
conditioning, shaping, modified PHBs and other QoS policies to be managed individually. More
importantly, unless there is some catastrophic, universal problem in the IP/MPLS core, each customers
traffic can be troubleshooted separately too. This is the P for Private in VPRN. Effectively the
customers traffic flows in virtual, private, managed layer 3 domains in the same way the with the VPLS
service the customers traffic flows in virtual, private, managed layer 2 domains.
Since the SDPs tie into the providers IP/MPLS core and are signalled within the context of IP tunnels or
MPLS/RSVP, they are sensitive to bad design and other faults in the underlying layer 3 infrastructure.
Also, another troubleshooting nexus is the use of the customers own IP address structure, often using their
own RFC 1918 addresses when connecting to sites through the service providers service tunnels. This
often means that troubleshooting is done on two planes, the service provider network and the customers
virtual reality network.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 16/89
IES Service
Internet
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Company C
PE C CE C
PE A
CE A Service Provider
Network
Company A PE B
CE B
Company B
From the customers perspective the IES provides a direct connection to the Internet. The Service provider
can apply all billing, ingress/egress shaping and policing to the customer. Unlike the VPRN service, there
is no separation at the network layer for the customer and the service provider. This coupling of customer
traffic to the service providers network may make troubleshooting more problematic in some
circumstances as natural demarcation points may be hard to find at the network layer. However, using
hierarchical routing protocols, good route summarization techniques and redistribution between the access
layer and the edge may aid in fault isolation.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 17/89
The Transport Layer
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
General
In an IP network there are two transport layer protocols, TCP and UDP. TCP is connection-oriented and
UDP is connectionless. TCP and UDP share source and destination port numbers that can be used as
discriminators for a variety of filtering and QoS decisions. These port numbers are associated with and
identify application layer protocols on the server side of the connection such as Telnet, FTP, SMTP, etc. In
addition, because TCP is connection oriented, it has additional fields that manage the state of the
connection (the control field) as well as sequencing (sequence number and acknowledgement number) as
well as receiving windows size, etc. As a consequence, TCP has more overhead per unit data than UDP.
From a troubleshooting standpoint, what makes TCP somewhat problematic is that the devices inside the
service providers network tend to be ignorant of the TCP session parameters since this is largely the
responsibility of the endpoints of the TCP connectionthe customers devices. While these TCP
parameters may used for input classification decisions, for example, as part of a QoS solution, congestion
issues in the service providers network can be exacerbated by the retransmission of data segments and
other spurious signalling initiated by the customer end systems, causing a cascading effect in extreme
situations. Furthermore, customers applications may time out if there is not sufficient priority given to
TCP connection setup in the QoS solution offered by the service provider.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 18/89
The Application Layer (TCP/IP = Layers 5, 6, and 7 of the OSI Model)
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
General
Layers 5, 6, and 7 of the OSI model are often represented as a layer 5 of the TCP/IP stack. Since there is
no RFC that defines an application layer nor for that matter a layered network architecture, this
simplification is debatable and can often lead to errors when troubleshooting a network. For example, layer
5 of the OSI model, the session layer, is a management layer that indicated to the transport layer what level
or type of service is required for the transport of application layer protocols.
The requirement for encryption at the transport layer, connection-oriented vs. connectionless transport, etc.,
is indicated by the session layer. While, some may view this as an unnecessary distinction in
troubleshooting a service-oriented solution which does not classify traffic based on any information
above layer 4.knowledge of timers, synchronization and other application layer behaviours will be useful
in conceptualizing problems. For example, knowing how a VoIP call, is setup, calls progress and then
sessions are torn down might prove useful.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 19/89
While classification and differentiated services for an application layer protocol occur at a lower layer of
the OSI model in the Alcatel-Lucent services model, the symptoms of inadequate network provisioning and
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
outages will be seen first at the application layer. This, of course, is what prompts our customers to call us
in the first place.
As was indicated earlier, as we progress up through the layered network protocol stack, more specialized
knowledge of the protocols is required.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 20/89
Lab 1 Network discovery and backing up images
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Objective:
Verify the operation and physical connectivity of the routers, which are connected according to the
following diagram.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 21/89
Pod Number Router Name Connect Address
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Syntax:
Commands required for this exercise are found in Table 1-2. Detail may be found in Module 1, IGP
Review. Each command may have additional parameters possible. Use the ? character for help and to
explore all command line options. Other commands may also be used, including those from previous
courses.
Exercise:
1. Connect to the routers in your Pod using the addresses provided by your instructor. Fill in the
required fields for Table 1. The username and password for all devices is admin. If you are
unable to connect to any of the routers, notify your instructor.
2. Verify the router has the initial configuration uploaded.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 22/89
Verification:
1. Determine if you can connect to your routers
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 23/89
Section 1.2 - IP Connectivity and IGP Configuration
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Objective:
Identify the IP addressing and routing protocol for your pods topology.
Exercise:
Determine the IP addressing and pod configuration for your assigned pod.
1. Identify the naming convention for each router in your pod
2. Document the IP addressing and subnet masking on each interface of each router and fill in the
diagram above. This addressing will not change throughout the entire course. This diagram can
be used as reference for the labs that follow.
3. Ensure network connectivity by use of Ping, and ssh
4. Determine what routing protocol is configured on the router (note: The routing protocol will
change in labs that follow)
5. Ensure routing is operational
6. Document the network topology
Verification:
1. Verify that all required interfaces are active and operational.
2. Verify that all adjacencies are up
3. Verify that all expected routes are populated in the routing table
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 24/89
4. Make sure your lab topology is complete and accurate. The following labs will not provide
addressing. It is your responsibility to use your documentation from this lab for the subsequent
troubleshooting labs.
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Objective:
On both the Core and Edge router save your configuration files to the TFTP server
Exercise:
Ensure you can access the TFTP server and then save the configuration files to it.
1. Your instructor will provide you with the IP address of the TFTP server for each pod.
2. Ping the TFTP server and ensure its operational
3. Save your Core routers configuration file using the following naming convention:
a. podXcore-student
4. Save your Edge routers configuration file using the following naming convention
a. podXedge-student
5. If possible access the TFTP server and ensure the file has been saved
Verification:
Make sure the files have been saved on the TFTP server.
1. Access your TFTP server and look for the file you saved
2. If unable to access the TFTP server directly, ask your instructor to verify the file has been
saved to the TFTP server.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 25/89
Questions:
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
What command can be used to determine the naming convention used for the interfaces? ______________
______________________________________________________________________________________
What command can be used to determine if all routers are in the same area?
__________________________________________________________
Notes
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 26/89
Lab 2 Setting event logs and debugs
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Objective:
Configure event logs on your R1 router.
Syntax:
Commands required for this exercise are found in Table 2-1. Detail may be found in Module 2,
Troubleshooting Methodology. Each command may have additional parameters possible. Use the ?
character for help and to explore all command line options. Other commands may also be used, including
those from previous courses.
Exercise:
For this exercise each student will create a log on one of the routers in their POD.
1. Configure an event log on the R1 and R2 router using log-id 1
2. define the source as all events from the main event stream
3. define the destination to memory with a maximum size of 100
4. Create log filter so only link down and link up events are logged.
Verification:
1. Using show commands ensure the log is created in your router
2. Clear log 1 and log 99
3. On R1 shutdown the port facing R2 within your pod
4. Examine log 1 and confirm only interface down and interface up events are logged
5. Compare this to the information displayed in log 99
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 27/89
Section 2.2 Use Debugs on the router for analysis
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Objective:
Using Debugs examine the output and determine usability.
Exercise:
For this exercise each student will enable debug on one of the routers in their POD.
Use Debug for packet analysis. It is always a good idea to have multiple sessions open when enabling
debug.
1. Open 2 sessions to a router within your POD
2. From one of your sessions create a log file with log-id 2
3. Define the source of information that of debug-trace
4. Forward the information to session
5. Enable debug for IP packet
6. Initiate a ping between the R1 and R2 router in your pod
7. Notice from the debug output there is a lot more information than the ping
8. From the session that is not being used for debug administratively shutdown the log that is being
used for debug (note: this releases any system resources that were being used for debug). Other
methods for turning debug off are disconnecting the session that is being used for debug or using
the command no debug.
9. Disable the debug for IP packet
10. Enable debug for IP packet again but this time only enable it for icmp traffic on the interface
between R1 and R2 in your pod. What command did you use to do this?
____________________________________________________________________________
11. Turn off debug
Objective:
Log packets to memory with a filter.
Exercise:
For this exercise each student will create filter to log packets to memory on one of the routers in their POD.
1. Create a filter log to log to memory
2. On your R1 router Configure an IP filter to match the address of your R1 system address and your
R2 system address for ICMP protocol. The goal is to log echo requests and echo replies between
your R1 and R2 router. The default action should be forward and the action for each entry should
be forward. The same exercise can be done on your R3 router for pings between the R3 router and
R2 router in your POD.
3. clear the filter log to endues it is empty
4. Initiate a ping from your R1 router to the system address of your R2 router
Verification:
1. show the contents of the filter log.
2. You should only see icmp traffic with the source and destination address as your R1 system
address or R2 system address
3. You should see icmp traffic egressing and ingressing on the interface the filter was applied to.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 28/89
Questions:
1. What must be created prior to activating debug on the router? _______________
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
_________________________________________________________________
4. To log telnet debug traffic to a file what command would you execute when
activating the debug? _______________________________________________
__________________________________________________________________
Notes
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 29/89
Lab 3 Troubleshooting OSPF
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Objective:
Troubleshoot network problems in a OSPF routing environment
Syntax:
In this lab we will troubleshoot some typical OSPF problems. Several troubleshooting tools will be used.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 30/89
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Exercise:
Load the configuration file pxry_ospf1.cfg into the routers.
In this section the students will work on their own POD in pairs.
After the routers are pointing to this configuration and the routers have been rebooted, routers pxr3 will be
inaccessible for the students. Take a few minutes to look at the configuration set up.
1. Are all adjacencies up? If not, which adjacencies are down?
2. What is the state of the adjacency? Can you find the different adjacency states the router went
through in the log?
3. When the adjacency gets stuck in this state, what is generally the cause of it? Start a debug trace to
prove this. What OSPF message are you looking for? What command can you use to show the
locally configured operational value for this parameter?
4. Provide 2 ways to solve this problem.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 31/89
Section 3.2 OSPF Route Redistribution Problem
Exercise:
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Notes
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 32/89
Section 3.5 OSPF route redistribution problem on NSSA
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
A network administrator is trying to redistribute the 192.168.1.0 and 192.168.3.0 networks into Area 1 and
2 respectively. Users in area 1 however can not reach 192.168.1.0 and users in area 2 can not reach
192.168.3.0.
Exercise:
Load the pxry_ospf2.cfg configuration into all routers.
Isolate the problem.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 33/89
Lab 4 Troubleshooting RIP-OSPF redistribution
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Objective:
Troubleshoot some typical problems that arise in RIP networks and when redistribution is used for RIP and
OSPF.
Before you start with the individual sections, load the pxry_ospf_rip.cfg configuration on each
router. Find the problem in each section but only solve the problem after the teacher tells you to.
There are several problems in this setup. Each problem will be covered in a separate section.
Section 4.1
Exercise:
When you logon to pxr2 or pxr3 you can see that not all routes from the backbone or from the other pods
can be found in the route table.
1. Whats wrong?
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 34/89
Section 4.2
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Exercise:
The system address of pxr3 does not appear in the route-table of the other routers. Router pxr3 however
does receive RIP routes from the other routers.
1. Whats wrong?
Section 4.3
Exercise:
The link address of pxr1-pxr2 does not appear in the route-table of routers in other PODs.
1. Whats wrong?
Section 4.4
Exercise:
The system address of p2r2 and p2r4 does not appear in the route-table of the other routers.
1. Whats wrong?
Notes
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 35/89
Lab 5 Troubleshooting ISIS
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Objective:
Trouble shoot ISIS related network problems
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 36/89
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Exercise:
In the IS-IS topology all networks and routers should be reachable from all routers. There is currently a
problem within the network preventing this from happening. Load the configuration file pxry_isis-1.cfg
into the routers.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 37/89
Section 5.2 ISIS Network Traffic Flow
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Exercise:
There have been reports of asymmetrical traffic flow in the above network. Confirm the reports and
identify where this is occurring in the network.
Load configuration file pxry_isis-2.cfg.
1. Identify all link metrics and fill in the information in space provided on the topology
2. What is the cause of the asymmetrical traffic flow?
Exercise:
Load the pxry_isis-3.cfg configuration in each router.
The topology is the same as in the previous sections. All networks and routers should be accessible from all
routers. The client opened a ticket because not all networks are reachable. Since this is a live network, you
are not allowed to make changes to the configuration.
1. What is the problem?
2. What are useful commands or tools to troubleshoot this problem?
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 38/89
Lab 6 Troubleshooting BGP
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Objective:
Troubleshoot BGP network related problems.
Section 6.1
Exercise:
Load the pxry_bgp-1.cfg configuration in each router.
In the above network P4R1 is the preferred exit point for all traffic destined outside of AS 6510. All
networks and routers should be reachable from AS 6510. Notice P3R1 and P4R1 are the only routers
within AS 6510 running BGP. Both P3R1 and P4R1 are injecting default routes into the IGP.
1. Identify which routers and networks are not reachable
2. What is the result of the current configuration
3. Identify 2 possible solutions or recommendations
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 39/89
Section 6.2
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Exercise:
Load config pxry_bgp-2.cfg
In the above network P1R1 and P3R1 are router reflector servers for AS 6500. All networks in all 3
autonomous systems should be reachable from any router in any autonomous system. So far it has been
determined that there is no connectivity between AS 6510 and AS 6520.
Note. There are a number prefixes being advertised by AS 6510 and AS 6520 which will not be reachable
but are used to populate the routing tables with a larger number of entries. All networks that are physically
on each router should be reachable.
1. Identify if there are any other connectivity problems in the above network.
2. Identify the BGP peering sessions
3. Which prefixes are being populated in the routing table?
4. What is the Flag value of the BGP routes that are not being installed? What command did you use
to find this?
5. Identify 2 possible solutions to the problem
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 40/89
Section 6.3 BGP peering problem
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Objective:
Troubleshoot BGP network related problems.
Before you start with the individual sections, load the pxry_bgp-3 configuration on each router. Find
the problem in each section but only solve the problem after the teacher tells you to.
There are several problems in this setup. Each problem will be covered in a separate section.
Exercise:
There seems to be a problem with the BGP peering sessions on pxr3.
1. Whats wrong?
2. On top of show commands and checking configurations, what extra tools can you use to
troubleshoot these problems?
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 41/89
Section 6.4 BGP Route redistribution problem
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Exercise:
Solve the problem in Section 6.3. All BGP peering should be up when you start section 6.4.
The routers in AS 6500 can not reach the p1r3 or p3r3 router.
1. Whats wrong?
2. How could you solve this problem
Exercise:
Solve the problem in Section 6.4. All routers should be reachable now.
In this section there is a problem with 192.168.[1,2,101,102].0/24 routes inserted on routers P1R3 and
P3R3.
Work in 2 teams, 1 looking at the problem with routes 192.168.1.0/24 and 192.168.2.0/24, the other team
looking at the routes 192.168.101.0/24 and 192.168.102.0/24.
The administrator wants to prepend the AS-path for the 192.168/16 routes. The administrator also wants the
routers in AS 6500 to choose P3R1 as the exit point for 192.168.102.0/24, P4R2 as the exit point for
192.168.101.0/24, P1R1 as the exit point for 192.168.2.0/24, and P2R2 as the exit point for 192.168.1.0/24.
A policy has been implemented on P1R2 and P3R2 but it does not seem to functioning as intended.
1. Whats wrong?
2. How can you solve this problem?
3. What command can you use to see the changes between the RIB in and RIB out?
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 42/89
Lab 7 MPLS LSPs With RSVP Signaling
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Objective:
Troubleshoot MPLS RSVP signaled LSPs.
Section 7.1
Exercise:
Load pxry_fullmesh_lsps.cfg on all routers.
In the above network a full mesh of LSPs has been setup between all routers. For this exercise each student
will work a on a single router.
1. What command can be used to view RSVP sessions?
2. View only originating RSVP sessions.
3. Enable debug for only one of the originating RSVP sessions. Use debug router rsvp lsp-id
<lspname>::<pathname> packet path detail
4. Disable Debug for the single LSP
5. View only terminating RSVP sessions
6. Enable debug for only one of the terminating RSVP session. Use debug router rsvp lsp-id
<lspname> packet path detail
7. Disable debug for the single LSP
8. View only transit RSVP sessions
9. Enable debug for only of the terminating RSVP sessions. Use debug router rsvp lsp-id <lspname>
packet path detail
10. What other options are available for limiting the debug output for RSVP packets?
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 43/89
Section 7.2
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Before you begin this section the instructor must implement a change.
Exercise:
There are several LSPs that are down. Use the tools perform router mpls cspf command to isolate where
the problem is in the network. Refer to the RSVP Signaled LSP Problems section in the appendix of the
lab guide. Try the options that are available in the command. Each node has visibility of the entire TE
network. There is no need to go to each router in the network to see which path cspf will return to an
endpoint. The from option can be used to determine what another router will return as its path to an
endpoint
1. Where in the network is problem?
Section 7.3
Before you begin this section the instructor has to make changes.
Exercise:
There are a number of LSPs that are down in the network.
1. Determine which LSPs are down. What do all the LSPs have in common?
2. What is the failure code of the LSP Path (show router mpls lsp <lspname> path <path name>
detail)?
3. What are the LSPs relying on to get the information required to signal the LSP?
Section 7.4
Exercise:
Before this lab begins the instructor must make changes
There is a mesh of LSPs between all R2 routers with FRR enabled. All expected detours are not coming
up.
1. View the LSPs and path configuration
2. For each of the FRR enabled LSPs determine where the expected detours should be.
3. Isolate where the problem is in the network.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 44/89
Lab 8 Troubleshooting VPLS Service Connectivity
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Objective:
Troubleshoot VPLS service connectivity.
Exercise:
Load pxry-VPLS4000.cfg on all R1 and R2 routers. The R3 routers are used so the SAPs on the R2
routers are operational
All sites within VPLS 4000 are not reachable. From your R2 router follow the steps below to familiarize
yourself with a logical approach to troubleshooting layer 2 services.
1. View the basic service information show service id 4000 base
2. The output of the above command indicates the operational states of the service and the
components the service is bound to.
3. View the details of the component that is in the down state. In this case there are 1 or 2 mesh-sdp
bindings down. show service id 4000 sdp <sdp> detail
4. The output of this command will have further details for the reason the mesh-sdp is down. The
Flags in the output indicate the SDP is operationally down
5. View the details of the SDP that is operationally down. show service sdp <sdp-id> detail
6. From the output of this command we can see that the flags indicate the transport tunnel is down.
The associated LSP is also displayed in the output of the above command.
7. View the detailed output of the LSP that the SDP is bound to. show router mpls lsp <lsp-name>
detail. This command will output the paths that the LSP is using
8. View the detailed output of the LSP path. show router mpls lsp <lsp-name> path <path-name>
detail.
9. From the output of the command above we can see the explicit hops that the LSP is taking.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 45/89
10. Use the tools perform router mpls cspf to <address> command each explicit hop in the list. In
this case the second one fails. The 3rd explicit hop takes a path that is not expected. This is a good
starting point to help isolate the problem
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
11. Open a connection to the router that is the second in the path
12. Are all expected interfaces operationally up? show router interface
13. Can you ping the adjacent interface on the link? ping <address>
14. Are all routing adjacencies up? show router ospf neighbor
15. Is MPLS enabled on the right interfaces? show router mpls interface
16. What is the problem?
Exercise:
A fix for the previous sections must be implemented before continuing with this section.
Connectivity between all sites in VPLS 4000 is down. Work in groups of 2 for this exercise.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 46/89
Section 8.3 Troubleshooting H-VPLS
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Exercise:
Load pxry_m-vpls.cfg on all routers
In the above diagram VPLS 6000 is fully meshed in Metro 1. VPLS 8000 is fully meshed in Metro 2. The
2 metro networks are connected via redundant Spokes between P1R1 and P3R1 and between P2R1 and
P4R1. Management VPLS 10000 has been created for the redundant Spoke connectivity. There are reports
that some sites are not reachable. It seems there maybe packets getting dropped. The R3 routers are CE
devices in this topology. Each is configured with a router interface on the 192.168.10.x/24.
1. You have full access to all equipment
2. Verify the reports of instability
3. Identify the cause of the problem
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 47/89
Notes
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 48/89
Section 8.4 Tracking packet flow in a VPLS with layer 3 termination
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Objection:
Trace the path of a packet through a VPLS with layer 3 termination
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 49/89
Lab 8.4 Command list
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Exercise:
Load pxry_trace.cfg on all routers.
Trace the path taken when initiating a ping from 192.168.1.2 to 192.168.3.2. Use 192.168.1.2 as the
source address when initiating a ping from P1R3. Include the return path from 192.168.3.2 to
192.168.1.2.
Notes
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 50/89
Lab 9 VPRN problems
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Objective:
Troubleshoot VPRN services.
Exercise:
Load pxry_vprnhns.cfg on all routers
There are 4 Hub and Spoke VPRNs configured, 100, 200, 300 and 400. Students will work in pairs to
troubleshoot one of the 4 VPRNs. P1R2 is the Hub for VPRN 100, P2R2 is the Hub for VPRN 200, P2R3
is the Hub for VPRN 300 and P2R4 is the Hub for VPRN 400. All traffic between Spoke sites route via the
Hub Site. All sites should be reachable. The R3 routers are not used for this lab but are required to have the
SAP operational on the R2 routers.
1. Login to each PE router
2. Document the VPRN configuration that you are troubleshooting
3. All VPRN interfaces should be reachable.
4. Determine which site(s) are not reachable
5. Isolate the problem
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 51/89
Section 9.2 Basic VPRN problems
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Exercise:
The instructor will implement a change before you begin this section.
VPRN 100, 200, 300 and 400 are configured in the same way as section 9.1. All VPRN interfaces should
be reachable from all VPRN interfaces within the same VPRN. Refer to the Layer 3 VPRN Problem
section in the appendix.
Exercise:
Load pxry-services1.cfg
The Pxr3 routers are CPE routers that are connected through a VPRN service. They have an interface
192.168.x.100/24 that is connected to the PxR2 router. The CPEs can not ping each other though...
In this lab the students will be working in couples per POD. There are several problems in this lab. The
questions will guide the student through the problems. Note down the commands you used for each
of the problems.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 52/89
Section 9.4 VPRN spoke-sdp termination problems
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Exercise:
Load pxry-services1.cfg
The Pxr3 routers are CPE routers that are connected through a VPRN service. They have an interface
10.99.x.1/24 that is connected to the PxR2 router. The CPEs can not ping the gateway 10.99.x.100.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 53/89
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
The following CLI commands are commonly used for checking the detailed configuration of cards, MDAs
or ports. Refer to Section 4 for more information on hardware operational status.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 54/89
MDA show mda
configuration & status show mda detail
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Ethernet ports:
You can NOT loop Ethernet ports using CLI commands.
SONET/SDH ports:
You can use CLI command to loopback a SONET/SDH port.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 55/89
NOTE:
1) The SONET/SDH port must be in a shut down state to activate any type of loopback.
2) When you loop back a SONET/SDH port, make sure it is not line timing.
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Description:
line Set the port into line loopback state.
internal Set the port into internal loopback state.
TDM ports:
You can use CLI to put a specified TDM port or channel into a loopback mode.
NOTE:
1) The corresponding port or channel must be in a shutdown state in order for the loopback mode to be
enabled. The upper level port or channel or parallel channels should not be affected by the loopback mode.
2) When you loop back a port, make sure it is not line timeing.
3) The loopback setting is never saved to the generated/saved configuration file.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 56/89
A.2. OSPF Problems
This section provides information on how to troubleshoot an OSPF related problem.
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Important Notes:
1) Before enabling debug, the user must make sure a log is created to view the debug result. The
following is an example log created to view debug results.
Note that if the log destination is session, when the session is closed, the log (log-id) will not be saved.
For example, log 3 is created to view the debug result:
SR12>config>log>log-id 3
SR12>config>log>log-id$ from debug-trace
SR12>config>log>log-id$ to session
SR12>config>log>log-id$ no exit
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 57/89
2) To stop the debug, use either of the following commands to stop the debug at different levels:
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Command Explanation
debug router ospf no Disable debugging for OSPF packets
packet
debug router no ospf Disable debugging for all OSPF messages
no debug Disable debugging for all applications
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 58/89
A.2.2. OSPF Adjacency Does Not Come Up
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 59/89
4. Mismatched subnet Check the router and its neighbors interface to see if the subnet mask
mask or IP address or IP address matches each other. Use the command:
show router interface
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
5. Interface not To verify if the interface has been configured in OSPF, use the
configured in OSPF commands:
show router interface to display router interfaces
show router ospf interface to display router interfaces in
OSPF
6. Router-id not unique Make sure the router has a unique Router ID.
Note: If the router-id Normally a router uses its system interface as its Router ID. A router
has to be changed the ID can also be configured specifically. If neither the system interface
OSPF process has to be or router ID are implicitly specified, then the router ID is inherited
shutdown/no shutdown from the last four bytes of the MAC address.
for the new route-id to To view the router-id, use the command:
take effect show router ospf status
To view the system(loopback) interfaces, use the command:
show router interface system
To add system interface(loopback) to OSPF, use the command:
config router ospf area <area-id> interface
system
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 60/89
8. Incorrect area To view the area of the interface, use the command:
show router ospf interface
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Task Command
Check for network stability. show router ospf spf
This command will display spf calculation statistics. The total number of
spf runs should not continually increment when the output is refreshed in
a stable network. This command can be used to determine if network
instability is caused by networks external to OSPF or internal to OSPF.
Note: In multi area environments it will be required to check the source of the summary LSAs. Once the
source of the summary LSAs has been determined run the above commands from the source of the
summary LSAs.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 61/89
View the ISIS related alarms/logs
To view the ISIS related alarms or log messages, use the command:
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Important Notes:
1) Before enabling debug, the user must make sure a log is created to view the debug result. The
following is an example log created to view debug results. Note that if the log destination is session, when
the session is closed, the log (log-id) will not be saved.
For example, log 3 is created to view the debug result:
SR12>config>log>log-id 3
SR12>config>log>log-id$ from debug-trace
SR12>config>log>log-id$ to session
SR12>config>log>log-id$ no exit
2) To stop the debug, use either of the following commands to stop the debug at different levels:
Command Explanation
debug router isis no Disable debugging for ISIS packets
packet
debug router no isis Disable debugging for all ISIS messages
no debug Disable debugging for all applications
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 62/89
A.3.2. ISIS Adjacency Does Not Come Up
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
3. Mismatched subnet Check the router and its neighbors interface to see if the subnet mask
mask or IP address or IP address matches each other. Use the command:
show router interface
debug router isis adjacency
4. Interface not To verify if the interface has been configured in ISIS, use the
configured in ISIS commands:
show router interface to display router interfaces
show router isis interface to display router interfaces in
ISIS
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 63/89
5. System-id not unique Make sure the router has a unique System ID.
Note: If the system-id Normally a router uses its system interface as its System ID. A router
has to be changed the ID can also be configured specifically. If neither the system interface
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
ISIS process has to be or router ID are implicitly specified, then the router ID is inherited
shutdown/no shutdown from the last four bytes of the MAC address.
for the new route-id to To view the system-id, use the command:
take effect show router isis status
To view the system(loopback) interfaces, use the command:
show router interface system
To add system interface(loopback) to isis, use the command:
config router isis interface system
6. Neighbor is If the routers ISIS neighbor is configured for authentication, the router
configured for must be configured to match the authentication. To view the
authentication authentication configuration of an interface, use commands:
config router isis interface <int-name>
config>router>isis>if# info detail
To configure the authentication on the interface level, use commands:
config router isis interface <int-name> hello-
authentication-key <authentication key|hash-
key> [hash|hash2]
config router ospf interface <int-name>
authentication-type {password|message-digest}
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 64/89
Task Command
Check for network stability. show router isis statistics
This command will display ISIS statistics. The total number of spf runs
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 65/89
messages
[no] packets - Enable/disable debugging for all BGP
packets
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Important Notes:
1) Before enabling the debug, the user must make sure a log is created to view the debug result.
2) To stop the debug, use either of the following commands to stop the debug at different level:
Command Explanation
debug router bgp no Disable debugging for all BGP Keepalive messages
keepalive
debug router no bgp Disable debugging for all BGP messages
no debug Disable debugging for all applications
3) The debug will stop if a router is rebooted for some reason.
2. Local or Peer AS To verify if the local or Peer AS is configured correctly, use command:
configured improperly show router bgp neighbor <neighbor address>
note: If the local-as has Use config router bgp command to modify AS number if it is the
to be changed the BGP problem.
process has to be For example: to modify the local AS number, use command:
shutdown/no shutdown config router bgp local-as <as-number>
for the AS to take effect. To modify the (group level) AS number for the remote peer, use command:
config router bgp group <name> peer-as <as-number>
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 66/89
3. BGP neighbor address To verify if a BGP neighbor address is configured correctly, use command:
mis-configured show router bgp neighbor
Use config router bgp group <name> neighbor <ip-
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
4. EBGP neighbor is not By default the ttl for BGP packets for EBGP sessions is 1. Modify the ttl for
directly connected the BGP neighbor with the following command
config router bgp group <name> neighbor <address>
multi-hop <ttl value>
For each prefix in the routing table, the routing protocol selects the best path. Then, the best path is
compared to the next path in list until all paths in the list are exhausted. The following parameters are used
to determine the best path:
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 67/89
Commands to adjust BGP attributes for load balancing
Local Preference Attribute Local preference can be set at the global level:
or group level:
or neighbor level:
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 68/89
always-compare-med config router bgp always-compare-med {zero |
infinity}
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Route policies allow you to configure routing according to specifically defined policies. You can create
policies and entries to allow or deny paths based on various parameters such as destination address,
protocol, packet size, and community list.
Policies can be as simple or complex as required. A simple policy can block routes for a specific location or
IP address. More complex policies can be configured using numerous policy statement entries containing
matching conditions to specify whether to accept or reject the route, control how a series of policies are
evaluated, and manipulate the characteristics associated with a route.
There are no default route policies. Each policy must be created explicitly and applied to a policy, a routing
protocol, or to the forwarding table. Policy parameters are modifiable.
Process of provisioning a basic router policy
The following diagram shows the process of how to provision a basic route policy.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 69/89
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
The following example is focused on how prefix lists are configured and used in a route policy, and how
this route policy applied to BGP. Other parameters such as AS-path, community list and damping
parameters are disregarded.
1) create/edit route policy
SR12>config>router>policy-options#
SR12>config>router>policy-options# begin
2) create/edit prefix lists
SR12>config>router>policy-options# prefix-list Deny-routes
SR12>config>router>policy-options>prefix-list# prefix 0.0.0.0/8 longer
. . .
SR12>config>router>policy-options>prefix-list# exit
SR12>config>router>policy-options# prefix-list "permit-routes"
SR12>config>router>policy-options>prefix-list$ prefix 10.10.1.0/30 exact
SR12>config>router>policy-options>prefix-list$ prefix 10.10.2.0/24
. . .
SR12>config>router>policy-options>prefix-list$ exit
3) create/edit route policies
SR12>config>router>policy-options# policy-statement "Service Provider-
IN"
SR12>config>router>policy-options>policy-statement$ entry 1
SR12>config>router>policy-options>policy-statement>entry$ from prefix-
list "D
eny-routes"
SR12>config>router>policy-options>policy-statement>entry# exit
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 70/89
SR12>config>router>policy-options>policy-statement>entry# action reject
SR12>config>router>policy-options>policy-statement>entry# exit
SR12>config>router>policy-options>policy-statement# default-action
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
accept
SR12>config>router>policy-options>policy-statement>default-action# exit
SR12>config>router>policy-options>policy-statement# exit
The begin command puts the node (not just the session) in a route policy edit mode.
Once begin is entered, until a commit is executed, subsequent users executing the begin
command will be warned that a policy configuration is in progress.
commit
A commit will save all policy configuration in progress on a node, this include all
session that have entered begin without having exited with a commit regardless of the state of
the route-policy under configuration.
A commit terminates edit mode for all users that are currently in edit mode.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 71/89
Troubleshooting Route Policies
To verify how the policy is configured, use command: show router policy
To verify how prefix list is configured in the policy, use command: show router policy prefix-
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
list <name>
========================================================================
=======
Route Policies
========================================================================
=======
Policy Description
------------------------------------------------------------------------
-------
Service Provider-IN
Service Provider-OUT
------------------------------------------------------------------------
-------
Policies : 2
========================================================================
=======
SR12#
SR12# show router policy prefix-list
==================================
Prefix Lists
==================================
Prefix List Name
----------------------------------
Deny-routes
permit-routes
==================================
SR12# show router policy prefix-list Deny-routes
prefix 0.0.0.0/8 longer
. . .
SR12# show router policy prefix-list permit-routes
prefix 10.10.1.0/30 exact
prefix 10.10.2.0/24 exact
. . .
SR12#
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 72/89
2. Check if the route is in the FIB. Use command show router fib <slot-number> [<ip-
prefix/mask]> [longer]]
3. Verify the routing policies for inaccuracies to ensure that packets are not getting filtered.
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Important Notes:
1) Before enabling the debug, the user must make sure a log is created to view the debug result. 2) To
stop the debug, use either of the following commands to stop the debug at different level (more choices
can be found by clicking ? at any level of the CLI syntax):
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 73/89
Command Explanation
debug router ldp interface <int-name> Disables debugging for specific LDP packets
no packet
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
debug router ldp no interface <int- Disables debugging for LDP interface
name>
no debug Disables debugging for all applications
3) The debug will stop if a router is rebooted for some reason.
Command Explanation
show router ldp bindings To display LDP bindings information
show router ldp bindings active To display LDP active bindings. An active binding must exist for a
prefix in order for an LSP to be active
show router ldp discovery To display LDP discovery information
show router ldp interface To display LDP interface information
show router ldp parameters To display LDP configured and operation parameters
show router ldp peer To display LDP targeted peer information
show router ldp session To display LDP session information
show router ldp status To display LDP operational information
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 74/89
<lsp-name> : [80 chars max]
<sender-address> : a.b.c.d
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
<endpoint-address> : a.b.c.d
<tunnel-id> : [0..4294967295]
<lsp-id> : [1..65535]
<ip-int-name> : [32 chars max]
Important Notes:
1) Before enabling debug, the user must make sure a log is created to view the debug result. The
following is an example log created to view debug results. Note that if the log destination is session, when
the session is closed, the log (log-id) will not be saved.
For example, log 3 is created to view the debug result:
SR12>config>log>log-id 3
SR12>config>log>log-id$ from debug-trace
SR12>config>log>log-id$ to session
SR12>config>log>log-id$ no exit
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 75/89
2) To stop the debug, use either of the following commands to stop the debug at different levels:
Command Explanation
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 76/89
A.8.2. RSVP MPLS Tools Perform Commands
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Command Explanation
tools perform router mpls Syntax :tools perform router mpls cspf
cspf to <ip-addr> [from <ip-addr>] [bandwidth <bandwidth>]
[include-bitmap<bitmap>] [exclude-bitmap <bitmap>] [hop-
limit <limit>] [exclude-address <excl-addr> [<excl-
addr>...(upto 8 max)]]
<ip-addr> : a.b.c.d
<bandwidth> : [1..100000] in Mbps
<bitmap> : [0..4294967295] - accepted in decimal,
hex(0x) or binary(0b)
<limit> : [1..255]
<excl-addr> : a.b.c.d (system or egress ip-address)
Context : tools>perform>
Description: This command does a manual CSPF calculation
based on the constraints provided in the command string. This
tool is very useful for troubleshooting LSPs that are not in the
up state or are not using the optimal path. This command can
only be used if Traffic Engineering is enabled in the IGP as It
relies on the traffic engineering database. The output of the
command is strictly informational and has no impact on any
LSPs. If the CSPF calculation fails it indicates that there is no
path in the TE database to reach the endpoint. The Traffic
Engineering database will not be fully populated if TE is not
enabled on all nodes in the network and if interfaces are not
MPLS or RSVP enabled. This tools command can be used
similar to ping to isolate the source of the problem. For
example, if the CSPF calculation fails to the desired endpoint
the next node in the path can be specified as the endpoint. This
can be continued until the failing node is isolated.
Parameters:
To Address- Endpoint IP address to run CSPF calculation to.
From Address- Starting point IP address for CSPF calculation.
This can be used to determine what path another node in the
network will return based on the constraints in the command
string
Bandwidth- 1..100000 in Mbps. This is used to add bandwidth
as a constraint when performing the CSPF calculation.
Include-bitmap- 0..4294967295 accepted in decimal, hex(0x) or
binary (0b). Only interfaces belonging to the specified
administrative group can be used as part of the CSPF calculation
Exlude-bitmap-0..4294967295 accepted in decimal, hex(0x) or
binary (0b). Only interfaces NOT belonging to the specified
administrative group can be used as part of the CSPF calculation
Hop-Limit-[1..255]. This is used to add a hop limit as a
constraint when performing the CSPF calculation
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 77/89
Tools perform router mpls Syntax: tools perform router mpls resignal lsp <lsp-name> path
resignal <path-name>
<lsp-name> : [32 chars max]
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Note: When you are reading the Failure Code of a LSP, wait for Retry period because the Failure Code
could change for the next retry period since for every retry period we attempt to compute a path again if the
last attempt to setup a LSP failed.
In addition, type cli command show router rsvp session status down on
all the LSRs along the LSP path to check if a PATH message reaches all the
LSRs or not.
noRouteToDestination For a non cspf empty path LSP with Dest Addr not valid/no valid route to
destination.
Verify that MPLS is enabled on all the LSRs in the network.
Verify that the to address is in the RTM. (Reachable by IGP)
For a cspf LSP with invalid strict/loose path.
Verify that all the routes/hops in the strict/loose path are valid and
are in correct sequence according to the network topology.
If a cspf LSP is trying to reserve a BW that none of the end-to-end paths in
the network can fulfill.
Check if the requested BW is too big for the network to handle or
the resources has already being used by other LSPs. Use tools
perform router mpls cspf to verify that there is a cspf path that
can fulfill the BW constraint.
If a cspf LSP cant find an end-to-end path in the network that matches the
color constraint.
Check if link color is assigned correctly for all the routes in the
network. Use tools perform router mpls cspf to verify that there
is a cspf path that can fulfill the color constraint.
For a cspf LSP, downstream routers have ospf/isis te off or opaque LSA
disabled.
Enable te /opaque LSA on all the LSRs that the LSP traverses.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 78/89
noCspfRouteOwner For a cspf LSP with Dest Addr not valid/ no valid route to destination.
Verify that MPLS is enabled on all the LSRs in the network.
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
admissionControlError For a cspf LSP with BW requirement that finds a te path to a destination
from IGP and is trying to send a PATH message to setup a LSP, if there is a
BW change in a downstream router and the opaque LSA hasnt arrived at
the ingress node yet.
In this case, shutdown the LSP and wait for 30s for Routing DB to
converge. Then do a no shut of the LSP, if there is still a path in
the network that can fulfill the BW requirement of the LSP, the
LSP will be setup properly.
For a non cspf LSP with BW request that is trying to setup to the egress
LSR, none of the paths in the network can fulfill it.
In this case, double check if the request BW size is valid according
to your network planning tool. BW resources might have already
been used by other LSPs in the network.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 79/89
A.9. Service Distribution Path (SDP) Problems
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
<sdp-id> : [1..17407]
- event-type
{config-change|oper-status-change|keep-alive-tx|keep-alive-
rx|keep-alive-timeout}
- no event-type
{config-change|oper-status-change|keep-alive-tx|keep-alive-
rx|keep-alive-timeout}
Important Notes:
1) Before enabling debug, the user must make sure a log is created to view the debug result. The
following is an example log created to view debug results. Note that if the log destination is session, when
the session is closed, the log (log-id) will not be saved.
For example, log 3 is created to view the debug result:
SR12>config>log>log-id 3
SR12>config>log>log-id$ from debug-trace
SR12>config>log>log-id$ to session
SR12>config>log>log-id$ no exit
2) To stop the debug, use either of the following commands to stop the debug at different levels:
Command Explanation
debug service no sdp Disable debugging for sdp events
no debug Disable debugging for all applications
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 80/89
3) The debug will stop if a router is rebooted for some reason.
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
KAFailure If message-length is configured for SDP keep Alive verify the SDP MTU
with the following oam command:
oam sdp-mtu
- sdp-mtu <orig-sdp-id> size-inc <start-octets> <end-octets> [step
<step-size>] [timeout <timeout>] [interval <interval>]
<orig-sdp-id> : [1..17407]
<octets> : start-octets [40..9198] end-octets [40..9198]
<step-size> : [1..512]
<timeout> : [1..10] seconds
<interval> : [1..10] seconds
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 81/89
A.10.1. Commands common to Layer 2 Service troubleshooting
show commands used to check Service related configuration
The following commands are commonly used for checking service related configuration:
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
<service-id> : [1..2147483647]
Important Notes:
1) Before enabling debug, the user must make sure a log is created to view the debug result. The
following is an example log created to view debug results. Note that if the log destination is session, when
the session is closed, the log (log-id) will not be saved.
For example, log 3 is created to view the debug result:
SR12>config>log>log-id 3
SR12>config>log>log-id$ from debug-trace
SR12>config>log>log-id$ to session
SR12>config>log>log-id$ no exit
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 82/89
2) To stop the debug, use either of the following commands to stop the debug at different levels:
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
Command Explanation
debug service no service Disable debugging for service events
no debug Disable debugging for all applications
1) View the basic service information with show service id <id> base. From the output of this
command the administrative and operational status of the service and service components (SAPs
and SDPs) can be determined.
2) For any service component that is in the down state view the component specific information with
the following commands.
a. show service id <id> sdp <sdp-id>
b. show service id <id> sap <sap-id>
3) Use the tables in the following sections to determine troubleshooting steps to follow based on the
flag codes in the outputs of the commands used in step 2. For any SDP binding that is down go to
the Service SDP Binding Does Not Come Up Section. For any SAP binding that is down go to
the Service SAP Binding Does Not Come UP section.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 83/89
SdpBindAdminDown Verify the following is True
The status of the sdp binding within the service is administratively
enabled
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
PortMTUTooSmall This indicates that the port MTU is smaller then the Service MTU. The
SAP MTU must be equal to or greater then the Service MTU
Check the port MTU the show port <port-id> command.
Adjust the port MTU.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 84/89
A.11. Layer 3 VPRN Problem
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
This section provides information on how to troubleshoot VPRN Services. This section will be limited to
troubleshooting VPRN Operational status and the importing and exporting of routes into and out of the
VRF.
Important Notes:
1) Before enabling debug, the user must make sure a log is created to view the debug result. The
following is an example log created to view debug results. Note that if the log destination is session, when
the session is closed, the log (log-id) will not be saved.
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 85/89
For example, log 3 is created to view the debug result:
SR12>config>log>log-id 3
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
2) To stop the debug, use either of the following commands to stop the debug at different levels:
Command Explanation
no debug Disable debugging for all applications
Component Explanation
PE Router A PE router is on the Service Provider Edge. Each CE device is
attached, via some sort of attachment circuit, toone or more
Provider Edge (PE) routers.
P Router P Routers are Routers in the SP's network which do not attach to
CE devices. P routers have no knowledge of the VPN routes
unless the P router is a Route Reflector server for PE routers.
Transport Tunnel Used to transport the data for the VPRN service between PEs.
Can be delivered by RSVP signaled LSPs, LDP signaled LSPs
or GRE. Auto-bind ldp and Auto-bind GRE are options.
PE-CE Routing Protocol Static
Support RIP
OSPF
BGP
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 86/89
A.11.3. Route Selection Process for VPRNS when BGP is being used as the PE-
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
CE routing protocol
IP-VPNs MSE direct route comparison of BGP and MP-BGP learned routes provides the ability
to compare a route received from a CE peer (inside the VPRN context) to the same route prefix
received as a BGP VPN-IPv4 update from a PE peer. This is required when a CE router is dual
homed and advertises the same customer route prefix to two (or more) PE peers. Each PE router
needs to choose one of the prefixes, which was done previously, based on the Route Table
Preference as opposed to comparing the BGP attributes. The BGP route decision process takes
into account the attribute values of the two routes according following table to decide which is
chosen to be the best route to install in the VRF table:
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 87/89
3. Route-Distinguisher Verify that a router-distinguisher is configured.
is not configured show service id <id> base
Note: A router-distinguisher is required for local VPRNs as well.
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
4. Route-Targets are not Import targets on the local PE must match the export targets on remote
configured properly PEs.
Verify that the expected routes from remote PEs are in the global BGP
routing table (BGP Rib-In). If there are no VPRNs with an import
target on the PE that matches the target in a received route the PE
router will silently drop the advertised route.
show router bgp routes <prefix>
If the expected routes are not present verify that the import target
configured within the VPRN matches the export target on the remote
PE.
Another command to verify the number of VPN routes learned from
remote PE is.
show router bgp summary
Verify the route-targets that are being advertised by the local node.
show router bgp routes <prefix> hunt
Check the community of the output from the above command to
confirm the export target used.
Note: This command will display the RIB-IN and RIB-OUT for the
specified prefix.
5. Local interface is not Verify the status of the local interface
showing up in the VRF Show router <service-id> interface
6. Transport Tunnel is A transport tunnel to the VPN routes next hop must be active before
not operational to next the route will get installed into the VRF.
hop for VPN route A spoke-sdp , auto-bind ldp or auto-bind gre must be configured.
show service id <id> base
From the output confirm that a spoke-sdp is configured or auto-bind is
configured.
If spoke-sdps are used for transport use the following command to
verify the status of the sdp
show service sdp <sdp-id>
If the sdp is not operational go to the Service Distribution Path (SDP)
Problemssection.
Note: Inner Labels are signaled by MP-BGP for VPRNs and signaling
is not required for SDPs being used for a VPRN service
If auto-bind LDP is configured confirm that there is an active binding
for the VPN next-hop address
show router ldp bindings prefix <vpn next-
hop/32> active
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 88/89
Alcatel-Lucent Confidential for internal use only -- Do Not Distribute
http:/ /www.alcatel-lucent.com/src
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1 89/89