Issue 3, 2012
Preface
In this issue of ZTE's Maintenance Experience, we continue to pass on various field reports and resolutions that are gathered by ZTE engineers and technicians around the world. The content presented in this issue is as below:
Maintenance Experience
Bimonthly for Data Products No. 4 Issue 272, June 2012
Two Technical Special Documents Eight Maintenance Cases of ZTE's Data Products
Have you examined your service policies and procedures lately? Are you confident that your people are using all the tools at their disposal? Are they trained to analyze each issue in a logical manner that provides for less downtime and maximum customer service? A close look at the cases reveals how to isolate suspected faulty or misconfigured equipment, and how to solve a problem step by step, etc. As success in commissioning and service is usually a mix of both discovery and analysis, we consider using this type of approach as an example of successful troubleshooting investigations. While corporate leaders maintain and grow plans for expansion, ZTE employees in all regions carry out with individual efforts towards internationalization of the company. Momentum continues to be built, in all levels, from office interns to veteran engineers, who work together to bring global focus into their daily work. If you would like to subscribe to this magazine (electronic version) or review additional articles and relevant technical materials concerning ZTE products, please visit the technical support website of ZTE CORPORATION (http://ensupport.zte.com.cn). If you have any ideas and suggestions or want to offer your contributions, you can contact us at any time via the following email:
doc@zte.com.cn.
Thank you for making ZTE a part of your telecom experience!
Contents
Technical Special
PIM-SSM Overview......................................................................................................................................2
Maintenance Instances
SSM-Mapping Fault on 89 Series Switches.................................................................................................8 Dual-Path Multicast Fault Between DRM and 8902.....................................................................................12 Fault Caused by Insufficient Capacity of Multicast Routing Table on FEI Board. ........................................14 PQ Configuration on MPLS Network............................................................................................................18 Address Pool Conflict on T600 BRAS..........................................................................................................20 Route Failure Caused by IS-IS OL Feature and Common OL Applications.................................................22 Cross-VLAN Multicast Configuration for 39 Series Switches.......................................................................24 OSPF Connection Fault Between ZXR10 T1200 and NE80........................................................................26
June 2012
Issue 272
Technical Special
PIM-SSM Overview
Qian Yuemei / ZTE Corporation
Abstract: This section describes the work flow of PIM-SSM, IGMP v3 messages, the function of SSM Mapping, the configuration and maintenance of PIM-SSM. Key words: PIM-SSM, IGMP, Mapping, Multicast
1 PIM-SSM Overview
Protocol Independent Multicast-Source Specific Multicast (PIM-SSM) is a service model different from the traditional one. It does not identify a multicast session as traditional PIM-SM protocol does. PIM-SSM uses a binary group multicast source address and a multicast group address to identify a multicast session. It maintains the quick joining of hosts into the multicast group in PIM-SM mode. PIM SSM bypasses shared distribution trees and the RP connection stage in PIM mode. It directly joins the multicast group towards the multicast source direction and forms a shortest-path tree. In Protocol Independent Multicast - Sparse Mode (PIM-SM), the shared trees and Rendezvous Point (RP) stage use (*, G) group to represent a multicast session, in which (G) indicates a specific IP multicast group, and (*) is a source that sends packets to the G. The PIMSSM is an extension to the traditional PIM protocol. It uses the SSM model to specify the multicast source and uses PIM-SM protocol to form a Shortest Path Tree (SPT) between the multicast source and subscribers. Subscribers can receive multicast packets from the multicast source directly without an RP. Similarities and differences can be found between PIM-SM and PIM-SSM
through the above description. Similarity: PIM-SSM and PIM-SM use IGMP join and PIM join, which is efficient. Differences: 1. PIM-SM uses multicast group G to identify a multicast session that contains multiple multicast sources. PIM-SSM uses a multicast group G and a multicast source to identify a session that contains one multicast group and one multicast source. 2. PIM-SM uses shared trees to send join messages to RPs. The PIM-SSM uses source-based trees to initiate join messages to multicast sources.
Joining
i. A host obtains source address information of a multicast group to be joined from the multicast source directory server. For details, refer to broken lines shown in Figure 1. ii. The host sends a source-specific group IGMP join message, meaning the (S, G) join, to the direct-connect router. The host initiates IGMPv3 join to R5, as shown in Figure 1. iii. After the router that directly connects to the
Maintenance Experience
www.zte.com.cn
host receives the IGMP join message, it initiates a PIM join message to its upstream router in the multicast source direction. After receiving the IGMP join message from the host, R5 initiates a PIM join message to R1, as shown in Figure 1. iv. The last hop router sends a (S, G) join message to the multicast source directly. The host does not join the group and no shared tree is generated. v. The multicast source sends the multicast traffic to the router. In accordance with the above process, join messages are determined by the (S, G) binary group instead of the group alone. In the join process, shared trees are not used. Source-based trees are joined directly, which simplifies the PIMSM protocol process.
by IGMP membership report messages. ii. When the router directly connected to the receiver receives the leave message, it checks whether there are the (S, G) users under its interface. If there is no user, it stops sending (S, G) multicast traffic to this interface. R5 cuts off the multicast traffic destined for the host, as shown in Figure 2. iii. R5 checks whether other interfaces need the multicast traffic regarding this (S, G). If there is no such interface, R5 does not need this multicast traffic and it initiates a PIM prune message to the upstream. R5 initiates the prune message to R1, as shown in Figure 2. iv. When R1 receives this message, it cuts off the multicast traffic to R5 and checks whether other interfaces need this multicast traffic. In the PIM-SSM, neither RP nor shared trees or RP mapping is necessary. RP election, RP register and SPT switching are no longer necessary for the routers
Pruning
i. When a receiver does not need multicast traffic, it sends an IGMP leave message. In IGMPV3, there are IGMP membership report message and query message only, excluding IGMP leave message. Here the leave message is used to help for understanding. The leave is completed
Data Products
June 2012
Issue 272
Technical Special
on the network. This simplifies protocol complexity and lowers the requirements on the network devices than before. For the cross-domain multicast routing protocol, the network does not need the Multicast Source Discovery Protocol (MSDP) to discover sources between RPs in different domains. The PIM-SSM protocol can be deployed directly.
IGMPv3, the leave message is not available. The leave is completed by the membership message. IGMPv3 includes query message (type 0x11) and membership message (0x22).
3 IGMP v3
In accordance with the above description of the protocol, before a host joins a group, it should know the source address. However, in Internet Group Management Protocol (IGMP)v1 and IGMPv2, there is no information regarding the source, so it is necessary to extend the IGMP protocol into IGMPv3. In IGMPv2, three types of messages are available: membership message, query message, and leave message. In
status of group members. When the Group Address fields in the query message are all zeros, the message is the General Query.
Source-Specific Query share the similar functions as the Group-Specific Query in GMPv2. When the router discovers that a member in a group leaves, it sends a Group-Specific Query message. When the
Maintenance Experience
www.zte.com.cn
router discovers that a source-specific member of some group leaves, it sends a Group-and-Source Specific Query message.
4 SSM Mapping
In accordance with the description of IGMP v3, IGMP v3/ Multicast Listener Discovery (MLD) v2 is a complicated protocol. It has changed a lot from the old versions. Implementation of the SSM based on IGMP v3 requires the operating systems, applications on the hosts and routers of the direct-connect receivers to support IGMP v3. This is not easy. So far, a lot of routers and terminal devices do not support IGMP v3. However, people could not just wait until IGMP v3 is fully available. Instead, for this situation, there is an interim solution, meaning SSM Mapping, to meet users demand for PIMSSM. SSM Mapping maps (*, G) information included in IGMP v1 or IGMP v2 report into (G, INCLUDE, (S1, S2)) information through configuration on the routers of the direct-connect receivers. With regard to the multicast source, this solution has no special requirements on applications that are responsible for advertising multicasts. Similarly, with regard to the receiver, there are no special requirements either. Operating systems and applications on the receiver do not need to support IGMP v3.
host sends a membership report containing the new group information. If there is no outgoing interface on the interface of the router for the group information, a new outgoing interface is be added.
sends a membership report to the router. In the information containing in the message, the group to leave is deleted. When the router receives such a membership report, it compares it with its own interface status. If the interface has information about this group, a Group-Specific Query message is sent. If the membership report regarding this group is not received within the specified time, the group is deleted from this interface. If received, the group maintains.
of the user has changed, the host also sends a membership report to the router. After receiving the report, the router sends a Group-Specific Query message to check whether the old multicast group and source have members. If they have, the outgoing interface remains. If not, the interface is deleted. The router compares the new multicast group and source with that on the local interface. If the local interface does not contain an outgoing interface for the multicast group and multicast source, a new outgoing interface is added.
Figure 3. Working Principles of SSM Mapping Mode
Data Products
June 2012
Issue 272
Technical Special
Figure 3 shows the working principles of SSM Mapping mode. As shown in Figure 3, host A, host B and host C run IGMP v1, IGMP v2 and IGMP v3 respectively. Versions on host A and host B are not allowed to be upgraded to IGMP v3. To provide SSM multicast service for host A and host B, it is necessary to enable IGMP SSM Mapping and configure corresponding mapping rule on router A. After that, when receiving an IGMP v1 or IGMP v2 report message from the hosts, router A first checks group address G in the message, and takes different actions in accordance with the results.
rules for the G, it maps (*, G) information in the report message into (G, INCLUDE, (S1, S2...)) information in accordance with the rules and provides the SSM multicast service.
If G is not in SSM-Mapping group address range, ASM multicast service should be provided.
SSM Mapping rules for the G, the SSM multicast service cannot be provided, and the message is discarded. ii. If router A has IGMP SSM Mapping
Maintenance Experience
www.zte.com.cn
Flags: T- SPT-bit set,A- Forward,J- Join SPT,U- Upsend , Jns- Pim Joins Macro,LAst- Pim Lost_assert Macro, Lcd- Pim Local_receiver_include Macro Timers:Uptime/Expires(Upstream State) Ind: 1/Jns: 0/LAst: 0/Imo: 1/Lcd: 1 Iif: NULL, RPF nbr: 0.0.0.0 Oif: vlan1, LocalIn / ImoXG
Macro state: Ind- Pim Include Macro,Exd- Pim Exclude Macro, Imo- Pim Immediate_olist Macro,Ino- Pim Inherited_olist Macro,
(*, 224.1.1.1), 00:00:09/00:00:00(JOINED), RP address: 0.0.0.0, Ind: 1/Jns: 0/LAst: 0/Imo: 1/Lcd: 1 Iif: NULL, RPF nbr: 0.0.0.0 Oif: vlan2, LocalIn / ImoXG
(*, 224.1.1.2), 00:00:08/00:00:00(JOINED), RP address: 0.0.0.0, Ind: 1/Jns: 0/LAst: 0/Imo: 1/Lcd: 1 Iif: NULL, RPF nbr: 0.0.0.0 Oif: vlan3, LocalIn / ImoXG
Parameter Connected Pruned RP-bit set Register flag SPT-bit set Up Send Join SPT Uptime/Expires RP flag Incoming interface RPF nbr Outgoing interface list
Description Specifies direct-connect member is available in the multicast group or on the interface Indicates that there is no next-hop for this entry Indicates that the (S,G) entry is available in RPT Indicates that the entry can send Register message from directly connected multicast source Indicates that the route entry receives a multicast packet sent from SPT Indicates that multicast packet is up-sent to this entry Indicates that the received traffic is switched to SPT Indicates the uptime and expiring time of entry/outgoing interface Indicates the RP available for (*, G) entry generated by PIM-SM Indicates the status of multicast route entry Indicates incoming interface for route entry Indicates RPF neighbor of the route entry Indicates the list of outgoing interfaces
Data Products
June 2012
Issue 272
Maintenance Instances
1 Network Description
At an IPTV site, the customer network runs PIM-SSM, as shown in Figure 1. The PE at the upstream of two border nodes 8902 isolates IPTV multicast and unicast. The PE connects to the unicast service from the subnet 172.*.*.* (with private network addresses) through MPLS VPN. It connects to the multicast service from the whole network multicast source of the central node VS8000C through the public network address 123.*.*.*. Each border node 8902 connects to the unicast service with private network address 72.*.*.* of VS8000C through L3 access, with
a default route pointing to the customers PE device. For the multicast service that uses the public network address, each border node 8902 transparently transmits multicast VLAN to the customers PE in L2 transparent transmission mode. Current VS8000C does not support IGMPV3. Each border node 8902 needs to use the SSMMAPPING function to map the IGMPv2 packets received by VS8000C multicast into IGMPv3 packets and to forward them to each node PE.
2 Symptom
1. During the testing of the border node IPTV traffic, peer PE fed back that it could not receive the IGMPv3 packets from 8902 that had been mapped. As a result, the PE could not receive the multicast traffic from the central node. 2. Some border nodes 8902 successfully mapped the IGMPv2 report packets sent from VS8000c. After a while, the traffic was interrupted and could not recover.
3 Fault Analysis
For symptom 1: 1. Engineers captured packets on 8902 and mirrored VS8000C to 8902 port. The IGMPv2 packets could be received, as shown in Figure 2. 2. Engineers mirrored 8902 to the PE port. The IGMPv3 query packets sent from the PE could be received. The IGMPv2 packets sent from VS8000C and converted into igmpv3 packets could not be
Figure 1. Network Topology
Maintenance Experience
www.zte.com.cn
3. From the output on the switch, IGMPv3 packets had been sent out.
IGMP SNOOPING Rcv General Query Msg: From Vlan 400, Port xgei_1/1. IGMP SNOOPING send v3 Report 232.0.0.29 Group in vlan 400 to xgei_1/1. IGMP SNOOPING send v3 Report 232.0.0.29 Group in vlan 400 to xgei_1/1. IGMP SNOOPING send General Query in vlan 400 to gei_2/19. IGMP SNOOPING send General Query in vlan 400 to gei_2/20. IGMP SNOOPING send General Query in vlan 400 to smartgroup1. IGMP SNOOPING Rcv 232.0.0.29 Group Report Msg: From Vlan 400, Port gei_2/19. 18:55:45 09/27/2009 18:55:44 09/27/2009 18:55:44 09/27/2009 18:55:44 09/27/2009 18:55:43 09/27/2009 18:55:33 09/27/2009
RemainTime
Version
8902-01#show ip igmp snooping host-source-filter vlan 400 Index VLAN Group ID 1 400 Ports
FilterMode SourceList
5. Engineers checked the PE side configuration and information on a border PE node. In accordance with the debugging information, IGMPv3 packets from 8902 had been received. Engineers at the PE side said that the source address in the mapped IGMPv3 packets was improper. The PE could not identify the packets. After the IGMPv2 packets sent from VS8000C were mapped by 8902, the source address of the report packets changed into the MNG management address of 8902, as shown in Figure 4.
Data Products
June 2012
Issue 272
Maintenance Instances
6. There was no route to the 8902 MNG management address on the PE. The PE could not receive the traffic. 7. Engineers modified the 89 MNG management address to the VS8000C address. The test channel established by the VS8000C received the multicast traffic immediately. For symptom 2: 1. Engineers displayed debugging information on 8902 and found that 8902 could perform mapping properly at the beginning.
IGMP SNOOPING Rcv 232.84.1.27 Group Report Msg: From Vlan 400, Port gei_2/15. IGMP SNOOPING send v3 Report 232.84.1.27 Group in vlan 400 to xgei_1/1. 11:12:34 10/31/2009 11:12:31 10/31/2009
RemainTime
Version
2. After a while, the SSM-mapping function on 8902 was abnormal. 8902 did not map the IGMPv2 report packets that needed to be mapped. It transparently transmitted the packets to the uplink device directly.
IGMP SNOOPING Rcv General Query Msg: From Vlan 400, Port xgei_1/1. IGMP SNOOPING send General Query in vlan 400 to gei_2/14. IGMP SNOOPING send General Query in vlan 400 to gei_2/15. IGMP SNOOPING send General Query in vlan 400 to gei_2/16. IGMP SNOOPING send General Query in vlan 400 to smartgroup1. IGMP SNOOPING Rcv 232.84.1.27 Group Report Msg: From Vlan 400, Port gei_2/15. IGMP SNOOPING send v2 Report 239.255.255.250 Group in vlan 400 to xgei_1/1. IGMP SNOOPING send v2 Report 232.84.1.27 Group in vlan 400 to xgei_1/1. IGMP SNOOPING Rcv 239.255.255.250 Group Report Msg: From Vlan 400, Port 11:06:16 10/31/2009 11:04:19 10/31/2009 11:04:12 10/31/2009 11:04:11 10/31/2009 11:04:11 10/31/2009 11:04:11 10/31/2009 11:04:11 10/31/2009 11:04:11 10/31/2009 11:04:11 10/31/2009
gei_2/14.
10
Maintenance Experience
www.zte.com.cn
3. Besides IGMPv2 report packets of the 232.1.84.* segment sent from VS8000C, 8902 also received the v2 report packets with address 239.255.255.250.
IGMP SNOOPING Rcv 239.255.255.250 gei_2/14.
7. 8902 could not map the IGMPv2 report packets of the 232.1.84.* segment properly. The interrupted traffic could not get recovered. 8. Engineers modified the configuration of backward compatible with IGMPv2 on the PE to maintain sending IGMP v3 query packets to the downstream. Interruption of the traffic never happened again.
4. The packets with address 239.255.255.250 were sent by a test PC connecting to 8902. 8902 did not map the IGMPv2 report packets from 239.255.255.250. It forwarded them to the uplink PE device.
IGMP SNOOPING send v2 Report xgei_1/1.
4 Conclusion
1. 89 series switches configured with the SSM-Mapping function will replace the host source IP address in the report packets with the IP address of the switch after receiving IGMPV1/V2 report packets from a host. 2. If there is VLAN L3 interface configuration on a 89 series switch, the host source IP address in the report packets will be modified to the VLAN L3 interface IP address on the switch. If there is no VLAN L3 interface configuration, the host source IP will be modified to the MNG management IP address of the switch. 3. On 8902, IGMP report packets are forwarded in accordance with the mrouter port. If the mrouter port receives IGMPv3 query packets, 8902 will map IGMPv2 report packets matching mapping rules into IGMPv3 packets and send them to the mrouter port. If the mrouter port receives IGMP v2 query, 8902 will not map the igmpv2 report packets, even if they meet mapping rules.
5. The PE was configured to be backward compatible with IGMPv2. After the PE received the IGMPv2 packets, the type of the query packets that the PE sent to the downlink was IGMPv2.
8902-01#show ip igmp snooping mr-portinfo
Version
State
6. On 8902, the IGMP report packets were forwarded in accordance with the mrouter port. If the mrouter port received IGMPv3 query packets, 8902 would map the IGMP2 report packets that matched mapping rules into IGMPv3 packets and send them to the mrouter port. If the mrouter port received IGMPv2 query packets, 8902 would not map the iIGMPv2 report packets (including the packets matching mapping rules).
Data Products
11
June 2012
Issue 272
Maintenance Instances
Notes: 1. SSM-Mapping is only applicable to the 232.0.0.0/8 group address, not for the other group addresses. 2. Enable the ssm-mapping function before configuring rules. 3. The SSM-mapping function on 89
series switches supports a maximum of 25610, meaning a maximum of 256 groups and 10 source addresses in each group. 4. Enable the IGMP protection function on the ports that need to process IGMP packets by executing the ipv4 protocol-protect mode igmp enable command.
1 Network Description
The DRM encryption system is normally used for IPTV multicast service. When the dual-path of DRM send a multicast request, both the active and standby 8902 switches need to process the request. In this case, the 8902 communicates with the uplink router through PIM-SSM multicast protocol, as shown in Figure 1. RTES is the name of DRM server. RTES1 is active, and RTES2 is standby. Port 5 is active, and port 6 is standby. RTES active/standby work as follows: On RTES, the active port (port 5) sends a multicast request and 8902-1 multicasts
Figure 1. Network Topology1
it to RTES1 and RTES2. After processing the multicast message, RTES1 forwards it to other devices. RTES2, as a standby device, only receives multicast messages, not processes or forwards them. Active/standby switching of RTES1 and RTES2 is controlled by another server.
12
Maintenance Experience
www.zte.com.cn
2 Symptom
After the router received the multicast request, it sent multicast messages. An exception occurred to RTES1-5 port. RTES1-6 port became the active and began to send the multicast request and receive multicast messages. The uplink port of 8902-2 was not available. The two multicast request paths initiated by DRM are:
user VLAN. The command for disabling mrouter interface learning was not configured in VLAN 19. After the hybrid mode was enabled, the multicast egress interface aged slowly. The DRM received two copies of multicast traffic from 8902-2, so the video become blocked. Engineers modified the configuration. The fault was solved by unicast route convergence and RPF inspection.
RTES1-6 -> 8902-2 -> ROUTER RTES2-5 -> 8902-1 -> ROUTER
The uplink port was enabled on 8902-2. PIMSSM used the shortest path for multicast. The test of interrupting uplink ports on the switches respectively for 8902-1/8902-2 master/slave switching ended in failure. 8902-1 was able to change over to 8902-2, but most of the channels became blocked for 3 minutes during the switching. During the changeover back to 8902-1, some channels were blocked and could not continue to play.
4 Solution
Engineers disabled the hybrid mode in VLAN 111 and disabled mrouter interface learning inVLAN19. The fault was solved.
5 Conclusion
Te d i o u s r e d u n d a n c y f e a t u r e s o f IPTV service multicast add difficulty to configurations on the switches. Relevant configuration should be added to ensure stability of services. Various changeover tests should be performed for verifying the function.
3 Fault Analysis
8902-1 and 8902-2 communicated through VLAN 111. The ip pim smdm hybrid mode was enabled on the switches. VLAN 19 was a multicast
Data Products
13
June 2012
Issue 272
Maintenance Instances
1 Network Description
The IPTV network of a city is shown in Figure 1. The stream media server serves as a server to receive multicasts. It also provides TV services for terminal users through unicast. T64G-1 connects to S8505 through a GE link. PIM-SM is enabled to import multicasts into the stream media. T64G-1 and T64G-2 connect to two S8512 devices through GE links to form a rectangle-like topology. T64G-1 and T64G-2 run OSPF to learn default routes from S8512 and provide TV services for terminal users. The IPTV network uses three RPs, of which two from SMG are responsible for providing TV programs in area A. S8505 is a RP for providing education programs in area B. Two decoders, 10.1.5.12 and 10.1.5.14, are directly connected to S8505. The decoder 10.1.5.12 uses a multicast address 225.1.1.10, and the other uses 225.1.1.14.
14
Maintenance Experience
www.zte.com.cn
2 Symptom
The customer added program sources 225.1.1.12 and 225.1.1.14. The customer used 222.83.5.58 (S8505) as the RP. The stream media could receive traffic from 225.1.1.12, but failed from 225.1.1.14. The stream media under S8505 could receive traffic from 225.1.1.14.
enable
protocol-protect mode igmp multicast untag-vlan 11 negotiation auto hybrid-attribute copper switchport mode trunk switchport trunk native vlan switchport trunk vlan 11 switchport qinq normal
11
3 Fault Analysis
1. Engineers checked configuration on T64G-. There was no problem. IGMP, PIM-SM and IGMP protection were configured on the port of T64G-1 connecting to the stream media. A static RP was configured in PIM-SM. ACL configuration was correct. The configurations are as follows:
interface vlan 1000 ip address 1.1.1.2 255.255.255.252 description TO_wengguan1 ip pim sm ! !
interface gei_2/48
description TO_wengguang1
enable
protocol-protect mode igmp multicast untag-vlan 1000 no negotiation auto duplex full
ip address 222.83.10.74
interface gei_5/11
protocol-protect mode igmp enable multicast untag-vlan 11 negotiation auto hybrid-attribute copper switchport mode trunk switchport trunk native vlan 11 switchport trunk vlan 11 switchport qinq normal switchport trunk vlan 100 smartgroup 9 mode on
interface gei_5/12
Data Products
15
June 2012
Issue 272
Maintenance Instances
2. Engineers checked the multicast routing table on T64G-1. They found that there were only (*, g) entities in the
225.1.1.12 and 225.1.1.14 groups. There were no (s, g) entities. The incoming and outgoing ports were correct.
T64G-1#show ip mroute group 225.1.1.12 IP Multicast Routing Table Flags:D -Dense,S -Sparse,C -Connected,L -Local,P -Pruned, M -MSDP created entry,N -No Used,U -Up Send, * -Assert flag
R -RP-bit set,F -Register flag,T -SPT-bit set,J -Join SPT, A -Advertised via MSDP,X -Proxy Join Timer Running,
Statistic:Receive packet count/Send packet count Timers:Uptime/Expires Interface state:Interface,Next-Hop or VCD,State/Mode (*, 225.1.1.12), 128d4h/00:03:35, RP 222.83.5.58 , 25071935/25071935 , flags: SC Incoming interface: vlan1000, RPF nbr 1.1.1.1 Outgoing interface list:
vlan11, Forward/Sparse, 82d0h/00:03:16 C T64G-1#show ip mroute group 225.1.1.14 IP Multicast Routing Table Flags:D -Dense,S -Sparse,C -Connected,L -Local,P -Pruned, M -MSDP created entry,N -No Used,U -Up Send, * -Assert flag
R -RP-bit set,F -Register flag,T -SPT-bit set,J -Join SPT, A -Advertised via MSDP,X -Proxy Join Timer Running,
Statistic:Receive packet count/Send packet count Timers:Uptime/Expires Interface state:Interface,Next-Hop or VCD,State/Mode (*, 225.1.1.14), 128d4h/00:03:35, RP 222.83.5.58 , 25071935/25071935 , flags: SC Incoming interface: vlan1000, RPF nbr 1.1.1.1 Outgoing interface list: vlan11, Forward/Sparse, 82d0h/00:03:16 C
Engineers checked the unicast routing table. They found that there was no route destined to other source addresses. Unicast RPF detection of PIM-SM failed, so there was no (s, g) entity. After static
routes were added manually, the routing table of the 225.1.1.12 group was normal. The routing table of the 225.1.1.14 group did not change. The stream media could not receive traffic from 225.1.1.14. The stream media could receive traffic from
16
Maintenance Experience
www.zte.com.cn
225.1.1.12, so the problem was not caused by the unicast routes. 3. Engineers checked the IGMP entities. There was no problem. The stream media was sending IGMP requests to T64G. 4. Engineers doubted that S8505 failed to send traffic from 225.1.1.14 to T64G-1, so (S,G) entities were not generated on T64G-1. Engineers captured packets on the uplink port of T64G-1. They found that traffic from 225.1.1.14 had been sent to T64G-1. 5. The faulty was located on T64G-1. T64G-1 received IGMP requests from the stream media, and generated IGMP entities correctly. It also received the multicast traffic from S8505, but it did not forward them. T64G-1 connected to S8505 through its 44FE+4GE board. This problem may be caused by insufficient capacity of the multicast routing table on the board. 6. Engineers executed the show ip mroute summary command to check the total entities in the routing table. They found that there were 268 routes and 110 groups in total. The capacity of sg+*g table on the board was 256. The total number of routes on the board was 268, exceeding the capacity of the table.
g) entity but (*, g) entities of 225.1.1.12 in the multicast routing table. Why was the stream media able to receive the multicasts properly? Answer: When the fault occurred, there was no (s, g) entity of 225.1.1.12 in the multicast routing table. This was because there was no unicast route of this multicast group source address in the routing table on the board, meaning the RPF detection failed. After the port enabled with PIMSM received an IGMP request, it could generate (*, g) entities. Note that RPF detection should be performed when routers were forwarding multicast packets. If the detection failed, the multicast packets would be discarded. At the end of the multicast forwarding, if there were IGMP entities on the router, the router would forward the packets in accordance with the IGMP table. RPF detection was not needed even if the detection failed. There were IGMP entities on T64G-1, so T64G-1 forwarded the packets in accordance with the IGMP entities directly. This procedure to troubleshoot such a fault is: 1. Check the IGMP entities to ensure that the device receives the IGMP requests and forms correct IGMP entities. 2. Check the multicast routing table to ensure that ingress port and egress port are correct. 3. The capacity of the routing table on the 44FE+4GE board is small (so is the capacity of the unicast routing table). If this board is used, do not use it as an uplink board.
4 Solution
Engineers replaced the port on T64G-1 connecting to S8505 with another one. They changed the uplink port to slot 4. The card in this slot is 12GE, and the multicast routing table capacity is 1K. The services were recovered. All entities on T64G-1 were normal. The fault was solved.
5 Conclusion
When the fault occurred, there was no (s,
Data Products
17
June 2012
Issue 272
Maintenance Instances
1 Symptom
On an MPLS VPN network, a T600 router works as a PE, as shown in Figure 1. The interface fei_5/1.2 connects to VPN1, and fei_5/2.2 connects to VPN2. Multilink1 is an egress, on which PQ is configured to guarantee traffic on VPN1 with priority. However, the PQ fails to provide the expected function.
2 Fault Analysis
This is an MPLS network. Data sent on multilink 1 is encapsulated with MPLS labels. The PQ configuration did not perform PQ scheduling in accordance with exp values, so it did not take effect on the data flows.
1. Set priority for colored packets in ingress direction. 2. Configure class-maps in accordance with MPLS exp labels (Routers will map the priorities of packets to the mpls exp labels automatically). 3. Set a policy-map, and put packets with different labels into different PQs. 4. Apply this PQ policy on the egress interface. Detailed configuration is as follows:
3 Solutions
Configure PQ to match MPLS exp values, as described below.
! !
encapsulation dot1Q 101 ip vrf forwarding VPN1 description VPN1 ip address 10.10.8.33 255.255.255.224
18
Maintenance Experience
www.zte.com.cn
rate-limit input localport cir 100000 cbs 1000000 pir 1000000 pbs 1000000
encapsulation dot1Q 202 ip vrf forwarding VPN2 description VPN2 ip address 10.10.10.65 255.255.255.224 rate-limit input localport cir 100000 cbs 1000000 pir 1000000 pbs 1000000
match mpls-exp 4
class-map test2 !
match mpls-exp 3
class test2
$ !
priority-queue low
4 Conclusion
Parameter descriptions:
The color priority on the ingress interface directly corresponds to the PQ priority effect. To prioritize the data flow on ingress interface 1, ensure that it has the highest precedence, and put the data flow into the high PQ when configuring the policy-map.
prec-transmit 4) of a packet set on the ingress interface is, the higher the precedence is.
mpls-exp value. This value is mapped from the priority field set on the ingress interface.
Data Products
19
June 2012
Issue 272
Maintenance Instances
1 Network Description
Five ZXR10 T600 (BRAS) devices connect to ZXR10 M6000. These devices run OSPF. The same address pool is configured on each BRAS. AAA server assigns IP addresses to fixed PPPOE users, so users connecting to different BRASs can obtain the same address.
subnet address from the five BRAS devices, so it could not respond to the ARP request of the host address properly. To solve this fault, engineers could configure static routes on T600 or advertise host routes on the BRAS devices. Setting static routes on T600 could solve the fault, but it was necessary to maintain the routes of these fixed IP users manually. Once a user obtained an address from a BRAS device, it was necessary to configure a static route pointing to this BRAS device on T600. If a user dialed up for access to the network frequently to change the BRAS device, it would be difficult for the T600 to maintain static routes. The solution was to advertise host routes on BRAS devices. T600 (BRAS) V2.08.23.B02P01 and later versions support VRF HOST user route distribution. The command is redistribute host.
2 Symptom
The M6000 learns a full segment of routes from the five BRAS devices. It could not forward packets with the host addresses obtained by PPPOE users properly.
3 Fault Analysis
The cause was that the five BRAS devices advertised the same network addresses. M6000 was determined by OSPF which can only advertise the whole segment of addresses configured for the device. The peer router only learned the address segment of the interface on which the addresses were advertised, even if the network 32-bit host address command was configured. T600 learned the same
4 Solution
Engineers ran OSPF on T600 and configured the redistribute host command.
router ospf 200 vrf test_ospf 0.0.0.0
20
Maintenance Experience
www.zte.com.cn
After the above configuration, engineers checked the routing table on the peer M6000:
M6000#show ip forwarding route IPv4 Routing Table: status codes: *valid, >best
metric-type Metric type peer Source peer route-map Route map reference tag Set tag for routes <cr> redistributed into OSPF
Dest Gw Interface Owner Pri Metric *> 192.168.110.0/24 192.168.100.1 gei-0/10/0/2 ospf 110 20 gei-0/10/0/2 ospf 110 20 *> 192.168.110.5/32 192.168.100.1
For T600 (BRAS), users must get online after the redistribute host command was configured. Otherwise, the host routes of users could not be advertised.
The redistribute host command advertised all the directly connected routes as 32-bit host route. To advertise specific segments as 32-bit host route, a policy routing could be used for filtering.
bras (config-router)#redistribute host ? as Source autonomous system number metric Metric for redistributed routes
5 Conclusion
The redistribute host command has no restrictions on devices. However, the advertised routes occupy the routing table of peer devices. In practical deployment, the route-map should be used for filtering if necessary.
Data Products
21
June 2012
Issue 272
Maintenance Instances
1 Network Description
As shown in Figure 1, the network is in an IS-IS L2 domain. A new T600 is added to the network, connecting to the SE800 through IS-IS.
2 Symptom
The IS-IS adjacency was in normal state. T600 could not learn routes advertised by peer IS-IS neighbors.
3 Fault Analysis
In theory, the cause may be:
22
Maintenance Experience
www.zte.com.cn
resources were released. LSPs could be advertised properly, and routes could be learned.
route calculation exception. 1. Engineers logged in to the device to check the IS-IS adjacency. It was normal.
T600#show isis adjacency SNPA(802.2) Pri MT 0022.9354.3FE0 64 Interface System id State Lev Holds gei_1/1 BL-WT-1.MAN.SE800 UP L2 7
5 Conclusion
When an LSP with the OL set to 1 is received by a router, the LSP does not take part in route calculation on the router. The router still forwards packets to the network to which the router sending the LSP connects to. However, the router will not be used as a transit router for forwarding data packets. Users can set the OL to get the data transport path bypass a certain router. In addition, the OL bit is most widely used on BGP networks (IGP uses IS-IS). When a new router is added into a network, IGP converges prior to BGP. If another router determines the added router is the next hop on the BGP path in accordance with the converged IGP route, while BGP convergence has not been completed on the added router, a route black hole forms. Before the BGP convergence is completed, users can set the IS-IS OL bit to avoid such a fault. Other routers will bypass this new router and choose another path. Once BGP convergence is completed, the OL bit will be cleared. It is recommended that users use the setoverload on-startup command to specify the number of seconds (300 seconds to 500 seconds), indicating the duration that the OL bit needs to set after IS-IS is enabled. Users also can add the wait-forbgp key word into the command to get the OL bit cleared once the BGP convergence is completed. However, once the BGP does not get UP due to some reason, the OL bit will never be cleared. So it is better to set duration.
2. Engineers checked the device configuration. The metric-style wide command was configured on the local device, which was consistent with the peer. Routes could be advertised properly. 3. Engineers checked the IS-IS database information on T600. The key information is shown below.
T600#show isis database IS-IS Level-2 Link State Database: LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OL 571 0/0/1 BL-WT-1.MAN.SE800-00-00 0x12d 0xf4a4
Engineers found that T600 received the LSP advertised by the SE800. However, the OL (overload bit) was set to 1. In general, it should be 0. The value 1 meant that the overload occurred to the router. Overload indicates that the router cannot provide enough system resources (including CPU and memory) to process routing and switching messages. The LSP with the OL set to 1 will not be flooded on the network. After receiving such LSP, other routers do not consider this type of LSP during the route calculation. As a result, this type of LSP does not take part in route calculation on T600, which caused the route learning fault.
4 Solution
Engineers restarted the peer SE800, and some
Data Products
23
June 2012
Issue 272
Maintenance Instances
1 Description
39 series switches experience a big change in cross-VLAN multicast configuration from D18 version. Configuration example: multicast VLAN60, unicast VLAN3800. Fei_1/1 is the uplink interface (connecting to the multicast router). Fei_1/2 is the user interface (connecting to users for crossVLAN multicast). The following compares configurations for two different versions of 39 series switches.
nas mode*/ /*Enter nas configuration /*Enable iptv control enable
IPTV control to provide cross-VLAN multicast function*/ create iptv channel general 1024
/*Define a general channel, which does not have restrictions on multicast*/ iptv channel 1024 mvlan 60 /*Specify the VLAN where multicast meaning the multicast VLAN*/ create iptv cac-rule 1 means all VLANs*/
rule 1, not followed by a specific VLAN iptv cac-rule 1 right order 1024 /*Bind rule 1 to general channel 1024*/ create iptv cac-rule 2 port gei_2/1 port. When this port receives query set to mr-port while other ports do not process the query. */ iptv cac-rule 2 right query 1024 /*Bind rule 2 to general channel 1024*/
/*Create rule 2 for specifying mroute from the multicast router, it will be
/*Enable igmp snooping querier globally, meaning enable the querier function*/ vlan 3800
VLAN, enable the querier. On multicast VLAN 60, do not enable the querier*/ igmp snooping querier
interface fei_1/1
24
Maintenance Experience
www.zte.com.cn
protocol-protect mode igmp protocol protection on the swtichport access vlan 3800 iptv service start /*Enable IPTV service*/ /*Enable IGMP
meaning enable the querier function*/ vlan 3800 /*On the unicast VLAN,
enable the querier. On multicast VLAN 60, do not enable the querier*/ igmp snooping querier
/*Configure
You should know multicast address indicates there are 255 channels in total from 239.255.40.1 to
Data Products
25
June 2012
Issue 272
Maintenance Instances
1 Symptom
T1200 and NE80 run OSPF. The connection between the two devices was disconnected. The OSPF neighbor relationship was re-established. The OSPF neighbor status on T1200 was FULL but that on NE80 was always loading.
Sum-Asbr 192.168.25.181 192.168.25.11 80000001 451 80000001 451 80000001 451 80000001 451 80000001 451 80000001 451 80000001 451 80000001 451 Sum-Asbr 192.168.25.217 192.168.25.11 Sum-Asbr 192.168.25.1 192.168.25.11 Sum-Asbr 192.168.25.10 192.168.25.11 Sum-Asbr 192.168.25.12 192.168.25.11 Sum-Asbr 192.168.25.15 192.168.25.11 Sum-Asbr 192.168.25.16 192.168.25.11 Sum-Asbr 192.168.25.69 192.168.25.11
2 Fault Analysis
The two devices were in an OSPF NSSA area. 1. Engineers checked the OSPF request list and found that NE80 was requesting some Type-4 LSAs (Sum-Asbr);
The Router's Neighbor is Router ID 192.168.25.68 Address 192.168.23.214 0.0.1.96
2. On NE80, engineers executed the debug ospf update command. T1200 kept sending Type-4 LSAs to NE80E and the LSA Age was 3600. The OSPF area type was NSSA. As a result, when NE80 received Type-4 LSAs, it discarded the LSAs. NE80E kept requesting these LSAs and the OSPF neighbor relationship was always in loading status.
6.3860500465 SXTY-PCR2 RM/6/RMDEBUG: FileID: 0x70178024 Line: 1859 Level: 0x20 OSPF 1: RECV Packet.
26
Maintenance Experience
www.zte.com.cn
80000001 451
6.3860500465 SXTY-PCR2 RM/6/RMDEBUG: 6.3860500465 SXTY-PCR2 RM/6/RMDEBUG: Ver# 2, Type: 4 (Link-State Update) Length: 336, Router: 192.168.25.68 Area: 0.0.1.96, Chksum: c8df AuType: 00 6.3860500465 SXTY-PCR2 RM/6/RMDEBUG: 6.3860500465 SXTY-PCR2 RM/6/RMDEBUG: 6.3860500465 SXTY-PCR2 RM/6/RMDEBUG: 6.3860500465 SXTY-PCR2 RM/6/RMDEBUG: Key(ascii): 0 0 0 0 0 0 0 0 LSAS: 11 6.3860500465 SXTY-PCR2 RM/6/RMDEBUG: # 6.3860500465 SXTY-PCR2 RM/6/RMDEBUG: LSA Type 4 6.3860500465 SXTY-PCR2 RM/6/RMDEBUG: LS ID: 192.168.25.162 6.3860500465 SXTY-PCR2 RM/6/RMDEBUG: Adv Rtr: 192.168.25.11 LSA Age: 3600 6.3860500465 SXTY-PCR2 RM/6/RMDEBUG:
192.168.25.11 80000001 451 192.168.25.11 80000001 451 Sum-Asbr 192.168.25.10 Sum-Asbr 192.168.25.12 Sum-Asbr 192.168.25.15 Sum-Asbr 192.168.25.16 Sum-Asbr 192.168.25.69 192.168.25.11 80000001 451 192.168.25.11 80000001 451 192.168.25.11 80000001 451 192.168.25.11 80000001 451 192.168.25.11 80000001 451
The AdvRouter (advertisement router) of all these LSAs was 192.168.25.11, meaning that these LSAs were generated by NE80. 2. When the link was disconnected and being re-established, T1200 should send all LSA digests (DD) in local database to its neighbors. DD packets contained Ty p e - 4 L S A s t h a t N E 8 0 a d v e r t i s e d previously. 3. After receiving DD, NE80 requested detailed Type-4 LSAs from T1200. T1200 responded a reply to NE80. NE80 was in an OSPF NSSA area, in accordance with the implementation mode of NE80, it discarded the Type-4 LSAs received from peer devices directly. As a result, NE80 could never get LSA information and the neighbor relationship was always in loading status. In a word:
3. NE80E requested these Type-4 LSAs because it received DD packets that contained these LSAs from T1200 in DD exchange stage. Further analysis: 1. NE80 thought there were Type-4 LSAs in the local database on T1200, so it requested these LSAs. Why did the local database on T1200 contain these LSAs? In accordance with analysis on NE80:
The router ID of NE80 was 192.168.25.11; The Type-4 LSAs NE 80 requested are as follows:
Type LinkState ID AdvRouter Sequence Age Sum-Asbr 192.168.25.105 192.168.25.11 80000001 451 80000001 451 80000001 451 Sum-Asbr 192.168.25.132 192.168.25.11 Sum-Asbr 192.168.25.162 192.168.25.11 Sum-Asbr 192.168.25.181 192.168.25.11
The root of this fault was that NE80 advertised Type-4 LSAs when it was configured to an NSSA area. This did not comply with the rules defined in RFC recommendations.
Data Products
27
June 2012
Issue 272
Maintenance Instances
T1200 should drop them directly instead of sending them to NE80 in DD packets. In accordance with RFC, Type-4 LSAs could not be sent in an NSSA area, but could be received. Also, when an OSPF neighbor relationship is being established, all LSAs in local database should be sent to neighbors in DD.
3 Solution
NE80 offers a solution to prevent NE80 from advertising Type-4 LSAs when it is in an NSSA area.
4 Conclusion
In-depth understanding of implementation principles of protocols and relevant recommendations is vital for troubleshooting.
28
Maintenance Experience
Address: ZTE Plaza, No.55, Hi-tech Road South, Shenzhen, P.R.China Post code: 518057 Customer Support Hotline: +86-755-26771900 Tel:+86-755-26776049 Fax: +86-755-26772236 Customer Support Email: doc@zte.com.cn Technical Support Website: http://ensupport.zte.com.cn Publication Date: June 15, 2012