How Traceroute Works Definition of Traceroute

Traceroute is a network diagnostic tool originally written by Van Jacobson to determine whether routing problems exist on the network. Traceroute can be used to determine which path IP packets are taking to get from your computer to the remote computer. Traceroute shouldn't be used on a network without routers. It is not really useful unless there are at least two routers in the network. The Internet has thousands (dare we say millions?) of routers so traceroute is perfect for the Internet. Traceroute was designed to reveal when network failures such as routing loops and black holes occur and shows roughly where those failures exist.

How Traceroute Works

Traceroute Implementations
Windows uses an ICMP-only traceroute-like utility called 'tracert'. UNIX, Linux and BSD based systems use a UDP-based traceroute. Why is this difference important? Because there is a difference how these tools work and in how networks will handle the packets these different tools produce. It can be argued that the Windows tool isn't traceroute at all.

Windows 'tracert'
1. The user sitting in front of the computer BINKY runs the tracert command at the command line. The command line is the MS-DOS prompt in Windows. 2. The user types the word tracert followed by the destination address, which can be either a name or an IP address, and presses the enter or return key on the keyboard. For example: Windows / MS-DOS c:\> tracert login.oscar.aol.com Binky, running tracert on a Windows computer creates 3 ICMP echo (ICMP type '8') messages with the time to live in the IP Header set to 1 and addresses the packets set to the destination computer's IP address (we'll call the destination computer clown). Binky starts a timer. Binky sends the three messages destined for clown out to the network. Binky waits for a response. This response will be: o An ICMP Time Exceeded message - this means the host responding is not the destination. o An ICMP Destination Unreachable - this means the host responding doesn't know how to get to the destination IP address in the traceroute packets.

3.

4. 5. 6.

7. The computer on which the messages die because the time to live expired (somewhere between Binky and clown ) sends back ICMP Time Exceeded (ICMP Type '11') responses. These messages indicate to Binky that the traceroute messages have not yet reached the destination clown. 8. Binky receives those Time Exceeded messages, notes the time they arrived, compares that to the time the ICMP Echo Request was sent and shows the results of that round trip on the screen. 9. Binky increments the TTL in the IP Header by one, then repeats steps the previous six steps (creates 3 packets, sets the Time to Live to the next highest number, starts a timer, transmits the packets, waits for a response). This process is repeated until the packets reach the destination computer (clown) which Binky is tracing the route to. 10. When the destination computer (clown) receives the packets, it sends back an ICMP Reply (ICMP type '0') and the traceroute program stops.

UNIX, Linux, Cisco and BSD 'traceroute'

The *NIX process is a bit different because it uses the Van Jacobson modification of using a UDP port number and relying on port unreachable errors to signify the end of the traceroute. Only the outbound packets are sent to UDP ports starting with 33434. The returning packets are ICMP and the UDP port number on the outbound packet usually increments upwards from UDP 33434 to match the TTL set in the IP Header. This is why some firewalls block UNIX/Linux/BSD traceroute but let Windows traceroute through.

Regarding Routers and Traceroute

If a router finds a TTL value of 1 or 0, it drops the datagram and sends back an Internet Control Message Protocol (ICMP) Time-Exceeded message to the sender. Traceroute determines the IP address of the first hop by examining the source address field of the ICMP Time-Exceeded message. To identify the next hop, traceroute sends a UDP packet with a TTL value of 2. The first router decrements the TTL field by 1 and sends the datagram to the next router. The second router sees a TTL value of 1, discards the datagram, and returns the Time-Exceeded message to the source. This process continues until the TTL is incremented to a value large enough for the datagram to reach the destination host or until the maximum TTL is reached or the destniation host replies with an ICMP Echo Reply.

Van Jacobson's Traceroute

The Traceroute command didn't actually work very well originally because of the interpretation of RFC 791 by routing equipment vendors. Thus, to fix this, Van Jacobson wrote a variant to Traceroute that worked so well and reliably, it was ported to all systems and used as the default. Many college textbooks still refer to this application when describing the functionality inside traceroute.

The Van Jacobson version used outbound UDP datagrams from the host running traceroute instead of ICMP. This was the default on any system using the Van Jacobson version of Traceroute including most BSD and UNIX type systems. To determine when a datagram reached its final destination, traceroute set the UDP destination port in the datagram to a very large value (33434 or higher) that the destination host is unlikely to be using. When a host receives a datagram with an unrecognized port number, it sends an ICMP Port Unreachable error message to the source. The Port Unreachable error message indicates to traceroute that the destination has been reached.

A FOOTNOTE
The first version of Traceroute used ICMP but not all routers responded properly to the ICMP echo requests due to the strict interpretation of RFC 791. Thus, VanJacobson wrote a UDP version which was widely used on UNIX and Linux systems and was the only functional version of traceroute at the time this tutorial was originally written. Here is Mr. Jacobson's answer to the question of why he used UDP 33434-33535: The original ip spec (rfc791) said that you should never send an icmp error in reponse to an icmp packet. Several years later this was amended to "... in response to an icmp *error* packet" but, at the time that traceroute was written, most router vendors had implemented according to the original spec & wouldn't send an icmp time exceeded in response to an icmp echo or echo reply. I then tried using an unassigned ip protocol instead of udp but it turned out that crashed HPUX systems (remember this was ten years ago, IP was new & there were lots of flakey implementations). The only thing that worked & didn't appear to do damage was udp to a port range that wasn't (& still isn't) used very often. See also: Traceroute Messages, Traceroute example

Networking Basics: Traceroute and Ping Overview

Tracert and Ping
Tracert
Tracert (and ping) are both command line utilities that are built into Windows and most other computer systems. The basic tracert command syntax is "tracert hostname". For example, "tracert visualroute.com" and the output might look like:

Here is a similar trace route as it would appear in a VisualRoute table:

Discover the path: Tracert sends an ICMP echo packet, but it takes advantage of the fact that most Internet routers will send back an ICMP 'TTL expired in transit' message if the TTL field is ever decremented to zero by a router. Using this knowledge, we can discover the path taken by IP Packets. How tracert works:Tracert sends out an ICMP echo packet to the named host, but with a TTL of 1; then with a TTL of 2; then with a TTL of 3 and so on. Tracert will then get 'TTL expired in transit' message back from routers until the desination host computer finally is reached and it responds with the standard ICMP 'echo reply' packet. Try it yourself: To see this in action yourself, just use the '-i' option of ping, which allows you to set the TTL value of outgoing ping packets. For example, "ping -i 1 visualroute.com" and you will see "Reply from 199.70.3.58: TTL expired in transit" (where the router IP Address returned, 199.70.3.58, is specific to your Internet connection). Then again with "ping -i 2 visualroute.com", and get back "Reply from 199.70.3.49: TTL expired in transit", and so on. Finally at "ping -i 13 visualroute.com" you get "Reply from 192.41.43.189: bytes=32 time=198ms TTL=245", which is the destination host responding. Round Trip Times: Each millisecond (ms) time in the table is the round-trip time that it took (to send the ICMP packet and to get the ICMP reply packet). The faster (smaller) the times the better. ms times of 0 mean that the reply was faster than the computers timer of 10 milliseconds, so the time is actually somewhere between 0 and 10 milliseconds.

Packet Loss: Packet loss kills throughput. So, having no packet loss is critical to having a connection to the Internet that responds well. A slower connection with zero packet loss can easily outperform a faster connection with some packet loss. Also, packet loss on the last hop, the desination, is what is most important. Sometimes routers in-between will not send ICMP "TTL expired in transit" messages, causing what looks to be high packet loss at a particular hop, but all it means is that the particular router is not responding to ICMP echo.

Ping
The basic ping command syntax is "ping hostname". For example, "ping visualroute.com" and the output might look like:

TTL reply: Ping sends an ICMP echo packet (with the TTL value set to the host default) to the host listed on the ping command line. Ping expects back an ICMP 'echo reply' packet. The millisecond time displayed is the round trip time. The "TTL=245" above says that the incoming ICMP echo reply packet has its TTL field set to 245. Because this value was decremented by one at each hop on the way back, this tells us that visualroute.com is probably setting the initial TTL value to 255. TTL Expired in Transit: Most computers today initialize the TTL value of outgoing IP Packets 128 or higher. If you ever see a reply above with a "TTL=5" (or some other low TTL number) this tells you that the computer being pinged should most likely have its default TTL value increased. Otherwise, anyone trying to communicate with the computer that is at a hop count higher than the TTL will not be able to communicate with the computer. For example, if you are 40 hops away from www.xyz.com, and www.xyz.com sets TTL fields in IP packets that it sends out to 32, the IP Packets will not reach you. They will 'expire in transmit' before they reach you. Discover your TTL: To discover the default TTL value of your computer, 'ping localhost' and examine the TTL reply value. For older Windows machines this value is 32. For newer Windows machines, this value is 128. ==================================================================
1997-2009 Visualware Inc. All rights reserved. Legal Information

Commands for basic FOS troubleshooting

created by bijukrishnan on Mar 25, 2010 12:29 AM, last modified by bijukrishnan on Mar 25, 2010 12:47 AM Version 1

****************************************************************************** ****************************************************************************** ********* Disclaimer: The information in this Contribution is provided AS IS, without warranty of any kind. The author reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use.

The author shall have no liability or responsibility to any person or entity with respect to any loss, cost, liability, or damages arising from the information contained in this contribution.

****************************************************************************** ****************************************************************************** *********

There are lots of brocade commands that come handy when troubleshooting a fabric related problems. We divide these into

Marginal Links

switchshow - Indicators like no light or In_sync will be displayed portshow <port> - Provides report on errors on a port porterrshow - provides gist of port errors, look for delta of counters

portstatsshow <port> - Provides details of port errors portstats64show - Extension of portstatsshow when 32 bit counters do not suffice, look for tim_txcrd_z to observe BB_credit deltas

fabriclog -s - Helps isolate flapping ports, look out for offline and online sequences sfpshow - Provides information on SFP and their state, check for power

Zoning

cfgshow - Displays zoning configuration defzone --show - Shows default zone settings nsshow -t - Display name server entries with additional details

nodefind <alias> - To find specific device in fabric nodefind <WWPN> - To find specific device in fabric

fcping <initiator> <target> - helps isolate end to end device connectivity and also an easy method to validate zoning

Licensing

licenseshow

Firmware Download

firmwaredownloadstatus firmwareshow version

Configuration

configdefault - To reset switch configuration to default configdownload - To download a previously uploaded config file

switchshow - Lists connection information as well first indicator of fabric segmentations errshow -r - Look for config events, match events with list in Brocade Message reference guide for suitable actions fabstatsshow - flags any parameter mismatch during merging fabricshow - Helps identify domain id conflicts and gives an overview of all fabric members

configure - To set correct switch and flow control parameters

portcfgshow <port number> - Display port configuration portcfgspeed <port number> - Set or show port speed portcfgdefaults <port number> - Set port settings to default portcfgpersistentenable <port> - To enable port permanently portcfgpersistentdisable <port> - To disable port persistently (across reboots)

portdisable portenable portcfgeport - Disable E port portcfglport - Lock port to L port portcfggport - Lock port to E or F port

Log collection

supportshow supportsave
8399 Views Average User Rating (0 ratings)

How to Troubleshoot Fabric Events Warnings

sniperx3me Mar 28, 2004 7:18 PM

Everyday i check our brocade switches there are always a warning log on the Fabric Events for ex. switch_1:admin> errdump Error 64 --

<BR>0x1023b8d0 (tThad): Mar 29 10:16:44<BR> WARNING FW-BELOW, 3, fopportState011 (FOP Port State Changes 11) is below lo<BR>w boundary. current value : 0 Change(s)/minute. (normal)<BR> <BR>Error 63<BR>

0x1023b8d0 (tThad): Mar 29 10:15:36 INFO FW-CHANGED, 4, fabricFL000 (Fabric Fabric login) value has changed. cur rent value : 408 Login(s). (info) Although this warnings ends in a Normal State still it irritates me everyday everytime i see one or two ports have this kinds of warning activity. Id already read the v30fabwatch.pdf document but there are no troubleshooting guide on how can i resolve this kind of problems, it mentioned about thresholds above/below do i have to adjust this thresholds so that i could minimize/eliminate this warnings? I doubt that there are problem on our switch or link because it always end in a normal state condition. is it the FC host?? link?? switch?? Also i monitor & record the porterrshow everyweek for the port error summary.. as usual ports inused increases error... even if i see enc out/ bad eof / too shrt /etc errors increases, What causes(identify) this problem & what actions should we make about this to minimize/eliminate this problem Hope u could give us some tips/ info/ docs in resolving this... i only have records of err no solutions at all....

here are a few things that may help to troubleshoot... find out what device is connected to this port. one error message is reporting a large number of FLOGI's from the device. if your are seeing increasing errors such as mentioned (enc out, etc...), the cabling or transceiver media may be bad/dirty. also, check other ports/devices in the same zone as this device. HBA's and some targets register for state change notification so that if another device in the same zone goes offline, an RSCN will be transmited so that registered devices can take appropriate action.
o o o o

Report Abuse Like (0) Switch Error Messages

BLOOM 4: Link Timeout

golubetz Nov 21, 2003 10:47 AM

Hello Sometimes our sw3900 switch (FabricOS 4.1.1a) generate message in syslog BLOOM 4: S0,P6(42): Warn: pt6: Link Timeout Fabric Watch also produces 2 messages - State Changes above and below boundaries. Error counters aren't increased during this situations, but we also register SCSI tran_err on HBA using disk array conected to port 6. Does anybody know which conditions switch ASICs (BLOOM) register this situation and produce this message.

1. Re: BLOOM 4: Link Timeout

Benrick Nov 24, 2003 5:16 PM (in response to golubetz)

Not sure if this will help. I am having similar issues. I noticed that this only happened to me on ports 7 and 8. After some investigation I realized that these ports were previouly used as E ports to ISL to another switch. I have not rebooted the switch since this ISLs were removed. I am working with Brocade support on this. I f I have a resolution I will let you know.
Report Abuse Like (0) 2. Re: BLOOM 4: Link Timeout
o o

TSG-Team-1 Feb 24, 2004 1:20 AM (in response to golubetz)

link timeout messages can mean there is buffer credit problem. if an HBA/target is attached to the port, it may not be returning credits in a timely manner. check to make sure driver/firmware revisions are up to date. if a link timeout occurs on an E_Port, there may be a misconfiguration of fabric parameters on one or more switches in the fabric. extended fabrics can cause this if long distance parameters are incorrectly set.
o o

Report Abuse Like (0)

Go to original post

Brocade Data Collection

There are 2 types of Data Collection associated with Brocade switches, "supportshow" and "supportsave". If the problem is related to basic "port issues", a "supportshow" is sufficient. However, a more complex problem that involves Zoning, Performance, Reboots, CP Failovers, Fabric wide problems etc, requires a "SupportSave" type of collection. If the problem is multiswitch related (ISL connections, long distance configurations etc.) we need the supportsave from all relevant switches in the fabric. The amount and type of information displayed in a supportshow output is depending on the type of switch as well as the collection options enabled in the switch configuration. This can be configured with the "supportshowcfgenable [option]" and "supportshowcfgdisable [option]". The output from the "supportshowcfgshow" command shows which subscripts will be enabled. The default groups are always enabled. Only on special occasions you will be asked to enable additional groups if necessary.
Performance and Intermittent Error Related Issues

If there is no sign of any obvious physical issue there might be link related problems which can identify performance problems and/or protocol related errors. Brocade counters are cumulative and keep doing so until a certain counter wraps, a switch reboots or the statistics are manually cleared.

HDS support requires in these circumstances that a new baseline is created, a certain run-time has been achieved and separate commands are submitted against the suspected switch or switches. To create a new baseline with cleared counters do the following:
1. Log in to the switch via Telnet or SSH 2. Submit the "statsclear" command 3. Submit the "slotstatsclear" command

After the agreed (mostly around one hour) run-time capture a new supportsave and upload this to the HDS TUF website under your case-id.
SupportSave Data Collection (CLI Method)

When the problem is more sophisticated a supportsave from the switch is required. The supportsave command is available as of Fabric OS version 4.4 however, Fabric OS versions (> 6.2.x) provide a significant better collection of logs which represent the status overview of the switch and fabric. If you have a director class switch with two CP's and/or core plus function blades it will also collect information from all the blades. The supportsave will upload between 25 and 80 files depending on platform, Fabric OS level and enabled features to an FTP or SCP server. These will not be tarred or zipped into one file so it is important you create such an archive with a meaningful name. (ie. switchname-domainidfabricid.zip) Example Fabosv4.4switch:admin> supportsave -u anonymous -p password -h xxx.xxx.xxx.xxx -d /directory -l ftp This command collects RASLOG, TRACE, supportShow, core file, FFDC data and then transfer them to a FTP/SCP server or a USB device. This operation can take several minutes. NOTE: supportSave will transfer existing trace dump file first, then automatically generate and transfer latest one. There will be two trace dump files transferred after this command. OK to proceed? (yes, y, no, n): [no] y Saving support information for switch:BR4100_IP127, module:RAS... Saving support information for switch:BR4100_IP127, module:CTRACE_OLD... Saving support information for switch:BR4100_IP127, module:CTRACE_NEW... etc...... To upload the files you can specify the FTP parameters inline (as modeled above) or through the supportftp command (see Fabric OS V5.1 command Reference guide) .
1. Host IP: XXX.XXX.X.X (example 192.168.1.1)

2. 3. 4. 5.

User Name: admin Password: Remote Directory: tmp (example: tmp) Saving support information .

SupportSave Data Collection via DCFM and Brocade Network Advisor (GUI Option)

NOTE: The FTP location is determined by the FTP server configuration in the DCFM options
SupportShow Data Collection

This is a non disruptive procedure and can be performed by the CE or the customer. You can use your favorite terminal emulation utility. Refer to the respective documentation how to turn on capturing output to a file. Telnet or SSH
1. Telnet or SSH into Brocade switch 2. Enter username and password 3. Start logging to file on the Telnet session. NOTE: For Windows standard telnet, this is under the terminal pulldown menu. 4. Enter command: supportShow 5. Upload the telnet log to TUF. Quick Links

Copyright 2008 Hitachi Data Systems. Ideas, requests, problems? Send feedback

==================================================================== ========

Windows Knowledge Base

Home PMP PREP OWA SAN/Storage o DMX o Clariion o IBM Storage o NETAPP o Brocade o CISCO Networking o LAN o Others o Security Hardware o SAN o Server o Workstation UNIX o AIX o HP/UX o Solaris Windows o Vista/XP/2K o Visio o Exchange
o o

Latest Topics
[VMAX]VMAX Replication Technoledge - IBM Storage Line Product Counterparts

TimeFinder Disk Local replication Point-In-Time Copy [VMAX]VMAX Single Point of Failure - Engine ..., A Symmetrix system provides redundant hardware that allows the system to remainoperational in the [VMAX] Understand VMAX Virtualisation layers A block is the smallest unit on a disk drive -520 Bytes block size -512 [VMAX]What is new with VMAX? 1. Architect from Direct Matrix Architecture to V-Max: Virtual Matrix Architecturewith Virtual Matrix technology, the [VMAX] EMC VMAX - Some Key Points about VMAX The Symmetrix VMAX system is the high-end, scalable storage array with a systembay and separate PreNext You are here: Home SAN/Storage Brocade

[Brocade] Troubleshooting - Switch Port shown wrong mode

To correct a port that has come up in the wrong mode, first use portshow <Port Number> to display more detailed information. Below listed some troubleshooting guide when you see port mode is wrong: Disabled Check the output from the switchShow command to determine whether or not the switch is disabled. If the port is disabled (for example, due to persistent disable or security reasons), attempt to resolve the issue and then enter the portEnable command. Bypassed Check the output from the switchShow command to determine whether or not the port is testing. Loopback Check the output from the switchShow command to determine whether or not the port is testing. E_Port

If the opposite side is not another switch, the link has come up in a wrong mode. Check the output from the portLogShow or PortLogDump commands and identify the link initialization stage where the initialization procedure went wrong. F_Port If the opposite side of the link is a fabric device, the link has come up in a wrong mode. Check the output from portLogShow or PortLogDump commands. G_Port The port has not come up as an E_Port or F_Port. Check the output from portLogShow or PortLogDump commands and identify the link initialization stage where the initialization procedure went wrong. L_Port If the opposite side is not a loop device, the link has come up in a wrong mode. Check the output from portLogShow or PortLogDump commands and identify the link initialization stage where the initialization procedure went wrong. Copyright 2010(c), Innovite Consulting Inc.( Devision of SynergyGear Tech)

ISL - TRUNKING - How to

Am about to configure a port for ISL between 12k and 48K. For this 1. we need trunking license - licenseshow 1a. NEED TO DISABLE THE SWITCH 2. portcfgtrunkport (specify a specific port as trunk port) - has to done on 48K and 12K 3. connect the 12 k switch cable to that specific port on 48K switc. 4. Then ISL is ready I have an ISL already running between 12K and 48K i want to make one as slave and other as Master - does this happen AUTO or we have to configure this? The ports configured for ISL should it needs to continous (1/2,1/3,1/4) or random ports(1/2,4/4,3/2) can be used for ISL ?

Brocade] Brocade CLI Command

Info uptime date version - Same as unix uptime - Same as unix date - Gives versions of firmwares & OS

Hardware State faultshow fanshow psshow tempshow switchstatusshow - Show switch faults - Show switch FAN faults - Show switch POWER SUPPLY faults - Show switch TEMPERATURE values - Overall status of switch

Config agtcfgshow configshow gbicshow licenseshow supportshow switchshow zoneshow - Show SNMP config - Show switch config - Show GBIC slots and serial numbers - Show license data - Like Sun's explorer - gets many configs at once - Show switch ports and connections - Show zone and switch aliases

IP bcastshow ifmodeshow ifshow ipaddrshow interfaceshow Performance ifshow portperfshow portshow # portrouteshow # portstatsshow # - Like unix netstat -s - Show interface mode (duplex) - Show stats on a port - Show routes on a port - Show stats (netstat -s) on a port - Show broadcast routing - show interface mode (duplex) - Like unix netstat -s - Like unix netstat -i - Like unix ndd

Misc Show diagshow - Show diagnostics - paged output errdump - Show diagnostics - no paged output fabricshow - Show fabric fspfshow - Show FSPF protocol info mqshow - Show queues nbrstateshow - Show FSPF neighbor states nsshow - Show name servers nsallshow - Show all name servers porterrshow - Like mpstat - shows port info switchstatuspolicyshow - Show config at when errors are flagged