Support Training Department 2007 AudioCodes Diagnostic Tools Version 5.2 Module Objectives Troubleshoot AudioCodes products using the relevant tools from the variety of available utilities.
Correctly report a problem to AudioCodes support department
Upon completion of this module you will be able to perform the following tasks: Troubleshooting Guide Lines Understanding the problem
Collecting data
What are the expected results?
What are the actual results?
Understanding the Problem Collecting Data Test setup (hardware, call direction, etc.) Uploaded ini file Syslog (decent syslog server without missing messages) Unfiltered wireshark Advanced (per request): PSTN traces PSTN problems DSP traces problems related to voice quality, modem\fax, tones, DTMF etc.
The ini File Must be added to every support request
Can be downloaded from the Web Interface (Advanced Configuration-> Configuration File)
There is no need to use the Complete ini file; all information can be obtained from the regular ini file
Includes information on the software version, networking parameters and the configuration of the gateway
When a parameter doesnt appear in the ini file it indicates that it received a default value Agenda Syslog
Network monitors (Wireshark)
Debug Recording
Command Shell
CPT Wizard
DSP Core Dump Chapter I
AC Syslog
AC Syslog Q: What does ACSyslog application is used for? A: The ACSyslog application is used to collect the trace messages that are sent by the gateway and send them as txt file.
Syslog Definition: The Syslog is a collection of error messages, warning, and/or other system messages that are sent to a central server using UDP port 514. Syslog protocol is defined in the IETF RFC 3164 standard. AC Syslog Configuration Set Log File Name Enabling Syslog Server Note: Enter the IP address of your Syslog server (PC where the Syslog installed) [Syslog] SyslogServerIP = 10.2.0.136 EnableSyslog = 1 SyslogServerPort = 514 Enable SIP Traces Always set the debug level to 5 if debug traces are needed GwDebugLevel = 5 Remember to turn off after troubleshooting.
Via Web Interface Protocol ManagementAdvanced ConfigurationMGCP Debug Level = High
Via ini File When enabling MGCP/MEGACO traces via ini file the traces are available from startup.
Enable MGCP / MEGACO Traces Online Syslog - Web Should only be used for simple debugging Doesnt include the IP address of the gateway Misses messages
Syslog Call Example IP addresses are 10.33.45.xx
MP1 (.72) (IP) TP1 (.65) (ISDN) TP2 (.69) (IP) MP2 (.61) Traces\sip_mp-tp-mp.log Syslog Applications Examples of Syslog servers available as shareware on the Internet:
Kiwi Enterprises: http://www.kiwisyslog.com/ The US CMS Server: http://uscms.fnal.gov/hanlon/uscms_server/ TriAction Software: http://www.triaction.nl/Products/SyslogDaemon.asp Netal SL4NT 2.1 Syslog Daemon: http://www.netal.com Wireshark cyclic buffer
Chapter II
Network Monitors (Wireshark)
Wireshark Network Monitor Q: What is Wireshark application used for? A: Wireshark is a free network protocol analyzer for Linux, Unix and Windows. It allows you to examine data from a live network or from a capture file on disk. You can interactively browse the capture data, viewing summary and detail information for each packet. Network Monitors Applications Use hub or port mirror Unfilttered Plugins The Wireshark plugins were tested with Wireshark version 0.99.4
WinPcap Application WinPcap is an architecture for packet capture and networkanalysis for the Win32 platforms. It includes a kernel-level packet filter, a low-level dynamic link library (packet.dll), and a high-level and system-independent library.
http://winpcap.mirror.ethereal.com/ Wireshark Capturing Packets Main menu: Capture Interfaces Select the network interface that is currently being used by the PC Wireshark Capturing Packets (Cont.) Select the Display options and press Start Wireshark Filters Filter specific addresses or / and protocol For example: sip and ip.addr==10.33.6.100 Select a specific field on Packet Details and apply it as a filter Wireshark Coloring Rules Different colors can be assigned for each protocol for better visualization. The Coloring list is processed until a match is found, therefore place general rules (like TCP and UDP) at the bottom of the list. Wireshark Generating Call Flow
Visual representation of different calls Main menu: Statistics VoIP Calls Wireshark Playing RTP Stream
From Wireshark version 0.99.4 Play back G.711 conversations Main menu: Statistics VOIP Calls Wireshark Analyzing the RTP Data
Wireshark can be used to analyze the RTP data stream and extract the audio from the data packets (only for G.711). Main menu: Statistics->RTP-> Stream Analysis menu item Wireshark Extracting the Audio
Save the audio payload of the RTP stream to a file. Save Payload as a .pcm file Select the forward option Can be played using programs such as CoolEdit
Wireshark Call Example MP1 (.100) (IP) MP2 (.101)
Traces\Basic analog call.pcap
MP1 (.100) (IP) TP (.179) TP (.179) MP2 (.101)
Traces\Basic MP to MP via TP call.pcap
Chapter III
Debug Recording
What is Debug Recording Q: What is Debug Recording used for? A: The Debug Recording (DR) mechanism is used to capture:
- Media streams (such as RTP and T.38) - Networking streams (such as HTTP) - Signaling traffic (such as SIP, MGCP and ISDN) - Other internal board information (such as DSP Events).
DR is relevant from version 5.2 and is used instead of the obsolete mechanisms that were used prior to this version to collect DSP / TDM and PSTN traces.
The Main Advantages of DR
Can be used to record all IP traffic that is sent / received from / by the gateway
Can be used to record the actual messages that are exchanged between the gateway and the PBX/PSTN switch
Can be used to record the actual voice signal that arrives from the TDM (before it enters the DSP)
Can be used to record network traffic in environments in which hub / port mirroring isnt available
Can be used to record internal traffic between two endpoints on the same gateway Collecting DR Messages
The client that is used to capture the DR packets is the open source Wireshark application.
AudioCodes proprietary plugins (supplied in the software kit) must be placed in the 'plugin' folder of the installed Wireshark version (typically, c:\Program Files\WireShark\plugins\xxx\, where xxx is the installed version).
The plugins were tested with Wireshark version 0.99.4
Use the following filter ACDR or udp.port==925 to view the DR messages that are sent by the gateway. Activating the DR
Start a CLI management session (via the FAE page)
At the prompt, type DR to access the DebugRecording directory.
At the prompt, type STOP to terminate all active recordings, if any.
At the prompt, type RTR ALL to remove all previous recording rules.
At the prompt, type RT ALL to remove all DR targets (i.e., client IP addresses) from the list.
At the prompt, type AIT <IP address of the target> to define the IP address of the PC (running Wireshark) to which the gateway sends its debug packets. Record Media Streams
Records: Internal DSP packets and events PCM - voice signal that arrives from the TDM (before it enters the DSP) RTP, RTCP, T.38 syslog GW is sending DR packets to external IP IP Record Media Streams (cont.)
AddNextCallTrace records the next media calls. AddTrunkBchannelTrace records media calls according to trunk and B-channel. AddChannelIdTrace records media calls according to CID. To record all media traffic (including PCM) of the next call enter: ANCT ALL-WITH-PCM 1 At the prompt, type START.
Note that currently PCM record on MP is available only on version 5.0 DR on internal MP call See Trace Files: Record Media Streams (cont.)
RTP/PCM Use filter rtp and acdr Record PSTN Messages Records ISDN, CAS or SS7 messages that are exchanged between the gateway and the PSTN/PBX switch.
First enable PSTN traces; via cmd shell enter: pstn PstnCOmmon PstnSetTraceLevel <TrunkId> <BChannel> <TraceLevel> For example, to enable PSTN traces on the first trunk: PstnSetTraceLevel 0 -1 1
For SS7 only, set the Trace Level of the SNs and Links tables to 1.
At the prompt, type AddPstnSignalingTrace <packet type - ISDN, CAS, or SS7>. For example, APST ISDN.
At the prompt, type START. PSTN Trace with DR See Trace Files: Record PSTN Messages (cont.) ISDN call flow can be viewed using the filter: acdr and q931
Record PSTN Messages (cont.) The trace can also be converted to a graph: Statistics->VoIP Calls Record Control Messages Records SIP, MEGACO, MGCP or TPNCP messages.
AddIPControlTrace TracePoint ControlType
Trace point (not relevant to SIP): Net2Host = Inbound non-media traffic Host2Net = Outbound non-media traffic
ControlType: SIP, MEGACO, MGCP or TPNCP
At the prompt, type START. Record IP Traffic
Used to record IP streams that are not associated with media or control. Such as HTTP and SCTP.
Trace Point: Net2Host or Host2Net PDUType: UDP, TCP, ICMP or IPType# Any other protocol type (as defined by http://www.iana.org/assignments/protocol-numbers). A = All traffic types. SourcePort: datagrams source port number (or A for all ports). DestPort: datagrams destination port number (or A for all ports).
To record SCTP traffic: aiptt n2h 132 a a aiptt h2n 132 a a DR General Notes
PSTN, DSP, Control and IP recording can be performed simultaneously
All DR rules are deleted after the gateway is reset.
Save the information that was captured in Wireshark and send it to AudioCodes.
DR recording must only be used for debugging purposes. On normal operation (if not requested differently from AudioCodes support) DR should be disabled.
To stop the DR, enter the command STOP in the command shell Chapter IV
Command Shell
Command Shell Q: What is Command Shell application used for? A: The Command Shell is primarily used for debugging purposes and to obtain various information about the gateway. For example, to check PSTN clocks synchronization.
Accessing the Command Shell via Web
Type the ip_address /FAE in the address fields of your web browser. Accessing the Command Shell via Telnet Enable the Embedded Telnet Server: When using the ini file, set the parameter TelnetServerEnable to 1 (standard mode) or 2 (SSL mode). When using the Embedded Web Server, set the parameter Embedded Telnet Server to Enable (Unsecured) or Enable Secured (SSL) and save the changes so they are available after a power fail.
Reset the gateway.
Use a standard Telnet application to connect to the gateways Embedded Telnet Server. Note that if the Telnet server is set to SSL mode, a special Telnet client is required on your PC to connect to the Telnet interface over a secured connection.
Login using the same username (default Admin) and password (default Admin) you use for the Embedded Web Servers Administrator level. Accessing the Command Shell via RS-232 Can be used when the gateway cannot be accessed via IP
Connect the RS-232 port to your PC
Use a serial communication software (e.g., HyperTerminalTM) with the relevant communications port settings (according to the gateway).
Commands Shell and Passwords There are three groups of commands:
Commands that are always available
Advanced commands. To enter this mode type: AdvancedMode. Use 'ADVancedmode OFF to return to normal mode.
Commands that are locked by password. The password is calculated by AudioCodes FAE from the IP address of the gateway. To enter this mode type: PW xxxxx Were xxxxx is the password. Use PW OFF to return to normal mode.
Command Shell Syntax Command line syntax is: [Command Name] <Parameters>
Enter h at the CLI prompt for help on global commands and enter h <command name> for information on a specific command.
Use two consecutive dots (i.e., ..) to access a higher directory level.
You can use the upper case of each command / directory as a shortcut. For example, enter CONF instead of CONFiguration and GPD instead of GetParameterDescription.
How to Get the Gateway's Exceptions When the gateway resets itself during operation without an explicit command from the user, an exception is burned to Flash. This exception can be viewed in the syslog server immediately after the gateway starts up and can be retrieved from the command shell using the following procedure: Access the FAE page (xx.xx.xx.xx/FAE) On the left hand menu press Cmd Shell In the Command Line type the following commands BSP EXC EI PH PSTN Clock Synchronization Several problems (such as, echo, fax and modem failures) can be caused if the E1/T1 clocks arent synchronized. To verify if the clocks are synchronized, use PSTN Performance Monitoring (PM). Note that PM is enabled by default. Access the command shell In the Command Line type the following commands AdvancedMode Cd PSTN Cd Physical PstnGetPerformanceMonitoring trunknumber 0 send the results to AudioCodes (for example, PstnGetPerformanceMonitoring 1 0) Chapter V
Call Progress Tones Wizard What is the CPT File The Call Progress Tones (CPT) configuration file is a binary file (with the extension dat) which contains the definitions of the Call Progress Tones (levels and frequencies) that are detected / generated by the gateway.
Most common tones are: Dial tone Ringback tone Busy tone Reorder Tone
Tones_File Example.ini CPT Wizard Q: What is CPTWizard used for? A: The Call Progress Tones wizard helps detect the Call Progress Tones generated by your PBX (or telephone exchange). The software automatically creates a basic Call Progress Tones: ini file. AudioCodes MP-11x FXO 2 Analog Line IP Network PBX Note: The version of the CPTWizard must match the software version of the gateway.
Getting Started with the CPT Wizard To use this wizard:
Connect the MediaPack/FXO gateway to your PBX with two physical phone lines
To install the CPTWizard, copy the files from the supplied installation kit to any folder on your PC.
The FXO gateway must be configured with factory-default settings and shouldnt be used for phone calls during the operation of the wizard. CPT Wizard- Initial Settings
Execute the CPTWizard.exe file. Enter the IP address of the MP-11x/FXO. Select the gateways ports that are connected to your PBX, and specify the phone number of each extension. Insert Invalid phone number field, that generates a fast busy.
CPT Wizard - Automatic Mode Press the Start Automatic Configuration button. The wizard starts the Call Progress Tones detection sequence (the operation takes approximately 60 seconds to complete) CPT Wizard - Automatic Mode (Cont.) Sets port 1 offhook, listens to the dial tone.
Sets port 1 and port 2 offhook, dials the number of port 2, listens to the busy tone.
Sets port 1 offhook, dials the number of port 2, listens to the Ringback tone.
Sets port 1 offhook, dials an invalid number, listens to the reorder tone. CPT Wizard - Recording Screen The wizard then analyzes the recorded Call Progress Tones. 4 Call Progress Tones are saved as standard A-law PCM at 8000. CPT Wizard - Manual Mode Manual mode enables you to record and analyze tones in addition to those tones analyzed when in automatic mode. Converting ini to dat Convert the ini (text) file to binary mode using the Dconvert software. Chapter VI
DSP Core Dump DSP Core Dump Q: What is a DSP Core Dump? A: A DSP Core Dump is a file that is created after a DSP restart occurs (indicated in the syslog). It is used to debug DSP restart problems.
DSP Core Dump This option is available: On version 4.8 from baseline F4.80.052.002 On version 5.0 from baseline F5.00.026.007 On all 5.2 versions (and higher).
The DSP dump file will be replaced automatically (by another DSP restart event) only after the previous DSP core dump file was downloaded.
To download the DSP Core Dump file: 1. Access the FAE page. 2. Press the Get DSP MON File link, and download the current Core Dump file.