Connection
BRKUCC-1900
Bryan Shapess
2
Agenda
• Troubleshooting Methodology
• Troubleshooting Tools
• CUC Traces
• Case Studies
– Message Waiting Indicator
– Visual Voicemail
– Single Inbox Message Delivery
• Note: Versions used for this presentation are Unity Connection 9.1.
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Troubleshooting Methodology
Where Does Unity Connection Fit?
• Unity Connection Integrates with:
– CUCM via SIP or SCCP
– Third party PBX via PIMG/TIMG
• Can optionally use External LDAP Directory
• Can optionally store copy of messages in External Message Store
Message Unity
Store Directory
Telephony
(CUCM, SIP, PIMG)
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Start by Gathering Information
• Detailed problem description
– Exact time (last week, yesterday, this morning around 9:00)
– What was the user doing? (What options did the user select or what digits were
entered?)
– What did the user hear?
– What did the user expect to happen?
• Determine frequency
– Is it reproducible?
– How often does it occur?
– Is there a Pattern?
• Isolate separate issues and prioritize
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Examine Available Information
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Prepare for Next Occurrence
• Configure macro and/or micro traces
• Notify user(s) of what information to capture
– Time of issue
– What they were trying to do
– Digits entered
– Recording(s) heard
• Be prepared to collect traces
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Troubleshooting Tools
Troubleshooting Tools
• Remote Port Status Monitor
• Cisco Utilities Data Link for Informix (CUDLI)
• Real Time Monitoring Tool (RTMT)
• Packet Capture
• EWS Editor
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Remote Port Status Monitor
• Available on
CiscoUnityTools.com
• Alternates color with
each new call
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Cisco Utilities Data Link for Informix (CUDLI)
• Found on www.ciscounitytools.com
• Read-only access to UnityDirDB, UnityRptDB, UnityDynDB and UnityMBX1DB
• Provides additional information on tables and columns within those tables
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
CUDLI: Column Info
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
CUDLI: Mouse Over for More Information
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
CUDLI: Options
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
CUDLI: Search
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
CUDLI: Search
Search Results
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
CUDLI: Has a Built-In Query Builder
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
RTMT
• Downloaded from Cisco Unity Connection Administration System Settings
Plug-ins
• Provides performance monitoring data (CPU, memory, disk usage, etc.)
• Provides alerting capabilities
• Syslog Viewer provides logging information for the server similar to Windows
Event Viewer
• Trace and Log Central is used to collect trace files
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Syslog Viewer Sample
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Syslog Viewer Sample
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
RTMT: Collect Files
Select services as
needed.
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
RTMT: Collect Files Cont’d
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
RTMT: Collect Files
• Select Date/Time Range
• Specify the Download File Directory
• Select if you want to Zip Files
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
RTMT: Scheduled Trace Collection
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
RTMT: Scheduled Trace Collection
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
RTMT: Scheduled Trace Collection
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
RTMT: Scheduled Trace Collection
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
RTMT: Scheduled Trace Collection
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Packet Capture
Why would you use a packet capture?
• To see packet headers and payload
• Help diagnose issues when traces don’t seem to help
• Provide product neutral data
• Make sure packets are making it to intended destination
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Packet Capture
• Built-in to Unity Connection admin cli
• Syntax:
– utils network capture [page] [numeric] [file fname] [count num] [size bytes] [src
addr] [dest addr] [port num] [host protocol addr]
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Packet Capture
• Can download packet
capture from RTMT or
from CLI
• Must analyze packet
capture in 3rd party
tool
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Packet Capture
• To view available capture files from CLI
admin:file list activelog platform/cli
cuc.cap
dir count = 0, file count = 1
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
EWS Editor
• A Microsoft tool which simulates the EWS connection to Exchange
• Helps diagnose Unity Connection single inbox connection issues to Exchange
• https://ewseditor.codeplex.com/
• Not TAC supported
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
EWS Editor
• Service URL:
– http(s)://<Exchange CAS
Server>/EWS/Exchange.asmx
• Requested Exchange Version:
– Exchange2007_SP1
• Credentials:
– Same credentials entered in CUC for
Unified Messaging Service
• Impersonation:
– Email Address of user’s mailbox
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
EWS Editor
• Will receive this error if you
are:
– Using self signed certificate on
Exchange
– Don’t have root certificate
trusted on workstation
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
EWS Editor
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
EWS Editor
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
CUC Traces
Macro Traces: Unity Connection
• A collection of preselected micro
traces
• Used to troubleshoot general areas
of functionality
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Micro Traces: Unity Connection
• Have to select one micro trace
grouping at a time
• Used for more in depth
troubleshooting
Select
Component
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Micro Traces: Unity Connection
Check
Individual
Tracing Levels
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
What Traces Should I Enable?
• Diagnostic Traces section in Troubleshooting Guide -
http://www.cisco.com/en/US/docs/voice_ip_comm/connection/9x/troubleshooting
/guide/9xcuctsg010.html
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
What Traces Should I Enable?
• When in doubt enable the following
Macro Traces:
– Call Flow Diagnostics
– Call Control (Miu) Traces
– Conversation Traces
• Collect Traces from RTMT:
– Connection Conversation Manager
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Unity Connection Traces
• UC traces end with extension .uc
• Plain text
• Format:
– 16:13:57.240 |29183,CUCM-SIP-1-
002,EE862E7085FA47719E2349568CBE8332,CDE,1,aDtmfAccessId = 1008
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
UC Trace Deciphered
• Format is:
• Time | Thread, Port, Call GUID, Component, Level, Function or Process
• Trace shows:
–16:13:57.240 |29183,CUCM-SIP-1-
002,EE862E7085FA47719E2349568CBE8332,CDE,1,aDtmfAccessId = 1008
• Event occurred at 4:13 PM
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
UC Trace Deciphered
• Format is:
• Time | Thread, Port, Call GUID, Component, Level, Function or Process
• Trace shows:
–16:13:57.240 |29183,CUCM-SIP-1-
002,EE862E7085FA47719E2349568CBE8332,CDE,1,aDtmfAccessId = 1008
• This process uses thread 29183
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
UC Trace Deciphered
• Format is:
• Time | Thread, Port, Call GUID, Component, Level, Function or Process
• Trace shows:
–16:13:57.240 |29183,CUCM-SIP-1-
002,EE862E7085FA47719E2349568CBE8332,CDE,1,aDtmfAccessId = 1008
• Call involved Port 2 on Port Group CUCM-SIP-1
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
UC Trace Deciphered
• Format is:
• Time | Thread, Port, Call GUID, Component, Level, Function or Process
• Trace shows:
– 16:13:57.240 |29183,CUCM-SIP-1-
002,EE862E7085FA47719E2349568CBE8332,CDE,1,aDtmfAccessId = 1008
• The Call GUID is a unique identifier; using a grep tool, you can filter all
associated traces for this call
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
UC Trace Deciphered
• Format is:
• Date Time | Thread, Port, Call GUID, Component, Level, Function or Process
• Trace shows:
–16:13:57.240 |29183,CUCM-SIP-1-
002,EE862E7085FA47719E2349568CBE8332,CDE,1,aDtmfAccessId = 1008
• Low Level (1) Conversation Engine (CDE) trace
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
UC Trace Deciphered
• Format is:
• Date Time | Thread, Port, Call GUID, Component, Level, Function or Process
• Trace shows:
–16:13:57.240 |29183,CUCM-SIP-1-
002,EE862E7085FA47719E2349568CBE8332,CDE,1,aDtmfAccessId = 1008
• The DTMF Access ID (extension) is 1008. This happens to be the calling
number for this call.
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Case Studies
53
Case Study 1 – MWI Failure
• Problem Description:
– A user reports that the red light on the phone is off but when they called Unity
Connection it said they had a new message
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Case Study 1 – MWI Failure
• What trace to enable?
– Macro Traces
• Call Control (Miu) Traces
• Traces for MWI problems
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Case Study 1 – MWI Failure
• Open diag_CuNotifier and search for Failed.
Phone
Systems with
port count
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Case Study 1 – MWI Failure
View with
subscribers’
and their
phone system
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Case Study 1 – MWI Failure
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Case Study 1 – MWI Failure
• Change phone system to proper phone system.
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Case Study 2 – Visual Voicemail Failure
• Problem Description:
When trying to play a message using Visual Voicemail you receive an error “No Line
Available To Play Message”. Issue happened around 8:55pm
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Case Study 2 – Visual Voicemail Failure
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Case Study 2 – Visual Voicemail Failure
• What trace to enable?
– Macro Traces
• Call Control (Miu) Traces
– Micro Traces
• VMWS (All)
– Phone Service debug logging
– CallManager Detailed traces
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Case Study 2 – Visual Voicemail Failure
• Set Phone Service VisualVoicemail log_level to debug
• Collect console logs from phone web page
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Case Study 2 – Visual Voicemail Failure
• Find calls to Reverse Trap Number
• Locate call which matches time user experienced problem
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Case Study 2 – Visual Voicemail Failure
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Case Study 2 – Visual Voicemail Failure
• Failure caused by defect CSCuj81059
– Symptom:
• On RT phones, having phone firmware 9.2.2 onwards, when line having a busy line/shared line the
VVM does not play and gives the following error: "No line available to play message“
– Resolved:
• 8.6.2SU4
• 9.1.2ES27
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Case Study 3 – Single Inbox Failure
• Problem Description:
A user reports that when they get a new message it doesn’t show up in their Outlook
Inbox
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Case Study 3 – Single Inbox Failure
• What traces to enable?
– Macro: Single Inbox Trace
– Micro: EWS (All)
• What traces to retrieve?
– Connection Mailbox Sync
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Case Study 3 – Single Inbox Failure
• Open trace file CuMbxSync
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Case Study 3 – Single Inbox Failure
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Case Study 3 – Single Inbox Failure
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Case Study 3 – Single Inbox Failure
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
Case Study 3 – Single Inbox Failure
• Check Exchange for users whose mailbox is over quota
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Case Study 3 – Single Inbox Failure
• To better understand single inbox messages, review single inbox annotated logs
• http://www.ciscounitytools.com/Training/Connection/CUC8_5.html
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Complete Your Online Session Evaluation
• Give us your feedback and you
could win fabulous prizes. Winners
announced daily.
• Complete your session evaluation
through the Cisco Live mobile app
or visit one of the interactive kiosks
located throughout the convention
center.
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Continue Your Education
• Demos in the Cisco Campus
• Walk-in Self-Paced Labs
• Table Topics
• Meet the Engineer 1:1 meetings
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 77