IBM Software Group

Troubleshooting WebSphere Message Broker Hints & Tips

Prasad Imandi WebSphere Message Brokers Level 2 Support, IBM imandi@us.ibm.com

WebSphere Support Technical Exchange

IBM Software Group

Notes
The purpose of this presentation is to discuss various troubleshooting techniques available in WebSphere Message Broker.

WebSphere Support Technical Exchange

IBM Software Group

Contents
How does the broker work ? Which log files do I check ? Tracing message flows Debugging message flows Collecting message flow statistics Advanced troubleshooting MustGather Information Websites to bookmark !!!

WebSphere Support Technical Exchange

IBM Software Group

Notes
The presentation starts with a brief introduction to WebSphere Message Broker. We will look at the various troubleshooting techniques available, which are not only helpful in problem determination, but also useful to check the health of the broker as well as verifying your flow logic and gather simple performance measurements. Finally there are a few must bookmark websites listed, that are a good source of information on WebSphere Message Broker.

WebSphere Support Technical Exchange

IBM Software Group

How does the broker work ?

WebSphere Support Technical Exchange

IBM Software Group

Notes
This chart gives a quick introduction to all the components in WebSphere Message Broker and shows the connectivity and flow of information between these components. The Toolkit in WebSphere Message Broker or the Control Center in WMQI V2.1 is a single user interface provided to developers and adminstrators. Developers use this tool to build the message flows and messages as well as use the built in debugger for analysis and problem determination. Administrators use this tool for deploys, administration of broker domain. The Configuration Manager is an administrative component and manages a broker domain. A broker domain may consistent of one or more brokers. Configuration Manager communicates with the Toolkit and the broker in order to manages all administrative actions issued from the Toolkit. In WMB V6, the Configuration Manager does not require an external database. In WBI Brokers V5, the Configuration Manager has an administrative database to store administrative data, while in WMQI V2.1, the Configuration Manager has a configuration database and message repository, which holds the message flow and message definitions. Broker is the run-time component that runs the message flows and processes messages. The broker has one administrative process and one or more execution group processes. The administrative process manages all the execution groups defined in the broker. Each execution group process runs the actual message flows which process messages.

WebSphere Support Technical Exchange

IBM Software Group

Which log files do I check?

Messages and Errors

Administrative and runtime

Event Viewer Application log on Windows syslog on Unix, needs to be configured z/OS
WMQI V2.1, USS SYSLOG needs to be configured WBI Brokers V5 and WMB V6, console log or joblog for address space Component start/stop, exceptions, message processing exceptions, Trace node output if destination configured as Local Error Log

WebSphere Support Technical Exchange

IBM Software Group

Notes
Windows event viewer application log events are logged with the following event source WMQI V2.1 WMQIv210 WBI Brokers V5 BIPv500 WMB V6 - WebSphere Broker v6000 Unix and z/OS, messages are reported in the form of BIPnnnn where nnnn is the diagnostic message number Diagnostic message can be looked up on the below website http://publib.boulder.ibm.com/infocenter/wmbhelp/v6r0m0/topic/com.ibm.etools.mft.bipmsgs.do c/amsgtop_.htm On Unix, syslog needs to be configured by adding the following command to /etc/syslog.conf user.info /var/adm/user.log The file /var/adm/user.log must exist, and syslog daemon must be restarted to pick up the above command.

WebSphere Support Technical Exchange

IBM Software Group

Which log files do I check?

Messages and Errors

Application errors, Status messages Application errors

Toolkit Tasks view, Problems view in Broker Application Development perspective Logic errors in message flows, ESQL, message sets

Status messages
Toolkit Alerts view or Operations view in WMQI V2.1 Control Center Status of Broker, Execution Group, Message flows

WebSphere Support Technical Exchange

IBM Software Group

Which log files do I check?

Deploy messages Response messages for a deploy are logged in Toolkit broker
domain event log or Log view in WMQI V2.1 Control Center

Environment variable MQSI_SHOW_DEPLOY provides

information on the progress of deploy export MQSI_SHOW_DEPLOY=1 Broker logs BIP8099I message to event viewer/syslog at every logical step of deploy operation Helps isolating problem Broker restart needed to pick up environment variable

WebSphere Support Technical Exchange

10

IBM Software Group

Which log files do I check?

WebSphere Support Technical Exchange

11

IBM Software Group

Tracing
User Level Flow Trace

Traces the path of message in your flow Can be enabled from Toolkit or command line Must be formatted on the broker machine Time stamp can be used to find time taken in each node performance

Time stamp in trace could be used to check flow Trace is circular

WebSphere Support Technical Exchange

12

IBM Software Group

Notes
Start trace command mqsichangetrace <Broker> -u -e <Execution Group> -f <Msg Flow> -l <level> -r -c <trace file size> Note, the default trace file size is 4 MB. Its recommended this value be increased to an appropriate value to avoid trace getting overwritten, as it is circular. Stop trace command mqsichangetrace <Broker> -u -e <Execution Group> -f <Msg Flow> -l none Format trace commands mqsireadlog <Broker> -u -e <Execution Group> -f o trace.xml Mqsiformatlog -l trace.xml -o trace.log

WebSphere Support Technical Exchange

13

IBM Software Group

Tracing
Trace snippet
Timestamp Thread Id

2003-07-27 14:29:28.547000 3540 UserTrace BIP2632I: Message received and propagated to 'out' terminal of MQ input node 'HextoASCII.MQInput1'.
Flow name Node name

2003-07-27 14:29:28.636999 3540 Recoverable Exception BIP2549E: (16, 1) : Type mismatch assigning to variable; value ''test'' inconsistent with type 'BLOB'. The source expression produced a value of ''test'' but the target variable was declared with a type of 'BLOB'. Ensure that only values of the appropriate data type are assigned to declared variables.

WebSphere Support Technical Exchange

14

IBM Software Group

Tracing
Trace snippet
2004-02-06 15:08:29.685880 17 UserTrace BIP4080I: Message propagated to try terminal from try-catch node 'INTERFACE.WMQI_AUDIT_SINGLEDB1.TryCatch1'. 2004-02-06 15:08:39.725772 17 UserTrace BIP4124I: Message propagated to 'out' terminal of compute node 'INTERFACE.WMQI_AUDIT_SINGLEDB1.insert_audit_db'.

WebSphere Support Technical Exchange

15

IBM Software Group

Debugging message flows

Trace Node

Primitive node to trace content of message in the flow Add to the message flow and configure
${Root}, ${Body}, ${ExceptionList}, ${LocalEnvironment}

Writes the message tree out, at the point in the flow Snapshot of Properties, Headers, and the message tree Destination can be User Trace, File, Local Error Log Destination Local Error Log is limited to 1024 bytes on Unix

WebSphere Support Technical Exchange

16

IBM Software Group

Notes
Sample Trace node output :
(0x01000000):Properties =( (0x03000000):MessageSet = P258VC4002001' (0x03000000):MessageType = IN VOICE' (0x03000000):MessageFormat = 'XML'. . . (0x01000000):MQMD =( = 'TESTIN' = TRUE

(0x03000000):SourceQueue (0x03000000):Transactional

(0x03000000):Encoding = 546 (0x03000000):CodedCharSetId = 1208 (0x03000000):Format . . (0x01000010):XML =( (0 x01000000):MESSAGE = ( (0 x02000000): = ELEMENT1' ) = 'MQSTR

WebSphere Support Technical Exchange

17

IBM Software Group

Debugging message flows

Flow Debugger

Unit testing of the message flow by developer Validation of message flow logic Displays message tree at run-time Stop the flow at any point by adding a breakpoint Alter the message at breakpoint Track single message through a flow using breakpoints ESQL, Java, Mapping nodes can be debugged Debug perspective in Toolkit
18

WebSphere Support Technical Exchange

IBM Software Group

Debugging message flows

WebSphere Support Technical Exchange

19

IBM Software Group

Debugging message flows

WebSphere Support Technical Exchange

20

IBM Software Group

Debugging message flows

Flow Debugger

In Debug configuration wizard, ensure IBM Agent Controller

port number is correct

Must be the same port configured on Agent Controller server Default port on Agent Controller can be changed in
<RACInstallRoot>/config/serviceconfig.xml
See notes for configuration

Default port on Toolkit can be changed in

Window -> Preferences -> Message Broker Debug

WebSphere Support Technical Exchange

21

IBM Software Group

Notes
Agent Controller runs on the broker machine and facilitates debugging functionality by controlling the debugging communications between the Toolkit and the execution group. The port used by Agent Controller can be changed in the configuration file <AgentController InstallRoot>/config/serviceconfig.xml as follows: Search for element AgentControllerConfiguration, and change the value for the attribute port to the new port number. Note, Agent Controller must be restarted to pick up this change.

WebSphere Support Technical Exchange

22

IBM Software Group

Debugging message flows

WebSphere Support Technical Exchange

23

IBM Software Group

Debugging message flows

Flow Debugger

Do you have a firewall between Toolkit and Agent Controller ?

All ports need to be open between Toolkit and Agent Controller Agent Controller V6.0.1 allows communication through a firewall
See notes for configuration

Ensure port used for communication is open on the firewall

Ensure all hosts can connect to Agent Controller

Can be changed in the configuration file <RACInstallRoot>/config/serviceconfig.xml
See notes for configuration

Agent Controller and Broker must be running for flow

debugging to work
WebSphere Support Technical Exchange 24

IBM Software Group

Notes
Agent Controller V6.0.1 allows communication between Toolkit and Agent Controller server through a firewall. To enable this, change the configuration file <AgentController InstallRoot>/config/serviceconfig.xml as follows: Search for element AgentControllerConfiguration, and change value for the attribute isDataMultiplexed from false to true value for the attribute filePort from 10005 to 0 Note, Agent Controller must be restarted to pick up this change. During installation of Agent Controller, you must select the hosts that are allowed for communicating with the server. You can select all hosts, or provide the names of hosts that are allowed. This can be changed at anytime as follows: In configuration file serviceconfig.xml, change element <Allow host="LOCAL"></Allow> to <Allow host="ALL"></Allow> Note, Agent Controller must be restarted to pick up this change.

WebSphere Support Technical Exchange

25

IBM Software Group

Debugging message flows

Flow Debugger

If debugging a Java node, setup JVM debug port in Debug

configuration wizard

Ensure the check box Debug Java Source Code is checked Add source java project in Source panel in Debug configuration
wizard

Debugger will look for source code as configured in Source

panel

WebSphere Support Technical Exchange

26

IBM Software Group

Debugging message flows

WebSphere Support Technical Exchange

27

IBM Software Group

Debugging message flows

WebSphere Support Technical Exchange

28

IBM Software Group

Debugging message flows

Flow Debugger

JVM in execution group running the message flow must be

configured for the same debug port number as debugger
mqsichangeproperties BrokerName -e EGName -o ComIbmJVMManager n jvmDebugPort -v portNumber

JVM debug port must be unique Requires a broker restart

WebSphere Support Technical Exchange

29

IBM Software Group

Debugging message flows

Flow Debugger

Ensure message flow is deployed before debugging

Process a test message

Open message flow to add breakpoints Add a breakpoint before ESQL, Java, Mapping nodes, subflow
to debug

Step into code to debug ESQL, Java code Breakpoints can be set in ESQL, Java code, subflow Step into to debug a subflow Process a message to start debugging
30

WebSphere Support Technical Exchange

IBM Software Group

Collecting message flow statistics

Accounting and Statistics

Records dynamic information about runtime

behaviour of message flow

Provides information at the message flow, thread,

node and terminal level

Flow performance can be measured No changes needed to request statistics collection Enabled using command mqsichangeflowstats Output available in user trace, XML publication message or SMF117 records on z/OS
31

WebSphere Support Technical Exchange

IBM Software Group

Collecting message flow statistics

BIP2380I: WBIMB message flow statistics. ProcessID='2704', Key='21', Type='SnapShot', Reason='SnapShot', BrokerLabel='WMQI', ExecutionGroupName='default', MessageFlowName=BRIDEPL21', StartDate='2004-08-03', StartTime='22:33:59.749', EndDate='2004-08-03', EndTime='22:34:22.191', TotalElapsedTime='3096000', MaximumElapsedTime='1083000', MinimumElapsedTime='1000', TotalCPUTime='20029', MaximumCPUTime='10015', MinimumCPUTime='10014', CPUTimeWaitingForInputMessage='0', ElapsedTimeWaitingForInputMessage='19338000', TotalInputMessages='4', TotalSizeOfInputMessages='1532', MaximumSizeOfInputMessages='383', MinimumSizeOfInputMessages='383', NumberOfThreadsInPool='1', TotalNumberOfMQErrors='0', TotalNumberOfErrorsProcessingMessages='3', TotalNumberOfCommits='0', TotalNumberOfBackouts='0'.
32

WebSphere Support Technical Exchange

IBM Software Group

Notes
Sample command to start collection of statistics for a specific message flow mqsichangeflowstats <Broker> -s -e <EGName> -f <Message flow> -c active -t basic -n advanced -o usertrace This command enables the broker to produce accounting and statistics information for the specified message flow. The information produced will contain basic threadData and advanced nodeData. The accounting and statistics information generated will be written to a user trace, hence will need to be formatted using commands as follows: mqsireadlog <Broker> -u -e <EGName> -f o trace.xml mqsiformatlog -i trace.xml -o trace.log The command to stop producing the above accounting and statistics information is as follows: mqsichangeflowstats <Broker> -s -e <EGName> -f <Message flow> -c inactive -t basic -n advanced -o usertrace A re-deploy of the message flow with automatically cancel accounting and statistics on that flow.

WebSphere Support Technical Exchange

33

IBM Software Group

Advanced troubleshooting
Abend file

Generated in MQSI_WORKPATH/errors directory when

execution group terminates due to a severe error

Shows the stack back trace leading to the failure WMB V6 abend file logs the message flow name that caused
the severe error

Periodic clean up of MQSI_WORKPATH/errors directory helps

quickly correlate an abend to a problem

WebSphere Support Technical Exchange

34

IBM Software Group

Notes
+-----------------------------------------------------------------------------+ | First Failure Symptom Report | ======================== | Proc start time :- Mon May 01 09:46:01 2006 | Product Details | Vendor | Product Name | Program ID | Version . . | Failure Location | ++++++++++++++++ | Time of Report :- Mon May 01 10:02:03 2006 | Message Flow :- INVOICEPROC_ITEMS | Thread ID :- 0x0000126C . . | | | | | :- IBM :- WebSphere Message Brokers :- 5724-J04 :- 6001 | | | | | | | |

WebSphere Support Technical Exchange

35

IBM Software Group

Advanced troubleshooting
Service level tracing

Broker startup tracing allows tracing at startup

Must be enabled only if requested by IBM support Service level tracing allows internal tracing of message flow, execution group, broker Must be enabled only if requested by IBM support Performance intensive

WebSphere Support Technical Exchange

36

IBM Software Group

MustGather Information
Information requested by IBM support

Product maintenance level

mqsiservice -v Broker environment

Event log/syslogs Abend files if any related to the problem

MustGather website
http://www-1.ibm.com/support/docview.wss?rs=849&context=SSKM8N&context =SSKMAB&context=SS3GH2&q1=mustgather&uid=swg21209857&loc=en_US&cs=utf-8&lang=en

WebSphere Support Technical Exchange

37

IBM Software Group

Websites to bookmark !!!

WMB Home Page
http://www-306.ibm.com/software/integration/wbimessagebroker/

WMB support website

http://www-306.ibm.com/software/integration/wbimessagebroker/ support/

Supported Software
http://www-306.ibm.com/software/integration/websphere/mqplatforms/ supported.html

WebSphere Support Technical Exchange

38

IBM Software Group

Websites to bookmark !!!

Support Packs
http://www-306.ibm.com/software/integration/support/supportpacs/

Books and Manuals

http://www-306.ibm.com/software/integration/mqfamily/library/manualsa/

WMQ support website

http://www-306.ibm.com/software/integration/wmq/support/

WebSphere Support Technical Exchange

39

IBM Software Group

Websites to bookmark !!!

Red Books
http://www.redbooks.ibm.com/

News Groups
news://news.software.ibm.com/ibm.software.websphere.mqintegrator/ http://www.mqseries.net/

IBM Software Support Lifecycle

http://www-306.ibm.com/software/info/supportlifecycle/

WebSphere Support Technical Exchange

40

IBM Software Group

Additional WebSphere Product Resources

Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at: www.ibm.com/developerworks/websphere/community/ Learn about other upcoming webcasts, conferences and events: www.ibm.com/software/websphere/events_1.html Join the Global WebSphere User Group Community: www.websphere.org Access key product show-me demos and tutorials by visiting IBM Education Assistant: ibm.com/software/info/education/assistant Learn about the Electronic Service Request (ESR) tool for submitting problems electronically: www.ibm.com/software/support/viewlet/probsub/ESR_Overview_viewlet_swf .html Sign up to receive weekly technical support emails: www.ibm.com/software/support/einfo.html

WebSphere Support Technical Exchange

41

IBM Software Group

WebSphere Support Technical Exchange

42