Aw PSD FMW Wls Java Thread Dumps v7

Upcoming: Advisor Webcast:Troubleshooting WebLogic Server Thread Dumps
© 2011 Oracle Corporation – Proprietary and Confidential 1

AGENDA
• Presentation – approximately 45 minutes

• Q&A Session – approximately 15 minutes
• Web attendees can ask questions via Q&A panel

• Phone attendees can ask questions via Q&A panel or
phone (operator assisted)
© 2012 Oracle Corporation – Proprietary and Confidential

2
ATTENTION – AUDIO Options
You can
• either listen the audio broadcast on your computer
• or join teleconference (dial in)

3
Voice Streaming – Audio Broadcast
• Listen only mode

• Advantage: no need to dial in
• What about Questions?
Type your questions into WebEx Q&A panel
• If you prefer full audio access in order to
ask questions directly, please connect to
our teleconference
• Connect details on next slide

4
ATTENTION – AUDIO INFORMATION
Teleconference Connect details:

1. Conference ID: 70793487
2. International dial in:
UK: +44 (0) 1452 562 665
US: 1866 230 1938
3. List with national toll free numbers is available in
note 1148600.1
You can view this info anytime during the conference using
Communicate > Teleconference > Join Teleconference
from your WebEx menu

5
<Insert Picture Here>
Troubleshooting WebLogic Server Thread Dumps

Laurent Goldsztejn, Proactive Support
Twitter @weblogicsupport
6
AGENDA
• Presentation – approximately 45 minutes

• Q&A Session – approximately 15 minutes
• Web attendees can ask questions via Q&A panel only

• Phone attendees can ask questions via Q&A panel or phone
(operator assisted)

7
AUDIO INFO - Teleconference

UK: +44 (0) 1452 562 665
US: 1866 230 1938
note 1148600.1

8
3
Send your question
2
Who to ask?
Q&A panel
1
type your question here

9
your question pop-up here

10
Safe Harbor Statement
The following is intended to outline our general

product direction. It is intended for information
purposes only, and may not be incorporated into
any contract. It is not a commitment to deliver any
material, code, or functionality, and should not be
relied upon in making purchasing decision. The
development, release, and timing of any features
or functionality described for Oracle’s products
remains at the sole discretion of Oracle.

11
<Insert Picture Here>
Troubleshooting WebLogic Server Thread Dumps

Laurent Goldsztejn, Proactive Support
Twitter @weblogicsupport
12
Agenda
• When are thread dumps needed?

• Troubleshooting Thread Dumps: basics
• Deconstructing Thread Dumps (DEMO)
• Analysis tools (DEMO)
• Issues observed: examples
• What to do next?
• Other resources
• Q/A

13
When are thread dumps needed?

14
Thread Dumps
• If you found some stuck threads in the server logs,

then Thread Dumps are the most important piece of
information required to narrow down where the
problem lies
• A Java Thread Dump is a snapshot that reports what

every thread in the process is doing at a particular
point in time. It’s an essential debugging tool
• Thread dumps do not give a direct solution but they

help in identifying problems and hotspots

15
When Threads need to be dumped
• The server does not respond to new requests
• Requests time out
• Requests take longer and longer to process
• The server is no longer reported as running in

the Admin Console but its state changed
to unknown or failed

16
Different ways to take thread dumps… [ID 1098691.1]
screenshot – truncated – please go to note for full text

17
Different ways to take thread dumps… [ID 1098691.1]
screenshot – truncated – please go to note for full text

18
Troubleshooting Thread Dumps: basics

19
How to Read Thread dumps?
• Thread dumps could contain a lot of data. Trying to

read them in Text editor can be difficult
• Different JVM Vendors display the data in different

formats (markers for start/end of thread dumps,
reporting of locks and thread states, method
signatures) but the underlying data (stack) exposed
remains the same across vendors
• Many free tools help analyzing thread dumps

20
What does a Thread Dump contain?
• Date and time the dump was created as well as JVM

detailed info (version, architecture, mode…)
• Each thread is dumped with its complete java stack that

indicates the packages, classnames and method calls
involved
• All this gives an overview of the thread execution and the

nature of each method invocation
• What is getting executed by the thread can be understood

even by non developers

21
Thread Dump Analysis - High level steps
1. Make sure at least 3 Thread Dumps, 10 seconds apart,

are collected when the server hang issue is observed.
• That’s a way to identify patterns among threads and
see if some end up moving with their tasks
2. Collect Thread Dumps from all the servers involved
• Other clustered managed servers
• Managed Servers from other domains which may be
interacting with your domain
3. Look first at default and custom execute threads
4. Check if some execute threads are stuck in the same
state across multiple thread dumps

22
Thread Dump Analysis - High level steps
5. Ignore the idle threads

6. Identify the execute threads that are not moving along
and try to understand the reason behind that by looking at
the last method and java class invoked
7. In most cases, WebLogic internal threads such as Muxer
threads, listen threads and other threads spawned by the
Application should be ignored when investigating a
problem from a WLS perspective.
8. Check all the thread dumps from different managed
servers to find potential relationship between the stuck
threads
Example: A thread from one server could be waiting on another
thread on another managed server

23
Deconstructing Thread Dumps
DEMO

24
Thread states
• Idle Threads:
Available threads to process work request for any work manager
• Stuck Threads:
Threads blocked by other threads. They might be waiting for
some other threads to complete their operations. Deadlock
condition results in stuck threads.
• Hogging Threads:
Threads that are taking a very long time. WLS will generally
declare a thread as HOGGING before it is declared as stuck
(after the configured timeout is reached).
• Standby Threads:
Threads that are in Sleep. Those threads won't do anything as it
is in a state of monitoring.
• Active Threads: Threads actively working on requests.

25
Locks
• Locks are used to prevent that more than one

thread can access and work on the same object
• Only one thread can own a Lock object at a time
• Lock objects support a wait/notify mechanism
• With little contention a lock is considered Thin,

with more contention, it’s promoted to Fat Lock

26
Diagnosis Tools

27
Tools
• Samurai is a light weight open source tool which runs

as a Java Web Start application or from a command
prompt. It allows to identify thread dumps from stdout
and visualize them. It also helps in visualizing heap
usage and GC activities (verbose:gc data)
http://yusuke.homeip.net/samurai/en/index.html
• TDA (Thread Dump Analyzer) a is a Swing GUI for

analyzing Thread Dumps and Heap Information
http://java.net/projects/tda/pages/Home
• ThreadLogic built upon TDA by adding logic for

common patterns found in application servers.
28
DEMO - Samurai

29
DEMO - TDA

30
Observed Issues: Examples

31
Thread Dumps - Examples of thread
hanging on SocketRead/SocketWrite
[STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" id=76 idx=0x128 tid=15140 prio=1
alive, in native, daemon
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:46)[optimized]
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined]
at java/net/SocketInputStream.read(SocketInputStream.java:129)[optimized]
at java/io/FilterInputStream.read(FilterInputStream.java:111)[inlined]
at
com/wily/introscope/agent/probe/net/ManagedSocketInputStream.read(ManagedSocketInputStream.java:214)[opti
mized]
at com/sun/mail/util/TraceInputStream.read(TraceInputStream.java:97)
at java/io/BufferedInputStream.fill(BufferedInputStream.java:218)[inlined]
at java/io/BufferedInputStream.read(BufferedInputStream.java:235)[optimized]
^-- Holding lock: java/io/BufferedInputStream@0x291bf2b8[thin lock]
at com/sun/mail/util/LineInputStream.readLine(LineInputStream.java:75)
at com/sun/mail/smtp/SMTPTransport.readServerResponse(SMTPTransport.java:1440)
at com/sun/mail/smtp/SMTPTransport.openServer(SMTPTransport.java:1260)
at com/sun/mail/smtp/SMTPTransport.protocolConnect(SMTPTransport.java:370)
at javax/mail/Service.connect(Service.java:275)
at javax/mail/Transport.send0(Transport.java:168)
…

32
Thread Dumps - Example of thread stuck in JDBC call
<[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)' id=76

idx=0x128 tid=15140 prio=1 alive, in native, daemon
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
oracle.net.ns.Packet.receive(Unknown Source)
oracle.net.ns.DataPacket.receive(Unknown Source)
oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
oracle.net.ns.NetInputStream.read(Unknown Source)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1099)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1070)
oracle.jdbc.driver.T4CTTIoauthenticate.receiveOauth(T4CTTIoauthenticate.java:751)
oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:362)
oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:420)
oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
java.sql.DriverManager.getConnection(DriverManager.java:525)
java.sql.DriverManager.getConnection(DriverManager.java:140)
com.foo.sbm.util.ConnectionPool.createConnection(Unknown Source)
…

33
Thread Dumps - Examples of thread stuck in RMI call
"ExecuteThread: '52' for queue: 'default'" daemon prio=5 tid=0x4b3e40b0 nid=0x1170 waiting on
monitor [0x4c74f000..0x4c74fdbc]
at java.lang.Object.wait(Native Method)
at weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:72)
at weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:97)
at weblogic.rmi.internal.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:80)
at weblogic.rmi.cluster.ReplicaAwareRemoteRef.invoke(ReplicaAwareRemoteRef.java:262)
at weblogic.rmi.cluster.ReplicaAwareRemoteRef.invoke(ReplicaAwareRemoteRef.java:229)
at weblogic.rmi.internal.ProxyStub.invoke(ProxyStub.java:35)
at $Proxy6.lookup(Unknown Source)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:341)
atweblogic.servlet.internal.HttpServer.lookupROIDS(HttpServer.java:789)
at weblogic.servlet.security.internal.SecurityModule.getCurrentUser(SecurityModule.java:207)
at weblogic.servlet.security.internal.SecurityModule.checkAuthenticate(SecurityModule.java:235)
at weblogic.servlet.security.ServletAuthentication.weak(ServletAuthentication.java:271
at weblogic.servlet.security.internal.ServletSecurityManager.checkAccess(ServletSecurityManager.ja
va:124)at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.jav
a:2518)
at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2260)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)

34
Thread Dumps - Examples of thread stuck in LDAP
call awaiting LDAP response
"[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10

tid=0x000000004c8a2000 nid=0x7559 in Object.wait() [0x000000004296c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaaeced3718> (a netscape.ldap.LDAPSearchListener)
at java.lang.Object.wait(Object.java:485)
at netscape.ldap.LDAPMessageQueue.waitForMessage(LDAPMessageQueue.java:200)
- locked <0x00002aaaeced3718> (a netscape.ldap.LDAPSearchListener)
at netscape.ldap.LDAPMessageQueue.waitFirstMessage(LDAPMessageQueue.java:101)
- locked <0x00002aaaeced3718> (a netscape.ldap.LDAPSearchListener)
at netscape.ldap.LDAPConnection.sendRequest(LDAPConnection.java:1796)
- locked <0x00002aaaeced3818> (a netscape.ldap.LDAPConnection)
at netscape.ldap.LDAPConnection.search(LDAPConnection.java:2566)
at weblogic.security.providers.authentication.LDAPAtnDelegate.getDNForUser(LDAPAtnDelegate.java:3782)
at weblogic.security.providers.authentication.LDAPAtnDelegate.authenticate(LDAPAtnDelegate.java:3625)
at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:220)
at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
at java.security.AccessController.doPrivileged(Native Method)

35
Thread Dumps - Examples of idle thread
"[ACTIVE] ExecuteThread: '96' for queue: 'weblogic.kernel.Default (self-tuning)'"

id=313028 idx=0x1d0 tid=4686 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0x739b42e0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]
at java/lang/Object.wait(Object.java:474)[inlined]
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[inlined]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)[optimized]
^-- Lock released while waiting: weblogic/work/ExecuteThread@0x739b42e0[fat lock]
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace

36
What to do next?

37
Thread Requesting Locks
• If the lock is not held by another thread then the

thread becomes the new holder of the lock until it
relinquishes it
• If the lock is currently held then the thread

requesting it joins the blocked list of contenders for
this lock
• Threads holding locks should relinquish them as

soon as they stop needing them to avoid impairing
on other threads
38
Deadlocks
• Deadlocks describe a situation where two or more

threads are blocked waiting for each other (circular
dependency). Neither block will ever end because
each thread is waiting for the other
• Deadlocks can occur when multiple threads need the

same locks, at the same time, but obtain them in
different order
• Restart of the server is the only option

39
Open Lock Chains
• JRockit JVM automatically detects deadlocked,

blocked, and open lock chains among the running
threads
• Open lock chains represent a straight dependency,

thread A is waiting for B which is waiting for C, and
so on. If you have long open lock chains, your
application might be wasting time waiting for locks.
You may then want to reconsider how locks are
used for synchronization in your application

40
Deadlock detection with HotSpot
• Java HotSpot VM also provides a deadlock

detection utility invoked by a Ctrl+\ on the command
line while an application is running
• If any deadlocks are detected, additional information

after the thread dump on each deadlocked thread is
printed
• The utility detects Java-platform-level deadlocks,

including locking done from JNI, JVMPI, and JVMDI.

41
Locking Issues
• Analyze the dependencies between code and

locks
• Resolving bottlenecks due to locking
• In case of deadlocks, check the locking and try

to establish an order of locking or to avoid the
locks entirely

42
Stuck Threads
• Check health of backend database, network

communication, external services (e.g. LDAP)
or other running servers that could lead to
latencies in responding to requests
• Threads stuck in database SQL read/execute

can indicate db response issue or SQL tuning
issue

43
Stuck Threads
• Threads stuck on socket read calls indicate

slow read or backend services or resources not
responding in a timely fashion or socket
closures due to firewall or other bad socket
handling
• Threads stuck in socket connect calls indicate

an unavailable backend service or incorrect
connection info

44
Other causes of the stuck thread(s)
• Once the stuck thread(s) is/are identified, look
at the thread stack to understand what work is
being done. It could be any of the following,
but not limited to:
• EJB or WebService Call

• HTTP servlet or JSP
• JDBC call using WLS pooling connectivity
• A WebService invoking another WS
• A HTTP servlet invoking an EJB
• Application Deadlock

45
Origin of the problematic class
• If the thread is stuck on a class that’s part of

your application then you should look at the
method invoked and debug the class
• If the thread is stuck on a class outside your

control then you should check for known
defects or contact the vendor

46
GC and heap
• Make sure that GC cycles do not create

bottlenecks
• Use –verbose:GC combined with -

XX:+PrintGCDetails -XX:+PrintGCTimeStamps to
to log GC stats about each GC
• The -verbose:gc option can be dynamically

enabled at runtime using the management API or
JVM tool interface (TI)
47
Heap size
• Insure enough java heap is available especially at

peak times
• Not enough heap can lead to OutOfMemoryError

exceptions and impact the performance of the
Java application

48
JRockit/JRA
• Use JRockit Runtime Analyzer (JRA)

recordings to gather general data about the
application (GCs, Heap, Optimizations, Objects
stats etc…)

49
JConsole
• Java Monitoring and Management

Console included in JDK 5.0
• Heap Memory
• Threads
• Classes
• CPU Usage
• VM Summary
• Mbeans
TDA can be used as JConsole or VisualVM plugin
50
Other resources

51
• WebLogic Server Information Center [ID 1298857.2]
• JRockit Information Center [ID 1384214.2]
• Twitter @weblogicsupport

52
Oracle customers have reported
value-added outcomes including:
• up to 25% fewer problems
•40% faster problem
resolution, and
• dramatically streamlined
upgrades.

53
QUESTIONS
ANSWERS
54
3
Send your question
2
Who to ask?
Q&A panel
1
type your question here

55
Further Info & Help
• Advisor Webcast Archived Recordings (Doc ID 740964.1)

• Advisor Webcast Current Schedule (Doc ID 740966.1)
• MOS Community Middleware - Oracle Weblogic Server

56
57
Oracle FMW Support Newsletter - Index
1347149.1

58
Oracle FMW Support Newsletter - Current
1347075.1

59
THANK YOU

60
BACKUP

61
ATTENTION – AUDIO STREAMING IS
NOT AVAILABLE !
UK: +44 (0) 1452 562 665
US: 1866 230 1938
note 1148600.1

62

Aw PSD FMW Wls Java Thread Dumps v7

Diunggah oleh

Informasi Dokumen

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Aw PSD FMW Wls Java Thread Dumps v7

Diunggah oleh

Hak Cipta:

Format Tersedia

Upcoming: Advisor Webcast:Troubleshooting WebLogic Server Thread Dumps

© 2011 Oracle Corporation – Proprietary and Confidential 1

• Presentation – approximately 45 minutes

• Web attendees can ask questions via Q&A panel

Upcoming: Advisor Webcast:Troubleshooting WebLogic Server Thread Dumps

© 2012 Oracle Corporation – Proprietary and Confidential

Upcoming: Advisor Webcast:Troubleshooting WebLogic Server Thread Dumps

© 2012 Oracle Corporation – Proprietary and Confidential

• Listen only mode

Upcoming: Advisor Webcast:Troubleshooting WebLogic Server Thread Dumps

© 2012 Oracle Corporation – Proprietary and Confidential

Teleconference Connect details:

© 2012 Oracle Corporation – Proprietary and Confidential

Troubleshooting WebLogic Server Thread Dumps

• Presentation – approximately 45 minutes

• Web attendees can ask questions via Q&A panel only

© 2012 Oracle Corporation – Proprietary and Confidential

Teleconference Connect details:

© 2012 Oracle Corporation – Proprietary and Confidential

© 2012 Oracle Corporation – Proprietary and Confidential

© 2012 Oracle Corporation – Proprietary and Confidential

The following is intended to outline our general

© 2012 Oracle Corporation – Proprietary and Confidential

Troubleshooting WebLogic Server Thread Dumps

• When are thread dumps needed?

© 2012 Oracle Corporation – Proprietary and Confidential

© 2012 Oracle Corporation – Proprietary and Confidential

• If you found some stuck threads in the server logs,

• A Java Thread Dump is a snapshot that reports what

• Thread dumps do not give a direct solution but they

© 2012 Oracle Corporation – Proprietary and Confidential

• The server does not respond to new requests

• Requests time out

• Requests take longer and longer to process

• The server is no longer reported as running in

© 2012 Oracle Corporation – Proprietary and Confidential

screenshot – truncated – please go to note for full text

© 2012 Oracle Corporation – Proprietary and Confidential

screenshot – truncated – please go to note for full text

© 2012 Oracle Corporation – Proprietary and Confidential

© 2012 Oracle Corporation – Proprietary and Confidential

• Thread dumps could contain a lot of data. Trying to

• Different JVM Vendors display the data in different

• Many free tools help analyzing thread dumps

© 2012 Oracle Corporation – Proprietary and Confidential

• Date and time the dump was created as well as JVM

• Each thread is dumped with its complete java stack that

• All this gives an overview of the thread execution and the

• What is getting executed by the thread can be understood

© 2012 Oracle Corporation – Proprietary and Confidential

1. Make sure at least 3 Thread Dumps, 10 seconds apart,

© 2012 Oracle Corporation – Proprietary and Confidential

5. Ignore the idle threads

© 2012 Oracle Corporation – Proprietary and Confidential

© 2012 Oracle Corporation – Proprietary and Confidential

• Active Threads: Threads actively working on requests.

© 2012 Oracle Corporation – Proprietary and Confidential

• Locks are used to prevent that more than one

• Only one thread can own a Lock object at a time

• Lock objects support a wait/notify mechanism

• With little contention a lock is considered Thin,

© 2012 Oracle Corporation – Proprietary and Confidential

© 2012 Oracle Corporation – Proprietary and Confidential