Anda di halaman 1dari 35

IBM Software Group

Debugging and trouble-shooting


techniques for WebSphere Application
Server start/stop issues in V6.0 / V6.1
Vikram Thommandru

WebSphere Support Technical Exchange

IBM Software Group

Agenda
Different ways of starting/stopping Application Server in a
stand alone node.
Different ways of starting/stopping Application Server in a
Network Deployment.
Application Server monitoring policy.
Server start/stop issues , its trouble shooting techniques and
debugging tips.
Questions and answers.

WebSphere Support Technical Exchange

IBM Software Group

Starting and Stopping in Standalone


Application Server

WebSphere Support Technical Exchange

IBM Software Group

Starting Application Server Different options


Command line
Run from the <PROFILE_HOME>\bin directory
Example: startServer server1

Run from <INSTALL_HOME>\bin directory specifying profile name


Example: startServer profileName AppSrv01 server1

Logs in <PROFILE_HOME>\logs directory

Windows Start menu


Start
Programs
IBM WebSphere
Profiles
<Profile>
Start the Server

Application Server V6.0

If Server was registered as Windows Services


Registration using WASService.exe or at the time of creating the
profile
Control Panel
Administrative Tools
Services
4

WebSphere Support Technical Exchange

IBM Software Group

Start Server command


startServer.bat(sh) profileName <profile>
<serverName> <options>
Creates new Java Virtual Machine (JVM) to parse configuration
Launches Application Server specified by <serverName> in profile
specified by <profileName> with JVM options specified in server.xml of
that server
Refer to the V6 Information center for other options

-script option creates a launch script for server1,


does not start the server.
startServer server1 script launchServer1.sh
Launch Scripts can be used to start the server (JVM)
Reduces start time by not parsing configuration files
If the JVM settings are changed, create a new launch script
5

WebSphere Support Technical Exchange

IBM Software Group

Stop Server command


stopServer.bat (sh) profileName <profileName>
<serverName> <options>
Example: stopServer.sh profileName AppServ01
server1
Logs in <PROFILE_HOME>\logs directory

Creates new JVM to read configuration and send


message to server to shutdown
By default, the stopServer utility does not return control
to the command line until the server completes shut
down
User ID and password required to stop a secure
Application Server
6

WebSphere Support Technical Exchange

IBM Software Group

Section

Starting and Stopping WebSphere


Application Server Processes for a Network
Deployment Cell
(Application Servers, Node Agent, DMgr)

WebSphere Support Technical Exchange

IBM Software Group

Starting Application Server


Using startServer command: Same as Stand-alone
Application Server
Command: startServer profile <profileName>
<serverName> options

Node Agent must be running before starting the


Application Server
Can also stop from the Deployment Manager
(DMgr) Administrative Console or using wsadmin
command
8

WebSphere Support Technical Exchange

IBM Software Group

Stop Server command


Using stopServer command: Same as Standalone Application Server
stopServer.bat (sh) profile <profileName>
<serverName> options

Can also stop from the Administrative Console or


using wsadmin command

WebSphere Support Technical Exchange

IBM Software Group

Starting/Stopping DMgr and Node Agent


Start/Stop DMgr using command
startManager -profileName <DMgr Profile Name>
startServer - profileName <DMgr Profile Name> dmgr
stopManager -profileName <DMgr Profile Name>
stopServer - profileName <DMgr Profile Name> dmgr

Start/Stop NodeAgent using command


startNode -profileName <Managed Node Profile Name>
startServer - profileName <Managed Node Profile Name> nodeagent
stopNode -profileName <Node Profile Name>
stopServer - profileName <Managed Node Profile Name> nodeagent

Can stop NodeAgent from the Administrative Console


Cannot start a Node Agent from the Administrative Console

Additional start/stop options for Windows OS


From Windows services
From Start menu
10

WebSphere Support Technical Exchange

IBM Software Group

Section

Monitoring WebSphere Application Server


Processes

11

WebSphere Support Technical Exchange

IBM Software Group

Monitoring and Restarting


WebSphere
Process Type

Monitor and Restart


owner

Operating System
Window Services/
Unix daemons

Standalone Node
Operating System Windows Service or UNIX
daemon

Deployment Manager

Application Servers in a
Managed Node

Monitors

Node Agent and is based


on configurable setting for
each Application Server
(using the Server -> Java
and Process Management > MonitorPolicy settings)

Monitors

v6

V6
Application
Server
(Standalone)
Ex
pr
es
s

Node Agent for a


Managed Node

12

WebSphere Support Technical Exchange

IBM Software Group

Windows OS: Monitoring Server Process


Windows services

Optionally, created during defining


new Profile for Stand-alone Node,
Managed Node or DMgr
addNode command has option to
add Node Agent services
WASService.exe command to
insert service registry entry

User must belong to


Administrator group and have
the user rights for
Act as part of the operating
system
Log on as a service

Can configure the services to:


Automatic/Manual start
Recovery actions
13

WebSphere Support Technical Exchange

IBM Software Group

WASService.exe (Windows only)


WASService command line tool allows adding any WebSphere Application Server
process as a Windows service, and managing the service
WASService.exe command syntax for adding a service
WASService.exe -add <service_name> -serverName <Server> profilePath <servers
profile path> <options>
WASService.exe command syntax for managing services
WASService.exe -remove service_name
WASService.exe -start service_name
WASService.exe -stop service_name
WASService.exe -status service_name
For more details on WASService, refer to the V6 Information Center

14

WebSphere Support Technical Exchange

IBM Software Group

Unix: Monitoring Server Process


etc/inittab

rc.was
Unix
Machine

numRetries=3
launchScript=start_server1.sh

Sample rc.was shell script can be found in <WAS_ROOT>/bin directory


Starts a defined process using a launch script
Launch script can be created using startServer command with script option
E.g.: StartServer server1 script start_server1.sh

Restart process for non-zero exit


Number of retries can be customized (numRetries=3)

System administrator has to manually add an entry into inittab


was:2:once:/usr/WebSphere/AppServer/bin/rc.was >/dev/console 2>&1

Servers started using startServer command will not be monitored

15

WebSphere Support Technical Exchange

IBM Software Group

UNIX: Monitoring server process : rc.was Example


rc.was Example
launchScript=start_server1.sh
numRetries=3 ; RETRY=0
while [ $RETRY -lt $numRetries ]
do
$binDir/$launchScript
# launching server using $launchScript
rc=$?
echo exit code: $rc
# Increment retry count on anything other than a normal exit code
if [ $rc -gt 0 ]
then
RETRY=`expr $RETRY + 1`
fi
case $rc in
0) break ;;

Typically Stand-alone Application Servers, Node Agent and Deployment Manager


are monitored
Multiple scripts and entries in inittab, may be required
16

WebSphere Support Technical Exchange

IBM Software Group

Trouble Shooting and Debugging Tips

17

WebSphere Support Technical Exchange

IBM Software Group

Anatomy of a Server Start


Before we dive into some debugging techniques, it is useful to know just
what happens when a server is started. You may not know this, but
when you type
two JVMs are actually launched. The first JVM is the Systems
Management server launch utility. Its job is to locate the appropriate
configuration (e.g. server.xml) (note that some default properties are set
in the various systemlaunch.properties files) and spawn the second
JVM, which is the actual server process. Spawning is achieved by
constructing a command (java -Dxxx -classpath x;y;z etc) and executing
it. This server launching process waits until it receives status back from
the server process (unless you specified the -nowait parameter).
Example output from this launch process is:

18

WebSphere Support Technical Exchange

IBM Software Group

Example output from this launch process is:


ADMU0116I: Tool information is being logged in file
C:\WebSphere61\AppServer\profiles\AppSrv01\logs\server1\startServer.log
ADMU0128I: Starting tool with the AppSrv01 profile
ADMU3100I: Reading configuration for server: server1
ADMU3200I: Server launched. Waiting for initialization status <= At this point the server
launching JVM is waiting for status back from the server JVM
ADMU3000I: Server server1 open for e-business; process id is 4152

19

WebSphere Support Technical Exchange

IBM Software Group

In the server process, things boot up as follows:

!
"
#

$
& '

(
#"
)

NOTE: In the context of this article, both RAS and FFDC are
components and thus are not fully initialized until the last bullet above.
This means that for some period of time, we are "flying blind". Failures
during this early critical region can be challenging to debug.
20

WebSphere Support Technical Exchange

IBM Software Group

You might see a problem where websphere will not


write anything into the systemOut.log
/SystemErr.log files :
In the scenario like this you just want to get the server launcher out of the way. This is easily done by
generating a server start script. For example,
startServer.bat/.sh server1 script
This generates a .bat/.sh file named start_server1.bat/sh into the current directory which can be used to
directly start the server without going through the launcher. Sometimes you will see things in your
console (DOS window, xterm, ...) that do not appear in any of the logs. It also puts the JVM in the
foreground in the console and makes things like taking a JVM dump easier (Ctrl-Break on Windows).

Things which could lead to the problems :


User have specified invalid JVM generic arguments or custom property
First thing you may want to look is server.xml file for the application server and if you see any
genericJVMargument , try removing them and see if server starts fine. Make sure you take a
backup to server.xml file . In a network deployment environment make the changes from
adminconsole then perform synchronization .
Corrupted java
Check native_stdout.log or native_stderr.log file for any errors logged in . Most of the time if
there is a problem with java then these files might have some useful information to start
investigation with.

WebSphere Support Technical Exchange

21

IBM Software Group


stopServer launcher script unable to establish SOAP connection to the application
server JVM due to invalid credentials.
If the admin security turned on make sure username and password supplied
correctly when stopping the application server .
If there is a proxy enabled on the system ensure its not blocking any communication
from startServer / stopServer launcher to the application server JVM.
If stopServer operation fails with SSL Handshake failure when making a SOAP
connection to application server , check client truststore has a server personnel
certificate.
If startServer / stopServer launcher itself is hanging for some reason.
Collect the javacores against startServer / stopServer launcher java process id for
investigation but to bypass this launcher you can try creating a application script file
with a -script parameter to initialize the server directly.

WebSphere Support Technical Exchange

22

IBM Software Group

When Server Start Hangs :


When a server start hangs or you have an unexpected crash, you will want
to look at or provide a java dump. In the case of the unexpected server
crash, the javacore file should have been created. Usually the file will have
javacore in the name and it will be created in the %USER_HOME%
directory. In the case of a server hang, you will need to create this file. On
early versions of WebSphere on Windows you could just hit Ctrl-Break on the
command window and the thread dumps would appear. However, since
WebSphere now uses the Systems Management launcher utility as noted in
the Anatomy of a Server Start section above, this no longer works. So what
should you do in this situation? Well, in 90% of the cases, the WSADMIN
component will be up prior to the hang condition and therefore ready for use.
There is good documentation in the InfoCenter for how to retrieve thread
dumps using scripting.
If this doesn'
t work you can also try one of these methods:
If you are on a UNIX platform, you may be able to use kill -3 against the
server process to create a java core as well. Use ps -ef | grep java to find the
JVM processes. Use top to check CPU utilization.
On Windows, you can start the server via a script and use Ctrl-Break to
obtain the thread dumps, or use the jvmdump utility .

WebSphere Support Technical Exchange

23

IBM Software Group

The Example below shows logging entries in nodeagent trace.log file when the
application server is started from deployment manager adminconsole :
[11/18/08 17:52:10:250 EST] 00000023 NodeAgent
[11/18/08 17:52:10:250 EST] 00000023 NodeAgent
websphere.process.starting
server1

3 Launching a new server process: server1


> sendNotification Entry

[11/18/08 17:52:11:250 EST] 00000023 NodeAgent < isServerRunning Exit


false
[11/18/08 17:52:11:250 EST] 00000023 LaunchCommand 3 launching process with parameters: executableName =
C:/WebSphere61/AppServer/java/bin/java
-Declipse.security -Dwas.status.socket =2110 -Dosgi.install.area=C:/WebSphere61/AppServer Dosgi.configuration.area=C:\WebSphere61\AppServer\profiles\AppSrv07/configuration -Dosgi.framework.extensions =
com.ibm.cds -Xshareclasses:name=webspherev61,nonFatal -Xscmx50M -Dcom.ibm.nio.DirectByteBuffer.SilentRetry=true
Xbootclasspath/p:C:/WebSphere61/AppServer/java/jre/lib/ext/ibmorb.jar;C:/WebSphere61/AppServer/java/jre/lib/ext/ibmext
.jar -classpath
C:\WebSphere61\AppServer\profiles\AppSrv07/properties;C:/WebSphere61/AppServer/properties;C:/WebSphere61/AppS
erver/lib/startup.jar;C:/WebSphere61/AppServer/lib/bootstrap.jar;C:/WebSphere61/AppServer/lib/j2ee.jar;C:/WebSphere61/
AppServer/lib/lmproxy.jar;C:/WebSphere61/AppServer/lib/urlprotocols.jar;C:/WebSphere61/AppServer/deploytool/itp/batch
boot.jar;C:/WebSphere61/AppServer/deploytool/itp/batch2.jar;C:/WebSphere61/AppServer/java/lib/tools.jar -Dws.ext.dirs
=C:/WebSphere61/AppServer/java/lib;C:\WebSphere61\AppServer\profiles\AppSrv07/classes;C:/WebSphere61/AppServer
/classes;C:/WebSphere61/AppServer/lib;C:/WebSphere61/AppServer/installedChannels;C:/WebSphere61/AppServer/lib/e
xt;C:/WebSphere61/AppServer/web/help;C:/WebSphere61/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runti
me -Dderby.system.home =C:/WebSphere61/AppServer/derby -Duser.install.root
=C:\WebSphere61\AppServer\profiles\AppSrv07 -Djavax.management.builder.initial
=com.ibm.ws.management.PlatformMBeanServerBuilder -Dwas.install.root =C:/WebSphere61/AppServer Dpython.cachedir =C:\WebSphere61\AppServer\profiles\AppSrv07/temp/cachedir -Dserver.root
=C:\WebSphere61\AppServer\profiles\AppSrv07 Djava.security.auth.login.config=C:\WebSphere61\AppServer\profiles\AppSrv07/properties/wsjaas.conf -

WebSphere Support Technical Exchange

24

IBM Software Group


Djava.security.policy =C:\WebSphere61\AppServer\profiles\AppSrv07/properties/server.policy
com.ibm.wsspi.bootstrap.WSPreLauncher -nosplash -application com.ibm.ws.bootstrap.WSLauncher
com.ibm.ws.runtime.WsServer C:\WebSphere61\AppServer\profiles\AppSrv07\config vikram2Cell02 vikram2Node08 server1
workingDirectory = C:\WebSphere61\AppServer\profiles\AppSrv07
[11/18/08 17:52:11:281 EST] 00000023 GenericServer > savePidToFile Entry
7108
[11/18/08 17:52:11:312 EST] 00000023 NodeAgent
[11/18/08 17:52:11:312 EST] 00000023 NodeAgent
[11/18/08 17:52:46:390 EST] 00000023 NodeAgent
true
[11/18/08 17:52:46:390 EST] 00000023 NodeAgent

3 Launched process pid: 7108


1 addLaunchedChild
3 processName = server1 pid = 7108 status = 0 - started = true wait =
< launchProcess Exit

true

WebSphere Support Technical Exchange

25

IBM Software Group

More about monitoring policy

WebSphere Support Technical Exchange

26

IBM Software Group

The monitoring policy settings of the application server define how it is


monitored by the node agent. The main properties involved in this are:
Ping Interval
How often the nodeagent pings the application server process to ensure it
is still running and able to handle requests.
Ping Timeout
How long the node agent waits for a response from the app server before
determining it is "unreachable".
Automatic Restart
Should the node agent restart the application server if the application
server process fails.
Node Restart State
When the node agent starts up, does it also start the application server.

WebSphere Support Technical Exchange

27

IBM Software Group

High level component flow


The node agent monitoring the application server begins with the node agent startup. The "Node
Restart State" property defines whether or not the nodeagent spawns the application server process
during its startup procedure. If this property is set to RUNNING, then it will attempt to start the
application server:
ADMN1001I: An attempt is made to launch server1 on node WAS_60_Node01.
When the nodeagent spawns the application server process, the application server process will be
considered the child of the node agent, which is the parent process. When the nodeagent does not
start the application server (for example: startServer server1), the nodeagent will "adopt" the
application server as its child. The node agent will then wait for the number of seconds in the "Ping
Timeout" property for a response from the application server process that the startup was successful.
This comes in the form of a discovery message.
ADMD0023I: The system discovered process (name: server1, type: ManagedProcess, pid: 2556)
If the node agent does not discover the application server process in the time defined in the "Ping
Timeout" property, then it will assume the application server startup failed. Then the nodeagent will
spawn a new application server process if the "Automatic Restart" property is set to true. It will
continue this procedure for the number of times specified in the "Maximum startup attempts" property.
Once the node agent discovers the application server process, it updates it'
s routing table and begins
to monitor the child process. The nodeagent checks the application server process to ensure it is still
up and able to respond in intervals defined by the "Ping Interval" property.
The JMX communications are done via either SOAP or RMI. This is determined by the "Preferred
Connector" configuration property for each Application Server. If the Preferred Connector is SOAP
(default), then the communication is done through each server'
s SOAP_CONNECTOR_ADDRESS
port (default 8880). If the Preferred Connector is RMI, then the communication is done through each
server'
s BOOTSTRAP_ADDRESS port (default 2809).

WebSphere Support Technical Exchange

28

IBM Software Group

Configuration Files
The monitoring policy configuration settings are stored in each application
server'
s server.xml file. Example path for server1 would be:
install_root/config/MyCell/nodes/MyNode/servers/server1/server.xml
The monitoring policy configuration settings appear as follows:
<monitoringPolicy xmi:id="MonitoringPolicy_1141403412562"
maximumStartupAttempts="3" pingInterval="60" pingTimeout="180"
autoRestart="true" nodeRestartState="STOPPED"/>
Example Startup Trace:
Nodeagent Trace: The nodeagent reads the monitoring policy settings for the
application server during its startup time.

WebSphere Support Technical Exchange

29

IBM Software Group


[3/17/06 13:06:02:109 EST] 0000000a NodeAgent 3 process monitor policy:
server1
PingInterval: 60
PingTimeout: 180
MaximumStartupAttempts: 3
NodeRestartState: 1
PreviousState: -1
AutoRestart: true
[3/17/06 13:06:02:109 EST] 0000000a NodeAgent 3 pName = server1 currentPid = null
[3/17/06 13:06:02:109 EST] 0000000a NodeAgent 3 isRestartingAllServers = false
[3/17/06 13:06:02:109 EST] 0000000a NodeAgent 3 restartState = RUNNING
[3/17/06 13:06:02:109 EST] 0000000a NodeAgent 3 launching server1
<-- The nodeagent launches the server1 process since restartState = RUNNING
[3/17/06 13:06:02:109 EST] 0000000a NodeAgent > launchProcess Entry
server1

Nodeagent Trace: The application server is launched with process ID 2556.


[3/17/06 13:06:05:547 EST] 0000000a NodeAgent 3 Launched process pid: 2556
[3/17/06 13:06:05:547 EST] 0000000a NodeAgent 1 addLaunchedChild
serverName =server1
pid =2556
[3/17/06 13:06:05:547 EST] 0000000a NodeAgent > saveNodeState Entry
[3/17/06 13:06:05:547 EST] 0000000a NodeAgent 3 restartServers = false monitorFile =
C:\WebSphere60/profiles/AppSrv01/logs/nodeagent/monitor.state
[3/17/06 13:06:05:562 EST] 0000000a NodeAgent 3 launchedChildren 2556
[3/17/06 13:06:05:594 EST] 0000000a NodeAgent < saveNodeState Exit

WebSphere Support Technical Exchange

30

IBM Software Group


Example Monitoring Trace:
Nodeagent Trace: The nodeagent pings the application server to see if it is up and running using a queryNames JMX call.
[3/17/06 13:07:35:391 EST] 00000059 PidWaiter > contact Entry
2556
<-- PidWaiter contact Entry starts the monitoring process. Look for the final result when PidWaiter contact returns.
[3/17/06 13:07:35:391 EST] 00000059 SecurityHelpe 3 Getting server subject.
[3/17/06 13:07:35:391 EST] 00000059 AdminServiceI > queryNames Entry
WebSphere:type =Server,process=server1,*
[3/17/06 13:07:35:859 EST] 00000059 AdminServiceI > getAttribute Entry
WebSphere:name
=server1,process=server1,platform= proxy,node=WAS_60_Node01,j2eeType=J2EEServer,version=6.0.2.7,type=Server,
mbeanIdentifier =cells/WAS_60_Cell01/nodes/WAS_60_Node01/servers/server1/server.xml#Server_1141403412266,
cell=WAS_60_Cell01,processType= ManagedProcess
<-- Sends a JMX call to the application server to check if server1 is still running
AppServer Trace: The application server receives the nodeagent's ping and responds if it is running.
[3/17/06 13:07:35:922 EST] 0000002b AdminServiceI > getAttribute Entry
WebSphere:name=server1,process=server1,platform=proxy,node
=WAS_60_Node01,j2eeType=J2EEServer,version=6.0.2.7,type=Server,
mbeanIdentifier =cells/WAS_60_Cell01/nodes/WAS_60_Node01/servers/server1/server.xml#Server_1141403412266,
cell=WAS_60_Cell01,processType= ManagedProcess
<-- The application server receives the call
[3/17/06 13:07:35:922 EST] 0000002b AdminServiceI < getAttribute Exit
STARTED
[3/17/06 13:07:35:922 EST] 0000002b AdminServiceD < getAttribute Exit
[3/17/06 13:07:35:922 EST] 0000002b SOAPConnector 3 return object type = class java.lang.String; value = STARTED
<-- The application server sends back the status

WebSphere Support Technical Exchange

31

IBM Software Group

Nodeagent Trace: The nodeagent receives the server1's response.


[3/17/06 13:07:35:938 EST] 00000059 AdminServiceI < getAttribute Exit
STARTED
[3/17/06 13:07:35:938 EST] 00000059 PidWaiter < contact Exit
true
<-- The PidWaiter contact method returns true since the server is started.
[3/17/06 13:07:35:938 EST] 00000059 PidWaiter
= false alarmSyncObject = 0

3 Pid 2556: For server1, bContact = true isProcessStopping

Example trace showing server1 process was killed and restarted:


Nodeagent Trace: The nodeagent detects that server1 has been killed and restarts it.
[3/17/06 13:08:18:094 EST] 00000072 RoutingTable > RemoveChildThread.run Entry
[3/17/06 13:08:18:094 EST] 00000072 RoutingTable 3 RoutingListner.parentRemoved:
com.ibm.ws.management.event.ProcessListener
[3/17/06 13:08:18:094 EST] 00000072 ProcessListen > childRemoved Entry
{cell=WAS_60_Cell01, version=6.0.2.7, pid=2556, name=server1, node=WAS_60_Node01,
role=ManagedProcess}
[3/17/06 13:08:22:359 EST] 00000072 NodeAgentStat < childRemoved Exit
[3/17/06 13:08:22:344 EST] 00000059 PidWaiter
[3/17/06 13:08:22:344 EST] 00000059 PidWaiter
2556
[3/17/06 13:08:22:344 EST] 00000059 PidWaiter

WebSphere Support Technical Exchange

3 Pid 2556: Process is being relaunched


> reLaunchProcess Entry
A ADML0064I: Restarting an unreachable server "server1".

32

IBM Software Group

Summary
Application server can be started from from command line ,
adminconsole or from services.
startServer launcher is a separate JVM from application server JVM.
Make sure synchronization and discovery are working fine when
starting/stopping application server from adminconsole.
Nodeagent monitors application server so if application server
crashes or does not respond then it will get re-started automatically.

WebSphere Support Technical Exchange

33

IBM Software Group

Additional WebSphere Product Resources


Discover the latest trends in WebSphere Technology and implementation, participate in
technically-focused briefings, webcasts and podcasts at:
http://www.ibm.com/developerworks/websphere/community/
Learn about other upcoming webcasts, conferences and events:
http://www.ibm.com/software/websphere/events_1.html
Join the Global WebSphere User Group Community: http://www.websphere.org
Access key product show-me demos and tutorials by visiting IBM Education Assistant:
http://www.ibm.com/software/info/education/assistant
View a Flash replay with step-by-step instructions for using the Electronic Service
Request (ESR) tool for submitting problems electronically:
http://www.ibm.com/software/websphere/support/d2w.html
Sign up to receive weekly technical My Notification emails:
http://www.ibm.com/software/support/einfo.html

WebSphere Support Technical Exchange

34

IBM Software Group

Questions and Answers

WebSphere Support Technical Exchange

35

Anda mungkin juga menyukai