Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12

Siddharth Gandhi
Nov 2008

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 1
Contents

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12..........................1
Introduction.......................................................................................... ...................3
First Steps.......................................................................................................... ......4
Memory management.............................................................................................4
Garbage Collection Concepts..................................................................................5
Definition....................................................................................................... .......5
Generations................................................................................................. .........5
Types of Collectors...............................................................................................6
Measurement.......................................................................................................6
Few notable JVM parameters...................................................................................8
Monitor the JVM............................................................................. ........................10
Future changes - JRockit........................................................................................13
References and Quotes.......................................................................................... 14
Appendix.................................................................................... ...........................15

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 2
Introduction
In many of our support tickets we might have encountered performance and memory issues with the
middle-tier and the ubiquitous java.lang.OutOfMemoryError(s) statement in our stack trace.

We might be encountering these performance issues for many reasons impacted by hardware
(specifications, networking etc) or software (OS, application versions/patches/parameters) or just by
external parameters like user behavior, current number of active users, bandwidth etc.

In the Oracle Application ‘development’ space, plugging in these performance and memory issues will
majorly boil down to JVM tuning, stopping memory leaks, JDBC optimizations and few others.

In this whitepaper, we will brief upon

• Garbage Collection,

• Tuning JVM for better performance,

• Understanding few of the JVM tuning parameters provided in Jserv.properties and opmn.xml file

• Monitoring JVM

The document focuses only on the OACoreGroup JVM and not considers the JVM settings for other
Oracle services like Forms & Reports, Discoverer, Configurator etc. Also assumed is that the reader has
some familiarity with Java and E-Business Suite 11i and Release 12

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 3
First Steps
Quoting Sun’s documentation, Out of Memory (OOM) errors are usually thrown…

“… when the Java Virtual Machine cannot allocate an object because it is out of memory, and no more
memory could be made available by the garbage collector.”

And so follows the usual fix of increasing the JVM heap size,

java -Xms1024M -Xmx1024M ….

But in this quick-fix world, we comfortably forget the following two important questions,

1. Why Java Virtual Machine is out of memory?

2. Why garbage collector is not able to make more memory available?

Let’s remember the fact that physical memory is limited and on the other hand there are easier, faster and
cheaper approaches to consider then upgrading the hardware.

So before we proceed to tune our JVMs, let’s brush our basics for memory management and role of
Garbage Collection (GC)

Memory management
“Memory management is the process of recognizing when allocated objects are no longer needed, de-
allocating (freeing) the memory used by such objects, and making it available for subsequent allocation“

Explicit Vs. Automatic Memory Management

Java employs automatic memory management unlike C and few other programming languages where
memory management is developer responsibility. Explicit management many a times results into
unexpected or erroneous program behavior and crashes.

Whereas automatic management is handled by a program called a garbage collector (GC). GC

automatically frees all memory no longer referenced.

There are basically two ways you can alter the JVM’s behavior to improve performance. Firstly, you can
influence memory use by changing how memory is allocated and organized by the JVM. Secondly, you
can influence how garbage collection is performed.

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 4
Garbage Collection Concepts

Definition
Objects that are referenced are said to be live. Objects that are no longer referenced are considered
dead and are termed garbage. The process of finding and reclaiming the space used by these objects is
known as garbage collection.

The timing of garbage collection is up to the garbage collector.

Generations
Memory in the Java HotSpot virtual machine is organized into three generations:

1. Young generation,

2. Old generation(or Tenured), and

3. Permanent generation (or Perm)

This generational garbage collection, takes advantage of the observation that, most programs create:

• Many objects that have short lives (for example, iterators and local variables).
• Some objects that have very long lifetimes (for example, high level persistent objects)

The young generation consists of an "eden space" and two "survivor spaces." The VM initially assigns all
objects to the Eden space, and most objects die there. When the young generation area fills up, it causes
Minor Garbage Collection. Any objects survived after Minor GC is moved to Tenured Generation.

And when Tenured Generation area fills up it causes Major Garbage Collection i.e. Full GC.
Major GC is slow and expensive as compared to Minor GC since it involves evaluating all live objects.

The permanent generation holds objects that the JVM finds convenient to have the garbage collector
manage, such as objects describing classes and methods, as well as the classes and methods
themselves.

Figure 1

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 5
Types of Collectors
To reclaim memory, HotSpot has four different collectors as of J2SE 5.0 (and the option to enable them):

• The default or serial collector

• Throughput collector (-XX:+UseParallelGC)
• Concurrent low pause collector (-XX:+UseConcMarkSweepGC)
• Incremental collector (-XX:+UseTrainGC)

The default collector is a serial collector that pauses the application during minor and major collections. If
the host machine has a single CPU, the serial collector will likely be as fast or faster as the other
collectors.

The throughput collector uses the same collection mechanism as the serial collector for major collections,
but implements a parallel minor collector for the young generation space.

On the other hand, the concurrent low-pause collector attempts to perform most of the work of a major
collection without interrupting your application (pauses are low/short).

Finally the incremental collector, by careful book-keeping, collects just a portion of the tenured generation
at each minor collection, trying to spread the large pause of a major collection over many minor
collections. However, the incremental collector is deprecated and will eventually be removed.

Which collector works best for your application depends on the load of the application and the host
system. However, unless you have a system with multiple CPUs, tuning memory configuration rather than
garbage collection is likely to yield performance gains.

Measurement
Before we consider the metrics, let’s understand various parameters, on which it’s evaluated.

Throughput is the percentage of total time not spent in garbage collection, considered over long periods
of time.

Pauses are the times when an application appears unresponsive because garbage collection is occurring.
Usually tolerable, but in an interactive graphics program even short pauses may negatively affect the user
experience.

Promptness is the time between when an object becomes dead and when the memory becomes
available, an important consideration for distributed systems, including remote method invocation (RMI).

Footprint is the working set of a process, measured in pages and cache lines. On systems with limited
physical memory or many processes, footprint may dictate scalability.

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 6
Throughput and footprint are best measured using metrics particular to the application. On the other
hand, pauses due to garbage collection are easily estimated by inspecting the diagnostic output of the
virtual machine itself.

Now let’s understand that how and what garbage collection details are generated in the E-Business Suite.

Table 1 below briefs how and where to set the parameters in the appropriate file to generate the GC data
and view it for specific E-Business Suite version

Table 1

Suite Generate the GC data View the output {file location}

11i Provide the command line argument -verbose:gc in the file $IAS_ORACLE_HOME/Apache/Jserv/l
$IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties. For ogs/jvm/OACoreGroup.<core#>.stdout
instance,

wrapper.bin.parameters=-verbose:gc -hotspot -Xmx1024M

-Xms128M -XX:MaxPermSize=128M

or value of the s_jvm_options context variable in the file

$APPL_TOP/admin/<CONTEXT>.xml

<jvm_options oa_var=”s_jvm_options” osd=”Solaris”>-

verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=128M
</jvm_options>

R12 Set the argument -verbose:gc in the start-parameters, java- $LOG_HOME/ora/10.1.3/opmn/OC4J~

options element in the file oacore~default_group_*
$ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml. For
instance,

<ias-component id="OC4J">
<process-type id="home" module-id="OC4J" status="enabled">
<module-data>
<category id="start-parameters">
<data id=”java-options” value=”- server -verbose:gc
-Xmx512M -Xms128M. . ."/>
...
</category>
...
</module-data>
...
</process-type>
...
</ias-component>

Following are the sample GC details that are generated after setting the value -verbose:gc

[GC 325407K->83000K(776768K), 0.2300771 secs]

[GC 325816K->83372K(776768K), 0.2454258 secs]
[Full GC 267628K->83769K(776768K), 1.8479984 secs]

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 7
But what does these values means and how to measure them?

Figure 2
1) Here we see two minor collections, and a
2) Major collection
3) and 4) Indicate the combined size of live objects before and after garbage collection, respectively. After
minor collections the count includes objects that aren't necessarily alive but can't be reclaimed, either
because they are directly alive, or because they are within or referenced from the tenured generation.
5) Is the total available space, not counting the space in the permanent generation
6) The minor collection took about 24 secs to run

Simply said, the first statement of this output tells that we had 325407Kb used, but that after about quarter
of a second of work, GC has freed 242407KB so you now have 776768KB used.

Note: Additional flags like, -XX:+PrintGCDetails or -XX:+PrintGCTimeStamps can be used to print more
information about the collections, for instance,

[GC 111.042: [DefNew: 8128K->8128K(8128K), 0.0000505 secs]111.042: [Tenured: 18154K->2311K(24576K), 0.1290354

secs]

Few notable JVM parameters

Few key options that should be considered for useful GC logging are:

Table 2

Option Meaning
-verbose:gc Enable verbose GC logging.
-XX:+PrintGCDetails Provide detailed statistics about collections.
Give details about young generation survivor spaces and how many objects are
-XX:+PrintTenuringDistribution
promoted to the tenured space.

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 8
Following are few of the options for controlling how the JVM allocates and organizes memory (refer Sun’s
documentation for details)

Table 3

Option Meaning
-Xms Configure the initial Java heap size.
-Xmx Configure the maximum Java heap size.
Instructs the VM to analyze the current operating environment and attempt to optimize
settings for memory-intensive applications. This option also enables implicit garbage
-XX:+AggressiveHeap
collection and adaptive memory sizing. It is intended for machines with large amounts of
memory and multiple CPUs.
-XX:NewRatio Configure the ratio of new/old generation sizes
-XX:NewSize Configure the initial size of the young generation space.
-XX:MaxNewSize Configure the maximum size of the young generation space.

Now let’s look at some of the default JVM parameters provided in one of the configuration files. For
instance, the sample context file $APPL_TOP/admin/$CONTEXT_NAME.xml contains the following
options,

<jvm_options oa_var=”s_jvm_options” osd=”Solaris”>-verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=128M

-XX:NewRatio=2-XX:+PrintGCTimeStamps -XX:+UseTLAB </jvm_options>

Table 4

Option Meaning
-Xms128M Initiate with 128MB heap size
-Xmx512M Grow JVM heap size to maximum size of 512 MB
-XX:NewRatio=2 If the size of young generation is 10 MB then the size of old generation should be around
20MB
-XX:MaxPermSize=128M Maximum grow able size of permanent generation to 128M
-XX:+UseTLAB Uses thread-local object allocation blocks. This improves concurrency by reducing
contention on the shared heap lock.

But all this tuning might go for a toss, if the developer calls System.gc() since this forces a major collection.
Application can be prohibited to make this operational by providing -XX:+DisableExplicitGC

Few tips for JVM tuning in E-Business Suite

 Upgrade to highest possible certified JDK version (JRE 6 is latest certified with both 11i and R12).
To know current version of Java, search for the parameter s_jdktop in the file
$APPL_TOP/admin/$CONTEXT_NAME.xml
<JDK_TOP oa_var=”s_jdktop”>/oracle/apps/11i/bin/java/1.4/</JDK_TOP>
 For OACoreGroup consider no more than 100 active users per JVM (Refer Appendix for SQL
Script)
 There should be no more than 1 active JVM/OC4J instance per CPU. Ideally, there should be 1
JVM per 2 CPUs
 Size your maximum heap size as either 512 MB or 1 GB. If you start with 512 MB and find that
more memory is required, increase to a maximum of 1 GB. If more memory than 1 GB is
required, use multiple JVM’s of 512MB or 1024 MB. (For multiple JVM’s, edit <oacore_nprocs

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 9
 oa_var=”s_oacore_nprocs”>2</oacore_nprocs>) This is to reduce GC time since more heap size
means more time to garbage collect
 Setting -Xms and -Xmx to the same value avoid the need to memory allocation during runtime
 With multi-CPU servers using JDK 1.4 and higher, the -XX:+UseConcMarkSweepGC -XX:
+UseParNewGC parameters have been seen to improve performance and GC throughput on some
environments.
 Minor GC should be occurring at interval long enough to allow many objects to die young (i.e. lot
of objects should die between two minor GC).
 Review the frequency of collections, especially major collections (i.e. Full GC)
 If full GCs are too frequent, consider increasing Xms and Xmx
 Throughput is inversely proportion to amount of memory. Bigger the heap, more the time for GC,
meaning low throughput.
 Unless you have problems with pauses, try granting as much memory as possible to VM (512 is
good start and fine tune as per load testing results)
 100 users per OACore JVM has generally proven to be a practical and performant number
 If heap usage is steadily increasing, then there may be a memory leak which require tweaking the
application code
 Do not use -XX:+UseParallelGC with -XX:+UseConcMarkSweepGC
 Last but not the least, try experimenting with various JVM parameters referred in Table 2 and
Table 3 above and many more available at Sun’s documentation site (refer References section),
to arrive at the optimal solution

Monitor the JVM

The basic text based monitoring tool available to us are the log files which contains the data produced by
the JVM (using –verbose:gc option). For the details of these log files and how to understand the heap data
available in these files refer the Measurement section above.

Apart for this text based approach, there are many other visual tools available to us like JConsole,
VisualVM etc

The following section will discuss one of them, JConsole.

JConsole provide information on performance and resource consumption of applications running on the
Java platform using Java Management Extension (JMX) technology. This tool comes in JDK 1.5 and
provides graphical view of the JVM heap, usage, threads, classes and other statistics in real time.
Refer Figure 3 below.

You can run JConsole either locally (from the server that you are monitoring as localhost) or remotely via
a network connection with Oracle Applications. The safest and easier option is to run locally since using
JConsole remotely is more complex, insecure and it may introduce unacceptable risk for a production
environment.

The JConsole interface in JDK 5 is composed of the following six tabs:

1. Summary displays summary information on the JVM and monitored values.
2. Memory displays information about memory use.
3. Threads displays information about thread use.
4. Classes displays information about class loading.
5. MBeans displays information about MBeans.
6. VM displays information about the JVM.

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 10
For instance, to check for memory leaks in your application, you can start a load test to create a constant
load on the application. If the functionality you're testing during the load test does not contain a memory
leak, the memory usage should stay between a specific upper and lower limit.

Connecting Locally

Using JConsole with Oracle E-Business Suite 11i

Add the following lines to your jserv.properties file present under $IAS_TOP/Apache/Jserv/etc/ ,save the
changes and bounce Apache server

Note: To make this change permanent, refer Metalink note 270519.1 about AutoConfig

wrapper.bin.parameters=-Dcom.sun.management.jmxremote
wrapper.bin.parameters=-Dcom.sun.management.jmxremote.ssl=false
wrapper.bin.parameters=-Dcom.sun.management.jmxremote.authenticate=false

Figure 3

Connecting Remotely

Firstly, authentication is enabled by default so you will either need to configure your username/passwords
or could choose to defeat authentication entirely by using the jserv.properties value below.

Note: The following change allows everyone to attach to your JVM processes

wrapper.bin.parameters=-Dcom.sun.management.jmxremote.authenticate=false

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 11
The second issue to overcome is that each JVM needs its own TCP port to accept connections. One
possible approach is to customize the $IAS_ORACLE_HOME/Apache/Apache/bin/java.sh JVM startup
script and provide a port number range from which you allocate a TCP port number.
The following sample script provided by Steven Chan but may give you some ideas as to how you could
overcome this issue in your own environment. Add this modification just before the existing line

exec $JSERVJAVA $JAVA_ADDITIONAL_ARGS $ARGV 1>> $STDOUTLOG 2>> $STDERRLOG

#### author Steven Chan

#### start of modification for JConsole remote monitoring
#### Not supported or recommended by Oracle Corporation
####
#### If there is a clash of port numbers calculated, then
#### the JVM will not startup. Script therefore needs to be
#### improved to check port not already in use
####
#### Use port number range 9600 - 9650 as an illustration
if [ "$JDK_VERSION" = "1.5" ] ; then
mJMXPORTMIN=9600
mJXMPORTMAX=9650
mNUMPORTS=`expr ${mJXMPORTMAX} - ${mJMXPORTMIN}`
mRANDOM=$(((($RANDOM*${mNUMPORTS})/32767)+1))
mJMXPORT=`expr ${mRANDOM} + ${mJMXPORTMIN}`
JAVA_ADDITIONAL_ARGS=" -DCLIENT_PROCESSID=$$ -Dcom.
sun.management.jmxremote.port=${mJMXPORT}"
fi
#### end of modification

After starting your JVMs we then need to identify the TCP port they are listening on. This can be
determined by using the following command:

netstat -a | grep tcp | grep 96

The JConsole executable is present in $JDK_HOME/bin, where JDK_HOME is the directory where the
JDK is installed. Once JConsole is started a connection dialog box opens, with tab lists as 'Local',
'Remote' and ‘Advanced'. Check that under 'Remote' tab 'Host or IP' field value should be localhost (for
local connection) or the IP address of the remote machine. For the remote connection, we additionally
need to provide the port we wish to connect to and also the username/password if appropriate. Then click
on 'Connect' button and it will show the details of any JVMs running on the local/remote system started
with the same user ID as JConsole, along with their process ID and class/argument information and it can
be monitored.

Note: JConsole works with the JRockit JVM which will be part of Fusion stack in future. But it’s
recommended to use the tools that are packaged as JRockit Mission Control for monitoring JRockit JVM

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 12
Future changes - JRockit

Acquisition of BEA and introduction of BEA Weblogic Application Server and JRockit JVM

All the discussion above has been restricted to Sun HotSpot JVM. But with the acquisition of BEA,
JRockit JVM will also be used across the Oracle Fusion Middleware and Oracle Applications product
families.

This will bring additional changes to JVM tuning for E-Business Suite associated with JRocket

For instance, the Table 4 below lists options that work the same or similarly on both Sun Hotspot and the
JRockit JVM but have different names depending upon the JVM you are running.

Table 4

Sun Hotspot Option Name JRockit JVM Option Function

Name

-XX:+AggressiveHeap -XXaggressive:memory Configures the memory system for memory-intensive

workloads and sets an expectation to enable large
amounts of memory resources to ensure high
throughput. The JRockit JVM will also use large pages,
if available.

-verbose:gc -Xverbose:memory Prints out log information about the memory system

-Xmn, -XXNewSize, -Xns Sets the size of the young generation

-XXMaxNewSize

-XX:+UseConcMarkSweepGC -Xgc:singlecon Sets the garbage collector to use a concurrent strategy

-XX:+UseParallelGC -Xgc:parallel Sets the garbage collector to use a parallel strategy

Note: The certified plugs-ins for Forms is Jinitiator and Sun only.

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 13
References and Quotes

General

http://java.sun.com/j2se/reference/whitepapers/memorymanagement_whitepaper.pdf
http://java.sun.com/docs/hotspot/gc1.4.2/example.html
http://blogs.oracle.com/stevenChan/
http://java.sun.com/docs/hotspot/gc1.4.2/
http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html
http://www.oracle.com/technology/software/products/jrockit/FAQ.html
http://java.sun.com/developer/technicalArticSE/jconsole.html
http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html
http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp
http://edocs.bea.com/jrockit/geninfo/diagnos/migrate.html

On upgrading the JDK in E-Business Suite:

Note 246105.1- Upgrading to J2SE 1.4.2 with Oracle Applications 11i

Note 304099.1- Using J2SE Version 5.0 with Oracle E-Business Suite 11i
Note 401561.1- Using J2SE Version 6.0 with Oracle E-Business Suite 11i

Notes on JVM tuning with respect to E-Business Suite

Note 244040.1- Oracle E-Business Suite Recommended Performance Patches

Note 370583.1- Basic troubleshooting of JVM consuming CPU or too many JDBC connections in Apps
Note 244040.1- Oracle E-Business Suite Recommended Performance Patches

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 14
Appendix

Script to determine "active users" for OACoreGroup {run as APPS user}

REM
REM SQL to count number of Apps 11i users , refer Metalink Note: Note:427759.1
REM Run as APPS user
REM
select 'Number of user sessions : ' || count( distinct session_id) How_many_user_sessions from icx_sessions icx
where disabled_flag != 'Y'
and PSEUDO_FLAG = 'N'
and (last_connect + decode(FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'), NULL,limit_time,
0,limit_time,FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) > sysdate and counter < limit_connects;
REM
REM END OF SQL
REM

Tags for this document:

JVM Tuning
Java.lang.OutOfMemoryError
JRockit
JConsole
Garbage Collection

Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 15