Learn WebSphere Software

Collaborate Succeed

Network

Advanced Performance Tactics for

WebSphere Application Server ND V6x

© 2006 IBM Corporation © 2006 IBM Corporation

Conference materials may not be reproduced in whole or in part without the prior written permission of IBM.
Agenda

•JVM
•JMS
• HA
•EJBs
•HttpSession
•Dynamic Caching
•64 bit WAS
•Threading
•Hardware
WebSphere Software
2
© 2007 IBM Corporation
JVM Tuning
JVM Sizing

•Some starting points:

ƒ -Xmx=512M for WAS
ƒ -Xmx=1GB for WP

• Adjust to fit the application’s needs

ƒ Always keep the JVM(s) within the physical memory of the
server

• Minimum heapsize
ƒ Production systems set –Xms lower than –Xmx
• Gives headroom for emergencies
• In theory, results in more efficient object table
ƒ “Burst” testing requires –Xms=-Xmx

•1.4x IBM JVMs more efficient than predecessors

ƒ Shorter GC pauses, even for large heaps

WebSphere Software
3
© 2007 IBM Corporation
JVM Tuning
Controlling Memory Fragmentation with the IBM JVM

Pre-tuning Post tuning

• Red indicates heap fragmentation

• Small spikes are normal – look for overall trend (in this case near 0)
• Post tuning exhibiting desirable (almost “text book”) free space pattern
WebSphere Software
4
© 2007 IBM Corporation
JVM Tuning
Controlling Memory Fragmentation with the IBM JVM

•Pinned and dosed objects are unmovable during compaction

ƒ Example: Native calls generate pinned objects
•Heap fragmentation could become an issue
ƒ Especially true with large objects
ƒ OutOfMemory occurs even if there are free space in the heap
•kCluster is an area of storage for class blocks
ƒ Default size = 1280 (entries) x 256 (bytes)
ƒ Can be reset via –Xk<size>
•pCluster is an area for pinned objects
ƒ 16K by default
ƒ Newly created pCluster are 2K in size
ƒ Can be reset using –Xp<iiii>[K][,<oooo>[K]]
ƒ iiii = initial pCluster size
ƒ oooo for the size of subsequent overflow pClusters

WebSphere Software
5
© 2007 IBM Corporation
JVM Tuning
J9 JVM

• IBM clean-room version of Java

ƒ Built to the J2EE specification
ƒ Not built from Sun reference implementation
• J9 manages JNI objects differently
ƒ No pinned or dosed objects
ƒ Reduces fragmentation
• Two memory models available in J9
ƒ Flat model
• Similar to existing Sovereign model
ƒ Generational
• Similar idea as Sun and HP generational collectors
• Simplified model and tuning

• Improved JIT
ƒ Longer warm-up for optimized code
ƒ No longer disabled for remote debugging
• Faster WAS startup times with remote debugger attached

WebSphere Software
6
© 2007 IBM Corporation
Agenda

• JVM
• JMS
• HA
• EJBs
• HttpSession
• Dynamic Caching
• 64 bit WAS
• Threading
• Hardware
WebSphere Software
7
© 2007 IBM Corporation
JMS Tuning
Overview

ƒ JMS messaging now runs in-process with WAS in V6

•Reduces messaging overhead for light-weight messaging
•Non-persistent messaging requires no external process interaction
ƒ Reduces overhead in serialization to database

ƒ Correctness always the most important consideration

ƒ Use transactional support for multiple, inter-dependent messages
ƒ Otherwise, non-transactional messaging (default) more efficient

ƒ Use light-weight acknowledgement, if possible

WebSphere Software
8
© 2007 IBM Corporation
JMS Tuning
Non-Persistent Messaging

Nonpersistent Messaging Performance

V5.1 V6.0

20000 17214
14015
15000
ops/sec

10000
5196
5000 1879
0
MDB_PT P_NP_T NS MDB_PS_NP_ND_T NS

CP U% W AS 85 75 35 65

MDB = MDB receiver, stand-alone sender scenario

PTP = point-to-point messaging domain, PS = publish-subscribe messaging domain
NP = nonpersistent, ND = nondurable subscription, TNS = transaction not supported

S ystem conf iguration

Producer: xSeries 445 Server, 4 x 3.0 GHz Xeon, 2 GB ram, W indows 2003 Server Ent Edition
Consumer: xSeries 365 Server, 4 x 2.8 GHz Xeon, hyper threaded, 3.5 GB ram, W indows 2003 Server Ent Edition

ƒ Chart shows maximum steady-state message throughput

ƒ In-process messaging produces significant performance improvements

WebSphere Software
9
© 2007 IBM Corporation
JMS Tuning
Reliability Performance

Messaging Performance per Reliability

Levels for Point-to-point Messaging

20000 17214
18000
msgs/second

16000
14000 10512
12000 9676
10000
8000
6000 2830 4050 4093
4000 1869 1004 2183
2000
0
Best- Express Reliable Reliable Assured Reliable Assured Reliable Assured
effort NP NP TNS NP TNS P TR P TR P TR P TR P TR P TR
TNS (CS) (CS) (CS) (CS) (CS) (local (local (remote (remote
DB2) DB2) DB2) DB2)
CP U% W AS 75 74 71 92 62 99 97 97 88
CP U% DB 18 35

System configuration
Producer: xSeries 445 Server, 4 x 3.0 GHz Xeon, 2 GB ram, W indows 2003 Server
Consumer: xSeries 365 Server, 4 x 2.8 GHz Xeon, hyper threaded, 3.5 GB ram, W indows 2003 Server
Database: xSeries 365 Server, 4 x 2.8 GHz Xeon, hyper threaded, 3.5 GB ram, W indows 2003 Server
Ent Edition, DB2 v8.2

ƒ Reliability is the determining factor in throughput; the more reliable, the lower the throughput
ƒ Persisting to a local DB2 database is better than to a Cloudscape database, and better to a remote
DB2 database than to a local database

WebSphere Software
10
© 2007 IBM Corporation
JMS Performance Tuning
Reliability levels

High
•BEST_EFFORT_NONPERSISTENT
ƒ Messages are never written to disk
ƒ throw away messages if memory cache over-runs
•EXPRESS_NONPERSISTENT
ƒ Messages are written asynchronously to persistent storage if
memory cache overruns, but are not kept over server restarts
Reliability

ƒ No acknowledgement that the ME has received the message

•RELIABLE_NONPERSISTENT

Performance
ƒ Same as Express_Nonpersistent, except, we have a low level
acknowledgement message that the client code waits for,
before returning to the application with an OK or not OK
response
•RELIABLE_PERSISTENT
ƒ Messages are written asynchronously to persistent storage
during normal processing, and stay persisted over server
restarts.
ƒ If the server fails, messages might be lost if they are only held in
the cache at the time of failure.
•ASSURED_PERSISTENT
ƒ Highest degree of reliability where assured delivery is supported
High

WebSphere Software
11
© 2007 IBM Corporation
JMS Performance Tuning
Data buffers

Discardable data buffer

Message control structure

Message itself

Cache data buffer

JDBC

Messaging Engine Data store

•Each Messaging Engine has two data buffers to store messages (for
example a message in a queue)
ƒ Discardable data buffer (only used for Best Effort Non-persistent)
ƒ Cache data buffer (all other messages)

•Messages are discarded from the Databuffer using FIFO

ƒ Messages from Cache data buffer are still available from the database
ƒ Messages from Discardable data buffer are lost when discarded!

WebSphere Software
12
© 2007 IBM Corporation
JMS Performance Tuning
Sizing the data buffers

•Default size is 320 Kb for both data buffers

ƒ This is very small; likely that messages will be discarded from the buffers

•Discarded messages from the Discardable data buffer will throw an

exception
ƒ Look for com.ibm.ws.sib.msgstore.OutOfCacheSpace in SystemOut.log

•Discarded messages from the Cache data buffer will not throw an exception
ƒ But messages might need to be retrieved from the database, hurting
performance
ƒ Actively monitor PMI data

•Configure data buffers using custom properties for the SIBus

ƒ sib.msgstore.discardableDataBufferSize
ƒ sib.msgstore.cachedDataBufferSize

•PMI: Performance Modules > SIB Service > SIB Messaging Engines >
Messaging Engine > Storage Management > Cache
ƒ CacheStoredDiscardCount (for Cache data buffer)
ƒ CacheNotStoredDiscardCount (for Discardable data buffer)
WebSphere Software
13
© 2007 IBM Corporation
JMS Performance Tuning
Using DB2

•When optimizing message size between 3 KB – 30 KB

ƒ Put SIBnnn tables into a 32 KB page tablespaces (instead
of default 4 KB)
ƒ Change column width of the VARCHAR column “DATA” to
32032 bytes
ƒ This prevents message data to be stored in a BLOB
Throughput

referenced by column “LONG_DATA”

VARCHAR

BLOB

Message size
32032 bytes

WebSphere Software
14
© 2007 IBM Corporation
Agenda

• JVM
• JMS
• HA
• EJBs
• HttpSession
• Dynamic Caching
• 64 bit WAS
• Threading
• Hardware
WebSphere Software
15
© 2007 IBM Corporation
High Availability (HA)

• WAS v6 introduced the High Availability (HA) manager

ƒ WAS 6.02.5 allows the admin to disable this feature
ƒ Prior v6x versions, always on

• HA is a collaborative approach to failure management

ƒ Instances in regular communication (“heartbeats”)
ƒ On failure, other instances assume tasks of failed instance
• 2PC, persistent messaging, etc.
Throughput

• “Core groups” concept

ƒ Set of instances
VARCHAR working together to support HA

ƒ By default, all instances belong to default core group

• Core group bridging BLOB

ƒ Allows communication between core groups
ƒ Example: EJB location, etc. Message size
32032 bytes

WebSphere Software
16
© 2007 IBM Corporation
High Availability (HA)
Performance Considerations

• HA requires synchronization at start up

ƒ Each instance synch’s with every other instance in the core group
• O(n²) problem
ƒ CPU intensive as the core group size increases

• Recommendations
ƒ Keep the core group size to 50 or less
ƒ High performance sites might require smaller core groups
Throughput

• Instances may be marked down because of delayed heartbeat response

• Customers migrating
VARCHAR from v5.x with very large instance deployments
ƒ Make aware of this issue
ƒ Consider disabling HA if the customer does not require it
ƒ Re-allocate large environments to smaller core groups, if required
BLOB

Message size
32032 bytes

WebSphere Software
17
© 2007 IBM Corporation
Agenda

• JVM
• JMS
• EJBs
• HttpSession
• Dynamic Caching
• 64 bit WAS
• Threading
• Hardware

WebSphere Software
18
© 2007 IBM Corporation
EJBs
Tuning the EJB Cache

•Cache size = Maximum concurrent active instances

ƒ Entity Beans required per transaction * maximum expected
concurrent txns
ƒ Add maximum active Stateless Session Bean instances
•Don’t be afraid of large EJB caches if you have sufficient
heap
ƒ Benchmark caches routinely sized at 30K+ entries
•If the cache fills up, the container tries to passivate based on
LRU
ƒ Passivation triggers disk I/O – slow
•Remember, some EJBs may be long lived
ƒ Stateful Session Beans
•In WAS V6.0x
ƒ Lazy initialization of EJBs
ƒ Faster server startup with large EJB applications

WebSphere Software
19
© 2007 IBM Corporation
Agenda

• JVM
• JMS
• EJBs
• HttpSession
• Dynamic Caching
• 64 bit WAS
• Threading
• Hardware

WebSphere Software
20
© 2007 IBM Corporation
HttpSession
Memory-to-Memory replication

•Simplified configuration

•Terminology changes
ƒ No more replicas and partitions, instead we have client
and servers in a replication domain

•Topology changes
ƒ Replication domain is no longer collection of replicas

•Integration with WLM

ƒ Provides hot failover in peer-to-peer mode

WebSphere Software
21
© 2007 IBM Corporation
HttpSession
Peer-to-Peer Memory replication

WebSphere Software
22
© 2007 IBM Corporation
HttpSession
Client Server Memory replication

WebSphere Software
23
© 2007 IBM Corporation
Agenda

•JVM
•JMS
•EJBs
•HttpSession
•Dynamic Caching
•64 bit WAS
•Threading
•Hardware
WebSphere Software
24
© 2007 IBM Corporation
Dynamic Caching
Overview

•Consider Dynamic Caching during application design

ƒ Part of design rather than retrofitting later

•Organize page layout and design for caching

ƒ Servlet/JSP caching to hold repetitive, expensive page elements
ƒ Stateful elements vs. general use elements

•API Caching via Distributed Map services

ƒ Leverage Dynamic Caching services within the application

•Leverage new caching structure

ƒ Multiple internal caches now supported
ƒ Split objects into different caches to more efficiently distribute cache
data

WebSphere Software
25
© 2007 IBM Corporation
Dynamic Caching
V6.0x Functionality

• Two types of caching function

ƒ Policy Based ( defined via the cachespec.xml file)
• Servlet/JSP
• Commands
• Web Service
• Web Service Client – ( JAXRPC )
ƒ API Based
• Distributed Map
• Cacheable Servlet
• Cacheable Command
• Distribution features
ƒ Distribute cache contents to peer instances
ƒ Push content to HTTP servers and other “edge”
components

WebSphere Software
26
© 2007 IBM Corporation
Dynamic Caching
V6.0x Features

• Struts and Tiles Caching Support

• Dynamic Content Provider
ƒ Allows cacheable fragment to contain non-cacheable content
ƒ An exit calls out when fragment retrieved to get dynamic content
• Servlet and Object Cache Instances
ƒ Multiple cached instances of the same servlet/object in a single
server
• Distributed Object Cache enhancements
• Enhanced Cache Monitor tool
ƒ Support new caching features, including multiple caches
• Disk offload
ƒ Allows cache to overflow to the hard disk
ƒ Also, may save cache to disk when stopping WAS instance
• Expanded policy definitions
ƒ Including “static” caching
WebSphere Software
27
© 2007 IBM Corporation
Agenda

• JVM
• JMS
• EJBs
• HttpSession
• Dynamic Caching
• 64 bit WAS
• Threading
• Hardware

WebSphere Software
28
© 2007 IBM Corporation
64bit WAS
Overview

•64bit WAS available

ƒ 6.01
• Linux on Power (LoP)
• Linux on x86-64 Platforms
ƒ 6.02
• 6.01 platforms
• AIX on Power
• Windows Server 2003 Enterprise x64 Edition on x86-64

•Allows WAS JVM to grow well beyond 32bit process size boundaries

•Applications experiencing greatest benefit

ƒ Memory constrained
• Extra memory supports better caching strategy
• Avoids expensive queries, etc.
ƒ Computationally expensive code
• Security algorithms, etc.

•Many apps will not see a benefit, may increase overall footprint
WebSphere Software
29
© 2007 IBM Corporation
Agenda

• JVM
• JMS
• EJBs
• HttpSession
• Dynamic Caching
• 64 bit WAS
• Threading
• Hardware

WebSphere Software
30
© 2007 IBM Corporation
Threading
Channel Framework Tuning

•Channel Framework
ƒ Non-blocking I/O in WAS V6
ƒ Accepts very large numbers of concurrent in-bound requests
ƒ Unlikely to see “connection refused” errors at high loading
ƒ Stabilizes the Web Container even under bursty conditions

•Web Container thread tuning

ƒ Tuning practices unchanged with Channel Framework
ƒ Defines the maximum concurrent activity executing in the Web
Container
ƒ Limited by CPU, memory, remote resources, etc.
ƒ Tune to drive CPU to maximum
• Avoid exceeding more threads than CPU can usefully engage
• 50-75 threads about the “normal”

•Channel framework working with the Web Container

ƒ Handles requests exceeding available container threads
ƒ Response time will increase after throughput saturation point
ƒ However, web container will still do useful work and accept connections

WebSphere Software
31
© 2007 IBM Corporation
Agenda

• JVM
• JMS
• EJBs
• HttpSession
• Dynamic Caching
• 64 bit WAS
• Threading
• Hardware

WebSphere Software
32
© 2007 IBM Corporation
Hardware
pSeries

•Use AIX 5.3 SMT to get the most from POWER5 equipment
ƒ Gains of 40% in lab testing over AIX 5.2 or AIX 5.3 w/out SMT

Trade3 Results, EJBs on 4-way pSeries Systems

SMT NA/Disabled SMT Enabled

1250 1083
1008 1012
1000
reqs/second

766 741 692

750
527
500
250
0
AIX 5.3 - AIX 5.2 - AIX 5.3 - SLES 9 - RHEL 3
p630 p570 p570 p570 - p570
SUT CPU% 99 99 99 99 85 90 95
DB CPU% 28 37 39 58 34 52 53
R/T m s 74 50 52 35 57 38 38

System configuration
Driver: WebSphere Studio Workload Simulator on IBM Intellistation E Pro, 1000 MHz Pentium III
p630 SUT: IBM pSeries 630, 4 x 1453 MHz POWER4+
p570 SUT: IBM pSeries 570, 4 x 1656 MHz POWER5
Database: IBM pSeries 630, 4x 1453 MHz POWER4+, AIX 5.3, DB2 V8.2, DB2 Type 2 Universal JDBC Driver
(JCC)

WebSphere Software
33
© 2007 IBM Corporation
Hardware
pSeries Large Page

•Special memory options for large Power systems

•Large Page
ƒ 16MB page size
ƒ Pinned in memory – will not page
ƒ Kept in separate pool from traditional 4MB pages

•Useful for high traffic, heavy memory usage applications

ƒ Performance bump of 7-10%, usually

•See whitepaper for more details

ƒ http://www-
1.ibm.com/servers/aix/whitepapers/large_page.html#over
ƒ Requires a hardware reboot after setting

WebSphere Software
34
© 2007 IBM Corporation
Hardware
Intel Platforms

•Maximize the L2/L3 cache

•WebSphere runs significantly better with maximum

L2/L3
ƒEspecially important for 64bit WAS

•Plan ahead
ƒL3 cache is an ordered configuration option
ƒDifficult/impossible to add-on or upgrade
ƒRelatively inexpensive option
•Significant performance gains

WebSphere Software
35
© 2007 IBM Corporation
Summary

•WebSphere performance spans many areas and features

•High-end function and features provide performance

when used appropriately

•Hardware and OS selection play important roles in

performance

WebSphere Software
36
© 2007 IBM Corporation
References
•Stan J. Cox, et al: WAS 64bit Performance
ƒ http://www-306.ibm.com/software/webservers/appserv/doc/WAS_x86-
64-performance.pdf

•Kyle Brown and Keys Botzum: Improving HTTPSession

Performance with Smart Serialization
ƒ http://www-
106.ibm.com/developerworks/apps/transform.wss?URL=/developerworks/websph
ere/library/bestpractices/httpsession_performance_serialization.xml&xslURL=/de
veloperworks/websphere/xsl/bestpractice.xsl

•Best Practices for Large WebSphere Topologies

ƒ http://www.ibm.com/developerworks/websphere/library/techarticles/0710_largetopologies/0710_largetopologies.html

•Many thanks to Hendrik van Run, Cameron Martin, Bill Hines, Jessica
Man, Dipak Patel, and others for their contributions to this presentation.

WebSphere Software
37
© 2007 IBM Corporation
Further Reading
Performance Analysis for Java Web Sites
Joines, Willenborg, Hygh

IBM WebSphere Deployment and Advanced Configuration

Barcia, et al

IBM Web Sphere System Administration

Williams, et al

Enterprise Java Programming with IBM WebSphere

Brown, et al

WebSphere Software
38
© 2007 IBM Corporation
Questions?

•Please complete your session evaluation

•Thank you

WebSphere Software
39
© 2007 IBM Corporation