Page 1 6/17/2008

Performance Testing: Open Source Approach

Intr oduction :-

For any web application, the end to end response time for a user is a
critical issue. As the number of users accessing the same web application increases, the
performance of the application decreases due this concurrent user load on the server. To ensure
business continuity, organizations often want to know and understand the application behavior
under projected user load before it goes live. The challenges of successfully conducting an online
business are as follows:

1) High User Volumes

2) Low response times for web requests
3) The overall reliability of Service

Exi st ing Op tion s for Perf or mance Test ing

Some popular commercial tools available for performance testing and diagnostics are

i) Mercury Load Runner

ii) IBM Rational Performance Tester
iii) IBM Rational Robot
iv) Borland Silk Performer

These tools in general provide user with excellent GUI options to record the transaction between
client and server. These tools also provide enough flexibility to modify the scripts in order to
simulate the dynamic data passed between client and server. User can then create user\business
scenarios to generate load on the server and monitor the different aspect of server response like
response time, resource utilization etc.

The commercial tools also provide tuning consoles that can gather the data from different tiers of
the application and present an integrated view of performance of these layers\tiers under load.

Popular Open so ur ce Perf or mance Test ing

Sol ution s

Though there are many open source solutions available under GNU that provide an alternate
solutions to commercial tools, for the purpose of this paper, we have selected the most popular
tools for performance testing of web (HTTP\HTTPS) applications.

i) OpenSTA
ii) Apache JMETER
iii) WebLoad
 Page 2 6/17/2008

Open Sour ce T ool s vs. Co mmer cia l tool s

Practically, Open source tools lag from their commercial counterparts in the following areas

 Effective Integrated Monitoring and analysis capability

 User network speed simulation
 IP spoofing capability

T he Pr actica l A ppr oac h to us e O pen S our ce

Load Tes ting Too ls
The open source tools can be cost effective substitution for commercial tools to load test your
web application, provided the nooks are well taken care of. The load Test planning and scoping
assumes greater significance when testing with open source tools. A careful study of application
architecture is also important to understand the protocols between client and first server along
with the deployment model for monitoring. Limitations for open source tools have to be well
understood and a workaround have to be found. For example, if user bandwidth simulation is a
important criterion, third party bandwidth throttles can be integrated with open source tools to
provide the same functionality.
As a conclusion, a careful evaluation of the open source tool to understand it's limitations and
knowledge of available workarounds becomes very important for it's implementation success in
any performance testing project.

Open Source Tools for Performance Testing:

Comparative Study

Evaluation
WebLoad
Criteria Description OpenSTA Apache JMeter

The communication
protocols that can
be captured HTTP,FTP, HTTP/S, WAP, AJAX,
Protocols
manipulated and SOAP/XML- ActiveX, Java, Web
replayed by the HTTP 1.0 / 1.1 / HTTPS RPC,JDBC services. Multi-protocol
tool. (SSL), SOAP/XML scripts also possible
It’s a GUI tool. So
there are many GUI
Playback
Extended logging Listeners, which are
functions
Replaying of the supports view of used to capture the Extended log view,
script and script parameter values and record and replay which displays request
debugging facilities. Server messages. time messages. and response data.

The capability of
Recording tool to capture the
Protocol Data protocol specific
data being passed Available for Available for the
between client and HTTP\HTTPS Available for the supported protocols
First Server. supported protocols
 Page 3 6/17/2008

Scripting is done
Called SCL, it uses a using the
proprietary, “BASIC” like Beanshell/Java
language that has functions. However
special automation most of the WebLoad Test scripts
Scripting
The medium used scripting facilities. Is performance testing are created in Java
language
to represent the limited in available features like Cookie Script, a familiar
captured protocol functions, such as string Manager, Logic industry standard,
data and manipulation and Controller, Samplers eliminating the need to
manipulate the data supports direct DOM etc are available learn another
for play-back. addressing. using console. proprietary language.
In keeping with its
open standards
approach, WebLOAD
enables testers to
seamlessly embed
their own Java,
The Beanshell/JAVA ActiveX or COM
Extensibility functions can be objects in the test
defined and can be script. The flexible
used as Plug-in as framework of
SCL script modules can per the specific WebLOAD easily
be defined in requirements. enables its expansion
The ability to â??Includeâ?? files. in support of other
increase the Open Source, therefore protocols, using the
functionality of the new tool functionality can WebLOAD Extensibility
tool. be added using C++. SDK.
Has low-level HTTP
protocol view and
provides graphical tree
representation of the
Document Object Model
(DOM) structure.
Viewable captured HTML An intuitive interface
Scripting rendering and enables novice testers
Interface addressable server- They are many to create scripts with
header table. Some consoles provided by drag-and-drop
language sensitive, Meter which are used simplicity. Experienced
The interfaces syntax color coding to edit the script. The users will appreciate
supplied by the tool functionality Beanshell/JAVA the ability to edit the
application for the functions can also be script directly in the
purpose of script used to enhance familiar java script
editing. scripts. language.

SmartCopy feature of
WebLoad allows easy
to implement manual
correlation through
GUI feature without
actually needing to do
scripting for the same.
Correlation
However this feature is
Manual correlation using Manual correlation not available in Open
The task of graphical interactive using the console Source Version of
substituting values DOM structure. Facility known as Regular WebLoad
in dynamic data to to automatically generate Expression Extractor
enable successful script code to aid is use to capture the
playback. variable substitution. value in the variable.
 Page 4 6/17/2008

Automatic Sever
Cookie management
feature in WebLoad
allows user to isolate
himself from managing
the dynamic serve side
cookie.
Cookie
Depending on the
Management
“Script content” setting
Detection, recording HTTP header cookies in the “Record
and playback of are managed Options” dialog,
HTTP cookies. Both HTTP header cookies automatically using WebLOAD will record
tools need are managed the console known as the different headers
additional code to automatically and can be HTTP Cookie into the agenda,
manage JavaScript manipulated manually if Manager. including the cookie
generated cookies required. header.
Extensive facilities for
data entry including
wizard interface to
automatically generate
Automatically test data. Standard
changing dynamic functions for sequential,
data values passed random and pseudo- Facilities for data entry
Parameterization from client to the random data-file access. in the script through
server during POST Have standard common the Global data Entry
requests for more locking facilities for file. The data file can
accurate emulation maintaining uniqueness be shared across
of real users. Often of parameters for an Parameterization can virtual users running
essential for individual load injector or be done in the on the same machine
session across all injectors on a console known as even under multiple
management. distributed test. Users Parameters. iterations

Resource Monitoring is
supported by running
performance
measurements
manager (PMM) from
There are many behind the firewall fo
consoles which are the supported servers.
known as Listeners. (Windows NT/2000/XP
These consoles give Unix platforms using
Monitoring
Resource usage the performance of SNMP
information is the servers. Unix platforms using
captured during Very basic monitoring RSTATD
execution. It can be NT and SNMP Monitors for Tomcat servers Windows .NET
shown during are available for • Monitors threads Framework
execution and used monitoring the hardware and memory usage Microsoft IIS
to build related resource such as • More useful for Sun One (former
performance memory, CPU utilization stress testing than iplanet 6)
reports. etc. monitoring Apache using SNMP)

Not Possible in open

source version. The
load can only be given
from one box in open
source version of
Webload

Distributed tests
Supports multiple load-
injectors managed by a
single controller. Uses
The ability to TCP/IP if on the same
distribute the network or the Web-relay
generation of load feature uses HTTP to Supports multiple
across multiple control load-injectors load-injectors
load-injector located within remote managed by a single
machines. DMZs. console
 Page 5 6/17/2008

Extended Multi IP
support, which is of
two types.
Per Client - preserving
current behavior,
meaning different IP
per client but using the
same IP for all rounds.
IP Spoofing This is the default
The ability to value.
emulate the Per Round - support
behavior of different Not available out of for using a different IP
IP addresses box in JMeter. from the pool per
accessing a Not available. The However the same client, per round, until
system. Particularly requests going from one can be emulated by the pool is exhausted
useful with Load machine have the same writing custom then cycle back to
balancing systems. IP imprint functions in Beanshell start.

Cache related
functions are
available in
Caching The ability to Can control browser Beanshell/JAVA which
emulate the caching cache emulation during can be use to
of pages as playback and control emulate the caching
performed by a web setting for each of pages as Not available in
browser. individual vuser. performed by a web. webload OpenSource

It can be done with

The ability to the help of some the
User connection emulate the functions used in the
speed emulation different network No inbuilt features to Beanshell/JAVA but it
speeds that can be emulate user-connection is not easy and not Not available in
used by real users. speed emulation. yet used. webload OpenSource

WebLOAD Console
displays online reports
of load session
statistics. The user
Simple charts and may create report
graphs sufficient for views that display any
analyzing key load subset of the wide
related statistics and array of statistics
resource usage collected by
Reporting & monitors. Resource WebLOAD. The user
analysis usage monitors supports can toggle between a
graph overlays. Can be graphical or textual
The facilities to exported to Microsoft (table) view of each
examine and Excel. No license report.
investigate the restrictions on OpenSTA
results of a test distribution thus stats WebLOAD enables the
including timers and can be viewed by any user to load results of
monitored user with access. -Free a previously executed
resources and tools and excel macros It creates some of the session for comparison
display the results available through public graphs but reporting with the results of the
in graphical format. user-forums. is not available. current session

Main resource used is

The ability of the RAM. Tested for simple
tool to generate ASP pages, reaching up
numbers of virtual to 3,000 users on load Resource limitations Resource limitations
users and the generator of 1Gb RAM are number of are number of threads
corresponding on a single P4 processor threads and RAM. and RAM. On a
Scalability resource usage. & W2K. Unconfirmed Approx 1 vuser per machine with
Actual resource use report of a limit of 1664 ½ Mb RAM for processor speed of
depends on the VUs per Win2K machine NT/W2K. Windows 600/7333 MHZ and
number, size and for complex scripts. 95, 98 & Unix are less RAM of size 256MB,
complexity of the Suspected thread efficient. Max. Approx 100-150 threads could
scripts. limitation. No licensing 1,500 VUs per PC. be run successfully
limitations.
 Page 6 6/17/2008

References

1. www.testreflections.com
2. www.jakartha.apache.org
3. www.opensta.org
4. www.webload.org