Load Testing For Less

7 tips on affordable load testing for startups and SMB

WHITEPAPER: May 2010

Table of Contents
Load Testing: The Breaking Point for Software Companies. The 'Peak Performance' Dilemma ....... New Approaches..... Load Testing Fundamentals.... Software Stress Testing........ Hardware Stress Testing...... Soak Testing ....... Spike Testing ........ Configuration Testing ........ Platform Testing ....... Load Testing For Less.... Simulated Load Testing ..... Load Testing Tools Common Scenarios......... Live Load Testing ...... Common Scenarios. Hybrid Load Testing ..... Common Scenarios. Load Testing Tips..... Test What Matters Most .... Redevelope The Metric System Watch and Listen To Your Results .... Test In The Wild.. Know What You're Looking For .. Don't Test Too Much (or too little)... Be Environmentally Friendly.... 2 2 2 3 3 3 3 3 3 3 4 4 4 5 5 5 6 6 7 7 7 7 8 8 9 9

About uTest..

10

It's easy to cry "bug" when the truth is that you've got a complex system and sometimes it takes a while to get all the components to co-exist peacefully. ~ Doug Vargas

WHITEPAPER: Load Testing For Less

Load Testing: The Breaking Point for Software Companies

The Peak Performance Dilemma At what point will your applications performance begin to degrade? How many concurrent users can it support? How do the JavaScript-based components of your website behave with 50, 100 or 1000 active users? Where are the bottlenecks between your code base, database, CDN and load balancers? How is all this affecting the user experience? And your page views, bounce rates, and ultimately, your revenue? Answers to these vital questions - questions that cause even the most seasoned tech execs to lose sleep (and sometimes their jobs) - are determined by comprehensive load and performance testing. The problem here, especially for start-ups and small businesses, is that traditional load testing has proven to be prohibitively expensive and requires specialized skills, as even base packages can range from $20k to $50k. Unfortunately, far too many companies reach a point where the benefits of load testing are outweighed by the costs - hence the dilemma of testing and attaining peak performance.

New Approaches To Load Testing Innovations in crowdsourced testing and open-source tools have resulted in a load testing methodology that is far more adaptable to growing software companies. Startups and small businesses now achieve load testing coverage to match that of most large enterprises and they do it by combining real-world testers with the latest in automated technology. This brief whitepaper will outline the basics of affordable load testing. While discussing how this emerging methodology works, it also will address the fundamentals of load testing, including tips on: Targeting core features: Learn how to prioritize the most vital features of your website, especially when developing test scripts and common user scenarios. Defining key metrics: How should you gauge download speed? Active server pages? Processing time? Well show you how to define key metrics and avoid needless analysis. Load testing frequency: How often should your company run a load test? What are the big events that make it a necessity? Environments: Should you run load testing on a staging environment or on production? Well outline the benefits and danger of each approach. And more: Successful load testing requires the right combination of technology, people and processes and well show you the best way to leverage them each step of the way.

WHITEPAPER: Load Testing For Less

Load Testing Fundamentals

A unique characteristic of load testing is that its performed for many different reasons. Though its sometimes viewed as companies simply wanting to break things software companies rarely have the same objective. Here are a few more common load testing terms and scenarios: Software Stress Testing: In this instance, a company is looking to determine where the software begins to break down under extreme load. Simply put, it is people wanting to break things. Hardware Stress Testing: Similar to software stress testing, this procedure performed to determine where a benchmarked set of hardware starts to break down under extreme load. A company, in this instance, seeks to ensure that their system is robust under normal usage. Soak Testing: This is a simulation that often takes place over the course of days or weeks that tests for performance degradation caused by things like slow memory leaks or even something as simple as hardware fatigue or subpar hosting conditions. It focuses on long-term usage. Spike Testing: This is performed to determine scalability under a sudden increase in load. The ability of an application to shift gears and go from handling a moderate number of users efficiently, to handling a large number of users adequately is of course very important. If your company makes the front page of a news website or an aggregate like Digg or Slashdot and your normal load spikes heavily you will want to be prepared. Configuration Testing: Here a company would want to compare relative performance based on sets of server parameters. If a company is unsure of what hardware and software options are best for their application, they can run a series of tests with the exact same user load, and monitor the relative performance in each scenario. Platform Testing: This procedure is geared towards companies that want to test end-user functionality on a broad spectrum of hardware and software. This allows them to test their software's front end under a wide variety of hardware and software configurations. This helps them determine their software's requirements, which they can publish as their suggested minimum configuration.

We will now examine methods that will provide unique insight into your application, no matter what type of load testing you plan on performing

WHITEPAPER: Load Testing For Less

Load Testing For Less

Simulated Load Testing There are a number of simulated load testing tools on the market today - both commercial and free. Of course, there are many advantages to using the commercial tools, as they tend to have cleaner UIs, more efficient reporting tools and documentation, to name a few perks. However, be careful not to discount the impact of having an entire open-source community reviewing the code for bugs, making contributions, and making sure that the testing tools are spyware-free. That said, heres a short list of tools you may find helpful (listed alphabetically): BrowserMob Free Load Testing Tool BrowserMob uses real browser users for website load testing, providing better AJAX support, clearer reporting of errors and an experience that more accurately mimics that of the real user. Run concurrent tests on your website with real browsers, not simply HTTP requests, and catch errors that other low cost providers miss. The Grinder Generic Jython Load Tester The Grinder uses the Java implementation of Python (Jython), which is a script-only, feature rich load testing platform. It is possibly the most powerful lightweight free load testers on the market, but also the most difficult to master. No GUI to speak of, very little in the way of examples, but it has a terrific FAQ. It isn't useful for front end automation, but otherwise The Grinder is flexible and infinitely useful if you can master it. This is the programmer's preferred load testing tool. JMeter Graphical Load Tester JMeter is without a doubt, the most downloaded free load testing software on the planet. It has a GUI written in Java, which means you can use it on just about any operating system, so your testing is truly portable. It has a flexible scripting language, as well as plugins for all sorts of strange test scenarios. Best of all, it is one of the most completely documented load testing applications out there, with ample tutorials and great community support. It is useful for just about every test type you'll eventually run into. OpenSTA - HTTP and HTTPS Heavy Load Tester OpenSTA is one of the most flexible load testing tools on the market today. Not only can it use recorded scripts and play them back, but it can convert the recorded macro into a powerful script that is easily edited and injected as a hypertext stream. OpenSTA is therefore best of both worlds, and is suitable for just about every form of load testing. SOASTA Leader in Cloud Testing SOASTA leverages the cloud to offer on-demand load & performance testing, enabling you to test the real-world performance of your web apps. This functional testing can either be in separate from SOASTAs load testing runs, or performed while an app is under simulated load to test how it works under pressure.

WHITEPAPER: Load Testing For Less

Of course, these are just a couple of tools that you should have on hand for general debugging. Your operating system will have basic network usage monitoring, so you can see if you're having any sort of bandwidth issues on your end. But there is no better network interface snooping tool than Wireshark, so make sure you have it on hand, just in case you need to debug the IP stack. Common Scenarios for Simulated Load Testing Simulated load testing services are typically leveraged by companies that meet the following criteria:

High-Traffic Apps: If you're expecting a lot of concurrent visitors or users, this method can provide you with hundreds of thousands of simulated users - helping you to identify any loadrelated weaknesses and assure performance during peak usage. Fast-Approaching Deadlines: Most automated tools require weeks of planning and on-ramping. Using the tools and methods mentioned above, companies can set up and execute a simulated load testing cycle in a much shorter time span. Changing Apps: Recent changes to any link in your infrastructure chain (i.e. features, functionality or even content) Limited Internal Resources: Traditional load testing tools or services often burden in-house resources, especially QA and IT staff. These tools eliminate much of the heavy lifting, letting you concentrate on more urgent matters.

Live Load Testing Unlike the great majority of load testing methodologies, this procedure requires NO automated or synthetic testing tools. Instead, a team of real live testers will converge on the core features of your application simultaneously. Thanks to advances in crowdsourced testing, it has become extremely easy to engage a global community to quickly converge on whatever features you deem most important. Logging bugs and usability issues, live testers can provide an important new layer to your metrics. And because theyre professional testers, they know what to look for and how to report/document issues that they may encounter. Furthermore, many startups and small businesses - particularly B2B companies - arent the least bit concerned with preparing for 500,000 or a million simultaneous users (though it would be nice). Instead, they would much rather know whether or not 200 live users can simultaneously complete a checkout, sign in to their account, and perform other key actions without any noticeable deterioration. In other words, they want to test against real-world conditions. Common Scenarios Since simulated load tests only tell part of the story, here are a few situations where live load testing makes the most sense:

Flash and Multimedia Apps: When launching a web app that incorporates flash, video and other streaming media, your automated tests may indicate that all is well, even when it's not. Live testers, on the other hand, can determine if the sound quality is off, if the video is pixilated, the file takes too long to buffer or other problems that simulated load testing rarely catches. Limited Traffic Apps: As mentioned earlier, real-world conditions might not entail hundreds of thousands of users. Most load testing tools require too much effort and incur great costs to

WHITEPAPER: Load Testing For Less

leverage them for such a comparatively small assignment. If your app has moderate traffic and this is part of your business model it is much better to gauge how real visitors will experience your application. Startups and SMBs: For most small and medium-sized companies, traditional load testing tools are seen as overkill, not to mention time-consuming and requiring specialized skills. Forgoing such tools and relying on the expertise of professional testers enables you to simulate the optimal amount traffic in a quick, easy and cost-effective manner. End-to-End Load Testing: For companies that want in the wild testing that mirrors the TRUE user experience and to avoid complicated automation live load testing is an ideal solution.

Hybrid Load Testing Combining live testers with simulated load tools, hybrid load testing offers greatest insight into your applications performance. Here, a team of testers will execute test scripts and perform common actions on the front end of the software, while automated tools place your application under synthetic load. Rather than relying entirely on backend automation, real testers geographically dispersed can verify Flash-based components, cross-browser performance, feature sets, page load times and other functions. Common Scenarios Hybrid load testing is employed by companies that seek to maximize testing in the following areas:

Flash and Multimedia Apps: For companies that need complete load testing, but also want to verify that the Flash, video and other multimedia components of their web application are working properly with no degradation under peak load, hybrid testing makes the most sense. Load-Related Defects: Some bugs notably GUI bugs only show themselves when a web application is under heavy stress. Hybrid testing enables companies to discover these defects through first hand bug reports. Load-Related Feedback: Through hybrid testing, companies can sort through the data coming from synthetic load testing, while also gathering real-world feedback from software-savvy users all over the world adding an entirely new layer of performance-related feedback.

And now, the seven tips for affordable load testing

WHITEPAPER: Load Testing For Less

Tips For Affordable Load Testing

Test What Matters Most What are the most important actions your typical end user will take? What features will have the greatest negative impact if they fail? This process should not be taken lightly. For many companies, this step will include a thorough check of e-commerce features such as online checkouts, shopping carts, order forms and other features. If these actions are not so clear, begin by examining your web traffic through Google Analytics or something comparable. Redevelop the Metric System Page load speed isnt an issue for everyone. For others, it is a matter of life and death (financially speaking anyway). So while its easy - and perhaps necessary - to dwell over active server pages, requests per second ratios and other technical indicators, the data most worth analyzing is that which directly involves your end user, so it would be wise to tier your metrics accordingly. Here are a few suggestions for server-based load testing metrics: Number of users and/or sessions Average session time Number of page views Average page views per session Peak period Number of hits Average page size Most requested pages Average time spent on page New users vs. returning users Frequency of visits (e.g., 75% of users made one visit) Demographics Browser, browser version, Java script support, Java script enable/disable, etc. Watch AND Listen to Your Results When running simulated load testing, remember that such virtual systems rely on emulators for video and audio output. Also, it does not consume all of the RAM, but only a part of it. So it can be extremely confusing when you stated in your system configuration that your PC has 2GB RAM on board, but in fact your virtual machine is configured to use only 512MB. Realize that when it comes to audio and video, real users are essential.

WHITEPAPER: Load Testing For Less

Test In the Wild Most likely, you didnt build your application for automated users. You built it for real users. Why then do so many companies rely solely on automated testing results to verify functionality under stress? Until recently, it was because there was no such mechanism by which a company could utilize real testers while a system was under load. But as we outlined in the Hybrid and Live Load testing section of this whitepaper, that obstacle has been overcome. By leveraging real testers with automated tools, companies now have access to entirely new layer of data. Better yet, they are able to identify and resolve issues that would have been reported later in the development process.

Know What Youre Looking For For an effective load test and to avoid wasting thousands of dollars in the process its essential that you know what you need to get out of it. It may seem obvious, but its a routine mistake, especially for startups and small businesses. To reach any actionable conclusions (which is the entire point of load testing) companies must be able to identify where specific failures occurred in your delivery chain. Did the major failure first occur with your database or code? Or did the issues originate from your web server, load balancer, content delivery network or ISP? Companies that fail to pinpoint these issues will learn a valuable lesson in the costs of redundant testing. Dont be one of them.

(Sample chart from a real-world load test, with each line representing different transactions within the same app.)

WHITEPAPER: Load Testing For Less

Dont Test Too Much (or too little) How often should companies run load testing of their application? Of course, this answer will vary from company to company, but generally speaking, these are ideal times for startups and small businesses to consider load testing: Major product launches or new features Changes to your cloud configuration New servers, content delivery networks, etc. Significant spike in users and/or traffic (+50%)

Be Environmentally Friendly One of the more obvious questions to arise when preparing to run a load test is, Should I test on production? Again, there is no right or wrong answer. While it is certainly helpful to have a staging environment that is identical to your production, it is not an absolute necessity. But to prevent wrecking your production site, it is helpful to start small with your load testing efforts, as most defects are discovered by low-volume tests (and by live testers). Once these issues have been identified, you can then move on to more comprehensive load testing. At that point, it will be more helpful to have a staging environment.

WHITEPAPER: Load Testing For Less

*Special thanks to uTester Bill Ricardi for his contributions on load testing and performance*

About uTest
Headquartered near Boston, uTest is the world's largest marketplace for software testing services. The company provides real-world QA services including simulated, live and hybrid load testing services through its community of 25,000+ professional testers from over 160 countries around the world. To date, more than 500 companies - from web start-ups to enterprise software firms - have signed up to get their apps tested by the uTest community. uTest enables companies to launch higher quality products; get their desktop, web and mobile applications to market faster; and control the cost of testing. Customers specify their requirements for tester experience, location, language, OS and browser, and uTest selects the right testers for each project. And because uTest is on-demand, companies only pay for completed test cycles. A brief online video is available at www.utest.com/intro. uTest can be contacted at: uTest, Inc. 153 Cordaville Road Southborough, MA 01772 p: 1.800.445.3914 e: info@utest.com w: www.utest.com

WHITEPAPER: Load Testing For Less

10