WHITE PAPER

Cloud-enabled vis--vis on-premise Performance Testing: A face-off

Abstract
In this white paper we talk about how you can integrate on-premise and cloud-based options to build an effective performance testing strategy. We will also discuss the available cloud-based performance testing solutions and whether they can effectively fill the gap areas created by the on-premise approach. Finally, we share our recommended strategy which incorporates both on-premise and the cloud approach and presents a robust performance testing plan.

Impetus Technologies, Inc. www.impetus.com May - 2012

Cloud-enabled vis--vis on-premise Performance Testing: A face-off

Table of Contents
Introduction .............................................................................................. 2 Limitations of the on-premise approach .................................................. 3 Benefits of cloud-based load testing ........................................................ 3 Challenges facing the cloud-based approach ........................................... 4 Data security ................................................................................ 4 Testing intranet applications ....................................................... 5 Repeatability and consistency ..................................................... 5 Debugging and bottleneck identification .................................... 5 Cloud-enabled solutions ........................................................................... 6 Using on-premise tools in the cloud ............................................ 6 Provisioning the tool from the cloud ........................................... 7 The Hybrid approach ................................................................... 7 Impetus recommendations ..................................................................... 8 Conclusion............................................................................................... 11

Introduction
Performance engineers worldwide are harnessing the infinite computing power of the cloud to test the performance of their web, mobile and cloud applications. The immense popularity of cloud-enabled performance testing is based on the fact that it helps to overcome the limitations of a test strategy that relies entirely on on-premise performance testing. However, before moving to cloud-based performance testing, it is important to examine the on-premise model, so called, because all its components including the tool, recorder, controller, system under test, etc., are deployed within the local environment.

Cloud-enabled vis--vis on-premise Performance Testing: A face-off

Limitations of the on-premise approach

Administrators need to be hired to maintain and support the environment. Since setting up this environment is a costly affair, there is a possibility that you may be requested to share this set up with other teams or for other activities, such as for a staging environment or for User Acceptance Testing. Also, it is very difficult to scale the test environment to simulate sporadic peak loads because it involves procuring additional servers, licenses etc. This might result in extra expenditure. However, this is mostly about the infrastructure considerations.
The first and the very basic step of on-premise performance testing are setting up a test environment. This involves putting together a suitable infrastructure, procuring servers, and software in order to host the load generation hardware and system under test. Setting up this kind of an environment requires considerable investment and effort and has overheads such as license and maintenance costs.

The other significant aspect that we have to consider is that the on-premise model does not measure up to the realistic simulation of user behavior. Since your application will get hits from all corners of the globe, it is important that its performance is measured from multiple geographies. Also, while conducting the tests from within the local environment, we ignore aspects such as network latency, firewall behavior, DNS servers and other important components in the end-to-end delivery chain. By doing this, we might not be testing the actual user experience. These are some of the limitations of the traditional on-premise approach. cloud-enabled performance testing can address these gap areas. At Impetus, we believe that performance testing with the cloud enables testing teams to take a big step forward in conducting more efficient and realistic, large scale tests. Moreover, cloud technology enables organizations to realize significant savings in cost and time.

Benefits of cloud-based load testing

The cloud can help set up the test environment and have it running in minutes. You just do not have to worry about the licensing costs of the software, Operating System, tools, etc. Furthermore, it eliminates the need to spend on the purchase and maintenance of the hardware and load testing tool that you would otherwise have to buy in the on-premise model. Using the pay-as-you-go approach, you can rapidly set up the testing infrastructure that you need, when you need it, and only for as long as you need

Cloud-enabled vis--vis on-premise Performance Testing: A face-off

it. From a business standpoint, the cloud lowers the Total Cost of Ownership (TCO), while increasing flexibility. Often, the web application experiences sporadic surges in traffic. Generating the load for large scale tests to mimic these unanticipated spikes in the production traffic, may require tens or even hundreds of machines. While purchasing and configuring these systems requires a significant investment in time and money, with the cloud, you can rapidly set up as many load generating machines as you need, on-demand using automated provisioning of required hardware. Automated provisioning ensures that the load generators are consistent and optimized to create the desired load. Another advantage that the cloud offers is the realistic simulation of test scenarios. With the cloud, you can execute load tests that access your web application as your users willfrom the outside of your firewalland validate all the infrastructure components, including the firewall, DNS, network equipment, and ISPs. These tests are significantly accurate and enable you to evaluate the real-world effects of third-party components, such as content delivery networks, analytics servers, and ad servers. And since your users will not access your application from one particular geographical location, a realistic load test will require simulation from different geographies. This can be achieved via the cloud. Cloud-based solutions can also be accessed via browsers from multiple devices, such as your PCs, Smartphones, Tablets, etc., and therefore, you can choose to run and analyze performance tests from anywhere, anytime. While cloud-based solutions offer several benefits for performance testing, the cloud does have its own set of limitations. Lets see what they are.

Challenges facing the cloud-based approach

Data security
We will start with data security, which is a major concern for any performance testing activity happening via the cloud. This is a critical challenge as the test scripts, test data, and test results are moved to the cloud. There is a need to figure out ways by which sensitive data is not accessed via any unauthorized source.

Cloud-enabled vis--vis on-premise Performance Testing: A face-off

Testing intranet applications

Another problem area is testing intranet applications on the cloud. Performance testing service providers actually need to work with the clients IT department in order to receive permission to access the application from the cloud. This involves exposing the intranet servers to the external world and requires relevant security mechanisms in place, to avoid any unauthorized access.

Repeatability and consistency

The other challenges are repeatability and consistency of the test results. Often in performance testing, you need to precisely measure the effect of changes made to the application code or configurations. For example, you may need to determine any improvement that results from resolving a specific defect, or evaluate performance for a range of cache sizes and other related settings. With cloud load testing, such precise measurements are difficult because of the variations in Internet traffic and bandwidth availability. Such variations can make it almost impossible to duplicate uniform test conditions. On the other hand, on-premise testing can achieve consistent and repeatable results by isolating effects that are due to the application or infrastructure.

Debugging and bottleneck identification

Debugging and bottleneck identification is another major challenge when the tests are conducted from the cloud. When load testing uncovers a problem, the next step is identifying which layer in the application chain is causing the problem. You can use monitors to check the performance metrics such as hits, average response time per request, and average bandwidth for each layer in the chain. You can also employ application performance management (APM) solutions to identify bottlenecks in the code. These techniques work reasonably well when the environment is controlled and free from external factors. Let us now move on to the next problem areaTest Assets Management. Since the cloud instances may not be available all the time, it is important to have a pertinent test data management strategy in place, as well. A proper back up mechanism for test scripts, test data and test results has to be created, so that they are available for reference, at a later stage. After going through the benefits and challenges of using the cloud for performance testing activities, let us look at the solutions available in the market today that can help in optimally leveraging the cloud.

Cloud-enabled vis--vis on-premise Performance Testing: A face-off

Cloud-enabled solutions
These solutions can be classified into three major categories. The first and simplest solution is to move the current on-premise tool to the cloud, as it is. A few companies also offer automated Ccoud provisioning portals for load generation, as well as executing and controlling tests. The first two categories have a few limitations and hence a third category called the Hybrid approach is in place. This involves using on-premise tools to generate on-demand load, using the cloud. It supports high concurrency testing while still providing the benefits of a controlled environment and security. Impetus recommends a two-phase approach, which we believe brings the best of on-premise and cloud options and can help you to build a successful performance testing strategy. Let us examine each of these strategies in detail:

Using on-premise tools in the cloud

As we know, there are some standard components of performance testing tools such as controllers, load generators, recorders and analyzers. The variation in the deployment models depends on where all these components are placed when we conduct the tests. You need to choose the appropriate deployment model based on your needs. In the case of the first model, all components of the tool are moved to the cloud. This is the simplest way to leverage the cloud for performance testing and is used by tools like JMeter, Grinder and Load Runner. This solution helps in generating a realistic simulation, unlike the on-premise model. You also have the option of instantiating as many load generators as you require, quickly and on-demand. Since there is no hardware procurement involved, you are unlike to incur massive expenditure on setting up the infrastructure. Another benefit of this approach is that there is no learning curve involved. The teams are already familiar with the tool and can start testing quickly. Also, there are no license costs involved. The process becomes simplified and you are only charged for the period you use the cloud infrastructure. However, there are several limitations in this model. It requires manual configuration of instances and knowledge about cloud

This model is suitable to test cloudbased applications or Internet applications, since these get hits from all parts of the world. The scenarios can be designed in a manner that simulates a realistic load across multiple geographies.

Cloud-enabled vis--vis on-premise Performance Testing: A face-off

APIs. The instances can only be accessed via RDP, Putty or SSH, which involve physical connection to the machines. Also, the test data resides in the cloud and after the instance is terminated can be lost. The user has therefore to come up with an appropriate data archival strategy before terminating the instances.

Provisioning the tool from the cloud

Lets now talk about another popular approach, which is provisioning the tool from the cloud. This involves using the tool provisioned on the cloud for load generation, execution and test management. All the components of a load testing tool are deployed on the cloud and the user can access the solution from the browser. This provides an easy interface that can be accessed from anywhere, at anytime. Though this approach is very popular, it has a few limitations that you need to know about. Since the tool is accessed via the Internet, data security is one of the major concerns. All the test artifacts and results are stored on the cloud, thereby, presenting a challenge to data confidentiality. Also, since the tests are run over the Internet, appropriate firewall access needs to be provided for the intranet applications. Due to varying network conditions, the tests might not be repeatable.

Impetus SandStorm enables you to provision the tool from the cloud. One of the major benefits of this approach is on-demand load generation, which means the user can scale its load tests at any given time. Pre-created instances help in quick scaling of the load tests. This process is suitable for mobile and cloud-based applications that have a large user base, and scales quickly.

It is also difficult to identify performance bottlenecks in such cases, as the number of variables is relatively high. This approach is well suited for measuring the end user experience and exercising the entire delivery chain including firewalls, ISP, CDN, routers etc. in a typical application deployment. Now, let us talk about the Hybrid approach that tries to address the limitations of the earlier two approaches.

The Hybrid approach

In this case, the components of the performance testing tool are deployed either on the cloud or on-premise. There are many combinations possible here, but let us take up the most likely scenario to understand this better.

Cloud-enabled vis--vis on-premise Performance Testing: A face-off

This is because regular testing can be done on premise not only to get the performance numbers, but also to determine the bottlenecks in the application. In case there is need to test an application with peak volumes, high concurrencies and more transactions, the Hybrid model can be used.

We have the load generators on the cloud and all the other components, such as, the controller, recorder and analyzer on-premise. This model is more suitable for organizations which have already invested in performance test labs and have a hardware set up. These organizations can use the Hybrid model when they need to scale

Impetus SandStorm and a few other tools offer this option. The controller is deployed on-premise in this model and enables you to use the integrated resource monitoring and diagnostic features of the tool. This also helps address any security concerns over the crucial data as the controller remains in-house. Since the data is available locally, it can be archived and referred to at any suitable time. This model too suffers from a couple of limitations related to remote access and parallel test executions. Since, the controller is installed within the local network; the tests can only be executed via physical access to these machines.

Impetus recommendations
We will now talk about some of the work we have done and the best practices we have deployed successfully across multiple customer engagements. These best practices can help you maximize the advantages, and minimize the challenges of load testing with the cloud. As we discussed earlier, while the on-premise approach has numerous limitations, it cannot be replaced completely by the cloud. And while cloud offers scalability along with cost-effectiveness, it also has some pertinent challenges associated with it. Therefore, let us examine the fourth approach that integrates the cloud and on-premise solutions for a successful and effective performance testing strategy. The on-premise model is suitable for initial performance tests and debugging and forms the first stage of our process. You can use the local environment to create test scripts, prepare test data, and validate the test scripts for runtime data changes. Then, can you conduct internal tests with a medium load to quickly identify and resolve preliminary issues.

Cloud-enabled vis--vis on-premise Performance Testing: A face-off

The source of any issue identified in the first stage is clearly within the firewall (because no other external systems are involved in the test, as of now). Therefore, it is easier to pinpoint and fix internal problems when they are not being compounded by other issues that will originate outside the firewall. With this two-stage process, you do not have to wait for the application to be deployed and accessible from the Internet to test it. You can test the application internally, and earlier in the application lifecycle, when the defects are easier and less expensive to fix. With internal testing, you have much more control over the environment, so you can precisely measure the effect that a code or a configuration change might have on the application performance. The tests are repeatable, and therefore, the tuning/optimization changes can be quickly validated. This approach also helps in creating scripts locally, and retains them for scaling at a later stage, using the cloud. After the performance has been validated internally, you can proceed to the second stage, which is cloudbased load testing. The cloud can be used to test large scale, a peak load that validates the entire delivery chain of the application. You can compare the results of the same test scenario run internally and from the cloud, to get a better handle on how the application server and network infrastructure contribute to the overall response times. cloud testing is based on pay-per-use model, and since you have already tested the application internally on your existing hardware setup, you have an option to reduce the amount of cloud testing and save on costs. You can also detect potential network issues at the cloud provider level. And having gone through this model, we will talk about the best practices associated with using cloudbased performance testing solutions. As discussed, one of the primary concerns of users is data security. We recommend encrypting the communication between your controller and load generators. This will help secure the data sent to the load generators during the test (including account information), as well as the data that is retrieved (including error messages). If possible, use SSL to secure the communication between the browser and the test server. Finally, ensure that your load generators are secured with their own firewalls to protect them from outside threats. To ensure that your load generator machines in the cloud are capable of generating larger loads, we would recommend tuning the system to support the creation of a high number of sockets and threads per process. Additionally, allocating an appropriate heap size for Java-based load generators will help in such scenarios. The default settings for a typical machine allow all programs to share resources fairly. In the case of load generators, the machine is dedicated
9

Cloud-enabled vis--vis on-premise Performance Testing: A face-off

to a single task, so you can improve the performance by allocating a significantly larger share of the available resources to the load generation tasks. Automated provisioning can also save time and efforts to scale the test environment to support high concurrency and transaction volumes. It also makes sure that the load generators are consistent and optimally tuned as they are generated from a single image. We recommend using an integrated solution for on-premise and cloud. If the solutions are interoperable, it reduces the learning curve for the team. The test scripts, scenarios can be re-used internally as well as on the cloud. Having covered the four available cloud-based performance testing solutions, we would like to point out that Impetus two-phased approach appears to meet all the criteria of a robust performance testing strategy. Almost all other approaches, other than cloud provisioning, need tool familiarity. Even the cloud provisioning option does not allow reuse of the scripts created locally for large scale testing from the cloud. A major advantage that the cloud provisioning and Impetus approaches offer is the ability to run multiple tests in parallel. This is very critical for large organizations where multiple parallel releases are happening for different applications. Another advantage is that they both provide browser-based testing to simulate the end user experience. This is very important, owing to the rapid changes taking place in client side technologies like HTML5, FLEX, etc. Each of these four solutions that we have spoken about have their own merits and demerits. The selection of specific tools depends on factors such as costeffectiveness, type of application, performance objectives, etc. We recommend that you establish the performance testing goals and then decide on the preferred solution.

10

Cloud-enabled vis--vis on-premise Performance Testing: A face-off

Conclusion
At the end it can be said that organizations that use only lab testing or only cloud testing are at a disadvantage. This also holds good for the companies that use different cloud-based solutions for these activities. We at Impetus believe that a solution that integrates the on-premise and cloud-based approaches will be the most beneficial. We have put together a quick check list to help you decide when to choose an on-premise option vis-a-vis the cloud. The on-premise environment is a highly controlled one, hence it is well suited to preliminary, low-scale testing. You can start testing early in the development cycle when the code is not yet ready for high concurrency testing. This helps in resolving performance issues early. Also, since the environment is consistent, it ensures repeatability of the tests. Any code change and tuning can be quickly certified for performance improvements in case of on-premise. The cloud, meanwhile, offers elasticity on demand i.e. large scale tests with high concurrency can be executed anytime. There is no need to procure additional hardware, licenses etc. This is well suited for applications that can scale quickly, such as mobile and cloud applications. From the cloud, we exercise the application under test from an external world, thus bringing in the CDN, ISP and routers etc. in the picture. This is very similar to how an end user will be using the system.

About Impetus Impetus Technologies offers Product Engineering and Technology R&D services for software product development. With ongoing investments in research and application of emerging technology areas, innovative business models, and an agile approach, we partner with our client base comprising large scale ISVs and technology innovators to deliver cutting-edge software products. Our expertise spans the domains of Big Data, SaaS, Cloud Computing, Mobility Solutions, Test Engineering, Performance Engineering, and Social Media among others. Impetus Technologies, Inc. 5300 Stevens Creek Boulevard, Suite 450, San Jose, CA 95129, USA Tel: 408.213.3310 | Email: inquiry@impetus.com Regional Development Centers - INDIA: New Delhi Bangalore Indore Hyderabad Visit: www.impetus.com

Disclaimers
The information contained in this document is the proprietary and exclusive property of Impetus Technologies Inc. except as otherwise indicated. No part of this document, in whole or in part, may be reproduced, stored, transmitted, or used for design purposes without the prior written permission of Impetus 11 Technologies Inc.