Page 1 of 10

10th Annual International

Software Testing Conference in India 2010

Maximizing ROI Through Reusability In Test Automation Across Products

- A Case Study

Vinay Jagadabi & Anitha Bimal

EMC Data Storage Systems (India) Private Limited Bagmane World Technology Center, Marathahalli Ring Road, Mahadevapura, Doddanekundi Village, KR Puram Bangalore 560 048

vinay.jagadabi@emc.com; anitha.bimal@emc.com

Page 2 of 10 Abstract:
This study addresses some of our observations made in a handful of projects while certifying the product suite, and more specifically, in GUI automated testing. It documents, analyzes and resurrects what we consider to be of vital interest. The major findings can be summarized in the lessons learned, covering state of automation, test coverage, test strategy, productivity measurements, testability, build verification, daily integration, migration to a new methodology (of scripting), framework reusability, comparative analysis of test executions manual vs. automation, and best practices utilized while spearheading with automation coverage. The report starts with a brief description of three sample projects and the automation coverage. Then, we discuss our observations and experiences and illustrate the findings. Adaptability of reusable methods/functions/classes is discussed to highlight the impact on savings observed and increasing return on investment (ROI). Tangible and intangible advantages observed are: Early detection of issues, product knowledge enhancement; commonality in devising the methods/classes/functions for reusability; better team coordination; increased execution cycles; enhanced regression scope; enhanced process efficiency; reduced dependency on manual test execution; savings in time, effort and money; and timely status updates to stakeholders. The report concludes with a synopsis of these experiences and with suggestions for future test automation endeavors.

Introduction

Page 3 of 10
An automated software testing system signifies an organizations progress towards establishing a mature quality assurance program. Numerous factors must be considered when planning for software test automation. Automation impingements an organization with respect to the product feature(s), testing tasks performed or test approaches adopted. It is important to comprehend the potential costs and benefits before adopting test automation. Test automation is justified by identifying its potential return-on-investment (ROI). ROI can be calculated as Total benefit derived from automation/Total cost of automation. It is important to enhance the core strategic assets - People, Process, and. Technology in order to make test automation successful and reap its benefits (ROI). In this case study we are considering 3 products namely Product A, Product B and Product C. Product A is an outlook integrated plug-in used for content management and archiving. Product B provides transparent access to content management through Windows explorer. Product C provides transparent access to content management through Windows explorer even when user is offline.

Products

Total number of Test cases

Total number of Automated cases 800 320 365

Savings Observed (Time) 54 man days 22 man days 24 man days

Savings Observed (Cost) 81000 (INR) 33000 (INR) 36000 (INR)

Product A Product B Product C

1500 500 600

Page 4 of 10

Graph depicts ROI observed over a period time

An automation frame work was designed keeping in mind multiple applications and multiple platform support with reusability across different products. In our frame work we separated automation scripts into test driver/test engine, test scripts, and object repository.

Page 5 of 10
The test driver does not contain any application related mechanism. It was an independent component which acted as an interface between test scripts and the application under test. It accepts functions or methods along with its arguments and converts them into GUI actions without depending on the type of application/test case/GUI object.

The object repository contains all application related object information and required methods or function to perform GUI actions on the application. In this case, we have used object oriented concepts that enable reusability across different projects. It was also observed that the maintenance cost was low even when GUI objects changed.

The test scripts contain actual test logic. Minimal automation knowledge is required to write test scripts as it is separated from application GUI objects, methods and functions.

Page 6 of 10
Testing experts

Test scripts

Results

Test Engine
[ Recovery, error handling and results reporting ]

GUI Object repository and common function

Automation experts

Different classes (Word, Explorer, Excel, and Notepad) were made for all common GUI objects and used among the three projects. This helped in fast development and less cost as its shared by three different projects. Reusability matrix for different products:

Page 7 of 10

Return on Investment
Automated testing involves higher upfront costs and should be looked at as a long-term investment where the pay-offs come anywhere between 2-4 years down the road. One has to keep in mind that there are various intangible benefits associated with automation. Performing a return on investment (ROI) for your planned automation can however help you understand right at the beginning the actual returns that you will get from your investments and you can weigh those against the benefits you will gain from automation. An ROI analysis will not only help you determine the various elements associated in calculating the ROI and the approximate cost and benefit involved but it can also help you decide on the types of automation you want, the areas that you can potentially automate, the tools and the skill levels of the testing resources that will be required.

Page 8 of 10 Calculating ROI

Some key parameters to be considered while initializing the ROI exercise:

o Automation focus with respect to test coverage, cost benefits, reduced

testing cycles etc.,

o Evaluate the existing test suite to determine effort and cost involved. o Evaluate the test automation tools
o o Exhaustive knowledge of the product/technology. Test Automation Framework design

ROI Calculator
A ROI calculator aids in understanding the possible returns on Test Automation investment. To calculate ROI, it is important to analyze the effort and cost involved in automation, based on the existing manual testing details. ROI on Test Automation is an output of transition parameters like manual testing cost and test automation investment. A higher ROI will also come through with the following factors:

o o o o o o

Formulated testing process. Increased test coverage to ensure consistency in the deliverables. Elimination of the risk of over sight of quality in repeated tests. Test coverage across multiple environments. More time to focus on new features Increased product quality

Page 9 of 10

Conclusion
Test automation provides significant benefits in terms of savings in execution cycle time. This can be further enhanced by reusing the scripts across different products. The benefits of test automation are not readily quantifiable. However over a period of time, it will be of high value to any product teams.
Test

Page 10 of 10 References:
[i] Douglas Hoffmanstudy explores the cost benefit analysis in test automation with examples. The paper details out the metrics techniques used for test effectiveness computation, setting realistic expectations for management, and ROIfactors that need attention. A careful assessment of various factors relevant for the journey of test automation needs to be looked into. [ii] Keanes whitepaper presents an overview of test automation an approach for ROI calculation.

[iii] www.myprocessanalyst.com [iv] www.methodsandtools.com

Images Source www.myprocessanalyst.com www.methodsandtools.com

About the Authors

Author 1: Vinay Kumar is currently working as Software Quality Engineer in EMC Data Storage Systems (India) Private Limited, Bangalore, India. He has 3+ years of IT experience in functional automation testing using SilkTest. Developed and implemented frameworks for SilkTest and RFT tools. Worked as consultant for SilkTest to many organizations like Sun Microsystems, GE, DeShaw and Wipro. He is certified in Java programming (SCJP).

Author 2: Anitha Bimal is currently working as an Associate Manager in the Intelligence Information Group at EMC Software and Services India Pvt Ltd, Bangalore, India. She has over 10 years of work experience as an Associate Manager, Team Lead and QA Engineer.