Dennis Jah

10/3/2007
INL 660

A Critical Review of Ramler and Wolfmaier’s Economic Perspectives in Test

Automation: Balancing Automated and Manual Testing with Opportunity Cost

Proceedings of the 2006 international workshop on Automation of software test

ACM 2006, pp 85-91

According to studies, Quality Assurance measures, such as testing are a major

cost factor in software development projects. Since testing is imperative, managers, IT

leaders and other decision makers are seeking cost effective ways in carrying on software

testing. In that direction, many have advanced test automation as the most cost effective

way of carrying on software testing as opposed to manual testing. Furthermore, test tools

vendors seizes such moment to advertise their tools as solutions to reducing testing cost.

In a paper presented during the Economics of test automation: test cost and

effectiveness session of an international workshop on Automation of software test

conducted in Shangai, China, Ramler et al discussed when should a test be automated or

manually tested and the trade-off between the two. The authors observe the need for cost

reduction in software testing as such need is exacerbated by pressure mounted on

software testing by growing international competition, shrinking budgets, and the fast

paced development process. The paper critically examines the cost models for automated

software testing used by proponents of automated test to argue in favor of test automation

as a cost reduction solution. Instead it proposes an alternative model based on opportunity

cost. The drive to come up with an alternative model of test automation is based on the

authors’ own experience in supplying consulting services and support and report from

practitioners, which speaks of failures in attempts to automate tests as means of reducing

cost. The study is not conclusive or gives a final verdict that test automation is

categorically not the solution to reduce testing cost, but its aim is to “stimulate

discussion… to support researchers and practitioners to reflect on proposed automation

approaches.” By outlining factors based on the alternative model the authors hope to

influence the decisions on whether or not to invest in test automation.

Ramler et al begin their article by emphasizing the cost factor involved in quality

assurance quoting studies that software testing accounts for more than 50% of total

budget spent on projects. Due to this high cost, the need for finding more cost effective

ways which automated testing proposes to solve cannot be over simplified. While testing

tools vendors and the cost model for software testing supports this preposition, the

authors counter such argument first by analyzing the model calling it “overly simplistic”

and providing an alternative one. The authors assert that the cost model for software

testing and the “universal formula” which support automated testing over manual testing

for cost effectiveness are flawed because they take into account only costs and leave out

benefits. They argue “cost and benefits are both required for an accurate analysis.”

Benefits in testing can be measured in terms of the defects they address or risk mitigation.

Other reasons given for the model’s flaws include the incomparability of automated and

manual testing since the two are really two different processes rather two ways of doing

the same thing. They further criticize the model because additional cost factors such as

the cost of abandoning automated test after changes in the functionality are not taken into

account, etc. its inability to consider entire project context, and that the model

mistakenly considers all test cases equally important. To counter the “overly simplistic”

model, the authors propose an alternative model based on opportunity cost. Borrowing
the concept of opportunity cost from Economics, the authors maintain that the decision as

to which testing alternative to adopt is a trade off between using automated testing or

manual testing. They likened economists’ production possibility frontier to the restricted

budget of testing to show the opportunity cost of test automation as measured in terms of

manual test execution. They argue that management should decide as to which testing

method will yield the maximum benefits. There is always going to be some automation

forgone for every manual test, “once the efficient points on the frontier have been

reached; the only way of getting more automated test cases is to reduce manual testing.”

They are convinced that the opportunity cost model includes both costs and benefits

which makes the analysis more rational. It approaches high cost of testing debacle not

from the standpoint of minimizing the cost of testing but maximizing its benefits. They

suggest that decision makers faced with the problems of either to manually test or

automate must build on the potential benefit of a test case so as to allow a realistic

comparison between alternatives.

Ramler et al provide a more realistic way of comparing manual testing and

automation in terms of making decision for a cost cutting alternative. First, as supported

by “Reconciling Manual and Automated Testing: the Auto Test Experience” (2007),

Leitner et al agree that both manual and automated testing strategies are distinct but

complementary since one may capture cases that the other might not find. So the claim

that automated testing is more cost effective and therefore should replace manual testing

to save cost is a wild one. The authors’ charge that the argument in favor of automated

testing leaves out benefits is in place. Managers and IT leaders need not decide to choose

one way of testing over the other because of cost alone rather the benefits of carrying on
one test over the other. Consideration on optimal benefits should be the driver of

choosing one test over the other. The opportunity cost models captures most if not all of

the factors ignored by the cost model. With both cost and benefits analyzed, managers

and IT leaders can be confident in making the decision

The paper has achieved the objective it set out to achieve, that is to address the

issue of when a test should be automated and the trade off between the two methods.

While the authors maybe convincing, the paper fell short of presenting real world data to

support their proposed alternative. As the authors have mentioned that the paper is only

aiming to stimulate discussion in a different direction, one issue that is left is that the

solution has not been tested in the real world. To be convincing, I would love for the

authors to compare two companies using both approaches and analyze the cost and

benefits.

Like the cost model, the opportunity cost model does not take into account all the

influencing factors. The Authors agreed that incorporating additional factors to the

opportunity cost model for completeness will add further complexities and thereby

reduce its practicality. For further studies, the authors plan to “include further influencing

factors to make the alternative model more realistic.

 Bibliography

Andreas Leitner, I. C. (2007). Reconciling Manual and Automated Testing: The

Auto Test Experience. ACM, 87-102

Rodulf Ramler, K.W.(2006) Economic perspectives in test automation: balancing

Automated and manual testing with opportunity cost. ACM, 85-91