Platform-Independent And Tool-Neutral Test
Descriptions For Automated Software Testing
Chang Liu
Information and Computer Science
University of California, Irvine, Irvine, CA 92697, USA
liu@ics.uci.edu
ABSTRACT
Current automatic test execution techniques are sensitive to
changes in program implementation. Moreover, different
test descriptions are required by different testing tools. As a
result, it is difficult to maintain or port test descriptions. To
address this problem, we developed TestTalk, a
comprehensive testing language. TestTalk test descriptions
are platform-independent and tool-neutral. The same
software test in TestTalk can be automatically executed by
different testing tools on different platforms. The goal of
TestTalk is to make software test descriptions, which
represent a significant portion of a software project, last as
long as the software project.
Keywords
software testing, test description, automatic test execution.
1 INTRODUCTION
Automatic software testing is gradually becoming accepted
practice in the software industry. One reason is that more
practitioners are exposed to the growing number of test
automation tools and have become aware that test
automation is a viable option. Another reason is that test
automation tools are more mature, have more features, and
are thus more appealing than simple capture-playback
tools. The most important reason, however, is that the ever-
shrinking development cycle and higher expectation of
software quality is forcing project teams to seek help from
automation.
Software test automation techniques and tools promise to
achieve shorter turnaround time and cut costs on frequently
repeated tests. There are, however, pitfalls on this path to
higher productivity [1]. Automated test cases are typically
expensive to develop. Those written in programming or
scripting languages are usually hard to understand and
sensitive to changes. Switching testing automation tools,
porting to other platforms, or upgrading the implementation
Permission to make digital or hard copies of part or all of this work for
personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, to republish, to post on servers, or to redistribute to lists, requires
prior specific permission and/or a fee.
ICSE 2000, Limerick, Ireland
© ACM 23000 1-58113-206-9/00/06 …$5.00
713
of the application-under-test often become nightmares for
test automation engineers. In Section 2 of this extended
abstract, we provide a detailed analysis of these problems.
We are developing TestTalk, a platform-independent and
tool-neutral testing language to alleviate these problems.
TestTalk is introduced in Section 3. Section 4 presents one
completed case study and two on-going case studies that
use TestTalk. Section 5 lists related work. Finally, a
summary is presented in Section 6.
2 CURRENT PROBLEMS IN AUTOMATED
SOFTWARE TESTING
Problem 1: Expensive to develop
Current state-of-the-practice in software test automation
uses general-purpose or tool-specific programming or
scripting languages to automate test cases. For example,
general-purpose languages such as C/C++, Perl, Tcl/Tk,
and Expect are widely used in writing automated test cases.
Tool-specific languages, such as BASIC for Rational
Visual Test and 4Test for SilkTest/QAPartner, are also
gaining popularity mainly because of the popularity of the
test automation tools. Programming and scripting
languages are powerful. Test designers have full access to
the power of these languages to achieve their goals. While
this is desirable, writing an automated test case in this way
requires a complete development cycle including design,
implementation, debugging, and testing (testing of the test
case itself), just as any other software artifacts. This task is
demanding in terms of both human resources and
computational resources. Consequently, test automation is
applied to only a very small number of test cases in many
projects.
Problem 2: Sensitive to changes
Automated test cases are indeed programs that drive an
application-under-test by simulating an end user. This is
usually achieved by simulating user interface events, which
may include hot-keys, menu-items, mouse events, and
keystrokes. Naturally, automated test cases are tied to the
semantics of these low-level user interface events, which
are subject to changes such as changed bindings of menu
item hot keys. These changes may happen frequently and
might happen very late in the application development
cycle as application developers fine-tune the application for
release. For test automation engineers, these changes are
equivalent to requirement changes if their task is viewed as
software development. We all know, as a basic software
engineering problem, that requirement changes are hard to
deal with when implementation is already underway. Not
surprisingly, one of the biggest challenges of today's test
automation engineers is to keep pace with the ever-
changing face of the application-under-test.
Problem 3: Difficult to port
Implementation changes are not the only source of
"requirement changes" for test automation engineers. If the
application-under-test is ported to another platform, the
project team should perform the same automated testing on
the new platform. This presents test automation engineers a
significant challenge because: first, very few test
automation tools are cross-platform; second, even fewer
test automation tools behave identically on different
platforms; third, the application-under-test is probably also
adjusted according to the different flavor of the new
platform.
Switching test automation tools can also potentially force
rewriting of all automated test cases. Switching is needed,
among other situations, when a new test automation tool
comes up with a desirable feature that a team's old favorite
tool lacks.
3 OUR SOLUTION: TESTTALK
To alleviate the problems described in Section 2, we
propose a platform-independent and tool-neutral testing
language to describe automation-friendly software tests.
The language that we developed for this purpose is
TestTalk [2,3], a comprehensive testing language based on
XML (http://belmont-shores.ics.uci.edu/TestTalk).
In TestTalk, a test case consists of a series of test actions
and test oracles. A test action represents a particular test
step or a particular series of test steps that testers would
perform to carry out a test or a part of a test. A test oracle
represents a verification step or a series of verification steps
that testers would perform to verify parts or all of test
outputs. A tester can define customized test actions and test
oracles suitable in her application domain. A collection of
tester-defined tests actions and test oracles is called a
dialect. A tester can use both her own dialect as well as
vocabulary provided by the TestTalk system to describe her
test suites and test cases. Since dialects are defined by
testers in the field, who know the application domain better
than language designers, the test descriptions are likely
easier to write and easier to understand.
An example TestTalk test case SimpleText is shown in Fig.
1. In this test case, there are only two simple test actions:
RestartServer and GetPage. Only one simple test oracle
Text is used. When this test case is executed, it first restarts
714
the Apache web server1 on the host-under-test, and then it
tries to get a page from this host machine. After it gets the
page, it checks if a text string "It Worked!" is present in the
returned page. Various test execution tools may be used to
execute this test case automatically.
<TestCase>
<Name>SimpleText</Name>
<Action-RestartServer/>
<Action-GetPage>
<Url>$TestHost</Url>
<Oracle-Text>
<Text>It Worked!</Text>
</Oracle-Text>
</Action-GetPage>
</TestCase>
Fig. 1. A TestTalk test case from the Web Testing
Framework.
TestTalk test descriptions describe only what software tests
do and what they expect as desired outputs, not how to
execute the tests or how to verify the results. We
intentionally separate test automation details from test
descriptions, because test automation details usually have
close ties to specific platforms or specific test execution
tools. This separation ensures that TestTalk test
descriptions are not sensitive to changes in application
implementation details and are portable.
While maintaining high understandability, maintainability,
and portability, TestTalk test descriptions are also
automation-friendly. For each test action and test oracle,
testers can define how it can be implemented on various
platforms using various tools. One way to define this is to
write transformation rules that can translate TestTalk test
actions and test oracles into complete or partial programs or
scripts. For each testing tool on each platform, a set of
transformation rules are developed to implement every test
action and oracle. With a transformation rule set, a test case
description can be transformed to a program or script. This
generated program or script implement the test case on a
particular platform using a particular tool. To use another
testing tool or to test on another platform, only a new set of
transformation rules is needed. All test descriptions remain
the same. Therefore, not only are TestTalk test descriptions
automation-friendly, the exact same test descriptions can
also be automatically executed on different platforms using
different tools.
4 CASE STUDIES
To validate the TestTalk language, we used it in a web
testing framework project. Currently, we are using
TestTalk in two other case studies. In this section, we
briefly introduce these case studies.
Case Study 1: Web Testing Framework
The Web Testing Framework is a framework for Apache-
based web testing. The Web Testing Framework can be
1
Apache is an open-source HTTP server software package.
used by web developers to write test cases that can be
automatically executed to test a web site. It can also be
used by developers of the Apache web server to quickly
verify if a new build of the Apache server functions as
expected. Test cases in the Web Testing Framework are all
written in TestTalk. Web developers or Apache developers
can use a set of predefined test actions and test oracles to
quickly put together a test case. They can choose to execute
the test case by lynx2, Expect3, lwp4, wget5, or curl6. They
can also choose to execute the test case by any other tool
that they think is suitable. Furthermore, they can even add
they own test actions and test oracles. To implement a new
test action or oracle, or to implement an existing test action
or oracle using a new tool, a web tester only needs to
provide a segment of Perl code that complies with a pre-
defined interface protocol. This ability to incorporate new
code is also the reason why the project is named "Web
Testing Framework" instead of "Web Testing Tool".
Expandability was a major design goal.
Our experience with this project showed that TestTalk
supported multi-platform and multi-tool test automation.
Case Study 2: Securibot
Many computer systems were compromised after attackers
exploited known security problems. To solve this problem,
we are developing a framework for automated security
checking and patching. The framework, named as
Securibot, provides a self-operating mechanism for security
checking and patching. Securibot performs security testing
and analysis using security profiles and security updates,
both of which are specified in TestTalk. Securibot can also
detect compromised systems using attack signatures, also
specified in TestTalk. Most important, the Securibot
framework allows system vendors to publish recently
discovered security weaknesses and new patches in
TestTalk so that the Securibot system running on deployed
systems can automatically check out security updates and
apply the patches.
Case Study 3: E-Fluency Test for E-Speak
E-Speak is an on-going project at Hewlett-Packard
(http://www.e-speak.hp.com/). The goal is to provide
uniform services over the Internet. E-Speak defines a
uniform service interface (API) and uniform service
interaction (E-Speak engine) that allows e-services to
dynamically interact to discover, negotiate, broker and
compose themselves to solve a business to business or
business to consumer service request. Since E-Speak is
2
Lynx is a text-based web browser.
3
Expect is a testing tool based on scripting language TCL.
4
Lwp is libwww-perl, the Perl World-Wide-Web library
(http://www.perl.com/CPAN).
5
GNU wget is a network utility to retrieve files from the
World Wide Web.
6
Curl is a command line tool for getting data from a URL.
715
essentially a set of protocols, implementations other than
the HP implementation are expected. It is critical to quickly
and accurately verify conformance of a new E-Speak
implementation.
We are working on the E-Fluency Test project
(http://belmont-shores.ics.uci.edu/E-Fluency), which is a
certification test suite for E-Speak. Because of the
heterogeneous nature of E-Speak, E-Fluency Test has to be
platform independent. TestTalk is an ideal language to use
in such a situation.
5 RELATED WORK
Ostrand and Balcer's TSL (Test Specification language) is a
language designed to describe functional test suites by
specifying categories. It provides property and selector
mechanism, which testers can use as constraints to annotate
the test specification so that they can control the
complexity and number of the tests. TSL does not address
test oracles and test result verification.
TDD (Test Data Definition language) from Sun
Microsystems has three types of elements to help generate
more effective test data by specifying test categories. These
three types of elements are properties, test variables, and
test directives. TDD does not provide support for test
automation.
4Test is a commercial testing language from Segue. While
it supports a higher abstraction level so that test
descriptions are not closely tied to the GUI of the
application-under-test, it is tool-specific.
6 SUMMARY
Software test automation has great potential in improving
software quality and reducing software development costs.
However, understandability, portability, and
maintainability of executable test descriptions must be
addressed before the potential of test automation can be
fully realized. TestTalk is an attempt in this direction. The
Web Testing Framework case study shows that a
customizable test framework can be built on top of
TestTalk. We are working on two more case studies to
collect further evidences.
REFERENCES
[1] Chang Liu, Debra J. Richardson, "Programming
Languages Considered Harmful in Writing Automated
Software Tests", Technical Report 99-09, Inform. &
Comp. Sci., University of California, Irvine, Feb. 1999.
[2] Chang Liu, Debra J. Richardson, "TestTalk, A Test
Description Language: Write Once, Test by Anyone,
Anytime, Anywhere, with Anything", Technical Report
99-08, Inform. & Comp. Sci., UC Irvine, Feb. 1999.
[3] Chang Liu, "TestTalk: A Comprehensive Testing
Language", the 14th IEEE International Conference on
Automated Software Engineering, Doctoral
Symposium, Oct. 1999, Cocoa Beach, Florida, USA.