AUTOMATED TEST

GENERATION
Muhammed . KALKAN

Automatic Test Generation (ATG)

Index
What is ATG?
ATG Types
Why do we need ATG?
What are the main benefits of ATG
ATG Methods
Testing Methodologies
Automatic Test Pattern Generation
Relevant Examples

What is ATG?

Testing an engineered system is the

most important part of the product
revealed.
Engineers are working on and
putting alot of effort into this issue.
Since testing process is required for
all obtained engineering products,
engineers decided to develop an
automated test mechanism instead
of generating test cases manually.

What is ATG?

Automated Test Generation includes

test cases , scenarios and the
evalution of the obtained result such
as passed and failed.
Given inputs and obtained outputs
are evaluated automatically.
The Main Idea :
TEST COMPLETELY
AUTOMATICALLY

ATG or ATPG?

ATG is an engineering subject which

can be applied to both Computer
Sciences and other engineering
specialities like Electronics or
hardware circuits.
ATG is a term which is generally
used for Computer Sciences
especially software testing.
On the other hand for hardware
testing Automatic Test Generation is
usually referred as Automatic Test
Pattern Generation.

ATG or ATPG?

ATPG is defined as follows:

ATPG is an electronic pattern
automation method/technology used
to find an input (or test) sequence
When applied to a digital circuit, it
enables testers to distinguish
between the correct circuit behavior
and the faulty circuit behavior
caused by defects.

ATG.But WHY??

As years have passed,computer hardwares

and its capabilities grow bigger and bigger.
Result 1: Software types and their
capabilities are varied.
Result 2: New software demand on market
for jobs.
Result 3: Bigger income ,more firms to
compete for market ration.
Result 4: Requirement for strong,bug free
and developed in small amount of time
softwares

ATG.But WHY??

Result 5 : Software Engineering

and Testing studies.

ATG.But WHY??

Software testing is important

Software

errors cost the U.S.

economy about $59.5 billion each
year
Improving testing infrastructure
could save 1/3 cost

Software testing is costly

Account

for even half the total cost

of software development

Benefits

Reduces Testing Time

Reduces Testing Cost
Reduces human errors while
testing

ATG Methods

Data-Driven ATG
xUnit Frameworks for ATG
Examples

: JUnit and NUnit

Model-Based ATG
Keyword-Driven ATG

xUnit

Various code-driven testing

frameworks have come to be
known collectively as xUnit.
Based on a design by Kent
Beck, originally implemented for
Smalltalk as SUnit, but are now
available for many programming
languages and development
platforms.

xUnit

The overall design of xUnit frameworks

depends on several components.
Test Fixtures
A test fixture is the set of preconditions or
state needed for a test to run. Also known
as a test context.
Test Suites
A test suite is a set of tests that all share the
same fixture.

Test Execution
The execution of an individual unit
test proceeds as follows:

xUnit

The setup() and teardown() methods serve to

initialize and clean up test fixtures.

Assertions
An assertion is a function or macro that verifies
the behavior of the unit under test. Failure of an
assertion typically throws an exception,
aborting the execution of the current test.

Data-Driven ATG

Data-driven testing is a methodology

used in Test automation
Test scripts are executed and verified
based on the data values stored in one or
more central data sources or databases.
Databases can range from datapools,
ODBC sources, csv files, Excel files, DAO
objects, ADO objects, etc.
Data-driven testing is the establishment of
several interacting test scripts together with
their related data results in a framework
used for the methodology.

Data-Driven ATG

Variables are used for both input values and

output verification values: navigation through
the program, reading of the data sources, and
logging of test status and information are all
coded in the test script. Thus, the logic
executed in the script is also dependant on the
data values.

An example of xUnit Framework Automatic

Testing by using Data-Driven Test Methodology
is as follows:

Example

Put the data that

varies from test
to test into the
Data-Driven Test
file that the
interpreter reads
to execute the
tests
For each test it
does the same
sequence of
actions to
implement the
Four-Phase Test

Example (FPT)

Set-up the test fixture

Try the test case
Evaluate the result
Release the fixture

Keyword-Driven ATG

It is also known as table-driven testing, actionword testing

It can be used for manuel testing as well as
automated testing
The advantages for automated tests are the
reusability and therefore ease of maintenance
of tests which have been created at a high
level of abstraction.
The Keyword-Driven ATG methodology divides
test creation into two stages.

Planning Phase
Implementing Phase

Keyword-Driven ATG
Planning Phase

The application (or the

requirements for the application)
is analyzed to determine which
operations and objects will need
to be tested.

Example
A web-questionnaire application will
require a large amount of text entries. By
identifying which operations should be
encapsulated into keywords, the efficiency
and maintainability of the tests are
maximized.

Keyword-Driven ATG
Implementation Phase

The implementation stage differs depending on the

tool or framework used.
Often, automation engineers implement a
framework that provides keywords like check and
enter.
Testers or test designers (who dont have to know
how to program) write test cases based on the
keywords defined in the planning stage, which have
been implemented by the engineers.
The test is executed using a driver which reads the
keywords and executes the corresponding code.

Keyword-Driven ATG

Other methodologies use an all-in-one

implementation stage.
Instead of separating the tasks of test
design and test engineering, the test
design is the test automation.
Keywords, such as enter or check are
created using tools in which the necessary
code has already been written.
This removes the necessity for extra
engineers in the test process, because the
implementation for the keywords is already
a part of the tool.

Keyword-Driven ATG

Benefits

This methodology requires more planning and a

longer initial time-investment than going directly to
the test creation stage and recording a test.
It does make the test creation and test maintenance
stages more efficient and keeps the structure of
individual tests more readable and easier to modify.
The more abstract keywords are, the more reusable
they are, and therefore the easier a test is to
maintain.
As well as reducing the cost and time spent
maintaining and updating tests, the modular
structure of Keyword-Driven ATG means that new
tests can easily be created from pre-existing
modules.

Keyword-Driven ATG

Another advantage is the reduction

in technical know-how required for
the test automation process.
In the first approach, technical knowhow is only required by the
engineers that implement the
keywords.
In the second approach, even this is
not required, which means that the
test team is capable of entirely
automating tests, even without
programming knowledge.

Model Based ATG

Model-based testing is software testing in

which test cases are derived in whole or in part
from a model that describes some (usually
functional) aspects of the system under test
process

The model is usually an abstract, partial

presentation of the system under test's desired
behavior.

The test cases derived from this model are

functional tests on the same level of
abstraction as the model

Can not run abstracted test cases.Need

executable ones.

Model Based ATG

Diagram description :
Model-Based Test Systems

Model Based ATG

Many ways to produce test cases.

No best method for producing tests.
Use your test interest to define your test
cases.
Design decisions is often known as
"test requirements", "test purpose" or
even "use case"

Model Based ATG

Use model and test requirements to

derive abstract test cases
Implementation Extra Information to
abstract test suit to produce executable
test cases

Model Based ATG

Execute the test

Evaluate the reports

Model Based ATG

There are a few ways to deploy the

Model-Based Testing
Online Testing: A model-based testing
tool connects directly to a system
under test and tests it dynamically.
Offline Generation of Executable
Tests: A model-based testing tool
generates test cases as a computerreadable asset that can be later
deployed automatically.

A collection of Python classes that embodies

the generated testing logic

Model Based ATG

Offline Generation of Manually

Deployable Tests: A model-based
testing tool generates test cases as a
human-readable asset that can be later
deployed manually.
A PDF document in English that
describes the generated test steps

Another Step :
Test

Generation

Model Based ATG

Model-Based Testing has a good potential in

aspect of automation.
Test case generation is one of the issues that
ATG is subject to
To find test cases, Model-Based structure is
interpreted as a kind of Finite State Machine to
see different states that the system might be in
There are a few technics for generating test
cases.Most knowns are as follows:

Model Based ATG

Test case generation by theorem proving

System model is defined as predicates,kinds of

logical expressions
These expression are processed and proved
afterwards

Test case generation by constraint logic

programming

Selecting test cases satisfying specific

constraints by solving a set of constraints over a
set of variables

Model Based ATG

Test case generation by model checking

Test case generation by symbolic execution

We provide a model of the system under test and a

property we want to test to the model checker
Searching for execution traces in an abstract model.
In principle the program execution is simulated using
symbols for variables rather than actual values

Test case generation by using an event-flow

model
With a graphical user-interface (GUI) front-end
is called the event-flow model that represents
events and event interactions

Example

TestMaster automates the generation of

tests for call processing features developed
for the 5ESS-2000 Switch
This test uses Model-Based Test Generation

Results obtained here as follows.

A test generation productivityimprovement

of just over 90%

Automatic Test Pattern

Generation

ATPG is an electronic design automation

method/technology used to find an input (or
test) sequence that, when applied to a digital
circuit, enables testers to distinguish between
the correct circuit behavior and the faulty circuit
behavior caused by defects
The generated patterns are used to test
semiconductor devices after manufacture,
In some cases to assist with determining the
cause of failure

Automatic Test Pattern

Generation

The effectiveness of ATPG is

measured by the amount of modeled
defects, or fault models, that are
detected and the number of
generated patterns.
We can estimate test quality by
effectiveness of ATGP

Automatic Test Pattern

Generation

A defect is an error introduced into a device

during the manufacturing process.
A fault model is a mathematical description of
how a defect alters design behavior.
A fault is said to be detected by a test pattern if,
when applying the pattern to the design, any
logic value observed at one or more of the
circuit's primary outputs differs between the
original design and the design with the fault.
There are two steps that ATPG should take to
detect fault.

Automatic Test Pattern

Generation

Fault activation: Establishes a signal

value at the fault model site that is
opposite of the value produced by the
fault model
Fault propagation: Moves the resulting
signal value, or fault effect, forward by
sensitizing a path from the fault site to a
primary output.

Sequential ATPG

Searches for a sequence of vectors to detect a

particular fault through the space of all possible
vector sequences

Vector Sequence: A sequence of values for

circuit input

In Sequential Circuits due to the presence of

memory elements, the controllability and

observability of the internal signals (in
general)are much more difficult than those
in a combinational circuit
Complexity of sequential ATPG much higher
than that of combinational ATPG.

Algorithmic Methods

Testing very-large-scale integrated circuits with

a high fault coverage is a difficult task because
of complexity.
Many different ATPG methods have been
developed to address combinatorial and
sequential circuits.
Some examples:

Pseudorandom test generation is the simplest

method of creating tests. It uses a
pseudorandom number generator to generate
test vectors

Algorithmic Methods

The D Algorithm was the first practical test

generation algorithm in terms of memory
requirements. The D Algorithm introduced D
Notation which continues to be used in most
ATPG algorithms.
Fan-Out Oriented Algorithm: It limits the
ATPG search space to reduce computation
time and accelerates backtracing.

Automatic Test Pattern

Generation

ATPG can fail to find a test for a

particular fault in at least two cases.
The fault may be intrinsically
undetectable
it is possible that a pattern(s)
exist, but the algorithm cannot find
it

References

Automated Test Generation, (From a Behavioral

Model), James M. Clarke, Lucent Technologies
Automatic Test Pattern Generation ,
http://en.wikipedia.org/wiki/Automatic_test_pattern_
generation
Data Driven Tests, Gerrard Meszaros,
http://xunitpatterns.com/Data-Driven%20Test.html,
2007
Data-Driven Testing,
http://en.wikipedia.org/wiki/Data-driven_testing

References

Test Automation,
http://en.wikipedia.org/wiki/Test_automation
Test Automation Framework,
http://en.wikipedia.org/wiki/Test_automation_frame
work
A Survay on Automatic Test Case Generation,
M.Prasanna, S.N. Sivanandam, R.Venkatesan,
R.Sundarrajan, Department of Computer Science
and Engineering, PSG College Of Technology,
http://www.acadjournal.com/2005/v15/part6/p4/
Keyword-Driven Testing ,
http://en.wikipedia.org/wiki/Keyword-driven_testing

References

Model Based Testing,

http://en.wikipedia.org/wiki/Model-based_testing

Improving Automation in Developer Testing:

Achievements and Challenges, Tao Xie,
Department of Computer Science, North Carolina
State University

xUnit , http://en.wikipedia.org/wiki/XUnit

The End

Thank You
Q&A