Dynamic White Box Testing: Code coverage

Laboratorio dei sistemi software

Andrea Bei

Autore delle slide: Dott.Marco Bianchi

Test case

Specification of inputs and outputs required to reveal defects

Consists of the pre-test state and environment, the test inputs and expected outputs and state. Expected outputs may include data, messages or exceptions generated by the application under testing.

Test suite

A collection of test cases, typically related by a testing goal or implementation dependency.

Homework Review

Develop the following program:

The program reads three integer values from an input dialog. The three values represent the lengths of the sides of a triangle. The program displays a message that states whether the triangle is scalene, isosceles, or equilateral

Create a set of test data for the program

Test case Test suite

Homework review

A valid scalene triangle? e.g. (1,2,3) and (2,5,10) are not possible dims. A valid equilateral triangle? A valid isosceles triangle? e.g. (2,2,4) are not possible dims. At least three among all permutations? e.g. (3,3,4), (3,4,3), (4,3,3) One side has a zero value? One side has a negative value?

Homework review

The sum of two of the numbers is equal to the third? At least three among all permutations? The sum of two of the numbers is less than the third? At least three among all permutations? All sides are zero (0,0,0) Noninteger values such as (2.5,3.5,5.5) Wrong number of values Specify the expected output from the program? ...

Homework results

Highly qualified professional programmers score, on the average, only 7.8 out of possible 14. Point of the exercise: the testing of even a trivial program is not an easy task.

Test Case Design

Question:

Given constraints on time and cost, the key issue of testing becomes: What subset of all possible test cases has the highest probability of detecting the most errors?

Test strategy

An algorithm and/or a collection of heuristics used to identify interesting test cases for an application under testing.

The two basic strategies

Methods

Equivalence partitioning Boundary-value analysis Cause-effect graphing Error guesting

Statement coverage Decision coverage Condition coverage Decision-condition coverage Multiple-condition coverage

Testing Task

Code Coverage

Test coverage attempts to address questions about when to stop testing, or the amount of testing that is enough for a given program. Ideal testing is to explore exhaustively the entire test domain, which in general (and in practice) is impossible. It is a dynamic white box testing technique.

Code Coverage

A code coverage model calls out the parts of an implementation that must be exercised to satisfy an implementation-based test model.

Coverage, as a metric, is the percentage of these parts exercised by a test suite.

Hundreds of coverage models have been published and used since the late 1960s. Nearly all support implementation-based testing.

Control flow graph

As such, most coverage models rely on control flow graphs, which give an abstract representation of the code. Control flow graph mainly relies on the following concepts:

code segment predicate condition

Code segment

A code segment consists of one or several contiguous statements with no conditionally executed statements.

That means once a segment is entered, all the statements involved will execute. The last statement in the segment must be another predicate, a method exit, a loop control, a break, or a goto. The last part of a segment includes the predicate or exit expression that selects another segment but does not include any of subsequent segments code.

Predicate and condition

A predicate expression contains one or many conditions that evaluate to true or false. One condition corresponds to each boolean operator in the predicate expression.

Predicates are used in control statements: if, case, do, while, do until, for, and so on. The evaluation of a predicate results in transfer of control to one or many code segments. A predicate with multiple conditions is called a compound predicate.

Examples: code segments in canonical loop structures

Control flow graph representation

A control flow graph (CFG) describes code segments and their sequencing in a program. It is a directed graph in which:

A node corresponds to a code segment; nodes are labeled using letters or numbers. An edge corresponds to a conditional transfer of control between code segments; edges are represented as arrows.

The entry point of a method is represented by the entry node, which is a node with no inbound edges. The exit point of a method is represented by the exit node, which is a node with no outbound edges.

Flow graphs for canonical loop structures

For Loop

Flow graphs for canonical loop structures

While Loop

Flow graphs for canonical loop structures

Until loop

Example CFG for a method

Path expression

A path corresponds to a sequence of segments connected by arrows.

A path is denoted by the nodes that comprise the path. Loops are represented by segments within parentheses, followed by an asterisk to show that this group may iterate from zero to n times.

An entry-exit path is a path starting with the entry node and ending with the exit node

Examples of Entry/Exit paths

ABH ABCH A(BCDEFG)*BH A(BCDEG)*BH A(BCDG)*BH A(BCDG)*BCH

Compound predicates

CFG with collapsed nodes

1. Triangle (a, b, c: Integer): String 2. IsATriangle: Boolean 3. begin 4. if (a<=b+c or b<=a+c or c<=b+c) 5. then IsATriangle := true 6. else IsATriangle := false fi; 7. if (IsATriangle) 8. then 9. if (a=b and b=c) 10. then return Equilateral 11. else 12. if (ab and ac and bc) 13. then return Scalene 14. else return Equilateral fi; fi; 15. else return Not a triangle fi; 16. end
3. begin 4. (a<=b+c or b<=a+c or c<=b+c)?
yes

no

5. IsATriangle := true

6. IsATriangle := false

7. IsATriangle?
yes no

9. (a=b and b=c)?

yes no

15. Not a triangle

10. Equilateral

12. (ab and ac and bc)?

yes no

13. Scalene
16. end

14. Equilateral

Test coverage analysis

Test coverage analysis uses some adequacy criteria to guide the testing process.

This increases the confidence that an implementation has been thoroughly tested.

Common coverage criteria

Statement coverage Decision coverage Condition coverage Decision-condition coverage Multiple-condition coverage

Statement coverage

Execute every statement in the program at least once. (A, B, C) := (2, 0, 3) (A, B, C) := (2, 0, X)

Statement coverage

In a CFG with each compound predicate collapsed in a node, each node is executed

3. begin 4. (a<=b+c or b<=a+c or c<=b+c)?

yes no

5. IsATriangle := true

6. IsATriangle := false

7. IsATriangle?
yes no

9. (a=b and b=c)?

yes no

15. Not a triangle

10. Equilateral

12. (ab and ac and bc)?

yes no

13. Scalene 16. end

14. Equilateral

Decision (or branch) coverage

You must write enough test cases that each compound predicate (decision) has a T and a F outcome at least once.

ace & adb | acd & abe

(2, 0, 4)&(1, 1, 1) (3, 0, 3)&(2, 1, 1)

ace & adb

(2, 0, X)

Decision (or branch) coverage

In a CFG with each compound predicate collapsed in a node, each edge is executed

3. begin 4. (a<=b+c or b<=a+c or c<=b+c)?

yes no

5. IsATriangle := true

6. IsATriangle := false

7. IsATriangle?
yes no

9. (a=b and b=c)?

yes no

15. Not a triangle

10. Equilateral

12. (ab and ac and bc)?

yes no

13. Scalene 16. end

14. Equilateral

Condition coverage

You must write enough test cases that each condition into a compound predicate (decision) has a T and a F outcome at least once

Condition coverage

In a CFG each edge is executed

Multiple condition coverage

All possible combinations of conditions

ace abe abe adb

We missed acd!!

Relationships between coverage strategies

statement coverage

decision coverage

condition coverage

all paths

multiple condition coverage

Test coverage, in practice

In practice, the three basic criteria most commonly used are statement coverage, decision coverage and condition coverage.

It has been suggested that the combination of these three criteria can achieve 80-90% or more coverage in most cases.

It is important to note that test coverage is not enough by itself:

100% test coverage cannot guarantee achieving error-free software.

Test coverage, in practice

In practice, coverage data are collected by instrumenting the code.

Instrumentation is typically done on a copy of the source code, while debugging changes are made to the un-instrumented code. Manual instrumentation is error-prone and very time consuming. In contrast, automatic instrumentation is easy and cost-effective. Automatic instrumentation is performed by a coverage analyzer, which is part of typical test automation environment.

Final Remarks

It is recommended not to use a code coverage model as a test model.

Instead, established test strategies (e.g., equivalence, domain) should be used to devise test suites, while coverage are used at the same time to analyze generated test suites adequacy. Coverage reports can point out a grossly inadequate test suite

Bibliography

The Art of Software Testing (second edition) by Glenford J. Myers et. al., John Wiley & Sons, Inc. (2004). Cap. 4 (p. 44-52). Some examples and figures are extracted from:

Lecture notes of Software Testing (2IW30) course (technisce universiteit eindhoven) by Judi Romijn (2006). Lecture notes of A Course on Software Test Automation Design by Doug Hoffman (2003). Practical Software Testing, Ilene Burnstein, Springer (2002).