BITP 3213:

Software Testing &

Quality Assurance
Week 6: Test Design Techniques

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Overview

• Design techniques
• Black Box techniques
• White Box Techniques
• Experienced Based Techniques
• Factors in Choosing Techniques

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Terminology
Test case
• a set of test inputs, execution conditions and expected
results developed for a particular objective;
• smallest entity that is always executed as a unit, from
beginning to end
A test procedure are detailed instructions for the set-up,
execution, and evaluation of results for a given test
case.
Testware are any deliverable associated with the test
process, e.g. test
cases, documentation, test results.

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Example of test case

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 3 Types of Design Techniques
• Structured base (White box)
• Create tests primarily by analysis of the structure of the
component or system
• Attempt to identify logical errors and verify test coverage
which typically reveal bugs the way system is built
• Specification based (Black-box)
• Create tests primarily by analysis of the test basis
• Look for bugs in the way system behaves
• Experience based (attacks, checklists, exploratory)
• Create tests primarily based on understanding of the system,
past experience, and educated guesses about bugs
• Look for bugs in the places other system have bugs
• Black box tests are specification – or experience based,
functional or non functional.
Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi
White-Box Black Box

Test Approach Test Approach

Structural testing approach Functional testing approach
focuses on application internals. focuses on application externals.
Program based. Requirements or specification
based
Characteristics Characteristics
Module design Functional
Implementation Requirements, standards
Do modules/functions meet Correctness
functional and design
specifications?
Do program structures meet Business forms
functional and design
specifications?
How does program work? Does system meets business
requirements?

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

White-Box Black Box

Type of Test Type of Test

Unit Testing System Testing
Integration Testing User acceptance testing
System Testing (menu structure,
web page structure)
Test Design Techniques Test Design Techniques
•Fault Insertion •Equivalence Analysis
•Error Handling •Boundary Analysis
•String testing •Path Analysis
•Statement Coverage •Use case
•Decision Coverage •Decision tables
•Condition coverage
•Path coverage
•Cycomatic Complexity

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 White Box – Structural Based
• Important of structural based : Is a best practice to measure the
extent of structural coverage of specification based and experience
based, so that you can look for important pieces of system’s
structure which are not being tested.
• Used system structures to derived the test cases (e.g code,
design, database tables, queries, system design..etc.)
• 3 Main ways to design structural:-
1. To analyze control flows in the code – to achieve desirable
code coverage
2. To analyze data flows using code and data structure – to
achieve desirable data flow coverage
3. To analyze interfaces, classes, call flows by looking at the
API,;hardware, software, network system design, database
tables, integrity constrains – To achieve desirable design
coverage
Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi
 White Box – Structural Based
Structure-based tests are based on how the system works
inside
• They guarantee that all independent paths within a
module have been exercised at least once
• They exercise all logical decisions on their true and
false side
• They execute all loops at their boundaries and within
their operational bounds
• They exercise internal data structures to ensure their
validity

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 White Box – Fault Insertion

• Forcing return codes to indicate errors and

seeing how the code behaves.
• Popular method
• return NULL in a function.
• Erroneous input testing (valid/invalid/values outside
the boundary range of parameters)
• Offers a way to measure the effectiveness of
false tests being performed.

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 White Box – Error Handling

• The test engineer needs to ensure that the

application returns error message properly.
• Example:-
• “Common Object Request Broker Architecture
[COBRA] user exception error” – has little value to
the end user
• Error handling tests seek to uncover instances
where the system application does not return a
descriptive error message, but instead crashes
and reports run time error.
Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi
 White Box-String Testing

• Involves the examinations of a related group of

modules that constitute a software functions –
interactions of the functions
• It determines whether modules work successfully
to form a cohesive unit and whether the software
unit produces accurate and consistence result.
• Parameter passed from function to function within
the module
• Correctness in term of data type and data validity

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

White Box- Statement Coverage
• The technique requires that every statement in
the program be invoked at least once.
• Weakness:-
• Does not verify decisions (path/results) – Boolean
expression but is affected for computational
• E.g logical operator (AND /OR)
• E.g. comparison operator (> Vs >=)

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 White Box- Decision Coverage

• The technique requires that every possible

decision in the program to be invoked at least
once.
• Also known as branch coverage

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Example of code coverage

Read p IF A>B THEN

Read q C=A-B
IF p+q > 100 THEN ELSE
Print “Large” C=A+B
ENDIF ENDIF
IF p>50 THEN READ D
Print “pLarge” IF C=D THEN
ENDIF Print “Error”
ENDIF
Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi
White Box - Basic Paths and Tests

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 White Box – MC Cabe’s
Cyclomatic Complexity
• Theory – Complexity increases with the increase
number of decisions in the modules
• Thus, prone to more bugs

Rule of thumb for quality risk analysis to say that

we expect higher likelihood of bugs in those
parts of the system which are high complexity

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

Black Box – Specification Based
• Concerned on the behavior or features of the software
product : the inputs and the outputs
• Synonymous with system testing & acceptance
testing. BUT can also occur in unit or integration
testing
• User participant is important to black box testing – as
the users are more familiar with the results that can be
expected from the business functions ; Thus, during
data generation phase, it is imperative that the end
users give as much input as possible.
• Sets of inputs are derived mainly from functional
requirements/specifications

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Black Box – Equivalence
Partitioning
• Inputs of a software are divided into a groups
that are expected to exhibit similar behavior.
• Can be found for valid and invalid data.
Partitions can also be identified for outputs,
internal values, time related values
• The techniques helps tester to identify data
(input/output) for testing

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

Example :Equivalence Partitioning
Testing problem – the specification for a
software system for validating expenses
claims for hotel accommodation includes the
following requirements :
 There is an upper limit of RM 70 for accommodation
claims
 Any claims above RM 70 should be rejected and
cause an error message to be displayed
 All expense amounts should be greater than RM 0
and an error message should be displayed if this is
not the case
Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi
Example :Equivalence Partitioning

0 70

Hotel 0< Hotel Charge Hotel Charge

Charge<=0 <=70 >70

Test Case Hotel Charge Partition Tested Expected

ID Output
1 50 0< hotel charge <=70 OK
2 -25 Hotel charge <=0 Error Message
3 89 Hotel charge >70 Error Message

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Black Box : Boundary Value
Analysis
• A refinement of equivalence partitioning that selects
the edges or end points of each partition for testing.
• Equivalence partitioning looks for bugs in the code
that handles each equivalent class
• Boundary values are members of equivalence
classes that also look for bugs in the definitions of
the edges (LLV, LLV-delta, HLV, HLV-delta)
• Can only be used when the elements of the
equivalence partition are ordered
• Non functional boundaries (capacity, volume, etc.) can
be used for non functional testing too.
• Can be used for test data selection
Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi
 Example :Boundary Analysis

0 70

-1 +1 69 71
Test Case Expected
Hotel Charge Boundary Tested
ID Output
1 -1 Error Message
2 0 0 Error Message
3 1 OK
4 69 OK
5 70 70 OK
6 71 Error Message
Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi
Black Box : Use Cases/Scenario
Tests
• Use case is used to show the user interaction
with system, any action which of tangible value
to the user
• Design various cases that reflect challenging real
world uses of product
• Each flow can be mapped to one test case.
• Some object oriented design methodologies
include use cases, so this can be an easy source
of tests

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Screen Prototype

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Use Case

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Black Box : Decision Tables

• Helps to select actions/conditions/logic

• Mainly apply for business rules-complex rules.
• Deciding how to process an order based on size,
stock on hand, state to which to ship, and so forth is
often in business rules
• These can be shown as flow charts or as tables
• The decision tables can make for instant test cases
• Strength – it creates combinations that might not
otherwise have been exercised during testing

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 ATM Decision Tables
Condition 1 2 3 4 5
This decision table shows
Valid card N Y Y Y Y the business logic for an
ATM, Notice that the
Valid Pin - N N Y Y
dashes indicate conditions
Invalid PIN=3 - N Y N N that aren’t reached as part
of this rule. The rules are
Balance OK - - - N Y mutually exclusive, in that
only one rule can apply at
any one moment of time
Action
Reject Card Y N N N N Notice that the business
Reenter PIN N Y N N N logic layer is usually under
the user interface layer, so
Keep Card N N Y N N at this point the basic
Reenter N N N Y N sanity checking of the
Request inputs should have been
done.
Dispense Cash N N N N Y

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Black Box – State Transition
• To view the system in terms of states, transition
between states, the inputs/events that triggers
state changes(transition) and the action which
result from those transitions
• Much used within embedded software industry
and technical automation in general.
• The technique may also suitable for modeling a
business object having specific states or testing
screen dialogue flows.

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Black Box – State Transition
• Advantage:-
• Reveal undefined situation
• Terms:-
• State – situations that persists, and will persists until
some event occurs
• Event – is an occurrence, something that concludes
almost instantly or in some fixed constant period
• Action – is an occurrence; the result

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Black Box - State Transition Diagram

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Black Box- State Transition Tables

Current State Event[Condition] Action New State

Logging In Password[Invalid] Error Logging In
Logging In Password[Valid] Open register Waiting
Logging In Customer [Undefined] [Undefined]
Logging In Scan[Any] [Undefined] [Undefined]
Logging In Pay[Any] [Undefined] [Undefined]
Logging In Shift end [Undefined] [Undefined]

A state-transition table can represent complex state-transitions that

won’t fit on the graph. (However, complexity might indicate bad
design). It can also reveal undefined situations as does this portion
of the table of the graph on the preceding page.

** Each row represents 1 test case

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi
 Example : Path Analysis

EXAMPLE : A person who is married and has worked for 5 years will follow the
path B1/A2/B2/A3, resulting in a salary of 7000
Specifications Explanation

IF worked = Y AND number of  Decision B1, consisting of

years > 10 conditions C1 and C2
THEN salary := 10,000  Action A1
ELSE salary := 5,000  Action A2
 Decision B2, consisting of
IF married = Y condition C3
THEN salary := salary + 2,000  Action A3

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Example : Path Analysis

Start
1
B1
2 3
A2 A1

4
B2 A3
6 5
End
Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi
 Example : Path Analysis

• Steps to determine test paths

– Coded uniquely the individual decision points, conditions and
actions (B1,B2 for decision points, and A1,A2 for actions)
– Translate the algorithm into flow chart to show the test path
– Create action combinations for each decision points [B1:(1,2);
(1,3)],[B2: (2,4,5);(2,6);(3,4,5);(3,6)]
– Put the action combination in ascending order :
• (1,2) ; (1,3) ; (2,4,5) ; (3,4,5) ; (3,6)
– Create paths by linked action combinations from the start to the
end of the algorithm
• Path 1 : ( 1,2,4,5 )
• Path 2 : ( 1,3,4,5 )
• Path 3 : ( 1,2,6 )
• Path 4 : ( 1,3,6 )

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

Test Action Expected Actual
No. Test Case Result Result (Y/N)
C1 C2 C3
1 Path 1 : (1,2,4,5) Y 11 Y A1 & A3
2 Path 2 : (1,3,4,5) Y 10 Y A2 & A3
3 N 11 Y A2 & A3
4 N 10 Y A2 & A3
5 Path 3 : (1,2,6)
6 Path 4 : (1,3,6)

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Experienced Based
• Common elements include:
• The knowledge and experience used to derive test
cases
• Can consider knowledge and experience of the
software, its usage and its environment or..
• Knowledge about historical and likely defects and
their distribution
• Example include:
• Attacks
• Checklists
• Exploratory

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Experience Based Techniques

• Key concepts
• Reasons for writing test cases based on intuition,
experience, and knowledge
• Comparing experienced based techniques with
specification based techniques

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Experience Based Tests
• Experienced based tests are based on the tester’s ..
• Skill and intuition
• Experience with similar applications
• Experience with similar technologies
• Rather than being pre-designed, experience based
tests are often created during test execution
• Tests are frequent “time-boxed” (i.e brief periods of
testing focused on specific test conditions)
• Example includes error guessing, bug hunting,
“breaking software”, based on checklists or bugs
taxonomies and exploratory testing

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Common Experience Based
Approaches
• Most professional approaches to experience based
testing do not create tests entirely during execution
• May be guided by:
• Checklist - descriptions of area, characteristics and features to test
• Bug taxonomy – hierarchy of bugs
• List of attacks – list of bugs that we seek
• Bug hunt approach – catalog of common problem
• Set of test charters – description of test to be done, similar to
checklists

• These guidelines are prepared in advance to some level of

detail
• Purely on-the-fly testing (ad hoc testing) is common, but
usually (ineffective) manual random testing
Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi
 Experience Based Advantages
and Disadvantages
Disadvantages
Advantages
• Gappy coverage
• Effective at finding bugs especially under pressure
• Resists pesticide paradox • Difficult to estimate
due to high variance
• No bug prevention
• Efficient (light weight
record keeping) • Extensive debriefing and
discussion doesn’t scale
• Good check on prepared to large teams
tests
• Not all testers have the
• Fun and creative necessary level of skill,
experience
Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi
Factors in Choosing Techniques
• Type of system
• Regulatory standards
• Customer or contractual requirements
• Level and type of risk
• Test objectives
• Documentation available
• Knowledge of testers
• Time and budget
• Development life cycle
• Previous experiences on type of defects found
Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi
 The Dynamic/Pre-designed
Spectrum
No documentation; often Used for highly regulated
associated with amateur test industries; follows specific
efforts. templates
Pros: Cheap and quick Pros: Traceable and auditable
Cons : No coverage Cons: High initial and on going
measurement costs; not always followed

Many test teams balance

precision and detail here,
“Pure “Charted IEEE 829 Style “Pure
getting adequate
Exploratory” Exploratory” Scripted”
reproducibility and
traceability within realistic
budgets

Lightweight documentation; a Standard documentation; a

sophisticated technique widely described technique
Pros: Inexpensive but Pros: Reproducible and
measureable traceable
Cons: Requires skilled testers Cons: Significant resources to
and managers develop and maintain

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi

 Test Case Detail and Precision
• Trade offs on the test documentation spectrum
• Precise tests allow less skilled testers, but not very
flexible
• Imprecise tests can cover more conditions, but not
very reproducible, especially across multiple testers
• Precise tests provide transparent test criteria, but are
hard and expensive to maintain
• Imprecise tests are quick to write, but coverage can
be hard to define and measure
• The degree to which a test effort is dynamic can be
measured by counting the number of words of
documentation (in your test cases and test procedures)
per test hour of test execution
Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi
 Thank You

Semester 2, 2009/2010 Fakulti Teknologi Maklumat dan Komunikasi