Overview of

Testin
g

Chapter 1

Objectives
End Goals

to identify the basic mindset of a tester, regardless of what

is being tested
to determine the correct motivations for testing in business
to explain some of the reasons why testing is undervalued
as a business practice
to explain what differentiates software testers from
software developers

Coverage
Overview of Topics

Objectives and Limits of Testing

The Value versus Cost of Testing
Relationship of Testing to the Software Development Cycle
Tester versus Developer Roles in Software Testing

Introduction
The Software Crisis

16%
53%

31%

Successful
Cancelled
Challenged
(Delayed/Exceed
Budget)

*Software Engineering, Roger Pressman, 5th

edition, 2001

Introduction
The Software Crisis
60%
50%

53%
46%

40%

53%
44%

40%

30% 31%
20%

49%

51%

34%

33%
27%

28%
26%

29%

28%
23%

16%

32%
24%

15%

Succeeded
Failed
Challenged

18%

10%
0%
1994

1996

1998

2000

2002

2004

2009
*Standish Chaos Report listed
by year

One solution
A Comprehensive Testing Approach

22.2B can be
saved by applying
a comprehensive
testing approach

$$
$

Estimated 59.5B annual loss

in US alone
*Software Testing: Testing Across the Entire
Software Development Life Cycle,
Everett&Mcleod, 2007

Objectives and Limits of Testing

The Mind of a Tester

Four kinds of thinking

Practical

Technical

Creative

Critical

ACTIVITY
The Kings Challenge

Objective: To check if people think alike.

Required: Three volunteers to come up to the board to write
their solution
Once upon a time, a mighty king wanted to determine which
of his three court wizards was the most powerful. So he put
the three court wizards in the castle dungeon and declared
whoever escaped from his respective dungeon cell first was
the most powerful wizard in all the kingdom.

User Level Testing

Buying an Automobile and Testing It

User Level Testing

Determining the Testing Objectives

TEST
DRIVE

Validate affordability
Validate attractiveness
Validate usefulness
Break the car?
Improve the cars
design?

Validate comfort
Validate performance

User Level Testing

Other Testing Approaches when Buying a Car

examining the sticker price and sale contract

trying out the radio, the air conditioner, and the lights
trying acceleration, stopping, and cornering

User Level Testing

Other Testing Approaches when Buying a Car

examining the sticker price and sale contract

trying out the radio, the air conditioner, and the lights
trying acceleration, stopping, and cornering

TRY Performance
Testing
Work different
features of the car by
actually driving the car

EXAMINE Static
Testing
Observe, read, review
without actually
driving the car

TRY OUT Functional and

Structural Testing
Work different features of
the car without actually
driving the car

Developer Level Testing

Building an automobile and Testing It

Developer Level Testing

Determining the Testing Objectives

Validate design via

scale models
Validate operation of
prototypes
Validate mass assembly
plans from prototypes

BASIS = Requirements
Document

Developer Level Testing

Other Testing Approaches when Building a Car

plan the tests based on requirements and design

specifications.
examine blueprints and clay models.
perform and analyze wind tunnel tests.
perform and analyze safety tests.
perform and validate prototype features.
drive prototype and validate operations.

Developer Level Testing

Other Testing Approaches when Building a Car

plan the tests based on requirements and design

Test Plan
specifications.
and Test
examine blueprints and clay models.
Cases
perform and analyze wind tunnel tests.
Static Testing
perform and analyze safety tests.
Observe, read, or
perform and validate prototype features.
review without
drive prototype and validate operations.actually building
the car

Performance Testing
work different
features of the car in
the prototypes

Functional and
Structural Testing
work different features
of the car models, mockups, and manufactured
subassemblies

The Objectives of Testing

The Common Goals

Testing can be applied to a wide range of development

projects in a large number of industries
The primary motivation for testing all business
development projects is to reduce the risk of unplanned
development expense, or worse, the risk of project failure.
A Risk Assessment is performed to know the size and
probability of occurrence of development risks that could
cause tangible loss to the business.

The Objectives of Testing

The Common Goals

The Risk motivation is divided into four interrelated testing

objectives
Identify the magnitude and sources of development risk
reducible by testing
Perform testing to reduce identified risks
Know when testing is completed
Manage testing as a standard project within the development
project

The Objectives of Testing

The Common Goals

What if you cant test it all? What if you run out of time?
Prioritize the outstanding defects, and prioritize features that
are most important
Review test results and determine clusters or trends to allocate
resources better

Development Axiom
Testing is not abrakadabra for quality

Quality Must Be Built In Because Quality Cannot Be Tested

In

The Value versus Cost of Testing

Return on Investment
Benefit from the Investment

Like any business project, testing requires the same ROI

decision
Testing should be done only when the test results can show
benefit beyond the cost of performing the tests.

Estimating the Cost of Failure

The Ford Pinto Story

Estimating the Cost of Failure

The Different Kinds of Business Losses

Revenue or profit
Cost associated with testing, testing basically reduces the total
profit of the final product

Testing resources (skills, tools, and equipment)

Testers need good training, good tools, and good testing
environments

Customers
Loss of customers due to poor quality

Litigation
An unhappy customer can do a company as much financial
damage as an injured customer

Basili and Boehms Rule

Exponentially Increasing Costs to Correct New Software

The Myth
Quality can be tested into a software product at the end of the
development cycle
NOTE: Quality in this context means software that exhibits zero
defects when used by the customer

The Ignored Truths

Testing should be started as early as possible to have the
greatest impact on the quality of the product
You cannot test in quality period!

Basili and Boehms Rule

Exponentially Increasing Costs to Correct New Software

Basili and Boehms Rule

Exponentially Increasing Costs to Correct New Software

The dot.bomb crash

2000 2002 business gold rush for internet presence
Software development teams tried to find ways to cut corners
and save time to market and viewed testing as a speed bump
Result? The obvious. It wouldnt be called a crash for
nothing.

Lessons
You cant take too many shortcuts when developing software
You will pay for testing now or later, but the cost of testing is
unavoidable. Testing now is always less expensive than testing
later.

Relationship of Testing to the SDLC

The Evolution of Software Testing

A Technology Profession

1950 1960

reactive
debugging

many
corrections and
refinements to
software

1960 - 1970

Captain Grace
Murray Hopper
found first
bug

1970 - 1980

captured best
software
development
practices,
repeatable level of
software reliability

The Y2K
Sword of
Damocles

1990 ->

The Ten Principles of Good Software Testing

Thank you Y2K

Y2K testing did not start in a vacuum. Computer

professionals already realized the need to develop a full
repertoire of software testing techniques by the mid 1980s.
The Y2K frenzy directed a spotlight on larger issues,
processes, and strategies for full development life cycle
software testing
These principles are an amalgam of the professional testing
experience from the 1980s and 1990s and the Y2K
experience to yield the following underlying software
testing principles.

The Ten Principles of Good Software Testing

Principle 1

Business risk can be reduced by finding defects

What will it cost if the software does not work as it is
advertised?
A strong connection should be made early in the process
between looking for defects and avoiding risk.

The Ten Principles of Good Software Testing

Principle 2

Positive and negative testing contribute to risk reduction.

Positive testing is simply the verification that the new software
works as advertised.
Negative testing is simply the verification that customers can
not break the software under normal business situations.

The Ten Principles of Good Software Testing

Principle 3

Static and execution testing contribute to risk reduction.

There are a large number of documents produced during
software development that, if reviewed for defects (static
testing), could significantly reduce the number of execution
defects before the code is written.
The best programmers in the organization cannot overcome
bad requirements or bad specifications by writing good code.

The Ten Principles of Good Software Testing

Principle 4

Automated test tools can contribute to risk reduction.

Manual testing has severe limitations and the non-repeatability
of test results because no one performs a manual test exactly
the same way twice.
The last 5 years of automated performance test tool maturity
has prompted the strong consideration of testing tools to
replace other kinds of manual testing when conditions are
favorable.

The Ten Principles of Good Software Testing

Principle 5

Make the highest risks the first testing priority.

It is important to ensure that there are sufficient testing
resources to address at least the top business risks

The Ten Principles of Good Software Testing

Principle 6

Make the most frequent business activities (the 80/20 rule)

the second testing priority.
It is common industry knowledge that 80% of any daily
business activity is provided by 20% of the business system
functions, transactions, or workflow. This is known as the 80/20
rule.
Concentrate the testing on the 20% that really drives the
business. The other 80% of the business system typically
represents the exception transactions that are invoked only
when the most active 20% cannot solve a problem.

The Ten Principles of Good Software Testing

Principle 7

Statistical analyses of defect arrival patterns and other

defect characteristics are a very effective way to forecast
testing completion.
The intelligent use of these statistical models enables the
tester to predict within 10%20% the total number of defects
that should be discovered in a software implementation.

The Ten Principles of Good Software Testing

Principle 8

Test the system the way customers will use it.

Let me tell you a story.

The Ten Principles of Good Software Testing

Principle 9

Assume the defects are the result of process and not

personality.
This principle presents an organizational behaviour challenge
for the tester.
The tester must find a way to focus on the software defect
without seeking to place blame.

The Ten Principles of Good Software Testing

Principle 10

Testing for defects is an investment as well as a cost.

Most executives, directors, and managers tend to view testing
only as an expense, and to ask questions such as How many
people? How many weeks delay? How much equipment? and
How many tools?
Although these cost factors represent a legitimate part of the
overall business picture, so do the tangible benefits that can
offset the testing costs, business risk reduction
notwithstanding.

The Game of Gossip

The Source of Software Defects

Tester versus Developer Roles in

Software Testing

The Role of Testing Professionals

Technical Skills

The best software testers must have advanced skills drawn

from many software professions including software
developers, database developers, network developers, and
system administrators.
The testers role
The Verifier
provides an objective look independent of the document authors
and code developers

The Manager and Planner

Identify critical areas of software to be tested and the most
efficient ways to complete that testing
Plan, Schedule, Manage

The Role of Test Tool Experts

The Use of Automated Test Tools

Tools enable software testers to do testing more effectively

than by using any manual procedure
The impetus behind test tool experts expanding their tool
expertise is the software testing communitys recognition that
no single test tool can support all the different kinds of tests
that are necessary across the entire development life cycle.

Who Is on the Test Team?

A mixture of skill levels

Skill classifications
Entry level skills
Intermediate level skills
Advanced skills

Who Is on the Test Team?

A mixture of skill levels

Skill classifications
Entry level skills
Intermediate level skills
Advanced skills

With the advice and

mentoring of the
senior testers, a mix
of intermediate-level
and entry-level
testers executes the
tests.

Responsible for the test

planning, scheduling,
and analysis of test
results.

Work within the test

plan to create the test
scenarios, cases, and
scripts that follow the
plan

 Testin
Overview of
END

Module Updates
Document Version Details

Versio
n

Date Updated

1.0

7/30/2011

Changes/Details
Created document

Author
Engr. K. Fajardo

This document must not be used for other purposes outside the USC
Department of Computer Engineering without permission.