PRODUC

T
Test
Definition Purpose
metric
The total number of remarks
found in a given time
period/phase/test type. A
remark is a claim made by One of the earliest indicators to measure
Number of test engineer that the once the testing commences; provides
remarks application shows an initial indications about the stability of the
undesired behavior. It may or software.
may not result in software
modification or changes to
documentation.

The total number of remarks A more meaningful way of assessing the

found in a given time stability and reliability of the software
Number of
period/phase/test type that than number of remarks. Duplicate
defects
resulted in software or remarks have been eliminated; rejected
documentation modifications. remarks have been done.

The status of the defect could

vary depending upon the
defect-tracking tool that is
used. Broadly, the following
statuses are available: To be Track the progress with respect to
solved: Logged by the test entering, solving and retesting the
Remark engineers and waiting to be remarks. During this phase, the
status taken over by the software information is useful to know the number
engineer. To be retested: of remarks logged, solved, waiting to be
Solved by the developer, and resolved and retested.
waiting to be retested by the
test engineer. Closed: The
issue was retested by the test
engineer and was approved.

Provides indications about the quality of

The severity level of a defect
the product under test. High-severity
indicates the potential
defects means low product quality, and
Defect business impact for the end
vice versa. At the end of this phase, this
severity user (business impact =
information is useful to make the release
effect on the end user x
decision based on the number of defects
frequency of occurrence).
and their severity levels.

Defect An index representing the Provides a direct measurement of the

severity average of the severity of the quality of the product—specifically,
index defects. reliability, fault tolerance and stability.

Shows how fast the defects are being

Time to
The effort required to find a found. This metric indicates the
find a
defect. correlation between the test effort and
defect
the number of defects found.
 Provides an indication of the
Time to Effort required to resolve a
maintainability of the product and can be
solve a defect (diagnosis and
used to estimate projected maintenance
defect correction).
costs.
This metric is an indication of the
Defined as the extent to
completeness of the testing. It does not
Test which testing covers the
indicate anything about the effectiveness
coverage product’s complete
of the testing. This can be used as a
functionality.
criterion to stop testing.
Test case The extent to which test This metric provides an indication of the
effectiven cases are able to find effectiveness of the test cases and the
ess defects. stability of the software.
This metric indicates the quality of the
product under test. It can be used as a
Defects/ The number of defects per
basis for estimating defects to be
KLOC 1,000 lines of code.
addressed in the next phase or the next
version.
PROJECT

Ratio of the planned This metric helps in detecting issues

Workload
workload and the gross related to estimation and planning. It
capacity
capacity for the total test serves as an input for estimating similar
ratio
project or phase. projects as well.

Test
planning The planned value related to
Shows how well estimation was done.
performan the actual value.
ce

Test effort is the amount of

The effort spent in testing, in relation to
work spent, in hours or days
the effort spent in the development
Test effort or weeks. Overall project
activities, will give us an indication of the
percentag effort is divided among
level of investment in testing. This
e multiple phases of the
information can also be used to estimate
project: requirements, design,
similar projects in the future.
coding, testing and such.

An attribute of the defect in

relation to the quality
attributes of the product.
Defect Quality attributes of a product This metric can provide insight into the
category include functionality, usability, different quality attributes of the product.
documentation, performance,
installation and
internationalization.
PROCES
S
Are we able to find the right defects in
Should be An attribute of the defect ,
the right phase as described in the test
found in indicating in which phase the
strategy? Indicates the percentage of
which remark should have been
defects that are getting migrated into
phase found.
subsequent test phases.
 An estimate of the number of The goal is to achieve a defect level that
Residual
defects that may have been is acceptable to the clients. We remove
defect
unresolved in the product defects in each of the test phases so that
density
phase. few will remain.

Provides an indication of the level of

Ratio of the number of
Defect understanding between the test
remarks that resulted in
remark engineers and the software engineers
software modification vs. the
ratio about the product, as well as an indirect
total number of remarks.
indication of test effectiveness.
Percentage of valid remarks
during a certain period. Valid
Valid remarks = number of defects
Indicates the efficiency of the test
remark + duplicate remarks +
process.
ratio number of remarks that will
be resolved in the next phase
or release.
Percentage of the number of
Indicates the effectiveness of the defect-
resolved remarks that
Bad fix resolution process, plus indirect
resulted in creating new
ratio indications as to the maintainability of the
defects while resolving
software.
existing ones.
Indicates the efficiency of defect removal
Defect The number of defects that
methods, as well as indirect
removal are removed per time unit
measurement of the quality of the
efficiency (hours/days/weeks)
product.

Defined as the number of

Shows the effectiveness of the defect
defects found during the
removal. Provides a direct measurement
Phase phase of the development life
of product quality; can be used to
yield cycle vs. the estimated
determine the estimated number of
number of defects at the start
defects for the next phase.
of the phase.
Backlog The number of remarks that Indicates how well the software
developm are yet to be resolved by the engineers are coping with the testing
ent development team. efforts.
The number of resolved
Backlog remarks that are yet to be Indicates how well the test engineers are
testing retested by the development coping with the development efforts.
team.
Scope The number of changes that Indicates requirements stability or
changes were made to the test scope. volatility, as well as process stability.
 How to calculate

Total number of remarks found.

Only remarks that resulted in modifying the

software or the documentation are counted.

This information can normally be obtained

directly from the defect tracking system
based on the remark status.

Every defect has severity levels attached to

it. Broadly, these are Critical, Serious,
Medium and Low.

Two measures are required to compute the

defect severity index. A number is assigned
against each severity level: 4 (Critical), 3
(Serious), 2 (Medium), 1 (Low). Multiply
each remark by its severity level number
and add the totals; divide this by the total
number of defects to determine the defect
severity index.
Divide the cumulative hours spent on test
execution and logging defects by the
number of defects entered during the same
period.
 Divide the number of hours spent on
diagnosis and correction by the number of
defects resolved during the same period.

Coverage could be with respect to

requirements, functional topic list, business
flows, use cases, etc. It can be calculated
based on the number of items that were
covered vs. the total number of items.
Ratio of the number of test cases that
resulted in logging remarks vs. the total
number of test cases.

Ratio of the number of defects found vs.

the total number of lines of code
(thousands)

Computation of this metric often happens in

the beginning of the phase or project.
Workload is determined by multiplying the
number of tasks against their norm times.
Gross capacity is nothing but planned
working time, determined by workload
divided by gross capacity.

The ratio of the actual effort spent to the

planned effort.

This metric can be computed by dividing

the overall test effort by the total project
effort.

This metric can be computed by dividing

the defects that belong to a particular
category by the total number of defects.

Computation of this metric is done by

calculating the number of defects that
should have been found in previous test
phases.
 This is a tricky issue. Released products
have a basis for estimation. For new
versions, industry standards, coupled with
project specifics, form the basis for
estimation.
The number of remarks that resulted in
software modification vs. the total number
of logged remarks. Valid for each test type,
during and at the end of test phases.

Ratio of the total number of remarks that

are valid to the total number of remarks
found.

Ratio of the total number of bad fixes to the

total number of resolved defects. This can
be calculated per test type, test phase or
time period.
Computed by dividing the effort required for
defect detection, defect resolution time and
retesting time by the number of remarks.
This is calculated per test type, during and
across test phases.

Ratio of the number of defects found by the

total number of estimated defects. This can
be used during a phase and also at the end
of the phase.

The number of remarks that remain to be

resolved.

The number of remarks that have been

resolved.

Ratio of the number of changed items in

the test scope to the total number of items.