Qu n l ch t l ng ph n m m

Software Quality Management GV: Nguy n Ng c T Email: nntu@hoasen.edu.vn

Are we building the right software for the need ? Are we building the software right ?

Chapter 09

DEFECT CLASSIFICATION AND ANALYSIS

SQS SQM SQA SRE
NNTu - SQM W2009

The software is done. We are just trying to get it to work Unknown

Hoa Sen University HoaSen University 2

AIMS
(actual/potential) defect or quality current and future products. in

NNTu - SQM W2009

HoaSen University

Contents
General Types of Defect Analyses
Defect distribution analysis
What: Distribution over defect types Where: Distribution over defect locations General observations about defect distribution

Defect trend analysis and defect dynamics model Defect causal analysis

DEFECT CLASSIFICATION AND ODC

ODC concepts Defect classification using ODC: A comprehensive example Adapting ODC to analyze web errors

DEFECT ANALYSIS FOR CLASSIFIED DATA

One-way analysis: Analyzing a single defect attribute Two-way and multi-way analysis: Examining cross-interactions

NNTu - SQM W2009

HoaSen University

Introduction
Defect CA
help both developers and testers to detect and remove potential defects help other project personnel to improve the development process, to prevent injection of similar defects and to manage risk better

defect data
from the main QA activities

NNTu - SQM W2009

HoaSen University

General Types of Defect Analyses

A defect is discovered,
various individual analyses can be performed.

defect data are accumulated over time

collective analyses can be performed

General defect analyses:

Questions: what/where/when/how/why? Distribution/trend/causal analyses.

NNTu - SQM W2009

HoaSen University

General Types of Defect Analyses [2]

Analyses of classified defect data:
Prior: defect classification. Use of historical baselines. Attribute focusing in 1-way and 2-way analyses. Tree-based defect analysis

NNTu - SQM W2009

HoaSen University

General Types of Defect Analyses [3]

What
The identification and classification of the discovered defects can be performed to identify what they are and classify them by some consistent scheme

Where
Where was the defect found or discovered? This information can be used to provide valuable feedback to the development process through defect distribution analysis.

When
The identification of the exact time or associated development phase or subphase when a defect is injected and when it is discovered is important, because it provides information to analyze the overall defect trend and serves as the basis for quality prediction into the future.

NNTu - SQM W2009

HoaSen University

General Types of Defect Analyses [4]

Pre- or Post release
An important extension to the when question is whether a defect is a pre-release defect or a post-release defect

How and Why

How was the defect injected into the software, and why? These two questions are closely related, both pertaining to the cause of the discovered defects

NNTu - SQM W2009

HoaSen University

Defect distribution analysis

can help us answer the what and where questions
distribution of defects over different defect types can find out the distribution of defects over different areas or product components

typically deal with faults or defect fixes instead of failures or errors Defect fixes
is typically used if actual fixing of discovered problem took place before defect analyses were performed

NNTu - SQM W2009

HoaSen University

10

Defect distribution analysis

Type A B C D E F G H I J K All types Description permission denied no such file or directory stale NFS file handle client denied by server configuration file does not exist invalid method in request invalid URL in request connection mod_mime_magic request failed script not found or unable to start connection reset by peer #failures 2079 14 4 2 28,631 0 1 1 1 27 0 30,760

http://lyle.smu.edu/~tian/class/8317.09f/webM.pdf

NNTu - SQM W2009

HoaSen University

11

What: Distribution over defect types

Type .gif .class directory .html .jpg other All Errors 12489 4913 4425 3656 1323 394 28631 % 43.62 17.16 15.46 12.77 4.62 1.38 100

Table 20.2 Characterizing web errors by file types

NNTu - SQM W2009

HoaSen University

12

Where: Distribution over defect locations

DF= 0 1 2 3 4 5 6 7 8 9 10-19 2037 7 50 14 3.9 1.1 673 417 28.4 17. 6 all 1295 100 2367 100

module 771 174 102 63 31 29 23 25 16 # % 58.8 13.4 7.9 4.9 2.4 2.2 1.8 1.9 1.2 0.5 DFsum 0 174 204 189 124 145 138 175 128 63 % 0 7.4 8.6 8 5.2 6.1 5.8 7.4 5 2.7

Table 20.3 Distribution of DF for a commercial product LS NNTu - SQM W2009 HoaSen University 13

General observations about defect distribution

DF= modul e# % DFsum % 0 23 2.3 0 1.6 7 1 13 1 13. 2 13 1 2.8 6 2 3 4 5 6 7 8 9 10-19 20-49 >50 11 12 99 94 68 50 38 32 147 68 13 2 0 11 12 9.9 9.4 6.8 5 3.8 3.2 14.8 6.8 1.3 all 995 100

22 36 39 47 40 35 30 28 2109 2040 910 7824 4 0 6 0 8 0 4 8 4.6 5.1 6 5.2 4.5 3.9 3.7 3.1 26.96 26.07 11.6 100 3

NNTu - SQM W2009

HoaSen University

14

Defect trend analysis and defect dynamics model

defect data contains some timing information
Minimum: pre-release or post-release
Injection phase requirement specification design coding testing post-release all phases
NNTu - SQM W2009

req. 10

spec. 22 10

Removal Phase design coding testing 8 20 52 0 2 120 198 5 0 32 320 58 415

post-rel 2 1 5 46 7 2 63

10

32

80
HoaSen University

320

all phases 47 33 209 564 65 2 920

15

Defect causal analysis

two forms
logical analysis
is a deterministic analysis that examines the logical link between the effects and the corresponding causes, and establishes general causal relations

statistical analysis
is a probabilistic analysis that examines the statistical link between causes and effects and deduces the probable causal relations between the two

NNTu - SQM W2009

HoaSen University

16

Defect causal analysis [2]

The effects
the observed failures or discovered (or fixed) faults

the corresponding causes

are the faults that caused the failures or the errors that caused the injection of the faults, respectively.

causal relations
are determined by the developers or code owners faults are typically determined through dedicated defect causal analysis

NNTu - SQM W2009

HoaSen University

17

Defect causal analysis [3]

Root cause analysis
is human intensive, and should be performed by experts with thorough knowledge about the product, the development process, the application domain, and the general environment

Gilb inspection for all the critical defects

NNTu - SQM W2009

HoaSen University

18

Defect causal analysis [4]

Statistical analysis
is based on empirical evidence collected either locally or from other similar projects can be fed to various models to establish the predictive relations between causes and effects

employ various statistical models

correlation analysis
Eg. the number of defects per module may be closely correlated to module control flow complexity
NNTu - SQM W2009 HoaSen University 19

DEFECT CLASSIFICATION AND ODC

various detailed information can be collected and recorded regarding the problems or the defects part of this information
is usually derived from explicit or implicit root cause analysis

Such information
can be organized in a systematic way for further analyses
more valuable and specific feedback: use statistical models
NNTu - SQM W2009 HoaSen University 20

DEFECT CLASSIFICATION AND ODC [2]

The systematic classification and analysis of defect data bridge the gap between causal analysis and statistical quality control, and provide valuable in-process feedback to the development or maintenance process and help assure and improve product quality

NNTu - SQM W2009

HoaSen University

21

DEFECT CLASSIFICATION AND ODC [3]

Orthogonal defect classification, or ODC, developed initially at IBM (Chillarege et al., 1992)
is the most influential among general frameworks for software defect classification and analysis.

to identify problematic areas, and to improve overall software product quality

NNTu - SQM W2009

HoaSen University

22

ODC
Key elements of ODC
Aim: tracking/analysis/improve Approach: classification and analysis Key attributes of defects Views: both failure and fault Applicability: inspection and testing Analysis: attribute focusing Need for historical data

NNTu - SQM W2009

HoaSen University

23

ODC concepts
ODC has a rich and extensive category of defect attributes, stemming from both the failure view and the fault view The attributes (8)
related to the former are typically completed by software testers or inspectors who initially observed problems and opened defect reports; while those related to the latter are typically completed by the software developers or system maintenance personnel who fixed the reported problems and updated the corresponding defect reports
NNTu - SQM W2009 HoaSen University 24

ODC
8 attributes
Activity
refers to the actual process step (code inspection, function test, etc.) when defects are discovered.

Trigger
describes the environment or condition that had to exist to expose the defects.

Impact
refers to either perceived or real impact on users.

Target
represents the high-level identity (i.e., design, code, ID, etc.) of the entity that was fixed.
ODC v5.11, IBM Center for Software Engineering NNTu - SQM W2009 http://www.ibm.com/developerworks/rational/library/aug06/gu/index.html HoaSen University 25

ODC
8 attributes
Type
represents the nature of the actual correction that was made.

Qualifier
specifies whether the fix that was made was due to missing, incorrect, or extraneous code or information.

Source
indicates whether the defect was found in code written inhouse, reused from a library, ported from one platform to another, or outsourced to a vendor.

Age
identifies the history of the target (i.e., design, code, ID, etc.) that had the defect. http://www.ibm.com/developerworks/rational/library/aug06/gu/index.html
NNTu - SQM W2009 HoaSen University 26

ODC concepts
failure view and information collected at defect discovery
Defect impact,
with attribute values covering functionality, reliability, etc.

Defect trigger,
with attribute values corresponding to the specific types of testing

Defect severity,
with commonly used attribute values: critical, major, minor, or or inspection activities or scenarios that triggered the defect detection. some numerical scale.

NNTu - SQM W2009

HoaSen University

27

ODC concepts
fault view collected at defect fixing
Defect type,
with attribute values: function, interface, algorithm, timing, etc.

Number of lines changed for the fixing.

Some additional causal analyses

Defect source,
with attribute values: vendor code, new code, base code, etc.

Where the defect was injected,

located to subsystems, modules, or components.

When the defect was injected,

typically identified with the development phase.
NNTu - SQM W2009 HoaSen University 28

Defect classification using ODC: ODC: A comprehensive example

various defect attribute data according to ODC were collected in the system testing stage a defect is detected,
a formal report (called Problem Tracking Report or PTR in IBM) is recorded and tracked until its final resolution

NNTu - SQM W2009

HoaSen University

29

Defect classification using ODC: ODC: A comprehensive example [2]

Label imp Name impact Possible Values or Categories & Labels c=capability, im=implementation, in=installation, ma=maintenance, mi=migration, p=performance, r=reliability, sec=security, ser=service, std=standard, u=usability i=installation, m=migration, s=stress, b=backup, c=communications, f=file ilo, co=coexistence, e=exception, hc=hlw config., sc=slw config., a=ad-hoc, ss=startup/shutdown, o=normal operation range from 1 (highest) to 4 (lowest) in severity week detected, counted from the start of the project o=other product, s=specification, hld=high-level design, lld=low-level design, c=code, b=build process a=add, d=delete, c=change b=base, v=vendor, n=new, c=changed, i=incremental (added to old), s=scaffolded, p=previous defect fix p=previous release, s=specification, hld=high-level design, Ild=low-level design, c=coding, ut=unit test, ft=function test, st=system test, d=customer usage
HoaSen University 30

trig sev wk ftype act src

trigger severity week fix type action code source phase injected

inj

NNTu - SQM W2009

Defect classification using ODC: ODC: A comprehensive example [3]

Defect impact
If this defect is not fixed, how will it impact the customer? Pre-defined impact categories (possible answers) include performance, reliability, etc.

Defect trigger categories

closely resemble test scenario classes used for managing the testing process for this product.

Defect severity
can be 1 (critical problem), 2 (major problem), 3 (minor problem), and 4 (minor inconvenience).

The week
when the defect is detected, counted from the start of the project.
NNTu - SQM W2009 HoaSen University 31

Defect classification using ODC: ODC: A comprehensive example [4]

The information collected at defect fixing
pertains to the actions taken by the developers to locate, identify and correct the faults that caused detected failures:
Fix type: fix to design, code, etc. Number of lines changed for the fixing. Fix action: adding, deleting, or changing to design or code.

Some simple causal analyses

Defect source: vendor code, new code, base code, etc. The development phase when the defect was injected: previous release or waterfall like development phases in the current release.
NNTu - SQM W2009 HoaSen University 32

Adapting ODC to analyze web errors

For web-based applications,
ODC-like defect classification can be defined and relevant defect data can be extracted from existing web server logs for analysis (Ma and Tian, 2003)

data collection is always a big hurdle that requires developers and testers to devote substantial time to analyze the defects and report the findings
NNTu - SQM W2009 HoaSen University 33

Adapting ODC to analyze web errors [2]

Attributes
Defect impact
corresponds to web error type, which indicates what problem was experienced by web users. It can be analyzed directly based on information extracted from the error logs or from response code used in web access logs (Kallepalli and Tian, 2001).

Defect trigger
corresponds to specific usage sequences or referrals that lead to problems recorded in the error logs. It can be analyzed by examining the referral pair information that can be extracted from the access logs (Ma and Tian, 2003).

Defect source
corresponds to specific files or file types that need to be changed, added, or removed to fix problems recorded in the error logs. It can be analyzed by examining both the specific errors and referral pairs.
NNTu - SQM W2009 HoaSen University 34

Adapting ODC to analyze web errors [3]

Various other attributes
can also be adopted or adapted from the original ODC attributes through a close examination of the web environment and data availability. Such adaptation to different environments can help people analyze problems or issues of concern to them and fulfill different purposes.

NNTu - SQM W2009

HoaSen University

35

DEFECT ANALYSIS FOR CLASSIFIED DATA

Various techniques most obvious and most straightforward analyses
are to apply defect distribution and trend analyses

one-way analysis
examines one attribute at a time, either its overall distribution or its trend over time

Two-way analysis
can be used to examine the crossinteraction of two attributes (Bhandari et al., 1993).

Higher-order analysis is also possible,

such as using tree-based modeling on all the ODC attributes (Tian and Henshaw, 1994). expected defect profile
NNTu - SQM W2009 HoaSen University 36

OneOne-way analysis: Analyzing a single defect attribute

For each defect attribute,
the overall distribution of its values can be examined. distribution of defects among the different defect impact areas is very non-homogeneous

When similar distribution data are available over time or different development phases,
can trace them to perform defect trend analysis

NNTu - SQM W2009

HoaSen University

37

OneOne-way analysis: Analyzing a single defect attribute

NNTu - SQM W2009

HoaSen University

38

OneOne-way analysis: Analyzing a single defect attribute

Figure 20.2 gives us the trend of type E errors for the SMU/SEAS web site over 26 days
NNTu - SQM W2009 HoaSen University 39

TwoTwo-way and multi-way analysis: multiExamining cross-interactions crossanalysis examines the interaction between two attributes, and can be applied to all the attributes in pair-wise fashion simplest form
is the conditional analysis of an individual attribute under the condition of another attribute taking a specific value.

NNTu - SQM W2009

HoaSen University

40

TwoTwo-way and multi-way analysis: multiExamining cross-interactions [2] crossImpact Capability Documentation Installability Maintainability Migration Performance Reliability Security Service Standards Usability 1 2 0 0 0 0 1 27 1 0 0 0 Severity 2 12 1 6 6 0 1 96 3 0 1 10 3 13 14 6 19 0 3 66 3 4 2 44 4 1 10 4 7 1 0 7 0 4 1 19

NNTu - SQM W2009

HoaSen University

41

Reference
[1]Chapter 20 [1]Chapter 13 [3]Chapter 5 [4]Chapter 04 ODC - http://www.research.ibm.com/softeng/ODC/DETODC.HTM IBM

NNTu - SQM W2009

HoaSen University

42

Q/A

NNTu - SQM W2009

HoaSen University

43