Software Failures Report 2

BSc.
(Hons) in Computing
Professional Issues in Computing The University of Bolton
Software Failures
What can be done to improve the quality of our
software and the consequences of this?
by Ooi Teong Kuan

Bolton ID: 0509033
Date: 13 November 2006
Professional Issues in Computing
Software Failures The University of Bolton
Acknowledgements
The author would like to thank the lecturers of SEGi College for contributing their
valuable advices and guidance on writing this report. They are Miss Jaya
Letchumi (our PIC lecturer), Mr. Tan Teik Thai (Head of Computer Department),
and Miss Jeannie, our former PIC lecturer who had left us in August 2006.
Another special thank to Miss Louise Blenkharn, who came all the way from UK in
giving us a great insight of writing a truly professional sound report as well as
good presentation skills. Also thanks to all the online publishers as listed in the
reference list and bibliography for sharing out their ideas and knowledge in
making this report a more convincing one with relevant supported facts. Last but
not least, many thanks to my fellow classmates, who always have the spirit of
sharing, and also my special friend, Miki Lee, for giving out her full support and
confidence on this report.
Prepared by: Ooi Teong Kuan

Bolton ID: 0509033 i
Summary
Not all software problems are considered software failures. Software failures have
been defined by J. C. Laprie, the author of ‘Dependability: Basic Concept &
Terminology’, as “the system service that no longer complies with the agreed
specification” (J. C. Laprie, 1992). Some of the possible failures are like system
crashes, inaccurate output, security vulnerability, and data integrity issues.
To improve the software quality, identifying root causes of the problems is

important as the starting point of the software improvement lifecycle, followed by
taking appropriate actions and solutions based on the proven approaches
developed by certain reputable organisations, such as CMMI, UML, RUP, and
Software Factory. After that, monitoring the progress and evaluating results are
important to ensure the intended objectives can be met. For continuous
improvements, repeating the entire software improvement lifecycle is necessary.
Consequences of software failures will lead to the following three impacts:

1. Consumer Impact
2. Industry Impact
3. Economy Impact
As a conclusion, it is wiser to invest more on software quality improvement plans

rather than causing losses to all the parties involved due to software failure. It is
also recommended that improving human quality is important as it is a key factor
to ensure that all the industry best practices can be implemented successfully.

Bolton ID: 0509033 ii
Table of Contents
1.0 TERMS OF REFERENCE...................................................................................................1
2.0 INTRODUCTION .................................................................................................................2
3.0 WHAT IS “SOFTWARE FAILURE”? ..................................................................................3
3.1 DEFINITION OF SOFTWARE FAILURE ....................................................................................3

3.2 EXAMPLES OF SOFTWARE FAILURES ...................................................................................3
4.0 HOW CAN WE IMPROVE THE SOFTWARE QUALITY? ................................................5
4.1 SOFTWARE QUALITY IMPROVEMENT LIFECYCLE...................................................................5

4.2 IDENTIFYING THE ROOT CAUSES.........................................................................................6
4.3 TAKING APPROPRIATE ACTIONS AND SOLUTIONS .................................................................6
4.3.1 Capability Maturity Model Integration (CMMI) ............................................................7
4.3.2 Unified Modeling Language (UML) .............................................................................8
4.3.3 Rational Unified Process (RUP) .................................................................................9
4.3.4 Software Factory .........................................................................................................9
4.4 MONITORING AND EVALUATING THE RESULTS ......................................................................9
4.4.1 Software Quality Assurance Test ..............................................................................10
4.4.1.1 Unit Test ...........................................................................................................10
4.4.1.2 Integrated Test..................................................................................................10
4.4.1.3 Stress Test........................................................................................................11
4.4.2 CMMI Software Product Quality Measurement ........................................................11
4.5 GOING THROUGH THE IMPROVEMENT LIFECYCLE ...............................................................12
5.0 WHAT ARE THE CONSEQUENCES?............................................................................14
5.1 CONSUMER IMPACT .........................................................................................................15

5.1.1 Impact to Standalone Software Users ......................................................................15
5.1.2 Impact to Online Transaction Users..........................................................................15
5.2 INDUSTRY IMPACT ............................................................................................................16
5.3 ECONOMIC IMPACT ..........................................................................................................16
6.0 CONCLUSIONS ................................................................................................................18
7.0 RECOMMENDATIONS .....................................................................................................19

Bolton ID: 0509033 iii
APPENDICES ................................................................................................................................21
APPENDIX I. – ARTICLE: CAPABILITY MATURITY MODEL ................................................................21

APPENDIX II. – PRESENTATION (EXTRACTED): MEASURING SOFTWARE PRODUCT QUALITY: THE
ISO 25000 SERIES AND CMMI .....................................................................................................33
APPENDIX III. – ARTCLE: SOFTWARE INDEPENDENT VERIFICATION AND VALIDATION
(IV&V)/INDEPENDENT ASSESSMENT (IA) CRITERIA.........................................................................38
APPENDIX IV. – ARTICLE: QUANTIFYING ECONOMIC IMPACT OF INFORMATION INTEGRITY FAILURES 46
BIBLIOGRAPHY ............................................................................................................................55
REFERENCE LIST.........................................................................................................................57

Bolton ID: 0509033 iv
Table of Tables
Table 1 – Some of the industry approaches and the software development processes that each of
them covers. .....................................................................................................................................7
Table 2 – CMMI Performance Results as of 15 December 2005 (Software Engineering Institute,
2005).................................................................................................................................................8
Table of Figures
Figure 1 – Software Quality Improvement Lifecycle.........................................................................5
Figure 2 – Relating Requirements, Evaluation, and Measurement (Dave Zubrow, 2004, p.11) ....12
Figure 3 – One of the observations from SEI after the year 2006 State of Software Measurement
Practice Survey (Mark Kasunic, 2006, p.37) ..................................................................................13

Bolton ID: 0509033 v
1.0 Terms of Reference

Before starting to explain the way of improving the software quality, first, it is
necessary to understand the actual meaning behind “software failures”. Hence,
this report has a kick start on defining the meaning of software failure based on J.
C. Laprie’s definition.
About the software quality improvement approach, this report is taking the
assumption that the software problems are the reasons for improvement.
Therefore, the terminology starts from identifying root causes of the problems,
followed by taking appropriate actions and solutions, which introduce some of the
proven approaches developed by certain reputable organisations, such as CMMI
from Software Engineering Institute (SEI).
The consequences have been divided into the following three major impacts:
1. Consumer Impact
2. Industry Impact
3. Economy Impact
Each of these impacts will be elaborated in detail together with the relevant
supported facts. Conclusions have been made based on the studies of this report
and some constructive recommendations have been given by the author as the
closing end of this report.
Prepared by: Ooi Teong Kuan 1

Bolton ID: 0509033
2.0 Introduction
As human’s day-to-day life has become more and more dependent on computer
software, efforts to reduce software failure and to improve its quality have become
vital importance. Unlike thirty years ago, computers were mostly used by
government agencies in the first world countries for scientific research purposes.
Today, the users range from government to education, business, and even
consumers that include small little kids, not to mention about the tremendous
growth of Internet users throughout the world. By having such volume of users,
and for those Internet users that typically accessing the same piece of server
application in particular, down of one server system will enough to harm millions of
user at a time.
Although there are many proven industrial approaches available in the market, it
seems no significant improvements until today and to the local made software in
the country of Malaysia especially. Therefore, instead of repeating the same
in-depth technical details in this report again, it is time to widen the area of
considerations and find out the root causes in making the approaches much more
worthwhile to be implemented.
Hence, this report attempts to target on readers who may have already heard
about and/or implemented some of the approaches but still unable to get a
satisfactory result. In addition, it is also suitable to those who are new to software
quality improvement techniques due to its non-in-depth, introductory level of
explanations. However, for those who would like to drill through the details, useful
references are also available at the end of this report.

Bolton ID: 0509033
3.0 What is “Software Failure”?
3.1 Definition of Software Failure
Not all software problems can be classified as ‘software failure’. According to J. C.

Laprie (1992), the author of ‘Dependability: Basic Concept and Terminology’, “A
system failure occurs when the delivered service no longer complies with the
specifications, the latter being an agreed description of the system's expected
function and/or service” (J. C. Laprie, 1992). Apparently, software problems which
are not being committed, and/or not fell under the same category of the agreed
deliverables in the specification, are not considered as software failures. This is
because every system has its own missions and promises to accomplish. If those
missions and promises have been stated in the specification and had been
agreed upon by the provider and end-users, the software are then need to comply
with the specification, otherwise they are considered ‘fail’ to deliver as promised.
For example, if the specification stated that ‘this software can be scaled up to 100
concurrent users without failing’ and the system stops responding when it reaches
101 concurrent users, it is not considered a software failure.
3.2 Examples of Software Failures
While only the software problems that do not comply with the specification are
legitimately able to classify as ‘software failures’, some of the following issues are
commonly classified as software failures by most of the software providers:
• Unreasonable crashes or hangs of the system

• Logically inaccurate output of information
• Security vulnerabilities that defeat the purpose of the security features
• Loss of data integrity, consistency, and accuracy due to program logic and
system design

Bolton ID: 0509033
The above issues may be considered as failures although they are not being
specified in the software agreement because they are preventing the fundamental
roles of the system from being accomplished. For example, 1 + 1 = 2 but the
printed output of the system appears as ‘3’. Apparently, this is an unacceptable
bug and should be classified as software failure even though a statement to
guarantee the output accuracy was not explicitly stated in the agreement.

Bolton ID: 0509033
4.0 How can we improve the software quality?
4.1 Software Quality Improvement Lifecycle
It is unable to deny that none of the software in this world is 100% bugs free.
However, its quality has to be improved if the bugs are not able to be fully
eliminated. When mentioning about ‘improvement’, it also implies the meaning of
‘continuous improvements’, hence the word ‘lifecycle’. Figure 1 shows the high
level of Software Quality Improvement Lifecycle:
Software Quality Improvement Lifecycle
Identify Root
Causes
Go through
the Lifecycle Take Appropriate
Again Actions &
Solutions
Monitor &
Evaluate
Results
Figure 1 – Software Quality Improvement Lifecycle
The sections below explain each of the processes as specified in the above
diagram.

Bolton ID: 0509033
4.2 Identifying the Root Causes
Root Cause Analysis (RCA) is not only important in the manufacturing industry but
also in software industry as well. Without investigating and identifying the root
cause of the software problems, it is almost impossible for a software company to
establish an effective solution or methodology in eliminating certain issues and
also for better improvements. Problems that repetitively occur will be the very
good examples in proving that the root of the problems is not being identified and
resolved with proper solution. As mentioned in the RCA seminar brochure
conducted by Ops A La Carte:
“Also, they become experts in fixing rather than preventing the problems. What is
left is very reactive method of fixing equipment rather than a pro-active method of
solving problems.” (Ops A La Carte, 2006, p. 1)
4.3 Taking Appropriate Actions and Solutions
After identifying the root causes, appropriate actions need to be taken, such as
seeking for proven software modeling, standards, best practices, and process
management techniques to solve the problems, once and for all. Most of these
approaches are not only help to design and develop better software but also
contain techniques to promote highly visible system specification and better
predictability of software issues. As far as the software quality is concerned,
Quality Assurance Test alone has never been enough. Other software
development processes, such as requirements studies, design, development,
deployment, and implementation, are processes that highly responsible to the
final quality of the software products. The following table shows some of the
well-known and proven approaches that software companies can apply to
improve their software development processes:

Bolton ID: 0509033
Requirements Evaluation & Deployment &

Approach Design Development
Studies Testing Implementation
Capability
Maturity Model
Integration 3 3 3 3 3
(CMMI)
Unified Modeling
Language (UML) 3 3 3
Rational Unified
Processed
(RUP)
3 3 3 3 3
Software
Factory 3 3 3 3
Table 1 – Some of the industry approaches and the software development processes that
each of them covers.
The sections below explain each of these approaches in more detail.
4.3.1 Capability Maturity Model Integration (CMMI)
CMMI is a process improvement approach developed by Software Engineering

Institute (SEI). It has been widely adopted by many software companies through
out the world, not only having the intention to improve their products quality and
internal management processes but also to gain trusts and confidence from their
customers. It consists of five levels that the companies can strive to achieve. They
are:
Maturity Level 1 – Initial

Maturity Level 2 – Repeatable
Maturity Level 3 – Defined
Maturity Level 4 – Managed
Maturity Level 5 – Optimising

Bolton ID: 0509033
For more detail of each maturity level, please refer to Appendix I – ARTICLE:
Capability Maturity Model or visit:
http://en.wikipedia.org/wiki/Capability_Maturity_Model
In order to show the effectiveness of their process improvement approach, SEI

has compiled quantitative evidence as shown in Table 2 below. The last row of
the table, that is the Return on Investment ratio, shows that the overall results are
quite significant and on productivity and quality in particular.
Performance Category Median Number of Data Points Low High
Cost 20% 21 3% 87%
Schedule 37% 19 2% 90%
Productivity 62% 17 9% 255%
Quality 50% 20 7% 132%
Customer Satisfaction 14% 6 -4% 55%
Return on Investment 4.7 : 1 16 2 : 1 27.7 : 1
Table 2 – CMMI Performance Results as of 15 December 2005

(Software Engineering Institute, 2005)
4.3.2 Unified Modeling Language (UML)
UML is a general purpose modeling language which is not only meant for software
engineering but also for business process modeling. It is not a proprietary
language and therefore it is opened freely to public to extend its potential and
capability, which they think best to describe their software in a graphical manner.
In other words, UML is meant to be customised. However, certain object modeling
rules and principles still need to be followed in order to reach the intended results
as expected by UML. The examples of UML-based modeling diagrams are like
class diagrams, components diagrams, and system behaviors diagrams.

Bolton ID: 0509033
4.3.3 Rational Unified Process (RUP)
RUP is quite similar to UML because it is in fact an extension of UML. As

mentioned in the previous section, UML is not only meant for business process
modeling but it is also widely used in software industry. In addition, it is also
opened for customisation to suite the industrial needs. Therefore, it is not
surprised that RUP was actually being designed based on the concept of UML.
The company that founded RUP is Rational Software Corporation, which now
becomes part of IBM.
4.3.4 Software Factory
The concept of software factory is very much like a manufacturing industry where
the final products will be assembled based on the reusable components. All these
smaller software components are well-tested, high quality, and flexible enough to
be plugged into a particular framework to form a complete software product. Such
components, although consists of standard functionalities, they can be blended to
different combinations in order to meet different product requirements.
4.4 Monitoring and Evaluating the Results
While the approaches stated above are proven and tested effective, different
groups or individuals may implement them differently on different environment.
Therefore, certain extents of progress monitoring and results evaluation are
important. The output quality needs to be tested to ensure that the intended
objectives can be met.
Software testing is also playing a very important role in this process. It ensures
that all the user requirements can be met and the reliability of the system during
deployment and implementation. The following sections introduce two popular
methods:

Bolton ID: 0509033
1. Software Quality Assurance Test

2. CMMI Software Product Quality Measurement
4.4.1 Software Quality Assurance Test
This is the most common and traditional way of testing the software quality. It
consists of three types of testing:
4.4.1.1 Unit Test
Unit Test is a detailed test that focuses on just a particular unit of a program, such
as a data entry form. It includes validity of the data input, behaviors, and displays
under which all sorts of criteria have been tested. However, it does not take into
consideration the affection of other units, modules, or source of the system. For
example, a unit test has been successfully conducted to change the credit limit of
a customer but the tester did not take into consideration the sales order portions
and how the changes affect the sales order transaction.
4.4.1.2 Integrated Test
The integrated test is an overall testing of the entire system. It includes testing of
program flows, data integrity, output accuracy that may be affected by the other
part of the system. However, problems detected during integrated test may be
harder to debug if the basic unit test was not conducted fully. A well-performed
unit test will help streamline the work of the integrated test and making the
problems easier to trace and debug.

Bolton ID: 0509033
4.4.1.3 Stress Test
Stress Test is a very important QA test to mission critical enterprise solutions in

particular. It is basically a performance or load test that test on high volume
transaction data and user concurrency effectiveness. It requires test environment,
manpower, and preparation of large volume of data. Due to the relatively high cost
of such testing, many organisations tend to ignore it.
4.4.2 CMMI Software Product Quality Measurement
In the past, many people believe that software quality can be assured by
performing adequate testing as mentioned in Section 4.4.1. However, the
question is: What if the requirements are not met even though the software is
extremely stable and 99.9% bugs free? CMMI Software Product Quality
Measurement is a modern way of measuring software quality. Below is how CMMI
defines Quality Requirements:
“The phrase ‘quality and process-performance objectives’ covers objectives and

requirements for product quality, service quality, and process performance.
Process performance objectives include product quality.” (Dave Zubrow, 2004,
p.5)
Therefore, they are not solely focus on physical and end product quality
inspection, but they also believe other software development processes, such as
Requirements Development, will affect the product quality as well. In Figure 2, it
shows that the Product Quality Measurement can be conducted during Product
Requirements Quality and Product Quality Evaluation processes.

Bolton ID: 0509033
Figure 2 – Relating Requirements, Evaluation, and Measurement (Dave Zubrow, 2004, p.11)
As the objectives for ‘quality’ are too conceptual, they have to be translated into
operational guidelines in order to achieve the desired quality in a realistic manner.
Measurement plays a very important role in this area. It can really provide an
efficient way of achieving the acceptance criteria based on the requirements
specification. The help of ISO 25000 series and GQ(I)M Indicator Template can
ease the implementation of this measurement. (Dave Zubrow, 2004, p.28)
4.5 Going through the Improvement Lifecycle
It is less likely that every software product is able to fully meet the expected
results during the first rollout. In fact, there are always rooms for improvement in
terms of functionality, stability, scalability, and performance. It is a good practice
by constantly going through the improvement lifecycle so that corrective actions
can be taken to refine the quality of the products.

Bolton ID: 0509033
Figure 3 – One of the observations from SEI after the year 2006 State of Software
Measurement Practice Survey (Mark Kasunic, 2006, p.37)
According to Mark Kasunic (2006), Software Engineering Institute (SEI) has

conducted a year 2006 State of Software Measurement Practice Survey and
observed that almost 20% of the respondents seldom or never took corrective
action when a measurement threshold is exceeded (see Figure 3). Also in Figure
3, it is highlighted that measurement has become useless unless the information
is used to take appropriate action.

Bolton ID: 0509033
5.0 What are the consequences?

In fact, the consequences of software failure and the severity of its impact are very
much depending on the following factors:
• The purpose that a particular software serves

• The type of industry that the software serves
• The users who use it and the number of users
• The roles that the software plays
• The size of the system
According to NASA (n.d.) Software Independent Verification and Validation

(IV&V), factors that contributing to the consequences of software failure may
include the following:
1. Potential for loss of life

2. Potential for serious injury
3. Potential for catastrophic mission failure
4. Potential for partial mission failure
5. Potential for loss of equipment
6. Potential for waste of software resource investment
7. Potential for adverse publicity
8. Potential effect on routine operations
If looking from other perspectives, some or all of the factors listed above may
impact the consumer, industry, and economic areas. The impact to each of these
areas will be discussed in more detail in the following sections.

Bolton ID: 0509033
5.1 Consumer Impact
5.1.1 Impact to Standalone Software Users
The severity of impact to consumer like home users due to software failure may
not be significant because the prices that charged to those individuals are typically
low. Furthermore, software being used for this group of individuals is usually less
critical compared to corporate users. However, due to the volume and consumer
market size, failure of personal software products may greatly impact the entire
industry businesses as a whole. This may also cause the increasing use of
unlicensed and pirated software due to disappointments and frustrations given by
the genuine products. More and more consumers are reluctant to risk their money
because of the past experiences with licensed software.
5.1.2 Impact to Online Transaction Users
Another group of consumer is online Internet users who perform buying and
selling over the Internet in particular. When mention about Internet, two major
failures will cause huge impact on online consumer:
1. Security
Password thefts and exposure of credit card numbers are the very common
issues happening to online users. This could be due to the failure of user
authentication system and the implementation of Secure Socket Layer (SSL).
Such failures will allow hackers to gain unauthorised access to owner’s
banking account or making use of the owner’s credit card number to perform
online shopping. For example, e1040.com, the popular tax-preparation Web
site, was being sued by their customers because the customers’ private
information, including passwords and Social Security numbers, were displayed
in plain text on the Internet due to a switching error in the site's encryption
software (Salimol Thomas, 2003, p.5).

Bolton ID: 0509033
2. Data Integrity
Web applications like online banking and online shopping usually involve
monetary transactions, for examples online fund transfer, bills payment, and
credit card payment. If the application failed to handle the transaction, money
can be deducted from the user’s account but the payee has not been received
the payment.
5.2 Industry Impact
As mentioned in Section 5.1, consumer impact due to software failure may also
introduce unwanted consequences to software industry, such as increasing use of
pirated software and unlicensed software. When this becomes a tragedy to the
software industry, the loss is not only directly come from this but millions of dollars
need to be spent in order to fight piracy and also to give more incentives to users
who buy genuine products. The incentives are like giving out free software
add-ons and provide free supports and services, which incur very high costs to the
software companies.
In large corporate sector, software failures can cause up to billions of dollar lost,
not only at the end-user sides but also to the software providers as well. For
instance, FoxMeyer sued SAP and Andersen Consulting for $500 million each
because of SAP's software failure (Salimol Thomas, 2003, p.5). Such
consequences will not only lead to loss of tangible assets to the corporate users
but also intangible assets such as tremendous loss of productivity, efficiency,
company image, and competitive advantage.
5.3 Economic Impact
As described in Section 5.1 and 5.2, consumer and industry impacts resulted by
software failure are definitely affecting the economy, not only in a country alone

Bolton ID: 0509033
but worldwide. While globalisation becomes a key factor for economic growth to a
particular country, demand for computer software in general and online
transaction processing applications in particular have become more and more
aggressive. To a development country like Malaysia, billions of dollars have been
invested to buy software products and bring in software experts from other
countries such as U.S. and India due to lack of local expertise. When majority of
software being used in the country are imported software, failures of those
software can really bring a huge impact to the economic growth of the country.
Imagine billions of dollars are wasted, return of investments (ROI) are in negative
values, loss of productivity, loss of income, and increase of liability costs.
Of course, such an economic impact is not involving the development countries

alone. “In the United States, billions of dollars are invested on software application
development. However, many of these projects fail due to poor design and
deployment errors. For example, in 1999, the failure of NASA’s Mars Lander
missions has cost a total loss of US$360 million. The cause of the failure was
traced back to a software error.” (Salimol Thomas, 2003, p.2)

Bolton ID: 0509033
6.0 Conclusions
By comparing the prices being paid for software quality improvement plans and
the losses caused by software failures, the following conclusions can be
confidently made:
1. Fixing bugs is more painful than preventing bugs

Bugs fixing may not be the only cost bared by the development team alone. It
may also mean losses that affect more than one party, such as loss of
investment, productivity, and efficiency at the end-user’s side and therefore
liability cost to the company. Therefore, “deliver first, fix later” is always a bad
practice.
2. Improving product quality is better than fighting piracy

The cost of anti-piracy campaign always comes together with the loss of
business and revenues. Therefore cost of improving software quality is far
better than spending money to fight software piracy.
3. Investing on quality improvements is better than paying for liability costs

By taking the example as specified in Chapter 5.0, Section 5.2, by spending
half of $500 million in improving the software quality, SAP could have avoided
the court case, saving up to $250 million dollars, and at the same time
maintaining good business relationship with FoxMeyer.
Therefore, since improving software quality can turn those possible losses into
revenues, why not doing it?

Bolton ID: 0509033
7.0 Recommendations
Regardless of how great that the industry approaches were being established, the
success of these approaches is still depending on people who:
1. Able to promote them in a very motivating manner

2. Able to transfer the knowledge effectively
3. Realise their benefits and potential
4. Appreciate their intended objectives
5. Willing to study and explore it
6. Passionate to make them effective and contributing
7. Discipline enough to follow the rules and guidelines being set by the
approaches
Besides, top management people and key persons who drive the organization are
also playing a very important role to ensure:
1. Serving customers with high quality software is always the top most priority;
2. Good company culture that promotes “work smart” to avoid employees from
being too exhausted, which may in turn lead to unproductive works,
inefficiency, and full of mistakes;
3. Good policies that encourage learning, R&D, knowledge sharing, and
attending trainings, seminars, and conferences;
4. Good company philosophy that realises on “product quality drives revenue”
instead of “revenue first then quality later”.
Without the above human qualities, the goal towards high quality state-of-the-art
software will be harder to achieve. Bear in mind that all computer software
instructed by human. In other words, the quality of the software is actually
reflecting the human quality. Therefore, instead of feeling doubtful to the
effectiveness of the industry approaches, why not putting more efforts in

Bolton ID: 0509033
improving people quality as the fundamental approach? Once the human quality
is improved, all the required attitudes and mindset will be in place. Naturally, all
the great approaches will be easily absorbed and the success of the software
products will just be around the corner.

Bolton ID: 0509033
Appendices
Appendix I. – ARTICLE: Capability Maturity Model
Title: Capability Maturity Model

Author: (Wikipedia’s Contributors)
Publisher: Wikipedia
Date: -
URL: http://en.wikipedia.org/wiki/Capability_Maturity_Model
Date Accessed: 15 September 2006

Bolton ID: 0509033
Capability Maturity Model
From Wikipedia, the free encyclopedia
Capability Maturity Model (CMM) broadly refers to a process improvement approach that is based on a process
model. CMM also refers specifically to the first such model, developed by the Software Engineering Institute (SEI)
in the mid-1980s, as well as the family of process models that followed. A process model is a structured collection
of practices that describe the characteristics of effective processes; the practices included are those proven by
experience to be effective. [1]
CMM can be used to assess an organization against a scale of five process maturity levels. Each level ranks the
organization according to its standardization of processes in the subject area being assessed. The subject areas can
be as diverse as software engineering, systems engineering, project management, risk management, system
acquisition, information technology (IT) services and personnel management.
CMM was developed by the SEI at Carnegie Mellon University in Pittsburgh. It has been used extensively for
avionics software and government projects, in North America, Europe, Asia, Australia, South America, and Africa.
[2]
Currently, some government departments require software development contract organization to achieve and
operate at a level 3 standard.
Contents
1 Maturity model
2 History
2.1 Context
2.2 Origins
2.3 Current state
2.4 Future direction
3 Levels of the CMM
3.1 Level 1 - Initial
3.2 Level 2 - Repeatable
3.3 Level 3 - Defined
3.4 Level 4 - Managed
3.5 Level 5 - Optimizing
3.6 Extensions
4 Process areas
5 Controversial aspects
5.1 Praise
5.2 Criticism
6 The most beneficial elements of CMM Level 2 and 3
7 Companies appraised against the CMMI
8 See also
9 References
10 Footnotes
11 External links
Maturity model
The Capability Maturity Model (CMM) is a way to develop and refine an organization's processes. The first
CMM was for the purpose of developing and refining software development processes. A maturity model is a
structured collection of elements that describe characteristics of effective processes. A maturity model provides:
a place to start
the benefit of a community’s prior experiences
a common language and a shared vision
a framework for prioritizing actions
a way to define what improvement means for your organization
A maturity model can be used as a benchmark for assessing different organizations for equivalent comparison. It
describes the maturity of the company based upon the project the company is dealing with and the clients.
History
The Capability Maturity Model was initially funded by military research. The United States Air Force funded a
study at the Carnegie-Mellon Software Engineering Institute to create a model (abstract) for the military to use as
an objective evaluation of software subcontractors. The result was the Capability Maturity Model, published as
Managing the Software Process in 1989. The CMM is no longer supported by the SEI and has been superseded by
the more comprehensive Capability Maturity Model Integration (CMMI), of which version 1.2 has now been
released.
Context
In the 1970s, technological improvements made computers more widespread, flexible, and inexpensive.
Organizations began to adopt more and more computerized information systems and the field of software
development grew significantly. This led to an increased demand for developers—and managers—which was
satisfied with less experienced professionals.
Unfortunately, the influx of growth caused growing pains; project failure became more commonplace not only
because the field of computer science was still in its infancy, but also because projects became more ambitious in
scale and complexity. In response, individuals such as Edward Yourdon, Larry Constantine, Gerald Weinberg,
Tom DeMarco, and David Parnas published articles and books with research results in an attempt to
professionalize the software development process.
Watts Humphrey's Capability Maturity Model (CMM) was described in the book Managing the Software Process
(1989). The CMM as conceived by Watts Humphrey was based on the earlier work of Phil Crosby. Active
development of the model by the SEI (US Dept. of Defense Software Engineering Institute) began in 1986.
The CMM was originally intended as a tool to evaluate the ability of government contractors to perform a
contracted software project. Though it comes from the area of software development, it can be, has been, and
continues to be widely applied as a general model of the maturity of processes (e.g., IT Service Management
processes) in IS/IT (and other) organizations.
The model identifies five levels of process maturity for an organisation:
1. Initial (chaotic, ad hoc, heroic) the starting point for use of a new process.
2. Repeatable (project management, process discipline) the process is used repeatedly.
3. Defined (institutionalized) the process is defined/confirmed as a standard business process.
4. Managed (quantified) process management and measurement takes place.
5. Optimising (process improvement) process management includes deliberate process
optimization/improvement.
Within each of these maturity levels are KPAs (Key Process Areas) which characterise that level, and for each
KPA there are five definitions identified:
1. Goals
2. Commitment
3. Ability
4. Measurement
5. Verification
The KPAs are not necessarily unique to CMM, representing - as they do - the stages that organizations must go
through on the way to becoming mature.
The assessment is supposed to be led by an authorised lead assessor. One way in which companies are supposed to
use the model is first to assess their maturity level and then form a specific plan to get to the next level. Skipping
levels is not allowed.
NB: The CMM was originally intended as a tool to evaluate the ability of government contractors to perform a
contracted software project. It may be suited for that purpose. When it became a general model for software
process improvement, there were many critics.
"Shrinkwrap" companies are also called "COTS" or commercial-off-the-shelf firms or software package firms.
They include Claris, Apple, Symantec, Microsoft, and Lotus, amongst others. Many such companies rarely if ever
managed their requirements documents as formally as the CMM described in order to achieve level 2, and so all of
these companies would probably fall into level 1 of the model.
Origins
The United States Air Force funded a study at the SEI to create a model for the military to use as an objective
evaluation of software subcontractors. In 1989, the Capability Maturity Model was published as Managing the
Software Process.
Timeline
1987: SEI-87-TR-24 (SW-CMM questionnaire), released.

1989: Managing the Software Process, published.
1991: SW-CMM v1.0, released.
1993: SW-CMM v1.1, released.
1997: SW-CMM revisions halted in support for CMMI.
2000: CMMI v1.02, released.
Current state
Although these models have proved useful to many organizations, the use of multiple models has been
problematic. Further, applying multiple models that are not integrated within and across an organization is costly in
terms of training, appraisals, and improvement activities. The CMM Integration project was formed to sort out the
problem of using multiple CMMs. The CMMI Product Team's mission was to combine three source models:
1. The Capability Maturity Model for Software (SW-CMM) v2.0 draft C

2. The Systems Engineering Capability Model (SECM)
3. The Integrated Product Development Capability Maturity Model (IPD-CMM) v0.98
4. Supplier sourcing
CMMI is the designated successor of the three source models. The SEI has released a policy to sunset the Software
CMM and previous versions of the CMMI. [3] The same can be said for the SECM and the IPD-CMM; these
models were superseded by CMMI.
Future direction
With the release of the CMMI Version 1.2 Product Suite, the existing CMMI has been renamed the CMMI for
Development (CMMI-DEV), V1.2.[1] A version of the CMMI for Services is being developed by a Northrop
Grumman-led team under the auspices of the SEI, with participation from Boeing, Lockheed Martin, Raytheon,
SAIC, SRA, and Systems and Software Consortium (SSCI).[2] A CMMI for Acquisition (CMMI-ACQ) is also
under development at the SEI.[3]
Suggestions for improving CMMI are welcomed by the SEI. For information on how to provide feedback, see the
CMMI Web site.
In some cases, CMM can be combined with other methodologies. It is commonly used in conjunction with the ISO
9001 standard. JPMorgan Chase & Co. tried combining CMM with the computer programming methodologies of
Extreme Programming (XP), and Six Sigma. They found that the three systems reinforced each other well, leading
to better development, and did not mutually contradict, see Extreme Programming (XP) Six Sigma CMMI.
Levels of the CMM

(See chapter 2 of (March 2002 edition of CMMI from SEI), page 11.)
There are five levels of the CMM. According to the SEI,
"Predictability, effectiveness, and control of an organization's software processes are believed to improve as
the organization moves up these five levels. While not rigorous, the empirical evidence to date supports this
belief."
Level 1 - Initial
At maturity level 1, processes are usually ad hoc and the organization usually does not provide a stable
environment. Success in these organizations depends on the competence and heroics of the people in the
organization and not on the use of proven processes. In spite of this ad hoc, chaotic environment, maturity level 1
organizations often produce products and services that work; however, they frequently exceed the budget and
schedule of their projects.
Maturity level 1 organizations are characterized by a tendency to over commit, abandon processes in the time of
crisis, and not be able to repeat their past successes again.
Level 1 software project success depends on having quality people.
Level 2 - Repeatable
At maturity level 2, software development successes are repeatable. The processes may not repeat for all the
projects in the organization. The organization may use some basic project management to track cost and schedule.
Process discipline helps ensure that existing practices are retained during times of stress. When these practices are
in place, projects are performed and managed according to their documented plans.
Project status and the delivery of services are visible to management at defined points (for example, at major
milestones and at the completion of major tasks).
Basic project management processes are established to track cost, schedule, and functionality. The minimum
process discipline is in place to repeat earlier successes on projects with similar applications and scope. There is
still a significant risk of exceeding cost and time estimate.
Level 3 - Defined
The organization’s set of standard processes, which is the basis for level 3, is established and improved over time.
These standard processes are used to establish consistency across the organization. Projects establish their defined
processes by the organization’s set of standard processes according to tailoring guidelines.
The organization’s management establishes process objectives based on the organization’s set of standard
processes and ensures that these objectives are appropriately addressed.
A critical distinction between level 2 and level 3 is the scope of standards, process descriptions, and procedures. At
level 2, the standards, process descriptions, and procedures may be quite different in each specific instance of the
process (for example, on a particular project). At level 3, the standards, process descriptions, and procedures for a
project are tailored from the organization’s set of standard processes to suit a particular project or organisational
unit.
Level 4 - Managed
Using precise measurements, management can effectively control the software development effort. In particular,
management can identify ways to adjust and adapt the process to particular projects without measurable losses of
quality or deviations from specifications. At this level organization set a quantitative quality goal for both software
process and software maintenance.
Subprocesses are selected that significantly contribute to overall process performance. These selected subprocesses
are controlled using statistical and other quantitative techniques.
A critical distinction between maturity level 3 and maturity level 4 is the predictability of process performance. At
maturity level 4, the performance of processes is controlled using statistical and other quantitative techniques, and
is quantitatively predictable. At maturity level 3, processes are only qualitatively predictable.
Level 5 - Optimizing
Maturity level 5 focuses on continually improving process performance through both incremental and innovative
technological improvements. Quantitative process-improvement objectives for the organization are established,
continually revised to reflect changing business objectives, and used as criteria in managing process improvement.
The effects of deployed process improvements are measured and evaluated against the quantitative process-
improvement objectives. Both the defined processes and the organization’s set of standard processes are targets of
measurable improvement activities.
Process improvements to address common causes of process variation and measurably improve the organization’s
processes are identified, evaluated, and deployed.
Optimizing processes that are nimble, adaptable and innovative depends on the participation of an empowered
workforce aligned with the business values and objectives of the organization. The organization’s ability to rapidly
respond to changes and opportunities is enhanced by finding ways to accelerate and share learning.
A critical distinction between maturity level 4 and maturity level 5 is the type of process variation addressed. At
maturity level 4, processes are concerned with addressing special causes of process variation and providing
statistical predictability of the results. Though processes may produce predictable results, the results may be
insufficient to achieve the established objectives. At maturity level 5, processes are concerned with addressing
common causes of process variation and changing the process (that is, shifting the mean of the process
performance) to improve process performance (while maintaining statistical probability) to achieve the established
quantitative process-improvement objectives.
Extensions
Recent versions of CMMI from SEI indicate a "level 0", characterized as "Incomplete". Many observers leave this
level out as redundant or unimportant, but Pressman and others make note of it. See page 18 of the August 2002
edition of CMMI from SEI. [4]
Anthony Finkelstein [5] extrapolated that negative levels are necessary to represent environments that are not only
indifferent, but actively counterproductive, and this was refined by Tom Schorsch [6] as the Capability Immaturity
Model:
Process areas
For more details on this topic, see Process area (CMMI).
The CMMI contains several key process areas indicating the aspects of product development that are to be covered
by company processes.
Key Process Areas of the Capability Maturity Model Integration (CMMI)

Abbreviation Name Area Maturity Level
CAR Causal Analysis and Resolution Support 5
CM Configuration Management Support 2
DAR Decision Analysis and Resolution Support 3
IPM Integrated Project Management Project Management 3
ISM Integrated Supplier Management Project Management 3
IT Integrated Teaming Project Management 3
MA Measurement and Analysis Support 2
OEI Organizational Environment for Integration Support 3
OID Organizational Innovation and Deployment Process Management 5
OPD Organizational Process Definition Process Management 3
OPF Organizational Process Focus Process Management 3
OPP Organizational Process Performance Process Management 4
OT Organizational Training Process Management 3
PI Product Integration Engineering 3
PMC Project Monitoring and Control Project Management 2
PP Project Planning Project Management 2
PPQA Process and Product Quality Assurance Support 2
QPM Quantitative Project Management Project Management 4
RD Requirements Development Engineering 3
REQM Requirements Management Engineering 2
RSKM Risk Management Project Management 3
SAM Supplier Agreement Management Project Management 2
TS Technical Solution Engineering 3
VAL Validation Engineering 3
VER Verification Engineering 3
Controversial aspects
The software industry is diverse and volatile. All methodologies for creating software have supporters and critics,
and the CMM is no exception.
Praise
The CMM was developed to give Defense organizations a yardstick to assess and describe the capability of
software contractors to provide software on time, within budget, and to acceptable standards. It has arguably
been successful in this role, even reputedly causing some software sales people to clamour for their
organizations' software engineers/developers to "implement CMM."
The CMM is intended to enable an assessment of an organization's maturity for software development. It is
an important tool for outsourcing and exporting software development work. Economic development
agencies in India, Ireland, Egypt, Syria, and elsewhere have praised the CMM for enabling them to be able
to compete for US outsourcing contracts on an even footing.
The CMM provides a good framework for organizational improvement. It allows companies to prioritize
their process improvement initiatives.
Criticism
CMM has failed to take over the world. It's hard to tell exactly how wide spread it is as the SEI only
publishes the names and achieved levels of compliance of companies that have requested this information to
be listed[4]. The most current Maturity Profile for CMMI is available online[5].
CMM is well suited for bureaucratic organizations such as government agencies, large corporations and
regulated monopolies. If the organizations deploying CMM are large enough, they may employ a team of
CMM auditors reporting their results directly to the executive level. (A practice encouraged by SEI.) The
use of auditors and executive reports may influence the entire IT organization to focus on perfectly
completed forms rather than application development, client needs or the marketplace. If the project is
driven by a due date, CMMs intensive reliance on process and forms may become a hindrance to meeting
the due date in cases where time to market with some kind of product is more important than achieving high
quality and functionality of the product.
Suggestions of scientifically managing the software process with metrics only occur beyond the Fourth
level. There is little validation of the processes cost savings to business other than a vague reference to
empirical evidence. It is expected that a large body of evidence would show that adding all the business
overhead demanded by CMM somehow reduces IT headcount, business cost, and time to market without
sacrificing client needs.
No external body actually certifies a software development center as being CMM compliant. It is supposed
to be an honest self-assessment ([6] and [7]). Some organizations misrepresent the scope of their CMM
compliance to suggest that it applies to their entire organization rather than a specific project or business
unit.
The CMM does not describe how to create an effective software development organization. The CMM
contains behaviors or best practices that successful projects have demonstrated. Being CMM compliant is
not a guarantee that a project will be successful, however being compliant can increase a project's chances
of being successful.
The CMM can seem to be overly bureaucratic, promoting process over substance. For example, for
emphasizing predictability over service provided to end users. More commercially successful methodologies
(for example, the Rational Unified Process) have focused not on the capability of the organization to
produce software to satisfy some other organization or a collectively-produced specification, but on the
capability of organizations to satisfy specific end user "use cases" as per the Object Management Group's
UML (Unified Modeling Language) approach[8].
The most beneficial elements of CMM Level 2 and 3

Creation of Software Specifications, stating what is going to be developed, combined with formal sign off,
an executive sponsor and approval mechanism. This is NOT a living document, but additions are placed in a
deferred or out of scope section for later incorporation into the next cycle of software development.
A Technical Specification, stating how precisely the thing specified in the Software Specifications is to be
developed will be used. This is a living document.
Peer Review of Code (Code Review) with metrics that allow developers to walk through an implementation,
and to suggest improvements or changes. Note - This is problematic because the code has already been
developed and a bad design can not be fixed by "tweaking", the Code Review gives complete code a formal
approval mechanism.
Version Control - a very large number of organizations have no formal revision control mechanism or
release mechanism in place.
The idea that there is a "right way" to build software, that it is a scientific process involving engineering
design and that groups of developers are not there to simply work on the problem du jour.
Companies appraised against the CMMI

Large numbers of IT companies across the world are making forays up the CMMI level ladder. In June, 1999
Wipro of India became the first software services company in the world to attain SEI CMM maturity level 5, the
highest maturity level. Every year many IT companies in the world enter into the CMMI regime or improve their
CMMI Levels. As of 2006, about 75% of the CMMI level 5 software centers are in India. Most of them are located
in the city of Bangalore.
For a complete list view the published SCAMPI results.
See also
Personal Software Process (PSP) and Team Software Process (TSP), two other process models also
developed by the Software Engineering Institute to address individual and team software development
respectively
Information Technology Infrastructure Library (ITIL), a framework of best practice approaches intended to
facilitate the delivery of high quality information technology (IT) services.
Capability Maturity Model Integration

People Capability Maturity Model
The SPICE project
One must be very skeptical about a company claiming that they have obtained a certain level (the higher level, the
more skeptical to be) of CMM at an "enterprise level." Usually this is used as a marketing technique that may
indeed apply to some project done by the company at some time, but most unlikely achieved by the enterprise.
References
Books
Chrissis, Mary Beth, Konrad, Mike, Shrum, Sandy (2003). CMMI : Guidelines for Process
Integration and Product Improvement. Addison-Wesley Professional. ISBN 0321154967.
Humphrey, Watts (1989). Managing the Software Process. Addison-Wesley Professional. ISBN
0201180952.
Websites
History of Process Models
Process Improvement: The Capability Maturity Model
ITNOW - September 2005: Capability model mature - or is it?
The PITCOM meeting of 22 March 2004 addressed one of the major trends affecting the UK
technology sector
Footnotes
1. ^ Capability Maturity Model®Integration (CMMI®) Version 1.2 Overview (PDF). SEI (2006). Retrieved
on 28 October 2006.
2. ^ What is CMMI? - Worldwide Adoption. SEI (2006). Retrieved on 28 October 2006.
3. ^ Sunsetting Version 1.1 of the CMMI® Product Suite. SEI (2006). Retrieved on 28 October 2006.
4. ^ August 2002 edition of CMMI (PDF). CMU/SEI-2002-TR-011. SEI (2002). Retrieved on 28 October
2006.
5. ^ Finkelstein, Anthony (2000-04-28). A Software Process Immaturity Model (PDF). University College
London. Retrieved on 28 October 2006.
6. ^ Schorsch, Tom (1996). The Capability Im-Maturity Model. The Air Force Institute of Technology.
Retrieved on 28 October 2006.
External links
CMMI Official Web Site
Capability Maturity Model® Integration (CMMI®) Overview [PDF]
A critical look at implementing CMM Level 2
Retrieved from "http://en.wikipedia.org/wiki/Capability_Maturity_Model"
Categories: Cleanup from January 2006 | Articles with unsourced statements | Software engineering | Information
technology management
This page was last modified 13:20, 28 October 2006.
All text is available under the terms of the GNU Free Documentation License. (See Copyrights for
details.)
Wikipedia® is a registered trademark of the Wikimedia Foundation, Inc.
Appendix II. – PRESENTATION (Extracted): Measuring Software

Product Quality: The ISO 25000 Series and CMMI
Title: Measuring Software Product Quality:

The ISO 25000 Series and CMMI
Author: Dave Zubrow
Publisher: Software Engineering Institute (SEI)
Date: 14 June 2004
URL: http://www.sei.cmu.edu/sema/presentations/
zubrow/esepg/esepg.pdf
Date Accessed: 25 September 2006

Bolton ID: 0509033
Appendix III. – ARTCLE: Software Independent Verification and

Validation (IV&V)/Independent Assessment (IA) Criteria
Title: Software Independent Verification and Validation (IV&V)/

Independent Assessment (IA) Criteria
Author: -
Publisher: NASA, United States
Date: -
URL: http://ivvcriteria.ivv.nasa.gov/Documents/ivvcrit.pdf
Date Accessed: 3 October 2006

Bolton ID: 0509033
Software Independent Verification and Validation (IV&V)/Independent
Assessment (IA) Criteria
1. Introduction
1.1 The purpose of this appendix is to provide quantifiable criteria for determining whether
IV&V should be applied to a given software development. Since software IV&V should begin
in the Formulation Subprocess (as defined in NPG 7120.5, paragraph 1.4.3) of a project, the
process described here is based on metrics that are available before project approval.
1.2 All projects containing software shall evaluate themselves against this criteria (which is also
available at http://ivvcriteria.ivv.nasa.gov) to determine if a software IA or IV&V is required.
1.3 See paragraph P.2 for applicability of these criteria.
2. Risk Factors and Consequences
2.1 Software IV&V is intended to assist mitigating risk; hence, the decision to do software
IV&V should be risk based. NPG 7120.5 defines risk as the “combination of 1) the probability
(qualitative or quantitative) that a program or project will experience an undesired event such as
cost overrun, schedule slippage, safety mishap, or failure to achieve a needed technological
breakthrough; and 2) the consequences, impact, or severity of the undesired event were it to
occur.” The exact probability of occurrence and consequences of a given software failure cannot
be calculated early in the software lifecycle. However, there are realistically available metrics
that give good general approximations of the consequences as well as the likelihood of failures.
2.2 In general, the consequences of a software failure can be derived from the purpose of the
software: i.e., what does the software control; what do we depend on it to do. Paragraph 2.2.1
contains a list of factors that can be used to categorize software based on its intended function
and the level of effort expended to produce the software. Paragraph 2.2.2 defines the boundaries
of four levels of failure consequences based on the rating factors from paragraph 2.2.1.
2.2.1 Factors contributing to the consequences of software failure include the following:
a. Potential for loss of life. Is the software the primary means of controlling or monitoring
systems that have the potential to cause the death of an operator, crewmember, support
personnel, or bystander? The presence of manual overrides and failsafe devices is not to be
considered. This is considered a binary rating: responses must be either yes or no. Examples of
software with the potential for loss of life include:
(1) Flight and launch control software for human-rated systems.
(2) Software controlling life support functions.
(3) Software controlling hazardous materials with the potential for exposing humans to a lethal
dose.
1
(4) Software controlling mechanical equipment (including vehicles) that could cause death
through impact, crushing, or cutting.
(5) Any software that provides information to operators where an inaccuracy could result in
death through an incorrect decision (e.g., mission control room displays).
b. Potential for serious injury. Serious injury is here defined as loss of use of digit or limb, or
sight in one or both eyes, hearing, or exposure to substance or radiation that could result in long
term illness. This rating is also binary. This rating considers only those cases where the
software is the primary mechanism for controlling or monitoring the system. The presence of
manual overrides and failsafe devices is not to be considered. Examples of software with
potential for serious injury include software controlling milling or cutting equipment, class IV
lasers, or X-ray equipment.
c. Potential for catastrophic mission failure. Can a problem in the software result in a
catastrophic failure of the mission? This is a binary rating. Software controlling navigation,
communications, or other critical systems whose failure would result in loss of vehicle or total
inability to meet mission objectives would fall into this category.
d. Potential for partial mission failure. Can a problem in the software result in a failure to meet
some of the overall mission objectives? This is a binary rating. Examples of this category
include software controlling one of several data collection systems or software supporting a
given experiment, which is not the primary purpose of the mission.
e. Potential for loss of equipment. This is a measure of the cost (in dollars) of physical
resources that are placed at risk due to a software failure. Potential collateral damage is to be
included. This is exclusive of mission failure. Examples include the following:
(1) Loss of a $5 million unmanned drone due to flight control software failure. (Assuming the
drone is replaceable, this would not be a mission failure.)
(2) Damage to a wind tunnel drive shaft due to a sudden change in rotation speed.
f. Potential for waste of software resource investment. This is a measure or projection of the
effort (in work-years, civil service, contractor, and other) invested in the software. This shows
the level of effort that could potentially be wasted if the software does not meet requirements.
g. Potential for adverse publicity. This is a measure of the potential for negative political and
public image impacts stemming from a failure of the system as a result of software failure. The
unit of measure is the geographical or political level at which the failure will be common
knowledge, specifically: local (Center), Agency, national, international. The potential for
adverse publicity is evaluated based on the history of similar efforts.
h. Potential effect on routine operations. This is a measure of the potential to interrupt business.
There are two major components of this rating factor: scope and impact. Scope refers to who is
2
affected. The choices are Center and Agency. The choices for impact are inconvenience and
work stoppage. Examples include the following:
(1) A faulty firewall that failed to protect against a virus resulting in a 4-hour loss of e-mail
capabilities at a Center would be a “Center inconvenience.”
(2) Assuming that the old financial management software was no longer maintainable, the
failure of the replacement system to pass acceptance testing and the resulting 2-year delay would
be a potential “Agency work stoppage.” This does not imply that workarounds could not be
implemented, but only that it has the potential to stop work Agencywide.
2.2.2 Software Consequences of Failure Rating.
2.2.2.1 Consequences of failure are considered “Grave” when any of the following conditions
are met:
a. Potential for loss of life – Yes.
b. Potential for loss of equipment – Greater than $100,000,000.
c. Potential for waste of resource investment – Greater than 200 work-years on software.
d. Potential for adverse publicity – International.
2.2.2.2 Consequences of failure are considered “Substantial” when any of the following
conditions are met:
a. Potential for serious injury – Yes.
b. Potential for catastrophic mission failure – Yes.
c. Potential for loss of equipment – Greater than $20,000,000.
d. Potential for waste of resource investment – Greater than 100 work-years on software.
e. Potential for adverse publicity – National.
f. Potential effect on routine operations – Agency work stoppage.
2.2.2.3 Consequences of failure are considered “Significant” when any of the following
conditions are met:
a. Potential for partial mission failure – Yes.
b. Potential for loss of equipment – Greater than $1,000,000.
3
c. Potential for waste of resource investment – Greater than 20 work-years on software.
d. Potential for adverse publicity – Agency.
e. Potential effect on routine operations – Center work stoppage or Agency inconvenience.
2.2.2.4 Consequences of failure are considered “Insignificant” when all of the following
conditions are met:
a. Potential for loss of life – No.
b. Potential for serious injury – No.
c. Potential for catastrophic mission failure – No.
d. Potential for partial mission failure – No.
e. Potential for loss of equipment – Less than $1,000,000.
f. Potential for waste of resource investment – Less than 20 work-years on software.
g. Potential for adverse publicity – No more than local visibility.
h. Potential effect on routine operations – No more than a Center inconvenience.
2.3 The probability of failure for software is difficult to determine even late in the development
cycle. However, Table 1 contains simple metrics on the software, the developer, and the
development environment, which have proven to be indicators of future software problems.
While these indicators are not precise, they provide order of magnitude estimates that are
adequate for assessing the need for software IV&V. (The Facility and the NASA Software
Working Group will further refine these indicators and their associated weighting factors as more
data become available.)
4
Factors Un-weighted likelihood of failure score Weighting Likely-
contributing Factor hood of
to likelihood failure
of software rating
failure
1 2 4 8 16
Software Up to 5 people Up to 10 Up to 20 Up to 50 More than 50 X2
team at one location people at one people at one people at one people at one
complexity location location or 10 location or 20 location or 20
people with people with people with
external external external
support support support
Contractor None Contractor with Contractor with Contractor with X2
Support minor tasks major tasks major tasks
critical to
project
success
Organization One location Two locations Multiple Multiple Multiple X1
Complexity(1) but same locations but providers with providers with
reporting chain same reporting Prime/sub associate
chain relationship relationship
Schedule No deadline Deadline is Non-negotiable X2

Pressure(2) negotiable deadline
Process Independent Independent Independent CMM Level 1 CMM Level 1 X2

Maturity of assessment of assessment of assessment of with record of or equivalent
Software Capability CMM Level 3 CMM Level 2 repeated
Provider Maturity Model mission
(CMM) Level success
4, 5
Degree of Proven and Proven but Cutting edge X1
Innovation accepted new to the
development
organization
Level of Simple - Stand Extensive X2
Integration alone Integration
Required
Requirement Well defined Well defined Preliminary Changing, X2
Maturity objectives - No objectives - objectives ambiguous, or
unknowns Few unknowns untestable
objectives
Software Less than 50K Over 500K Over 1000K X2
Lines of
Code(3)
Total
Table 1 Likelihood of Failures Based on Software Environment
The following notes and definitions apply to Table 1:
(1) “Organization Complexity” is an indirect measure of the software developer’s potential

communications difficulties. A single organization working from multiple locations faces a
slightly greater difficulty than an organization in one location. When the software development
is accomplished by multiple organizations working for a single integrator, the development is
significantly complicated. If the developing organizations are coequal such as in an associate
contractor relationship (or a similar relationship between government entities) then there is no
integrator. Experience has shown this arrangement to be extremely difficult as no one is in
charge.
5
(2) Under “Schedule Pressure,” a deadline is negotiable if changing the deadline is possible,
although it may result in slightly increased cost, schedule delays, or negative publicity. A
deadline is non-negotiable if it is driven by an immovable event such as an upcoming launch
window.
(3) As the problems identified in software IV&V are often mismatches between the intended use
and the actual software built, “Software Lines of Code” shall include reused software and
autogenerated software.
3. Risk Assessment
3.1 Combining the software consequences of failure and the likelihood of failure rating from
Paragraph 2 yields a risk assessment that can be used to identify the need for software IV&V.
The indication of whether software IV&V is required is obtained by plotting in Figure 2 the
intersection of the Consequences of Software Failure determination and the Total Likelihood of
Failure determination. Application of these criteria simply determines that a project is a
candidate for software IV&V – not the level of software IV&V or the resources associated with
the software IV&V effort. These will be determined as a result of discussions between the
project and the Facility.
Figure 2 Software Risk
3.2 Figure 2 shows a dark region of high risk where software consequences, likelihood of
failure, or both are high. Projects having software that falls into this high-risk area shall undergo
software IV&V. The exception is those projects that have already done hardware/software
integration. A software IV&V would not be productive that late in the development cycle.
These projects shall undergo a Software Independent Assessment (IA). An IA is a review and
analysis of the project/program’s software development lifecycle and products. The IA differs in
6
scope from a full software IV&V program in that IV&V is applied over the lifecycle of the
system whereas an IA is usually a one-time review of the existing products and plans.
3.3 Figure 2 shows three gray regions of intermediate risk. Projects having software that falls
into these areas shall undergo a Software IA. The Facility shall conduct the Software IA. One
purpose of the Software IA is to ensure that the software development does not have project-
specific risk characteristics that would warrant the performance of software IV&V. Should such
characteristics be identified, the Facility will make a recommendation for software IV&V
performance.
7
Appendix IV. – ARTICLE: Quantifying Economic Impact of

Information Integrity Failures
Title: Quantifying Economic Impact of Information Integrity

Failures
Author: Salimol Thomas
Publisher: Information Integrity Coalition (IIC)
Date: 14 February 2003
URL: http://www.informationintegrity.org/downloads/
econImpact-executiveSummary.pdf
Date Accessed: 3 October 2006

Bolton ID: 0509033
Quantifying Economic Impact of Information Integrity Failures
Executive Summary
by
Salimol Thomas
2
Executive Summary
Information integrity failure causes economic loss as well as loss of intangible assets. It results in loss of
investment, loss of income, loss of productivity and liability costs. The Standish Group estimates that
defective software code accounted for 45 percent of computer system downtime and cost U.S. companies
$100 billion in lost productivity in 2000 [1]. Indirect effects of information system failures have included
diminution of brand value and even legal action by law enforcement agencies. Pervan G. P. and Phua R.
[2] extensively studied the executive information systems (EIS) used in Australia. They identified that
inadequate technology, unavailability of data, doubtful data, and complexities were the main factors
responsible for the failure of EIS.
This report is organized in two sections. Section 1 examines the various economic aspects of lack of
information integrity. Section 2 reports the factors responsible for failure of information systems (IS).
1.0 Lack of Information Integrity Cost

The following section describes various tangible and intangible losses an organization can incur
because of information integrity failures.
1.1 Loss of Investment

In the United States, billions of dollars are invested on information technology (IT) application
development. However, many of these projects fail due to poor design and deployment errors.
For example:
• In 1999, failure of NASA’s Mars Lander missions , result ed in a total loss of $360 million.
The cause of the failure was traced back to a software error [3, 4].
1240 E. Diehl Road, Suite 300, Naperville, IL 60563

Tel: 630-505-5525 Fax: 630-505-1812
Toll Free: 866-CALL IIC
www.informationintegrity.org
2/14/2003_v03
3
1.2 Loss of Income
Loss of Income can occur because of:
1. Missed opportunity due to lack of valid information. For example, the information
system predicts no growth for a particular product for a particular year, and then the
company won't be anticipating any increase in demand. However, if this information is
incorrect, the company will lose the additional revenue.
2. Information integrity failure of the business support system. Today's business

decisions and operations are largely dependent on information handled by information
systems. Any failure/lack of integrity at any point in the system could lead to significant
loss in business.
The following examples illustrate the above points:
• The failure to produce reliable software to handle baggage at the new Denver airport is
costing the city $1.1 million per day [5].
• Nike's third quarter sales fell after faulty supply chain software (i2) delayed orders and
caused excess inventories. Revenues for that quarter were off by more than $100 million
[11]
• Hershey Foods’ $115 million SAP computer system implementation had problems,
crippled shipments and was blamed for a 12 percent decrease in third quarter sales in 1999
[6]
• eBay’s business was halted for 22 hours due to a system crash, resulting in a $4 million
revenue loss and $5.7 billion in market capitalization [3].

Tel: 630-505-5525 Fax: 630-505-1812
2/14/2003_v03
4
1.2 Brand Image
In today's world the image of an organization largely depends on the IT infrastructure. Any
failure of these systems conveys a wrong signal to investors.
1. Organizations can lose market value because of a decrease in stock price.

2. Diminution of the intangible brand image
The following examples prove the above points:
• Egghead.com's reputation was shattered when hackers were able to steal roughly 3.7
million credit card numbers by exploiting a security flaw in the company's e-commerce
software servers. Egghead customers left in droves after being notified of the theft and
shares of the company's stock fell to as low as $0.50 after trading as high as $13.50 [10].
• In the days following announcement of software problems at Nike, the company's stock
dropped nearly 20 percent per share [14].
• H&R Block’s reputation was adversely affected when a software failure allowed online
clients to view other client's tax returns [3].
• Online music retailer CDUniverse suffered brand damage, liability costs and loss of
revenue when a hacker exploited a software security flaw and stole 300,000 credit card
numbers [3].
1.3 Loss of productivity

Lack of integrity in an organization’s information system may halt normal business operations
and reduce productivity.
• Input errors are typically around 2 percent, and in excess of 10 percent under certain
conditions, when a user interface is not designed properly. Spreadsheet errors are estimated

Tel: 630-505-5525 Fax: 630-505-1812
2/14/2003_v03
5
to be 5 percent. These errors not only reduce productivity, but also spread inaccurate
information [13].
• The Standish Group [1] estimates that defective software code accounted for 45 percent of
computer system downtime and cost U.S. companies $100 billion in lost productivity in
2000.
• The New York Stock Exchange (NYSE) came to a crashing halt for one hour on June 6,
2001 [7]. It was due to system error.
1.4 Liability Costs

Absence of integrity not only causes loss of revenue, but it also invites the wrath of unsatisfied
customers. For example:
• Customers of the popular tax-preparation Web site e1040.com were preparing to sue after
their private information — including passwords and Social Security numbers — was
displayed in plain text on the Internet due to a switching error in the site's encryption
software [8].
• FoxMeyer sued SAP and Andersen Consulting for $500 million each because of SAP's
software failure [3]
1.5 Federal Government Action

Inaccurate information invites the wrath of law enforcing agencies. For example:
• In August 3, 1999, Securities and Exchange Commission (SEC) filed 26 cases against 82
defendants and respondents across the country for engaging in fraudulent microcap
schemes from which they profited by over $12 million. [15]

Tel: 630-505-5525 Fax: 630-505-1812
2/14/2003_v03
6
2.0 Failure factors
Pervan G. P. and Phua R. (1996) [2] have extensively studied the executive information systems
(EIS) used in Australia and identified the following factors that were collectively the leading cause
of the EIS’s failure.
Typical Failure Factors

(Pervan G. P. and Phua R., 1996)
No. Failure Factors Rate
1 Inadequate/Inappropriate Technology 4.75
2 Loss of Interest by Executives 4.25
3 Unavailability of Data 4.25
Poorly Planned System Evolution Leading to Inability to Demonstrate
4 3.75
Breakthroughs
5 Doubtful Data Integrity 3.75
6 Difficulty in Reaching Operational Status Due to Technical Difficulties 3.75
7 Information Systems Unable to Adapt to Changing Requirements 3.50
7 Inability to Cost Justify IT Systems 3.25
8 System is Difficult to Use 3.25
9 Lack of Commitment/Support by Top Management 3.00
10 Organizational Resistance to Change/Technology 3.00
11 System Objectives not Linked to Business Strategy 2.75
12 Perception of System by Executives as Unimportant 2.75
13 Insufficient Depth of Information 2.75
14 Lack of Executive Sponsorship 2.50
15 Insufficient/Incompetent System Support Staff 2.50
16 Failure of System to Meet Executive Information Requirements 2.00

Tel: 630-505-5525 Fax: 630-505-1812
2/14/2003_v03
7
3.0 References
1. Ricadela, A., Gilbert, A., “The state of the software,” Informationweek, 42-54, May 21, 2001.
2. Pervan , G.P. and Phua, R., “Executive Information Systems in Australia: Current Status and
Some Historical Comparisons,” Proceedings Of The 29th Annual Hawaii International Conference
on System Sciences, 1996.
3. Miles, P., Bryant, B. “Cigital Estimates a Half Trillion Dollars Was Lost by Businesses Due to
Software Failure,” http://www.cigital.com/news/halftrillion.ht ml , Jan 24th 2001.
4. “Second Science Committee Hearing on NASA's Mars Failures,” The American Institute of
Physics Bulletin of Science Policy News, Number 81: July 11, 2000,
http://www.aip.org/enews/fyi/2000/fyi00.081.htm,
5. “The CHAOS Report,” Standish Group , 1994.

http://www.eee.bham.ac.ukdsvp_gr/roxby/ee4a3/Lecture2/sld010.htm
6. English, L. P., “Plain English on Data Quality: The Information Quality Revolution,” DM
Review, February 2000.
7. Yasin, R., “Software Bug Halts Trading On NYSE ,” Reuters , June 8, 2001,
http://www.internetweek.com/story/INW20010608S0002
8. Sandoval, G. “Tax site leaves customer data exposed,” CNET News.com, Feb. 12, 2001
http://news.com.com/2100-1017-252457.html
9. Ricadela , A. ,“The State Of Software :QUALITY,” Information Week, May 21, 2001
10. Rosencrance, L., “Bulletin: Hacker breaks Egghead's security shell,” Computerworld December
22, 2000, http://www.computerworld.com/cwi/story/0,1199,NAV47_STO55529,00.html
11. Karpinski, R., “Don't Get "Nike-ed ,” InternetWeek hosted by CMP Media’s TechWeb section,
March 7, 2001
12. Fonstad, J., “Supply-chain model giving way to real-time inventory management,” CNNmoney
May 24, 2001
http://money.cnn.com/2001/05/24/redherring/herring_supply/index.htm
Tel: 630-505-5525 Fax: 630-505-1812
2/14/2003_v03
8
13. 13. Panko R. R., “What we know about spreadsheet errors,” Journal of End User Computing,
Vol.10, No. 2, 1998, pp.15-21.
14. 14. Wilson, T., “Supply chain debacle,” InternetWeek hosted by CMP Media’s TechWeb section,
March 1, 2001, http://www.internetweek.com/story/INW20010301S0002
15. “About Microcap Fraud,” Securities and Exchange Commission, August 3 1999,
http://www.sec.gov/divisions/enforce/microcap.htm,.

Tel: 630-505-5525 Fax: 630-505-1812
2/14/2003_v03
Bibliography
Dave Zubrow. (2004) Measuring Software Product Quality: the ISO 25000 Series
and CMMI [Online]. Available from < http://www.sei.cmu.edu/sema/presentations/
zubrow/esepg/esepg.pdf > [Accessed 25 September 2006].
J. C. Laprie. (1992) Dependability: Basic Concept & Terminology [Online].

Available from < http://srel.ee.duke.edu/sw_ft/node3.html > [Accessed 14
September 2006].
Mark Kasunic. (2006) 2006 State of Software Measurement Practice Survey

[Online]. p.37. Available from < http://www.sei.cmu.edu/sema/presentations/
stateof-survey.pdf > [Accessed 25 September 2006].
NASA. (n.d.) Software Independent Verification and Validation (IV&V)

/Independent Assessment (IA) Criteria [Online]. Available from
< http://ivvcriteria.ivv.nasa.gov/Documents/ivvcrit.pdf > [Accessed 3 October
2006].
Ops A La Carte. (2006) Root Cause Analysis [Online]. p.1. Available from
< http://www.opsalacarte.com/pdfs/courses/OALC_Root_Cause_Analysis_
Seminar.pdf > [Accessed 20 September 2006].
Robert N. Charette. (2005) Why Software Fails [Online]. Available from <
http://www.spectrum.ieee.org/sep05/1685 > [Accessed 14 September 2006]
Salimol Thomas. (2003) Quantifying Economic Impact of Information Integrity

Failures Executive Summary [Online]. Available from < http://www.
informationintegrity.org/downloads/econImpact-executiveSummary.pdf >
[Accessed 3 October 2006].

Bolton ID: 0509033
Software Engineering Institute. (2005) CMMI Performance Results [Online].

Available from < http://www.sei.cmu.edu/cmmi/results.html > [Accessed 15
September 2006].
The University of Bolton. (n.d.) Using Information [Online]. Available from

< http://www.bolton.ac.uk/bissto/infoskills/useinfo/index.htm > [Accessed 20
October 2006]
VA Software. (2005) VA Software Uncovers Impact of "Development Disconnect"

[Online]. Available from < http://www.vasoftware.com/news/press.php/2005/
1481.html > [Accessed 4 October 2006]
Wikipedia. (n.d.) Capability Maturity Model [Online]. Available from

< http://en.wikipedia.org/wiki/Capability_Maturity_Model > [Accessed 15
September 2006]
Wikipedia. (n.d.) Rational Unified Process [Online]. Available from

< http://en.wikipedia.org/wiki/RUP > [Accessed 15 September 2006]
Wikipedia. (n.d.) Software Factory [Online]. Available from

< http://en.wikipedia.org/wiki/Software_factory > [Accessed 15 September 2006]
Wikipedia. (n.d.) Unified Modeling Language [Online]. Available from:

http://en.wikipedia.org/wiki/Unified_Modeling_Language [Accessed 15
September 2006]

Bolton ID: 0509033
Reference List
J. C. Laprie. (1992) Dependability: Basic Concept & Terminology [Online].
Available from < http://srel.ee.duke.edu/sw_ft/node3.html > [Accessed 14
September 2006].
Ops A La Carte. (2006) Root Cause Analysis [Online]. p.1. Available from
< http://www.opsalacarte.com/pdfs/courses/OALC_Root_Cause_Analysis_
Seminar.pdf > [Accessed 20 September 2006].
Software Engineering Institute. (2005) CMMI Performance Results [Online].

Available from < http://www.sei.cmu.edu/cmmi/results.html > [Accessed 15
September 2006].
Dave Zubrow. (2004) Measuring Software Product Quality: the ISO 25000 Series
and CMMI [Online]. Available from < http://www.sei.cmu.edu/sema/presentations/
zubrow/esepg/esepg.pdf > [Accessed 25 September 2006].
Mark Kasunic. (2006) 2006 State of Software Measurement Practice Survey

[Online]. p.37. Available from < http://www.sei.cmu.edu/sema/presentations/
stateof-survey.pdf > [Accessed 25 September 2006].
NASA. (n.d.) Software Independent Verification and Validation (IV&V)

/Independent Assessment (IA) Criteria [Online]. Available from
< http://ivvcriteria.ivv.nasa.gov/Documents/ivvcrit.pdf > [Accessed 3 October
2006].
Salimol Thomas. (2003) Quantifying Economic Impact of Information Integrity

Failures Executive Summary [Online]. Available from < http://www.
informationintegrity.org/downloads/econImpact-executiveSummary.pdf >
[Accessed 3 October 2006].

Bolton ID: 0509033

Software Failures Report 2

Diunggah oleh

Informasi Dokumen

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Software Failures Report 2

Diunggah oleh

Hak Cipta:

Format Tersedia

BSc.

software and the consequences of this?

by Ooi Teong Kuan

Prepared by: Ooi Teong Kuan

To improve the software quality, identifying root causes of the problems is

Consequences of software failures will lead to the following three impacts:

As a conclusion, it is wiser to invest more on software quality improvement plans

Prepared by: Ooi Teong Kuan

1.0 TERMS OF REFERENCE...................................................................................................1

2.0 INTRODUCTION .................................................................................................................2

3.0 WHAT IS “SOFTWARE FAILURE”? ..................................................................................3

3.1 DEFINITION OF SOFTWARE FAILURE ....................................................................................3

4.0 HOW CAN WE IMPROVE THE SOFTWARE QUALITY? ................................................5

4.1 SOFTWARE QUALITY IMPROVEMENT LIFECYCLE...................................................................5

5.0 WHAT ARE THE CONSEQUENCES?............................................................................14

5.1 CONSUMER IMPACT .........................................................................................................15

6.0 CONCLUSIONS ................................................................................................................18

7.0 RECOMMENDATIONS .....................................................................................................19

Prepared by: Ooi Teong Kuan

APPENDIX I. – ARTICLE: CAPABILITY MATURITY MODEL ................................................................21

Prepared by: Ooi Teong Kuan

Prepared by: Ooi Teong Kuan

1.0 Terms of Reference

Prepared by: Ooi Teong Kuan 1

Prepared by: Ooi Teong Kuan 2

3.0 What is “Software Failure”?

3.1 Definition of Software Failure

Not all software problems can be classified as ‘software failure’. According to J. C.

3.2 Examples of Software Failures

• Unreasonable crashes or hangs of the system

Prepared by: Ooi Teong Kuan 3

Prepared by: Ooi Teong Kuan 4

4.0 How can we improve the software quality?

4.1 Software Quality Improvement Lifecycle

Software Quality Improvement Lifecycle

Figure 1 – Software Quality Improvement Lifecycle

Prepared by: Ooi Teong Kuan 5

4.2 Identifying the Root Causes

4.3 Taking Appropriate Actions and Solutions

Prepared by: Ooi Teong Kuan 6

Requirements Evaluation & Deployment &

The sections below explain each of these approaches in more detail.

4.3.1 Capability Maturity Model Integration (CMMI)

CMMI is a process improvement approach developed by Software Engineering

Maturity Level 1 – Initial

Prepared by: Ooi Teong Kuan 7

In order to show the effectiveness of their process improvement approach, SEI

Performance Category Median Number of Data Points Low High

Cost 20% 21 3% 87%

Schedule 37% 19 2% 90%

Productivity 62% 17 9% 255%

Quality 50% 20 7% 132%

Customer Satisfaction 14% 6 -4% 55%

Return on Investment 4.7 : 1 16 2 : 1 27.7 : 1

Table 2 – CMMI Performance Results as of 15 December 2005

4.3.2 Unified Modeling Language (UML)

Prepared by: Ooi Teong Kuan 8

4.3.3 Rational Unified Process (RUP)

RUP is quite similar to UML because it is in fact an extension of UML. As

4.3.4 Software Factory

4.4 Monitoring and Evaluating the Results

Prepared by: Ooi Teong Kuan 9

1. Software Quality Assurance Test

4.4.1 Software Quality Assurance Test

1987: SEI-87-TR-24 (SW-CMM questionnaire), released.

Capability Maturity Model Integration