From 2006 CSTE

Common Body of Knowledge

Study Session Outline Sept. 9, 2006

John Peterson
Vocabulary

Why Do We Test Software?

The Multiple Roles of the Software Tester

Life Cycle Testing

Quality Assurance versus Quality Control

– Quality control is concerned with a specific

product.

– Quality assurance helps establish processes.

– Quality assurance is sometimes called quality

control over quality control because it
evaluates whether quality control is working.
 The Cost of Quality

Cost of Quality is a term used to quantify the total cost of prevention and
appraisal, and costs associated with the production of software.

Prevention Costs
- Up-front costs for benefits that will be derived later
- Establishing procedures, training, tools and planning
- Spent before the product is actually built

Appraisal Costs
- Review completed products against requirements
- Includes the cost of inspections, testing, and reviews.
- After the product is built but before it is shipped to the user

Failure Costs
- Defects that make it to the user or to production.
- Repairing products to make them meet requirements.
- Cost of operating faulty products and operating a Help Desk.

The majority of costs associated with the Cost of Quality are associated with
the identification and correction of defects.
 Software Quality Factors

The software quality factors are attributes of the software that, if they are
wanted and not present, pose a risk to the success of the software..

Identifying the software quality factors is done by surveying the decision makers.
Their answers prioritize testing.

Software Quality Factor Examples

Correctness
Authorization
File Integrity
Audit Trail
Continuity of Processing
Service Levels
 How Quality is Defined

Quality is frequently defined as meeting the customer's requirements the first

time and every time.

Quality is also defined as conformance to a set of customer requirements

that, if met, result in a product that is fit for its intended use.

The common thread that runs through today's quality improvement efforts is
the focus on the customer and, more importantly, customer satisfaction.
Why Do We Test Software?
Developers are unable to build defect-free software.
 Developers are not Good Testers

The disadvantages of a person checking their own work using their own
documentation are as follows:

Misunderstandings will not be detected, because the checker will assume that
what the other individual heard from him was correct.

Improper use of the development process may not be detected because the
individual may not understand the process.

The individual may be “blinded” into accepting erroneous system specifications

and coding because he falls into the same trap during testing that led to the
introduction of the defect in the first place.

Information services people are optimistic in their ability to do defect-free work

and thus sometimes underestimate the need for extensive testing.

Without a formal division between development and test, an individual may be

tempted to improve the system structure and documentation, rather than allocate
that time and effort to the test.
 What is a Defect?

A defect is an undesirable state. In order to know what a defect is we must first

define a desirable state.
What is Quality Software?

IT evaluates software against requirements.

User’s of software view of quality software means fit for use.

Just because software meets IT requirements doesn’t mean a user will

find it fit for use.

When software doesn’t meet requirements, it is an IT gap

When software isn’t fit from the user’s perspective, it is a user gap.

To close the user’s gap, the IT quality function must understand user needs
with customer surveys and customer testing.

Testers need to ensure that software not only meets the requirements, but is in
fact usable by the customer.
 Why Does a Development Process Produce Defects?

Ideally, the software development process should produce the same results each
time the process is executed.

The example..
If a process that produces one function-point-of-logic in 100 person hours the
first time, that process should produce one function-point-of-logic in 100
hours when ran again. If the process takes 110 hours to produce one function-
point-of-logic, then there is “variability” in the software development process.

Can anyone explain what a “function-point-of-logic” is?

Variability is the “enemy” of quality – maturing a software development process

should reduce variability.
The concept of measuring and reducing variability is commonly called
statistical process control (SPC).
Variation is measured with statistics like standard deviation.
Standard deviation is basically how spread out the values in a data set are.
Examples of Typical Sources of Variation
Measurement Components Material Components
Counting Forms
Sampling Suppliers

Machine Components Method Components

Office equipment Procedures
Computers Policies
Software

People Components Environment Components

Training Temperature
Experience Humidity
Attitude Noise Level
Aptitude Lighting

What are some things that could be varying in figures 5 and 6?

(What occurrence might have been counted and plotted?)
Special causes of variation are not typically present in the process. They occur
because of special or unique circumstances. If special causes of variation
exist, the process is unstable or unpredictable. Special causes must be
eliminated to bring a process into a state of statistical control. A state of
statistical control is established when all special causes of variation have been
eliminated.

Someone please help with this. I don’t think this really says to ignore special
causes so the statistics look good.

The book doesn’t give examples of special causes of variation.

Can anyone provide examples?
I think that if we some answers to “variation in what?” by this point, this
section will be more understandable.

Bringing a process into a state of statistical control is not really improving the
process; it is just bringing it back to its typical operation. Reducing variation
due to common causes is process improvement and the real essence of
continuous process improvement.

As previously mentioned, variation due to special causes must be identified

and removed to create a stable process. However, a stable process may not
be an acceptable process. If its variation, due to common causes results in
operation of the process beyond specifications, the process is called
“incapable.”

The process can be taken from incapable to capable by reducing variation

due to common causes, retargeting the process, or both.
Software testing does not add a lot of value to the
business if all they are doing is validating that
incorrect requirements are implemented correctly.
 Reducing the Frequency of Defects in Software Development

Department of Defense (DOD) had Carnegie Mellon University’s Software

Engineering Institute (SEI) study what makes software development more
effective. The result was the capability maturity model.

The Five Levels of Maturity

As the model moves from Level 1 to Level 5, the variability in the process is
significantly reduced. Thus, those at Level 5 have minimal variability in their
software development process, while Level 1 organizations have significant
variability.

The cost differences to produce a function point of logic between a Level 1

and Level 5 organization may vary by 100 times. In other words, what a
Level 1 organization may spend on building software, for example $1,000,
may only cost $10 for a Level 5 organization.
 Level 1 – Ad Hoc
Ad hoc means unstructured, inconsistent levels of performance. At the ad hoc
level, tasks are not performed the same way by different people or different
groups.

Level 2 – Control
There are two major objectives to be achieved at Level 2. The first is to instill
discipline in the culture of the information organization so that through the
infrastructure, training, and leadership of management individuals will want to
follow defined processes. The second objective is to reduce variability in the
processes by defining them to a level that permits relatively constant outputs.

Level 3 – Core Competency

At this level, an information organization defines its core competencies and
then builds an organization that is capable of performing those core
competencies effectively and efficiently.
 Level 4 – Predictable
This level has two objectives. The first is to develop quantitative standards
for the work processes based on performance of the Level 3 stabilized
processes. The second objective is to provide management the dashboards
and skill sets needed to manage quantitatively. The result is
predictable work processes.

Level 5 – Innovative

At Level 5, the information organization wants to be a true leader in the

industry. At this level, the organization is looking to measure itself against the
industry through benchmarking, and then define innovative ways to achieve
higher levels of performance. Innovative approaches can be achieved through
benchmarking other industries, applying new technology in an innovative way,
reengineering the way work is done, and by constantly studying the literature
and using experts to identify potential innovations. This level is one in which
continuous learning occurs, both in individuals and the organization.
 The Multiple Roles of the Software Tester

The major factors that affect the tester’s role are:

People relationships
Scope of testing
When should testing occur
How should testing be performed
Testing constraints
 People Relationships

Organizations are finally becoming convinced that testers truly can not test in
quality at the end of the development process and that the traditional waterfall
methodology has lead to many of the issues surrounding testing today.

I’m rusty on the waterfall methodology. Anyone want to explain this statement?

Some attitudes that have shaped a negative view of testing and testers are:
Testers hold up implementation.
Giving testers less time to test will reduce the chance that they will find defects.
Letting the testers find problems is an appropriate way to debug.
Defects found in production are the fault of the testers.
Testers do not need training; only programmers need training.

Essential testing skills include test planning, using test tools (automated and
manual), executing tests, managing defects, risk analysis, test measurement,
designing a test environment, and designing effective test cases.
 People Relationships

Top ten people challenges:

Training in testing
Relationship building with developers
Using tools
Getting managers to understand testing
Communicating with users about testing
Making the necessary time for testing
Testing “over the wall” software
Trying to hit a moving target
Fighting a lose-lose situation
Having to say “no”

Any thoughts about the red items?

 Scope of Testing

The scope of testing is the extensiveness of the test process. A narrow scope
may be limited to determining whether or not the software specifications were
correctly implemented. The scope broadens as more responsibilities are
assigned to software testers.

Among the broader scope of software testing are these responsibilities:

1. Determining if the user’s needs have been met regardless of specs
2. Finding defects early when they cost less
3. Removing defects prior to going into a production
4. Identifying weaknesses in development process to improve process and
make the development process more mature.

In defining the scope of software testing each IT organization must answer

the question, “Why are we testing?”
 When Should Testing Occur?

When testing is constrained to a single phase and confined to the later stages
of development, severe consequences can develop.

An error discovered in the latter parts of the life cycle must be paid for four
different times.
The first cost is developing the program erroneously, which may include writing
the wrong specifications, coding the system wrong, and documenting the
system improperly.
Second, the system must be tested to detect the error.
Third, the wrong specifications and coding must be removed and the proper
specifications, coding, and documentation added.
Fourth, the system must be retested to determine that it is now correct.

Verification must not be isolated to a single phase in the development process,

but rather, incorporated into each phase of development.
 When Should Testing Occur?

The recommended test process involves testing in every phase of the life cycle.

During the requirements phase, the emphasis is upon validation to determine

that the defined requirements meet the needs of the organization.

During the design and program phases, the emphasis is on verification to

ensure that the design and programs accomplish the defined requirements.

During the test and installation phases, the emphasis is on inspection to

determine that the implemented system meets the system specification.

During the maintenance phases, the system will be retested to determine that
the changes work and that the unchanged portion continues to work.
 When Should Testing Occur?

Requirements

The adequacy of the requirements must be thoroughly analyzed and initial test
cases generated with the expected behavior

Generating these tests and the expected behavior of the system clarifies the
requirements and helps guarantee that they are testable.

Design
Test plan is produced.
Test schedule with observable milestones is constructed

Validation support tools should be acquired or developed

Perform design inspections using design walkthroughs.

 When Should Testing Occur?

Program (Build/Construction)

Actual testing occurs during the construction stage of development.

As opposed to some kind of non-actual testing?

Test Process

Test sets, test results, and test reports should be catalogued and stored in
a database.

Installation
Placing tested programs into production is an important phase normally
executed within a narrow time span.
Testing during this phase must ensure that the correct versions of the
program are placed into production.
 When Should Testing Occur?

Maintenance

As the system is used, it is modified either to correct errors or to augment

the original system. After each modification the system must be retested.
Such retesting activity is termed regression testing.

If test cases from initial development have been catalogued and preserved,
duplication of effort will be minimized.
 How the Test Plan Should be Developed

The applicable test objectives would be listed and ranked, and the phases of
development would be listed as the phases in which testing must occur.

Four steps must be followed to develop a customized test plan:

1. Select and rank test objectives.

Customers or key users of the system in conjunction with the test team should
define and rank the test objectives
2. Identify the system development phases.

3. Identify the business risks

4. Place risks in the matrix

The risk team should determine the test phase in which the risk needs to be
addressed by the test team and the test objective to which the risk is
associated.
 Testing Constraints

Budget and Schedule Constraints

May limit the ability complete the test plan

Costs dramatically increase the later a defect is found

The risk of under testing is directly translated into system defects present in the
production environment.

The risk of overtesting is the unnecessary use of valuable resources

At some point, an over test condition begins and the cost of

testing to uncover defects exceeds the losses from those defects.

Most problems associated with testing occur from one of the following
causes:
Failing to define testing objectives
Testing at the wrong phase in the life cycle
Using ineffective test techniques
 Testing Constraints
Incomplete Statement of Work
If requirements are lacking or poorly written, then the test team must have a
defined method for uncovering and defining test objectives.

Changes in Technology
Integration
Technology is being more closely integrated into day-to-day business
System Chains
Problems in one can cascade into and affect others.
The Domino Effect
One problem condition, can cause hundreds or even thousands of similar
errors within a few minutes.
Reliance on Electronic Evidence
The validity of transactions is dependent upon controls, and thus a control
error may result in extensive losses.
Multiple Users
Systems belong to multiple users, making it difficult to identify a single
organizational unit responsible for a system.
 Testing Constraints

Limited Tester Skills

Testers should be competent in all ten knowledge categories to be effective.
Does anyone know the list of ten skills?
Software Risks
One of the most effective methods to reduce computer system strategic risk is
testing.

Risks deemed important to reduce become the areas for testing.

Software Design Defects

Designing software with incomplete or erroneous decision-making criteria

Failing to program the software as intended

Omitting needed edit checks for determining completeness of output data.

Data Defects

Poor quality data can adversely affect the computer-directed actions.

Finding Defects

Testing focuses on discovering and eliminating defects or variances

from what is expected.

Testers need to identify these two types of defects:

Variance from Specifications

A defect from the perspective of the builder of the product.

Variance from what is Desired

A defect from a user (or customer) perspective.
Why Are Defects Hard To Find?

There are at least two reasons defects go undetected:

Not Looking
Looking, But Not Seeing
 Life Cycle Testing

Life cycle testing involves continuous testing of the solution even after software
plans are complete and the tested system is implemented.

Testing should not use the same methodology that was used for developing
the system.

Testing does not stop once you’ve completely implemented your system; it
must continue until you replace or update it again!