Anda di halaman 1dari 33

Quality Assurance and Testing

Review
1
2
Benefits of quality
User: meets needs and expectations
Measurable attributes (for example, response time)
Fitness for use = does the job
Trust in good manufacturing processes
Value for money; return on investment
Transcendent feelings (latest technology, fun to use)
Vendor: stay competitive and survive
Reduce costs
Speed response to the market
Grow customer bases and market share
Increase profit
Improve employee motivation
Foster a productive, creative, happy environment
Stock owner, investor
Profits
Society, employees
The good side of progress
Good working environment with scope for personal growth



Quality and Process
Quality is about doing things better, with less
cost and more enjoyment
It provides business opportunities and is
essential to developer competition and survival
3
Process
Product
- Goods
- Services
- Software
Quality helps bring success
Goal: survival of the software vendor
Revenue: customer satisfaction and loyalty
Cost: effective and efficient development
QA Message
Good processes yield good products

What makes development successful?
Product delivered on time
Development costs within budget
Product meets requirements and fit for use






5 5
Quality assurance and control
Quality Assurance PROCESS
Discipline to maximizes chances software products
conform to requirements, as perceived by the users
Defining and documenting procedures and standards
Ensuring they are followed
Monitoring success and seeking to improve
Quality Control PRODUCT
Build quality into the software througout the software
development life cycle
Reviewing documents and testing code
Managing defects
Gathering metrics


6 6
Assessing maturity with CMMI
CMMI is a process improvement approach
to:
Guide process improvement across a project, a
division, or an entire organization
Help integrate traditionally separate
organizational functions
Set process improvement goals and priorities
Provide guidance for quality processes
Provide a point of reference for appraising
current processes
Must to be implemented at corporate level
Management initiative, from the top down
7
CMM software maturity levels
F
i
v
e

d
e
f
i
n
e
d

L
e
v
e
l
s

Ad hoc and chaotic process. A projects
success depends upon heroes and luck.
Project level thinking. Reactive.
Similar projects can repeat past
success.
Organization level thinking.
Proactive. Documented and
standardized.
Controlled Process. Detailed measures
and understanding of process and project
quality.
1. Initial
2. Repeatable
3. Defined
4. Quantitatively
Managed
5. Optimizing
Continuous process improvement through
quantitative feedback and new approaches.
ISO 9001 certification
Seven steps to implement and maintain a quality
management system (QMS)
1. Engage top management
Quality policy and organizational objectives come from the top
2. Identify key processes and interactions to meet quality objectives
3. Use process management techniques to implement QMS
4. Perform gap analysis between QMS and ISO 9001 requirements
and then add required activities, procedures and controls
5. Implement the system, train staff, verify effective operation
6. Manage the QMS
Focus on customer satisfaction
Monitor operation and strive for continual improvement
7. Seek third party certification/registration or issue self-declaration
of conformity

8
Software Quality Characteristics
ISO/IEC 9126:2001
Functionality
Suitability, Accuracy, Interoperability, Compliance, Security
Reliability
Maturity, Recoverability, Fault tolerance
Usability
Learnability, Understandability, Operability
Efficiency
Resource efficiency, time efficiency
Maintainability
Stability, analysability, Changeability, Testability
Portability
Installability, Replaceabiltiy, Adaptability, Conformance

9
10
Recall the general testing principles
1. Testing shows the presence of defects
The primary purpose of testing is to find defects
2. Exhaustive testing is impossible
Except in trivial cases 100% coverage is impossible
3. Early testing
Start early and focus on defined objectives
4. Defect clustering
Focus effort where early testing show weakness / manage risk
5. Pesticide paradox
Repeating the same tests tends to finding zero defects
Constantly vary test data, update scripts
6. Testing is context dependent
Load test Web, try to break security, test extreme conditions
7. Absence of errors fallacy
No defects does not guarantee product is fit for use
11
V-model concepts of testing
Validation
The process of evaluating a system or component during or at the end
of the development process to determine whether it satisfies specified
requirements



Verification
The process of evaluating a system or component to determine
whether the products of a given development phase satisfy the
conditions imposed at the start of that phase
Does the product do the right things?
Fitness for use
What the users want and need
Did developers build the product right?
Correctness
Built to design, assuming design fulfils requirements
V Model
12
Introduce early
testing into the
waterfall model
Recalling the 4 levels of test
The V-model introduced levels:
1. Unit / Component
Class, package, Web asset, module
2. Integration
Package, module, collections of assets, module
Vertical or horizontal slices through application
3. System
Verification on mirror of production environment
4. Acceptance
Validation
13
Application Life cycle Management
Agile Life Cycle management
Modern ALM replaces waterfall with:
Increments add functionality
New use cases
Additional capability
More slices of the application
Iterations refine existing code
Improve on code from previous or current increment
Refactoring
Each increment may include several iterations
Agile manifesto sets out a philosophy
Implementations such as DSDM, SCRUM, RUP,
provide workable approaches
14
Three Agile implementations
COMP 311
15
Dynamic Systems
Development Method
(DSDM)
Extreme Programming
(XP)
SCRUM
An outgrowth of, and
extension to, rapid
application development
(RAD)
Boasts best-supported
training/ documentation of
any Agile method.
DSDMs principles include:
Active user involvement
frequent delivery
Team decision making
Integrated testing
throughout SDLC
Reversible changes in
development.
Values of community,
simplicity, feedback, and
courage.
Emphasis on cost of
change and technical
excellence through
refactoring and test-first
development.
A proven system of
dynamic practices, with
integrity as a holistic unit:
Pair programmming
Daily cycles
Direct involvement of
customer
Provides a project
management framework
Cycles are 2-4 week
Sprints drvien by specified
Backlog features.
Team lead is Scrum
master.
Daily 15-minute team
meetings for coordination
and integration.
Scrum has been in use for
nearly ten years may now
be most widely
usuccessfully used Agile
practice.
Agile methodologies involve
Feature-driven development: business
stories scenarios to define functionality
Involving users
all the time
Continuous development
with shared code
ownership
Test-driven development
writing tests first
Frequent integration

16
Tests used throughout SDLC
Functional quality characteristics
Functional testing includes
Suitability, Interoperability, Security, Accuracy, Compliance
Non-functional quality characteristics
Non functional testing includes
Performance, load, stress testing
Structure or architecture
Measuring thoroughness code coverage
Testing related to changes
Confirmation testing to verify success of corrective
actions
Regression testing to ensure changes introduce no new
defects
17
I
S
O
/
I
E
C

9
1
2
6

Software configuration management
Version control is task of tracking and controlling changes in
the software Team collaboration
Revision control
Establishment of baselines
A place to store artifacts repository
Allow developers to check in, check out quickly and easily
Most SCMs support optimistic locking with merge to resolve conflicts
Most SCMs support branching and merging of branches
A historical record of changes over time
Old versions are never removed
Most SCMs store changes, not full versions
Essential for development, the test process, defect tracking, frequent
incremental builds,
Distributed Version Control Systems (DVCS) support distributed
repositories


18
# P22
# P22
# P22
Defect life cycle
Test Incident Fix Build
19
New
Version
Built
System
TESTING
Designer/
Programmer



Fixed
Fault
NEW
Software
Problem
Report
#P22
SPR # P22
Details of fix
- why
- where
How long
Insert new
Code fix
New: A164
Testing A163
SCM
System build concept
A time for delivery of system components is agreed
At regular intervals
Start or end of day
Overnight can take advantage of time zones
A new version of a system is built from a configuration of
components, for:
New baseline for developers of the product
Used also by developers of dependent software
End of phase or iteration in incremental development
Configuration to test
Release of product
Code freeze
no more development, just text and fix
Continuous Integration is a new approach supported
by some enhanced SCM tools that can continually feed
released source into a build stream
20
Defect Management
Defects reports maintained in an SCM
Reported
Rejected
Opened
Assigned
Fixed
Deferred
Reopened
Closed
R
e
w
r
i
t
t
e
n

B
a
d


r
e
p
o
r
t

V
e
r
i
f
i
e
d

A
c
c
e
p
t
e
d

22
Testing consists of the following
main activities
1. Planning and control
When goals of test are defined
2. Analysis and Design
When test cases are specified
3. Implementation and execution
When test suites are run
4. Evaluating exit criteria and reporting
When test results are analyzed
5. Test closure activities
When test artifacts are archived

Planning &
Control
Analysis &
Design
Implementation
& Execution
Evaluation
& Reporting
Test closure
Reviewing the test process
Summary of test process
23
Planning &
Control
Analysis &
Design
Implementation
& Execution
Evaluation
& Reporting
Test closure
Entry
criteria
Exit
criteria
24
Test Design Activities
For each goal/objective from
planning stage, define:
Test conditions
Refine generic goals or objectives
Test Cases
Add specific data
precondition + action postcondition
Test suites / scripts
Combine sets of tests for automatic
execution

Test procedures that testers follow
Should ensure that high level goals are
tested

Test
Conditions
Test
Cases
Test
Procedures
Test Case Specification - overview
Be explicit and detailed
Dont hesitate to state the obvious
Should be explicit enough for outsourcing implementation
and performance of test to a third world country
Often restate requirements
What objective or business rule is being tested
criteria for passing the test
Always
Define data to set up test preconditions
State what the tester does action
State the expected result postcondition


25
Every company has its own format
ISO/IEEE 829 specification has standard templates
Test Case Specification Format used in this course
Identifier:
Uniquely identify test case with a number or headline-style title
Criteria:
Paraphrase or state the full business rule or requirement being tested
A bank account owner can withdraw up to the balance of the account. The system records
every transaction and applies a service charge of $1.50 per transaction for all acounts with a
balance of less than $5000.
Precondition:
All data and with specific values to set up before running the test
Martha Pennas has account 6513-5913 with a balance of $4250
Action:
External activity that causes software system to perform some functionality
Martha Withdraws $300
Post condition or expected result:
Result of performing the requested functionalitiy, often update state of the system
Account 6513-5913 has balance of $3948.50 and a transcation record is logged.

Writing the Test Case Specification
26
27
Overview of testing techniques
Dynamic
Peer/SME
Reviews
Walkthroughs
Static Analysis
Data Flow
Control Flow
Formal
Inspection
Specification
based
Equivalence
Partitioning
Boundary
Value Analysis
Decision Tables
State transition
Use case testing
Experience based
Structure based
Exploratory
testing
Error
guessing
Architecture
Statements
Paths
Black box
White box
Static
Non-code
artifacts
Black and white box testing
Black box
Dynamic tests that do not involve looking at source
To check the outcome of using the system to
perform externally triggered actions
Performing steps in use cases/scenarios
Verify result for different test data

White box
Reviews and static test look at code
Standards, readability, cyclomatic complexity

Reviews look at all documents
28
private Seat assignSeat(SeatingClass sClass) {
int seatNumber;
// seat assignment from random number generator
seatNumber = seatFinder.nextInt(sClass.getNumSeats()) +
sClass.getIndexFirstSeat();
ArrayList<Seat> seats = plan.getSeats();
// random numbers may issue same seat twice.
// if that happens take first available seat in section
Seat seat = seats.get(seatNumber);
if (seat.getTicket() != null) {
seat = findFirstEmptySeat(seats, sClass);



29
Static White-box testing: Reviews
Technical reviews
Informal
Getting feedback from a SME
Peer review within the team
Walkthrough
Guided tour through documents
Demonstration of proposed system or prototype
Inspection
Formal review to find and fix defects
Has its own process
30
White box summary

White box tests of code
Analysis of structure for code coverage
Goal to test every path through code
Cyclomatic complexity = number of paths through a method
Data flow for state transition
Creation/Destruction/state of objects = values of fields
Static analysis for coding standards and error-prone habits
White box code analysis can help dynamic black-box test design
Static tests of other documents
Five types of reviews
Informal: peer reviews or technical reviews by an SME
Walkthroughs: such as presentation by evangelist
Inspections: formal process to identify defects and follow-up
Audits
Management reviews



31
Black box test design
Intuitive
Specification based
Use case testing
Techniques to improve coverage
Equivalence partitioning
Boundary value analysis
Decision tables
State transition
more not covered on this course
Experience based testing
Informal but effective compliment to the above
Useful when test cases hard to design security



Test metrics: coverage
Coverage is a measure of how much of the
component/system is exercised
Often expressed as percentage (%) of code tested
How do you measure coverage of dynamic tests?



32
Test Coverage = * 100%
Number of tests cases run
Total number of test cases
Success rate = * 100%
Number of tests passed
Number of tests cases run

Effectiveness = * 100%
Defects found by test
Total number of Defects

Total defects from all sources: test, customers, developers, maintenance,
History of QA
Early contributors to QA principles in the USA from 1920
Adoption of QA leads to success of Japanese products after WW II
American manufacturing adopts the Total Quality Movement from 1970s
QA in software and software services has additional challenges
ISO certification (ISO 9001 for software and services) in 1990s
QA gurus: names to remember
Walter Shewhart: USA 1930s
Statistical quality control (sampling); originator of the PDSC/PDCA cycle
W. Edwards Deming: father of Quality Assurance
Took QA message to Japan after World War II (1950s-1970s)
Promoted quality as a driver of business success
Joseph M. Juran: also in Japan after World War II
Qualtiy as a management concern, no just technicolgy
Wrote Quality Control Handbook; forerunner of Six Sigma and Lean methodologies
Phil Crosby: management consultant 1970s and 1980s
Quality as conformance to requirements or absense of defects
Author of Quality is free (1979) and a great source of quotations
33