Software Quality Assurance

Software Testing

Lecture 7
International Islamic University Islamabad
By Waqar Hussain
Software Testing: Definitions
 Software Testing is the process of
executing a program or system with
the intent of finding errors. [Myers79]

 Or, it involves any activity aimed at

evaluating an attribute or capability
of a program or system and
determining that it meets its required
results. [Hetzel88]
 Typical Software Quality
Factors
Functionality Engineering Adaptability
(exterior quality) (interior quality) (future quality)
Correctness Efficiency Flexibility

Reliability Testability Reusability

Usability Documentation Maintainability

Integrity Structure

Typical Software Quality Factors [Hetzel88] 

 Testing Classification
Classification by Purpose By Scope
Correctness testing Unit testing
Performance testing Component testing
Reliability testing Integration testing
Security testing
System testing
By Life-cycle phase
Requirements phase testing
Design phase testing
Program phase testing
Evaluating test results
Installation phase testing
Acceptance testing

Maintenance testing
Correctness testing
 Correctness is the minimum requirement of software, the
essential purpose of testing.

 Correctness testing will need some type of criteria , to tell

the right behavior from the wrong one.

 The tester may or may not know the inside details of the
software module under test, e.g. control flow, data flow,
etc.

 Either a white-box point of view or black-box point of view

can be taken in testing software.

 Note: Black-box and white-box ideas are not limited in

correctness testing only
Correctness Testing: (a) Black Box Testing
(Input/Output driven or requirements-based
Testing)

 Test data are derived from the specified

functional requirements without regard to
the final program structure.
 It is also termed data-driven, black-box
testing also mainly refers to functional
testing -- a testing method emphasized on
executing the functions and examination of
their input and output data.
 only the inputs, outputs and specification
are visible, and the functionality is
determined by observing the outputs to
corresponding inputs.
Correctness Testing: (b) White Box Testing
(Glass-box / Logic Driven/ Design Based
Testing)
 software is viewed as a white-box

 structure and flow of the software under test are visible to

the tester.

 Testing plans are made according to the details of the

software implementation, such as programming language,
logic, and styles. Test cases are derived from the program
structure.

 There are many techniques available in white-box testing

 Statement coverage: executing each line of code at least once
 Branch coverage: traverse every branch statements
 Multiple condition coverage: cover all the possible
combinations of true and false condition predicates
Performance Testing
 Usually includes:
 Resource usage,
 Throughput
 Stimulus-response time
 Queue lengths detailing the average or maximum number of
tasks waiting to be serviced by selected resources.
 Typical resources considered
 Network bandwidth requirements
 CPU cycles
 Disk space
 Disk access operations
 Memory usage
 The goal of performance testing
 performance bottleneck identification,
 performance comparison and evaluation, etc.
 Typically done by using a benchmark -- a program, or trace
designed to be representative of the typical system usage
Reliability testing

 refers to the probability of failure-free operation of a

system.
 related to many aspects of software, including the
testing process.
 Testing is an effective sampling method to measure
software reliability.
 black-box testing can be used to obtain failure data,
along with an estimation model is used to estimate
the present reliability and predict future reliability.
 the developers can decide whether to release the
software, and the users can decide whether to adopt
and use the software
Security testing
 Software quality, reliability and security are tightly coupled.
 Flaws in software can be exploited by intruders to open
security holes.

 With the development of the Internet, software security

problems are becoming even more severe.

 The purpose of security testing of these systems include

 identifying and removing software flaws that may potentially
lead to security violations,
 Validating the effectiveness of security measures.

 Simulated security attacks can be performed to find

vulnerabilities in the software.
Testing automation
 Software testing can be very costly.
 Automation: way to cut down time & cost.
 Software testing tools & techniques usually
suffer from a lack of generic applicability &
scalability
 Still significant amount of human
intervention is needed in testing.
 The degree of automation remains mainly
at the automated test script level.
Various Types of Software Testing
 Black box testing
 not based on any knowledge of internal design or code. Tests
are based on requirements and functionality.

 White box testing

 based on knowledge of the internal logic of an application's
code. Tests are based on coverage of code statements,
branches, paths, conditions.

 unit testing
 the most 'micro' scale of testing; to test particular functions or
code modules. Typically done by the programmer and not by
testers, as it requires detailed knowledge of the internal
program design and code.

 incremental integration testing

 continuous testing of an application as new functionality is
added;
Various Types of Software Testing
 integration testing
 - testing of combined parts of an application to determine if
they function together correctly. The 'parts' can be code
modules, individual applications, client and server applications
on a network, etc. This type of testing is especially relevant to
client/server and distributed systems.
 functional testing
 - black-box type testing geared to functional requirements of
an application;
 system testing
 - black-box type testing that is based on overall requirements
specifications; covers all combined parts of a system.
 end-to-end testing
 - similar to system testing; the 'macro' end of the test scale;
involves testing of a complete application environment in a
situation that mimics real-world use, such as interacting with a
database, using network communications, or interacting with
other hardware, applications, or systems if appropriate.
Various Types of Software Testing
 sanity testing or smoke testing
 typically an initial testing effort to determine if a new software version
is performing well enough to accept it for a major testing effort.

 regression testing
 Re-testing after fixes or modifications of the software or its
environment. Automated testing approaches can be especially useful
for this type of testing.

 acceptance testing
 final testing based on specifications of the end-user or customer, or
based on use by end-users/customers over some limited period of
time.

 load testing
 testing an application under heavy loads, such as testing of a web site
under a range of loads to determine at what point the system's
response time degrades or fails.
Various Types of Software Testing
 stress testing
 - term often used interchangeably with 'load' and 'performance' testing.
Also used to describe such tests as system functional testing while
under unusually heavy loads, heavy repetition of certain actions or
inputs, input of large numerical values, large complex queries to a
database system, etc
 performance testing
 - term often used interchangeably with 'stress' and 'load' testing.
Ideally 'performance' testing (and any other 'type' of testing) is defined
in requirements documentation or QA or Test Plans.
 usability testing
 - testing for 'user-friendliness'. Clearly this is subjective, and will
depend on the targeted end-user or customer. User interviews,
surveys, video recording of user sessions, and other techniques can be
used. Programmers and testers are usually not appropriate as usability
testers.
 install/uninstall testing
 - testing of full, partial, or upgrade install/uninstall processes.
Various Types of Software Testing
 recovery testing
 testing how well a system recovers from crashes, hardware failures, or
other catastrophic problems.
 failover testing
 typically used interchangeably with 'recovery testing'
 security testing
 testing how well the system protects against unauthorized internal or
external access, willful damage, etc; may require sophisticated testing
techniques.
 compatability testing
 testing how well software performs in a particular
hardware/software/operating system/network/etc. environment.
 exploratory testing
 often taken to mean a creative, informal software test that is not based
on formal test plans or test cases; testers may be learning the software
as they test it.
 ad-hoc testing
 similar to exploratory testing, but often taken to mean that the testers
have significant understanding of the software before testing it.
Various Types of Software Testing
 context-driven testing
 testing driven by an understanding of the
environment, culture, and intended use of
software. For example, life-critical medical
equipment softwarer a low-cost computer game.

 user acceptance testing

 - determining if software is satisfactory to an

end-user or customer.
 comparison testing
 - comparing software weaknesses and strengths
to competing products.
 alpha testing
 - testing of an application when development is nearing
completion; minor design changes may still be made as a
result of such testing. Typically done by end-users or others,
not by programmers or testers.
 beta testing
 - testing when development and testing are essentially
completed and final bugs and problems need to be found
before final release. Typically done by end-users or others, not
by programmers or testers.
 mutation testing
 - a method for determining if a set of test data or test cases is
useful, by deliberately introducing various code changes
('bugs') and retesting with the original test data/cases to
determine if the 'bugs' are detected. Proper implementation
requires large computational resources