Software Testing - ISO Standards

Many organizations around the globe develop and implement different standards to improve the
quality needs of their software. This chapter briefly describes some of the widely used
standards related to Quality Assurance and Testing.

ISO/IEC 9126
This standard deals with the following aspects to determine the quality of a software
application −

 Quality model
 External metrics
 Internal metrics
 Quality in use metrics

This standard presents some set of quality attributes for any software such as −

 Functionality
 Reliability
 Usability
 Efficiency
 Maintainability
 Portability

The above-mentioned quality attributes are further divided into sub-factors, which you can
study when you study the standard in detail.

ISO/IEC 9241-11
Part 11 of this standard deals with the extent to which a product can be used by specified users
to achieve specified goals with Effectiveness, Efficiency and Satisfaction in a specified context
of use.

This standard proposed a framework that describes the usability components and the
relationship between them. In this standard, the usability is considered in terms of user
performance and satisfaction. According to ISO 9241-11, usability depends on context of use
and the level of usability will change as the context changes.
/IEC 25000:2005

ISO/IEC 25000:2005 is commonly known as the standard that provides the guidelines for
Software Quality Requirements and Evaluation SQuaRESQuaRE. This standard helps in
organizing and enhancing the process related to software quality requirements and their
evaluations. In reality, ISO-25000 replaces the two old ISO standards, i.e. ISO-9126 and ISO-
14598.

SQUARE is divided into sub-parts such as −

 ISO 2500n − Quality Management Division

 ISO 2501n − Quality Model Division
 ISO 2502n − Quality Measurement Division
 ISO 2503n − Quality Requirements Division
 ISO 2504n − Quality Evaluation Division

The main contents of SQuaRE are −

 Terms and definitions

 Reference Models
 General guide
 Individual division guides
 Standard related to Requirement
Engineering i.e.specification,planning,measurementandevaluationprocessi.e.specificatio
n,planning,measurementandevaluationprocess

ISO/IEC 12119

This standard deals with software packages delivered to the client. It does not focus or deal
with the clients’ production process. The main contents are related to the following items −

 Set of requirements for software packages.

 Instructions for testing a delivered software package against the specified requirements.
Miscellaneous
Some of the other standards related to QA and Testing processes are mentioned below −

Sr.No Standard & Description

IEEE 829
1 A standard for the format of documents used in different stages of software testing.

2 IEEE 1061

A methodology for establishing quality requirements, identifying, implementing,

analyzing, and validating the process, and product of software quality metrics.

3 IEEE 1059

Guide for Software Verification and Validation Plans.

4 IEEE 1008

A standard for unit testing.

5 IEEE 1012

A standard for Software Verification and Validation.

6 IEEE 1028

A standard for software inspections.

7 IEEE 1044

A standard for the classification of software anomalies.

8 IEEE 1044-1

A guide for the classification of software anomalies.

9 IEEE 830

A guide for developing system requirements specifications.

10 IEEE 730

A standard for software quality assurance plans.

11 IEEE 1061

A standard for software quality metrics and methodology.

12 IEEE 12207

A standard for software life cycle processes and life cycle data.

13 BS 7925-1

A vocabulary of terms used in software testing.

14 BS 7925-2

A standard for software component testing.

 Capability Maturity Model

The Capability Maturity Model (CMM) specifies an increasing series of levels of a software
development organization. The higher the level, the better the software development process,
hence reaching each level is an expensive and time-consuming process.
Levels of CMM

 Level One: Initial - The software process is characterized as inconsistent, and

occasionally even chaotic. Defined processes and standard practices that exist are
abandoned during a crisis. Success of the organization majorly depends on an
individual effort, talent, and heroics. The heroes eventually move on to other
organizations taking their wealth of knowledge or lessons learnt with them.

 Level Two: Repeatable - This level of Software Development Organization has a

basic and consistent project management processes to track cost, schedule, and
functionality. The process is in place to repeat the earlier successes on projects with
similar applications. Program management is a key characteristic of a level two
organization.

 Level Three: Defined - The software process for both management and engineering
activities are documented, standardized, and integrated into a standard software
process for the entire organization and all projects across the organization use an
 approved, tailored version of the organization's standard software process for
developing, testing and maintaining the application.

 Level Four: Managed - Management can effectively control the software development
effort using precise measurements. At this level, organization set a quantitative quality
goal for both software process and software maintenance. At this maturity level, the
performance of processes is controlled using statistical and other quantitative
techniques, and is quantitatively predictable.

 Level Five: Optimizing - The Key characteristic of this level is focusing on

continually improving process performance through both incremental and innovative
technological improvements. At this level, changes to the process are to improve the
process performance and at the same time maintaining statistical probability to achieve
the established quantitative process-improvement objectives.
 Software Reliability
A test is considered reliable if we get the same result repeatedly. Software Reliability is
the probability of failure-free software operation for a specified period of time in a specified
environment. Software Reliability is also an important factor affecting system reliability.

Reliability Testing

It is about exercising an application so that failures are discovered and removed before the
system is deployed. The purpose of reliability testing is to determine product reliability, and to
determine whether the software meets the customer’s reliability requirements.

 Reliability refers to the consistency of a measure. A test is considered reliable if we get

the same result repeatedly. Software Reliability is the probability of failure-free
software operation for a specified period of time in a specified environment. Software
Reliability is also an important factor affecting system reliability.
 Reliability testing will tend to uncover earlier those failures that are most likely in
actual operation, thus directing efforts at fixing the most important faults.
 Reliability testing may be performed at several levels. Complex systems may be tested
at component, circuit board, unit, assembly, subsystem and system levels.

Software reliability is a key part in software quality. The study of software reliability can be
categorized into three parts:

1.Modeling
2.Measurement
3.Improvement

1. Modeling: Software reliability modeling has matured to the point that meaningful results can
be obtained by applying suitable models to the problem. There are many models exist, but no
single model can capture a necessary amount of the software characteristics. Assumptions and
abstractions must be made to simplify the problem. There is no single model that is universal to
all the situations.

2. Measurement: Software reliability measurement is naive. Measurement is far from

commonplace in software, as in other engineering field. “How good is the software,
quantitatively?” As simple as the question is, there is still no good answer. Software reliability
cannot be directly measured, so other related factors are measured to estimate software
reliability and compare it among products. Development process, faults and failures found are
all factors related to software reliability.

3. Improvement: Software reliability improvement is hard. The difficulty of the problem stems
from insufficient understanding of software reliability and in general, the characteristics of
software. Until now there is no good way to conquer the complexity problem of software.
Complete testing of a moderately complex software module is infeasible. Defect-free software
product cannot be assured. Realistic constraints of time and budget severely limit the effort put
into software reliability improvement.
 Project Quality Plan
To address all the quality requirements, standards and quality assurance mechanisms in a
project, a document called 'project quality plan' is developed by the project team. This plan
acts as the quality bible for the project and all the stakeholders of the project should adhere to
the project quality plan.

The Components of a Project Quality Plan

Depending on the nature of the industry and the nature of the project, the components or the
areas addressed by a quality plan may vary. However, there are some components that can be
found in any type of quality plan.

Let's have a look at the most essential attributes of a project quality plan.

Responsibility of Management
This describes how the management is responsible for achieving the project quality. Since
management is the controlling and monitoring function for the project, project quality is
mainly a management responsibility.

Document Management and Control

Documents are the main method of communication in project management. Documents are
used for communication between the team members, project management, senior management
and the client.

Therefore, the project quality plan should describe a way to manage and control the
documents used in the project. Usually, there can be a common documentation repository with
controlled access in order to store and retrieve the documents.

Requirements Scope
The correct requirements to be implemented are listed here. This is an abstraction of the
requirements sign-off document. Having requirements noted in the project quality plan helps
the quality assurance team to correctly validate them.

This way, quality assurance function knows what exactly to test and what exactly to leave out
from the scope. Testing the requirements that are not in the scope may be a waste for the
service provider.

Design Control
This specifies the controls and procedures used for the design phase of the project. Usually,
there should be design reviews in order to analyse the correctness of the proposed technical
design. For fruitful design reviews, senior designers or the architects of the respective domain
should get involved. Once the designs are reviewed and agreed, they are signed-off with the
client.
With the time, the client may come up with changes to the requirements or new requirements.
In such cases, design may be changed. Every time the design changes, the changes should be
reviewed and signed-off.

Development Control and Rigor

Once the construction of the project starts, all the processes, procedures and activities should
be closely monitored and measured. By this type of control, the project management can make
sure that the project is progressing in the correct path.

Testing and Quality Assurance

This component of the project quality plan takes precedence over other components. This is
the element, which describes the main quality assurance functions of the project. This section
should clearly identify the quality objectives for the project and the approach to achieve them.

Risks & Mitigation

This section identifies the project quality risks. Then, the project management team should
come up with appropriate mitigation plans in order to address each quality risk.

Quality Audits
For every project, regardless of its size or the nature, there should be periodic quality audits to
measure the adherence to the quality standards. These audits can be done by an internal team
or an external team.

Sometimes, the client may employ external audit teams to measure the compliance to
standards and procedures of the project processes and activities.

Defect Management
During testing and quality assurance, defects are usually caught. This is quite common when it
comes to software development projects. The project quality plan should have guidelines and
instructions on how to manage the defects.

Training Requirements
Every project team requires some kind of training before the project commences. For this, a
skill gap analysis is done to identify the training requirements at the project initiation phase.

The project quality plan should indicate these training requirements and necessary steps to get
the staff trained.
 Automation Testing
Automation Testing means using an automation tool to execute your test case suite. The
automation software can also enter test data into the System under Test, compare expected and
actual results and generate detailed test reports. It demands considerable investments of money
and resources.

Successive development cycles will require execution of same test suite repeatedly. Using a test
automation tool, it's possible to record this test suite and re-play it as required. Once the test
suite is automated, no human intervention is required. This improved ROI of Test Automation.
The goal of Automation is to reduce the number of test cases to be run manually and not to
eliminate Manual testing altogether.

Automated software testing is important due to the following reasons:

 Manual Testing of all workflows, all fields, all negative scenarios is time and money
consuming
 It is difficult to test for multilingual sites manually
 Automation does not require Human intervention. You can run automated test
unattended (overnight)
 Automation increases the speed of test execution
 Automation helps increase Test Coverage
 Manual Testing can become boring and hence error-prone.

Automated Testing Process:

Following steps are followed in an Automation Process

Test tool selection:-

Test Tool selection largely depends on the technology the Application under Test is built on.
For instance, QTP does not support Informatics.

So QTP cannot be used for testing Informatics applications. It's a good idea to conduct a Proof
of Concept of Tool on AUT.
Define the scope of Automation:-

The scope of automation is the area of your Application under Test which will be automated.
Following points help determine scope:

 The features that are important for the business

 Scenarios which have a large amount of data
 Common functionalities across applications
 Technical feasibility
 The extent to which business components are reused
 The complexity of test cases
 Ability to use the same test cases for cross-browser testing

Planning, Design, and Development

During this phase, you create an Automation strategy & plan, which contains the following
details-

 Automation tools selected

 Framework design and its features
 In-Scope and Out-of-scope items of automation
 Automation test bed preparation
 Schedule and Timeline of scripting and execution
 Deliverables of Automation Testing

Test Execution:-

Automation Scripts are executed during this phase. The scripts need input test data before there
are set to run. Once executed they provide detailed test reports.

Execution can be performed using the automation tool directly or through Test management
tool which will invoke the automation tool.

Example: Quality center is the Test Management tool which in turn it will invoke QTP for
execution of automation scripts. Scripts can be executed in a single machine or a group of
machines. The execution can be done during the night, to save time.

Maintenance:-

As new functionalities are added to the System under Test with successive cycles, Automation
Scripts need to be added, reviewed and maintained for each release cycle. Maintenance
becomes necessary to improve the effectiveness of Automation Scripts.
 Project Management Tools

When applying any of the techniques to the project you need to accomplish, you also need to
use specific tools for successfully implementing the technique. Here’s a list of software tools
that are used in project management on different work steps.

Organizing Workflow & Planning

It is the core part of starting a project: it defines how the project will be performed, and how the
quality of its outcome will be ensured. Large companies often use such comprehensive solution
as MS Project. For smaller teams, various alternatives are available. They don’t provide all the
rich functionality typical to complex and expensive tools, but they have planning and roadmap
features that are sufficient for visualizing future project progress. There are also free project
management solutions that suit best for start-ups and small businesses.

Communication

Being the key point in many techniques and methodologies, communication within a project
team needs to be properly organized. While using email for formal communication and
important messages, it’s also essential to have a corporate messenger – Slack and Skype are the
most popular ones. And, if your team members use different tools to communicate, eliminate
the pain of having multiple messengers by integrating them.

Scheduling

When allocating resources and planning for future, it’s crucial to know who on
the team is available for specific dates. Use scheduling software for that! Such tools
as actiPLANS provide a clear visual chart of absences for upcoming dates, and allow to see all
necessary details to team members’ leaves and time off.

Time management

Knowing where your team’s time goes not just helps managing current project risks. It also
provides valuable information for future planning and estimating. Time management tools,
such as actiTIME, help managers understand both individual time expenses and team’s results
for any period. Informative reports with time & cost summaries and notes to the logged time
provide insights into how time is used and what can be optimized.
Finance & Accounting

For any project manager and business owner, understanding financial outcome of the projects is
crucial for analysis and future planning. Most used accounting tools – QuickBooks, Zoho, and
Fresh books – help collect this information. For smaller project teams, other accounting
solutions can be reasonable. They require less investment, but also provide insights on project
profitability, teams’ performance, and estimation accuracy.