The research register for this journal is available at

http://www.mcbup.com/research_registers/quality.asp

The current issue and full text archive of this journal is available at
http://www.emerald-library.com

Quality assurance and

effectiveness of audit systems

Quality
assurance and
audit systems

Stanislav Karapetrovic

Daltech/Dalhousie University, Halifax, Nova Scotia, Canada, and

Walter Willborn

University of Manitoba, Winnipeg, Manitoba, Canada

679
Received September 1999
Revised December 1999

Keywords Quality assurance, Reliability, Maintenance, Effectiveness

Abstract Quality audits are prominent and proven management tools for assessing compliance
and effectiveness of quality systems. They are commonly used in the effort to improve overall
business performance. However, similarly to any other physical or conceptual system, they may
fail to achieve objectives set forth, raising concerns among auditees and clients alike. Argues for
the provision of adequate confidence to various interested parties in the quality of auditing
services. A quality audit is conceptualized using the systems approach. Subsequently, a quality
assurance framework based on the application of the 1994 and 2000 versions of the ISO 9001
standard in auditing is presented. Audit system effectiveness is modeled using the concepts of
audit reliability, availability and suitability. Audit failures, risks and maintainability are addressed
in detail. It is concluded that audit managers must ensure appropriate levels of quality and
effectiveness of quality audit systems.

Introduction
Quality audits have gained prominence in the last 20 years as a tool for
assessing the effectiveness of quality assurance efforts and, more recently, for
the evaluation of compliance with applicable quality standards, such as
ISO 9000. Quality auditors examine, in several stages, whether or not quality
processes, resources and objectives are what they should be. First, they assess
compliance of quality assurance procedures and related documentation with
applicable standards and guidelines (also called ``audit criteria''). Then they
typically evaluate whether actual quality assurance activities conform to the
documented procedures, and are effectively implemented and suitable to
achieve quality objectives. The evaluation of the system effectiveness can be a
powerful management tool for quality improvement. In fact, many authors
argue that one of the primary purposes of audits is continuous improvement
(Burr, 1997; Hunt, 1997; Willborn and Cheng, 1994; Russell, 1997; Russell and
Regel, 1996; Walker, 1998).
In performing auditing activities, quality auditors must objectively and
independently collect and verify audit evidence, evaluate it against audit
criteria, and report their findings. Objectivity and independence are two
separate, yet interrelated, fundamental principles of auditing. Objectivity
relates to the consistency of the auditing process and results, materiality of
evidence, the use of appropriate methodology (e.g. statistical sampling,
flowcharts, and checklists), the application of a systematic approach to
auditing, as well as being free from bias. Consistency, for instance, means that

International Journal of Quality &

Reliability Management,
Vol. 17 No. 6, 2000, pp. 679-703.
# MCB University Press, 0265-671X

IJQRM
17,6

680

two auditors auditing the same system against identical criteria should come
up with similar conclusions. On the other hand, independence refers to both the
auditor's organizational position and his/her state of mind. In order to conduct
effective and efficient audits, and do this objectively, auditors must not be
directly responsible to the function or organization being audited. Here lies the
fundamental difference between an auditor and a consultant. While consultants
assist an organization in establishing a quality system, they cannot objectively
and independently assess whether the system is implemented effectively and in
compliance with applicable standards. However, consulting and auditing have
one common characteristic: both are, in essence, services rendered to
management. As such, both are subject to quality assurance.
The issue of quality assurance and reliability of audits bears a particularly
important impact on the fundamental principles of objectivity, independence
and continuous improvement. For instance, how does an auditor ensure a
reliable and objective collection and evaluation of evidence against audit
criteria? Due to time, organizational, financial and other constraints, these
processes normally involve only a selected sample of activities and
documentation. Thus, an auditor must collect a representative sample of
evidence from the population, and evaluate it against set criteria. Inherent in
sampling procedures is a risk of accepting a wrong finding, or rejecting a
correct one. This concept, called an audit risk, must be taken into account when
assuring the audit client of the reliability of findings and audit results with
some degree of confidence. Naturally, audit errors and inconsistencies do
happen, and audits do fail (Druckman, 1997; Stratton, 1995). For instance, if
audit objectives are deemed unattainable, an audit is terminated (Willborn,
1993). In such a case, how can clients be assured that audits will be restored to
the operational mode and ultimately achieve audit policy and objectives of
continuous improvement? Audit maintainability deals with related questions.
Another related and important issue is the suitability of audits and audit
systems to adapt to changing environments, audit requirements and criteria in
achieving set goals.
The whole notion of quality assurance and quality of auditing activities has
been relatively unexplored in the realm of quality audits (Willborn, 1993).
While earlier national auditing guidelines, such as the Canadian CSA Q395
(1981) and American ANSI/ASQC Q1 (1986), contained provisions for quality
assurance of audits, the contemporary international standard on quality audits,
namely ISO 10011 (1990), does not discuss this issue. In the same vein, the
current work on the revision of quality and environmental audit guidelines,
under the auspices of the International Organization for Standardization, does
not point specifically to addressing audit quality assurance. Unfortunately, this
has created a situation where implied quality assurance of auditing activities
rests solely with the adequate qualification and competence of auditors, and
conformance of the auditing process to the existing audit guidelines. It has also
left a serious gap in the available guidance for quality assurance and control of

auditing programs and systems. In the wake of increased concerns about the
effectiveness of quality audits to achieve stated ``continuous improvement''
objectives, questions about the overall usefulness of sporadic quality system
registration audits, as well as difficulties in achieving and improving the
consistency of both external and internal audits, quality assurance of auditing
services must be seriously examined and implemented.
All this is happening while the cutting edge of quality auditing seems to lie
in the use of new information and computer technologies, the systems approach
to auditing, and the harmonization and integration with other auditing
activities and systems, such as environmental, safety, ergonomic, maintenance
and financial audits. The last area mentioned, namely integration with
financial audits, bears a particularly interesting impact on quality auditing.
The financial (also known as ``internal'' or ``accounting'') audit has evolved as an
established element of the accounting profession. Thus, the concepts of audit
risk, uncertainty, materiality, statistical sampling, reliability of findings, and
audit errors are well known and continuously researched in the accounting
literature (e.g. Colbert, 1996; Friedlob and Schleifer, 1999; Shailer et al., 1998;
Busta and Weiberg, 1998; Woodhead, 1997; Karim and Siegel, 1998). These
concepts are crucial for proper understanding and application of audits,
regardless of the particular discipline (e.g. finances, quality, environment or
safety) addressed. However, they are not given appropriate recognition in the
quality auditing literature. For example, the body of knowledge for certified
quality auditors by the American Society for Quality (ASQ), does not even
mention audit risk, materiality, reliability, maintainability, or for that matter,
quality assurance of auditing activities (ASQ, 1999).
This paper attempts to present a systematic view of quality assurance in
auditing. The systems approach to auditing is briefly presented, followed by a
discussion on quality assurance and control of the various elements of the audit
system. A model for evaluating audit effectiveness is presented, and reliability,
maintainability and availability of the audit system are addressed.
Systems approach to auditing
Quality audits are documented systems for independent and objective
collection and verification of audit evidence, and evaluation of the evidence
against audit criteria. Audit evidence relates to all information and material
gathered for the purpose of achieving audit objectives. Audit criteria includes
identified and agreed on procedures, guidelines, and standards, such as
ISO 9001 or QS 9000, that serve as a benchmark for an individual audit.
Under the systems approach (Karapetrovic and Willborn, 1998), an audit is
viewed as a set of interdependent processes (or activities), using human,
material, infrastructural, financial, information and technical resources to
achieve objectives related to the continuous improvement of performance
(Figure 1: top). In other words, an audit is a system (Hirzel, 1998; Bishara and
Wyrick, 1994).

Quality
assurance and
audit systems
681

IJQRM
17,6

682

Figure 1.
Audit system

Characteristics of the systems approach in auditing include (Figure 1, bottom):

.

Individual audits are harmonized and aligned toward achieving a

strategic common goal (for instance, continuous improvement of
performance), rather than a set of unrelated tactical goals.
Individual audits are subsystems of audit programs (e.g. a program for
auditing a quality management system may include a cycle of 12
individual quality audits over a period of three years).

Audit programs are subsystems of the overall generic audit system (for
example, an organization can have several audit programs forming an
audit system, including environmental, safety, financial and quality, as
well as special external, internal or supplier audit programs).

Quality
assurance and
audit systems

A generic audit system is a subsystem of the overriding management

system, and is managed according to the same management guidelines,
methodologies and principles as other systems (such as the quality
management system and the environmental management system).

683

Audits are interdependent, meaning that, while focused on achieving the

same global goal and sharing the same resource base, output of one or
more audits can be input into another.
Audits are open and dynamic systems, meaning that the parameters and
constraints under which they operate inevitably change with time
(Willborn, 1990; Peters, 1998).
Audits are adaptive systems, being able to accustom themselves to
changing operational environments.

These characteristics bear an important impact on quality assurance and

effectiveness of audits. Being managed in the same manner as other
management systems, audit systems can make use of generic management
guidelines, such as ISO 9000, to provide assurances of quality to their
customers (called ``audit clients''). Thus, it is foreseeable, and indeed desirable,
to independently and objectively audit (or ``assess'') the audit programs and
systems themselves for compliance to audit criteria (say ISO 10011 (1990)
Guidelines for Quality Auditing Systems) and suitability to achieve audit
objectives. Because audits are dynamic and adaptive systems, audit objectives
and criteria can be gradually improved with time, causing quality
improvements in the audit system. The interdependence of audits allows for
the focus on the ``global goal'', which in turn reduces redundancies and waste in
the audit systems. For instance, instead of scheduling audits at pre-set time
intervals, even when the situation does not render a particular audit necessary,
flexible scheduling based on risk management and the prioritization of audit
findings is more useful (Harral, 1998). Also, as Gardner (1997) states, ``greater
gains in performance should be made because of the systemic view of the
enterprise''. On the other hand, since audit systems are composed of various
audit activities and resources, the reliability, availability and maintainability
(RAM), and ultimately the effectiveness of the system, depend on the RAM of
each of its elements. In other words, the audit system, much like a chain that
always fails at its weakest link, is only as good (meaning as reliable, available
and maintainable) as its weakest element. The following sections discuss
quality assurance and audit system effectiveness in more detail.

IJQRM
17,6

684

Quality assurance in audit systems

In order to ensure quality of audit processes and outcomes, relevant measures
for quality assurance should be applied. We will now proceed with a discussion
of the reasons, benefits and concerns of applying formal assurance based on the
ISO 9001 quality system in auditing, as well as suggestions on the framework
for such an application.
Building confidence
Quality assurance (QA) relates to the provision of confidence to customers that
their requirements for the quality of product or service are met continuously.
Since quality auditing is a service rendered to internal (executive management)
or external (customer or third party) clients, a quality audit system must
provide such confidence with respect to stated audit requirements. For
example, the requirements may include:
.
Specific time frames for conducting audits.
.
Provisions for the management of audit programs.
.
Maintenance of adequate auditor competence.
.
Availability of audit resources to plan and conduct risk-based audits
(e.g. see Colbert and Alderman, 1995).
.
Proper co-operation with the client and auditee.
The auditing process commonly involves parties other than clients who have
substantial interests and stakes. For instance, the auditee is very interested in
the results of the audit, and relies on the competence and quality of the auditor
and the audit system for the provision of useful feedback. In environmental
auditing, the list of interested parties is broadened to include the local and
global community, the company's personnel (who are not necessarily the
auditees) and the government. Adequate QA efforts must be aimed at building
confidence and awareness in all audit participants and interested parties. In
order to meet such an audit QA objective, audit organizations (both external
registrars, and internal audit department or individual auditors) may
document and implement an assurance system for auditing services (ANSI/
ASQC, 1986), or focus on adopting total quality management principles and
practices (Rezaee, 1996). For implementing quality assurance in auditing, a
framework based on the ISO 9001 quality system is presented in the next
section.
ISO 9001-based auditing framework
We have mentioned earlier that, under the systems approach, individual
quality audits are viewed as interdependent subsystems of audit programs,
and, ultimately, of the generic audit system of an organization. To provide
adequate confidence to customers (external or internal clients) that auditing
services meet their expectations, quality assurance (QA) efforts are required

at all three levels: individual audits, audit programs and the generic audit
system. The conceptualization of QA at the individual audit level is depicted in
Figure 1, and will be discussed later in the paper, with the presentation of
proper audit methodologies, as well as the discussion of audit effectiveness and
auditor competence. The ISO 9000 scheme, although undoubtedly having an
impact on individual audits, is primarily aimed at the audit program/system
levels. This is because the value of a series of interrelated audits directed at
performance improvement by far outweighs the benefits of a single audit
performed for compliance purposes only (Willborn and Cheng, 1994).
Organizations that utilize their audits in a kaizen-like manner, focusing on
small, but steady improvements, will greatly benefit from a structured QA
approach that ISO 9000 brings. They will manage the audit system by
concentrating first on the global auditing policy and objectives, and
transforming them into a meaningful framework of quality, environmental,
safety, financial, ergonomic, maintenance, and other audit programs, which
will be brought to fruition by conducting individual audits.
When such a system of interrelated audits is established, an ISO 9000
standard could be applied to further strengthen the system efficiency and
effectiveness. Gardner (1997) illustrates the benefits of an ISO 9000-based audit
administration in terms of increased accuracy and confidence of audit results,
organizational measurements, as well as greater confidence in corrective
actions resulting from an audit. The implementation on the basis of the current
ISO 9001 (1994) standard and its revised year 2000 version is presented in
Figure 2. The illustrated framework is structured around the main system
elements: audit objectives, processes and resources. Overall audit system policy
and objectives are determined and reviewed first. For instance, the policy of a
manufacturing organization could be to manage audits that will consistently
provide opportunities for a measurable improvement of quality, environmental,
health and safety, ergonomic and financial performance. Identified policy and
objectives will determine the types of audits to be conducted, as well as the
suitable audit requirements. In the case of the above-mentioned policy, separate
quality, environmental, safety, ergonomic and financial (accounting) audit
programs may be designed, and corresponding standards, guidelines,
regulations and statutes are used as audit requirements. The ISO 9001
standard is an example of a requirement in auditing quality management
systems, while ISO 14001 and related environmental standards can be used for
the assessment of environmental performance. A smaller organization may
require only a single audit program, which can relate to a particular discipline
(say quality), or an integrated approach with quality, environmental and safety
audits done simultaneously.
The next stage involves audit system planning and design, in which audit
programs are prepared and reviewed for effectiveness to meet stated objectives,
and specific quality assurance procedures for conducting individual audits are
designed. These procedures need not be complicated and detailed. For instance,

Quality
assurance and
audit systems
685

IJQRM
17,6

686

Figure 2.
ISO 9000 application in
auditing

they can simply state that audits are planned and conducted in compliance
with the existing audit guidelines, such as ISO 10011 (1990) and ISO 14010/11/
12 (1996). Audit methodologies, such as discovery, acceptance and estimation
sampling (Hill et al., 1962), computer-aided audit tools, risk or procedure-driven
auditing, flowcharts and checklists are also prepared at this time. The choice of

methods depends on the stated objectives. For example, if compliance to a

management system standard is of primary importance, the emphasis will be
placed on checklists that adequately represent the contents and the meaning of
the standard, as well as appropriate sampling techniques that will assure the
management, with a high confidence level, of the required compliance.
The planning stage is followed by the allocation and deployment of
resources necessary to achieve the audit system objectives. Auditors and the
persons responsible for the management of audit programs are identified, their
qualifications and competence determined and reviewed, and audit
assignments for particular audit programs are given. Other resources, such as
the financial support for the audit program/system, are acquired and allocated.
It is important to note that resource allocation and deployment in this manner
relates to the audit program/system, and not to the individual audit. In the
overall audit system, the planning, conducting and reporting on individual
audits is actually a part of the ``implementation and control'' stage of the audit
process (Figure 2). Here, individual audit plans are prepared, audits are
executed, and audit reports are provided to the client. After several cycles of
audits are performed, the people responsible for the management of the audit
system may analyze the performance of the system, and prepare a report on the
overall system efficiency and effectiveness. As an input into the analysis (we
may refer to it as ``review of the audit system'' or simply an ``audit review''),
results of the internal quality audits on the audit system should be taken into
account. This report should then be submitted to the top management, who
should review it and attempt to find possibilities for improvement. Table I
presents a list of questions, categorized by each element of the audit system,
which could help in performing internal audits (assessments) of the audit
system, as well as in conducting audit reviews.
Naturally, questions about the applicability of a formal ISO 9000 system in
auditing may be raised. The following section discusses some of these issues.
ISO 9000 applicability
It does not take a large organization with a separate auditing department, or
even a registrar or a consulting firm with external auditing as the most
important part of their business, to develop an ISO 9000 system for audits. On
the contrary, small businesses with perhaps only one or a few internal auditors
could benefit the most from this. Much too often, due to the lack of people and
available time, ISO 9000-registered small businesses rely on sporadic,
compliance-geared external audits to support their registration. Surveillance
audits are performed every six months to a year, with only a small sample
(several unrelated elements) of the quality system being checked each time.
Hastily organized internal audits are completed just before the external ones,
``to make sure that everything is OK before the registrar finds something''. In
such a situation, how can the management be assured that audits are doing
what they are supposed to do? They are just living a big lie (Beeler, 1999).

Quality
assurance and
audit systems
687

IJQRM
17,6

System element
Objectives
Policy

688

Objectives

Scope

Feasibility

Criteria

Resources
Auditor

Methods

Table I.
Material
Sample quality
Information
assurance questions for
audit system elements

Sample quality assurance questions

Is the overall audit policy defined?

Is it aligned with the organization's overall strategic goals/policies?
Is a regular audit policy review in place, and is it effective?
Do audit objectives follow the framework set in the audit policy?
Are they quantifiable/measureable
Are audit objectives attainable within the given scope/criteria?
Are procedures in place for identification, review and communication
of the scope/extent of audit activities for each individual audit and
audit program?
Are procedures in place to deal with changes in scope?
Are regular feasibility reviews for individual audits and audit
programs in place, and are they effective?
In the case of non-feasibility of an audit/audit program, have
adequate corrective/preventive actions been identified, acted on, and
reviewed?
Are audit criteria, including applicable standards, regulations,
guidelines and laws, identified for each individual audit?
Are audit criteria readily available to all parties concerned?
Are procedures for establishing auditor qualifications and competence
in place, and are they effective?
Are procedures for auditor/audit team leader training and continual
professional development established, and are they effective?
Are procedures for the evaluation of auditor qualifications and
competence available, and are they effective?
Is adequate supervision/management of auditors' activities in place?
Are records of evaluation of qualifications and competence kept?
Are adequate discovery/estimation/acceptance sampling procedures
used in order to identify risk and confidence intervals for audit
evidence/findings?
Are principles of risk management used when planning and
conducting audits?
Are checklists and flowcharts used in the auditing process?
Are auditors' checklists regularly updated with respect to changes in
the audit criteria, types and scope of audit, business activities
audited, location and other pertinent factors?
Is auditing methodology regularly reviewed for adequacy in meeting
audit objectives/policy
Is computer-aided auditing in place and is it reviewed for adequacy?
Are required material resources identified, acquired and deployed?
Are required information resources identified, acquired and deployed?
(continued)

System element
Processes
Management

Planning

Execution

Improvement

Sample quality assurance questions

Are roles, responsibilities and authorities for the management of

individual audits, audit programs and the audit system identified and
reviewed?
Are individual audits/audit programs planned according to the
applicable audit standards and guidelines, such as ISO 10011 and
ISO 14010/11/12?
Are planning activities and results, including the audit programs,
regularly reviewed for effectiveness to meet stated audit objectives
and policy?
Are individual audits/audits programs executed according to the
applicable audit standards and guidelines, such as ISO 10011 and
ISO 14010/11/12?
Are procedures in place for continuous review of the suitability of the
audit system to meet the audit policy?
Are methods for improvement including statistical quality control
techniques and design of experiments, identified and used regularly?
Are quality audits used to identify areas of non-compliance of the
audit system with stated policies/guidelines and corrective actions
identified?

The point is just the opposite: audits are supposed to find something. No
system is perfect, and there is always room for improvement. A system of
interdependent audits should be established in order to identify areas of
non-conformance to audit criteria and areas for improvement. In some cases,
the auditor might even find that the audit criteria, such as prescribed
calibration procedures, might have to be improved (Willborn, 1990).
``Interdependent'' means that the outputs of an audit (audit findings and
conclusions) are directly related to the input of a subsequent audit, and that
individual audits are aligned to achieve the same ``global'' objective. For
example, a quality audit has found that a particular type of service receives
very low ratings in repeated customer surveys. A corrective action identified
problems in the service delivery area as the primary cause, and these were
corrected. These findings were used in the subsequent audit, which again
examined the service. Although some improvement in the ratings was
accomplished, they were still mediocre. The following action was directed at a
better understanding of the customers' needs for the service, and an improved
communication within the company. The ratings were further improved, but
the audit also recognized problems in the methodology of conducting the
survey. In this situation, all audits were directed toward the achievement of the
``global'' continuous improvement objective.

Quality
assurance and
audit systems
689

Table I.

IJQRM
17,6

690

As we emphasized earlier, the focus of the application of ISO 9000 in

auditing is on the systematic effort to make audits work better, and not on
creating a bureaucratic quagmire of documents, procedures and work
instructions for auditors, which serve no useful purpose. This is why it is
important to conceptualize and implement a declarative, rather than
procedural, quality assurance system (Karapetrovic and Willborn, 1999). While
procedural systems focus on fixed and centralized mechanisms of control,
declarative systems are organized in a dynamic and adaptive fashion. To
illustrate the difference between the two, an example of a document control
process in an ISO 9000 quality system is used. In the procedural system,
document control is a centralized function. Commonly, this means that only one
person (the ISO 9000 management representative or a designate), is in charge of
the document control, and may for instance, replace obsolete pages in the
quality manual. This approach is very good for strict control of controlled
copies of documents, but can be very cumbersome if rapid changes are
required, especially if the person responsible for control is not available. On the
other hand, in order to change the document control process itself, you would
need to change the procedure that guides the process. A declarative system
would make document control more decentralized, with a number of people
involved. Knowing the common objective of the system, in this case adequate
control over quality system documentation, including the availability of
exclusively up-to-date documents and records of changes, people could form
and absolve ad hoc groups to deal with specific emerging document control
tasks. In other words, instead of using a fixed procedure to manage the
document control system, the system is guided by a common objective and
flexible links among the people who control it. The interested reader is referred
to Karapetrovic and Willborn (1999) for a more detailed description of a
declarative quality assurance system.
Flexibility, lack of formal procedures, and adaptation to the changes in the
environment, are probably the most important characteristics of the declarative
approach that can help to improve audit management. For instance, priorityand risk-based auditing, which are becoming increasingly prominent (e.g. see
Colbert and Alderman, 1995; Harral, 1998; Shailer et al., 1998; and Friedlob and
Schleifer, 1999), inherently require a dynamic and adaptive audit system. In
priority-based auditing, audit activities are focused mostly on the areas of
highest importance to the company. These could be the departments that are
the most profitable, or the products that have the highest market share, or
simply elements of the quality system that are considered to be crucial for the
development of the company. Risk-based auditing is similar in the sense that
the effort is prioritized, but in the areas carrying the largest risk of noncompliance with the audit criteria or where not enough information is available
to ensure a correct finding. Because prioritization objectives and areas can
quickly change, a declarative system is required to ensure the effectiveness of
the system.

Audit system effectiveness

``System'' and ``effectiveness'' are probably two of the most often used (and
conversely misused) words in the area of quality management. For instance,
phrases such as: ``We have a system in place, and we are continuously striving
to improve our effectiveness and efficiency'' are common in the workplace
today. Both these words are powerful and lucrative, but what exactly do they
mean in the auditing realm? While we discussed the ``audit system'' earlier in
the paper, we will now focus on the conceptualization of its effectiveness.

Quality
assurance and
audit systems
691

Effectiveness
Providing confidence to the audit customers that their requirements for the
audit quality are met is a necessary, but not the only sufficient condition to
ensure a good audit. The audit system must be able continuously to meet everchanging audit policy and objectives. In other words, it has to be effective.
Naturally, system effectiveness must be measured somehow in order to be
improved. A particularly useful way of expressing effectiveness is in terms of
probability that a system will fulfill a set objective within a given time frame,
when it is functioning under specified conditions and scope (Vujanovic, 1990;
Zelenovic and Todorovic, 1990). Several models from the systems theory are
available to measure effectiveness defined in such probabilistic terms
(Vujanovic, 1990). For the purposes of illustrating the effectiveness of the audit
system, we have selected the model which expresses it as a product of system
reliability, availability and suitability (Figure 3).
Reliability
Audit system reliability can be defined as a probability that the system will
adequately function within a given time frame when it is functioning under
specified conditions and scope. For example, an organization can prepare a
three-year audit plan, with a number of individual audits to be conducted in

Figure 3.
Audit system
effectiveness measures

IJQRM
17,6

692

this time period. Reliability refers to the characteristic of such a series of audits
that their intended function is performed continuously over the stated period of
time. Implied in this definition is that an audit (or a series of audits) can fail,
meaning (in simple terms) that it does not do what it is supposed to.
Unfortunately, audits can and do indeed fail in myriad different ways. At the
individual audit level, failures (also called ``errors'') may be indicated by:
.

Errors in the audit planning stages.

Use of unqualified or incompetent auditors to conduct a specific audit.

Absence of opening and closing meetings.

Inadequate and improper use of sampling methods and other audit

methodologies when collecting evidence.

Lack of a sufficient amount of audit evidence.

Deficient or missing verification of evidence.

Biased evaluation of audit evidence against audit criteria.

Inconsistencies in audit findings.

Acceptance of a non-compliant or ineffective management system in

registration audits.
Rejection of a compliant and effective management system in
registration audits.
Subjective, biased or undue-influenced audit report.

In addition to the above-mentioned failures, audit reliability can be damaged at

the program or system level, where:
.

Audit program management is inadequate.

Audit objectives do not reflect the underlying policy.

Audit errors remain undetected.

Audits are declared feasible when actually they are not.

Deficiencies in material resources and lack of available time exist.

In fact, much like any physical system, audit systems are prone to failure when
any of their constitutive elements fail. For instance, audit objectives, processes
and resources are connected in a series-like fashion, and when an element fails,
it is likely that the whole system will be damaged. Possible audit failure modes,
representing different mechanisms or ways an audit could fail, can be analyzed
using several techniques, including the fault tree analysis (FTA) and failure
mode, effect and criticality analysis (FMECA). An example of a fault tree,
corresponding to the three main audit system elements, is presented in
Figure 4. These techniques help us identify possible causes of errors, and focus
the effort of improving the system reliability by eliminating the root causes.

Quality
assurance and
audit systems
693

Figure 4.
Fault tree diagram for
an audit (example)

Interestingly, due to the analogy between physical and audit systems, many
proven techniques in reliability and maintenance engineering can be
conceptualized in improving the reliability of audits. For instance, physical
systems are continuously monitored in order to detect any changes that could
negatively affect reliability, i.e. cause the system to fail. This technique is called
``condition-based monitoring''. An analogous method in monitoring audit
systems would be an internal audit of the audit system and related programs.

IJQRM
17,6

694

Figure 5.
The bathtub curve in
auditing

In this manner, an audit system is continuously observed and assessed for

potential problems. When such problems are identified, corrective and
preventive actions are immediately taken to support proper functioning of the
system. For example, an internal audit on the audit program may reveal that
some auditors feel uncomfortable with their level of training and competence in
assessing new, technologically advanced processes and products. This by itself
does not yet represent a major failure, but can indicate a potential problem of
having incompetent auditors evaluating such new systems in the future.
Therefore, an immediate corrective action resulting in adequate auditor
training with respect to the function and use of these advanced systems would
be suitable.
Another well-known reliability engineering concept can be applied to
auditing, namely the bathtub curve. The bathtub concept relates to the number
of failures the system experiences over time (Figure 5). Experience suggests
that the system is more likely to fail in its infancy, while during most of the
system's life, the rate of failures is constant, and due to random/chance causes.
Finally, toward the end of life, due to the wear and tear of its elements, the
system exhibits an increasing number of failures. The situation is similar in
auditing. At the inception of a new audit program, such as the introduction of
internal quality auditing in an ISO 9000-driven small business, auditors are
commonly inexperienced, program and individual audit objectives may be
incompatible, new and relatively undeveloped systems are examined, the level
of co-operation of auditees is low, and so on. Therefore, a high number of errors
and misjudgements may be expected. With the passage of time, as the audit
system becomes increasingly mature, and sound audit methodologies are
introduced, the rate of errors decreases. After a while, the audit system reaches
its ``steady-state'', characterized by experienced auditors and efficiently used
methods and processes. At one point, however, audit failures may start to

increase. This may be caused, for example, by the audit management's

insistence on adherence to invalid or obsolete audit criteria (e.g. standards) and
objectives (Willborn and Cheng, 1994). The audited management system
appears to be compliant, when in fact it is not effective. This, in essence, is a
type II or  error (accepting a wrong decision). Therefore, the audit may not
adequately identify problems or areas of possible improvement, which causes
the audit failure rate to increase. When such invalid criteria and objectives are
replaced, audit failures will fluctuate around a steady rate, and ultimately
stabilize, according to the Drenick's limit theorem (Coetzee, 1998). The
fluctuation is caused by simultaneous decreases due to the appropriate
standards being used, and increases due to the ``infancy'' failures prone to the
lack of experience when auditing against new criteria and under new
objectives.
The concept of audit reliability pertains to several related concepts that are
essential in determining audit effectiveness. Three of them, namely audit
availability, risk and maintainability are discussed in the following section.
Availability
Availability (sometimes referred to as ``operational readiness'') is the
probability that the audit system will adequately function or be prepared to
function in a given instant. It is a complex measure, involving both reliability
and maintainability (Ebeling, 1997). This characteristic is very important in
determining the ability of the audit system and/or program to perform required
audits when needed. In a procedural audit system, audits are performed
according to a set schedule and on the basis of fixed procedures. Therefore, it is
not difficult to determine when and where the audit will need to be conducted.
However, in a risk- or priority-based system of interrelated audits, audits can
be carried out on very short notice, thus making it harder to plan for
availability. Planning for availability involves estimations of the risk of failure
and the risk of the system being inoperable when required. While the first risk
involves reliability, the latter is related to maintainability.
The probability that the audit will result in an incorrect finding (result) is
commonly called the audit risk (CSA, 1994). Accounting literature contains a
number of models addressing audit risks (e.g. see Woodhead, 1997; Friedlob
and Schleifer, 1999; Shailer et al., 1998; Colbert and Alderman, 1995). Most of
these models can be applied to the realm of quality auditing. For instance,
accounting auditors commonly address inherent, control and detection risk,
and obtain the total audit risk by multiplying these three elements (Friedlob
and Schleifer, 1999). In quality auditing, inherent risk is the likelihood that an
error (or a noncompliance with specified requirements) exists within a quality
system, product, service or process (Willborn, 1996). Control risk is the
probability that this error is not detected by management controls or an
internal quality audit. The detection risk is the probability that an external
quality audit fails to identify the error. An opposite situation may occur where

Quality
assurance and
audit systems
695

IJQRM
17,6

696

a non-existent error is assumed to be real by both internal and external

controls, in which case false rejection of a compliant management system will
occur (Woodhead, 1997). When applied specifically to address audit failures
and availability, this audit risk model may function in the following manner.
Inherent risk arises when an individual audit, audit program or system has
failed to adequately function (e.g. an auditor is not qualified or competent).
When the audit management fails to recognize this error, the control risk is in
play. Finally, the probability that the failure will pass through an audit of the
audit system or program is the detection risk. Audit risks mostly arise from
lack of information (Friedlob and Schleifer, 1999) or the application of
inadequate audit methodologies (Willborn, 1996). Therefore, careful application
of estimation, discovery and acceptance sampling techniques, paired with the
adequate use of flowcharts, checklists, and computer-aided methods (e.g. see
Glaccum, 1991), should reduce the audit risk and increase audit availability.
For a detailed outline of these and other audit methodologies, the interested
reader is referred to Willborn and Cheng (1994) and Hill et al. (1962).
Maintainability is the ability of the system to be maintained, or the ease
of returning the system to the operational state after it has failed. In simple
terms, audit maintainability answers an important question: ``What
happens when the audit fails?'' It is related to availability due to the fact that
after an error is detected, the audit system should be returned to the
functional state as soon as possible in order to be available to perform its
intended function when required. In most cases, after an error is found,
corrective actions must be performed to remove the error and avoid related
consequences. For instance, if the auditor used a smaller sample size than
required, which resulted in a questionable acceptance of the quality system
compliance, a corrective action may require an adequate determination of
the sample size and confidence level in the audit. Naturally, the sooner an
error is detected, the easier it is to correct it, which in turn makes the audit
system maintainable. Compliance with the auditor code of ethics and due
care standards play a significant role here (e.g. see Ecton et al., 1996; Grant
et al., 1996). Audit maintainability can also be increased with the adequate
use of audit techniques and methodologies, inducing the prevention, rather
than correction of possible errors.
Suitability
We have mentioned earlier that the role of a quality auditor is to assess
compliance of a quality system with audit criteria, as well as to examine the
suitability of the system to achieve set quality objectives. When quality and
effectiveness of the audit are evaluated, the same two roles should be
performed. Compliance of the audit system with appropriate auditing
guidelines, such as ISO 10011 (1990) and ISO 14010/11/12 (1996), is a good
indication of the audit quality. Nevertheless, the evaluation of effectiveness
should extend beyond compliance to the measurement of the suitability of

the auditor and the audit to achieve set audit objectives. Audit suitability
refers to the ability of the audit system to adapt itself to changes in audit
scope, requirements, criteria, or suggested practices, and still achieve the
overall audit policy. For instance, if the audit criteria have changed from the
ISO 9001 standard to the automotive counterpart of QS-9000, does the
auditor have the ability to audit against this newly introduced standard in
his/her company? Or if an audit program was designed to assess a quality
system, its effectiveness in doing the same for an environmental
management system, i.e. after a change in the scope, could be questioned.
Such changes in scope and criteria may even involve the addition of new
concepts that were previously thought to be outside the auditor's realm. For
example, Hunt (1997) suggests the inclusion of the examination of corporate
culture in the auditing process. ``Organizational culture'' is a fairly evasive
term, especially for a technically oriented auditor. While an auditor must be
familiar with the prevailing language, customs, attitudes, and other aspects
of the manner in which an audited organization operates, if for no other
reason than to establish and maintain proper communication with the
auditee, it is unclear how such ``cultural analysis'' would be incorporated in
the set of relatively straight forward audit criteria (such as an ISO 9001
standard). For instance, what would be the objective evidence the auditor
should look for, and how would such evidence be analyzed against audit
criteria to form audit findings? The emergence of new management system
standards, which for an auditor means a new set of audit criteria, in such
areas as social responsibility (SA 8000) and occupational health and safety,
is bound to further complicate these matters.
As we can see from these examples, although audit suitability depends on
many audit elements, the onus is usually on the auditor (or the audit team) and
his/her qualifications and competence. The auditor must assess his/her
qualifications and competence, for instance whether he/she is competent to
conduct both quality and environmental management system audits, in order
to achieve adequate suitability. The evaluation of auditor qualifications and
competence follows the chain of concepts performance qualification/
competence auditor (Figure 6). Competence may be defined as the
demonstrated and recognized ability of a qualified auditor to consistently
achieve audit objectives to the satisfaction of client and auditee, while
qualifications refer to the auditor's education, training and experience. This
implies qualification as a pre-condition for competence. Relevant principles and
methods for the competence evaluation, demonstration, recognition and
development are illustrated in Table II, representing a possible framework for
quantifying audit suitability.
Effectiveness revisited
Audit system effectiveness is a complex issue involving the ability of the
auditor to conduct an audit free of errors, ready to operate when required, and

Quality
assurance and
audit systems
697

IJQRM
17,6

698

Figure 6.
Concept diagram related
to auditor competence

suitable to achieve set objectives. If any one of these three elements is missing,
the system effectiveness will plunge. Thus, auditors and audit managers must
design and implement their systems to maximize the reliability, availability
and suitability of all system elements, while still providing adequate confidence
to clients and auditees. In this effort, trade-offs are inevitable. For instance,
regular audits of the audit system will increase reliability, but may in turn
reduce availability of auditors to perform other required audits. Nevertheless,
audit quality and effectiveness are factors that nobody can afford to lose.
Another related point is audit cost-effectiveness. Not only does the audit
have to be functionally effective, it also must justify the cost of performing the
audit activities. Cost-effectiveness relates to the ability of the audit to achieve
objectives while minimizing the associated spending. When an audit is
designed and conducted in a manner that ensures its suitability, availability
and reliability, reduction of cost comes as a natural consequence. For instance,
an auditor can spend a lot of time (and time is money) auditing a trivial issue
that does not contribute in any significant way to the auditee's business,
thereby reducing both the audit functional and cost-effectiveness. A good way
to judge the cost-effectiveness of an audit is to evaluate its contribution to the
improvement of the organization's bottom-line (i.e. profit), and reduction of the
overall costs, such as waste, scrap, the costs related to defective products, and
so on. While the aim of this paper was to discuss the audit functional
effectiveness, it is recognized that the analysis of various methods of
evaluating cost-effectiveness would be a very interesting and useful area of
further research.

Evaluation principles
Application of relevant auditing principles
Competence development and maintenance
Conducted by competent evaluator
Conducted on request of auditor or audit management
Performed periodically for all auditors
Provision of follow-up training as required
Evaluation against relevant and known criteria
Application of proper evaluation methods
Report of the result to the evaluated auditor

Quality
assurance and
audit systems
699

Demonstration
Based on sound principles
In the entire auditing process
For specific audits normally conducted by the auditor
In audit assignments accepted by the auditor
Prior to an audit assignment
During and after training and professional development programs
Methods
Direct observation of auditor's performance during several relevant audits
Application of formal, prepared, and known checklists
Review of audit plans and reports prepared by the auditor
Monitoring specific competence indicators (criteria) under known conditions
Requesting reports from evaluators who are known and accepted by the auditor
Recognition
Auditor personally (self-assessment) based on performance evaluation
Client when being able to monitor the auditor's performance
Auditee when directly participating in the audit by the auditor
Qualified audit team leader supervising the auditor in several relevant audit assignments
Peers of the auditor in several relevant audit assignments
Qualified audit program manager having conducted a proper formal evaluation
Development (evaluation after each stage)
After basic auditor qualification start as auditor-in-training
Join an audit team as an auditor conducting audit assignments with increasing complexity
Conduct several complete audits
Decide future career as an auditor; i.e. generalist or specialist
Conduct audits in the selected area under various relevant conditions
Prepare, qualify for, and conduct audits as the audit team leader
Continue professional development

Conclusions and further research

The last decade of the twentieth century brought many developments in the
domain of quality auditing. An international quality audit standard, namely
ISO 10011 (1990), was issued at the beginning of the decade, and since then has
been widely used for guidance in conducting individual audits, managing audit
programs, and assessing auditor qualifications. ISO 9000 quality system

Table II.
Auditor competence

IJQRM
17,6

700

registrations have topped 300,000 worldwide, causing at least that many

internal quality audit programs to develop. Riding the wave of harmonization
and integration of management system standards, an integrated quality and
environmental management system audit guideline is currently being
prepared. With success, however, drawbacks inevitably come about. Many
multinational companies have vowed to declare self-compliance to ISO 9000
standards by the year 2005, paving the way for repudiation of external audit
and registration schemes (Zuckerman, 1998). Concerns are being raised about
the consistency of auditors and audit results, overall quality of both external
and internal audits, and their ability to achieve improvement objectives.
This paper argues that part of the answer to such concerns lies in the
establishment of meaningful quality assurance in individual quality audits,
audit programs and audit systems. The ISO 9000 framework was
particularly discussed as a possible solution. In addition to providing
confidence in quality externally, audit systems must be managed for
internal effectiveness. A model for measuring effectiveness of audit systems
was adapted and presented. Audit reliability, availability, maintainability,
suitability and risk represent the concepts that were addressed in most
detail. Particular attention was paid to the prevention and correction of
possible audit failures, as well as the adequate qualifications and
competence of auditors.
Further research is suggested in the domain of statistical sampling
techniques in quality auditing, modeling of audit maintainability and
sustainability, as well as the use of quality assurance schemes for auditing
in smaller companies. Research and development of an audit risk model for
quality auditing would be a worthy exercise. Particular attention should
also be given to the development of a quantitative method for assessing
audit effectiveness through reliability, availability, and suitability
measures. While in technical systems it is relatively simple to empirically
model these measures of equipment effectiveness, e.g. by running a
reliability test of a number of identical components until failure,
accomplishing such a task for audit systems may prove to be challenging.
This is because various constraints and factors that do not come into play in
the analysis of technical systems, certainly have a significant impact on a
combined human-hardware-software system such as auditing.
Psychological, procedural and organizational factors are just three
examples that must be taken into account here. Ultimately, whenever a
system includes people as a crucial resource for accomplishing its tasks, we
must deal with the inherent variability that exists in the system. For
instance, we can test 100 identical fuel pumps until failure, record the times
of failure for each, and calculate appropriate reliability characteristics, such
as the mean time to failure (MTTF) or the variance of the estimated failure
distribution. But can we do the same with 100, or for that matter even ten,
quality auditors? Probably not, at least not in the same manner. What we

can do, however, is attempt to localize the most important factors that
influence audit reliability, availability and suitability, and test their effects
for a variety of settings (e.g. using design of experiments). Subsequently, we
could model the associated probabilities, and finally obtain the aggregate
score for audit effectiveness using the framework presented in this paper.
An empirical study addressing these issues in further detail is suggested,
and would contribute to the research not only in quality auditing, but also in
financial (internal), health and safety, environmental, maintenance and
other areas of auditing practice.
References
ANSI/ASQC (1986), Q1 Generic Guidelines for Auditing of Quality Systems, American Society for
Quality Control, Milwaukee, WI.
ASQ (1999), Certified Quality Auditor: Body of Knowledge, American Society for Quality,
Milwaukee, WI (available at http://www.asq.org/standcert/certification/cqa.html#cqabok)
Beeler, D.L. (1999), ``Internal auditing: the big lies'', Quality Progress, Vol. 32 No. 5, pp. 73-8.
Bishara, R.H. and Wyrick, M.L. (1994), ``A systematic approach to quality assurance auditing'',
Quality Progress, Vol. 27 No. 12, pp. 67-70.
Burr, J.T. (1997), ``Keys to a successful internal audit'', Quality Progress, Vol. 30 No. 4, pp. 75-7.
Busta, B. and Weinberg, R. (1998), ``Using Benford's law and neural networks as a review
procedure'', Managerial Auditing Journal, Vol. 13 No. 6, pp. 356-66.
Coetzee, J.L. (1998), Maintenance, Maintenance Publishers, Hatfield.
Colbert, J.L. (1996), ``International and US standards audit risk and materiality'', Managerial
Auditing Journal, Vol. 11 No. 8, pp. 31-5.
Colbert, J.L. and Alderman, C.W. (1995), ``A risk-driven approach to the internal audit'',
Managerial Auditing Journal, Vol. 10 No. 2, pp. 38-44.
CSA (1981), Quality Audits, (Can3-Q395-81), National Standard of Canada, Canadian Standards
Association, Etobicoke, Ontario.
CSA (1994), Guidelines for Environmental Auditing: Statement of Principles and General
Practices (Can-Z751-94), National Standard of Canada, Canadian Standards Association,
Etobicoke, Ontario.
Druckman, D. (1997), ``Auditors' contribution to failed audits'', Proceedings of the 6th Annual
Quality Audit Conference, Los Angeles, CA, pp. 13-18.
Ebeling, C.E. (1997), An Introduction to Reliability and Maintainability Engineering, McGraw-Hill,
New York, NY.
Ecton, W.W., Houston, M. and Reinstein, A. (1996), ``Improving the due professional care
standard'', Managerial Auditing Journal, Vol. 11 No. 1, pp. 7-13.
Friedlob, G.T. and Schleifer, L.L.F. (1999), ``Fuzzy logic: application for audit risk and
uncertainty'', Managerial Auditing Journal, Vol. 14 No. 3, pp. 127-37.
Gardner, E.R. (1997), ``Applying ISO 9000 principles when auditing'', Logistics Information
Management, Vol. 10 No. 5, pp. 208-13.
Glaccum, J.F. (1991), ``Combining multilevel sampling plans and personal computers'', Quality
Progress, Vol. 24 No. 9, pp. 75-8.

Quality
assurance and
audit systems
701

IJQRM
17,6

702

Grant, J., Bricker, R. and Shiptsova, R. (1996), ``Audit quality and professional self-regulation: a
social dilemma perspective and laboratory investigation'', Auditing A Journal of Practice
and Theory, Vol. 15 No. 1, pp. 142-56.
Harral, W.M. (1998), ``The roles and inter-relationships of risk management and quality
management systems auditing'', Proceedings of the 7th Annual Quality Audit Conference,
Louisville, Kentucky, pp. 150-61.
Hill, H.P., Roth, J.L. and Arkin, H. (1962), Sampling in Auditing, Ronald Press, New York, NY..
Hirzel, R.C. (1998), ``A systems approach to auditing systems'', Proceedings of the 7th Annual
Quality Audit Conference, Louisville, KY, pp. 50-5.
Hunt, J.R. (1997), ``The quality auditor: helping beans take root'', Quality Progress, Vol. 30 No. 12,
pp. 27-33.
ISO 9001 (1994), Quality Systems Model for Quality Assurance in Design, Development,
Production, Installation and Servicing, International Organization for Standardization,
Geneva.
ISO 9001 (2000), Quality Management Systems Requirements: Committee Draft 2, International
Organization for Standardization, Geneva.
ISO 10011 (1990), Guidelines for Auditing Quality Systems: Parts 1, 2 and 3, International
Organization for Standardization, Geneva.
ISO 14010 (1996), Guidelines for Environmental Auditing General Principles of Environmental
Auditing, International Organization for Standardization, Geneva.
ISO 14011 (1996), Guidelines for Environmental Auditing Audit Procedures-Part 1: Auditing of
Environmental Management Systems, International Organization for Standardization,
Geneva.
ISO 14012 (1996), Guidelines for Environmental Auditing Qualification Criteria for
Environmental Auditors, International Organization for Standardization, Geneva.
Karapetrovic, S. and Willborn, W. (1998), ``Integrated audit of management systems'',
International Journal of Quality & Reliability Management, Vol. 15 No. 7, pp. 694-711.
Karapetrovic, S. and Willborn, W. (1999), ``Holonic model for a quality system in academia'',
International Journal of Quality & Reliability Management, Vol. 16 No. 5, pp. 457-85.
Karim, K.E. and Siegel, P.H. (1998), ``A signal detection theory approach to analyzing the
efficiency and effectiveness of auditing to detect management fraud'', Managerial Auditing
Journal, Vol. 13 No. 6, pp. 367-75.
Peters, J. (1998), ``Some thoughts on auditing'', TQM Magazine, Vol. 10 No. 1, pp. 4-5.
Rezaee, Z. (1996), ``Improving the quality of internal audit functions through total quality
management'', Managerial Auditing Journal, Vol. 11 No. 1, pp. 30-4.
Russell, J.P. (Ed.) (1997), The Quality Audit Handbook, American Society for Quality Control
(ASQC), Quality Press, Milwaukee, WI.
Russell, J.P. and Regel, T. (1996), ``After the quality audit: closing the loop on the audit process'',
Quality Progress, Vol. 29 No. 6, pp. 65-7.
Shailer, G., Wade, M., Willett, R. and Yap, K.L. (1998), ``Inherent risk and indicative factors: senior
auditors' perceptions'', Managerial Auditing Journal, Vol. 13 No. 8, pp. 455-64.
Stratton, J.H. (1995), ``Auditor consistency: what improvements are underway?'', Proceedings of
the 49th Annual Quality Congress Transactions, Cincinnati, OH, pp. 1064-5.
Vujanovic, N. (1990), Theory of Technical Systems Reliability, VINC, Belgrade.

Walker, A.J. (1998), ``Improving the quality of ISO 9001 audits in the field of software'',
Information and Software Technology, Vol. 40 No. 14, pp. 865-9.
Willborn, W. (1990), ``Dynamic auditing of quality assurance: concept and method'', International
Journal of Quality & Reliability Management, Vol. 7 No. 3, pp. 35-41.
Willborn, W.O. (1993), Audit Standards, A Comparative Analysis, Quality Press (ASQC),
Milwaukee, WI.
Willborn, W. (1996), ``Report on audit methodologies and other audit standards'', ISO/TC176/
SC3/WG7/N26, International Organization for Standardization, Geneva.
Willborn, W. and Cheng, T.C.E. (1994), Global Management of Quality Assurance Systems,
McGraw Hill, New York, NY.
Woodhead, A.D. (1997), ``The other audit risk: the impact of false rejection on audit planning'',
Managerial Auditing Journal, Vol. 12 No. 1, pp. 4-8.
Zelenovic, D. and Todorovic, J. (1990), Effectiveness of Mechanical Systems, Naucna Knjiga,
Belgrade.
Zuckerman, A. (1998), ``58 multinationals question ISO 9000 registration'', Quality Progress,
Vol. 31 No. 8, pp. 16-21.

Quality
assurance and
audit systems
703