Evaluating Operational

Performance

INTRODUCTION
In Chapter l, we introduced the concept ofthe 3 Es ofeffectiveness, efficiency and
economy using the model and definitions shown in Figure 2.1.
f,

ACTUAL IEfficiency] ACTUAL

(the ratio ot actual
INPUTS OUTPUTS
inputs to actual
outruts) I
I

IEconomy] lefecti,lrenessl
(the ratiobetween (the ratio ol ac-tual
planned inputs and ouFrrts to danned
actual inputs in ouFrts)
terms ol unit costs) I

I
I

PLANNED
PI.ANNED OUTPUTS
INPUTS [i.e.Objectives]

Effectiveness means "doing the dght things" i.e. achieving

objectives.
-
Efficiency means "doing them well" for instance with good
systems which avoid waste and rework.
-
Eanomy means "doing them cheaply' with, for instance, unit
-
costs for labour, materials, etc. being under control.

Figure 2.1 The three E's of effectiveness, efficiency and economy

 EVATUATING OPERATIONAL PERFORMANCE

In this chapter we build on this premise and explore the related subjects of
measuring operational performancc and value for money auditing.
trn order to bring all the relevant issucs closer to home, we conclude this chapter
by applying thc principles of performance measurement to the internal auditing
function itself.

PRODUCTIVITY AND PER,FORMANCE MEASUREMENT SYSTEMS

Overview

Organisations are likely to have in place a number of key performance measures,

so urs to, among other things, assess the achievement of their objectives and.goals,
assess their progrcss, and compare relative performanoe (for example, over time).
The nature and form of such measures will, of course, vary between types of
organisation and inded specific specialised forms of measurement may apply in
certain industries or scctors. However, there are a number of general measures of
effectiveness, efficiency and economy which usually apply universally and we shall
look at some examples later in this chapter.
Measurement methods can be applied in order to identify whether there is any
initial potential for improvement, and then subsequently used to monitor that the
required levels of performance are maintained. The need to apply effective and
realistic performance measurement methods is often generated as a by-product of
fundamenial change processes where, for example, an organisation is re-focusing
its stratery and position.

The Audit lmplications for Measurement

During thc course of a review of an operational area, the auditor is often faced
with the nccd eithcr to set the review findings into an appropriate context, or to
indicatc tbc performance of the arca under review against thc criteria previously
establishcd by management.
In most cases, it is prcferable to utilisc the measurement standards and criteria
put in plaa by management as this results in the auditor using a oommon dnd
compatiblc language when communicating results and points of concern.
Conversely, if the auditor chooses to us€ a new, alternativc or perhaps radical
form of performanae measure, this may influence or jeopardise management's
view of the auditor's findings. This is not to say that auditors should only adopt
the prevailing measurement criteria established by management, as therc may be a
compelling reason for introducing anothcr objective form of performance
ass€ssment in some cases. Whatever the form of measurement applied, its use
must be founded on both a@urate and reliable data and a provbn method,
otherwise thc credibility of internal audit will suffer.
Although it is important to establish a reliable and meaningful vocabulary for
the measureinent of performance in key operations, auditors must not losc sight
of the fact that iuch measures can only point to potential areas of improvement
and do not of themselves offer solutions.l Assuming that the conclusions drawn
 rHE OPERATIONAL AUDITING HANDBOOK

from the review ofsuch criteria are accurate and relative, they can then be used to
frame and support audit recommendations and the appropriate corrective
action(s).
In their use of performance measurement, auditors should be careful not to
supplant management's use and interpretation of the same criteria. On the one
hand, it may be legitimate for an auditor to investigate further the lack of
management response to an adverse measurement indicator, but this does not
necessarily mean that management has abdicated their basic responsibility for
monitoring and control. This underlines a basic truism, in that measurement
data is provided for interpretation and unless there is a formal measurement
protocol in place, there may be the potential for differing conclusions to bc
drawn from the same data. This stresses the importance of formally
ctablishing, for the organisation, a performance measurement policy and
framework so that all concerned are clear about the nature of the data and
how to usc it in practice. Additionally, the creation and communication of
corporate targets and goals can remove (or at least contain) some of the
anbiguity associated with the required level of performance and expected level
of associated achievement.
Each operational audit review project will present the auditor with a
challenge to identify the most appropriate and meaningful performance
measures to utilise, whether or not such criteria are already applied within
the organisation.

Example Performance Measures

When establishing performance measures, it is logical to structure them on a

hierarchal basis with the macro-level indicators being broken into more detailed
(micro-level) measures relative to specific areas or suMivisions of either the
operations or organisation. This should be borne in mind when considering the
following example pcrformancc meznurcs. The examples in this section were drawn
from Internal Auditing by Chambers, Selim and Vinten (1987), pages 361-2.

Workload/Demand Pertormance Measures

Indicate thc volume of output, whether services, products or other, and when
linked to mcasures of input of resources, give useful information on quality or
quantity matters.

Exarnples:

o Number of users.
o Number of units produced.
o Number bf books in a library.
r Percentage offirst class degrees in a university.
 €VATUATING OPERAT]ONAL PERFORMANCE I5

Economy Pertormance Measurs

These may hiShtight waste in the provision of rcsourccs indicating that the same
resources may bc provided more cheaply or that morc enterprise may be
conducted at thc samc cost.

Exonples:
o Cost of actual input in comparison with planned input.
o Cleaning costs per hour worked.
r Maintenance costs per unit area.
o Cost of thc finance function pcr 100 staff.
o Cost of the chief exccutive's department per l00O clients.

Effici e n cy Pe rfo rmancq Measu res

These may higblight potential opportunities to convert given resources to end

product with less waste. Many performance measures will point to either
uncconomic or ineffcient practices, or both. It is often not possible to distinguish
between one and the other.

Examples:
o Ratio of actual input to actual output.
o Breakdown per production day.
o Accidents at work per 1000 pcrsonnel.
o Dcgree su@ess in comparison to school examinatioo grades.

Effectiveness Pertormance Measure

These performanoe measures focus on how objectives arc bcing achicved-

regardless of cconomy, etfciency or equity (exccpt where the objectives relate
specifcally to cconomy, ef[ciency and equity).

Exatnples:
o Actual output in comparison to planned output.
o Degree su@ess (in a college or university).
o Rescarch o.utput per 100 research staff.
o Ratio of customer complaints to sales.

EquW Performance Measures

. These performanbe measures draw attention to unfairness or potential social

irresponsibility in terms of corporate policy and practice.
 THE OPERATIONAT AUDITING HANDBOOK

Exarnples:

o Departmental grant per member of staff.

o Number of library books per category of user.
o Proportion of female employees.
o Proportion of disabled employees.

VATUE FOR MONEY (VFM) AUDITING

Valuc for money auditing takes account of the 3 Es. It frequently makes extensive
use of performance indicators in the form of ratios and other statistics to give an
indication of value for money-espccially whcn trends are explored in these
performance indicators over time, or variations in performance are identified and
explained between different operating units.
The term value for money is often applied to public sector spending in the UK,
where there is an implied obligation placed on public bodies to ensure that they
obtain and provide services on the most economic grounds. This process
invariably involves elements of competition where cost comparisons are made
between parties being invited to supply goods and services. For exampte, many
services within UK local government have been put out to tender in order to
obtain the "best deal", and very often this tendering process has also included the
internal department or function that had previously been supplying the scrvice.
This striving for procurement on a least casl basis appears to b€ very logical
and represent common sense, especially where the expenditure of public funds is
involved. However, it is equally important to consider whether the porcntial
scrvice provider (or supplier or contractor) can meet the required quality and
pcrformance standards as well. Therefore, any consideration of value for money
must take in quality and performance achievement factors as well, as Oere may be
serious commercial or operational implications if the relevant services/goods arc
not up to a given standard.
Value for moncy auditing will involve the assessment of an appropriate range of
performance measurement criteria. It could be asked that unless management
have clearly established their own basis for measuring and assessing the supply of
goods and the provision of services, why did they embark on the pr(rcess in first
place? In other words, what was their driving motivation in cither fulfilling the
requirements or seeking alternatives?
In both the management and audit assessment of matters of value for money,
the usual approach is to make comparisons with a range of options or possible
solutions to thc principal problem. These comparisons should be conducted as
scientifically and objectively as possible and utilise appropriate measurement
means. This part of the process begins with realistically idintifying all the
practical options and alternatives (perhaps including doing nothing at alt).
In a more formal environment (for example, where acquiring new computing
facilities) it may be necessary and desirable to go through a detailed feasibility study
as part of an overall project appraisal process. This can then incorporate the
appropriate cost and performance comparisons which underline the determination
 fVALUATING OPERATIONAI PERFORMANCE l7

of value for money. In such scenarios, it is important that the auditor is content with
the chosea assessment mechanism and measurement criteria so that, taken together,
the appropriate reassurance can be derived that the process is sound and accuratc. In
somc instances it may be neccssary for the auditors to recommend improvements in
thesc areas to add value to the proccss, whilst avoiding usurping managernent's
ultimate responsibility for their system.
Whethcr or not a formal procedure is in place to dctermine generally the
achievement of value for money, the internal audit function may be required (or
indeed obliged) independently to assess such matters on behalf of management.
Auditors should always avoid taking on activities which should, in the fust place,
bc the rcsponsibility of management. Howcver, wherc internd audit has a
legitimate role to play, auditors should endeavour to identify all the probable
options and the most suitable basis on which they should be measured and
assessed in value terms.
In order to avoid any potential problems at the conclusion of their assessment,
auditors should consider discussing their proposed assessment and measurement
critcria with managernent at the outs€t, and furthcrmore to obtain the agreemcnt
of uranagement on the applied methodology. In certain sectors and industries,
recognised criteria may already exist and so it may not be necessary for auditors
to develop their own process.

BENCHMARKINC

Benchmarking can be defined simply as a comparison of one's own pcrformance

in a specific area with that applied by others in compatible circumstances. As a
technique it is founded on the premisc that therc may be viable alternative ways of
performing a process and fulfilling a requirement.
For a benchmarking cxercisc to bc meaningful, it is necessary to understand
fully thc existing prooesses, systems and activities as a firm basis for subsequent
comparison with erternal poins of refcrene (sueh as industry or professional
standards). This process of rcalisation often incorporatcs the establishment of
critical suocess factors for an operation (or part thereof). Thc principal objectives
of benchmarking are likely to include
o maintaining a competitive advantage in the appropriate market
o establishing currcnt methods, best practicc and related trends
o ensuring the future survival of the organisation
r maintaining an awareness of customer expectations (and being ablc to address
them)
o ensuring that the organisation has the appropriate approach to quality issues.

The focus of a benchmarking exercise can be varied in relaiion to the

fundamental justification and objectives of the process. For example, if the
objective was primarily to examine the existing processes within the organisation
as a means of identifying cornmon factors and best practices to apply throughout
the company, thti focus could be said to be downward and inward. Alternatively,
if the organisation was seeking views on the strengths and weaknesses of
 18 THE OPE,RATIONAL AUDITING HANDBOOK

competitors this is outward looking in nature, and could involve one-to-one

competitor benchmarking, industry benchmarking, or best-in-class benchmarking
methods.
trnternal audit departments can often benefit from participating in benchmark
comparisons with other audit functions; such involvement can coitribute to their
understanding of:

r the internal auditing trends and practices as applied by the companies surveycd
o the implications and porential of the nnoings foi the p"iti"ip"rrt,, o*o
organisation
r the validity of the participant's own stance on internal auditing in relation to
that apparent from the survey data.

Involvement in such exercises will enable participants to take a view of the need
for change or review of their own organisation's approach to internal auditing in
ligbt of the survey data.
of course, benchmarking is not an end in itself, but rather one platform used to
idgntify and subsequently launch the required or neoessary proorr", of change
within a department, function, activity, process or organisatiron.

EFFECTIVE MEASURING OF INTERNAT AUDITING'S CONTRIBUTTON

TO THE ENTERPRISE'S PROFITABILITY

lntroduction

At a time when internal auditing is being challenged by outsourcing alternatives

and by other methods of reviewing managerial effectiveness, it is particularly
important to be able to measure its contribution to the enterprise's piofitabilitl,.
In this section wc take a look at performance measures forlnternal auditingi
mealinres of inputs (economy), proccss (efficiency) and espccially outp-uts
(effectiveness). Appropriarc spccific measures are recommended. -doing -this
In
we will be identifying the kcy aspects of internal auditing which need to be focused
upol in order to improve internal auditing's contribution to the enterprise's
profitability. We consider the diffculties of reaching retiable measures of internal
audit perfonnance, and distinguish between qualitative and quantitative
measurcs. we suggest a value-for-money approach to assessing internal audit
performance.
It will be n@essary to identify the categories of performan@ measures which
may be used to evallllte internal audit performance and the strengths and
weaknesses of each. we give advice on their interpretation. we place the
measurement of internal audit in context with: (a) the general business
environment, (b) professional standards for internal auditing, and (c) good
management practice on planning and control. Lastly, we present a parti-uhr
approach to'the use of performance measures in value for money auditing which
may be applied to assessing internal auditing performance
 €VALUATING OPERATIONAL PERFORMANCE 19

The General Business Environment-The Recession

flistorically the growth of internal auditing as a business service has been

"counter cyclical", though there are indications that this may not be so during the
present recession. By this we mean that in the past internal auditing has dcveloped
rnost strongly during times of cconomic constraint. It may bc that dircctors and
managements considcr that investmcnt in internal auditing is particularly
important in constrained timcs as an antidote to the control risks sometimes
associated with stringent cost cutting. Or it may be that managements have turned
their attention away from financial, accounting and operational control (to which
internal audit can contribute) when extra profits have been more easily secured by
burgeoning sales.
' Whethcr or not internal audit prospers in constrained times, it certainly
behoves internal audit to be able to demonstrate that its function is cost-effective
and is managed so as to maximise its cost-effectiveness. Where internal auditing is
not a mandatory requirement by statute law or by regutation, there is added
pressure for audit to be able to demonstrarc its worth.2
There are indications that managements are now placing internal auditing
under a microscope with the intention of determining whether it pays its way.
Internal auditing is a costly service to run. Each productive day ofinternal auditor
time may cost typically some €400--*learly sizable amounts of profits from sales
are needed to resource an internal audit function.

Competitive Tendering

Even where internal auditing is a mandatory requirement the current vogue for
compctitive tendcring makes inrcrnal auditing a prime candidatc for market
testing and contracting out.3 Established in-house internal auditing functions find
thcy arc rcndering compctitivcly against firms of public accountants, consultants
spocialising in intcrnal auditing and other in-housc internal auditing functions
who have bccn given thc frccdom to tendcr for extcrnal work. To win the
contract, thesc outside parties may bc willing to bid at marginal cost-+specially if
they have surplus capacity. Bids at €150 (compared with the f400 quoted abovc)
per day are not unknown.
The many arguments for and against an cnterprise contracting out its intcrnal
auditing are summarised below. An aspcct that has been largely overlooked is that
performance me:rsures for internal audit are particularly important for providing
the means of establishing performance-related contracts for internal audit
provision, and for monitoring its ongoing provision after thc contract has been
let.
with the bias being towards accepting lowest cost bids, it is particularly
important to devise and use internal audit performanc€ measures which focus
upon outpuls first, process second and tnputs last-this categorisation is
followed in this discussion. Each of these three is, of course, important. senior
general managembnt and the board responsible for contracting out decisions
should ensure that this sort of internal audit performance monitoring is in
 rHE OPEMTIONAL AUDITING HANDBOOK

place. In-house heads of internal audit can influence management and the
board towards this and, in so doing, should be maximising their own
opportunities for securing into the future the internal audit work for their in-
house internal audit departrnents.
If management and the board allow daisions on letting contracts for internal
audit work to be made on price alone, rather than value for money, they are
acting irresponsibly. Decisions on price alone betray a lack of commitment to the
value of internal auditing-perhaps merely a resignation to the provision of a
skeletal internal audit service due to statutory or regulatory obligations. Even in
enterprises with acute cash flow problems, decisions on price alone are unjustified
as it is espccially important that such businesses maximise value.
Since (a) mandatory obligation to have internal auditing and (b) cash flow
difficulties often come together within the public sector, it is within that sector
that wc are currently experiencing most pressure to contract out internal auditing
on price grounds alone.

Potential advantages of contracting out internal auditing

l. The business can more readily vary its spend on internal auditing, according
to what it can afford, from time to time.
2. The contractor is motivated to perform well and can be held to account for
that performance.
3. The provider can be changed more easily.
4. The servicc may be provided at a lower price.
5. An external provider may have a wider understanding as to how other
enterprises tackle similar business issues.
6. An external provider may have more extensive audit support resour@s to
draw upon.
7. An external provider may be able to develop the enterprise's own staff.
8. The actual and perceived independence of au cxtemal provider may bc
greater-leading to more confidence in the results of the audit work.

Potential advantages of in-house internal audit provision

l. A deeper grasp of the enterprise's affairs.

2. A finer adjustment of internal audit emphasis to the enterprise's needs.
3. "On the spot" responsiveness to management and the board; better able to
take on unplanned work.
4. A training ground for the future senior executives.
5. Confidentiality.
6. More likely to have a genuine internal audit orientation as distinct from an
exernal orientation.
7. Unable to "walk away".
 EVALUATINC OPERATIONAL PERFOR,\,1ANCE

Professional Standards for lnternal Auditing

The Standards of The Institute of Internal Auditorsa have a specific standard that
"the chief internal auditor should establish plans to carry out the responsibilities
of the internal auditing department". Supporting guidelines require that "these
plans should be consistent with the internal auditing department's charter and
with the goals of the organisation", and that "the goals of the iaternal auditing
department should be capable of being established within specified operating
plans and budgets and, to the extent possible should be measurable. They should
be accompanied by measurement criteria and targeted dates of accomplishment."
These Standards also require that "The chief internal auditor should establish and
maintain a quality assurance programme to evaluate the opcrations of the internal
auditing department", and this is elaborated ia, inter alia, the statement that "A
key criterion against which an internal auditing department should be measured is
its charter". Quality assuran@ is seen as being preservcd by means of supervision,
intemal reviews and external reviews. These rcviews are "performed to appraise
the quality of the department's operations" and should be "structured to evaluate
the degrec of compliancc with the Standards" themselvcs.
So it is clear that the Institute of Internal Auditors eonsiders that there should
be performan€ measures of an internal auditing department and that these
should include an evaluation of compliance with the Charter of the department
and also with the Standards of the Institute; they should also include a
measurement of the achievement of target dates.
The UK's Consultative Committee of Accountancy Bodies (CCAB) is less
forthcoming. Their guidance to their memberss mercly says: "The head of internal
audit should establish arrangements to evaluate the performance of the internal
audit unit. He [sfcJ may also prepare an annual report to management on the
activity of the internal audit unit in which he [srl gives an assessment of how
effectively the objectives of the function have been met."

Categories of Performance Measures

Measures of intcrnal audit pcrformancc have tended to focus upon inpul and
process rathet than upon output. Auditors will understand the association between
thesc three and economy, effciency and effectivene.rs, respectivcly, as illustrated in
Figure 2.1. The greatest challenge is now to develop a rangc of measures which
throw light upon internal audit effectiveness (i.e. output measures).

Quantitative, Quasi-quantitative and Qualitative Measures

Another way to categorise performancc measures is according to whether they are
quantitative or qualitative. In reality each performance measure can be conceived
as being somewhere on a gradation between the extremes of objective
(quantitative) and subjective (qualitative). An important characteristic of
quantitative performance measures is that their measurement is objectively
THE OPERATIONAL AUDITINC HANDBOOK

determined. Yet even in a very clear+ut case of a quantitative performance

measure6 subjectivity is not avoided as the selection and design
'performance measure will have been based upon a judgernent that
that of
provided lt
relevant guidance on relative internal audit performance, ind the interpretation
of
the resulting data also will be very subjective. Soft, subjective measures are often
grv€n an aura of objectivity, so that they may ue termea quasi-quantitative.
For instance, a satisfaction survey of internal audit clients, asking questions
similar to those suggested in Figure 2.2 may be analysed numerically and
trends compared over time or between different audit iections. The numeric
presentation of the data tends to mask the high degree of subjectivity inherent
within this performance measure. Thc client satisfaction survey is discussed
further below.

"i.:ii:r::'

t.)- i

;:+:?
.ry t*tn ouFcq , lBopc
.i:l.,ta lw !t!l?;... '., ,. ,... .x,:;,,;;:
n - .' '..+',;.:; ,, ,;ii.t::,..!, r!:i :t:liL:,i1,f--1;i, . t:

bcon your dlscucslons wltfunrdlt at

;"9,,;:,1ffi_ffi
I,F""W
yqq.;dlsanrrldns

Figure 2.2 Cl ienVmanagement satisfaction survey

 €VATUATINC OPEMTIONAL PERFORMANCE

Performane measures with a higher degree of objwtivity than others are Dot
necessarily the preferred ones to use: the criteria for selection of a pcrformance
measure should include a matching to the aspects of internal audit performance
which are most important and which need to be monitored most.
Of crucid importance is to determine which aspects of internal audit
performance are most important and which need to be monitored most. Strictly
speaking, an aspect of internal audit performance could be of first importance
while not noeding to be monitored so closely as other aspects-y'its achievement
were assured or, occasionally, if it were outside the scope of management to
regulate its achievement.
Using our modcl of input, process and output, we now considcr for each of these
caiegories the most important aspects of performance as thcy relatc to internal
audit. Some measures of performancc infonn about more than one of these
categories. For instance, the success of the internal auditing function in
completing its planned progr:unme of audits relates closely to whether the
function has achicved its objectives (planned outputs) but it also gives potential
insighs as to whether the function has approached its work cfficiently (process).
Another example of this overlap bctrpeen categories of performance measure is
the climt satisfaction survey (see Figure 2.2 for an example) which provides data
on the reputation of audit. To some extent this will result from the judgement that
management has made about the professionalism of internal auditors they have
observed in action (audit process); to somc cxtent tlte answers will depend on
management's experience of thc valuc of audit findings and recommendations
(audit output). Some of the questions put to management in the survey will be
targeted more to process than to output, and vice versa; but the impressions that
management have about the professionalism of the audit process are likely to
colour their answers to questions targeted at audit outpug and thcir satisfaction
with the audit output is likely to colour their impression of the audit process.

lnput Measures

Thcse pcrforrnanoe measures throw ligbt upon the oconomy with which the
intcrnal auditing activity is provided-whether by in-house provision or by
external providers. Possible candidates for use as oconomy measures are:

o Numbcrs of auditors per 1000 staff comparcd to sector average.

r l-cvels of expenditure:

r budget actual
r cost per auditor day
o ratio ofpayroll to other costs
. comparison between audit sections
. comparison with previous periods.

o Allocation of productive time according to type of work (audit type; audit and
non-audit work'[such as firefighting] etc.)
o Extent audit staff are stretched.
 rHE OPERATIONAL AUDITINC HANDBOOK

Those measures selected may need to be adapted if internal auditing is

contracted out. For instance the "number of auditors per 1000 staff compared to
the sector average" would require a conversion to full-time staff quivalents based
on the time ,that the outside consultants were spending in performing internal
audit work. Data on the norms for each business sector are available from the
impressive surveys of internal auditing conducted by the Institute of Internal
Auditors.T A ratio of one internal auditor to every 50 or 100 total staff employed
by the enterprise might be typical of financial institutions where tight control is an
absolute priority, whereas 1:1000 or l:2000 is more typical of civil engineering
constructing firms-probably on account of other personnel, such as quantity
surv€yors, being engaged in quasi-intcrnal auditing tasks.
As a bald measure of economy, the number of auditors employed is useful, but
there may still be diseconomies to be identified. Audit expenditure may be out of
control--cither audit payroll expenditure or non-payroll costs. Some sections of
the audit department may be more costly than othen, perhaps without
justification. Even where there is justification for differential costs between audit
sections, this is useful information for management. as it may point to possible
opportunities to obtain better value for money in certain parts of the total audit
programme than in other parts. Whether or not this is so will depend not just on
cost considerations, but also on the potential for audit effectiveness in the various
parts of the total audit programme. So measures of economy must be interpreted
together with measures of effectiveness (outputs) and efficiency (process) before
appropriate management action can be determined'
We suggest among our input measures a measure of the extent to which audit
staff are stretched. It is arguable that this should be categorised as an efflciency
(process) measure. If audit staff are not bing extended it is likely that the staff
input is unduly costly. Ensuring that audit staff are extended is a matter of
managing the audit fuaction efficiently. If staff are extended they are likely to
perform better and the effectiveness (outputs) of the audit function may be
improved.
Audil departments are now frequently calculating the cost of each audit. Audit
reports often highlight this figure. An increasing number of audit functions are
charging out the cost of the audit to the activity which has been audited. This
practice encorrages auditors to pcrform well in order to keep clients satisfied, and
encourages clients to take the audit process more seriously as they are paying for
it. It also more accurately reflects the total costs of running the different parts of
the business. On the other hand, since line managemgnt should not determine
whether or not an audit is conducted, nor what resources are allocated to it, it is
arguable that those costs should not be charged against their budgets.

Process Measures

The emphasis with respect to process measures is the efficiency with which the
internal audit activity functions. The efficiency analogy with an automobile is
whether it ruris as a well-oiled, well-maintained machine. This is distinct from the
costs associated with running the automobile, which are matters of economy.ltis
 EVALUATING OPERATIONAL PERFORMANCE 25

also distinct from whetler or not the automobile achieves thc objectivcs set for
it-such as luxury, prestige, timeliness, ctc.-which is a matter of effectiveness. of
course, these threoverlap, as we have'said before: a poorly maintained
automobile is less likely to be effective, for instance.
See Figure 2.3 for a breakdown of proccss performanoe measures.

Figure 2.3 Process performance measures

our model in Figure 2.1 shows that eficiencybnks economy with effectiveness.
Perhaps a good overall measure ofaudit efficiency is therefore the average cost of
each implemented audit recomnendation.
Insigbt into the audit function's ovcrall efficiency will come from exploring the
achievement of target dates and the extent that audit management has bcen
successful in maximising auditors' time actually spent conducting airdits and,
within that productive time, the way it has becn allocated and supervised.
The audit client may also have some useful impressions about the
professionalism of thc audit approach which can be explored in a survey-see
Figure 2.2. T\e main measure of professionalism of internal auditing is generally
held to be The Standards for The Professional Practice of Internal Auditing of the
26 THE OPEMTIONAT AUDITINC HANDBOOK

Institute of Internal Auditors: performance measures can be devised to assess the

extent to which an internal auditing function complies with these Standards. It
should be pointed out that compliance with them requires commitment to them
by the internal auditing function, but also needs support by senior general
management and the board.

Output Measures

Here we are considering (a) whether or not internal audit achieves its objectives,
and (b) indeed, even whether it achieves the right objectives.
The Charter of the internal auditing function, as a statement of the distinctive
rights and obligations of the audit function, is an important yardstick against
which audit elfcctiveness or output should be measured. C.ertain elements of the
Standards also relate to audit effectivcness as distinct from audit process.
Audit output is hard to measure. Internal auditors are knowledge workers
whose output is not always tangible. Knowledge workcrs conventionally issue
reports and internat auditors are no exccption. Internal audit reports are a
repository of information on audit output. Perhaps the principal objectives of
internal auditors is to reassure management that their systems of intcrnal control
are sound and, where they are not, to persuade management to implement their
rcommendations. Figure 2.4 summarises the implicit or explicit objectives of
internal auditing.

The implicit and explicit objectives of internalauditing:

1. To reassure managment that internalcontrolis sound.
2. To identify non-compliance and urge future compliance.
3. To identify s)lstem weaknesses and make re@mmendatjons for
improement.
4. To persuading managment to accept and implement successfully
the audit recommendations icr improvement.

Figure 2.4 The objectives of internal auditing

Figure 2.5 outlines output performance measures.

The existence of an audit function with broad coverage provides a measure of
reassurance to management and the board with respect to point I in Figure 2.4
and discourages future abuse due to the deterrent effect of audit. perhaps the
nearest we can get to measuring this type of audit effectiveness is to measure the
planned coverage ofinternal audit and the extent to which internal audit succeeds
in completing its planned programme of work.
 EVALUATING OPEMTIONAT PERFORM^NCE 27

:ii}
.;"i

'.il

figure 2.5 Output periormance measures

An analysis of the findings in audit reports can measure the success of the
department i'r identifying non-compliancc with esscntial controls-perhaps
comparing with the previous year, or comparing the success of differcnt audit
teams, or comparing the succcss of the audit function in certain areas of audit
work compared to other areas.
With rcgard to points (3) and (4) in figure 2.5, a similar analysis of (a) past audit
reports and O) audit records of audit follow-up should allow a neasurement
similar to the example in Tablc 2.1. Admittedly this is an inexact set of
m€a$ufcs-it Presumce, for instanoe, that success can be asscsscd. Even where it
can be asscssod, the time delay is likely to be too great to make it a useful measure
of inrcrnal audit performance. So it might be more practical to measure in
accordance with Tablc 2.1 but stopping short of trying to evaluatc wbether or not
an implemented audit recommeadation was successful.
It nay bc possiblc to attach money values to cost savings which follow
management's acceptance and subseguent conrction/implementation of audit
findings and recommendations in points (2), (3) and (4) of Figure 2.4. Ir will,
Table 2.1 Example of internal audit reporting success

Recommendations Acceptances lmplementations Successes

Nurnber 1000 800 7W 550

Losses 200 100 50
Loss rate (o/o) 20 12.5 7.1
 2a THE OPERATIONAL AUDITINC HANDBOOK

however, never be possible to account for the total value of the audit function to the
business as a whole in terms of cost savings. The impact on costs of many accepted
and implemented audit recommendations is indeterminable, as usually we will never
know what would have happened if management had not so acted. Nevertheless, a
historical record of known cost savings which have followed from audit work can
give one indication of audit value for money. It is, however, human nature to
overlook the additional costs which are often associated with points (2), (3) and (4)
Internal audit also tends to take credit for good suggestions from line staff.
Certainly, management and staffshould be given credit for successful implementa-
tion of audit rwommendations. In measuring cost savings it is difficult to determine
the length of time into the future that the audit department should compute the
saving: the decision is arbitrary. For instancc, if the audit department takes credit for
savings over a twelve-month period, this overlooks that the business may continue
to benefit from that audit finding indefinitely. Despite these objections, measuring
cost savings does have a place in the assessment of audit effectiveness.

I nterpretation of Performance Measures

Any performance measure may mislead if it is interpreted on its own. For

instance, the number of internal auditors per 1000 staff employed may show a
very economic approach to internal auditing, but other measures may indicate
that lnternal auditing is not very effective. Completion of all audits and reports by
their target dates may be at the expense of useful findings and recommendations
bcing made. Measures of reporting success should be linked with measures of cost
savings, time utilisation and the achievement of audit plan.
We also need to be cautious about placing too much confidence in our
performance measures. It might be that they indicate a high degree of audit
su@ess and yet overlook important issues which bear upon internal audit
effcctiveness. Here we highlight just two possible issues of this sort.
Ftrst, audit independence. This is a prerequisite of successful internal auditing.
Seondly, the scope of internal auditing work. Two quotations are helpful here:
\f,ihether or not audit is able to perform the full range of audit functions effectively and
elficiently largely depends upoo management attitude and support which is itself largely
influenced by status and independence. The rcal sign ofindependencc is that auditors are
not impeded in their efforts to examine any area within the organisation whereas status
oftcn determines the significance attached to audit fiodings by management.E

There is no persuasivc reason why...internal auditing should not [appraise operations

gcnerally, weighing actual results in the light of planncd resultsl. Perhaps the only limiting
factors are the ability to alford so broad an audit, the difficulty of obtaining people who
can do a broad type ofaudit, and the very practical consideration that individuals may not
likc to be reported upon. While persons rcsponsible for aocounts and for the safeguarding
of company assets have learnt to acoept audit, those responsible for far more valuable
things-the execution of plans, policies and procedures of a company-have not so readily
learnt to accept the idea.e

Figure 2.6 highlights some of the issues which affect audit independence.
 €VATUATINC OPERATIONAL PERFORMANCT 29

1.

2.
8.

a
a
7.

&

3.
I ili1:':
'. .;. l
;.1[:
':.1.i

Figure 2.6 Factors affecting internal audit independence

30 THE OPERATIONAL AUDITING HANDBOOK

lntegrating Performance Measures with Good Management Practice

The performanc€ tneasures we use to evaluate internal audit should harmonise
with thos€ applicable to the enterprise as a whole:
The importancc of the objectivcs identified for audit is that these should underpin an
orgaoisation's overall aims and objective.s, so that audit's achievements aid the
dcvelopment of the organisation as a whole.ru
Top managemcnt and the board should take the trouble to satisfy themselves that
this is so. The Chartered Institute of Public Finance and Accountancy (CIPFA)
suggests there are four fundamental questions to be asked of internal auditing,
without which performance measures for internal audit have little meaning:
l. Does internal audit have agreed and cstablished goals?
2. [s the work planned and resourced in such a way as to make achievement a
realistic PossibilitY?
3. Does the achievement of these goals contribute to the attainment of the
corporate objcctives, i.e. establishing and maintaining internal control?
4. Does internal audit achieve its defined goals?lr
Thesc questions should be addressed by the audit committee, by management,
by the head of internal audit, and by external audit. The charter of the internal
auditing department should ensure positive answers to these four questions.

A Value for Money Approach to Evaluating lnternal Audit, Using Performance Measures
In esscnce, value for money auditing endeavours to assess economy, efficiency and
effectiveness, making use of carefully chosen and carefully interpreted
performane measures. So the approach we havc taken is a value for money
approach to evaluating the internal auditing function.
A refinement of the value for money audit approach is to organise the chosen
performaace measures into three hierarchies, where the more junior levels of
performanoe measures are intended to interpret the measurement of the more
senior ones. The most scnior measure in each hierarchy is intended to most
aocurately rcflect the most important measure of cconomy (or effciency, or
effectiveness). Examples of these structures are given in Figures 2.7,2.8 and 2.9.

Cost of audit as a
proponion ol totral @rporate
operating costs

Comparison Ratio of audit

of audit budget payroll costrs to
to actual costrs other audit costs

Figare 2J Economy (inputs) measures in the context of a value for money approach
 TVALUATING OPEMTIONAL PERFORMANCE

Accounting
and financial
audits

Figure 2.8 Efficiency (processl measures in the context of a value for money approach

Extenl lhal cdtical

systerns were
covered in the
audil plan

Figure 2.9 Effectiveness (outputs) measures in the context of a value for money approach
32 THE OPERATIONAL AUDITINC HANDBOOK

It is no longer sufficient for audit to view the historic reasons for its establishment as
justification lor its continued existcnce. Audit must and should be prepared to provide
broof of its worth and value for money to the organisation as part of the organisation's
continual growth.l2

NOTES

l. A. D., Selim, G. M. and Vinten, G. (1987) Internal Auditing 2nd edn.

Chambers,
Pitman, London, p. 86.
2. CIPFA (Chartered Institute of Public Finance and Accountancy) (January 1992)
Measuring the Performance of Interncl Audit' Page 3.
3. The UK ixpression contracting or, is the equivalent to the US cxpression outsourcing,
4. Thc fifth'stindard of the Institute of Internal Auditors is on "Managcment of Internal
Auditing Dcpartment", and in this paragraph we have quoted specifc standards 520
and 560, togither with reference to supporting guidelines 520.01, 520.03 and 560.01.
We have also used material from "The Institute of Internal Auditors' Statement on
Internol Audirw Standards ISIAS] No 4 on "Quality Assurancc" @ecember 1986).
5. Consultative Committee of Accounting Bodies, (June 1990) Guidance for Internal
Auditors, Auditing Guideline 308. London.
6. Such as, for instance, the elapsed time between ending audit frcldwork and issuing the
audit report.The decision to include this as one of the performaoce measures would bc
bound io be a matter of judgement; similarly there may be judgement involved in
deciding the start and end points of the elapse time which is being measured.
Judgemint also has to bc cxcrcised in interpreting lhe resultant data-for instance.
what elapse time is accePtable?
7. Sec for example, Survey of Internal Auditing in thc United Kingdom & Eire, /985
(Research nei"rt No. 15, Thc Institute of Internal Auditors-UK) and similar
intcrnational surveys with a US bias by IIA Inc.
8. Chartercd Institute of Public Financc and Accountancy (1981) An Approach to the
Measurement of the Performance of Internal Audit, Audit Occasional Paper No. 3.
9. Koonu, Harold and O'Donnell, Cyril (1976) Management-A Systems and
Contingency Analysis of Mougerial Functions, 6th edn (Inrcrnational Student
Fditioa). McGraw-Hill, Tokyo, pp. 67G-1.
10. CIPFA (lgEl) An Approach to the Measurement of the Performance of Internal Audit,
Audit Occasional Paper No. 3, p. 2.
ll. Ibid., p. 6.
t2. Ibid., p. l.