Policy Study Interactive

Performance Audit
and Policy Evaluation

in the Western Balkans
On the Same or Parallel Tracks?
Abstract
From a conceptual perspective, performance audit and policy
evaluation are very close fields, with highly converging goals,
methods and tools. At the same time, in the Western Balkans
these two fields have been evolving without connection and
reference to each other. How can the two processes be brought
closer together in the three studied WB countries – Macedonia,
Montenegro and Serbia – with a view to ensure efficiencies
and synergic development of these rather novel performance
management instruments?
Milena Lazarević, Miloš Đinđić,

Simonida Kacarska, Jovana Marović,
Marko Sošić, Kristina Cuculoska
Issuer and copyright holder:
European Policy Centre (Centar za evropske politike – CEP)
For publisher: Nebojša Lazarević
Authors: Milena Lazarević, Miloš Đinđić, Simonida Kacarska
Jovana Marović, Marko Sošić, Kristina Cuculoska
Prepress and printing: La Mantini d.o.o.
Design: Marko Milin
Belgrade, 2015
ISBN: 978-86-89217-06-3
The RRPP promotes social science research in the Western Balkans (Albania, Bosnia
and Herzegovina, Kosovo, Macedonia, Montenegro and Serbia). Social science research
aids in the understanding of the specific reform needs of countries in the region
and in identifying the long-term implications of policy choices. Researchers receive
support through research grants, methodological and thematic trainings as well as
opportunities for regional and international networking and mentoring. The RRPP is
coordinated and operated by the Interfaculty Institute for Central and Eastern Europe
(IICEE) at the University of Fribourg (Switzerland). The programme is fully funded
by the Swiss Agency for Development and Cooperation (SDC), Federal Department of
Foreign Affairs.
The views expressed in this study are those of the authors and do not necessarily
represent opinions of the SDC and the University of Fribourg.
www.rrpp-westernbalkans.net
www.facebook.com/RRPP.Balkans
http://vimeo.com/rrpp
3
About the Project and Partners
About the Project

Aim of the project was to analyse the weaknesses, strengths, connecting
spots and similarities between performance audit and policy evaluation in
Macedonia, Montenegro and Serbia so as to provide policy makers and
supreme audit institutions in these countries with evidence-based policy options
and recommendations for the promotion of good governance principles and
contribution to the improvement of public administration performance. The
inspiration for embarking on such analysis in these countries was found in: a) the
need for synergies and efficiencies in reforming public administration especially
in the event of pressing EU accession requirements and deficient resources for
fulfilling them and b) in the acknowledgment that performance audit and policy
evaluation are valuable tools for improving performance of public administration
but remain underutilised or underdeveloped in these three countries.
The study before you is the final product of a research project which lasted over
twelve months. The project “Performance Audit and Policy Evaluation: On
the Same or Parallel Tracks?” is implemented in the framework of the Regional
Research Promotion Programme in the Western Balkans (RRPP) with financial
support of the Swiss Agency for Development and Cooperation (SDC).
Draft policy study was presented at the regional conference in Belgrade on
September 09-10. This final version of the policy study incorporates contributions
of stakeholders from the conference.
About the Partners

Implementing partners of this project are member organisations of the Think for
Europe Network of Think Tanks and EU Policy Research Centres in South East
Europe (TEN). Evidence based policy making and promotion of EU integration lie
at the heart of TEN. Its members are non-profit, independent think-tank or policy
research organisations focusing on the EU integration process and committed to
evidence-based policy making and advocacy. The founding members of TEN are:
European Policy Centre (CEP), Belgrade, SRB - www.europeanpolicy.org
European Policy Institute (EPI), Skopje, MK - www.epi.org.mk
Institute Alternative (IA), Podgorica, MNE - www.institut-alternativa.org
About the Project and Partners

5
Contents
About the Project and Partners 3

Abbreviations Table 7
0 Executive Summary 9
1 Introduction: The EU Drive to Performance 16
1.1 Background: New Public Management and the Western
Balkans16
1.2 Purpose and Method of the Policy Study 17
1.3 Structure 21
2 Context: What Are the Preconditions for Improved
Performance of Public Administration? 23
2.1 Administrative Culture 24
2.2 Macro-level Strategic and Policy Planning 25
2.3 Performance Budgeting 26
2.4 Policy Monitoring Practices 29
2.5 Openness and Reliability of Data in the WB 31
3 Policy Evaluation in the Western Balkans:
A Nascent Concept 33
3.1 Policy Evaluation in Macedonia – the Challenge
of Embedding It in the Policy Cycle 35
3.2 Policy Evaluation in Montenegro – the Missing Link
of the Policy Cycle 39
3.3 Policy Evaluation in Serbia - from an Ad Hoc towards
a Systemic Approach 44
3.4 Highlights from the EU Case Studies – Evaluation Systems 52
3.5 Conclusion: The Pervasive Influence of the Legalistic
Tradition56
4 Performance Audit in Western Balkans:
The Third “E” Dilemma 58
4.1 Performance Audit in Macedonia - towards a Decade
Long Practice 60
4.2 Performance Audit in Montenegro - a Thorny Road
towards Performance 67
4.3 Performance Audit in Serbia - Accountability before
Effectiveness 76
4.4 Highlights from EU Case Studies 86
4.5 Conclusion: A Long Way from the Third “E” 90
Contents
6
5 Interaction Between Performance Audit

and Policy Evaluation 92
5.1 The Conceptual Relation: The Converging of the Twin-Tracks 92
5.2 Macedonia - a Promising (yet Still Unattained) Objective 98
5.3 Montenegro: Foes Rather than Friends 100
5.4 Serbia - the Newcomer in Audit and Evaluation 102
5.5 Highlights from the EU Case Studies 105
5.6 Conclusion: PA and Policy Evaluation as Estranged Friends
in the WB 111
6 Which Direction to Take in the WB? Modalities of Interaction
and Evaluation of Implementation Options 113
6.1 Modalities of Establishing Interaction between Performance
Audit and Policy Evaluation 113
6.1.1 D
escription of Identified Modalities 113
6.1.2 Discussion of the Implementation Modalities based
on Stakeholder Meetings 117
6.2 Evaluation of Implementation Options 121
7 Conclusion and Policy Recommendations 127
7.1 Concluding Remarks 128
7.2 General Recommendations 130
7.3 Specific Recommendations for Macedonia 133
7.4 Specific Recommendations for Montenegro 134
7.5 Specific Recommendations for Serbia 137
7.5.1 Recommendations pertaining to the first modality
of establishing interaction 138
7.5.2 Recommendations pertaining to the second modality
7.5.3 Recommendations pertaining to the fourth modality
Bibliography143
Annexes153
Annexe 1 – Methodology 153
Annexe 2 – Semi-structured questionnaire 157
Annexe 3 – L ist of Interviewed institutions and organisations per
country 160
PERFORMANCE AUDIT AND POLICY EVALUATION IN THE WESTERN BALKANS - On the Same or Parallel Tracks?
7
Abbreviations Table
AWPG Annual Work Programme of the Government

CDC French Court of Audits (originally Cour des Comptes)
CoA Court of Audit
EEF Educational Endowment Foundation
GAWP Government Annual Work Plan
GWAs Government-wide audits (GWAs)
INTOSAI International Organisation of Supreme Audit Institutions
ISSAI International Standards of Supreme Audit Institutions
IT Information Technology
LCSE Law on Civil Servants and Employees
MF Ministry of Finance
MFAEI Ministry of Foreign Affairs and European integration
Ministry for Information Society and Administration
MIOA (Macedonia)
MIS Management Information System
NAD Needs Assessment Document
NCA The Netherlands Court of Audit
NICE National Institute for Health and Care Excellence
NIPAC Secretariat of the National IPA Coordinator
NPM New Public Management
PA Performance Audit
PAR Public Administration Reform
PE Policy Evaluation
Abbreviations Table
8
PIA Policy Impact Assessments

PIFC Public Internal Financial Control
PIS Project Information System
PMO Prime Minister Office
PPS Public Policy Secretariat
RBB Results Based Budgeting
RIA Regulatory Impact Assessment
RoP Rules of Procedures
SAI State Audit Institution
SAO State Audit Office
SEE South-East Europe
SEIO Serbian European Integration Office
The General Secretariat for Public Policy Modernisation
SGMAP (France)
Social Inclusion and Poverty Reduction Unit of the
SIPRU Government (Serbia)
SPD Sector Planning Documents
TA Technical Assistance
VBTB Policy Budget and Policy Accountability Reform
VFM Value for Money
WB Western Balkans
9
0 Executive Summary
About the Study

The study addresses the issue of public sector performance in three
Western Balkan countries - Macedonia, Montenegro and Serbia, with
the goals of connecting two instruments for the improvement of policy
learning and overall delivery of public administration. Performance
audit and policy evaluation, despite their converging goals, methods
and tools, have been evolving in isolation from each other, and without
institutional or practical connections. How can the two processes
be brought closer together in these countries, with the purpose of
ensuring efficiencies and synergic development of these rather novel
performance management instruments?
There is increased pressure on these states to adapt to the EU (as well as
other international) requirements and models, and inter alia to introduce
performance audit and policy evaluation functions into their systems.
Given that resources in public administration systems of Macedonia,
Montenegro and Serbia are too scarce to allow for the fulfilment of all
requirements in the accession process at once, the authors of this study
researched the connecting points between performance audit and
policy evaluation in the three countries, in order to show their potential
synergies for the benefit of public sector performance. Moreover, the
complementarity and the similarities between these two fields have been
emphasised in international literature and in comparative practice for
decades, but have remained understudied in the context of the Western
Balkans.
Performance audit and policy evaluation differ from an institutional
perspective, while the former is performed by Supreme Audit
Institutions (SAIs), the latter is a competence of policy evaluators
within, or commissioned by, policy departments or units of public
administration bodies. Still, they tackle the same phenomenon, i.e.
assessment of governmental activities, by focusing on a comparison of
final results to the initially defined goals. However, this is conducted
through diverging philosophies, terminology, methodologies and tools.
Therefore, the purpose of the study is to examine the existing, as well
as the potential, conceptual, methodological and institutional relations
between performance audit and policy evaluation in the three countries,
0 Executive Summary
10
which are the forerunners in the Western Balkan enlargement process.

The study relies on a qualitative thematic analysis of documents and
data from interviews with key stakeholders, who are involved and shape
the processes of evaluation and performance audit in these countries.
In addition, six EU Member States are analysed in order to gain insight
into the practices of establishing links between performance audit and
policy evaluation in countries with different administrative cultures
and diverse administrative experiences (the Netherlands, France,
United Kingdom, Finland, Estonia and Slovenia). Data was collected
through document analysis which included legislation and relevant
strategic documents on both policy evaluation and performance audit
in the countries under examination, as well as the existing secondary
literature on performance audit and its linkages to policy evaluation
in order to locate this study in conceptual and empirical literature.
Moreover, semi-structured interviews with stakeholders selected from
SAIs and ministerial departments responsible for policy analysis or
evaluation in countries under examination were conducted in order to
collect first-hand information.
Performance Audit and Policy Evaluation -

Emerging Concepts in WB
Firstly, the study focuses on policy evaluation systems or existing policy
evaluation practices in the studied Western Balkan countries, and
seeks not only to elaborate on the development of these practices and
systems, which differ in terms of institutionalization and their goals
and impact, but also to explore to what extent the existing evaluation
practices are conducive to linkages and synergies with performance
audit. It is concluded that policy evaluation is a novel concept that lacks
a systemic approach, with the exception of Macedonia where (internal)
ex-post evaluation of legislation has recently been introduced as a
formal requirement. In contrast to Macedonia, in Serbia and Montenegro
no formal requirements or a systemic approach to evaluation have
been introduced to date. The existing policy evaluation practices are
sporadic and ad hoc, often driven by external pressures (e.g. technical
assistance projects). The EU accession, as a common denominator
for all three countries, is expected to further push towards systemic
solutions that would focus on the effectiveness of policies.
Further on, the study provides an analytical insight into the
development of performance audit, but, similarly to the previous
chapter, its primary intention is to determine the possibilities for
bringing performance audit closer to policy evaluation. Namely, it
examines whether performance audit can become more “evaluative” in
nature. In all three countries performance audit is developing in overly
11
legalistic administrative cultures, which influence supreme audit

institutions (SAIs) to focus more on traditional accountability aspects
of audit, whereas the audit of effectiveness of government undertakings
is still far from reach. Differences among countries extend beyond
the simple number of performance audits conducted to date, to the
general audit focus. Examples are the focus on individual institutions
in Montenegro, the preference for specific policy programmes in
Macedonia, or the audit of cross-cutting topics and their management
aspects in Serbia. All in all, performance audit is not fully evaluative
in nature and effectiveness studies would not be feasible at this stage.
The current circumstances in the three countries studied regarding
performance audit and policy evaluation provided the basis for the
examination of potential links between the two processes, which
could potentially benefit the overall public sector performance, as
well as policy making practices. Little or no evidence was found on the
existing links, the awareness of the key stakeholders on the possibilities
of establishing links, or even their potential usefulness. Different
institutional roles, discrepancies in the level of institutionalization and
formality, a lack of understanding of the roles and work of “the other
side” have, inter alia, influenced such perceptions in all three of the
studied countries. However, a positive finding for each country is that
professionals from both sides, despite certain reservations, strongly
support the idea that the two fields and their actors should be brought
closer together.
The EU case study examples are particularly emphasised in the chapter
on the interaction of performance audit and policy evaluation, as they
have been used as a source of inspiration for defining the modalities of
interaction between the two fields considered in the Western Balkan
countries. The practices of the six selected EU countries demonstrated
a variety of models for establishing more or less formalised links
between performance audit and policy evaluation.
Modalities and Options for Western Balkan Countries:

Which Way to Go?
The modalities describe the different approaches in establishing
linkages and synergies between performance audit and policy
evaluation. The five modalities are thus structured so as to present
a progression from more simple to more challenging and complex
modalities, in terms of their implementation.
The study also elaborates on the implementation options which
policy makers can consider, with the purpose of institutionalising
0 Executive Summary
12
these modalities. The options are, thus, defined in terms of the policy
instruments through which the modalities can be implemented:
1) Status Quo Approach, which does not require new
instruments to be adopted;
2) Soft Law Approach, relying on recommendations and
guidelines;
3) Hard Law Approach, proposing a complex regulatory
activity, i.e. adoption of binding primary and/or secondary
legislation.
The first modality of establishing interaction between performance
audit and policy evaluation was largely inspired by the informal
practice of Dutch performance auditors and policy evaluators, who
seek to connect the two professions and network with each other as
much as possible, using resources at hand. This modality could be
implemented without an additional regulatory activity, i.e. the Status
Quo implementation option would be sufficient for its achievement.
Informal cooperation could be supported by a more proactive role of the
actors involved in performance audit and policy evaluation processes.
This modality was supported by all interviewed stakeholders.
Inspired by the practice of the Netherlands Court of Audit, the second
modality envisions a more proactive role of the SAIs in fostering an
evaluation culture through the execution of systemic performance
audits, which would identify the gaps in public performance
management and measurement. This modality could be implemented
through the Soft Law Approach (internal guidelines and methodologies
of the SAIs) and was generally acceptable for the stakeholders. This
modality would require that SAIs and governmental institutions act as
allies in the shared task of improving the public sector performance,
which is not always the case.
The third modality leans on the experience of the United Kingdom
and the Netherlands where SAIs occasionally conduct meta-
evaluations of policy evaluation systems or individual evaluations in
respective countries, which essentially means assessing the quality
of the government’s evaluations. There is some reluctance among
stakeholders in terms of whether this option is feasible or even
desirable. In the current systems, where both performance audit and
policy evaluation need further strengthening (and institutionalisation),
this option is seen as too ambitious.
The fourth modality finds inspiration in the Estonian model where
policy planning units, which are units in charge of policy evaluation
13
within the ministries, actively use performance audit reports and their
findings in conducting their analyses. There is a general understanding
among stakeholders in the SAIs and governmental institutions that
referring to performance audit reports when conducting policy
evaluation is useful and can substantively contribute to the quality
of evaluations. In the current legal systems of the WB countries, the
implementation of this option would most probably require the hard
law approach, i.e. a systemic regulation of policy evaluation or at least a
regulation of the requirement for policy evaluators to refer to relevant
performance audit reports, and comment or report on whether the
auditors’ recommendations have been implemented by the relevant
policy makers.
Finally, the fifth modality was inspired by the practice of the French
Court of Audit, which has introduced policy evaluation as its own
standalone activity, alongside performance audit. This modality
entails two parallel, but complementary evaluation processes (one
conducted by the SAI and the other by governmental institutions).
Although attracting attention among stakeholders, there is unanimous
agreement that, within the present circumstances of scarce resources
or insufficient capacities of the SAIs, this modality would constitute an
unnecessary duplication of functions.
The selection of the preferred modality and implementation option will
be strongly influenced by the specificities of the country in question.
Therefore, specific recommendations for each country will be proposed
in the study in order to address local contexts and developments.
Concluding Remarks and General Recommendations

In conceiving the strategies for interlinking performance audit and
policy evaluation, policy evaluators and performance auditors in each
of the three countries will have to take into account the differences in
government and audit agendas, be cautious when it comes to institutional
roles and duties (particularly with respect to independence of SAIs),
and start working in the direction of institutional awareness raising
and learning about each other’s roles and activities. Additionally, in
order to get the most out of the potential synergies, current policy
developments and public administration reform efforts, which might
differ across countries, should be taken advantage of for creating a
space for meaningful interaction between performance audit and
policy evaluation (e.g. performance-based budgeting).
General recommendations highlight several steps that need to be taken
in all three countries:
0 Executive Summary
14
Recommendations to SAIs:
ÊÊ It is recommended that SAIs monitor the developments in the
area of policy evaluation and collect and analyse information
about the conducted evaluations, as part of their PA planning
methodologies and processes;
ÊÊ To ensure that SAIs are better informed about the ongoing
and completed policy evaluations, the policy units and/or
centre-of-government institutions in charge of the evaluation
systems could be obliged or recommended to regularly send
such updates to SAIs;
ÊÊ SAIs should devote a part of their performance audit work
to system-oriented issues or causes of failures of policy
implementation;
ÊÊ It is important to establish follow-up mechanisms for
monitoring the implementation of performance audit
recommendations, which are distinct from other types of
audit and sensitive to specific approaches of the three SAIs and
to the stakeholders’ needs in these countries;
Recommendations to SAIs and governments:

ÊÊ Stakeholder consultations among auditees and performance
auditors should be organized as often as possible, before the
audit starts and throughout the stages of the performance
audit process, provided that independence of SAI remains
intact;
ÊÊ The process of preparation of the final performance audit reports
should be made inclusive by including the concerns and remarks
of the auditees in collaboration with performance auditors;
Recommendations to governments:
ÊÊ Consulting performance audit reports should be applied in
the work of ministries and other administration bodies in
the process of designing and evaluating policies, taking into
account country-specific conditions;
ÊÊ The formal requirement for conducting ex-post policy
evaluations should be clearly emphasised by the legislation
(primary or secondary), stipulating general types of ex-post
evaluations to be performed, the level at which they will be
conducted (policy, budget programme, law, etc.), and their
scope and frequency;
15
ÊÊ It is recommended to transfer evaluation-related competence

to a single institution at the centre of government. This
institution would not be in charge of conducting evaluations,
but would act as the “driving engine” behind the development,
implementation, oversight and quality assurance.
0 Executive Summary
16
1 Introduction: The EU Drive to

Performance
1.1 Background: New Public Management and the

Western Balkans
The EU accession process is triggering and driving a plethora of
reforms in candidate countries, including the increasingly important
and emphasized public administration reform. Acceding countries
have adopted different models ranging from the more classical (neo)
Weberian models of public administration to approaches inspired
by New Public Management (NPM)1 and the subsequently emerging
perspectives, such as New Public Governance or Neo-Weberian State.2
What they have in common is an attempt to introduce mechanisms and
instruments for improving and measuring the performance of public
administration. The experience of new Member States that joined
the EU in 2004 and 2007 showed that NPM applied as a one-size-
fits-all model has a debilitating impact in countries that do not have
a well-functioning public administration based on a rule of law and
a democratic administrative tradition with established management
practices.3 Therefore, it has been proposed that in emerging, new
democracies advantage should be given to the building of a more
classical – Weberian – merit-based administrative system, which
would ensure proper embedding of accountability systems, respect for
the rule of law, professionalism and transparency, before embarking
on more “modern” models of public management.4 Nevertheless,
elements stemming from NPM are still being applied in the current
EU enlargement to the Western Balkans (WB), albeit in a different
atmosphere – one focusing much more on strengthening Weberian
administrative concepts.
1
Christopher Hood, “A Public Management for All Seasons?”, Public Administration, 69.1 (1991),
3–19.
2
See, for example, Christopher Pollitt and Geert Bouckaert, Public Management Reform: A
Comparative Analysis - New Public Management, Governance, and the Neo-Weberian State, 3rd ed.
(Oxford: Oxford University Press, 2011).
3
See, for example, Jane Finlay and Marek Debicki, eds., Delivering Public Services in Central and
Eastern Europe: Trends and Developments. (NISPAcee Press, 2003). And Wolfgang Drechsler, “The
re-emergence of “Weberian” public administration after the fall of New Public Management:
The Central and Eastern European perspective,” Administrative Culture 06 (2005), 94-108.
4
For a discussion on this matter see: Tiina Randma-Liiv, “New public management versus
the Neo-Weberian state in Central and Eastern Europe,” The NISPAcee Journal of Public
Administration and Policy 1.2 (2008), 69-81.
17
Macedonia, Montenegro and Serbia, as the case studies of this research,

are under increased pressure to adapt to EU (and other international)
formal and informal requirements and models. Some of those elements
are quite explicitly stated by SIGMA in the Public Administration
Principles published in November 2014 under the auspices of the
European Commission. Cases in point are policy monitoring and
evaluation as well as performance audit, which, albeit from different
perspectives, have been made the necessary ingredients of public
administration reform (PAR) in countries seeking EU membership.5
In the international context, the development of both of these fields
was greatly accelerated with the advent of NPM reforms across the
globe and, although the Western Balkans were never captured by the
NPM reform zeal which captured some of the CEE countries in their
transitional period, the drivers for their development are still very
much present. Thus, although one may argue that NPM never arrived in
the Western Balkans, some of its ideas, those which have merged well
into the Neo-Weberian state concept, have certainly come to stay.6 The
Neo-Weberian state model started to gain attention as the one which
keeps certain elements of the NPM, while retaining crucial aspects
of the Weberian model such as hierarchy, legality, or professional
civil service.7 This model is often seen as a rival of the NPM model,
as it favours a strong over a minimal state, stability over flexibility,
regulation over deregulation, et cetera.8
1.2 Purpose and Method of the Policy Study

Performance audit is defined by the International Organisation of
Supreme Audit Institutions (INTOSAI) as an “independent, objective
and reliable examination of whether government undertakings,
systems, operations, programmes, activities or organisations are
operating in accordance with the principles of economy, efficiency
and effectiveness and whether there is room for improvement.”9 On
the other hand, evaluation is defined by the European Commission as
the procedure of “assessing the relevance, efficiency, effectiveness,
5
See: The Principles of Public Administration, OECD/SIGMA, available at: http://www.
sigmaweb.org/publications/Principles-Public-Administration-Nov2014.pdf.
6
See: Wolfgang Drechsler, “The Re-emergence of “Weberian” Public Administration after the
Fall of New Public Management: The Central and Eastern European Perspective,” Administrative
Culture 06 (2005), 94-108.
7
Wolfgang Drechsler, “The Rise and Demise of the New Public Management: Lessons and
Opportunities for South East Europe,” Uprava, letnik VII (2009), 12.
8
See: Tiina Randma-Liiv, “New public Management versus the Neo-Weberian State in Central
and Eastern Europe,” The NISPAcee Journal of Public Administration and Policy 1.2 (2008), 69-81.
9
ISSAI 300, Fundamental Principles of Performance Auditing, 2.
1 Introduction: The EU Drive to Performance

18
impact and sustainability of policies and programmes.”10 From an

institutional perspective, performance audit and policy evaluation are
separate processes. While the former is performed by independent
Supreme Audit Institutions (SAIs),11 the latter is a competence of policy
evaluators within, or commissioned by, policy departments and/or
units of state administration bodies. And yet, both tackle the same
phenomenon i.e. assessment of governmental activities by focusing on
the comparison of final results to the initially defined goals, although
through diverging philosophies, terminology, methodologies and tools.
In an environment of limited resources and increasing pressure
to downsize the public sector and “do more with less”, it becomes
paramount to inquire how both of these performance related
requirements built in the EU accession process can be met both
effectively and efficiently, and by ensuring maximum synergies
between the two fields. Based on this assumption, the purpose of
this study is to examine the existing, as well as potential, conceptual,
methodological and institutional relations between performance audit
and policy evaluation in three Western Balkan countries, all of which
are candidates for EU accession, namely, Macedonia, Montenegro and
Serbia. These three countries are the forerunners in the Western Balkan
enlargement process; whereas Macedonia is the longest standing
candidate for EU accession in the region since 2005, Montenegro and
Serbia are already negotiating EU membership. Therefore, the analysis
of these three countries brings insight into the incentives that operate
at various stages of the EU accession process. At the same time, the
three countries share a common pre-1990 administrative system that
has developed, as will be shown, in somewhat different directions,
thus shedding light of the importance of contextual factors.
In order to look for comparative practices at the EU level and possible
inspiration for our recommendations to WB policy makers, we
analysed the relationship between performance audit and policy
evaluation in six EU member states. Based on preliminary research
of all EU member states and after applying the criteria that countries
should have both developed PA and policy evaluation systems, as
well as taking into consideration the availability of information and
data, we selected France, the Netherlands, United Kingdom, Slovenia,
Estonia and Finland as out case studies. Additionally, the choice was
10
European Commission, Quality of Public Administration – A Toolbox for Practitioners,
Directorate-General for Employment, Social Affairs and Inclusion, 2015, 50. Available at:
http://ec.europa.eu/esf/main.jsp?catId=3&langId=en&keywords=&langSel=&pubType=434.
11
There is a separate and strong tradition of establishing internal audits in public sector
organisations. Internal audits are, however, not the competence of SAIs but a function
embedded within the public administration bodies and budget beneficiaries.
19
influenced by the consideration of the need to include countries from

diverse institutional traditions as well as create a combination of “old”
(pre-2004) and “new” (post-2004) member states. Such an approach
in selecting the case studies promised to provide us with sufficient
breadth of data for devising potential policy options and alternatives
for the Western Balkans.
The research questions we asked and responded to in this study are:
• What is the nature of the relationship between
performance audit and policy evaluation?
• How is that relationship mirrored in the existing practices of
the three Western Balkan states and the six EU case studies?
• What can policy makers, public administration
employees and SAIs in the WB learn from conceptual
and empirical literature and the practices, from
each other and from the six EU case studies?
In combining the traits of exploratory and descriptive research,
justified by the novelty and underdevelopment of the studied concepts
in the three WB countries, we relied on qualitative thematic analyses
of documents and data from interviews with key stakeholders who
are involved and shape the processes of evaluation and performance
audit in these countries, as well as the EU case studies analysed. We
adopted a qualitative approach given its suitability for “[engaging] in
research that probes for deeper understanding rather than examining
surface features.”12 In terms of the former, we studied the legislation
and relevant strategic documents on both policy evaluation and
performance audit in the countries under examination. These include
constitutions, laws (for example on state audit institutions), as well
as relevant reports, including performance audit reports and reports
on policy evaluation, all of which were subjected to thematic analysis
(for detailed methodology, see Annex). Furthermore, the existing
secondary literature on performance audit and its linkages to policy
evaluation was studied in order to locate this study in conceptual and
empirical literature.
In order to study the operation of the two processes (PA and policy
evaluation) in practice, and the potential and need for their linkage,
we conducted “elite” and “expert” semi structured interviews using a
common interview guide with key stakeholders in the three countries.13
12
Scott D. Johnson, “From the Editor - Will Our Research Hold up Under Scrutiny?” Journal of
Industrial Teacher Education, 32.3 (1995), 4.
13
All of the interviews were transcribed, supplemented with notes and then subject to thematic
analysis.

20
The objective of the interviews was to focus on implementation in

order to supplement desk research conducted for the paper, as well as
to increase awareness of the processes on both of the sides studied. In
addition, “interviews may also help in the process of identifying which
documents have been deemed to be important, read and acted upon.14
In this direction, “one of the strongest advantages of elite interviews is
that researchers can interview first-hand participants of the processes
they are investigating and obtain accounts from direct witnesses to
the events in question”.15 While recognizing the limitations of elite
studies, the study adopts a view of elites as “experts which can provide
high quality information and guidance,” which is of primary concern
to this research since the policies studied in this study depend largely
on elites.16
The interlocutors were selected from SAIs and ministerial departments
responsible for policy analysis and evaluation through a combination
of non-probability purposeful sampling and the snowball technique.
The sample of the potential interlocutors is very small and hence
random sampling was not an option for this research. The focus
was not only on appointed officials, but also civil servants who hold
expertise relevant for the research. While there are limitations to
these sampling techniques, the researchers encompassed a variety of
stakeholders in the interviews and ensured that all relevant elements
of the performance systems in the three countries are represented in
the sample. Reliability was ensured by using data gathered both from
secondary sources and from document analysis and interviews.
The paper fills a gap in both the conceptual and empirical literature.
In relation to the former, the linkages and complementarities between
performance audit and policy evaluation remain debated and largely
understudied in general, but also for non-WB countries specifically.
For this purpose, scholars have urged for the incorporation of
performance information into the financial and policy cycle as there is
lack of comparative data and an omitted link.17 Empirically, the paper
fills a gap in the study of performance audit and policy evaluation in
the context of the Western Balkans. As such, it brings insight into the
14
Lisa Harrison and Wolfgang Deicke, “Conducting Interviews in Political Research,” in Political
Research - An Introduction, ed. Lisa Harrison (London: Routledge, 2001), 94.
15
Oisín Tansey, “Process Tracing and Elite Interviewing: A Case for Non-Probability Sampling,”
PS: Political Science & Politics, 04 (2007), 767.
16
George Moyser and Margaret Wagstaffe, “Studying elites: theoretical and methodological
issues,” in Research methods for elite studies, ed. Wagstaffe, G. M. A. M. (London: Allen & Unwin,
1987), 16-17.
17
See: Christopher Pollitt and Geert Bouckaert, Public Management Reform: A Comparative
Analysis - New Public Management, Governance, and the Neo-Weberian State, 3rd ed. (Oxford:
Oxford University Press, 2011).
21
operation of performance audit and policy evaluation in the conditions

of post-communist transformations and under the pressure of the EU
accession process.
1.3 Structure
The structure of the study is as follows:
• Chapter 2 sets out the wider context for the research of policy
evaluation and performance audit, their linkages and interaction
from the perspective of necessary preconditions for improved
public administration performance. The emphasis is put on four
aspects or preconditions related to performance management in
Macedonia, Montenegro and Serbia.
• Chapter 3 provides an analysis of the state of play in the
development and implementation of policy evaluation in three
Western Balkan counties – Macedonia, Montenegro and Serbia,
focusing on the patterns of development of policy evaluation and
their potential for accommodating linkages with performance
audit.
• Chapter 4 analyses the development and state of play in the area
of performance audit in the three countries, primarily focusing on
the patterns of development of the field in relation to the potential
future linkages with policy evaluation.
• Chapter 5 focuses on the linkages and interactions between PA
and policy evaluation, starting from a conceptual discussion and
followed by a discussion of the main findings of the field research
pertaining to the existing and perceived potential linkages in the
three WB countries, as seen by the interviewees.
Chapters 3 to 5 conclude with key findings from the six European
Union member states case studies presented as concise text boxes,
with the purpose of illustrating examples of good practice that could
serve as inspiration to WB policy makers in further developing the two
fields. The EU case studies are particularly emphasized in Chapter 5,
given that they represent the source and inspiration for defining the
modalities of cooperation presented and discussed in the following
chapter.
• In Chapter 6 specific modalities of cooperation between
performance audit and policy evaluation are elaborated based
on the main findings of comparative case studies (six EU member
states). Based on the results and conclusions from discussions
with key stakeholders conducted in the final stage of research,

22
options for implementation of the preferred modalities are

constructed and evaluated for each of the three WB countries.
• Chapter 7 is the concluding chapter, which provides evidence-
based policy recommendations for the policy makers and SAIs in
the three WB states. Recommendations are structured in a way
to reflect issues which are common to all three countries but at
the same time the largest number of recommendations are tailor-
made, fitting for particular context of a single country owing to
the differences in institutional practices and level of experience;
• Annexes including detailed methodology, questionnaires and a
list of interviewed institutions and organisations.
23
2 Context: What Are the Preconditions

for Improved Performance of Public
Administration?
Although the principal intention of this policy study is to focus on

performance audit and policy evaluation, as well as the possibilities
for linking these two processes in order to ensure better coordination
and a more efficient approach in improving the targeted policy areas, a
few words need to be said about the overall context within which these
two process (should) operate. These contextual questions are not only
relevant for a better understanding of the place of performance audit
and policy evaluation in the overall public administration reform
agendas of the WB countries, but are also useful to introduce, as they
may serve as inspiration for future, follow-up research in the field.
On the whole, the concept which provides a wider context for both
performance audit and policy evaluation, and which also integrates
several important linked issues which were identified during
the research for this study, is that of performance management.
It can be defined as “an integrated set of planning and review
procedures which cascade down through the organization to provide
a link between each individual and the overall strategy of the
organization.”18 Essentially, performance management in its purest
form represents a framework of processes, methodologies, actors
and responsibilities which serve to plan, formulate, implement,
monitor and evaluate the policies, organisations and people in a
government in order to maximise achievement of results in the
most efficient manner. A performance management framework uses
“interrelated strategies and activities to improve the performance of
individuals, teams and organizations. Its purpose is to enhance the
achievement of agency goals and outcomes for the government.”19
The key objectives of performance management include “(re-)
18
Steve Rogers, Performance Management in Local Government (Harlow: Longman, 1990), 16,
as quoted in: Geert Bouckaert and John Halligan, “A Framework for Comparative Analysis of
Performance Management” (paper for presentation to Study Group on Productivity and Quality
in the Public Sector, Conference of European Group of Public Administration, Università
Bocconi, Milan, September 6-9, 2006).
19
“Management Advisory Committee, Performance Management in the Australian Public
Service: A Strategic Framework” (Canberra: Commonwealth of Australia, 2001), 14, as quoted
in Geert Bouckaert and John Halligan, “A Framework for Comparative Analysis of Performance
Management” (paper for presentation to Study Group on Productivity and Quality in the Public
Sector, Conference of European Group of Public Administration, Università Bocconi, Milan,
September 6-9, 2006).
2 Context: What Are the Preconditions for Improved Performance of Public Administration?
24
allocation of resources (of which savings is one specific variant),

increase in the performance of the public sector as such, and enhanced
accountability.”20
Performance audit and policy evaluation, as tools for assessing the
efficiency, economy and effectiveness of government actions, play
their roles in the wider performance management concept. In order
for these processes to be successfully conducted, to thrive and yield
results, certain preconditions are preferable. This brief chapter, based
on a combination of archival research, interviews and notes from
several expert discussions,21 provides a succinct overview of the state
of several important preconditions for and/or elements of performance
management in Macedonia, Montenegro and Serbia, focusing on those
considered as the most important for the development of performance
audit and policy evaluation.22 Therefore, the overview focuses on 1)
administrative culture, 2) overall strategic and policy planning at the
macro-level (government level), 3) performance budgeting, 4) policy
monitoring practices, and 5) lack of open and reliable data, in order to
set out the context in which performance audit and policy evaluation
functions are developing.
2.1 Administrative Culture

As explained above, the concept of administration which is rule- and
compliance-oriented, and resistant to change, coupled with a strong
state that plays a major role in the economy and society, drives the
administration away from highly performance-oriented paradigms,
such as the NPM, reliant on the concept of the minimal state with
application of business and market principles in the public sector,23 and
shifts the focus towards a Neo-Weberian state model. For Macedonia,
Montenegro and Serbia, it is clear that the EU accession process is
influencing and will continue to influence the future direction of public
administration reform. Nonetheless, in the absence of a clear model
at EU level, the legalistic administrative culture and tradition of these
countries impede attempts of taking bolder steps towards introduction
of performance management.
20
Ibid., 23.
21
This chapter arose primarily out of the discussions during the Regional Conference
“Improving Public Administration Performance in the Western Balkans: Current Practices
and Future Perspectives,” held in Belgrade on 9-10 September 2015, as part of the same
project under which this study has been developed. For the report on the conference and its
proceedings, please visit: http://ten.europeanpolicy.org.
22
For example, the overview does not include a discussion of individual performance
assessments and evaluations.
23
Drechsler, “The Rise and Demise of the New Public Management: Lessons and Opportunities
for South East Europe,” 8.
25
Namely, the administration and legal systems of the countries in

question are largely based on the continental legal tradition. As put
forward by Thomas and Bojicic-Dzelilovic, “in looking at the formal
policy making process in the Western Balkan countries […], a picture
emerges of a fundamentally legalistic approach that is rooted in the
historical traditions of the Austro-Hungarian and the civil law system
of the former Socialist Federal Republic of Yugoslavia.”24 Rabrenovic
argues that “‘over-legalisation’ poses problems for the functioning of
the system which lacks flexibility in its operation, as the ‘rules of the
game’ can often be changed only by Parliamentary amendments.”25
Generally, legalistic administrative culture is considered to be resistant
to the introduction of new practices which do not need some degree of
regulation.
2.2 Macro-level Strategic and Policy Planning

Coherent and integrated policy and a strategic planning process are
necessary elements of any comprehensive performance framework, as
they create hierarchy and consistency of priorities at all government
levels, functional links between different levels of prioritisation as
well as a connection with the budgetary framework.
In Macedonia the introduction of the strategic planning system
commenced as early as 2000 with the establishment of the General
Secretariat of the Government of the Republic of Macedonia, and
presently the strategic planning system seems to be functioning well.
SIGMA assessment in 2014 addressed the policy planning system in
detail and concluded, inter alia, that:
- There is no formal hierarchy of central planning documents,
although the Government Political Programme is considered
as the document with the highest status. At present, the main
strategic document of the Government is the Pre-accession
Economic Programme.26
- The planning system established by the rules of procedure of
the Government and the Budget Law provides an effective link
between the setting of strategic priorities, the budget process
and the development of an annual work programme.27
24
Margo Thomas, ed., Public Policy Making in the Western Balkans (Dordrecht: Springer
Netherlands, 2015), 8.
25
Aleksandra Rabrenovic, Financial Accountability as a condition for EU Membership (Belgrade:
Comparative Law Institute, 2007), 174.
26
SIGMA Assessment, the Former Yugoslav Republic of Macedonia, 2014, 8. Available at: http://
www.sigmaweb.org/publications/FYRoM-Assessment-2014.pdf.
27
Ibid., 9.
26
- The linkage between policy and financial planning is less clear

with sectoral and horizontal strategies. Strategies are often
considered as declarative documents and are not accompanied
by action plans or fiscal impact assessments.28
In Montenegro, there is no centre for coordinating strategic priorities at
the administrative level and analyses of government work performance
are not done centrally, although reports on the implementation of
government conclusions and reports on the work of ministries are
prepared.29 There are persistent problems of inconsistency and lack
of coordination among various sector strategies. Shortcomings of this
system are best reflected in problems in medium-term policy resource
planning, which is not developed to capture sector policies or their
resource needs. Even when financing needs are presented in a sector
strategy, they are not consistent with the annual budget or the IPA
programmes.
In Serbia, the planning process at the centre of the government
has gradually evolved due to technical assistance projects which
have been implemented for this purpose, albeit some decade-long
problems are still present. Overall, strategic planning remains largely
unconsolidated and top-down prioritisation is missing. There is no
clear hierarchy between planning documents and different levels
of prioritisation (strategic, policy and operational), and functional
links between different strategies are weak. At the same time, the
budget process is not reflected in the planning process as there is
no “real and effective or, more importantly, causal link between the
GAWP, the Budget Act and budget planning process and management
of the Government.”30 The absence of a connection with the budget
has far-reaching consequences for policy implementation, as well as
monitoring and evaluation.
2.3 Performance Budgeting

Performance budgeting is another important component of a
comprehensive performance framework, and one which can be an
important precondition for the development and quality of performance
audit and policy evaluation (although it cannot be considered
28
Ibid., 10.
29
See: Klas Klaas, “Policy Making Review Montenegro,” SIGMA Papers, No. 51, OECD Publishing,
2014, http://dx.doi.org/10.1787/5jz15qwrt2vb-en.
30
Milena Lazarevic and Marko Obradovic, “Map of Policy Cycle at Central Government Level
in Serbia,” Project Reforming Policy Coordination and the Centre of the Government – Third
Phase, EuropeAid11SER01/03/21, 22.
27
an absolute prerequisite).31 Regardless of the form performance

budgeting might take, it is the only government document produced
with the intention of connecting goals and indicators with financial
resources in a binding manner. What is more, one of the basic features
of performance budgeting is the emphasis on measurable results. In the
research conducted for this study, stakeholders in all three countries
referred to performance budgeting as an important precondition
for performance management, and hence better implementation of
performance audit and policy evaluation.
Programme budgeting has been developing in Macedonia together
with strategic planning in co-operation with the IMF and a Twinning
Project,32 and international donors have supported programme-
based budget related trainings in the past few years. As a result of
this capacity building process, a number of budget beneficiaries have
started implementing a programme budgeting structure. However, the
approach undertaken in the implementation of programme budgeting
is sectoral, i.e. through the adoption of pilot projects through selected
ministries.33 In that sense, there is “partial political will because PBB
[performance-based budgeting] benefits have been identified at a
case-mix (sectoral) level.”34
On the implementation side, benefits of introducing programme-based
budgeting are still missing. In Macedonia, budget beneficiaries lack
output indicators and their correlation with result indicators. For
instance, “50 out of 92 budget users, which is 54% of all budget users,
have not specified output indicators in the 2015 budget (the situation
is the same or even poorer in the budgets for previous years).”35 When
these indicators are in place, they are “poor and inconclusive (usually
only one output indicator is referred) and cannot determine the
performance, achievement and fulfilment of the planned objectives/
goals. The most commonly encountered indicators are the number
of procedures, reports, projects, or permits to be issued/ or their
issuance initiated, nevertheless, the output indicators do not include a
part referring to the successful accomplishment of the goals.”36
31
See: LinkedIn discussion “Performance Budget: Connecting Spot for Performance Audit and
Policy/Programme Evaluation,” within the Group “Performance Based Budgeting for Government,”
https://www.linkedin.com/grp/post/4518607-6041524285556695045?trk=groups-post-b-
title.
32
SIGMA Assessment Report, The Former Yugoslav Republic of Macedonia, 2013, 30.
33
Aleksandra Maksimovska et al., “Performance-Based Budgeting in South-Eastern Europe:
A Legal and Economic Perspectives,” Central European Journal of Public Policy, 8.1, (2014), 64.
34
Ibid.
35
Marjan Nikolov, Borce Trenovski, and Biljana Tasevska, “Feasibility study: Performance Based
Budgeting for Enhanced Budgeting Process and Increased Transparency and Accountability”,
Center for Economic Anaysis – CEA, Institute for Democracy Societas Civilis, 2014, 28.
36
Ibid., 29.
28
The introduction of programme budgeting in Montenegro appears

to be slow and hesitant. Although all spending units have their
budget programmes on paper currently, those programmes still do
not contain performance indicators to monitor the attainment of
programme goals. Apart from dividing the existing line-item budget
into programmes, non-financial elements of the programme budget
(such as goals, starting positions, targeted indicators, etc.) have not
been developed. Although announced in 2007 by the Ministry of
Finance, the Action Plan for Introducing Programme Budgeting has
never been adopted.
The main features of the process of implementing programme
budgeting thus far have been overly ambitious regulatory solutions
which are impossible to carry out in practice. All of this leads to a
conclusion that this reform does not stem from the capacities of
Montenegrin administration, as well as that it fails to include an
analysis of the desired outcomes of the reform process. Therefore, the
hitherto process of implementing programme budgeting has led not
to greater accountability of spending units for the funds allocated to
them, but to an even greater centralisation of responsibility within the
Ministry of Finance.
In its 2012 report, SIGMA criticised the way in which programme
budgeting was set up in Montenegro, claiming that it was not
conducive to promoting accountability within spending units.37
Furthermore, SIGMA recommended a reconsideration of the approach
to programme budgeting, seeing it as unsustainable and overly
ambitious at the time when some simple issues (such as multi-annual
planning and consolidated planning in line with the changes in the
structure of spending units) have not been resolved. Thus far, these
recommendations have not been taken into consideration. Additionally,
the Ministry of Finance is insisting that spending units should have
as few budget programmes as possible (limiting it to five),38 with the
underlining rationale being that having fewer programmes simplifies
planning, monitoring and reporting.39 The consequence of this practice
is a small number of large programmes which do not follow the division
of competencies or an organisational structure with the spending
units and the Ministries.
37
SIGMA, Overall Assessment Montenegro, 2012, 16-22.
38
Instructions for Preparation of Non-Financial Elements of the Programme Budget, in
National Budget Circular, Part D, “Instructions and Guidelines for Preparation of Non-Financial
Information for Budget Users,” 2014.
39
Decision on the Methodology of Preparation and Content of the Programme Budget, “Official
Gazette of Montenegro” 38/2008.
29
Similar to Macedonia and Montenegro, performance budgeting in

Serbia has been introduced as programme-based budgeting. It is a new
concept, formally applied to the 2015 budget of the Republic of Serbia
for the first time, and is yet to be fully implemented. Capacity building
for establishing a performance structure has been implemented
to a large extent for ministries and other budget beneficiaries.
Programme structure consists of programmes, as the first level, and
programme activities and projects as the second.40 The development
of objectives and indicators is required for both levels. Programmes
are usually defined according to sectors in the ministries (usually
3-5 programmes per ministry) and are defined broadly enough not
to be influenced by changes in competences of the ministries. Joint
programmes spanning a few budget beneficiaries are also introduced
in order to execute programme activities and projects that require
interventions from different ministries. Defining objectives and
output and outcome indicators is currently one of the major issues in
the early stage of implementation, and capacity building for improving
skills in that regard is ongoing.41 Preliminary analysis of nine-months
of implementation in 2015 showed that the realisation of budget
appropriations was only 20-30%, which indicates low development of
mid-term planning of budget beneficiaries.42
There is still a need to familiarise all budget users with this concept
and improve definitions of indicators and objectives. In 2015 a review
of programme structure and system of indicators and objectives
will be performed. First monitoring reports will be delivered to the
Ministry of Finance in 2016. In other words, in the inception year the
main focus was on the introduction of a programme-based budgeting
structure. The focus on results and performance is expected as soon as
the system of objectives and indicators as well as mid-term operational
planning at the level of budget users are improved.
2.4 Policy Monitoring Practices

Policy monitoring is generally viewed and accepted as a concept
inherently connected to policy evaluation. Monitoring is more closely
associated to the concept of performance measurement, it entails the
systematic collection and analysis of data in the implementation of
a policy (or a programme or project) in order to verify whether the
course of action is kept (process monitoring) and whether results set
40
Guidelines for producing a programme-based budget, available at: http://bit.ly/1W4hENc.
41
Interview with USAID Business Enabling Project representative. USAID BEP is working
together with the Ministry of Finance on the introduction, capacity building and development
of programme-based budgeting.
42
Interview with USAID Business Enabling Project representative.
30
are being achieved (result based monitoring). Generally, without some

kind of policy monitoring, policy evaluation can be quite difficult, as
the necessary inputs into the evaluation process might not be in place.
Nevertheless, evaluation can be conceived of even if monitoring is
not done systematically, especially in cases where information about
achievements and measurement of performance indicators can be done
post factum.
Policy monitoring in Macedonia has been formally introduced in
the Government rules of procedure, especially in terms of top-down
monitoring (Government work plan), but it is not done systematically
at all levels, and monitoring and reporting systems are very weak
within ministries.43 Nevertheless, some of the basic preconditions
for building up the monitoring practices in Macedonia have been
met, as all ministries have formed Units for Strategic Planning,
Policy Making and Monitoring, which currently differ with regards
to capacity, but which do have an explicit responsibility to monitor
policy implementation.
In Montenegro, although the Law on State Administration generally
enlists monitoring as one of elements of the law implementation function
of the administration,44 the function itself has not been sufficiently
elaborated and developed within the administration. Implementation
of the Government’s Work Programme is monitored by the GS, which
prepares a quarterly report on its implementation and delivers it to the
Government, but medium-term horizontal planning is missing, as well
as the requirements to ensure that policy implementation is followed up
by monitoring and analysis.45
Similarly, the Serbian Law on State Administration states “monitoring
and determining of the situation in their areas of competence”
as one of the tasks of the administration,46 but the function is not
further elaborated within the individual institutions and through
standardized job descriptions and/or methods of work. Specific centre-
of-government institutions are active in monitoring certain documents
(plans or strategies), such as monitoring of the implementation of
the Government Annual Work Plan by the General Secretariat of the
Government, or of the National Plan for the Adoption of the Acquis
43
SIGMA Public Administration Reform Assessment of the Former Yugoslav Republic of Macedonia,
2014, 11. Available at: http://www.sigmaweb.org/publications/FYRoM-Assessment-2014.pdf.
44
Article 15, Law on State Administration, “Official Gazette of Montenegro” no. 38/03, 22/08,
42/11.
45
SIGMA Country Assessment, Montenegro, 2013, 6. Available at: http://www.sigmaweb.org/
publicationsdocuments/MontenegroAssessment_2013.pdf.
46
Article 13, Law on State Administration, “Official Gazette of the Republic of Serbia” no.
79/2005, 101/2007, 95/2010 and 99/2014.
31
by the Serbian European Integration Office. However, no systematic

approach exists at the ministry level, ensuring that individual, sectoral
policies are regularly monitored and reported on, and the capacities of
the ministries to engage in data collection, processing and analysis are,
overall, very weak.47 The Public Policy Secretariat, established in April
2014 as a new centre-of-government institution with the mandate
to manage the reform of the policy planning and making system,
has developed a legislative package on the planning system, which
includes a methodology for policy management, on the basis of which
this function can be expected to be developed more substantially in
the coming period.48
2.5 Openness and Reliability of Data in the WB

One of the major problems affecting the overall performance
management capacity of the WB countries, which also featured
prominently in the discussions during the Belgrade regional
conference,49 is lack of open and reliable data. The statistical offices
in these countries are undergoing reforms and catching up with EU
methodologies and practices, but there is still a strongly prevailing
doubt over the quality of the statistics produced by them. At the same
time, administrative bodies rarely produce consistent and reliable
data of their own, which was observed as one of major obstacles in the
implementation of the first round of baseline assessments by SIGMA/
OECD based on the new framework of Principles of Public Administration
in early 2015.50
SIGMA country assessment reports also put emphasis on the absence
of reliable data in budget and policy processes for different priority
domains for SIGMA. For illustration purposes, the assessment for 2014
for Montenegro claims that the “lack of a long time series of data in
Montenegro contributes to weaknesses in economic forecasting.”51 For
Macedonia, the country report states inter alia that, even when a formal
mechanism is in place, the quality of data and estimation may vary.52
47
Sena Maric, Jelena Zarkovic Rakic, Ana Aleksic Miric and Milena Lazarevic, How to get Results in
Public Policies: Monitoring and Evaluation with the Evidence Supplied by the Civil Society (Belgrade:
Foundation for the Advancement of Economics and European Policy Centre, 2014), 39.
48
At the moment of the finalization of this study, the legislative package was still in the public
consultation process and not yet adopted.
49
See Conference Report at: http://ten.europeanpolicy.org/2-news/32-ten-held-regional-
conference-on-improving-performance-of-public-administration.html.
50
Based on the discussions at the Regional School of Public Administration (ReSPA) conference
“Monitoring and Evaluation of Public Policies,” held in Danilovgrad, Montenegro, October 13-14 2015.
51
SIGMA Country Assessment Report, Montenegro, 2014, 7. Available at: http://www.sigmaweb.
org/publications/Montenegro-Assessment-2014.pdf.
52
SIGMA Country Assessment Report, Former Yugoslav Republic of Macedonia, 2014, 10.
Available at: http://www.sigmaweb.org/publications/FYRoM-Assessment-2014.pdf.
32
Similarly, for Serbia, systematic and regularly updated databases

are missing for specific areas.53 Depending on sector policies, data
accessibility and reliability among WB countries may differ, but as a
rule, there is a general lack of systematic and consistent data which
additionally makes comparability between WB countries harder.54
53
SIGMA Country Assessment report, Serbia, 2013, 17. Available at: http://www.sigmaweb.
org/publicationsdocuments/SerbiaAssessment_2013.pdf.
54
Multiple assessments conducted for specific policies in the WB highlight the lack of reliable
data in some instances and challenges for drawing conclusions based on this data. For example,
see: ‘Overview of the research and innovation sector in the Western Balkans’ by the World
Bank, available at: http://bit.ly/1LLppGA; or ‘The Western Balkan in Transitions’ paper by
European Commission, available at: http://bit.ly/1RnYVec.
33
3 P
olicy Evaluation in the Western
Balkans: A Nascent Concept
Policy (or programme, in the US tradition) evaluation is a part of the

learning stage of the policy process, in which data and information
collected in the process of implementing a policy are analysed in order
to assess the worth of that policy (see illustration below). Evaluation
should provide credible and useful information for incorporating the
lessons learned in decision-making and policymaking processes.55
Numerous types and approaches to policy evaluation have been
recorded in international literature and practice. Howlett and Ramesh
distinguish between administrative, judicial and political evaluations,
with the first type being the most widely studied one. Administrative
evaluations are “undertaken within the government” and they are
usually “restricted to examining the efficient delivery of government
services and attempting to determine whether or not ‘value for money’
is being achieved.”56 Effort (or input-oriented) evaluations, performance
(or output-oriented) evaluations, effectiveness evaluations (or
adequacy of performance), efficiency evaluations (which are mainly
concerned with the relationship of inputs/costs and outputs), and
process evaluations (dealing with process streamlining) are the most
common types of administrative evaluations.57 Evaluation is deeply
related to concerns with performance measurement and improvement
in public administrations.
Due to the novelty of the evaluation concept in the Western Balkan
countries and the lack of wider debates on the types and purposes
of different types of evaluations, this chapter treats as evaluation
any activity, procedure or methodology aiming at analysing ex-post
(i.e. after the initiation of the policy implementation) the effects
(or results) of that policy in the broadest sense. Furthermore, as
the concept of the policy cycle is still in development in the studied
55
Jody Zall Kuzek and Ray C. Rist, “Ten Steps to a Results-Based Monitoring and Evaluation
System,” The World Bank, 2004, 12.
56
Michael Howlett and M.Ramesh, Studying Public Policy: Policy Cycles and Policy Subsystems
(Oxford: Oxford University Press, 1995), 210-11.
57
Ibid., 211-12. Another categorization of programme (policy) evaluations is offered by the
Government Accountability Office – the US SAI: 1) process evaluations; 2) outcome evaluations;
3) impact evaluations; 4) cost-benefit and cost-effectiveness evaluations. See: ISSAI 3000
Standards and Guidelines for Performance Auditing based on INTOSAI’s Auditing Standards
and Practical Experience, 24-25.
3 Policy Evaluation in the Western Balkans: A Nascent Concept

34
countries, the objects of evaluation are defined in the broadest

possible manner: strategies and plans, budgets and budget
programmes, legislation, technical assistance programmes and
projects, in order to take stock of the various existing practices and
trajectories of development of future/new evaluation systems. In
this regard, practices and procedures of relevance for evaluation
are, in some cases, taken into consideration in the analysis, for
example regulatory impact assessment (RIA) as a type of ex-ante
evaluation of legislation.
Illustration 1. Evaluation in the Policy Learning Stage of the Policy Cycle
• Definition of priorities - • Deciding on the

agenda setting proposed policy
• Problem recognition • Preparation of legal
and analysis drafts or other polisy
• Policy design - instruments, including
Policy Policy
preparation of polisy continued regulatory
Formulation Realisation
proposals and impact assessment
formulation (RIA)
of policy alternatives Policy • Coordination at the
• Ex - ante impact planning centre of government
assessment, incl. (macro • Further consultations
weighing of polisy level)
options
• Choise of policy
option
Policy
• Inter-ministerial
Learning
consultations
• Continuous
consultations
• Policy implementation
• Policy monitoring
• Pollicy evalution
& re-launch of the
policy cycle
The purpose of this chapter goes beyond taking stock – it sets out to
determine which of the existing policy evaluation practices (and those
under development) in the three WB countries are most conducive
to establishing stronger linkages with performance audit, studied
in the subsequent chapter. In that regard, the chapter focuses inter
alia on the existing formal or informal practices of policy evaluation,
actors and their capacities for conducting policy evaluation as well
as the impact achieved so far. Reference is also made to the ongoing
policy developments in all three WB countries of relevance for ex-
post policy evaluation. Furthermore, it offers the most relevant
highlights from the EU case studies, in particular having in mind the
35
performance audit link, selected according to their relevance for WB

countries studied.
3.1 Policy Evaluation in Macedonia – the Challenge

of Embedding It in the Policy Cycle
The ex-post policy evaluation was systematically introduced in
the Republic of Macedonia with a Methodology for evaluating
the implementation of legislation, adopted by the Government in
March 2013.58 On the basis of the Methodology, the Manual for its
implementation was also adopted.59 However, prior to the adoption of
the Methodology, ad hoc evaluations were conducted primarily with
the support of external actors such as the World Bank and the OSCE.60
In fact, OSCE has been supporting the introduction of ex post evaluation
in Macedonia since 2010, starting with the project “Strengthening the
Process of Legal Drafting”. Within the framework of this project, an
assessment of the necessity for introducing an ex post policy evaluation
system was undertaken.61
Systemic ex-post evaluation of legislation: the starting point

The Methodology for evaluating the implementation of legislation
outlines the process of evaluation as the last phase in the policy
cycle. Through ex post evaluation of implementation of legislation,
ministries and other institutions can monitor and determine the level
of success or deficiency in the process of legislation implementation,
as well as the elements which influenced the outcomes. The aim of the
evaluation is to produce drafts of future legislation which are higher in
quality. One of the expected benefits of introducing ex-post evaluation
58
Methodology for Evaluating the Implementation of Legislation, Ministry of Information
Society and Administration, Republic of Macedonia, March 2013, http://www.mio.gov.mk/
files/pdf/Ex_post_Metodologija.pdf.
59
Manual for the Implementation of the Methodology, Ministry of Information Society and
Administration, Republic of Macedonia, March 2013, available at: http://mioa.gov.mk/files/
pdf/Priracnik_za_primena_na_Metodologijata_mk.pdf.
60
Between 2003 and 2004 an ex-post policy evaluation on the impact assessment of the
SME’s policy was conducted. The evaluation resulted in a recommendation for the legislative
amendments to meet the needs of businesses. In 2004-2006, within the framework of the WBO
regional office in Macedonia, ex-post policy evaluations were undertaken on the national policy
for the support of the education of Roma population and education policy concerning research and
development. The policy evaluation on education policy and the connection between education
and research and development resulted in the establishment of the Innovation Fund in 2010.
61
Within this comprehensive project there was evaluation of the necessity of introducing an
ex-post evaluation system (i.e. system of policy evaluation). In 2010, OSCE established a WG on
ex-post policy evaluation which has since analysed numerous laws, such as the Criminal Code,
the Criminal Procedure Code, legislation concerning work safety, and legislation concerning
social welfare.

36
is the promotion of “performance-based management.”62 In the course

of ex-post evaluation, the ministries and institutions should plan the
implementation of legislation in a way that enables them to achieve
better results and increase the efficiency of implemented legislation.
The Methodology prescribes a 6-month time frame for ex-post
evaluation, except for systematic laws where the evaluation process
may take longer to complete. In the implementing phase, the department
responsible for implementation of a certain piece of legislation within
the ministry performs the ex-post evaluation. A working group, which
includes representatives of all institutions which participate in the
implementation of a specific legislative piece, is established for the
evaluation process, but this approach remains essentially internal.
Each ministry that performs ex-post evaluation determined in the
AWPG is obliged to produce an Evaluation Report, which contains the
results of the ex-post evaluation and lessons learned, as well as the
proposed recommendations for the legislative process, ranging from
amendments to enactment of a new law, or to status quo.
Since the adoption of the Methodology in 2013 two extensive ex-post
policy evaluations have been conducted. These include evaluations on
the Law on Anti-Discrimination and the Law on the Promotion and
Protection of the Rights of Minorities representing less than 20% of the
population of the Republic of Macedonia. The analysis in this section
is based primarily on the latter, which we analysed using document
analysis and interviews with the evaluators.
The evaluation on the implementation of the Law on the Promotion and
Protection of the Rights of Minorities representing less than 20% of
the population of the Republic of Macedonia was finalized in 2014 and
we analysed that evaluative process as an illustrative example for this
study. The beneficiary was the Agency Minority Rights Realization,
which established a working group of stakeholders for the purposes
of evaluation.63 All of our interviewees emphasised the inclusion of a
wide set of stakeholders as fundamental to obtaining a high-quality
62
The need for increased awareness and a culture of performance measurement and its
benefits has already been established in the literature, encompassing all parties involved in
the policy cycle in Macedonia. See: Marjan Nikolov, Borce Trenovski and Biljana Tasevska,
“Feasibility study: performance based budgeting for enhanced budgeting process and
increased transparency and accountability,” Centre for Economic Anaysis – CEA, Institute for
Democracy Societas Civilis, 2014.
63
The Agency with OSCE assistance and an external consultant formed a Work Group for
evaluation, consisting of representatives from the Agency, Ministry of Justice, Ministry of
Local Self Government, Ministry of Foreign Affairs, Secretariat for Implementation of the Ohrid
Framework Agreement, Agency for development and promotion of the languages of minorities,
Agency for promotion of the culture of minorities as well as representatives from the associations
of all minorities groups living in the Republic of Macedonia as well as the NGO’s working in this
field.
37
and objective evaluation report. This specific evaluation produced

numerous recommendations such as the need to adopt an Action
Plan for the implementation of the Law, change the internal structure
of the organisation, and increase the budget and visibility of the
Agency. The main recommendation was for a legislative amendment
and it is currently being followed up as a working group which was
established within the Ministry of Justice for the purpose of drafting
the amendments to the Law.
The process of evaluation planning is performed in parallel with
the strategic planning process and the drafting of the Annual Work
Programme of the Government (AWPG). Each ministry plans the
implementation of at least two ex-post evaluations on an annual basis
within the strategic planning process.64 The proposals for ex-post
evaluations are integrated in the AWPG. SIGMA has considered the
“introduction of ex-post analysis for two laws per ministry per year is
a proportionate approach that enables a staggered introduction”.65 In
the planning phase, each ministry responsible for the implementation
of the legislation, which will be subject to evaluation, is supposed
to draft a plan for conducting the ex-post evaluation, which is to
be published on the web page of that ministry and which includes
the necessary resources and the time frame for the evaluations.
Nevertheless, none of these plans has thus far been made available
on the ministries’ websites.
The introduction of ex-post policy evaluation was supported with
a generic training for representatives from the key ministries and
independent bodies.66 According to our interviewees, further trainings
are essential and necessary for all civil servants in the ministries
dealing with ex-post policy evaluation, as is the hiring of additional
staff dedicated specifically to this function. As it stands at this point,
there was consensus among our interviewees that the ministries do
not have the capacity for extensive data gathering and processing, as
is needed for a substantial policy evaluation.
Furthermore, the monitoring by the ministries of their respective
policies is usually weak and undermines the potential for substantive
64
Initially ex post evaluation was envisaged for all laws, including in-depth analysis for some
(for example the Packaging Law). However, after careful consideration, the Methodology
includes ex post evaluation for only two laws per year.
65
SIGMA, Public Administration Reform Assessment of the Former Yugoslav Republic of
Macedonia, 2014. Available at: http://sigmaweb.org/publications/FYRoM-Assessment-2014.
pdf.
66
Such as the Ministry of Information Society and Administration, Ministry of Justice, Ministry
of Agriculture, Forestry and Water Management, the State Audit Office, Ministry of Economy,
General Secretariat, Ministry of Environment, and others.

38
evaluations. Most ministries have units for policy creation and analysis,
however they do not conduct monitoring of policies, a challenge
underlined by a number of our interviewees. The comments on weak
monitoring were usually coupled with remarks on the lack of action
plans for the implementation of legislation, which is fundamental for
ex-post policy evaluation as a key segment of the strategic planning
process.67 Strategic planning deals with formulating government
programmes and should be followed up with the monitoring of policy
implementation, and concluded with ex-post policy evaluation.
Linking ex-post evaluation to the ex-ante RIA

Interviewed evaluators and officials coming from state institutions
viewed the introduction of ex-post evaluation as a ‘logical’ step following
the introduction of the ex-ante regulatory impact assessment (RIA).68
Since 2009 there is an obligation for all new laws to be subjected to
RIA, although the quality of these ex-ante assessments is still not
satisfactory given that the documents have been largely superficial, i.e.
without offering a substantial assessment of impact. On a similar note,
the 2014 European Commission Progress Report on Macedonia notes
that, whereas “procedures for policy development and coordination
across the sectors are well defined, [they] are not always efficiently
implemented,” and that “thorough analysis and assessment of policies
is often lacking.”69 Indicators of success, which could be potentially
used for the evaluation of a specific law or policy, are supposed to be
determined as a part of RIA, however this has not been put into practice
so far. In fact, the two laws which were subjected to ex-post evaluation
were adopted without an accompanying RIA report, which could have
been beneficial for the purposes of evaluating the implementation of
legislation as well. According to one of the interviewees, “RIA is very
important, as well as the quality of RIA reports, since it creates a
baseline containing indicators that are later on extremely useful for
ex-post evaluation of policies. What was defined as an indicator is very
important, since that will be measured later, in the performance audit
and policy evaluation reports.”70
67
Interview with external evaluator, May 2015.
68
The introduction of RIA was supported by the United Kingdom since 2005. RIA is defined as
a policy tool which is used to make policies effective and efficient by providing stakeholders
with high quality regulation. RIA in the Republic of Macedonia is part of the broader regulatory
reform undertaken since 2006. The reform, still in progress, has been conducted in two phases.
Regulatory Guillotine took place in the first phase, while RIA occurred in the second phase.
69
European Commission, Working Document Progress Report on the Former Yugoslav
Republic of Macedonia, 2014, 8. Available at: http://ec.europa.eu/enlargement/pdf/key_
documents/2014/20141008-the-former-yugoslav-republic-of-macedonia-progress-report_
en.pdf.
70
39
Both the civil servants and the evaluators interviewed viewed RIA and
ex-post evaluation as largely complementary in the approach adopted
in Macedonia. However, different recommendations were provided in
terms of how to advance and link these two processes. In the opinion
of the interviewees, the ministries do not have the capacity to conduct
meaningful and substantial RIA for all of the laws, hence it was
recommended to limit the number of laws for the sake of quality. On the
other hand, given the gradual introduction of ex-post evaluations by
focusing on two laws per year, as a logical follow up, recommendations
were given to extend ex-post evaluations to all legislation adopted.71
Overall, policy evaluation has been formally introduced in the policy
making process in Macedonia and a track record of largely donor-
supported evaluations has been produced. The process is still a novelty
in civil service and the approach of gradual introduction limited to two
laws per year per ministry is likely to provide a more quality- (rather
than quantity-) oriented evaluation system. The methodology for
evaluation formally includes a requirement for consulting independent
reports (for example performance audit reports), which indicates an
existing linkage for the purposes of our study. Its materialisation in
practice is yet to be established due to the recent introduction of a
formal policy evaluation process.
3.2 Policy Evaluation in Montenegro – the Missing Link

of the Policy Cycle
Systematic ex-post policy evaluation has still not been established
within the Montenegrin administration, although the country has been
negotiating its membership in the European Union since 2012. The
central part of the accession negotiation process is the harmonization
with the EU acquis, but not all procedures that are supposed to
increase the quality of regulations are in place. The Government Rules
of Procedure (RoP) set the steps for the preparation of new legislative
proposals, but do not prescribe a mandatory follow-up procedure for a
ministry to assess the implementation and suitability of a policy.
Ex ante assessments as the antecedent of evaluation

Unlike the ex-post evaluation, ex ante mechanisms have already been
established and some progress has been made so far. Since 2012,
when it was formally introduced, the ministries have been conducting
regulatory impact assessment (RIA) for all legislative acts, and this is

71

40
the one of two continuous and obligatory evaluations of the regulations

or programmes.
The requirement to apply RIA was introduced by the Government’s RoP,
which specifies that, in the preparation of laws, ministries are obliged to
carry out an analysis of the regulatory impact assessment.72 It further
stipulates that the form of the RIA has to be approved by the Ministry of
Finance, including an assessment of whether the analysis is carried out
properly. At the same time as the obligation to apply this rule began, the
application process was defined by specific instructions.73 Only during
the first year of the RIA implementation, the Ministry of Finance issued
its opinion on 299 regulations, of which 203 were laws and 96 were
by-laws. Despite the three-year long implementation process, RIA has
not been fully implemented. Moreover, the quality of the RIA analysis
is not high and it is usually prepared prior to submitting the draft
law to the Government, and not before the public consultations.74 Our
interviewees believe that the reports should be made more extensive
and comprehensive through the use of the standard cost model.75
If properly conducted and accompanied by specific indicators of
success, RIA reports could become a suitable basis for extending the
system towards ex-post assessments. Montenegrin policy makers
can learn from the Macedonian experience of introducing legislative
evaluations, thus making sure that a more explicit link between the
two methodologies is put in place early on in the establishment of a
possible future evaluation system.
At the same time, although performance-based budgeting was
introduced in Montenegro six years ago, no requirement for evaluating
budget programmes has been put in place, and this despite the fact
that the same Ministry – Ministry of Finance – is in charge of both the
existing evaluative practices in the policy cycle and the development of
the performance budget system and procedures.
72
RIA is part of a broader regulatory reform with two additional pillars, the guillotine and the
ease of doing business reform. The first action plan prepared to allow the regulations to provide
simplicity, efficiency and legal security, was adopted by the Council for Regulatory Reform and the
Improvement of Business Environment during 2009. As part of the project Guillotine a number of
recommendations were prepared, but the majority has not yet been implemented, and this is re-
defined as a priority in the draft of the new Public Administration Reform Strategy for the period
2016-2020, which is currently at a public hearing. Therefore, a new and unique overview has to
be prepared with all the recommendations that have been implemented and regulations repealed.
73
Instructions on Compiling Reports on Regulatory Impact, “Official Gazette of Montenegro”
no. 09/2012.
74
Klaas, “Policy Making Review Montenegro,” 6.
75
Interview with representative of the Ministry of Finance, March 2015.
41
Ad hoc evaluations – the donor projects

Another established and ongoing evaluation practice is the one
which refers to the projects financed from IPA funds required by
the framework agreement between Montenegro and the European
Commission. As an instrument for monitoring and evaluation of
projects financed from IPA funds, the Ministry of Finance has developed
a so-called Project Information System (PIS) which is processing and
recording all financial data and information in electronic format.
However, this is just a step toward the establishment of a computerized
Management Information System (MIS), which will be made accessible
to all interested stakeholders.
In addition, the evaluation system includes a mandatory establishment
of the “evaluation structure”.76 Therefore, the evaluation is also
facilitated, to a certain extent, by the existence of units that manage IPA
funds in all ministries and which are obliged to carry out the evaluation.
The coordination unit is located within the Ministry of Foreign Affairs
and European integration (MFAEI). In the EU Screening Report, it is
stated that Montenegro “is gathering experience in monitoring and
evaluating EU co-funded actions under the different IPA components,
both at programme and project level”.77
Ad hoc examples that lead the way

Even without a formally established framework, methodology and
responsibilities, there are certain developments which could be
considered as individual evaluation initiatives of the relevant PA bodies,
an example being the one which is implemented and connected to the
improvement of public administration reform. In fact, during 2014 and
2015 the Ministry of Interior, in cooperation with SIGMA, has prepared
reports on the evaluation of the Law on Civil Servants and Employees
(LCSE) and the Public Administration Reform Strategy implementation,
both of which are mainly result-based. This dynamic is conditioned
by at least two factors, the first being related to the prioritization of
public administration reform by the European Union as one of the three
pillars of the criteria relevant for assessing the country’s progress
in the negotiation process, while the other is linked to its “attention
towards the extent of implementation of acquis-related legislation in
Montenegro”.78 Moreover, OECD SIGMA has managed the process of
preparing the methodology, in cooperation with independent local
76
Screening report Montenegro Chapter 22 – Regional policy and Coordination of Structural
Instruments, 2013, 2. Available at: http://bit.ly/1LEMrvk.
77
Screening Report Montenegro Chapter 22 – Regional Policy Coordination of Structural
Instruments, 7.
78
Klaas, “Policy Making Review Montenegro,” 37.

42
experts, and closely monitored the preparation of the evaluation report.

The result of this process are, when it comes to the implementation of the
LCSE, ten specific recommendations.79 Immediately after the adoption
of the report by the Government in June, the Ministry of Interior started
establishing a working group which will, until the end of the year, work
on amendments, with a special focus on the process and the period of
employment, selection of candidates, assessment and promotion.
Activities which were conducted earlier, and which were focused on
analysing the implementation of laws, did not serve their purpose.
Moreover, prepared reports were not methodologically directed
towards identifying problems in the implementation process, were
missing clearly defined indicators, and contained only brief directions
for improvement, which were already recognized in the state institutions
or EU reports.80 Therefore, the evaluation of the Public Administration
Reform Strategy and the Law on Civil Servants implementation for the
first time in a substantially different manner determined the impact
of regulations and problems in implementation. The evaluation was
made in accordance with clear methodological guidelines and in
cooperation of state and non-state actors. Additionally, this is a good
example of setting priorities in evaluation where the two documents
are somehow connected. These positive developments can be a good
basis for furthering activities intended to gradually introduce ex-post
evaluation in the system.
Capacities and actors involved

As previously noted, the preparation of strategic documents and other
legislative acts in Montenegro is usually not based on previous analysis,
which would be able to demonstrate all deficiencies and problems in
the previous implementation and incite policy learning. “It is hence
evident that Montenegro does not have mechanisms for ensuring
institutional memory. Consequently, at the start of each preparation
of a new document, stakeholders are faced with the lack of analytic
material that would make their job easier. This problem points to
another one: weak analytical capacities of the state administration.”81
RIA and evaluation of IPA programmes are usually conducted with
79
Information on the Law on Civil Servants and Employees Implementation, June 2015, 51,
accessed on September 1, 2015, http://bit.ly/1krpTrb.
80
Information on the Implementation of the Code of Criminal Procedure, The Ministry of
Justice of Montenegro, December 2012, accessed on September 2, 2015, http://www.gov.me/
files/1230044941.doc.
81
”State administration reform in Montenegro – between ambitious plans and real possibilities”,
Institute Alternative, December 2012, 18. Avaiable at: http://bit.ly/1jU4cPR.
43
own PA capacities, while in all other cases external consultants and

experts are engaged by the Government through a tender procedure.
Since there is no systematic approach to the process of policy evaluation
in Montenegro, many different actors are involved in this process. Taking
into account the aforementioned approaches, this process includes all
ministries, but primarily the MF and the MFAEI. Institutions82 involved
in policy evaluation hold that the General Secretariat of the Government
should play a central role in coordinating and initiating individual
evaluations of public policies, and emphasize the need to strengthen
their capacities. Also, as indicated by the European Commission with
regard to the mechanisms established in the ministries, the necessity
for strengthening the evaluation capacity in public administration
and the audit functions is of increasing importance.83 Support in
strengthening the public administration capacity for monitoring
and evaluation of public policies is provided by the NGO sector, from
developing methodological guidelines for the evaluation of strategic
development programmes to the implementation of training for
parliamentary staff and that of the line ministries. Thus, for example,
with the support of international donors and in cooperation with state
institutions, NGOs presented in late 2012 the results of an evaluation
of the education reform in Montenegro.84
To conclude, the implementation of the ex-post evaluation in
Montenegro can be described as sporadic or ad hoc, with isolated
cases pushed by the European Union and other stakeholders.
These initiatives are often financed by international donors and
backed by external support, NGO or foreign experts. While the
relevant ministries are currently working on improving the existing
regulations in line with the evaluation of the previous scope, it still
remains uncertain to what extent the new legislation will respond
to the identified problems, and how this will affect future activities.
The logical sequence of stimulating RIA should be its linking with ex-
post evaluation. Its gradual introduction could replace the current
ad hoc schedule with priority laws defined on an annual basis until
a system-wide evaluation framework is in place. Donor assistance in
the preparation of these analyses is possible. The current challenges
are related to insufficient capacity and a lack of training, experience,
and understanding the concept as such, which was recognized by all
interviewees.
82
Interviews were conducted with representatives of the Ministry of Finance, the Ministry of
Foreign Affairs and European Integration and the Ministry of Interior.
83
Screening Report Montenegro, Chapter 22-Regional Policy and Coordination of Structural
Instruments, 9.
84
Available at website of the Ministry of Education: http://bit.ly/1RWeQBb.

44
3.3 Policy Evaluation in Serbia - from an Ad Hoc

towards a Systemic Approach
Similarly to the other two WB countries, the concept of evaluating
the effects of policies is a novelty in the administration system of
Serbia and its policy making practices. Policy evaluation, as a distinct
part of the policy cycle, is at present not separately treated through
any particular piece of legislation or policy instrument. Introducing
an obligation to perform policy evaluation would definitely be
beneficial in terms of starting an overarching approach to evaluation
and assessing policy effects in Serbia. However, even if a systemic
framework was put in place, the process of making it fully functional
would occur with difficulties. Using interviews with civil servants, we
conducted an analysis of the current practices concerning evaluation,
which provides an insight into the main characteristics of an ad hoc
approach to evaluation. This means that evaluation needs are mostly
conditioned by externally funded projects or programmes, which
are oriented towards specific policy interventions, and limited in
scope and time. This approach is based on the current or burning
needs and requirements of the EU accession process or development
assistance. The aim of this section is to analyse specificities of the
current approach and to see if it contributes to the establishment of a
well-defined evaluation system in Serbia. We focus on three aspects –
external drivers, capacities and actors.
Nature of current evaluation practices: External drivers, poor

sustainability
According to the interviewees, the evaluations conducted so far
have mostly been externally initiated. This setting typically means
that an external agency (EU, The Global Fund, Swiss Agency for
Development and Cooperation etc.) provides financial and technical
assistance to a domestic institution to implement a concrete project
or programme towards the achievement of certain goals. In order
to prove that these goals have been met and determine their actual
effects, evaluations constitute an obligatory part of the project
activities. For example, the Social Inclusion and Poverty Reduction
Unit of the Government (SIPRU), as the project-based unit at the
centre of the government, started producing analyses in the form
of ex-post evaluation in 2008 in different domains that fall within
SIPRU competences (social protection and education, mapping of
social services etc.) in the period when the implementation of the
Poverty Reduction Strategy was nearing its end.85 Newly produced
85
The Poverty Reduction Strategy was adopted for the period 2003-2008.
45
evaluations are in the form of so-called policy impact assessments

(PIA) and their basis is defined in the project documentation.86
SIPRU has been hiring experts for the production of evaluations on
a contractual basis, as well as coordinating the evaluations process
(producing contract documentation, tendering, monitoring etc.)
and following up on the results. Reports have been produced on
the implementation of recommendations for the first round of ex-
post evaluations. The results of these reports were rather weak in
the sense that most recommendations were not implemented and
institutions/organisations in charge did not take ownership of
the evaluation reports and their recommendations (areas of work
of SIPRU are responsibility of different state institutions). These
obstacles were taken into account in the sense that the approach to
evaluation as a project activity has changed in the second phase of
the evaluations.87 On the other side, these evaluation reports have
been referenced many times (according to the SCIndex88) and, as an
exception, amendments of legal acts have taken place occasionally.89
The lesson learned from these results, as stated in an interview, is
that SIPRU should take a further step and start participating in the
implementation of recommendations.
Additionally, there are other institutions that coordinate externally
driven evaluations and, if there are differences among practices,
they are mostly concerned with technical aspects or specificities of
different policy areas. For instance, the Serbian European Integration
Office (SEIO) has a responsibility to coordinate process of evaluation
of IPA programmes and other development assistance. As in the
previous example, these evaluations are posed as a requirement by
the assistance provider/donor (e.g. European Union) and produced by
the external evaluators. Tasks of SEIO included tendering, monitoring,
ensuring access to institutions as well as consulting stakeholders, etc.90
Evaluation report results are integrated in the planning documents
for development assistance, as well as used for the next phases of
assistance programming.91
A different example shows that evaluation can be a product of domestic
demand. The former Office for Regulatory Reform of the Government
had to perform impact analysis (evaluation) of the Regulatory Reform
86
These ex-post analyses and PIA’s are available at SIPRU’s webpage: http://socijalnoukljucivanje.
gov.rs/en/category/documents/.
87
Interview with SIPRU representative.
88
Serbian Citation Index, available at: http://scindeks.ceon.rs/?lang=en.
89
Interview with SIPRU representative.
90
Interview with SEIO representative.
91
Interview with SEIO representative.

46
Strategy for the period 2004-2008.92 The analysis, which represented

an ex-post evaluation of the implementation of the Action Plan of the
Strategy, was produced in 2011, as envisaged by the same Action Plan.
Results of the analysis were used in drafting a new strategic document
on regulatory reform.93 Although a product of domestic demand, this
type of analysis serves as a particular, time-constrained purpose.
In this type of setting, however, a few different elements limit the
possibility for a spill-over to the national level. Firstly, externally
driven evaluations are concentrated on very specific policy areas and
are tailored to adjust to a certain policy instrument (in the example
of SIPRU, instruments such as state support for the poor, pre-school
programmes etc.). Potential policy evaluation requirements at the
systemic level are expected to be more general in nature, i.e. outlining
the types of evaluations to be done, objectives, responsible parties,
timeframes, et cetera. If a general requirement for evaluation is in
place, individual evaluations could be tailored later on, when they are
applied to a specific instrument. The development of a methodology,
analysis and preparation of the solutions for improvement are
typically handed over to consultants from various organisations,
institutes, universities, consultancies, etc. who are proven experts in
their respective fields. As already mentioned, the process of hiring
experts is conducted on a contractual basis through open calls or
single sourcing, and their tasks and obligations are clearly defined
through the terms of reference.
Capacities - missing, actors - undefined

The need for improving the analytical skills of civil servants in the
Serbian public administration is acute and relevant for enhancing
overall policy making capacities. This is of vital importance in order
to establish not only capable evaluation units in the future, but also to
equip units that should be with policy analysis, collection of evidence,
problem analysis and other skills irreplaceable for producing high-
quality and evidence-based policies.94 In other words, critical skills
92
Implementation of the Strategy was extended to 2011.
93
Interview with representative of the Public Policy Secretariat of the Republic.
94
“High-quality, obligatory policy analysis and writing skills trainings should be developed
and implemented throughout the public administration. This type of training should be
regarded as ‘induction training,’ necessary and obligatory for all civil servants (or at least
a large majority), as analytical thinking and good writing skills should be regarded as a
standard in public administration.” Milena Lazarevic, Sena Maric and Amanda Orza, Policy
Making and EU Accession Negotiations – Getting Results for Serbia, (Belgrade: Deutsche
Gesellschaft, fur Internationale Zusammearbeit (GIZ) GmBH, 2013), 121. This study gives an
in-depth assessment of policy making capacities in Serbia and provides recommendations for
better policy formulation and coordination, especially in the EU accession context, based on
the comparative case studies of the EU member states.
47
in all stages of the policy cycle, needed to properly evaluate policy

effectiveness, are in fact missing. Given that they cannot be acquired
overnight, but rather through an extensive public administration
reform, the production of in-house evaluations is still far from home
in Serbia.
Nonetheless, evaluation skills are not necessarily attached to
administration bodies. Outsourcing to external actors is a viable
option even in developed policy making systems of older EU member
states (see the EU case studies highlights below). Instead, evaluation
units could coordinate and monitor evaluations, making sure that,
from start to finish – designing, structuring, timing, quality control
– evaluations are done according to certain pre-set requirements
and are tailored to the specific policy needs. In that sense, evaluation
units could undertake the role of “clients” ordering a specific product
(evaluation report), to be produced using defined standards and
quality control,95 given that, in their work “evaluators are influenced
by and can influence entity’s expectations for evaluation – its
evaluation goals, its participants, its standards of quality and so on.”96
Nonetheless, building internal organisational skills for producing
evaluation should, ideally, be a viable option as well. In that sense,
it remains to be determined in practice which option – outsourcing
experts, building own capacities, or a combination of the two – is the
best one for the administration system in question when it comes to
deploying resources.
At the moment it is not an easy task to delineate a clear hierarchy
or a network of institutional actors performing evaluation in the
administration system of Serbia. Although in their inception phase,
policy evaluation practices extend across various actors including
ministries and other government agencies, while contracted evaluators
are currently not regarded as actors in an institutional sense. So far,
evaluation has been most institutionalised through the competence
of the Serbian European Integration Office and, more precisely, its
Department for Planning, Programming, Monitoring and Reporting on
EU Funds and Development Aid. The competence to monitor and report
on the utilisation of IPA funds is the closest to the description of public
95
According to the HM Magenta Book, quality control and assurance are crucial dimensions
of evaluation. Quality control ensures that “the evaluation design, planning and delivery are
properly conducted, conform to professional standards (such as ethical assurance), and that
minimum analytical standards are adhered to.” Also, “quality control will ensure consistency
in data collection, methodology, reporting and interpretation of findings.” In: The Magenta
Book: Guidance for Evaluation (London: HM Treasury, 2011), 32.
96
Leslie J. Cooksy et al., “Evaluation Policy and Evaluation Practice: Where do We Go from
Here?”, in Evaluation Policy and Evaluation Practice, New Directions for Evaluations, ed. W.M.K
Trochim, M. M. Mark and L.J. Cooksy, 2009, 107.

48
agencies as ordering parties of evaluation. However, it should be noted

that this institution has a central role in the EU integration of Serbia and,
as the Secretariat of the National IPA Coordinator (NIPAC), also assumes
responsibility to arrange evaluations of IPA programming and planning.
The institution does not only focus on EU funds, but on overall bilateral
and multilateral development assistance, i.e. a co-managed process of
evaluating effectiveness and efficiency of all development assistance to
Serbia per sector in the period 2007-2011. This was the first time that
an evaluation covering all official development assistance for the whole
country was conducted.97
Given this particular role of SEIO in the evaluation of EU funds, the
role of establishing a central system and requirements for policy
evaluation for all administration bodies should be assumed by another
institution, with a more general policy management and coordination
responsibility.
This role was recently taken up by a novel institution in the Serbian
system – the Public Policy Secretariat of the Republic (PPS).98 As a
special organisation operating at the centre of government (its head
being responsible to the Prime Minister), the PPS has been put in
charge of improving policy coordination at the strategic level and policy
making in general, which ought to include policy evaluation. The PPS
has strong potential to provide top-down impetus for administration
bodies to establish, develop and align evaluation practices across
the board. It is additionally emphasized when considering that RIA
unit in Serbia is already part of the PPS, and a link to the ex-post
evaluations and analysis can be clearly made. The interviewees with
PPS representatives have shown that this institution does see itself
as responsible for setting up a future systemic approach to policy
evaluation. Its recent initiative to regulate the country’s planning
system by law and set some standards for policy making practices
through secondary legislation seems to be a step in this direction,
although common rules, structures and methodologies for policy
evaluation are not yet a part of those emerging documents. While
still in its drafting stage, secondary legislation aims to introduce a
methodology for better policy making practices by focusing on impact
assessments of measures contained in policy papers and legal acts for
the implementation of policies. The intention is to make these (ex-
post and ex-ante) assessments obligatory and assess the possible and
97
Final report Evaluation of Effectiveness and Efficiency of Development Assistance to the
Republic of Serbia per Sector is available at: http://www.evropa.gov.rs/Documents/Home/
DACU/5/194/Final%20Evaluation%20Report.pdf.
98
Established by Article 33 of the Law on Ministries, “Official Gazette of the RS” no. 44/2014,
14/2015 and 54/2015.
49
real impact of measures of these documents. Although not conceiving

systemic policy evaluation as such, once in place, this mechanism
will certainly represent a step forward, given that no monitoring and
evaluation of the implementation of laws and their effects has been
performed thus far.99 The potential value of this draft legislation is
that, once in place, it should provide a link between the ex-ante and the
ex-post part of impact assessment through the introduction of common
standards for different policy documents.
The absence of systemic solutions in Serbia has certainly acted as an
impediment for the introduction of evaluation. An unconsolidated
policy planning system at the central level and a loose hierarchy of
strategic and planning documents lie at the heart of the undeveloped
policy making system, and, consequently, of policy evaluation as well.
The high number of sectoral strategies with mutually mismatching
objectives, which do not take into account development goals of the
country, impede proper implementation and diminish evaluation
efforts.100 Additionally, a financial framework for implementing these
strategies, as well as clear performance indicators, are missing, while
budgetary planning is “not linked to the policy efforts, both strategic
and operational.”101 Finally, “continuity in the budget planning process
has been severely undermined by government changes and reshuffles
and some elements of the process have never been implemented,”102
which has rendered evaluation even more difficult to achieve.
Despite these circumstances, recent developments have created
a more promising situation. The Public Administration Reform
Strategy adopted in 2014 emphasizes that the existing monitoring
and evaluation system is not regulated in a systematic way, and that
this is the precondition for improving efficiency and effectiveness in
the achievement of policy goals.103 The Strategy also differentiates
between internal evaluations and independent external evaluations
through the involvement of expert institutions, civil society
organisations, and international organisations as adequate
independent control bodies.104 Evidence produced within these
evaluations will be used for introducing corrective measures in the
implementation of Strategy.
99
Lazarevic and Obradovic, “Map of Policy Cycle at Central Government Level in Serbia,” 87.
100
Maric et al., How to Get Results in Public Policies? Monitoring and Evaluation with the Evidence
Supplied by Civil Society, 39.
101
102
Ibid.
103
Section IV - Management of Public Administration Reform of the Public Administration
Reform Strategy, “Official Gazette of the RS” no. 9/14, 42/14.
104
Section IV - Management of Public Administration Reform of the Public Administration
Reform Strategy.

50
A legacy for the future: What is the impact?

Only a systemic approach to the issues analysed in this section would
pave the way for policy evaluation as a clearly structured mechanism
for policy improvements. Currently, evaluation is essentially non-
existent despite the ad hoc evaluation practices which develop with
no systemic repercussions, as they are mostly externally driven and
created for a single purpose. There is, however, a vaguely defined
authority of ministries and other state administration bodies to monitor
and assess activities within their competences, examine consequences
and take certain actions, or to propose measures to the Government.105
In practice, monitoring is mostly organized for strategies and action
plans, in the form of reports on the state of implementation, but even
this is not done on a regular basis.106
Consequently, it is not easy to measure the impact the evaluations
conducted so far had on the policy making system in terms of its
improvement. In a few cases, impact achieved so far could be analysed
from the perspective of organisations that have been coordinating
evaluations for the purposes of their own work. Nevertheless, there
are some positive, visible trends when it to comes to the real impact
of evaluations in specific domains. In that sense, SIPRU demonstrated
that the evaluation efforts could yield results when it comes to their
actual influence. Firstly, the results of the first round of policy impact
assessments (PIAs) have been presented to government bodies and
civil society organisations as a precondition for further actions in
terms of legislative improvement and policy corrections. In the analysis
of how new measures, policies and/or legal acts in the domain of social
inclusion and poverty reduction were influenced by the evidence
provided by PIAs, it was determined that new legal framework
in education policy was indeed informed by these analyses.107
Additionally, in the development of the strategic framework for rural
development in Serbia, certain recommendations were applied, while
the legal framework on employment introduced a mechanism for an
annual planning cycle, which is a direct influence of the respective
PIA.108 In addition, findings and recommendations of PIAs were used
in drafting a report on Millennium Development Goals, as well as in the
preparation of IPA projects.109
105
Article 13 of the Law on State Administration, “Official Gazette of the RS” no. 79/2005,
101/2007, 95/2010 and 99/2014.
106
107
Interview with representative of SIPRU.
108
109
51
Case by case evaluations managed by the SIPRU had their impact, mostly
in terms of the improvement of the legislative/strategic framework.
What is perhaps more important is that, in conducting evaluations,
lessons have been drawn for the future. These lessons pertain to:
1. Better communication with decision makers in order to boost
ownership and create greater demand (definition of evaluation
subject and goals, organisation of standing meetings, and an
increasing the role of decision makers in consultation processes
with stakeholders);
2. Identification of all implementation agencies and interested
parties, as well as better coordination among them (there is
usually no single institution in charge for the policy domain);
3. Improvement of analysis through the modification of the
selection procedure for hiring experts, taking into account their
credibility and knowledge of the local context and capacities.110
With regards to evaluation work coordinated by the SEIO, effects of
evaluations of EU and other development assistance implemented in
Serbia were mostly used for the purposes of improving development
assistance in the future, i.e. for the next programming period.111
In other words, evaluation results have been incorporated into
planning and programming documents such as the Needs Assessment
Document (NAD) or the Sector Planning Documents (SPDs). In this
regard, evaluation results are useful for the definition of baseline and
target indicators that are used in the programming process and the
production of planning documents.
The examples of SIPRU and SEIO prove that it is possible to achieve
impact through an approach that is ad hoc and confined to serve
the specific needs of a single institution or external requirements.
Together, these two institutions were involved in managing more than
30 evaluations and there is a strong belief among the staff that they
improved the legislative framework and the internal management
processes. Still, these are exceptions and, from the perspective of
evaluating ex-post policy implementation, there is no clear structure
nor obligation. The system of responsible actors and their competences
with a clear “leading” institution is missing in order to make policy
evaluation more structured and transform it into a coherent systemic
obligation, instead of an isolated activity.

110
Interview with representative of SEIO, Sector for Planning, Programming, Monitoring and
111
Reporting on EU funds and development assistance. List of all evaluations conducted so far is
available at: http://www.evropa.gov.rs/Evropa/PublicSite/Documents.aspx.

52
3.4 Highlights from the EU Case Studies – Evaluation

Systems
Evaluation System in the Netherlands: Ever-growing Demand
for Evaluation
Systematic evaluation in the Netherlands was the product of a multi-
year period of demand and a series of attempts. It is said that the
reasons behind the systematization and revival of evaluation are
to be found in different influences. As Leeuw states, “evaluation
blossomed in the Netherlands and informal evaluation policy not only
was formulated but began to evolve into formal policy […] this dramatic
transition can be attributed to four factors: (1) demands by Parliament,
(2) new vigour in the NAO, (3) the New Public Management movement,
and (4) the Dutch administrative culture.”112 As these demands and
pressures were increasing, with a special role of the national audit
office (The Netherlands Court of Audit - NCA) that will be explained
further, formal institutionalization of evaluation started in the 1990s.
Few things were, inter alia, clear at the beginning of this process:
firstly, the importance of and the need for evaluation was clearly
emphasized; secondly, there was an agreement that there is no need
for a separate law; and lastly, evaluation had to be a responsibility of
the top political and administrative management.113
Overall, the institutional design of evaluation is in the hands of
the Ministry of Finance as the driving engine114. It is responsible
for legislation and the assignment of roles and responsibilities. The
correlation of evaluation with the budget is also visible through the
Policy Budget and Policy Accountability reform (VBTB) in 1999, which
introduced performance indicators in order to connect budgetary
decisions to policy objectives (programme structured budget). In
this regard, each minister is responsible for informing the NCA
on policy effectiveness.115 However, the experience has shown
that, while performance indicators and a new way of structuring
112
Frans L. Leeuw, “Evaluation Policy in the Netherlands,” in Evaluation Policy and Evaluation
Practice. New Directions for Evaluation, ed. W.M.K. Trochim, M.M. Mark, & L.J. Cooksy, 2009, 90.
113
Ibid., 93-94.
114
The strong role of the Ministry of Finance in the overall accountability and control structure
is also visible through the Central Audit Service, part of the Ministry, which conducts audits
on behalf of the ministries and is therefore a shared service organization for the central
government. “Public Finance in the Netherlands” (2013), the National Academy for Finance and
Economics, Ministry of Finance, 47.
115
Mickie Schoch and Corina den Broeder, “Linking Information on Policy Effectiveness and
Efficiency to Budget Decisions in the Netherlands”, OECD Journal on Budgeting, 12.3 (2013), 6.
53
and reporting on budget were enough to ensure transparency of

budgetary decisions, evaluations were needed in order to assess
efficiency and effectiveness.116 In other words, an additional reason
for pushing towards more evaluation work was to see what is behind
the figures. The table below illustrates the architecture of the current
evaluation system in the Netherlands:
Table 2. Type of evaluations in the Netherlands
Type of Evaluation Description Responsible agency
Ex ante evaluations
Analysis of costs and benefits Relevant ministry (most

Social cost-benefit in a society, expressed in frequently Ministry
analysis monetary terms of Infrastructure and
Environment)
Other types of ex-ante Cost-effectiveness studies; multi-

Relevant ministry
studies criteria analyses etc.
Ex-post evaluations
Periodic evaluations of the

Policy review current policy programmes (4-7 Relevant ministry
each year)
Effect evaluations assess to

what extent the changes in
Effect evaluations policies can be attributed to Relevant ministry
a particular intervention incl.
counterfactual analysis
Evaluations to identify savings and reform options
Identification of policy options

Spending review for future savings and/or for Ministry of Finance
more value for money
Comprehensive Multiple spending reviews Cabinet (initiated by the

spending review carried out simultaneously. Ministry of Finance)
From: Schoch and den Broeder, “Linking Information on Policy Effectiveness and Efficiency to
Budget Decisions in the Netherlands,” 7.
Ibid., 4.
116

54
Evaluation System in France: Making Scenarios for the Future

With the public policies becoming more complex and inter-dependent,
coupled with the growing public sector expenditures (government
expenditures make 57% of the French national GDP in 2015)117, the
necessity for a more robust system of public policy evaluations inspired
the creation of an institution in charge of steering all public policy
evaluations as well as providing methodological support and quality
control. The General Secretariat for Public Policy Modernisation
(Le secrétariat général pour la modernisation de l’action publique,
abbreviated from French as SGMAP) was established by a Prime
Minister’s Decree in 2012,118 with the aim of acting as an essential
instigator for the implementation of public sector reforms in the state.
While the SGMAP is the main coordinator of policy evaluations, the
principal actor in charge of conducting them is the ministry with
primary authority over the evaluation topic.119 This lead ministry
is appointed by the Prime Minister, who makes the decisions on
evaluation topics based on the ministries’ needs. The role of the lead
ministry is to compose and preside over the evaluation committee
(comprised of other ministries, territorial entities, social partners,
associations, etc.) in charge of providing guidelines, as well as to
appoint the evaluation coordinator and its team, who represents the
operational actor responsible for carrying out the evaluation. What
makes this evaluation system distinct is the fact that the evaluation
teams consist not only of the experts, but also of beneficiaries, whose
inputs are collected, assessed and taken into account in the course of
conducting the evaluations.
Upon providing a “diagnosis” on the state of play of the topic being
evaluated, each evaluation should develop and offer different policy
options, or “scenarios for the future reform”, with the explanation on
their potential impacts, arguments for and against the implementation
of each scenario and an estimation of costs.120
117
Index of Economic Freedom, 2015, available at: http://www.heritage.org/index/
explore?view=by-variables.
118
The SGMAP is under the direct authority of the Prime Minister and reports to the Secretary
of State for State Reform, the Minister of Decentralisation and the Civil Service, the Minister
of Finance and Public Accounts and the Minister of Economy, Industrial Renewal and Digital
Economy.
119
For example, the Ministry of Interior, together with the Ministry of Social Affairs, would be
the lead ministries in charge of conducting evaluation on the policy towards newly arrived
migrants in France.
120
See, for example, the scenarios developed in the framework of evaluation of policy
towards newly arrived foreigners in France, conducted in 2013 by the Ministry of Interior
and the Ministry of Social Affairs and Health. Evaluation report is available at: http://www.
modernisation.gouv.fr/sites/default/files/epp/epp_primo-arrivants_rapport.pdf.
55
Policy Evaluation in the UK: Multi-actor Approach

to Producing Evaluations
The ex-post evaluations in the UK are produced by analysts in
government departments, academics, consultancies, and other
organisations commissioned by the government. In some cases, the
government has set up arm’s-length bodies which commission or
synthesise evaluation evidence, with varying levels of autonomy and
independence. These include the Educational Endowment Foundation
(EEF), the National Institute for Health and Care Excellence (NICE),
and the Independent Commission for Aid Impact (ICAI).
Recently, the UK government has set up a network of ‘What Works’
centres, which are responsible for synthesising evaluation evidence
on the effectiveness of policy in a range of fields. Furthermore, think
tanks, research centres and consultancy firms all offer reviews of
public policy initiatives. However, independent evaluators coming
outside of the government experience difficulties accessing a range of
official and administrative data, which could be used to evaluate the
impact of government interventions.
The Finnish Evaluation System: Centrally Developed
Performance Indicators
In Finland, the Government Policy Analysis Unit is responsible for the
development of the indicator service – Findicator, in cooperation with
Statistics Finland and other ministries. Findicator is a service opened in
2009 provided by the Prime Minister Office’s and Statistics Finland and it
includes approximately 100 indicators for social progress. “The Findicator
service is targeted at everyone needing up-to-date, reliable information
on social progress in their work or other activities: decision makers,
public servants, specialists, teachers, journalists, citizens. The indicators
are selected in consultation with user groups and data providers.”121
This Unit also produces data and prepares publications in support of the
government work on economic policy decision making and foresight.
The Unit prepares the Government annual report which contains
information on the performance and self-evaluation of ministries and
agencies. It is responsible for monitoring the implementation of the
Government programme, and is used to communicate performance
issues through a ‘’traffic light’’ reporting system (RAG - red, amber,
green, or inadequate, reasonable and ideal).122
Findicator, available at: http://vnk.fi/en/findicator.

121
OECD, Finland. Working Together To Sustain Success, Public Governance Reviews, 2010,
122
158-159. Available at: http://unpan1.un.org/intradoc/groups/public/documents/UN-DPADM/

UNPAN042992.pdf.

56
Evaluation in Estonia: Through Budget Reform

to Improved Effectiveness
An important part of developing the evaluation process in Estonia has
been the reform of the budget system towards more performance-
based budgeting. Estonia has reformed its public finance system and
has “developed a strategic planning process based on a medium-
term framework and incorporating performance information.”123
Although the budget law has the traditional line-item form, it also
entails performance information. Performance information that is
included in the explanatory note accompanying the budget law entails
the ministries presenting their main goals, performance targets and
activities as a part of their budget requests, indicating how much
different activity areas would cost.124
The government system is in charge of developing methodologies
on ex-post evaluations. This is important in order to improve policy
making and facilitate policy learning in the administration. NAO of
Estonia is trying to systematically ‘’make the Government’’ improve
their methodology and, after the Government has conducted ante
and ex-post evaluation, regardless of whether it comes in the form
of a strategy, legislation, policy paper, NAO forces it to develop the
methodologies and conduct the evaluations. Representatives of NAO
state that the Government must be forced to improve its methodology
and that is the biggest impact NAO has in Estonia.
3.5 Conclusion: The Pervasive Influence of the

Legalistic Tradition
The current practices of policy evaluation, which is still a novel and
little-known concept in the Western Balkans, are dominated by an
ad hoc approach, with Macedonia being the only country which has
formally introduced a requirement, albeit limited to an evaluation
of legislation. The current debate in Serbia seems to be moving in a
similar direction, although the common methodology which is being
developed by the Public Policy Secretariat pertains to the policy impact
assessment methodology. The relevant documents are currently
in the public discussion phase, which leaves space for instigating a
debate on the various possible types of evaluations, as well as on the
123
Dirk-Jan Kraan, Joachim Wehner and Kirsten Richter, “Budgeting in Estonia,” OECD Journal
on Budgeting, 8.2 (2008), 1-40.
124
Raudla Ringa, “Pitfalls of Contracting for Policy Advice: Preparing Performance Budgeting
Reform in Estonia,” Governance: An International Journal of Policy, Administration, and
Institutions, 26. 4, (October 2013), 605–629.
57
target documents for evaluations. There seems to be little initiative in

Montenegro at the present moment in moving towards a more systemic
approach to evaluation, although recent examples of evaluations of
individual strategic documents do show a growing understanding of
the need to assess the results and impacts of policies. External drivers
and pressures, especially those coming from the EU accession process,
to introduce evaluations are a feature in all three countries, which
may mean that, with the acceleration of EU accession, bolder moves
towards creating more systemic solutions can be expected. The impact
of implemented evaluations in all three countries can be determined
within the described common context which is lacking a well-rounded
policy evaluation structure and where experience is limited. However,
in all three countries evaluation recommendations have influenced
or are expected to influence changes in legislation, as well as specific
implementation aspects of policies in question.
Despite these recent developments, there seems to be little debate in
all of the three countries with regards to different crucial evaluation-
related questions. Firstly, different types of evaluations are essentially
not defined. The highlights of the Dutch evaluation system demonstrate
that different types of evaluation serve different purposes, and pertain
to different levels and aspects of policy implementation. Secondly,
the relationship between the overall strategic planning systems and
policy evaluations is not tackled in WB countries. For example, the
French system opted for an evaluation approach that includes the
central authority (SGMAP) as an evaluation coordination unit that ties
together all actors across different levels (government, line ministries,
and operational units). Last but not least, there is no debate on how
performance-based budgeting affects and sustains evaluation. In the
cases of the Netherlands and Estonia the initial spark for evaluation
actually arose from the need to reform the budget process in a direction
of stronger performance information and linkage to policy objectives.
In the absence of answers to these questions, in all three WB countries,
the new evaluation-related initiatives tend to opt for an extension
of what is already familiar, e.g. RIA (developed and introduced
similarly, under external pressure) as the only formal mechanism for
assessing impacts of new policy instruments. Therefore, the focus of
new evaluation requirements is mostly on legislation, leaving policy
documents and (budget) programmes outside of the loop.

58
4 P
erformance Audit in Western
Balkans: The Third “E” Dilemma
As the most novel form of external state audit, performance audit (PA)
has developed in the context of government reforms and innovations
that mainly had state modernizing, streamlining or even minimizing
as the main objective.125 Thus, PA is just one of the audit categories
which tries to investigate the effectiveness of government actions,
i.e. whether goals of the organization are achieved. In other words,
effectiveness, referred to as the third “E”, requires an auditing activity
that would go beyond measuring the input-output relation of a budget
activity (economy and effectiveness), which is characteristic for
financial and compliance audits. Financial and compliance audits are
much more common overall, not only for the WB countries, and are
seen as self-evident audit activities for many SAIs.
The relevant INTOSAI standards and guidelines summarize the
definition and scope of PA:
Performance auditing is concerned with the audit of
economy, efficiency and effectiveness and embraces: (a) audit
of the economy of administrative activities in accordance
with sound administrative principles and practices,
and management policies; (b) audit of the efficiency of
utilization of human, financial and other resources, including
examination of information systems, performance measures
and monitoring arrangements, and procedures followed by
audited entities for remedying identified deficiencies; and
(c) audit of the effectiveness of performance in relation to
achievement of the objectiveness of the audited entity, and
audit of the actual impact of activities compared with the
intended impact.126
Whereas most SAIs around the world, including Europe, presently
implement PA within their mandates, it is widely understood that
“the form of public management employed will necessarily influence
priorities in performance auditing. In countries where public
125
See: Christopher Pollit et al., “Performance or compliance? Performance Audit and Public
Management in Five Countries,” OUP Catalogue (1999), 87.
126
ISSAI 3000, Standards and Guidelines for Performance Auditing based on INTOSAI’s Auditing
Standards and Practical Experience, 11.
59
management is mainly concerned with means and less involved with

ends, audits also tend to focus on whether rules have been observed
and enforced rather than whether the rules serve or are seen to serve
their intended purpose.”127 According to Pollitt and Bouckaert, among
the twelve developed countries in their study, vast differences were
observed in regard to the level of ambition, with some countries still
maintaining a highly legalistic general culture.128
Additionally, this type of culture can certainly influence the willingness
and readiness of a SAI to experiment and borrow the methods and
approaches from the evaluation field and social science research, and
generally engage in more evaluation-type studies. In PA, it is emphasized
that it is more difficult to define and apply specific requirements and
expectations, and that increased flexibility is necessary for all aspects,
including the methods applied. Finally, PA should use “a wide selection
of investigative and evaluative methods and operate from a highly
different knowledge base to that of traditional auditing.”129 As the
ISSAI guidelines recognise, in the process of performance auditing
“specific skills may be required, such as knowledge of evaluation
techniques and social science methods, and personal abilities such as
communication and writing skills, analytical capacity, creativity and
receptiveness. Auditors should have sound knowledge of government
organisations, programmes and functions.”130
The analysis of the development and state of play in performance audit
in the three studied WB countries in this chapter is conducted mainly
from the perspective of understanding its closeness to evaluation and
the capacities for a development in that direction. The general traits
of PA, such as its origin, development, the overall approach to PA, and
the specific aspects of audit implementation and internal organisation
are described to the extent necessary to understand the specificities
of the field in each country. The chapter proceeds by “telling the story”
of each country’s PA, given that direct comparisons of the different
traits would have limited value (and would, to an extent, be practically
impossible to actualize), due to fairly divergent levels of experience with
PA. Additionally, the impact of PA conducted so far is, in this chapter,
analysed to the extent allowed by the experience of the respective
SAIs. Following the WB case analyses, the main findings from the EU
case studies are provided, selected so as to serve as good practices for
127
Ibid., 20.
128
See: Christopher Pollitt and Geert Bouckaert, Public Management Reform: A Comparative
Analysis - New Public Management, Governance, and the Neo-Weberian State, 3rd ed. (Oxford:
Oxford University Press, 2011).
129
ISSAI 3000, 12.
130
ISSAI 300, 8.
4 Performance Audit in Western Balkans: The Third “E” Dilemma

60
WB countries in their further development of PA. A concise conclusion

is offered at the end of the chapter, summarising the most important
findings with regard to the patterns of development of PA in the WB
from the aspect of contributing to policy learning.
4.1 Performance Audit in Macedonia - towards a

Decade Long Practice
Origin and Development. Performance audit was first introduced in the
Republic of Macedonia in 2005, whereas the first annual Programme
was adopted and the first performance audits were conducted in 2006.
The introduction of this type of audit, according to the staff of the
SAO, was highly influenced by its international (European) contacts
at the time.131 Between 2006 and 2014, a total of 48 performance
audits were conducted, placing Macedonia as the frontrunner in terms
of its experience in this type of assessment compared to the other
two WB countries.132 In addition, in 2014, for the first time since its
introduction, the performance audits dominated the work of the SAO.
A large performance audit on the financial management and control
system, as well as the internal audit system, was conducted, covering
80 entities and resulting in a report offering recommendations for
each of the auditees.133
Table 1. Number of performance audits conducted per year in Macedonia134
134
2006 2007 2008 2009 2010 2011 2012 2013 2014
3 3 3 3 6 7 10 7 6
Requirements and Guidelines. With regards to the legislative

framework for conducting PA, pursuant to the 2010 State Audit Law135,
the State Audit Office (SAO) of Macedonia is in charge of conducting
regularity audit (financial audit) and performance audit. The State
131
Interviews with state auditors, April 2015.
132
According to our interviewees, a number of performance audits are ongoing in 2015 as well.
133
See Annual Report of the State Audit Office of the Republic of Macedonia, available at: http://
www.dzr.mk/en/Uploads/SAO%20Annual%20Report%202014.pdf.
134
T he performance audit on the system of financial management and control and internal audit
adds to this number. As mentioned before, this performance audit was conducted on 80 entities
and 80 audit reports were produced. See Annual Report of the State Audit Office of the Republic
of Macedonia, available at: http://www.dzr.mk/en/Uploads/SAO%20Annual%20Report%20
2014.pdf.
135
State Audit Law, “Official Gazette of the Republic of Macedonia” no. 66/2010. The first Law
on State Audit was enacted in Macedonia in 1997.
61
Audit Law prescribes the conditions and the manner of conducting

state audit, as well as the SAO organization and competences. The state
audit shall be conducted in accordance with the audit standards of the
International Organization of Supreme Audit Institutions (INTOSAI),
and the rules determined in the INTOSAI Code of Ethics, published
by the Minister of Finance in the “Official Gazette of the Republic of
Macedonia.”136 Consequently, INTOSAI standards for performance
audit are practically directly applicable in Macedonia. Pursuant to this
law, the performance audit is defined as an assessment of the economy,
efficiency and effectiveness of the operation and use of funds in a
defined area of activities or programmes.137
SAO is guided by the following documents in conducting performance
audit: the audit standards, PA standards and the State Audit Law.
The first phase of conducting PA is the planning process which
encompasses: preliminary research on the topic; risk assessment;
approach and scope; aim of the PA; a plan of activities along with a
timeframe for performing activities, as well as a proposal for PA. The
implementation phase encompasses: audit evidence (which needs to
be competent, reasonable and reliable), methodology and analysis. The
communication phase relates to the characteristics of the PA report,
its quality, structure and manner of writing, which need to be easily
understandable to a wider, non-expert audience. Quality control and
quality assurance are conducted by a special organizational unit within
the SAO, which is comprised of the most experienced state auditors. All
the phases are implemented according to a well-established procedure
in the PA Manual, one of the most important documents regulating
PA.138
Planning of PA. There is a specific procedure for programming PA in the
internal SAO Guidelines for drafting its Annual Programmes. Firstly,
each of the auditors individually conducts an analysis of a sector or
a government programme that needs improvement, and then drafts
a summary. This 2-3 pages long summary includes the analysis, the
aim of the possible PA, risk assessment, among other elements. After
discussing the summary at a SAO staff meeting, a decision is adopted
regarding choosing the sector which will be subject to PA for the
following year. An annual programme for performing PA is adopted
136
Article 18 of State Audit Law, “Official Gazette of the Republic of Macedonia” no. 66/2010.
137
State Audit Law, “Official Gazette of the Republic of Macedonia” no. 66/2010.
138
However, the PA Manual, as well as other internal documents for conducting PA, are not
publically available on SAO’s website. The first version of the Manual was adopted in May 2004.
In December 2014, the SAO adopted a new PA Manual which is currently in force. The new
Manual regulates the planning process, implementation, communication, quality control and
it contains annexes for all the phases of the process.

62
in the period between September and November for the forthcoming

year, and the PA is conducted in accordance with this programme.
The internal manual for PA includes guidelines on how to conduct the
research (analysis of the sector or the government programme), which
can be included in the annual programme. Once one sector/programme
was subject to PA, there is no repeat audit considering there are many
other areas which were not subjected to PA. The SAO has identified
that the PA process in Macedonia lacks a follow-up study, which is
essential for monitoring the implementation of recommendations.
This is expected to be dealt with in a new SAO Manual for follow-up of
the PA reports’ recommendations, which is currently in preparation.
Alignment with standards. According to the most recent report on the
level of alignment of SAO’s work with the audit standards of INTOSAI
(ISSAI), SAO is 69% aligned with the standards, 29% partially aligned
with the standards, and 2% of the standards are not applicable in the
Republic of Macedonia. On the basis of this Report, the SAO adopted
a Strategy for Implementation of ISSAI standards for the 2013-2017
period.139 The prevailing strategic goal of SAO is the further development
of the quality of state audit, including the sub-goal of enhancing the
efficiency of the planning process and the process of programming PA,
as well as improving the quality of performance audits.
Internal Organisation. Concerning the SAO organizational structure for
conducting PA, from 2005 until 2009, there was a separate department
conducting PA. This department consisted of a relatively small number
of state auditors and could therefore produce only a limited number of
annual PAs. In 2010, the SAO Organisational Chart was amended and PA
has since been conducted by all departments horizontally. The auditors
that had conducted PA within a separate PA department (for the period
2005-2009), after the change of the SAO organisational structure, were
relocated to different departments, facilitating the transfer of know-
how to other auditors. SAO concluded that this manner of managing
PA, i.e. disseminating it to all departments, is more constructive.
Training and Background. The SAO state auditors have a practice of
attending a five months in-service training in the EU Court of Auditors,
which operates as a train-the-trainers scheme, with an obligation for
these auditors to continue working for the SAO for a number of years.140
Nonetheless, this is a general obligation for all auditors and there are
no specific training and qualification requirements in order for a
139
SAO Strategy for implementation of ISSAI standards is based on the following standards:
ISSAI 100 – basic principles of public sector audit; ISSAI 300 – Fundamental Principles of
Performance Auditing; ISSAI 3000 – Standards and Guidelines for PA, and ISSAI 3001.
140
16 auditors have completed this training so far.
63
person to work on PA, other than the general and specific conditions
one has to meet to become an auditor. These include an educational
background in economics, law, information sciences, or other relevant
education for conducting audit, including an exam which can be taken
after 5 years of experience in accounting or financial matters or three
years of working experience on issues of audit or control.141 From that
criteria, and its application in practice, it is clear that there is no trend
of diversification of the skills of the auditors, which are indispensable
for conducting performance audit, as foreseen in the ISSAI standards.
Capacity Building and TA. The SAO has benefited from capacity/
institutional building activities for many years, which was, according
to our interviewees, fundamental for establishing the capacities for
conducting PA, both in order to fulfil the SAO mandate pursuant to the
State Audit Law and to establish PA as a corrector of policies.142 One of
the areas encompassed by foreign assistance was the development of PA
methodology and practice. Between 2003 and 2005, the introduction of
PA was reinforced by a World Bank project which supported trainings of
SAO staff. After the World Bank project, the most substantial assistance
was the technical assistance coming from the Netherlands.143 In 2010,
the Netherland Court of Auditors signed a cooperation project with
SAO for the period of five years, which is still on-going. Within this
cooperation project, the Court of Auditors of the Netherlands provides
technical assistance to SAO for the implementation of performance
audits, regularity audits, quality management, human resources
management, and other topics. As a part of the set of performance
audits, the SAO has also audited donor projects directed to national
institutions, and has assisted and participated in coordinated
international audits.144 For example, in 2007, the performance audit of
the Ministry of Justice project for procuring equipment, designated for
141
Articles 14 and 15 of the State Audit Law, “Official Gazette of the Republic of Macedonia” no.
66/2010.
142
This was the common opinion of all 8 interviewed state auditors from the SAO in Macedonia.
143
For the period 2005-2008 there was a twining project and from 2008-2010 there was
a MATRA project for technical assistance to SAO. The main aim of the MATRA project was
strengthening SAO capacities for producing high quality audit reports in accordance with
the EU acquis, as well as with the relevant international standards and best practices. The
project activities were divided into four components: improving the legislative framework,
strengthening the organizational and administrative capacities of the SAO, enhancing the audit
skills and IT audit. SAO was also a beneficiary of a twinning project by the Netherlands Court of
Auditors that lasted for 2 years (2011-2012).
144
An international coordinated audit combines several audits on the same topic conducted
by different SAIs in their respective countries according to an integrated planning approach.
See: Victor Hart, Luciano dos Santos Danni, Paula Hebling Dutra, and Brazilian Court of
Audit (TCU), “Coordinated Audits as a Capacity Building Strategy,” International Journal of
Government Auditing (October 2013), accessed September 4, 2015, http://www.intosaijournal.
org/highlights/audits_capacity_building_oct2013.html.

64
the courts in Macedonia as a part of the 2000 PHARE programme, was

realised.145 The Ministry of Justice coordinated and implemented the
Public Administration Reform – Judicial Administration project, while
the European Agency for Reconstruction concluded and implemented
the procurement contracts for goods and services, with funding
coming from EU programmes for Macedonia. In the course of 2009,
there was an international PA for a climate change project, while in
2010 a PA for the policies for employment of people with disabilities
and a PA for managing public debt were conducted. Currently, that is,
in 2015, SAO is participating in two international PA, one for tourism
development policies with Croatia and the second for energy efficiency
and renewable energy sources with Serbia and other countries from
the region. State auditors consider the coordinated international
audits as a very useful tool for capacity building.
Cooperation and Communication. As a rule, the institutions subject
to PA cooperate with performance auditors. However, the level of
cooperation depends on the subject matter and the ministries/
institutions involved. The institutions are obliged to, within 90 days
of the performed audit and submitted audit report, inform the SAO on
what measures/activities they will undertake in order to implement
the SAO recommendations. As argued by our interviewees, the
implementation of the SAO recommendations is issue/institution-
specific and depends primarily on the awareness of the management
structure of the ministry/institution. The communication in all three
phases of PA is specifically regulated by the PA Manual. Auditors
regularly communicate with the institutions that have an on-going
PA. After the PA is finalized, SAO organizes a presentation attended by
all of the institutions involved. The draft reports are submitted to all
relevant stakeholders. The PA reports are published on the SAO web
page and are submitted to the Assembly of the Republic of Macedonia,
as well as to all relevant stakeholders subject to PA.
Follow-up. In terms of follow-up, since 2009, the Government has been
devoting a special session to the annual report and other audit reports of
the SAO, signalling the increased importance of SAO’s recommendations.
However, as the SIGMA 2013 report notes, “the implementation of SAO’s
recommendations has apparently had no visible impact on the average
number of auditees affected by an overly high level of irregularities.”146
Still, as was outlined by our interviewees, the focus of these cases was
145
EU Project EuropaAid/112995/D/S/MK. PHARE programme 2000 – Procurement of
IT equipment for Public Administration Reform- Judicial Administration with a value of
508.183,00 EUR.
146
The Former Yugoslav Republic of Macedonia Assessment Report 2013, SIGMA Country
Assessment Reports 2013/10. Available at: http://bit.ly/1LojOkK.
65
foremost on the regular financial audits, rather than PA per se. The
interviewed SAO auditors also hold the opinion that institutions are
more responsive to the performance audit reports than to the financial
reports. A specific positive example in this respect is the 2010 PA
report on the programmes and measures for increasing employment
of persons with special needs, which determined that the provisions
of the laws were not correctly implemented. The PA report focused
on three main points in order to improve the legislative framework:
strengthening the control over the implementation of the laws,
ratifying the Convention on the Rights of persons with disabilities, as
well as the European Social Charter, and determining another mode of
funding. According to the interviewees, the PA report contributed to
the adoption of a new law in line with the recommendations. Several
other examples such as changes in the laws on health care were also
given. They also emphasise that the degree of implementation of the
recommendations from the PA report depends on the area and the
character of the measures that should be adopted. Interviewees viewed
that the main challenges in the implementation of the PA reports, in
addition to increasing their number in general, are: raising awareness
among the institutions, improving the dissemination of the reports,
public/open discussions, and introducing mechanisms for monitoring
of the implementation. The institutions need time to implement some
of the measures, especially concerning specific and sensitive issues.
Concerning the follow-up, the institutions have 90 days to inform the
SAO on whether they have implemented the recommendations from
the PA Report. Presently, the SAO is drafting a new Manual planned to
be adopted in September 2015, which will introduce a new manner of
follow-up regarding the recommendations from the PA reports.
Quality. In 2007, the German state audit institution, in its Peer Review
Assessment of the State Audit Office of Macedonia, assessed that “the
audits did not address government performance in a wider sense but
rather the compliance of the programmes with the set objectives or
EU rules. The question as to whether the effects achieved are in a
reasonable proportion to the allocated funds and whether there are
alternative, more efficient options, did not play a major role. The audits
generated a large number of findings that rather address regularity
aspects e.g. the compliance with requirements for approval or with
reporting duties”.147
The essential question is to what extent the SAO has managed
to improve the quality of its performance reports. Although the
Bundesrechnunghof, Peer Review Concerning the Macedonian State Audit Office Final
147
Report, Gz.: I 5 – 2007 - 1009 Bonn, 2007.

66
interviewees have assessed the quality of PA reports positively, local

research reveals several shortcomings. Performance audit reports in
Macedonia are “missing a vital part to complement performance based
budgeting, i.e. the efficiency and effectiveness audit in comparison with
the set/planned performance/goals and evaluation of the taxpayers’
value for money”.148 As is the case in other countries as well, the lack
of performance-based budgeting has been identified in local research
and confirmed by our interviews as an impediment to achieving the
full potential of performance audit in general.
Our comparison of the performance audit reports in 2007 and the
performance audit reports in 2014 leads to the conclusion that the
quality of the audits has improved.149 The goal of the audit is more
clearly set in terms of economic, efficiency and effectiveness criteria.
Although compliance is still in focus, performance is analysed much
more thoroughly, based on the identification of risks, which results
in more elaborate audit findings that can be valuable for policy
evaluation. Still, it cannot be claimed that audits address government
performance in a wider sense. In addition, the SAO does publish the
audit findings, but does not publish the applied methodology, which
is annexed to the Audit report. The focus of SIGMA assessments has
been on regulatory and compliance audit. SIGMA has assessed that
performance audit is “developing gradually, but at a steady pace.”150
This general assessment corresponds to our findings.
Overall, it can be concluded that, with a decade long experience
of PA, Macedonia can serve as a valuable source of experiences and
lessons learnt for other countries in the region. Performance audit
is well-established both in procedural terms and in practice, largely
because of the pro-active role of the SAO. The main challenges lie in
improving the quality of performance audit, introducing mechanisms
for monitoring implementation of the recommendations, and raising
awareness among the institutions on the role of performance audits
in general.
148
Nikolov, Trenovski and Tasevska, “Feasibility study: performance based budgeting for
enhanced budgeting process and increased transparency and accountability,” 37.
149
Two performance audits were compared: Performance audit of the programme for the
protection of children in 2005 and 2006 (2006) and Performance audit on the efficiency of the
measures to ensure ambient air quality in the Republic of Macedonia (2014).
150
OECD/SIGMA, The Former Yugoslav Republic of Macedonia Assessment Report, 2013, 40.
Available at: http://bit.ly/1GAk8fb.
67
4.2 Performance Audit in Montenegro - a Thorny

Road towards Performance
Background. Recently marking its ten year anniversary, the State
Audit Institution in Montenegro is a relatively young institution which
has produced over 150 audit reports on budgetary units, funds, public
companies, municipalities, et cetera. A great majority of these reports
are compliance and financial audits, while there are some follow-up
(control) audits, as well as very few performance audits, taking into
consideration that some of the early work classified as PA does not
really satisfy the relevant international standards. SAI classified
some of its early audits as “performance audits”, simply because they
stepped out of the narrow financial and compliance audit barriers.
It seems as something that the SAI always wanted to do, but did not
know exactly how to or had the resources to conduct. It is important
to emphasize that SAI’s mandate to specifically conduct the audit of
“success” or performance of managing state funds and property has
been enshrined in the Constitution itself.151 The current situation is
not much better and SAI faces both internal constraints and outside
barriers to performance auditing. Besides lacking experienced staff
and resources, SAI is operating in an environment that does not have a
culture of measuring performance or thinking about goals in relation
to public funds allocated for the implementation of policies.
The usual domestic audiences of SAI’s work in Montenegro (journalists,
NGOs, MPs, independent experts and, more rarely, government
officials) do not exert pressure upon SAI to conduct more performance
audits. The emphasis is still on compliance and financial audits that
are more straightforward and often carry revelations of gross and
significant misuses of public funds, misdemeanours and irregularities.
Additionally, there are constant requests for more reports and the SAI
must increase the number of audit reports published annually, in order
to cover as much of the public sector as possible. Performance audit in
this kind of environment is perceived as too advanced, whereas the
simpler forms of audit reveal results that are more “sensationalist,”
easier to understand and report on. The pressure to develop
performance audit seems to be coming externally, mainly as a part of
the EU accession process and membership negotiations.152 The Chapter
151
Article 144 of the Constitution of Montenegro, “Official Gazette of Montenegro” 01/07. In
some countries, the constitution or legislation in force do not always confer on the SAI the
authority to audit “effectiveness” or “efficiency” of the financial management of the executive.
152
The latest, 2014 European Commission Progress report on Montenegro emphasizes
low capacities for PA. See: Page 59 of http://ec.europa.eu/enlargement/pdf/key_
documents/2014/20141008-montenegro-progress-report_en.pdf.

68
32 Screening Report has indicated the low level of development of

performance audits as one of the stumbling blocks.153
Organisational setup. Montenegrin SAI has wandered around a bit
when it comes to institutional set up for conducting performance
audit. SAI’s Strategic Development Plan outlines the introduction of
performance audit as one of the goals for the period from 2012 to 2017,
but does not go into much detail about the specific goals that are to
be achieved in this area.154 The first scenario, which was tried out
in 2011, was based on the model previously used for the audit of IPA
funds. It consisted of setting up a department that would be tasked
with developing a performance audit methodology which was not
attached to any of its five sectors,155 and was headed by one of the senior
auditors. This body was cooperating with the Swedish NAO in terms of
expert help, while its task was to develop methodological guidelines
for performance audit, as well as to conduct pilot audits of this type.
Its first performance audit was focused on the topic of the system for
collecting the income tax.156 However, this audit was abandoned and
never published, as this whole working mode proved to be ineffective.
In February 2014, a new PA Department was created as a special unit
within one of the SAI’s sectors, directly run by one of the SAI Senate
members. The Department has since conducted two performance
audits and played a key role in defining the methodological guidelines
for performance audit which were adopted in 2015. It is safe to say that
even this solution is not viable in the long term, due to the difficulties
that will arise from the fact that the PA department operates within a
sector which clearly has a limited scope of audit entities, in accordance
with the general organisation and division of work in the SAI.157 As
a result, it is not clear if the PA department can encroach on another
sector’s turf with its audits, or if it is limited in its scope to the audit
entities which fall under its sector’s purview. Although it is too early to
discuss radical organisational changes caused by the PA requirements,
it is expected that the SAI will eventually end up forming a special
sector which will conduct only this type of audit, run by one of the
153
Screening report Montenegro, Chapter 32 – Financial control, 2013, available at: http://
ec .europa .eu/enlargement/pdf/monteneg ro/screening _ repor t s/screening _ repor t _
montenegro_ch32.pdf.
154
SAI’s Strategic Plan of Development 2012-2017, 34. Available at: http://www.dri.co.me/1/
index.php?option=com_wrapper&view=wrapper&Itemid=52&lang=en.
155
SAI is divided in five sectors, each covering a number of audit entities or thematic areas,
and headed by one Senate member. The division is not clear-cut, and certain areas spill over to
different sectors, while others, such as municipal budgets or the security and defence sector,
belong to specific sectors as a whole.
156
SAI’s 2012 Annual report, 208. Available at: http://www.dri.co.me/1/images/godisnji%20
izvjestaj.pdf.
157
The list of Sector Two’s audit entities is available here: http://bit.ly/1M6SF6i.
69
Senate members. The question to be answered here is whether it is

realistic to expect that the expertise for conducting performance
audits can be developed across the institution or whether this should
be reserved for a special department, while the rest of the SAI deals
with compliance and financial audits.
Human resources. Montenegrin SAI is a relatively small institution,
with 80 work positions systematised. Out of that number, 76 positions
are intended for audit staff, while the rest is administration. Only
47 auditors have been hired, while the remaining 29 posts remain
vacant.158 That means that the SAI currently operates with around
60% of its planned human resources, and that the PA department
is understaffed as well. Currently, there are five work positions
systematised in the performance audit section, only two of which
are filled. These two persons are experienced auditors that had
previously worked in the same sector on financial and compliance
audits, and have been transferred to conduct PA. There are various
issues related to filling these vacant positions, they range from the
shortage of qualified staff on the labour market, to a great disparity
in wages between the state and commercial audit sector, which
makes SAI less attractive,159 as well as a complicated procedure
in which the SAI is obliged to have the approval of the Ministry of
Finance for each new hiring.160 In these conditions, it is particularly
demanding to plan human resources for the performance audit
division. Issues of capacities for PA are even more important when
the ratio between the number of staff and obligatory annual audits
are taken into account.161 On the other hand, performance audit can
be seen as a tool for increasing the audit scope (coverage of public
spending), due to the fact that it allows for an approach focused on
topics covering a large number of entities at the same time. This was
one of the recommendations of the SAI’s peer review report, seen as
158
Data from the latest SAI’s Annual Report, 319. Available at: http://bit.ly/1MEkXFL.
159
The only available data on this disparity (from the 2010 Government’s answers to the EC
Questionnaire) shows that the average salary of a state auditor is three times lower than that
of an auditor in the private sector.
160
As an additional burden for its capacities, for almost five years SAI has been functioning
with only four out of five of its Senate members (who are collegially running the institution,
with the Senate President being the first among the equals), due to the inability of the political
parties to agree upon a suitable candidate.
161
For example in 2012, the Law on Financing of Political Parties was amended and an obligation
for SAI to conduct audits of annual financial statements of all political parties was included,
along with the obligation to conduct an audit of electoral campaign financial reports. Along
with the obligatory audits of the final national budget statement (year-end budget report), this
represents a significant burden on the capacities of SAI.

70
a way for a small institution such as SAI to meet its challenging audit
remit.162
SAI’s official examination tests for passing the state auditors’ exam
or its official training plan do not cover performance audit topics. SAI
is in charge of conducting official national tests for certification of
state auditors, as well as conceiving the programme, contents and
procedure of the tests.163 Its outdated programme for the official
examination process (adopted back in 2006) does not mention
performance audit, nor does it cite any resource which indicates
that the examinee has to be informed about the performance audit
methodology or standards.164 On the positive side, there are no
formal barriers for people with diverse backgrounds to become state
auditors. The Law does not exclude those with a background other
than economics and law to become state auditors. An individual
coming from any educational or professional background can become
a state auditor, as long as the official test is passed, along with
some formal requirements set out by the Civil Service Law.165 This
is important for the future development of human resources in the
field of performance audit given that they should encompass a staff
incorporating a variety of backgrounds. Additionally, there is a legal
possibility for the SAI to contract services from external experts for
conducting audits,166 however, this option is rarely used, and has not
been utilised for the purposes of PA so far.167
One of the ways external pressure acts as a motivation for developing
capacities for performance audit is the regional process of learning
through practice in the form of conducting parallel performance
audits. The European Court of Auditors has supported the network of
candidate countries’ SAIs in carrying out a parallel performance audit
on energy efficiency,168 by providing experts on the subject matter
and on the methodology of performance auditing.169 In addition,
financial aid from the EU has helped achieve concrete steps in building
organisational capacities and human resources for performance audit.
162
Peer Review of the National Audit Institution of Montenegro, 2011, 26. Available at: http://
bit.ly/1hTP9nT.
163
Law on State Audit Institution, “Official Gazette of the RS,” no. 101/2005, 54/2007, 36/2010,
Article 38.
164
Rulebook for the programme of passing the state auditor’s exam, “Official Gazette of
Montenegro” 44/06. Available at: http://www.dri.co.me/1/doc/Pravilnik-o-programmeu-
polaganja-ispita-za-drzavnog-revizora.pdf.
165
Law on SAI of Serbia, Article 45.
166
Ibid., Article 46.
167
Information from interview with SAI staff, March 2015.
168
See more at: http://bit.ly/1W5N3nH.
169
European Court of Auditors’ 2014 Activity Report, 41. Available at: http://bit.ly/1INTWl1.
Additionally, Montenegrin SAI 2013 Annual Report, 380. Available at: http://bit.ly/1jTLjwx.
71
An EU-funded project has, since 2009, laid the foundation for the
development of performance audit in Montenegro. It produced the first
seminars on this topic and the formation of a special performance audit
department in the SAI.170 This work was continued with the EU-funded
project on audit quality control, during which the organisational set-
up was redefined and methodological guidelines for performance
audit were adopted.171
What has been done so far - classification troubles. When
determining the number of performance audit reports, one can
have a particularly hard time coming up with an exact figure. This
is an indicator of the thorny road of the Montenegrin SAI towards
developing its PA methodology. In its work so far, SAI has officially
conducted two performance audits, the first dealing with the work
of the Secretariat for Development,172 and the other with the Tax
Administration.173 These are the only reports which can be found
under the Performance Audit heading at the SAI’s website,174 although
there are other audit reports classified as performance audits from
earlier years. The first performance audit, focusing on the Human
Resource Management Authority, was published in 2009, barely five
years after the establishment of the institution. In the meantime,
other institutions, mainly smaller budgetary units, have been
subjects of PA: the Property Administration, the Fund for Minorities,
Montenegrin Agency for Foreign Investment Promotion, and the
Department for International Scientific, Cultural, Technical and
Educational Cooperation. However, PA in Montenegro is objectively
“still in its infancy.”175 The reason why these earlier performance
audits “don’t count” anymore is not only the adoption of a special
methodology for PA in 2015,176 but also an acknowledgment of the
fact that these reports did not exactly adhere to the international
170
SEA EU Project, ‘Strengthening of external audit in Montenegro, 09ME01.10.1, entailed a
realization of the institutional cooperation with the National Audit Office of Sweden referring
to conducting performance audit, setting up the Performance Audit Authority for development
of performance audit within the organisational structure of the SAI, as well as drafting a Pre-
study for pilot performance audit and beginning the process of performing this audit.
171
IPA 2012 Twinning Light Project, “Audit Quality Control in the State Audit Institution of
Montenegro.“
172
Performance Audit Report on Secretariat for Development Projects “Monitoring the Status
and Dynamics of Investment Projects“, 2014. Available at: http://bit.ly/1NTZxuh .
173
Performance Audit Report on Efficacy of the Tax Administration in payment of Tax Debt,
2015. Available at: http://bit.ly/1W4fhdq.
174
Performance audit section at the SAI’s website: http://www.dri.co.me/1/index.
php?option=com_content&view=article&id=165&Itemid=212&lang=sr.
175
SIGMA’s Public Administration reform assessment of Montenegro, April 2014, 26. Available
at: http://www.sigmaweb.org/publications/Montenegro-Assessment-2014.pdf.
176
Full title: Instruction on Methodology for Conducting Performance Audit, adopted in January
2015.

72
standards of performance auditing. Before the adoption of special

methodological guidelines for PA and the establishment of the special
PA department, there seemed to be a tendency to introduce some
elements of the performance audit methodology in the “classic” audits.
In some of the compliance and financial audits, special attention was
paid to the elements of performance, while others were initiated as
compliance audit, only to turn into a performance audit during the
process. While the auditing standards foresee the possibility of such
an overlap, the newer tendencies in the SAI adhere to a more strict
methodological division.
Performance vs. financial/compliance audit. Methodological
differences between the compliance and financial audit on one hand,
and the performance audit on the other, as understood by the SAI
through it methodological guidelines, are numerous. Perhaps most
importantly, the process of preparing and proposing an audit for the
annual audit plan is different. Performance audit requires the auditor
to prepare the so called pre-study, a detailed research concept that
would prove why the chosen topic is important and whether the
conditions in place would enable SAI to conduct a performance audit.
In a very detailed procedure, the auditor must conduct a thorough
research into the topic and present the case to the head of the Sector,
who then decides whether the actual performance audit is necessary
or feasible. Only then can the auditor create a detailed audit plan. This
procedure is completely different and much more demanding than the
one required for a compliance or financial audit to be proposed for the
SAI’s annual audit plan. One of the differences in the methodology for
writing audit reports that distinguishes the performance audit, is the
exclusion of the opinion (adverse, qualified or unqualified) which is
obligatory for the compliance and financial audit reports.177
Scope of the performance audits. The policy basis of government
programmes has not been tackled in the SAI’s performance audits.
All published performance audits are related to the functioning of
individual institutions (audit entities) and not to horizontal processes.
This is perhaps a reflection of a narrower definition of performance
audit that is given in SAI’s audit reports (not present in the legal acts or
methodological papers), and which states that the goal of the PA is to
see whether the “institutions are doing the right job and are they doing
it in the right way, with the least expenses possible.”178
177
As foreseen by the Instruction on Methodology of Conducting Financial and Compliance
Audit, “Official Gazette of Montenegro” no. 07/15.
178
Performance Audit Report on the Secretariat for Development Projects “Monitoring the
Status and Dynamics of Investment Projects“, 2014, 3-4.
73
Quality of PA reports and follow-up. One characteristic trait of

Montenegrin PA reports is that they cite international good practices.
In both of the audit reports, secondary literature (SIGMA’s papers,
European Commission’s and World Bank’s materials, academic papers
and best practices from various sources) was used, in addition to the
usual audit source materials. Also, in both reports, comparative insights
and examples from other countries are used as additional arguments
as to why a change is necessary and why the model proposed by the
SAI is viable. Additionally, these reports contain the audit question
with sub-questions which were defined in the beginning of the audit
process, and which the whole process should give an answer to, as well
as including a detailed elaboration of the methodology.
In the Audit of the Secretariat for Development Projects, one of the two
audit question the SAI answers is “Does the Secretariat have an impact
on the enhancement of investment and business environment?” However,
instead of measuring the impact of the Secretariat’s work, which
would entail a wider study of its role, position and consequences of its
activities, the SAI narrows down its focus to a slightly modified version
of a compliance audit, in which it inspects whether the Secretariat is
doing its job and fulfilling all of the obligations and competencies it
has, with an overview of the overlapping of competencies of other state
bodies in conducting this policy. The audit of the Tax Administration is
a similar case which, instead of examining the effects of the operations
of the Tax Administration, comes down to inspecting whether it
implements all the legally prescribed procedures in order to lower
the level of tax debt. In both cases, however, auditors do come up with
systemic recommendations about the impact of certain actions of the
auditees and, in some cases, ambitious recommendations on what
needs to change.
SAI’s recommendations in PA reports resemble those of a policy paper,
with multiple policy options given for certain areas, though with little
or no rationale about the advantages and disadvantages that the choice
of a certain option would have. In addition, as it is sometimes the case in
other types of audits, recommendations stemming from performance
audits demand action from a variety of actors, which then makes it
unclear on who should report to SAI. At the end of each audit report,
SAI demands that the audit entity reports back after a certain time
period (usually six months). It is the legal obligation of audit entities to
report back to SAI on what they have done to tackle the problems and
implement the obligations stemming from SAI’s recommendations.179
179
Law on SAI, Article 15, “Official Gazette of the Republic of Montenegro” no. 28/04, no. 27/06,
no. 78/06, no. 17/07, no. 73/10, no. 40/11, and no. 31/14.

74
However, there is no defined template for this report, which is abused

by the auditees, with some of them sending in only yes-no answers with
little or no explanation of which concrete steps have been taken. Unlike
the usual audit report follow-up demands, in PA there is an obligation
for the audit entity to prepare a plan for the implementation of the
recommendations as well. However, according to our findings, neither
of the institutions targeted by the two aforementioned performance
audits has yet delivered such a plan.
Performance-based budgeting as a precondition for performance
audit. An important factor which influences the use of performance
auditing in Montenegro is that very little emphasis is placed on results-
oriented functioning of the public administration and the goals of
public finance management.180 Accordingly, one of the great barriers to
a more successful and methodologically sound performance auditing
is the lack of performance-based budgeting. In international literature,
it has been observed that “in general it appears that performance
auditing and performance budgeting have developed along separate
paths, but recently have been brought more closely together through
increasing emphasis on published performance indicators.”181 Although
this reform of the public finance system has been ongoing for over ten
years, it has not brought any results thus far. This means that the budget
users at the central level do not plan their work in accordance with
goals and activities, therefore not identifying indicators for measuring
their success. This lack of strategic planning in the administration as
well as a complete absence of goals and set targets, is a considerable
180
“In countries where public management is mainly concerned with means and less involved
with ends, audits also tend to focus on whether rules have been observed and enforced rather
than whether the rules serve or are seen to serve their intended purpose. In countries that have
acknowledged management by objectives and results, the audit focus is different. Public sector
management generally displays a combination of these philosophies.“ ISSAI 3000 Standards
and Guidelines for Performance Auditing based on INTOSAI’s Auditing Standards and Practical
Experience, 20. Available at: http://www.issai.org/media/13224/issai_3000_e.pdf.
181
David Shand, “Performance Auditing and Performance Budgeting,’ in Performance
Budgeting: Linking Funding and Results,” ed. Marc Robinson (International Monetary Fund,
2007), 88-112.which aims to link the funding of government agencies to the results they deliver
to improve the efficiency of public spending. In a combination of thematic studies and case
studies, it clearly presents the diverse range of contemporary performance budgeting models
and examines their effectiveness. Its coverage is truly international, spanning developed,
developing, and middle-income countries. Reflecting this, its case studies range from the
cutting-edge use of performance targets in the budget process in the United Kingdom to the
implementation of performance-based university funding in Ethiopia. The prerequisites for
effective performance budgeting--including the development of good performance measures
and accounting systems for costing the results delivered by government--are systematically
treated, as is implementation strategy. For more information on how to purchase a copy of this
title, please visit http://www.palgrave.com/economics/imf/index.asp.”,”ISBN”:”978-1-4639-
2070-8”,”shortTitle”:”Performance Budgeting”,”language”:”en”,”author”:[{“family”:”Robinso
n”,”given”:”Marc”}],”issued”:{“date-parts”:[[“2007”,10,17]]}}}],”schema”:”https://github.com/
citation-style-language/schema/raw/master/csl-citation.json”}
75
obstacle for the development of both evaluation and performance audit.

There are announcements that, during the next two years, this reform
of public finance will have high priority for the Ministry of Finance.
Any progress made in this area can only help performance auditors
do their work better and more systematically, but can also create the
currently non-existent culture of measuring performance in the public
administration.
Impact of PA reports. It is fairly early to discuss the impact of the two
performance audit reports published by the SAI due to the fact that
both were published in the last 12 months, the first being published in
July 2014, and the second in June 2015. Experience from the country
cases such as Estonia and Finland shows that actual changes in the
administration, as a consequence of SAI’s reports, take several years
to be implemented.182
While it is still early to talk about the substantial impact of the reports,
in the terms of changing policies, regulation or the way the institutions
function, both of them had topics that were relevant in the daily
political debate as well as in policy cycles. It is important to note that
there were no public arguments over the audit reports and that the
audit entities did not dispute the SAI’s findings.
The Audit of the Secretariat for Development Projects has a number
of recommendations that aim to solve the problems that have been
registered by the SAI and which relate to imprecise and overlapping
functions of several institutions, as well as poor oversight of the
concluded contracts and projects in implementation. The situation has
not yet changed in this regard, although the report has had a lot of
publicity and various actors (such as the Speaker of the Parliament and
the smaller member of the ruling coalition183) are pushing for it to be
implemented.
The performance audit on the work of the Tax administration has
revealed new and confirmed the previously rumoured issues in the
work of this institution, as well as the whole system of tax revenue.
The Ministry has published a press release stating that they agree
with the findings and the recommendations in the SAI’s report, and
182
Reports which achieved the most impact in Estonia were the ones which dealt with
auditing state companies and the report on sustainability of the pension system. The latter
was produced in 2010, while the most important changes occurred in 2014. State officials,
such as the Minister of Finance, the Minister of Social Affairs, the Minister of Defence, and the
Minister of Interior expressed their accordance with the NAO findings defined in the Report
and promised to conduct certain measures in order to fulfil NAO recommendations.
183
See: “Krivokapić: Parliament is not stopping Government’s projects,” Vijesti, October
4, 2015, http://www.vijesti.me/vijesti/krivokapic-netacno-da-skupstina-koci-vladine-
projekte-854251.

76
that they will act in order to implement them.184 This explicit statement
was issued after extensive media coverage of the report that brought
to light many inefficiencies in the tax collection system and register. A
couple of months later, the Head of the Tax Administration submitted
his resignation, citing personal reasons for such a decision.185
4.3 Performance Audit in Serbia - Accountability

before Effectiveness
Origin and Development. The Serbian SAI has taken a cautious approach
with regards to developing performance audit, as it is methodologically
the most complex type of audit and an “advanced management tool.”186
It was introduced very recently, in 2013, and hence represents a novel
mechanism for improving the management of public funds in the
country. The State Audit Institution (SAI) is still developing its internal
performance audit capacities and is yet to accumulate more substantial
experience in this field of audit. Moreover, the concept of performance
per se is still underappreciated in the Serbian public sector as a whole
and, if there is a notion of performance in the public sector at all, it
only echoes in the distance rather than being clearly defined, the
administrative culture being “based on rule compliance rather than
performance and characterised by a high degree of risk aversion.”187
This has affected the work of SAI, which, in its first 6-7 years, primarily
focused on financial irregularities and non-compliance aspects of the
operations of public entities, with a goal of merely creating order in
the management of public finances. While almost all the workload
pertains to the financial and compliance audits, SAI has managed to
fully implement two pilot performance audits. Accordingly, there
has been a strong dependency on experience derived from financial
auditing in conceiving and implementing the first performance audits.
It can be said that this focus on compliance is not without a reason
and that the strengthening of the performance audit function is not
184
See: “Ministry and the Tax Administration will implement SAI’s recommendations,” Vijesti,
June 16, 2015, available at: http://www.vijesti.me/vijesti/ministarstvo-finansija-i-pu-ce-
postovati-preporuke-dri-838424.
185
See: “Lakićević resigns, succumbing to criticisms and pressure,” Vijesti, September 19, 2015,
available at: http://www.vijesti.me/vijesti/podlegao-pritiscima-i-kritikama-milan-lakicevic-
podnio-ostavku-852011.
186
Dalia Daujotaite and Irena Macerinskiene, “Development of performance audit in public
sector” (paper presented at the 5th International Scientific Conference Business and
Management, Vilnius, May 2008), 184.
187
Vid Štimac and Mina Lazarević, “Public Management Work Context and Reform in Serbia,
Results from a Large Scale Survey of Senior Public Executives,” Belgrade Fund for Political
Excellence, The Social Inclusion and Poverty Reduction Team of the Government of the Republic
of Serbia (2013), 48.
77
only dependent on the willingness of SAI. The screening report of

the European Commission states that it takes a lot more work on
the systemic level, meaning that “Serbia needs to ensure parallel
improvements in the overall public financial management system for
the SAI to assess performance in a meaningful manner.”188
Capacities. The initial workforce of SAI consisted almost exclusively
of professionals in law and accounting, with previous work experience
in the public sector. Thus, the overall state of capacities, in terms of
staff and skills needed, have probably influenced the delay in the
introduction of performance audit, while it has become evident that
capacity development is necessary. The performance audit function
of SAI has been considerably strengthened since the beginning of the
work of SAI in 2007, owing to a great extent to development assistance.
In general, there are two methods commonly used for strengthening
staff capacities for performance auditing, recruitment of new staff for
performance auditing purposes only, and/or retraining of financial/
compliance auditors at disposal.189 Serbian SAI opted for the latter
option which was conducted through the EU and other technical
assistance. It can be discussed if the selected option was preferable
in terms of advancing the nascent performance auditing function,
however, it should be noted that “newly recruited staff, with a different
background than those recruited in the past for financial auditing,
would have to undergo training in performance auditing, as this is
hardly a profession taught at universities, but a highly specialised type
of investigative work which has to be understood in the context of the
national accountability framework and the internal procedures of the
SAI concerned.”190 In that regard, there is no right or wrong approach
for SAI’s staffing policy, but a combination of these approaches should
be taken into consideration if performance auditing is to be further
developed. This variety in thinking and approaches can be particularly
beneficial in the development of performance audit, in view of its
specificity with regards to other types of audit, emphasized also in
the international auditing standards.191 In that sense, it is important
188
Screening Report Serbia, Chapter 32 – Financial Control, 2014, 10. Available at: http://
ec.europa.eu/enlargement/pdf/key_documents/2014/140429-screening-report-chapter-32-
serbia.pdf.
189
Mose Apelblat, “The Status of Performance Auditing in the Enlargement Countries – Results
of Findings in Questionnaire to the Supreme Audit Institutions,” European Commission, DG
ELARG, draft discussion note (2013), 6.
190
Ibid.
191
ISSAI 300 Fundamental principles of Performance Auditing state that “[…] it should be borne
in mind that performance auditing focuses on activity and results rather than on reports or
accounts, and that its main objective is to promote the economy, efficiency and effectiveness,
rather than report on compliance,” 4.

78
for Serbian SAI, as a newcomer in performance auditing, to carefully

assess these aspects.
Normative focus. Overall, external audit in Serbia is characterized
by its normativity. However, this is something that characterizes
audit as such: “One of the characteristics of auditing is the normative
perspective where discrepancies between ‘the norms and the reality’
– the actual findings – are expressed explicitly, and assessments
and recommendations are provided as ‘normative’.”192 Performance
auditing, on the other hand, could be more flexible. One of the basic
questions performance auditing is trying to answer – i.e. “Are the right
things being done? – demonstrates this clearly. This question goes
beyond addressing the stated objectives and goals of a policy, also
implying how a policy change influences a specific group or the society
as a whole. Thus, if necessary, it tends to bypass normative rigour.
Analysis of the first performance audit reports produced by the Serbian
SAI shows that focus is still to a large extent on compliance with norms
or their absence. Broadly speaking, dealing with the management of
public vehicles, the first performance audit yielded recommendations
for: the adoption of necessary legal acts and documents, keeping
regular and updated records of the usage of vehicles, the introduction
of necessary (cost) analysis before using, procuring or disposing of
vehicles, et cetera.193 In other words, the discrepancy between the
norms and the actual state of affairs was emphasized, coupled with
an actual absence of rules to adhere to. For this particular topic it was
clear that goal-achievement of policy was hard to measure simply
because there had been no clearly defined goals (or a clearly defined
policy to begin with). Moreover, SAI itself contributed to the normative
side by defining the goal of the audit as “to gain insight into financial
materiality and management of public vehicles of direct beneficiaries
of the budget of the Republic of Serbia and to give recommendations
for the improvement of public vehicles management systems.”194
From the definition of the audit goal it is clear that the focus of PA
was on the operational efficiency of the audited policy, rather than on
effectiveness. However, this approach can be easily understood given
an environment with little to no performance information on the
delivery of policies, or even the majority of public services. In such an
environment, in order to audit effectiveness of a policy, the SAI would
need to assume what the objectives and performance indicators of a
192
ISSAI 3000, 12.
193
Performance Audit Report on the Management of Public Vehicles, 2014, 3. Available at: http://
www.dri.rs/images/pdf/revizija2014/konacan%20izvestaj%2025%2006%202014%20sajt.
pdf.
194
Ibid., 6.
79
policy are. Furthermore, it was probably a wise choice for the Serbian
SAI to begin its PA work with a topic which is relevant for the use
of public finances in terms of economy and efficiency, but not really
defined as a distinct government policy, as a result of which it did not
require an inquiry into the appropriateness of goals and objectives.
A similar approach is followed in the second performance audit which
deals with the management of public property (immovable property),
although the topic itself is fairly more complex and sensitive due to
the higher budgetary implications. This time, however, SAI came
closer to a policy debate in terms of its findings and recommendations.
After establishing the existence of severe breaches of procedures,
miscommunication among managing authorities, and a lack of
coordination, it focused its recommendations on the adoption of strategic
documents, as well as normative corrections, coordination issues as well
as procedures for inventorying, monitoring and managing immovable
public property.195
The topics of the two audits were mostly focused on management
practices (vehicles and real estate) and auditing of cost-efficiency
was expected. However, the SAI’s findings in both of these audits
also illustrate that the normative approach of SAI is conditioned
by the culture and operation of the Serbian administration itself,
i.e. inefficient control mechanisms and an absence of performance
measurement systems. Thus, the prevailing “normative mindset”
is the result of two parallel features – the limited experience of SAI
staff coupled with a traditional, norm-oriented administration.196
Still, these first performance audit reports also illustrate a high level
of ambition by the Serbian SAI to take up system-wide issues and
problems in selecting performance audit topics (wide, cross-cutting
policies rather than policies constrained to one or a few institutions/
bodies). The possibility to promote and generate change is more
achievable if the audit approach is systemic in nature, which is line
of distinction between performance and compliance audits.197 Lastly,
195
Performance Audit Report on the Management of Public Property in Serbia, 2015, 1. Available
at: http://dri.rs/images/pdf/revizija2015/izvestaj_rns2015.pdf.
196
A majority of SAIs in the Western Balkans are of the opinion that these are the reasons why
performance audit is lagging behind. The most repeated explanations are: a lack of adequately
skilled personnel, lack of performance measurement at audited bodies, focus on building
capacities for financial and compliance auditing, lack of programme-based budgeting, etc. In:
Apelblat, 7.
197
A ssessment report concerning the first performance audits in the Western Balkans, done by
the SAI of Sweden in 2011, claims that the “feature characterising the reports is their orientation
towards audit of compliance, i.e. whether rule and regulations are followed or violated […] By
focusing the audits on compliance the actual causes to problems are not addressed […] Because
of the limited approach of these audits the possibility to improve systems and promote change
is limited.” In: Apelblat, 18.

80
the debate on the “normative” focus of Serbian SAI should not be

erroneously narrowed to a debate on the “right” or “wrong” approach.
In the opinion of SAI Performance Audit management, every audit
needs to begin with a compliance with the rules as the starting point,
performance audit included.198
Methods. INTOSAI standards suggest that performance audit
should not be streamlined, as it is a “complex investigatory work
that requires flexibility, imagination and a high level of analytical
skills. Streamlined procedures, methods and standards may in fact
hamper the functioning and the progress of performance auditing.”199
When choosing a methodology, one has to take into consideration
the audit goals, scope and limitations. So far, the Serbian SAI has
chosen audit methodologies in relation to these considerations. For
example, in auditing the management of public vehicles, SAI mostly
relied on analysing data previously requested from and sent by 126
budget users, regarding the different features of vehicles (number,
brand, procured/current value, consumption, expenses etc.). In the
next step, 15 budget users were subject to a detailed analysis on the
basis of predefined benchmarks. The sample was deemed plausible
having in mind that it contained 88% of all public vehicles and 84%
of total expenses. Conversely, the second performance audit relied on
a case study methodology due to certain objective limitations such
as the low quality of available data, inappropriate record keeping
of immovable property and shortcomings in the central registry.
Case studies included 5 categories of immovable property and their
users were selected as sources of information. The selection of users
was conducted through scoring according to determined basic and
derived criteria. Pollitt suggests that use of case studies in PA is an
innovative approach, one that was not apply in the early period of PA
development in Western Europe,200 which points to the openness of the
Serbian SAI to apply a wider spectrum of social science methods, albeit
carefully adjusted to the current needs and resources. This conclusion
has been corroborated by our SAI interviewees, who explained that
they had already used international policy evaluation guidelines and
methodologies as references in preparing the pre-studies for the two
performance audits.
INTOSAI guidelines suggest that there are two traditions when it comes
to the auditing approach. These two approaches are a results-oriented
and a problem-oriented approach. According to these guidelines, in the
198
Interviews with a Performance Audit Sector Manager and a certified state auditor.
199
ISSAI 3000, 29.
200
Christopher Pollitt, “Performance Audit in Western Europe: Trends and Choices,” Critical
Perspectives on Accounting, 14.1 (2003), 159.
81
result-oriented approach “the auditor studies performance (concerning

economy, efficiency and effectiveness) and relates observations to the
given norms (goals, objectives, regulations, etc.) or the audit criteria
(more or less precisely defined before the main study begins)”.201 In
a nutshell, this approach is considered as more likely to determine
“deviations” from norms, and is thus more normative in nature.
Conversely, the problem-oriented approach “deals primarily with
problem verification and problem analysis, normally without reference
to predefined audit criteria. A major task in the audit is to verify the
existence of stated problems and to analyse their causes from different
perspectives (problems related to economy, efficiency, and effectiveness
of government undertakings or programmes).”202 The guidelines refer
to this approach as more analytical and instrumental in comparison to
the previous one, since it is based on an analysis and causes of problems.
Broadly speaking, SAI of Serbia could belong to both traditions if we
consider the audit work implemented so far. There is still insufficient
information in order to group SAI into one of these two traditions,
especially knowing that performance audit work will enlarge in scope
and that capacities will evolve in the future. Published audit reports
clearly reveal that the main audit questions, as well as the audit goals
raise concerns on management and compliance. For example, the audit
goal of the first report seeks to improve management systems, whereas
one of the audit questions is to examine whether inappropriate record
keeping, monitoring and control increase the risks of mismanagement?
Auditors were already certain and hypothesized that there are
irregularities or errors on the basis of previous analysis. While it is
common for performance audit to prepare a pre-study before starting
the real audit work, the previous research in this case was actually the
audit performed beforehand by the SAI. Both reports state that the
reasons for selecting this topic go back to the findings of financial and
compliance audits in the past. Firstly, this demonstrates that, in the
absence of vast experience, SAI is utilizing all disposable resources for
accelerating performance audit. Secondly, it reveals that the financial
and non-compliance aspects of audit do heavily influence SAI’s work.
The interviewed performance audit managers and auditors themselves
stated that the focus of performance audit work is, for the time being,
mainly on efficiency and economy, while effectiveness is still far from
reach.203 When it comes to the actual “approach” of Serbian SAI in
performance audit, it is still in the making and dependent on the future
development of SAI and the Performance Audit Sector.
ISSAI 3000, 26.

201
Ibid.
202
Interview with certified state auditor from Performance Audit Sector of SAI.
203

82
Qualifications and skills. The qualifications and skills required for

auditors dealing with performance are more demanding than in
financial audits. They include knowledge of methods applicable to
performance auditing, knowledge of the government, specifically
tailored communication skills, as well as “special skills depending on
the nature of the specific audit (e.g. statistics, information technology
(IT), engineering etc., or expert knowledge of the subject matter
concerned).”204 Ensuring the necessary qualifications and skills of its
employees for PA seems as one of the major challenges for the Serbian
SAI. In practice, limitations are at least twofold. Firstly, the education
and certification requirements for state auditors are almost entirely
conceived for professionals in the fields of law and finance/accounting.
Secondly, a majority of the staff in the performance audit sector
previously worked in other audit departments, dealing with financial
and compliance audits,205 which may require a robust retraining
scheme for acquiring critical skills. Although highly skilled in audit and
with extensive knowledge of the public finance system and the public
administration, there is a need for more diversity among auditors, which
would mean opening up to hiring of people with different backgrounds
and skills (e.g. in social science or statistics). Interviewees from SAI
agree that this is indispensable. Knowing that performance audit is
more than just determining “right” or “wrong” answers, recruitment of
staff for this particular sector is definitely the first in line for changes.
Of course, this entails changes in the SAI philosophy, as well as a clear
emphasis on the performance auditor’s role. Ideally, there should be a
minimum requirement for a university degree with the possibility of
hiring experts from different fields.206 A university degree requirement
with an additional emphasis on the experience in investigative and
evaluation work is also required by the INTOSAI standards, in addition to
personal qualities of the auditor which are of “considerable importance,”
such as creativity, analytical and social skills, receptiveness, judgment,
oral and writing skills etc.207
The internal structure of SAI resembles the current philosophy
and state of capacities – out of the seven audit sectors, only one is
conducting performance audit, whereas six others are occupied with
financial and compliance audits in different fields. It is not known at
the present moment whether performance auditing in the future will
become horizontal and cross-cutting in all audit departments. For now,
even the interviewed performance auditors are convinced that this
204
ISSAI 3000, 38.
205
206
207
ISSAI 3000, 37.
83
would be premature given that performance audit represents just a

small portion of the overall audit work.
This comparison of INTOSAI guidelines with the performance audit
practice in Serbia served to examine to what extent this type of audit
is structured in a way which is conducive to influencing policies, and
therefore contributing to policy evaluation, rather merely than holding
individuals accountable. While it is not the intention of Serbian SAI to
simply ensure accountability when conducting performance audits,
it is still early to measure how valuable performance audit findings
are for policy learning. Nevertheless, before we continue with the next
section, two more things should be considered in performance audit
planning and conducting, both of which are important in this regard –
topic selection and the relationship with auditees.
Planning of audit. The topics for the two pilot performance audits were
drawn from the findings of financial audits. Procedurally, performance
audit management proposes topic(s) to the SAI Council, whose role
is decisive in approving the annual audit programme. However,
performance audit management is free to nominate any topic that is
deemed relevant for their work. This opens the door for a more creative
approach and interviews have shown that potential sources for topic
selection are also found in the media, and even through social networks.
Tapping into popular demand and the social-political surrounding for
clues is definitely a new way of thinking for SAI of Serbia. After selecting
the topic, the sector prepares three to four preliminary studies each
year as the basis for the final Council decision.208 These preliminary
studies represent a kind of ex-ante risk assessment to see whether an
audit is needed for the matter in concern. They are needed before the
main study in order to better understand the problem, as well as to
justify the audit. Preliminary studies represent an actual spot where
audit becomes more “evaluation-oriented,” i.e. based on research,
possibly with the introduction of a variety of methods.
Relationship with auditees. Partly due to the established practice of
the Serbian SAI instigating misdemeanour and criminal proceedings
in the aftermath of its audit work, auditors tend to be perceived
as investigators or inspectors whose task is to check for errors
and determine individual responsibility rather than produce
recommendations for improved work. Although this is mainly
relevant for compliance/financial auditors, performance auditors
have experienced the same reactions in their recent work. However,
208
In the organizational concept of the Serbian SAI, the heads of the individual sectors are not
members of the SAI Council and they do not participate directly in the decision-making process
of selecting performance audit topics.

84
it is the intention of the performance auditors to take on a different

approach to auditees and foster a stronger sense of cooperation in
order to increase the level of trust. They apply several approaches to
achieve this: they carefully explain what the purpose of performance
audit is, they keep regular communication with the auditees and take
into account that deadlines for implementing SAI recommendations
should be more loose (as performance audit often concerns systemic
and deeper policy issues), and lastly, as an internal rule of thumb,
they do not initiate proceedings before courts because it is not the
ultimate audit goal (one which is strongly emphasized by the other
departments). This approach by performance auditors suggests that,
despite its strong orientation towards accountability and sanctioning,
SAI has also – through PA – created a space concentrating on policy
learning.
Follow-up and learning. To conclude this brief analysis of the
development of performance audit in Serbia, it is useful to look beyond
what SAI needs to improve or do. It is clear that capacity development
will go on and that SAI will need to develop strategies for informing
and “educating” stakeholders and the public on the role of performance
auditing. There is, however, an important question of who exactly
are the users of performance information produced by the SAI. In
the current state of affairs, the Parliament remains only formally
interested in discussing SAI reports in its plenary and committee
sessions, while the executive bodies and agencies remain defensive,
and the public at large is too diffuse to claim that there are specific
demands or pressures. In addressing the demand for performance
information in democracies, Pollit suggests that politicians in general
do not just pick reports containing performance information and apply
recommendations.209 However, performance audit reports demand at
least a procedural response (within a rather short, legally prescribed
deadline of 90 days), but this does not guarantee impact, let alone
policy change. This describes the case with SAI of Serbia – performance
audit is developing in the context of low domestic awareness and the
responses to the audit reports are still a formality. With the increasing
number of performance audits conducted, it should be expected that
the general awareness will also increase.
Impact. The impact can be assessed with respect to the early stage
of development and small number of performance audits conducted.
In that regard, the assessment of impact is approached in a way as
to analyse the impact of the two individual audits that have been
209
Christopher Pollitt, “Performance Information for Democracy - The Missing Link?”,
Evaluation, 12.1 (2006), 48.
85
performed recently, since there is no sufficient data to assess the overall

impact of performance audits on the system. Analysis will be focused
on the first performance audit on the management of public vehicles,
which was officially published in 2014. Given its novelty, assessing the
impact of the second performance audit would be premature.
The audit results arrived at the same time when the Government
started to newly regulate and manage the usage of public vehicles. In
its report, SAI advised that the Government should adopt a legal act
as the foundation for a comprehensive and sound management of all
aspects of utilising public vehicles. Even before the final report of SAI
was prepared and published, the Government adopted the Regulation
on using official vehicles.210 It can be said that the performance audit
already exercised an impact during the auditing process. It is, however,
noteworthy that the Regulation exempts ministries in charge of defence
and internal affairs, and that these ministries were part of the second-
tier sample of SAI. In accordance with this Regulation, a Government
Commission will be formed for granting approvals for using vehicles.211
It is not only that the management of vehicles was regulated anew, but
also that the Government decided in the beginning of 2015 to put on
the market over 1,500 vehicles in official use (most of which constitute
middle and low category vehicles). There is still insufficient data on
how many vehicles have been sold and what the consequences are in
terms of economy, efficiency and effectiveness, which should be the
ultimate appraisal of this effort. Ex-post analysis of these measures
will surely provide more detailed findings. Nevertheless, although the
government measures came in about mostly as “austerity measures,”
the undisputed impact of SAI was that it raised this topic publicly and
influenced the agenda. Lastly, a certain level of impact on auditees was
already achieved during the audit process. This includes the usage
of GPS devices in the Ministry of Economy, the adoption of internal
procedures and instructions for the usage of vehicles by a few budget
beneficiaries, as well as new cost monitoring mechanisms in the Tax
Administration.212 In a nutshell, performance audit did exercise an
impact that is tangible and measurable in the sense that it provoked
measures of the government institutions. It still remains to be seen
what the consequences are in terms of better management of vehicles
in official use in general, as well as in terms of effectiveness of the
budget users’ transactions.
210
Regulation on Using Vehicles in Official Use, “Official Gazette of the RS” no. 49/14. Available
at: http://www.uzzpro.gov.rs/doc/podzak/uredba-o-nacinu-koriscenja-sluzbenih-vozila.pdf.
211
Regulation on Using Vehicles in Official Use, Article 9.
212
From the summary of the performance audit report. Summary accessible at SAI website:
http://bit.ly/1JUTdLl.

86
4.4 Highlights from EU Case Studies

Performance Audit in the Netherlands: The “Mother”
of Evaluation
The types of performance audits conducted are sometimes referred
to as systemic and substantive audits. “With regard to the substantive
audits the focus is not only on programmes and activities but also on
organizations as such (how well are they organized and do they achieve
the goals set?). With regard to the systemic audits the Netherlands
national audit office also highlights the so called government-wide
performance audits (GWAs) which focus on policy instruments,
programmes and activities relevant for all or most of the ministries.
These audits are comparative in nature. Since the late 1980s they have
formed an important part of the agenda of the Netherlands Court of
Audit. GWAs mirror the current state of the art with regard to different
policy instruments such as subsidies, grants, levies, loans and public
information campaigns.”213
Furthermore, the NCA has been known as innovative due to the
introduced improvements in its work. With regard to performance audits,
these innovations came out of necessity, i.e. the need to make the audit
work more in line with real world policy problems – “an important risk
of performance audits that focus only firmly on objectives and indicators
is that the auditor misses the true explanation of disappointing agency
or programme performance. Such performance audits may even become
less relevant. The objective-and-indicators-driven approach also carries
the risk of (1) misunderstanding of stakeholders’ needs, (2) resistance
to change, and (3) dysfunctional strategic behaviour.”214
One of the examples which demonstrates the development of this
participatory and more open approach in accordance with the needs
of those whom policies affect are reality checks. The idea behind the
reality checks, as a specific type of activity that lies between audit and
evaluation, is to examine the problems that end users of government
policies face (citizens, businesses etc.) and see whether the policy
making process incorporates these problems into the formulation of
policy measures or objectives. These audits concern all stakeholders,
213
Frans L. Leeuw, “Performance Auditing, New Public Management and Performance
Improvement: Questions and Answers,” Accounting, Auditing & Accountability Journal, 9.2
(1996) 95.
214
Peter van der Knaap, “Making Performance Audits More Responsive,” International Journal
of Government Auditing (2012), 10.
87
not just government bodies. 215 After such audits are completed, the
results are presented to the policy makers. 216
The UK Value for Money Audits: Efficiency over Effectiveness

Performance audit in the UK is called value for money (VFM) audit.
As the second stream of audit, it aims to stimulate a change, make
recommendations and give a clear conclusion on a given problem.
Each study covers a major area of government expenditure, and the
main goal is to form a judgment on whether value for money has been
achieved. The VFM audits must provide independent and rigorous
analysis to the Parliament on the way in which public money has been
spent to achieve policy objectives. The VFM concept does not question
government policy objectives, which is an activity that should be
accomplished through policy evaluation. With the conducted PA/
VFM, recommendations are made on how to achieve better value for
money and to improve the services under examination. In particular,
the NAO examines the management practices (in 92% of its reports).
Regarding effectiveness, it is seldom examined in the reports. 217 In
the UK, management issues are priorities, whereas policy evaluation
is secondary. 218 The NAO examines the management level (strategic
planning, implementation, performance measurement, etc.) and
the relationship between the government and the customers. The
main norm for NAO is a customer-oriented, efficient, and effective
management. 219 NAO Recommendations are addressed solely to
departments or agencies.
Performance Audit in Finland: In the Service of Performance

Management
The process of developing performance measurement in Finland has
been one of the more important functions of the Finnish NAO, which
215
The following questions have been used: “How do these stakeholders relate to the
objectives, criteria, and intervention logic (or policy theory) of central government? Do they
recognize the relevance and value of those elements, and do they use them? How do they
appraise the actual interventions or policy tools and the way the policy programme has
been implemented? To what degree did the targeted spending actually land on target? What
concrete effect has the policy had on those who are directly concerned? What information
is provided by central government on the effectiveness of the policy measures taken?” Peter
van der Knaap, “Making Performance Audits More Responsive,” International Journal of
Government Auditing (2012), 11-12.
216
Examples can be found in Peter van der Knaap, “Sense and Complexity: Initiatives in
Responsive Performance Audits,” Evaluation, 17.4 (2011), 358-360.
217
Jeremy Lonsdale, Peter Wilkins and Tom Ling, Performance Auditing: Contributing to
Accountability in Democratic Government (Cheltenham: Edward Elgar Publishing, 2011), 78.
218
Ibid.
219
Ibid., 81.

88
has been a proponent of strengthening the performance management

system. Whole of government performance reporting is contained in
the Report on State Finances and Adherence to State Budget. This report
is a constitutional requirement, containing the main performance
information and analyses of the effectiveness of policies, economy
and productivity of the operations, together with general information
on the implementation of the budget. In its work, NAO has audited the
performance of ministries against their stated objectives. These reports
have been critical of the way the information about the effectiveness of
the budget spending has been produced as well as used in the planning
of work of the administration. Apart from issues in the methodology and
the way the data is measured, collected and presented, a major dispute
concerning the state of the performance management system is about the
use of information. Other issues include the lack of consequences at the
organizational level of the administration,220 after a failure or success to
meet strategic objectives, as well as too much focus on detailed processes
instead of strategic actions to support the Government programme.
Based on the NAO reports, reviews of performance management system
have been conducted by the Ministry of Finance.221 In April 2009, a
project to evaluate the performance management system was launched
by the Ministry of Finance following the recommendation of the annual
report of the NAO. This project evaluated the operability of the present
performance management system from the perspectives of steering
public service production, managing fiscal policy, and setting and
evaluating performance targets set across the various sectors.222
Performance Audit in Estonia: Making a Change that Matters
The performance audit department of the Estonian NAO has a 15-
year long tradition, and has gradually become the largest department
of the Institution, consisting of 30 auditors (out of the total number
of 80 auditors).223 It comprises of people coming from diverse
220
Ministries and agencies which are legally required to report on the achievement of results-
targets together with their financial results. NAO is mandated to audit the performance
information, as well as the way the performance management and budgeting systems are
implemented. However, failure to achieve the desired results-targets does not give rise to
legal liability, and no sanctions or rewards are generally connected to performance – neither
“organisational sanctions” (for example, changes in the level of funding for agencies) nor
“individual sanctions” for staff and management. OECD, Finland. Working Together To Sustain
Success, 155.
221
Ibid.
222
OECD, The Call for Innovative and open Government: An Overview of Country’s Initiatives,
2011, 97.
223
According to the Annual report on work of the NAO in 2010, which is the latest report issued
on a website of the Institution, there were 31 positions in the department of a performance
audit, including 1 director of audit, 5 audit managers, 10 senior auditors and 15 auditors. There
were no vacancies at the start of 2010. The department hired 4 auditors and 1 senior auditor in
2010, and 1 senior auditor left the department. By the end of the year, 35 positions were filled.
89
educational background structure, some of whom are teachers, public

administrators, economists, lawyers, etc. NAO usually conducts 12-
15 performance audits per year and they are written in simple and
understandable language. 224 According to the representatives of NAO,
this is one of the reasons why the media follows the performance
audit results with great interest, resulting in significant publicity for
the reports. 225
NAO is working on improving performance auditing, thus a follow-
up model was created for monitoring the effects of individual audits
after their completion. Hence, NAO is capable of collecting data on
improvements and implemented measures by the auditees two years
after a certain performance audit has been conducted.
A recent survey targeting Estonian public officials shows that the
perceived usefulness of performance audit is significant.226 The
survey included representatives of auditees who were asked about
their impressions of the impact that performance audit has had on the
functioning of their institutions and it showed that 40% of respondents
“found performance audit to be useful,” while a smaller percentage
(21%) agreed with the statement that performance audit led to the
adoption of changes in the audited organisations.227
224
‘’Our reports without exception get attention. Those which are politically interesting get
more attention. We are getting harder to be ignored’’, Mr Urmet Lee, Advisor to the NAO of
Estonia.
225
Interview with the Head of Performance audit department in the NAO of Estonia, Mr Tarmo
Olgo.
226
Ringa Raudla, Külli Taro, Cherlin Agu and James Douglas, “The Impact of Performance Audit
on Public Sector Organizations: The Case of Estonia” (paper presented at EGPA conference,
Speyer, September 10-12, 2014), 2.
227
Ibid., 8.

90
4.5 Conclusion: A Long Way from the Third “E”

Given the recent introduction of performance audit in all Western
Balkan countries (regardless of the currently visible advantage of
the Macedonian SAI, PA is still a new and developing practice in all of
the countries) and the legalistic public administrative culture, SAIs
of the region still mainly recur to the examination of the economy
and efficiency when conducting PA. The work of the Serbian and
Montenegrin PA is still strongly influenced by compliance audit. The
Serbian SAI has been particularly cautious when introducing PA and it
only approached this complex form of audit work when proper external
support and guidance was made available from experienced European
SAIs, which is generally a positive practice (also supported by SIGMA/
OECD). Conversely, the audit work in Montenegro was dubbed as
performance audit long before external support or methodological
manuals, or even a specialised department, were at hand, showing
that the term was perhaps used with less caution than it deserves.
The most recent PA work has shown significant improvements, though
effectiveness is still far from sight and might not even be in the planning
stage, especially given the sole focus on individual entities/institutions
in PA.228 However, this point is potentially misleading if conditions
within the local SAIs are not taken into consideration. For example,
Estonian NAO produces over ten performance audit reports annually
with performance audit department being the largest one, and with
almost fifty percent of audit staff educated in diverse disciplines. This
is not the situation with the three WB countries and it could be easily
argued that the audit of effectiveness is something that SAIs in WB do
not have the capacity for pursuing, as opposed to not being willing to
try it. Based on the analysis of the PA in the WB countries, the following
summary graph, inspired by the illustration in the international ISSAI
standards on PA, is proposed.
In any case, true effectiveness studies are not as frequent even in the
SAIs of developed Western European countries, as was found by Pollitt
in the study of five SAIs (Finland, France, the Netherlands, Sweden
and the UK).229 Even in the cases of countries with a longer tradition
in performance management, certain SAIs centre their performance
audit activities around improving management practices, as described
above in the case of the UK NAO. Effectiveness studies are admittedly
much more costly and they do require support from a shift in the overall
228
Pollitt suggests that, in order to study effectiveness, SAIs need to “shift the focus of audit
from individual institutions to policies and programmes (because individual institutions
seldom make sense as units in which to analyse effectiveness).” See: Pollitt, “Performance Audit
in Western Europe: Trends and Choices,” 166.
229
Pollitt, “Performance Audit in Western Europe: Trends and Choices,” 160.
91
public management culture, as suggested by Pollitt. Nevertheless,

the Dutch scenario does show that the NCA was in the driver’s seat
of improving the public management and especially performance
management culture in the Netherlands. One more conclusion inspired
by the Dutch scenario is that, if purposeful, creativity in thinking
by the performance auditors could lead to a better implementation
of performance audits and greater ownership among auditees
(reality checks). This “creativity” however, even if allowed by the
international standards, is not a necessity per se. It should lead to a
better understanding of the nature of performance audit in general
and of specific audit goals and engage auditees through dialogue and
ownership.
Illustration 2. Selected approaches by the SAIs in three WB countries
focus on compliance with
laws and regulations to secure
implementation
sai sai
montenegro serbia
sai
focus on macedonia focus on
individual goverment
organisational undertakings/
units programs
focus on effectiveness to
contribute to change and
renewal
Despite the differences in numbers of PA among WB countries and the

different approaches of the SAIs, the general impact of PA would be
easier to identify once the quality of PA and performance management
in the government improves and the understanding of PA in public
administration increases. For now, the impact of individual PAs
could only be assessed in relation to the number of recommendations
implemented.

92
5 Interaction Between Performance

Audit and Policy Evaluation
This chapter opens with a discussion mainly focused on the

conceptual relationship between performance audit and policy
evaluation, based on the gathered international academic and
professional literature. The section serves to set the framework for
understanding why the two fields can – and should – be institutionally
connected, in particular in environments of very limited financial
and human resources in the wider areas of performance management
and policy making, the Western Balkans being a fitting example. The
discussion goes on to present the findings of field research in the
three WB countries on the existing linkages between the two fields,
and the perceptions of the interviewed stakeholders with regards
to the need and the possibilities of linking them. Additionally, a
selection of cases from EU member states is offered with the purpose
of establishing links between PA and policy evaluation. The cases
were selected according to their potential to serve as good practices
for WB countries in envisioning the potential interaction between
the two fields. The chapter ends with a short summary outlining
the main arguments and, lastly, conclusions drawn for Macedonia,
Montenegro and Serbia.
5.1 The Conceptual Relation: The Converging of the

Twin-Tracks
Whereas their origins differ and are based in different disciplines, 230
both performance audit and policy or programme evaluation evolved
in the age of New Public Management, which was characterized
by a strong focus on performance in the public sector and which
sought to introduce private sector management tools into the
public management sphere. As early as the second half of the
1980s, a body of literature appeared in the Anglo-Saxon part of the
world (mainly the USA and Australia) focusing on the question of
relating policy evaluation and performance audit. Since these were
countries with considerable experience with both policy evaluation
and performance audit, researchers and practitioners started to
Eleanor Chelimsky, “Comparing and Contrasting Auditing and Evaluation Some Notes on
230
Their Relationship,” Evaluation Review, 9.4 (1985), 483–503.
93
inquire about the common traits and differences between the two.
There were several reasons for conducting these inquiries. For some
authors, the motivation to write about this topic was to offer guidance
as to whether and when policy makers should choose one or the
other field, 231 whereas others inquired about their contribution to
accountability and/or performance improvement. 232 Others simply
sought to differentiate between the two, given that both fields were
being developed in very homogenous policy environments. 233 Few
authors from the identified literature tackled the question of an actual
convergence between PA and evaluation in their implementation, 234
how they could be better integrated, and what efficiencies would
be gained from that. 235 Whatever their motivation for studying this
topic, most authors focused on the methodological and conceptual
similarities and differences of the two fields.
The differences between the two fields pointed out by different
authors vary from their different origin, both in terms of age and
the actual fields from which they have developed, 236 the level of
development of quality control rules and professional norms (both
in favour of PA), 237 the level of independence in choosing the topics
they work on, 238 to the obvious differences such as the legal basis
and the institutional setting, 239 the level of authority, 240 the level of
independence from the executive, as well as the types and targets
of reporting. 241 The INTOSAI guidelines for PA strongly emphasise
231
Dwight F. Davis, “What Do You Want a Performance Audit or a Program Evaluation?,” Public
Administration Review, 50.1 (1989), 35-41.
232
See, for example, Ian McPhee, “Evaluation and Performance Audit: Close cousins – or Distant
Relatives?”, Australian National Audit Office (February 2006), 3; Frans L. Leeuw, “Performance
Auditing, New Public Management and Performance Improvement: Questions and Answers,”
Accounting, Auditing & Accountability Journal, 9.2 (1996), 92–102.
233
See, for example, Ian McPhee, “Evaluation and Performance Audit: Close cousins – or distant
relatives?”
234
Leeuw, “Performance Auditing, New Public Management and Performance Improvement,”
95-96.
235
Pat Barrett, “Evaluation and Performance Auditing: Sharing the Common Ground,”
Australian National Audit Office (October 2001), 30-31.
236
Chelimsky, 484-7. It should be noted, however, that Chelimsky was comparing more
traditional forms of audit (compliance and financial) to evaluation, and not performance audit.
237
Davis, “Do You Want a Performance Audit or a Programme Evaluation?” 38.
238
Pollitt and Summa, “Performance Audit and Evaluation: Similar Tools, Different
Relationships,” New Directions for Evaluation, No. 71, (Fall 1996), 47.
239
For Arthur et al., “one of the main differences seems to be the legal mandate of the institution
that performs the study: performance audits are conducted by SAIs with the legislatures as main
recipients, while evaluations, tailored to address the clients’ research questions, are carried
out by various organizations”. See: Arnfrid Arthur, Lars Tore Rydland and Kristin Amundsen,
“The User Perspective in Performance Auditing—A Case Study of Norway,” American Journal of
Evaluation, 33.1 (2012), 47.
240
Relationships,” 32.
241
McPhee, “Evaluation and Performance Audit: Close cousins – or distant relatives?” 17-18.
5 Interaction Between Performance Audit and Policy Evaluation

94
these latter elements, choosing in particular the aspects that most

vividly differ from policy evaluation:
Legitimacy and trust are essential values in all
government undertakings, and performance auditing
may contribute to strengthening these values by
producing public and reliable information on the
economy, efficiency, and effectiveness of government
programmes. This is facilitated by the fact that
performance auditing is independent of the government
ministries whose activities are subject to the audit.
In this way, an independent and reliable view of the
performance of the audited programme or objects is
obtained. The performance audit does not represent any
vested interest and has no ties, financial or otherwise,
to the audited objects. By producing independent
assessments, performance auditing may also serve as a
basis for decisions on future investments and activities. 242
Unlike the incontestable institutional differences, most of the
more conceptual differences remain contested among various
authors and sources. One of the most important ones relates to the
question of whether performance audit can, and should, address
the effectiveness or impact of a policy, and to what extent it should
aspire to do so. 243 In fact, the effectiveness and impact question
may be at the root of the most relevant differentiation between PA
and policy evaluation – what we might call the “level of ambition”
with regards to actually influencing or changing a policy. Pollitt
and Summa call this the difference between the “What” question,
whether what has been done has conformed to standards, and the
“Why” question, i.e. trying to understand what would produce
certain desired or undesired effects. 244 Put differently, while
evaluation often has a strong focus on policy and is able to make
a qualitative assessment of policy effectiveness, performance audit
is focused on evaluating the economy, efficiency and effectiveness
of public administration.”245 One may safely state that an approach
which questions the effectiveness and the impact of a policy under
review – which is what evaluations do – actually demonstrates an
242
ISSAI 3000, 13. Emphasis by the authors of this study.
243
Davis, 37. Since 1990, Davis argued that: “Auditors are more likely to address programme
impacts now than they used to be, but they are still less likely to do so than evaluation
researchers.”
244
Relationships,” 31.
245
Author’s emphasis. See: McPhee, “Evaluation and Performance Audit: Close cousins – or
distant relatives?” 17-19.
95
ambition for affecting the design or even the very existence of the
evaluated policy. Hence, whereas evaluation can pose questions
regarding the very appropriateness of a policy (and could thus even
lead to an abandonment of a policy), PA remains restricted to the
questions of whether a policy has been implemented in accordance
with the three E’s. 246 In other words, PA must “take government
policy as given,”247 but it can question the “operational effectiveness”
of a policy or programme. 248 At the same time, INTOSAI standards
clearly state that, despite the general requirement to take “political
decisions and goals established by the legislature” for granted, SAIs
may “make critical comments on the goals, for example if they are
inconsistent or if it proves impossible to follow up the extent to which
they have been achieved. Consequently, a performance audit report
may in fact question the merits of existing policies or decisions.”249
The international standards even go as far as to propose that the
second question posed in PA - “Are the right things being done?”
can be interpreted as to “imply that a government undertaking – or
a chosen measure to achieve a certain objective – runs the risk of
being contested.”250
These discrepancies in the proposed conceptual differences
(leaving aside the obvious institutional differences) are, on one
side, connected to the differences in approaches and perspectives in
performance audit which were discussed in Chapter 2. On the other
side, they are related to the angle of observation, given that many
professional auditors and evaluators participate in this debate and,
therefore, might not be completely unbiased in their observations.
The original distance between the two fields, due to their separate
origins and development paths, has certainly negatively affected the
extent of communication and even mobility between them, which
plays in favour of such possible bias.
On the other hand, when discussing the conceptual similarities
between PA and policy evaluation, many authors start from
observations related to the general purpose of both processes,
i.e. that “[both] auditing and evaluation research are concerned
with assessing the worth of government actions.”251 In fact, some
246
Author’s emphasis. For a discussion on this issue see, for example, Barrett, 30.
247
Richard Mulgan, “Policy versus Administration: The Auditor General’s clash with Finance
and the Minister,” Canberra Bulletin of Public Administration, Issue 101, September 2001, 39,
as quoted in: McPhee, “Evaluation and Performance Audit: Close cousins – or distant relatives?”
18.
248
McPhee, “Evaluation and Performance Audit: Close cousins – or distant relatives?”21.
249
ISSAI 3000, 34.
250
Ibid., 14.
251

96
authors, who approach performance audit from the perspective of

its contribution to “wider policy debates”252 or its contribution to the
“evaluative state”253 rather than focusing strictly on its accountability
role, essentially treat performance audit as a type of policy or
programme evaluation. 254 One such definition describes performance
audit as “an evaluative activity which produces assessments
regarding performance or information about performance.”255 These
authors argue that the purpose and the role of performance audit
extend far beyond the improvement of accountability and enter the
sphere of (policy) learning. Contrary to the idea that PA is a type
of evaluation, but still corroborating the idea about the closeness of
the two fields, ISSAI 3000 states that “[programme] evaluations are
one type of study that might be executed by a SAI under the general
heading of performance audits.”256 Without entering the “chicken
and the egg” debate, we can conclude that the scope of both fields
allows for a complementarity hypothesis to be raised.
From the institutional and the methodological perspective,
several authors emphasise that both fields put great emphasis
on the importance of independence of their professionals from
“what they evaluate,”257 thus impugning the INTOSAI’s and several
audit-originating authors’ argument that this is a distinct feature
of performance audit. Whereas the institutional position of SAI
certainly makes it easier to ensure independence, the evaluator’s
profession also requires adherence to certain standards, including
independence. Nevertheless, this argument might be easier to
support in the case of external, contracted evaluators, while in
the case of internal administrative evaluations conducted by civil
servants, such an argument might be more difficult to uphold.
A number of authors focus on the fact that PA and evaluation apply
similar tools in their work. 258 Whereas evaluation is traditionally
embedded in social science research, performance auditors
252
Raudla, Taro, Agu, and Douglas, “The Impact of Performance Audit on Public Sector
Organizations: The Case of Estonia,” 2.
253
David Clark, ‘The Changing Face of Audit and Evaluation in Government: A Franco-British
Comparison’’, Public Policy and Administration, 16.4 (2001), 20–33 (23)..
254
See: Clark, “The Changing Face of Audit and Evaluation in Government”; Davis, “Do You
Want a Performance Audit or a Programme Evaluation?” 35-41; Barrett, “Evaluation and
Performance Auditing,” 30; and McPhee, “Evaluation and Performance Audit: Close cousins – or
distant relatives?”, 21.
255
Jan-Eric Furubo, as quoted in Raudla et al., “The Impact of Performance Audit on Public
Sector Organizations: The Case of Estonia,” 2.
256
ISSAI 3000, 23.
257
258
See: Pollitt and Summa, 2; McPhee, “Evaluation and Performance Audit: Close Cousins – or
Distant Relatives?” 2.
97
increasingly borrow methods and techniques from evaluation and, in

a number of SAIs of Western Balkan countries, they have immensely
expanded the range of methods and data collection techniques they
employ. 259 INTOSAI supports this claim by stating that “guidelines
in performance auditing cannot comprehensively embrace all
possible approaches, methods and techniques, since in practice that
would include everything in the social sciences.”260 Nevertheless,
comparative studies have observed differences among SAIs in terms
of the tools and the methods applied, 261 which probably means that
PA and evaluation come closer in those terms only in countries with
“bolder” and more ambitious SAIs, i.e. those willing to innovate.
All authors agree on the importance and necessity of both approaches
to serve the policy makers’ and managers’ needs. As Dwight
puts it, “[policy] makers need both process/managerial data and
impact/outcome data for full evaluation of policy alternatives and
priorities,”262 and goes on to explain that different concerns should
be addressed at different stages of policy development. A “toolbox
for practitioners” developed by the EU claims that PA has filled in a
part of the gap in the EU countries which was created by insufficient
development of policy evaluation, and, in fact, does not even make
the semantic differentiation between the two fields with regards to
whether operational or overall effectiveness of a policy is tackled
by each of them. 263 To conclude, the extent of the contribution and
influence (whether intentional or as a “side-effect”) of PA on policy
learning depends certainly on the concrete practices of a particular
SAI, 264 the methods and the tools applied by a SAI, or even the
educational or training background of auditors. 265
5.2 Macedonia - a Promising (yet Still Unattained)

Objective
As has been shown above, the processes of performance audit and
policy evaluation have been institutionally developed independently
259
260
ISSAI 3000, 7.
261
262
263
Quality of Public Administration: A Toolbox for Practitioners, European Commission, 50.
264
Pollitt suggests four different roles that SAIs (can) perform with regards to their PA activity:
1) public accountant; 2) management consultant; 3) scientific or research based organization;
4) judge or magistrate. See: Pollitt, “Performance Audit in Western Europe: Trends and
Choices,” 164.
265
Pollitt gives examples from France, where performance auditors are usually trained as
lawyers, the UK, where they have an accounting background, and Sweden where they are likely
trained as social scientists. See: Pollitt, “Performance Audit in Western Europe: Trends and
Choices,” 161.

98
in the context of Macedonia, even though the two processes have

been largely influenced by the prospect and the processes of EU
accession, as well as by external financial support. Furthermore,
different aspects of the policy cycle have been tackled by different
interventions and projects, resulting in a scattered introduction
and implementation of the different phases and aspects (strategic
planning, policy analysis, monitoring and evaluation).
With the 2013 Methodology for ex-post evaluation, the examination
of independent reports, performance audit reports among others, is
embedded in the process of conducting evaluations of policies. Hence,
one can formally conclude that a link has been established between
the two processes in the context of policy making in Macedonia.
Nonetheless, the practice of evaluation is too recent for us to be
able to derive a meaningful conclusion as to how this provision
is implemented in practice. The civil servants and evaluators
interviewed for this study supported this provision and considered
the PA reports as crucial for any successful evaluation.
On the other hand, the process of performance audit is largely linked
with policy per se, while policy evaluation is, at the moment, more
linked to a specific legal act, than to a policy. In addition to the largely
legalistic culture of the public administration, this is an impediment
for further enhancing policy evaluation and, consequently, linking
performance audit with policy evaluation.
In terms of the perceptions of the actors involved, the possibilities
for linking were recognised by both evaluators and performance
auditors. As was to be expected, performance auditors were
more cautious about the possibilities for linkages due to their
independence and their constitutional role which places them outside
of the executive. They did not view the independence of the policy
evaluation reports as a given since it is partly conducted by the same
actors that create the very policies. 266 Yet, they did not completely
reject the potential for linking the two processes or building
professional ties with evaluators. Still, given that the systematic
introduction of ex-post evaluations is fairly recent in comparison to
performance audit in the country, the interviewees believed that a
certain track record of evaluations needs to be established to see
how such reports operate in practice.
The caution of independent bodies, such as the Audit office, to involvement in “policy” of any
266
kind, can also be explained by the lack of linguistic and conceptual differentiation between
“politics” and “policy”, for which the same word is used in the languages of the region.
99
On the other hand, the civil servants and evaluators we interviewed

were very familiar with performance audit reports and their impact
in terms of changing policies. In addition, the civil servants and
evaluators were aware of the requirement in the Methodology
for evaluation to consult independent reports for the purposes of
evaluation. The performance audit reports were considered to be
beneficial for policy evaluation, yet it was underlined that this is
an assessment from an independent body which is wider in scope
than policy evaluation (currently conducted solely on legislation).
While this group of interviewees was also supportive of the
potential bridging of the two processes, they considered the need
for increasing the capacity for policy evaluation before this could be
achieved.
On a more general level, however, a systemic problem of policy
planning was mentioned as an impediment to the linking of the two
processes. For either of these reports to be substantially included
in policy planning and analysis, the problem identification phase
needs to be much more developed than is currently the case in the
country. 267 Most legislation is adopted under shortened procedures,
which do not allow for a complete policy cycle to take place before
the enactment of new measures. This problem has also been
highlighted in the SIGMA 2014 Public Administration Assessment,
where the experts underline that “in terms of regulatory and
legislative matters, government and parliamentary procedures
that were designed for exceptional cases are being used routinely;
this has several implications: less time is devoted to developing
policy proposals, and legislation is being developed prematurely,
prior to any appraisal of the impact being undertaken”. 268 In such
circumstances, however, both the policy evaluation reports and the
performance audit reports can add a much needed missing link in
the policy planning process.
5.3 Montenegro: Foes Rather than Friends

There are no institutional mechanisms in place which would
ensure the linking of policy evaluation and performance audit in
Montenegro. The key topic that concerns SAI is how to ensure that
its recommendations have an impact and are implemented by the
audit entities. The high pressure stemming from the public has
267
Interview with representative of the Ministry of Information Society and Administration,
May 2015.
268
OECD/SIGMA, Public Administration Reform Assessment of the Former Yugoslav Republic
of Macedonia, 2014. Available at: http://sigmaweb.org/publications/FYRoM-Assessment-2014.
pdf.

100
guided the Government to establish a practice of adopting an annual

action plan of implementing SAI’s recommendations. Furthermore,
in 2014, the Government formed a cross-sectoral, Coordination Body
for monitoring the implementation of this Action Plan, which can
perhaps serve as a nucleus for linking the two processes.
However, this process and the significant public attention given to the
particularly critical audit reports (financial and compliance ones),
has created a kind of barrier between the SAI and the government
officials dealing with evaluation. Performance auditing has still not
gained ground among the policy makers and government experts as
an activity of the SAI which is different from usual audits. In the
usual audits, SAI is always perceived as someone that is “out to get
them“, rather than to provide evidence-based data to feed the policy
making process. The fact that performance audit has been centred
on single institutions, more than on processes or policies, results in
the impression that SAI is “trying to find the guilty ones”, i.e. to find
proof of the management’s mistakes, rather than to help.
Interviewed officials dealing with evaluation believe that SAI needs
to enhance its methodology in order for their reports to be more
applicable and objective (i.e. tailor-made to the specifics of the
institution or the process being audited). They claimed that SAI
deservedly enjoys a relatively high level of respect and acceptance
in the society, but emphasize the fact that its methodology for
performance audit reports and classical audits with elements of
performance auditing has never been truly contested by the expert
public, which has either ignored it or taken it for granted. Among
some of our interviewees, we registered a dose of reservation
towards SAI’s work and its findings, specifically concerning the basis
of the expertise of auditors and the credibility of their methodology.
On the other hand, the representatives of SAI are almost completely
unaware that there are policy evaluation efforts going on in the
state administration. In cases where they are informed about them,
they appear sceptical of their value, objectivity and methodology.
There seems to be professional rivalry and enmity arising between
performance auditors and evaluation experts, primarily based on a
lack of communication and cooperation.
The representatives of the Government also claim that a greater
interaction with the SAI is necessary in the phase of adopting the
final version of the audit report. They want their suggestions on
the preliminary audit report to be taken more seriously by the SAI
and to actually have an impact on the published report. One of the
101
steps needed for achieving better cooperation and exchange of

communication is to enhance this part of SAI’s methodology, and
have more intense cooperation with the audit entities during the
formulation of the final report. This is particularly important for
performance audit, whose recommendations are generally more
ambitious, advanced and demanding than those from other types of
audit, and thereby require a higher awareness and understanding
from the audit entity in order to increase the chances of their
implementation.
One interlinking point for these two processes is the implementation
of performance-based budgeting. SAI has been one of the most vocal
proponents of intensifying the implementation of performance-
based budgeting, which has come to a standstill around 2009 when
the budget programmes were formally introduced into the Budget
Law, without any progress being made since. 269 One of the significant
cooperation opportunities is the process of defining the so-called
non-financial elements of the budget, goals and performance
indicators, which is supposed to start with financial assistance from
the EU in the near future, as part of the preparations for the direct
budget support. It would be productive to have SAI on board from
the start, as an evaluator of draft non-financial elements, at least for
a sample of budget users.
The potential of internal audit as a tool for horizontal implementation
of the SAI’s recommendations is not recognized nor utilised. Internal
audit (as part of the Public Internal Financial Control system) should
also be utilised to ensure that the findings of the policy evaluation
are taken into account and actually implemented. Although it is not
directly associated with policy evaluation, it can serve to enhance
relations with the executive.
One of potential actions that can be taken is to amend the SAI’s
procedure for formulating the annual audit plan, in order to
somewhat align the priorities with the Government regarding the
performance audit. Without infringing on the SAI’s constitutional
independence, it would be beneficial if SAI contributed through
performance to the policy changes the Government is planning in
the future, and that pertain to the amendments of systemic laws
or institutional reconfigurations. The approach of SAI is clear —
rather than waiting for a policy window to open or trying to create it
single-handedly through the strength of its report and presentation,
269
See: “Budgeting the Cost of Reforms – Programme Budget for Police and Prosecution,” Institut
Alternativa, Podgorica, 2014. Available at: http://media.institut-alternativa.org/2014/12/
ProgrammeskoBudzetiranje-ENGL.pdf.

102
alignment with the Government’s schedule increases SAI’s chances

to exert an impact and meaningfully contribute to the policy cycle.
Additionally, the executive should, in principle, be interested in hard
data and evidence to accompany the process, even if its own form of
evaluation is also ongoing. This kind of interaction would increase
contacts between SAI and the policy evaluation cycles, and perhaps
foster mutual cooperation or at least increase the attention that
both sides are devoting to strengthening their methodologies and
capacities in this regard.
5.4 Serbia - the Newcomer in Audit and Evaluation

The interaction between performance audit and policy evaluation
can be approached in different ways. Institutionally, two processes
are characterised by a different institutional position, design and
competences, this being one of the main concerns of actors on both
sides. As for the actors, there are two relevant aspects when speaking
about the possibility of interaction: what is the general perception
of actors (i.e. how auditors perceive evaluation and vice versa) and
where do they see interaction or opposition. It is this perception
which will be illustrated in this section.
Before we proceed, it is useful to look back to approaches elaborated
in the previous sections with regard to Serbia. Differences in
approaches relate to many aspects of the processes, but a few should
be highlighted. Firstly, timing is very important. SAI decides on
the subjects of audit on an annual basis and there is small room for
working around it. The timing of evaluation, on the other hand, is
dependent on the implemented project or programme, and follows
the logic of the policy cycle. Understandably, different priorities
influence differences in timing, which can be seen as a potential
deadlock in interaction. Secondly, audit and evaluation work can
vary greatly in terms of research means, but performance audit
has the potential of being more “evaluative” if it focuses on effects
and problem solution. What is encouraging is that performance
auditors have started utilising techniques that generally belong to
the social science domain (such as interviews, case studies etc.) and
undertaking a less rigid approach in relation to the auditees. Thirdly,
it is evident that skills may also differ significantly, owing to the
specific requirements for state auditor status, on one side, and on
the diversity of the professional background of the evaluators, on the
other. It can be said that evaluators have more options in terms of
methodology, whereas performance auditors are more authoritative
103
in their work. Lastly, the range of performance audit and evaluation

can be quite different. The first two performance audits that were
conducted, focused on horizontal issues and overall management
and procedural aspects, whereas evaluations were more specific
or narrower. All these differences are influencing the possibility of
interaction and the individual perceptions of actors.
It is important to stress that performance audit staff and management
are generally inclined to have a positive attitude towards evaluation
as a process which can improve transactions of budget users. The
biggest concern, however, is the “commercial” aspect of evaluation.
Outsourcing a single evaluation activity for a stand-alone project
lacks reliability in the eyes of the auditors. Normally, supreme audit
institutions are safeguarding their independent position and are
relying on their high professional standards, as well as on their
integrity and ability to provide authoritative recommendations.
Furthermore, the perception of interaction with policy evaluation
is influenced by the constitutional role of SAI of Serbia and de
jure responsibility before the parliament and the public, whereas
evaluation is perceived as a predominantly government domain.
On the different side of the spectrum, there are two tendencies
when it comes to perceptions of interviewed civil servants. The
first tendency is that audit is perceived in a narrow sense as a
control or inspection activity. Unsurprisingly, the only experience
they had with SAI is in financial and compliance audits, and with
accountability mechanisms the Serbian SAI has on disposal, which
include initiating misdemeanour and criminal proceedings. Indeed,
all audit work so far has been conducted in a way as to determine the
damage done. The second tendency shows that performance audit,
given its recent introduction, is barely distinguished from other
types of audit. This is, again, definitely not a surprising finding, but it
is an important tendency since it directly influences the perception
of SAI. The more performance audits SAI implements in the future,
the greater the chance of a change of perception.
Finally, where do the actors see room for interaction and opposition
if we take into account all that has been said thus far? In the eyes
of performance auditors and management, there are currently no
direct links with policy evaluation, nor does communication between
them exist. However, when it comes to establishing an initial link,
there is general acceptance among performance audit staff of the
idea that consultation mechanisms between stakeholders can
promote performance audit findings before the decision-makers, as

104
well as promote SAI as an institution. Consulting SAI through the

entire policy cycle could help stakeholders realise the potential of
performance audit for policy making in general, and therefore for
policy evaluation as well.
Civil servants from the interviewed institutions have not had the
chance or the need to use the performance findings of the audit reports,
although they can hypothesise on the benefits of performance audit
in terms of improving evaluation. Examples vary and include benefits
of performance audit results in developing assistance programming
or in comparing methodologies and data. Yet, in individual cases,
findings from financial and compliance audit reports which indicated
deficiencies in the legal framework, the methodologies or the
implementation mechanisms, were used as inputs for the production
of documents as sources of information.270 Performance audit reports
could be even more useful sources of information, provided that there
is more of them in the future.
In a nutshell, interviewees from both sides agree that the overall link
between the processes could be useful. It is clear from the interviews
that additional awareness raising of the role of SAI’s is needed since,
in most cases, audit is equalised with accountability in a strict sense,
without connection to the learning aspect. Moreover, insufficient
experience in conducting both performance audit and evaluation is
influencing the way interaction is perceived. Without filling the void
in experience, this perception will remain contemplative in nature
and prevent more elaborated approaches in bridging performance
audit and evaluation.
270
Data from financial or compliance audit reports have no use for evaluation in most cases.
However, SIPRU used findings from SAI report in producing the study on the establishment of
social inclusion fund (Interview with SIPRU representative).
105
5.5 Highlights from the EU Case Studies

The Netherlands: The Perennial Link
It is said that the NCA has been one of the major drivers for increasing
attention to evaluations. This trend started with the so called
government-wide audits (GWAs) which represented an innovative way
of conducting performance audits. According to Leeuw (2009), these
comparative studies were benchmarks in the sense that they, at least
once per year, inventoried and audited what government bodies knew
about the economy, efficiency and effectiveness of the policies they
implement.271 It appeared that the government and their agencies had
no or very little information by their side.
One of these audits found that “information about financial costs and
benefits of new laws was hardly available, despite the fact that hundreds of
new laws were implemented over a couple of years […] it turned out that, of
the more than 700 central government subsidies and grants, ministries had
information on the three Es for only a tiny fraction of them. Further, in 1989
it was discovered that 28 inspectorates […] knew little about the impact of
what they were doing and how it related to the costs involved”272 etc.
Although not in direct connection with policy evaluation (which was
not formally established when first GWAs were implemented), these
audits pushed policy evaluation on the agenda. The NCA was not alone
in pursuing more performance-based government actions. This came in
line with more proactive parliamentary scrutiny, as well as with changes
in the philosophy of public administration and management. The NCA
could not introduce evaluation mechanism by itself and could not
intervene into the policy domain more than its competences allow. Yet,
the credibility of the institution, as well as the specific relationship with
the parliament and the government, had their role and the NCA actually
co-framed the introduction of evaluation in the Dutch administration.
This lesson can be valuable for the Western Balkan countries in the sense
that supreme audit institutions of these countries could put emphasis on
performance related issues and especially effectiveness of government
spending and policies in general.
A relatively recent performance audit practice of the NCA is auditing
evaluation systems of the ministries. These audits are not traditional
audit types as they summarize evaluation requirements posed on
ministries, examine their evaluation work and, in the end, provide
recommendations for the improvement of evaluation practices. In this
Leeuw, “Evaluation Policy in the Netherlands,” 91.

271
Ibid.
272

106
sense, they represent specific type of performance audits which aim to

improve the effectiveness of the evaluation work of the government. In
this regard, there is a practice of sending evaluation reports to the NCA
for their examination even though there is no legal requirement to do
so.273
The latest such performance audit report was published in 2012.274 Apart
from summarizing evaluation requirements on the part of ministries,
this report emphasises the importance of effectiveness research and
concentrates only on evaluations that measure effectiveness, i.e. focus
on causality between the policy instruments and the achieved or
intended social effects. As can be expected from the nature of audit,
the report also calculates how much policy expenditures have been
evaluated for effectiveness.
The NCA’s role in the evaluation analysis is sometimes referred to
as twofold in nature. In that regard, the NCA can make use of meta-
evaluation and evaluation syntheses. The first is concerned “with
assessing and enhancing the quality of evaluative information”275
provided by the ministries, while evaluation synthesis serves “to
encourage ministries themselves to undertake this kind of study. By giving
examples of how this might be done and by making public the manual of
the NCA (HANDAR) on performing evaluation syntheses, the NCA tries to
reach this goal.”276
France: Evaluation as the Sui Generis Activity of CDC

Since 2011, evaluation of public policies has become a constitutional
obligation for the French Court of Audits (Cour des Comptes, hereinafter
referred to as CDC).277 This is a particularity of the French system, and
of the Finnish one to a lesser extent, since, at the international level,
there is no consensus over the question whether the evaluation of
273
Interview with employees of the NCA performance audit directorate.
274
Algemene Rekenkamer, Effectiviteitsonderzoek bij de rijksoverheid (2012), http://bit.
ly/1W4D1OA.
275
Andrea Kraan and Helenne van Adrichen, “The Netherlands Court of Audit and Meta-
Research: Principles and Practice”, eds. Robert Schwartz and John Mayne, Quality Matters:
Seeking Confidence in Evaluating, Auditing and Performance Reporting, Comparative Policy
Evaluation, Volume XI, (New Brunswick, New Jersey: Transaction Publishers, 2005), 115.
276
Kraan and van Adrichen, The Netherlands Court of Audit and Meta-Research: Principles and
Practice,” 115.
277
Namely, article 47(2) of the French Constitution stipulates that “The Court of Accounts
assists the Parliament in controlling the Government’s action. It assists the Parliament and
the Government in controlling the enforcement of finance laws and the implementation of laws
on the financing of social security as well as for the evaluation of public policies. It contributes
to informing the citizens through its public reports.” Available at: http://www.assemblee-
nationale.fr/connaissance/constitution.asp.
107
public policies should become a part of INTOSAI professional standards

on performance audit.278
The evaluations can be carried out either at the request of the
Parliament, or on the basis of the Court’s self-initiative. Since 2012,
four comprehensive evaluations have been conducted on topics that
span various public policies and different governmental and non-
governmental stakeholders, such as the anti-smoking policy, social
housing and the anti-alcohol policy. 279 There are several features that
distinguish the work of CDC from the work done by SGMAP (i.e. the
ministries and the Government) on public policy evaluations:
1) Comprehensive approach - although the evaluations steered by
SGMAP normally cover more than one public policy, the Court seeks
to evaluate topics that concern several complex public policies, which
require greater time and human resources than required for policy
evaluation conducted by the ministries. While the average duration
of evaluation conducted in the case of SGMAP is 7 months, for CDC it
takes 12 to 18 months;
2) Independence - the independent status allows the Court to provide
critical assessment of the ministries’ results and performance, which
may be a considerable comparative advantage to the evaluations
conducted by the ministries themselves, and
3) Easy coordination - the Court as a single institution does not
encounter issues such as conflict of interest or difficulties in
coordination, which commonly emerge when the ministries conduct
evaluations.
In order to avoid duplication and ensure that existing resources are
used efficiently, the Court follows closely the activities of SGMAP
when deciding on the evaluation topics. Aside from the differences in
length and complexity mentioned above, the two evaluation systems
differ in the sense that the ministries’ evaluations offer policy
options or scenarios as a final result, while the Court evaluations
provide policy recommendations. Moreover, evaluation “is not aimed
at apportioning blame, still less at designating who is responsible. It
starts with a neutral outlook and seeks to highlight positive aspects
as well as criticising negative aspects. It may help to confirm a public
policy or call into action its very existence, if this policy does not appear
to be appropriate.”280
Phone interview with the magistrate at the Cour de Comptes, June 11 2015.
278
Evaluations are available at: http://bit.ly/1LLKDUz.

279
Evaluations are available at: http://bit.ly/1LLKDUz.

280

108
The UK: Assessing Strengths and Weaknesses of Evaluation

through PA (1)
According to NAO, the two main approaches to evaluation are
summative and formative evaluation. 281 Summative evaluation -
impact evaluation, asks questions about the impact of a specified
programme on a specific group of people. Summative evaluation
examines how the impact compares to the original objectives,
to some other programme, or to doing nothing at all. Formative
evaluation - process evaluation, is more oriented towards how and
why a programme has worked or failed. This type of evaluation is
typically oriented towards the process of policy development, as well
as its implementation and delivery.
In December 2013, the National Audit Office (NAO) adopted a Report
on Evaluation in government assessment of the quality of cost-
effectiveness evaluations.282 The Report is a part of a wider NAO project
on the use of cost-effectiveness evidence in the Government, and is
based on a retrospective review of a selection of 35 UK government
evaluations in policy areas of active labour markets, business support,
education and spatial policy. According to this report, the three main
uses for policy evaluation are: to inform strategic resource allocations,
such as in spending review; to inform decisions about policies and
programmes, in terms of the design of new programmes, and improving
or stopping existing programmes; and to support accountability, by
demonstrating the costs and benefits of spending.283
The Report highlighted the strengths and weaknesses of policy
evaluation, assessed the usefulness of evaluations for policy makers,
and suggested improvements. One of the major findings was that
the quality of cost-effectiveness reports varies widely both within
and across policy areas. High quality evaluations were conducted
in the areas of the active labour market and education. In contrast,
evaluations in areas of business support and spatial policy were
considerably weaker (none of the business support or spatial policy
evaluations provided convincing evidence of policy impacts).284 The
same report from NAO finds that the government spends significant
resources on evaluating the impact and cost-effectiveness of its
spending programmes and other activities. The coverage of evaluation
281
See: “Evaluation,” UK National Audit Office, accessed September 21, 2015, http://www.nao.
org.uk/successful-commissioning/monitoring-evaluation/evaluation/.
282
“Evaluation in Government,” UK National Audit Office, accessed September 12, 2015, http://
www.nao.org.uk/wp-content/uploads/2013/12/10331-001-Evaluation-in-government_NEW.
pdf.
283
Ibid., 27.
284
“Evaluation in Government,” 7.
109
evidence is incomplete and the rationale for what the government

evaluates is unclear.285 Finally, the report finds that evaluations are
not always robust enough to identify the impact, and the government
fails to learn from these evaluations, with regard to how it can improve
impact and cost-effectiveness.
Slovenia: Assessing Strengths and Weaknesses of Evaluation
through PA (2)
Probably the best systematic report on policy evaluation and impact
analysis has been done by the Court of Audit in Slovenia in 2007.286
The Court recommended that the government should adopt guidelines
for the preparation and implementation of ex-ante impact analysis,
consider to monitor the effects of already adopted legislation, and, in
particular, encourage the work of the inter-ministerial working group,
which should provide efficient dissemination of regulation for impact
assessment in the administration.
A response to this Report by the Government came in 2009. The
Resolution for drafting legislation was prepared that regulated the
procedure for adoption of the legislation. The Resolution included
policy evaluation, involvement of the stakeholders in the procedure,
and a framework for evaluation of the already adopted legislation.
Additionally, it states that, in formulating and implementing policies
which include administrative, organisational, and political activities,
it is of crucial importance to conduct a regulatory impact assessment,
which should be a reflection of the policy in a given field. However, in
2012, the Court of Audit published its second Report on the Impact
Assessment in which it assessed that, although there have been legal
changes, the situation has not changed substantially.
Finland: Getting Auditors and Policy Makers Together
“On Board”
The Advisory Board of the NAO in Finland is prescribed by law and the
Board invites representatives of key cooperation partners, as well as
experts on financial administration and public finances. The Advisory
Board also includes the Auditor General and one member elected to
represent the National Audit Office’s staff, and has a maximum term of
three years. The Advisory Board maintains and strengthens the NAO’s
connections with cooperation partners, presents initiatives to develop
auditing, and monitors the effectiveness of audits and their ability to
Ibid., 45.
285
Bojan Radej, “Osnove vrednotenja politik za občasne uporabnike Inštitut za politike

286
prostora,” Slovensko drustvo evaluatorjev (2010), http://www.sdeval.si/attachments/

article/298/Osnove-vrednotenja%20(XI%202010)).pdf.

110
serve different cooperation partners. This board is composed of 19

members, among whom are members of the Government, Parliament,
and the academic community.
The Finnish NAO has also been formally given the mandate to
perform independent monitoring and evaluation of the fiscal policy,
which is their regular, obligatory task and which effectively makes
the SAI an integral part of the country’s policy evaluation system.
As a part of this task, “the National Audit Office is responsible for
monitoring the drawing up and implementation of the multi-annual
plan for general government finances […]” and monitoring and
evaluation results are “submitted to the Parliament in the middle
and at the end of electoral period. The Government must either
defer to the opinions publicly adopted by the National Audit Office
or publicly state why not.”287
Estonia: PA Feeding into the Heart of Policy
When it comes to the cooperation between performance audit and
policy evaluation, connection is currently mostly functioning one-way,
in terms of performance audit being used to feed the policy evaluation
process. One of NAO’s annual reports state that the main objectives
of the performance audit department was to give the Parliament
and the public an overview of the sustainability (…) the success, the
effectiveness (…) of state’s activities.288 However, the main “consumers“
of the performance audit reports are the policy planning units in the
Ministries and the Prime Ministers’ Office.
Additionally, each ministry has a policy planning department which
takes performance audit reports into consideration. At the central
level, the State Secretary at the Prime Minister’s Office is an example
of a department which significantly takes into consideration the
performance audit findings.
In conducting their performance audits, auditors from NAO are
using materials produced within the policy evaluation system,
special manuals, methodologies, and regulatory impact assessments
conducted by the executive.289
287
National Audit Office of Finland, accessed on September 9, 2015, https://www.vtv.fi/en/
functions/fiscal_policy_evaluation.
288
See: Annual Report NAO of Estonia, 2010, 9.
289
Interview with Mr. Urmet Lee, Advisor to the NAO of Estonia.
111
5.6 Conclusion: PA and Policy Evaluation as

Estranged Friends in the WB
Although conceptual and comparative literature suggests a very
high degree of closeness and convergence between PA and policy
evaluation, with EU case studies showing a wide range of possibilities
to create interactions between the two fields, the study of the practices
in the three WB countries reveals little to no existing links, or indeed
awareness of the key actors that such links and interactions are both
possible and useful. While in Macedonia at least an indirect link was
created through a general obligation for evaluations of legislation to
take into account the reports of independent bodies (without, however,
substantially elaborating this general obligation and its purpose), in
Montenegro and Serbia – inter alia due to the current ad hoc approach
to policy evaluation – no such requirements or recommendations
exist. Moreover, interviews have shown that the main actors are often
not even aware that there are evaluative efforts going on outside of
their own branch of work and have, in some cases, shown very little
knowledge or understanding of each other’s roles. At the same time,
it is encouraging that the learning effect of the interviews was rather
strong and that most interviewees expressed strong support for the
idea that the two fields and their practices should be connected in a
tangible way.
Comparative case study examples given above demonstrate that it takes
time for meaningful and sustainable correlation to be established. It is
usually the case that SAIs try to investigate how evaluation systems
in the government work and, according to the audit findings, propose
recommendations for a better functioning of the system and follow-
up on the evaluations (UK and Netherlands). Similarly, SAIs try to
push governments to adopt necessary legislation and procedures for
conducting impact analysis or monitoring already adopted legislation
(Slovenia). In some cases, a more institutionalised relation between
performance audit and policy units is already achieved through
consultative bodies (Finland) or through an understanding of evaluation
as an activity that should fall within the competences of SAIs (France).
Finally, the example of Estonia demonstrated that an increased level of
performance culture in the administration is necessary if performance
audit findings are to be meaningfully observed when conducting
evaluation, even if there is no formal requirement to do so. However,
regardless of the approach, the tangible results of the links between
performance audit and policy evaluation are not necessarily obvious,
which means that the work on performance issues is steep and narrow.
Also, the work on performance audit and evaluation can be very costly,
which was emphasised in the interviews with performance auditors

112
and public administration officials in case study countries. This is

one of the reasons why SAIs in WB countries give priority to financial
and compliance audits, especially in the light of capacity and financial
constraints.
113
6 W
hich Direction to Take in the WB?
Modalities of Interaction and Evaluation
of Implementation Options
This chapter is comprised of two main sections. The first one

elaborates on the modalities for establishing the interaction and
cooperation between the fields of PA and policy evaluation, based on
all relevant research findings (from WB and EU case studies alike,
although the latter were a major source of inspiration). Following
the description of the five modalities, we offer a discussion on the
feasibility of their application in each of the three WB countries,
based on the meetings with stakeholders which were organised in
the final research phase. The second section of the chapter turns to
a discussion of implementation options for the preferred modalities,
where three basic options are discussed. That part also offers an
evaluation of each of the options for the three WB countries in table
format, based on the experiences from similar fields and reforms
that have been implemented in the three countries in the given
administrative and legal culture.
6.1 Modalities of Establishing Interaction between

Performance Audit and Policy Evaluation
6.1.1 Description of Identified Modalities
I. Communication and professional cooperation
Even though the status quo still prevails due to a lack of a systematic
approach to policy evaluation and the early development phases of
performance audit, this interaction modality between the two fields
uses the available opportunities for creating relationships of trust
and cooperation between actors in the area of performance audit
and the decision and policy makers, without interfering in their
institutional roles and jurisdictions. In other words, communication
and professional cooperation implies a more intensive interaction
between the actors involved in PA and policy evaluation, an interaction
that is not exhausted in the auditor-auditee relationship. The value of
this modality lies in mutual learning and in raising the awareness of
the similarities and differences between performance audit and policy
evaluation. Moreover, a positive aspect of this modality is that the
first interactions are bottom-up initiatives, initiated by performance
auditors and policy evaluators themselves, or by respective managers.
6 Which Direction to Take in the WB? Modalities of Interaction and Evaluation of Implementation Options
114
In the absence of regulated ways for interaction, auditors and

evaluators are creating demand for more exchange among professions
for the benefit of both sides.
According to this modality, performance auditors and decision
makers or policy evaluators create a common informal professional
network through participation at conferences and other expert events
in the country and abroad, as well as by connecting on professional
social networks and using other opportunities to communicate. The
Netherlands are an example of such cooperation, where performance
auditors and evaluators actively use opportunities provided by the
associations and initiatives for exchanging expertise and knowledge
(European Evaluation Society, LinkedIn, auditors and evaluators
associations etc.). Experience of performance auditors of the
Netherlands Court of Audit also shows that few other SAIs participate
in these and similar initiatives.
II. SAI develops an evaluation culture through systemic
performance audits
Following the example of The Netherlands Court of Audit model which
started conducting broad, system-oriented performance audits a
few decades ago, SAI can also initiate performance audits that would
have a more systematic character, where the focus would be shifted
from sector issues or specific institutions within a certain field,
to the performance of public administration in general. Such audit
would include processes, procedures and measurements that public
administration bodies use in order to collect information on all three
aspects of organisational performance, namely economy, efficiency
and effectiveness. The result of such specific performance audit is
that it would point out potential gaps in the realization of activities
and the budget spending of auditees in relation to the defined goals.
In cases where an inconsistency between the proclaimed goals of
the organisation and the actual budget is proved, or where there is
an absence of adequate measurement and performance monitoring
systems, SAI can recommend establishing adequate monitoring and
evaluation procedures for a specific case, as well as at a systemic level.
Moreover, SAI could use its institutional relation with the parliament
and the committees in charge of the budget and the public accounts to
further present and discuss the findings of such systemic performance
audit reports. This in turn could produce more reverberation of these
reports in the public and, most importantly, among policy makers.
Systemic performance audits are implemented with full SAI autonomy
in defining the audit programme, and they are not one-time activities,
but instead take place regularly. Having in mind that performance audit
115
has the potential to be more flexible than other types of audit, public
administration bodies are supposed to implement recommendations
from systemic performance audit reports both in the mid- and the long-
term, whereas recommendations are treated as a body of knowledge for
improving their own work. This can be made possible through a special
mechanism introduced by SAI for monitoring the implementation of
performance audit report recommendations. Monitoring follow-up
to the recommendations is different from financial and compliance
audits in terms of the deadlines and obligations, thus influencing the
increase in trust and mutual cooperation in the relation between the
performance auditor and the auditee.
III. SAI conducts PA of the policy evaluation systems or of individual
evaluations (meta-evaluations)
In the process of planning performance audit topics, special attention is
given to evaluation systems and evaluation activities in the ministries
and other state bodies. In other words, SAI conducts an audit of quality
and usefulness, as well as of the fiscal impact of the evaluation reports.
Quality checking includes methodologies for data collection, relevance
of sources, findings and data, resources used for conducting the
evaluation, expenditures, as well as the usefulness of these reports for
decision makers and the sustainability of recommendations in terms
of improving the effectiveness of budget spending. The case studies of
the Netherlands and the United Kingdom show that this sort of “meta-
evaluation” is a significant instrument for revealing the shortcomings
in the implementation and the effects of the evaluations. In that sense,
they provide incentives for the improvement of individual evaluations.
As evidenced from these case studies, SAI could concentrate on policy
evaluations that deal with effectiveness research, i.e. those examining
the usage of budget allocations and policy instruments for achieving
socially desirable goals. As an alternative, it could put more emphasis
on cost-effectiveness or cost-efficiency studies, or investigate the
performance of an organisation’s management and operations.
Moreover, in this modality SAI can also decide to audit multiple policy
evaluations at once, provided that its capacities allow it. This modality
is especially useful in administrations which allocate significant
budget resources to evaluation studies, as they can examine whether
such studies are worthwhile.
IV. Policy evaluation system actively and systematically uses
performance audit reports
Inspired by the findings of the Estonian model, this modality implies
that policy planning units, which are in charge of policy evaluation
within the ministries, actively use performance audit reports and their
116
findings in conducting their analyses. This modality presupposes that

a policy evaluation system has already been established and causes
certain consequences in the policy-making process. The actors and
institutions which influence the policy evaluation process actively
work in the direction of its improvement, including the centre off
government institutions in charge of steering policy coordination,
planning and evaluation (Government Secretariats, Policy Secretariats,
etc.). A systematic usage of performance audit reports by policy units
means that this is not an ad hoc activity which depends on the tastes
and affinities of policy analysts and evaluators in administration. If
not regulated in a way which makes the usage of performance audit
mandatory, this exchange between performance audit and the policy
domain can function based on a mutual understanding of the benefits
of the performance audit findings for improving policy making in
general and policy evaluation in particular.
V. Evaluation as a distinct activity of the SAI
Policy evaluation as a separate, sui generis activity of the supreme
audit institution has recently been introduced in France. According
to this modality, in addition to performance audit and other types
of audit, SAI also has a constitutional and legal obligation to conduct
evaluations upon the parliament’s or its own request. Such a system
consists of two parallel evaluation systems which provide mutually
complementing insights on the performance of public administration
bodies – one that is introduced at the level of executive and the other
which is a part of external audit. In this setting, SAI capacities are
strengthened to a large extent, satisfying the needs for expertise and
knowledge in different fields, as well as the need for diversity when it
comes to the methods used and the analytical, social and writing skills.
Evaluations conducted by the SAI in this modality would need to
differ from those produced by the administration, by being more
comprehensive, but at the same time more objective, given that
they are conducted by an institution that is not a part of the policy
implementation process. These two systems can be differentiated by
the length and complexity of evaluations, as well as by the purpose
of recommendations and their focus. In order to avoid a duplication
of evaluation activities, actors in these two evaluation systems need
to be in constant communication, while simultaneously performing
counselling and information exchange regarding methodologies and
organisational issues. The undeniable quality of this modality is the
existence of parallel evaluation systems which are not conflicted, but
which complement each other. More specifically, by engaging directly
117
in policy evaluation, SAI advocates for a system of double-checking

policy effects.
6.1.2 Discussion of the Implementation Modalities based on

Stakeholder Meetings
The discussions with key stakeholders (SAIs and evaluation related
authorities mainly) in each of the three countries have yielded the
following conclusions:
Serbia
ÊÊ All stakeholders agreed that the first modality should be
implemented in the shortest possible timeframe and expressed
interest in connecting with their colleagues from the other
field. Nevertheless, they could not predict exactly how the
professional connections would be facilitated, who would be
in charge of organising meetings and events, etc. Therefore,
the instigation of such professional networks and networking
events would need to rely on actors outside of the government
sector, i.e. mainly on non-governmental organisations and
their initiatives.
ÊÊ Most stakeholders agreed that the second and third modality
could be implemented in some kind of a sequence, with the
second modality clearly preceding and being the precondition
for the third one. In fact, it was emphasised that, through the
implementation of the second modality, the SAI would serve
as a powerful ally to the governmental institutions in charge
of developing and/or reforming strategic planning or policy-
making systems and practices. The established practice within
the Performance Audit Sector of Serbia to refrain from filing
requests for misdemeanour proceedings and proceedings
for criminal offences, as well as SAI’s focus on system-wide
problems and topics in the two PAs conducted so far, positively
contribute to the development of the second modality.
ÊÊ One of the stakeholders expressed doubt as to whether the
implementation of the third modality in the medium term would
be cost-effective, given that it would presuppose focusing two
different evaluative efforts on the same policy. At the same time,
concerns were raised as to the poor quality and applicability of
many of the (ad hoc) evaluations which have been produced so
far, which could be remedied if meta-evaluations were produced.
Nevertheless, there seemed to be a prevailing agreement that
118
the third modality is still out of reach and would be viable only
in the longer run.
ÊÊ The fourth modality was well-received by both stakeholders from
SAI and from the government institutions, with some concerns
raised by one of the stakeholders from the latter group as to the
quality and policy learning potential of SAI’s reports. This issue
certainly corroborates the conclusion from the interviews that
SAI needs to work on its image and the awareness of the key
stakeholders of its work, particularly of the specificities of PA
compared to traditional forms of audit. Naturally, an important
caveat was stressed on both sides – in order for this modality to
be implemented, a more systemic approach to policy evaluation
would firstly need to be put in place (although PA results can
generally be used in policy formulation as a source of policy
learning). Nevertheless, the stakeholders mainly agreed that
the very design and development of the evaluation system
would in fact be the proper time to think about installing a
requirement (a substantive one, rather than a vague and general
one) for policy evaluators to refer to relevant performance audit
reports and comment or even report on whether the auditors’
recommendations have been implemented.
ÊÊ The fifth modality both attracted the most attention
(especially on the part of the SAI interlocutor) and raised the
most concerns as to the feasibility of its implementation in the
mid-term. The government interlocutors unanimously agreed
that the Serbian SAI would not be able to apply such a model,
with which the SAI interlocutor partly agreed, although she
expressed a strong affinity towards the consideration of
the “French model” one day, when SAI’s capacities are much
stronger.
Macedonia
ÊÊ In the discussion with stakeholders it was explained that
the first modality is ongoing in continuity (according to staff
of Macedonian SAO as well as of the Ministry of Information
Society and Administration). In the view of civil servants in
the ministry it can be accepted as the preferred modality given
that policy evaluation is in the early stage of development. It
was noted in the discussion that the voluntary aspect of the
modality can generally be considered as a continuation of the
status quo.
119
ÊÊ According to SAO representatives, the second modality

is implemented through a systemic approach to financial
management and control systems, including internal audit.
However, the level of success in its implementation is
dependent on the capacities of the institutions in question.
Additionally, when conducting performance audit SAI requests
opinion from all relevant stakeholders, e.g. in the ongoing
international performance audit on tourism, universities
and professional associations are consulted. For SAO staff,
every performance audit is an evaluation in the sense that the
objective is to evaluate the three Es, the possibility of which
depends on the circumstances. On the other hand, a lack of
indicators for performance measurement in the programme
budget impedes the process of further developing systemic
performance audits.
ÊÊ According to the SAO, the third modality is not feasible at the
current stage of development. It can be achieved only far into
the future given that the development of policy evaluation is
still in its early stage. However, this modality was not discarded
and was considered as a long term goal.
ÊÊ According to the stakeholders that deal with policy evaluation,
the number and scope of performance audits need to increase,
and SAO’s reports need to be made more accessible and visible
in order to fully make use of the fourth modality. Therefore,
the implementation of this modality is possible in the medium
term with significant training efforts for the staff in charge
of policy evaluation, guided by the Ministry of Information
Society and Administration. It is suggested that a manual
for cooperation would be useful for achieving the goal of
consulting performance audit reports in the practice of policy
evaluation.
ÊÊ The fifth modality is not feasible in the opinion of SAO staff
given the institutional tradition of the Macedonian SAO
and the potential duplication in the circumstances of scarce
resources. In the Ministry in charge of administration, which
is the central institution in the country in charge of policy
evaluation, the idea of having two parallel evaluation systems
was an option, although they considered it imperative for the
executive to engage in the policy evaluation process. For the
latter, the institutions that draft a policy have to evaluate it in
order to discern the errors in its implementation.
120
Montenegro
ÊÊ Regarding the first modality, both the auditors and the state
administration personnel involved in policy evaluation
have no objection to establishing this kind of cooperation
in Montenegro. However, the problem with this modality in
Montenegro is that the SAI does not know whom to dial when
they want to reach the policy evaluation structures. This is due
to the fact that there are no specific policy evaluation bodies
and no professionalised policy evaluation officials, meaning
that it is hard to establish permanent forms of consultation
and networking. Also, policy evaluation practitioners in
the administration are at the same time assigned different
functions, sometimes even senior positions. This in turn could
make them reluctant to cooperate with SAI, considering that
auditors are sometimes perceived as investigators.
ÊÊ At present, the kind of scope for a performance audit suggested
by the second modality is too ambitious for the Montenegrin SAI,
whose audits are still almost exclusively focused on individual
institutions. Currently, the only topic that was considered
for such a wide ranging audit was the internal audit, as SAI
plans to increase its cooperation with this segment of the PIFC
system. The ambition of SAI is to continue auditing the internal
audit system in the future, as well as to conduct regular checks
of its functioning. On the other hand, the development of a
comprehensive system to monitor the follow-up to reports is
being considered in the SAI, with plans to set the foundations
by developing methodological guidelines for verifying the
implementation of recommendations. This mechanism, once
set up, would serve for future developments in the performance
audit sector, including systemic performance audits.
ÊÊ There were considerable objections on the part of staff in charge
of policy evaluation to the kind of SAI intervention proposed by
the third modality. This is particularly due to their conviction
that SAI is still not in the position to assess their work and
lacks the knowledge and experience for such assignments.
Similarly, SAI methodology for choosing performance audit is
detailed and extensive and stakeholders from SAI are sceptical
that such a topic would be given priority, especially having in
mind the low capacities of the Performance Audit Department.
In these circumstance, the third modality is given a long-term
priority.
121
ÊÊ In order for the fourth modality to be in place, a more

structured approach to policy evaluation is required in the
first place. This was recognised by the stakeholders in the
public administration. Thereby, it is not feasible to implement
this modality in the short term.
ÊÊ Neither the stakeholders in the state administration nor those
in SAI consider the fifth modality as a viable option. On one
side, stakeholders within the SAI claim that it is in the mandate
of SAI to conduct evaluations as such, while on the other, public
administration representatives are of the opinion that SAI’s
capacities are insufficient for conferring it to policy evaluation
competences.
6.2 Evaluation of Implementation Options

Based on the identified modalities in the previous section, and taking
into account the discussions with relevant stakeholders in the three WB
countries, we have established three basic options for the implementation
of the preferred set of modalities for all three countries. Given that
the preferences of the local stakeholders differ substantially when it
comes to the modalities which they consider desirable and feasible
within their national contexts, the evaluation of options is done for each
country individually. Implementation options are not modelled after
strategic choices and/or policy directions, but after the alternatives for
implementing the preferred options. The three options identified are: A)
Status Quo; B) Soft Law Approach; C) Hard Law Approach. What follows
is a concise delineation and description of each option, followed by their
evaluation, and based on a predefined set of criteria.
Option A: Status Quo

The first option implies that no new policy instruments are designed
and adopted in order to ensure the implementation of the preferred
modalities. This option, thus, continues the current state of affairs
in terms of regulation and institutionalisation of links and relations
between PA and policy evaluation, which, in practice, means that no
regulated links are achieved (with a partial exception of Macedonia
where a link has been determined through the obligation to consider
the reports of independent bodies prescribed by the Methodology for
evaluating legislation).
122
Option B: Soft Law Approach

The soft law approach entails the mildest possible way to regulate
a relationship between PA and policy evaluation. It is distinct from
Option A, as it does require a new policy instrument (one or several)
to be prepared and adopted, although a number of possible sub-
options could be designed for the exact forms of such instruments.
What distinguishes this option from Option C (see below) is that it
proposes a soft instrument, which is not binding for its addressees,
but only provides recommendations and guidelines.
There are various ways to put Option B in practice. One way is for
the SAIs to develop their internal guidelines for identifying topics for
performance audit in a manner which is more conducive to better
coordination and/or interaction with policy evaluation, i.e. preference
for systemic audits rather than institutional ones or a focus on the
audit of government performance systems (such as management of
the programme budget, civil service appraisal systems, internal audit
systems, etc.). Another way would be for the institutions coordinating
policy evaluations in the administrations (e.g. Government
Secretariats, Policy Secretariats, Ministries in charge of public
administration) to adopt guidelines, methodologies, or checklists for
evaluations which require consultation of performance audit reports.
The hardest sub-option in this soft-law approach would be for the
Government to adopt guidelines recommending all ministries and
other administrative bodies to consult the performance audit reports
when conducting policy evaluations, which would also potentially
improve the overall culture of follow-up to audit reports. All of these
variants of the option would only provide recommendations (rather
than obligations).
Option C: Hard Law Approach

The hard law approach presupposes the adoption of primary
(statutory) or secondary legislation, and is thus the most sophisticated
option in terms of the complexity of the regulatory activity which
needs to be invested in its implementation. Unlike in the Option B, the
policy instrument of this option would be binding for all ministries
and other administrative bodies conducting evaluations.
In most administrative systems originating from the ex-Yugoslav
system, secondary legislation (regulations, bylaws) can only be
adopted by the Government if there is a clear basis for it provided in
the law. Therefore, a combination might be needed of a more general
legal basis in primary law and a more specific provision in bylaw. It
should be specified here that such a legal basis could be provided for
123
in any relevant law which regulates the policy process and represents
the basis for passing a bylaw on the process for policy formulation,
monitoring and evaluation (e.g. in Serbia the Law on the Planning
System which is currently in the adoption procedure, or even a more
generic law, such as the Law on State Administration, if it gives the
basis for setting out the policy evaluation process). In such a case,
the obligation for the evaluation process (in both the planning and
the realisation phase) to consult performance audit reports would be
included in the bylaw defining the process of policy evaluation.
Evaluation of options has been done on a predefined set of criteria,
comprising four questions:
Q1 Can the option ensure implementation of preferred modalities of
interaction?
Q2 How does the option fare against perceptions and preparedness of
key stakeholders?
Q3 How does the option fare within the existing legal and administrative
culture?
Q4 Can the option be implemented in the medium term (2-3 years)?
Evaluation of implementation options per country

It should be kept in mind that this evaluation table refers to the
implementation options (status quo, soft law approach and hard law
approach), and how those options can help achieve the preferred
modalities of interaction between PA and policy evaluation.
124
Macedonia Montenegro Serbia
Allows for implementation Allows for Allows for

of the first modality, implementation of the implementation of the
which does not normally first modality, which first modality, which
require any regulation. does not normally does not normally
As evidenced by the require any regulation. require any regulation,
interlocutors, there is The stakeholders but depends on
Q1
already communication need to be more voluntary actions and
in place, however it familiarised with the role external (CSO) support
can be increased in the of performance audit and initiative. Cannot
future. Cannot ensure the and policy evaluation ensure implementation
implementation of other in order to successfully of other preferred
preferred modalities. implement this option. modalities.
Option Generally positively Generally negatively Generally positively
1: Status both within SAO and the with both the SAI with SAI, less positively
Quo Q2
government stakeholders. and the government with government
stakeholders. stakeholders.
Voluntary aspect makes it In a generally Poorly – in the Serbian
clear that solutions that do administrative culture, legal and administrative
not require any degree of the lack of regulation culture it is widely
regulation could actually increases the likelihood considered that most
Q3 impede more meaningful that the modality is not new practices need
interaction. given adequate priority some degree of
or that it will not be regulation, otherwise
implemented once it is they will not happen.
agreed.
Q4 N/A N/A N/A
The soft law approach Currently, the level of It would not cover the
is preferred among development of policy first modality, as it does
relevant stakeholders evaluation and the not need regulatory
in the implementation focus of performance action. The second
of modalities. The audit in Montenegro do modality can be
fourth modality can not render the soft law implemented only with
be implemented (or approach adequate. a formulation of internal
strengthened given SAI recommendations
that there is an actual related to the scope
obligation to consult and target of PA and the
Q1
independent reports when ensuing increased focus
conducting evaluation) on performance systems
through the introduction of and policymaking
a manual for cooperation processes. The fourth
which would provide more modality is not likely
detailed guidance on how to be implemented
Option 2: to incorporate findings through the soft law
Soft Law from performance audit approach.
Approach reports when practicing
policy evaluation.
There is no resistance There is no resistance There is no resistance
among stakeholders nor explicit approval of among stakeholders
towards the soft law the soft law approach on either side in
approach. However, it is among stakeholders. applying the soft law
clear that there is a need They do not object, approach. However,
for more capacity building but also fail to see there is no obligation
on both sides in order the benefits for their for SAI to participate
Q2 for the soft solutions to respective work by in any activity initiated
work. Extensive trainings implementing this by the government
and capacity building/ option, and therefore, bodies. In that sense,
HR development are do not “own” the option this approach would
necessary. Furthermore, but would implement be based on the
informal dissemination it on paper if it was voluntariness and
should be used. required of them by a flexibility of the SAI
higher instance (the EU). management and staff.
125
Macedonia Montenegro Serbia
In the administrative The use of soft In general, soft law is

culture where the law approach not a good instrument
introduction of new in Montenegro in the Serbian legal and
solutions into the system to arrange inter- administrative culture,
requires a certain amount institutional relations and frequently fails to
of time to work and generally signals the deliver results. A sense
there is a need for more intent or obligation of obligation is usually
capacity building and to implement needed to ensure
learning, soft instruments something, without results, although an
do not necessarily deliver the real intention of active implementation
Q3 the results that are following through monitoring role by
intended. and establishing a specific body can
a new practice. help. A government-
An additional adopted guideline
Option 2: problem that was document, monitored
Soft Law registered is possible by the competent
Approach inconsistency that centre-of-government
would exist due to institution could,
the different actors therefore, produce
developing their own results.
guidelines.
This option can be This option could be Yes, especially
implemented in the implemented in the given that soft law
medium term with medium term, but documents produce
technical assistance. stakeholders on both less reaction and
Q4 sides would need the resistance from within
expert assistance the conservative
from the civil society elements in the
organisation that has administrative system.
raised the initiative.
As this option is Application of Apart from the first
not preferred, this option would modality, this option
implementation most likely ensure would be the one
through this option is not implementation of the most likely to ensure
Q1
considered. fourth modality, except the implementation
for the first which does of the other preferred
not require a hard law modalities (2 and 4).
approach.
Soft law approach There is no resistance It resonates well
is preferable among to the hard law within both the
stakeholders both in the approach among government and the
administration and within stakeholders in SAI. The opinion of
Q2 the SAI. Montenegro, but rather the stakeholders is
an ambiguity related that for any solution
Option 3: to the selection of a to be implemented
Hard Law preferred modality. with certainty, hard
Approach regulation is required.
Given the highly Given the highly Given the highly
regulatory culture, this regulatory culture, this regulatory culture, this
Q3
would be the preferred would be the preferred would be the preferred
option on this criterion. option on this criterion. option on this criterion.
Yes, if extensive capacity Hardly likely due to the If the currently ongoing
building and trainings complex processes of process of adopting
for the staff engaged getting new legislation the Law on Planning
in policy evaluation are on the agenda, if it is System and its bylaws
implemented. not directly required (methodologies) is
Q4
by the EC and the not stopped by a lack
negotiation process at of political will or the
the moment, in exactly announced general
this form of hard law elections, it is feasible.
approach.
126
Out of the three proposed option for the implementation of modalities,

the one that stood out as the preferable one in the discussions with
stakeholders in Montenegro and Serbia, is the Hard Law Approach,
i.e. the one requiring binding regulatory activity. The reasons for this
preference are discussed previously in the study and pertain to the
legalistic administrative cultures. Highly regulatory cultures focus
on binding solutions as the foundation for any action. Conversely, in
Macedonia, the Soft Law Approach has been the preferred one among
the stakeholders. Justification for this choice is found in the opinion of
stakeholders from Macedonia that some of the modalities are already in
place, at least partly. Therefore, what is necessary is not a new binding
provision, but the improvement of the existing solutions, such as the
requirement of the Methodology for ex-post evaluation to consult the
reports of independent bodies.
As was expected regarding the modalities for establishing a link
between PA and policy evaluation, not all of them are preferable or even
possible to implement. In Montenegro, there is an ambiguity among
stakeholders when it comes to selecting a single preferred modality
and the only modality that did not raise considerable objections was
the first one. In Macedonia, the most preferred one is the first modality
due to the early stage of development of policy evaluation. At the same
time, second preference is given to the fourth modality due to the
already institutionalised requirement to consult reports of independent
bodies in legislative evaluation. However, before full acceptance of this
modality there is a need for extensive capacity building for evaluation.
In Serbia, the second and the fourth modality received the most praise.
The second was well-received in the SAI owing to the readiness of the
Performance Audit Sector of SAI to further develop its own work. Still,
whether or not implementation of this modality is feasible will depend
on the SAI Council as the decision-making body. The fourth modality
was also well-received among all stakeholders, particularly having
in mind the first steps in the introduction of the evaluation system
in Serbia through PPS. However, this modality faces two conditions –
the necessity to 1) establish a system-wide evaluation system, and 2)
improve and further develop performance audit work of SAI.
127
7 Conclusion and Policy

Recommendations
The authors of this study have raised the question of the possibility of
establishing links between performance audit and policy evaluation
in the three Western Balkan countries that are in the process of EU
accession – Macedonia, Montenegro and Serbia. Argumentation was
based on two driving factors. Firstly, while the complementarity
and the similarities between the two fields have been emphasised in
international literature, they remained understudied in comparative
practice in the context of the Western Balkans. Secondly, the EU
accession process itself requires acceding countries to develop effective
performance audit and policy evaluation functions for the purposes
of achieving greater effectiveness of the public sector delivery. The
authors gave an analytical insight into the development, functioning
and consequences of performance audit and policy evaluation alike
for each of the countries. This was done not only for the purpose of
describing the local contexts and the nascent nature of these two
functions, but also in order to research the possibilities of creating
synergies and links between the two fields, for the benefit of better
performance of the public sector and policymaking that is enrooted
in credible evidence and analysis. On the basis of this analysis, the
authors examined the current state of interaction between these two
functions, as well as the related obstacles and opportunities. Moreover,
the study incorporated elements of six case studies (EU Member
States), and their experience and practice in linking performance audit
and policy evaluation. This was added in order to provide a body of
evidence for constructing the modalities of interaction between the
two functions, as well as the options for implementing the preferred
modalities. Modalities of interaction and implementation options were
developed so as to present multiple choices in terms of their complexity
and types of policy instruments required for implementation. In this
final chapter, we draw general concluding remarks, followed by general
recommendations focusing on commonalities and identified parallels
in all three countries. Finally, we offer specific recommendations for
each country, based on interviews, discussions and own insight, in
order to account for the specific Macedonian, Montenegrin and Serbian
local contexts.
7 Conclusion and Policy Recommendations

128
7.1 Concluding Remarks

Performance audit and policy evaluation have different origins with
regards to the disciplines they evolved from, but the question of their
relationship started to emerge in the advent of transformations in the
public sector philosophy a few decades ago. This change was reflected
in the efforts to make the public sector more performance-oriented,
and outlining conceptual, institutional, as well as methodological
similarities and differences between performance audit and policy
evaluation has become a part of the academic agenda. While some
of the differences, such as the institutional ones, have remained
undisputed, discussions on conceptual similarities are still ongoing.
Similarities are found in the general purpose of both processes,
which extends beyond the improvement of accountability and enters
the policy learning arena. Additionally, the possibility to apply
similar tools and methods in their work, emphasised by a number of
authors, builds on the conceptual similarities and highlights another
connecting point. While the actual influence of performance audit
on policy learning depends on the respective administrative culture
and approach of local SAIs, there seems to be an agreement that
both performance audit and policy evaluation should also serve to
improve policy results.
There are no predefined concepts leading towards the creation of
links between performance audit and policy evaluation. As can be
seen from the case study examples, the applied approaches are mainly
contextually driven in order to satisfy domestic needs and complement
institutional arrangements, taking into account institutional memory
and the administrative culture. In that sense, the Western Balkan
countries will have to look into these (in)variables, while at the same
time learning from past experiences. However, parallels between
Macedonia, Montenegro and Serbia shed light on a few aspects of
performance audit and policy evaluation that need to be considered
before proposing recommendations.
Namely, in order to establish links between performance audit and
policy evaluation it is useful to consider the differences between
government and performance audit planning. Government priorities
could be observed more closely by SAIs during the process of planning
performance audits. However, in the current circumstances, when
performance audit experience and orientation differ across WB
countries (focus on institutions, specific sector/programmes or
horizontal issues), and policy evaluation practices differ in terms of
their institutionalisation and focus (project, programme, or legislation),
each country would have to approach the matter of “aligning” agendas at
129
both ends and take domestic nuances into account. The question which
remains is in what way and to what extent should audit planning monitor
government priorities when SAIs have their own risk assessments and
methodologies for selecting the audit topics. Furthermore, low quality
of strategic and policy planning would hamper the efforts to “align the
agendas” in the sense that it would prevent the SAIs from having a clear
understanding of government priorities and how public administration
organisations measure and monitor results.
Moreover, independence is a critical feature safeguarded by the
international audit standards, and one of the major concerns of SAIs
in the studied Western Balkan countries. It was made evident by the
interviewees from SAIs that independence in organising their audit
work is crucial for the credibility of the external audit function and their
institutional integrity. In general, however, there is support for the idea of
bringing performance audit and policy evaluation closer together, as long
as this relation does not thwart institutional roles. In some cases, there
are already visible traces of how these relations could be established
without questioning the independence of SAI and its neutral position, as
could be seen in the example of the Methodology for ex-post evaluation in
Macedonia. It could be hypothesised that the concern over independence is
influenced by the need to reiterate the distinct position and status of SAIs,
since these are (relatively) new institutions in the studied WB countries.
Consequently, the need for mutual learning and institutional awareness
raising is pending and equally relevant, regardless of the differences
in their experience, for all three countries studied. Perceptions of
stakeholders are influenced by limited experience in communication,
hence, they have resulted in a narrower understanding of each other’s
roles among performance auditors and representatives of public
administration bodies alike. Given that support from within the relevant
institutions is key for the sustainability of any links created between
the two fields (regardless of the modality or the approach chosen), it is
crucial for the institutions in charge of performance audit and policy
evaluation to communicate with each other. That communication
should focus on covering their objectives, approaches and the tools
they apply, so as to get the most out of potential synergies. Two
major conditions need to be met as the first step in the establishment
of connection: 1) raising the awareness among stakeholders that
performance audit is crucially different from financial and compliance
audit, and 2) acknowledgment by the SAIs that policy evaluation is a
complementary activity in improving public sector performance.
Finally, complementary policy developments in the respective countries
could provide a solid foundation for identifying points for interaction and

130
thus facilitate synergies. Several interviewees emphasised the importance

of performance-based budgeting (or programme-based budgeting), both in
the context of policy evaluation and performance audit. They considered it
as an important precondition supporting development and quality in both
of the fields we have studied. The logic is clear: the programme budget is
the only government document which fully connects goals and indicators
with financial resources in a binding manner, and whose execution is
regularly monitored. While the programme budget is officially introduced
in all three WB countries, there are efforts to improve its implementation.
If properly implemented in all three WB countries, the programme budget
could, in the future, represent a strong connecting point between PA and
policy evaluation, as professionals in both fields should seek to analyse
and compare budget information with other relevant sources of data
and information. Similarly, the reform of the official statistics systems
and the resulting improvement of the availability and quality of data
was particularly emphasised as a critical condition for making progress
in any area which requires the measurement of performance against set
indicators.
7.2 General Recommendations

The SAIs are autonomous in deciding the performance audit agenda
and should be able to decline any requests coming from external
actors. Nevertheless, “maintenance of the SAI’s independence does not
preclude requests to the SAI from the executive, proposing matters for
audit.”290 The institutions in charge of establishing and coordinating
the policy evaluation system could potentially be a useful source of
information for the identification of performance audit topics, based on
the focus and findings of evaluation reports, which could then result in
the establishment of a meaningful link between PA and evaluation. As
the case studies have shown, it is not uncommon for SAIs to examine
policy evaluation reports sent by policy units in the ministries.
Recommendation to SAIs:
 It is recommended that SAIs monitor the developments in the
area of policy evaluation and collect and analyse information
about the conducted evaluations, as part of their PA planning
methodologies and procedures. Information from evaluation
reports could indicate risks and uncertainties in the execution
of government programs as well as a lack of reliable data. This in
turn, could increase the need to take up performance audit.
ISSAI 3000, 44.

290
131
Recommendation to governments:
 To ensure that SAIs are better informed about the ongoing and
completed policy evaluations, the policy units and/or centre-
of-government institutions in charge of the evaluation systems
could be obliged or recommended to regularly send such updates
to SAIs.
As suggested by ISSAI 3000, performance auditing may differ in
approaches and it should not be streamlined, in order to preserve its
creativity and flexibility.291 The EU practices have shown that it is up to
individual SAIs to decide which approach is to be applied and what are
the goals of individual performance audits. The research has revealed
significant differences between the countries – both EU and WB ones
– in the orientation of PA towards single institutions and specific
programmes, or towards more horizontal, system-wide issues. At the
same time, a shift in focus towards systemic issues could be beneficial
in terms of avoiding the “name and blame” game or pure accountability
checks, and instead ensuring a better understanding of the underlying
policy problems. A systemic focus of PA can, thus, increase the level
of usefulness of performance audit reports for the improvement of
policy implementation, i.e. for policy evaluation, and render them more
relevant for policymakers and the overall policy cycle.
 SAIs should devote a part of their performance audit work
to system-oriented issues or causes of failures of policy
implementation. Such audits could examine the existing
performance measurement systems and approaches, provide
recommendations for improving them, and lead to the creation
of more systemic approaches to policy monitoring, as well as
policy evaluation.
The study has shown that the deadlines for follow-up in the performance
audit process do not differ from those in compliance and financial
audits in the studied countries. In order to facilitate meaningful and
real policy learning, it is important to consider extending deadlines
and approaching the follow-up to recommendations in a significantly
different manner, given that recommendations of performance
audit reports usually tackle issues which go beyond compliance and
institutional procedures, requiring more systemic and policy-based
interventions.
Ibid., 29.
291

132
 It is important to establish follow-up mechanisms for monitoring
the implementation of performance audit recommendations,
which are distinct from other types of audit and sensitive to
specific approaches of the three SAIs and to the stakeholders’
needs in these countries.
A recurrent obstacle for establishing a link between the two processes
is found in rigid existing ties between SAIs and public administration
bodies, where auditees feel they are in a position to only account for
potential mismanagement. While fully applicable to Western Balkan
countries, this obstacle is also common for the EU case studies.
Increased communication throughout the performance audit process
could increase the sense of ownership of the performance audit reports
among the stakeholders, which would consequently “contribute to
better knowledge and adequate improvements.”292
Recommendations to SAIs and governments:

 Stakeholder consultations among auditees and performance
auditors should be organised as often as possible, before the
audit starts and throughout the stages of the performance
audit process, provided that the independence of SAI remains
intact. It can help stakeholders understand the concept and the
goals of performance audit, and respond to performance audit
requirements in a more meaningful manner;
 The process of preparation of the final performance audit reports
should be made inclusive by including the concerns and remarks
of the auditees, in collaboration with performance auditors. The
focus should be less on procedural aspects, and more on the
substantive aspects of communicating findings.
Systemic ex-post policy evaluation (targeting policy programmes,

instruments etc.) is still to be fully introduced into the administrative
systems of all studied WB countries. This, however, does not prevent
potential efforts to address performance audit reports, specifically in
different stages of the policy cycle.
Recommendation to governments:
 Consulting performance audit reports should be applied in the
work of ministries and other administration bodies in the process
of designing and evaluating policies, taking into account country-
ISSAI 3000, 72.

292
133
specific conditions. The first steps in this direction have been taken
in Macedonia through the Methodology on ex-post evaluation
and the consultation of reports of independent agencies – “In the
process of ex-post evaluation the ministries acquire information
from all the independent bodies that perform external evaluation
of the level of successfulness of the policies and the efficiency of the
regulations.” In Montenegro, the first traces are found through the
Action plan for the implementation of SAI recommendations and
the Coordination body which is in charge of monitoring. However,
the need to consult performance audit reports specifically should
be emphasized and integrated into current and future reform
processes, especially performance management-related reforms,
including policy planning and management reforms.
The evaluation structure in the studied countries is either missing
(Montenegro, Serbia) or the focus is put on legislation (Macedonia).
Major efforts still need to be made for the introduction of a clear ex-
post policy evaluation structure, with clear competences and division
of duties.
Recommendations to governments:
 The formal requirement for ex-post policy evaluation should be
clearly emphasized by the legislation (primary or secondary),
stipulating general types of ex-post evaluations to be performed,
level at which they will be conducted (policy, budget programme,
law, etc.) and their scope and frequency;
 It is recommended to transfer evaluation competence to a single
institution at the centre of government. This institution would not
be in charge of conducting evaluations, but acting as the “driving
engine” behind the development, implementation, oversight
and quality assurance. This institution would assign roles and
responsibilities to lower tiers of the evaluation structure. A
clear hierarchy of competences would facilitate monitoring the
functioning of the evaluation system.
7.3 Specific Recommendations for Macedonia

The main challenges for performance audit in Macedonia are:
improving the quality of performance audit, introducing mechanisms
for monitoring the implementation of the recommendations, and
generally raising awareness among the institutions regarding the role
of performance audits. As for the ex-post evaluation, there is a need
to gradually increase their number and to enhance the capacities for
policy evaluations in the ministries.

134
Therefore, it is recommended to SAO to:

 Continue and intensify the process of increasing the number, quality
and visibility of all conducted performance audits;
 Monitor the existing obligations of the ministries to publish on their
websites the plans for conducting ex-post evaluations (for 2 laws
per year);
 Increase the cooperation with the parliamentary committees
(especially the Finance and Budget Committee). In addition to
this, there is a need for strengthening the oversight capacity of the
Parliament, especially with regards to deliberation on performance
audit reports.
It is recommended to the government and other

administration bodies to:
 Gradually introduce the concept of programme-based budgeting,
including performance indicators and objectives. This would allow
the performance audit to focus on efficiency and effectiveness in
comparison with the planned goals, and evaluation of the taxpayers’
value for money;
 Implement capacity building activities with the goal of enhancing
the capacities of the ministries for conducting policy evaluation,
especially gathering data and data analysis, which includes
analysing the findings of the relevant performance audit reports.
In this regard, it is necessary to improve the communication
and cooperation between SAO and the units in charge of policy
evaluation in the ministries, especially the Ministry of Information
Society and Administration (MIOA), as it is the key institution
responsible for the process of introducing policy evaluation;
 Gradually increase the number of ex-post evaluations while
improving their quality, taking into account the lessons learned
from the earlier policy evaluations;
 Improve the process of follow-up for performance audit reports,
with a greater focus on embedding the lessons learned in the
policy cycle.
7.4 Specific Recommendations for Montenegro

Considering the long road ahead of Montenegro regarding developing
and reaching full potential of policy evaluation and performance audit,
it is especially important to ensure a correlation between the priorities,
objectives, form and structure of these reports. In order to achieve
135
progress in these areas, a multifaceted set of reforms is needed. An

analysis of policy options and the stance of main stakeholders towards
different modalities of cooperation and coordination, shows a high
degree of mistrust and low level of understanding of each other’s work.
Before having more advanced forms of cooperation, thorough reforms
are needed in both PA and PE, as well as gaining common ground in
terms of mutual communication and trust. Additionally, there are
horizontal issues that deal with the development of the medium-term
budget planning, performance budgeting and policy coordination that
should be dealt with for a proper development of both PA and PE.
Given that Montenegrin stakeholders did not clearly opt for any of the
proposed modalities of establishing interaction between PA and PE,
we propose recommendations which would first ensure that these
two fields achieve a level of development where they are recognisable
in their respective and each other’s professional communities, which
would ensure a better understanding of each other’s roles. The
first step in this direction is to provide a channel of communication
between performance auditors and civil servants who deal with policy
evaluation.
Regarding performance budgeting, the Ministry of Finance

should in the short term:
 Establish a separate Programme Budgeting Unit, which will be
tasked with the development and technical support to spending
units in terms of defining the non-financial elements of programme
budgeting, and offering training programmes in the future;
 Prepare an action plan for implementing the programme budgeting
for a three-year period, in which special attention would be
given to the plan of developing the non-financial elements of
programme budgeting and to the programme objectives and
indicators in several pilot institutions.
Regarding policy evaluation development, the short term

priorities should be:
 Adopt amendments to the Government’s Rules of Procedures and
formally make ex-post policy evaluation obligatory;
 Identify priority areas for evaluation in the Government’s working
plan, or in a separate document which will be prepared by the
General Secretariat of the Government (GSG) in cooperation with
the policy units in the ministries;

136
 Improve the capacity of the GSG, since this body could potentially
be the administrative and professional core of providing support to
other policy units related to policy evaluation;
 Ensure that existing experience with RIA is used as the basis for
developing ex-post evaluation; units in ministries that implement
RIA and evaluation of the IPA funds could be trained for the
implementation of policy evaluation;
 With expert support, formulate methodological guidelines for
policy evaluation which could also define how to use the findings
of the performance audit in these processes.
Regarding the further development of performance audit, the

short to mid-term recommendations are:
 Performance audit department should be strengthened
regarding human resources, both in terms of the size and the type
of personnel;
 The number of positions systematised for the Performance Audit
Department should be increased, namely, doubled. Therefore, the
decision on the amendment of the Regulation on systematisation
and internal organisation should be adopted by the SAI Senate;
 The Law on SAI should be amended in order to allow professionals
with a background in fields other than law and economics to gain
the state auditor certificate, and thus provide the needed basis for a
multidisciplinary approach in performance audits;
 In the mid-term period, the SAI should make a transition from
institution-based to policy-based performance audit, moving
further away from the patterns of financial and compliance audit;
 Government’s Coordinating Body for monitoring the
implementation of SAI’s recommendations should expand its remit
in order to include performance audit as well, and report quarterly
on what has been done to implement the recommendations;
 SAI should do more to follow up on the implementation of its
performance audit recommendations, mainly through monitoring
the work of the administration, and making public all of the documents
received from the auditees, such as the plan for implementing the
recommendations and follow-up reports; follow-up audit on the
performance audit reports should be mandatory, however, this
should be granted a longer time for implementation compared to
financial and compliance audits, whose recommendations require
less time to be implemented;
137
 SAI should adopt a methodological manual for formulating the

recommendations and policy options, in order to make them more
operative and avoid hitherto experienced problems of vague,
abstract or non-implementable recommendations;
 SAI should amend its methodological guideline and practice in
order to be more extensive, and devote more time to discussing
and shaping the preliminary performance audit report and its
recommendations in cooperation with all relevant auditees, as well
as the related institutions before its final adoption;
 Parliamentary Committee for the economy, budget and finance
should review each performance audit report at a separate session,
and engage in monitoring the execution of the action plan for
implementing the SAI’s recommendations.
For first steps in establishing the link between PA and PE, it is

necessary to:
 Improve communication between the policy units in the
ministries and the GSG on one hand, and the Performance Audit
Department of the SAI on the other. This could be done through
regular consultations or online communication through interactive
tools and instruments;
 Amend SAI’s procedure for formulating the annual audit plan, with
the goal of creating some alignment with the Government work
plan, but without infringing on SAI’s constitutional independence.
This would mean that performance audits could be planned in
policy areas where more radical changes are planned by the
Government in the coming period, where systemic legislation
will be changed or amended, or where restructuring of the
institutional framework is planned. That way, the results of PA
reports could directly feed into the policy formulation stage and
serve an evaluative purpose.
7.5 Specific Recommendations for Serbia

In the discussion of the implementation of modalities with stakeholders,
three of them stand out as the most preferable ones – modalities 1, 2
and 4. In this section, recommendations for the implementation of these
modalities are provided. For the reasons discussed in the evaluation
of implementing options for Serbia, the hard law approach is held as
the most appropriate one for establishing links between performance
audit and policy evaluation, especially for the implementation of
modality 4. At the same time, modality 2 is more suitable for a soft-law

138
approach, whereas modality 1 cannot really be regulated and could

only be achieved through voluntary actions of performance auditors,
policy evaluators, and interested third parties (civil society, technical
assistance projects, etc.).
Recommendations for the implementation of preferred options are
designed in a way as to emphasise who is called for intervention, SAI
or government bodies, as well as to propose the time needed for their
implementation.
7.5.1 Recommendations pertaining to the first modality of

establishing interaction
Currently, there is a lack of communication between performance audit
and policy evaluation, i.e. performance auditors and civil servants
whose tasks relate to policy evaluation are mostly unaware of each
other tasks and roles, and, consequently, of the potential impact that
mutual communication would entail. Stakeholders expressed that the
first modality of cooperation is the most feasible one, since it does not
require implementing regulations.
Short-term recommendations to SAI:

 It is recommended that SAI of Serbia launches an overall performance
audit promotion campaign. This could be done through the existing
channels of communication (media and online presentation)
or through specifically tailored tools, such as a subdomain for
performance audit within the SAI official internet presentation,
promo materials, public events for raising awareness of performance
audit in general or specifically focused on meetings with policy
evaluation-related units and institutions within the administration;
 SAI of Serbia should use existing institutional connections
with the National Assembly to promote the PA function and its
importance for the improvement of policy design, implementation
and effectiveness. In particular, this could be done through the
Committee on Finance, State Budget and Control of Public Spending
or, more specifically, the Subcommittee for the Consideration of
Reports on Audits Conducted by the SAI. Relevant stakeholders
from the public administration should be invited to participate in
sessions of the committees;
139
Short-term recommendations to SAI and government:

 It is recommended to the SAI, Public Policy Secretariat, as well as
civil servants engaged in policy analysis tasks, to join and participate
in domestic and international professional bodies and discussion
forums on evaluation in the broadest sense. Participation in such
venues could ensure a better mutual understanding, sharing of
practices and experiences, and result in stronger professional
bonds between the two fields.
 It is recommended to the SAI of Serbia and relevant state
authorities to participate in the events and project activities
of civil society organisations which promote public sector
performance (policy monitoring and evaluation, performance-
based budgeting etc.).
7.5.2 Recommendations pertaining to the second modality of

The potential of SAI to contribute to the development of an evaluation
culture through its performance audit function is currently not
recognised. In order to stress this potential role of SAI, it is necessary
to consider changes in the way performance audit work is done,
especially in the early stage of development of this function of SAI.
Mid-term recommendations to SAI:

 It is recommended that SAI of Serbia issues internal guidelines
related to the scope and target of performance audit, which would
include performance management systems and policymaking
processes as potential topics for PA. Implementation of this
could be achieved either through the adoption of a separate
internal document of SAI, which explicitly contains such
provisions, or through the amendments of the SAI manual on
performance auditing. This should be done in close consultation
and with participation of the Performance Audit Sector;
 SAI of Serbia should adopt and make available an internal
instruction or methodology for planning performance
audit, outlining a wide range of tools and methods for the
implementation of PA. This would strengthen the evaluative
aspect of performance audit and make SAI a relevant actor in the
promotion of the evaluation culture and related practices;

140
 The Law on State Audit Institution should be amended so as to

allow for:
• Engagement of professionals of multiple disciplines in audit
work.293 Entry requirements should be tailored to suit the
specific needs of individual sectors. In this case, the Performance
Audit Department should be allowed to employ professionals of
any field relevant for the performance audit function, including
a range of social science disciplines or even wider;
• Follow-up to SAI recommendations which takes into account
the stakeholders’ needs, in terms of the time needed to address
policy and systemic issues which those recommendations
often tackle. Strict timeframes for implementation
might cause poor implementation of more systemic and
strategic recommendations.294 A PA-specific mechanism for
communicating with auditees regarding the implementation of
recommendations could also be introduced. This could include
the development of action plans for implementation by the
auditees, in a consultative and iterative process with the SAI.
• Exemption of PA from the possibility to file requests for
(misdemeanour and criminal) proceedings. Although this has
not been the practice of the Performance Audit Department
so far, such a legislative change would provide a positive
signal to government stakeholders that PA is conceptually
different from financial and compliance audits;
• In essence, an overall clearer distinction between PA and
other types of audit.295
7.5.3 Recommendations pertaining to the fourth modality of

The utilisation of audit reports by the public administration bodies
in policy development is exceptional and rare. In the case of PA, we
cannot even speak about such practice yet, given that only two reports
have been published so far. In order to incorporate PA findings into
293
Currently, from the perspective of making synergies between performance audit and policy
evaluation, staff recruitment is narrowly focused on experienced individuals in the field of
(traditional) audit through the state exam (Article 28 of the Law on State Audit Institution).
294
A rticle 40 of the Law on State Audit Institution, prescribes that SAI can require response
reports from auditees in the period between 30 and 90 days, which may diminish the
possibilities for implementing more demanding performance audit recommendations.
295
A rticle 2 of the Law on State Audit Institution presently defines than clearly distinguishes
between audit types.
141
policymaking in a meaningful manner, new solutions are needed in the

evaluation domain.
Short-term recommendation to government:

 Draft legislation on the planning system in Serbia should contain
a clear provision indicating policy evaluation as the integral part of
the policy cycle. Consultation with the Ministry of Finance, as well
as with the State Audit Institution, is deemed valuable in order to
adapt the newly introduced mechanisms to the important existing
processes, such as performance audit and programme-based
budgeting;
 The bylaw on methodology for policy management, impact
assessment and content of policy papers (which includes ex-post
evaluation) should explicitly state at what point of the process, and in
what way do the PA reports need to be addressed, provided that PA
has been conducted in a respective policy area. More specifically, in
conducting policy impact assessments, it should be made obligatory
to check and make notes on whether SAI recommendations have
been implemented, provided that PA has been conducted for the
policy domain in question;
 Public administration bodies should be obliged to disseminate
evaluation reports to the Performance Audit Department of the SAI
of Serbia, so the auditors could familiarise themselves with the
ongoing evaluation work and the results achieved so far. More
precisely, public sector units responsible for the coordination of
evaluation should disseminate these reports to SAI once they are
completed and submitted. This should be stipulated by the bylaw
which regulates the methodology for policy management, impact
assessment and content of policy papers that is currently being
drafted, or by the legal act introducing a binding ex-post policy
evaluation requirement once it is adopted by the government.
Mid-term recommendation to government:

 In the mid-term, policy evaluation needs to be systemically
developed, which would include institutional changes at the level
of each ministry, as well as a set of procedures and processes for
conducting policy evaluation. There is a need for an institution which
would be in charge of steering and monitoring the whole system, as
well as coordinating, especially when the policy evaluation system
is newly introduced. The Public Policy Secretariat is seen as an
institution at the centre of the government, which would naturally
initiate and later on coordinate the evaluation structure at the

142
systemic level and perform quality assurance of the system.

Therefore:
• A government decision on establishing a system-wide ex-
post policy evaluation requirement should be made and
materialised;
• In the process of drafting policy evaluation requirements,
necessary consultations and coordination between all
relevant institutions should be made, i.e. ministries in charge
of public administration, the ministry in charge of finance,
and other institutions at the central level of importance such
as the SEIO and the PPS;
• PPS should be assigned the leading role in the evaluation
structure owing to the competences of this institution in
facilitating policy making and coordination at the central
level, as well as connecting strategic planning and budget
programming. Competences should include assistance to
public administration bodies (especially in the formative years
of evaluation), monitoring and reporting to the government;
• More detailed elaboration on the evaluation system and its
necessary elements should be made through the adoption
of a regulation by the government, prepared through the
assistance of the PPS. This secondary act would be a backbone
of the evaluation system which derives from a more general
requirement prescribed in the law, distinguishing between
evaluations types and tiers in more detail. This piece of
legislation would clearly state the duties and competences of
PPS, as well as those of the units/organisations conducting
evaluation;
• Government guidelines for evaluation should be adopted,
defining the main mandatory features of the evaluation
process. Moreover, in drafting the guidelines, it would be
useful to include professional opinions of the SAI Performance
Audit Department and, consequently, start creating synergies.
The monitoring of implementation of these guidelines should
be delegated to the PPS, with responsibility for the overall
policymaking reform, which could in turn inform not only
the government, but also the SAI on the state of performance
in the public sector annually.
143
Bibliography
Books
Christopher, Pollitt and Bouckaert, Geert. Public Management Reform: A
Comparative Analysis - New Public Management, Governance, and the
Neo-Weberian State. 3rd edition. Oxford: Oxford University Press,
2011.
Cooksy, L. J., Mark, M. M. and Trochim, W.M.K. “Evaluation Policy and
Evaluation Practice: Where do we go from here?” In Evaluation policy
and evaluation practice. New Directions for Evaluation, edited by
Trochim, W.M.K., Mark, M. M. and Cooksy L. J. 2009.
Della Porta, Donatela, “Comparative Analysis: Case-Oriented versus Variable-
oriented Research.” In Approaches and Methodologies in the Social
Sciences. Cambridge: Cambridge University Press, 2008.
Devine, Fiona, “Qualitative methods.” In Theory and Methods in Political
Science, edited by D. Marsh and D. Stoker. New York: Palgrave
Macmillan, 2002.
Finlay, Jane and Debicki, Marek, editors. Delivering Public Services in Central
and Eastern Europe: Trends and Developments. NISPAcee Press, 2003.
Harrison, Lisa and Deicke, Wolfgang, “Conducting Interviews in Political
Research.” In Political Research – An Introduction. London: Routledge,
2001.
Howlett, M. and Ramesh, M, Studying Public Policy: Policy Cycles and Policy
Subsystems. Oxford: Oxford University Press, 1995.
Lazarevic, Milena, Maric, Sena and Orza, Amanda. Policy Making and EU
Accession Negotiations – Getting Results for Serbia. Belgrade: Deutsche
Gesellschaft, fur Internationale Zusammenarbeit (GIZ) GmBH, 2013.
Lonsdale, Jeremy, Wilkins, Peter and Ling, Tom. Performance Auditing:
Contributing to Accountability in Democratic Government. Cheltenham:
Edward Elgar Publishing, 2011.
Maric, Sena, Zarkovic-Rakic, Jelena, Aleksic Miric, Ana and Lazarevic, Milena.
How to get Results in Public Policies: Monitoring and Evaluation with
the Evidence Supplied by the Civil Society, Belgrade: Foundation for the
Advancement of Economics, European Policy Centre, 2014.
Moyser, George and Wagstaffe, Margaret, “Studying elites: theoretical and
methodological issues.” In Research methods for elite studies, edited
by Wagstaffe, G. M. A. M. London: Allen & Unwin, 1987.
Kraan Andrea and van Adrichen, Helenne, “The Netherlands Court of Audit
and Meta-Research: Principles and Practice.” In Quality Matters:
Bibliography
144
Seeking Confidence in Evaluating, Auditing and Performance Reporting,

edited by Robert Schwartz and John Mayne, Comparative Policy
Evaluation, Volume XI. New Brunswick, New Jersey: Transaction
Publishers, 2005.
Pollitt, Christopher, Girre, Xavier, Lonsdale, Jeremy, Mul, Robert, Summa,
Hilka and Waerness, Marit. Performance or Compliance? Performance
Audit and Public Management in Five Countries. Oxford: Oxford
University Press, 1999.
Rabrenovic, Aleksandra, Financial Accountability as a Condition for EU
Membership. Belgrade: Comparative Law Institute, Belgrade, 2007.
Thomas, Margo and Bojicic-Dzelilovic, Vesna, editors. Public Policy Making in
the Western Balkans. Dordrecht: Springer, 2015.
Journal Articles
Arnfrid, Arthur, Rydland, Lars Tore and Amundsen, Kristin. “The User
Perspective in Performance Auditing - A Case Study of Norway.”
American Journal of Evaluation 33.1 (2012): 44–59.
Berry Frances S., Turcotte, John W., Latham, Sarah C. “Program Evaluation
in State Legislatures: Professional Services Delivered in a Complex,
Competitive Environment.” New Directions for Evaluation 2002.95
(2002): 73-88.
Brooks, Roger A. “Blending two Cultures: State legislative Auditing and
Evaluation.” New Directions for Evaluation 1996.71 (1996): 15-28.
Bukkems, Gerard and de Groot, Hans. “Learning by auditing: mission
impossible?” Paper presented at the 5th biennial conference of the
European Evaluation Society, Sevilla, October 10-12 (2002).
Chelimsky, Eleanor. “Auditing and Evaluation: Whither the Relationship?”
New Directions for Evaluation 71 (1996): 61-67.
Chelimsky, Eleanor. “Comparing and Contrasting Auditing and Evaluation
Some Notes on Their Relationship.” Evaluation Review, 9.4 (1985):
483–503.
Clark, David. “The Changing Face of Audit and Evaluation in Government: A
Franco-British Comparison.” Public Policy and Administration 16.4
(2001): 20–33.
Davis, Dwight F. “What Do You Want a Performance Audit or a Program
Evaluation?” Public Administration Review 50.1 (1989): 35-41.
Divorski, Stan. “Differences in the Approaches of Auditors and Evaluators
to the Examination of Government Policies and Programs.” New
Directions for Program Evaluation 71 (1996): 7-14.
Drechsler, Wolfgang. “The Re-emergence of ‘Weberian’ Public Administration
after the Fall of New Public Management: The Central and Eastern
European Perspective.” Administrative Culture 06 (2005): 94-108.
145
Drechsler, Wolfgang, “The Rise and Demise of the New Public Management:
Lessons and Opportunities for South East Europe.” Uprava, letnik VII
(2009): 7-27.
Gunvalden, Jens A. and Karlsen Rita. “The Auditor as an Evaluator. How to
Remain an Influential Force in the Political Landscape.” Evaluation
5.4 (1999): 458-467.
Hart, Victor, dos Santos Danni, Luciano, Hebling Dutra, Paula, and Brazilian
Court of Audit (TCU). “Coordinated Audits as a Capacity Building
Strategy.” International Journal of Government Auditing (October
2013). Accessed September 4, 2015. http://www.intosaijournal.org/
highlights/audits_capacity_building_oct2013.html.
Hood, Christopher. “A Public Management for All Seasons?” Public
Administration 69.1 (1991): 3–19.
Kraan, Dirk-Jan, Wehner, Joachim and Kirsten Richter. “Budgeting in Estonia.”
OECD Journal on Budgeting 8.2 (2008): 1-40.
Leeuw, L. Frans. “Auditing and evaluation: bridging a gap, worlds to meet?”
New Directions for Evaluation 71 (1996): 51-61.
Leeuw, L. Frans. “Performance Auditing and Policy Evaluation: Discussing
Similarities and Dissimilarities.” Canadian Journal of Program
Evaluation 7.1 (1992): 51-68.
Leeuw, L. Frans. “Performance Auditing, New Public Management and
Performance Improvement: Questions and Answers.” Accounting,
Auditing & Accountability Journal 9.2 (1996): 92–102.
Leeuw, Frans L. “Evaluation Policy in the Netherlands.” New Directions for
Evaluation 123 (2009): 87-102.
Lonsdale, Jeremy. “Balancing Independence and Responsiveness. A
Practitioner Perspective on the Relationships Shaping Performance
Audit.” Evaluation 14.2 (2008): 227-248.
Maksimovska-Veljanovski, Aleksandra and Stojkov, Aleksandra.
“Performance-Based Budgeting in South-Eastern Europe: A Legal
and Economic Perspectives.” Central European Journal of Public
Policy, 8.1 (2014): 50-77.
Mayne, John. “Audit and Evaluation in Public Management: Challenges,
Reforms, and Different Roles.” Canadian Journal of Program Evaluation
21.1 (2006): 11-45.
Pollitt, Christopher. “Performance Audit in Western Europe: Trends and
Choices.” Critical Perspectives on Accounting 14.1 (2003): 157–70.
Pollitt, Christopher. “Performance Information for Democracy - The Missing

Link?” Evaluation 12.1 (2006): 38–55.
Pollitt, Christopher and Summa, Hillka. “Performance Audit and Evaluation:
Similar Tools, Different Relationships.” New Directions for Evaluation
71 (Fall 1996): 29-50.
Bibliography
146
Pollitt, Christopher and Summa, Hilkka. “Reflexive Watchdogs? How Supreme

Audit Institutions Account For Themselves.” Public Administration 75
(1997): 313-336.
Randma-Liiv, Tiina. “New Public Management versus the Neo-Weberian
State in Central and Eastern Europe.” The NISPAcee Journal of Public
Administration and Policy 1.2 (2008): 69-81.
Raudla, Ringa. “Pitfalls of Contracting for Policy Advice: Preparing
Performance Budgeting Reform in Estonia.” Governance: An
International Journal of Policy, Administration, and Institutions 26. 4
(October 2013): 605–629.
Reichborn-Kjennerud, Kristin. “Performance Audit and the Importance of the
Public Debate.” Evaluation 20.3 (2014): 368-385.
Richard Mulgan, “Policy versus Administration: The Auditor General’s
Clash with Finance and the Minister.” Canberra Bulletin of Public
Administration 101 (2001): 38-40.
Rouse, Paul and Putterill, Martin.” An integral framework for performance
measurement.” Management Decision 41.8 (2003): 791-805.
Schoch, Mickie and den Broeder, Corina. “Linking Information on Policy
Effectiveness and Efficiency to Budget Decisions in the Netherlands.”
OECD Journal on Budgeting 12.3 (2013): 1-22.
Scott D., Johnson. “From the Editor - Will Our Research Hold up Under
Scrutiny?” Journal of Industrial Teacher Education 32.3 (1995): 3-6.
Tansey, Oisín. “Process Tracing and Elite Interviewing: A Case for Non-
Probability Sampling.” PS: Political Science & Politics (2007): 765-772.
Van der Knaap, Peter. “Making Performance Audits More Responsive.”
International Journal of Government Auditing (2012). Accessed
September 5, 2015. http://www.highbeam.com/doc/1P3-2646190671.
html.
Van der Knaap, Peter. “Sense and Complexity: Initiatives in Responsive
Performance Audits.” Evaluation 17.4 (2011): 351–363.
Legislation
Constitution of Montenegro, “Official Gazette of Montenegro” 01/07.
Decision on the Methodology of Preparation and Content of the Programme
Budget, “Official Gazette of Montenegro” 38/2008.
Information on the Implementation of the Code of Criminal Procedure.
The Ministry of Justice of Montenegro, December 2012. Accessed
September 2, 2015. http://www.gov.me/files/1230044941.doc.
Information on the Law on Civil Servants and Employees Implementation.
June 2015. Accessed September 1, 2015. http://bit.ly/1krpTrb.
Instructions on Compiling Reports on Regulatory Impact, “Official Gazette of
Montenegro”, no. 09/2012.
Instruction on Methodology of Conducting Financial and Compliance Audit,
“Official Gazette of Montenegro” no. 07/15.
147
Instruction on Methodology for Conducting Performance Audit, Montenegro,

January 2015.
Law on Ministries, “Official Gazette of the Republic of Serbia” no. 44/2014,
14/2015 and 54/2015.
Law on State Audit Institution, “Official Gazette of the Republic of Montenegro”
no. 28/04, no. 27/06, no. 78/06, “Official Gazette of Montenegro” no.
17/07, 73/10, 40/11, No. 31/14.
Law on State Audit Institution, “Official Gazette of the Republic of Serbia”, no.
101/2005, 54/2007, 36/2010.
Law on State Administration, “Official Gazette of Republic of Montenegro” no.
38/03 of 27.06.2003, 22/08, 42/11.
Law on State Administration, “Official Gazette of the Republic of Serbia” no.
79/2005, 101/2007, 95/2010 and 99/2014.
Methodology for Evaluating the Implementation of Legislation, Ministry of
Information Society and Administration, Republic of Macedonia,
March 2013. Accessed August 30, 2015.
Manual for the Implementation of the Methodology, Ministry of Information
Society and Administration, Republic of Macedonia, March 2013.
Accessed August 30, 2015.
Public Administration Reform Strategy, “Official Gazette of the Republic of
Serbia” no. 9/14, 42/14.
Regulation on Using Vehicles in Official Use, “Official Gazette of the Republic
of Serbia” no. 49/14.
Rulebook for the Programme of Passing the State Auditor’s Exam. “Official
Gazette of Montenegro” 44/06.
State Audit Law, “Official Gazette of the Republic of Macedonia” no. 66/2010.
Documents
Algemene Rekenkamer, Effectiviteitsonderzoek bij de rijksoverheid, the
Netherlands Court of Audit (2012). http://www.rekenkamer.
nl/Publicaties/Onderzoeksrapporten/Introducties/2012/05/
Effectiviteitsonderzoek_bij_de_rijksoverheid.
Apelblat, Mose. “The Status of Performance Auditing in the Enlargement
Countries – Results of Findings in Questionnaire to the Supreme Audit
Institutions.” European Commission, DG ELARG, draft discussion
note (2013).
Barrett, Pat. “Evaluation and Performance Auditing: Sharing the Common
Ground.” Australian National Audit Office (2001).
Bundesrechnunghof, Peer Review Concerning the Macedonian State Audit
Office Final Report, Gz.: I 5 – 2007 – 1009, Bonn, (2007).
“Budgeting the Cost of Reforms – Programme Budget for Police and
Prosecution,” Institut Alternativa, Podgorica, (2014). http://media.
institut-alternativa.org/2014/12/ProgrammeskoBudzetiranje-
ENGL.pdf.
Bibliography
148
Daujotaite, Dalia and Macerinskiene, Irena. “Development of Performance

Audit in public Sector.” Paper presented at 5th International Scientific
Conference Business and Management, Vilnius (2008).
European Commission. Progress Report on Montenegro (2014). http://
ec.europa.eu/enlargement/pdf/key_documents/2014/20141008-
montenegro-progress-report_en.pdf.
European Commission. Screening report Montenegro, Chapter 22 – Regional
policy and Coordination of Structural Instruments (2013). http://
ec.europa.eu/enlargement/pdf/montenegro/screening_reports/
screening_report_montenegro_ch22.pdf.
European Commission. Screening report Montenegro, Chapter 32 – Financial
control (2013). http://ec.europa.eu/enlargement/pdf/montenegro/
screening_reports/screening_report_montenegro_ch32.pdf.
European Commission. Screening Report Serbia, Chapter 32 – Financial
control (2014). http://ec.europa.eu/enlargement/pdf/key_
documents/2014/140429-screening-report-chapter-32-serbia.pdf.
European Commission. The Western Balkan in Transitions. Occasional Papers
No. 46 (2009). http://ec.europa.eu/economy_finance/publications/
publication15155_en.pdf.
European Commission. Working Document Progress Report on the Former
Yugoslav Republic of Macedonia (2014). http://ec.europa.eu/
enlargement/pdf/key_documents/2014/20141008-the-former-
yugoslav-republic-of-macedonia-progress-report_en.pdf.
European Commission. Quality of Public Administration – A Toolbox
for Practitioners, Directorate-General for Employment, Social
Affairs and Inclusion,” (2015). http://ec.europa.eu/esf/main.
jsp?catId=3&langId=en&keywords=&langSel=&pubType=434.
European Court of Auditors. Activity Report (2014). http://www.eca.europa.
eu/Lists/ECADocuments/AAR_14/AAR_14_EN.pdf.
“Evaluation in Government.” UK National Audit Office. Accessed September
12, 2015. http://www.nao.org.uk/successful-commissioning/
monitoring-evaluation/evaluation/
Evaluation of Effectiveness and Efficiency of Development Assistance to the
Republic of Serbia. http://www.evropa.gov.rs/Documents/Home/
DACU/5/194/Final%20Evaluation%20Report.pdf.
Guidelines for Producing Programme-based Budget in Serbia (2014). http://
www.skgo.org/files/fck/File/2015/PB2016/Uputstvo%20za%20
izradu%20programskog%20budzeta.pdf.
International Organisation of Supreme Audit Institutions. ISSAI 300 -
Fundamental Principles of Performance Auditing. http://www.issai.
org/media/69911/issai-300-english.pdf.
International Organisation of Supreme Audit Institutions. ISSAI 3000 -
Standards and Guidelines for Performance Auditing based on
149
INTOSAI’s Auditing Standards and Practical Experience. http://

www.issai.org/media/13224/issai_3000_e.pdf.
Klaas, Klas. “Policy Making Review Montenegro”, SIGMA Papers, No. 51, OECD
Publishing (2014). http://dx.doi.org/10.1787/5jz15qwrt2vb-en.
Kuzek, Jody Zall and Rist, Ray, C. “Ten Steps to a Results-Based Monitoring
and Evaluation System.” The World Bank (2004).
Lazarevic, Milena and Obradovic, Marko. “Map of Policy Cycle at
Central Government Level in Serbia.” Project Reforming Policy
Coordination and the Centre of the Government – Third Phase,
EuropeAid11SER01/03/21 (2014).
LinkedIn discussion. “Performance Budget: Connecting Spot for
Performance Audit and Policy/Programme Evaluation,”
within the Group “Performance Based Budgeting for
Government”. https://www.linkedin.com/grp/post/4518607-
6041524285556695045?trk=groups-post-b-title.
McPhee, Ian. “Evaluation and Performance Audit: Close Cousins – or Distant
Relatives?” Australian National Audit Office (2006).
Ministry of Interior and Ministry of Social Affairs and Health of France.
Evaluation of policy towards newly arrived foreigners in France
(2013). http://www.modernisation.gouv.fr/sites/default/files/epp/
epp_primo-arrivants_rapport.pdf.
Ministry of Finance of Montenegro. National Budget Circular - Part D -
Instructions and Guidelines for Preparation of Non-Financial
Information for Budget Users (2014).
Nikolov, Marjan, Trenovski, Borce and Tasevska, Biljana. “Feasibility study:
Performance Based Budgeting for Enhanced Budgeting Process and
Increased Transparency and Accountability.” Center for Economic
Anaysis – CEA, Institute for Democracy Societas Civilis (2014).
OECD. Finland - Working Together To Sustain Success. Public Governance
Reviews (2010). http://unpan1.un.org/intradoc/groups/public/
documents/UN-DPADM/UNPAN042992.pdf.
OECD. The Call for Innovative and Open Government: An Overview of
Country’s Initiatives (2011).
OECD/SIGMA. Peer Review of the National Audit Institution of Montenegro
(2011). http://bit.ly/1hTP9nT.
OECD/SIGMA. The Principles of Public Administration (2014). http://www.
sigmaweb.org/publications/Principles-Public-Administration-
Nov2014.pdf.
Radej, Bojan “Osnove vrednotenja politik za občasne uporabnike Inštitut za
politike prostora,” Slovensko Društvo Evaluatorjev (2010). http://
www.sdeval.si/attachments/article/298/Osnove-vrednotenja%20
(XI%202010)).pdf.
Raudla, Ringa, Külli, Taro, Agu Cherlin and Douglas, James. “The Impact of
Performance Audit on Public Sector Organizations: The Case of
Bibliography
150
Estonia,” Paper presented at EGPA conference, 10-12 September,

Speyer, Germany (2014).
SIGMA Country Assessment Report. The Former Yugoslav Republic of
Macedonia (2013). http://bit.ly/1LojOkK.
SIGMA Country Assessment Report. The Former Yugoslav Republic of
Macedonia (2014). http://www.sigmaweb.org/publications/FYRoM-
Assessment-2014.pdf.
SIGMA Country Assessment Report. Montenegro (2013).
h t t p : // w w w . s i g m a w e b . o r g / p u b l i c a t i o n s d o c u m e n t s /
MontenegroAssessment_2013.pdf.
SIGMA Country Assessment Report. Montenegro (2014). http://www.
sigmaweb.org/publications/Montenegro-Assessment-2014.pdf.
SIGMA Country Assessment report. Serbia (2013). http://www.sigmaweb.
org/publicationsdocuments/SerbiaAssessment_2013.pdf.
SIGMA Overall Assessment Montenegro (2012). http://www.sigmaweb.org/
publications/Montenegro_Assess_2012.pdf.
SIGMA Public Administration Reform Assessment of Serbia (2014). http://
www.sigmaweb.org/publications/Serbia-Assessment-2014.pdf.
SIGMA Public Administration Reform Assessment of the Former Yugoslav
Republic of Macedonia (2014). http://www.sigmaweb.org/
publications/FYRoM-Assessment-2014.pdf.
SIGMA Public Administration Reform assessment of Montenegro
(2014). http://www.sigmaweb.org/publications/Montenegro-
Assessment-2014.pdf.
”State Administration Reform in Montenegro – Between Ambitious Plans and
Real Possibilities.” Institute Alternative. Podgorica (2012). http://
media.institut-alternativa.org/2013/03/state-administration-
reform-in-montenegro.pdf.
State Audit Institution of Montenegro. Annual report (2012). http://www.
dri.co.me/1/images/godisnji%20izvjestaj.pdf.
State Audit Institution of Montenegro. Annual Report (2013). http://www.
dri.co.me/1/doc/Godisnji%20izvjestaj%20DRI%20za%202013.pdf.
State Audit Institution of Montenegro. Annual Report (2014). http://bit.
ly/1MEkXFL.
State Audit Institution of Montenegro. Strategic Plan of Development 2012-
2017 (2012). http://www.dri.co.me/1/index.php?option=com_
wrapper&view=wrapper&Itemid=52&lang=en
State Audit Institution of Montenegro. Performance Audit Report on
Secretariat for Development Projects “Monitoring the Status and
Dynamics of Investment Projects“(2014). http://bit.ly/1NTZxuh.
State Audit Institution of Montenegro. Performance Audit Report on Efficacy
of the Tax Administration in Payment of Tax Debt (2015). http://bit.
ly/1W4fhdq.
151
State Audit Office of the Republic of Macedonia. Annual Report (2014). http://
www.dzr.mk/en/Uploads/SAO%20Annual%20Report%202014.pdf
State Audit Institution of Serbia. Performance Audit Report on the
Management of Public Vehicles (2014). http://www.dri.rs/images/
pdf/revizija2014/konacan%20izvestaj%2025%2006%202014%20
sajt.pdf.
State Audit Institution of Serbia. Performance Audit Report on the
Management of Public Property in Serbia (2015). http://www.dri.rs/
revizije/izvestaji-o-reviziji/izvestaji-o-reviziji.136.html?&offset=1.
Štimac, Vid and Lazarević, Mina. “Public Management Work Context and
Reform in Serbia, Results from a Large Scale Survey of Senior Public
Executives.” Belgrade Fund for Political Excellence, The Social
Inclusion and Poverty Reduction Team of the Government of the
Republic of Serbia (2013).
Shand, David. “Performance Auditing and Performance Budgeting,” in
Performance Budgeting: Linking Funding and Results, ed. Marc
Robinson, International Monetary Fund (2007).
The Magenta Book: Guidance for Evaluation. London: HM Treasury (2011).
The National Academy for Finance and Economics. Ministry of Finance of the
Netherlands. “Public Finance in the Netherlands” (2013).
Western Balkans Regional R&D Strategy for Innovation. Overview of the
Research and Innovation Sector in the Western Balkans. World
Bank Technical Assistance Project (P123211) (2013). http://bit.
ly/1LLppGA.
Internet pages
Court of Audit, France: https://www.ccomptes.fr/.
Court of Audit, Slovenia: http://www.rs-rs.si.
Findicator: http://vnk.fi/en/findicator.
Index of Economic Freedom, 2015: http://www.heritage.org/index/
explore?view=by-variables.
Intersectoral Development Assistance Coordination Network, Republic of
Serbia: http://www.evropa.gov.rs/Evropa/PublicSite/Documents.
aspx.
International Organization of Supreme Audit Institutions. http://www.
intosai.org.
Ministry of Information Society and Administration, Republic of Macedonia:
http://mioa.gov.mk/?q=frontpage.
Ministry of Education, Republic of Montenegro: http://www.mpin.gov.me.
National Audit Office of Estonia: http://www.riigikontroll.ee/Avaleht/
tabid/36/language/en-US/Default.aspx.
National Audit Office of Finland: https://www.vtv.fi/en
National Audit Office of the Republic of Macedonia: http://www.dzr.mk/en.
Bibliography
152
National Audit Office of the UK: http://www. https://www.vtv.fi/ennao.org.

uk.
Serbian Citation Index: http://scindeks.ceon.rs/?lang=en.
Social Inclusion and Poverty Reduction Unit, Republic of Serbia: http://
socijalnoukljucivanje.gov.rs/en/category/documents/.
State Audit Institution of Montenegro: http://www.dri.co.me/1/index.
php?lang=en.
State Audit Institution of Serbia: http://dri.rs/home.4.html
Think for Europe Network (TEN): http://ten.europeanpolicy.org.
News Article
“Krivokapić: Parliament is not stopping Government’s projects.” Vijesti,
October 4, 2015. Accessed October 10, 2015. http://www.vijesti.
me/v ijest i/ k r ivok apic-net acno-da-sk upst ina-koci-vladine-
projekte-854251.
“Ministry and the Tax Administration will implement SAI’s recommendations.”
Vijesti, June 16, 2015. Accessed September 1, 2015. http://www.
v ijest i.me/v ijest i/minist arst vo-f inansija-i-pu-ce-postovat i-
preporuke-dri-838424.
“Lakićević resigns, succumbing to criticisms and pressure.” Vijesti, September
19, 2015. Accessed September 26, 2015. http://www.vijesti.me/
vijesti/podlegao-pritiscima-i-kritikama-milan-lakicevic-podnio-
ostavku-852011.
153
Annexes
Annexe 1 – Methodology
This research relied on a qualitative methodology, i.e. thematic analysis
of documents and interview data. The researchers in the three country
teams had employed two parallel research strategies depending on the
case studies concerned. The three case studies of the Western Balkans
were subject to comprehensive explorative research of the processes
of performance audit and policy evaluation, as well as the link between
the two. The EU countries studied were analysed for lesson drawing
purposes with a more targeted approach and with a limited scope.
This annex briefly outlines the methods of data collection and analysis
employed in the study.
Data collection
Desk research
For the conceptual approach policy and academic literature on
performance audit and policy evaluation, as well as on the link
between the two processes, was consulted. A common reference list of
key articles was compiled and shared among the researchers from the
three teams. For the purposes of data collection in the three Western
Balkans case studies, each country team relied on a common set of
sources of data collection. These included:
• Relevant legislation;
• Strategic documents on performance audit/
policy evaluation in the specific countries;
• Performance audit reports;
• Reports on policy evaluation;
• EU guidelines and recommendations on
performance audit and policy evaluation;
• EU progress reports;
• SIGMA reports on Public Expenditure Management
and Control (external audit) and Policy Making
and Co-ordination (policy evaluation).
Annexes
154
A similar set of documents (albeit less extensive) was consulted in the

case of the EU member states studied.
Field work - semi structured interviews

In order to supplement the findings from the desk research, we have
conducted 54 semi-structured interviews with stakeholders in all of
the countries studied (list of interviewed institutions in Annexe 3).
In the Western Balkan case studies, the interviews were conducted
face to face, whereas in the EU countries the researchers interviewed
stakeholders via Skype. The interviews in all countries were conducted
using the same interview guide, which was developed in cooperation
with the three research teams (see below). The research guides were
slightly adapted and shortened for use in the EU countries.
In addition to supplementing the findings of the desk research, the
interviews were essential for this research since the understanding
of both processes studied among the interlocutors set the basis for
devising policy options and recommendations. Moreover, in the last
phase of the research, the interviews were also used for testing the
policy options of the study, i.e. checking for the level of acceptance
among key stakeholders. Our interviewees include:
• Staff from state audit institutions that have
engaged in performance audit;
• Staff from the institutions at the centre of government, in
charge of horizontal overview of the policy making process
(for example, Public Policy Secretariat in Serbia);
• Staff in policy analysis/evaluation departments;
• Staff in the institutions which were subject to audit;
• International staff that has acted as expert support through
projects to both SAIs and policy evaluation departments;
• External evaluators.
The interlocutors were selected from SAIs and ministerial departments
responsible for policy analysis and evaluation through a combination
of nonprobability purposeful sampling and snowballing technique.
The sample of the potential interlocutors is very small and hence
random sampling was not an option for this research. The focus
was not only on appointed officials, but also civil servants that hold
expertise relevant for the research. While there are limitations to
these sampling techniques, the researchers encompassed a variety of
stakeholders in the interviews and ensured that all relevant elements of
155
the performance systems in the three countries are represented in the

sample. The first interviewees were the state auditors/civil servants
in policy evaluation departments. This sample of interviewees has
provided us with enough background information (technical details),
assessment of both processes studied as well as the relationship
between them.
The interviews were subject to detailed note taking so as to ensure
consistency in the analysis. The notes were prepared in native
languages of the respective countries, as all of the researchers are fairly
familiar with each other’s languages for the purposes of consultation.
They were identified solely according to the position of the individual
relevant for the research. Access to the interviews was only available
to the research teams of the project.
Data analysis
The basic approach of this study is qualitative because “it seeks to
understand the experiences and practices of key informants and locate
them firmly in context”.296 In light of the research question and the
need for in-depth analysis of both cases, this study is based on a case
oriented research, as it seeks to understand complex units.297
The data collected from documents and interviews were subjected to
thematic framework analysis. On the basis of the data collected the
researchers set common themes which were used for analysis of the
specific case studies. In our analysis, we study the impact of the context
in which an organization operates, allowing more leeway in developing
policy proposals. At the same time, we use the distinction between the
formal (legal) and substantive aspects of the processes and policies
studied. The discordance between the two, i.e. a de jure and de facto
situation, has been a key theme in our analysis, not surprising because
of the novelty of many of the processes and policies studied.
The documents and the interview data have served different purposes
in the analysis. First, the primary documents (audit reports, evaluation
strategies etc.) were used to set the formal aspects of performance
audit and policy evaluation. They set the context and modalities of
performance audit and policy evaluation in the Western Balkans
countries, while providing potential lessons from the EU countries
studied. The document analysis also served as a baseline for studying
296
Fiona Devine, “Qualitative methods,” in Theory and Methods in Political Science, ed. D. Marsh
and D. Stoker, (New York: Palgrave Macmillan, 2002), 197-215.
297
Donatella della Porta, “Comparative analysis: case-oriented versus variable-oriented
research,” in Approaches and Methodologies in the Social Sciences, ed. D. Della Porta and M.
Keating (Cambridge: Cambridge University Press, 2008), 198.
Annexes
156
the positioning of both performance audit and policy evaluation within

the broader policy cycle of the case studies.
The data from the interviews has provided insight into the substantive
operation of the studied processes, as well as the link between
performance audit and policy evaluation in practice in all countries
studied in the research. In addition, the understanding and positions of
the interviewees in both the SAI and the state administration, were, in
fact, a key element of the analysis and crucial for our recommendations
and analysis of policy options. Hence, as a last step in this research,
the policy options were discussed with the stakeholders in order to
assess the likelihood of their acceptance. Reliability was ensured by
using data gathered both from secondary sources and from document
analysis and interviews.
157
Annexe 2 – Semi-structured questionnaire

General guiding questions for SAI staff
1. When and why was performance audit introduced in the country?
1.1. I s there a separate unit within the SAI conducting performance
audit?
1.2. How equipped is it technically and personnel-wise? Is this
sufficient?
1.3. A re there special training/qualification requirements in order
to work in performance audit?
2. How do you prepare the annual programmes for performance
audit?
3. Do audited organizations generally cooperate with performance
auditors?
4. What are the major challenges you have encountered in your work?
5. What have been the most common responses to the reports (public
response, institutional in terms of Parliament, audited institution
etc.)?
a. Are the recommendations from the reports implemented?
b. Are there mechanisms for monitoring implementation?
6. What kind of impact do you think performance audit has or had in
the country?
7. What measures do you think should be taken to improve
performance audit in the country?
8. Have you been through any training on policy evaluation in
general?
8.1. If yes, are you familiar with the concept of the policy cycle?
8.2. If no, show and explain graph. Where do you see your work
fitting in the policy cycle?
9. In your opinion, is there a link between your work and policy
evaluation?
a. If yes, elaborate.
b. If no, why?
10. What do you see as the key contributions and differences between
the two processes?
Annexes
158
11. Should there be a link between performance audit and policy

evaluation overall?
12. If yes, are there any ways in which you think the link between the
two can be improved?
13. Would you say that the performance auditing has led, or that it
could, lead to improved governance in the respective country?
Additional (optional):
14. Has the SAI received foreign assistance for the purposes of
introducing or developing the performance audit methodologies
and practice? Which EU member states have been involved in the
TA?
15. What usually happens after the performance audit report is
published in terms of responses from the audited institutions?
16. How would you assess the fit between performance audit and
the institutional and political culture in the country? (optional
depending on the interviewee)
General guiding questions for staff of evaluation departments

1. How long have you been doing policy evaluation?
2. What kind of preparation have you had for the purposes of policy
evaluation?
3. What are the primary guidance documents you use in the policy
evaluation?
4. What are the major challenges you have encountered while
conducting policy evaluation?
5. What would you say is the role of the policy evaluation reports in
practice and what should it be?
6. Have you ever used information/reports from the SAI in policy
evaluation and how?
7. What about specifically performance audit reports? If yes, how? If
no, why?
8. What did you think of these reports and was there something
missing in order to increase their use for policy evaluation?
9. How applicable are the recommendations from the performance
audit reports in practice, from the perspective of evaluating and
improving the audited policies or programmes?
159
10. Do you think performance audit is important for policy evaluation

and, if yes, how?
11. In your opinion, is there a link between your work and the
performance audits in the policy cycle and in practice in the
country?
a. If yes, elaborate.
b. If no, show and explain graph. Where do you see the work of
performance auditors fitting in the policy cycle?
12. What do you see as the key contributions and differences between
the two processes?
13. Should there be a link between performance audit and policy
evaluation overall?
14. If yes, are there any ways in which you think the link between the
two can be improved?
Additional (optional):
15. What sources of information or documents do you use when
conducting policy evaluation?
16. Is the department/office sufficiently equipped to organize, manage
and support evaluations?
17. What kind of trainings do you think you need for advancing your
capacity for evaluations?
Annexes
160
Annexe 3 – List of Interviewed institutions and

organisations per country
Estonia
• The National Audit Office (NAO)(Department for International
Relations, Department for Performance Audit)
• Ministry of Finance, Deputy Secretary
General for Public Governance Policy
• Ministry of Education and research
• Secretary of State - Government Office’s Head of Communication
Finland
• Prime Minister´s Office
• Ministry of Social Affairs and Health
(Administration and Planning Department)
• Ministry of Employment and the Economy (Employment
and Entrepreneurship Department)
• National Audit Office (NAO) (Performance Audit
and Fiscal Policy Audit Department)
France
• The Court of Audit
• The General Secretariat for Public Policy Modernisation (SGMAP)
Macedonia
• Agency of Realisation of Minority Rights
• External evaluators
• Ministry for Agriculture, Forestry and Water Supply
• Ministry for Information Society and Administration (MIOA)
• State Audit Office (SAO)
Montenegro
• State Audit Institution (SAI) (Senate,
Performance Audit Department)
• Audit Authority (for the Audit of EU funds)
161
• Ministry of Finance (Central Harmonisation

Unit, Budget Directorate)
• Ministry of Foreign Affairs and EU Integration
• Ministry of Interior
Serbia
• Audit Authority Office of EU Funds
• Ministry of Health (former employee on
the project within the Ministry)
• Ministry of Labour, Employment, Veterans and Social Affairs
• Public Policy Secretariat (PPS)
• Serbian European Integration Office (SEIO)
• Social Inclusion and Poverty Reduction
Unit of the Government (SIPRU)
• State Audit Institution of Serbia (SAI)
• Resident Adviser, EU Twinning Project to SAI
• USAID Business Enabling Project
The Netherlands
• Ministry of Economic Affairs
• Ministry of Finance
• The Netherlands Court of Audit (NCA)
Annexes
162

Policy Study Interactive

Diunggah oleh

Informasi Dokumen

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Policy Study Interactive

Diunggah oleh

Hak Cipta:

Format Tersedia

Performance Audit

and Policy Evaluation

Milena Lazarević, Miloš Đinđić,

About the Project and Partners

About the Project

About the Partners

About the Project and Partners

About the Project and Partners 3

5 Interaction Between Performance Audit

AWPG Annual Work Programme of the Government

PIA Policy Impact Assessments

About the Study

which are the forerunners in the Western Balkan enlargement process.

Performance Audit and Policy Evaluation -

legalistic administrative cultures, which influence supreme audit

Modalities and Options for Western Balkan Countries:

Concluding Remarks and General Recommendations

Recommendations to SAIs and governments:

ÊÊ It is recommended to transfer evaluation-related competence

1 Introduction: The EU Drive to

1.1 Background: New Public Management and the

Macedonia, Montenegro and Serbia, as the case studies of this research,

1.2 Purpose and Method of the Policy Study

1 Introduction: The EU Drive to Performance

impact and sustainability of policies and programmes.”10 From an

influenced by the consideration of the need to include countries from

1 Introduction: The EU Drive to Performance

The objective of the interviews was to focus on implementation in

operation of performance audit and policy evaluation in the conditions

1 Introduction: The EU Drive to Performance

options for implementation of the preferred modalities are

2 Context: What Are the Preconditions

Although the principal intention of this policy study is to focus on

allocation of resources (of which savings is one specific variant),

2.1 Administrative Culture

Namely, the administration and legal systems of the countries in

2.2 Macro-level Strategic and Policy Planning

- The linkage between policy and financial planning is less clear

2.3 Performance Budgeting

an absolute prerequisite).31 Regardless of the form performance

The introduction of programme budgeting in Montenegro appears

Similar to Macedonia and Montenegro, performance budgeting in

2.4 Policy Monitoring Practices

are being achieved (result based monitoring). Generally, without some

by the Serbian European Integration Office. However, no systematic

2.5 Openness and Reliability of Data in the WB

Similarly, for Serbia, systematic and regularly updated databases

Policy (or programme, in the US tradition) evaluation is a part of the

3 Policy Evaluation in the Western Balkans: A Nascent Concept

countries, the objects of evaluation are defined in the broadest

Illustration 1. Evaluation in the Policy Learning Stage of the Policy Cycle

• Definition of priorities - • Deciding on the

performance audit link, selected according to their relevance for WB

3.1 Policy Evaluation in Macedonia – the Challenge

Systemic ex-post evaluation of legislation: the starting point

3 Policy Evaluation in the Western Balkans: A Nascent Concept

is the promotion of “performance-based management.”62 In the course

and objective evaluation report. This specific evaluation produced

3 Policy Evaluation in the Western Balkans: A Nascent Concept

Linking ex-post evaluation to the ex-ante RIA

3.2 Policy Evaluation in Montenegro – the Missing Link

Ex ante assessments as the antecedent of evaluation

Interview with external evaluator, May 2015.

3 Policy Evaluation in the Western Balkans: A Nascent Concept

About the Project and Partners 3

5 Interaction Between Performance Audit

1 Introduction: The EU Drive to

2 Context: What Are the Preconditions