Synthesis in Policy Impact Evaluation

Abstract: Many of those involved in the impact evaluation of large-scale and multi-scope policy proposals have had
difficulties in aggregating detailed assessments of individual impacts into a summative evaluative conclusion. There
is a disagreement over assumptions about the aggregation which endorse different end results. The problem arises
from the fact that different policy impacts are not commensurable (i.e. in money) neither across social scales (micro-
meso-macro) nor across social scopes (economic-social-natural) of evaluation. A new approach to synthesis is
proposed that takes incommensurability of policy impacts into account. For this reason detailed impact observations
are organised into an input-output matrix of evaluation scopes from where correlation between unintended (or
secondary) impacts plays a central role in synthesis. The aggregation problem is studied by the comparative
evaluation of the sustainability of development program for Pomurje region with three methods: micro (no
aggregation), macro (full aggregation) and meso (partial aggregation) approach. Only the latter is found unbiased.
Meso approach displays interesting characteristics, which are discussed in the conclusions.
Keywords: Incommensurability, scope, scale, meso, matrix, impact evaluation.
JEL Code: A13, A10, F12, A19, C50, H40

1. Introduction
To evaluate policy impacts is to make a value judgment of the policy proposal by collecting
evidence and systematic assessment of its “worth or merit” (Scriven, 1994) to determine if
evaluation standards or criteria have been met. Despite widespread impact evaluation of policy
proposals, such as programs, public budgets or legislations, governments have been experiencing
systemic failures in managing complex social issues (Bar-Yam, 2003). The difficulty is that with
the increasing complexity of society, it is increasingly hard to provide for substantively diverse
aspirations in public sphere. Inability of a government to assess its proposals’ indiscriminately
results in unsuccessful application of substantively diverse aspirations in public sphere.
Evaluation of social complexity involves judgment which is based on information that is
obtained from many sources, through many means, as well as the use of multiple criteria. There
are essentially diverse viewpoints which value a given policy document’s “worth and merit”
because that they apply different scales of evaluation (from micro, meso, or macro perspective)
and also make judgement from different scopes of evaluation, such as from environmental,
economic or social aspect. In the face of a complex configuration of the social reality,
qualitatively different issues are equally important (incommensurability in scope), but also one
and the same issue can be perceived differently, either in local specifics or in its global entirety
(incommensurability in scale).
This heterogeneous corpus of evaluation information is rendered sensible through the process of
synthesis (Encyclopedia of Evaluation, 2004). However, as the observed policy impacts are not
commensurable between each other, they can not be reduced to a common denominator
(Funtowicz, Ravetz, in Martinez-Alier et al., 1998) so their synthesis is not simply putting a
puzzle together to create a whole. Social incommensurability (Munda, 2004) implies that
different principles of legitimacy and social primacy must be reckoned with and reconciled
(Wacquant, 1997) in evaluation – and so different »numeraires« or measures need to be applied
in their appraisal. This methodologically complicates currently prevailing approaches to
evaluation of worth and merit of policy proposals.
Social incommensurability is accompanied with value comparisons in the public domain as well
as with formal systems and knowledge structures. The latter decisively differ in the propositions
they take as their axioms (Kuhn, 1970; Feyerabend, 1975). Kuhn asserts that sciences organize
and integrate information in different ways. Different theories weigh the appearances of the same
world differently and different types of models are best suited for representing the judgment
processes of a given complex social issue. This induces disagreements concerning the

1
appropriate way to sum up what we know about reality. When confronted with multidisciplinary
issues even competent, honest and disinterested scientists may arrive at different problem
framing and conclusions because of systematic differences in the way they summarize available
information (Mumpower et al., 1996). With the incommensurable theories, no objective basis
exists for rational choice between opposing facts, and as a result no neutral observation of social
reality is possible. When values are irreducibly plural, value conflicts are ‘un-decidable’
(Mouffe, 2000 in Trainor, 2008) i.e. there can be ‘no rational resolution of the conflict’, no clear
cut judgment about “merit and worth” of policy proposals.
Even though social incommensurability is a determining structural characteristic of complex
social systems, this concept is strange to contemporary evaluation research for the reason that we
often think of two aspects of incommensurability – in scale and in scope – as coupled because of
the most common ways we encounter them. Bar-Yam gives an example (2004): “Consider
observing a system through a camera that has a zoom lens. For a fixed aperture camera, the use
of a zoom couples scope and resolution in the image it provides. As we zoom in on the image we
see a smaller part of the world at a progressively greater resolution. This leads to a particular
relationship of observations of parts and wholes, suggesting that when observing details of the
system, the whole is not being observed. (…We) must allow a decoupling of scope and
resolution, so that the system as a whole can be considered at differing resolutions as well as part
by part. For this purpose, scale can be considered as related to the focus of a camera—a blurry
image is a larger scale image—whereas scope is related to the aperture size and choice of
direction of observation.” What one sees is always predefined in scope and in scale of
observation and so these two need to be treated separately.
One of immediate consequences of this finding is the need to look at the foundations of the
aggregation methodology (Scriven, 1994) in impacts evaluation of large-scale and multi-scope
(LS-MS) policy proposals. The lack of explicit justification of the aggregation procedure is the
Achilles heel of the evaluation effort (Scriven, 1994). A useful example of the aggregation
problem in social research is brought forward with the theory of intersectionality in sociology
(Kimberlé, 1994; Hill Collins, 1993) which is studying patterns of gender inequality.
Intersectional theory examines how various socially and culturally constructed categories of
discrimination which are incommensurable, such as gender, race, and class, interact on multiple
areas, contributing to systematic inequality that may be hidden on the surface. Intersectionality
constitutes a critical alternative to additive claims (commensurability) when studying complex
issue involving multiple threats. For instance, it dismisses the additive claim that black women
are twice as badly off than white women due to both sexism and racism. According to Prins
(2006), intersectionality emphasizes that »the complexity of processes of individual identification
and social inequality cannot be captured by such arithmetical frameworks«. In studying such
issues simple aggregation ignores essential distinctions between incommensurable categories of
research and evaluation.
Idea that public policy initiatives should be evaluated from the different aspects of social
incommensurability has been widely recognized (Leopold, 1971; Ekins, Medhurst, 2003;
Rotmans, 2002; Weaver, Rotmans, 2006; Munda, 2004) but poorly implemented in policy impact
evaluation – in particular in matrix based methodologies. Standard methods are designed for the
appraisal of assumingly homogeneous policy interventions with only rudimentary distinction
between scope and scale of evaluation (Elbers et al., 2007; Rotmans, 2002), while governments
intervene into social issues that are heterogeneous in scope and in scale. Policy-makers deal with
numerous sectoral policies as well as with a great number of goals – all of which may hinder,
support or reinforce each other (Wildavsky in Carlsson, 2000). Hence, a cumulative evaluation of
such complex situations is far from trivial (Veen, Otter, 2002).

2
Different aggregation procedures yield different evaluative conclusions. When evaluation results
are presented in a too disaggregated or too aggregated form, they in particular lack relevance for
the strategic considerations. Majority of standard approaches are found to avoid addressing
incommensurable oppositions in the evaluation of programs, such as EU’s strategic impact
assessment (2001/42/EC), Impact Assessment Guidelines (SEC(2005)791), the territorial impact
assessment (TIA; ESPON - 3.2, 2006), and ex-ante assessment of the contribution of the EU
structural funds to regional sustainability (GHK et al., 2002). Poor handling of
incommensurability may be well linked to really disappointing conclusions by the Impact
Assessment Board (2009), that up to 80% of impact assessments studies currently provided to the
European Commission supply the kind of information that does not inform top-level policy-
makers whether their global objectives can be met. Linked to this, there is an apparent paradigm
crisis in impact evaluation (Virtanen, Uusikylä, 2004; Hertin et al., 2007).
Crisis in policy evaluation is in part seen as a result of unresolved aggregation problem (Foster,
Potts, 2007). There is a disagreement over assumptions about the aggregation of impacts across
multiple evaluation scopes and scales. Older methodological approaches strictly decline
aggregation of the detailed assessment results. Luna Leopold et al. (1971) were the first to
address the issue in this way. They proposed a detailed expert-based assessment method at the
micro level from which synthesis of results and the macro aspect remain entirely absent. Beside,
Leopold’s assessment matrix is concerned with two scopes only – economic and nature – and
assesses only the possible impact or side effects of the former on the latter (second chapter).
Ekins and Medhurst (2003) recently proposed a macro evaluation approach in the complete
multi-scope perspective. They appropriately claim that majority of policy impacts are “normal” –
they conform to the system thresholds. Diverse policy impacts are thus comparable to a certain
degree at least until they stay within critical system thresholds. So Ekins and Medhurst propose a
vertically and horizontally aggregated version of the Leopold matrix, but with the impact scope
dimension expanded from two to four scopes (social, human, economic, natural; this approach is
named here as the Leopold-Ekins-Medhurst impact matrix or LEM); they further allow for
aggregation of assessed impacts for all evaluation criteria within each of four evaluation scopes
that are placed in columns of the LEM. So they aggregate policy impacts internally, within each
impact scope, but not between them. In this way they obtain four aggregated or composite
indicators of policy impact – one for each evaluation scope. Their work is an important step
towards cumulative evaluation methodology in the framework of social incommensurability
(second chapter). An analogous approach to LEM is now conventionally applied in various
standard impact assessment procedures, such as EU’s strategic impact assessment (2001/42/EC),
territorial impact assessment (ESPON – 3.2, 2006), and assessment of the contribution of EU
structural funds to sustainability of regional development (GHK et al., 2002).
Nevertheless, LEM’s summation approach is inappropriate in its second step, when it allows for
the summation of all policy measures’ impacts irrespective of their diverse policy scopes
(economic, social,… policy measures) on given impact or evaluation scope (economic, social, …
impact). The problem is that given policy’s impacts on different evaluation scopes are not
qualitatively the same – such as an impact of economic policy on economy is not of the same
quality as its impact on nature – so they are not homogenous (Rotmans, 2006). It is shown in this
paper that fragmented assessment results can be aggregated in LEM only by the given source and
scope of impact, i.e. partially, because policy impacts are not strongly but only partially or
weakly commensurable. This suggests reorganizing LEM into the square input-output matrix of
Leontief (1970) with equal number of rows as policy scopes and columns as evaluation scopes
(third chapter).
The methodological divergence between summation approaches, as well as the resulting disparity

3
in their end results is illustrated by a comparative evaluation of a practical example. The
development program for the Slovenian region Pomurje for 2007 – 2013 (RDPP; Radej, 2008) is
first evaluated with Leopold’, next with LEM’s and finally with input-output method. Pomurje is
the least advanced Slovenian region (at NUTS 2; with 6.6% of the national territory and 4.3% of
national GDP), bordering Croatia (south), Hungary (east) and Austria (north). This is a region
with a strong cultural and ecological identity – more than a third of its territory is protected
nature areas including also unique landscape along the River Mura. Its economic capital is weak
but improving since the mid-nineties. Social capital is very weak and further degrading. For half
a century, the region had been surrounded with cold war borders. Since the beginning of the
market transition in the early nineties, the region has found itself on the main European transport
corridor which increased its geo-strategic importance and exposed it to international flows of
people and goods. The accession of Slovenia to the EU also imposed a more restrictive border
regime between Pomurje and Croatia (EU candidate country), which previously, in Yugoslavia,
were traditionally close. Regional development lags have accumulated despite increased inflow
of resources earmarked for less advanced regions from the national or European budget in the
past two decades, because not enough emphasis was placed on genuine local needs. This
additionally burdened social capital, leading to continued depopulation, brain-drain, long-term
unemployment, prolonged health and social risks for vulnerable groups (the majority of the
population is officially classified as vulnerable).
Analysis of development indicators (1995-2006) suggests that policy choice has failed to address
critical regional trends and trade-offs appropriately. Because of systemic failures to cope with
regional complexity in scale and scope, policy might become one of the obstacles for regional
sustainability. This leads to the following working hypothesis: a large deal of inconsistency in
public management in general and specifically in impact evaluation is not caused by the
complexity of the public domain itself but by inappropriate dealing with this complexity. The
distinction between commensurable and incommensurable social events is crucial, but not
sufficient. A new approach to evaluation synthesis is proposed that is based on meso explanation
which distinguishes between weakly commensurable and weakly incommensurable policy
impacts. The paper concludes that evaluation of complex issues requires placing synthesis
concerns into the centre of methodological efforts (see Lipsey, 2009).

2. Standard approach to evaluation

The first generation methodologies for matrical impact assessment of large-scale and multi-scope
policy proposals started with the suggestion by geologist Luna Leopold et al. (1971). In their
approach, assessment of policy impacts involves only two evaluation scopes (economic policy
impacts on environmental assessment criteria) and only micro scale – assessing impacts of
specific economic measures on specific environmental assessment criteria. This approach is very
interesting because it goes beyond plain monitoring of policy performance – i.e. how economic
policy impacts economic assessment criteria – but evaluates side effects or secondary impacts of
economic policy on environmental conditions.
In the finest analytical manner, Leopold et al. aim to comprehend a large picture of the complex
policy issue with detailed description of its elementary ingredients. They listed the 100 most
important economic policy measures horizontally and 88 areas of environmental impact
vertically. This created a matrix with 8,800 cells – each further divided into four sections that
describe every impact by its size (large/medium/small), direction (positive/negative/neutral),
probability (high/low) and the amount of risk (critical or not). In this way, impacts are assessed
in sufficient detail to enable maximally informed decisions of elected politicians about the
impacts of assessed policy proposal.

4
Recognizing the incommensurability between economic and environmental scope of evaluation,
Leopold explicitly rejected the summation of multifarious impacts into a cumulative impact
indicator. They claim that detailed assessment results should be presented disaggregated, leaving
policy-makers with full responsibility for the evaluation synthesis and for drawing its policy
implications. For Leopold refusal of aggregation is essential as it draws demarcation line
between evaluator and policy-makers to protect the former from the value judgments and
political interference (Kunseler, 2007). This argument has been accepted as an evaluation
standard. Today, some major procedures follow similarly fragmented approach such as the EU’s
Impact Assessment Guidelines (SEC(2005)791) which is even more detailed; territorial impact
assessment (ESPON 3.2, 2006), the European Strategic Environmental Assessment Directive
(2001/42/EC), and several others (COM(2002)276; SEC(2004)1377).
The rejection of summation in evaluation and shifting this task to policy-makers is problematic.
Fragmented evaluation results make decisions more informed but not necessarily easier
(Diamond, 2005). Refusing to summarize is “letting the client down at exactly the moment they
need you most” (Scriven, 1994). It is exactly politicians’ failure as social aggregators that call for
policy evaluation in the first place – see Arrow’s impossibility theorem (1951). Fragmented
results that are left unrelated in their interpretation, fail to satisfy information needs at the
strategic level; they produce banal answers to multi-dimensional societal problems (Virtanen,
Uusikylä, 2004) and offer little value to decision-makers at strategic level. Without explanation
of how opposing views of a complex phenomenon work together, it is impossible to substantiate
evaluation findings and this leaves evaluation utterly exposed to manipulation. In complex
considerations, the whole emerges as the outcome of interactions between parts, so impacts
should not be studied in isolation, one by one. Evaluation which simply produces non-
overlapping information tends to underplay inherent system contradictions, legitimizing
disregard of stakeholders’ issues in policy-making (Stake, 2001).
It is logically impossible to derive strategic policy advice directly from fragmented assessment
results. So, the question is how the big picture of policy overall impacts can be obtained from
fragmented assessment results. This is illustrated on the case of impact assessment of Regional
Development Program for Pomurje 2007-2013 with Leopold’s approach which is modified for
our specific purpose so that it is extended to three evaluation scopes. Table 1 had been obtained
in a standard way. Group of experts convened a workshop (via internet) and applied Delphi
method to assess possible impacts of proposed 47 RDPP’s measures on a selected set of
assessment criteria. Experts assessed impacts against the scale with only three possible values:
positive, neutral or negative impact. Behind the estimation of the matrix there is a large deal of
methodological difficulties which are simply skipped in this paper because they would focus our
attention on the micro level, while the paper is concerned with multi-level evaluation.
Table 1
Results in Table 1 can be summarised following three lines of reasoning: (i) prevalence of
positive impacts suggest that a majority of the program measures will positively contribute to
regional development, which supports endorsement of the proposal; (ii) negative impacts focus
evaluator’s attention on the weakest elements of the proposal which ought to be improved or
abandoned; (iii) neutral impacts (0) are not problematic provided that they are not the result of
unresolved disagreements among involved experts about direction of an impact. These three lines
of reasoning would suggest evaluator to focus his/her attention on negative impacts of the
proposed program. However, this would not be appropriate. Positive (and sometimes neutral)
impacts should not be always accepted as unproblematic, nor negative impact as such can be
treated as disagreeable. Cumulative policy impact evaluation is far more twisted.
Prevalence of positive impacts is not sufficient evidence for a conclusion that policy is adequate

5
in the proposed form but it only indicates that it is prepared by basically competent public
authority. Policy proposals can not be treated as if they were outcome of bad fortune. In fact they
are carefully studied as well as painfully negotiated propositions among dissimilar group
interests usually much earlier then they are submitted to impact evaluation. It is only when
systematic evidence of positive impacts is obtained, evaluator can decide about the
appropriateness of the overall proposal. But “systematic evidence” can only be identified at
higher levels of evaluation when detailed results of assessment are the first properly aggregated.
There are some additional reasons why positive impacts shall not by themselves lead evaluators
to switch green light to policy proposal. Impacts are sometimes assessed against the criteria that
are selected by formally responsible implementation agencies themselves so their achievements
can not be automatically seen as »neutral« from a wider social aspect. Even when this is not the
case, impacts are assessed against individual criteria and thus in isolation from each other.
Successful realization of separate policy measures can not by itself guarantee positive social-
wide impact of the policy when its goals (or assessment criteria) are in conflict. This is exactly a
situation which is normal in public sector. So it is important to see that prevalence of positive
impacts in Leopold matrix can only inform policy-makers about their effectiveness in atomistic
perspective (micro view) while it does not enable, and more importantly, does not jet allow for a
systematic conclusion about the proposal’s impact on the overall society.
The imperative of aggregation of assessment results ignites two methodological concerns. They
are linked to two limitations of commensurability of impacts and these limitations define
conditions under which negative impacts may be tradable with certain positive impacts in
aggregation of detailed assessment results. The first relates to situations when different experts
cannot reach consensus on direction of impact (positive or negative). The Leopoldian tradition
requires that positive and negative impacts should be presented separately in the evaluation
report but this even further fragmentises evaluation results. Some other approaches such as CAF
(Common Assessment Framework; 2006) go further suggesting that oppositions between
experts’ assessments need to be discussed with the aim of reaching consensus among them about
direction and even about intensity of impact. However, forcing consensus for every single
assessment detail is risky because it could invoke asymmetries within the assessment team – such
as different negotiating power. Imperative of consensus exhibits a strong tendency to revert to a
kind of closed, exclusive process (Connelly, Richardson, 2004) where the dominant actor
prevails. It would suffice to invite experts to verify arguments for their disagreements and discuss
them – not to reach consensus but at least to reach “rational disagreement” (Sankey, 1995) and
confirm validity of arguments that stand behind them. If conflicting expert assessments are well
founded, disagreement between them is justified and irresolvable. So another option which is
also applied in case study, is simply netting-out the opposing assessments. Different expert
opinions are equally valid and only partial claims when observed from a wider regional
perspective, and so in the pursuit of emerging wider judgment, they may well be netted-out.
The second difficulty is related to aggregating positive and negative impacts of a given policy
measure on various assessment criteria. For instance in Table 1, may negative impact of
entrepreneurship promotion on employment in Pomurje be outweighed with positive impacts of
entrepreneurship promotion on migration? Or a more radical example: is it permitted to outweigh
additional tons of greenhouse pollution with additional purchase of tradable pollution permits? If
impacts were commensurable, a positive policy effect on one assessment criteria would outweigh
any other policy’s negative effect judged on any other criteria. But we know that greenhouse
emissions cause irreversible changes in the atmospheric conditions and that the economic and
climate aspects of welfare are incommensurable in scope. So, a trade-off between greenhouse
gases and money is not adequate as a general principle. However, one needs to reason in the
multi-scale perspective, where local judgments are different from the global one. Trade-offs

6
between income and greenhouse emissions are not accompanied by incommensurability in every
single case, or at least people and their communities are not willing to treat them as such.
To incorporate this peculiarity in public choice, system thresholds – such as ecological and social
standards – have emerged (for a survey of literature see Muradian, 2001). A policy proposal is
not allowed with any of its impact to cross system thresholds because this could endanger the
basic integrity of the social system. Thresholds simply account for the fact that there are
discontinuities in the measurement of value-based phenomena and value addition (Mason, 2006)
that comply with discontinuity in individual and social values.
The concept of system thresholds is closely linked to incommensurability. As Wiggins (1997)
explains, two values are incommensurable if there is no general way in which A and B trade-off
in the whole range of situations of choice and comparison in which they figure. But such
examples are rare in social context. Social phenomena are usually incommensurable ‘only’
beyond (or below or both) their threshold values. Within the safety limits an agent either does not
sense the difference between two qualitatively different social conditions or refuses to declare a
preference for one or the other (Luce, 1956 in Munda, 2006), as in the cases of minor
environmental damage that stays within ‘safe’ ecological standards. System thresholds thus
define a social space of normality where elements of the system freely interact so their
acceptance can only be resolved locally as it depends largely on specific considerations among
those directly concerned. Submission to system thresholds is obligatory for authors of policy
proposals so it would be unusual in evaluation to deal with basically illegal situations with high
probability of incidence of negative policy impacts involving critically high system risk. If this
danger is averted with the introduction of system thresholds then evaluation may be quite
simplified as it considerably alleviates the aggregation problem. Within the space of system
normality assessed negative impacts of different policies are by definition not critical so they
may be acceptable and therefore at least conditionally tradable and thus aggregatable.
Recognition of system thresholds allows methodological step forward in the cumulative impact
evaluation. One of the first macro-evaluation methods was strategic environmental assessment
(Sadler, Verheem, 1996; SEA Directive, 2001/42/EC), but it gives no instruction on how to
cumulate environmental impacts and parallel them in aggregate way to economic ones. The
missing link has been contributed by Ekins and Medhurst (2003) in their novel approach to
evaluation of the EU structural funds’ impact on regional sustainable development. They
proposed the “four capitals model” which evaluates in parallel economic, social, environmental
and human impacts. This particular four scope logic can be traced back to the Brundtland report
(WCED, 1987), and to the conference on sustainable development in Rio de Janeiro (UNCED,
1992; Munasinghe, 1992; Ekins, 1992).
Ekins and Medhurst have proposed a macro-assessment method that acknowledges
incommensurability of development in scope. They have proposed a highly compacted form of
the Leopold matrix. The columns are reduced from 88 fields of possible environmental impacts
to four scopes of (regional) sustainability, each covered by smaller number of evaluation criteria
(only two in Table 1; in Table 2 these are already summed up). For simplicity, the model is
reduced in this paper to a “three capitals model” because it is even in its reduced form entirely
sufficient to explain the aggregation problem in multi-scope perspective. The number of rows in
LEM matrix may be as large as in the case of Leopold matrix, but it is also reduced, as suggested
in Ekins and Medhurst, from the level of 47 program measures (Table 1) to the level of (six)
main regional policies – see Table 2. It presents RDPP’s impacts on much wider range of values
(compared to Table 1) from the most robust positive composite impact (+++) to the most
negative impact (---), with all other five intermediate possibilities. When in dilemma how to
round aggregate impact from Table 1 to Table 2 a decision was taken on comparison of the

7
financial weight of the related measures. At the bottom Table 2 presents aggregated impacts of
program on each of three evaluation scopes.
Table 2

Table 2 presents impacts of each sectoral policy involved in the RDPP on three scopes of
evaluation. Infrastructure development will be the most welfare enhancing policy, followed by
rural development policy. The most problematic is the negative impact of value added growth on
social welfare. The impact of health policy and tourism are disappointing because their measures
mostly relate to preparation of plans and regional organization structures. But the overall impacts
of the program which is presented in aggregate at the bottom row of Table 2 do not appear
excessively problematic. The program will improve regional sustainability in all three respects
even tough more in the economic aspect (++) than in social and nature ones (+). Summary
impacts of the program are positive; the differences between scopes are relatively small. In this
way evaluation concludes that the program will have a rather acceptable overall impact.
However, the main task of policy evaluation in macro perspective is not to examine how policy
makers achieve their narrowly planned goals. LEM can not say much about RDPP’s impact on
integrity of the regional development; it does not tell how the program affects the overall
development from the aspect of its internal incommensurable oppositions. In a complex setting,
evaluator also needs to judge obtained detailed assessment results in the perspective of conflicts
and synergies between its scopes. This is not possible in LEM because it constructs macro level
view mechanistically with simple accumulation of elementary information from micro level
where micro and macro are seen only as quantitatively (!) different perspectives. This kind of
reductionist over-simplification of multi-level structure of social complexes is entirely consistent
with the thesis about “micro-foundation of macro”, which is conventional in the mainstream neo-
Keynesian economics (Mankiw, Romer) and with older Samuelson’s neo-classical synthesis.
Even though Ekins and Medhurst have made an important summative step forward in macro
evaluation methodology, one needs to conclude that they essentially remained followers of
micro-based and reductionist evaluation philosophy put forward by Luna Leopold. This example
confirms again that judgment of worth and merit of policy proposals is far from trivial in
complex circumstances.

3. Complex approach to evaluation

If detailed assessment results are not cumulated, such as in Leopold's method, the assessment
produces findings that are too fragmented for evaluator of system-wide impact. In contrast, full
aggregation, such as in LEM, causes findings that are too amassed for the judgment about
policy’s inherent structural tensions. Both micro and macro type of assessment waste essential
information for evaluation of complex social issues. In this section we first discuss the reasons
for inconsistent aggregation in LEM and than “negotiate a compromise” between micro and
macro approaches. This intermediate solution goes beyond micro-macro dualism and even unites
both aspects of evaluation.
Ekins and Medhurst have overlooked that the assessed impacts are vertically not fully
aggregatable despite their (eventual) conformity to system thresholds. Column aggregation
assumes the homogeneity of the impacts of different policies on a given evaluation scope.
However, many studies demonstrate that given policy does not influence different areas of
impact in the same way (Schnellenbach, 2005). Policy impacts can be direct or indirect. They are
direct when they affect the target impact area, but indirect or unintended when they, usually
unpredictably, impact on areas which are not primarily targeted (Rotmans, 2006) but fall under
jurisdiction of other policies with different sectoral scopes (Rotmans, 2006). So policy impacts

8
are not neutral in scope! This is confirmed even for those policies that had previously been taken
as the most homogenous in impact such as monetary (Lucas, 1972) and tax policy (Leith,
Thadden, 2006). For instance, interest rates in the monetary policy are mechanism which is
directly more beneficial for the profit sector than for the non-profit one (Kennedy, 1995). In
principle, policy interventions should be always addressed in terms of their inadequacy due to
their sectoral specialization as it can be exposed against the general interest they are supposed to
serve (Donzelot, in Burchell et al., 1991). Yet unintended or secondary impacts routinely fade
out in policy impact evaluation because it is assumed that they are too complex and thus
impossible to track. Absence of secondary impacts in policy considerations might help us
understand why good individual policies, based on strong values and even common sense, often
lead to disappointing overall system results (Chapman, 2004).
Sectoral specialisation of public policies implies that distinction between incommensurable
impact scopes must be preserved in evaluation (Ostmann, 2006). Vertical summation in LEM is
only permitted partially within the given source of impact (and area – as seen earlier). For
example, economic and social policy’s impacts on the environment are not commensurable so
they are aggregated separately (economic impacts on nature separately from social impacts on
nature). This rule will be referred to henceforth as a partial aggregation rule.
This rule seems at odds with the strong version of incommensurability thesis. But Martinez-Alier
et al. (1998, in Stagl, 2007) point out that in situations when there is an irreducible value conflict,
we can only search for weak comparability as a facilitator of collective discourse. Some authors
have argued against the strong incommensurability thesis (Griffin, 1986 in Morgan, 2007; Nola,
Sankey, 2000). They proposed to make a distinction between relations of strong and weak
in/commensurability. Impact is said to be weakly commensurable when specific limitations are
imposed in summation procedure such as with the partial aggregation rule. Impacts that are
weakly commensurable in two or more incommensurable scopes of the evaluation, like socio-
economic impacts which take place when a given policy measure simultaneously produce weakly
commensurable impacts in economic as well as in social scope (“secondary” impact), are said to
be weakly incommensurable. These are evaluated against two otherwise incompatible set of
judgments so their nature is hybrid, which introduces a delicate possibility of translation between
them. This means that evaluator needs to distinguish between: (i) diverse primary impacts – due
to their incommensurability; (ii) between primary and secondary impacts – due to weak
commensurability of the latter; (iii) and between different sorts of secondary impacts – due to
their weak incommensurability.
Taking into account partial aggregation rule all policy measures in the Leopold matrix (rows in
Table 1) need to be first regrouped in the same way as impact areas (columns) – by their three
scopes. This divides the Leopold matrix into nine sub-sections. When weakly commensurable
impacts are partially aggregated within sections, cumulative impacts are obtained for each sub-
section. These can be presented in a square “input – output” or a Leontief's matrix (its “central
quadrant”). Leontief initially developed his matrix to facilitate inter-sectoral studies, because it is
suitable for an explicit presentation the tension between sectors, direct and indirect. For instance,
if we have agriculture, industry and services as three sectors in an economy, we can show in a
matrix, by rows and columns, how the sectors are directly linked. The agriculture requires some
capital goods and chemicals from the industry, and at the same time it supplies its product output
to the industrial and service sectors as their intermediate input. Additional agricultural needs for
industrial goods will indirectly induce also demands for services which are needed to enable
increased industrial production.
Square matrix exists hierarchically above the micro-level (Leopold matrix) because it is
aggregated from it. At the same time, as a partial aggregate, the matrix exists at a lower level

9
than the macro (LEM). The input-output matrix thus presents an intermediate or meso view of
the assessed program impacts. This approach to evaluation is therefore meso-matrical. Meso-
matrix is of central importance because the complex social system is built upon meso (Dopfer et
al, 2004) as it intersects two axes in definition of social complexity – horizontal or scope aspect
(E, S, and N in input-output matrix) with vertical or scale aspect (meso, micro and macro).
Square matrix is “a plane of inter-paradigmatic standards” (evaluation scopes; Kordig, 1973)
which “equally tries to tread the path between difference and sameness” (Allmendinger, 2002). It
displays “plural-relativistic” view (Geertz, in Renselle, 2007) of social complex, which means
that it covers many parallel views of one closed reality containing many (pre)existing substantial
contexts (scopes). Meso is “intersectional” as it possesses hybrid characteristics (Schenk, 2006)
of its intersecting dimensions (scopes and scales). With its plural characteristics meso is situated
in “the un-excluded middle” (Wallerstein, 2004) of the social complexity. It enables mid-level
articulation of constitutive oppositions that accompany public choice and it facilitates the
understanding of the basis of deep system oppositions (Mertens, 1999). It not only crosses
complex system divides but also intermediates between them (Knauft, 2006) and integrates them.
Evaluator would, for instance, intervene from meso perspective in conflicts to help actors with
different belief systems understand where their disagreements have epistemological and ethical
roots and help expose the meaning systems by which these facts are being interpreted (Bovens et
al., 2008). As plural, intersectional and non-excludive, meso is a perspective from where the
modelling of complexity is the most tractable “a priori” (Easterling, Kok, 2002).
A meso-matrix refocuses evaluator’s attention from the achievements of sectoral policies on
impact-driven changes in the relations among evaluation scopes - their system oppositions and
synergies. It shows policy proposal in the perspective of overlapping between its policy scopes
(inputs, in rows) and its impact or evaluation scopes (outputs, in columns). Intersection between
input and output scopes is denoted with the intersection sign ‘∩’ from the set theory. For
example, economic policy impact on the social scope of evaluation is denoted as E∩S (E
intersects S), while the impact of social policy on the economy is denoted as S∩E. Input-output
matrix of RDPP’s impacts is presented in Table 3.
Table 3
Table 3 is irrelevant and entirely useless for everyone who is only narrowly concerned with the
assessment of performance of his/her own actions and is not able or willing to question its
system-wide impact. Table 3 is therefore relevant only for the highest ranking policy-makers
who are concerned with overall policy consistency taking into account its wanted and unwanted
impacts indiscriminately. Table 3 thus produces a qualitatively different policy perspective which
is emergent, which means that its message can not be reduced back to the fragmented pieces of
performance assessment from which it is acquired. Table 2 does not produce qualitatively
different view compared to Table 1 since they both only present effectiveness of each policy
(measure) assessed against some uniform set of performance criteria.
To understand how qualitatively different perspective emerges in the meso-matrical view so to
say “out of nothing”, one need first to correlate ingredients of Table 3. A correlation is applied
when one studies connectedness (“overlaps”) between pairs of variables in causal models. In our
case “to correlate” means to link one directional overlaps between scopes, such as E∩N, with
their diagonally symmetric opposites, N∩E, in the aim to obtain a bi-directional or reciprocal
impact (NE as nature-economic overlap) that explains how two evaluation scopes work together.
The square matrix consists of two distinctive classes of relationship: (i) intra-scope or primary
impacts (E∩E, S∩S, etc) are located on the diagonal (top left to bottom right); (ii) and inter-
scope or secondary overlaps (E∩N etc) are located below or above this diagonal. The diagonal

10
elements provide narrow evaluation of primary effectiveness – how one particular policy scope
impacts “its own” evaluation scope. Diagonal elements are strongly incommensurable between
each other. So they can not be aggregated any further and are interpreted as they are. Regional
economic policy would be very successful in achieving its own primary goals with the
implementation of RDPP (three pluses); social policy would be moderately successful (two
pluses). Nature conservation would be only faintly effective (one plus) either because it is not
foreseeing clearly its main challenges or it pursues its primary aims inconsistently. Comparison
of diagonally placed impacts indicates that social, environmental and economic scopes of
regional development are not treated indiscriminately in RDPP. This observation does not match
with the previously obtained conclusion from summary row of Table 2 that suggested a broadly
balanced impact of the program on the three evaluation scopes.
Broader insight into RDPP’s internal consistency is obtained “under the surface” of detailed
assessment results with the correlation of non-diagonally located and weakly incommensurable
sub-aggregates. Correlation produces three correlates in Table 4: SE, for the ‘socio-economic’
intersection of impacts, denoting bi-directional overlap between S∩E and E∩S; NE and SN, as
the socio-natural correlate. The negative SE correlate is a result of a socially narrow-minded
economic growth policy. Low correlation in SN results from a socially narrow-minded nature
conservation policy. Here we recall the previous observation that nature protection policy will
not be very effective in pursuing its primary goals – so the proposed program will impose a
social burden at least in relative terms for only scanty aspirations in environmental sustainability.
This shows that a narrow minded economic and nature protection policy both take further
advantage of already very fragile regional social capital. Beside, the economic scope very poorly
integrates with S and N; similarly can be said about the feeble secondary impacts of N. This
information can not be extracted from Tables 1 and 2. Table 4 shows that in RDPP, S is involved
in interactions that are more beneficial for E and partly for N. Information about anti-social
character of RDPP is entirely absent from the surface view (Table 1 and 2).
Table 4
In Table 4 evaluative conclusion on program’s “merit and worth” is not derived solely from the
aggregation of impacts but also from the co-relation between sub-aggregates of weakly
commensurable impacts. The program exhibits anti-social character, which is highly problematic
taking into account the baseline conditions with S already deeply depressed in comparison with E
and N. The conclusion is that the RDPP is not contributing to regional sustainability. This
conclusion is just the opposite to the one obtained from Table 2.
Disagreements in conclusions derived from Tables 1, 2 and 4 are, of course, not due to different
detailed expert multi-criteria assessment at the micro level, which remains the same in all three
cases. It emerges solely from the summative endeavour of the evaluator. Policy failure to
implement more cohesive policies is therefore not necessarily a result of intentional bias, but can
be a consequence of inadequate strategic evaluation of options in the presence of deep value
oppositions between policy stakeholders resulting from social complexity in scale and scope.
Paper shows that different aggregation assumptions bring about different evaluative conclusions,
and inappropriate assumptions lead to misleading policy advice. This confirms the working
hypothesis that a large deal of difficulties arising from inconsistent impact evaluation is not
caused by the complexity of the public issues but at least sometimes by an inadequate approach
to social complexity, with flawed assumptions about the nature of (in)comparability of social
issues. In this way standard matrical evaluation methodologies serve only as sophisticated,
expensive and time consuming tools for a strategic deception of decision-makers.

4. Conclusions

11
This paper has explored how standard approaches in matrical impact evaluation comprehend
social complexity in scale and in scope and how they take it into account in formation of
evaluative conclusions about “worth and merit” of the policy proposal. The meso approach is
proposed as an alternative against both relativization of evaluation results (as in the Leopold) as
well as against their generalization beyond limited validity (as in the LEM). Leopold has
accurately differentiated environment and economy as two incommensurable scopes of
valuation; but he failed to see that the assessment is actually concerned with secondary impacts,
which are weakly commensurable, so they can be partially aggregated. Refusal of aggregation is
therefore inappropriate. Ekins and Medhurst have accurately observed that aggregation of
impacts within a given evaluation scope is formally correct. They provided a logically justified
procedure in cumulative evaluation and offered firmer ground for making system-wide
evaluative conclusions. But they failed to apply incommensurability consistently on the input
side of policy scopes. At the end they remain “micro-foundationalists” when they construct
macro perspective through simple aggregation of micro level observations instead of from
(correlation of) meso level sub-aggregates.
Paper demonstrates that social incommensurability is not an irresolvable obstacle to more
cohesive social research, but only “a safety mechanism” which reminds researcher that social
issues are complex in scale and in scope. But strong differentiations are important for a very
small number of social issues. Even though contemporary societies are built on incommensurable
oppositions which cause strong social fractures and invoke strong relationships of dis/agreement,
majority of links important for the reproduction of everyday social life are weak. The case
illustrated that the policy may be quite differently evaluated from the aspect of primary and
secondary effectiveness. This finding justifies our research concern.
Sectoral policies are biased but also equally important, and so there is no mechanism to install an
optimal public policy. In such a situation, a policy proposal that is the most secondary effective
ought to be chosen (compare with Demsetz, in Schnellenbach, 2005). So the yardstick by which
policy proposals are compared and selected must be their secondary impacts in weakly related
welfare concerns. As societies grow more complex, policy-makers should be increasingly aware
not only of their own agency’s primary objectives narrowly defined, but also of wider
implications and unwanted effects of their (in)activity. The same principle is relevant to the
thought of both Hayek and Popper, who take the view that the unintended consequences of action
are the principal concern of social science and that the existence of unintended consequences is a
precondition for the very possibility of the scientific understanding of a complex society
(Vernon, 1976).
Meso approach to evaluative synthesis is not simply a cognitive process of drawing general
lessons from local projects (Geels; 2007). Synthesis requires a careful framing of the research
problem which calls for a formalism that theoretically remains faithful to the framework of study
(Barroso et al., 2003, in Bondas, 2007). This is a crucial precondition if we wish synthesis to
become a tool for identifying, avoiding or tackling systemic contradictions and systemic
exclusions such as due to injustice (Pulido, in Hilding-Rydevik, Theodórsdóttir, 2004), gender
inequality, or unsustainability of development. Today majority of evaluation studies try to earn
their neutrality in the first evaluation step of scientifically objective assessment of individual
impacts. Paper sheds light on the cases when neutrality should be earned also in the synthesis of
evaluation results. This experiment not only confirms the need for the cumulative evaluation of
complex policy proposals, but places synthesis concerns, via neutrality considerations, into the
centre of the efforts pertaining to improvement of policy advice by policy impact evaluation.

Acknowledgements: I acknowledge valuable comments given to the previous version of the

12
paper by professors Mojca Golobič and Srečo Dragoš, both from the University of Ljubljana.

Bibliography
Allmendinger P. Towards a Post-Positivist Typology of Planning Theory. Sage, Planning Theory, 1/1(2002):77–99.
Arrow K. 1951. Social Choice and Individual Values, (2nd ed., 1963), New Haven: Yale University Press. 124 pp.
Bar-Yam Y. 2003. Multiscale Variety in Complex Systems. Cambridge, Mass.: NECSI - New England Complex
Systems Institute, Technical Report, 17 pp., http://necsi.org/projects/yaneer/multiscalevariety.pdf, [VII/08].
Bar-Yam Y. A Mathematical Theory of Strong Emergence Using Multiscale Variety. Wiley, Complexity, 9/6
(2004):15-24.
Bondas T., E.O.C. Hall. Challenges in Approaching Metasynthesis Research. Sage, Qualitative Health Research,
17/1(2007):113-21.
Bovens M., P. ’t Hart, S. Kuipers. 2008. The politics of policy evaluation. The Oxford Handbook of Public Policy,
Chapter 15, pp. 317-33.
Burchell G., C. Gordon, P. Miller (eds). 1991. The Foucault Effect: Studies in governmentality. Hertfordshire:
Harvester Wheatsheaf, 303 pp.
Carlsson L. Non-Hierarchical Evaluation of Policy. Sage, Evaluation, 6/2(2000):201–16
Chapman J. 2004. System failure - Why governments must learn to think differently. Second edition, London:
Demos, http://www.demos.co.uk, [IV/09].
Common Assessment Framework. 2006. Brussels: EIPA – European Institute of Public Administration,
http://www.eipa.eu/en/campaigns/go/&tid=20, 51 pp., [VI/09].
Connelly S. T. Richardson. 2004. Exclusion: The necessary difference between ideal and practical consensus.
Journal of Environmental Planning and Management 47/1(January 2004):3-17.
Diamond J. Establishing a Performance Management Framework for Government. Instituto de Estudios Fiscales,
Presupuesto y Gasto Público, Working Paper 40(2005):159-83.
Donzelot J. 1991. The Mobilisation of Society, in Burchell et al., p. 169-79.
Dopfer K. 2006. The Origins of Meso Economics Schumpeter's Legacy. Jena: Max Planck Institute of Economics,
Evolutionary Economics Group, The Papers on Economics and Evolution No. 610, 44 pp
https://papers.econ.mpg.de/evo/discussionpapers/2006-10.pdf [IV/07].
Dopfer K., J. Foster, J. Potts. Micro–meso–macro. Springer, Journal of Evolutionary Economy 14/3(2004):263–79,
http://papers.ssrn.com/sol3/Delivery.cfm/SSRN_ID721599_code114131.pdf?abstractid=721599&mirid=1,
[VI/07].
Easterling W.E., K. Kok. Emergent Properties of Scale in Global Environmental Modelling – Are There Any?
Integrated Assessment 3/2–3(2002):233–46,
http://www.ingentaconnect.com/content/routledg/nias/2002/00000003/F0020002/art00012;jsessionid=18gjekrjr
n3oh.alice, [XII/06].
Ekins P. 1992. A Four-Capital Model of Wealth Creation; in Real-Life Economics: Understanding Wealth Creation
(in Ekins P., M. Max-Neef, eds.). London, New York: Routledge, p. 147-55.
Ekins P., Medhurst J., 2003. Evaluating the Contribution of the European Structural Funds to Sustainable
Development. Presented at the 5th European Conference on Evaluation of Structural Funds, Budapest, June 26-
27, 48 pp, http://europa.eu.int/comm/regional_policy/sources/docgener/evaluation/rado_en.htm, [XI/04]
Elbers C., J.W. Gunning, K. de Hoop. Assessing Budget Support with Statistical Impact Evaluation: a
Methodological Proposal. Amsterdam: Tinbergen Institute, Discussion Paper 2007-075/2, 29 pp.,
http://ssrn.com/abstract=1016945, [VII/08].
ESPON - 3.2, Vol. 5. 2006. Spatial Scenarios and Orientations in Relation to the ESDP and Cohesion Policy -
Territorial Impact Assessment, Final Report, p. 2-97,
http://www.espon.eu/mmp/online/website/content/projects/260/716/file_2786/fr-3.2_April2007-full.pdf,
[IX/07].
European Commission. 2001. SEA Directive 2001/42/EC on the assessment of the effects of certain plans and
programmes on the environment, http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?
uri=CELEX:32001L0042:EN:NOT, [X/06].
European Commission. 2006. Impact Assessment Guidelines. SEC(2005)791,
http://ec.europa.eu/governance/impact/docs/key_docs/sec_2005_0791_en.pdf, [X/06].
Feyerabend P. 1975. Against method : outline of an anarchistic theory of knowledge. New York : Humanities Press.
Foster J., Potts J. 2007. A micro-meso-macro perspective on the methodology of evolutionary economics:
integrating history, simulation and econometrics. The University of Queensland, School of Economics,
Discussion Paper No. 343, 24 pp.
Geels F.W. Feelings of Discontent and the Promise of Middle Range Theory for STS Examples from Technology
Dynamics. Sage, Science, Technology, & Human Values, 32/6(November 2007):627-51.
GHK, PSI, IEEP, CE, National Evaluators. 2002. The Contribution of the Structural Funds to Sustainable
Development: A Synthesis Report to DG Regio, EC. 2002. Volume 1-2. London, Brussels,
http://ec.europa.eu/regional_policy/sources/docgener/evaluation/doc/sustainable_annexes.pdf [III/05]
Granovetter M. The Strength of Weak Ties. American Journal of Sociology 78(1973):1360-80.

13
Granovetter M. The Strength of Weak Ties: A Network Theory Revisited. Sociological Theory 1(1983):201-33,
http://rfrost.people.si.umich.edu//courses/SI110/readings/In_Out_and_Beyond/Granovetter.pdf, [IX/09].
Hertin J., A. Jordan, M. Nilsson, B. Nykvist, D. Russel, J. Turnpenny. 2007. The practice of policy assessment in
Europe: An institutional and political analysis. EU/FP6 Project MATISSE Working Paper 6, 52 pp.,
http://www.matisse-project.net/, [VII/08].
Hilding-Rydevik T., Á.H. Theodórsdóttir. 2004. Planning for Sustainable Development – the practice and potential
of Environmental Assessment. Proceedings from the 5 th NEA Conference, Reykjavik, 25 – 26 August 2003,
Stockholm: Nordic council, Nordregio Report 2/2004, 324 pp. http://www.nordregio.se/Files/r0402.pdf,
[IV/07].
Hill Collins, P. Toward a New Vision: Race, Class, and Gender as Categories of Analysis and Connection. Race,
Sex & Class, 1/1(1993):25–45.
IAB. 2009. Impact Assessment Board report for 2008. Third strategic review of Better Regulation in the European
Union. SEC(2009)55, 17 pp., http://ec.europa.eu/governance/impact/key_en.htm, [II/09].
Kennedy M. 1995. Interest and inflation free money: creating an exchange medium that works for everybody.
Okemos: Seva International.
Kimberlé C. W. 1994. Mapping the Margins: Intersectionality, Identity Politics, and Violence against Women of
Color, in Fineman, M.A., R. Mykitiuk (eds.). The Public Nature of Private Violence. New York: Routledge, pp.
93-118.
Knauft B.M. Anthropology in the middle. Sage, Anthropological Theory, 6/4(2006):407–30.
Kordig C. R. Discussion: Observational Invariance. Philosophy of Science, 40(1973):558-69,
http://www.jstor.org/pss/186288.
Kuhn T.S. 1970. The Structure of Scientific Revolutions. Chicago: The University of Chicago Press, 210 pp.,
http://sciencepolicy.colorado.edu/about_us/meet_us/carl_mitcham/courses_taught/5110/classic_sts/structure_of
_scientific_revolutions.pdf, [VII/08].
Kunseler E. 2007. Towards a new paradigm of science in scientific policy advising. Environment and Health
Department. NUSAP.net, 5 pp., http://www.nusap.net/downloads/KunselerEssay2007.pdf, [VII/08].
Leith C., L. von Thadden. 2006. Monetary and fiscal policy interactions in a new Keynesian model with capital
accumulation and non-Ricardian consumers. Working Paper Series No 649, 42 pp,
http://ssrn.com/abstract_id=908620, [VII/07].
Leontief W. Environmental Repercussion and the Economic Structure: An Input-Output Approach. Cambridge
Mass.: The MIT Press, Review of Economics and Statistics, 52/3(1970):262-71.
Leopold L.B., F.E. Clarke, B.B. Hanshaw, and J.R. Balsley. 1971. A procedure for evaluating environmental impact.
Washington: Geological Survey Circular 645, 13 pp., http://eps.berkeley.edu/people/lunaleopold/(118)%20A
%20Procedure%20for%20Evaluating%20Environmental%20Impact.pdf, [VII/08].
Lipsey M.W. Identifying interesting variables and analysis opportunities. In H. Cooper, L. V. Hedges, & J. C.
Valentine (Eds). The Handbook of Research Synthesis and Meta-Analysis (2nd edition). NY: Russell Sage
Foundation, 2009. Chapter 8, pp. 147-58.
Lucas R.E. Jr. Expectations and the neutrality of money. Blackwell, Journal of Economic Theory, 4/2(1972):103-24.
Martinez-Alier J., G. Munda, J. O’Neill. Weak comparability of values as a foundation for ecological economics.
Elsevier, Ecological Economics 26(1998):277-86.
Mason E. 2006. Value Pluralism; entry in Stanford Encyclopedia of Philosophy (in Zalta N.E, ed.), Stanford: Centre
for the Study of Language and Information, Stanford University, http://plato.stanford.edu/entries/value-
pluralism/, [II/07].
Mertens D.M. Inclusive Evaluation: Implications of Transformative Theory for Evaluation. Sage, American Journal
of Evaluation, 20/1(1999):1-14.
Morgan D.L. Paradigms Lost and Pragmatism Regained: Methodological Implications of Combining Qualitative and
Quantitative Methods. Sage, Journal of Mixed Methods Research, 1/1(2007):48-76.
Mumpower J.L., T.R. Stewart. Expert Judgement and Expert Disagreement. Taylor and Francis, Thinking and
Reasoning, 2/2-3(1996):191-211.
Munasinghe M. 1992. Environmental Economics and Sustainable Development. Paper presented at the UN Earth
Summit, Rio de Janeiro, Washington: World Bank, Environment Paper No. 3.
Munda G. 2006. A NAIADE based approach for sustainability benchmarking. Inderscience, International Journal of
Environmental Technology and Management, 6/1-2(2006):65–78,
http://www.inderscience.com/storage/f113210112864597.pdf, [VI/07].
Munda G. Social multi-criteria evaluation: Methodological foundations and operational consequences. European
Journal of Operational Research, 158/3(2004):662-77.
Muradian R. Ecological thresholds: a survey. Elsevier, Ecological Economics, 38/1(2001):7–24.
Nola R., H. Sankey, eds. 2000. After Popper, Kuhn & Feyerabend: Issues in Theories of Scientific Method.
Australasian Studies in History and Philosophy of Science, Kluwer. http://philosophy.wisc.edu/forster, [II/08].
Ostmann A. 2006. The aggregate and the representation of its parts. Bonn: Max Planck Institute for Research on
Collective Goods, Preprints of the Max Planck Institute for Research on Collective Goods 2007/11b, 38 pp.,
http://ssrn.com/abstract=1024681, [IX/07].
Prins B. Narrative Accounts of Origins: A Blind Spot in the Intersectional Approach? European Journal of Women’s
Studies 13/3(2006):277-90.

14
Pulido L. 2000. Rethinking environmental racism: white privilege and urban development in southern California.
Annals of the American Association of Geographers 90(1):12-40.
Radej B. 2008. Meso-Matrical Synthesis of the Incommensurable. Ljubljana: Slovenian Evaluation Society,
Working paper 3/2008, 25 pp, http://www.sdeval.si/Publikacije-za-komisijo-za-vrednotenje/Meso-Matrical-
Synthesis-of-the-Incommensurable.html, [VII/07]
Renselle D. 2007. Review of Clifford Geertz' Available Light (Princeton University Press, 2000, 271 pp.). Carmel:
Quantonics, http://www.quantonics.com/Review_of_Clifford_Geertz_Available_Light.html, [III/07]
Rotmans J. Scaling in Integrated Assessment: Problem or Challenge? Swets & Zeitlinger, Integrated Assessment,
3/2–3(2002):266–79.
Rotmans J. Tools for Integrated Sustainability Assessment: A two-track approach. Vancouver: University of British
Columbia, The Integrated Assessment Journal, 6/4(2006):35–57.
Sadler B., R. Verheem. 1996. Strategic environmental assessment: Status, challenges and future directions. Hague:
Ministry of Housing, Spatial Planning, the Environment. 188 pp.
Sankey H. The Problem of Rational Theory-Choice. Epistemologia, 18(1995):299-312,
http://www.philosophy.unimelb.edu.au/staff/Sankey/howard/howardpaper4.PDF, [III/09].
Schenk N.J. 2006. Modelling energy systems: a methodological exploration of integrated resource management.
Groningen: University of Groningen, PhD Dissertation, Chapter 6, p. 97-115,
http://dissertations.ub.rug.nl/FILES/faculties/science/2006/n.j.schenk/06_c6.pdf, [I/07].
Schnellenbach J. The Dahrendorf hypothesis and its implications for (the theory of) economic policy-making.
Cambridge Journal of Economics, 29/6(2005):997-1009.
Scriven M. The Final Synthesis. Sage, American Journal of Evaluation, 15/3(1994):367-82.
Synthesis. Encyclopedia of Evaluation. 2004. SAGE Publications. 3 Apr. 2010. http://www.sage-
ereference.com/evaluation/Article_n536.html [IV/10].
Stagl S. 2007. Emerging Methods for Sustainability Valuation and Appraisal - SDRN Rapid Research and Evidence
Review. A report to the Sustainable Development Research Network. Final Report (January), 66 pp,
http://admin.sd-research.org.uk/wp-content/uploads/2007/04/sdrnemsvareviewfinal.pdf, [VI/07].
Stake R.E. A Problematic Heading. American Journal of Evaluation; 22/3(2001):349–354.
Tae-Hee J. 2008. The Micro-Macro Synthesis in Veblen’s Institutional Economics. Buffalo State College,
http://historyofeconomics.org/Conference08/papers/jo.pdf [V/09]
Trainor B.T. Politics as the quest for unity. Perspectivism, incommensurable values and agonistic politics. Sage,
Philosophy & Social Criticism, 34/8(2008): 905–24.
UNCED. 1992. Agenda 21. United Nation Conference on Environment and Development,
http://www.un.org/esa/sustdev/documents/agenda21/index.htm, [IX/06].
Veen A. Van Der, H. S. Otter. Scales in Space. Swets&Zeitlinger, Integrated Assessment, 3/2–3(2002):160–6.
Vernon R. The »Great Society« and the »Open Society«: Liberalism in Hayek and Popper. Canadian Journal of
Political Science/Revue canadienne de science politique, 9/2(1976):261-76,
http://www.jstor.org/stable/3230923, [III/09]
Virtanen P., P. Uusikylä. Exploring the Missing Links between Cause and Effect A Conceptual Framework for
Understanding Micro–Macro Conversions in Programme. Sage, Evaluation, 10/1(2004):77–91.
Wacquant L. 1997. Reading Bourdieu's «Capital». Foreword to the English-language translation of The State
Nobility. Cambridge: Polity Press, http://www.homme-moderne.org/societe/socio/wacquant/capital.html, [I/07].
Wallerstein I. 2004. The Uncertainties of Knowledge. Philadelphia: Temple University Press, 224 pp.
WCED. 1987. Our Common Future. Oxford and New York: Oxford University Press, World Commission on
Environment and Development – WCED, 400 pp, http://www.un-documents.net/wced-ocf.htm, [IX/06].
Weaver P.M., J. Rotmans. 2006. Integrated Sustainability Assessment: What? Why? How? EU/FP6 Project
MATISSE, Working Paper 1 (October), 22 pp., http://www.matisse-
project.net/projectcomm/uploads/tx_article/Working_Paper_1_03.pdf, [VII/08].
Webster’s Third New International Dictionary. 1971. Chicago: Encyclopaedia Britannica Inc.
Wiggins D. 1997. Incommensurability: Four Proposals, (in Chang R. (ed.), Incommensurability, Incomparability,
and Practical Reason), Cambridge, Mass.: Harvard University Press, http://plato.stanford.edu/entries/value-
incommensurable, [VII/08].

15