EVO - Evaluation- A Management Tool for Improving Project Performance (a logical

framework) -3/97-

Tomado de: http://www.iadb.org/ove/engbook/evaii.htm

II. EVALUATION AND PROJECT PREPARATION

A. EVALUATION AND PROJECT DESIGN

This chapter explains how evaluation can improve project design and planning, and
can set the stage for evaluation activities throughout the project cycle. It begins by
reviewing the steps to ensure that the project is addressing the relevant development
problem and that it has a clearly defined purpose, as these two attributes are
important for enhancing project performance and facilitating evaluation activities. The
chapter also describes the evaluation products that are generated at the design stage
of the project. It must be emphasized that, at the project design stage, some of the
more vital aspects are:

1. Establishing a clear understanding of the development problem;

2. Building into the project design lessons from previous operations; and
3. Setting the stage within the project design for effective evaluation both during
the monitoring and ex-post stages.

The above considerations do not replace, and are only complementary of the Bank's
economic, financial and technical analyses.

1. Evaluation Framework for Project Design

The Logical Framework is an evaluation tool that may be used at the ex-ante or
project design phase of the project evaluation process. This handbook cannot do
justice to all the details of Logical Framework Analysis, but a brief synopsis is
provided. For further details, see Annex 1 of this handbook.

2. Preparing for the Development of a Logical Framework

A central part of any evaluator's work is to determine whether a project has been
successful in addressing the development problem it was designed to solve. In too
many past projects this has been difficult because the development problem was
never well understood when the project was designed, nor was a link established to
the solution provided by the project.
3. The Stakeholder or Interested Party Analysis

This analysis clarifies which groups are directly or indirectly involved in a specific
development problem, their respective interests in relation to it; their perceptions of
the difficulties related to the development problem; the resources (political, legal,
human, and financial) they may contribute toward resolving the development
problem; their respective mandates with regard to the problem situation; their
reactions to a possible project strategy; and the existing or potential conflicts among
stakeholders. The stakeholder analysis is an important source of information for the
evaluation of the project during its execution, and thus it is important to identify the
stakeholders and understand their roles in the execution of the project.

4. The Problem Tree

The problem tree is an important aid to understanding the development problem. It

considers the negative conditions perceived by the stakeholders in connection with the
development problem. It arranges the principal problems according to their cause-
and-effect relationships, thereby clarifying upon which objectives the project should
focus. The definition of the chain of problems allows for improvements in project
design, a better monitoring of the "assumptions" of the project during its execution,
and, upon project completion, facilitates the task of the evaluator, who must judge
whether these problems have improved or worsened as a result of the project. A
simplified example of a problem tree is presented on Figure 3. It addresses the issue
of a city bus service and identifies the cause and effect relationships between the
principal problems.

5. The Objectives Tree

The development problems identified in the problem tree are transposed into project
objectives as part of the initial stage of designing a solution. The objectives identified
as the outputs of a project become the means for addressing the identified
development problem and provide a means to assess performance. An example of an
objectives tree is on Figure 4, using the problems identified in the problem tree.
 B. THE LOGICAL FRAMEWORK

The logical framework is one of the principal tools used nowadays by agencies for
project design and planning. Developed for the U.S. Agency for International
Development (USAID) in the late 1970s, the logical framework provides a matrix
within which the evaluator can examine aspects of project performance at all stages of
the project. The strengths of such a framework are that it provides:

 a clear means-ends analysis of project inputs leading to outputs for set

purposes in support of a goal
 specification of inputs and costs for project activities;
 objectively verifiable indicators of performance and sources of verification;
 specification of the key assumptions or risks underlying the project; and
 a framework for introducing lessons learned to be incorporated in future
projects

The logical framework is a tool that helps project designers better understand the
problems they are trying to solve. The framework is based on two basic principles:
first, cause-effect relationships between different parts of a problem which correspond
to the four levels (or rows) of the framework which relate to activities (or inputs),
components (or outputs), the purpose and the goal as the set of hierarchical project
objectives; second, the principle of correspondence, which links the four levels of
objectives to the measurement of attainment (indicators and means of verification)
and conditions which may affect attainment (or assumptions.)

In order to follow the description below, please refer to Table 3.

1. Vertical Logic

The logical framework helps systematize and apply a rational approach to the design,
execution and evaluation of projects. In Tables 3 and 4 the vertical logic postulates
that if we contribute certain inputs we will deliver certain outputs: thus, there is a
necessary and sufficient relationship between inputs and their corresponding outputs,
as long as the assumptions are confirmed in reality. At the next level of vertical logic
of the framework we again make a causal inference. If the project delivers those
outputs (or components), and the assumptions hold, the purpose (a hypothesis) will
be achieved (i.e., the outputs are necessary and sufficient conditions). Continuing to
the final step, if the purpose is achieved, and the assumptions at the purpose level
hold, we will have contributed significantly to the attainment of the goal (i.e., the
purpose is necessary but not sufficient).

2. Horizontal Logic

In practical terms, the horizontal dimension is a description of how project managers,

Country Office staff responsible for monitoring the project, and evaluators measure
the attainment of results expected at each level of objectives. In the Tables mentioned
above, the second column includes what are called "indicators". These are
predetermined, quantitative and qualitative measures that indicate the status of input
or output delivery, the achievement of the purpose (project impact) or the extent of
contribution toward attainment of the goal. The third column explains how they will be
measured, by specifying sources of information and methods to be employed. The
fourth column describes the assumptions or risks that must hold in order to ensure the
achievement of the activities or products of each level, and to proceed to the next
level in the hierarchy of objectives.

3. Indicators

Indicators provide the focus for evaluation data collection, so they need to be specified
carefully and sparingly. The logical framework must include indicators for all the
important objectives, but, at the same time, indicators should be chosen carefully to
ensure their measurability. In setting indicators, the quantity, quality and timing
required to achieve the next level of objectives must be considered.

These indicators are very important because they provide an operational response to
the issues being addressed by the project. Furthermore, they provide a focus for data
collection at the preparation stage and a map to guide project monitoring and
evaluation later on. The former function is presented later in this chapter; the latter
are the subjects of chapters IV, V, and VI.

A general introduction to the logframe concepts is presented on Table 4. It is followed

by a sample logical framework that builds on the example used for the problem and
objectives trees presented earlier in this chapter.
4. Assumptions

As mentioned in previous sections, for the "vertical" relationships to hold, project

managers rely on the verification or confirmation of the establiched critical
assumptions which, as stated previously, in the logical framework matrix are outlined
on the far right column for each level. These assumptions are the critical factors not
controlled by the project managers or the executing agency, but which influence its
implementation and chances for success.

Table 4 : THE STRUCTURE OF THE LOGICAL FRAMEWORK

Objectives Indicators Means of Verification Assumptions

GOAL

The Goal is a statement of

how the project or program
will contribute to the
solution of the problem (or
problems) of the sector.
PURPOSE
The indicators at Goal level
describe how the overall
impact of the project shall be
measured. They are specific
in terms of quantity, quality,
and time (target group and
location if relevant).
The means of verification are
the sources of information that
can be used to verify that the
targets were achieved. They
can include published material,
visual inspection, sample
surveys, etc.
The assumptions indicate
the important events,
conditions, or decisions
necessary for the
sustainability in the long
run of the benefits
generated by the project.

The Purpose is the direct

impact to be achieved as a
result of the Outputs
produced by the project. It
is a hypothesis about the
impact or benefit that the
project attempts to achieve.
OUTPUTS
The indicators at the
Purpose level describe how
the direct impact of the
project shall be measured.
They should include targets
reflecting the end of project
status (EOPS). They are
specific in terms of quantity,
quality, and time (target
group and location if
relevant).
The means of verification are
the sources to which that the
executor and evaluator can
refer to see if the targets are
being achieved. They can
indicate that there is a problem
and suggest the need for
changes in project Outputs.
They can include published
material, visual inspection,
sample surveys, etc.
The assumptions indicate
the events, conditions, or
decisions that must occur
in order for the project to
contribute significantly to
the achievement of the
Goal.

These are the goods,

services, and training that
the project executor is
required by contract to
complete. They should be
expressed as work
completed (systems
installed, people trained,
etc.).
ACTIVITIES
The indicators for Outputs
are succinct, but clear,
descriptions of each of the
Outputs that has to be
completed during execution.
Each should specify
quantity, quality and timing
of the goods, services, etc.
to be delivered.
This cell tells where an
evaluator can find the sources
of information to verify that the
products/services contracted
have been delivered. Sources
can include site inspection,
auditor's reports, etc.
The assumptions are the
events, conditions, or
decisions that have to
occur in order for the
Outputs will achieve the
Purpose for which they
were undertaken.

Activities are the tasks that

the executor must carry out This cell contains the budget This cell tells where an The assumptions are the
for each Output produced by evaluator can obtain events, conditions, or
in order to produce each of the project.
the Outputs of the project
and that denote costs.
Activities are listed in
chronological order for
each Output.
Table 5 (a): LOGFRAME FOR IDB PROJECT DESIGN AND EVALUATION
COUNTRY:
PROJECT:
ESTIMATED PROJECTG START DATE:
ESTIMATED PROJECT COMPLETION DATE:
NARRATIVE SUMMARY
PROJECT GOAL
Use of the CBS by the
population increases.
PROJECT PURPOSE
The service offered by the
CBS is reliable.
PROJECT OUTPUTS
1. Drivers drive carefully.
2. Buses are in good
condition.
3. Scheduling and use of
information on whether the decisions that have to
budget was spent as planned. occur in order to complete
It is usually the accounting the Outputs of the project.
records of the executing unit.
Example: Transportation Project Case
PROJECT PLANNING TEAM __________:
PROJECT NO.:
INDICATORS MEANS OF VERIFICATION ASSUMPTIONS
l Number of passengers l Audited City Bus Company
increases from X0, in base statistics reported to the City
year, to X3, at the end of the Council.
year 3, X4 at the end of the
year 4, X5, by December
2002 and X6, by December l Results of passenger
surveys.
2003.
l The number of passengers
complaints decreases from
B0, in base year, to B3, at the
end of year 3, B4, at the end
of year 4, B5, by December
20002 and B6, by December
2003.
l The rate of accidents l City Department of (l The relative price of the
decreases from Y0, in base Highways statistics. gas is stable).
year, to Y1, at the end of
year 1, Y2, at the end of year
2, Y3, at the end of year 3
and Y4, at the end of the
project (December 2001). l Statistics from the City
Police Department.
l The number of delays (+/.5
minutes) drops from Z0, in
base year, to Z1, at the end
of year 1, Z2, at the end of
year 2, Z3, at the end of year l Audited City Bus Company
3 and Z4, at the end of the statistics reported to the City
project (December 2001). Council.
l Infractions of safety l Audited City Bus Company (l Special bus lanes are
regulations decline from S0 statistics reported to the City approved by the City
in base year, to S1 at the Council. Council.)
end of year 1, S2 at the end
of year 2, S3, at the end of
year 3 and S4, at the end of l Audited City Bus Company
statistics reported to the City
the project (December
2001). (1) Council.
l Breakdowns decline from l Baseline data from the
J0, in base year, to J1, at the census; updated from the
end of year 1, J2, at the end monthly current population
of year 2, J3, at the end of survey carried out by the
year 3 and J4, at the end of National Statistical Office.
the project (December
2001). (2)
l City residents living within
 buses is optimized. 1/2 mile of rush-hour bus
stops increases from R0, in
base year, to R1, at the end
of year 1, R2, at the end of
year 2, R3, at the end of year
3, and R4, at the end of the
project (December 2001). (3)
PROJECT ACTIVITIES (l Adequate road
maintenance and
reconfiguration by the City
1.1 Train bus drivers. Public Works Department.)
1.2 Introduce incentives to
drive carefully.
1.3 Improve working (l The drivers union is in
conditions. agreement with the project
1.4 Introduce safety strategy.)
regulations and inspection
system.
(l Import duties on vehicle
parts do not increase.)
BUDGET BUDGET EXECUTING
DOCUMENTS
(l Revenues from bus fares
2.1 Store repair equipment are sufficient to allow
and parts. replacement of buses.)
2.2 Improve the repair
facility.
2.3 Establish schedule for (l The City Council approves
replacement of buses. the revised scheduling and
routes.)

3.1 Optimize the routes and

scheduling.
3.2 Equip buses with radios
for communication.
3.3 Establish a
communication station at the
central bus terminal.
3.4 Collect statistics
regarding compliance with
schedules and safety
regulations.

4.1 Undertake continuing

passenger survey program.

C. EVALUATION AND PROJECT APPROVAL

Everyone involved in project approval should ensure that each new project will be able
to benefit from future evaluation processes. An evaluability assessment must be
incorporated at the design stage, and project documents submitted for approval
should include the Logical Framework and other elements that ensure "evaluability".
"Evaluability" is a review undertaken by the project team to assess the extent to
which the design described in project documents is able to support monitoring and
evaluation activities. An evaluability assessment will:

 support the design team in ensuring the project is of the highest quality
 ensure that logical framework standards are being adhered to
 ensure that the project plan provides adequate criteria for monitoring and
evaluation
 assess the extent to which lessons and best practices have been incorporated

The following guidelines should be used for this evaluability assessment.

Table 6: Evaluability Guidelines

Evaluability Requirements Paragraph
No.
Objectives
Problem or need that the project is attempting to address has been identified and analyzed.
Identification of whose problem or need it is.
Causes of the problem or need have been identified and ranked.
Expected objectives have been consistently defined.
Lessons learned from previous operations and evaluations have been taken into account.
Indicators
Conditions (physical, institutional, economic, social) prior to the execution of the project have been
described.
Baseline data on the conditions prior to the execution of the project have been included.
If no baseline data provided, project design includes data collection.
Benchmarks, target figures, or other evidence to monitor progress and determine attainment of the
objectives have been provided.
Outputs
Goods or services that the project will generate have been identified and described.
Description of how and when the beneficiaries will use the goods and services generated by the project
has been provided.
Benefits to be derived from the use of the goods and services generated by the project have been
identified.
Assumptions
Individuals, groups, institutions, and organizations that could positively or negatively affect the execution
of the project have been identified.
Events or elements that are outside the direct control of project management and that could affect project
viability, outputs and objectives have been identified and described.
Provisions have been made to review financial and economic feasibility analyses if the project
experiences implementation delays which will negatively impact the indicators of project success.

Table 7: PROJECT STRUCTURE: PERFORMANCE BENCHMARKS

PROJECT CYCLE TYPES OF PROJECT ISSUES EXAMPLES OF MEDIOS DE
STAGE BENCHMARKS INDICATORS VERIFICACIÓN
 CONDITION AT PROJECT START-UP
SITUATION: BASELINE (a) problem or need (a) high mortality of Specific means of
BENCHMARKS: identified and analyzed. children between ages of verification must
0 and 5 in rural areas, of be identified for all
to be changed as a which "x" % with skin indicators (i.e.
result of the project or zero of the project (b) causes of the infections,"y" % with hospital and school
and its objectives indicator system. problem identified and gastrointestinal diseases, records, social and
(ex-ante). analyzed. etc. demographic
sources,etc). With
(For Logframe (c) data on initial the timeliness and
(b) (1) sanitary conditions
users: refers to conditions of the problem in hospitals; (2) sanitary periodicity
Stakeholder and set (physical, economic, conditions in the home; (3) required. If these
Problem Analysis) social, gender, financial, children's sources do no
institutional, exist the
malnourishment; (4) low generation of
environmental, etc.) level of education of validating
mothers. information will
(d) identification of have to be
project stakeholders (i.e. (c) (1) "x" number of introduced as a
population that will hospitals without basic project activity.
benefit from the project, sanitary modules in
interested public and children's wards; (2) "x" %
private institutions, public of nursing staff with only
and private institutions "y" number of years of
that could become training; (3) "x" % de
impediments to the mothers with only "y"
project) and listing of number of years of
assumptions on primary schooling; (4) "x"
behaviour of % of children under five
stakaholders and/or years with only "y" level of
events that could affect nutrition, etc.
project execution and/or
development impact
upon completion. (d) (1) women's
employment must
increase in the
(e) Lessons learned from region/country/city, etc; (2)
previous operations. Health Ministry functions
will be decentralized; etc

(e) (1) no new hospitals

are needed --
rehabilitational
improvement of existing is
sufficient to attain project
outcome; (2) no new
physicians need to be
trained -- training of
nursing staff is sufficient;
etc.
PRODUCTION OF COMPONENTS AND PROJECT PERFORMANCE
OUTPUTS: MONITORING (a) goods and services (a) Mid-term: (1) "x"
BENCHMARKS: the project will delivernumber of hospitals
mid-term upon rehabilitated at "y" water
completion. quality level by years "t1",
"t2", "t3", etc; (2) "x"
or products to be number of nursing staff
(b) assumptions on trained in "y" quality
generated as a (a) measure the behavior of stakeholders techniques by years
result of project progress and and events that are
activities during efficiency of project "t1"..., etc.;
outside the direct control
execution and at execution (by adhering of project management.
completion, or final to expected schedules (a) Upon completion: (1) a
disbursement (ex- and products) at mid- total of "x" hospitals
dure). term and completion rehabilitated and capable
stages, of sustaining "y" quality
(For Logframe standards of medical
users: refers to attention to the 0 to 5 year
Activity and age grupo by year "z"; (2)
Component levels) a total of "x" nursing staff
(b) assess the trained at "y" technical
behavior of critical standard and capable of
assumptions as they assisting in training of
 may affect project other nurses by year"z";
execution. etc.

(b) (1)"x" number of

hospitals budgets are
adjusted to "y" level by
years "t1"...etc. to facilitate
rehabilitation programs;
(2) "x" number of schoold
budgets and teaching
programs are improved by
"y" levels to facilitate
training of mothers by
years"t1" ...etc.
PERFORMANCE DEVELOPMENT IMPACT
OUTCOME: TARGET (a) Use beneficiaries will (a) (1) "x" % of hospital
BENCHMARKS: make of the goods and staff uses "y" % improved
services of the project basic services in "z"
and benefits they derive; number of hospitals; (2)
"x" % of better trained
or desired situation nurses provide care to "y"
(b) Solution, or level to "z" number of
after project measure the use and contribution to the
completion as effectiveness of children within target age
solution, of the problem group per year; etc.
products are used project products and identified at situation
by beneficiaries (ex- assess the behavior of stage above;
post) assumptions that (b) decrease of child
could affect project mortality in rural areas: (1)
development impact. (c) List of assumptions rate of infections of
on behavior of events in children in target age
the hands of the cohort falls from "x" to "y"
(For Logframe borrower/beneficiaries, % between years "z" and
users: refers to or beyond their control, "v"; (2) number of
Purpose and Goal after project completion hospitalizations of children
levels) (final disbursement). in target age group falls
from "x" % to "y" %
between years "z" and "v";
etc.

( c) (1) all assumptions of

project completion stage
continue to operate and
project sustainability is
attained by beneficiaries
contributions to upgraded
hospital, training and
nutrition programs.

D. BASELINE BENCHMARKS

These benchmarks, and their corresponding indicators (see Table 8), are most
important and are intended to provide a picture of the situation before project
intervention. They describe the situation by quantifying the levels of selected
indicators which can be re-visited later on. Changes in the levels are expected to have
a plausible link to the effects of the project. Baseline benchmarks provide a basis for
the measurement of change and test the reliability, validity and feasibility of certain
types of information upon which monitoring and evaluation can be built.

E. ESTABLISHING BASELINE BENCHMARKS

Unless they already exist, baseline benchmarks may have to be collected for at least
some of the indicators identified in the project logical framework. The following
checklist can be used to identify necessary baseline data and to indicate when it
should be updated. It often happens that when projects are evaluated, at mid-term
and ex-post levels, appropriate data are difficult, impossible or too costly to acquire.
Careful attention at the preparation stage can often do much to alleviate these
problems.

The first step in completing the checklist is to identify the status of data on each
indicator:

 What data are available now?

 Should additional baseline data be collected before the project is implemented?
 Will data be required for this indicator for project monitoring activities?
 Will data for this indicator be required for mid-project, end-of-project, and/or
impact evaluations?

The following checklist has been completed for two of the indicators identified in the
logical framework presented earlier in the Chapter.

Table 8: Baseline Benchmark Checklist

Indicators Baseline Data When will updates of data be required?
Data Data to be Monitoring Evaluation
Available Collected Mid BEP Impact
Number of 2,317 fare- N/A N/A ü ü
Passengers paying
using CBS passengers,
August 1994
(CBS
statistics)
Reliability Require data ü ü ü
of CBS on delays (+/-
service 5 minutes) for
representative
 sample of
rush-hour bus
stops

SUMMARY POINTS

 Through its contribution to a sound logical framework

and integration of lessons learned, evaluation is a
fundamental part of project design.
 Evaluability assessment provides decision-makers with
vital information for those in charge of project approval
and those involved in the project's execution.
 Baseline data are essential for sound planning and
subsequent evaluation.

Top