|
EX-POST EVALUATION
OF OPERATIONS
GENERAL
POLICY
The ex-post evaluation of operations at the Bank aims to assess the
extent to which the development objectives of IDB-financed operations
have been attained. The ex-post evaluation of operations also aims to
assess the efficiency with which those objectives have been attained.
It is called an Ex-Post evaluation since the purpose is to evaluate
the results of an operation, particularly in terms of its outcomes and/or
impact, after it has been completed.
Ex-post evaluations of operations are performed by:
- Management, which
is responsible for real-time evaluation of all loan operations, up to
the moment of project completion. To this end, Management prepares a
Project Completion Report (PCR) on all completed IDB-loan operations.
- The Office of Evaluation
and Oversight (OVE), who carries out independent reviews of the PCRs
produced by Management and produces independent evaluations of completed
projects.
- Any Borrower, who
elects to do so, may undertake ex-post evaluation of operations, taking
into account the institutional capacity to carry them out.
Monitoring and Evaluation activities at the IDB are a shared responsibility
between the Bank and the Borrowing Country. Within the Bank, Management's
primary focus is on real-time evaluation, up to the moment of completion.
As part of this function, Management prepares the PCR at the end of
each project execution, including information on outputs and outcomes,
and ratings on its performance, sustainability and probability of achieving
development objectives. 1/
OVE carries out
an independent PCR Review on all completed projects, including a validation
on the ratings presented in Management´s self-evaluation. OVE
also will perform, on a sample basis, in-depth ex-post evaluations of
the results (impact and/or outcome) of IDB-financed operations, two
or more years after completion. OVE will produce, as part of this sample:
- Ex-Post Performance
and Sustainability Assessments (EPSAs) on at least 20% of completed
projects, aimed at reviewing the project performance and continuity
of benefits, two years or more after completion. 2/
- Ex-Post
Impact Evaluation Reports
(IERs) on at least two projects
completed in the previous four
or more years. Such evaluations
seek to establish the long-term
flow benefits from the project,
and to estimate the value added
by the Bank
to the long-term results obtained.
OVE
produces country, thematic,
strategy and policy evaluations
based on, among other things,
the results from the ex-post
evaluation of operations
in the different areas. In
addition, OVE´s Annual
Reports will provide information
on the ex-post evaluation of
operations carried
out during the year.
Under the principle
of shared responsibility on the activities related to monitoring and
evaluation of operations, Borrowers have the main responsibility of
collecting the basic information required for the preparation of the
PCR and ex-post evaluations. Such information is systematically gathered
for all projects up to the level of outcomes. Additional information
needed for longer-term, in-depth ex-post evaluation of results should
be identified, whenever possible, during project design, and the capacity
to collect it should be in place during execution. As an integral part
of the execution of all Bank projects, the executor must collect the
information and indicators for monitoring and evaluation agreed during
project design. Reliable information should be the basis for justifying
the projects' ratings at different stages and increase their objectivity.
The Bank also provides
technical assistance to Borrowers for capacity building in evaluation
and encourages them to undertake ex-post evaluation of their projects.
Several instruments are available at the Bank to support Borrowing Countries
to develop institutional capacity in ex-post evaluation, including projects
in the area of Modernization of the State, as well as technical assistance
for ex-post evaluations of operations. OVE also coordinates with Management
and the Borrowers the planning and programming of ex-post evaluation
of operations to be carried out.
PURPOSE
Ex-post evaluation of operations is part of the Bank's evaluation system,
whose aim is to establish evaluation as a tool for institutional learning
and as an instrument for systematic assessment of the effectiveness
of the Bank's development policies and programs, and their related processes.
The main objectives of ex-post evaluation of operations are:
- To provide borrowing
countries with both self and independent evaluations of the results
of programs and projects, and help them strengthen their monitoring
and evaluation systems.
- To learn from the
execution of programs and projects, so that recommendations for improvements
can be incorporated into the preparation, analysis, and execution of
new projects.
- To provide the Bank
and Borrowing Countries with basic information and analysis to assess
the development effectiveness of Bank activities.
- To provide a suitable
evaluation tool to support the decision-making process by the Bank and
borrowing countries.
BASIC GUIDELINES
The evaluation system is a participative process that must involve the
interested parties, so as to generate added value that meets the needs
of the Borrowers and the Bank. The selection of operations that will
be subject to longer-term, in-depth ex-post evaluation of results (impact
and/or outcome) by the Bank will be discussed annually during the preparation
of OVE´s Work Plan. The Work Plan will identify each year the
projects that will be evaluated and under what methodology.
Ex-post evaluations may range from a desk study focused on short-term
outcomes, preliminary impact and continuity of benefits, carried out
two years after project completion, to a longer-term, in-depth evaluation
of results, carried out several years after completion. In the case
of these more in-depth impact evaluations, OVE will attempt to identify
operations that will be subject to this type of evaluation as early
as possible during the project cycle (i.e. during programming and design
of the operations). The information needed to successfully undertake
this type of evaluation, especially for purposes of estimating benefits
and carrying out counterfactual analysis, requires that efforts be made
at project preparation, to make sure that this information is collected
during project execution and operation. To this end, coordination with
Management and Borrowers is required.
The criteria to
decide what projects should be subject to ex-post evaluation and the
specific methodology to apply in each case include:
- OVE's priorities
based on the areas targeted for evaluation.
- The innovative aspects
of the operation and its potential for drawing lessons learned.
- The strategic relevance
at the Bank, sector, or national levels.
- The particular interest
and capacity of the Borrower, especially considering how the evaluation
exercise may help to strengthen the country's evaluation capacity.
When
a Borrower decides to perform an
ex-post evaluation,
its content, the type
of information required,
and the indicators
to be used will
be defined in conjunction
with the Bank. The
Bank should establish whether
the Borrower has the
capacity to perform
the evaluation, and if it
does
not, the Bank should define
the assistance that
the Borrower will require
with this activity. When a Borrower
decides to undertake an evaluation or agrees on collecting additional
information for the Bank´s evaluation, the loan proposal will
include a section on ex-post evaluation, specifying, on a case by case
basis:
- The parties responsible
for gathering data and for carrying out the evaluation.
- The methodology
to be used, both in terms of the analytical approach to be utilized,
as well as the application of other evaluation tools, such as Peer Reviewers,
Focus Groups, Control Groups, surveys, etc.
- The information
requirements and measures to adequately collect it at the time the information
is needed.
- The timeframe involved
in all these activities.
- The measures to
be taken to ensure the existence of adequate Borrower capacity to collect
the information and carry out the ex-post evaluation commitments, when
needed.
- A sound estimate
of the costs involved and how they will be financed.
The findings of evaluations
and the lessons learned
from them should be incorporated
directly and systematically
into project programming,
identification, preparation,
analysis, approval, and
execution. All
the evaluation activities and
outputs produced by the
Bank's evaluation system,
including ex-post evaluations
of operations, should be
accounted
for in reports that analyze issues
and provide information
at a more aggregate level,
such as country, sector,
theme strategy policy,
and
annual reports on evaluation
activities, and a summary
of their findings and recommendations
should be incorporated
into these documents. In
terms of the Annual Reports,
individual evaluations
should provide basic
input into the overall assessment
of the development effectiveness
of the Bank's interventions
in the borrowing countries. REPORTING
Project Completion
Reports (PCRs) prepared by Management for the purpose
of evaluating all completed programs or projects in the respective countries.
They contain a general summary of project execution and serve as the
basis for reviewing the technical, financial, and administrative problems
encountered in a program or project, its achievements, and the lessons
learned. They also include information on outputs and outcomes, and
ratings on its performance, sustainability and probability of achieving
development objectives, reporting on the decision to carry out an ex-post
evaluation of the operation.
PCR Reviews, prepared by OVE on all PCRs produced by Management within
a year of their delivery, providing an independent view of Management's
self-evaluation in terms of its consistency and main conclusions, and
validating the ratings reported in the PCR.
Ex-Post
Performance and Sustainability
Assessments (EPSA) prepared by OVE, on
a sample basis, two years or more after project completion, aimed at
assessing the performance and short to medium terms results of a program
or project, particularly in terms of its outcomes, the institutional
sustainability, and whether benefits continue to flow. They also review
the ratings assigned by Management to the project in the Project Completion
Report (PCR) and the previous validation of the PCR Review.
Ex-Post
Impact Evaluation Reports (IER) prepared selectively by OVE preferably
four years or more after project completion, aimed at evaluating the
longer term results and sustainability of a program or project, including
its impact and/or outcomes, and the effectiveness and efficiency in
achieving the development objectives, drawing lessons for future operations.
They also include an in-depth analysis of the project performance and
further validation of the project ratings assigned in the PCR and validated
in the PCR Review.
Borrowers
Ex-Post Evaluations (BEP), prepared by Borrowers on a voluntary basis,
preferably four years or more after project completion, aimed at evaluating
the results and sustainability of a program or project, including its
impact and/or outcomes, and the effectiveness and efficiency in achieving
the development objectives, drawing lessons for future operations.
NORMS
The specific guidelines for ex-post evaluations are set out in the Operations
Administration Manual, and the Country Offices Manual.
________________________________________________________________
1/
Management is not prohibited, however, from undertaking studies or even
ex-post evaluations on finished projects outside of this policy framework,
for purposes of its own research and lesson learning, which could also
complement the work carried out by OVE and the Borrowers.
2/ Based on the total number of projects completed during or after 2003.
________________________________________________________________
Prevailing
Reference Documents:
GN-2254-5, September 2003
Resolution DE-23/94, March 1994
GA-151-1, March 1994
Memorandum CON/OEO-84/93:"Modifications to the Borrowers' Ex-post
Evaluation System", March 1993
RE-238 Corr: Strengthening Oversight and Rebuilding Evaluation in the
Bank", June 1999.
*
The operational policies of the Inter-American Development Bank are
intended to provide operational guidance to staff in assisting the Bank's
borrowing member countries. Over the course of the Bank's more than 40 years of
operations, the approach to developing operational policies has taken
various forms, ranging from the preparation of detailed guidelines to
broad statements of principle and intent. Many policies have not been
updated since they were originally issued, and a few reflect emphases
and approaches of earlier years which have been superseded by specific
mandates of the Bank's Governors, the most recent being the
Eighth Replenishment mandates of 1994.
In
accordance with the Bank's information disclosure policy, the Bank is
making all of its operational policies available to the public through
the Public Information Center. Users please note that the Bank's operational
policies are under a process of continuous review. This review process
includes preparation of best practice papers summarizing experience
at the Bank and other similar institutions, and sector strategy papers.
|
|
©
2007 Inter-American Development Bank. All rights reserved. Terms
and Conditions