(disponível em inglês)
General Policies
The ex-post evaluation of operations at the Bank aims to assess the extent to which the development objectives of IDB-financed operations have been attained. The ex-post evaluation of operations also aims to assess the efficiency with which those objectives have been attained. It is called an Ex-Post evaluation since the purpose is to evaluate the results of an operation, particularly in terms of its outcomes and/or impact, after it has been completed.
Ex-post evaluations of operations are performed by:
Management, which is responsible for real-time evaluation of all loan operations, up to the moment of project completion. To this end, Management prepares a Project Completion Report (PCR) on all completed IDB-loan operations.
The Office of Evaluation and Oversight (OVE), who carries out independent reviews of the PCRs produced by Management and produces independent evaluations of completed projects.
Any Borrower, who elects to do so, may undertake ex-post evaluation of operations, taking into account the institutional capacity to carry them out.
Monitoring and Evaluation activities at the IDB are a shared responsibility between the Bank and the Borrowing Country. Within the Bank, Management's primary focus is on real-time evaluation, up to the moment of completion. As part of this function, Management prepares the PCR at the end of each project execution, including information on outputs and outcomes, and ratings on its performance, sustainability and probability of achieving development objectives.
OVE carries out an independent PCR Review on all completed projects, including a validation on the ratings presented in Management´s self-evaluation. OVE also will perform, on a sample basis, in-depth ex-post evaluations of the results (impact and/or outcome) of IDB-financed operations, two or more years after completion. OVE will produce, as part of this sample:
Ex-Post Performance and Sustainability Assessments (EPSAs) on at least 20% of completed projects, aimed at reviewing the project performance and continuity of benefits, two years or more after completion.
Ex-Post Impact Evaluation Reports (IERs) on at least two projects completed in the previous four or more years. Such evaluations seek to establish the long-term flow benefits from the project, and to estimate the value added by the Bank to the long-term results obtained.
OVE produces country, thematic, strategy and policy evaluations based on, among other things, the results from the ex-post evaluation of operations in the different areas. In addition, OVE´s Annual Reports will provide information on the ex-post evaluation of operations carried out during the year.
Under the principle of shared responsibility on the activities related to monitoring and evaluation of operations, Borrowers have the main responsibility of collecting the basic information required for the preparation of the PCR and ex-post evaluations. Such information is systematically gathered for all projects up to the level of outcomes. Additional information needed for longer-term, in-depth ex-post evaluation of results should be identified, whenever possible, during project design, and the capacity to collect it should be in place during execution. As an integral part of the execution of all Bank projects, the executor must collect the information and indicators for monitoring and evaluation agreed during project design. Reliable information should be the basis for justifying the projects' ratings at different stages and increase their objectivity.
The Bank also provides technical assistance to Borrowers for capacity building in evaluation and encourages them to undertake ex-post evaluation of their projects. Several instruments are available at the Bank to support Borrowing Countries to develop institutional capacity in ex-post evaluation, including projects in the area of Modernization of the State, as well as technical assistance for ex-post evaluations of operations. OVE also coordinates with Management and the Borrowers the planning and programming of ex-post evaluation of operations to be carried out.
Purpose
Ex-post evaluation of operations is part of the Bank's evaluation system, whose aim is to establish evaluation as a tool for institutional learning and as an instrument for systematic assessment of the effectiveness of the Bank's development policies and programs, and their related processes. The main objectives of ex-post evaluation of operations are:
To provide borrowing countries with both self and independent evaluations of the results of programs and projects, and help them strengthen their monitoring and evaluation systems.
To learn from the execution of programs and projects, so that recommendations for improvements can be incorporated into the preparation, analysis, and execution of new projects.
To provide the Bank and Borrowing Countries with basic information and analysis to assess the development effectiveness of Bank activities.
To provide a suitable evaluation tool to support the decision-making process by the Bank and borrowing countries.
Basic Guidelines
The evaluation system is a participative process that must involve the interested parties, so as to generate added value that meets the needs of the Borrowers and the Bank. The selection of operations that will be subject to longer-term, in-depth ex-post evaluation of results (impact and/or outcome) by the Bank will be discussed annually during the preparation of OVE´s Work Plan. The Work Plan will identify each year the projects that will be evaluated and under what methodology.
Ex-post evaluations may range from a desk study focused on short-term outcomes, preliminary impact and continuity of benefits, carried out two years after project completion, to a longer-term, in-depth evaluation of results, carried out several years after completion. In the case of these more in-depth impact evaluations, OVE will attempt to identify operations that will be subject to this type of evaluation as early as possible during the project cycle (i.e. during programming and design of the operations). The information needed to successfully undertake this type of evaluation, especially for purposes of estimating benefits and carrying out counterfactual analysis, requires that efforts be made at project preparation, to make sure that this information is collected during project execution and operation. To this end, coordination with Management and Borrowers is required.
The criteria to decide what projects should be subject to ex-post evaluation and the specific methodology to apply in each case include:
OVE's priorities based on the areas targeted for evaluation.
The innovative aspects of the operation and its potential for drawing lessons learned.
The strategic relevance at the Bank, sector, or national levels.
The particular interest and capacity of the Borrower, especially considering how the evaluation exercise may help to strengthen the country's evaluation capacity.
When a Borrower decides to perform an ex-post evaluation, its content, the type of information required, and the indicators to be used will be defined in conjunction with the Bank. The Bank should establish whether the Borrower has the capacity to perform the evaluation, and if it does not, the Bank should define the assistance that the Borrower will require with this activity.
When a Borrower decides to undertake an evaluation or agrees on collecting additional information for the Bank´s evaluation, the loan proposal will include a section on ex-post evaluation, specifying, on a case by case basis:
The parties responsible for gathering data and for carrying out the evaluation.
The methodology to be used, both in terms of the analytical approach to be utilized, as well as the application of other evaluation tools, such as Peer Reviewers, Focus Groups, Control Groups, surveys, etc.
The information requirements and measures to adequately collect it at the time the information is needed.
The timeframe involved in all these activities.
The measures to be taken to ensure the existence of adequate Borrower capacity to collect the information and carry out the ex-post evaluation commitments, when needed.
A sound estimate of the costs involved and how they will be financed.
The findings of evaluations and the lessons learned from them should be incorporated directly and systematically into project programming, identification, preparation, analysis, approval, and execution. All the evaluation activities and outputs produced by the Bank's evaluation system, including ex-post evaluations of operations, should be accounted for in reports that analyze issues and provide information at a more aggregate level, such as country, sector, theme strategy policy, and annual reports on evaluation activities, and a summary of their findings and recommendations should be incorporated into these documents. In terms of the Annual Reports, individual evaluations should provide basic input into the overall assessment of the development effectiveness of the Bank's interventions in the borrowing countries.
Reporting
Project Completion Reports (PCRs) prepared by Management for the purpose of evaluating all completed programs or projects in the respective countries. They contain a general summary of project execution and serve as the basis for reviewing the technical, financial, and administrative problems encountered in a program or project, its achievements, and the lessons learned. They also include information on outputs and outcomes, and ratings on its performance, sustainability and probability of achieving development objectives, reporting on the decision to carry out an ex-post evaluation of the operation.
PCR Reviews, prepared by OVE on all PCRs produced by Management within a year of their delivery, providing an independent view of Management's self-evaluation in terms of its consistency and main conclusions, and validating the ratings reported in the PCR.
Ex-Post Performance and Sustainability Assessments (EPSA) prepared by OVE, on a sample basis, two years or more after project completion, aimed at assessing the performance and short to medium terms results of a program or project, particularly in terms of its outcomes, the institutional sustainability, and whether benefits continue to flow. They also review the ratings assigned by Management to the project in the Project Completion Report (PCR) and the previous validation of the PCR Review.
Ex-Post Impact Evaluation Reports (IER) prepared selectively by OVE preferably four years or more after project completion, aimed at evaluating the longer term results and sustainability of a program or project, including its impact and/or outcomes, and the effectiveness and efficiency in achieving the development objectives, drawing lessons for future operations. They also include an in-depth analysis of the project performance and further validation of the project ratings assigned in the PCR and validated in the PCR Review.
Borrowers Ex-Post Evaluations (BEP), prepared by Borrowers on a voluntary basis, preferably four years or more after project completion, aimed at evaluating the results and sustainability of a program or project, including its impact and/or outcomes, and the effectiveness and efficiency in achieving the development objectives, drawing lessons for future operations.
Norms
The specific guidelines for ex-post evaluations are set out in the Operations Administration Manual, and the Country Offices Manual.
________________________________________________________________
1/ Management is not prohibited, however, from undertaking studies or even ex-post evaluations on finished projects outside of this policy framework, for purposes of its own research and lesson learning, which could also complement the work carried out by OVE and the Borrowers.
2/ Based on the total number of projects completed during or after 2003.
________________________________________________________________
Prevailing Reference Documents:
GN-2254-5, September 2003
Resolution DE-23/94, March 1994
GA-151-1, March 1994
Memorandum CON/OEO-84/93:"Modifications to the Borrowers' Ex-post Evaluation System", March 1993
RE-238 Corr: Strengthening Oversight and Rebuilding Evaluation in the Bank", June 1999.