Effectiveness for Improving lives
measuring results and impact
IDB investments need to produce tangible and positive results in the lives of beneficiaries and address the most important development challenges facing our 26 borrowing members in Latin America and the Caribbean. In order to achieve these goals, the IDB has put in place a system that allows us to design sound development projects, monitor their progress, and measure their results and impact.
At entry, proposals identify the development challenge to be addressed by the project and provide an analytic diagnosis that leads to a proposed solution with a clear logic. Solutions are evidence-based, in that their effectiveness has been documented, or a sound rationale for the intervention and provisions exists to generate knowledge about their effectiveness. Proposals also quantify the size of the problem and the relevance of the intervention. All this information is presented alongside a results matrix and a monitoring and evaluation plan. The proposals are rated for alignment with strategic priorities, evaluability (which encompasses logic, economic and risk analyses, as well as monitoring and evaluation), and additonality, through the Development Effectiveness Matrix (DEM).
Development Effectiveness Matrix (DEM)
The Bank measures the evaluability of development interventions with a set of development effectiveness matrices or DEMs. These matrices, based on the Good Practice Standards produced by the Evaluation Cooperation Group (ECG), are a checklist of analytical and informational requirements. The DEMs allow an assessment of whether products meet a minimum set of information requirements such that reliable and credible monitoring may be conducted during implementation, and reporting results from the interventions in a rigorous manner at completion through evaluation is possible.
Monitoring and Evaluation (M & E)
Monitoring and Evaluation (M & E) includes processes, systems and tools to analyze performance of the Bank’s development interventions. The monitoring function identifies the status of a project with regard to its expected outputs and estimated time and cost parameters, while evaluation uses empirical evidence to identify to what extent outcomes and impacts are achieved. Evaluation has shifted from reflexive comparisons to more rigorous methodologies. There are several methods for conducting evaluations, the most rigorous being the use of random assignment to create experimental and control groups. If random assignment is not feasible the IDB may use alternative methods to create a credible comparison (quasi-experimental approaches).Economic Rate of Return (ERR)
Many of the IDB’s operations include an economic analysis using the Economic Rate of Return (ERR) to evaluate contribution of Bank’s efforts to economic development. The ERR compares the interest rate of an operation with the costs of capital and benefits for countries discounted over its life. Other cost-effectiveness indicators are also utilized as proxies to track contributions to economic development over the life of the project.
Implementation is monitored using a quantitative approach to track the achievement of a project’s outputs and outcomes relative to its estimated time and cost parameters, through the Project Monitoring Report (PMR). At 18 months from execution, a Loan Results Report (LRR) will be prepared to assess potential implementation issues. Progress Monitoring Report (PMR)
The Progress Monitoring Report (PMR) is a tool to enable results-based management, shifting the focus of monitoring of implementation from inputs to outputs and outcomes. The PMR strives to identify delays and deviations early on during project implementation, and changes needed during execution, using a quantitative approach to track the achievement of a project’s outputs and outcomes relative to its estimated time and cost parameters. The PMR helps measure corporate performance and provide information on delivered results during project execution.
Loan Results Report (LRR)
To measure the effectiveness of Bank’s development interventions each project team prepares the Loan Results Report (LRR) 18 months after the first disbursement is made. This report includes three key sections: a) the results report, b) the review of implementation including risks and safeguard experience, and c) the annual operating plan with a timeline of activities for the next 12-18 months. The LRR may also include the evaluability rating of the Development Effectiveness Matrix (DEM).
Results at completion are reported in the Project Completion Report (PCR), and are externally validated by the Office of Evaluation and Oversight (OVE). The Bank’s strong commitment to produce evidence on “what works” has resulted in an increased number of operations that include rigorous impact evaluation components at project design.
Project Completion Report (PCR)
The Project Completion Report (PCR) assesses results achieved by a project, its long-term sustainability, and lessons learned to improve the design and execution of future operations. The PCR constitutes a building block to improve management for results, learning and greater accountability. The PCR is designed to help Bank staff comply with the nine core standards for project completion reporting outlined by the Good Practice Standards produced by the Evaluation Cooperation Group (ECG-GPS). Key to these standards is that all PCRs results need must be validated by the Office of Evaluation and Oversight (OVE).
An Impact Evaluation (IE)
An Impact Evaluation (IE) determines the net causal effect of an intervention on an indicator of interest. A key concept in IE is counterfactual analysis—what would have happened in the absence of the project? The net attributable impact is the difference between the observed value and the counterfactual. However, a fundamental challenge in IE is the impossibility to observe the same units (households, schools, firms, etc.) with and without the intervention. While it is possible to observe the units at different moments in time—as in before and after scenarios—it is possible that other factors could explain the change and therefore this scenario is not a valid counterfactual. Rigorous impact evaluations ask not only if there was a net causal effect but also why there was or was not an effect. In this regard, it is important to make the project logic explicit and consider other mechanisms and variables.
The Office of Strategic Planning and Development Effectiveness (SPD) engages in the evaluation of projects at different levels according to the demands from project teams and the strategic relevance of a particular operation. SPD’s role is to strengthen the design and implementation of IE, and its involvement can occur at three different levels: a) advising teams on general evaluation questions; b) reviewing terms of reference and/or results matrices and supporting the team in the definition of the evaluation design (sample size, definition of indicators, etc.); and c) in a more substantive role, in which a SPD member participates as full team member, responsible for the evaluation component of the project.
Office of Evaluation and Oversight (OVE)
The Office of Evaluation and Oversight (OVE) undertakes independent and systematic evaluations of the Bank's strategies, policies, programs, activities, delivery support functions and systems. OVE disseminates findings of these evaluations so that recommendations for improvement can be used in the design, appraisal and execution of new operations. OVE has organizational and behavioral independence, and is free from external pressure and conflicts of interest, as stipulated by the criteria established by the Evaluation Cooperation Group (ECG) of the Multilateral Development Banks, of which the IDB is a founding member. OVE's findings, analyses, and conclusions are free from management’s influence at all stages of the process, including the planning of work programs and budget, formulation of terms of reference, staffing of evaluation teams, execution of evaluations and approval of reports.
As part of the Ninth General Capital Increase (GCI), a Results Framework (RF) was developed to provide the Bank with an accountability mechanism to measure and report on progress on institutional priorities. It includes best practices adopted by other multilateral and bilateral development organizations. Based on the Bank’s five institutional priorities, the RF will allow shareholders to monitor the Bank’s contribution towards selected regional development goals, as well as desired progress on Bank’s output indicators and operational effectiveness and efficiency. Monitoring the set of proposed indicators will contribute to increased transparency and accountability.
The Quarterly Business Review (QBR) is a report to provide management with a status of the Bank’s internal performance. The QBR reports on outputs, lending program priorities and organizational indicators on a quarterly basis to allow management to track the achievement of expected results. This constant monitoring allows more timely identification of deviations and effective implementation of measures to address them.