EVALUATION OFFICE REPORTS - REs

Español

Table of Contents

Acronyms
Summary
1. A New Paradigm for Evaluation
2. The Bank Evaluation System
3. Enhancement of the Bank Evaluation System
4. EVO's activities

Appendices

Appendix 1: List of Borrower Ex-Post Evaluations and Mid-Term Evaluations
Appendix 2: List of Project Completion Reports
Appendix 3: List of evaluation reports prepared by in 1994
Appendix 4: Performance Benchmarks Matrix
Appendix 5: LFA Structure

Top

Acronyms

   

BEP	Borrower Ex-post Evaluation
BES	Bank Evaluation System
CPE	Country Program Evaluation
ECG	Evaluation Coordinators Group
EMD/DEV	Staff Development Section - Administrative Department
	Evaluation Office
NET	Bank Evaluation Information System
GCO	General Consulting Office
IDB	Inter-American Development Bank
IDB-8	Eighth Replenishment of the IDB
LFA	Logical Framework Analysis also known as Logframe
OVE	Operations Evaluation Office
ORE	External Review and Evaluation Office
PCR	Project Completion Report
PRA/PAO	Project Advisory Office - Project Analysis Department
TAPOMA	Task Force on Portfolio Management

Top

Summary

0.1 Evaluation in the Inter-American Development Bank is a multiple stakeholder system involving producers and users of evaluation products. These stakeholders are the borrowing countries, executing agents, Bank staff and Management, the Board of Executive Directors, the Evaluation Office and the general public audience. The purpose of the evaluation system is to contribute toward improved performance of the operations funded by the Bank, to institutional learning and to the transparency of results achieved by the activities of the Bank and its development partners.

0.2 During 1993, the Board of Executive and management actively reviewed and remodeled the evaluation function of the Bank. The aim of this effort was in support of the bank's commitment to "manage for results", as highlights by the TAPOMA report.

0.3 1994 was the first year of operation of the new Evaluation Office (). Priority was given to the establishment the Office. dedicated significant efforts to introducing to the Bank a new evaluation paradigm which is necessary to meet development changing environment and expectations of member countries. also insured that evaluation studies, training and oversight activities were kept at levels comparable to those of the previous years.

0.4 The activities of the Evaluation Office may be summarized as follows:

• in areas of high priority in the Eighth Replenishment, such as Gender in Development, Primary Education, Public Sector Reform and the Health Sector;

• in sectors and sub-sectors that are of traditional interest to the Bank, such as Rural Roads, Roads Maintenance and Agricultural Sector Reform; and

• at the project level by undertaking Project Performance Reviews and reviewing the Project Completion Reports done by Management.

0.4.2 Support the revitalization of the Bank Evaluation System (BES) by

• disseminating the new vision of the evaluation function and setting up the institutional framework for the establishment of the BES and,

• promoting evaluation capacity in borrowing countries through seminars and on-site workshops.

• training Bank staff and counterpart officers in design, monitoring and evaluation techniques.

0.5 This first annual evaluation report provides:

• an overview of the Bank Evaluation System

• a reviews of the status of the main components of the BES; and,

• a synopsis of the 1994 work program of the Evaluation Office.

Top

1  A New Paradigm for Evaluation

In a world of increasingly scarce resources, governments are under unprecedented scrutiny from their citizenry for increased performance. Countries have dedicated a considerable amount of funds to international development assistance and are now demanding better performance for dollar invested and greater transparency of the results achieved. There are similar pressures on borrowing nations from their citizens, who are still recovering from the debt crises of the 1980s. Governments have responded to these demands by undertaking public sector reforms aiming at improving public services and investments. To support these endeavors, the evaluation function can play a cost-effective role and provide a unique contribution to demonstrate value-for-the-investment.

1.2 The success of the IDB's efforts in helping members countries achieve sustainable development depends on a set of circumstances, many of which are beyond its control. There is no definitive way or apparent path of pursuing development. Hence, one may expect that some IDB's activities may fall short of meeting the expected results, while others may exceed them. There is a need to ensure that the lessons learned from the activities become part of the Bank's knowledge base and lead to improved performance.

1.3 The broad spectrum of development activities funded by IDB makes measurement of development results - the impact on the socio-economic development of borrowing countries - a challenging task. In some cases, such as projects of a physical nature, results are easily quantified. In other cases, such as institution building, results are more qualitative and long-term in nature. In this context it is essential to have a sound information base which allows the Bank to assess the effectiveness of its operations: the degree to which activities achieve the expected results and how efficiently they are delivered.

1.4 The IDB, in an effort to meet the needs of its members for strengthening their evaluation capacity and increasing the effectiveness and efficiency of Bank funded operations, is re-examining the guiding principles of its evaluation function. This re-examination is part of a Bank-wide re-engineering processes responding to the challenge of increasing Bank lending to the social sectors. To maximize its impact, the Bank is seeking to ensure that its activities contribute to borrowing member countries' efforts to reduce poverty and to pursue social and economic reforms. In order to meet this challenge, the Bank is reviewing and adjusting its processing capabilities, both in terms of staff skills, internal procedures and organizational set up required to improve its portfolio management.

1.5 The evaluation function within the IDB has evolved over time. Initially there was the Group of the Three Controllers, followed by a dual system comprising the Office of External Review and Evaluation and the Operations Evaluation Office. As elsewhere, the evaluation function found its genesis in the controller ship functions.

1.6 Presently, controller ship withing the Bank is comprehensively ensured by many entities exercise control functions. The Board of Governess and the Offices of Executive Directors perform a continuous oversight function. Representatives from executing agencies, lobbying groups and the media informally contribute t this oversight function. IDB's Independent Investigation Mechanism, the Office of Ombuudeperson and the Administrative Tribunal review complaints that may be bade about Bank actions. The External Auditor of the Bank and the Office of the Auditor General verify if proper management controls are in place and audit if the Bank complies with its policies and regulations. Bank managment has its own managerial self-controls and is presently enhancing its ability to perform portfolio assessment. The Office of the Controller supports Managment in matters of organizational and system improvement and in programming and loan reviews. The Evaluation Office reports on the extent to which the bank achieves expected results. In addition, there are research functioned that explore new ways and seek best practices. For example, reference can be made to the activities undertaken by the Office of the Special Advisor on Social Affairs, the Office of the Chief Economist, the Inter-American Institute for Social Development and the Instituted for the Integration of Latin America.

1.7 With the unification and re-examination of this evaluation function, the Bank was presented with a unique opportunity to redesign this function insight of the new challenges facing the Bank, giving evaluation a singular role within the controller ship spectrum. Taking into consideration the existence of numerous oversight entities existing within the Bank, has adopted a new approach to evaluation breaking away from the general legacy of the inspection paradigm in favor of greater institutional learning.

1.8 Recent portfolio reviews of the international financing organizations all point to the need for designing sustainable approaches to development. In the development evaluation community, there is a consensus that keys to successful developmental activities include: local ownership of the project, design answering to the real demand of beneficiaries, gradual implementation, which take into account the absorptive capacity of the beneficiaries, step-wise approaches enabling on-going assessment, and adjustments aiming at end-results, client satisfaction and cost-effectiveness.

1.9 Given the social content of many Bank-financed projects, there is a need to adapt the approach to project evaluation in such a way that performance measurement is not based solely on quantitative results of physical deliverables, but may be explained with a qualitative assessment of the development processes. This calls fro regular monitoring throughout the life of a project, leading to periodical realignment of benchmarks as progress towards project outcomes are or may not be achieved.

1.10 Considering the above, one may view the evaluation system as a continuous function. It begins with formulation of objectives of the project and assessment of critical assumptions necessary for the successful achievement of the objectives. During project execution one has to periodically appraise the effectiveness and impact of the project and reading the benchmarks in order to optimize the end results. Ex-post evaluation has to go beyond the simple measurement of the ex-ante blueprint and the final result. It needs to assess the extent to which project implementation was responsive to changing environments. The integrated continuum of ex-ante assessment and bench marking, of mid-term evaluation and ex-post evaluation is essential to result-driven and knowledge-based management.

1.11 The approach leads to the following principles:

• evaluation is a learning tool that empowers the stakeholders;

• evaluation is an investment which must have a short payback period and its results can be used for decision-making;

• evaluation is participatory, involving borrowers and beneficiaries;

• evaluation draws on best practice, facilitating learning by doing;

• evaluation concentrates on performance results; and

• evaluation serves all management levels.

1.12 In fact, the evaluation system calls into question the very existence of rigid policies and programs. Through on-going assessment, it aims at improving the overall effectiveness of a program in carrying out policy objectives: its design, its results, its impact on the beneficiaries and its effectiveness when compared with alternative means of program delivery.

1.13 The most recently internationally approved definition of evaluation, served as the starting point of the Bank vision for evaluation. The definition is as follows:
 

1.14 Within this new evaluation paradigm, Borrowing countries and Bank Management play a key role. They set the objectives and benchmarked of the project. During implementation they evaluate the fulfillment of critical assumptions and realign the benchmarks. At the time of ex-post evaluation they take stock of the results achieved and draw lessons to improve the design of projects and to better their delivery.

1.15 For its part, the Evaluation Office provides an independent assessment of the performance of the Bank and its activities. The information is used to improve decision -making and external reporting on results achieved. The assessment of developmental impact is carried out with participation of the involved. This is done in the spirit of learning not blaming. Evaluation findings and syntheses provide the President and the Board of Executive Directors with credible, timely and impartial information enabling them to:

• assess results, continued relevance and cost-effectiveness of strategies, policies, programs, projects and systems;

• determine the consistency of the programs with Bank policies, strategies and priorities;

• take informed decision in allocating Bank resources;

• determine the adequacy of follow-up action on key evaluation findings.

1.16 The new paradigm is that evaluation has a unique added-value within the controller ship function, as a learning tool. Evaluation ought not to duplicate or be used in lieu of the inspection and audit functions. The systemic approach to evaluation recognizes the singular contribution that Borrowing countries and Managment provide within the continuum of self-evaluation. The mandate of the Evaluation Office is to undertake independent evaluation and oversee the Bank Evaluation System. The challenge and opportunity is to integrate all the components of the evaluation system into a systematic approach for institutional learning, decision making and better performance.

Top

2  The Bank Evaluation System   

Description of the BES

2.1 Until December 1993, the Bank had a two-track evaluation system, comprised of an external evaluation function (External Review and Evaluation Office, ORE) and an internal evaluation function (Operations Evaluation Office, OVE), The first answered directly the board of executive directors and undertook evaluations commissioned by the board. The second, under the Office of the Controller, was responsible for ex-post evaluations of operations and distillation of lessons learned for the improvement of future operations.

2.2 On January 7, 1993, the Board of Executive Directors det the terms of reference of the Directors of the new Evaluation Office() and the objectives of the Bank evaluation system (BES). The new mandate integrated into a single system the functions of evaluations. The official structure of was approved by the Board on March 23, 1994.

2.3 The director is directly answerable to the Board of Executive Directors and reports to the Board of the President of the bank. The director manages the Evaluation Office ans is also responsible for overseeing and enhancing the efficiency and effectiveness of the Bank Evaluation System, for promoting donor cooperation in this area and for addressing evaluation concerns and needs of member countries.

2.4 The Evaluation Office is mandated to:

• Undertake independent and system evaluations of the Bank's strategies, policies, programs, projects and systems, according to the work plan approved by the Board of Executive Directors and the President.

• Disseminate findings of the evaluations so that recommendations for improvement can be used in the design, appraisal and execution of new operations; and

• Oversee and support the enhancement of the effectiveness of the Bank Evaluation System and provide technical support to the various management levels, including IDB Country Offices and the borrowing countries.

2.5 Borrowing countries participate in on-going mid-term and ex-post project evaluations, taking advantage of the technical capacity and support offered by the Bank. In addition, countries collaborate with Bank Management in the preparation of the Project Completion Reports (PCRs)., and may undertake their own ex-post project evaluations (BEPs). Countries also contribute to evaluations conducted by .

2.6 Management performs many evaluation activities such as: portfolio assessment, progress reports, PCRs, mid-term and ex-post evaluations. Management also undertakes reviews at aggregate levels (review of themes, policies, and programs).

2.7 The Board and Management may commission to undertake evaluations at the various operational levels of the Bank (strategies, policies, programs, projects and systems). Taking into account requests made by the Board of Executive Directors ad evaluation activities contained in the Business Plants prepared by Management, develops its work plan proposal. 's work plan is approved by the Board of Executive Directors an by the President. Activities related to the improvement of the BES (dissemination, training, methodological support, etc) are identified in close consultation with Management.

2.8 There are three key evaluation fora within the Bank. The Evaluation Coordinators Group chaired by the Director of is a consultative inter-departmental working group. The Audit and Evaluation Committee chaired by the Executive Vice President is the Senior Management Committee responsible for matters related to evaluation. The Board Matters and Evaluation Committee of the Board is charged with the consideration of evaluation activities.

2.9 The objective of the Bank evaluation system (BES) is to use evaluation as a tool for institutional learning in support of management decision-making and as an instrument for systematic assessment of Bank development policies, the results of Bank-financed activities, and related processes. The BES draws on the valuation experience of borrowers, non-borrowing countries, and the Bank's own operations, and seeks to incorporate the lessons learned into improving ongoing activities. In this connection, the Bank also is promoting and supporting in-country capacity building and facilitating cooperation in evaluation activities with other development agencies.

2.10 These changes, involving evaluation activities in more than the traditional ex-post area, make the evaluation system an integral part of the Bank's business processes. The evaluation Office is no longer the only producer of evaluation. Evaluation itself must be part of the whole project cycle: what is planned to happen (ex-ante), what is happening (monitoring and execution) and what has happened (ex-post). Such an approach, however, introduces the need for a BES oversight function in the Evaluation Office and independent evaluation reporting system for the Board and for Management.

2.11 Based on the principles outlined above, during 1994, as systemic description of the BES was developed. Considering the new mandate of , past institutional experience of OVE and ORE, and existing Management/Borrower evaluation activities, a proposal on evaluation-related objectives, activities and responsibilities was made and discussed at meetings of the Evaluation Coordinators Group (ECG). The final version of these proposals will be submitted in 1995 to Board Management for approval. This documents should provide the guiding principles of the new BES.

2.12 A framework for accountability for the main evaluation activities within BES has been developed. It clarifies the responsibilities of the BES stakeholders with regards to evaluation products. This matrix, which follows, shows:

• the importance of horizontal and vertical coordination among the stakeholders;

• the importance of supportive approaches by to the activities of other stakeholders;

• the need for progressiveness and selectivity in establishing future Bank-wide evaluation priorities, given the complexity of the BES and the limited resources available.

• the requirement for an oversight function.

Current status of the BES

2.13 has used the Bank Evaluation System matrix to make some initial observations on the status of the BES for this Annual Report. While the effort being made to improve the portfolio in an effort for which management must be commended, the BES is not yet a system in operational. Hence, greater efforts must be made to make it so.

2.14 Although preliminary in nature and based on empirical observation, 's intention as a culmination of 1994 BES oversight activities, is to provide an initial status of the "system". This serves to take an integrated took of the situation, create a baseline and from there identify the major issues to be faced over the next few years.

2.15 Each section examines the status of key evaluation products. The BEP, done by the borrower, is an ex-post evaluation performed several years after the completion of an operation. The PCR is report written by the Country Office three moths after the last disbursement of the project. Mid-Term and ex-post evaluations are evaluations programmed in the project approval documents, or when the Borrower, Management and Board of Directors, or think than an evaluation would be useful. Project Performance Reviews (PPRs), and Operation Evaluation Reports (OERs) are ex-post evaluations undertaken by . Sector Summaries are evaluation reports produced by , which combine the results of five of six individual PPRs and OERs to make an assessment of the Bank's performance in an operational sector.

2.16 Although the task ahead seems daunting when the BES is viewed in its comprehensive operational integrity, in the light of the results of the Bank's 1994 evaluation activities it can be concluded that substantial progress is being made and that the next few years will result in the evaluation function becoming closer and more useful to operational main streaming and to the decision-making process.

The BEPs  

2.17 The Bank has traditionally encouraged borrowers to undertake their own ex-post evaluation of operations (BEPs). Though the BEPs were not intended to be mandatory, they were often written as a contractual requirement in Bank funded operations. Due to borrowers' limitations with regards t evaluation (institutional capacity, data collection capabilities, shortage of funds and staff) the BEP system has not worked as hoped for.

2.18 In document GN-1826 of January 14, 1994, the Board of Executive Directors was informed by Management, on the basis of a study conducted by OVE, that the Borrowers Ex-post Evaluation System was not achieving its objectives. On March 17, 1993, the Management's Coordination Committee had approved changes to the system, as follows:
 

2.19 As a result of this, it was not possible to establish a base-line for the expected production of BEPs, nor ascertain what had in fact been produced in 1994. During 1994, out of 18 responses received from country Offices,, only two BEPs were produced. There is no central instrument available in the Bank to measure how many BEPs should ideally have been produced. The list of BEPs produced are listed in Appendix 1. will begin to retrieve duplicates of the reports produced and include their electronic version, when available, into the Evaluation Information System.

2.20 There is a need to examine the relevance and incentives of the BEP system in the context of country evaluation capacity building.     

The PCRs  

2.21 's review of Project Completion Reports (PCRs) prepared by Management confirmed findings similar to those made in Annual Reports on Operations Evaluation (OVE) produced by the Operations Evaluations Office, from 1982-1992. PCRs under the old and new guidelines were found to be largely descriptive, not analytical and of uneven quality. As suggested in the OVE's 1992 report, the improvements south in the quality of PCRs and the feedback y Project Teams and the former Country Teams has not occurred in the manner expected. On January 14, 1994, Management informed the Board of Executive Directors that the Pcr system was not achieving its objectives, was not an integral pat of the Bank's project cycle and provided little by way of lesson-learning information.

2.22 To better meet the objectives of the PCR, a new system had been designed and approved by Management and had come into effect in January 1992. The principal characteristics of the new system are:
 

2.23 's review covered 40 Project Completion Reports prepared under the 1992 guidelines and received by during the 18-month period from January 1,1993 to June 30, 1994. The PCRs received by covered 29 loans, 4 sector loans, 3 small projects, and 3 technical cooperation operations. These PCRs were reviewed for their quality and content. The degree of compliance for to timely preparation of PCRs was difficult to measure exactly because there is no central tracking system. Appendix 2 lists the PCRs that were reviewed.

2.24 The overall quality of PCRs was poor -- even though "good" PCRs show an improvement over those prepared under the old format. In the majority of cases, PCRs were inconsistent, lacking necessary basic data or contained significant contradictions between text and data. PCRs often were ambiguous and did not proved a clear description or explanation of the operation's implementation and related factors, even on fundamental issues such as loan cancellations.

2.25 In general, PCRs lacked the critical analysis sought by the 1991 guidelines. As it was the case in the past, PCRs continue to measure the accomplishment of objectives solely by focussing on realized physical targets and disbursements and in many cases, did not measure outcomes against those originally planned.

2.26 The review done by reveals that the guidelines of 1992 were not Being applied by Bank stage, nor was the spirit of the guidelines followed. Recognizing the importance of PCRs in their role to provide lessons learned, management welcomed 's recommendation that a Task Force be set up to expeditiously diagnose the causes creating the PCR situation and recommend alternatives to create incentives for the improvement of the PCR system.

2.27 Issues to be studied by the Task Force will proved useful insights explaining the perceived the relevance of PCRs withing the Bank and proved leads for the improvement of their content and use. Some of the issues are:

• Use of PCRs in the preparation of new operations

• Priority given to the preparation of the PCRs:

• Ownership and accountability for PCRs

• Adjustment of the Guidelines following the reorganization of the Bank,

• Commitment of Management to the sue of PCRs

2.28 it is in the Bank's interest to clarify the relevance of the PCR product and process, and make tem an important component of the Bank Evaluation System.   

The Mid-term/ex-post evaluations  

2.29 More and more frequently, mid-term, end-of-phase and ex-post evaluations are included in the approval documents submitted to the Board of Executive Directors. The funding of these evaluations are part of the operations budget submitted for Board approval. At times Executive Directors ask for an evaluation to be conducted.

2.30 At present, there is no central system available to track these evaluations. There is no record made of Executive Directors requesting for an evaluation, unless such a request is recorded in the minutes of the Committee of the Whole. It was impossible for to obtain a list of the evaluations that would normally be due in 1994. The business plans of the operational departments made no reference to such evaluations. There is no system to manage these bank-wide, nor exist in the manuals of the Bank. There are no norms and standards set for them. There is no mechanism for their review or for ensuring useful feedback as part of the Bank learning processes.

2.31 In the view of , the mid-term, end-of-phase and ex-post evaluation which are included in the operations of the Bank need to receive higher priority and support. From an investment point of view, they respond to a genuine need expressed by the borrower. Their timely production and review will yield short term benefits to the borrower by allowing adjustments to be made to rectify the course of the project, thus ensuring greater positive results. Mid-term evaluations can be efficient means for participatory management and greater ownership of the project.

2.32 For 1994, the Country Offices reported to that 45 Mid-Term reviews have been completed. The list may be found in Appendix 1. As of 1995, will attempt to log these evaluations and report findings to the Board of Executive Directors. With regards to the evaluation requested by Executive Directors, will maintain a record to facilitate follow-up in the future.

PPRs and OERs

2.33 Prior to the reorganization of the evaluation function, the internal evaluation system entailed five components: BEPs, PCRs, Project Performance Reports (PPRs), Operations Evaluation Reports (OERs) and Sector Summaries. PPRs and OERs were prepared for a sample of projects. OERs are ex-post evaluation studies which contain an in-depth analysis of the design, execution, financial, institutional and economics aspects of the projects as well as an assessment of their socioeconomic impacts. PPRs are similar to OERs. The main difference is that PPRs do not normally include a socioeconomic impact analysis of the project. Usually PPRs are prepared two or three years after the project whereas OERs are done five or seven years after the project is terminated.

2.34 Improving the Bank's formal ex-post evaluation will need careful consideration in the future. This type of evaluation is important not only for its impact on strategy and policy issues but also because of their use for country programming and project design recommendations.

2.35 In 1994, the selection of projects which were reviewed by means of a PPR, was based on their usefulness as sample for the thematic and sector evaluation under way. No OER was produced in 1994.   

Evaluability of the operations approved in 1994  

2.36 The Evaluation Office conducted a review to determine the "evaluability" of the 40 Loan Proposals submitted to the Board during 1994. The review was not directed at the quality of the financial, economic, and other technical analyses. This exercise was only limited to the review of the documents presented for approval by the board of Executive Directors. These documents normally serve as authorization reference for the preparation of the contracts with borrowers and beneficiaries.

2.37 A contributing factor to delays in project execution, and later on, disagreements during ex-post evaluation exercises, stems from the lack of precision as to what the developmental objectives of the project are, and how to measure success. In the documents submitted to the Board 53% of the projects had clearly enunciated pre-set standards of success. 40% of project proposals had explicit ex-ante benchmarks, target figures or indicators to determine if and when the operation will attain its development objectives. Compared to documents submitted a few years ago, these results indicate a new trend for which Management is commended for.

2.38 Another positive finding was tat 64% of the project proposals provided a description of the conditions (physical, economic, social and institutional) prior to the execution of the project. Obviously, this is a useful baseline against which end-results may be compared.

2.39 Other important contributing factors to execution difficulties are those related to institutional weaknesses and unforeseeable events. A very small percentage of the proposals documents contained an explicit description of the beneficiary groups, institutional and organizations that could positively or negatively affect project execution. Few project proposals explained hose events that are outside the direct control of project management and that could frustrate of significantly delay achievement of the project objectives.

2.40 Noteworthy is the fact 30% of the project proposals indicated that an ex-post evaluation was required by the borrowers. A growing trend which needs to be arranged by the Bank.   

Success rating of operations  

2.41 Viewed from the systemic perspective that 's oversight responsibility mandates, the Bank Evaluation System has in place the components which should support the Bank to reach the critical performance and sustain ability goals to TAPOMA and IDM-8. The challenge is to ensure that a systematic and qualitative use of these components occurs in an integrated fashion.

2.42 Until recently, the only systematic success rating portfolio management effort started reporting system that measured project activities against the timeliness of their disbursements schedule. In 1994, there was an indication that serious measures are being taken by the Bank to increase the success rate of the operations. The portfolio management assessment effort started in 1994, despite its initial systemic shortcomings, has created a very supportive momentum in favor of the paradigm shift toward managing fro results. No doubt, this ins only the beginning of a process which will necessitate much commitment from all stakeholders.

Top

3  Enhancement of the Bank Evaluation System   

New Board/Management relations

 
3.1 In October of 1993 the Report of the Task Force on Portfolio Management for the IDB (TAPOMA) recommended that "the Board and Management should review their respective roles in operational matters, including project approval and portfolio management". On June, 22, 1994 the Board of Executive Directors approved the report submitted by the Working Group on Board-Management Relations (Document GN-1838).

3.2 Over the years, the Board has been concerned with the decline in the developmental and sectoral policy-making role of the Board, as well as its lack of consideration of overall country programs. The Working Group concentrated on ways to facilitate the Board's dialogue with Management by focussing less on detailed consideration of individual loans and technical assistance operations, and concentrating the Board's attention on:
 

3.3 Many of the recommendations approved by the Board, have a significant impact on the Bank Evaluation System. The Board has generated a greater expectation for services to be received from the evaluation function. They are the following:
 

•  A diagnosis of the current country situation (national, regional and global context), including country risk assessment,

• An assessment of the current portfolio (based on the ex-ante criteria set at the project approval stage),

• An assessment of the activities of other international development organizations in that country, (which have an impact on the Bank's portfolio) and of the Bank's comparative advantages,

• The upcoming (about 12 months) current pipeline of loans, small projects and technical cooperation operations for the particular country, which is to include a concise description of each operation showing the logical links between the operation and the Bank's policies and country strategy.

3.4 On June 29, 1994, the Board of Executive Directors approved document DR-304-5 setting new terms of reference for the committees of the Board. Very significant changes were made which affect the evaluation function. They are the following:

3.5 The intent of the Executive Directors is to ensure consistency between the strategic, policy program and project levels and to give due consideration to each of these levels of management. There is an expectation that the evaluation system has to serve at all these levels.

3.6 Current efforts to implement the recommendation on performance indicators of the Report of the Task Force on Portfolio Management (TAPOMA) will reinforce efforts at integrating the BES as an operating system. The introduction of benchmarks and means of verification at project design level, and the adoption of monitoring principles will improve the possibility of more successful country portfolio evaluation exercises

3.7 The TAPOMA report recommended that Management submit to the Board of Executive Directors annual portfolio management reports based on country portfolio assessments. According to TAPOMA, these reports should
 

3.8 The TAPOMA report recommended that appropriate indicators should be developed at the time of appraisal and used for tracking project progress in meeting development objectives. In that regard, the Bank's approach to portfolio management was an excellent innovation. To further the effectiveness of this exercise, important methodological enhancements to the project rating and information systems of the Bank are required to address more directly the development impact of projects. The legacy from the past is that many projects still active in the Bank's pipeline contain neither development objectives nor indicators to show how the development success of a project can be measured, monitored or evaluated. The First Annual Report to the Board of Executive Directors on Portfolio Management identified the need to review the experience of bilateral and other international organizations in the use of methodological approaches for assessing the achievement of development objectives.

3.9 The 1994 methodological approach to portfolio assessment was applied on a pilot program basis to all the projects in the portfolio-- in the transportation, education and sanitation sectors. This new approach allowed to better measure and report project potential impact on the development process. The development objective ratings are based on judgements by sector specialists in the country offices about the expected development performance of projects in different stages of implementation. This was the first time that the Bank staff has attempted to systematically classify projects by their likelihood of achieving development objectives. Being the first time that such an approach was applied, there was evidence that Bank staff did not consistently understand and follow the instructions for the preparation of the portfolio assessment. Nevertheless, this is a laudable effort of learning by doing. Management efforts warrant commendation to the methodological progress achieved this year.

3.10 The results of the Portfolio management assessment showed that the Bank's portfolio of loan in execution as of December 321, 1994 was 352 projects (444 loans) with current commitments totaling US$26.7 billion. Based on the Bank's system of reviewing the execution status of projects, the portfolio is in good shape. Only 12.8 percent of the projects were experiencing major obstacles to successful implementation and disbursements. Because 1994 was the first year to attempt to assess the likelihood of achieving development results, the results of the pilot program (77 projects) covering three sectors should be interpreted with caution. The results suggest a good performance. A significant percentage (about 78%) of the projects covered by to program are expected to achieve most of their major development objectives. Since the pilot program was limited to just three sectors, it would be inadvisable to draw conclusions about the qualitative aspects of the active portfolio from this sample.

Establishment of ex-ante criteria for success  

3.11 In 1994, during meetings of the Committee of the Whole, Executive Directors have frequently reiterated the need to establish ex-ante criteria of success. These criteria help to define the expected development results, can be used as yardsticks for monitoring progress during the implementation and as indicators for assessing the results achieved.

3.12 In 's view, the lack of clarity of a project objectives and data limitations of the project information system are the main hindrance to an effective evaluation system. Hence is very supportive of Management's action plan, based on the 1994 portfolio management report to require project teams to identify development objectives and verifiable indicators and explicitly assess project risk factors for all new projects and for all projects in the Bank's portfolio for which disbursement have not yet begun.

3.13 has collaborated with the Office of the Executive Vice-President and to develop a simple performance benchmarks framework fro structuring projects. The Executive Vice-President has asked Bank staff to use these benchmarks. These ex-ante benchmarks will enhance the quality of project design and provide greater clarity on expected outcome. From a systemic perspective, this is the cornerstone of the new Bank Evaluation System.

3.14 As can be seen in Appendix 4, the table entitled Project Structure: Performance Benchmarks is based on concepts borrowed from the Logical Framework Analysis (L.A.) approach. The L.A. approach is well known and widely used by development professionals in bilateral and multilateral development organizations.   

Policy development approaches  

3.15 While it can be said that the logical framework approach constitutes the cornerstone of the BES, the strategy and policy level constitutes the keystone of the institutional system. While the logical framework provides the tool for measuring effects at project level, clear concepts of strategies and policies determine what we are measuring through bench marking and monitoring at the aggregate level.

3.16 The basic approach involves the development of means and instruments to expand from the present implementation focus resting primarily on operational transaction targets, to strategic management oriented toward developmental outcomes or results. Since at present there is little in-house experience, this approach must be developed gradually, building upon initial efforts through country programs and portfolio management.

3.17 Three concurrent endeavors need to be envisaged. The first is to foster that the main IDB-8 strategic priorities as articulated through the Mid-term Operational Plans, related business plans, and strategic frameworks for new areas of intervention with intermediate level means of measurement and follow-up of implementation. Second is the formulation of country programs as means to strengthen the nexus with broader strategic goals in accord with country circumstances. This should underscore the dimension of development goals in key areas of Bank support, in conjunction with explicit indicators of desired results or outcomes. Lessons from a first country program evaluation (CPE), to be conducted in 1995, should also be instrumental in designing future benchmarks at the country program level.

3.18 A third undertaking is to develop a capability to monitor progress in implementing strategic objectives so that the Board and the Management can assess performance results as a basis for reporting achievements and supporting decision-making on strategies and policies. Conceptual and design work is being initiated, with pilot work planned in 1995. This is a multi-year effort, with a developmental phase scheduled so that strategic reporting can come on stream for mandated reporting of IDB-8 strategic objectives.

3.19 The Board of Directors decided (document GN-1838-1), early in 1995, to create a meaningful policy framework, following the IDB-8 strategic framework, for the purpose of clarifying, updating, and formulating operational policies into a coherent vision. The initial review "Policy development Approaches and Options in the context of IDB-8" was a first step to support this decision. Work is underway to develop a common agenda, between the Executive Board and Management, leading to a policy environment more expressly linked to the IDB-8" strategic priorities. This has implications in terms of updating existing policies and ascertaining their consistency and/or present relevance; identifying those policy changes needs for strategic reasons; and designing new policies where gaps may be perceived.

3.20 BES activities to support the policy agenda involve:
 

Evaluation activities included in the business plans  

3.21 Before the recent Bank reorganization, the preparation of business plans by Bank Departments and offices was not standard practice. Therefore, was not able to prepare a list of the evaluation activities undertaken by the various entities of the Bank. In the past, the paradigm within the Bank was that operational departments had the responsibilities for the production of PCRs and BEPs, and beyond that, evaluation was the business of OVE and ORE. Through its numerous representations, began to disseminate the concept that evaluation is a responsibility which is shared with all levels of management. Hence, for the 1995 business plans, some Departments included evaluation activities.

3.22 In 1994, only one evaluation was done by Management to be reported to the Board of Executive Directors. It was the evaluation of the new Project Preparation Facility, conducted by the Office of the Controller.   

Mid-term restructuring of projects  

3.23 's preliminary review of the BES at monitoring levels during execution shows an uneven situation. Certain types of projects such as sector loans and time-slices have specific review and decision-making activities introduced as part of a project's execution. Other projects, in general show an unsystematic approach to mid-term review as a basis for monitoring project performance quality.

3.24 On the basis of a 1994 & EMD/DEV training experience, in which four projects in execution were subjected to a log frame exercise, it was proven beneficial to create improved monitoring conditions even at this stage in the project cycle. Better monitoring would appear to be feasible if projects are designed in function of execution and performance, and appropriate protocols and methods are introduced not only at start-up but also at project execution levels. A selective approach to the latter, based on the completion of the current log frame training exercises, would permit the Bank to introduce improved monitoring in the short-term. 

Training courses

3.25 Training in logical framework approaches to project design, stakeholder and project execution monitoring, had been initiated by PRA/PAO in October/November of 1993 in response to Recommendation IX of the 1991 Representatives Report. This activity was also supported by TAPOMA.

3.26 Given the importance of training in these methods, as a basis for improving the BES, these functions were transferred to during the Bank's restructuring last September. While PRA/PAO staff engaged in these activities were mostly transferred to other functions, training activities were continued in 1994 thanks to the use of consultants funded out of generous contributions from the and EMD/DEV budgets.

3.27 During the period October 1993 to November 1994 a total of 559 professionals were trained (see following table). Fifteen basic logical framework courses were given for Country Offices and two at Headquarters, at which 502 professionals (of which 48 headquarters staff, 143 country staff, and 311 national counterparts were trained). An additional 34 professionals received training in the use of the log frame at project monitoring and execution level of which 25 country staff, one headquarters professional and 8 counterparts (project managers and deputy managers). Finally, 33 professionals received training in the use of the log frame method to improve stakeholder analysis of which 16 country staff and 7 counterpart professionals.

3.28 On the basis of this experience, work plans were prepared for 1995 designed to train another 550 professionals, thus ensuring a trained critical mass for future applications of these methods which should permit the production of projects in the light of the requirement set in document GN-1837-1 A Criteria and norms for Board/Management Relations@. The basic log frame table is shown in Appendix 5, as an example of what the training exercises are directed toward at project formulation level and as an instrument for project monitoring evaluation.   

Evaluation handbook  

3.29 An Evaluation Resource Book is being prepared to be used for training and dissemination purposes as of 1995. It will consist of three parts:
 

3.30 The intention is to use this resource book to promote a Bank-wide standard for project evaluation.

Top

4  Evo's activities   

Rationale underpinning the 1994 work plan  

4.1 On may 18, 1994, the President and the Board of Executive Directors approved the work plan. The work plan had four priorities:

• completion of evaluation studies which had been initiated by the former evaluation offices (ORE and OVE).

• redesign and start up of the bank Evaluation System together with a review of each component for their revitalization.

• the promotion of in-country capacity in evaluation,

• the organization of the Evaluation Office, serving both as an independent evaluation office and as center of expertise in support of the Bank Evaluation System.

4.2 This chapter provides a description of the activities conducted by in fulfillment of its strategy for 1994.

Completion of ongoing ORE and OVE activities

4.3 The 1994 Work Plan include twelve evaluation projects which had been commissioned by the Board or Management in 1993, which inherited from ORE and OVE.

CORE 1994 WORK PLAN COMPLETION DATES

start-up and coordination of BES

4.4 The activity concerning Management (Project 13), includes such activities as establishing the organization and recruiting new staff. These are, of course, necessary start-up activities that should need only occasional updating in future work plans. The activities of the Advisory Panel (Project 14) provided valuable expert advice on the establishment of and BES. The International Evaluation Network (Project 15) was designed to keep informed on current evaluation activities and findings.

4.5 Corporate Evaluation Services activities also received a high priority in 1994, because they provide the backbone of a well-organized and systematic evaluation system. The creation of an Evaluation Network within IDB (Project 16) will become less time consuming in later years. Training in Evaluation (Project 17) is another area which required considerable start-up activity, but which should later become routinized within Bank operations. On the other hand, Evaluation Methodology Guides (Project 18), will undoubtedly continue to be a dynamic area of activity, as will Dissemination Activities (Project 19).

4.6 The development and implementation of NET (Project 20), an integrated information system which includes existing data bases with lessons and best practices derived from evaluation reports produced by IDB and by other agencies, will be a valuable tool to access information and to learn lessons from other institutions. When this information system is completed, the Board, Management, Bank staff, Borrowers, and the Evaluation Community, will be linked together in a common information network.

4.7 Finally, initiated activities supporting the new thrust of promoting In-country Evaluators (Project 21). In that regards, the Follow-up to the Quito Seminar on Evaluation (Project 22), and the Regional Technical Cooperation on Evaluations (Project 23) were undertaken during 1994.

4.8 The work plan also includes activities for the review f the quality of PCRs (project 24), as well as the preparation of project ex-post evaluation, PPRs (Project 25).   

Coverage by 's evaluation studies

4.9 In the following table, the shaded areas show the thematic coverage of =s 1994 evaluation reports within the major development sectors and within major areas of summative evaluation.   

Year-end Situation of Consideration of Evaluation Reports by Bank Committees

4.10 Processing evaluation reports through the various level of committees takes a significant amount of time. Here are the four main steps:
 

• First, the evaluator submits the draft to a peer review for a methodological review of the rigor of the approach and for confirmation of the validity of the information used for the analysis. The report is then submitted for internal review by management.

• Second, a revised draft is submitted to the Evaluation Coordinators Group (ECG). This facilitates a Bank-wide exchange on the findings and their systemic and policy implications. ECG discusses the relevance of the recommendations and their practicability for implementation.

• Third, the formal draft report is submitted for consideration by the Audit and Evaluation Committee and by the Board Matters and Evaluation Committee.

• Ultimately, the final report is submitted to the President and the Board of Executive Directors for formal consideration, leading to approval of recommendations and Management=s commitment to implement them.

4.11 This review process by committees may appear slow but it yields high benefits. Namely, through such a process, evaluation is used as a learning tool. The review process of the report is institutionally and beneficially as important as the formal approval of the report per se. Indeed, committee discussions on an evaluation report create an opportunity for institutional sharing on findings and prepare the ground for buy-in by those who will have to champion the required change.

Performance with regards to evaluation activities

Completed studies

4.12 In 1994, 13 draft reports were prepared, 9 were reviewed by ECG and 2 have been considered by Management =s Audit and Evaluation Committee, and 5 by the Board Matters and Evaluation Committee. The list may be found in Appendix 3. These reports will be submitted to the Board of Executive Directors during the first quarter of 1995. Executive Directors during the first quarter 1995. The chart on the following page summarizes the status of each evaluation study, as of December 1994.

On-going studies

Primary Education (Project 1)

4.13 The need to expand lending for basic education as mandated by the Eighth Replenishment, together with the Bank's focus on project quality, led to the design of this evaluation, which aimed at learning lessons from experiences found in Bank-financed primary education projects that address improved learning as the ultimate project goal.

4.14 The evaluation focuses on identifying the best conditions for learning, in the context of Bank financed projects. It is carrying out a review of ongoing projects, approved in the recent past, that address issues of education quality and contain components to improve learning. After meeting with project execution professionals, the evaluation team focused its research on consultations with stakeholders (teachers, parents, children) to find out their views on learning, and assess how such views compare with state of the art knowledge on learning, and with Bank financed components. Two countries were visited in 1994, and two more are scheduled for January 1995. The final report with conclusions and recommendations is expected in mid-1995.

Reform of Development Banks in the Financial Sector Loans (Project 2.3)

4.15 This evaluation, which focuses on country case studies of Bolivia, Peru, Uruguay and Venezuela, addresses the reform of the financial sector with special emphasis on the development banking system, and its implication for IDB activities in the sector. The reforms supported by these programs basically aim at reducing public sector intervention, and restructuring, selling, or liquidating public sector financial institutions.

Public Sector Reform (Project 25.4/25.5)

4.16 Two Project Performance Reviews (PPRs) of sector loan operations related to public sector reform were initiated in 1994, namely the Public Sector Reform Loan of Argentina and the Financial Sector Loan of Venezuela. These studies, in addition to two other PPRs to be finalized in 1995, will help prepare an evaluation summary of this sector by the end of that year.

Environmental Management of a Multi-purpose Water Resource Development Project in Ecuador

(A Project Performance Review) (Project 25.6)

4.17 is completing an assessment of the rationale, effectiveness, efficiency and Sustain ability of the environmental management and protection component of a multi-purpose water resource development project. The evaluation highlights relevant issues for the borrowing government, the executing agency and the Bank regarding watershed management in the area of influence in particular, and other watershed protection and management projects more generally. This evaluation is the first in a series on the Bank's experience, to date, with renewable natural resource management, of which watershed management, plays a large role.   

Performance with regards to Bank-wide support

Dissemination Activities (Project 19)

4.18 Using evaluation reports prepared before 1994, six dissemination activities occurred during the year. They are described hereafter. also held numerous sessions for the purpose of disseminating the new evaluation paradigm and of building a broader consensus on the vision for the Bank Evaluation System.

Agricultural Sector Reform (Project 7)

4.19 In July 1994, organized a Seminar with IDB staff, to discuss implementation issues and policy implications of IDB Sector Loans in the Agricultural Sector of Guyana, Honduras, Jamaica and Mexico. To facilitate that discussion, a Consultant Report was presented, based on previous evaluation work undertaken by the Operations Evaluations Office in 1993.

Environmental Impacts of Rural Roads: Methods and Cases (Project 8.2)

4.20 The cases are from ex-post evaluations of completed projects. The working paper summarizes the experience of , in assessing ex-post environmental impacts of rural road projects from a pragmatic viewpoint, including ease of implementing the assessment methodology, information requirements, reliability, timeliness and usefulness to decision-makers.

Health Sector Workshop (Project Profile #4)

4.21 In September 1994, held a seminar to discuss a draft Health Sector Summary Report. Interested members from relevant Bank departments attended the seminar to make comments on the draft and to add their own experience in Bank health projects to the Report's findings. Also discussed was the general status of the health sector throughout the Region.

Seminar on Women in Development: Rural Case Study (Project 5.1)

4.22 A seminar was held in June of 1994 to discuss the findings and conclusions of this study. Participants included a wide representation of Bank staff, mainly from the agricultural and women in development offices. The discussion focussed on the importance of recognizing the needs, resources and responsibilities of men and women from the very early stage of project identification and design in order to improve effectiveness and results

Workshop on Rural Financial Series for the Rural Poor (Project 6)

4.23 In December, and the Social Programs and Sustainable Development Department sponsored this workshop to review and discuss possible strategies to reach small and micro rural producers with credit and other financial services. Presentations included a background paper prepared for the event, and three cases of different strategies on ongoing projects. This workshop was intended as a first step of a systematic review, that is expected to result in consideration of a modified IDB Policy in rural finance.

Infrastructure Maintenance Workshop (Project 8.1)

4.24 was a member of the technical advisory committee for the Infrastructure Maintenance Workshop which was held on July 25 and 26, 1994 at Headquarters. Although the event was organized by the then Project Analysis Department, was an active participant in the development of the technical agenda as well as in the identification of presenters. The findings from evaluation studies on the maintenance of rural roads, water and sanitation programs, irrigation, and health programs were presented and discussed at different forums of the workshop. The purpose of these forums was to present officials of ministries and enterprises present at the workshop with evaluation findings on causes and effects of poor infrastructure maintenance. This workshop renewed the call for the Bank and its borrowers on the importance of long term maintenance management planning both at the regional and country levels.

Establishment of the Evaluation Network within the Bank (Project 16)

Organizing the Bank Evaluation System

4.25 Using a broad consultative approach and benefitting from a participatory process involving the Evaluation Task Force, prepared a proposal for the Bank Evaluation System. It is summarized in document BE-12 of November 1994. At its meeting of December 12, 1994, the Board Matters and Evaluation Committee agreed to recommend to the Committee of the Whole that it approve the proposal.

4.26 Drafts of procedures for coordinating evaluation document production and processing as well as for defining stakeholder responsibilities and activities in the BES have been prepared by the end of 1994. These procedures should be the object of discussion with Management and the Board early in 1995.

International Evaluation Network (Project 15)

OECD Expert Group on Evaluation

4.27 The Inter-American Development Bank has an Observer status within the Expert Group on Aid Evaluation of the Development Assistance Committee (DAC) of the Organization for Economic Co-operation and Development (OECD); only country delegations have the status of member. Nevertheless the observer status enables to fully participate to the activities of the Group. The heads of evaluation units are the representatives to the Expert Group on Aid Evaluation.

4.28 The agenda of the Expert Group is quite relevant to the preoccupations of the Bank.

• An evaluation report on DAC Women in Development policy and programs of its members was concluded.

• Follow-up activities to the Africa-Asian-Latin American & Caribbean Regional Seminars were reviewed and coordinated.

• The Expert Group launched a DAC evaluation of programs promoting participatory development and good governance.

• The Group also took stock of the state-of-the-art in methodological approaches to country program evaluations. The feasibility of a success rating system was discussed leading to a commitment to attempt to establish criteria. A multi-year work plan was agreed to.

Inter-Agency Working Group on Evaluation

4.29 is a member of the Inter-Agency Working Group on Evaluation (IAWG). This Group brings together under the chairmanship of the United Nations Development Program (UNDP) the Heads of evaluation of thirty two Specialized Agencies of the United Nations and the Heads of evaluation of the Multilateral Development Banks.

4.30 The agenda of IAWG covered evaluation issues that are of interest to the Bank, such as:

Professional Associations

4.31 staff have been active members of the Central American Evaluation Association, of the Canadian Evaluation Society and of the American Evaluation Association. was represented in the annual meetings of these association.

4.32 These associations have decided to unite with the Australian-Asian Evaluation Association and the new European Evaluation Group to host a major conference on international evaluation. This event is scheduled for October 1995 in Vancouver. Some 2000 delegates are expected. is supporting the organizing committee.

Cooperation with international organizations

4.33 In October, co-sponsored with the Inter-American Dialogue (IAD) and the International Development Research Center (IDRC) a seminar on Social Policy Research in Latin America. The objective was to disseminate results of research conducted by a distinguished group of Latin American social scientists. This seminar was organized for the benefit of Bank staff and Washington based organizations involved in Latin American social development.

NET (Project 20)

Evaluation Feedback System (EFS)

4.34 In 1994, began to integrate existing data bases from ORE and OVE into an electronic evaluation text base, which will (I) provide up-to-date information on the evaluation findings of the Bank and other international agencies; (ii) ensure that evaluation information can be fed back into new project and program design; (iii) facilitate the ongoing evaluation activities of the office, and; (iv) offer a frame of reference for the findings and recommendations derived from evaluations by sector, country and subject area, so that a better performance assessment of Bank activities can be made.   

Performance with regards to in-country capacity building

Quito Seminar Follow-up (Project 22)

4.35 A Regional Seminar on Monitoring and Evaluation in Latin America and the Caribbean, jointly sponsored by the DAC and the Inter-American Development Bank was successfully held November 9-13, 1993 in Quito, Ecuador. In attendance were representatives from 24 countries in the region and officials from 13 OECD member countries as well as from the European Community, World Bank and the Pan-American Health Organization. Agreement was reached on the need to strengthen the evaluation function as an integral part of public sector management and as a means to facilitate transparency and accountability in the operations of government.

4.36 During 1994, contributed towards the follow-up to the Quito Seminar by organizing two sub-regional seminars:

• The Caribbean Regional Seminar on Monitoring and Evaluation was held in Barbados, October 6-7 co-sponsored with the Caribbean Development Bank. Eight Caribbean nations, Surinam and Belize participated.

• The Regional Seminar on Monitoring and Evaluation for Central America, Panama and the Dominican Republic was held in San Jose, Costa Rica on November 30 - 2 December. It was co-sponsored with the International Fund for Agricultural Development (FIDA) and organized by the Central American Evaluation Association.

In-Country Evaluation (Project 21)

4.37 During the Annual Meeting of the Bank in Guadalajara, Mexico, several country delegations indicated an interest in receiving the support of the Bank in evaluation capacity building. The requests covered a wide range of activities from the provision of technical support to the supply of methodological evaluation tools.

4.38 The Minister of Planning and the Minister of Finance of Costa Rica visited the Bank last June and asked that lead a mission to Costa Rica to assist them in the development of a National Performance Evaluation System (NPES). The mission which occurred soon it was done with the cooperation of the Head of evaluation of the Planning Ministry of Colombia, of the Deputy Head of Evaluation of USAID and a Senior Evaluator of the Treasury Board of Canada. Working in a very participatory fashion with the staff of the Planning Ministry of Costa Rica, the mission designed the NPES and drafted an administrative decree which was put into effect a few months after. coordinated the development of a plan of action for the short term which included providing further consultancies on the organizational and institutional design of the evaluation system. also assisted in the development of staff training.

4.39 In addition, was involved in supporting the launching of a Masters Program in Evaluation at the University of Costa Rica. This program aims at training local expertise in evaluation to be used by the Government and be available for the organizations funding international development activities.

4.40 has also supported activities enhancing the institutional strengthening of the Central American Evaluation Association (CAES). This Association facilitates the participation of civil societies in the evaluation debate. The contracting by of the CAES for the organization of the Regional Seminar on Monitoring and Evaluation held in San Jose, has given an impetus to the CAES.

4.41 In response to requests from the Caribbean countries, negotiated an administrative arrangement with the Caribbean Development Bank. This arrangement is also a follow-up to the Regional Seminar on Monitoring and Evaluation held in Barbados. The activities included in the agreement involve the participation of the University of West Indies, of the University of Pittsburgh, of the George Washington University and the University of Maryland. The Government of Chile is also supporting this initiative.

Regional Technical Cooperation Project: Evaluation Methodology of Social Program (Project 23)

4.42 During 1994 headed the project team that prepared a regional Technical Cooperation proposal for a grant of $500,000 to assist Latin American and Caribbean member countries in the evaluation of their social reform programs. This operation, which was approved in January of this year, will support the evaluation of 10 projects and programs in the reform of health and basic education, and community development for groups at risk, such as children, women and indigenous groups. The project will identify areas of success and difficulties, develop methodologies for execution, monitoring and evaluation, and disseminate the results.

4.43 The project is to be executed by a research center from a borrowing member country, to be selected jointly by IDB and the International Development Research Center (IDRC) which also contributes $500,000 to the project. After approval by the Board of Directors, the project will be supervised by the Social Programs and Sustainable Development Department. The results of these evaluations will be useful for the Bank to learn lessons on best practices for the preparation of future similar projects, especially from the standpoint of new innovative activities/programs in the social area and the related IDB-8 priorities.   

Performance with regards to administrative considerations

4.44 The Organizational Structure of the Evaluation Office was approved by the Board of Directors of the Bank at its meeting of March 23, 1994 (see GA-151-1 of April 4, 1994) The structure allocated a total of 28 staff positions to the office including the Director, Deputy Director, and the three Principal Evaluation Officers who would lead the Evaluation "Teams" for Strategy and Policy Evaluation, Program and Project Evaluation, and Corporate Evaluation Services. The Board also approved 's proposal that some evaluation officers might be placed in Country Offices (as regional evaluators) provided that this could be achieved within 's budget ceiling. The organizational chart may be found in Appendix 6.

4.45 The Director of , Mr. Jean S. Quesnel and the Deputy Director, Mr. Richard Fletcher assumed their duties in January 1994. The three Principal officers arrived during the year: Jean Michel Houde, formerly Alternate Special Representative, Europe (August); Brian Thomson, formerly Chief, Project Advisory Office of the Project Analysis Department (September); and Paul Köhling, formerly Country Representative of the Bank in Chile (December).

4.46 In October, vacancy announcements were issued on the basis of internal/external competition for positions of Junior Evaluation Officers, Evaluation Officers and Senior Evaluation Officers; and on the basis of internal competition for the position of Administrative Officer. Six new professional staff members were hired as a result of these competitions.

4.47 's 1994 Work Plan was approved by the President and the Board in May 1994. The Plan comprised 27 Projects/Activities with a total budget of $3.57 million. Actual expenses for the year totaled $3.46 million or 97% of the budgeted amount (see Table 1). estimates that of the 1994 expenses, 48% were spent on evaluation reporting, 23% on corporate evaluation support services (such as training and dissemination), 17% on in-country capacity building and 12% on administration of the office.

4.48 The resource allocations for the 1994 Work Plan were not to be considered a reflection of future annual allocations. Rather, they allowed, for 1994, a much heavier investment to the establishment of the Evaluation Office and a one-time effort to define relationships between stakeholders within the Bank evaluation system. Over the next years, fewer activities, and more resources will be dedicated for the production of ex-post and impact evaluation activities.

E N D N O T E S

1. TAPOMA is the report of the task Force on Portfolio Management entitled Managing for Effective Development published October 1993 by the Inter-American Development Bank.

2. The Wapenhans Report by the World Bank's Portfolio Management Task Force, the TAPOMA Report on the activities of the IDB, the Schulz Report by the Asian Development Bank and the Knox Report by the African Development Bank, all point to the necessity for the Multilateral Development Banks to evolve from a control and approval culture to a result-based execution management culture.

3. This definition was approved by the Ministerial level of the Development Assistance Committee (DAC) of the Organization for Economic Co-operation and Development (OECD), in December 1991, as part of the Principles for Evaluation of Development Assistance.

4. A description of the evaluation system under the responsibility of the Office of External Review and Evaluation may be found in document AB-714 of September 1979.

5. The description of this internal evaluation system was described in the Operational Manual in Chapter OP-305.

6. Document RE-185-2 of 7 January provides the terms of reference for the Director of the Evaluation Office.

7. The official structure of the Evaluation Office may be found in document GA-151-1.

8. Reference is made to document GA-151-1 of April 4, 1995 entitled Organization of the Evaluation Office approved by the Board of Executive Directors.

9. See document BE-19 of March 17, 1995 entitled "Systemic implications of the review of project completion reports received in 1993 and the first half of 1994"and the report of the "Review of project completion reports received 1993/mid-1994: A Study of Content and Quality" prepared by the Evaluation Office.

10. Management presentation to the Board of Executive Directors of the status of the Project Completion Reports system and on the Borrower's Ex-Post Evaluation system may be found in document GN-1826

11. The evaluation conducted by the Office of the Controller was submitted to the Board of Executive Directors as document GN-1865 on March 17, 1995.

12. The project number refers to the Project Profiles which were submitted to the President and the Board of Executive Directors for approval, in document RE-193.

13. Once the proposal for the Bank Evaluation System was prepared the Evaluation Task Force became the Evaluation Coordinators Group.

Top