2. Findings and Lessons from Country Programs

A. Introduction

2.1 The Bank's overall economic and social development strategy for the borrowing member countries of the region is articulated in the Agreement Establishing the Inter-American Development Bank, and in the reports on the General Increases in Resources of the Bank (i.e., the "Replenishment documents"). The latter identify the principal areas where the Bank should allocate its financial and technical resources during the respective Replenishment period.

2.2 The Bank defines its development strategy and programs its activities through the programming documents: Country Papers (CPs) - formerly known as Country Program Papers (CPPs) - the Programming Mission Reports, and the Profiles I. The purpose of the programming documents is to articulate the Bank's strategy to achieve, within a programming period and with a set amount of resources, specific objectives that will contribute to the IDB goal of "accelerating economic and social development" in the borrowing member country.

2.3 The evaluation of the Bank's programs and strategies is an important element of the mandate of the Evaluation Office. This mandate has been articulated in Annex I of the Provisions of the Agreement Establishing the Bank, and in the Board of Governors' Report on the Eighth General Increase in the Resources of the Inter-American Development Bank. The assessment of the design, implementation, and results of the country-specific programs and strategies of the Bank is known as Country Program Evaluation (CPE).

2.4 In 1997 completed the CPEs of the Dominican Republic and of Ecuador. This chapter summarizes their main elements, findings, conclusions, and recommendation.

B. The Case of Ecuador

2.5 's Country Program Evaluation (CPE) for Ecuador, while specific to that country, was part of the process to enhance the effectiveness of the Country Program Cycle and improve the policy dialogue between the country and the Bank. This is the first CPE undertaken by .

2.6 This evaluation was a retrospective review of IDB 's operations in Ecuador over the last decade. Attention was given to the country program's strategic rationale, sustainability and ownership over the long-term. Based on the analysis of this experience, conclusions were drawn, and recommendations made to improve the efficiency and effectiveness of the Bank/Ecuadorean cooperation.

2.7 A conceptual framework of analysis was developed to guide this evaluation. It takes into account the Bank's institutional setting for the country program cycle, and its relationship with Ecuador.

2.8 The evaluation of the achievement of the country program's objectives consisted of a comparative evaluation of the distribution of IDB/Ecuador portfolio, according to strategic sectors and priority areas.

2.9 The objective was to assess to what extent the country programs have achieved replenishment goals. The knowledge of counterpart Ecuadoreans about IDB strategy and policy, and the degree of consultation about the Country Paper and its implementations as the Country Program with local authorities was reviewed, using the results from a stakeholder's survey.

2.10 The coverage concentrated on IDB's traditional type of lending instruments involving investment projects. The evaluation of the outcome of the fast disbursing policy-based loans was not analyzed as they are presently in the disbursement phase. In addition, as part of this experimental and learning phase, a field stakeholder's survey was conducted. The survey results are used to convey client perceptions about Bank performance and operational services, to assess knowledge of IDB procedures, and to obtain recommendations for improvement. This survey sample covered the view of over a hundred Ecuadoreans representing 37 institutions, and 28 projects with a portfolio value equal to US$1,200 millions, or 59% of the loans approved above US$ 1 million in the period 1984 to 1995.

2.11 The report examined the quality and content of program design. This was done through a compa-rison of the content of the Country Paper. The report evaluated IDB's "de facto" institutional strengthening and project execution strategy using execution units and technical cooperation.

2.12 The report reviewed the experience with program delivery through project preparation and negotiation. IDB's operational services were thereby evaluated. This included an analysis of the use of consultants to prepare projects and used material from the stakeholder survey for this purpose. The report examined the issue of conditionality and the possible imposition of unnecessary contractual clauses on IDB clients.

2.13 Finally, the report reviewed retrospectively the Bank/Ecuador portfolio execution performance.

1. Findings

a) On Country Program Objectives

2.14 Overall, about 60% of the IDB/Ecuador investment portfolio (1961- 1994) was concentrated in three major Sectors:

• Agriculture US$1,244 million (34%);
• Transport US$515 million (14%); and
• Energy US$409 million (11%).

2.15 Environment/health and industry/mining each account for approximately US$350 million, or 10%. Education and Urban Development each represent about US$180 million, or 5%.

2.16 In 1990-1994, new patterns appeared in which environment/health increased to 24% and agriculture declined to about 16%-17%. Urban development is irregular, but transportation remains strong at between 14% to 24%. The "other" category, which includes sector lending becomes important and accounts for 25%.

2.17 Ecuador has performed at a level similar to its "peer countries". The total value of the disbursements (D) over the past three and a half decades (1961-1996) was equal to US$2,600 million. Total approvals (A) are equal to US$3,300 million. The resulting D/A ratio was 77%. Using the disbursement/approval ratio (D/A) to measure the efficiency of the absorption of development finance, Ecuador's D/A ratio has generally performed at a level similar to its "peer" C and D country groups. The resulting US$60 million average annual approval-disbursement gap has contributed to the current total of undisbursed resources equal to approximately US$700 million. The source of financing of the IDB/Ecuador portfolio has changed. In the beginning it was 93% concessional. It now consists of 97% from Ordinary Capital.

2.18 IDB disbursements were relatively significant to the Ecuadorean economy. Disbursement of US$175 million in 1994 were equal to: 1.0% of the GDP; 4.4%of the annual government budget; 16% of the government's investment budget; and US$16.30 per capita. Due to the influence of fast-disbursing policy based loans, disbursements in 1995 increased to US $250 million. Disbursements in 1996 were equal to US$160 million; the proportion from policy-based loans was negligible in that year.

2.19 The macroeconomic analysis covering the 1970-1994 period did not demonstrate that IDB disbursements had a significant effect on the variations in Ecuador's capacity to import, which behaved in a variable and uneven manner. To the extent that the growth of the Ecuadorean economy is dependent on its capacity to import, volatility of this capacity can have a negative impact. A negative association was detected between government savings and increases in external financing.

b) The Country Program Design

2.20 The content and potential effectiveness of the Country Paper and Programming Mission Report (PMR) was evaluated for the 1987-1994 period. The evaluation criteria, which has developed in 1997 and adapted to the Ecuadorean context, emphasized strategic and operational objectives, performance indicators, targets or benchmarks, the composition and performance of the portfolio in execution and its linkages to IDB/Ecuador priorities. Institutional capacity and risk were also included. The strategy content of the Country Paper and PMR, as reflected in the quality of the economic analysis, has improved in the documents produced in 1993 and 1994. Previously the Country Paper and PMR emphasized new projects and did not analyze the condition of the current portfolio in terms of an operational strategy with performance indicators to assess the achievement of goals. In general, the Country Paper was not used as a basic planning and strategy instrument to prepare and implement a plan of action based on agreed upon IDB/Ecuadorean goals. It was used largely as a general reference to justify and to modify the content of some projects which were already designated a priority by the country.

2.21 The Country Paper/PMR was not used as the basis to define alternative strategies, to review the current status of measurable achievements and, based on this assessment, define a short, medium and long-term plan of action. The Country Paper documents had a low evaluability content because they did not present measurable goals or establish time limits within which goals were to be met.

2.22 With regard to the priorities set in the replenishment, no systematic attempt was observed in the Country Papers to translate the priorities of the replenishments into specific guidelines. A better definition of what the IDB and Ecuador set out to achieve, in particular at the sectoral level, would have made the effects of the Country Papers planning exercise more susceptible to being measured for Ecuador. Clear and measurable objectives and specific monitoring and evaluation variables could contribute to the effectiveness and efficiency of operations in Ecuador. The Country Program strategy did not fully take into account the experience and potential advantage of IDB's intimate knowledge of its borrower clients, the sector they represented, or the IDB's credibility as a long-term partner to support change. As a consequence they did not contribute as much as they could have to the efficacy and efficiency of the Ecuadorean Country Program.

2.23 The "de facto" institutional strategy employed used two instruments: executing units and technical assistance. These were essentially short-term solutions to facilitate project execution. In those earlier years, IDB as a whole, was not viewed as setting a high priority on institutional strengthening or having a long-term plan or strategy.

c) On Country Program Delivery

2.24 To evaluate program delivery, which involves project preparation and negotiation, the stakeholders' survey asked questions about the performance of different aspects of the project teams, consultants, IDB training and the negotiation process.

2.25 The technical quality of IDB Project Teams was viewed favorably by clients, and the role of the legal advisors on both sides was seen as positive. But three-fourths of the Ecuadorean respondents felt that they had been treated autocratically during the negotiation process and that unnecessary contractual conditions were included which were not essential to the success of the project. Also, during project preparation there was confusion in the role played by consultants, to whom they reported and how they represented the opinion of IDB. Knowledge by Ecuadoreans of Bank procedures and contract contents was low. Ecuadoreans felt disadvantaged during negotiations due to a lack of experience and knowledge about IDB procedures. They also felt that the presentation of premature orincomplete projects contributed to their weak bargaining position.

d) On Country Program Execution and Achievement of Results

2.26 Ecuador's program execution status was evaluated using as a source the Annual Reports on Portfolio Management data from over four-hundred active projects during the three years from 1993 to 1996. Comparisons were made with IDB as a whole and the various country groups (A, B, C, and D). In 1995 and 1996, the percentage of the number of projects for Ecuador ranked as "normal" was 65% and 62%, respectively.

2.27 The evaluation of the achievement of project objectives was based on Project Completion Reports from 55 loans and technical cooperations which repre-sented US$1,569 million in disbursements and was equal to 46% of the approvals and 68% of the disbursements made during the 1975-1995 period. Comparable information was not available for IDB as a whole or for country groups.

2.28 In terms of sectoral allocations, the best performers were industry/mining and energy, followed by preinvestment and education. The transport portfolio and urban development were next. In agriculture, only 1.0% of disbursed resources were in projects which achieved their objectives; approximately one-half achieved some objectives and the other half fell very short.

2.29 The 1995 and 1996 Reports on Portfolio Management provided estimates of the projected likelihood of achieving developmental goals in the future for a sample of projects from Ecuador and the rest of IDB. Based on this data, the Ecuador program is expected not to be above average performer.

2.30 This means that the Ecuador program, in terms of achieving developmental project goals, is projected to perform not better than the Bank as a whole, and its comparator C and D country groups. Until 1995, Ecuador's record of problems during execution was higher than the IDB's average.

2.31 In contrast to the continued relative performance in projected achievement of developmental objectives, project execution improved in 1995 and 1996.

2.32 Contractual extensions and cancellations were the manifest symptoms of the main problems. A partial list of some of the factors which first lead to delays and finally to cancellations include :

• Recurring shortcomings in project design, in which the risks have not been fully assessed or taken into account.
• Weak executing institutions.
• Agreement to optimistic and unrealistic execution schedules or other contractual clauses, which the executing agency was unable or unwilling to fulfill.
• Delays from reformulation and reprogramming, due to optimistic execution schedules.
• Contractual clauses which are not complied with in time.
• Complicated and multiple bidding procedures, which can frequently originate in differences between IDB and Borrower procedures.
• Incomplete knowledge about IDB procedures and policies.
• Slowness on the part of the borrower and IDB to make decisions.
• Failure to internalize on the part of the executing agencies the financial and economic costs of delays.
• Constraints placed on continued access to soft funds of the IDB.
• Executing units which want to prolong their existence beyond the execution phase of projects.
• High turnover in middle and lower management levels leading to loss of institutional memory at executor level; lack of experienced, adequately paid and well trained, full-time professional civil service.

2. Lessons Learned

2.33 The evaluation drew main lessons to improve the efficiency, effectiveness, definition, and content of IDB/Ecuador program. The general direction of the lessons is to improve the strategic and sectoral focus of the Country Program, to increase ownership, to provide sustainability, to respond to client needs, and to strengthen institutional capacity in the long-term. To accomplish this, long-term programming is necessary. Below are five main lessons for the Bank and Ecuador:

• Strengthen the Strategic content of the Country Program; provide the rational for the identification of the priority sectors, the modes of development support and the definition of the measurable goals to be achieved.
• Include a Plan of action which focuses on institutional building and on a more efficient use of external credit and counter-part funds.
• Develop a greater degree of cooperation, participation, and ownership during the country planning cycle.
• Develop a stronger degree of sustainability and ownership framework during project preparation and negotiation.
• Strengthen monitoring and evaluation of the achievement of country program effectiveness and of sector and project goals.

C. The Case of the Dominican Republic

2.34 A total of US$675 million was available to finance the execution of 12 projects during the 13-year period of analysis of this CPE. The twelve projects

were distributed among 11 sectors/subsectors. To date, only one of those projects has been fully executed. There is an evident sectoral/subsectoral dispersion in the Bank-financed projects and in the .objectives that those projects pursue in the Dominican Republic

1. Findings

2.35 The review of approximately 150 Loan Progress Reports revealed that the lack of counterpart funds and institutional weaknesses have seriously impacted the normal execution of the 12 Bank-financed projects. As a result, one-third of them have been under execution for more than 7 years - of which two have been under execution for more than 10 years. The documents reviewed offer no convincing proof that the Bank has forcefully addressed these issues, or that alternative solutions were examined.

2.36 The Bank's strategy for the period 1984-1987 sought to promote the modernization of the productive sectors by increasing economic efficiency and productive competitiveness in international markets. This was to be achieved by stimulating production and structural change, and supporting the development and consolidation of export-oriented industries.

2.37 Nine projects were approved during the 1984-1987 programming period, but six of them were canceled due to the Dominican Congress' refusal to ratify them. The remaining three were responsive to the intent of the strategy. Of these, only one of the operations has been fully disbursed and the other two have experienced serious execution problems.

2.38 The 1988-1991 strategy aimed at lowering unemployment and increasing foreign exchange earnings through increased exports of diversified agricultural products, manufactured products and increased tourism. Three projects were approved and brought to execution: two in the education sector (technical-vocational and basic education) and one in the energy sector. None of the approved projects appear to be directly responsive to the strategy. Only the technical-vocational education operation seems to have been executed according to schedule; albeit also experiencing execution delays at its onset.

2.39 The third strategy, encompassing the years 1992-1995, intended to increase foreign exchange earnings, reduce unemployment, and alleviate poverty. Six projects were approved during the programming period - two of them address the need of increasing foreign exchange earnings, and one of them addresses poverty alleviation. The other three (roads, financial sector reform, and basic education) were predicated in response to objectives not clearly sought by the strategy. Execution delays have been experienced in the roads and financial sector reform projects.

2.40 The current strategy, for the period 1997-1999, seeks to preserve economic stability, stimulate efficiency, promote equity and strengthen governance. Although the proposed projects are responsive to the strategy, the expected number of approvals appears optimistic, and the sectoral coverage is too encompassing.

2.40 While there is a marked improvement in the design of the 1996 Country Paper, the four country strategies reviewed share some significant weaknesses that merit attention: The expected resource levels seems to play no role in the formulation of the strategies, in the selection of their objectives and in the choice of the means to attain them. It is not possible to hold any of the country development strategies accountable for the achievement of a single objective. None of the strategies seemed to emerge out of a process of identifying and weighing strategic options and their respective pros and cons None of the strategies offered an analysis of the most critical risks that could affect achievement of the objectives. No attempt was made in any of the strategies to identify which development issues/ constraints were most binding at the various points in time.

2.42 The 1996 CP identifies a list of economic and social problems which are reminiscent to those in the 1984 Country Paper, It is not fully clear to what extent the Bank activities of the past 13 years contributed to the improvement of the economic and social conditions in the Dominican Republic. It is difficult to determine the operational and strategic value that the Bank assigns to the CP exercise. None of the documents appeared to have given any weight to the previous development strategies, or attempted to assess whether and why the previous strategies achieved or failed to attain their objectives. Until appropriate benchmarks/ indicators have been incorporated into the strategies, such an effort does not appear to be cost-effective. Future CPEs should continue to focus on the linkages between the Bank's mandates, the country strategies, the operational programs, and the portfolio in execution.

2.43 The findings, conclusions and lessons of this CPE refer exclusively to the programming process in the Dominican Republic during the period of analysis. This analysis does not attempt to project these to the programming processes of other countries.

2.44 High levels of unemployment and insufficient foreign exchange earnings are two development problems that were consistently highlighted throughout the various programming periods. The strategy papers give no indication whether the intensity of any of these constraints has ameliorated with time and whether it has or not as a result of the Bank's efforts.

2.45 Country Papers did not provide the elements necessary to judge the Bank's performance in a clear, unambiguous and objective manner. To that effect, it is necessary to arrange the principal development issues/ constraints according to their cause-and-effect relationships, thus defining the objectives upon which the strategy should focus. This would entail conducting selective in-depth economic and sector analysis, and monitoring the evolution of the economic and social development issues/constraints that are being addressed. Sound economic and sector work is needed to assure the macro/sectoral linkages of lending operations.

2. Lessons Learned

2.46 Lessons with regard to the selection of the "focus" of the strategies: The first step in the design of a development strategy is the selection of its "focus." This should be determined, among others, by: (i) the composition and the progress of the current portfolio; (ii) a sense of the Bank's comparative advantages; (iii) the country's absorptive capacity; (iv) the experience and on-going commitments of other donors; (v) the development needs, constraints and opportunities; (vi) the expected resource levels; and (vii) the potential for a sustainable development impact within the programming period. In other words, a strategy should concentrate in a few areas where it is thought that Bank assistance can achieve significant program results within the programming period, and where those results can be expected to produce a sustainable development impact.

2.47 The strategy must also ensure that the resources available are commensurate with the objectives being selected, and that the recurrent cost implications are manageable given the country's budgetary constraints. It should be simple and correspondent with the country's implementation capacity, and not lead to increasing commitments in sectors or entities where there is persistent poor execution of ongoing projects.

2.48 Two additional lessons are offered to the preparation of the next Country Paper for the Dominican Republic:

• The first step in the design of a development strategy is the selection of its "focus." In other words, a strategy should concentrate in a few areas where it is thought that Bank assistance can achieve significant program results within the programming period, and where those results can be expected to produce a sustainable development impact. The strategy should ensure that the resources available are commensurate with the objectives being selected, and that the recurrent cost implications are manageable given the country's budgetary constraints.
• Strategy papers should not be written in isolation from previous efforts to articulate the Bank's development strategy in a country.

2.49 Lessons with regard to the objectives pursued by the strategies: In any kind of strategy, the objectives must be stated with specific targets and indicators with respect to results, otherwise there is no clear way to monitor progress, make corrections, or tell whether the strategy is achieving its objectives or not.

2.50 Lessons with regard to the risks to the strategies: The analysis of risks or of the key assumptions that must be met if the strategy is to achieve its objectives was consistently absent in all the CPPs and in the 1996 CP.

2.51 Comment: During formulation of a development strategy it is necessary to examine the risks or the probability of failure of the strategy. Therefore, the factors beyond the strategy's influence which may affect the success must be identified, as well as the critical assumptions upon which the success of the strategy depends. In the case of the Dominican Republic, issues such as the future availability of counterpart funds, the capacity of executing agencies, and the possibility that a sector is overburdened by commitments should be considered.

2.52 Strategy papers should not be written in isolation from previous efforts to articulate the Bank's development strategy in a country. Continuation of such practice diminishes the significance and credibility of the exercise. Projects should, to the extent possible, adhere to the objectives pursued by the strategy. When there are deviations, the subsequent Country Paper should explain the reasons for it and not ignore the deviation from the main objectives of the strategy - unexplained deviations raise more doubts as to the developmental and strategic value of the document.

2.53 From the above conclusions and comments, the following summary lessons are offered:

Top