3.  Findings and Lessons from Project Evaluations

 

This Chapter presents a summary of lessons learned from 1997 evaluations of projects. They are mainly based on the twenty-one Project Performance Reports (PPRs) prepared by the Evaluation Office ( )for the different sectors.

3.1 Unlike the lessons from the previous two Chapters on Policies and Country Strategies, lessons from project evaluations are tentative and often local in nature.

3.2 Over the year, conducted numerous evaluations in priority areas of IDB Lending. Evaluations and lessons were generated on specific topics such as:

• Urban Development and Sanitation
• Finance, Trade and Investment
• Reform of the Public Sector
• Environmental Issues in Infrastructure
• Reaching the Poor
• Other Lending Areas

3.3 Each topic includes an introduction, findings and lessons learned.

3.4 Project Performance Reviews (PPR's) are in-depth studies of Technical, Financial, Administrative, Socio-economic and Institutional aspects of IDB financed operations. They take into account the identification of actual results compared to projected achievements.

3.5 These studies are typically carried out a year or two after the respective project's final disbursement and their primary focus is related to project execution.

A. Urban Development and Sanitation

1. Introduction

3.6 Over the years the IDB has supported numerous programs to promote Urban Development by financing

Housing and Urban Infrastructure projects throughout the region. Initially, urban issues were understood to be a problem of housing and basic infrastructure shortages, the result of high rates of urban migration overwhelming the capacity of the cities to provide basic services. Many of these projects were reasonably successful in meeting their physical targets.

3.7 However, by concentrating most of the efforts on physical implementation these projects did not have a major impact on the broader issues of sustainability. Subsequent IDB's efforts emphasized the multi-sectoral character of urban problems by integrating housing and infras-tructure investments into urban planning.

3.8 This strategy led to the development of multi-sectoral inter-ventions many of which encountered implementation difficulties. The intrinsic complexity of these programs added to serious institutional weakness in the countries made the success of these operations problematic. By the mid-eighties, IDB increased its emphasis on poverty alleviation and began to focus on the problem of supplying affordable housing to low-income populations. In pursuing this agenda, the Bank relied largely on sites and services and slum upgrading projects.

3.9 In 1997, evaluated two programs in the Urban and Housing Sector. These loans, for a com-bined estimated amount of US$155.6 constitute relevant examples of IDB's experience with integrated urban development, and sites and services programs, respectively.

2. Major Findings and Lessons Learned

3.10 One Program set out to stimulate Public Investments following a decade of crisis and limited investment activity. As a part of the program, some thirty-three projects were approved, with a total cost of US$82 million and an average cost of about US$2.5 million. By late 1995 this Program had increased potable water by 60% for beneficiaries in eight urban centers, established more than 85% sewer coverage in five centers, and contributed to sanitary waste disposal in three. Five channeling and drainage projects were completed as were eleven transportation projects.

3.11 An interesting lesson with regard to the Integrated Urban Development operations was its significant impact on poverty alleviation despite some important implementation difficulties. Key factors were: project design, adequate targeting and community involvement. This seems to indicate that urban management needs to find the appropriate balance between equity and efficiency while ensuring effective use of resources; an objective that is central to all development efforts.

3.12 Sites and services programs sought to capitalize from an on-going process of self-construction initiated by poor settlers while tackling simultaneously affordability, tenure, security and sustainability. The available evidence shows the potential of the strategy as an effective instrument to meet the housing demand of the informal sector. Often, however, sites and services programs failed to achieve two crucial objectives: Cost-recovery and equity.

Questions and Lessons to be considered:

• Are IDB's Policies in the Sector too rigid?
• Were there deficiencies in the program's risk analysis and why? and,
• The dimensioning of the effective demand, and in the supervision and monitoring of program activities should be reviewed.

3.13 One evaluation found a need for a more syste-matic and integrated approach to the planning of social development objectives, results, and targeting and evaluation methodologies. Current reporting procedures between the program's executor and the IDB should be reviewed so as to strengthen and systemize the presentation of information related to specific targets and groups.

3.14 The cases analyzed in the urban and housing sector reinforce the general lessons that projects have a greater probability of success when:

Therefore, important lessons from these experiences are the need to:

(i) assure appropriate knowledge of the institutional and political forces involved in supporting the objectives of the operation;

(ii) provide greater flexibility in project design;

(iii) guarantee accuracy in preparation through sound evaluation of the possible obstacles to be confronted;

(iv) secure appropriate monitoring/supervision; and

(v) promote beneficiaries involvement.

3.15 Attention should be given to strengthening the capacities of governmental levels, the private sector, and civil society to effectively implement and negotiate decentralization reforms over the long-term.

3.16 Effective inter-governmental relations are an important consideration in the implementation of social sector decentralization reforms. These relations should be carefully assessed in program planning and mechanisms developed to encourage

capacity building in this area, including provisions for communications and clear lines of responsibilities.

3.17 Significant aspects must be considered in an analysis of the possibilities and limitations of urban development programs of such large scope:

1. the complexity of executing the program which results from the wide range of sectors involved,
2. the requirements of inter-institutional coordi-nation; and
3. the financing structure and cost recuperation mechanisms of private investments.

B. Finance, Trade, and Investment

1. Introduction

3.18 In 1997, conducted seven evaluations to analyze issues relating to the design, execution, sustainability, and impact of reforms in the finance, trade, and investment sectors. The evaluations represent US$ 1.11 billion in combined total program costs, and form part of the Second Stage of a larger evaluation of IDB sector loans. The first stage consisted of a working paper titled, "The First Stage of Sector Loans: Preliminary Outlook of IDB Experience" (RE-213). In this Second Stage, seven sector loans are being evaluated in three countries which were chosen because they represent a sample of different conditions found also in other countries of the Region.

2. Major Findings and Lessons Learned

3.19 An evaluation of a Financial Sector Adjustment Program found the program to be a success in restoring the solvency and stability of the financial sector in the country where the operation was executed. Furthermore, the program contributed to an improvement in the competitiveness of financial institutions, as well as to solving the external debt problem in that particular country.

3.20 One possible lesson to be looked into: the passage of a financial intermediation law increased the legal powers of the Central Bank, improved upon the organizational structure of the Superintendent of Financial Institutions, and reinforced its capacity to supervise and regulate banks.

3.21 One Program contributed to the reduction and restructuring of the country's external debt in coordination with the country's international commercial creditors. As a result, the country's external debt is considered to be of a moderate level, and its debt service obligations to be manageable for the coming years.

3.22 One evaluation of a financial and trade sector loan found the program to be unsatisfactory in its component related to financial sector reform and incomplete with regard to the trade reform component. The design of the program should have taken into consideration the weaknesses of the country's judicial system or the uncertainty and lack of definition with regard to property rights. Moreover, certain conditions within the program's design were should have been formulated more precisely.

3.23 Another evaluation of a loan to support trade reform found that the program helped to put in place the final pieces to facilitate international trade in the country where the operation was executed.

3.24 An evaluation of a Financial Sector adjustment programs generated lessons for the program's design. Among these:

• avoid designing the program apart from the existing operations in the country; but instead:
• make them a part of structural adjustment measures.

   

3.25 Another evaluation proposes that:

• Sector Programs should aim at smaller number of reforms but with greater depth.

C. Reform of the Public Sector

1. Introduction

3.26 In 1997, conducted evaluations whose objectives were to analyze issues relating to the design, execution, sustainability, and impact of public sector reforms supported by each loan operation with-in this sector. These evaluations form part of the Second Stage of a larger evaluation exercise of IDB sector loans. Total program costs were of US$120 million. The findings and lessons learned of a Work-ing Paper which is one of a series of products that is producing on the Decentralization of Social Services in the region is also included. is evaluating Decentralization because of the great number of projects currently being executed, or in the IDB pipeline which include components for the division of responsibilities between government levels.

2. Major Findings

3.27 Since the implementation of one country's "second generation of reforms", the relations among government levels and between government and civil society have been significantly altered. New actors are assuming new responsibilities and the roles of different levels and various institutions are changing.

3.28 In one country, since the passage of a Popular Participation Law, municipal expenditures have been concentrated in urban works and housing, followed to a lesser extent by social sector investment (basic sanitation and education). Indeed, almost 49% of investments identified in nearly three hundred municipal operating plans developed in 1995 (the year following passage of the popular participation law) were related to urban works. Agriculture projects received only a limited proportion of funding in the 1995 operating plans. Areas like Health and Energy were similarly under-represented either as budget priorities or in terms of numbers of projects chosen for municipal investments.

3.29 It was found that there is less enthusiasm and knowledge among the general population for the Administrative Decentralization and Regulatory Reforms which are taking place. Many reforms affect issues which are far from the daily concerns of the average citizen. Possible issues:

• The Administrative Decentralization Law is a very new reform and needs further development to meet many of the country's needs.
• Communication first: some sub-national Departments with increased responsibilities are only now developing the links needed for good communications with their municipalities.

   

3.30 In this same country, national development funds are also changing their role from working with the departmental level to working directly with municipalities and communities. There are concerns about whether the Funds should increase their support for productive projects, serve a poverty reduction role or as a severe co-financing agent, and move to support integrated or staged activities, rather than isolated projects.

3. Lessons Learned

3.31 Loan conditions in countries which experience large disequilibrium and unclear laws -such as the case of property rights in one country- ought to be less ambitious in scope and more focused in formation of human resources in order to attain a sufficient cadre of trained professionals.

3.32 The environmental issues should not be over-dimensioned in the context of the conditions imposed. Regulatory framework that is being proposed for some sectors in the country should be realistic in light of the other needs of the country. Environmental standards should not imposed on poor countries on the level of strictness applied to the more developed countries.

3.33 It is important that countries have privatization laws that favor trade reform and financial sectors reform, instead of laws which are seen to be obstacles to successful privatization programs.

3.34 Health Service Reforms should not be focused only on one aspect of the health system, while ignoring the rest. For example, if reforms are limited to decentralizing services without changing the financing mechanisms, the incentives, or the criteria for setting priorities, one is likely to simply multiply the problems of the previously centralized systems. Comprehensive approaches are required to deal with the health system as a whole.

3.35 Decentralization does not consist of federal government allocations and legal reform alone. Training is an important ingredient to any Decentralization Policy.

D. Science and Technology Sector

1. Introduction

3.36 IDB supports programs within the Science and Technology Sector of the member countries in order to facilitate the up to date expertise and technology developments within member countries.

3.37 In 1997, conducted several evaluations of programs within the Science and Technology Sector when combined represent US$443 million in program costs. Among the objectives of the loan operations evaluated were:

• halt the erosion of one country's scientific expertise by providing selective support for excellent or highly promising research centers,
• raise the volume of financing available for technology development in the private sector, in order to meet currently unsatisfied demand,
• strengthen and expand the network and system for technology and investment information and promotion,
• finance research and development as well as scientific and technological services projects in priority areas at public institutions, and by private enterprises in the areas that they determine.

What follows are the major findings and lessons learned from these loan operations.

2. Major Findings

3.38 It was found that a strategic approach is necessary, involving government, the Science and Technology Sector and Industry in programs that are directly aimed at generating potentially economically important results. The present combination of a rather impassive industry and scientists that to a large degree are content with purely academic research, does not look too promising.

3.39 In one country is was found that the program had clearly been successful in supporting research, training of personnel, creating laboratories, and in renewing equipment. The program has also had an educational impact in universities and in the productive sector in terms of increased know-how about the R&D process and its importance for society in general. Linkage, while still not sufficient, has improved considerably due to the program. There are already verified productive sector results emanating from the program.

3.40 In another case, the main conclusion was that the program had been successful from an academic point of view. The program has had a strong impact in terms of improvements of scientific infrastructure and in the development of human resources. The country is clearly now in a better position both in basic sciences, and in applied areas with potential industrial importance, than when the program started.

3.41 One Science and Technology Program has clearly initiated a process of linking the research community with the productive sector and increasing the awareness in the country of the importance of S&T. The program has had a strong impact in terms of improvements of scientific infrastructure at universities and development of human resources.

3.42 The funding for the Science and Technology Sector in this same country was raised to 0.8-1.0% of GDP during the IDB Program, and is still kept at that level even after the country has decided to finance Science and Technology solely with national funds.

3.43 This country now has a vision and the instruments to continue developing the Science and Technology Sector. Thus, perhaps the main impact of the IDB loan has been the creation and administration of government agencies as part of a national innovation system with a capacity to foster basic research, applied research, and Research and Development in private firms, and to use these activities for the good of the country.

3. Lessons Learned

3.44 The banking and financial systems of developing countries should increase investments in activities relating to the promotion of technical exchange. They should support the introduction of credit programs to ensure access to funds to private firms for activities such as Research and Development, the introduction of quality enhancement programs and transfer of technology.

3.45 Within such credit programs, there is a need to design the credit conditions, to ensure that participation is possible for small enterprises. It may often be necessary to develop specific financial instruments for small firms.

3.46 Especially in large countries, IDB client institutions need to be encouraged to find ways of making their services available throughout the country and not just in the principal metropolitan areas.

3.47 Financial institutions dealing with technical change need to be encouraged to adopt practices which allow their staff members to keep up-to-date with what is going on in firms. Visits to projects represent one useful means of maintaining awareness.

3.48 Patents are a potential source of information and can be very useful in identifying existing technology and avoiding duplications.

E. Environmental Issues in Infrastructure

1. Introduction

3.49 In 1997, conducted evaluations of the Environmental Management of IDB-supported public infrastructure investments. The purpose of these evaluations is to learn lessons about the usefulness and adequacy of IDB guidelines and to examine how it oversees compliance regarding infrastructure projects. Analysis and evaluation of environmental management impacts, conformity with loan contract stipulations, and effectiveness of enforcement instruments will lead to recommendations for Strengthening Country Capacity to implement IDB-supported investments. Among objectives, the evaluations aim to provide lessons to enhance Country Capacity to manage public infrastructure projects.

2. Major Findings

3.50 One project was designed to rehabilitate 1,400 kilometers of paved roads and to provide the State Roads Department with the means necessary to maintain and operate the road network efficiently and safely. The executing agency far exceeded the first objective by reaching a target of nearly 1,900 kilometers of improved roads. The evaluation of this project, however, focused on the second objective dealing with the agency's capacity for environmental management of public sector infrastructure investments and civil works maintenance.

3.51 Building on the initiatives of one State government to rehabilitate and maintain its road network, this same Program itself served as a catalyst not only for further road improvement operations, but also for institutional changes within the Highway and Transportation Department. Local counterpart funds for maintenance reached US$11.2 million, or 84% of total investments for this component. In addition, the Highway and Transportation Department was responsible for all training activities and actually allocated 30% more resources to this activity than originally budgeted.

3.52 Two years after initiation of one of the Program's activities, the Highway and Transportation Department altered its organizational structure, elimi-nated several upper-level positions, and began the process of overall staff reduction. The introduction of annual maintenance plans (in accordance with Loan Contract Clauses) for the duration of the Program, and beyond, contained an articulation of state road policy and its implications for implementation. Gradual staff reductions were accompanied by an intensification of training to increase personnel capacity to carry out refurbished mandates. The introduction of contracted maintenance workers also reflected decisions to stream-line operations for greater efficiency and effectiveness. The Department of Operations decentralized equipment and added responsibilities to the maintenance office in the field leading to collaborative efforts, mutual sup-port systems, and "horizontal" resources sharing diminishing an excessive dependence on the local Highway and Transportation Department Headquarters.

3.53 Creative solutions and initiatives were noted by the Team evaluating this same Project. Periodic rotation among Senior Field Office Staff is a vehicle for information exchange, standardization of norms and procedures, and the transfer of knowledge for the solution to road maintenance problems. The evaluation Team inspected, for example, specialized and improvised machinery which had been "invented" and constructed to respond to specific needs. Furthermore, some train-ing exercises exposed a serious need and demand for an adult literacy program to enhance the capacity building process. Recognition of this deficiency among a great number of workers led to the creation of such a program. Road work is scheduled from Monday through Thursday. Machine maintenance and repair take place on Fridays. The presence of road maintenance teams at the field office on this day provides the opportunity for literacy classes.

3.54 In another country where a road project was executed, the government has established the basis for environmental management and has enhanced its legal framework with binding regulations. The preparation, approval, and implementation of this road Project coincided with the government's establishment of the National Environmental Commission and Environ-mental Unit within the Department of Roads, National Environmental Legislation, and specific environmental regulations and assessment process.

3. Lessons Learned

3.55 Good examples:

• For one project, IDB guidelines and supervision with regard to environmental management were both appropriate and instrumental and may serve as a model.

   

• Within one country, the Executing Agency has taken initiatives by formulating its own procedures and standards for construction, road rehabilitation, and the mitigation of potential negative impacts on the natural surroundings in its jurisdiction. This process of institutional strengthening is successful and should be considered as a model for other operations.

   

F. Reaching the Poor

1. Introduction

3.56 In 1997, produced reports which form the Second Phase of the 1995 evaluation which examined activities for reaching the poor. While the first phase of this activity concentrated on IDB Activities for Reaching the Poor in the Social Sectors, namely projects which benefit children in especially difficult circumstances, this phase examines components of production-oriented projects/programs that benefit low-income groups.

2. Major Findings

3.57 A Program which aimed to Strengthen Cooperatives in one Region of the country in which it was carried out found that the program had a favorable impact with regard to volume of production, productivity, and the overall balance of payments of the country.

3.58 An analysis by the Office of Planning and Budget found that this same program generated approximately one thousand jobs, 75% of which were in the low-income sectors. These findings exceed expected results.

3.59 Measuring variations in income levels as a result of the project is uncertain, especially if the salaries of personnel employed by the cooperatives is factored in. However, according to reports from the technical personnel of the firms involved with the project, salaries received by low-income workers have increased as a result of the program.

3. Lessons Learned

3.60 A Technical Vocational Education Enhance-ment Program concluded that the support for the productive capacity of poor sectors, through job training and vocational education credit lending programs could be a more effective mechanism to target and affect low-income groups, if the designers of the project and/or its management:

• establish a strong link between the pro-ductive sectors and the academic institutions, and institute stronger work-study/ internship programs, as well as monitoring systems to track the graduates.
• identify institutional weaknesses in both the executing agency and the academic institu-tions and include an institutional strengthe-ning component for the executing agency and the academic institutions to mitigate problems experienced during this project's implementation stage.
• effectively estimate all the potential risks/ pitfalls inherent in credit lending programs.
• create a balance between achieving sustain-ability and providing the services in accordance with the project's mission, with-out sacrificing the services provided for low-income students (e.g., formulating credit policies not only to achieve sustainability, but also to ensure that target groups are not excluded; and discourage the participation of non-poor).
• ensure adequate participation of the poor through outreach programs in poorest communities, and factor costs associated with advertising and promoting the program in the project budget.
• verify commitment from the Government to invest in vocational programs and marketing programs to change the culture of supporting careers that do not respond to the demands of the labor market.
• require students participating in the program to commit to employment in the sector that corresponds to the student's area of study for the three years directly after the end of the vocational education certification, before allowing them to continue with their education.

   

3.61 A same program found that for programs such as this, with a strong "social" commitment to target low-income groups, it is crucial to establish mechanisms of support that will allow the executing agency to effectively implement the program while separating itself from the responsibility of assuming risks, such as loan defaults, arrears, and currency devaluations.

3.62 From this program, it was also determined that the support for the productive capacity of poor sectors, through Job Training and Vocational Education measures is an effective strategy to reduce poverty by improving the quality of the workforce. In vocational education credit projects, however, it is important for the IDB to determine to what extent the measures to achieve sustainability will affect the "social development component" of the project, i.e., providing credit to students of low-income households to enable them to participate in technical vocational programs. On a loan-finance basis, if sustainability is the only intent of the project, then these type of credit projects should not be part of the IDB strategy to reduce poverty.

3.63 Funding a vocational education project through a private, non-profit organization, without support of the government, limits its scope, its financial capacity, and potentially introduces economic risks.

3.64 In future vocational education projects, links between the productive sectors and academia must be established at the onset of the project and support throughout the life of the project in order to ensure its success.

3.65 Initiatives to open trade, such as the Southern Common Market (MERCOSUR), imply deep changes within the economies of the countries involved, as well as with how productive resources within a country are assigned. Although it is hoped that such an initiative is beneficial to the country as a whole, it is recognized also that it will negatively affect some sectors. IDB ought to take a look at these microeconomic impacts and propose measures to mitigate such impacts in programs and projects that is has financed or studied in other countries.

G. Other Evaluations

1. Introduction

3.66 In 1997, conducted evaluations whose objectives were to analyze issues relating to the design, execution, sustainability, and impact of reforms supported by each loan operation. These evaluations form part of the second stage of a larger evaluation of IDB Sector Loans. What follows are the major findings and lessons learned of one of these evaluations. Furthermore, as part of its 1996 Work Plan, was assigned the responsibility to examine three projects whose objectives were to increase the incomes and opportunities for gainful employment within population groups of scarce resources. The major findings and lessons learned of a credit and technical support program for agricultural producers is also presented below. Together these programs evaluated represent US$120 million in program costs.

2. Major Findings

3.67 Sector Loans are not loans where the funds are targeted at institutional and economic reforms. They are general loans whose conditions are designed to back such reforms. From the point of view of the borrowing country, the conditions stipulated within the loans are the equivalent of restrictions to attain the loan. In this sense, the degree to which the borrower ought to fulfill the conditions of the loan should remain explicit and clear within the clauses of the loan.

3.68 In general terms, the conditions of one loan were fulfilled. Although the conditions stipulated within the loan were necessary ones, they were clearly not sufficient to achieve the objective of backing the institutional and economic reforms in the different sector areas.

3.69 In spite of the above, the agricultural sector of this country has shown clear signs of improvement in the last few years. This is a consequence of the relative stability of the macroeconomic climate which has been sustained by strong international support. The challenge will consist of maintaining this growth within an environment of decreased assistance.

3.70 Another loan was completed as well. The beneficiaries of the program, initially estimated in the project report to be 4,800 small producers, reached, up to the date of the evaluation, a level of some 5,700 producers. Of this total, 18% were women.

3.71 The median annual income per family of 5.1 persons was $7.482 Argentinean pesos at the beginning of this same program. By the end of the program's execution, this median family income approached $11.461 Argentinean pesos.

3.72 The labor employed outside the unit of production exhibited an important shift toward salaried work which was not in evidence before one program was executed. A significant number of those employed were drawn from jobs held in urban areas.

3.73 The majority of technology transferred went toward labor intensive crops such as vegetables in greenhouses, open-air cultivated vegetables, fruit, crop management, and to aspects related to product marketing.

3. Lessons Learned

3.74 One of the lessons learned form the analysis of the problem of land holdings in one country is that one can not expect the executive branch to be able to resolve a particular problem when by nature it involves the legislative branch, judicial branch, and other government institutions such as the police force.

3.75 Another lesson learned from the same program is that conditions referring to international trade ought to be framed within any international treaties to which the borrowing country has subscribed. In this particular case, the tariff issue ought to be analyzed in light of the Central American Common Market Agreements.

3.76 One Agricultural Credit Program found that it is not prudent to design such credit programs in which banks and official financial institutions do not participate in the analysis and risks of the operation.

3.77 Rural Credit can be an efficient instrument to promote rural development. Nevertheless, its use should be directed only at those beneficiaries whose properties exhibit solid agricultural potential. Small farmers with properties of scarce or no productive potential should receive additional training. A similar lesson was proposed by the Evaluation Office in the mid eighties (RE-129).

Top