EVO - Evaluation- A Management Tool for Improving Project Performance (a logical framework) -3/97

The Inter-American Development Bank is entering a new era in which project performance is being emphasized. The Bank is committed to investing in development with better performance and results. It is looking for ways to improve performance in its overall strategy and policies, and country projects and programs; particularly those that deal with low-income beneficiaries, women in development, environmental management, and support for microentrepreneurs.

The IDB Board of Governor's "Report on the Eighth General Increase in the Resources of the IDB" states that the Bank will "begin to work more systematically with borrowing member countries to evaluate the financial sustainability of social sector projects." Properly executed evaluations, involving managers and staff, borrowers and executing agencies, strategically linked to all stages of the project cycle, can be a vital tool to improve the performance of IDB-sponsored projects.

Evaluations can assist in improving project performance at all stages of the project cycle and provide information feedback to improve Bank policies and procedures. Carefully planned and executed evaluations can also 1) help analyze the Bank's performance in the project identification and planning stages, and in its supervision of project implementation; 2) measure progress toward meeting objectives, which is essential for good management; 3) support the performance of implementing agencies through both monitoring and training in evaluation methods; 4) measure the extent to which expected results are achieved and analyze the factors that helped or hindered this achievement; and 5) assess the effects and impacts projects have on their intended beneficiaries.

In addition, project evaluation generates products to help managers and directors: 

• improve the performance of new and ongoing projects
• choose among various development alternatives
• learn about best practices in a given field
• enhance project sustainability

Evaluation involves the application of rigorous methods to assess the progress of a project in meeting its objectives. The evaluation process combines many types of information with the judgements and perspectives of people involved or affected. It relies on tools from such fields as statistics, economics and anthropology, and is fundamentally rooted in the concepts and procedures of research methodology. However, most of all, internal evaluation within an institution such as the Bank also relies on an institutional evaluation context provided by all the stakeholders in the Bank Evaluation System (BES).

While there are various definitions of evaluation, all incorporate the following three concepts:

Project evaluation has traditionally been seen as focussing on ex-post analysis. Current practice considers evaluation as a vital ongoing activity that not only provides learning at project level, but also provides an understanding of issues at program, policy and strategy levels. Furthermore, the closer evaluation gets to policy, the further it moves from a quantitative approach to a qualitative approach.

Over the last hundred years, evaluation has evolved from a one-dimensional focus carried out by a single evaluator, who generally came from the technical field being evaluated, to a multi-dimensional activity focusing on multiple levels of the project, involving a wide range of producers of evaluation products, and users of the evaluation results. Currently, rather than relying only on the perspectives of the single evaluator at different stages, a much larger group of stakeholders helps shape the evaluation and interpret its findings.

These "stages" should be viewed as summary and introductory concepts as reality is somewhat more complex, particularly, currently as evaluation begins to evolve from mostly an ex-post activity to a tool used more and more for monitoring purposes. The goal of evaluation has essentially evolved from audit and culpability to the current goal of understanding and learning.

Since the late 1970s, international development agencies have integrated aspects of evaluation into project design, particularly through the Logical Framework Analysis approach. This has advanced the use of evaluation models which link planning, design, execution, monitoring and project completion performance. The Bank is increasingly using the Logical Framework Analysis methodology to link evaluation to projects, as developed in Chapter IV of this handbook.

Based on the relationships between the project cycle and the evaluation tools and products described in this book, the following table looks at the two principal types of evaluation: "formative" evaluation that occurs during and after the project preparation and implementation stages; and "summative" evaluation that is conducted at, and after the project completion stage.

The following table shows each project cycle and evaluation stages and their corresponding formal evaluation tools. Final evaluation products consist of the documents themselves, as produced at each stage; and the "process results", as generated through strategy and policy decisions of the Board of Executive Directors and the Management of the Bank. The table shows that evaluation reports need not be restricted to "summative" learning after project execution. Formative evaluations ¹ also play a very important role. Evaluation services should be extended more effectively to the preparation and implementation stages of the project cycle through formative evaluation methods and techniques.

The evaluation process will not do much to improve performance unless it is carefully planned and managed and is integrated fully into all stages of the project cycle. If supported adequately and managed appropriately, evaluation should yield improved participation, accountability and development performance of projects and programs.

The evaluation process generally requires an array of internal and external resources working together for strong project performance. The investment of resources in project evaluations is often underestimated. This may lead to poor quality results. The time required of the major stakeholders involved in conducting an evaluation can become a costly investment if the evaluation is not executed effectively. By applying the principles of the Bank Evaluation System (BES) ³, the generation of "evaluable" projects is ensured at project start-up.

If the evaluation process is to contribute to performance, it must also ensure stakeholders that it is:

Impartiality implies neutrality, transparency and fairness of the evaluation process, analysis and findings. There can be no vested interest or conflict of interest by the evaluator -- the impartiality should come through in all aspects of the evaluation process.

The first essential for credibility is the trust stakeholders place on the evaluator's expertise and impartiality. Good evaluation research follows standards of the discipline related to the appropriateness of the evaluation design and the reliability and validity of the data, ensures the thoroughness of the data analysis, and makes explicit the logical connection between the findings, the conclusions and the recommendations. Furthermore, it does so with sensitivity to the needs and perspectives of the various evaluation stakeholders. While each of these matters are complex and extend beyond this introductory description, it is possible to have some understanding of the rigor of the evaluation by examining the content and presentation of the evaluation report.

The full measure of an evaluation process is its effect on the people and organizations who learn from it. Effects on evaluation consumers can scarcely be expected to be positive unless the evaluation products are timely contributions to decision-making presented in clear and concise language that stakeholders can easily understand.

Contemporary evaluation practice places great emphasis on participation of all involved stakeholders, including mutual sharing of experience at all levels. Thus, the evaluation should reflect different stakeholder interests, needs and perceptions.

The evaluation must be seen as a process that feeds back into decision-making and organizational learning. Thus, the evaluation manager has particular responsibility for ensuring the systematic dissemination of findings to stakeholders, including project planners and policy-makers.

As an investment, evaluations must balance the need for thoroughness and validity with getting an appropriate return. They should add value to stakeholder experience with the project and tend to have the greatest impact when stakeholders invest their own resources in the evaluation.

The emerging model of project evaluation involves a partnership among the producers and consumers of the evaluation process. Stakeholders are groups involved in the design and execution of the project - as well as those involved in producing, implementing, and using the evaluation. Stakeholders may also be positively or negatively affected in some way by the results of the evaluation. Different stakeholders will have different claims, concerns and issues. It is the task of the evaluator to identify and address stakeholders' interests in the process of evaluation.

Stakeholders in IDB project evaluations can be categorized into three groups: the evaluation sponsor, consisting of IDB Executive Directors, Management, the Borrower, and the Office of Evaluation (EVO); the primary Stakeholders, including Beneficiaries, IDB Country Offices, Executing Agents and IDB Governors; and the secondary stakeholders, or partners in development, lobbyists, and partners in research.

Those most directly involved in project evaluations are the evaluation's "primary stakeholders" (refer to diagram). Typically, primary stakeholders include the organization that manages or sponsors the evaluation (typically the Bank or borrower), and the people and organizations that receive the evaluation report. "Secondary stakeholders" are wide-ranging and may vary depending on the particular project and evaluation focus. As is the case when planning any development project, the evaluation sponsor needs to analyze the characteristics of the stakeholders.

Evaluation at the IDB is a multiple stakeholder system of producers of evaluable products, generators of evaluations and users of evaluation products. Such a system begins with the formulation of the project, continues during the project's execution, assessing its effectiveness and impact, and ends after termination of the project, with an assessment of the project's development impact.

In practice, this system is based on:  

• the stages of the project cycle (as expressed in Bank objectives, procedures, and activities);
• the operational structure of the Bank;
• the actions required of the different stakeholders at each intersection of stage and structure.

All of these elements constitute the "matrix" of the Bank Evaluation System (BES). The following table presents the conceptual structure of this system.

1. The Bank is committed to using evaluation to build a capacity for verifying performance at all stages of the project cycle. The diagram below shows the project cycle, the contributions of the evaluation function at each stage of the project cycle, and the central role of capacity development.

There are three basic evaluation frameworks used by the Bank in evaluation-related activities:

• The Logical Framework and benchmark/indicator guidelines, introduced at the project preparation (or design) stage;
• A framework incorporating the concepts of continued rationale, efficiency, effectiveness, effects and lessons learned, which is used during implementation/completion and after the completion stages for ex-post evaluations; and
• A framework featuring areas of summative evaluation (strategy, policy, program, design, monitoring and execution, procedures, performance, sustainability) used in the ex-post stage for impact evaluations.

These evaluation functions become less complex to execute if, at ex-ante design and execution monitoring levels, appropriate benchmarks /indicators and data collection/reporting are generated. Participation of all BES stakeholders is essential to ensure this.

Top  

------------------------------------------------------------