EVO - Evaluation- A Management Tool for Improving Project Performance (a logical framework) -3/97-

In addition to the monitoring activities discussed in the previous Chapter, formal reports are often produced during the execution of the projects as well as at and after project completion. Hence, completion of project execution is not the end of the project's contributions to the development process. Although, at this stage, the project may have succeeded in generating the components, the purpose and goal level development objectives are sometimes far from being reached as many assumptions at these levels still need to be confirmed. Additionally, an increasingly important aspect of project performance is its contribution to future Bank lending operations, and what lessons the Borrower learns from the experience.

As stated in the first chapter, the Bank is committed to investing in development with better performance and results, by applying lessons learned from previous projects. The inclusion of parameters to measure the project's financial, environmental and social impact, in line with the Bank's commitment to these sectors, is becoming increasingly important. In fact, the Bank's Women in Development (WID) Policy Document (OP-761) specifically states that evaluation "guidelines and criterias for ex-post evaluation... should be revised to reflect the main issues outlined" in the policy.

Organizational learning can be achieved through well prepared and properly disseminated Project Completion Reports (PCRs), completed 2-3 months after the final disbursement; Project Performance Reviews (PPRs), typically completed 1-2 years after the end of the project; Borrower's Ex-Posts (BEPs), conducted as requested; and Operations Evaluation Reports (OERs) or development impact evaluations, completed 3-4 years after the end of the project.

The evaluation instruments that should be used for formal implementation and completion evaluations in the Bank (i.e. mid-term, ex-post, and impact) are discussed in this chapter. This chapter will also provide sample outlines for ex-post and development impact evaluation reports.

The PCR has a dual role. It is the last component of ongoing monitoring and is the first instrument for use in the ex-post evaluation. The PCR is prepared by the Country Office with input from the project team within three months of the date of final disbursement. The report brings together the project experience which can then be shared with the Executive Directors and other stakeholders.

The following outline describes the types of information that may be included in PCRs as indicated in the 1991 PCR guidelines (CON/OVE, 1991).

Mid-Term and Ex-Post project evaluations should address five major sets of issues which, of course, should be adapted to the specific reasons for evaluation in particular projects: 

• Continued rationale for the project
• Project efficiency
• Project effectiveness
• Project effects and impacts
• Lessons learned

The first four of these major evaluation issues are linked to specific sets of cells in the logframe. The fifth, lessons learned, covers the whole logframe.

Those managing evaluations should be familiar with this framework and the types of questions addressed therein. Annex 1 includes a brief description of the logical framework.

 Indicates evaluator' s initial search

Continuing Rationale refers to the extent to which the project design: 

• continues to reflect the country's development priorities;
• relates to the policy framework at the IDB; and
• contributes to the IDB's development priorities.

Sample evaluation questions that relate to continuing rationale are: 

• Does the project continue to relate to the policy framework at IDB?
• Does it continue to reflect the recipient country's development priorities?
• Does it contribute to IDB-8's development priorities? (eg. poverty reduction; environment; modernization of production structures, public sectors, and the state; integration of regional and sub-regional markets)
• Are the objectives still relevant?
• Among the array of alternatives to meet project objectives, was this an appropriate choice?
• Will achievement of the project's outputs ensure attainment of its purpose/goal?
• Are the project assumptions, as stated in the logical framework, still valid?
• Are there any unstated assumptions which are, or could become, problematic?

 Indicates evaluator' s initial search

 Indicates evaluator's additional search.

Efficiency relates to the activity and component level during project execution and refers to the extent to which: 

• inputs were supplied and managed
• activities were organized in the most appropriate manner
• the above was done at the least cost to produce the expected outputs

Sample evaluation questions that relate to efficiency are: 

• How well has project implementation been managed?
• How appropriate were the inputs and were they managed in a cost-effective way?
• Were there better ways of achieving the same results at less cost or in less time?
• Were the most cost-effective alternatives used in managing the project?
• Did the project duplicate the work of other organizations or donors?
• Was there adequate coordination with other interventions?
• How did inclusion of IDB-8's development priorities affect efficiency?
• If the outputs were not produced, what changes in the pattern of inputs might have remedied this?
• How would improvements in outputs attained enhance project purpose and goal achievement?
• What should be done, with what financial implications, to modify or improve the project?
• Do project activities complement, duplicate or work at cross-purposes with other projects/programs sponsored by IDB, the borrower, or other donors?

Indicators will be found at the activity and component levels of the logframe. Assumptions must be verified to assess the degree to which their behavior may have affected project execution.

 Indicates evaluator' s initial search

 Indicates evaluator's additional search.

Effectiveness relates to the purpose and goal levels after completion, and should refer to the extent to which:

 

• the project produced the expected outputs;

   

   

• the outputs achieved lead to project purpose; and

   

   

• purpose attainment contributed to the goal.

   

Sample evaluation questions that relate to effectiveness are: 

 

• How timely and cost-effective has the performance of input delivery been?

   

   

• How well has the project performed in terms of producing the expected outputs?

   

   

• How well has the project performed in achieving the purpose?

   

   

• To what extent has the project contributed to the goal of the program or sector it supports?

   

Indicators and possible causes should be found in the indicator and assumption columns.

 Indicates evaluator' s initial search

 Indicates evaluator's additional search.

Whereas effectiveness relates to the direct project objectives universe, effects and impacts refer to the effects of the project in other areas. Project effects and impacts refer to the planned and unplanned consequences of the project. In general:  

 

• effects relate to purpose-level planned and unplanned consequences.

   

   

• impact relates to goal-level planned and unplanned consequences.

   

Sample evaluation questions that relate to effects and impacts: 

 

• What has happened as a result of the project?

   

   

• What are the impacts and effects on the included IDB development priorities?

   

   

• What are the unplanned effects?

   

   

• What are the probable long term project effects on the program and sector?

   

   

• Why were anticipated effects not achieved?

   

In order to understand effects and impacts, in addition to the logframe's indicators, useful information could be generated by reviewing previously verified project assumptions At this level, evaluation analysis should also be related to country program objectives - at least within the sector or sectors related to the project being evaluated. The study of these issues is most important at OER or impact evaluation level.

 Indicates evaluator' s initial search

Lessons learned are of two types: Operational Lessons which tend to relate to efficiency, effectiveness and rationale; Developmental Lessons which focus on effects and impact. In the logframe method, the first would be found in the lower six cells, whereas the second would be found in the upper six cells. Lessons learned are conclusions (which can become future hypotheses) based on the findings of a particular evaluation that may be useful for future policy or practice. This field is so large (as can be seen, it covers the whole logframe), that the evaluation must establish relative search criteria, or identify the questions to be answered, early on. Otherwise the investigation could be never ending for this component.

Sample evaluation questions that relate to lessons learned are: 

• What are the lessons learned about the project's relevance?
• What has been learned about the project's performance in terms of achieving its objectives?
• What are the lessons learned about the project's efficiency in utilizing resources?
• What was learned about the usefulness of replicating this type of project?
• What has been learned about EVO's contributions to the project and evaluation?

 Until the ex-post phase, evaluation processes tend to focus at the input and output levels of the logical framework. The special role of an ex-post evaluation is to take a serious look at the purpose level and begin to analyze project contributions to the goal.

Carefully executed BEPs and PPRs can provide a wealth of invaluable information on the development performance of the Bank. The detailed specification of procedures and standards for BEPs is covered thoroughly in Chapter VI.

A sample outline of an ex-post evaluation, at PPR or BEP level, is shown on Table14.

Unlike the BEP and PPR, which look at the project itself, the OER or impact evaluation is a vital way for the Bank to gain a deeper understanding of its development impact and to use this both to celebrate its successes, and to learn from projects whose impact was modest, or negative. No matter how sound the design or how successful the project execution, only by studying the longer term impact can we fully understand project performance. Impact evaluations follow similar procedures to the other formal evaluations discussed in Chapter VI, though the issues move beyond implementation to the project's achievement of the purpose and contributions to the goal.

The impact evaluation framework suggested for use by IDB focuses on six major types of impact:

• Technological
• Institutional
• Economic
• Political
• Social and Cultural
• Environmental

The way these are treated in practice is suggested by the sample outline shown on Table 15.

Top