EVO - Evaluation- A Management Tool for Improving Project Performance (a logical framework) -3/97-

IV

PROJECT COMPLETION AND EVALUATION REPORTS

 

In addition to the monitoring activities discussed in the previous Chapter, formal reports are often produced during the execution of the projects as well as at and after project completion. Hence, completion of project execution is not the end of the project's contributions to the development process. Although, at this stage, the project may have succeeded in generating the components, the purpose and goal level development objectives are sometimes far from being reached as many assumptions at these levels still need to be confirmed. Additionally, an increasingly important aspect of project performance is its contribution to future Bank lending operations, and what lessons the Borrower learns from the experience.

 

As stated in the first chapter, the Bank is committed to investing in development with better performance and results, by applying lessons learned from previous projects. The inclusion of parameters to measure the project's financial, environmental and social impact, in line with the Bank's commitment to these sectors, is becoming increasingly important. In fact, the Bank's Women in Development (WID) Policy Document (OP-761) specifically states that evaluation "guidelines and criterias for ex-post evaluation... should be revised to reflect the main issues outlined" in the policy.

 

Organizational learning can be achieved through well prepared and properly disseminated Project Completion Reports (PCRs), completed 2-3 months after the final disbursement; Project Performance Reviews (PPRs), typically completed 1-2 years after the end of the project; Borrower's Ex-Posts (BEPs), conducted as requested; and Operations Evaluation Reports (OERs) or development impact evaluations, completed 3-4 years after the end of the project.

 

The evaluation instruments that should be used for formal implementation and completion evaluations in the Bank (i.e. mid-term, ex-post, and impact) are discussed in this chapter. This chapter will also provide sample outlines for ex-post and development impact evaluation reports.

 

 

A. PROJECT COMPLETION REPORTS (PCRS)

 

The PCR has a dual role. It is the last component of ongoing monitoring and is the first instrument for use in the ex-post evaluation. The PCR is prepared by the Country Office with input from the project team within three months of the date of final disbursement. The report brings together the project experience which can then be shared with the Executive Directors and other stakeholders.

 

The following outline describes the types of information that may be included in PCRs as indicated in the 1991 PCR guidelines (CON/OVE, 1991).

 

Table 13: Sample Outline
Project Completion Report

Section 1 - Project Summary

1.1 Project Identification

1.2 Project Description

1.3 Project Results

1.4 Major Lessons Learned

1.5. Recommendations

 

Section 2 - Project Details

2.1 Project Objectives and Description

2.1.1 Summary of objectives

2.1.2 Project Components

2.1.3 Changes since approval

 

 

2.2 Project Results

 

2.2.1 Rationale for Changes Since Project Approval

2.2.2 Results Achieved

2.2.3 Differences Between Planned and Actual Results

2.2.4 Unplanned Outcomes

2.2.5 Aspects That Have Minimal Chances of Being Achieved

2.2.6 Impact of External Consultant Services

2.2.7 Internal Level of Economic Return

 

2.3 Lessons Learned from the Project

2.3.1 Design

2.3.2 Execution

2.3.3 Role of the Bank in the project

2.3.4 Role of the beneficiary Institution

2.3.5 Conditionality

 

2.4 Recommendations for Future Projects

2.4.1 Other lessons

2.4.2 Application of lessons learned

 

2.5 Additional Comments

 

Annexes on Baseline Data, Costs, Financing

 

 

 

B. THE IDB EVALUATION FRAMEWORK FOR MID-TERM AND EX-POST EVALUATIONS

 

Mid-Term and Ex-Post project evaluations should address five major sets of issues which, of course, should be adapted to the specific reasons for evaluation in particular projects: 

  • Continued rationale for the project
  • Project efficiency
  • Project effectiveness
  • Project effects and impacts
  • Lessons learned

 

The first four of these major evaluation issues are linked to specific sets of cells in the logframe. The fifth, lessons learned, covers the whole logframe.

 

Those managing evaluations should be familiar with this framework and the types of questions addressed therein. Annex 1 includes a brief description of the logical framework.

 

CONTINUING RATIONALE

NARRATIVE SUMMARY

INDICATORS

MEANS OF VERIFICATION VERIFICACIÓN

ASSUMPTIONS

GOAL

­ ­ ­ ­ ­

­ ­ ­ ­ ­

­ ­ ­ ­ ­

PURPOSE

­ ­ ­ ­ ­

­ ­ ­ ­ ­

­ ­ ­ ­ ­

OUTPUTS

   

­ ­ ­ ­ ­

INPUTS

   

­ ­ ­ ­ ­


­ ­ ­ ­ ­

 Indicates evaluator' s initial search

 

 

1. Continuing Rationale of a Project

Continuing Rationale refers to the extent to which the project design: 

  • continues to reflect the country's development priorities;
  • relates to the policy framework at the IDB; and
  • contributes to the IDB's development priorities.

 

Sample evaluation questions that relate to continuing rationale are: 

  • Does the project continue to relate to the policy framework at IDB?
  • Does it continue to reflect the recipient country's development priorities?
  • Does it contribute to IDB-8's development priorities? (eg. poverty reduction; environment; modernization of production structures, public sectors, and the state; integration of regional and sub-regional markets)
  • Are the objectives still relevant?
  • Among the array of alternatives to meet project objectives, was this an appropriate choice?
  • Will achievement of the project's outputs ensure attainment of its purpose/goal?
  • Are the project assumptions, as stated in the logical framework, still valid?
  • Are there any unstated assumptions which are, or could become, problematic?

 

EFFICIENCY

NARRATIVE SUMMARY

INDICATORS

MEANS OF VERIFICATION VERIFICACIÓN

ASSUMPTIONS

GOAL

     

PURPOSE

     

OUTPUTS

­ ­ ­ ­ ­

­ ­ ­ ­ ­

u u u u u

INPUTS

­ ­ ­ ­ ­

­ ­ ­ ­ ­

u u u u u


­ ­ ­ ­ ­

 Indicates evaluator' s initial search


u u u u

 Indicates evaluator's additional search.

 

 

2. Project Efficiency

Efficiency relates to the activity and component level during project execution and refers to the extent to which: 

  • inputs were supplied and managed
  • activities were organized in the most appropriate manner
  • the above was done at the least cost to produce the expected outputs

 

Sample evaluation questions that relate to efficiency are: 

  • How well has project implementation been managed?
  • How appropriate were the inputs and were they managed in a cost-effective way?
  • Were there better ways of achieving the same results at less cost or in less time?
  • Were the most cost-effective alternatives used in managing the project?
  • Did the project duplicate the work of other organizations or donors?
  • Was there adequate coordination with other interventions?
  • How did inclusion of IDB-8's development priorities affect efficiency?
  • If the outputs were not produced, what changes in the pattern of inputs might have remedied this?
  • How would improvements in outputs attained enhance project purpose and goal achievement?
  • What should be done, with what financial implications, to modify or improve the project?
  • Do project activities complement, duplicate or work at cross-purposes with other projects/programs sponsored by IDB, the borrower, or other donors?

 

Indicators will be found at the activity and component levels of the logframe. Assumptions must be verified to assess the degree to which their behavior may have affected project execution.

 

EFFECTIVENESS

NARRATIVE SUMMARY

INDICATORS

MEANS OF VERIFICATION VERIFICACIÓN

ASSUMPTIONS

GOAL

­ ­ ­ ­ ­

 

­ ­ ­ ­ ­

PURPOSE

­ ­ ­ ­ ­

 

­ ­ ­ ­ ­

OUTPUTS

u u u u u

u u u u u

­ ­ ­ ­ ­

INPUTS

     

­ ­ ­ ­ ­

 Indicates evaluator' s initial search


u u u u

 Indicates evaluator's additional search.

 

 

3. Project Effectiveness

Effectiveness relates to the purpose and goal levels after completion, and should refer to the extent to which:

     

  • the project produced the expected outputs;

     

     

  • the outputs achieved lead to project purpose; and

     

     

  • purpose attainment contributed to the goal.

     

 

Sample evaluation questions that relate to effectiveness are: 

     

  • How timely and cost-effective has the performance of input delivery been?

     

     

  • How well has the project performed in terms of producing the expected outputs?

     

     

  • How well has the project performed in achieving the purpose?

     

     

  • To what extent has the project contributed to the goal of the program or sector it supports?

     

 

Indicators and possible causes should be found in the indicator and assumption columns.

 

EFFECTS AND IMPACTS

NARRATIVE SUMMARY

INDICATORS

MEANS OF VERIFICATION VERIFICACIÓN

ASSUMPTIONS

GOAL

­ ­ ­ ­ ­

­ ­ ­ ­ ­

u u u u u

PURPOSE

­ ­ ­ ­ ­

­ ­ ­ ­ ­

u u u u u

OUTPUTS

     

INPUTS

     

­ ­ ­ ­ ­

 Indicates evaluator' s initial search


u u u u

 Indicates evaluator's additional search.

 

 

4. Project Effects and Impact

Whereas effectiveness relates to the direct project objectives universe, effects and impacts refer to the effects of the project in other areas. Project effects and impacts refer to the planned and unplanned consequences of the project. In general:  

     

  • effects relate to purpose-level planned and unplanned consequences.

     

     

  • impact relates to goal-level planned and unplanned consequences.

     

 

Sample evaluation questions that relate to effects and impacts: 

     

  • What has happened as a result of the project?

     

     

  • What are the impacts and effects on the included IDB development priorities?

     

     

  • What are the unplanned effects?

     

     

  • What are the probable long term project effects on the program and sector?

     

     

  • Why were anticipated effects not achieved?

     

 

In order to understand effects and impacts, in addition to the logframe's indicators, useful information could be generated by reviewing previously verified project assumptions At this level, evaluation analysis should also be related to country program objectives - at least within the sector or sectors related to the project being evaluated. The study of these issues is most important at OER or impact evaluation level.

 

EFFECTS AND IMPACTS

NARRATIVE SUMMARY

INDICATORS

MEANS OF VERIFICATION VERIFICACIÓN

ASSUMPTIONS

GOAL

­ ­ ­ ­ ­

­ ­ ­ ­ ­

­ ­ ­ ­ ­

PURPOSE

­ ­ ­ ­ ­

­ ­ ­ ­ ­

­ ­ ­ ­ ­

OUTPUTS

­ ­ ­ ­ ­

­ ­ ­ ­ ­

­ ­ ­ ­ ­

INPUTS

­ ­ ­ ­ ­

­ ­ ­ ­ ­

­ ­ ­ ­ ­


­ ­ ­ ­ ­

 Indicates evaluator' s initial search

 

 

5. Lessons Learned from the Project

Lessons learned are of two types: Operational Lessons which tend to relate to efficiency, effectiveness and rationale; Developmental Lessons which focus on effects and impact. In the logframe method, the first would be found in the lower six cells, whereas the second would be found in the upper six cells. Lessons learned are conclusions (which can become future hypotheses) based on the findings of a particular evaluation that may be useful for future policy or practice. This field is so large (as can be seen, it covers the whole logframe), that the evaluation must establish relative search criteria, or identify the questions to be answered, early on. Otherwise the investigation could be never ending for this component.

 

Sample evaluation questions that relate to lessons learned are: 

  • What are the lessons learned about the project's relevance?
  • What has been learned about the project's performance in terms of achieving its objectives?
  • What are the lessons learned about the project's efficiency in utilizing resources?
  • What was learned about the usefulness of replicating this type of project?
  • What has been learned about EVO's contributions to the project and evaluation?

 

 

C. EX-POST EVALUATION

 Until the ex-post phase, evaluation processes tend to focus at the input and output levels of the logical framework. The special role of an ex-post evaluation is to take a serious look at the purpose level and begin to analyze project contributions to the goal.

 

 

D. PPRS AND BEPS

Carefully executed BEPs and PPRs can provide a wealth of invaluable information on the development performance of the Bank. The detailed specification of procedures and standards for BEPs is covered thoroughly in Chapter VI.

 

A sample outline of an ex-post evaluation, at PPR or BEP level, is shown on Table14.

 

Table 14: Sample Report Outline -
Project Performance Review (PPR)
Ex-Post Project Evaluation (BEP)

 

Table of Contents

Abstract

 

1.0 Introduction

1.1 Background

    • Purpose of the Evaluation
    • Major Stakeholders

1.2 Project Description

    • Summary of logical framework
    • General Status (start, finish date)

1.3 Evaluation Methodology

1.3.1 Design/General Approach

1.3.2 Sources of Data

1.3.3 Instruments

1.3.4 Evaluation Team

1.3.5 Limitations

1.4 Organization of Report

 

2.0 Project Rationale

2.1 Introduction

2.2 Changes in Project Context and Review of Assumptions

2.3 Validity of Project logic

2.4 Continued Project Relevance

2.5 Implications for Project Continuation/Re-Formulation

 

3.0 Effectiveness and Effects

3.1 Achievement of Project Outputs

3.1.1 Planned and Actual

3.1.2 Implications of any Shortfalls

3.2 Project Outputs and Effects and the Project Purpose

3.2.1 Planned Effects and Contribution to Purpose

3.2.2 Unplanned Effects and consistency with purpose and IDB priorities

3.3 Contributions to the Project Goal

3.4 Conclusions and Recommendations

 

4.0 Efficiency

4.1 Introduction

4.2 Management of Project Inputs

4.2.1 Standards and quality control

4.2.2 Costs and cost control

    • management costs
    • costs of other inputs

4.3 Reporting through timely and useful information on

    • project finances
    • input scheduling and delivery
    • achievement of project outputs
    • project problems and risks

4.4 Problem-Solving

    • effective identification
    • timely solution

4.5 Resource Utilization

4.5.1 Cost Effectiveness Analysis

4.5.2 Cost Benefit Analysis

4.6 Conclusions and recommendations

 

5.0 Conclusion

5.1 Introduction

5.2 Project Sustainability

5.2.1 Institutional Capacity

5.2.2 Recurrent Costs

5.3 Lessons Learned

5.3.1 Operational Performance

5.3.2 Development Performance

5.4 Implications for Future Loans

5.5 Summary of Recommendations

 

References

Appendices

 

 

 

E. OPERATIONS EVALUATION REVIEWS OR IMPACT EVALUATION

Unlike the BEP and PPR, which look at the project itself, the OER or impact evaluation is a vital way for the Bank to gain a deeper understanding of its development impact and to use this both to celebrate its successes, and to learn from projects whose impact was modest, or negative. No matter how sound the design or how successful the project execution, only by studying the longer term impact can we fully understand project performance. Impact evaluations follow similar procedures to the other formal evaluations discussed in Chapter VI, though the issues move beyond implementation to the project's achievement of the purpose and contributions to the goal.

 

 

F. IMPACT EVALUATION FRAMEWORK

The impact evaluation framework suggested for use by IDB focuses on six major types of impact:

  • Technological
  • Institutional
  • Economic
  • Political
  • Social and Cultural
  • Environmental

 

The way these are treated in practice is suggested by the sample outline shown on Table 15.

 

Table 15 (a): Sample Outline
Project Impact Evaluation Report

Table of Contents

Abstract


1.0 Introduction

 

1.1 Background

 

  • Purpose of the Evaluation
  • Major Stakeholders

 

1.2 Project Description

 

  • Summary of Logical Framework
  • Completion Date
  • Other related projects since completion

 

1.3 Evaluation Methodology

 

 

1.3.1 Design/General Approach

 

1.3.2 Sources of Data

1.3.3 Instruments

1.3.4 Evaluation Team

1.3.5 Limitations

 

1.4 Organization of Report

 


2.0 Technological Impact

 

2.1 Introduction

 

2.2 Changes in Technology

2.3 Effects of Technology Change on Performance


3.0 Institutional Impact

 

3.1 Introduction

 

3.2 Effects on Institutional Capacity

3.3 Effects on Institutional Performance


4.0 Economic Impact

 

4.1 Introduction

 

4.2 Effects on Families

4.3 Effects on Communities

4.4 Effects on Sector, Economy, etc.

4.5 Cost-Benefit Analysis


5.0 Political Impact

 

5.1 Introduction

 

5.2 Effects on Political Parties

5.3 Changes in Method of Governance

5.4 Effects on increasing Good Governance


6.0 Social and Cultural Impact

 

6.1 Introduction

 

6.2 Effects on Families (size, structure, roles)

6.3 Effects on Employment Patterns

6.4 Community Effects (demographic)

6.6 Effects on Health

6.6 Effects on Education


7.0 Environmental Impact

 

7.1 Introduction

 

7.2 Effects on Quality of Environment

7.3 Effects on Genetic Diversity


8.0 Project Purpose and Goal

 

8.1 Introduction

 

8.2 Achievement of the Project Purpose

8.3 Realization of the Project Goal

8.4 Lessons Learned

8.5 Conclusion


References

Appendices

 

SUMMARY OF POINTS

  • Effective Mid-Term and BEP/PPR type Ex-Post Evaluations study continued rationale, effectiveness, efficiency, effects and impacts, and contribute lessons learned.
  • Ex-post impact evaluations examine technological, institutional, economic, political, social/cultural and environmental effects and impacts.

 

Top