EVO - Evaluation- A Management Tool for Improving Project Performance (a logical frame work) -3/97 -

III

EVALUATION AND PROJECT IMPLEMENTATION

 

It is frequently true that project managers have to deal with ten times more information than they need for making key decisions while they still lack the critical information they would need for making those decisions. Effective monitoring can easily reverse this. This chapter introduces the benefits of monitoring and presents the steps required for successful monitoring activities.

 

A. WHAT IS MONITORING?

Monitoring is a procedure for checking the effectiveness and efficiency of implementing a project by identifying strengths and shortcomings and recommending corrective measures to optimize the intended outcomes.

 

 

B. THE ROLE OF THE BANK'S COUNTRY OFFICE

The Country Office is responsible for ongoing monitoring and has the prime responsibility to identify performance problems and work towards their successful resolution. Effective monitoring enables both the Bank and the borrower to act before the problems get too far advanced for effective solution. The Country Office is tasked with following this monitoring process and reporting to management on important developments.

 

 

C. BENEFITS OF MONITORING

The benefits of monitoring are that it:

  • identifies flaws in the design and execution plan;
  • establishes whether project is carried out according to plan;
  • continually reviews the project assumptions thereby assessing the risk;
  • establishes the likelihood of output achievement as planned;
  • verifies that outputs continue to support the purpose;
  • identifies recurrent problems that need attention;
  • recommends changes to the project implementation plan;
  • helps identify solutions to problems;
  • identifies supplements to project required to enhance its effectiveness; and
  • establishes links between the performance of the operations in progress and the Bank's future assistance to the country.

 

Table 9
THE MONITORING PROCESS

  • Project Familiarization
  • Determine Information Requirements and Client Information Needs
  • Establish a Management Information System
  • Report to Bank Management and Executing Agency
  • Intervene to Improve Project Performance

 

 

D. STEPS REQUIRED FOR SUCCESSFUL PROJECT MONITORING

 

The monitoring role of the Country Office in coordination with the counterpart project manager is key to improved project performance. The five steps for successful monitoring, outlined in the Table 9, help ensure effective and successful project monitoring. Though some of these procedures are currently being applied to projects, it is useful to review them once again for the purposes of this handbook.

 

1. Project Familiarization 

If the project monitor (assigned Country Office professional) is not familiar with the project, s/he needs to understand its context and history. To do so, the monitor must:

  • review project documents, including the logframe, and available reports;
  • conduct interviews with the project design and project management teams/professionals;
  • visit the project site and interview executing agency personnel.

 

 

2. Determine the Information Requirements and Stakeholders' Information Needs

It is important that the right information be provided to the right people at the right time in order to ensure that monitoring assists in enhancing project performance. The project monitors must:

  • identify the primary and secondary stakeholders of monitoring;
  • clarify, with the counterpart project manager and corresponding Bank supervisor (or project team leader), who should receive monitoring reports and who should be copied (if done at the outset, protocol difficulties will be avoided later on);
  • establish, with the project team leader, the frequency of formal reporting; and
  • ascertain, with the counterpart project manager, whether the monitoring stakeholders have particular issues or concerns that should be incorporated.

 

 

3. Establish a Management Information System (MIS)

The monitoring information needs should be used to develop a management information system (MIS) for the project. This process will be simplified if a logframe has been developed for the project.

In order for an MIS to be effective, the information collected must:

  • be pertinent, in that it is the appropriate information;
  • be timely, in that it arrives when you need it;
  • be cost effective to collect;
  • answer strategic questions; and
  • streamline monitoring, evaluation, and special reporting activities.

 

The required steps for putting together an MIS plan are:

  • identify information users;
  • clarify users needs;
  • identify priority areas of information;
  • link information needs and sources (ie. determine which ongoing and existing data can be used and what needs to be generated);
  • determine appropriate methods for meeting information needs (ie. establish data collection methods);
  • identify roles and responsibilities;
  • identify reporting requirements and formats;
  • identify resources (human, financial, technological) required to make the system reliable and credible; and
  • establish feedback procedures.

 

 

4. Report to Monitoring Stakeholders

Reporting is both formal and informal. Many of these reporting procedures already exist, and good monitors know that the earlier they can identify and communicate issues (even informally), the easier the issues may be to resolve. However, we feel that the checklist below may be useful.

In general, data collected for monitoring must answer the following questions:

  • Are the outputs or components (eg. goods, services, technical cooperation, training and policy conditions/measures) being achieved as planned, in terms of quality, schedule and cost?
  • Are inputs (eg. bank disbursements, counterpart funds, co-financing, project management, project implementation staff, goods and services, etc.) being delivered/provided in a timely and cost-effective manner?
  • To what extent are the project assumptions, as identified in the logframe, still valid? Have certain assumptions become risks that may hinder the project's performance or progress?
  • To what extent is the borrower complying with contractual clauses?
  • To what extent will the project likely achieve its development objectives?

For its Annual Report on Portfolio Management, the Bank has developed a new project performance review form which summarizes the status of these issues, as shown on Table10.

 

Table 10 (a)

PROJECT PERFORMANCE MONITORING REPORT (PPMR)

I. BASIC DATA (Amounts in US$ millions)

Country:
Borrower:
Executing Agency:
Sector:

Project Title: 

Project Number:
Loan Number(s):

Date of Board Approval:
Date of Contract:
Date of Eligibility:
Date of Final Disbursement:

Sector Specialist: 

Report Update: 

Headquarters Staff Member Assigned: 

Date of Last Review by Representative:

Years in Execution: 

Cumulative Extension of Final Disbursement Date (months): 

Cancellation(s): Date Amount

Original Loan(s) Amount(s):
Current Amount:
Disbursements:
% Disbursed:
Original Cost:
Current Cost:

Loan Modality:
[ ] Investment
[ ] Time Slice
[ ] Sector
[ ] TC
[ ] Hybrid
[ ] Other

Sector Loans - Amounts
First Tranche:
Second Tranche:
Third Tranche:

Disbursement Dates - Expected

Actual

Cofinancing

Source(s):
Counterpart:

Amount

II. PROJECT PURPOSE(S)/DEVELOPMENT OBJECTIVE(S) AND KEY PERFORMANCE INDICATORS 

The development objective(s)/purpose(s) must describe the specific impact or results expected from the components of the project. Performance indicators should be few and readily measurable. There must be an agreement with the borrower/executing agency on objective(s) and performance indicators.

Project Development Objective(s):

1.
2.
3.

Key Performance Indicators:

1.
2.
3.

Have project development objective(s) changed since Board approval?

Have project development objective(s) and indicators been agreed to with borrowers/executing agencies?

Have project development objective(s) and indicators changed since the last report?

Has the project been officially restructured?

Is the borrower maintaining performance indicator data?

[ ] yes


[ ] yes


[ ] yes


[ ] yes

[ ] yes

[ ] no


[ ] no


[ ] no


[ ] no

[ ] no

 

 

 

 


Date:

III. PROJECT IMPLEMENTATION PROGRESS

Implementation of project components relates directly to the achievement of development objective(s)/project purpose(s). When classifying implementation progress, please take into consideration the physical progress and the quality of the components, as well as the performance indicators for delivery

Components/Outputs

Key Delivery Performance Indicators

Classification

HS

S

U

VU

1.
2.
3.
4.
5.
6.

1.
2.
3.
4.
5.
6.

 

 

 

 

Implementation Progress Summary Classification (IP):

[ ] Highly Satisfactory (HS)
[ ] Satisfactory (S)

[ ] Unsatisfactory (U)
[ ] Very Unsatisfactory (VU)

Check key reasons for very unsatisfactory or unsatisfactory classifications:

[ ] Legislative approvals
[ ] Borrower commitment
[ ] Counterpart funding shortfall
[ ] Central Government
[ ] State/Provincial/Local
[ ] Executing agency institutional capacity

[ ] Community/political opposition
[ ] Executing agency staff
[ ] Consultant performance
[ ] Inter-agency coordination
[ ] Supplier/contractor performance
[ ] Project/component design

[ ] Contract condition compliance
[ ] Bank efficiency
[ ] Procurement difficulties
[ ] Environmental issues
[ ] Cost overrun

Identify causes, other reasons, or additional comments:

 

IV. KEY ASSUMPTIONS

 

The achievement of project development objectives and the delivery of components often depends on assumptions which are outside the control of the project manager. In assessing whether such assumptions will be met, one needs to take both quality and timeliness into consideration. Identify and report only on the assumptions that may affect the achievement of development objectives/delivery of components. These assumptions may be implicit or explicit (an example of an implicit assumption might be that the regulatory authority permits timely adjustment of tariffs of a public enterprise). When reporting on key assumptions, please take into consideration compliance with contractual conditions that are of particular importance to the project and that may have been formulated to increase the probability that an assumption will hold true. The summary classification refers to the probability that the assumptions will occur and permit meeting of project objectives.

Assumptions

Assessment of Probability of Occurrence

High

Low

Assumptions Related to Development Objectives

1.
2.
3.
4.
5.
6.

 

 

Assumptions Related to the Implementation of Components

1.
2.
3.
4.
5.
6.

 

 

Summary Assumptions Classification:

[ ] High Probability

[ ] Low Probability

V. ACHIEVEMENT OF DEVELOPMENT OBJECTIVES 

To classify the likely achievement of development objective(s), take into consideration the project's development objective(s), the performance indicators, components performance, and key assumptions.

List major factors on which the development objective(s) classification is based:

 

Development Objective(s) Assessment (DO):

[ ] Highly Probable (HP)
[ ] Probable (P)

[ ] Low Probability (LP)
[ ] Improbable (I)

VI. SUMMARY OF PROJECT STATUS, ISSUES AND ACTIONS

Project Status:

 

Identify the major issues affecting project implementation and/or achievement of development objectives:

1.
2.
3.
4.
5.
6.

 

 Although the format may change over time, the PPMR is designed to track the status of inputs/ouputs, borrower compliance clauses, assumptions/risks, and the likelihood of the achievement of development objectives. Regardless of the format, it is important that the information be: 

  • timely;
  • as succinct as possible and convey the information essential to its users;
  • of an adequate technical quality in terms of content, presentation, credibility, action-orientation.

 

5. Intervening to Improve Project Performance

Thus far, two actors have been mentioned in conjunction with project monitoring: the counterpart Project Manager and the Country Office. However, there may be cases where specialized monitoring assistance may be required, in addition to these operational monitors. Examples of these are shown on the table below. Monitors are therefore not merely passive collectors and communicators of information. Effective evaluation monitors need to be available to support project management and Country Office monitors in many formal and informal ways. These specialized services are provided by, or through, EVO's advisory functions and may be used to contribute expertise to management. On other occasions, EVO may be asked to undertake special monitoring assignments itself to assist project management, such as special monitoring reports or mid-term evaluations.

 

Both in their regular and their special work, monitors need to turn their findings into appropriate action.

 

Table 11

Findings by Operational Monitors

Monitoring Actions

1. No significant deviation from plans

1. This confirms short term approach in the execution of the project; no modification required at this stage.

2. Deviations from schedules for provision of inputs and/or achievement of outputs due to difficulties with implementation

2. Monitor needs to assess the effects of slippage; probable causes of slippage; ascertain whether changes required in project implementation and what type to recommend.

3. Evidence of problems related to project management

3. Monitor needs to assess seriousness of problems; analyze and understand the likely causes of problems; search for solutions at the lowest possible level of intervention; plan adjustments and modifications. A Mid-term evaluation may be required.

4. Significant deviations from schedules for provision of inputs and/or achievement of outputs due to inadequate project design

4. Monitor needs to determine whether the design can be suitably revised so that the purpose (if valid) can still be achieved or recommend project cancellation. An Execution and Monitoring Review may be required.

5. Significant cost over-runs

5. Monitor needs to understand causes, effects, alternative courses of action; recommend such measures as audit, major re-design or cancellation.

6. Lack of project performance, however nature and causes are unclear

6. Recommend and/or conduct a mid-term evaluation.

 

Out of the many tools used by the Bank to identify and address performance problems, two formal evaluation procedures are particularly useful:

 

a. Execution Monitoring Review

This mechanism addresses design problems by involving project stakeholders in an intensive workshop to re-formulate the project. The logframe is the major tool for re-analyzing the development problem, the stakeholders, project objectives and benchmarks, indicators, and assumptions. Moreover, economic, financial and technical specialists should also participate in this exercise. The earlier such a review can be initiated, the better, so that effort is not wasted on trying to implement an ineffectual design.

 

b. Mid-Term Evaluation

A mid-term evaluation can take place at any point during implementation. It is frequently used to analyze implementation problems more fully, or to aid decision-making in time-slice programming. The requirements and methods to be incorporated in mid-term and impact evaluations are described fully in Chapter VI. The framework used for mid-term and ex-post evaluations is an important one and is presented fully in Chapter V. The issues addressed in a mid-term evaluation depend on the reasons for performing this type evaluation, but they typically follow a generic outline. A sample report outline for a mid-term evaluation is shown on Table 12.

 

Table 12: Sample Report Outline - Mid-Term Evaluation

 

Table of Contents

Abstract

1.0 Introduction

1.1 Background
    • Purpose of the Evaluation
    • Major Stakeholders

1.2 Project Description
    • Summary of logframe
    • General Status (start, finish date)

1.3 Evaluation Methodology

1.3.1 Design/General Approach

1.3.2 Sources of Data

1.3.3 Instruments

1.3.4 Evaluation Team

1.3.5 Limitations

1.4 Organization of Report

 

2.0 Project Rationale (Note: Sometimes this follows discussion of efficiency and effectiveness)

2.1 Introduction

2.2 Changes in Project Context and Review of Assumptions

2.3 Validity of Project logic

2.4 Continued Project Relevance

2.5 Implications for Project Continuation/Re-Formulation

 

3.0 Efficiency

3.1 Introduction

3.2 Management of Project Inputs

3.2.1 Standards and quality control

3.2.2 Costs and cost control
    • management costs
    • costs of other inputs

3.3 Reporting through timely and useful information on

3.3 Reports on:

    • project finances
    • input scheduling and delivery
    • achievement of project outputs
    • project problems and risks

3.4 Problems in Project Implementation
    • identification
    • analysis and proposed solution

3.5 Conclusions and recommendations

 

4.0 Effectiveness

4.1 Achievement of Project Outputs

4.1.1 Planned and Actual to Date

4.1.2 Implications of any Shortfalls

4.2 Project Outputs and Effects and the Project Purpose

4.2.1 Planned Effects and Contribution to Purpose

4.2.2 Unplanned Effects and consistency with purpose and IDB priorities

4.3 Conclusions and Recommendations

 

5.0 Conclusion

5.1 Introduction

5.2 Continued Project Relevance

5.3 Measures to Increase Performance

5.4 Lessons Learned

5.4.1 Operational Performance

5.4.2 Development Performance

5.5 Implications for Future Loans

5.6 Summary of Recommendations

References

Appendices

 

 

SUMMARY POINTS

 

  • Valid and timely monitoring information may improve project implementation
  • Useable mechanisms to collect and distribute information facilitate decision-making for improving project performance
  • Execution and monitoring reviews can help re-formulate projects for improved performance
  • Mid-term evaluation can be an effective means to diagnose performance problems in implementation

 Top