EVO - Evaluation- A Management Tool for Improving Project Performance (A logical framework) -3/97-

This annex is just one part of EVO's overall approach to improve project performance through evaluation. It fits within a logical approach shown in the abbreviated logical framework below.

The annex provides a set of tools to help project teams plan for monitoring and evaluation (M&E) throughout the project cycle, and to incorporate this plan into the project design.

For those who approve loans, the annex provides an analysis of what to look for in loan documents to ensure evaluability.

The result should be better monitoring and evaluation (M&E) plans in loan documents.

There are seven sections in this annex. Section 2 provides a brief discussion of monitoring and evaluation and its benefits. Section 3 presents an overview of what an M&E plan should contain and provides a sample. Sections 4 through 6 assist project designers in ensuring that a project has what is required to be evaluated at each stage of the project cycle.

An unfortunately high number of the Bank's loans experience problems with performance - problems that range from difficulty with executing loans that have been approved, to active loans that continue to support projects that are not achieving the intended results.

If invested in adequately and managed appropriately, monitoring and evaluation can provide a considerable return on investment by improving participation, accountability and development performance. To be effective, monitoring and evaluation must be considered throughout the project cycle, from design to completion.

At the design stage, projects need to be planned so that they can be successfully monitored and evaluated later on. Unfortunately, relatively few loan proposals contain the basic elements required to monitor project progress or to enable later determination of whether or not the objectives were achieved. This book examines the notion of incorporating a Monitoring and Evaluation Plan into project designs.

At the approval stage, everyone involved in reviewing loans and project approval has a responsibility to ensure that each new project will be able to benefit from future evaluation processes. At this stage, an analysis of evaluability will help confirm that the project design is sound and based on a supportable logical framework analysis, that it incorporates lessons learned and best practices from past experience, and that it provides specific criteria for monitoring and evaluation throughout the project. An evaluability assessment must be part of the process of loan approval. This book includes an "evaluability" checklist for board members involved in loan approval.

The vital function at this stage is to ensure that approved loans become active.

Valid project designs are not enough - projects must also be implemented effectively. It would appear, however, that many of the Bank's loans are experiencing implementation difficulties. When monitoring and evaluation are used as tools by both the project manager and the funder, the effectiveness of implementation can be increased. Concrete planning includes scheduling and allocating resources for standard monitoring and evaluation activities and planning for follow-up. It is also important to have a plan for how to handle unexpected performance problems - the conditions, for example, that signal the need for a Mid-Term evaluation if this was not part of the original plan.

Evaluation information gathered at the project completion stage is an indispensable source of knowledge for future projects. Planning for the generation and dissemination of the collective lessons is an important element in the ongoing cycle of projects and loans. Effective Ex-Post and Impact evaluations are important tools to help improve long-term performance.

A solid plan for monitoring and evaluation must be completed in the design phase of a project. The process of planning for M&E involves detailed scrutiny of each stage of a project - Design, Start-up, Implementation, Completion - and a detailed analysis of M&E requirements at each of those stages.

During the design phase, the primary M&E concern is to design a project that can be evaluated later.

Throughout the implementation of a project, mechanisms are needed to track the loan's performance - the ongoing relevance, effectiveness, and efficiency of the project.

At the end of a project, the M&E function is responsible for measuring and assessing a project's results, its impact, and to contribute lessons learned for the future.

Designing an evaluable project is discussed in Section 4. A step-by-step examination of M&E at each stage of a project is presented in Section 5. These sections will give you tools and question checklists to complete your detailed analysis.

The result of detailed planning should be a concise plan that satisfies the five standard requirements for monitoring and evaluation:

Effective plans include standard and special checkpoints throughout the life of the project. The types of checkpoints used by IDB are: 

• Design - Baseline Studies; Logical framework analysis and incorporation of lessons learned

• Start-Up - Monitoring of start-up delays; completion of baseline data base as required

• Implementation - Semi Annual Progress Reports; Early Warning Reports; Mid-Term Evaluations

• Completion - Project Completion Reports (PCR); Ex-Post Evaluations; Impact Evaluations

To support the country in taking responsibility for a loan, the M&E plan should indicate who will be responsible for M&E activities. This may vary from country to country, but should stipulate the roles of a wide range of players: project team members, external resources, and other project stakeholders.

To ensure that monitoring activities can be carried out as planned, resources must be allocated to these activities as a project cost in the initial design. The cost of monitoring and evaluation activities throughout the project cycle includes fees, expenses and participants' time.

EVO can help suggest a range of acceptable costs, or planners can do so, based on either: 

• the historical average for projects of this type, or
• an actual project budget including a person-day analysis.

It is difficult to generalize about the resources required for monitoring and evaluation as a percentage of the project budget. So much will depend on the sector involved, the geographic scope of the project, its complexity, etc. Mid-term evaluations are generally cost-effective, as they provide an opportunity to refocus a project and make necessary corrections mid-stream. Project designs should, at a minimum, allocate sufficient resources for a mid-term evaluation. As a rule of thumb, a mid-term evaluation requires approximately 100 person days of consultant time.

It is important not only to collect M&E information, but to report on this to the appropriate groups. Who gets M&E information? When? What do they do with it? M&E reports should conform to IDB and EVO standard report formats.

M&E has both standard reports and special situation reports. The M&E plan should indicate the timing and dissemination of standard reports and should indicate the conditions which will signal the need for a special type of monitoring report.

Monitoring and evaluation activities are only as valuable as the information they generate and the actions they stimulate. Projects need mechanisms to correct and adjust the project to keep it performing. The M&E Plan should indicate how the project will follow up on monitoring information.

C. A SAMPLE M&E PLAN - ABBREVIATED NARRATIVE

The M&E plan should be part of the project design and should be presented in the loan document. It should be at a suitable level of detail and in a form that can be easily understood and evaluated by the Board. The following sample M&E plan is in narrative form and suggests the components that should be included in a Loan Proposal.

Existing baseline studies include a nutrition survey, an IDB poverty study and a school inventory analysis. The poverty study is to be updated by extension to the Parador Region of Qualiquó, previously omitted. This will be contracted by the Executing Agency at a cost of $50,000 charged to the loan and reported on in the first semi-annual progress report.

2.Lessons Learned

The project design responds to an important lesson learned in prior education projects in Qualiquó: that equity cannot be served without incorporating private sector education, that accounts for upwards of 20% of education of low-income children.

The Executing Agency will report on project disbursements to date and for the period, and projections for the fiscal year semi-annually.

The only plans at this time is for the Ministry of Education to report to IDB one year from the date of approval of the loan if project implementation has not yet been initiated. Such a report will provide a full analysis of the reasons for delay and recommended actions.

The IDB will conduct annual project performance reviews to evaluate the achievement of targets as outlined in the Logical Framework of each project. The reviews will provide information about the amounts and the nature of annual budgetary allocations to recurrent and investment expenditures, including the Borrower's commitment to allocate resources for preventative maintenance of rehabilitated schools.

A mid-term evaluation is scheduled for approximately two years after project start-up. It will assess project performance in the national context, and will review project assumptions. In particular, the Borrower undertakes to achieve agreed performance and expenditure targets for education, and this needs to be reviewed mid-term in order to adjust the schedule of project outputs to the situation at that time. Several factors may require that changes be made in the project:  

• the performance and status of the $3 million fund for community participation in education;
• evidence that the Borrower has created a "Fondo Editorial" to generate income from sales of curriculum that will support future curriculum revisions; and
• government progress in developing legislation for participatory governance of education.

This will be contracted by the Executing Agency with terms of reference developed jointly by the Ministry and IDB and conducted by an external evaluator - $100,000 has been allocated for the purpose. IDB will receive the evaluation report and jointly with the Executing Agency will agree on plans of action to improve project performance in response to the evaluation findings.

The PCR will be submitted to IDB by the Executing Agency within 90 days after the final disbursement of the loan.

The Borrower may undertake to commission an ex-post evaluation within six months of project completion. The evaluation will examine project effectiveness in terms of the Logical Framework indicators and the Baseline study, project efficiency in terms of adherence to cost projections and through an analysis of its execution, and operational and developmental lessons learned that can inform the Government's and Bank's future work in education for low income groups. This will be conducted by external evaluators with the report submitted to the Bank. A sum of $150,000 has been allocated within the loan.

While there are no specific plans at this stage, the project falls within the Bank's priority of poverty alleviation in this country and is recommended for inclusion in future Bank impact evaluations of interventions to address poverty in Qualiquó.

M&E requirements can easily be represented in the project cycle as shown in Table 3.

The project design should include the collection of baseline data which can provide a starting point for the measurement of change throughout a project. It often happens that when projects are evaluated, appropriate data are difficult or impossible or too costly to acquire. Careful attention at the preparation stage can often do much to alleviate these problems. Baseline studies are often conducted before a project proposal is written, but if not, baseline data can be collected as an initial project activity.

The indicators identified in the logframe determine the type of information you need. Unless they already exist, baseline data may have to be collected for at least some of the key indicators identified in the project logical framework. The M&E plan should indicate what baseline data are required and how they will be obtained and/or updated. The plan should identify the status of data on each indicator.

Baseline data are often used in developing indicators to measure project success. Benchmarks are targets you are aiming for, and can be either "comparative" or "absolute".

A comparative benchmark is usually a standard or best practice that has been achieved within the relevant industry or sector. Some examples of comparative benchmarks: 

• The pupil:teacher ratio in the highest ranked primary schools is 15:1.
• The safest railways in North America invest $0.82 per $1 of revenue.

An absolute benchmark is generally based on scientific statistical evidence or on a professional standard. Some examples of absolute benchmarks: 

• The acceptable parts per million of a toxin in drinking water (scientific standard)
• All teachers must be licensed (professional standard)

1. What data are required for the indicators in the logical framework?
2. What data are available now?
3. Should additional baseline data be collected before the project is implemented?
4. Will data on these indicators be required for project monitoring activities? For end-of-project, and/or impact evaluations? Should a database be created?
5. What is the standard error (tolerance) in measurement? What sample size is required to provide estimates within this level of tolerance?
6. What are the most efficient ways of collecting the data? Is local expertise available?
7. What will be the cost of collecting the data relative to project size and available resources?
8. Is the economic viability of the project dependent on the baseline findings?

Lengthy delays in project start-up are a fairly accurate indicator of future project problems. Delays may be the result of many factors - political, environmental problems, lack of stakeholder commitment, or poor management. Whether these are within the control of the project or result from outside forces, they may present a serious threat to the success of the project and the performance of the loan. A good M&E plan will include a mechanism to flag unreasonable start-up delays.

Some questions to ask about start-up are: 

1. What is the expected loan approval date?
2. What is a reasonable timeframe for project start-up? Is this realistic given past history? Does start-up depend on season of the year?
3. What action will be taken to get project going or to cancel it? Who will make this decision?
4. If adjustments in the project design are required to get it going, what conditions will be put on these adjustments? (e.g. Will indicators need to be revised? Will new baseline data be required? How long will project be given to fulfill conditions?)

Project performance can be enhanced by using systems to monitor progress and make needed corrections. The following schema shows how the various reports relate to project activities: Progress reports are scheduled at specified intervals; achievement reports coincide with outputs; mid-term evaluations are scheduled for the project mid-point; the completion report is scheduled for 90 days from the end of project; ex-post evaluations are conducted within a year of project completion.

These reports are normally scheduled to start six months after eligibility or the first disbursement.

Some questions to ask about semi-annual progress reports are: 

1. Are these scheduled?
2. Who will be responsible for completing them? Who will participate?
3. How long will it take to complete these reports?
4. Who receives them?
5. What follow-up may be required? Who will be responsible?

Project performance reports should coincide with the completion of each project output/component.

Early warning reports should be prepared when the Project Manager feels that outputs will not be produced on time; when the quality of outputs will not meet expectations; and/or when the outputs are not likely to achieve the purpose for which they were intended.

Some questions to ask about the PPRF are:

1. Who will be responsible for completing this? Who will participate?
2. How long will it take to complete the report?
3. Who receives the PPRF?
4. What follow-up will be required? Who will be responsible?

A Mid-Term evaluation should be part of the project design. Mid-term evaluations are generally cost-effective, as they provide an opportunity to refocus a project and make necessary corrections mid-stream.

If a Mid-Term evaluation was not included in the project design, the M&E plan should consider the following:

1. What will flag the need for a Mid-Term evaluation?

• Time between loan approval and project eligibility (e.g. if more than one year delay, a mid-term evaluation may be required to review and validate the design.)
• Slippage in execution (e.g. If disbursements less than 50% of target in any component, a mid-term evaluation may be used to determine the cause and make corrections.)
• Delays in standard reporting (e.g. If reports do not arrive, a mid-term may be required to diagnose and suggest how to address the problem.)
• Public controversy (e.g. Public protests about environmental effects, corruption, can be investigated using a mid-term evaluation.)
• Audit problems - (e.g. Auditors' reports raise concerns.)

2. If a Mid-Term is required, who will conduct it?

3. How much will need to be budgeted for this?

4. Who will be responsible for managing the contract?

5. Who will receive the evaluation report?

6. Who will be responsible to follow-up on the recommendations of the report?

The most valuable long-term benefits of monitoring and evaluation activities are the knowledge they generate for the future. A good logical framework reflects lessons learned and best practices from previous projects. This knowledge comes from the M&E activities conducted at the project completion stage.

The M&E Plan should indicate how lessons learned from the project will be distilled and applied to future projects.

Some questions to ask about the Project Completion Report (PCR) are: 

1. When will this be done?
2. Who will be responsible for completing this? Who will participate?
3. How long will it take to complete the report?
4. Who receives the PCR?
5. Who will be responsible to disseminate the findings of the report?
6. Who is responsible for ensuring the PCR is prepared and received?

Some questions to ask about the Ex-Post Evaluation (BEP) are:

1. Who will conduct it?
2. How much will need to be budgeted for this?
3. Who will be responsible for managing the contract?
4. Who will receive the evaluation report?
5. Who will be responsible to disseminate the findings of the report?

Using monitoring and evaluation to improve project performance is a shared responsibility that involves all the partners in the enterprise at all stages of the project cycle. Borrowers and Bank officials need to work collectively so that the contributions of each partner inform the other and helps achieve the Bank's development agenda. Evaluation is a powerful tool if used well. EVO is pleased to assist the process when requested to do so.

Top |  Foreword | Introduction