EVO - Evaluation- A Management Tool for Improving Project Performance (a logical framework) -3/97

This chapter is intended for those who are, or may be, required to manage formal evaluations at Mid-Term and Ex-Post (PCR, PPR, BEP, and OER) levels. Sometimes these evaluations are conducted by external evaluators contracted for the purpose. In such cases, this section will provide the evaluation manager and the evaluator a step-by-step process to help the evaluation manager control the quality of an evaluation, to ensure a successful, effective evaluation. In other situations, formal evaluations are conducted by Bank staff and there is no need for contracting and less formal terms of reference can be utilized. Thus, the reader should adapt the step-by-step approach outlined here to suit the evaluation needs of the particular situation.

Successful evaluations require the context of an operational BES and the combined expertise of evaluators, managers, and stakeholders. The project evaluator is responsible for the content and presentation of the evaluation report. The evaluation manager is responsible for managing the process of evaluation. Other stakeholders are expected to provide data as requested and respond to the findings, recommendations and lessons learned presented in the evaluation report.

Sometimes the need for an evaluation is identified during the project approval process. At other times, it arises in response to problems experienced during implementation. In some instances the need for an evaluation may be as part of EVO's responsibility to evaluate selected projects. Whatever its origin, once the need has been established, the evaluation must be managed.

Professional evaluators have the necessary "technical" evaluation skills and knowledge (such as appropriate evaluation designs and research methods, data collection and analysis techniques). But the evaluator's expertise must also fit into the IDB's existing processes - a task for the evaluation manager (who may be a CO professional, a Headquarters operational department professional, an EVO specialist, etc.). Evaluation management is necessary at all stages of an evaluation: at the design stage, to identify and administer the human and financial resources to carry out the evaluation; throughout the evaluation, to assure quality control; and once the evaluation is completed, to ensure that its results are disseminated to stakeholders.

Effective management of the evaluation process is key to producing high quality evaluations.

The evaluation manager needs to ask some basic questions before moving to a precise description in a formal terms of reference:

WHY IS THE EVALUATION BEING DONE?

• What are the big questions to be addressed by the evaluation.

WHO ARE THE PRIMARY AND SECONDARY STAKEHOLDERS OF THE EVALUATION?

• Who needs what answers provided by the evaluation?

WHAT ARE THE MAJOR EVALUATION ISSUES RELATING TO THE BANK'S FRAMEWORKS FOR MID-PROJECT AND EX-POST EVALUATIONS?

• How do the big questions map into the IDB frameworks and the BES?

WHO WILL CONDUCT THE EVALUATION?

• What type of expertise is required to conduct the evaluation? Will it be conducted by an internal or external team?

WHO WILL PARTICIPATE IN THE EVALUATION?

• Are there particular demands for time and resources from the borrower, the implementors, and/or the beneficiaries?

WHEN WILL THE EVALUATION TAKE PLACE?

• When is the information required? When is it feasible to conduct the evaluation given participants' schedules, seasonal effects on travel and access to beneficiaries?

HOW MUCH WILL THE EVALUATION COST IN TERMS OF FEES, EXPENSES AND PARTICIPANTS' TIME?

• What level of effort is required given the anticipated benefits of the evaluation? How much time and effort can reasonably be demanded of stakeholders?

The Terms of Reference (TORs) define the evaluation and are the major contractual control on its outputs and quality. TORs may be prepared by various people, and this section clarifies what formal TORs should contain. The evaluator (either an individual or a team) is sometimes asked to comment on draft TORs, and the evaluator will need to discuss and clarify them before attempting to prepare a workplan as discussed later on in this chapter.

1.0 PROJECT BACKGROUND

This section of the TORs should include:

• The goal and purpose of the project from the logframe
• Context and rationale of the project
• Inputs by type and dollar value
• Start and planned completion dates

2.0 REASONS FOR EVALUATION

This section should present the main reasons for which the evaluation is being conducted. Examples of reasons for conducting an evaluation are:

• To improve performance during implementation
• To review the design after major changes take place in the context
• For accountability at a milestone, for example the mid-point or Ex-Post, or, a time-slice after a particular disbursement.
• To trace long term effects and impact
• To learn from successes and shortcomings for future programming

3.0 EVALUATION STAKEHOLDERS

In this section, the primary and secondary stakeholders of the evaluation should be identified. Primary stakeholders would typically include the beneficiaries, the Country Office, Executing Agents, and the IDB Board of Governors. Secondary stakeholders are all the other interest groups - Partners in Development, Partners in Research, Lobbyists, NGOs, etc.

4.0 SCOPE AND FOCUS

This section should identify which of the project components will be covered in the evaluation and the major issues for the evaluation to examine. These issues will normally reflect the issues in the appropriate IDB Evaluation Framework (see Chapter V) suitably tailored to reflect the reasons for this evaluation.

5.0 CONDUCT OF EVALUATION 

Requirements for conducting the evaluation that must be followed by the evaluators are identified here. The requirements should pertain to: 

• Type and timing of the Workplan
• Methodology requirements

6.0 COST 

The TORs should provide an estimate of the cost of the evaluation.

7.0 DESCRIPTION OF REQUIRED EVALUATOR(S) 

Required qualifications (eg. sectoral expertise, evaluation expertise, etc.) for the evaluator. In the case that the evaluator will be a team of people, its size and the roles and positions of team members might also be indicated in the TORs.

8.0 SCHEDULE 

A schedule for the evaluation should be outlined, including:

• Duration of evaluation
• Allocation of time at the various sites to be visited throughout the evaluation (i.e. in the field, at headquarters, etc.), if appropriate

9.0 AUTHORITY AND RESPONSIBILITY

The roles and levels of authority must be stated:

• Roles of Bank personnel
• Role of evaluator
• Procedures to amend TORs
• Reporting relationships

10.0 EVALUATION PRODUCTS

The products to be produced as a result of the evaluation process must be identified in the TORs. These typically include:

• Workplan
• Draft report
• Evaluation briefing
• Final report

The key to getting a job done well is to assign it to a competent person. As IDB evaluations become more complex, the skills required to conduct them are becoming more numerous and more specialized.

A good evaluator will demonstrate the following characteristics:

TECHNICAL EXPERTISE

• Understanding of, and experience in, the required evaluation methodologies.

SECTORAL EXPERTISE

• Expertise in the sectoral area of the project being evaluated (education, sanitation, environment, etc.).

CREDIBILITY

• Recognized as expert judge(s) by the main stakeholders of the project and its evaluation.

IMPARTIALITY

• No conflict of interest with any of the parties involved in the project evaluation.

COMMUNICATION

• Able to communicate the evaluation results in a manner that is easily understood by all parties.

INTERPERSONAL SKILLS

• Able to interact with all parties in a sensitive and effective way.

AVAILABILITY

• Be available to conduct the evaluation at the required level of depth in the specified timeframe.

Because of their complex nature, some projects may require a team of experts. In such instances, the evaluation team also needs:

LEADERSHIP

• An identified team leader accountable for project results and for all levels of quality control is required.

COMPLEMENTARY SKILL SET

• Evaluation teams should be composed of evaluators from multiple disciplines matched to the requirements of the project being evaluated.

ABILITY FOR TEAMWORK

• Unless the team has a track record of working together, there is always the risk that the benefits of a team will be outweighed by non-functional disagreements and incoherent reporting.

EFFECTIVE MECHANISMS FOR COORDINATION AND COMMUNICATION

• When team members are widely dispersed, procedures must be developed to ensure they have good communication and time together for joint planning, analysis and report preparation.

Both internal and external evaluators are useful for an evaluation system such as the BES. The usefulness of both cannot be discussed in the abstract. Therefore such usefulness must be considered within the framework of the BES. In such a context it is obvious that the internal evaluator is the only one who not only has an understanding of the process and procedures which lead to project identification, design and execution, but is also involved in ensuring the operation of the BES and therefore the generation of evaluable products. However, given the diversity of potential technical areas, the need to assign resources on an intensive basis in many specialized fields, plus the need to find innovative approaches to new problems, the use of external evaluators as well may be justified. Objectivity alone cannot be a reason for hiring an external evaluator as it would imply that an internal evaluator is biased and cannot be trusted.

Evaluators need a number of skill sets as outlined above. Additionally, the evaluator (be it a team or an individual, whether internal or external) selected should meet the following requirements:

1. Know the Bank, its programs, operations and evaluation procedures.
2. Be free from organizational bias and serve as arbitrators/facilitators between parties.
3. Be directed to ensuring follow-up on recommendations and lessons learned.
4. Be available for intensive work within required timeframes.
5. Always bring fresh perspectives, insights, experience and recent state-of-the-art knowledge.
6. Must not avoid looking for facts or forming conclusions that are negative or reflect badly on the IDB
7. Be aware of constraints on feasibility of recommendations.
8. Be free for full participation or intensive work.
9. Should not be perceived as an adversary, arousing unnecessary anxiety.
10. Should not be constrained by organizational role conflict
11. Should be not be unfamiliar with local political, cultural, and economic environment.

The evaluation workplan is normally the first output for the evaluator. It is the evaluator's response to the TORs and is the proposal for conducting the evaluation.

In order to adequately prepare a workplan, the evaluator should review the TORs with the manager of the evaluation in order to clarify the requirements. Additionally, the evaluator will need basic information on the project that can be obtained through reviewing major project documents and consulting with the country office.

The workplan will be submitted to the manager of the evaluation for approval. The approved workplan then directs the evaluation process, although slight departures from the workplan might become necessary as the evaluation develops. Procedures for approving such deviations should be established when the workplan is first approved.

1.0 PROJECT OVERVIEW

The workplan should begin with an overview of the project that is around 1-2 pages long, and:

• Summarizes the project's background and context
• Identifies the project's goal, purpose, inputs and outputs, by referring to its logframe (if the project was designed/executed without a logframe, the evaluation should attempt to reconstruct one on the basis of available project/loan documents)
• Incorporates the results of the evaluation team's preliminary review of major project documents and consultations with the project team.

2.0 EVALUATION MANDATE

This section essentially reiterates the reasons for the evaluation and its scope and focus, as identified in the TORs. The information is included in the workplan in order to provide, within a single document, a complete record of the expectations for the evaluation. The objectives of the evaluation are often expanded and/or clarified in this section of the workplan.

3.0 EVALUATION MATRIX

One of the major functions of the workplan is to re-conceptualize the TORs into a more operational framework that applies directly to the project evaluation. The workplan turns the evaluation questions introduced in the TORs (examples of such questions are presented in "Scope and Focus" of the TORs section earlier in this chapter) into a tighter and more project specific evaluation matrix such as the one depicted on Table 17.

i) The major questions should relate with each level of the logical framework as follows:

ii) The additional questions should emerge either as possible questions, as a result of the expected results of the major questions (very speculative during the first stages), or as a result of the specifice sectoral or technical characteristics of the project.

The evaluation matrix summarizes the scope of the whole evaluation by identifying the: 

• Issues suggested by the TORs.
• Questions suggested by the TORs.
• Essential sub-questions which need to be answered.
• Objectively verifiable indicators.
• Basis on which evaluation judgements are to be made
• Sources of data and the methods of collecting data

This matrix will assist in the development of lessons learned from the project, and will facilitate information feedback at the mid-term evaluation phase for possible project redesign, and at the impact evaluation phase for consideration when designing future projects.

4.0 METHODOLOGY

Some workplans summarize the general approach of the evaluation at the introduction of the methodology section. In a paragraph or two, the general design or type of evaluation and major methods of data collection and analysis are stated. A rationale for the choice of methodology is also provided.

Sub-sections should provide more detail on the data collection and analysis, namely:

• Sources of Data: Although sources of data are included in the evaluation matrix, they should also be summarized here.
• Instruments: Data collection instruments that will be used in the evaluation should be described.
• Data Collection Procedures: Although methods of data collection are listed in the evaluation matrix, details on the overall strategy for data collection should be provided in the workplan.
• Data Analysis: Major methods of analysis of qualitative and quantitative data should be described.

5.0 EVALUATION TEAM

When the evaluator consists of a team, rather than one individual, the name, role and responsibilities of each team member should be indicated.

6.0 ACTIVITY/EFFORT ANALYSIS

The workplan must list the required activities for the evaluation, as well as an estimate of the level of effort necessary (in person-days) to complete each one.

7.0 SCHEDULE OF ACTIVITIES

An evaluation schedule should be proposed in the workplan. The schedule must, of course, link to the activities and effort analysis presented in the workplan. The dates of milestones (field visits, submission of workplan and evaluation report, etc.) should be clearly indicated. The schedule can be further broken down by evaluation team member, if appropriate.

8.0 BUDGET

A budget, detailing fees, travel and other expenses, should be included in the workplan. It should be linked to both the strategy for data collection and analysis, as well as the activities and effort analysis.

9.0 REPORT OUTLINE

A proposed outline for the evaluation report should be included in the workplan. Although it will invariably change throughout the evaluation process, the outline at least indicates what will be covered in the report and gives an idea of how issues will be presented. (Sample outlines for Mid-Project and Ex-Post evaluations are presented in Chapters IV and V, respectively.)

Successful evaluations report on their findings in many ways. They often incorporate briefings with stakeholders, opportunities to assess conclusions and recommendations, electronic dissemination of conclusions, recommendations, lessons learned; however, virtually all professional evaluations produce a written evaluation report. It becomes the official record of what happened and records the judgements made. The report is also the major official stimulus to project re-formulation and to data systems which collectively contribute to knowledge about development performance. For these reasons, the evaluation report is the most important output of the evaluation process.

International standards of good evaluation practice suggest that evaluation reports should conform to the standards specified below. While some very good evaluations may not have all the required elements, it is unlikely that an evaluation whose report lacks many of these elements would be considered good by accomplished professional evaluators.

1. Standards of Language and Presentation 

• Have a cover or title page indicating the name of the project, author and date
• Have a table of contents linked to page numbers of the report
• Have a summary or abstract outlining the major findings, conclusions and recommendations
• Be divided into chapters and/or sections which relate to the table of contents
• Be clearly and succinctly written in good literary style
• Evaluation abstract, recommendations and lessons learned are separate and distinct and written to conform to IDB requirements
• All figures, graphs, tables are clearly presented with complete and suitable titles and labels
• Provide complete and accurate information on any references included in the report

2. Technical Standards 

• Describe the purpose of the evaluation and append the terms of reference
• Answer the questions posed in the terms of reference
• Describe the methodology used to collect and analyze the data
• Indicate any limitations of the evaluation or its methodology
• Indicate the reliability and validity of the data used
• Describe any samples drawn by method of sampling, number of targeted and available elements
• Include the major data, suitably analyzed, on which the conclusions are based

There are two separate elements in the utilization of evaluation results:  

• implementation of specific recommendations;
• dissemination of information; and
• a combination of implementation and dissemination.

Recommendations are more easily absorbed into future project activities where there is a direct link with the evaluation. In a mid-project evaluation the recommendations should be incorporated in the re-design/planning and implementation documentation where relevant. There should be evidence provided in the documentation which shows how recommendations are incorporated into the new or re-formulated project.

Within the Bank, implementation refers to Management actions in response to either a) policy decisions by the Board of Executive Directors which stemmed from recommendations of evaluation reports, or, b) suggestions and recommendations from mid-project evaluations which Management devided to include in the design and planning of projects or programs. There should be evidence provided in the documentation which shows how these recommendations and suggestions are incorporated into the new or reformulated project.

The dissemination of evaluation methods is the link to other projects and other country offices, and applies particularly to end-of-project and impact evaluations. Appropriate dissemination of results is important for all evaluations. It is this activity in the project cycle which should transmit the relevant lessons learned from evaluations for utilization in subsequent projects. The effectiveness of the dissemination process, therefore, is important and is measured in terms of their use and availability for future projects.

Dissemination of results, lessons learned and best practices most often refers to suggestions on project preparation, design, implementation and effectiveness of projects. It may also refer to impacts and overall goal attainment. Results may be particulat to projects or programs in one country or lending area, while others are general in nature, with implications for Bank policies and strategies.

Ethics are becoming increasingly important as people are more frequently subjected to all forms of social science research. Bank evaluators are expected to be particularly respectful of:

Being sensitive to the beliefs, manners and customs of the people involved both in data collection and reporting.

Respecting people's right to provide information in confidence and ensure that routine data cannot be traced to its source.

Not putting local evaluators in the position of being seen to be critical of their society requires team responsibility.

Ethically, evaluators have a responsibility to bring to light significant issues and findings which may not relate directly to the TORs.

Individual performance reviews are not normally a part of Project Evaluations; the evaluation of management functions must be balanced with this general principle.

Evaluations sometimes uncover evidence of wrongdoing which should be reported to the responsible national or Bank officials.

Data should be validated through use of multiple measures or sources.

Evaluators should make reasonable efforts to share evaluation results with those who have a right to know without denying their loyalty to the evaluation sponsor.

Top