At the design stage, you will define your evaluation questions, identify an appropriate methodology, and plan and budget for the evaluation activities. Use the following guidelines to help design and plan your impact evaluation.
a. Introductory readings:
- Impact evaluation in practice
- Evaluating the impact of development projects on poverty. A handbook for practitioners(Baker, Judy; World Bank, 2000).
b. Intermediate readings:
- Evaluating anti-poverty programs (Ravallion, Martin; World Bank, 2005).
- Program evaluation and spillover effects (Angelucci and Di Maro; IDB, 2010).
- Handbook on impact evaluation: quantitative methods and practices (Khandker et al.; World Bank, 2009).
- Recent developments in the econometrics of impact evaluation (Imbens and Wooldridge; Journal of Economic Literature, 2009).
- The mystery of vanishing benefits: an introduction to impact evaluation (Ravallion, Martin; World Bank Economic Review, Vol. 15, No. 1).
a. Experimental methods:
- In pursuit of balance. Randomization in practice in development field experiments (Bruhn and McKenzie; World Bank, 2008).
- Using randomization in development economics research:a toolkit (Duflo et al.; Center for Economic Policy Research, 2007).
b. Quasi-experimental methods:
- A primer for applying propensity-score matching (Heinrich et al; IDB, 2010).
- Instrumental variables and the search for identification: from supply and demand to natural experiment (Angrist and Krueger; Journal of Economic Perspectives, 2001).
- Regression discontinuity designs:a guide to practice (Imbens and Lemieux; National Bureau of Economic Research, 2007).
Agriculture
- Designing impact evaluations for agricultural projects (Winters et al.; IDB, 2010).
Early childhood development
- Methodologies to evaluate early childhood development programs (Behrman et al.; World Bank, 2007).
Education
- Impact evaluation for school-based management reform (Gertler et al.; World Bank, 2007).
- Guidelines for impact evaluation in education using experimental design (Bando, Rosangela; IDB, 2013).
Housing
- Impact evaluation for slum upgrading interventions (Field and Kremer; World Bank, 2008).
Industry
- Evaluating the impact of cluster development programs (Giuliani et al.; IDB,2013).
Labor markets
- Building in an evaluation component for active labor market programs: a practitioner’s guide(Card et al.; IDB, 2011).
Land property
- Impact evaluation for land property rights reforms (Conning and Deb; World Bank, 2007).
Microfinance
- Impact evaluation for microfinance (Karlan and Goldberg; World Bank, 2007).
Nutrition
- Methodologies to evaluate the impact of large-scale nutrition programs (Habicht et al.; World Bank, 2006.
Regional development
- Evaluating the impact of regional development programs (Winters and Sitja; IDB, 2010).
Science and technology
- Evaluating the impact of science, technology and innovation programs: a methodological toolkit(Crespi et al.; IDB, 2011).
Tourism
- Technical guidelines for evaluating the impacts of tourism using simulation models (Taylor, Edward; IDB, 2010).
Transportation
- Conducting impact evaluations in urban transport (Boarnet, Marlon; World Bank, 2007).
- Impact evaluation of rural road projects (Van de Walle, Dominique; World Bank, 2008).
Water and sanitation
- A guide to water and sanitation sector impact evaluations (Poulos et al.; World Bank, 2006).
- Power calculations spreadsheet: a tool for determining sample sizes.
- Random numbers generator.
- Impact evaluation Gantt chart: a spreadsheet to assist in the scheduling of impact evaluation activities.
- Data collection Gantt chart: a spreadsheet to assist in the scheduling of data collection activities.
- Budget template: a tool for estimating the costs of an impact evaluation.
- Concept note template: a document that describes the details of the impact evaluation methodology.
- Design template: a blank presentation to guide the main steps and components of an impact evaluation design.
- Impact evaluation checklist: a list of core activities to be carried out during an impact evaluation.
Prior to launching data collection, it is imperative that the safety and protection of survey participants be considered. Institutional Review Boards (IRBs) help ensure that research involving human participants is conducted in an ethical manner. This includes verifying that risks to participants are minimized, that their selection is equitable, that they are fully informed of what the survey entails, and understand the potential risks and benefits. Information about IRBs as well as available services is included in the following links:
- IRB FAQs
- Definition of IRB
- NIH IRB Training
- Association for the Accreditation of Human Research Protection Programs (AAHRPP). Provides training and certifies IRBs. The webpage includes a list of currently certified IRB
- Biomed IRB.
- Advarra IRB. Independent IRB services
- IRB Review Solutions. Independent IRB services
- NRC-National Research Center. Provides services to implement surveys, program evaluation, performance measurement, policy building, etc. Includes IRB in consultancy.
- Office of Human Research Protections (OHRP) at the United States Department of Health and Human Services (HHS). Located within HHS, this office has leadership in human subjects and provides a list of laws, regulations, and guidelines on human subjects protection in over 100 countries.
- Pearl IRB. Independent IRB services
- Advarra IRB. Independent IRB services – Only US and Canada
- Sterling IRB. Independent IRB services
- Western IRB. Independent IRB Services - International
- Open Science Framework
- Berkeley Initiative for Transparency in the Social Sciences
- Meta Research Innovation Center at Stanford
- ClinicalTrials.gov: registry and results database
- The American Economic Association’s registry for RCTs
- Equator Network: Enhancing the Quality and Transparency Of health Research
- 3ie’s Registry for International Development Impact Evaluations (RIDIE)
Terms of Reference (TORs) establish the roles and responsibilities, activities, products, and schedules of the parties involved in an impact evaluation. The following examples can be adapted based on the needs of each particular program:
Examples of TORs:
The 3ie's expert roster and the J-PAL affiliates' webpage include the names and areas of expertise of impact evaluation specialists.
High-quality data are a key input for impact evaluations. These survey materials can be adapted for your data collection needs using these instructions. Also, see our questionnaire designer manual and the data entry manual for guidance on preparing your survey.
HOUSEHOLD | COMMUNITY | HEALTH FACILITY | SCHOOL | FARMERS |
|
Household Survey Training Materials
- Training Agenda
- Introduction
- Section 00. Main Page
- Section 01. Household and household members characteristics
- Section 02. Employment in the last 12 months
- Section 03. Housing and basic services
- Section 04. Health
- Section 05. Education
- Section 06. Formal Employment
- Section 07. Family Business/Independent Activities
- Section 08. Agricultural activities
- Section 09. Non-labor income
- Section 10. Food consumption
- Section 11. Non-food spendings
- Section 12. Household goods
- Section 13. Credit access
- Section 14. Shocks
- Section 15. Enumerator comments
- Section 16. Contact information
- Section 17. Wellbeing perception
- Section 18. Migration
- Section 19. Anthropometrics
- Section 20. Vaccination
- Section 21. Fertility
- Section 22. Time Use
Further examples of survey questions, as well as questionnaires, can be found at:
- Demographic and Health Surveys (DHS)
- Family Life Surveys: Mexican Family Life Survey (MXFLS), Guatemala Survey of Family Health (EGSF)
- Fundación SEPI: Survey on business strategy
- Living Standards Measurement Surveys (LSMS)
- National Household Survey Network(IHSN) Catalog of Survey Questionnaires
- OAS Victimization Surveys
- Standardized Tests
- Early Childhood Development
- Education
- Designing Household Survey Questionnaires for Developing Countries: Lessons from 15 years of Living Standards Measurement Study (Grosh and Glewwe; World Bank, 2000): A comprehensive handbook with insights for customizing surveys and improving data quality.
- The Survey Quality Assessment Framework (SQAF) and reference guide (Spanish) assist survey managers with good practices for planning, implementing, and documenting surveys. The guidelines can be used to develop new surveys and evaluate existing surveys. SQAF advise for impact evaluations can be found here.
- Improving the Quality of Data and Impact-Evaluation Studies in Developing Countries (Stecklov and Weinreb, IADB, 2010).
The document, store and disseminate your data with the following tools:
Once the data required to conduct an impact analysis are available, the data will be analyzed using statistical software and appropriate estimation strategies. In this section, you can find useful codes and guidelines to assist you with your data analysis.
1. Validate survey data and potential attrition
- Article on attrition analysis of surveys in developing countries "Attrition in Longitudinal Household Survey Data: Some Tests for Three Developing- Country Samples"
- An example of the attrition analysis of a long-term panel survey is "An Analysis of Sample Attrition in Panel Data: The Michigan Panel Study of Income Dynamics"
2. Check the internal validity of the data
- Balancing check
- Placebo analysis regarding clinical trials
- Lack of spill-over effects. SPD’s guide "Program Evaluation and Spillover Effects"
- Case study of deworming project "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities"
3. Impact estimation
- Robustness checks “ A Practitioner’s Guide to Cluster-Robust Inference ”
4. Results presentation
- Report template: outline of key content for a full impact evaluation report makers and consumers of evaluation results.
- Policy brief template: outline of the content included in a short brief intended for policy
Exercise
External resources
Once the data is analyzed, the results are then reported through presentations, evaluation reports, papers, and policy briefs. In addition, evaluation results are increasingly communicated using multimedia. Examples of these dissemination strategies are presented below.
Tools for Reporting Evaluation Results
- Policy brief template: outline of the content included in a short brief intended for policymakers and consumers of evaluation results
- Report template: outline of key content for a full impact evaluation report
- Examples of infographics for impact evaluations:
Videos and Multimedia
This page includes links to impact evaluation training courses and workshops as well as training materials.
IDB
- Impact Evaluation Design Week 2018 - July (only available in Spanish)
- Impact Evaluation Design Week 2017 - October
- Impact Evaluation Design Week 2017 - June (only available in Spanish)
- 2014 Impact Evaluation in Practice Workshop: Productive Sectors[ROMF1]
- Advanced Topics in Impact Evaluation Seminar:
- Statistical Theory for Meta-Analysis (Sebastian Galiani, University of Maryland)
- Using Models and Experiments to Learn about Counterfactuals (Sebastian Galiani, University of Maryland)
- Regression Discontinuity Design (Matias Cattaneo, University of Michigan)
- Synthetic Controls (Alberto Abadie, Harvard University)
EXTERNAL
Methodologies
- Causal inference (Spanish)
- Differences in differences (Spanish)
- Impact evaluation from start to finish (Spanish)
- Instrumental variables (Spanish)
- Matching (Spanish)
- Measuring impacts: impact evaluation methods (Spanish)
- Randomized trials (Spanish)
- Regression discontinuity design (Spanish)
Sampling
- Conceptual foundations of scientific sampling(Spanish)
- Limitations of simple random sampling (Spanish)
- Power calculations for experimental evaluations (Spanish)
- Sampling topics relevant to impact evaluations (Spanish)
Fieldwork
- Calendar and budget (Spanish)
- Organization of fieldwork (Spanish)
- Questionnaires and other instruments (Spanish)
- Surveys: the keys to their design (Spanish)
Quality and data management
- Data management, quality assurance and documentation (Spanish)
- Surveys: the notion of total quality (Spanish)
Stata Exercise
- Exercises: impact evaluation methods (Spanish)
- Database
- Answer dofile | Answer logfile Workshop checklist
External
In this section you will find a series of six lessons to help you learn and review the basics for completing an impact evaluation. Each lesson contains a series of videos, presentations, exercises and tests to go over the material.
* from Gertler, Martinez, Premand, Rawlings, and Vermeersch (2010) Impact Evaluation in Practice, Washington, DC: The World Bank
At the design stage, you will define your evaluation questions, identify an appropriate methodology, and plan and budget for the evaluation activities. Use the following guidelines to help design and plan your impact evaluation.
a. Introductory readings:
- Impact evaluation in practice
- Evaluating the impact of development projects on poverty. A handbook for practitioners(Baker, Judy; World Bank, 2000).
b. Intermediate readings:
- Evaluating anti-poverty programs (Ravallion, Martin; World Bank, 2005).
- Program evaluation and spillover effects (Angelucci and Di Maro; IDB, 2010).
- Handbook on impact evaluation: quantitative methods and practices (Khandker et al.; World Bank, 2009).
- Recent developments in the econometrics of impact evaluation (Imbens and Wooldridge; Journal of Economic Literature, 2009).
- The mystery of vanishing benefits: an introduction to impact evaluation (Ravallion, Martin; World Bank Economic Review, Vol. 15, No. 1).
a. Experimental methods:
- In pursuit of balance. Randomization in practice in development field experiments (Bruhn and McKenzie; World Bank, 2008).
- Using randomization in development economics research:a toolkit (Duflo et al.; Center for Economic Policy Research, 2007).
b. Quasi-experimental methods:
- A primer for applying propensity-score matching (Heinrich et al; IDB, 2010).
- Instrumental variables and the search for identification: from supply and demand to natural experiment (Angrist and Krueger; Journal of Economic Perspectives, 2001).
- Regression discontinuity designs:a guide to practice (Imbens and Lemieux; National Bureau of Economic Research, 2007).
Agriculture
- Designing impact evaluations for agricultural projects (Winters et al.; IDB, 2010).
Early childhood development
- Methodologies to evaluate early childhood development programs (Behrman et al.; World Bank, 2007).
Education
- Impact evaluation for school-based management reform (Gertler et al.; World Bank, 2007).
- Guidelines for impact evaluation in education using experimental design (Bando, Rosangela; IDB, 2013).
Housing
- Impact evaluation for slum upgrading interventions (Field and Kremer; World Bank, 2008).
Industry
- Evaluating the impact of cluster development programs (Giuliani et al.; IDB,2013).
Labor markets
- Building in an evaluation component for active labor market programs: a practitioner’s guide(Card et al.; IDB, 2011).
Land property
- Impact evaluation for land property rights reforms (Conning and Deb; World Bank, 2007).
Microfinance
- Impact evaluation for microfinance (Karlan and Goldberg; World Bank, 2007).
Nutrition
- Methodologies to evaluate the impact of large-scale nutrition programs (Habicht et al.; World Bank, 2006.
Regional development
- Evaluating the impact of regional development programs (Winters and Sitja; IDB, 2010).
Science and technology
- Evaluating the impact of science, technology and innovation programs: a methodological toolkit(Crespi et al.; IDB, 2011).
Tourism
- Technical guidelines for evaluating the impacts of tourism using simulation models (Taylor, Edward; IDB, 2010).
Transportation
- Conducting impact evaluations in urban transport (Boarnet, Marlon; World Bank, 2007).
- Impact evaluation of rural road projects (Van de Walle, Dominique; World Bank, 2008).
Water and sanitation
- A guide to water and sanitation sector impact evaluations (Poulos et al.; World Bank, 2006).
- Power calculations spreadsheet: a tool for determining sample sizes.
- Random numbers generator.
- Impact evaluation Gantt chart: a spreadsheet to assist in the scheduling of impact evaluation activities.
- Data collection Gantt chart: a spreadsheet to assist in the scheduling of data collection activities.
- Budget template: a tool for estimating the costs of an impact evaluation.
- Concept note template: a document that describes the details of the impact evaluation methodology.
- Design template: a blank presentation to guide the main steps and components of an impact evaluation design.
- Impact evaluation checklist: a list of core activities to be carried out during an impact evaluation.
Prior to launching data collection, it is imperative that the safety and protection of survey participants be considered. Institutional Review Boards (IRBs) help ensure that research involving human participants is conducted in an ethical manner. This includes verifying that risks to participants are minimized, that their selection is equitable, that they are fully informed of what the survey entails, and understand the potential risks and benefits. Information about IRBs as well as available services is included in the following links:
- IRB FAQs
- Definition of IRB
- NIH IRB Training
- Association for the Accreditation of Human Research Protection Programs (AAHRPP). Provides training and certifies IRBs. The webpage includes a list of currently certified IRB
- Biomed IRB.
- Advarra IRB. Independent IRB services
- IRB Review Solutions. Independent IRB services
- NRC-National Research Center. Provides services to implement surveys, program evaluation, performance measurement, policy building, etc. Includes IRB in consultancy.
- Office of Human Research Protections (OHRP) at the United States Department of Health and Human Services (HHS). Located within HHS, this office has leadership in human subjects and provides a list of laws, regulations, and guidelines on human subjects protection in over 100 countries.
- Pearl IRB. Independent IRB services
- Advarra IRB. Independent IRB services – Only US and Canada
- Sterling IRB. Independent IRB services
- Western IRB. Independent IRB Services - International
- Open Science Framework
- Berkeley Initiative for Transparency in the Social Sciences
- Meta Research Innovation Center at Stanford
- ClinicalTrials.gov: registry and results database
- The American Economic Association’s registry for RCTs
- Equator Network: Enhancing the Quality and Transparency Of health Research
- 3ie’s Registry for International Development Impact Evaluations (RIDIE)
Terms of Reference (TORs) establish the roles and responsibilities, activities, products, and schedules of the parties involved in an impact evaluation. The following examples can be adapted based on the needs of each particular program:
Examples of TORs:
The 3ie's expert roster and the J-PAL affiliates' webpage include the names and areas of expertise of impact evaluation specialists.
High-quality data are a key input for impact evaluations. These survey materials can be adapted for your data collection needs using these instructions. Also, see our questionnaire designer manual and the data entry manual for guidance on preparing your survey.
HOUSEHOLD | COMMUNITY | HEALTH FACILITY | SCHOOL | FARMERS |
|
Household Survey Training Materials
- Training Agenda
- Introduction
- Section 00. Main Page
- Section 01. Household and household members characteristics
- Section 02. Employment in the last 12 months
- Section 03. Housing and basic services
- Section 04. Health
- Section 05. Education
- Section 06. Formal Employment
- Section 07. Family Business/Independent Activities
- Section 08. Agricultural activities
- Section 09. Non-labor income
- Section 10. Food consumption
- Section 11. Non-food spendings
- Section 12. Household goods
- Section 13. Credit access
- Section 14. Shocks
- Section 15. Enumerator comments
- Section 16. Contact information
- Section 17. Wellbeing perception
- Section 18. Migration
- Section 19. Anthropometrics
- Section 20. Vaccination
- Section 21. Fertility
- Section 22. Time Use
Further examples of survey questions, as well as questionnaires, can be found at:
- Demographic and Health Surveys (DHS)
- Family Life Surveys: Mexican Family Life Survey (MXFLS), Guatemala Survey of Family Health (EGSF)
- Fundación SEPI: Survey on business strategy
- Living Standards Measurement Surveys (LSMS)
- National Household Survey Network(IHSN) Catalog of Survey Questionnaires
- OAS Victimization Surveys
- Standardized Tests
- Early Childhood Development
- Education
- Designing Household Survey Questionnaires for Developing Countries: Lessons from 15 years of Living Standards Measurement Study (Grosh and Glewwe; World Bank, 2000): A comprehensive handbook with insights for customizing surveys and improving data quality.
- The Survey Quality Assessment Framework (SQAF) and reference guide (Spanish) assist survey managers with good practices for planning, implementing, and documenting surveys. The guidelines can be used to develop new surveys and evaluate existing surveys. SQAF advise for impact evaluations can be found here.
- Improving the Quality of Data and Impact-Evaluation Studies in Developing Countries (Stecklov and Weinreb, IADB, 2010).
The document, store and disseminate your data with the following tools:
Once the data required to conduct an impact analysis are available, the data will be analyzed using statistical software and appropriate estimation strategies. In this section, you can find useful codes and guidelines to assist you with your data analysis.
1. Validate survey data and potential attrition
- Article on attrition analysis of surveys in developing countries "Attrition in Longitudinal Household Survey Data: Some Tests for Three Developing- Country Samples"
- An example of the attrition analysis of a long-term panel survey is "An Analysis of Sample Attrition in Panel Data: The Michigan Panel Study of Income Dynamics"
2. Check the internal validity of the data
- Balancing check
- Placebo analysis regarding clinical trials
- Lack of spill-over effects. SPD’s guide "Program Evaluation and Spillover Effects"
- Case study of deworming project "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities"
3. Impact estimation
- Robustness checks “ A Practitioner’s Guide to Cluster-Robust Inference ”
4. Results presentation
- Report template: outline of key content for a full impact evaluation report makers and consumers of evaluation results.
- Policy brief template: outline of the content included in a short brief intended for policy
Exercise
External resources
Once the data is analyzed, the results are then reported through presentations, evaluation reports, papers, and policy briefs. In addition, evaluation results are increasingly communicated using multimedia. Examples of these dissemination strategies are presented below.
Tools for Reporting Evaluation Results
- Policy brief template: outline of the content included in a short brief intended for policymakers and consumers of evaluation results
- Report template: outline of key content for a full impact evaluation report
- Examples of infographics for impact evaluations:
Videos and Multimedia
This page includes links to impact evaluation training courses and workshops as well as training materials.
IDB
- Impact Evaluation Design Week 2018 - July (only available in Spanish)
- Impact Evaluation Design Week 2017 - October
- Impact Evaluation Design Week 2017 - June (only available in Spanish)
- 2014 Impact Evaluation in Practice Workshop: Productive Sectors[ROMF1]
- Advanced Topics in Impact Evaluation Seminar:
- Statistical Theory for Meta-Analysis (Sebastian Galiani, University of Maryland)
- Using Models and Experiments to Learn about Counterfactuals (Sebastian Galiani, University of Maryland)
- Regression Discontinuity Design (Matias Cattaneo, University of Michigan)
- Synthetic Controls (Alberto Abadie, Harvard University)
EXTERNAL
Methodologies
- Causal inference (Spanish)
- Differences in differences (Spanish)
- Impact evaluation from start to finish (Spanish)
- Instrumental variables (Spanish)
- Matching (Spanish)
- Measuring impacts: impact evaluation methods (Spanish)
- Randomized trials (Spanish)
- Regression discontinuity design (Spanish)
Sampling
- Conceptual foundations of scientific sampling(Spanish)
- Limitations of simple random sampling (Spanish)
- Power calculations for experimental evaluations (Spanish)
- Sampling topics relevant to impact evaluations (Spanish)
Fieldwork
- Calendar and budget (Spanish)
- Organization of fieldwork (Spanish)
- Questionnaires and other instruments (Spanish)
- Surveys: the keys to their design (Spanish)
Quality and data management
- Data management, quality assurance and documentation (Spanish)
- Surveys: the notion of total quality (Spanish)
Stata Exercise
- Exercises: impact evaluation methods (Spanish)
- Database
- Answer dofile | Answer logfile Workshop checklist
External
In this section you will find a series of six lessons to help you learn and review the basics for completing an impact evaluation. Each lesson contains a series of videos, presentations, exercises and tests to go over the material.
* from Gertler, Martinez, Premand, Rawlings, and Vermeersch (2010) Impact Evaluation in Practice, Washington, DC: The World Bank