Explore our hub for assessing development projects
CEA compares the cost per unit of effect in a particular project or program option with the costs per unit of effects for alternatives. The comparison between costs and effectiveness will allow the ranking of the alternatives or a comparison with similar interventions or projects. It is important to underline that cost-effectiveness measures allow for the ranking of interventions that result in the same effect.
The measure of effectiveness (and cost-effectiveness) should be directly linked with the stated objective of the project. In any project, there are many possibilities as to which measure of effectiveness to choose from. Effectiveness selection should be guided by two criteria: reliability and validity.
- An indicator is said to be reliable if it yields similar results when applied repeatedly to the same individuals or populations. As most measures of effectiveness do not have entirely reliable indicators, it is suggested that the cost-effectiveness analysis includes a reasonable range for these effects.
- An indicator is said to be valid if it has a close match to the underlying impact it is trying to capture.
When the analysis uses intermediate outcomes that have limited validity, it is suggested that the analysis – and justification – for the selected indicator be based on empirical evidence from secondary sources.
Effectiveness can be measured at both outcome and impact levels. In some cases, it can be necessary to resort to intermediate outcomes, if relevant outcome or impact information is scarce. If intermediate outcomes are used, evidence should be presented on the relationship and correspondence of these intermediate outcomes to the expected impact, if the objective is stated in those terms.
Se puede conseguir satisfacer cualquier necesidad particular (el agua potable, por ejemplo) o resolver cualquier problema específico (reducir la tasa de delincuencia, por ejemplo) de diversas maneras. Siempre se deberían especificar las alternativas al abordar el mismo problema y solucionar el mismo problema.
Las alternativas se enfrentan al mismo problema y difieren en cuanto a la solución. Las alternativas sólo se pueden comparar cuando abordan el mismo problema o problemas similares. Dado que cualquier problema puede abordarse de diversas maneras, debería tenerse en cuenta que el número de alternativas consideradas tiene consecuencias en términos de costos y plazos y que éstas deberían ser limitadas. Las alternativas normalmente implican opciones sobre las escalas, el grado de participación (no actuar, no hacer más, mejorar), los plazos de la inversión, los diseños institucionales y la opción tecnológica.
Una vez que se han calculado los ratios de costo-efectividad, es necesario evaluar las alternativas basándose en los coeficientes calculados. Las alternativas deberían clasificarse según sus coeficientes de costo-efectividad.
Todos los enfoques alternativos considerados deberían tener coeficientes de costo efectividad que reflejen el mismo impacto y debería seleccionarse la alternativa con el coeficiente más bajo. Si se incluye más de un impacto, debería prestarse especial atención a la asignación de un costo a cada uno; todos los costos y efectos deberían expresarse en términos marginales o incrementales.
There are different approaches in identifying costs including the ingredient method which relies on the identification of all resources or ingredients consumed in an intervention and the valuation of each ingredient. The specification of ingredients is often facilitated by dividing ingredients into categories including 1) personnel, 2) facilities, 3) equipment and materials, 4) other program inputs, and 5) beneficiary or client inputs.
For Bank-funded projects, public sector costs are identified in project documents and the ingredient method is most useful in identifying other costs such as operation and maintenance costs and non-public sector societal costs.
Identifying costs in the project
Here are some of the costs the project must identify:
First: public sector costs included for the project investments
Project documents lay out investments and non-recurring costs explicitly as part of project preparation and there should be consistency between what is noted in the project and what is identified as project costs, including costs that are part of the loan as well as counterpart costs.
Second: additional operational and maintenance costs or other recurring costs to the public sector
If the project establishes new government entities or invests in new infrastructure that requires funds for operation and maintenance or other recurring costs beyond the project timeframe these should be included. These are incremental costs beyond what would have been paid for in operation and maintenance without the project.
Third: costs to society that should be incorporated - non-governmental organizations, firms, households or individuals
Any additional costs to non-public entities that result from the project should be included. In general, all inputs (costs) that contribute to the generation of benefits should be included. Otherwise, benefits would have to be adjusted accordingly. These costs depend largely on the project and linked to the response required by project beneficiaries to obtain the benefit of the project.
Fourth: total flows of costs in the project
All types of costs should be noted and aggregated to get the total cost flows and the present value of those cost flows. It should be clear that all of these costs are incremental and the manner in which they are measured should be transparent. In the case of shared or common costs (with other projects or among components), these should be distributed and assigned according to a clear allocation rule, which needs to be explicitly stated in the assumptions section.
Sensitivity analysis allows the testing, under different scenarios, of the strength and robustness of the assumptions, the data and the logic of the intervention that underlie the economic analysis and to acknowledge the underlying uncertainty of the expected results. Any forward looking estimation presumes behavioral assumptions, assumptions on the vertical logic, and on the causality links that allow the attribution of a particular benefit or cost to the goods and/or services delivered or used by the project.
In addition it is necessary to undertake sensitivity analysis to compensate for the natural tendency for project appraisers to be overly optimistic. Many project parameters are affected by optimism, overstating benefits and understating timings and costs, both capital and operational.
- In many instances both costs and benefits are driven by a few critical factors. Identifying these factors – that can be called “drivers” - is a central element in performing a good sensitivity analysis. The assumptions behind the future performance of these drivers need to be critically assessed and the sensitivity analysis can illuminate at which points in these assumptions the project is no longer worth pursuing. These values are typically called “switching values¨.
- In a sensitivity analysis, it is important to focus on “what if” situations, particularly in adverse or more stringent and acid scenarios than those assumed under a Base Case scenario. This focus will allow the analysis to illuminate those values in critical variables and assumptions that might “switch” the recommendation from Go to No Go (or vice-versa) or to reformulate project design and / or components in order to make the project viable. As there are many sources of uncertainty and numerous risks, it can be very useful to identify the key sources of uncertainty and base the analysis on those. Sometimes having too much information does not provide guidance as to the viability of the project under different situations.
- The sensitivity analysis, which is really a risk assessment, should be coherent with the Risk Matrix and other risks discussed in the project proposal.
- Many projects face significant start and implementation delays. If warranted, a simulation should be undertaken delaying start and implementation dates.
- In a CBA derived from impact evaluations – sensitivity can be done by ranking alternatives by their relative net benefit indicators (NPV, ERR or C/B) using the point estimate of their impact, and then re-compute this ratio using the lower and upper bounds of the impact estimate.
A Base Case reflects the most plausible estimates for unknown quantities and prices and should reflect the average expectation of their future behavior, particularly of cost and benefit drivers. This average expected behavior should be empirically based. The more useful sensitivity analysis are those that present few explicit and reasoned changes on the assumptions of the most critical values of the drivers of the dominant benefits and costs, many times in a “best case, worse case” situation.
- Sensitivity analysis can be “one way” or “multi-way”.
- One-way sensitivity analysis allows for the variation of one critical variable at a time, holding everything else constant. Some authors call this “partial sensitivity analysis¨. This kind of analysis is most appropriately applied to situations in which the analyst believes there is one critical driver.
- Multi-way allows for simultaneous variations of more than one variable at a time. In this type of analysis, variable combinations can be combined into best-case and worst-case scenarios. Some projects lend themselves to simulations, where combinations of different scenarios and their probability of occurrence can be modeled and estimations of critical values can be synthesized in terms of Expectations or Expected Values. This is typically undertaken with decision trees and Monte Carlo Simulations. In these cases, means and variances can indicate project risk profiles.
- In a single-factor sensitivity analysis, it is generally assumed that all other factors stay constant. In many projects, this might not be true and the variables might not be independent. In those cases, a discussion on correlation issues is warranted.
Sensitivity analyses can also help, if the feasibility of a project is very sensitive to a particular assumption on the value of a variable that is uncertain, in identifying mitigating actions that should be considered. If there is exceptional uncertainty, the project might have to be redesigned if implemented on a pilot basis.
Sensitivity analysis is most useful if the proposed scenarios and adjustments are empirically based and/or reflect the unique characteristics of the project in hand—how good and reliable is the data. As with the initial assumptions used in an analysis, ideally, the scenarios are empirically based.
CEA compares the cost per unit of effect in a particular project or program option with the costs per unit of effects for alternatives. The comparison between costs and effectiveness will allow the ranking of the alternatives or a comparison with similar interventions or projects. It is important to underline that cost-effectiveness measures allow for the ranking of interventions that result in the same effect.
The measure of effectiveness (and cost-effectiveness) should be directly linked with the stated objective of the project. In any project, there are many possibilities as to which measure of effectiveness to choose from. Effectiveness selection should be guided by two criteria: reliability and validity.
- An indicator is said to be reliable if it yields similar results when applied repeatedly to the same individuals or populations. As most measures of effectiveness do not have entirely reliable indicators, it is suggested that the cost-effectiveness analysis includes a reasonable range for these effects.
- An indicator is said to be valid if it has a close match to the underlying impact it is trying to capture.
When the analysis uses intermediate outcomes that have limited validity, it is suggested that the analysis – and justification – for the selected indicator be based on empirical evidence from secondary sources.
Effectiveness can be measured at both outcome and impact levels. In some cases, it can be necessary to resort to intermediate outcomes, if relevant outcome or impact information is scarce. If intermediate outcomes are used, evidence should be presented on the relationship and correspondence of these intermediate outcomes to the expected impact, if the objective is stated in those terms.
Se puede conseguir satisfacer cualquier necesidad particular (el agua potable, por ejemplo) o resolver cualquier problema específico (reducir la tasa de delincuencia, por ejemplo) de diversas maneras. Siempre se deberían especificar las alternativas al abordar el mismo problema y solucionar el mismo problema.
Las alternativas se enfrentan al mismo problema y difieren en cuanto a la solución. Las alternativas sólo se pueden comparar cuando abordan el mismo problema o problemas similares. Dado que cualquier problema puede abordarse de diversas maneras, debería tenerse en cuenta que el número de alternativas consideradas tiene consecuencias en términos de costos y plazos y que éstas deberían ser limitadas. Las alternativas normalmente implican opciones sobre las escalas, el grado de participación (no actuar, no hacer más, mejorar), los plazos de la inversión, los diseños institucionales y la opción tecnológica.
Una vez que se han calculado los ratios de costo-efectividad, es necesario evaluar las alternativas basándose en los coeficientes calculados. Las alternativas deberían clasificarse según sus coeficientes de costo-efectividad.
Todos los enfoques alternativos considerados deberían tener coeficientes de costo efectividad que reflejen el mismo impacto y debería seleccionarse la alternativa con el coeficiente más bajo. Si se incluye más de un impacto, debería prestarse especial atención a la asignación de un costo a cada uno; todos los costos y efectos deberían expresarse en términos marginales o incrementales.
There are different approaches in identifying costs including the ingredient method which relies on the identification of all resources or ingredients consumed in an intervention and the valuation of each ingredient. The specification of ingredients is often facilitated by dividing ingredients into categories including 1) personnel, 2) facilities, 3) equipment and materials, 4) other program inputs, and 5) beneficiary or client inputs.
For Bank-funded projects, public sector costs are identified in project documents and the ingredient method is most useful in identifying other costs such as operation and maintenance costs and non-public sector societal costs.
Identifying costs in the project
Here are some of the costs the project must identify:
First: public sector costs included for the project investments
Project documents lay out investments and non-recurring costs explicitly as part of project preparation and there should be consistency between what is noted in the project and what is identified as project costs, including costs that are part of the loan as well as counterpart costs.
Second: additional operational and maintenance costs or other recurring costs to the public sector
If the project establishes new government entities or invests in new infrastructure that requires funds for operation and maintenance or other recurring costs beyond the project timeframe these should be included. These are incremental costs beyond what would have been paid for in operation and maintenance without the project.
Third: costs to society that should be incorporated - non-governmental organizations, firms, households or individuals
Any additional costs to non-public entities that result from the project should be included. In general, all inputs (costs) that contribute to the generation of benefits should be included. Otherwise, benefits would have to be adjusted accordingly. These costs depend largely on the project and linked to the response required by project beneficiaries to obtain the benefit of the project.
Fourth: total flows of costs in the project
All types of costs should be noted and aggregated to get the total cost flows and the present value of those cost flows. It should be clear that all of these costs are incremental and the manner in which they are measured should be transparent. In the case of shared or common costs (with other projects or among components), these should be distributed and assigned according to a clear allocation rule, which needs to be explicitly stated in the assumptions section.
Sensitivity analysis allows the testing, under different scenarios, of the strength and robustness of the assumptions, the data and the logic of the intervention that underlie the economic analysis and to acknowledge the underlying uncertainty of the expected results. Any forward looking estimation presumes behavioral assumptions, assumptions on the vertical logic, and on the causality links that allow the attribution of a particular benefit or cost to the goods and/or services delivered or used by the project.
In addition it is necessary to undertake sensitivity analysis to compensate for the natural tendency for project appraisers to be overly optimistic. Many project parameters are affected by optimism, overstating benefits and understating timings and costs, both capital and operational.
- In many instances both costs and benefits are driven by a few critical factors. Identifying these factors – that can be called “drivers” - is a central element in performing a good sensitivity analysis. The assumptions behind the future performance of these drivers need to be critically assessed and the sensitivity analysis can illuminate at which points in these assumptions the project is no longer worth pursuing. These values are typically called “switching values¨.
- In a sensitivity analysis, it is important to focus on “what if” situations, particularly in adverse or more stringent and acid scenarios than those assumed under a Base Case scenario. This focus will allow the analysis to illuminate those values in critical variables and assumptions that might “switch” the recommendation from Go to No Go (or vice-versa) or to reformulate project design and / or components in order to make the project viable. As there are many sources of uncertainty and numerous risks, it can be very useful to identify the key sources of uncertainty and base the analysis on those. Sometimes having too much information does not provide guidance as to the viability of the project under different situations.
- The sensitivity analysis, which is really a risk assessment, should be coherent with the Risk Matrix and other risks discussed in the project proposal.
- Many projects face significant start and implementation delays. If warranted, a simulation should be undertaken delaying start and implementation dates.
- In a CBA derived from impact evaluations – sensitivity can be done by ranking alternatives by their relative net benefit indicators (NPV, ERR or C/B) using the point estimate of their impact, and then re-compute this ratio using the lower and upper bounds of the impact estimate.
A Base Case reflects the most plausible estimates for unknown quantities and prices and should reflect the average expectation of their future behavior, particularly of cost and benefit drivers. This average expected behavior should be empirically based. The more useful sensitivity analysis are those that present few explicit and reasoned changes on the assumptions of the most critical values of the drivers of the dominant benefits and costs, many times in a “best case, worse case” situation.
- Sensitivity analysis can be “one way” or “multi-way”.
- One-way sensitivity analysis allows for the variation of one critical variable at a time, holding everything else constant. Some authors call this “partial sensitivity analysis¨. This kind of analysis is most appropriately applied to situations in which the analyst believes there is one critical driver.
- Multi-way allows for simultaneous variations of more than one variable at a time. In this type of analysis, variable combinations can be combined into best-case and worst-case scenarios. Some projects lend themselves to simulations, where combinations of different scenarios and their probability of occurrence can be modeled and estimations of critical values can be synthesized in terms of Expectations or Expected Values. This is typically undertaken with decision trees and Monte Carlo Simulations. In these cases, means and variances can indicate project risk profiles.
- In a single-factor sensitivity analysis, it is generally assumed that all other factors stay constant. In many projects, this might not be true and the variables might not be independent. In those cases, a discussion on correlation issues is warranted.
Sensitivity analyses can also help, if the feasibility of a project is very sensitive to a particular assumption on the value of a variable that is uncertain, in identifying mitigating actions that should be considered. If there is exceptional uncertainty, the project might have to be redesigned if implemented on a pilot basis.
Sensitivity analysis is most useful if the proposed scenarios and adjustments are empirically based and/or reflect the unique characteristics of the project in hand—how good and reliable is the data. As with the initial assumptions used in an analysis, ideally, the scenarios are empirically based.
