COVID-19: What can we learn from past responses to public health crises?

Around the world, the COVID-19 pandemic is bringing unprecedented economic and humanitarian impacts. To address the situation effectively, response and recovery efforts must be guided by evidence. This is where evaluation comes into the picture: evaluation can tell us what works, what doesn’t, and why.

Last June, OVE convened a panel of evaluators who had assessed their institution’s response to past public health crises such as Ebola and Avian Flu: Maya Vijayaraghavan (Asian Development Bank), Beth Plowman (UNICEF), and Stephen Hutton (World Bank). Ivory Yong-Prötzel, OVE’s Director, moderated the discussion.

The panel’s focus was on evaluative lessons drawn from responses to past public health crises that could be relevant to addressing the COVID pandemic. These were the main takeaways:

Shift toward preparedness: Multilateral development agencies often struggle to maintain the focus on crises-related agendas after each crisis has subsided, which limits their ability to learn and apply lessons. That´s why it is important to shift toward a pandemic preparedness approach, similar to the shift from emergency relief to emergency preparedness in the case of natural disasters. It is also key to use the time between crises to build capacity, as it is virtually impossible to address all the issues of an emergency while the crisis is unfolding (Hutton, World Bank).

Engagement with communities and communication: Public health emergencies are distinctly different from other types of emergencies because the crisis can move rapidly from one community to another. The involvement of medical anthropologists and sociologists played a crucial role in better understanding communities and in helping UNICEF engage more effectively with them. Based on that experience, UNICEF has set up a social science research platform to better communicate new findings around Ebola and currently around the COVID-19 pandemic (Plowman, UNICEF).

Collaboration and knowledge sharing: Learning from other emergencies (e.g., natural disasters, epidemics) helps to respond quickly and better—for example, by harnessing expertise from across the institution and by streamlining approval and operational processes. The “One ADB approach” is based on the idea of bringing together knowledge and expertise from across the organization (Vijayaraghavan, ADB).

Feedback in rapidly changing contexts: The real issue in health crises is a lack of sufficient feedback during the implementation of crisis response programs. For the COVID-19 pandemic it would be useful to know whether programs work, why or why not, and what can be done to improve them while they are being implemented (Vijayaraghavan).

Adaptive management: It is not possible to have adaptive learning without adaptive management. Whether an organization’s management is flexible and willing to adapt is determined by the institutional culture and attitude toward risk. Making fast decisions with limited information can be challenging in some institutions, especially if a culture of risk avoidance is prevalent. UNICEF’s evaluation department has recently started to provide management with so-called “good enough” information to guide real-time decisions about the COVID-19 response (Plowman).

Emergent learning: Real-time learning on communications and public awareness was particularly relevant in inducing behavioral change by informing effective messaging on the avian flu at the World Bank. That´s why institutions operating in the context of a global crisis need to practice emergent learning. This calls for defining, in a before-action meeting, what success would look like and what pitfalls could exist, and verifying outcomes in an after-action review that would help extract lessons and shift tactics while the project (or similar ones) is still being implemented (Hutton).

Collection of data before projects are completed: The evaluation of UNICEF’s response to Ebola, conducted at the end of the project cycle, was “too late to pick up all of the important adaptations that had been made and the issues that were emergent throughout the outbreak” (link video 1:19:04-1:24:07). It is crucial to start collecting information in preparation for an evaluation while the response to the crisis is under implementation, to develop a good grasp of the nuances of the implementation process (Plowman).

Early planning and real-time evaluation: To evaluate responses to the COVID-19 pandemic evaluators should plan early, keep track of the institutional response (financing and processes), and prepare for the possibility of a real-time evaluation to gauge the likeliness that the response will be effective (Vijayaraghavan).

Close work with implementation teams: It is important that evaluations in a crisis setting inform projects up front and during implementation about what works and what does not. This approach may require a new mindset regarding the independence of evaluation offices because real-time evaluation requires working closely with implementation teams (Hutton).

As OVE´s Director pointed out in her closing remarks, it is important that, despite shifting priorities, institutions be prepared for future emergencies so that they can effectively support countries during crises, with a focus on populations that are most vulnerable to and affected by the crisis. Sharing knowledge between institutions and regions is key to navigating complex crises like the COVID-19 pandemic.