Evaluating Humanitarian Response in Rapid Onset Crises: The Power of Pre-Positioning Research
When it comes to evaluating humanitarian programming and emergency responses, we have a conundrum. The urgency of humanitarian response is widely thought to be at odds with lengthy evaluation processes. In the case of a rapid-onset humanitarian crisis, such as a natural disaster or armed conflict, setting up research partnerships, designing a methodology, obtaining ethical approval, and collecting baseline data are challenging, if not impossible.
Given this, the evidence base within rapid-onset crises is limited, reducing the information humanitarian agencies have to inform and design their relief activities. Evidence to inform how to optimize humanitarian programming is increasingly important as humanitarian need expands and funding to meet that need struggles to keep pace. While it may be possible to conduct monitoring of the outputs of such programming, routine and robust beginning-to-end evaluations are infeasible without a shift in how evaluations are procured.
Pre-positioning partnerships and research designs in advance of a rapid-onset crisis helps overcome these time constraints. Navigating these timeline constraints is critical for allowing evaluations that require ex-ante design work to take place. Additionally, pre-positioning research ensures that data collection can happen more quickly after a crisis, which mitigates recall bias and ensures that findings are timely enough to be relevant to the situation and the people in need of assistance.
With foresight and preparation, evaluation research can begin immediately following a pre-agreed-upon trigger event. Prior to the 2022 hurricane season, GiveDirectly and Causal Design did just this. To evaluate GiveDirectly’s unconditional cash transfers to those impacted by a hurricane, GiveDirectly and Causal Design pre-positioned their research partnership by laying the groundwork for the evaluation in advance of the identified trigger event, the anticipated hurricane season.
Below we highlight the key steps to pre-positioning research ahead of a crisis as demonstrated through the evaluation of GiveDirectly’s humanitarian response in light of the hurricanes. While this example focuses on a U.S.-based natural disaster, these lessons can be applied to contexts of other rapid-onset humanitarian crises, such as regions often afflicted by floods, cyclones, or tsunamis.
Preparation for rapid-onset crisis response and evaluation
Step 1: Determine the trigger event
The first step in pre-positioning an evaluation of a rapid-onset crisis is determining what will trigger the research to begin. For this evaluation, the research team connected the evaluation trigger to the parameters that would initiate GiveDirectly’s cash transfer intervention. Though there was no way to know when and where the anticipated hurricanes would occur, GiveDirectly designed an intervention that could target and deploy in near real-time. In this case, when a hurricane makes landfall, GiveDirectly would visually assess the extent of the damage using National Oceanic and Atmospheric Administration (NOAA) data and Google’s SKAI model, a damage prediction technology developed with the World Food Programme. Once GiveDirectly determined that the damage caused by a hurricane met their criteria for a trigger event, they would begin disbursing aid. Causal Design then planned for data collection three to four weeks after the first cash disbursement.
Step 2: Set up a contracting mechanism
Once a trigger event is determined, a contract should be put in place that outlines what evaluation work should be completed ahead of the crisis and what will be deployed once the crisis takes place. Causal Design and GiveDirectly specified that the research protocol development and ethical clearance would take place in advance of the trigger event and data collection and reporting would only begin following the event. In this case, they relied on a master service agreement (MSA) to outline these parameters. With an MSA, additional contracts do not need to be renegotiated, yet the basics of the initial agreement are included in all future contracts, facilitating quick adaptations in response to a crisis.
Step 3: Develop the research design
Once a trigger event is determined, the research partner develops a research design. While the specific crisis may be unknown, as long as there is a clear understanding of the evaluation questions and general context, the research team can design a relatively comprehensive research protocol.
Based on the research objectives, Causal Design created an interview guide to collect qualitative data through semi-structured, in-depth interviews focused on how and when cash supported participants’ recovery process. In addition, the interviews probed for insights regarding how recipients heard about, accessed, and made use of GiveDirectly’s assistance. The interview questions were written such that they could apply to any recipient of GiveDirectly’s assistance in the region of interest.
Even if the specific context is not known, guidance on how a sample will be drawn can be determined. For example, the population of interest was GiveDirectly program participants, which GiveDirectly planned to identify based on financial need (as determined by enrollment in government-provided Supplemental Nutrition Assistance Program benefits) and residence in areas damaged by the hurricanes. GiveDirectly then would notify people in the areas that they may be eligible for additional assistance. Participants who opt for assistance are then contacted, with their consent, to complete a brief interview approximately three weeks after receiving the transfer. Causal Design proposed to interview 40 recipients based on the expected subgroups of interest.
Step 4: Obtain ethical approval for the research
Research with human subjects requires review and approval by an institutional review board (IRB), an administrative body established to protect the rights and welfare of human research subjects. It can take weeks or even months for an IRB to grant ethical clearance for research to proceed. Studies sometimes require extensive back-and-forth with an IRB to ensure that the research does not endanger or psychologically harm participants. However, with sufficient foresight and commitment by the donor, it is possible to finalize a design for immediate deployment alongside humanitarian assistance.
Causal Design and GiveDirectly began preparations early to obtain IRB approval. While the specific context was not known, there was sufficient knowledge about the broad area where the crisis was expected, the evaluation objectives, and the methodology. The IRB reviewed the design to ensure that interviewees’ privacy was protected and that the research would not cause undue stress — this included providing contact information for counseling services and additional assurance that all questions were optional and participants could stop responding at any time. The IRB concluded that this was an exempt study, i.e., one posing minimal risk to human subjects. Such studies typically take between one and three weeks to approve.
Fiona and Ian Make Landfall
Hurricane Fiona made landfall in Puerto Rico on September 18, 2022, and Hurricane Ian made landfall in Florida on September 23, 2022. GiveDirectly moved to quickly offer relief. As planned, GiveDirectly and the SKAI team assessed the damage to a sample of approximately 500 structures to determine that this constituted the trigger event. Once confirmed, from September through November 2022, GiveDirectly sent $700 in emergency funds to 4,748 low-income households impacted by Hurricane Ian and Hurricane Fiona. The funds were intended to cover the cost of common post-disaster recovery items, such as food, baby formula and diapers, first aid supplies, toiletries, portable generators, and fuel.
Evaluating the Response
GiveDirectly then provided Causal Design with a redacted roster of recipients indicating when they received the transfer, where they were located, their primary language, and if they had consented to be interviewed. Interviewees from these two strata were randomly selected for an interview.
Causal Design asked recipient households what their most pressing needs had been at various intervals following the hurricane. Immediately after the hurricanes hit, the top three needs in Puerto Rico and in Florida were the same: food, water, and electricity.
One striking difference between the two sites is that the majority (65%) of Puerto Rican respondents were still without electricity one to two weeks after Hurricane Fiona. By contrast, only 5% of Floridians were without power in that same period after Hurricane Ian. While food and water still ranked in the top three needs in both locations, by one to two weeks after Hurricane Ian, Florida respondents had moved on to home repair as a priority need.
As a result of pre-positioning the research, the research team was able to assess this change week-by-week and construct a more accurate timeline of transfer recipients’ experiences, needs, and observations. This provided valuable insight for GiveDirectly as they were interested in how cash assistance fits into the timeline of recipients’ needs.
In particular, one useful insight was the importance of the timing of cash assistance vs. in-kind aid. Some noted that receiving cash earlier in the recovery process would not have been particularly helpful because the roads were impassible, or because stores’ shelves were not well-stocked. As one recipient in Florida described it, “Most places were closed. You couldn’t use your card to purchase anything if you happened to find a place that was open that had what you needed. Everything was cash only. It was kind of impossible to get what you needed because there wasn’t any way for you to get cash.” Many respondents stated that in-kind assistance, such as direct provision of food and water, was the most appropriate form of help in the immediate aftermath, but that once those basic needs had been met, cash was what they needed to continue their recovery. These types of insights can inform the timing of GiveDirectly’s cash transfers in future hurricane seasons to maximize the usefulness of the assistance.
Pre-positioning as a model for evaluating rapid onset crisis programming
Pre-positioning is necessary for conducting timely research in the context of rapid-onset crises. As highlighted above, it allowed for the generation of more accurate findings to inform the optimal timing window for the delivery of GiveDirectly’s assistance.
This approach can easily be applied in other contexts where rapid-onset disasters can be anticipated. While it does require a shift in how funders procure research, as there is an upfront cost associated with the design, partnering, and approval of evaluation research, it does not need to constitute a large financial risk for funders. There are many contexts where disasters can be reasonably anticipated during certain times of the year, for example, cyclones in South Asia or tropical storms in Southern Africa. In these situations, evaluations can be pre-positioned with reasonable certainty and reduced risk for donors. While some level of risk may be unavoidable, pre-positioning is critical for expanding the limited evidence base on humanitarian response in rapid-onset crises to ensure that relief has as great an impact as possible.
