With support from the William and Flora Hewlett Foundation, IDinsight published a 3ie Working Paper on the “Future of Impact Evaluation.” We really like the team over at IDinsight and think they are spot-on with this work. Causal Design’s primary focus is on providing decision-focused evaluations like they describe in the paper.
Their main arguments are listed below; here are links to their blog and the original paper:
1. The field of rigorous impact evaluation in international development has been dominated by “knowledge-focused evaluations” to date, with a number of notable successes. However, a number of changes to the status quo have the potential to significantly increase the extent to which impact evaluations improve social outcomes.
2. In the future, we need to distinguish between “decision-focused evaluations” (DFEs) and “knowledge-focused evaluations” (KFEs):
a. DFEs are evaluations that are fundamentally demand-driven and responsive to the timelines, budgets and operational constraints of implementers – DFEs are deliberately designed to directly inform the scale-up decisions of funders and implementers.
b. KFEs are evaluations that are designed with a primary goal of contributing to a “global body of evidence” or “development theory”, usually through publication in an academic journal.
c. Both DFEs and KFEs are rigorous, counterfactual-based quantitative impact evaluations.
3. Need to greatly increase the number of “decision-focused evaluations”, as a tool to directly inform the decisions of a specific implementer in a specific geography at a specific time.
4. Use “knowledge-focused evaluations” where they add most value – when the primary objective is to contribute to development theory or when there is likely to be high external validity. Given the growing body of evidence around the low generalizability of impact evaluation evidence (e.g., Bold et. al. (2013); Pritchett and Sandefur (2014); Vivalt (2015)), the high cost and longer timelines of KFEs are likely to be best justified when the primary evaluation objective is to development theory. However, this means that KFEs are likely not appropriate if the primary objective is to guide a specific funding or implementation decision in a specific context.
5. Clear objectives, appropriate evaluation type – All parties involved in an evaluation (funder, evaluator and implementer) should explicitly agree from the outset on the primary objective of an evaluation, and then design the evaluation (including the extent to which it will have characteristics of KFE vs. DFE) accordingly.
View Dr. Neil Buddy Shah’s full presentation at 3ie’s Evidence Week, the presentation slides, and read the paper.
