Abt Associates: Bold thinkers driving real-world impact
Researchers at Abt Associates are breaking new ground with their cutting-edge evaluation methods that allow deeper delving into the “black box” to learn “what works” in social programs. Their forthcoming Member Forum with the Association for Public Policy Analysis and Management (APPAM), “On the Frontier of the ‘How’ of Social Experiments: Getting Inside the Black Box,” explores how to identify “impact drivers” — the various features of social program interventions that cause favorable impacts or make them greater in magnitude. As one session of five in the day-long in-person Forum, this session will be available via webinar from 2–3:30 p.m. on April 24 for those unable to attend in person.
In this Q & A, Laura Peck, Ph.D., Abt Associates principal scientist and a member of the APPAM Policy Council, and Stephen Bell, Ph.D., principal scientist and a vice president in Abt’s Social and Economic Policy Practice, offer a preview of their presentation.
Q1: What's wrong with the social experiments we've been doing; why do we still need new frontiers after four decades in this business?
Steve Bell: We’ve gotten really good at estimating the average effect of a policy intervention in selected places, but we need to do more to “open up the black box” to determine which program features generate larger impacts for participants. A priority should also be placed on gathering data reflective of the entire U.S. without having to include the entire U.S. in our research. Lastly, we need to test programs at a point in their maturation, and at a scale, that give a good chance of uncovering successful interventions.
Q2: You are looking for "drivers" of successful social programs. What do you mean by that? Why are they so hard to find?
Laura Peck: Impact “drivers” can be considered the ingredients that comprise an intervention’s recipe. Social welfare programs tend to be multifaceted, and we evaluate them as such. Randomizing to the program recipe as a whole instead of to selected ingredients prevents easily detecting what it is about the recipe that makes it delicious. But those who design and administer such programs find it useful to know what about their programs are the essential ingredients, and so we must advance evaluation methods to provide this information.
Q3: Can we better use the data we have from social experiments to learn "what works" and what doesn't?
Laura Peck: Yes. Experimental evaluation design is a powerful thing. Thankfully we have many past and ongoing social experiments whose data we can use (1) to further develop methods for uncovering what works and thereby (2) to inform the field about what works. As with any evaluation, an experimentally-designed evaluation is an investment in understanding whether our program dollars are being well spent. I believe we should continue to use existing data from social experiments to grow our evaluation toolkit so that future evaluations can access those tools and answer more policy questions for the same investment.
Q4: How can we design more ambitious social experiments to learn more in the future?
Steve Bell: More of the facets of an intervention that could be difference-makers or "drivers" need to be varied at random to isolate their individual effects. That is, we should try multiple combinations of ingredients within a given program recipe to compare them head to head. We can do this in experimental evaluations by creating three or more randomized groups, staggering randomization over time as participants flow through the stages of a program, and/or crossing one program factor with another in a randomization matrix. These less commonly used designs hold promise by themselves and also as a foundation for future analytic innovations.
To attend the webinar, register by clicking here.
Abt Associates is a mission-driven, global leader in research, evaluation and program implementation in the fields of health, social and environmental policy, and international development. Known for its rigorous approach to solving complex challenges, Abt Associates is regularly ranked as one of the top 20 global research firms and one of the top 40 international development innovators. The company has multiple offices in the U.S. and program offices in more than 40 countries.