Exploring Variation in Impacts to Learn More from Impact Studies
Most impact studies in education and other policy areas are designed to estimate the average effects of the program. However, average effects can often mask variation in effects across individual units — variation that can often be more relevant to policy makers. Further, focusing on average effects can mask variation in effects across sites that may be useful for intervention developers and program implementers to understand. Abt Associates researchers are developing cutting-edge approaches to understand both the overall and more specific effects of educational interventions — to learn more than just what works, but also for whom, and under what conditions. Abt researchers are developing:
- Systematic approaches to measuring implementation fidelity so that our evaluations can better answer questions about whether variation in fidelity is related to program impacts;
- Methods for mediational analysis in impact studies using instrumental variables; and
- Methods for moderation analysis in impact studies when subgroups are only observed for treatment groups, such as implementation, fidelity, dosage, and receipt of particular intervention components.