Sometimes, the Story is in the Subgroups
Program evaluators are often asked to determine whether a given program or program had its intended effects.
Although answering the question “did it work?” is often the primary goal of evaluation, policy makers and program managers also want to know “for whom and under what circumstances did the program work?” Very often, evaluations reveal that the story is in the subgroups.
For example, women might benefit from a particular program more than men, or program impacts might be more favorable for those participants who had certain experiences before enrolling. Understanding how program impacts vary across subgroups is critical to helping policy makers and program administrators improve and target their services more carefully to those most likely to reap maximum benefits. This is especially important as we consider how to craft a more equitable society: those subgroup impacts help identify the extent to which an intervention might benefit some participants over others, or close pre-existing gaps (for example, those that exist due to longstanding social inequities).
Within the context of an experimental design, if the subgroups of interest are defined by characteristics that are observed before the study entry (such as prior work experience) or do not vary with time (such as race), then these types of subgroup analyses are generally straightforward. We call these “exogenous” subgroups. Analyzing impacts on these subgroups is essential to telling a complete story about a program’s impacts; and evaluations should be sure that they collect these data and have sufficient sample sizes to permit generating this evidence.
We may also want to know about subgroups that are defined by post-random assignment behaviors, which we refer to as “endogenous” subgroups. Here, analyses are less straightforward. For example, a school superintendent interested in implementing an after-school tutoring program might want to know about the effectiveness of the program for improving test scores among those students who, had they not enrolled in the program, would not have found some other source of tutoring. The evaluator can observe whether a control-assigned student found some alternative form of tutoring but cannot observe whether a treatment assigned student would have done so had he or she been assigned to the control group.
Methodological research has made advances on this challenge (see here, here, and here); and those tools for examining endogenous subgroup impacts are being applied in evaluations to improve the evidence base regarding those what works best for whom kinds of questions. Approaches for analyzing program effects on endogenous subgroups include principal stratification, Analysis of Symmetrically-Predicted Endogenous Subgroups (ASPES), and extensions of these leveraging machine learning tools (an early example in the literature is here).
Program and policy decisions are informed both by evidence of the average treatment effect and of how impacts vary across groups. An overall negative or null finding may mask large favorable impacts to some subgroups and correspondingly large negative impacts for some other subgroup. These types of subgroup analyses can provide critical information to support program improvement and the refinement of policy, strengthening the tools with which we might gain insights into those important questions of “for whom” and “under what circumstances” programs work.
Subscribe to our bimonthly newsletters, with information about our work, staff, and current job openings, and other periodic mailings.
