Assessing the External Validity of Multi-Site Impact Studies
Impact studies, whether in education or other areas, are often conducted in a small sample of sites – such as schools or districts – that were not selected randomly. Indeed, most impact evaluations select a convenience, or “purposive” sample of sites for the study. Therefore study results may not generalize to the broader populations of sites or to the individuals within those sites who are of interest to policymakers. Abt Associates, in collaboration with Johns Hopkins University, has been conducting methodological research on the external validity of multi-site impact studies conducted in purposive samples. We have also been exploring methods to make impact studies more useful in informing policy decisions. This work has been funded by grants from the Department of Education and the National Science Foundation, and has been published in the Journal of Policy Analysis and Management. Read the journal article: External Validity in Policy Evaluations that Choose Sites Purposively.