Do random assignment evaluations measure the right thing?
Do random assignment evaluations—which compare outcomes of families or individuals who participate in a social program to outcomes for otherwise identical families/individuals excluded from it--provide a reliable indication of the program’s contribution to addressing the nation’s social problems?
If members of the excluded “control group” participate in similar services provided by another branch of government or by nonprofit agencies in the community, as is inevitable in a decentralized, fragmented system such as exists in the United States, this can be called into question.
Is this—as some would contend—a fatal flaw of the random assignment evaluation approach since, unlike medical trials (wherein an inert placebo is administered to to control group members) social experiments compare their “treatments” to whatever other services exist in the community?
“No,” says Abt Senior Fellow and Principal Scientist Stephen Bell. “It constitutes a strength rather than a weakness of the random assignment approach by giving experimental studies the same character as the real-world programs they evaluate.”
How so? Because, according to Bell, some of the people who apply for a specific agency’s assistance would, in the absence of that program, obtain similar assistance from other sources. If they subsequently achieve the same results in terms of their employment, food security, or educational attainment (or whatever other outcome the intervention seeks to enhance), the money spent on the particular program under study could be cut back without consequence.
“Looking at its own programs compared to everything else that’s out there is exactly what a government agency should be doing—identifying the parts of its portfolio that make a difference,” said Bell. “Interventions that add nothing to the existing landscape can be cut back without consequence, allowing money to be moved to programs that do benefit participants more than the alternatives and hence benefit taxpayers.”