February 15, 2017
Vice President, Social & Economic Policy
Principal Scientist, Social & Economic Policy and Director, Research & Evaluation Expertise Center Abt evaluation experts Stephen Bell and Laura Peck recently shared insights and “hot tips” on the American Evaluation Association blog as part of the AEA365’s Experiments Topical Interest Group Week. In it, they examined concerns about social experiments and provided ways to avoid common pitfalls. Here’s a look:
The Ethics of Using Experimental Evaluations in the Field
In this piece, Stephen Bell and Laura Peck address concerns about the ethics surrounding experimental evaluations and randomizing access to government services. “Giving all deserving applicants an equal chance through a lottery, is the fairest, most ethical way to proceed when not all can be served,” they write.
What Can Experimental Evaluations Tell Us? And Why We Should Not Be So Doubtful About What They Won’t Tell Us
Laura Peck recaps what experimental evaluations typically tell us and highlights recent research that helps tell us more.
More Things You Thought You Couldn’t Learn from a Randomized Experiment…But You Can
Stephen Bell addresses three apparent drawbacks in what experiments can teach us and notes how multi-stage random assignment can be used to answer questions about the effects of different treatment components. “Don’t let naysayers turn society away from experimental designs without first thinking through what is achievable,” he advises.
Do Social Experiments Inevitably Distort the Programs They Set Out to Study?
In this post, Bell asks whether randomization necessarily distorts the intervention that an experiment sets out to evaluate. He offers options for evaluators and concludes that “social experiments need not distort the programs they set out to study.”
Biggest Complaints: Experiments Have Limited External Validity, Take Too Long, and Cost Too Much
Laura Peck discusses the most common complaints about experimental evaluations and discusses how they don’t have to be more time-consuming or costly than other kinds of impact evaluation. “The future is bright for experimental evaluations to meet high standards regarding external validity,” she writes.
Additional reading on these and other points appear in Bell and Peck's recent scholarly article on the topic.