Every now and then, I like to stop and think about how things have changed in policy research, at least from a quantitative methods perspective. We see increases today in the use of machine learning—the use of computers to analyze and categorize vast amounts of data—something that didn’t exist when I started out in the field.
In my first real research job as a graduate student at Columbia University I examined administrative data on community college students, and I’ve gravitated toward the quantitative side of things ever since. Back in 2008, our research group had what we thought was a considerably large dataset: information on 250,000 community college students from 57 institutions in nine states. With this information, we tracked how students generally moved from one “level” of remedial math and English to another. We showed this using some clever descriptive statistics, ran a few regressions and demonstrated that students did not move very well from level to level.
At that point, there was no real talk of using machine learning in policy research. Some definitions linked it to artificial intelligence, but it was all a bit murky. Remember, this was a time in policy research when a one gigabyte dataset was unprecedented, and you’d be lucky if you used a server to store and analyze your data. These were the general limitations of the era. Even private industry, particularly in marketing and e-commerce, didn’t have the tools to invest in machine learning applications. The Netflix Prize, which provided $1 million to the team with the best algorithm to predict movie enjoyment back in 2009, was one of the first publicized instances of machine learning use.
After 2010, machine learning started to gain attention beyond purely academic circles. In 2011, IBM’s Watson made waves when it easily beat Jeopardy! champion Ken Jennings in a televised match. That changed opinions on the potential of machine learning, and artificial intelligence more broadly. In 2012, a team of researchers fed YouTube videos to a machine learning algorithm to train it to identify cats without providing any prior information to the algorithm on what a cat should look like. Although humorous in its application, this was a huge step in machine learning’s ability to understand what something should look like and accurately identify it.
Meanwhile, policy research continued to progress as it had over the previous decades: running descriptive statistics and regressions and using tools to improve causal inference. For example, instrumental variables estimation was still very much in use around 2010. Regression discontinuity designs had gained prominence. Propensity score matching methods had become increasingly popular. However, randomized controlled trials were and remain the ideal means for establishing causality for impact evaluation questions.
To use any of these methods with appropriate power, data involving several thousands of individuals would usually suffice. As a result, there was no real need for machine learning applications to process our data. Ingesting thousands of YouTube videos for a cat-detecting algorithm today requires immense amounts of data (a pixel on an image frame is a single data point), on a scale that easily dwarfs the largest datasets used in policy research. But the policy research world is changing with the larger world around it, and it is beginning to understand the cost-effective advantages in analysis precision and speed. In this new world, a place like Abt has tremendous advantages: our content and methods expertise allow us to not only take part in but to help shape this world using machine learning and data science.
Read more in this blog series:
Machine Learning Today: How Far Have We Come?