The idea of using behavioral theory – and especially low-cost “nudges” informed by behavioral science – to influence individuals’ decisions has been attracting a lot of interest lately. Popular examples abound: Your utility company may have nudged you to save electricity by comparing you with your more-efficient neighbors on your recent bill (complete with frowny-face emoticon for extra oomph); or you may have noticed that you were automatically enrolled in a matching-contribution 401k plan last time you took a job rather than being asked to opt in. Nudges like these, many of which are outlined in Richard Thayler and Cass Sunstein’s recent book, Nudge: Improving Decisions About Health, Wealth, and Happiness, are alluring to policymakers due to the combination of their voluntary nature, low cost, and often-surprising effectiveness.
Consistent with these initiatives, the U.S. Department of Labor (DOL) has recently used behavioral insights for a variety of purposes: to help employers understand and respond to citations for health and safety violations, to encourage DOL employees to increase their retirement contributions, and to encourage unemployment insurance claimants to participate in a targeted reemployment program. At Abt, we helped DOL test whether behavioral theory could be used to increase demand for one of the Occupational Safety and Health Administration’s (OSHA’s) voluntary workplace-safety programs. But instead of simply testing whether the behavioral-insight-driven approach worked as a whole (it did!), we designed the study to test which specific components were responsible for the bulk of the impact. In other words, we tried to look inside the black box to discover which parts of the marketing were effective – was it the behavioral theory-driven components or something else?
Using Nudges to Market OSHA’s Program
Our goal was to increase demand for OSHA’s On-Site Consultation (OSC) program. Through OSC, OSHA provides free and confidential advice to small and medium-sized businesses on how to prevent workplace injuries and illnesses. DOL believed that many businesses that could benefit from OSC had not signed up. But why? One possibility is that businesses did not have sufficient information to make the “right” decision (e.g., they didn’t know the program exists). A second is that they knew all about the program but they made the wrong decision anyway—perhaps they were suspicious of OSHA’s motives, or they just procrastinated and never got around to signing up. Our marketing was designed to address both of these possibilities, by providing information about the program as well as behavioral nudges intended to increase the use of the OSC program.
Evaluation considerations required us to use marketing methods that could be targeted to specific worksites in key industries. After considering various marketing approaches, we decided to mail and email brochures with important design features. Specifically, we designed brochures to incorporate—and test—three things that could be thought of as behavioral-theory driven: The content of the marketing, the presentation of that content, and the distribution method for getting the content into the right hands.
The content of the brochures was designed by behavioral marketing experts to frame the OSC program in various ways – in essence, “good for you if you sign up” versus “bad for you if you don’t.” In particular, each brochure incorporated a single message appealing to one of three distinct emotions: fear, self-determination, or expectancy (essentially the anticipation of positive outcomes).
Each of these messages was designed to be presented in three different ways: a “dialogue” format showing two employers talking to each other, a “myth/fact” format in which myths about the program were debunked, and a “future orientation” format in which employers were invited to imagine what things would be like after going through OSC. All told, Abt’s creative services team created nine brochures to be tested – three formats for each of the three messages. To get an idea of how the formats look, here is an example of the first panel of the three brochures developed using the expectancy (“Safety Pays”) message:
We distributed the resulting brochures via two modes: mail and email. We did this to test what behavioral economists call “channel factors;” in this case, removing seemingly minor obstacles to scheduling the visit. One of the things we did to make it easier for employers to follow through was to design emails that complemented the brochures. We hoped that recipients would be more likely to schedule a request if they were already sitting at their computer. We were also worried that finding the right contact information for scheduling an appointment might inhibit response. To combat this, we included state-specific contact information on each brochure and provided a link to the relevant state’s OSC website in each email, where a request could (often) be scheduled.
The study used a multi-arm randomized trial called a “factorial design” that tested each marketing component (message, format, and distribution method) separately. This rigorous design allowed us to test:
- the behaviorally-informed marketing as a whole compared with both no marketing and with an existing informational brochure that did not utilize behavioral nudges;
- each message and format compared with every other message and format;
- whether the email had an incremental impact on the marketing’s success, above and beyond the letter.
The study’s very large sample size (97,182 worksites across 48 states and Washington D.C.) meant that we could detect even very small differences between different kinds of marketing.
What We Learned
The findings from this study were simple and striking. The marketing was very effective overall, nearly doubling the request rate for consultations from OSHA’s program. However, there was no difference at all between brochures with and without behavioral nudges. Contrary to our expectations, the email (which we thought would allow recipients to schedule a consultation more easily, while sitting at their computer) appears to have had no effect. Likewise, the behavioral-theory-based emotional appeals did not seem to improve response. Thus, it appears that the brochures’ success was driven not by the cleverly-designed behavioral nudges, but rather by something common to all the brochures, such as the basic informational content or the friendly cover letter from DOL’s assistant secretary that accompanied each brochure. It is worth noting that we previously tested OSHA’s high-rate letters, which are sent as part of the Site-Specific Targeting program and convey both information and the threat of an inspection, and found that they are also effective at increasing the request rate.
We can only speculate as to why our nudges were not effective—e.g., perhaps a different medium, such as TV or radio, would have been more compelling. But it appears that printed brochures are simply not a very effective medium for conveying the kinds of emotional appeals tested in this study.