Abt’s approach to rapid cycle evaluation (RCE) is guided by the objective to improve program effectiveness within a program cycle, and rapidly! We use RCEs to assess alternative approaches to implementing a program before scaling up any one approach, particularly when the evidence doesn’t point to an obvious “right” approach. As Abt expands and implements ever-larger programs that reach thousands and millions of beneficiaries, clients are partnering with us to implement RCEs to help improve the programs’ impact.
RCEs are completed using the most rigorous approach feasible to assess the effectiveness of two or more implementation approaches. These evaluations can be as rapid as the time period needed for short-term indicators to change, and they also generate evidence to inform implementation within the program cycle. The short-term indicators used to assess the effectiveness of alternative approaches must have evidence that links them to long-term impact indicators. Data for an RCE ideally come from the program’s monitoring or administrative databases—rather than from conducting a survey—to keep costs low. Using RCEs, Abt helps our clients improve cost-effectiveness even as we generate impactful data.
Abt uses a range of experimental and non-experimental research methods combined with rigorous qualitative methods to implement RCEs. We start with qualitative, cutting-edge research to substantiate the alternative approaches to test and then articulate the intervention’s theory of change. During this process, we engage deeply with stakeholders and sometimes using learning collaboratives to help inform the research and interpret the results, using them to provide feedback. We draw on our evaluation staff who have decades of experience determining the right evaluation approach and our subject matter experts covering a vast range of topics, including health agriculture, governance and environment and energy.
Helping Save the Children in Cambodia
Client: USAID Development Lab
In Cambodia, Abt helped improve Save the Children’s efforts to reduce the number of children unnecessarily separated from their families. Fueled by well-intentioned donations, residential care institutions (RCIs) have become a lucrative business in Cambodia. So much so that the RCIs often recruit children away from families that are poor. We conducted two RCE pilots that assessed whether social and behavior change communication (SBCC) can effectively convey the value of families staying together, particularly when compared with placing children in RCIs.
One pilot tested whether adding SBCC to social work services was more effective than social work alone in communicating the importance of keeping families together. We only found a difference in attitudes toward some factors related to RCIs in villages that received SBCC. These families were less likely to report a situation in which they would send their child to an RCI in the future. The pilot also generated information that the program is using to improve how it identifies at-risk households and the type of support that they provide to those households. On the flip side, another RCE pilot helped identify the optimal social media strategy to change donor behavior. In that pilot, we found that the SBCC Facebook ads had a small effect on research participants’ perceptions of RCIs, but we did not detect an effect from the same ads when seen on Google. The RCEs provided the benefit of giving information within the program cycle, enabling the implementers to improve their effectiveness. Further, the RCEs provided information on the cost-effectiveness of interventions, which enabled the programs to make decisions about continuing and scaling-up interventions within the initial stages of the project.
Evaluation and Monitoring of the Bundled Payments for Care Improvement Initiative
Client: Centers for Medicare & Medicaid Services (CMS)
CMS, through the Center for Medicare & Medicaid Innovation, implemented the Bundled Payments for Care Improvement (BPCI) Initiative in 2013. Using RCE, Abt helped determine if participating in BPCI led to better quality, more coordination and lower cost relative to Medicare. BPCI comprises four broadly defined models linking payments for all services that Medicare beneficiaries receive during an episode of care. In partnership with the Lewin Group, Abt conducted a mixed methods RCE for three of the BPCI models, covering 48 different clinical episodes of care per model. The evaluation was designed to determine if participating in BPCI models led to higher quality and more coordinated care at a lower cost to Medicare. Abt conducted site visits to providers participating in BPCI, quarterly interviews with BPCI participants and expert interviews. We also collected quantitative data on patient experience, satisfaction and functional status through a beneficiary survey, as well as through Medicare claims, enrollment data and post-acute care clinical assessment data. Early returns from the BPCI evaluation helped CMS design and implement a new bundled payment model, which decreased Medicare spending by 3.3% per episode of care in the first performance year.
Mobilizing Households for Indoor Residual Spraying To Prevent Malaria: A Case Study in Mali
For USAID’s Africa Indoor Residual Spraying (AIRS) project, Abt used an RCE in Mali to test whether mobilizing recipients of anti-malaria insecticide spraying to prepare for treatment by mobile phone could provide a successful alternative to using door-to-door mobilization. To answer this question, Abt used a randomized controlled trial to evaluate home preparedness in villages where beneficiaries were mobilized through text messages compared with villages that were prepared for spray through door-to-door mobilization Within the program cycle, the RCE was able to quickly determine that the mobile phone-based option was both less effective and more costly per structure than the traditional door-to-door approach. Because the RCE was conducted quickly within the program cycle Abt knew to continue using the less costly and more effective door-to-door methods, providing greater value—and better results—for our clients.