Democratizing Evidence: Leveraging Advances in Systematic Evidence Reviews
More than 28,000 peer-reviewed journals publish research in English, and more than 2.5 million articles are published worldwide each year. With such a massive amount of knowledge being produced every day, it can be nearly impossible for policymakers and practitioners to stay on top of the research in their fields and make informed choices based on the research. So how can we make sense of this overwhelming amount of information?
The answer is systematic evidence reviews. These reviews use standardized protocols to identify all published and unpublished evidence on a given topic, determine which evidence is relevant, assess the rigor of the evidence, and summarize evidence that meets review standards. Systematic evidence reviews allow us to translate research more effectively, to decrease time-to-market for research findings, and, ultimately, to become smarter, faster.
During the American Evaluation Association’s national conference in New Orleans, Abt presented three exciting advances in systematic evidence review methods and applications that would help democratize evidence. Each has the potential to be truly transformative in advancing connections between research, policy, and practice.
- International Applications: Systematic review efforts are gaining momentum in international development work, though they face unique challenges in that context. Relative to U.S. domestic systematic reviews, international systematic review efforts tend to have a lower volume of experimental and quasi-experimental research to review, so traditional evidence standards—which typically focus on these designs—may fall short. Moreover, because systematic reviews in the international development context have more diffuse audiences, greater resource constraints, and weak feedback loops between evidence and practice, there is a great need to engage stakeholders to adapt evidence across contexts—and continents. Abt recently developed a resource guide on Adapting Evidence-Based Practices for Under-Resourced Populations that can help address inherent challenges in generalizing evidence in both domestic and international contexts.
- Equity-Driven Approaches: Systematic evidence reviews ultimately are about knowledge building, so it is essential that we amplify the voices of under-resourced populations that are so often participants in research. There are a number of concrete things researchers can do to bring an equity lens to their systematic reviews. More inclusive knowledge-building strategies include: (a) incorporating lived expertise into the planning, analysis, and dissemination of systematic reviews, (b) pursuing the authentic engagement of stakeholder groups who could benefit from results and who implement recommended practices, (c) getting beyond mean-centric data presentations to address what works for research participants who are facing the greatest challenges, (d) addressing implicit biases in measurement, such as in standardized testing, (e) identifying characteristics that may signal inequities such as race, ethnicity, gender, sexual orientation, culture, education, and socioeconomic status and documenting findings based on these characteristics, and (f) distilling evidence from interventions to discrete practices in order to expand opportunities for knowledge building and ensure equitable data dissemination.
- Core Components Analysis: An inherent challenge we face in maximizing the utility of systematic evidence reviews is that only a limited number of users can act on the findings because such reviews often provide evidence on expensive multi-component interventions which have a relatively limited number of people with the power to purchase them. But what if we could expand the audience for systematic reviews by distilling this evidence down to discrete components that may be implemented at low or no cost? This would effectively “democratize” evidence by pushing actionable evidence to the front lines of service delivery. Abt is a leader in core components analysis, which is a rapidly growing method to unpack evidence. This method involves systematically coding a group of core components—which can involve intervention content, intervention strategies, and participant characteristics—and including these components as independent variables in a meta-analysis. Abt has completed core components analyses for the U.S. Department of Labor on effective elements of career pathways programs, and for the U.S. Department of Health and Human services on core components of effective youth programs.
Together, these advances have great potential to transform the ways we use evidence to transform policy and practice—and to do so in a more equitable manner.
