This page is optimized for a taller screen. Please rotate your device or increase the size of your browser window.

Getting the Word Out: How Can Formative Evaluation Impact Program Direction?

April 6, 2018

city streetAbt Associates has been working with the Conrad N. Hilton Foundation since 2011 to evaluate their Chronic Homelessness Initiative — a 10-year, multi-faceted strategy to address chronic homelessness across Los Angeles County. Through our work, we learned the value of implementing an evaluation design that allowed us to provide regular feedback while it could still inform the effort’s direction.

From 2011 to 2015 – phase one of the initiative – the Abt evaluation team established a formative evaluation of the initiative. Unlike summative evaluations, which delay presenting results until the program is complete, formative evaluations provide ongoing learning and continuous feedback to help stakeholders achieve their goals. 

These ongoing, quick turnaround findings provided several advantages to us:
 

Getting the Information Out There

Experience our digital dashboard report for the Conrad N. Hilton Foundation’s Chronic Homelessness Initiative.

Quick-turnaround pieces, particularly those with an emphasis on lessons-learned, usually mean less formal, shorter reports or infographics that people can readily understand and actually apply to their work. Back in 2012, our annual reports began looking a lot like traditional evaluation reports: long and, admittedly, a bit dry! Over the years, however, we’ve shifted our focus to making our findings readable, easy to understand, and useful in the field.

Figuring Out How to Get the Biggest ‘Bang for your Buck’

Philanthropic funding, while critically important, is small relative to public funding sources. Formative evaluation can assist staff and boards to think through where to wield their funding to make the biggest impact. The Foundation’s grantmaking initially emphasized pilot projects to foster both recognized and exploratory approaches to serving chronically homeless people, prioritizing them for housing, and helping them retain that housing. While the Foundation had always funded both direct programs and opportunities to drive systemic change, over the course of the Initiative, with feedback from the evaluation team, the Foundation shifted from an emphasis on funding individual supportive housing programs, finding more opportunities to catalyze sustainable funding and systems change.
 

Documenting Best Practices and Challenges

Before a model is ready for a formal evaluation – or even a broad scale rollout – a formative evaluation can provide crucial rolling feedback about what’s working and what lessons can be carried forward. By documenting practices in the field – with challenges and successes – as agencies are rolling out strategies, we were able to produce a blueprint for scaling up.

Our report on Los Angeles’ Flexible Housing Subsidy Pool documents how private and public partners worked together to support a locally-driven funding model for permanent supportive housing, an approach that has since rapidly scaled up within L.A., and which could readily be adopted in other cities and counties.

Testing Small and Planning Big

Foundation funding often supports innovative programs and interventions. Formative evaluation is a great way to reap the most benefits from small scale pilot projects and ensure that the valuable lessons learned don’t end up collecting dust on a shelf. Our short report and infographics on the Coordinated Entry System (CES) for Los Angeles document a pilot phase in Skid Row, as well as early attempts to build policy and infrastructure for streamlined access to homeless services countywide.

Following the initial year and half of private support for the project, the CES was adopted by the City/County Continuum of Care, the Los Angeles Homeless Services Authority, and is being formalized to meet U.S. Department of Housing and Urban Development requirements. Public funders could use small pools of funding to employ this approach with their own resources, or consider collaborating with a private funder to test and refine before deploying large pools of public funding.
 

Sharing Results with Others

The response to our evaluation approach has been quite positive. The Foundation and its partners have shared our mini reports within the community and with other cities and funders interested in adopting some of the strategies. We are honored that Glasspockets #OpenForGood selected the final Phase One report as their staff pick in December 2017, noting, “the combination of a summary ‘dashboard’ with easily digestible infographics about both the process of the evaluation and its findings, and a clear summary analysis for each strategic goal, makes this evaluation stand out from the crowd.”

Our first five-year partnership with the Foundation has taught us that ongoing, responsive feedback leads to more community engagement with the evaluation. Even evaluators using more traditional approaches may find that incorporating some of these principles increases the impact of their findings. We’re looking forward to seeing what we’ll learn in the next five years!