Impact evaluations – and randomized control trials, in particular – are the core of Abt Associates’ evaluation work. Previously, we developed custom programming routines and reporting templates for most of our impact evaluations.
But in 2014 we began developing Guidelines for Reporting and Formatting (GRAF) in 2014 after recognizing the potential for a standardized analysis and reporting system to reduce our time and costs and increase the clarity and consistency of evaluation results. GRAF provides guidance and statistical software code for statistical impact analysis, as well as code for producing publication-ready tables exhibiting results from that analysis.
So far Abt has used GRAF in evaluations for the U.S. Department of Labor’s Occupational Safety and Health Administration, the U.S. Department of Agriculture’s Food and Nutrition Service, the U.K. Department for International Development, the U.S. Social Security Administration, and others.
GRAF is structured with twin goals: Best practices and flexibility.
Best Practices: GRAF guides project analyses towards Abt’s view of best practices in impact evaluation. This is true at each stage: –how the analysis should be done, what results should be reported and how they should be reported.
Flexibility: GRAF also recognizes that not all projects are the same, so GRAF provides several analysis templates, options for further tweaking analysis and reporting, and underlying source code for further customization. Some clients and some analyses have strong preferences for the specific format of output, so GRAF’s formatting programs have several standard reporting templates, options for further tweaking output, and underlying source code for further customization.
Core GRAF Tasks
This common core of GRAF tasks includes:
Descriptive statistics; overall, treatment/control, and by subgroup: Associated with these descriptive statistics are balance tests — variable by variable and a global test across all variables.
Impact analyses; overall for multiple outcomes, by subgroups, with regression adjusted means for treatment and control, impact estimates, estimates of precision, and tests for homogeneity of inputs: Analyses are robust to unequal weights, stratification, blocking, and clustering, as well as multi-level designs.
Clear reporting: Corresponding to each of these analyses is a reporting format that clearly presents the key statistics and their variability.