Development of the Patient Education Materials Assessment Tool (PEMAT): A new measure of understandability and actionability for print and audiovisual patient information
Article
August 5, 2014
Objective
To develop a reliable and valid instrument to assess the understandability and actionability of print and audiovisual materials.
Methods
We compiled items from existing instruments/guides that the expert panel assessed for face/content validity. We completed four rounds of reliability testing, and produced evidence of construct validity with consumers and readability assessments.
Results
The experts deemed the PEMAT items face/content valid. Four rounds of reliability testing and refinement were conducted using raters untrained on the PEMAT. Agreement improved across rounds. The final PEMAT showed moderate agreement per Kappa (Average K = 0.57) and strong agreement per Gwet's AC1 (Average = 0.74). Internal consistency was strong (α = 0.71; Average Item-Total Correlation = 0.62). For construct validation with consumers (n = 47), we found significant differences between actionable and poorly-actionable materials in comprehension scores (76% vs. 63%, p < 0.05) and ratings (8.9 vs. 7.7, p < 0.05). For understandability, there was a significant difference for only one of two topics on consumer numeric scores. For actionability, there were significant positive correlations between PEMAT scores and consumer-testing results, but no relationship for understandability. There were, however, strong, negative correlations between grade-level and both consumer-testing results and PEMAT scores.
Conclusions
The PEMAT demonstrated strong internal consistency, reliability, and evidence of construct validity.
Practice implications
The PEMAT can help professionals judge the quality of materials (available at: http://www.ahrq.gov/pemat).
To access audio and slides for the article, visit: http://audioslides.elsevier.com/getvideo.aspx?doi=10.1016/j.pec.2014.05.027
To develop a reliable and valid instrument to assess the understandability and actionability of print and audiovisual materials.
Methods
We compiled items from existing instruments/guides that the expert panel assessed for face/content validity. We completed four rounds of reliability testing, and produced evidence of construct validity with consumers and readability assessments.
Results
The experts deemed the PEMAT items face/content valid. Four rounds of reliability testing and refinement were conducted using raters untrained on the PEMAT. Agreement improved across rounds. The final PEMAT showed moderate agreement per Kappa (Average K = 0.57) and strong agreement per Gwet's AC1 (Average = 0.74). Internal consistency was strong (α = 0.71; Average Item-Total Correlation = 0.62). For construct validation with consumers (n = 47), we found significant differences between actionable and poorly-actionable materials in comprehension scores (76% vs. 63%, p < 0.05) and ratings (8.9 vs. 7.7, p < 0.05). For understandability, there was a significant difference for only one of two topics on consumer numeric scores. For actionability, there were significant positive correlations between PEMAT scores and consumer-testing results, but no relationship for understandability. There were, however, strong, negative correlations between grade-level and both consumer-testing results and PEMAT scores.
Conclusions
The PEMAT demonstrated strong internal consistency, reliability, and evidence of construct validity.
Practice implications
The PEMAT can help professionals judge the quality of materials (available at: http://www.ahrq.gov/pemat).
To access audio and slides for the article, visit: http://audioslides.elsevier.com/getvideo.aspx?doi=10.1016/j.pec.2014.05.027