Plan, Evaluate and Improve Your Health
Literacy Interventions and Activities
Sabrina Kurtz-Rossi, M.Ed.
Health Literacy Consultant
April 10, 2013
Objectives
• After attending the session, participants will
be able to:
– Conceptualize a health literacy program evaluation
– Write measurable health literacy outcome objectives
– Identify indicators to assess both process and outcomes
– Identify user-friendly evaluation resources for future use
Olney, C. and Barnes, Susan (2006). Planning and Evaluating Health Information
Outreach. NN/LM Outreach Evaluation Resource Center. Seattle, WA.
Ten Attributes of Health Literate
Health Care Organizations
–
Brach C, Keller D, Hernandez LM, et al. Ten Attributes of Health Literate Health Care Organizations.
Washington DC: National Academy of Sciences. 2012
Case Study
Providence, RI
Dorcas Place Adult and Family
Learning Center Computer Lab
Western Maine
Auburn Public Library
Computer Training Class
Research vs. Evaluation
• Research: To draw causal links between
observed phenomena and to add to the
knowledge base of the field
• Evaluation: To demonstrate effectiveness, and
determine what works and what does not for
the purpose of program improvement
Boulmetis, J and Dutwin, P (2005). The ABCs of Evaluation. Jossey-Bass. San Francisco, CA
Health Literacy Program /
Evaluation Planning
• Step 1: Know Your Audience
• Step 2: Identify Goals and Objectives
• Step 3: Develop Program (and evaluation) Plan
• Step 4: Implement and Evaluate
• Step 5: Apply and Share Results
Step 1: Know Your Audience
• Start with an Assessment
–
–
–
–
Identify needs in context
Guide your program
Build on already existing capacity
Identify champions within your organization
• Methods
– Informal – discussion, observation
– Formal – Key informant interviews, focused group
discussions, survey
Attribute 1:
In a Health Literate Organization leadership
makes health literacy integral to its mission,
–
structure, and operations.
Case Study
• Methodology:
Leaders at community-based organizations interviewed. Semistructured interview protocol designed to explore community confidence in using
the Internet to find health information and local resources to support access. All
interviews conducted in person. Data were coded and analyzed into themes.
• Key Findings:
•
Maine has a growing senior population and community-based organizations
support public health efforts. Public schools, adult education, and libraries
offer computer access and support (n=9).
• Latinos get their health information from sources they know and trust,
including the radio. A network of minority health programs offer computers
access, training and support (n=10).
Step 2: Identify Goals & Objectives
• Goal -- Why you are doing what you
are doing and for whom
• Example:
– Increase use of health literacy interventions
– Improve community access to reliable online
health information
Attribute 2:
A Health Literate Organization integrates
health literacy into planning, evaluation,
–
patient safety, and quality improvement.
Case Study
Focused Health Information Outreach Project
Goal: Increase access to health
information among communities
experiencing health disparities
National Network of Libraries of Medicine –
New England Region (NN/LM NER) http://nnlm.gov/ner/
Step 3: Develop Program(and
Evaluation) Plan
Logic Model: A planning tool
connecting activities to outcomes
Resources
If you have
these
resources
Activities/Outputs
and conduct
these activities
Outcomes
you will get
these
outcomes
Why a Logic Model?
• Provides a framework for planning
• Helps organize your thoughts
Resources
Activities
Outcomes
Outputs
• Clarifies intended outcomes
• Links activities to outcomes
• Acts as a communication tool
Attribute 3:
A Health Literate Organization
prepares its workforce.
Case Study
Goal: Increase access to health information among communities
experiencing health disparities
Resources
Consultant
Outreach
coordinators
NLM materials
Activities
Develop training
materials
Conduct
trainings
NLM funding
Working student
Librarians in
communities
Distribute
materials at
community
events
Outputs
Number of
trainings held
Number
trained
Types of
materials
developed
Number of
materials
distributed
Outcomes
Increase
knowledge of
NLM resources
Increase use
of NLM
resources
Increase
community
experts to help
others
Activity
• Choose a goal (or use your own)
– To increase staff awareness and use of health literacy interventions
– To improve community access to reliable online health information
– To improve health information literacy skills among patients and their
families
• Complete the logic model worksheet - identify
outcomes, activities, outputs, and resources
Attribute 4:
A Health Literate Organization
involves the population served
in evaluation of health
information and services.
Case Study
• Community Assessment
– Key informant interviews
• Program
Western Maine
(rural)
– Tailored efforts based on findings
– Community-based partnerships
• Evaluation
– Outcome assessment: pre-post training evaluations,
follow-up assessment
– Process assessment: story-based evaluation,
community partner exit interviews
Providence, RI
(Latino)
Evaluation Plan
• Process
– To determine what worked and what didn’t
– For program improvement
• Outcome
– To demonstrate effectiveness, value, and influence
– For comparison purposes, to determine which programs
are more effective
Attribute 5:
A Health Literate Organization
meets the needs of the populations
and avoids stigmatization.
Process Assessment
• Methods
–
–
–
–
Activity logs
Teacher feedback forms
Focused group discussion
Interviews
Outcome Assessment
• Methods
–
–
–
–
–
–
Existing records
Pre/post tests
Follow-up surveys
Observation
Interview
Story-based
Attribute 6:
In a Health Literate Organization
providers and staff use health literacy
strategies in verbal communication
and confirm understanding.
Case Study
Outcomes & Indicators
Outcomes (what you want)
Indicators (how you will know)
Increased knowledge of NLM
resources
Will name resources on postsurveys
Increased use of NLM resources
Will report use two weeks later
of follow up
Increased community experts to
help others
Will document helping others on
story-based evaluation forms
Measurable Objectives
Four components of a measurable objective
1. Condition
• Following the training…
• After watching the video…
2. Who
• …the learner…
• …the patient…
• …the caregiver
Measurable Objectives
3. Performance (knowledge, attitudes,
behavior)
• …will name…
• … will agree…
• … will use…
4. Criterion (quality/quantity)
• …5 out of 6…
• … at least 2…
• … the correct way…
Characteristics of a
Measurable Objective
• Is audience focused, not intervention focused
(the participant will…)
• Includes action verbs
• Targets one expectation
• Includes observable criteria for evaluation
• Matches goals to activities/strategies
Attribute 7:
In a Health Literate Organization
consumers have easy access to health
information and services and
navigation assistance.
Bloom’s Taxonomy
• Cognitive – explain, recall, list, predict, select,
describe, define
• Affective – agree, avoid, help, join
• Psychomotor – use, develop, design, repair,
taste, use, move
Verbs to Avoid
Know
Comprehend
Understand
Appreciate
Study
Gain knowledge of
Learn
Attribute 8:
A Health Literate Organization
develops plain language materials that
are easy to understand and act on.
Case Study
Writing Measurable Objectives
Condition
Who
Performance
Criterion
Immediately after
training
Two weeks after
training
participants
will name
NLM resources
on post-survey
participants
will report using
NLM resources on follow
up interview
Three months
training
participants
will write a story
Describing how they helped
others find information on
story-based evaluation form
Activity
• Choose an outcome (or use your own)
– Increase confidence in using plain language/teach-back
– Increased awareness of available online health
information resources
– Increased interaction with diabetes educators
• Complete the writing measurable objectives
worksheet - identify condition, audience,
performance, and criterion
Step 4: Implement and Evaluate
• Expect the Unexpected
– Be flexible
– Adjust to the environment
– Address learners’ needs
– Tweak your notion of success
Attribute 9:
A Health Literate Organization
addresses health literacy in high-risk
situations, including transitions
and about medicines.
Case Example
Maine: Pre/Post: Ever Used NLM Resources
45%
Rhode Island:
Pre/Post: Confidence in Ability to Find
Health Information on the Internet
100%
88%
Pre N= 520
Post N=503
80%
39%
40%
Pre N=72
Post N=55
35%
29%
30%
25%
60%
47%
40%
20%
Pre-Training
20%
Post-Training
15%
Pre-Training
Post-Training
10%
0%
Confident
5%
2%
1%
0%
MedlinePlus
NIHSeniorHealth
Case Example
• Successes
– Training participants shared resources with friends and
neighbors
– Partners will continue to promote and distribute resources
– Providers used resources to find accurate information to
use in their work
• Challenges
– Pre- and post-evaluations were hard to administer
Step 5: Apply and Share Results
• Apply Lessons Learned
– Make changes based on what you learned
– Replicate with similar or different audiences
• Share Your Findings
– Partners
– Colleagues
Attribute 10:
At a Health Literate Organization they
clearly communicates about health plans
and health care costs.
Evaluation Resources
• Planning and Evaluating Health
Information Outreach Projects
• Measuring the Difference: Guide to
Planning and Evaluating Health
Information Outreach
http://nnlm.gov/evaluation/guide/index.html
Questions & Comments
Sabrina Kurtz-Rossi, M.Ed.
Principal | Health Literacy Consultant
781-835-6488
sabrina@kurtz-rossi.com
Kurtz-Rossi & Associates
Health*Literacy*Education*Evaluation
www.kurtz-rossi.com
Member, Clear Language Group
www.clearlanguagegroup.com
Health Literacy Leadership Institute, June 10-14, 2013
http://healthliteracyleadreship.com