AMCHP-CDC Skills Building Suite:
Evidence-based Practice: Moving
along the Continuum from Selection
to Sustainability
Friday, September 13, 2013
Starr Banks, MPH
Kathi Wilson, PhD, MPH, CHES
Quick Overview
How to Use Web Technology
• Mute your line by using
the mute function on your
phone or by using *6 to
mute/un-mute
• Asking a Question
– You can type your
questions into the chat
box (shown right)
– Raise your hand. Using
the icon at the top of
your screen (example
shown right)
• Lastly active participation
will make sure today’s
webinar a success!
2
Evaluation and Sustainability
of
Your Evidence-Based Programs
Starr Banks
Kathi Wilson
Community Guide Branch,
Centers for Disease Control and Prevention
Division of Epidemiology, Analysis, and Library Services (proposed)
Disclaimer
Any views or opinions expressed by the speaker do not
necessarily represent the views of the CDC, HHS, or any other
entity of the United States government. Furthermore, the use of
any product names, trade names, images, or commercial sources
is for identification purposes only, and does not imply
endorsement or government sanction by the U.S. Department of
Health and Human Services.
4
Acknowledgements
This training is brought to you by
5
Objectives
•
Name your most important outcomes.
•
Define different types of evaluation.
•
Match your evaluation methods with your objectives.
•
Identify components of sustainability.
•
Assess your program’s sustainability .
•
Recognize the importance of incorporating a sustainability plan for your evidence-based
approach.
6
Where do We Stand?
Community
Assessment
Establishing
Goals &
Objectives;
Planning for
Evaluation
Finding
Evidence
Selection
Adaptation
Sustainability
Implementation &
Evaluation
7
EVALUATION PLANNING
Why collect evaluation data?
•
Communicate findings to staff, partners and funders
•
Contribute to the evidence base. Create practice-based
evidence
•
When findings indicate that objectives were not met, explore
and address the reasons
•
When findings indicate objectives were met, celebrate!
Causes of Program Success or Failure
How intervention is designed
Sound
How the
intervention is
implemented
Faulty
Sound
Intervention
Program success
problem
Faulty
Implementation
Program failure
problem
What outcomes are most important?

In designing an evaluation plan,
find balance between
• Desired outcomes
• Feasible outcomes
Outcomes
•
Are the changes that result from your program
•
Are linked to either activities or objectives
•
Can be
• Immediate
• Short-term
• Long term
Poll Question #1
13
General Evaluation Procedure
Planning
Develop the questions, consult with the program
stakeholders or resources, make a timeline.
Data Collection
Pilot testing. How will the questions be asked?
Who will ask them?
Data Analysis
Who will analyze the data and how?
Reporting
Who will report and how? Who will receive the
data and when? How will it affect the program?
Application
How could your results be applied in other
places?
Objectives Contribution to Evaluation
Objective
Type
Desired Result
(Examples)
What to Evaluate
Process
•
Intervention delivered
as planned
• Target audience
engaged
•
Intervention delivery
• Attendance or awareness
• Satisfaction
Learner
(Immediate)
•
Increased knowledge
• Attitude change
•
-Behavior
change
-Policy or
system change
(Short-term)
•Increased
activity
•Increase number of
places to buy fresh food
in the community
•
Overall
(Long-term)
•
Improved QOL
• Reduce morbidity,
mortality
•
Changes in knowledge
• Changes in attitudes
Behavior changes
• Policy changes
• Cause of changes
Health status changes
• Cause of changes
Poll Question #2
Write SMART objectives
•
Specific: Boy and girls between the ages of 10-13
•
Measureable : Decrease obesity rate in adolescents from
35% to 5%
•
Achievable: Make sure a 30% decrease is possible in the
time fame given
•
Realistic: Adolescents in 3 middle schools in Oconee County,
GA
•
Time-framed: In 2 years
Activities Achieve Objectives
Activities are the specific things that you do to accomplish your
objectives.
• Link activities to objectives
• Describe actions
• Specify timing and intended population
BOTTOM LINE:
Each activity leads in an obvious way to an objective.
Evaluation Planning:
Evaluation Phases and Types

Process evaluation
• Are all activities being implemented as planned?
• Are you reaching the intended audience?

Short-term outcome evaluation
• Does behavior change?
• Is a policy documented? Is it enforced?

Long-term outcome evaluation
• Does health improve?
• Do morbidity and mortality decrease?
Process Evaluation

Can find problems early

Includes an assessment of the staff, budget and how well
the program is doing overall

Includes measurement of adoption, reach, participation,
exposure, implementation, satisfaction, barriers and
facilitators, and maintenance

Data collection methods may include:
•
•
•
•
databases
registration sheets
surveys
interviews
Windsor et al., Evaluation of health promotion, health education, and disease prevention programs, 2003.
Short-term Outcome Evaluation

Assesses short-term effects on awareness, knowledge,
attitudes, behavior, setting or environment

Can assess adoption or enforcement of a policy (including in
schools, worksites, faith organizations, etc.)

Measures progress in meeting your objectives

Data collection methods may include:
• surveys
• interviews focus group
• observation
• program implementation logs
Poll Question #3
Evaluation Methods & Tools





Tracking logs: Keep an ongoing record of activities or
outcomes
Existing data: Use data that has already been collected for
another purpose.
Surveys or interviews: mailed, handed, or emailed to clients,
patients, providers either one time or both before and after
an activity.
Focus groups: Open ended questions asked of a small group
of similar people to explore questions more deeply.
Observation: site visits to document extent to which activities
are implemented as planned.
23
Long-term Outcome Evaluation

Assesses if the long-term goals were met



Measures sustained program or strategy changes

Use common data collection methods
Measures longer term changes in rates of illness,
death, or health status of your population
Activity #1
Sharing Evaluation Results

Evaluation findings need to reach all stakeholders
• In language that everyone understands
• In time to make decisions

Program maintenance v. adaptation based on
evaluation findings

Broader dissemination and application
Sample 1: Evaluation Worksheet
Sample 1: Evaluation Worksheet Instructions

Program Goal and Objectives column
• Write a major objective of the program.

Overall Intervention column
• Write at least one activity that will help you achieve your objective.

Evaluation Activities column
• Look for possible evaluation activities. They should be based on the
goals and objectives as well as planned intervention activities.

Evaluation Results column
• Brainstorm about what you hope to learn from the evaluation.

Evidence of Activities and Quality column
• Brainstorm about what kinds of materials will be needed.
• For example, a focus group would require a question guide and
consent forms. You might need instruction materials for the
participants. Are they available?

Evidence of Results column
• Think about what kind of results would show that an objective has
been met.
Sample 2- Smoke-free Policy Evaluation Methods
Type of
Evaluation
Evaluation Question
Data
Data Collection
Method
Process
Evaluation
What materials were
created for smoke-free
policy formation?
How many organization
were targeted?
Number of materials
or organizations
Interview with staff
Survey of organizations
Process
Evaluation
Did policy staff implement
all process steps
Checklist of conduct
of each policy
process step
Policy implementation checklist
Process
Evaluation
What are barriers or
facilitators to
implementation or
enforcement?
Barriers, Facilitators
Organizational survey
Key informant interviews
Short-term
Outcome
Evaluation
How many organizations
adopted the policy?
Number of
organizations
Survey of participants (after the
program) about healthy eating
Review of church policy
Outcome
Evaluation
What are the smoking rates
after the smoke-free rule
adoption?
Number of
employees who
smoke
Annual employee survey
Evaluation Resource List


CDC Framework for Program Evaluation
http://www.cdc.gov/eval/framework/index.htm

American Evaluation Association http://www.eval.org/

CDC Demonstrating your Program’s Worth
http://www.cdc.gov/ncipc/pub-res/demonstr.htm

Reach Effectiveness Adoption Implementation Maintenance (RE-AIM)
http://cancercontrol.cancer.gov/is/reaim/

The Evaluation Center at Western Michigan University
www.wmich.edu/evalctr

University of Wisconsin - Extension: Program Development and
Evaluation http://www.uwex.edu/ces/pdande/index.html

W.K. Kellogg Foundation Evaluation Handbook http://www.wkkf.org
Evaluation References




Glasgow, R.E., Vogt, T.M., & Boles, S.M. (1999) Evaluating the
public health impact of health promotion interventions: The
RE-AIM framework. American Journal of Public Health, 89,
1322-7.
Audrey, S., Holliday, J., Parry-Langdon, N., & Campbell, R.
(2006). Meeting the challenges of implementing process
evaluation within randomized controlled trials: The example
of ASSIST (A Stop Smoking in Schools Trial). Health Education
Research, 21, 366-77.
Beresford, S.A., Shannon, J., McLerran, D., & Thompson, B.
(2000). Seattle 5-a-Day work-site project: Process evaluation.
Health Education & Behavior, 27, 213-22.
Grembowski, D. (2001). The practice of health program
evaluation. Sage: Thousand Oaks, CA.
SUSTAINING YOUR
INTERVENTION
Where do We Stand?
Community
Assessment
Establishing
Goals &
Objectives;
Planning for
Evaluation
Finding
Evidence
Selection
Adaptation
Sustainability
Implementation &
Evaluation
33
Planning for Sustainability
Sustainability is…
… the existence of structures and processes which allow a program to
leverage resources to most effectively implement evidence-based policies
and activities.
Sustainability capacity is…
…the ability to maintain programming and its benefits over time.
Sustainability requires…
… knowledge of factors critical to program success.
Luke D, Calhoun A, Herbers S. Building & Managing Sustainability in Public Health Programs. CDC Roundtable Presentation. Feb. 23, 2012
Why is planning for sustainability important?

WITHOUT planning for sustainability
• Waste $ and other resources
• Damage trust between program and community
• Limit ability to succeed

WITH planning for sustainability
• Lasting outcomes
• Healthier communities
35
Sustainability Planning Resources
http://ctpr.wustl.edu/sustainability/
Sustainability Tool Definitions

Program – refers to the set of formal organized activities that
you want to sustain over time.

Organization – encompasses all the parent organizations or
agencies in which the program is housed.

Community – refers to the stakeholders who may benefit from
or who may guide the program.
37
Program Sustainability Assessment Tool

Purpose –
• Assess program’s current capacity for sustainability across a number of
factors
• Identify strengths and challenges to sustainability
• Guide action to improve sustainability as part of program planning
38
Program Sustainability Domains









Funding Stability – making long term plan on stable funding
Political support – internal and external political environments
Partnerships – connection between program and community
Organizational Capacity – resources needed
Program Adaptation – ability to adapt and improve
Program Evaluation – monitoring of processes and outcomes
Communications – strategic distribution of successes
Public Heath Impacts – program’s effect
Strategic Planning – process that defines program directions
Sustainability Activities - Examples






Seek additional funding
Maintain or secure a program champion
Continue partnerships developed or build new ones to share
costs/implementation
Referrals for further service, if needed
Make intervention a part of organizational services
Surveillance of the impact of the PSE
(e.g., tracking of smoking rates, BMI of children)
Poll Question #4
41
Homework Challenge

Complete Sustainability Plan - Center TRT
42
Questions & Discussion
43
Thank you!
Kathi Wilson
Starr Banks
Community Guide Branch,
Centers for Disease Control and Prevention
Kathi Wilson kwilson@cdc.gov
Starr Banks smbanks@cdc.gov
Center for Surveillance, Epidemiology, and Laboratory Services (proposed)
Division of Epidemiology, Analysis, and Library Services (proposed)