Evaluating for Comprehensive
Cancer Control Policy, Systems,
and Environmental Change
Kellee Thorburn McCrory, MPH
The University of Iowa
National Resource Center for Family Centered
Practice
• What is PSE? Why must we have so many
acronyms?
• Calculating reach
• How do you evaluate a PSE program or
intervention?
• Challenges of evaluating PSE
What is PSE?
• Policies: Policy interventions may be a law, ordinance,
resolution, mandate, regulation, or rule (both formal and
informal)
Example: Organizational policies that provide time off
during work hours for physical activity.
• Systems: Systems interventions are changes that impact all
elements of an organization, institution, or system.
Types of systems include: school districts, transportation, parks
and recreation, health care clinics, etc.
• Environmental: Environmental interventions involve
physical or material changes to the economic, social, or
physical environment.
Example: Incorporating sidewalks, paths, and recreation areas
into community design.
( Healthy South Carolina, 2012)
Increasing
Population
Impact
Counseling
and
Education
Increasing
Individual Effort
Needed
Clinical
Interventions
Long-lasting
Protective Factors
Changing the Context to
Make Individuals’ Default
Decisions Healthy
Socioeconomic Factors
A Framework for Public Health Action: The Health Impact Pyramid
Health Promotion VS. PSE Strategies
• Health promotion program
• Change one person at a time
• PSE strategies
• Change the environment in a way that makes
healthy behavior easier
Health Promotion Program (One)
PSE Strategies (Many)
Education regarding tooth brushing
Add fluoride to water system
Smoking cessation education
No-smoking policies for public places
Healthy nutrition classes
Adding calorie info to menus
(Brega, 2012)
Program vs. Policy Change
Program Characteristics
PSE Change Characteristics
One-time
Ongoing
Often results in short-term behavior
Often produces behavior changes over time
Individual level
Policy level
Not part of an ongoing plan
Part of an ongoing plan
Short-term in duration
Long-term in duration
Not sustainable
Sustainable
What have been
some PSE changes in Iowa?
Importance of Evaluation
• Assessing the impact of an intervention (PSE)
• Has many benefits
• Are you meeting your goals?
• Have the effect you expected?
• Information to share with community and funders
(Brega, 2012)
Baseline and Target Measures
Baseline data: collected prior to intervention; essential to monitor
and track change. The purpose is to:
• Compare what happens before and after intervention
• Assess the effect of the program
• Provide a foundation for showing performance improvement
Target data: provides info on level of change desired over a given
time period that represents success. It is not a guess. It involves
knowing where you are, what you are trying to achieve and what is
realistic to get there.
Most important criteria:
1) data availability
2) use of an informed and systematic approach
Where can I find this data?
Secondary data
• National, state, local surveys, surveillance systems and
registries
• Special studies by national orgs or partners
• University or academic studies
• Mass media
• CDC data
• Project records
Primary data
• If secondary is not available, you will need to collect new
data
USE SECONDARY DATA! IT IS
AVAILABLE AND USUALLY
RELIABLE
Reach: is the absolute number, percentage,
and proportion of settings and people
affected by the intervention.
How does this help evaluation?
Evaluators want to know the so what and
how many of an intervention. Reach can
answer the how many.
How to calculate reach
Reach calculation is a two-step process:
1) Assessing adoption
2) Determining those served
1. Calculate the # of targeted systems/settings
that adopt intervention
2. Calculate the # of or % of people within
those settings that are served by the
intervention
Here are some examples
Number of settings that adopt
40 FQHCs participating
= 50% of FQHCs adopted
80 FQHCs targeted (total # in state)
Number of people served by the intervention
4,000 patients participating
10,000 patients targeted*
= 40% of patients served
*10,000 patients who are served by 40 FQHCs; of those
10,000, 4,000 were reached by the intervention.
Number of settings that adopt
3 Farmers markets established
4 farmers markets planned
= 75% of FMs established
Number of people served by FM
3,000 people shop at FM
= 50% shopped FM
6,000 people in surrounding census blocks
*6,000 shoppers who are served by 3 FMs; of those 6,000,
3,000 were reached by the intervention.
Impact of a PSE Strategy
Intervention
Implement
a statewide
tobacco tax
increase
PSE
Change
Higher
cost for
tobacco
products
Behavior
Change
•Reduction
in tobacco
use
•Reduction
in secondhand
smoke
Improved
Health
Outcomes
Distinctions from Health Promotion Model
• Addition of PSE change stage
• Timeline of intervention activities and effect
Reduction of
tobacco
related
morbidity and
mortality
Implications for Evaluation
Intervention
PSE
Changes
Process
Evaluation
Behavior
Change
Improved
Health
Outcomes
Outcome Evaluation
Short-term
Intermediate
Long-term
(Brega, 2012)
Intervention Plan Template
GOAL: By September 30, 2014, increase the number of school
districts with healthy food procurement policies from 0 to 10.
Direction
of change
Unit of
measure
ment
What will be Baseline
measured?
Increase
School
district
# of districts
with healthy
food
procurement
policies
0
Target
Time
frame
Data source
10
9/29/13
thru
9/30/14
School
Board/School
procurement
policy
• Identify exactly what you want to measure
• Think about data sources from the beginning
(Brega, 2012)
Evaluation Example
GOAL: By September 30, 2014, increase the number of schools
that have healthy food procurement policies from 0 to 10.
Activity 1: Form an advisory board
to develop model food procurement
standards.
Activity 2:Work with 20 school districts
to promote adoption of standards.
Process Measures
• Advisory board formed
• •Board meets regularly
• •Standards developed
• # meetings held with
school district
leadership
• # of school board
meetings when policy
was discussed
• Policy approved
Challenges of Evaluating PSE
Change
•
•
•
•
•
•
Have we actually implemented a PSE project?
Can we access data?
Is the data right?
What is the source?
Was reach calculated correctly?
PSE can take time – what can we learn as
change is happening?
Let’s try it!
• Move into groups of five
• Decide what you would like to see change at
a policy, system, or environmental level
• Use the intervention diagram to plot it out
• Complete how you would evaluate it
THINK BIG! BE A DREAMER!