
Define evaluation & different types of evaluation

Know how to use different methods to evaluate
programs, policies or strategies

Match your evaluation methods with your
objectives
CPCRN National Training Group, 2012
Evaluation is the systematic application of
social science research methods for assessing the:





conceptualization
design
implementation
impact
efficiency
of health related policies or programs.
Adapted from: Rossi and Freeman, 2004
3
Conceptualization
 What is the nature and scope of the problem?
Design
 What interventions are likely to alleviate the problem?
Implementation
 Is the program/policy being implemented as planned?
Impact
 Was the program/policy effective?
 Cause-effect relationship?
Cost Effectiveness (Efficiency)
 Was the program/policy cost effective?

Why evaluate?

When should you start thinking about evaluation?

Who will be interested &/or involved in the evaluation?

What are you evaluating?

How do you go about conducting the evaluation?
 Determine whether objectives were met
o
When findings indicate that objectives were not met, explore/address reasons
•
o
Document program strengths and weaknesses
When findings indicate that objectives were met, celebrate!
 Contribute to the evidence base
◦ Create practice-based evidence
 Communicate findings to staff, partners and funders
◦ Promote good public relations & raise awareness of critical issues
◦ Meet grant or contract requirements
 Identify issues/groups to focus on in future programs
◦ Determine the extent to which a program or its components are appropriate
for other populations or settings
Adapted from Windsor et al., Evaluation of health promotion,
health education, and disease prevention programs, 2003

From the very beginning
◦ Considered throughout all phases of program
planning & implementation

Begins with your community assessment
◦ Identify specific community needs
◦ Provides baseline data
 Collected prior to program implementation
 Allows you to measure the effect that your
program has on your audience
Adapted from Boyle et al. National Cancer Institute. Using What
Works: Adapting Evidence-Based Programs. Available at:
http://cancercontrol.cancer.gov/use_what_works/start.htm
A “practice run” using a small group
that is similar to your target audience
Conducted after the community assessment
and prior to program implementation




Determine whether adaptations are appropriate
Examine issues related to feasibility
Identify need for further/additional adaptation(s)
Predict how effective your program is likely to be
Adapted from Boyle et al. National Cancer Institute. Using What Works:
Adapting Evidence-Based Programs. Available at:
http://cancercontrol.cancer.gov/use_what_works/start.htm

Who are the stakeholders for your program/s?

How can you best obtain their buy-in &/or engage them
in the evaluation?
◦ Why would/should they value your evaluation activities?
◦ What questions and outcomes matter most to them?
◦ What procedures are favored?
◦ Obtain permission up-front
CPCRN National Training Group, 2012
In designing an evaluation plan, aim to find balance:
A) Which outcomes are most important?
B) What evaluation methods are most feasible?
 What program or policy do you want to evaluate?
 For what population?
 Delivered or implemented by whom?
 Under what conditions?
 For what outcome?
 How did it come about?
Paul G, as cited in Glanz et al. (eds) Health Behavior and Health
Education (pp. 531-2). 2002: Jossey-Bass; San Francisco, CA.
You will NEED pre-defined, clear,
measurable indicators that include
a focus on:
 Delivery, Participation/Uptake
 Health behavior change
 Policy or practice change
 Health change
 Objectives
 Process
 Short-term
outcomes
 Long-term
outcomes

Process (Formative)
◦ Assessing the nature and scope of problem
◦ Documenting the implementation of policy or program
 Are activities being implemented as planned?
 Is the program reaching the intended audience?
◦ Important for assessing WHY program did/didn’t work

Outcome (Impact/Summative)
◦ Assessing impact/effect of policy or program
◦ Did policy or program achieve intended outcomes?
 Important to quickly identify implementation issues
 Includes measurements of:
◦
Adoption
◦
Reach
◦
Participation
◦
Exposure
◦
Implementation
◦
Satisfaction
◦
Barriers & Facilitators
◦
Maintenance
 Includes assessment of: staff, budget, overall program
Adapted from Windsor et al., Evaluation of health promotion,
health education, and disease prevention programs, 2003.
Data collection methods may include:
◦
◦
◦
◦
◦
◦
◦
Databases
Administrative Data
Registration Sheets
Observation & Tracking Sheets
Program Implementation Logs
Surveys
Interviews
Adapted from Windsor et al., Evaluation of health promotion,
health education, and disease prevention programs, 2003.

Short-term outcome evaluation
◦ Measures progress toward meeting long-term objective
• Do attitudes, awareness, knowledge, setting,
environment &/or behaviors change?
• Is a policy documented? Is it enforced?

Long-term outcome evaluation
◦ Does health improve?
◦ Do morbidity and mortality decrease?
 Consider using prior evaluation tools and methods
◦ What outcomes were measured?
◦ How rigorous was the original evaluation?
◦ Do you have resources to do a similar evaluation?
 How much have you adapted?
◦ The more adaptation, greater need for rigor in evaluation
◦ With few changes, focus can be on process & short-term outcomes
Body & Soul
◦ Church-Based Program
◦ Designed to increase fruit and vegetable consumption
◦ Developed for use in African American Churches

Church-wide nutrition activities
◦ Kick-off event
◦ Form project committee
◦ At least four nutrition-focused events
 One involving the pastor
◦ At least one policy change
 E.g., establishing guidelines for types of food served at church functions
or changing snacks served at youth camps

Motivational counseling calls to church members

Additional church-wide activities
◦ At least 2 per member
◦ Made by lay volunteers
◦ Host guest speakers, cooking demonstrations, and taste tests
◦ Provide self-help materials
(e.g., cookbook, video, educational pamphlets)
What type of process evaluation data would you
want to collect?

Number of participants

Number of nutrition events completed

Number of lay counselor nutrition sessions completed

Were the core elements of the program implemented?

When?

In what order?
20
What type of short-term outcome evaluation data
would you want to collect?



Did fruit and vegetable consumption increase?
Did awareness of the nutritional value of fruits and
vegetables increase?
Compliance with new policy to have healthy foods at
church events
Planning
•Consult
with the program stakeholders or resources
•Develop the questions
•Make a timeline
Data
Collection
•Pilot
test
•How will the questions be asked?
•Who will ask them?
Data
Analysis
•Who
Reporting
•Who
will analyze the data and how?
will report and how?
•Who will receive the data and when?
•How will it affect the program?
Application •How could your results be applied in other places?
CPCRN National Training Group, 2012
EVALUATION WORK PLAN
SMART
Objective
Question/s
Data /
Data Source
Data Collection
Method
Result Sharing
Plan
1.
1.
3.
4.
5.
23
BODY & SOUL EVALUATION WORK PLAN
SMART
Question/s
Data /
Data Source
Objective
1. Engage 30% of
parishioners in Body &
Soul program activities by
the end of the program
period.
How many
parishioners
participated?
(Process)
2. a. Form project committee
of 6-8 church members by
April, 2015.
Did you implement
all six core
elements?
Likely would want
one question per
element
(Process)
•
•
•
•
•
•
What were reactions
to the program?
(Process/ Shortterm outcome)
• Satisfaction,
b. Host kick-off event
following Sunday service
by June, 2014.
c. Etc.
3. 80% of participants think
that the program is:
•
Informative
•
Appropriate
•
Feasible
• Impactful on their habits
4. Participants in Body & Soul
consume more 2 fruits and
vegetables per day
during/after B&S than
before.
5. Two significant foodrelated alterations to
church policy are adopted
midway through the
program.
What changes
occurred in eating of
fruits and
vegetables?
What changes
occurred in church
policies for healthy
eating?
• Number
of
participants by
session
Project Committee
Kick-off event
3 church-wide events
1 event with pastor
1 food habit change
2 motivational
counseling call rounds
• Attitudes
about
program
•#
of fruits &
vegetables eaten
• Policies
(e.g.,
healthy offerings at
dinners)
Data Collection
Method
• Session
Result
Sharing Plan
sign-in
sheet
• Activity
Log of
events in each core
element (date,
event, description,
participants, etc.)
• Participant
survey
(during the
program)
• Participant
survey
(after the program)
• Review
policy
of church
Provide summary to Pastor
& Project Committee;
Share w research team;
Include in final report to
funders;
Share with churches that
hope to enroll in next
session of the program

Organizational policy
◦ Regulate types of foods served at meetings and events
◦ Provide 30 minutes of paid time daily for physical
activity

Public policy
◦ Prohibit smoking in all bars and restaurants
◦ Allocate funding for farm-to-school initiatives

Document policy process
◦
◦
◦
◦

Number of entities that adopt
Implementation of policy as intended
Reactions to policy
Changes in policy over time
Document short-term outcomes
◦ Enforcement of the policy
◦ Environmental change (improved air quality)
◦ Health behavior change (reduction of smoking)

Document long-term outcomes
◦ Changes in health outcomes
SMOKE-FREE POLICY EVALUATION WORK PLAN
SMART
Data Collection
Method
Data /
Data Source
Question/s
Result Sharing
Plan
Objective
1. Educational materials
about smoke-free policies
are created and
distributed to 80% of
businesses in catchment
area by 2 months by
December, 2014.
What materials were
created for this policy
formation?
How many
organization were
targeted?
2. 75% of affected
businesses identified in
Obj. 1 have been trained
in implementing smokefree policy by March,
2015.
3. 65% of businesses in
catchment area adopt
smoke-free policy by July,
2015..
4. Smoking rates in the
catchment area are 20%
lower 2 years post-policy
enactment.
5. Lung cancer rates are
5% lower 20 years postpolicy enactment.
•
•
•
How many businesses
were trained?
Which representatives
of those businesses
attended trainings?
How many businesses
adopted a smoke-free
rule?
•
•
Materials created
Examine materials
for content
•
•
List of
organizations
targeted
•
Sign-in sheets from
trainings
•
Log sheets of
materials
distribution to sites
Sign-in sheets from
trainings
Send materials & list
of sites to policy
stakeholders
 Evaluation
findings need to reach all
stakeholders
◦ In language that everyone understands
◦ In time to make decisions
 Program
maintenance v. adaptation based
on evaluation findings
 Broaden
dissemination and application
Fidelity-Adaption Continuum
HIGH
FIDELITY
MINOR
ADAPTATION
E
V
A
L
U
A
T
I
O
N
NEED
LOW
FIDELITY
MAJOR
ADAPTATION
Image retrieved from
http://letstalkaboutscience.files.wordpress.com/2014/02/colorful_question_mark.jpg
30
31
Evaluation is the systematic application of
social science research methods for assessing the:





conceptualization
design
implementation
impact
efficiency
of health related policies or programs.
Adapted from: Rossi and Freeman, 2004
32
From Merriam Webster dictionary…
eval·u·ate transitive verb \i-ˈval-yə-ˌwāt, -yü-ˌāt\
Definition of EVALUATE
1 : to determine or fix the value of
2: to determine the significance, worth, or condition of
something usually by careful appraisal and study
— eval·u·a·tion noun
Campbell & Stanley (1963): Experimental and QuasiExperimental Designs for Research

Premise: evaluations should adhere to highest
standards of research design
◦ Pre-test post test control group design = gold
standard

Internal validity prioritized over all other factors

Reaction to “failure” of field of education to adopt
rigorous scientific methods since 30s


Cronbach (of internal consistency fame)
published an evaluation text in 1982
◦ Evaluation is an art
Cronbach argued that “relevance” should top
internal validity in evaluation-- reaction to
Campbell & Stanley
“No single best plan for an evaluation, not even for an inquiry
into a particular program, at a particular time, with a
particular budget”
“For any evaluation many good designs can be proposed, but
no perfect ones.”

Evaluation is a systematic determination of a subject's
merit, worth, and significance, using criteria governed by a
set of standards.

The primary purpose of evaluation, in addition to gaining insight
into prior or existing initiatives, is to enable reflection and
assist in the identification of future changes.

Evaluation is often used to characterize and appraise subjects of
interest in a wide range of human enterprises, including the
arts, criminal justice, foundations, non-profit organizations,
government, health care, and other human services.
- Wikipedia, 2013
Evaluation is an ART in many ways:
• learn from experience, gut feeling, etc.
Unintended effects:
• must be prepared to detect and measure
Political and Social Climate:
• can seriously influence success or failure of
evaluation
Cost of evaluation:
• must be assessed before deciding to undertake

Examples:
◦ Low completion rate for brief feedback questionnaire in
clinic-based study
◦ Care seeking behaviors of Korean participants
Rand retracts report on pot clinics and crime
Think tank says researchers failed to realize that data used in the study did not
include LAPD statistics. It plans to recalculate its analysis.
October 24, 2011 |By John Hoeffel, Los Angeles Times
The researchers reviewed crime reports from the 10 days before the city's medical
marijuana ordinance took effect on June 7, 2010, with the 10 days after, when at
least some of the more than 400 illegal dispensaries shut down.
They found a 59% increase in crime within 0.3 of a mile of a closed
dispensary compared to an open one.
Usher and Greenberg criticized the researchers for not relying on LAPD crime
data, but also questioned the short time frame and the assumption that
dispensaries actually closed.
The Rand researchers relied on data posted by CrimeReports.com, which they
mistakenly believed included LAPD data. Knopman said Rand was not blaming
the website. She said Rand reviewers, digging deeply into the data, only recently
discovered that it did not include LAPD reports.
Also from Wikipedia…

In the social sciences, unintended effects (a.k.a.
unintended consequences, unanticipated
consequences ) are outcomes that are not the ones
intended by a purposeful action.

The concept has long existed but was named and
popularized in the 20th century by American sociologist,
Robert K. Merton.
See also
The road to hell is paved with good intentions
Three types:
 A positive, unexpected benefit (luck, serendipity, windfall).

A negative, unexpected detriment occurring in addition to the
desired effect of the policy (e.g., while irrigation schemes
provide people with water for agriculture, they can increase
waterborne diseases that have devastating health effects,

A perverse effect contrary to what was originally intended
(when an intended solution makes a problem worse)
Examples:
 Expanding hours in school-based vaccination clinic

Opening up positions to improve unemployment rates in
community
Association of Health Care Journalists
Program for disabled kids has unintended effects
BY PIA CHRISTENSEN | JAN. 21, 2011

Patricia Wen, of The Boston Globe, examines the Supplemental
Security Income (SSI) program for children and finds the program as
“a fast-growing alternative welfare system.”

Wen found that the program (intended to serve children with severe
physical disabilities) mainly serves children with behavioral, learning
and mental conditions & carries financial incentives to put children on
psychotropic drugs.

Another, unintended, effect of the program is that teenagers on the
program are declining to take part-time jobs despite their desire to
start careers and not be dependent on the government. Wen explains
“they fear working will jeopardize their disability checks. Their families
are poor and need the money.
A few more quotes from Cronbach….
“When he can afford to choose, the social
scientist ought to undertake an evaluation
only if it appears that the political system
will give a serious hearing to all findings,
welcomed and unwelcomed”
“The political world is not often in the market
for the whole truth”

A program initiated for social reasons may not be
influenced by negative evaluation

Programs under the most scrutiny is also political
issue/decision

Some programs are never evaluated
◦ Political pressures would prevent major changes
(aspects of Medicare, FDA regulations)
◦ Doing something may pacify constituents; don’t want to
know if “it works”
Los Angeles teachers' evaluation victory
bucks a trend
Los Angeles teacher appraisals won't be based on 'value added,'
increasingly being used across the country. Now a key question is how
test scores will figure in.
December 09, 2012|By Teresa Watanabe and Howard Blume, Los Angeles Times
NPR-Frontline
Despite Show of Support, Federal
Funding Ban on Needle Exchange
Unlikely to Be Lifted Anytime Soon
Study after study has shown that needle-exchange programs
do not increase drug use, and the American Medical
Association, the Federal Centers for Disease Control and
Prevention, the National Institutes of Health, the General
Accounting Office and the National Academy of Sciences have
all said the programs work to help reduce the spread of HIV.
Federal scientists can again research gun
violence
By Joe Davidson,January 17, 2013
George Bush's sex education failure
Teenage pregnancies and STDs in the US are on the rise again,
thanks to George Bush's abstinence-only obsession
Truth or DARE: 20 Years of Great Press, But Statistical Failure
1-C National Story
Qualitative Study Revealed Reasons for Continued Interest in D.A.R.E.:
School officials are skeptical that any low-input short-term program like D.A.R.E. can
change adolescents’ drug-taking behavior.
Improvement in relationships between police and children is a main reason for many
districts’ continued implementation of D.A.R.E. (although usually not assessed)

Cost is an increasingly important aspect of
evaluation
◦ Cost of evaluation itself not just cost-effectiveness

Rigorous evaluations are expensive

Evaluation needs to be built into the initial
design & budget of every intervention program

Funding cuts are common when implementing
programs and evaluation often suffers
School districts' costs for teacher evaluations outstrip
funding
By Paul Riede | priede@syracuse.com
on March 07, 2013 at 12:16 PM
School districts across New York are spending far more on new teacher
and principal evaluation systems than they are getting in federal money
Teacher evaluation systems proving costly
November 22, 2012 By JO NAPOLITANO jo.napolitano@newsday.com
The newly mandated teacher and principal evaluation system is costing
Long Island school districts tens of thousands of dollars per year in training,
testing and materials, even as they struggle with effects of the property-tax
cap and putting in place other required education reforms.
Program Design: Logic Model
What is a Logic Model?
A logic model is a graphic illustration of an intervention (e.g.
program, campaign, etc.) and how its various components
are linked to one another sequentially, logically and
realistically. There are also different formats for logic models.
Some can be very broad and simple; others are more detailed and
complicated.
Source: University of Wisconsin - Extension
Components of a Logic Model
People use different words to describe the components of a logic model. For
the sake of clarity, these are the terms that we will work with today to refer to
the main components of a logic model:
•
Inputs
•
Outputs
•
Outcomes and Impact
INPUTS
OUTPUTS
Also Known Resources
Activities /
As…
 Tangible (staff,
Objectives
funding, equipment,
space, etc.)
 Intangible (experience,
relationships, etc.)
OUTCOMES
Impact/Goals
 Short-term
 Medium-term/
Intermediate
 Long-term
Logic Model: Inputs
• What do you need or already have to carry out these
activities? What potential barriers can you see and what do
you need to address those barriers?
• Funding
• Staff and volunteers
• Past experience with after-school programs
• Existing relationships with teachers and parents
• Access to school facilities
• Curricula on math and language skills
• Strengthened relationship with school district as program
partner
Logic Model: Outputs
OUTPUTS:
• Program activities, with units of service
• Target Population
How they relate to
Outcomes:
Frequency and
intensity of program
activities needed to
achieve outcomes
(Dosage)
How they relate to
Inputs:
What, and how many,
resources would you
need to support your
program activities?
Logic Model: Outcomes
Changes in your target population that your program is intended to bring
about. All outcomes, starting with long-term outcomes, should be
aligned with the mission of the organization and/or priorities in
strategic plan.
ShortTerm
•
•
•
•
Awareness
Attitude/Belief
Knowledge
Skills
• Increase in awareness
about options for
mammograms
• Increase in skills to do
self-exams
• Improved attitude
about one’s power over
her own health
LongTerm
Intermediate
•
•
•
Health Behavior
Policy Changes
Community Norms
•
Increased # of self•
exams and/or
mammograms
Policies adopted to
remove barriers for
women to health
insurance
Reduced stigma about
breast cancer
•
•
•
Health Status or
Condition (Individual
or Community)
Women survive
breast cancer, stay
healthy, and have
high quality of life
Connecting Model Components
Conventional logic model building starts with the end and plans
backward.
• What is your ultimate goal or long-term impact?
• What are the changes in behavior or condition in your target
population that will lead to the long term impact?
• What are the changes in attitude, knowledge, or perception that
you want your target population to show that will lead to the
changes in behavior or condition?
• What are the indications that these changes have occurred?
• What are the program activities that would lead to these changes?
• Refine your target population.
• What do you need or already have to carry out these activities?
Sample Logic Model
Inputs
Outputs

Funding

Staff
Target Population

Volunteers

Past experience with
after-school
programs
High school freshmen and
sophomores with an average of
GPA between 2.0 – 3.0 from
low-income families




Existing
relationships with
teachers and parents
Access to school
facilities
Curricula on math
and language skills
Strengthened
relationship with
school district as
program partner

Program Activities
(Units of services)
 Weekly math classes
 Weekly lang. skills classes
 Monthly skill-building
workshops
Outcomes
(Short - and IntermediateTerm)
 Improve study skills
 Improve academic
performance
 Decrease law enforcement
involvement
 Increase motivation for
learning
 Improve math and
language skills
(knowledge)
 Improve aspiration for
higher ed.
 Bi-monthly academic
counseling
Impact (Long-Term)
 Quarterly field trips to local
colleges
Youth have access to
higher education.
Types of Evaluation
Or, How Do You Define Success?
Process
Evaluation
How does it define
success?
How well is program being
implemented?
What does it measure?




Common evaluation
tools/methods
Participant profile (e.g.
demographics, etc.)
Program flow (e.g. new
participants, dropouts, etc.)
Service Usage (e.g. # of times
service is being provided, # of
participants used services, etc.)
Participant Satisfaction
Client intake/enrollment forms
Sign-in sheets
Outreach logs
Satisfaction surveys
Outcome
Evaluation
How well do participants achieve
changes you intended?
Depending on outcomes,
improvement in:

Knowledge

Skills

Attitude

Perceptions

Behavior

Status/Conditions
Pre-/post- or retrospective
surveys
Standardized assessments
Focus groups or interviews
Observation checklists
Needed in conjunction with ALL evaluation designs
Lacking an appropriate process evaluation may lead to:

Rejecting an effective program

Accepting a program that is inadvertently different
than the one that you had planned to implement
60
FEASIBILITY
What’s possible for our organization?
• Cost of evaluation
• Capacity of organization to evaluate programs
• Acceptability of evaluation design to stakeholders
How confident are we that we know whether
or not a program is effective?
• Many different evaluation designs (roadmaps) to
choose from
• Ideally, use most rigorous evaluation design
SCIENTIFIC
RIGOR
61
Promoting Sun Safety at Public Pools
Target Populations: Children ages 5-10 in swim lessons
Desired Outcome: Adherence to recommended sun protection
practices (sun screen, hats, shade seeking, long sleeves, sunglasses)
Core components:
Content: skin cancer info.; sun protection info.
Activities: staff training, sun safety educational activities for kids,
environmental changes (shade structures, sunscreen availability)
Study design: Implemented the intervention in three pools,
measured sun protection practices before and after program via
parent surveys and on-site observations
62
INTERNAL VALIDITY:
How confident you are that the outcomes you
achieved are due to the program?
Did sun protective behaviors among children
and staff at the pool(s) increase because of the
sun safety intervention?
63
EXTERNAL VALIDITY:
How confident are you that the outcomes from
the program in one particular setting will be
achievable in other settings?

“Generalizability”

How sure are that the intervention will be effective
in other populations of children?
64
Threats to INTERNAL validity:
 Events, scenarios, and situations which reduce
confidence that outcomes are due to the program
and not some other cause
Threats to EXTERNAL validity:
 Events, scenarios, and situations that reduce
confidence that program outcomes will be
achievable in other settings
65

Threats to validity can be controlled for using
a scientifically rigorous evaluation design

Importance of rigorous evaluation:
◦ Understand whether or not your program works
 Deliver the most effective services possible to your clients
 Use your organization’s resources effectively
 Be competitive in grant applications
◦ Understand if it would work for other settings
 Contribute to evidence base of community programs
66
Events (other than those planned) that happened
during the program
 During program implementation, did:


 a new pool manager take over?
 a celebrity die or suffer negative effects of skin cancer? become a
spokesperson for sunscreen? (a Hugh Jackman effect…)
 a new public awareness campaign about skin cancer &/or sunsafety start up?
If one group design, “history” is a threat to internal validity
If have comparison group, “history” could be external validity
threat (if effect of program depends on historical context)
67
Changes that program participants experienced
due to the passage of time

Does not directly apply to our sun safety program
Better examples…
◦ Age-related gains in reading ability in the context of
skills-based reading program for kinders
◦ Age-related reduction in body fat among 3 year olds enrolled
in obesity control intervention


In one group study, “maturation” is a threat to internal validity
If have a comparison group, threat is controlled
68
Baseline assessment (Pre-Test)
Did the baseline assessment….
 Inadvertently educate parents of participating children about the
detriments of too much sun exposure?
 Get participants interested in sun protection?
 Remind participants of various ways that you can protect
yourself/pool users from sun?
 Change the way participants reacted to the intervention?
• If no comparison group (one-group design), “testing” is
a threat to internal validity
• If have a comparison group, “testing” could be an external
validity threat (if program effect depends on having test)
69
Change in instrumentation/measurement
Did something change in our measurement from
baseline to follow-up?
 Research staff stopped adhering to observation protocol
 Different staff were hired and trained differently
 Different parent completed follow-up survey compared to
baseline (ex. mom at baseline, dad at follow-up)
• If no comparison group (one-group design),
“instrumentation” is a threat to internal validity
• If have a comparison group, should be controlled for as
long as both study groups use same protocols
70
Uneven drop out of the program (only select
group of those who start complete follow-up)



Were kids who skipped last week of swim lessons…
 Starting public school?
 Leaving on Hawaiian vacations?
 Of a different race or ethnic group?
 Have parents with higher or lower incomes?
 More or less interested in sun protection?
If one group design, drop out (a.k.a. “study mortality”) is an
internal validity threat
If have comparison group, need to be concerned about
whether drop out is similar between two groups
71
Pre-existing differences between the groups
(intervention vs. control)
Note: primarily a concern for two-group design, so LET’S ASSUME…
2 pools get the program & 1 pool serves as comparison….
 Were the children at pools that received the program different than
children at comparison pool?
 May differ on SES
 Race/ethnicity
 These pre-existing differences may lead to different outcomes
• “Selection” can also be a threat to external validity,
participants in both program and comparison pools are different
than rest of the population
72

Events, scenarios, and situations that reduce confidence
that program outcomes will be achievable in other settings

Anything that makes the setting in which you originally
evaluate a program different vs. rest of world
◦ Presence of a influential baseline survey
◦ Artificiality of “research setting”: outside observers,
special protocols, timelines, incentives, and general hoopla
◦ Highly select population and setting being studied: (often
evaluate Cadillac programs under best conditions)
73
Control Group:
◦ A group of people as similar as possible to the treatment group
◦ Do not receive the program/intervention

Greatly enhances quality of program’s evidence

Not always feasible to use control group
◦ Organizational resources
◦ Acceptability to stakeholders
74

For some evaluation designs, the intervention group
can ALSO serve as their own control group

Delay giving the intervention to the control group
◦ For example: Intervention group receives from Jan – June
and control group receives from July-Dec

Alternative intervention to the control group
In practice, normally provide both groups with both
programs (switch interventions after evaluation complete)
◦ May be a good solution if already have another program
developed and it is relatively easy / low-cost to implement
75

In the most “rigorous” evaluation designs,
participants are assigned to the intervention or
control by a process similar to flipping a coin
◦ Randomization or Random Assignment

Randomization is not always possible
◦ Important that the control group is as similar as
possible to the group that receives the program (the
intervention group)

Find a similar group of people—another clinic, church,
neighborhood, school, etc. to use as comparison group
76

Commonly used in experimental research
Intervention Group:
Control Group:
Pre-Test
Pre-Test
X
Ø
Post-Test
Post-Test
◦ Participants randomly assigned to group A or B
KEY
X = Intervention
Ø = No intervention
77

When individual randomization is not feasible
Intervention Group:
Control Group:
Pre-Test
Pre-Test
X
Ø
Post-Test
Post-Test
◦ Participants are in pre-existing groups
(e.g., churches, schools, clinics,
neighborhoods, etc) and are NOT
randomly assigned
KEY
X = Intervention
Ø = No intervention
78

When Pre-Test Data is not available
Intervention Group:
Control Group :


X
Ø
Post-Test
Post-Test
Benefits of this design?
Limitations of this design?
KEY
X = Intervention
Ø = No intervention
79

Commonly used in public health practice,
although does not control for most threats
Pre-Test


X
Post-Test
KEY
X = Intervention
Benefits of this design?
Limitations of this design?
80
81

Control Group is not available

Multiple pre- and post-tests on one group
Pre-Test1
Ø Pre-Test2
X
Post-Test3
Ø
Post-Test4
Note: There may be any number of Pre- and Post- Tests


Benefits of this design?
Limitations of this design?
KEY
X = Intervention
Ø = No intervention
82
Check-In
• What are issues that you need additional
clarification on?
• What are some remaining questions that we didn’t
discuss?
• What are some obstacles and challenges that you
see in including evaluation into your work?
• What kind of help would you anticipate?
• What would be the next steps for your program or
organization, if you want to move forward with
integrating evaluation into your work and using
evaluation data optimally?
84

Read one of two peer-reviewed journal articles
(Swartz 2005 or Simons-Morton 1991)

With a partner or alone, respond to 8 questions on
the worksheet

Share your answers with the larger group
85

Define evaluation & different types of evaluation

Know how to use different methods to evaluate
programs, policies or strategies

Match your evaluation methods with your
objectives
CPCRN National Training Group, 2012
Image retrieved from
http://letstalkaboutscience.files.wordpress.com/2014/02/colorful_question_mark.jpg
87
88








Community assessment
Objectives
Core program elements
Adaptable components
Evaluation design
Process measures
Outcome measures
Cost effectiveness
89

Read one of the Nat’l Association of County & City
Health Officials (NACCHO) ‘promising’ practices
descriptions or the ‘CLEAR’ program from CDC
(one of best practices for HIV/AIDS)

With a partner/small group, respond to the 8
questions on the handout and fill out the logic
model if it is useful to your process.
90
Image retrieved from
http://letstalkaboutscience.files.wordpress.com/2014/02/colorful_question_mark.jpg
91

Evidence-based training course materials
◦ Events>Training Materials

Contact information

Center news & information
http://healthequity.ucla.edu/
CPCRN National Training Group, 2012