Evidence-Informed Program
Improvement:
Using principles of effectiveness to enhance the quality
and impact of youth and family programs
STEPHEN SMALL
Professor of Human Development & Family Studies,
University of Wisconsin-Madison
and
Extension Human Development & Family Relations Specialist
University of Wisconsin-Extension
Iowa State Webinar
November 17, 2009
Agenda
I.
II.
III.
What are evidence-based programs, practices, and
organizations?
What are some common principles of effective
evidence-based youth and family programs?
How can Evidence-Based Program Improvement
(EIPI) be used to improve existing programs?
Evidence-based
Programs and Practices
Years of research have demonstrated that specific approaches,
practices and strategies can reduce problem behaviors and
enhance positive developmental outcomes
The most effective programs and practices are termed
“evidence-based” – these are the “gold standard”
Why the growing interest in evidencebased programs and practices?
• Accountability: Increasingly required by funders within
•
•
•
•
•
political climate
Efficiency: Reduces development costs associated with creating
a new program
Availability of good science: Body of scientific evidence has
reached a critical mass
Cost-effectiveness: Increased likelihood that limited resources
will be used wisely
Marketability: Documented evidence can help sell and justify
program
Ethics: Responsibility to use what is known to be effective
The Evidence-Based Programs and
Practices Continuum
Evidence-based Programs
Evidence-based Kernels
and core elements
Evidence-based Practices
Evidence-based Principles
Evidence-based Organizational
Systems
Evidence-Based Programs
A new class of programs that:
• Are based on a sound scientific evidence and theory
• Have been carefully implemented, evaluated and replicated
using rigorous scientific methods
• Have evaluation findings that have been subjected to critical
review
• Often “endorsed” as evidence-based by a federal agencies or
respected research organizations (e.g., Campbell
Collaboration, SAMHSA, CDC)
Terminology
Evidence-based  Research-based
Evidence-based = Research-based + Rigorously tested
Examples of Evidence-based
Programs
Family/Parent Education Programs
 Nurse Family Partnership: 15-year follow-up showed ↓
arrests (54%) among adolescents; $3.59 return on the
dollar
 Strengthening Families Program for Parents and Youth 1014: ↓ aggression, alcohol, tobacco, marijuana, meth; $7.82
return
Examples of Evidence-based
Programs
School-based programs
 Life Skills Training: 6-year follow-up showed ↓ alcohol,
tobacco, marijuana, narcotics, hallucinogens; $25.61 return
on the dollar
 Olweus Bullying Prevention Program: ↓ in bullying and
victimization, vandalism, fighting, truancy; ↑ in school
attachment
Examples of Evidence-based
Programs
Mentoring
 Big Brothers Big Sisters: ↓ in violence, alcohol and drug use;
↑ in school performance; $1.01 return
The effect of the Strengthening Families 10-14
Program on teen aggressive and hostile behaviors
1.4
1.2
1
Control group
0.8
Strengthening
Families Program
0.6
0.4
0.2
0
6th
grade
pretest
6th
grade
posttest
7th
grade
8th
grade
10th
grade
SOURCE: Spoth, R., Redmond, C., & Shin, C. (2000) Reducing adolescents' aggressive and hostile behaviors: Randomized
trial effects of a brief family intervention four years past baseline. Archives of Pediatrics and Adolescent Medicine 154,
1248-1257
What are some limitations of
evidence-based programs?
 Not always easy to find
 Can be costly to implement in both time & $
 May not address particular issues or audiences
 Tend to be problem focused
 May downplay local knowledge and community ownership
 Overlooks value of promising grassroots programs
 Often difficult to transport to new settings
 Other?
The Evidence-Based Programs and
Practices Continuum
Evidence-based Programs
Evidence-based Kernels
and core elements
Evidence-based Practices
Evidence-based Principles
Evidence-based Organizational
Systems
Evidence-Based Kernels
• Kernels = units of behavioral influence and practice that
have been shown through experimental evaluation to
produce reliable effects on behavior
• Kernels = Active ingredients of effective programs
The Evidence-Based Programs and
Practices Continuum
Evidence-based Programs
Evidence-based Kernels
Evidence-based Practices
Evidence-based Principles
Evidence-based Organizational
Systems
Definitions of
Evidence-based practices
 “The integration of the best available research with
clinical expertise in the context of patient characteristics,
culture and preferences.” (American Psychological Association)
 “The integration of best-researched evidence and clinical
expertise with patient values.” (Institute of Medicine of the
National Academies)
Evidence-Based Practice
Practitioner
Expertise & Experience
EvidenceBased
Practices
The Evidence-Based Programs and
Practices Continuum
Evidence-based Programs
Evidence-based Kernels
Evidence-based Practices
Evidence-based Principles
Evidence-based Organizational
Systems
EVIDENCE-INFORMED PROGRAM IMPROVEMENT:
Using principles of effectiveness to enhance the quality
and impact of family-based prevention programs
February 2009, Family Relations, 58, 1-13.
http://whatworks.uwex.edu
Principles of evidence-based youth and
family programs
1. Design & Content
2. Relevance
3. Delivery/Implementation
4. Assessment & Quality Assurance
Program design and content
Effective programs…
 Target relevant assets and/or risk and protective factors
 Are theory-driven
 Are of sufficient dosage and intensity
 Are comprehensive
 Use active learning approaches & appropriate delivery
formats
Theory-driven
 Based on strong, scientific theory
 Target relevant risk, protective factors
and assets to accomplish objectives
 Logical, well-developed program theory
A well thought out recipe
is essential
Theory driven - Example
 Nurse-Family Partnership (early childhood/parent ed) -
grounded in theories of human ecology (Bronfenbrenner,
1979, 1992), self-efficacy (Bandura, 1977, 1982), and
human attachment (Bowlby, 1969)
Theory-driven
 Based on strong, scientific theory
 Target relevant risk, protective factors
and assets to accomplish objectives
 Logical, well-developed program theory
A well thought out recipe
is essential
Have sufficient dosage and intensity
Enduring change
takes effort
Thinking about sufficient dosage
and intensity…
For a mentoring program to be effective, the relationship on
average has to exist for at least…
A. 2 weeks
1 month
C. 6 months
D. 9 months
E. 1 year
B.
Comprehensive
Simple solutions rarely work
Use active learning techniques and
appropriate delivery format
Program relevance
Effective programs…
 Are developmentally appropriate
 Are socio-culturally relevant
 Are appropriately timed
 Are sensitive to local context
 Are responsive to individuals
One size does not fit all
Program delivery
Effective programs…
•
•
•
•
Foster good relationships
Are delivered by well-trained and committed staff
Address environmental barriers to program delivery and
behavior change
-Provide a safe setting for the intervention
Are embedded in a supportive organization infrastructure
Well trained and
committed staff - Examples
 Big Brothers, Big Sisters (mentoring program) –
Orientation, volunteer screening, youth assessments,
matches, supervision
 Promoting Alternative Thinking Strategies (PATHS,
classroom program) - Teachers receive training in a two- to
three-day workshop and in bi-weekly meetings with the
curriculum consultant.
Program assessment and quality
assurance
Effective programs…
 Are well-documented
 Focus on evaluation and refinement
 Provide opportunities for ongoing feedback
Make sure the program is
well-documented
 It’s important to understand the program in order to
implement it well
 Maintaining program fidelity requires that people
know what the program is about
 Program replication and evaluation is difficult if the
program isn’t well documented
Focus on
evaluation & refinement
 See evaluation as your friend
 Evaluation takes many forms
 Evaluation isn’t always easy
 Consider EIPI as a first step
Evidence Informed Program Improvement
http://whatworks.uwex.edu/attachment/whatworks_manual.pdf
SUMMARY
Principles of Effective Youth and Family
Prevention Programs
 Program design and content
 Program relevance
 Program delivery
 Program assessment and quality assurance
The Evidence-Based Programs and
Practices Continuum
Evidence-based Programs
Evidence-based Kernels
Evidence-based Practices
Evidence-based Principles
Evidence-based Organizational
Systems
Evidence-based organization
• Promotes an organizational culture that values science and evidence-based
approaches
• Creates an infrastructure that supports effective practices and programs:
• Uses existing evidence-based programs when available and appropriate
• Employs a strong theoretical/science approach to new program design
• Provides regular opportunities to reflect on programs and effective practices
• Uses scientific tools to assess needs/assets, understand local context and adapt
programs
• Provides ongoing technical support for selecting, implementing and evaluating
effective programs and practices
• Builds organizational and community capacity to identify relevant goals and
implement appropriate and effective approaches to address these goals
Questions/Discussion