From Bench Science to Systems Change:
Reconciling Evidence-based Practice with
Practice-based Evidence
Brian Bumbarger & Stephanie Bradley
PRC Seminar
January, 2013
Investigators and Authors:
Brian Bumbarger
Mark Feinberg
Louis Brown
Michael Cleveland
Jennifer Sartorious
Brendan Gomez
Stephanie Bradley
Mark Greenberg
Brittany Rhoades
Wayne Osgood
Damon Jones
Julia Moore
Richard Puddy
Elizabeth Campbell
Collaborative Policy Innovators:
Mike Pennington
Clay Yeager
James Anderson
Keith Snyder
The EPISCenter and research described here are supported by grants from the Pennsylvania Commission on Crime
and Delinquency. Special thanks to the staff of the Office of Juvenile Justice and Delinquency Prevention (OJJDP)
2
We know a great deal about how youth problems develop,
and how to effectively prevent them (& reduce prevalence)
• Known risk & protective factors
• Multiple domains of influence (community,
family, school, peer, individual)
• Multifinality and equifinality
(a public health approach to public safety)
• Different trajectories (early vs. late starters)
• Criminogenic impact of intervention
From Lists to Improved Public Health…
• Synthesis and translation of research to practice,
(and practice to research)
• EBP dissemination, selection, and uptake
• Ensuring sufficient implementation quality and fidelity
• Understanding adaptation and preventing program drift
• Measuring and monitoring implementation and outcomes
• Policy, systems, and infrastructure barriers
• Coordination across multiple programs and developmentally
• Sustainability in the absence of a prevention infrastructure
Bumbarger, B. and Perkins, D. (2008). After Randomized Trials: Issues related to dissemination of evidence-based interventions. Journal of
Children’s Services,3(2), 53-61.
Bumbarger, B., Perkins, D., and Greenberg, M. (2009). Taking Effective Prevention to Scale. In B. Doll, W. Pfohl, & J. Yoon (Eds.)
Handbook of Youth Prevention Science. New York: Routledge.
Moving From Prevention Science . . .
Implement &
Evaluate
Programs
Develop &
Test
Interventions
Identify Risk
& Protective
Factors
Define the
Problem
Problem
Response
…To Prevention Service
Provide
Technical
Assistance
Set & Collect
Performance
Measures
Monitor Quality
of Program
Implementation
Assess
Public Health
Impact
Response
Pennsylvania’s EBP Dissemination Model
• Prevent dependency, delinquency, and ATOD use to the greatest
degree possible (primary prevention)
• Intervene effectively with youth for whom primary prevention is not
sufficient
• Allow communities flexibility to select strategies that best meet local
needs
• Create community-level infrastructure for strategic prevention
planning and coordination
• Provide accountability and use scarce resources efficiently
ULTIMATELY….
• To “move the needle” on key indicators of (behavioral)
health at the POPULATION level
9
Multi-Agency Steering Committee
(Justice, Welfare, Education, Health)
Intermediary and
State-level Prevention Support System
Support to
Community
Prevention
Coalitions
Support to
Evidence-based
Programs
Improve Quality of
Local Innovative
Programs and
Practices
A unique partnership between policymakers, researchers, and
communities to bring science to bear on issues of public health and
public safety
The EPISCenter is a project of the Prevention Research Center, College of Health and Human Development, Penn State University,
and is funded by the Pennsylvania Commission on Crime and Delinquency and the Pennsylvania Department of Public Welfare
as a component of the Resource Center for Evidence-Based Prevention and Intervention Programs and Practices.
The Menu of EBPs in This Project*
•
Olweus Bullying Prevention Program (OBPP)
•
Promoting Alternative Thinking Strategies (PATHS)
•
Big Brothers Big Sisters of America (BBBS)
•
Multidimensional Treatment Foster Care (MTFC)
•
Strengthening Families Program 10-14 (SFP)
•
Project Towards No Drug Abuse (Project TND)
•
Life Skills Training (LST)
•
Incredible Years (IYS)
•
Functional Family Therapy (FFT)
•
Multisystemic Therapy (MST)
•
Aggression Replacement Training (ART)
Multifinality:
Equifinality:
There are Poor
known
outcomes
risk
are
factors,
associated
each of which
with a iscombination
associated of risk
with a factors
range ofand
poor
developmental
outcomes pathways
Creating Fertile Ground for EBPs
Risk-focused Prevention Planning
(the Communities That Care model)
Form local coalition
of key stakeholders
Re-assess risk
and protective
factors
Collect local data on
risk and protective
factors
Leads to community
synergy and
focused resource allocation
Select and implement
evidence-based program that
targets those factors
Use data to
identify priorities
Pennsylvania’s CTC coalitions 2012
Multi-Agency Steering Committee
(Justice, Welfare, Education, Health)
Intermediary and
State-level Prevention Support System
Support to
Community
Prevention
Coalitions
Support to
Evidence-based
Programs
Creating “fertile
ground” for EBP
selection and adoption
Improve Quality of
Juvenile Justice
Programs and
Practices
Multi-Agency Steering Committee
(Justice, Welfare, Education, Health)
Intermediary and
State-level Prevention Support System
Support to
Community
Prevention
Coalitions
Support to
Evidence-based
Programs
Ensuring high-quality
implementation and
sustainability
Improve Quality of
Juvenile Justice
Programs and
Practices
Pennsylvania’s EBP
dissemination in 1999…
Pennsylvania’s EBP
dissemination in 2012…
Measuring Population-level Impact
• Cross-sectional quasi-experimental study of
98,000 students in 147 communities
• Used propensity score matching to minimize potential
selection bias
• Found youth in CTC communities reported lower rates of
risk factors, substance use, and delinquency than youth
in similar non-CTC communities (7x as many as by
chance)
• Communities using EBPs showed better outcomes on
twice as many R/P factors and behaviors (14x as many
as by chance)
5 year Longitudinal Study of PA Youth
% Change of CTC/EBP Youth Over
Comparison Group
419 age-grade cohorts over
a 5-year period:
youth in CTC communities
using EBPs had significantly
lower rates of delinquency,
greater resistance to
negative peer influence,
stronger school
engagement and better
academic achievement
40
33.2
30
20
16.4
10
0
-10.8
-10.8
-10
-20
Delinquency
Nega ve Peer Influence
Academic Performance
School Engagement
Feinberg, M.E., Greenberg, M.T., Osgood, W.O., Sartorius, J., Bontempo, D.E. (2010). Can Community Coalitions Have a
Population Level Impact on Adolescent Behavior Problems? CTC in Pennsylvania, Prevention Science.
Impact on Juvenile Court Placement Rates:
Comparison of Placement Rates for Counties* With and Without an EBI
11.00
10.50
10.53
10.00
10.05
9.89
9.50
9.76
No EBI
9.00
9.07
8.50
8.70
Adopted
EBI
8.00
7.50
7.79
7.78
2009
2010
7.00
2007
2008
Bumbarger, B. K., Moore, J., & Rhoades, B. (2010). Impact of evidence-based interventions on delinquency placement rates.
Presentation at 2011 Society for Prevention Research annual meeting. Washington, DC.
Justice Reinvestment Realized
24
Pennsylvania’s EBP
dissemination in 2012…
Programs/services can be placed along a continuum of
confidence based on their evidence or theory
*Bumbarger & Rhoades, 2012
Ineffective
“This program has been
evaluated and shown to have
no positive or negative effect”
Very
Confident
HARMFUL
Ineffective
Iatrogenic (Harmful)
“This program has been
rigorously evaluated and
shown to be harmful”
 Best Practices
 Research-based
“We’ve done it
and we like it”
unknown
“This program is based on sound
theory informed by research”
Promising
 Promising Approaches
“We really think this
will work… but we need
time to prove it”
EFFECTIVE
Very
Confident
 Evidence-based
“This program has
been rigorously evaluated
and shown to work”
How confident are we that this program or practice is a good use of resources
AND improves outcomes for children and families?
26
Multi-Agency Steering Committee
(Justice, Welfare, Education, Health)
Intermediary and
State-level Prevention Support System
Support to
Community
Prevention
Coalitions
Support to
Evidence-based
Programs
Improve Quality of
Juvenile Justice
Programs and
Practices
Developing “practicebased evidence”
A 2-pronged approach…
Promoting
Evidence-based Programs
Developing
Practice-based Evidence
• Theoretically-based
• Not an EBP for every
community
need/context
• Demonstrated effects
(confidence)
• Sponsored lists
– E.g., Blueprints
• Funding requirements
• Many
programs/services
already in widespread
use
• Some of those may be
effective
• Local expertise/fit
Background
• More than one way to establish “evidence-based practice”:
1. Model program implementation
2. Evaluation of local program
3. Meta-analysis of research on program “types”
Background
• Generic intervention types abundant in juvenile justice
• Research shows some types effective at reducing recidivism
• Main disadvantage of generic programs is they…
“lack the specificity that comes with the protocol for a
model program and the associated training and support
systems that are also often available from the developer”
(p. 3).
Howell, J.C., & Lipsey, M. W. (2012) Research-based guidelines for juvenile justice programs.
Justice Research and Policy, (14) 1, p.1-18.
Integrated Meta-analysis of Program Types
• Mark Lipsey, Peabody Research Institute, Vanderbilt Univ.
• 548 studies - program impact on recidivism
• Juveniles aged 12-21, in programs to reduce delinquency
• Most effective program types? Characteristics of them?
Lipsey, M. W. (2009). The primary factors that characterize effective interventions with juvenile
offenders: A meta-analytic overview. Victims and Offenders (4), 124-147.
Meta-analysis Results
Howell, J.C., & Lipsey, M. W. (2012) Research-based guidelines for juvenile justice programs.
Justice Research and Policy, (14) 1, p.1-18.
Meta-analysis Results
• Four factors most strongly related to recidivism reduction:
• Program type: therapeutic vs. control-oriented
• Quality of service delivery: clear protocol, monitoring,
staff training
• Amount of service: duration and total contact hours
• Risk: greater impact with higher risk youth
Youth Level of Risk (25pts):
Service/program groups (35pts):
% of youth served at target risk
1. Individual counseling, jobYouth
Level of Service (YLS)
training
42 item,
Quality
ofinterview-based
Service
Delivery
(20pts):
2.
Community
service,
remedial
Riskacademics
factors assessed:
1.
Written
Program
Protocol
•Amount
Prior
&
current
offenses
3. Family
skills
of couns.,
Service social
(20pts):
•2. Family
circumstances/Parenting
Staff Training
training, challenge programs
• of
Education/Employment
%
youth
meeting targets for
3.
Staff
Supervision
•4. Peer
Relations
Group
counseling,
mentoring,
1.
Duration
(10pts)
•4. Substance
Use
Response to Drift
contingency
mgmt
• Leisure/Recreation
2.
Contact Hours
(10pts)
• Personality/Behavior
5.
Cognitive-behavioral therapy
• Attitudes/Orientation
Example: Mentoring
Target weeks: 26
Target hours: 78
SPEP Score Validity
*Quality of service delivery not scored in this sample.
Howell, J.C., & Lipsey, M. W. (2012) Research-based guidelines for juvenile justice programs.
Justice Research and Policy, (14) 1, p.1-18.
Advantages
• Utilizes wider research base, and broadens scope of
vetted programs
• e.g., literature on “social skills training”
• e.g., other mentorship models, beyond BBBS
• Fills practical and political gap between “model
programs” and everything else
• Not a model program for every need
• Potential for significant public health impact
• Creates new possibilities in program research and
development
• Moving beyond the binary mode of “on-the-list” or not
Limitations
• Scoring is intensely qualitative, time consuming
• e.g., 53.5 man hours (2-3 staff, 3-4 days)
• What’s “behind the curtain”?
• Confusion around what “evidence-based” means now…
• Field just getting comfortable with vetted lists
EPISCenter’s Role
• “Ride along” with Berks & Lehigh during pilot phase
• How well does SPEP work in PA context?
• Prepare for later statewide rollout
• Develop infrastructure and streamline processes from
pilot phase to scale-up
• Facilitate the process of scoring across PA counties
(probably in phases/cohorts)
• Develop and support Program Improvement Plans
• Collect lessons learned and facilitate CQI of SPEP in PA
EPISCenter’s Role
• Main disadvantage of generic programs is they…
“lack the specificity that comes with the protocol for a
model program and the associated training and support
systems that are also often available from the developer”
(p. 3).
Howell, J.C., & Lipsey, M. W. (2012) Research-based guidelines for juvenile justice programs.
Justice Research and Policy, (14) 1, p.1-18.
40
Thank You!
Evidence-based Prevention and Intervention Support Center
Prevention Research Center, Penn State University
206 Towers Bldg.
University Park, PA 16802
(814) 863-2568
episcenter@psu.edu
www.episcenter.psu.edu