Evidence-Based Interventions:
Musings on Evidence, MetaAnalysis, and Programs for Juvenile
Offenders
Mark W. Lipsey
Peabody Research Institute
Vanderbilt University
Penn State
March 2011
Themes

An alternative perspective on evidencebased practice and its implementation for
social programs

Some of what I’ve learned from metaanalysis about interventions for at-risk
youth and research on those interventions

Effective interventions for juvenile
offenders
1
Conventional definition of evidencebased practice: ‘Brand name’ protocol
programs
Examples for juvenile offenders:

Functional Family Therapy (FFT)

Multisystemic Therapy (MST)

Multidimensional Treatment Foster Care

Aggression Replacement Training (ART)
Lists of “model” programs certified by
some group as supported by credible
research
Blueprints for Violence Prevention
 National Registry of Evidence-based
Programs and Practices (NREPP)
 Helping America’s Youth (HAYS)
community guide
 OJJDP Model Programs Guide

2
Protocol programs as evidence-based
practice
Evidence base:

Evaluation studies of implementations of
that protocol in different places (usually
only a few studies)
Assumptions:

Implemented with fidelity

Generalizable beyond the original studies
Specificity assumption for protocol
programs

Program A works

Similar Program B doesn’t work

The difference is because Program A has
effective ‘active ingredients’

Program B may be similar in other ways
but lacks those effective ingredients
3
Alternate perspective on what
constitutes a social program: Generic
intervention types
Examples for juvenile offenders:

Interpersonal skills training

Family therapy

Group counseling

Cognitive behavioral therapy
Within a type, similar activities and
interactions between providers and
juveniles
Allows broader consideration of what
differentiates similar effective and
ineffective programs
Evidence base:

Evaluation studies of different programs of
that type in different places (often many
studies).
Allows examination of the specificity
assumption and exploration of other
program characteristics that may be
associated with effectiveness
4
Case example: Programs for juvenile
offenders
Outcome of primary interest: reoffense rates
* Interventions that reduce reoffense rates
also have positive effects on:



Mental health indicators and interpersonal
adjustment
School participation and performance
Vocational accomplishments
Meta-analysis of existing studies of
interventions for juvenile offenders

548 independent study samples

Juveniles aged 12-21

Intervention intended to have positive
effects on subsequent delinquency

At least one delinquency outcome

Random assignment or, if not, pretreatment
differences reported or matched
5
Examples of information coded into
the meta-analysis database
Distribution of effect sizes for cognitive
behavioral programs
Average
recidivism
reduction of
25%
6
Mix of brand name & ‘home grown’
programs
R&R
MRT
ART
Distribution of effect sizes for family
therapy interventions
Family Interventions
Covariate-Adjusted Recidivism Effect Sizes (N=29)
>.00
Average
recidivism
reduction of
13%
-.40
-.30
-.20
-.10
.00
.10
.20
.30
.40
.50
.60
Median
Effect Size (zPhi coefficient)
7
Brand name model programs in this
distribution
Family Interventions
Covariate-Adjusted Recidivism Effect Sizes (N=29)
>.00
FFT
MST
-.40
-.30
-.20
-.10
.00
.10
.20
.30
.40
.50
.60
Median
Effect Size (zPhi coefficient)
Effect sizes analyzed as a function of study
and program characteristics
8
Method matters (a lot)
Effect size differences associated with:




Outcome measurement (type of recidivism,
timing)
Design (initial equivalence; design type)
Sample size
Publication source
Used as statistical control variables
Some characteristics of the juveniles
matter
Effect size differences associated with:


Delinquency risk (strong positive)
Aggressive history (moderate negative)
Effect size differences not associated with:



Mean age
Gender mix
Ethnicity
9
JJ supervision level doesn’t matter
much (with risk, etc. controlled)
Effect size differences not associated with:




No JJ supervision (prevention programs)
Diversion
Probation/parole
Incarceration
Type of program matters
Programs are identified first according to
their broad approach or “philosophy:”


Control philosophies
Therapeutic philosophies
And, second, by their generic type, e.g.,
group counseling, interpersonal skills,
cognitive behavioral therapy
10
Treatment “philosophy” matters
Therapeutic
interventions
Discipline
Deterrence
Surveillance
Restorative
Skill building
Counseling
Multiple services
-10
-5
0
5
10
15
% Recidivism Reduction from .50 Baseline
Type of intervention: Skill-building
Behavioral
Cognitive-behavioral
Social skills
Challenge
Academic
Job related
0
5
10
15
20
25
30
% Recidivism Reduction from .50 Baseline
11
Type of intervention: Counseling
Individual
Mentoring
Family
Family crisis
Group
Peer
Mixed
Mixed w/referrals
0
5
10
15
20
25
% Recidivism Reduction from .50 Baseline
How well the program is implemented
matters: Service amount and quality
Effect size differences associated with:



Duration of service
Total hours of service
Quality of implementation
Along with intervention type, the strongest
predictors of effects
12
These findings suggest simple ‘best
practice’ guidelines

Program type: Therapeutic approach and one
of the more effective intervention types

For a given intervention type:

Risk: Larger effects with high risk juveniles

Dose: Amount of service that at least matches the
average in the supporting research

High quality implementation: Treatment protocol
and monitoring for adherence
Operationalizing these practice
guidelines at scale: The Standardized
Program Evaluation Protocol (SPEP)

A rating system for each program type
within the therapeutic philosophies

Applied to individual programs based on
data about the services actually provided

Pilot projects with the juvenile justice
systems of Arizona, North Carolina, &
Tennessee
13
Standardized Program Evaluation Protocol (SPEP) for
Services to Probation Youth
Points assigned
proportionate
to the
contribution of
each factor to
recidivism
reduction
Target values
from the metaanalysis
(generic) OR
program
manual
(manualized)
Possible
Points
Received
Points
Primary Service:
High average effect service (35 points)
Moderate average effect service (25 points)
Low average effect service (15 points)
Supplemental Service:
Qualifying supplemental service used (5 points)
35
5
Treatment Amount:
Duration:
% of youth that received target number of weeks of service or more:
0% (0 points)
60% (6 points)
20% (2 points)
80% (8 points)
40% (4 points)
100% (10 points)
Contact Hours:
% of youth that received target hours of service or more:
0% (0 points)
60% (9 points)
20% (3 points)
80% (12 points)
40% (6 points)
100% (15 points)
Treatment Quality:
Rated quality of services delivered:
Low (5 points) Medium (10 points)
10
15
15
High (15 points)
Youth Risk Level:
% of youth with the target risk score or higher:
25% (5 points)
75% (15 points)
50% (10 points)
99% (20 points)
Provider’s Total SPEP Score:
20
100
[INSERT
SCORE]
Validity study: Does it work?

Arizona Juvenile Justice Services Division

Programs provided during 2005-06 to
juvenile probationers in five pilot counties

1490 juveniles who received services from
66 SPEP rated programs

6-month recidivism data on all; 12-month
recidivism for most
14
Distribution of SPEP scores
across programs
30
25
73% have
scores < 50
34.8%
Number of Programs
31.8%
20
15
16.7%
10
5
6.1%
4.5%
4.5%
60-69
70-79
1.5%
0
20-29
30-39
40-49
50-59
80-85
Total SPEP Score
Actual vs. predicted recidivism for
6-mo
providers
with scores ≥ 50 and < 50
recidivism
6-mo
recidivism
difference:
Low score
difference:
High score
6-Month Recidivism Difference
-0.12
-0.01
SPEP ≥ 50
SPEP < 50
12-Month Recidivism Difference
12-mo
recidivism
difference:
Low score
-0.13
-0.01
12-mo
recidivism
difference:
High score
-0.14
-0.12
-0.1
-0.08
-0.06
-0.04
-0.02
0
0.02
Actual Minus Predicted Recidivism Difference
15
Final musings

The name brand “model” program is not
the only approach to effective evidencebased practice

Meta-analysis is important for investigating
the factors associated with effectiveness

Key factors: Intervention type and quality
of implementation

Challenge of scaling up effective
intervention approaches
Thanks!
Questions?
Comments?
16