Evidence Based Practice – An Overview
Webinar for Reclaiming Futures
October 23, 2008
Randolph Muck, M.Ed.
CSAT/SAMHSA
Contact Info: randy.muck@samhsa.hhs.gov
240-276-1576
Evidence Based Practice
What is it?
Why do it?
Points to consider
Evidence Based Practice
The term evidence-based practice (EBP)
refers to preferential use of mental and
behavioral health interventions for which
systematic empirical research has provided
evidence of statistically significant
effectiveness as treatments for specific
problems. Alternate terms with the same
meaning are evidence-based treatment (EBT)
and empirically-supported treatment (EST).
Evidence Based Practice
Tested with good outcomes
Manual exists so it can be replicated/trained
A training program exists
Supervision leading to certification
Ongoing monitoring
Outcomes measurement
Ways of Viewing EBP

EBP is a process. EBP is a way of doing practice
that integrates the best evidence with clinical
expertise and consumer values. (EBP as a verb.)
(Sackett et al., 2000)
Practitioner
Expertise
Best
Evidence
EBP
Client Values
& Preferences
Ways of Viewing EBP

EBP is a product. An evidence-based practice
is any practice that has been established as
effective through scientific research according
to some set of explicit criteria. (EBP as a
noun.) (Drake, 2001)
–
–
EB Interventions. (A-CRA, MET/CBT5)
EB Skill sets. (CBT, Behavioral Parent Training)
Definition of Implementation
“…Specified
set of activities
designed to put into practice an
activity or program of known
dimensions…such that independent
observers can detect its presence
and strength.”
(Fixsen et al, 2004, p. 5)
Definition of Fidelity

Strategies used to monitor the faithful
delivery of a manual-guided behavioral
intervention

Important dimensions include
– adherence (i.e., extent to which
intervention procedures were delivered
as prescribed in the treatment manual)
– competence (i.e., qualitative measure of
the skillfulness in which intervention
procedures are delivered)
Different Types of Manuals

Session Driven

Procedure Driven

Principle Driven
Randomized Clinical Trials (RCT) are to
Evidence Based Practice (EBP) like
Self-reports are to Diagnosis





They are only as good as the questions asked (and
then only if done in a reliable/valid way)
They are an efficient and logical place to start
But they can be limited or biased and need to be
combined with other information
Just because the person does not know something
(or the RCT has not be done), does not mean it is
not so
Synthesizing them with other information usually
makes them better
Context

The field is increasingly facing demands from payers,
policymakers, and the public at large for “evidence-based
practices (EBP)” which can reliably produce practical and
cost-effective interventions, therapies and medications
that will
– reduce risks for initiating drug use among those not yet
using,
– reduce substance use and its negative consequences
among those who are abusing or dependent, and
– reduce the likelihood of relapse for those who are
recovering
NIDA Blue Ribbon Panel on Health Services Research
(see www.nida.nih.gov )
So what does it mean to move the field
towards Evidence Based Practice (EBP)?

Introducing reliable and valid assessment that can be used
– At the individual level to immediately guide clinical judgments
about diagnosis/severity, placement, treatment planning, and
the response to treatment
– At the program level to drive program evaluation, needs
assessment, and long term program planning

Introducing explicit intervention protocols that are
– Targeted at specific problems/subgroups and outcomes
– Having explicit quality assurance procedures to cause
adherence at the individual level and implementation at the
program level

Having the ability to evaluate performance and outcomes
– For the same program over time,
– Relative to other interventions
The Current Renaissance of
Adolescent Treatment Research
Feature
1930-1997
1997-2005
Tx Studies*
16
Over 200
Random/Quasi
9
44
Tx Manuals*
0
30+
QA/Adherence
Rare
Common
Std Assessment*
Rare
Common
Under 50%
Over 80%
40-50%
85-95%
Methods
Descriptive/Simple
More Advanced
Economic
Some Cost
Cost, CEA, BCA
Participation Rates
Follow-up Rates
* Published and publicly available
Issues to Consider

Juvenile Justice involved youth increasing
presence in the treatment system

Support for funding relies on ability to
demonstrate effectiveness

Treatment needs of the youth that we see
and the need to incorporate appropriate
and effective interventions for these needs

Continuing Care is as or more important
than the treatment delivered
Change in Referral Sources: 1993-2003
JJ referrals have doubled, are
53% of 2003 admissions and
driving growth
70,000
114%
80,000
140%
115%
90,000
120%
100%
60,000
Other sources of Referral have
grown, but less than expected
50,000
80%
60%
40%
12%
20,000
37%
37%
5%
30,000
41%
40,000
10,000
61%
growth
20%
Source: Treatment Episode Data Set (TEDS) 1993-2003.
Employee/EAP
Other Health
Care
Other SA Tx
Agency
Other
Community
Self/Family
School
0%
Juvenile Justice
-
1993
2003
Change
53% Have Unfavorable Discharges
Despite being widely
recommended, only
10% step down after
intensive treatment
Total
(61,153 discharges)
LTR
(5,476 discharges)
STR
(5,152 discharges)
Detox
(3,185 discharges)
IOP
(10,292 discharges)
Outpatient
(37,048 discharges)
0%
Completed
20%
Transferred
40%
60%
ASA/ Drop out
80%
100%
AD/Terminated
Source: Data received through August 4, 2004 from 23 States (CA, CO, GA, HI, IA, IL, KS, MA, MD, ME, MI, MN, MO, MT, NE, NJ, OH, OK, RI, SC, TX,
UT, WY) as reported in Office of Applied Studies (OAS; 2005). Treatment Episode Data Set (TEDS): 2002. Discharges from Substance Abuse Treatment
Services, DASIS Series: S-25, DHHS Publication No. (SMA) 04-3967, Rockville, MD: Substance Abuse and Mental Health Services Administration.
Retrieved from http://wwwdasis.samhsa.gov/teds02/2002_teds_rpt_d.pdf .
Median Length of Stay is only 50 days
Level of Care
Median Length of Stay
Total
(61,153 discharges)
50 days
LTR
(5,476 discharges)
49 days
STR
(5,152 discharges)
21 days
Detox
(3,185 discharges)
3 days
IOP
(10,292 discharges)
Less than
25% stay the
90 days or
longer time
recommended
by NIDA
Researchers
46 days
Outpatient
(37,048 discharges)
59 days
0
30
60
90
Source: Data received through August 4, 2004 from 23 States (CA, CO, GA, HI, IA, IL, KS, MA, MD, ME, MI, MN, MO, MT, NE, NJ, OH, OK, RI, SC, TX,
UT, WY) as reported in Office of Applied Studies (OAS; 2005). Treatment Episode Data Set (TEDS): 2002. Discharges from Substance Abuse Treatment
Services, DASIS Series: S-25, DHHS Publication No. (SMA) 04-3967, Rockville, MD: Substance Abuse and Mental Health Services Administration.
Retrieved from http://wwwdasis.samhsa.gov/teds02/2002_teds_rpt_d.pdf .
38%
Sex Under the Influence of AOD
32%
Multiple Sex partners
26%
Any Unprotected Sex
Victimized Physically, Sexually, or
Emotionally
21%
3%
Source: CSAT AT Outcome Data Set (n=9,276 adolescents)
100%
90%
80%
70%
84%
Sexually active
Any Needle use
60%
50%
40%
30%
20%
10%
0%
Past 90 day HIV Risk Behaviors
49%
School/Work Peers Getting Drunk Weekly+
28%
Others at Home Getting Drunk Weekly+
74%
Social Peers Using Drugs
65%
School/Work Peers Using Drugs
14%
Source: CSAT AT Outcome Data Set (n=9,276 adolescents)
100%
90%
80%
70%
60%
57%
Social Peers Getting Drunk Weekly+
Others at Home Using Drugs
50%
40%
30%
20%
10%
0%
Recovery Environment
54%
Conduct Disorder
45%
Attention Deficit/Hyperactivity Disorder
37%
Major Depressive Disorder
26%
Traumatic Stress Disorder
17%
59%
Ever Physical, Sexual or Emotional Victimization
47%
High severity victimization (GVS>3)
31%
Ever Homeless or Runaway
25%
Any homicidal/suicidal thoughts past year
Any Self Mutilation
16%
Source: CSAT AT Outcome Data Set (n=9,276 adolescents)
100%
90%
79%
Any Co-occurring Psychiatric
General Anxiety Disorder
80%
70%
60%
50%
40%
30%
20%
10%
0%
Co-Occurring Psychiatric Problems
82%
Any violence or illegal activity
69%
Physical Violence
66%
Any Illegal Activity
51%
Any Property Crimes
Other Drug Related Crimes*
Any Interpersonal/ Violent Crime
49%
45%
84%
Lifetime Juvenile Justice Involvement
68%
Current Juvenile Justice involvement
1+/90 days In Controlled Environment
39%
*Dealing, manufacturing, prostitution, gambling (does not include simple possession or use)
Source: CSAT AT Outcome Data Set (n=9,276 adolescents)
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Past Year Violence & Crime
Multiple Problems* are the Norm
100%
90%
80%
70%
Five to
Twelve
In fact, over
half present
acknowledging 5+
major problems
60%
50%
40%
30%
Most
20%
acknowledge
10% 1+ problems
0%
Four
Three
Two
One
None
Few present with
just one problem
(the focus of
traditional
research)
* (Alcohol, cannabis, or other drug disorder, depression, anxiety, trauma, suicide, ADHD, CD, victimization,
violence/ illegal activity)
Source: CSAT AT Common GAIN Data set
No. of Problems* by Severity of Victimization
100%
Those with high
lifetime levels of
victimization have
117 times higher
odds of having 5+
major problems*
90%
80%
70%
60%
50%
Five or More
40%
Four
Three
30%
Two
20%
One
10%
None
0%
Low (31%)
Moderate (17%)
High (51%)
GAIN General Victimization Scale Score (Row %)
Source: CSAT AT Common GAIN Data set (odds for High over odds for Low)
* (Alcohol, cannabis, or
other drug disorder,
depression, anxiety,
trauma, suicide, ADHD,
CD, victimization,
violence/ illegal activity)
Most Lack of Standardized Assessment for…





Substance use disorders (e.g., abuse, dependence,
withdrawal), readiness for change, relapse
potential and recovery environment
Common mental health disorders (e.g., conduct,
attention deficit-hyperactivity, depression,
anxiety, trauma, self-mutilation and suicidality)
Crime and violence (e.g., inter-personal violence,
drug related crime, property crime, violent crime)
HIV risk behaviors (needle use, sexual risk,
victimization)
Child maltreatment (physical, sexual, emotional)
Summary of Problems in the Treatment System







The public systems is changing size, referral
source, and focus – often in different directions by
state
Major problems are not reliably assessed (if at all)
Less than 50% stay 50 days (~7 weeks)
Less the 25% stay the 3 months recommended by
NIDA researchers
Less than half have positive discharges
After intensive treatment, less than 10% step
down to outpatient care
While JJ involvement is common, little is known
about the rate of initiation after detention
EBPs and Treatment for Youth
in the Juvenile Justice
System
Some Programs Have Negative or
No Effects on recidivism

“Scared Straight” and similar shock incarceration
program

Boot camps mixed – had bad to no effect

Routine practice – had no or little (d=.07 or 6% reduction
in recidivism)

Similar effects for minority and white (not enough data to
comment on males vs. females)

The common belief that treating anti-social juveniles in
groups would lead to more “iatrogenic” effects appears to
be false on average (i.e., relapse, violence, recidivism for
groups is no worse then individual or family therapy)
Source: Adapted from Lipsey, 1997, 2005
Meta Analysis of the Effectiveness of
Programs for Juvenile Offenders
N of
Offender Sample
Studies
Preadjudication (prevention)
178
Probation
216
Institutionalized
90
Aftercare
25
Total
Source: Adapted from Lipsey, 1997, 2005
509
Most Programs are actually
a mix of components
Average of 5.6 components distinguishable in program
descriptions from research reports
Intensive supervision
Prison visit
Restitution
Community service
Wilderness/Boot camp
Tutoring
Individual counseling
Group counseling
Family counseling
Parent counseling
Recreation/sports
Interpersonal skills
Source: Adapted from Lipsey, 1997, 2005
Anger management
Mentoring
Cognitive behavioral
Behavior modification
Employment training
Vocational counseling
Life skills
Provider training
Casework
Drug/alcohol therapy
Multimodal/individual
Mediation
Most programs have small effects
but those effects are not negligible

The median effect size (.09) represents a reduction of the
recidivism rate from .50 to .46

Above that median, most of the programs reduce
recidivism by 10% or more

One-fourth of the studies show recidivism reductions of
30% or more

The “nothing works” claim that rehabilitative programs
for juvenile offenders are ineffective is false
Source: Adapted from Lipsey, 1997, 2005
Major Predictors of Bigger Effects
1.
Chose a strong intervention protocol
based on prior evidence
2.
Used quality assurance to ensure
protocol adherence and project
implementation
3.
Used proactive case supervision of
individual
4.
Used triage to focus on the highest
severity subgroup
Impact of the numbers of Favorable
features on Recidivism (509 JJ studies)
Usual
Practice
has little
or no
effect
Source: Adapted from Lipsey, 1997, 2005
Program types with average or better
effects on recidivism
AVERAGE OR BETTER
BETTER/BEST
Preadjudication
Drug/alcohol therapy
Interpersonal skills training
Parent training
Employment/job training
Tutoring
Group counseling
Probation
Drug/alcohol therapy
Cognitive-behavioral therapy
Family counseling
Interpersonal skills training
Mentoring
Parent training
Tutoring
Institutionalized
Family counseling
Behavior management
Cognitive-behavioral therapy
Group counseling
Employment/job training
Individual counseling
Source: Adapted from Lipsey, 1997, 2005
Interpersonal skills training
Cognitive Behavioral Therapy (CBT) Interventions
that Typically do Better than Practice in Reducing
Recidivism (29% vs. 40%)










Aggression Replacement Training
Reasoning & Rehabilitation
Moral Reconation Therapy
Thinking for a Change
Interpersonal Social Problem Solving
Multisystemic Therapy
Functional Family Therapy
Multidimensional Family Therapy
Adolescent Community Reinforcement Approach
MET/CBT combinations and Other manualized CBT
NOTE: There is generally little or no differences in mean
effect size between these brand names
Source: Adapted from Lipsey et al 2001, Waldron et al, 2001, Dennis et al, 2004
Implementation is Essential
(Reduction in Recidivism from .50 Control Group Rate)
The best is to
have a strong
program
implemented
well
Thus one should optimally pick the
strongest intervention that one can
implement well
Source: Adapted from Lipsey, 1997, 2005
The effect of a well
implemented weak program is
as big as a strong program
implemented poorly
Moderate to large differences
in Cost-Effectiveness by Condition
$16
$20,000
$16,000
$12
$12,000
$8
$8,000
$4
$4,000
$0
$0
MET/ CBT5
MET/
CBT12
FSN
MET/ CBT5
ACRA
MDFT
CPDA*
$4.91
$6.15
$15.13
$9.00
$6.62
$10.38
CPPR**
$3,958
$7,377
$15,116
$6,611
$4,460
$11,775
* p<.05 effect size f=0.48
** p<.05, effect size f=0.72
Source: Dennis et al., 2004
* p<.05 effect size f=0.22
** p<.05, effect size f=0.78
Cost per person in recovery
at month 12
Cost per day of abstinence
over 12 months
$20
ACRA did better than
MET/CBT5,
and both did
Trial 2
better than MDFT
MET/CBT5 and
Trial 1
12 did better
than FSN
Choosing an EBP
 Best
evidence
 Practitioner experience
 Youth/Family values and preferences
 Readiness for change (buy-in at all
levels of agency, but particularly the
top)
 Cost/Resources
 Ability to implement well
What are the pitfalls of EBP?

EBP generally causes some staff turnover

EBP often shines a light on staff or work place problems
that would otherwise be ignored

EBP often impact a wide range of existing procedures and
policies – requiring modification and provoking resistance

EBP (and most organizational changes) will fail without
good senior staff leadership

EBP typically require going for more funds from grant or
other funders

On-going needs assessment will create demand for more
change and more EBP
A Fearless Appraisal…

We are entering a renaissance of new knowledge in this area, but are
only reaching 1 of 10 in need

Several interventions work, but 2/3 of the adolescents are still having
problems 12 months later

Effectiveness is related to severity, intervention strength,
implementation/adherence, and how assertive we are in providing
treatment

As other therapies have caught up technologically, there is no longer
the clear advantage of family therapy found in early literature reviews

While there have been concerns about the potential iatrogenic effects
of group therapy, the rates do not appear to be appreciably different
from individual or family therapy if it is done well (important since
group tx typically costs less)

Effectiveness was not consistently associated with the amount of
therapy over a short period of time (6-12 weeks) but was related to
longer term continuing care
Common Strategies you can do NOW







Standardize assessment and identify most common
problems
Pool knowledge about what staff have done in the past,
whether it worked, and what the barriers were
Identify system barriers (e.g., criteria to local access case
management, mental health) that could be avoided if
thought of in advance
Identify existing materials that could help and make sure
they are readily available on site
Identify promising strategies for working with the
adolescent, parents, or other providers
Develop a 1-2 page checklist of things to do when this
problem comes up
Identify a more detailed protocol and trainer to address the
problem, then go for a grant to support implementation
Evidenced Based Practice - Summary






Achieving reliable outcomes requires reliable measurement,
protocol delivery and on-going performance monitoring.
The GAIN is one measure that is being widely used by CSAT
grantees and others trying to address gaps in current knowledge
and move the field towards evidenced based practice.
Standardized and more specific assessment helps to draw out
treatment planning implications of readiness for change,
recovery environment, relapse potential, psychopathology,
crime/violence, and HIV risks.
Adolescents entering more intensive levels of care typically have
higher severity.
Multiple problems and child maltreatment are the norm and are
closely related to each other.
There is a growing number of standardized assessment tools,
treatment protocols and other resources available to support
evidenced based practices.