Efficacy to Implementation in the
Child Welfare System
J. Landsverk*, P. Chamberlain**, J. Reid**
(Eve Reider***, Chair)
PSMG
October 26, 2005
* Child and Adolescent Services Research Center, San Diego, CA
** Oregon Social Learning Center & Center for Research to Practice,
Eugene, OR
*** National Institute on Drug Abuse, Rockville, MD
•
•
•
•
•
•
•
•
Cascading Dissemination of a Foster Parent Intervention (R01 MH 60195)
Principal Investigator: Patti Chamberlain
Co-Investigators: Joe Price, John Reid, John Lansverk
Funding Agency: National Institute of Mental Health
Preventing Health-Risking Behavior in Delinquent Girls (R01 DA15208-1)
Principal Investigator: Patti Chamberlain
Co-Investigators: John Reid, Leslie Leve
Funding Agency: National Institute of Drug Abuse
Preventing Problems for Girls in Foster Care (R01 MH54257-6)
Principal Investigator: Patti Chamberlain
Co-Investigators: Leslie Leve, John Reid
Funding Agency: National Institute of Mental Health
Oregon Prevention Research Center (2 P30 MH 46690)
Principal Investigator: John Reid
Co-Investigators: Chamberlain, DeGarmo, Eddy, Fisher, Leve, Marinez, Fetrow,
Patterson, Forgatch, Bank, Shortt, Capaldi
Funding Agency: National Institute of Mental Health
Pathways Home: Reducing Risk in the Child Welfare System (1 P20 DA017592)
Principal Investigator: John Reid
Co-Investigators: Chamberlain, DeGarmo, Eddy, Fisher, Leve, Martinez, Fetrow,
Bronz, Sprengelmeyer, and Smith
Funding Agency: National Institute of Mental Health
Patterns of Youth Mental Health Care in Public Service Systems (U01 MH55282)
Principal Investigator: Richard Hough
Funding Agency: National Institute of Mental Health
Mental Health Services Across Child Welfare Agencies (R01 MH59672)
Principal Investigator: John Lansverk
Funding Agency: National Institute of Mental Health
Center for Research on Child and Adolescent Mental Health Services (P50 MH50313)
Principal Investigator: John Lansverk
•
Improving Care for Children in Child Welfare (R24 MH67377)
Principal Investigator: John Lansverk
Funding Agency: National Institute of Mental Health
•
Mixed-Methods Study of a Statewide EBP Implementation (R01)
Principal Investigator: Greg Aarons
Funding Agency: National Institute of Mental Health
Child Welfare Service Sector –
Characteristics Related to Design
Issues
Child Welfare as a Mission Context
• Effectiveness and implementation studies are service sector
context specific (usual care), efficacy studies usually are not.
• Mission of Child Welfare
– Child Safety
– Child Permanence
– Child Well-Being
– Safety & permanence services provided by CW
– Well-Being services provided by other sectors (mental health,
substance abuse services)
Service Sector Comparisons on Risk level &
Intervention Type, Reason for Entry & Sex Ratio
Risk Status
Level
Primary
Secondary
Tertiary
Intervention
Type
Universal
Sector of Care
Reasons for Entry
Sex Ratio
General Health
Physical Health Care
Even
Education
Education
Even
Child Welfare
Maltreatment/Caregiver
absence
Even
Juvenile Justice
Legal violations
M>F
Substance Abuse
Substance abuse
M>F
S.E.D.
Mental health problems
M>F
Mental Health
Mental health problems
M>F
Targeted
Clinical
Service Sector Comparison on
Age at Entry
Sector of Care
Infancy /
Toddlerhood
Childhood
General Health
Primary Care
Education
Child Welfare
Juvenile Justice
Substance Abuse
S.E.D.
Mental Health
= Most common entry age into service sector
Adolescence
Context of Child Welfare
• Yearly: 5,000,000 referred, 3,000,000 investigated
(4.5% of all children), 826,000 substantiated for
child abuse and neglect
• Multiple living environments for child when
services delivered: (1) home of origin, (2) out-ofhome care or foster care [relative/non-relative,
congregate care], (3) adoptive home
• Service trajectories may involve movement from
one child setting to another
Nationally Representative Child
Welfare Cohort Study (NSCAW)
• 92 primary sampling units, 6,000+ investigated
cases (substantiated or indicated), followed for 36
months
• 5 months after investigation
– 1% in residential care (congregate) (5.7% after 36
months)
– 10% in relative or non-relative family foster care
– 24% received services in parent’s home
– 65% no further services beyond investigation
Need and Opportunity for
Preventive Interventions
• High rates of mental disorders (42% for ages 6-17 – POC
San Diego study, highest rates for ADHD, ODD, CD),
developmental problems , social and cognitive problems
• At any given time, 1% of children under 18 live in out-ofhome care, estimated to be 3-7% of all children at some
time
• Externalizing problems, mission elements of safety and
permanence >>> parent mediated interventions
Child Welfare Context and Effectiveness
Study Design and Measurement Issues
• Multiple child settings – multiple environmental
contexts and multiple parent informants for
longitudinal designs
• Clustering at entry – multiple children removed
from same family and placed in out-of-home care
with confounding by maltreatment type ( more
clustering for neglect)
• Clustering for case worker functions – supervisory
units of 8-10 case workers
• Timing for identification, sampling and enrolling
because of early exit bias
Child Welfare Context and Effectiveness
Study Design Issues
• Little tradition of randomized study designs and use of
standardized measurement
• Labor force: (1) case workers - BA and MSW in social
work, (2) para-professional foster parent (relative & nonrelative), congregate care shift workers (BA level)
• Cost analysis complicated by multiple sector costs:
– safety and permanence costs are paid by child welfare
– parent training costs paid by education sector
(community college)
– ameliorative costs (mental health, substance use,
developmental) paid by other service sectors
Child Welfare Context and Effectiveness
Study Design – Organization of Services
• Multiple levels: state and county agencies, programs within
agency, supervisory units within programs, families within
case worker, children within families – adjustment for
clustering and nested designs
• Child nested within foster care and within biological family
• Service programs organized by type of child setting –
importance of change of setting and it’s meaning
– Entry into services
– Entry into out-of-home care
– Change of placement within out-of-home care
– Exit from out-of-home care
– Exit from services
Collaborations among Disparate
Research Traditions
• Intervention Development and Efficacy
Trials – relatively small n studies, “thick”
measures with triangulation
• Services Research and Clinical
Epidemiology – large N studies, “thin”
measures including administrative data
• Prevention Science Methodologists
Developmental modeling and
efficacy trials leading to
effectiveness and Implementation
trials of an intervention for foster
parents in the CWS
Early trials of PMT
• Crafted on basis of a great deal of clinical
and systematic observation in homes
• Case studies and small randomized trials.
• Very thick measurement
Replications Across
Developmental Level and Sites
•
•
•
•
•
OSLC: first graders through adolescents
Wahler: children
Webster-Stratton: preschoolers
Eyeberg
Forehand
Thick, Multi-Method
Methodology
• Naturalistic and laboratory observations
• Parent, child, and teacher reports
• Administrative data from schools and
juvenile courts
• Psychiatric data
• Analogue Tasks
Word Slide of Neglect Model
e5
e6
CEES
Congruence
CRSS
Congruence
D1
r 2 .15
SupervisionTracking
e1
Mother
Education
log Abusive
Index
Analog
Physicals Escalation
e3
e4
AchieveHEQACH
e9
D3
r 2 .30
Punitive
Discipline
e2 Social Status
HEQSS
e8
Anger
STAS - APT
e10
Inconsistent
Discipline
log IPC
Mother to
Child
Social
Status
e11
e12
CBCL T
e13
Child
Scenarios
e14
Interviewer
Rating
e15
TRF T
e17
D4
Mother
Occupation
r
Age
Care &
Environmental
2 .12 Neglect
Child
Aggression
r 2 .48
D2
Sex
e7
Neclect
Index
D4b
Child
Aggression
WB
cmin=5356.00,df=\df,p=\p,cmindf=\cmindf,cfi=\cfi
r 2 1.29
CBCL T
e13b
Child
Scenarios
e14b
Interviewer
Rating
e15b
TRF T
e17b
Mediated model for lift, playground etc
Obs 2
.36
e15
e16
Obs 3
Obs 1
Obs 2
Obs 3
.41
.42
.34
.3
7
e14
.4
4
Obs 1
e13
.39
e12
Aggress
Playground
Fall 5th
.52***
r2
e11
Agress
Playground
Spr. 5th
r2 .06
.26
*
-.32*
4*
.2
Sex of
Child
D5
.23*
Average
Substance
Use
D1
r2 .13
.60***
1*
2
.
-
Growth
Subtance
Use
-.26**
Assigned
Intervention
-.15*
-.16*
Intervention
X Sex
6*
.- 1
Accel.
Substance
Use
D3
e7
.25
e8
.72
r2
parents/
child
Prob.
Solving
Spr. 5th
mother/
child
e9
D4
.73
mother/
child
.48***
.71
.76
Prob.
Solving
Fall 5th
parents/
child
e10
.48***
.13
*
-.2
0*
D2
r2 .09
Academic Progress
14.0 12.0 10.0 8.0 6.0 4.0 2.0 0.0 -
10.5
2.3
Intervention
Control
1st Grade (p < .01)
14.0 12.0 10.0 8.0 6.0 4.0 2.0 0.0 -
12.4
Intervention
10.7
Control
5th Grade (n.s.)
CD
AP
DSM IV Criteria
14.0 12.0 10.0 8.0 6.0 4.0 2.0 0.0 -
12.6
5.5
Intervention
Control
1st Grade (p < .05)
14.0 12.0 10.0 8.0 6.0 4.0 2.0 0.0 -
3.6
Intervention
2.5
Control
5th Grade (n.s.)
Foster Parents as Agents of
Change
• Multi-dimensional treatment Foster Care
Multidimensional Treatment Foster
Care (MTFC)
 For youth placed in out-of-home care
 Youth are placed singly in intensively trained and
supervised community foster homes that are
contacted daily and supported 24/7 for 6–9 mo.
 Interventions are implemented using multiple
methods (e.g., family and individual therapy, skill
training, academic supports) in key settings
 Program supervisors carry a caseload of 10,
supervise foster parents, therapists, & skills
trainers, and work with parole/probation officers
 Youth attend public schools
MTFC Effects for Boys
 More time in program/fewer runaways
 Less time in “locked” incarceration in follow-up
 Fewer criminal offenses (½ the rate of GC boys)
 Less likely to commit violent crimes 2 years later
 Delinquency effects mediated by:
• Supervision
• Relationship with a mentoring adult
• Consistent non-harsh discipline
• Less association with delinquent peers
Chamberlain & Reid, 1998; Eddy & Chamberlain, 2000;
Eddy, Whaley, & Chamberlain, 2004
MTFC Effects for Girls
Delinquency
Deviant Peer Association
School Attendance & Homework Time


Leve, Chamberlain & Reid (in press), J of Counseling and
Clinical Psychology; Leve & Chamberlain (2005) J of Abnormal
Child Psychology;
Chamberlain & Leve, in preparation
24-month arrest outcomes
Mean number
of arrests
6
5
4
3
2
1
5.2
4.5
2.7
1.3
1.5
1.4
0
GC
0.8
MTFC
Intervention group
12 months pre-treatment entry
12 months post-treatment entry
24 months post-treatment entry
0.7
Mean number of
days in locked
settings
24-month locked settings outcomes
100
90
80
70
60
50
40
30
20
10
0
90
129
72
75
57
42
22
GC
MTFC
Intervention group
12 months pre-treatment
12 months post-treatment
24 months post-treatment
20
Efficacy to Effectiveness and
Implementation
Project Keep
Project KEEP
Cascading Dissemination of a
Foster Parent Intervention
A collaboration between the:
• San Diego Health and Human Services Agency,
• Child and Adolescent Services Research Center,
• Oregon Social Learning Center, and
funded by the National Institute of Mental Health.
The Goals of KEEP
• To increase the parenting skills of foster and
kinship parents
• To decrease the number of placement disruptions
• To improve child outcomes
• To increase the number of positive placement
changes (e.g. reunification, adoption)
• To test the “cascade” question-can the intervention
be implemented with equal effectiveness by a
second generation of interventionists?
Project KEEP aims to
accomplish these goals by-• Promoting the idea that foster parents can serve as
key agents of change for children.
• Strengthening foster parent’s confidence and skill
level so that they can successfully change their
own and their child’s behaviors.
• Helping foster parents use effective parent
management strategies and provide them with
support to do so
• Increasing short and long term positive child
outcomes in multiple domains and settings –
home, school, with peers.
Support and Training for Foster Parents
Child Demographic Information
Age
Gender
Ethnicity
Age at Baseline
Mean
8.8 (SD=2.2, n=700)
Age Range
4-13 (n=700)
Female
52% (n=364)
Male
48% (n=336)
Caucasian
29% (n=206)
African-American
25% (n=175)
Hispanic/Latino/a
33% (n=228)
Mixed/Other
13% (n=91)
Foster Parent Demographic Information
Mean
Language Spoken
Household Income
English only
60% (n=404)
Spanish only
8% (n=52)
Both English and Spanish
32% (n=122)
Less than 64,999
69% (n=483)
Over 65,000
17% (n=120)
Refused/Don’t Know
14% (n=97)
Foster Parent Demographics - 2
Mean
Employment
Currently Employed
(not including foster parenting)
# of hours works per week
(includes unemployed foster parents)
Education
Level
49% (n=343)
17.1
(SD=20.8,
n=696)
High School/GED or less
41% (n=285)
Some College
46% (n=325)
Vocational or Technical Degree
Bachelor’s Degree
Graduate Degree
1% (n=48)
7% (n=48%)
5% (n=32)
Average Number of Children in Home
Biological/Step Children
Adopted/Foster Children
Other Children
All Children
Control
(n=341)
Treatment
(n=359)
Both
(n=700)
.7
(1.1)
2.4
(1.9)
.7
(1.2)
2.6
(2.0)
.7
(1.2)
2.5
(2.0)
.3
(1.1)
3.5
(2.0)
.1
(.5)
3.5
(1.8)
.2
(.9)
3.5
(1.9)
Rates of child problems: Parent daily
report (PDR)
• PDR is collected by telephone from foster/kin
parents
• Each call takes 5-10 minutes
• We collect 3 calls at baseline, another 3 calls 4
months later, and a final 3 calls 6 months after that
(10 months after baseline)
• PDR produces data on the occurrence of child
behavior problems and foster parent stress
Greater than 5 problem behaviors per day at baseline predicts
placement disruption within the next 6 months
0
Fitted Log Hazard of Placement Disruption
1
2
3
After 5 behaviors, every additional behavior on the PDR
increases the probability of disruption by 13 %
0
5
10
Baseline PDR
15
20
Good news: We learned that…
• Foster/kin parents tolerate about as much
child problem behaviors as non-system
families do –5 behaviors
• PDR data is feasible to collect and is well
tolerated by foster and kin parents
• PDR data tells you who to concentrate the
intervention on given limited resources :
40/60
Percentage
Percentages of Exit Type by Group
15
13
11
9
7
Negative Exit
Intervention
Positive Exit
Control
PDR – Total # of Problem Behavior
(All Children)
Baseline
Termination
Control
Group
5.8
(4.0)
(n=265)
5.4*
(4.1)
(n=265)
Treatment
Group
5.9
(4.3)
(n= 299)
4.4
(3.9)
(n=299)
*P < .05
PDR – Total #of Problem Behaviors
(TC only)
Baseline
Termination
Control
Group
5.4
(3.7)
(n=229)
5.1*
(3.9)
(n=229)
Treatment
Group
5.7
(4.2)
(n= 241)
4.2
(3.8)
(n=241)
*P < .05
PDR – Total #of Problem Behaviors
(Substitute Child only)
Mean (SD)
Control Group
7.4 (4.9)
(n=36)
Treatment
Group
5.1 (4.2)*
(n=64)
PDR by interventionist cohort
control
baseline
developers 5.7
experimental
term
baseline
term
6.0
(4.3)
4.4
(4.0)
5.6
(4.2)
4.3
(3.6)
N=506
(4.0)
5.4
(4.2)
cascade
N=191
6.0
(3.9)
5.5
(4.1)
Intervention Effects on Parenting
Social Learning
parenting practices
Control
BL T
Treatment
BL T
Uses rewards (1=daily,
7=never)
% use point charts
3.7
4.0
3.6
18
22
26
51*
% who use time out
42
37
36
52*
Doesn’t warn/discuss
50
48
45
41
3.4*
Conclusions
• Preliminary data suggests KEEP group
participation increase foster and kin parent skills
• Increased parent skills translate into lower rates of
child problem behaviors
• Lower rates of child problem behaviors translate
into fewer placement disruptions & more frequent
placement transitions to family/relative care
Review of Design and Methods
Issues Raised
• Use of Administrative Data to measure outcomes
(CWS vs. Juvenile Court / School Records)
• Measurement of proximal or targeted processes
(Parent Reports and PDR)
• Moving from Thick to Thin Assessment
• Power when large community units are unit of
randomization
• How do we team up efficiently with PSMG