Ecodevelopmental and Systemic
Modeling and
Implementing High Fidelity
Interventions in Real World Settings
Guillermo Prado1,2, Hilda Pantin1, Seth
Scwartz1, Jose Szapocznik1 & Daniel J.
Feaster 1,2
1
Center for Family Studies,
University of Miami
2 Stempel School of Public Health,
Florida International University
1
The Ecodevelopmental Model and
Methodological Questions
2
The Center for Family Studies is
interested in the role of
ecodevelopmental context in the
prevention and treatment of adolescent
behavior problems, drug abuse, and
HIV/AIDS.
3
Ecodevelopmental Theory
Incorporates three primary, integrated
components:
(a) Social Ecological Theory
(b) Developmental Theory
(c) Emphasis on Social Interactions
4
Social Ecological Theory
Bronfenbrenner (1979, 1986)
Highlights the Multiple Influences on
Adolescent Development, including:
(a) Macrosystems (e.g., cultural & societal
values)
(b) Exosystems (e.g., parents’ exosystemic
stressors and social support for parents)
(c) Mesosystems (e.g., parental monitoring of
peers and collaboration with youth’s school)
(d) Microsystems (e.g. family functioning)
5
Developmental Component
Emphasizes the changing nature of youth,
contexts, & their interdependence over time
e.g., family functioning is influenced not only by parents’
current social support & work stress, but also by previous
levels of social support & work stress
e.g., current family functioning in turn influences both present
and future levels of adolescent behavior problems
6
Social-Interactional Component
Risk and protection are expressed in the patterns of
direct transactions between individuals within and
across the different contextual levels
e.g., when parents engage in supportive interactions with
individuals outside the family, they are more likely to parent
their children in supportive rather than harsh ways.
7
An Ecodevelopmental
Perspective on Prevention
8
Context of Adolescent Behavior Problems,
Drug Use, and Risky Sexual Behavior
Social-Cultural Context
Parental Resources/Stressors
Parents’ Social Support
Parents’ Work Stress
Cultural
Mismatch
Family Microsystem
Parent-adolescent communication about sex
Parent-Adolescent Communication
Parental Involvement
Family-School
Relations
Parental involvement in
school
Monitoring homework
School
School Bonding
Academic Achievement
Immigration
Policy
Positive Parenting
Marital Conflict
Family support
Language
Problems
Family-Peer Relations
Parental monitoring of peers
Supervision of situations of
sexual possibility
Peers
Substance use w/ friends
Sexually active friends
Poverty
9
Ecodevelopmental Model of Problem Behaviors
Social Support
for Parents
Adolescent
Acculturation
Family Exosystemic
Stressors
Parent
Acculturation
Parental Monitoring
of Peers
Parental-Adolescent
Communication about Sex
Peer Sexual
Behavior
Family Relations
Early Adolescent
Sexual Initiation
Social Cognitive
Mediators re. Sex
Early Adolescent
Substance Use
Peer Substance
Use
Early Adolescent
Problem Behaviors
Social Cognitive
Mediators re.Drug Use
10
Implications of Ecodevelopmental Theory
for Methodological Development
Ecodevelopmental research involves multiple
levels of nesting
 Repeated
Observations  Individuals  Families
 Ecosystemic levels across developmental
stages
 Statistically,
data are non-independent—
Substantively people and their social contexts are
interdependent
11
Implications of Ecodevelopmental Theory
for Methodological Development
Ecodevelopmental research involves
longitudinal processes
 Ecodevelopmental
processes are interrelated and
influence each other over time

How do we model these interrelationships over time?
 Ecodevelopmental
processes develop over time,
and a snapshot of such a process (e.g., family
functioning at baseline) is not accurate

How do we account for this in our model?
12
Implications of Ecodevelopmental Theory
for Methodological Development
Ecodevelopmental research involves
influences from multiple systems
 Ecodevelopmental
models involve two and
three level interactions
 How
do we model these moderation effects
when the variables are observed?
 How do we model these moderation effects
when the variables are latent?
13
Multiple levels of nesting in
effectiveness trials
Time is nested
within adolescents
who are nested within families
who are nested with therapist
who are nested both within treatment & site
Treatment crosses site
14
Implementing a High Fidelity
Systemic Treatment in Drug Abuse
Treatment Centers
15
Plan of Section
 Clinical
Trials Network (CTN)
 BSFT
 BSFT
Design Issues—Level of
Control
 BSFT Experience in High Fidelity
Implementation
16
Clinical Trials Network


NIDA funded network to test the effectiveness of
efficacious treatments in real world settings
As of 9/05:






Nodes = 17
States = 34+Puerto Rico
CTPs = 152
Protocols = 27 (11 closed to enrollment, 5 in development)
Currently 88 CTPs involved in 11 open studies
Mission: To implement science-based efficacious
treatments in community settings AND to show that
these implementations are an improvement over
current practice
17
Brief Strategic Family
Therapy—BSFT, CTN0014

Brief Strategic Family Therapy is a systemic,
process-focused family therapy
4
months with weekly sessions
 Up to 8 booster sessions over 8 months
 Focus on changing repetitive (maladaptive)
interactions within the family
Note that the focus is on underlying processes, not crises
 May use a crisis as a content focus, but therapy
addresses underlying “everyday” processes


BSFT has been the focus of over 30 years of
research at the Center for Family Studies
18
BSFT-Therapy Components
Three major techniques
 Joining
(Engaging Participants into Treatment)
• Balanced across all family members
• Must join with most powerful
 Diagnosis
(Family Relationships and Roles)
 Restructuring (Implementing the Treatment Plan)
• Work in Present—Enactments
• Reframing
• Shifting boundaries
19
BSFT CTN0014





8 Sites
60 participants per site on average
480 adolescent participants
Drug Use assessed monthly for 12 months
Full assessments at Baseline, 4 months, 8
months and 12 months
 Delinquent
Behaviors, Conduct Problems, Sexual
risk behaviors, Adolescent Pro-social Activities, &
Family Functioning
20
BSFT—Randomization
Participants will be randomized to BSFT
or the clinic’s standard outpatient
treatment
 Note, randomization is at the individual
level, not at the clinic level

21
Design Considerations for
Effectiveness Trials

Level of Control
 Homogeneity
of Study Population
(Szapocznik)
 Standardization & Monitoring of Treatment
(Szapocznik)


Standardization & Monitoring of Control (Feaster)
Handling of Site Variance (Feaster)
 Efficacy
study—Fixed Effect
 Effectiveness—Random Effect
22
BSFT:
Heterogeneous Treatment Population

Inclusion Criteria:
 12-17
years of age
 Any illicit drug use in last 30 days
 Lives or is expected to with “family”
 Reside in the same geographic area as CTP
 Signed consent & assent

Exclusion Criteria:
 Not
living with family (halfway house, institution, etc.)
 Suicidal or homicidal risk must be stabilized, first
 Current or pending severe criminal charges if likely
to lead to incarceration
 If already receiving drug treatment services
23
Choice for BSFT:
Full Control of Experimental
Condition

Extensive Training and Supervision
5
months of training
 Weekly supervision

Training and Supervision are
considered integral to the BSFT model
24
Multiple Levels of Clinical Supervision &
Adherence Monitoring

Clinical Supervision Weekly conference calls with a
BSFT supervisor. Supervision includes:



National Clinical Supervisor



weekly face-to-face sessions with each clinical supervisor
Regularly sits in on selected supervision calls with sites
Adherence Ratings of Videotapes



videotape review
case discussion and planning.
Randomly selected sessions
trained independent raters in Miami
Failure to adhere to the BSFT model


Definition-<70% adherence for 3 consecutive sessions
Consequences & Corrective Action



no new cases until 80% in two consecutive sessions
Increased supervision & retraining until meets criteria
If criteria not met before conclusion of current cases
withdrawn from study (at discretion of clinical supervisor)
25
Therapist Consent and
Selection Process

Identification of volunteers
 Consent
 Demographics
 Views

of Adolescent Drug Abuse – Q-sort
Selection
 Interviews
(Site PI, National Study Director/
Coordinator, and BSFT Head Training
Supervisor)
 Family Session

Randomization
 Academic
training
 Years of clinical experience
26
Training Phase for BSFT Therapists


Five-month clinical training program
Workshops
 Four
3-day workshops
Week 1 (Workshop 1) *Miami
 Week 3 (Workshop 2)
 Week 5 or 6 (Workshop 3)
 Week 13 (Workshop 4)


Supervision
 Weekly



group supervision
Each therapist will have ½ hour for videotape review
and ½ for case discussion
Pilot Cases: 2-4 cases for each therapist
Certification
27
Implementation Phase

Active Cases
 Caseload
builds over time
 Study caseload will range from 2-8 cases


(minimum = 0; maximum = 10)
Supervision
 Weekly
group supervision
Review of videotapes
 Review of clinical forms
 Treatment planning
 Each therapist (2 active) will have 30-45 minutes for
videotape review and 30-45 minutes for case
discussion

28
Adherence Ratings By Site
Node/Site
All Sites
15/301
7/802
7/302
7/604
13/701
9/200
3.5 (3.2, 3.9)
3.5 (3.2, 3.7)
-
3.4 (3.0, 3.7)
3.5 (3.3, 4.1)
3.8 (3.7, 4.0)
3.6 (3.1, 4.0)
108/130 (83.1%) 38/46 (82.6%)
-
21/26 (80.8%)
30/34 (88.2%)
4/4 (100.0%)
15/20 (75.0%)
3.8 (3.5, 4.0)
41/46 (89.1)
4.1 (4.0, 4.2)
6/6 (100.0%)
4.0 (3.8, 4.0)
16/16 (100.%)
4.0 (3.5, 4.3)
28/31 (90.3%)
3.8 (3.8, 3.8)
1/1 (100.0%)
3.7 (3.6, 4.0)
10/12 (83.3%)
4.0 (3.0, 4.0)
4.0 (3.0, 4.0)
108/112 (96.4%) 44/46 (95.7%)
4.0 (4.0, 4.0)
6/6 (100.0%)
4.0 (4.0, 4.0)
4.0 (3.0, 4.0)
16/16 (100.0%) 29/31 (93.6%)
4.0 (4.0, 4.0)
1/1 (100.0%)
4.0 (3.5, 4.0)
12/12 (100.0%)
Domain Average for Blind
Ratings
Median (25th, 75th)
% Good Adherence2
Domain Average for
Clinical Ratings
Median (25th, 75th)
% Good Adherence
3.9 (3.6, 4.0)
102/112 (91.1%)
Overall Supervisor’s
Ratings
Median (25th, 75th)
% Good Adherence
Good Adherence is  3
29
Adherence Ratings

Show adequate adherence

Some variability across sites

Supervisors’ ratings uniformly higher
than than independent raters’
30
31
Some Statistical Approaches
and
Areas for Further Research
in Systemic Modeling
and
Design of Effectiveness Trials
32
Plan of Section:
Specification of Site Effects & Bounds of
Inference
 Power analysis and Trial Planning
 Time Structure of Models & Reciprocal
Effects
 Need for Simulation Research
 Questions about Mplus & Simulation

33
Design Considerations for
Effectiveness Trials

Level of Control


Homogeneity of Study Population (Szapocznik)
Standardization & Monitoring of Treatment (Szapocznik)
 Standardization
(Feaster)

& Monitoring of Control
Handling of Site Variance (Feaster)
 Efficacy
study—Fixed Effect
 Effectiveness—Random Effect
34
Considerations for Choice of
Comparison Condition

Standardized Control Group
 Smaller
sample size (sites & participants)
 High internal validity
 Lacks ecological validity for CTPs

Treatment As Usual
 Larger
sample size (sites & participants)
 High external validity
 Highly variable across sites
 Minimum site size— within site comparisons
35
Choice for BSFT:
Treatment As Usual
Compare BSFT to the population of
treatments in the community
Currently drug abuse treatment has
considerable variability in treatment
approach and implementation
36
Handling of Site Variance


Fixed Effect (control for site & remove
variance from error): Cannot generalize
statistically
Random Effect: Can generalize effect
beyond the clinics included
 Both
site & site X treatment are random
 Implications on power of study
37
Choice for BSFT: TAU


Differences in TAU at each site implies
larger variance of the random Site X
Treatment effect
Sites need not be randomly assigned,
but need to describe the generality of
clinics (if not randomly selected)
38
Fidelity of Implementation
Variability in fidelity across sites will
increase the site and site by treatment
effects
39
Specifics of BSFT
Statistical Plan
Hypothesis 1: BSFT will be significantly
more effective than TAU in reducing
adolescent drug abuse, defined as the
percentage of days with positive drugs
use.
40
Example of Expected
Trajectories
Drug Use Over Follow-up
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
BSFT-Outpatient
0.1
TAU-Outpatient
0
T4
T8
T12
41
Secondary Hypotheses
BSFT will be significantly more effective
than TAU in :
– Reducing:
 Delinquent
behaviors & conduct problems
 Sexually risky behaviors
–
Increasing:
 Prosocial
activities (school, employment)
 Family functioning (parenting, parentadolescent communication)
42
Analysis Strategy


Randomization at the Individual Client Level
Multi-level Growth Curve with 3 levels
 Level
1 is within person—Time
 Level 2 is between persons (and incorporates
treatment assignment)
 Level 3 is between site


Controlling for baseline drug use (ANCOVAtype specification)
Note: not accounting for all levels of nesting
in primary hypotheses (therapist effects
examined in post-hoc analyses)
43
Proposed Model
Level 1—Time:
yijt   ij 0   ij1  aijt   ijt
Time, a, is centered at Month 4
Level 2—Between Adolescents:
 ij 0   0 j 0  1 j 0 BSFT  rij 0
 ij1   0 j1  1 j1BSFT  rij1
Level 3—Between Sites:
 0 j 0   000  u0 j 0
1 j 0   100  u1 j 0
 0 j1   001  u0 j1
1 j1   101  u1 j1
 000 
 100 
 001 
 101 
Grand Mean Month 4 Drug Use in TAU
Increment to Month 4 Drug Use for BSFT
Grand Mean Slope in Drug Use in TAU
Increment to Slope in Drug Use for BSFT
44
Multisite Power Analyses
(Raudenbush & Liu, 2001)
Procedure assumes single post test (not growth curve)
We have 12 monthly drug use assessments
However, drug use is not normally distributed
1.0
= 0.050
2
= 0.05
2
J= 8,n= 56, = 0.10
2
J= 8,n= 56, = 0.15
0.9
J= 8,n= 56,
0.8
0.7
P
o
w
e
r
0.6
0.5
0.4
0.3
0.2
0.1
0.20
0.40
0.60
Effect Size
0.80
1.00
45
Methodological Issues
Most research on design implications of
site effects has been done by
statisticians with drug trial experience
 Aim is to prevent site effects or justifying
ignoring them if they exist
 Therefore, little prior evidence published
on site variability & particularly on the
site by treatment interaction

46
Methodological Issues

Power Analysis
 Lack
of prior info makes difficult
 Lack of software (Raudenbush & Liu, 2001)
 Simulation in M-Plus

Have been doing simulations
 In
SAS, have difficulty with 8 sites identifying the
site & site X treatment variance in growth curve
framework
 Specifically, it may not be possible to identify the
covariance between the site and site X treatment
random components
 Simulations do not exactly match Raudenbush &
Liu
47
Resource Allocation Issue
Random site X treatment effect with
variability in Treatment as Usual
requires a large sample
 Necessity of effectiveness research if
to generalize to new clinics
 Some disagree (due in part to costs)

 Simulation
to show Type 1 error
 Look at potential mistakes of policy based on
inappropriate overly precise estimates
48
Reciprocal Effects
Person-specific cross-lagged panel
model
 Could be formulated in a latent
difference score framework
 Model is important for systemic
phenomenon like family interactions
 Illustrated within a person using coping
and distress data

49
“SETA-Fam” Hypotheses
b
SET
a
Family
Functioning
Target
Woman’s
Outcomes
d
e
c
Dotted Line Refers to
Hypotheses in SETA
Family
Members’
Outcomes
50
Within A Person
Distress 1
a
Distress 2
c
e
e
f
Coping 2
e
f
d
d
b
Distress 4
c
f
d
a
Distress 3
c
z
Coping 1
a
b
Coping 3
b
Coping 4
Test a through d as random effects
51
Within a Family
Distressj 1
a
Distressj 2
c
e
d
f
e
Distressk 2
Distressj 4
c
f
e
f
d
d
b
a
Distressj 3
c
z
Distressk1
a
b
Distressk 3
b
Distressk 4
Test a through d as random effects
52
Distress M=.61*
AR
V=.02*
Cross Lagged
w/AR Random
Distress
T1
Distress
T2
Distress
T3
.19*
.19*
.19*
.03
.03
.03
Coping
T1
Coping
T2
Coping
T3
Adj. BIC=2206.9
Coping M=.55*
AR
V=.03
Distress
T4
Coping
T4
53
How does Model Fit Compare to
Growth Curve Specification?
Distress
Baseline
.77*
Coping
Baseline
-.46*
Distress
Linear
Distress
Quadratic
V=0
.9999*
Coping
Linear
Adj. BIC=2219.5, *Difficulty with Psi
Coping V=0
Quadratic
54
Selected Issues to Address

Questions to be addressed:
 Parameter
interpretation
 Comparative Model Fit/Discrimination
 Initial Conditions
 Different time frames both within and between
individuals (data is collected on only one time
frame, though with variability in exact times of
collection)

Simulations
 Question,
is it possible to vary parameters in a
simulation within Mplus and save the parameter
settings to the generated data file?
 [AR@.3 to .8 by .1].
 AR @ 0 to 5 by .5.
55
56
57