Innovations in Prevention &
Implementation Science:
Implications for Improvement
in Schools
C Hendricks Brown, Northwestern University Feinberg School of Medicine
Sheppard Kellam, Johns Hopkins University
Jeanne Poduska, American Institutes for Research
Juan Villamar, Northwestern University Feinberg School of Medicine
Gracelyn Cruden, Northwestern University Feinberg School of Medicine
1
Acknowledgments: for Ce-PIM
• National Institute on Drug Abuse: CENTER FOR
PREVENTION IMPLEMENTATION METHODOLOGY FOR
DRUG ABUSE AND SEXUAL RISK BEHAVIOR (PI Brown
P30 DA027828)
• Substance Abuse and Mental Health Services
Administration
SAMHSA Ce-PIM Partnership
2
Acknowledgements: Work to Date
Funding Agencies:
• IES
• NIDA
• NIMH
• NICHD
3
Process in this Workshop:
Active Engagement
4
Workshop Outline
1. A Challenging Tour of Implementation Science and its
Home in Service Delivery Systems
2. Public Education and Public Health: Toward a System
of Systems
3. Systems Thinking and Partnerships in Daily Life
4. Illustrative Innovations of Ce-PIM
5. Summary: Building an Integrated Structure to
Support Children and Youth Reaching their Full
Potential
5
1. A Challenging Tour of Implementation
Science and its Home in Service Delivery
Systems
The Perspective at Ce-PIM
A. Implementation as an action strategy working on systems
change
B. Why Study Implementation?
C. Approaches to Moving Effective Interventions into Practice
D. Three Interacting and Evolving Components of
Implementation
E. A Systems Approach involving 3 interacting components
Small group exercise
6
Ce-PIM Mission: Building scientific and
methodologic rigor into implementation
research and practice.
AIMS:
1. Develop Systems Methods (Systems Science, Engineering,
Computational) Methods for Implementation Science
2. Infuse systems methods into the field of Implementation
Research
3. integrating methods within the practice of prevention
implementation at the federal, state, county, and local
levels.
7
A. What is Implementation Research?
According to NIH (2008):
The use of strategies to adopt and integrate evidence-based health
interventions and change practice patterns within and across specific
systems
Action Oriented
Within Settings or Systems
AND collects data
Chambers DA. Advancing the science of implementation: A workshop
summary. Administration and Policy in Mental Health and Mental
Health Services Research. 2008;35(1-2):3-10.
8
B. Why Study Implementation?
1. We know a lot about what works
10K reviewed studies in What Works Clearinghouse
2. We are short on implementation action strategies to put
what works into practice:
WWC Practice Guides:
Reducing Behavior Problems in the Elementary School
Classroom 2008
3. It takes too long for research to affect practice
9
Closing the Gap between What We
“Know” Works and What We Do.
“17 Year Gap” in Health Care
10
17 Years in Implementing an Statistical Methodology
Generalized Estimating Equations
(GEE: Brown et al., APMH 2012)
11
•Is the gap between research and
practice similar in education to that
existing in health?
Types of Gaps?
As long as?
As important to shorten? Which way?
As resistant to change?
12
C. Approaches to Moving Effective
Programs into Practice
13
Traditional Research Pipeline for
Implementation
Real World Relevance
Sustainment
Implementation
Implementation
Making a
Program
Work
Adoption / Preparation
Generalized knowledge
Exploration
Intervention
Local knowledge
Does a
Program
Work?
Could a
Program
Work?
Effectiveness
Studies
Efficacy
Studies
Preintervention
IOM 2009
Landsverk,
Brown et al.
2012
Aarons et al.,
2011
Traditional Translation Pipeline
14
A Focus on the Intervention itself is Not
Sufficient
•The use of effective interventions without
implementation strategies is like serum
without a syringe; the cure is available, but
the delivery system is not
Fixsen, Blase, Duda, Naoom, Van Dyke,2010.
15
D. Three Interacting and Evolving Components of
Implementation (Chambers et al., 2013 Imp Sci)
• Intervention: Program, Practice, Policy, Principles
• Practice Setting: Delivery Support System
• Ecological System: Population and Community/Cultural Context
16
Key Parameters on Implementation
Science
A. Transitions of Research Questions:
Effectiveness – Can a Program Work in real life systems?
Implementation – Making a Program Work
B. 4 Stages: Exploratory, Adoption, Implement w/ Fidelity, Sustainability
C. Focus on Interactions of Program, Delivery System, and Ecology
D. Science:
Produce Generalizable Knowledge rather than Local
Knowledge
Measurement, Modeling, and Testing Using Rigorous
Methods
17
E. Partnerships
Definition of System
A set of connected parts forming a
complex whole
18
Group Exercise: Produce a System
Map of the Educational System
Three Steps:
1. Who needs to be at the table? What are the components
needed represented?
2. How do they Interact with one another?
3. Where would you intervene to affect intervention usage?
Education Policy, i.e. Common Core Standards
Identify one person from your small group to bring this back
to the group
19
Follow-up Question on Systems:
Would your list of players and location
intervention differ based on implementing
New policy
Replacement of an existing curriculum
Removal of a expulsion policy
20
Summary of Part 1
• Multi-goal, Interacting, Dynamic and Multilevel
System
• System View Looks at Interactions between
components.
• Behavior of a system is not linear: complex.
• A useful way to understand a system is to intervene
and study its behavior.
21
2. Public Education and Public Health:
Toward a System of Systems
Sheppard G. Kellam, MD
Center for Prevention Implementation Methodology
Johns Hopkins Bloomberg School of Public Health
Society for Research on Educational Effectiveness
Washington, DC, September 4-5, 2014
The Baltimore Education and Prevention
Partnership

The Baltimore City Public School System (BCPSS) has collaborated in 3
generations of education and prevention field trials.

Trials were directed at helping children master obeying rules of behaving,
attending, academic learning, socializing appropriately in 1st grade
classroom.

Interventions were tested separately in 1st generation (our focus today),
then together in later trials.
23
Two Dimensional View of
Mental Health
• Social Adaptational Status—level of success or
failure in main social fields at each stage of life
• Psychological/physical Well-Being—includes
affect, cognitive development, i.e., internal
condition of the individual
24
Prevention Research and Service Strategies
DEVELOPMENTAL
EPIDEMIOLOGY:
COMMUNITY
PREVENTION:
directed at
early proximal targets
directed at Community &
School proximal targets
MORE IMMEDIATE RISK:
directed at
more recent
proximal targets
COMMUNITY /
SOCIETAL:
directed at
Policies & Laws as proximal
targets
25
Kellam/Langevin, 2003
Universal
Levels of Prevention
and Treatment
Selective
Indicated
Rx
Med, MH,
Soc Welfare
26
Current Problems in System
Coordination
•Prevention and treatment not integrated
•Service Agencies are Silos—minimal
collaboration
•No shared information system monitoring
developmental progress
•Mystery as to who gets “referred” for services
•Little follow-up of “no-shows”
27
Early Risk in Prevention Research
 Over the last four decades much has been learned about early
risk factors and paths leading to drug abuse, and other
behavioral, mental health, and school problems.
 Most if not all are strongly related to school failure, also a
major risk factor for later drug abuse, alcohol, tobacco,
depression, anti-social and other problem outcomes.
 Aggressive, disruptive behavior as early as 1st grade has been
repeatedly found a risk factor for later drug and alcohol abuse
and disorders, delinquency, violence, tobacco use, high risk sex,
school failure and other high risk behaviors.
28
When a child misbehaves…
29
The Baltimore Education and
Prevention Partnership
• The Baltimore City Public School System (BCPSS) has collaborated in
3 generations of education and prevention field trials.
• They were directed at helping children master key social task
demands in 1st grade classroom.
• Interventions were tested separately, then together.
• The 1st generation will be our main focus here, where the Good
Behavior Game (GBG) was tested by itself and the children were
followed up at ages 19-21 and beyond.
30
High Risk Children in Well vs.
Poorly-Managed Classrooms
(control classrooms only)
• If the top 25% of children on aggressive behavior were
in disrupted classrooms, their risk of severe aggressive
behavior by middle school was up to 59 times the
average child’s.
• If similar children were in well-managed classrooms, the
risk was up to 2.7 times the average child’s.
31
32
Kellam et al., Addiction Science & Clinical Practice; 6:73-84, 2011
The Role of School Information Systems in
our New System of Systems
• Following individual children over time, schools and
communities is required for assessing their progress,
needs and for integrating services across agencies
• Information systems tracking each child exist in most
school districts and can be extended and used to
integrate services and monitor progress
• Confidentiality is by law guaranteed by school
authorities and can be expanded to include data as
needed for services
33
Partnerships for Building and Overseeing
the New System of Systems
• Analyze political constituencies
• Engage and work through trust with each
• Learn their mission and vision
• Identify mutual self-interests and
• Encourage their coming together as oversight
committee
34
Whole Group Activity:
Partnerships
•What do we mean by “who” needs to be
included?
•The nature of the process of engaging and
developing the base of
support/collaboration.
35
3. Systems Thinking and
Partnerships in Daily Life
Jeanne Poduska, Sc.D.
American Institutes for Research
Society for Research on Educational Effectiveness
Washington, DC, September 4-5, 2014
36
Systems Concepts
• Introduction of programs and action strategies
requires adaptation to systems, e.g. staffing,
fiscal allocations
• We operate in complex systems: multi-level,
multi-actor
• Change occurs locally (even if simultaneously)
• Systems thinking focuses on interactions
37
Working Definition of Scaling-Out
“Scaling out is the deliberate use of
strategies to implement and sustain
evidence-based interventions through or
across settings to promote the greatest
public health impact”.
Ce-PIM Workgroup on Scaling Out, unpublished, 2014
38
Scaling Out Perspective on
Implementation Research
Scale Up
Generalized
knowledge
Scale Out Across Diverse Contexts
Local
knowledge
39
AIR’S mission is to conduct and apply
the best behavioral social science
research and evaluation towards
improving peoples’ lives, with a special
emphasis on the disadvantage.
40
Chambers et al, 2013 Implementation Science
41
Case Study: Good Behavior Game at AIR
American Institutes for Research
JHSPH
Baltimore
’83-’07
Prevention Science into….
Johns Hopkins: 1983-present
RCT 1: GBG | ML
RCT 2: GBG + curricula | family
AIR: 2003-2007
RCT 3: GBG + reading + family
Develop model of training and
support for GBG
Houston
‘08- present
Implementation Science
RCT 1:
-GBG: 2 models of PD
-Implement/Sustain
-”Adaptation”
Nebraska urban
’11- present
Implementation Science
Nebraska rural
’13- present
Implementation Science
Service contract: Face-to-face
training and support
Development grant: Develop
and pilot web-based tools to
deliver training to teachers
Under review: GBG for Middle
School
Service contract to develop
local GBG capacity: Face-toface training and support
Domitrovich et al., 2008; Poduska et al., 2009; Poduska et al., 2012; Poduska & Kurki, 2014
Under review: Develop and
pilot web-based tools for
coaching and on-going support
to teachers
Scaling Out Perspective on
Implementation Research
Scale Up
Generalized
knowledge
Scale Out Across Diverse Contexts
Local
knowledge
43
Who needs to be at table at AIR?
Legal
Contracts
Financial
Early Childhood Quantitative Analysts
Data Use
Mental Health Services
Professional Development
Colleagues:
Middle School
Researchers
Adolescence
Practitioners
Qualitative Analysts
Ed Tech
Social Emotional Learning
Leadership:
CEO/President
Board of Directors
Dir, Ed Program
Publication
Services
Web
Services
Human
Resources
Communications
?
?
Institutional
Review Board
(IRB)
Information
Technology
(IT)
??
Additional Systems Concepts
• Systems have developmental trajectories
• Systems are adaptive, fluid
• Adaptation occurs locally
• A note about silos: people may know one
another, talk with one another and not be
working towards a common goal
45
4. Innovations: Activities of
Ce-PIM
Juan Villamar, MS.Ed
Gracelyn Cruden, M.A.
C Hendricks Brown, Ph.D.
Center for Prevention Implementation Methodology
Department of Psychiatry and Behavioral Sciences
Northwestern University
46
General Model to Support
Implementation of Evidence Based
Intervention
Broker,
Intermediary
Brown et al, 2013 JAIDS
Implementation Agency or
Agencies
Target
7. Monitoring
Large-Scale
Implementation
Developer/
Purveyor
Funder /
Oversight
8. Modeling and
instituting
implementation
and oversight
policies
6. Intervention
R&D
4. Training,
fidelity
monitoring
and feedback
5. Improve
efficiency and
effectiveness of
implementation
delivery
2. Delivery
medium
3. Simplify and
support
intervention
delivery
1. Person,
Place, and
Time, Reach,
Engagement
47
Using a mobile application to improve fidelity of
the GBG – Rapid Development of a Prototype
48
Visualization Tool – Classroom
Disruptiveness Comparison Model
2 classrooms, 24 students
Chance that students will act
disruptively and their
disruptiveness will spread.
49
Major Gap – Sustainable Partnerships
• Partnerships between content experts and technology developers
“What do you recommend we use?” VS “Tell me what you want to see in a
report”
• Bring to the table those players that will interact with the report FIRST
• Partnerships between Research and Practice
Iterative testing with feedback loops
• Technology System, Educational System, Public Health System
Integrated System?
50
Understanding the System: Policy
• Problem: Dissemination of Evidence-Based Interventions
• Who can pay, sustainably  USPSTF
• How to influence this powerful entity?
• Understand how they came to power, to whom they report, and who is
required to listen
• Review their decision making process
• Concrete challenge: # of prevention programs in parenting that are
linked to primary care nearly 0 (despite IOM 2009 report)
• Is primary care even the best environment? What about schools, who have
a larger denominator?
Strategic Engagement
• Identify areas for involvement
• Short term –
• engage with AHRQ and USPSTF; identify brokers
• Long term –
• Scaling out research designs (primary care)
• Research design for increased knowledge utilization (ex: harmonized
measures)
• Publications in strategic journals
• Is there a similar influential agent in Education? How would you
begin to engage in the short and long term?
52
Innovation: Opening integration between
primary care and research
ODP
CMMI grants
AHRQ
Primary Care
Clinics
Schools
- Nurses
- School Wide
Programs
- Information Systems
USPSTF
NIH
53
Implementation Science can be conducted with
high rigor through:
• Measurement
• Modeling
• Testing
54
CAL-OH Head-to-Head Randomized
Implementation Trial
• Single evidence-based intervention:
• Multidimensional Treatment Foster Care (MTFC)
• 2 alternative strategies of implementing this same program
Standard implementation (Stnd)
Community Development Team (CDT)
• Randomize 51 Counties to implementation strategy*
• Evaluate implementation success using Stages of Implementation Completion
(Saldana IS 2014)
• Faster implementation
• more (# families served),
• better (fidelity)
Chamberlain P, et al. (2008). Engaging and Recruiting Counties in an Experiment on Implementing Evidence–Based Practice in
California. Administration and Policy in Mental Health and Mental Health Services Research, 35(4): 250-260.
55
Two-Arm Trials
Effectiveness vs. Implementation
Existing
Implementation
Supports for
County, Agency,
Group Home
Control
Condition
Youth
MTFC
Implementation
Supports for County,
Agency, Clinicians,
Parent
MTFC
Intervention
Standard
Implementation
Supports for County
MTFC
Intervention
CDT
Implementation
Supports for County
MTFC
Intervention
Youth
Youth
Youth
56
COHORT 1
COHORT 2
COHORT 3
COHORT 4
Randomize 51 Counties in CA and OH to
Implementation Strategy and Time (Cohort)
Randomized Roll-Out Design*
40 CA Counties
11 OH
Counties
CDT
Stnd
26 Wait
LIsted
Wait
Listed
13 Wait
LIsted
*Brown, et al. 2009 Ann Rev PH
57
Summary of Findings (Brown et al.,
under review)
• Mixed Results
Number Served Quantiles for CDT
Figure IND
4. Comparison
versus
(EQQ Plot) of Placement
• No evidence that
30
Quantiles for CDT and IND Counties
15
10
5
• CDT increased numbers
of families served
• CDT counties completed
implementation more thoroughly
0
• Evidence that
IND
20
25
• CDT affected rate of adoption
• CDT changed speed of implementation
0
5
10
15
20
25
30
CDT
58
5. Building an Integrated Structure to
Support Children and Youth Reach
their FULL Potential
Opportunities, Strengths, Next Steps
and Closing
59
1. Implementation Science involves a systematic study of
actions within and across systems
2. Requires partnerships that need to built on mutual self
interest
3. Systematic way to build such partnerships that coalesce
the interests of diverse systems, including public
education and public health
4. Implementation lives inside the organizations and
communities that are responsible for delivering or
overseeing services.
5. There are rigorous methodologic approaches to
Implementation Science that address different stages of
implementation with well define measures, modeling
approaches and research designs.
60
Contact Information:
• C Hendricks Brown,
hendricks.brown@northwestern.edu
• Sheppard G Kellam, skellam@jhu.edu
• Jeanne Poduska, jpoduska@air.org
• Juan Villamar, juan.villamar@northwestern.edu
• Gracelyn Cruden, gracelyn.cruden@northwestern.edu
Slides and References are Available:
http://cepim.northwestern.edu/products/presentatio
ns-and-workshops/
61