Toward Efficient and
Sustainable Delivery of
Interventions: The Stages of
Implementation Completion
Lisa Saldana, PhD
Oregon Social Learning Center
Institute of Medicine:
Innovations in Design and Utilization of
Measurement Systems to Promote Children’s
Cognitive, Affective, and Behavioral Health
November 6, 2014
FUNDING
PI: Saldana
NIMH R01 MH097748
NIMH R01 MH097748-S1
NIDA K23DA021603
PI: Chamberlain
NIMH R01MH076158
NIDA R01MH076158-05S1
NIDA P50 DA035763-01
SIC/COINS TEAM
Patricia Chamberlain, PhD
John Landsverk, PhD
Jason Chapman, PhD
M ark Campbell, M S
Holle Schaper, M S
Courtenay Padgett, M S
Vanessa Ewen
Katie Lewis
Co-I
Co-I
Co- I A nalyst
Research Economist
Statistician Coordinator
Project Coordinator
Data M anager
Editorial A ssistant
Off-Site I nvestigative Team:
Sonja Schoenwald, PhD Co-I
David Bradford, PhD Co -I
Larry Palinkas, PhD Co -I
A dvisory Board:
Greg A arons, Sarah Horowitz, Lonnie Snowden, Lynne M arsenich,
IMPLEMENTATION MEASUREMENT
Growing Body of Measures Targeting
Key Aspects of Implementation
 Organizational Culture and Climate
e.g. Glisson, Aarons, Steckler
 Organizational Readiness
e.g., Simpson, Weiner, Lehman, Helfrich
 Leadership
e.g., Aarons, Kivipõld
 Attitudes Toward EBPs
e.g., Aarons, Melnyk, Upton
 Research Evidence Use
e.g., Palinkas, Stomski
MEASUREMENT GAP
 Measure of Implementation Process
Rate of Implementation
Implementation Activities
Patterns of Implementation Behavior
 Measure of Implementation Outcomes
Implementation Milestones
Penetration
CHALLENGES IN MEASURING
IMPLEMENTATION
Implementation of EBP entails extensive
planning, training, and quality assurance
Involves a complex set of interactions
between developers, system leaders,
front line staff, and consumers
Recursive process of well defined stages
or steps that are not necessarily linear
IMPLEMENTATION PROCESS HAS BEEN
ASSUMED TO BE EBP SPECIFIC
Little is known about which methods
and interactions are most important
for successful implementation
Little is known about how and if the
process influences successful
outcomes
THE STAGES OF IMPLEMENTATION
COMPLETION (SIC)
Developed as part of an
implementation trial focused on
scale-up of MTFC to fill the gap in
the lack of measures available
Developed out of necessity while
also addressing the challenges of
measuring implementation process
Chamberl ain , P., Brown, C.H., & Sal dana, L. (2011). Observ ational
measure of impl ementati on progres s : The Stages of I mpl ementati on
Compl eti on (SI C). I mplementation Science, 6, 116.
OBSERVATIONAL OPPORTUNIT Y: CAL-OH
STUDY
(PI: CHAMBERLAIN)
 53 Sites observed from Engagement - Competency
 Span 3 Phases of Implementation
Pre-Implementation, Implementation, Sustainment
 Developed a Measurement Tool to measure
Rate of Implementation (Duration)
Thoroughness of Implementation (Proportion)
 Date Driven
 Stages of Implementation Completion
8 Stages from Engagement to Competency
Activities within Stages
SIC BASICS: THE SCIENCE OF
MEASURE DEVELOPMENT
 Developed through an iterative process
 Designed to target the general process and steps
of implementation
 Involves assessment of implementation behavior
of different levels of agents
 Initially designed to assess implementation
activities specific to MTFC
STAGES OF IMPLEMENTATION COMPLETION
(SIC)
Pre
Imp
8 Stages:
1. Engagement
2. Consideration of Feasibility
3. Readiness Planning
Involvement:
System Leader
System Leader, Agency
System Leader, Agency
4. Staff Hired and Trained
Agency, Practitioner
5. Adherence Monitoring
Practitioner, Client
Established
6. Services and Consultation
Practitioner, Client
7. Ongoing Services,
Practitioner, Client
Consultation, Fidelity, Feedback
Sus 8. Competency (certification ) System Leader, Agency,
Practitioner, Client
WITHIN STAGE ACTIVITIES
1.
Engagement
D a t e a g r e e d t o c o n si d e r i m p l e m e nt a ti o n
2 . C o n si d e r a ti o n o f F e a si bi li t y
D a t e o f s t a k e h ol d e r m e e t i ng # 1
3 . R e a d i ne ss P l a nni ng
D a t e o f c o s t c a l c ul a t o r /f u ndi ng p l a n r e v i e w
4. Staff Hired and Trained
D a t e o f i ni ti a l s u p e rv i so r t r a i ni ng
5 . A d h e r e nce M o ni t o ri ng E s t a b l is he d
D a t e f i d e l i t y t e c h n ol o gy s e t - u p
6 . S e r v i c e s a n d C o n s ul t a ti on B e g i n
D a t e o f f i r st c l i e nt s e r v e d
7 . O n g o i ng S e r v i c e s, C o n s ul t a ti on , F i de l i t y , F e e d b a c k
D a t e o f I m p l e m e nt a ti o n R e v i e w # 1
8 . C o m p e t e nc y (c e r ti f i ca ti o n )
D a t e o f f i r st c e r ti f i ca ti o n a p p l i ca ti on s u b m i t t e d
YIELDS THREE SCORES
Duration
Proportion
Stage Score
SCORING AND PSYCHOMETRICS
Challenges related to challenges of measuring
implementation
Recursive Nature Means Scoring is not Linear
Not Possible to total Duration ACROSS Stages
Rasch Based Modeling Helps Account for
Challenges
MTFC-SIC: PSYCHOMETRICS
Demonstrated Reliability Using Rasch Modeling Across All 8 Stages
 A ctivity Rel iability (Proportion) = .77
 Site Reliability (Proportion) = .92
 A ctivity Rel iability (Duration) = .91
 Site Reliability (Duration) = . 67
Demonstrated Face Validity
3 distinct cl usters of sites based on Pre -Implementation Behavior
Cl uster 1: High Proportion -Relatively Fast (23 Sites)
Cl uster 2: Low Proportion -Relatively Slow (22 Sites)
Cl uster 3: Non -Completers (8 Sites)
Demonstrated Predictive Validity
 Sites that both took l onger to complete each stage and
completed fewer activities had significantly lower hazard of
successful program start -up during the study period
HR = 0.090, p < 0.001
(Cox Proportional Hazard Survival M odel)
UTILITY: REAL-WORLD SITES
 75 most real-world teams
 Sites were successfully clustered
 Failed Sites spent significantly longer in pre implementation than successful sites
 Sites that took longer to complete Stages 1 -3
significantly lower hazard of successful program
start-up
HR = 26.50, p < 0.002
(Cox Proportional Hazard Survival Model)
MTFC-SIC UTILIT Y:
REGARDLESS OF IMPLEMENTATION STRATEGY
 Reliably distinguish good from poor
performers
 Reliability distinguishes between
implementation strategies
 Meaningful prediction of implementation
milestones
 Pre-implementation SIC behavior predicts
successful program start-up
 Pre-implementation SIC behavior predicts
discontinuing program
 Pre-implementation and implementation
behavior combined predict development of
Competency (Stage 8)
17
ADAPTATION
 Can the SIC be adapted to other treatments and
service sectors?
 Will similar utility be found?
 Is there a universality in implementation?
ADDITIONAL EBPS
SCHOOLS:
 C a m p - C o p e - A l o t : C o m p ute r A s s i s te d Ve r s i o n o f C o p i n g C a t
 C B I T S ( t r a um a i n t h e s c h o o ls )
 S o u rc e o f S t r e n g t h ( s u i c id e p r ev e n t i o n )
C H I L D W E L FA R E :
 KEEP
 Safe Care
 PTC
 L i n ke d E B P s
 M T F C ( O r i g in a l )
JUVENILE JUSTICE
 MST
S U S TA N C E A B U S E / J J
 MDFT
 BSFT
Additional Services
HIV Prevention- Mujer Segura
Adult Services-Collaborative
Care
Integration of Services—
Housing First
Medical Interventions—
perinatal care
M O R E S I M I L A RI T I E S T H A N D I F F E R E N C E S I N I M P L E M E N TAT I ON AC T I V I T I E S
19
ADAPTATION PROCESS
Iterative Process
Collaborative with Developers and End Users
Retrospective Data Collection
Does the adaptation match reality?
Modification
Prospective Data Collection
Assessment of Utilization
INFORMING IMPLEMENTATION STRATEGIES
Modification to Existing Practice
UNI-SIC PSYCHOMETRICS
 Evidence of Dimensionality
Item Bi-factor suggests distinct pre -implementation,
implementation, and sustainability phases
 Site Distinction
Can distinguish 3 “types” of sites
 Items Demonstrate Order Ef fects
Easier items early on, harder items in later stages
 Reliability– Taking into account EBP, site, and phase
Indication of high reliability (0.91)
 Noisiness
Out of 45 items, 4 appear to be noisy.
These are with purpose
WHERE DO WE GO FROM HERE?
 Standardization of Measuring Implementation
Process and Milestones
 Detection of Sites that are “at risk” for
Implementation Failure and targeted intervention
 Evaluation of Implementation Strategies (e.g., CDT vs
IND)
 COINS
THANK YOU
Lisa Saldana
lisas@oslc.org