Building Implementation Capacity
to Improve Youth Outcomes
Allison Metz, Ph.D.
Associate Director
National Implementation Research Network
University of North Carolina at Chapel Hill
Agenda
• The Challenge
• Implementation Science
• Plan for Change
• Active Implementation Frameworks
The Challenge: Recognizing the Gaps
• Science to Service Gap
– What is known to be effective is not what is selected
to help students
• Implementation Gap
– What is selected is not used with fidelity and good
outcomes
– What is used with fidelity is not sustained for a useful
period of time
– What is used with fidelity is not used on a scale
sufficient to broadly impact youth outcomes
Implementation Gap
RESEARCH
IMPLEMENTATION
PRACTICE
Implementation is defined as a specified set of
activities designed to put into practice an activity
or program of known dimensions.
Why Focus on Implementation?
RESEARCH
IMPLEMENTATION
GAP
PRACTICE
“Children, youth, and families cannot benefit from
interventions they do not experience.”
IMPLEMENTATION
INTERVENTION
Effective
Effective
NOT Effective
MarkInconsistent;
Lipsey’s 2009 Meta-analytic
Actual from
overview Not
of theSustainable;
primary factors that
characterize effective juvenile offender
Poor
Benefitsinterventions
– outcomes
“. . . in some analyses, the quality
with which the intervention is
Poor
hasoutcomes;
been as strongly
NOT Effective Unpredictable orimplemented
poor outcomes;related to Sometimes
harmful
recidivism effects
as the
type of program, so much so that
a well-implemented intervention
of an inherently less efficacious
type
can
outperform
a more on
(Institute of Medicine, 2000; 2001; 2009;
New
Freedom
Commission
efficacious
one thatinis poorly
Mental Health, 2003; National Commission
on Excellence
Education,1983; Department of Health implemented.”
and Human Services, 1999)
“Implementation Science”
“Implementation science is the systematic
study of variables and conditions that
lead to full and effective use of
evidence-based programs and other
effective innovations in typical human
service settings.”
—Blase and Fixsen, 2010
National Implementation Research Network
Download at:
http://www.fpg.unc.edu/~nirn/resources/publications/Monograph/
Best data show these methods, when used alone
Do not Result in Implementation as Intended
– Diffusion/ Dissemination of information
– Training
– Passing laws/ mandates/ regulations
– Providing funding/ incentives
– Organization change/ reorganization
5 to 10% return on investment
NECESSARY BUT NOT SUFFICIENT
Plan for Change: “Making It Happen”
Letting it happen
– Recipients are accountable
Helping it happen
– Recipients are accountable
Making it happen
– Purposeful use of implementation practice and science
– Implementation teams are accountable
—Based on Hall & Hord (1987); Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou (2004);
Fixsen, Blase, Duda, Naoom, & Van Dyke (2010)
To successfully implement and sustain evidence-based and
evidence-informed youth violence prevention interventions, we
need to know:
WHAT to do
What is the intervention (e.g.identified prevention, intervention
and enforcement strategies that are feasible and relevant)?
HOW to do it
Active and effective implementation and sustainability
frameworks (e.g. strategies to change and maintain behavior of
adults)
WHO will do it
Organized, purposeful, & active implementation support from
linked implementation teams
Q. How?
A. Effective Implementation
•
•
•
•
Changing the behavior of practitioners and administrators
Creating the setting conditions to facilitate these changes
Creating the processes to maintain and improve these
changes in both setting conditions and behavior of wellintentioned adults
So that youth benefit
Active Implementation
Frameworks
Implementation Drivers
Implementation Stages
Improvement Cycles
Implementation Teams
IMPLEMENTATION DRIVERS
Common features of successful supports to help
make full and effective use of a wide variety of
innovations
Positive Outcomes for Youth
Effective Youth Violence Prevention Practices
Staff capacity to
support youth
with the selected
practices
Institutional capacity to
support practitioners in
implementing practices
with fidelity
Core
Implementation
Components
Leadership
Capacity to provide
direction and vision
© Fixsen & Blase, 2008
Improved youth outcomes
Consistent Use of
Prevention Innovations
Interventions
meet
Implementation
Performance Assessment
(Fidelity)
Systems
Intervention
Coaching
Training
Selection
Adaptive
Integrated &
Compensatory
Technical
Facilitative
Administration
Decision Support
Data System
Leadership
© Fixsen & Blase, 2008
IMPLEMENTATION STAGES
Purposeful matching of critical implementation
activities to the stage of the process
Drivers
Drivers
Drivers
Stages AND Drivers
“DRIVERS”
EXPLORATION
Integrated &
Compensatory
Leadership Drivers
Stages of
Implementation
—Fixsen, Naoom, Blase, Friedman, & Wallace, 2005
Goals of Exploration
•
Create readiness for change
•
•
•
Changing hearts and minds
Examine degree to which the proposed
strategies and practices meet the needs
of our community and our youth
Determine whether the strategies,
practices, and implementation are
desirable and feasible
The Hexagon
An EBP Exploration Tool
The “Hexagon” can be used as a
planning tool to evaluate evidencebased programs and practices during
the Exploration Stage of
Implementation.
Download available at:
www.scalingup.org/tools-and-resources
Need in school, district, state
• Academic & socially significant Issues
• Parent & community perceptions of need
• Data indicating need
Capacity to Implement
•
•
Staff meet minimum qualifications
Able to sustain Imp Drivers
• Financially
• Structurally
Buy-in process operationalized
• Practitioners
• Families
•
NEED
Fit with current Initiatives
• School, district , state priorities
• Organizational structures
Community values
CAPACITY
FIT
EBP:
5 Point Rating Scale:
High = 5; Medium = 3; Low = 1.
Midpoints can be used and scored as a 2 or 4.
High
Med
Low
READINESS
Fit
Resources and supports for:
Readiness for Replication
•
•
•
•
•
•
Need
RESOURCES
Qualified purveyor
Expert or TA available
Mature sites to observe
Several replications
How well is it operationalized?
Are Imp Drivers operationalized?
EVIDENCE
•
•
•
•
•
•
•
Curricula & Classroom
Technology supports (IT dept.)
Staffing
Training
Data Systems
Coaching & Supervision
Administration & system
Resource Availability
Evidence
Evidence
Readiness for
Replication
Capacity to
Implement
•
•
•
•
•
•
•
Outcomes – Is it worth it?
Fidelity data
Cost – effectiveness data
Number of studies
Population similarities
Diverse cultural groups
Efficacy or Effectiveness
Total Score
© National Implementation Research Network 2009-2012
Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University
Integrated &
Compensatory
Leadership Drivers
Stages of
Implementation
—Fixsen, Naoom, Blase, Friedman, & Wallace, 2005
Goals of Installation
•
•
•
•
•
•
Structural and functional changes are
made to support implementation
Staff selection protocols developed
First ‘practitioners’ selected
Define and initiate training of first
cohort of practitioners
Develop coaching system and plans
Evaluate readiness and sustainability of
data systems (e.g. fidelity, outcomes)
INITIAL
IMPLEMENTATION
Integrated &
Compensatory
Leadership Drivers
EXPLORATION
Stages of
Implementation
Fixsen, Naoom, Blase, Friedman, & Wallace, 2005
Goals of Initial Implementation
•
•
•
•
•
Work through the Awkwardness
Provide training and coaching on the
evidence-based practice, re-organization
of school roles, functions and structures
Make use of improvement cycles to
resolve systems issues
Continue buy-in efforts and manage
expectations
All the components of the program or
innovation are at least partially in place
and the implementation supports begin
to function
FULL
IMPLEMENTATION
2-4
Years
Stages of
Implementation
Fixsen, Naoom, Blase, Friedman, & Wallace, 2005
Goals of Full Implementation
•
•
•
•
•
Maintaining and improving skills and
activities throughout the system
Components integrated, fully
functioning
Skillful practices by front line staff,
supervisors, administrators (50% meet
performance criteria)
Changes in policy are reflected in
practice at all levels
Ready to be evaluated for expected
outcomes
IMPROVEMENT CYCLES
Changing on purpose to support the new way of
work
PDSA Cycles
Improvement Cycles
Act
Plan
Study
Do
•
•
•
New practices do not fare well in existing
organizational structures and systems
Effective innovations are changed to fit the
system, as opposed to existing systems
changing to support effective innovations.
People, organizations, and systems. . .
•
Cannot change everything at once (too big;
too complex; too many of them and too few
of us)
•
Cannot stop and re-tool (have to create the
new in the midst of the existing)
•
Cannot know what to do at every step (we
will know it when we get there)
•
Many outcomes are not predictable (who
knew!?)
•
Plan-Do-Study-Act Cycles

Rapid cycle problem solving
(Shewhart; Deming)

Transformation Zone

Usability testing (Neilson; Rubin)

Practice-policy communication loops
Practice-Policy Communication Cycle
Feedback
Practice Informs Policy
Policy Enables Practices
Plan
Study - Act
External Implementation Support
Policy
Do
Practice
FORM SUPPORTS FUNCTION
Policy
Structure
Procedure
Practice
IMPLEMENTATION TEAMS
Organized, expert assistance to develop and sustain
an accountable and effective structure
Implementation Teams
• Provide accountable and effective structure to move
intervention through stages of implementation
• Scope of the initiative determines the number of teams
and the linked communication protocols needed
• Focus is on
–
–
–
–
–
Ongoing “buy-in” and readiness
Installing and sustaining the Implementation Drivers
Fidelity & Outcomes
Systems Alignment and Stage-based work
Problem-solving and sustainability
Implementation
Teams
Linked Team Structures
“We tend to focus on
snapshots of isolated parts
of the system and wonder
why our deepest problems
never seem to get solved.”
—Senge, 1990
State-based
Implementation Team
Regionally-based
Implementation Team
Community-based
Implementation Team
Site-based
Implementation Team
Implementation Teams
CORE
COMPETENCIES
Team Structure
Know the
Intervention
Know
Implementation
Know Improvement
Cycles
Know Systems
Change
Who Makes Change
Happen?
• INDICATORS
• Representative
• Accountable
SO THAT...
…can effect change
throughout the system and
make decisions
• Formal/Practice Knowledge
• Model fluency
• Fully operationalize Intervention
…can promote implementation
• Best practices for Implementation
Core components
• Stage-appropriate Work
…can guide implementation
and build capacity throughout
the organization and system
• Communicate Change
• Use of data for decision making,
problem solving and feedback loops
• Knowledge and skills for system
building and components
of core components,
adaptations, and infrastructure
…can develop and followthrough on action planning
…can support efforts to improve
access, reach or scale, improve
connection, influence decisionmaking
Examples of Success
Implementation results (higher fidelity) and intervention results
(improved outcomes) improve when Active Implementation
Frameworks are used purposefully and systematically
• Catawba County Child Wellbeing Project- 5 year period to develop, implement
and evaluate post-care service system for children exiting foster care; high fidelity
and early indicators of success
• Colorado EPIC Project- Scale-up of MI using Active Implementation Frameworks
in corrections
• EPIS Center at Penn State University - 419 age-grade cohorts over a 5-year
period: youth in CTC communities using EBPs had significantly lower rates of
delinquency, greater resistance to negative peer influence, stronger school
engagement and better academic achievement
• Evidence-Based Model Purveyors – NFP, MST, FFT use many components of
active implementation frameworks to achieve sustainable outcomes
Summary: “Making it Happen” for youth



Purposeful selection of an effective and feasible “What”
Conceptualize a change process so that effective interventions for
children and families can become embedded and sustained in socially
complex settings

“stage-matched activities” to guide the process

“implementation drivers” to build the infrastructure
Improvement processes are critical


the work is never done because the environment is in motion
Invest in the development of organized, “expert” implementation
support
Stay
Connected!
Allison.metz@unc.edu
nirn@unc.edu
nirn.fpg.unc.edu
www.scalingup.org
www.implementationconference.org