EVIDENCE BASED PRACTICE ALONE IS NOT
ENOUGH: THE USE OF IMPLEMENTATION
SCIENCE TO ENSURE SUCCESS
Maria Scannapieco, Ph.D., Professor & Director
Center for Child Welfare
Mountains & Plains Child Welfare Implementation Center
25 April 2011
Overview
Introduction to Implementation Centers
 Basic understanding of Implementation Science
 Reflections on lessons learned in systems change
and implications for states

Introduction to the Implementation
Centers
4
Implementation Centers and National Resource Centers
Funded by the Children’s Bureau to provide States and
Territories, Tribes, and Tribal Consortia individualized
training and technical assistance (T/TA) to:




Improve child welfare administration and practice
Meet Federal performance standards and implement effective
programs
Pursue sustainable and positive systems change
Improve outcomes for children, youth & families
Implementation Centers:
Filling the Gap
5
 States and Tribes are sometimes without the resources
necessary to implement comprehensive strategic plans
 National Resource Centers have limited resources to
provide intensive, long-term TA
 Implementation Centers provide in-depth and long-term
consultation and peer networking opportunities to States
and Tribes
Project Partner Roles
Building Capacity to Implement Sustainable Systems Change
Implementation Center
Child Welfare Agency
Conduct organizational and readiness
assessment to identify potential barriers to
implementation
Develop and implement strategies to address
organizational barriers and improve
readiness for implementation
Guide planning process
Develop project plan
Provide and coordinate technical assistance
resources to support organizational change
and facilitate implementation
Provide and coordinate organizational/
human resources to manage change and
support implementation
Provide technical assistance to promote best
practices in organizational and child welfare
systems change
Utilize best practices to align people,
process, structure, measurement/rewards,
and technology around the desired systems
change
Facilitate implementation
Implement plan
Develop methodology (e.g. benchmarks,
tools) to monitor implementation progress
Collect project data; use data reports to
monitor implementation and inform /refine
implementation strategies
Evaluate project outcomes
Use findings and lessons learned to inform
Geographic Assignments
Who are the Implementation Centers?
8
Implementation Center
Organization
Northeast & Caribbean
Implementation Center (NCIC)
University of Southern Maine, Muskie School of
Public Service
Atlantic Coast Child Welfare
Implementation Center (ACCWIC)
University of Maryland School of Social Work
Midwest Child Welfare
Implementation Center (MCWIC)
University of Nebraska-Lincoln, Center on
Children, Families & the Law
Mountains & Plains Child Welfare
Implementation Center (MPCWIC)
University of Texas at Arlington, partnering with
the University of Denver and The Native
American Training Institute (NATI)
Western & Pacific Child Welfare
Implementation Center (WPIC)
American Institutes for Research, partnering with
National Indian Child Welfare Association, Center
for the Study of Social Policy, National Technical
Assistance Center for Children’s Mental Health at
Georgetown University, and the Florida Mental
Health Institute at the University of South Florida
Mountains and Plains Child Welfare Implementation Center
MPCWIC
Geographical Service Area
Region 6: Arkansas, Louisiana, New Mexico, Oklahoma, Texas
Region 8: Colorado, Montana, North Dakota, South Dakota,
Utah, Wyoming
MPCWIC Projects




The Skun-eyah (Garden) Project is a collaborative project to be implemented by
two tribes, Three Affiliated Tribes (Mandan, Hidatsa and Arikara Nations) and
Turtle Mountain Band of Chippewa
The Osage Nation implementation project goal is three-fold: develop a business
mapping model, develop and articulate a culturally based family centered practice
model, and create a decision support data system to help facilitate the first two
Colorado Department of Human Services, Division of Child Welfare– Child Welfare
Practice Reform. This three-year project will explore, define and implement a
Colorado practice model, inclusive of specific practice strategies, methods and tools
to improve outcomes for children and families. Denver, Colorado
Developing a New Mexico Child Welfare Practice Modelwill be focused on the
development and implementation of a clearly articulated practice framework,
inclusive of vision, mission, values and operating principles, to guide all of the
change initiatives underway in the State
Key Elements of Systems Change
Change Management Model
Awareness of the need for change
 Desire to participate and support the
change
 Knowledge of how to change
 Ability to implement the required
skills/behaviors
 Reinforcement to sustain the change

Competencies for Leading Change
Creativity
and innovation
External awareness
Flexibility
Strategic Thinking
Vision
Understanding Implementation
Science
Implementation is not a Hobby!



There is a body of research around effective
implementation.
Child welfare is slow to apply the science.
Implementation Centers are utilizing
methodology to support project work with
states and tribes.
What do we know?

Implementation Science
 National
Implementation Research Network
(NIRN)

Phases of Change
 John

Kotter, Harvard Business School
Managing Change
 ADKAR
Model (Awareness, Desire, Knowledge,
Ability and Reinforcement)
Implementation Frameworks


Brief overview of the science of
implementation
Practice, program and systems change
through…
 Multi-dimensional,
 Implementation
fully integrated use of
Stages
 Implementation Drivers
 Implementation Teams
Applying Frameworks

The value of frameworks is
 To
promote the ability to generalize beyond the
immediate project or initiative
 To enhance communication among partners (e.g.
better understanding of one another)
 To more easily share and apply improvements
 To increase the relevance of the “lessons learned”
Building the Implementation
Platform


Brief overview of the science of
implementation
Multi-dimensional, fully integrated
 Implementation Stages
 Implementation Drivers
 Transformation Zones
Science to Service
SCIENCE
IMPLEMENTATION
GAP
SERVICE
What Do We Mean by
Implementation?


A specified set of activities designed to put
into practice a policy, activity, or program of
known dimensions
Implementation processes are purposeful and
defined in sufficient detail such that
independent observers can detect the
presence and strength of these “specified
activities”
Overall Challenge

Science to Service Gap
 What
is known often is not what is adopted to
help children, families, and caregivers
Implementation Gap
There are not clear pathways to
implementation
What is adopted often is not used
with fidelity and good effect
What is implemented often disappears
with time and staff turnover
From the Synthesis of the Literature We Know
That…

Implementation issues are common across
widely diverse domains
 Human
service prevention and intervention (e.g.
child welfare, substance abuse, mental health,
violence prevention, education)
 Advanced manufacturing technologies
 Research-based clinical guidelines
 Engineering (e.g. bridge maintenance)
 Hotel management
 National franchise operations
 Cancer prevention and treatment
Positive Intervention Outcomes
≠ Implementation
Implementation has not been achieved by doing
more or better research on interventions or on
curricula
The usability of a program or practice has nothing to
do with the weight of the evidence regarding that
program
–“Evidence” on effectiveness helps you
select what to implement for whom
–“Evidence” on these outcomes does not
help you implement the program or
practice
Insufficient Methods



Implementation by laws/ compliance by itself does
not work
Implementation by “following the money” by itself
does not work
Implementation without changing supporting roles
and functions does not work
Fixsen, Naoom, Blase, Friedman, Wallace, 2005
Insufficient Methods

Diffusion/dissemination of information by itself
does not lead to successful implementation

Training alone, no matter how well done, does not
lead to successful implementation
Fixsen, Naoom, Blase, Friedman, Wallace, 2005
So what does work?
What Works
INTERVENTION
The WHAT
IMPLEMENTATION – The HOW
Effective
Effective
NOT Effective
Performance
Implementation
(High Fidelity)
Paper Implementation
Procedure
Implementation
(Low Fidelity)
NOT Effective
Getting Here and Staying Here is the Challenge!
Pre-Requisite: Defining “IT”

From what current state to what future state?
 The “it” must be operationalized whether it is:
 An
evidence-based practice or program
 A best practice Initiative
 A broad systems change initiative

Operationalize
Part of Speech: verb Definition: to define a concept or
variable so that it can be measured or expressed
quantitatively
Webster's New Millennium™ Dictionary of English, Preview Edition (v 0.9.7)
Copyright © 2003-2008 Lexico Publishing Group, LLC
Child Welfare Practice Classification
System

Well-Supported, Efficacious Practice
 Evidenced

Promising and Acceptable Practice
 Best


Based Practice (EBP)
Practices
Innovative or Novel Practice
Concerning Practice
Stages of Implementation
Exploration
Design/Installation
Initial
Implementation
Full
Implementation
Sustainability
Adapted from Implementation Research: A Synthesis of the Literature (2008)
The National Implementation Research Network
Stages of Implementation
Core Activities
Exploration
• Conduct
organizational
assessment
• Garner leadership
and stakeholder
support
• Review literature
• Solicit multiple
perspectives on
needs and
opportunities
• Establish baseline
performance
• Develop
intervention model
Design/Installation
Initial Implementation
• Develop plans:
• Implementation
• Communication
• Change
management
• Define goals and
benchmarks
• Engage
stakeholders
• Conduct readiness
assessment
• Align organization
systems/supports
• Build staff
competency
• Communicate
project status
• Monitor fidelity of
implementation
• Monitor
implementation
climate
• Identify and
address barriers to
implementation
• Provide
performance
feedback to staff,
management, and
stakeholders
Adapted from Implementation Research: A Synthesis of the Literature (2008)
The National Implementation Research Network
Stages/Activities
Full
Implementation
•Apply lessons
learned from initial
implementation to
full operation
•Track fidelity
through quality
assurance and
performance
evaluation data
•Solicit feedback
from multiple
stakeholders and
consumers
•Evaluate impact on
child and family
outcomes
Innovation
Sustainability
• Adapt or adjust
intervention model
to reflect lessons
learned from
stakeholder
feedback, tracking,
evaluation
• Communicate
changes and
rationale
• Re-build
competency
around modified
model
• Establish long term
funding sources
• Align ongoing
quality assurance
and performance
measurement with
model
• Promote visibility
of new practice
and successful
outcomes
Adapted from Implementation Research: A Synthesis of the Literature (2008)
The National Implementation Research Network
National Implementation Research Network:
Implementation Drivers
Performance Assessment
(Fidelity)
Systems Intervention
Coaching
Facilitative Administration
Training
Selection
Integrated
&
Compensatory
Decision Support
Data System
LEADERSHIP
© Fixsen & Blase, 2008
Improved outcomes for children
and families
Implementation
Drivers
Performance Assessment
Systems
Intervention
Coaching
Training
Selection
Adaptive
Integrated &
Compensatory
Technical
Facilitative
Administration
Decision Support
Data System
Leadership
Graphics by Steve Goodman,2009
© Fixsen & Blase, 2008
System Change


Demonstrations or “pilots” are a place to start
for innovations (“it’s possible!”)
Don’t usually lead to sustainable service and
system change
 Random
acts of innovation
 Person and passion dependent
 Can “ghost” system its way to success
 Executed by the “extraordinary”
 No replicable implementation infrastructure
Transformation Zone
Focus on innovations
 And implementation infrastructure
development
 And organizational change
 And systems change processes

Transformation
Zone
Use Innovations
Develop
Implementation
Infrastructure
Change System
Transformation Zone

A “vertical slice” of the service system (from
the front-line to the Capitol)
 The “slice” is small enough to be manageable
 The “slice” is large enough to include all
aspects of the system (court, case work,
service system, urban, rural, frontier, diverse
communities)
 The “slice” is large enough to “disturb the
system” – a “ghost” system won’t work.
Transformation Zone

Transformation Zones provide opportunities
to…
 Manage
the risks (most innovations don’t work
at first) and experience intended and unintended
outcomes
 Limit the damage (quick detection, recovery,
repair)
 Document “what works”
 Consider the implications of scaling-up
Process for Implementing Change
TA Outcomes Evaluation Questions:
Research Question
Indicator
Perceptions of increased
To what extent do
state/tribal implementation
implementation project
stakeholders view MPCWIC capacity
technical assistance as
having increased their
State/Tribal capacity to
meet project goals?
What components of NIRN
are applicable in
implementation of child
welfare system change
projects?
Stages/Drivers alignment with
implementation success
Process Measures
Implementation Capacity
Assessment
Key Stakeholder Interviews
Implementation Process
Analysis
Global Attainment Scaling
Driver Analysis
TA Outcomes Evaluation Questions:
Research Question
Indicator
Process Measures
To what extent do
implementation project
stakeholders view MPCWIC
technical assistance as
helpful in addressing
state/tribal systems
barriers?
Perceptions of increased
state/tribal implementation
capacity to address systems
barriers
Key Stakeholder Interviews
Perceptions of Stages/Drivers
To what extent do
installation related to
implementation project
stakeholders view the NIRN implementation success
framework as helpful in
addressing system barriers?
Key Stakeholder Interviews
Implementation Capacity
Assessment
Implementation Process
Analysis
MPCWIC Outcome Evaluation







Individual Implementation Project Evaluation
Do State and Tribal systems improve with respect to the issue
addressed in their individual implementation projects?
Do State and Tribal systems make significant advancements
toward the development of successful practice models?
How do State and Tribal characteristics or organizational factors
contribute to the development of successful practice models?
What is the impact of the implementation project activities and
integrations of the practice model on State or Tribal CSFR and
PIP outcomes?
Individually, do State and Tribal partners accomplish the goals
set forth in their multi-year strategic plans for sustainable
system’s change?
Are State and Tribal partners successfully able to use practice
model frameworks with fidelity to guide daily practice?
MPCWIC Outcome Evaluation






Cross- Implementation Project Outcomes
As a group, do State and Tribal partners accomplish
goals set forth in their multi-year strategic plans for
sustainable systems change?
Do organizations develop successful practice
models?
What organizational factors are associated with
successful practice models?
Do project type and organizational characteristics
differentially affect outcomes?
What is the relationship between TA intensity and
type and overall outcomes, particularily those
related to culture, climate leadership, supervision
and practice?
Summary


Stage related activities prepare the system for a
successful change process
Competence needs to be developed and sustained


Selection, training, coaching, fidelity measures help change and
support new practitioner behavior and skills
Organizations and systems need to change



Data systems need to be used to make decisions
Facilitative administrative practices & systems interventions create
hospitable environments
Policy enables new practice but practice needs to inform policy