RE-EXAMINING THE ROLE OF
PROFESSIONAL DEVELOPMENT AND
TRAINING EVALUATION THROUGH
AN IMPLEMENTATION SCIENCE LENS
MICHELLE GRAEF & ROBIN LEAKE
NHSTES June 15, 2011
Presentation Overview
Children’s Bureau’s Child Welfare
Implementation Centers
 What do we mean by “implementation”?
 Overview of one Implementation Science
framework (NIRN)
 Implications of framework for training and
training evaluation

Implementation Centers and
National Resource Centers
Funded by the Children’s Bureau in 2008 to provide
States and Territories, Tribes, and Tribal Consortia
individualized training and technical assistance (T/TA)
to:




Improve child welfare administration and practice
Meet Federal performance standards and implement effective
programs
Pursue sustainable and positive systems change
Improve outcomes for children, youth & families
Implementation Centers: Filling the Gap
 States and Tribes are sometimes without the resources
necessary to implement comprehensive strategic plans
 National Resource Centers have limited resources to
provide intensive, long-term technical assistance
 Implementation Centers provide in-depth and long-term
technical assistance and peer networking opportunities to
States and Tribes
Geographic Assignments
What do we mean by
“implementation”?



The process of moving an idea from concept to
reality
“a specified set of activities designed to put into
practice an activity or program of known
dimensions” (NIRN)
Involves both:
 project
management (managing the tasks, resources,
budget, process –technical)
 change management (managing the people –
adaptive)
In other words…
Science of Implementation



There is a body of research on effective implementation,
mostly from research on evidenced-based practices
Child welfare systems have been slow to explore the
science, often using “paper implementation”
The Implementation Centers are using several models to
guide their work:
National Implementation Research Network (NIRN)
 Phases of Change (John Kotter)
 ADKAR (Awareness, Desire, Knowledge, Ability,
Reinforcement) - Prosci

National Implementation Research Network:
Implementation Drivers
Performance Assessment
Systems Intervention
Coaching
Facilitative Administration
Training
Selection
Integrated
&
Compensatory
Decision Support
Data System
LEADERSHIP
© Fixsen & Blase, 2008
Stages of Implementation
Core Activities
Exploration
• Conduct
organizational
assessment
• Garner leadership
and stakeholder
support
• Review literature
• Solicit multiple
perspectives on
needs and
opportunities
• Establish baseline
performance
• Develop
intervention model
Design/Installation
Initial Implementation
• Develop plans:
• Implementation
• Communication
• Change
management
• Define goals and
benchmarks
• Engage
stakeholders
• Conduct readiness
assessment
• Align organization
systems/supports
• Build staff
competency
• Communicate
project status
• Monitor fidelity of
implementation
• Monitor
implementation
climate
• Identify and
address barriers to
implementation
• Provide
performance
feedback to staff,
management, and
stakeholders
Adapted from Implementation Research: A Synthesis of the Literature (2008)
The National Implementation Research Network
Stages of Implementation
Core Activities (cont’)
Full
Implementation
•Apply lessons
learned from initial
implementation to
full operation
•Track fidelity
through quality
assurance and
performance
evaluation data
•Solicit feedback
from multiple
stakeholders and
consumers
•Evaluate impact on
child and family
outcomes
Innovation
Sustainability
• Adapt or adjust
intervention model
to reflect lessons
learned from
stakeholder
feedback, tracking,
evaluation
• Communicate
changes and
rationale
• Re-build
competency
around modified
model
• Establish long term
funding sources
• Align ongoing
quality assurance
and performance
measurement with
model
• Promote visibility
of new practice
and successful
outcomes
Adapted from Implementation Research: A Synthesis of the Literature (2008)
The National Implementation Research Network
So does any of this really happen
in child welfare agencies?


ICs have developed a common measure of our
implementation process, based on NIRN drivers but
includes others
Collecting data across all projects:
summarize implementation drivers that are salient,
 describe implementation activities happening over life
of projects, across multiple stages.

Intervention vs. Implementation
Outcomes


Important to differentiate between interventionlevel activity/outcomes and implementation-level
activity/outcomes (Proctor et al, 2010).
Examples of implementation outcomes:
 Acceptability
(user satisfaction)
 Adoption/uptake (utilization or intent to try)
 Feasibility (suitability for use, practical)
 Fidelity (adherence to program model)
 Penetration (spread across settings)
 Sustainability (level of institutionalization)
Implications for Role of Training


What is the role of training in the NIRN framework?
How is this different/not different from the way
training is usually conceptualized by CW agencies?
Training alone, no matter how well done,
does not lead to successful implementation
Fixsen, Naoom, Blase, Friedman, Wallace, 2005
Implications for Framework


How does overall professional development fit
within the NIRN framework?
Why is training depicted separately from coaching
in the triangle? Is there a better way to model this?
Implications for Role of Training
Evaluation



What is the role of training evaluation in this
framework?
What are the boundaries between training
evaluation and implementation evaluation? Is there
a distinction? Should there be?
What is the implication for the work that we do?
Learn More
Implementation Centers
Northeast & Caribbean Implementation
Center (NCIC)
Website
http://ncic.muskie.usm.maine.edu
Atlantic Coast Child Welfare Implementation www.accwic.org
Center (ACCWIC)
Midwest Child Welfare Implementation
Center (MCWIC)
www.mcwic.org
Mountains & Plains Child Welfare
Implementation Center (MPCWIC)
www.mpcwic.org
Western & Pacific Child Welfare
Implementation Center (WPIC)
www.wpicenter.org