Making the most of what we have:
The role of evidence, the importance of context
and working together to achieve high quality
implementation to improve outcomes for
children and families.
Robyn Mildon, PhD
Director of Knowledge Exchange and Implementation
Cradle to Kinder Evaluation Proposal
2
Parenting Research Centre
- non-government, non-profit, independent research, program
development, implementation specialist and evaluation
organisation
-support the use of evidence in practice and participatory
program development and evaluation
- use evidence from the science and practice of
implementation to support effective and full implementation
of quality services for children and families
Where are we at..
The ultimate goal of services and supports in child welfare
service systems is to decrease the prevalence of child abuse
and neglect and increase child permanency, safety and
wellbeing.
Over the two decades, thinking about how to accomplish
this has involved a number of strategies..
5
Evidence-informed programs consist of collections of
practices that are performed within known parameters
(philosophy, values, service delivery structure, and
treatment components) and are put together in the form of
a structured manual containing details of procedures and
tools (Fixsen et al., 2005).
Evidence-informed practices are effective skills, techniques,
and strategies that can be used by an individual practitioner.
Such practices describe core intervention components that
have been shown to reliably produce desirable effects and
can be used individually or in combination to form more
complex procedures or programs (Embry, 2004).
Evidence to practice gap
Despite these efforts, widespread implementation with
sustainment has been difficult to achieve across human services
What is implemented often disappears with time and staff
turnover
Moving effective interventions from development and research
settings to the practice setting, with fidelity and good effect,
involves far more then making the interventions available
(Fixsen et al, 2005; Kauffman Foundation Best Practices Project;
Rogers 1995; )
Only 9% of those who
completed all
3 phases of training
followed through to
implementation.
Alone, these will not do.
Identification and cataloguing of “evidence-based” practices
and programs.
Making laws and policy directives
Providing funding
One way dissemination of information (tip sheets, websites,
lectures like this one...)
Implementation without changing supporting roles and
functions does not work
Training alone
(Embry & Biglan, 2008; Fixsen et al., 2005 )
Children and families cannot benefit from
something they do not receive
10
The get the most out of what we have..
The role of evidence (the what)
Evidence-informed interventions and/or practices
High quality implementation (the how):
Effective implementation frameworks (e.g. strategies to change
and maintain behaviour of practitioners and create hospitable
organisational systems)
The importance of context, within which the programs and
practices will be delivered (community, organisations, political
context)
Effective interventions and practices
+
Effective implementation
=
Positive outcomes
Necessary conditions
Motivation: intent to change
- perceived advantages of innovation + culture + climate +
resources + leadership + structure
Organisational capacity
- context, culture, current infrastructure, organisational
processes (sufficient staffing, leadership, well established
network)
Innovation-specific capacity
- Knowledge, skills, abilities needed for the innovation; program
champion; specific implementation climate support
Scaccia, Cook, Lamont, Wandersman, Castellow, Katz & Beidas, in press
Implementation
...the planned and intentional use of strategies to put into practice
effective innovations within real-world settings (Fixsen et al. 2005;
Mitchell, 2011).
Implementation is a process, not an event, and should be
distinguished from adoption, which is defined as the formal
decision to use an effective practice/s (Mitchell, 2011).
Implementation Science
Is the study of methods to promote the integration of
research findings and evidence into health, education and
human services
15
What we choose to do matters
The What..
Human services field is moving toward providing services
that are demonstrably effective as one way to reduce the
prevalence of most common problems and increase
wellbeing.
In the main, efforts to do this have focused on identifying
and disseminating evidence-informed programs.
What next..
Increasing the adoption and implementation
of effective programs and practice
Start with outcomes
Pay attention to the “implementability” of EIPs
Considering programs AND a common elements approach
Considering core elements and adaptable characteristics
Starting with outcomes
Outcome: describes how the target population (who) might change based upon successful
completion of your strategies.
Improved parent and child safety
Improved parental and family functioning
Promote a secure, predictable and dependable relationship between the parents/children
Increase child and parent self-efficacy
Promote responsive and warm parenting
Improved parenting coping skills
Improve parent and child problem-solving skills
Increase positive communication
Improved child and adult social connections/resources
Identify and assess best fit evidence-informed programs/practices
1.
Clarify needs of children and families the agency support.
2.
Clarify desired aims and outcomes of the program or service.
3.
Collect and summarise the evidence base.
4.
Map practice.
5.
Select and adapt knowledge to local context.
6.
Select, tailor and/or design program or Practice Model.
21
22
Programs and practices
Where we are looking to achieve change outcomes with a
particular target group, and a manualised EIP has been
shown to achieve this, we should strive to effectively
implement that EIP with high fidelity.
This will often mean changing the service setting in which
the program will be implemented (staff skill, supervision,
organisational systems and structures, service
frameworks)
Challenges with EIPs for service system
•
challenges with respect to implementation in agencies serving children and
families with complex and multiple needs.
•
EIPs can be too constraining of practitioners’ practice as a result of rigid
manualization and structure (Lyon et al, under review);
•
there are difficulties in implementing an EIPs widely with fidelity and
maintaining effectiveness (Embry & Biglan, 2008);
•
in the real-world there are often difficulties in accessing adequate training in a
variety of ESIs that have been developed (Kazdin & Blase, 2011)
•
many existing interventions that are treated as EIPs may also have limited
effectiveness, modest effect sizes, issues with scalability, limited generalisation,
and difficulties with maintaining or sustaining the program without adaptation
in child welfare service settings (Embry & Biglan, 2008).
A common elements approach
Common elements refer to the individual treatment practices that reflect
specific content (e.g., psycho-education, exposure, and rewards) that
comprise an intervention
Treatment elements are selected to match particular child and family
characteristics, allowing for practitioners to establish therapeutic
alliances, utilize clinical judgment, and still follow evidence-based
practice protocols.
Practice elements are defined by their content, not by duration,
periodicity or location within a manual.
(Chorpita, Daleiden, & Weisz, 2005a; Chorpita, Daleiden, & Weisz, 2005b)
Common elements
- recognizes the benefits to be gained by understanding more
about current practice when trying to build effective services
across contexts
- offers a genuine possibility in terms of facilitating more effective
adoption and implementation
- conductive to a high level of adaptability to context and
reciprocal adaptation between intervention and context, and
these may serve to decrease the substantial barriers to adopting
EIP’s that exist in provider attitudes, and characteristics of client
populations
Fidelity and Adaptation
One of the most critical challenges in implementation of EIPs
- tension between adaptation (i.e., flexibility) and attaining
fidelity
Adaptation.
“To make fit (as for a specific or new use or situation) often
by modification”
(Aarons et al 2012)
27
Fidelity and Adaptation
Exposure : amount of an intervention that is offered to the
participants in relation to the amount prescribed in the validated
intervention model (the number of sessions or hours of
programmed activity offered).
Adherence: extent to which the techniques/practices were
delivered according to the program developer’s specifications for
content.
Quality of delivery: pertains to practitioners/manager
performance on dimensions that are thought to enhance delivery
of the intervention (e.g., enthusiasm, style, ability to facilitate
client participation, etc.).
Implementation matters
The How
“Evidence” on effectiveness helps you select what to
implement for whom
“Evidence” on these outcomes does not help you
implement the program or practice
Fixsen & Blase (2008)
Implementation Matters (from Fixsen et al., 2005)
Implementation: The How
Intervention – The
What
Effective
Effective
Not Effective
Actual benefits
Poor outcomes
Not Effective
Inconsistent; not
sustainable; poor
outcomes
Poor outcomes;
sometimes harmful
(Institute of Medicine, 2000; 2001; 2009; New Freedom Commission on Mental
Health, 2003; National Commission on Excellence in Education,1983; Department of
Health and Human Services, 1999)
Implementation matters
Durlak & DuPre (2008) review of over 500 implementation studies in
the field of prevention and health promotion programs for children
and youth
found that the magnitude of mean effect sizes were at least 2 to 3
times higher when programs were implemented well with few or no
problems in the implementation.
full implementation of programs was associated with better
outcomes, particularly when fidelity and dosage were used to
measure levels of implementation.
Implementation matters
“. . . in some analyses, the quality with which the intervention is
implemented has been as strongly related to recidivism effects as
the type of program, so much so that a well-implemented
intervention of an inherently less efficacious type can outperform a
more efficacious one that is poorly implemented.”
Lipsey et al (2010)
Frameworks for implementation
descriptions of the implementation process which include key
attributes, facilitators, and challenges related to implementation
provide an overview of practices that guide the implementation
process and provide guidance to researchers and practitioners by
describing specific steps to include in the planning and execution
of implementation efforts
Letting it happen
– Diffusion; networking; communication
Helping it happen
– Dissemination; manuals; websites
Making it happen (Implementation Science)
– Purposeful and proactive use of implementation practice
and science
Based on Hall & Hord (1987); Greenhalgh, Robert, MacFarlane, Bate, &
Kyriakidou (2004); Fixsen, Blase, Duda, Naoom, & Van Dyke (2010)
38
Quality Implementation Framework (Meyers, Durlak and
Wandersman, in press)
National Implementation Research Network (Fixsen,
Naoom, Blase, Friedman & Wallace, 2005)
Getting to Outcomes Framework (Wandersman, Imm,
Chinman, & Kaffarian, 2000)
Quality Implementation Framework
Phase One : Initial Considerations regarding the host setting (elements)
1. Conducting a Needs and Resources Assessment
2. Conducting a Fit Assessment
3. Conducting a Capacity/ Readiness Assessment
4. Possibility for Adaptation
5. Obtaining Explicit Buy-in from Critical Stakeholders and Fostering a
Supportive Organizational Climate
6. Building General/Organisational Capacity
7. Staff recruitment/ maintenance
8. Effective Pre-innovation Staff Training
Phase Two : Creating a Structure for Implementation
9. Creating Implementation Teams:
10. Developing an implementation plan:
Phase three : Ongoing Structure Once Implementation Begins
11. Technical assistance/ Coaching/Supervision:
12. Process evaluation
13. Supportive feedback mechanism
Phase Four : Improving Futures Applications
14. Learning from experience
Knowledge to Implementation Cycle
(based on Fixsen et al 2005)
Improved outcomes for children and families
Effective implementation (fidelity)
Performance Assessment
Coaching
Systems Intervention
Training
Staff selection
Facilitative Administration
Integrated &
Compensatory
Decision Support Data System
Leadership
Technical
Adaptive
© Fixsen & Blase, 2008
Implementation drivers
Competency Drivers
• develop,
improve, and sustain ability to implement an
intervention with fidelity and benefits to consumers. Drivers
include: Selection, Training, Coaching, and Performance
Assessment
Organizational Drivers
• create
and sustain hospitable organizational and systems
environments for effective services. Drivers include: Decision
Support Data System, Facilitative Administration, and Systems
Intervention
Leadership Drivers
• methods
to manage technical problems
TOWARD AN EVIDENCE INFORMED SYSTEM FOR INNOVATION
SUPPORT
Tools +
To
Achieve
Desired
Outcomes
Current
Level of
Capacity
+
Training +
GTO Steps: (1) Needs & Resources; (2)
Goals & Desired Outcomes; (3)
Science-based practices; (4) Fit; (5)
Capacity ; (6) Plan; (7)
Implementation & Process
Evaluation; (8) Outcome evaluation;
(9) Continuous Quality Improvement;
and (10) Sustainability
QI/QA +
From Barbee et al 2011.
TA +
=
Actual
Outcomes
Achieved
Context Matters
Implementation is affected by
organisational context
Relationship between organisational support for EBP, attitudes
towards EBP and use of EBPs in practice.
Findings:
• gap
between public and private sector organisations regarding
innovation and implementation.
• private
agencies provided greater support for EBP implementation
• staff
working for private agencies reported more positive attitudes
toward adopting EBPs.
• organisational
support was significantly positively associated with
attitudes toward EBP and EBP use in practice.
Aarons et al. (2009)
Stages of Implementation
50
51
IFSS Logic Model
Population of concern
Intervention Strategies
(WHAT)
Intervention Outcomes
Children and families
IFSS workers skilfully
where neglect is present. implement the evidencebased practices within
the IFSS Practice Model
Reduction in child
neglect according to the
‘Child Neglect Index’
52
The logic model continued
Population of
concern
Children and
families where
neglect is present.
Populations
IFSS workers
Intervention Strategies (WHAT)
Intervention Outcomes
IFSS workers skilfully implement
the IFSS Practice Model
Reduction in child neglect
according to the ‘Child
Neglect Index’
Implementation Strategies (HOW)
Implementation Outcomes
Team Leaders carefully select staff
and arrange for skilful, timely
training; coaching; performance
assessments; and use data in
supportive administrative
environments
IFSS workers competently
use the IFSS Practice Model
to improve child neglect
and strengthen families
53
The logic model continued
Population of
concern
Intervention Strategies (WHAT)
Families where
child neglect is
present.
IFSS workers skilfully implement the
IFSS Practice Model
Populations
Implementation Strategies (HOW)
Intervention Outcomes
Reduction in child neglect according
to the ‘Child Neglect Index’
Implementation Outcomes
IFSS workers
Team Leaders carefully select staff and
arrange for skilful, timely training;
coaching; performance assessments; and
use data in supportive administrative
environments
IFSS workers competently use the IFSS
Practice Model to improve child neglect
and strengthen families
Supervisors/Team
Leaders
Agency managers engage with technical
assistance providers; plan for release time
for training, coaching, and ongoing
consultation services; installation of data
systems to monitor fidelity; review and
revise agency policies and procedures
Team Leaders ensure access to skilful,
timely training; coaching; performance
assessment and supportive
administrative environments for workers
implementing the IFSS Practice Model
54
Population of
concern
Intervention Strategies (WHAT)
Intervention Outcomes
The logic model continued
Families where
child neglect is
present.
Populations
IFSS workers skilfully implement the IFSS
Practice Model
Implementation Strategies (HOW)
Reduction in child neglect according to
the ‘Child Neglect Index’
Implementation Outcomes
IFSS workers
Team Leaders carefully select staff and arrange
for skilful, timely training; coaching;
performance assessments; and use data in
supportive administrative environments
IFSS workers competently use the IFSS
Practice Model to improve child
neglect and strengthen families
Supervisors /
Team Leaders
Agency managers engage with technical
assistance providers; plan for release time for
training, coaching, and ongoing consultation
services; installation of data systems to monitor
fidelity; review and revise agency policies and
procedures
Team Leaders ensure access to skilful,
timely training; coaching; performance
assessment and supportive
administrative environments for
workers implementing the IFSS
Practice Model
Provider Agency
Managers
FaHCSIA funds and supports an Implementation
Support Service for IFSS: formal mechanisms
and structures created to enhance policypractice feedback loops (CIT): streamline
reporting requirements
Agency managers engage with
technical assistance providers; plan for
release time for training, coaching,
and ongoing consultation services;
installation of data systems to55monitor
fidelity; review and revise agency
Population
of concern
Children and
families.
PRC IMPLEMENTATION SUPPORT TEAM
Populations
Intervention, Strategies and
Measures (WHAT)
Intervention Outcomes
IFSS workers skilfully implement the IFSS
Practice Model
Reduction in child neglect
according to the ‘Child Neglect
Index’
Practice, Organisational and
Systems Change Strategies (HOW)
Implementation Related
Outcomes
IFSS Workers
Team Leaders carefully select staff and provide
or arrange for skilful, timely training; coaching;
performance assessments; and use data in
supportive administrative environments
IFSS workers competently use the
IFSS Practice Model to improve child
neglect and strengthen families
Team Leaders
Provider agency managers facilitate
agreements with IFSS trainers, quality
consultants, and technical assistance
providers; plan for release time for training,
coaching, and ongoing consultation services;
manage the installation of data systems to
monitor fidelity and Service outcomes.
Team Leaders ensure access to skilful,
timely training; coaching;
performance assessment and
supportive administrative
environments for caseworkers
implementing the IFSS Practice Model
Provider Agency
Managers
FaHCSIA develops a common mission for
supporting implementation of IFSS: formal
mechanisms and structures created to
enhance policy-practice feedback loops;
streamline reporting requirements; facilitating
collaborative relationships to support system
infrastructure.
Agency leaders ensure access to
skilful, timely training, and
implementation supports to Team
Leaders: creation of hospitable
organisations for the IFSS Practice
56
Model
The logic model continued
Population of
concern
Children and
families.
Populations
Intervention, Strategies and Measures (WHAT)
Intervention Outcomes
IFSS workers skilfully implement the IFSS Practice
Model
Reduction in child neglect according to
the ‘Child Neglect Index’
Practice, Organisational and Systems Change
Strategies (HOW)
Implementation Related Outcomes
IFSS Workers
Supervisors carefully select staff and arrange for
skilful, timely training; coaching; performance
assessments; and use data in supportive
administrative environments
IFSS workers competently use the IFSS
Practice Model to improve child neglect
and strengthen families
Team Leaders
Provider agency managers facilitate agreements
with trainers, quality consultants, and technical
assistance providers; plan for release time for
training, coaching, and ongoing consultation
services; manage the installation of data systems
to monitor fidelity.
Team Leaders ensure access to skilful,
timely training; coaching; performance
assessment and supportive
administrative environments for
caseworkers implementing the IFSS
Practice Model
Provider Agency
Managers
PRC Implementation Team develops a specific
training program; provides coaches and develops
measures and a data capture system so that
provider agencies have improved knowledge, skills
and resources in the delivery of the IFSS Practice
Model
Agency leaders ensure access to skilful,
timely training, and implementation
supports to Team Leaders: and the
creation of hospitable organisations for
the IFSS Practice Model
57
58
Contact Us
Level 5, 232 Victoria Parade East Melbourne, Victoria, Australia 3002
E: info@parentingrc.org.au
P: +61 3 8660 3500
F: +61 3 8660 3599