IMPLEMENTING
EVIDENCE-BASED PRACTICES
Prepared by the Justice Research and Statistics Association
THE IMPORTANCE OF
EVIDENCE-BASED PROGRAMS
• The evidence-based movement is one of the most
significant developments to occur in criminal and
juvenile justice in recent years.
• Evidence-based programs (EBPs) are important
because they can be expected to produce results and a
sound return on taxpayer investment.
• Funding sources are increasingly promoting
the use of EBPs.
Prepared by the Justice Research and Statistics Association
2
IMPLEMENTATION MATTERS
• Identifying and adopting an EBP is not enough to
achieve successful outcomes.
• An evidence-based program still has to be delivered
with fidelity/integrity in diverse and complex realworld settings.
Prepared by the Justice Research and Statistics Association
3
HOW ARE EBPS IDENTIFIED?
• EBPs are deemed to be effective based on rigorous
scientific research.
• While specific criteria may vary, all organizations
assess the quality and consistency of evidence when
determining program effectiveness.
Prepared by the Justice Research and Statistics Association
4
THE IMPORTANCE OF THE
PROGRAM MODEL
• Conclusions about a program’s effectiveness are
derived from evaluations of a specific set of program
components and activities, typically referred to as the
program model.
• An EBP should be implemented in a manner that is
consistent with its program model.
• Implementing an EBP in a way that deviates from the
program model changes the program and
invalidates expectations regarding
the program’s effectiveness.
Prepared by the Justice Research and Statistics Association
5
IMPLEMENTATION FIDELITY
• Implementation fidelity refers to the degree to which
a program’s real-world implementation matches the
program model.
• Research has shown that programs implemented with
a high degree of fidelity are more likely to be
successful.
• Deviating from the program model not only can
degrade program effectiveness, but also can result
in an otherwise effective program
doing more harm than good.
Prepared by the Justice Research and Statistics Association
6
ORDINARY CIRCUMSTANCE CAN BE
AN OBSTACLE TO IMPLEMENTATION
• What appears to be simple and straightforward in the
implementation process often turns out to be more
complex than anticipated.
• We often underestimate the number of steps involved,
the number of separate decisions that have to be made
or the number of participants whose preferences have
to be taken into account.
• Because of the complexity involved,
implementation, even under the best
circumstances, is exceedingly difficult.
Prepared by the Justice Research and Statistics Association
7
HIGH-QUALITY IMPLEMENTATION IS
DIFFICULT TO ACHIEVE
• A 2011 Westat study found that only about 3.5% of
all school-based programs to prevent youth substance
abuse and school crime are research-based and
well-implemented.
• A 2013 EPISCenter study found that nearly half of
the Blueprint program implementations in PA
involved adaptation, and the majority (53%) of these
adaptations were predicted to negatively
impact program effectiveness.
Prepared by the Justice Research and Statistics Association
8
IMPLEMENTATION SCIENCE
Research-based
insights and tools
that can be used
to support
high-quality
implementation
in complex
real-world settings
National Implementation
Research Network (NIRN)
Prepared by the Justice Research and Statistics Association
9
IMPLEMENTATION SCIENCE:
LESSONS LEARNED
• Implementation is a process, not an event.
• Implementation occurs in stages:
–
–
–
–
Exploration: acquisition of information
Installation: preparation, behind-the-scenes tasks
Initial implementation: small scale, trial change in practice
Full implementation: the innovation becomes accepted
practice; full staffing and client loads
• Entire process takes 2-4 years.
Prepared by the Justice Research and Statistics Association
10
TRADITIONAL STRATEGIES TO PROMOTE
EBPS ARE INSUFFICIENT TO ACHIEVE
SOUND IMPLEMENTATION
• Diffusion and dissemination of information
• Passing laws or using other mandates
• Providing funding, or using funding incentives
• Training
• Organizational development
Prepared by the Justice Research and Statistics Association
11
FACTORS THAT AFFECT
IMPLEMENTATION FIDELITY
• Complexity of the program
• Specificity of information in the program model
• Facilitation strategies available to the implementing
organization
• Responsiveness of clients and staff
Prepared by the Justice Research and Statistics Association
12
CORE PROGRAM COMPONENTS
• Core components are the parts of a program that are
vital for program success and should not be
eliminated or altered prior to or during
implementation
• When core components are known, organizations
should ensure they are implemented with the highest
degree of fidelity possible.
Prepared by the Justice Research and Statistics Association
13
ADAPTATION
• The deliberate or accidental modification of a
program
– Deletions, additions or enhancements of program
components
– Changes in the manner or intensity of administration of
program components
– Cultural and other modifications required by local
circumstances
• Excessive program adaptation is a common problem.
Prepared by the Justice Research and Statistics Association
14
ADAPTATION
• Finding the right balance between fidelity and
adaptation is difficult.
• An evidence-based program should be implemented
with the highest degree of fidelity possible.
• Adaptation is likely to be advantageous only
when it is highly strategic, pursued with
caution, and monitored to prevent
potentially harmful effects.
Prepared by the Justice Research and Statistics Association
15
SYSTEMS TRUMP PROGRAMS
• Systems are not very accommodating to innovation
and change.
• Systems will exert pressures to alter a new initiative
so it fits into existing structures with the
least amount of change.
Prepared by the Justice Research and Statistics Association
16
IMPLEMENTATION DEPENDS ON:
• Core components of implementation that facilitate
implementation fidelity
• Based on the commonalities found among
successfully implemented programs
• Integrated and compensatory in nature
Prepared by the Justice Research and Statistics Association
17
IMPLEMENTATION DRIVERS
•
•
•
•
Staff selection
Pre-service and in-service training
On-the-job coaching
Staff and program evaluation to identify problems
and guide mid-course corrections
• Facilitative administrative support to address
technical and adaptive problems
Prepared by the Justice Research and Statistics Association
18
OVERCOMING RESISTANCE
• Implementation teams
– Must have the knowledge, skill, freedom, and authority to
act within the larger organization or a collaboration of
agencies
– Actively support and facilitate implementation on a daily
basis
• Communities of practice
– Mechanisms for practitioners to interact and exchange craft
knowledge
• Practice to policy feedback loops
Prepared by the Justice Research and Statistics Association
19
ASSESSING IMPLEMENTATION
FIDELITY
• Program structure – the framework for service delivery
–
–
–
–
The presence of key program elements
Participation of requisite partner organizations
Resources allocated to the program
Staffing levels and characteristics
• Program processes - the way services are delivered
– Adherence to program protocols or procedures
– Frequency and intensity of services provided
– The quality with which program staff
delivers services
Prepared by the Justice Research and Statistics Association
20
ASSESSING IMPLEMENTATION
FIDELITY
• Context - prerequisites that must be in place for the
program to operate
– Staffing qualifications and numbers
– Practitioner-consumer ratio (i.e., caseload size)
– Completion of requisite training
• Compliance with the intervention components
prescribed in the program model
• Competence or level of skill shown by the program
staff
Prepared by the Justice Research and Statistics Association
21
ASSESSING IMPLEMENTATION
FIDELITY
• Adherence to the program model - is the program
being delivered as it was designed?
• Dosage or Exposure - do program participants receive
the amount of the intervention prescribed in the
program model?
• Quality of program delivery - do staff members
deliver the program with skill?
• Participant responsiveness - are program participants
engaged by program activities?
Prepared by the Justice Research and Statistics Association
22