Using Implementation Science
To guide the installation and
implementation of evidence-based
practice in NYC child welfare
programs
Children’s Justice Act and State Liaison Officers
Annual Meeting, April 28-29, 2014
Hyatt Regency. New Orleans, LA
PANEL
• Theresa Costello, MA, Director National Resource Center for
•
•
•
•
Child Protective Services
David Collins, LMSW, Assistant Commissioner, Program
Innovation and Planning, NYC Administration for Children's
Services (ACS)
Diane DePanfilis, PhD, MSW, Moses Distinguished Visiting
Professor, Silberman School of Social Work at Hunter College &
Professor, University of Maryland School of Social Work
Besa Bauta, LSW, MPH, Senior Director of Research and
Evaluation, Center for Evidence Based Implementation and
Research at Catholic Guardian Services
Paul Martin, LCSW-R, Director of Preventive Services, Leake and
Watts Services Inc.
2
Introductions:
Why are you here?
3
AGENDA for Session
• Purpose of NRCCPS TA in NYC
• ACS Evidence Based Practice Initiative in Child
Welfare
• Brief overview of Implementation Science
– Illustrations and conversations about how
implementation science is being used in NYC
4
A “reframe” of the Agenda through the lens of the Science
of the Positive
Copyright 2013 Jeff Linkenbach
Evidence based practice,
Implementation science
TA framework
Why is this
Important?
What & How is NYC
Implementing
Evidence based practice
In child welfare?
What does NYC expect to achieve?
5
Why is this
important?
Our SPIRIT
We want families to be
empowered to meet the basic
needs of their children and to
keep them safe without the
need for out-of-home
placement.
They/we have the best
chance of succeeding when
we implement efficacious
practices.
6
What is our science?
-TA framework, evidence based
practice, implementation science
7
A member of the National Training
& Technical Assistance Network,
a service of the Children’s Bureau,
U.S. Department of Health &
Human Services
PURPOSE of Technical
Assistance: To support
the installation &
implementation of EBP
with families at risk of
maltreatment and
placement.
8
Types of TA (examples)
• Outputs/Products:
– Alignment activities
with ACS policies
– Organizational
Assessment
– Fidelity assessment
tools
– Capacity building with
implementation teams
and workgroups
9
Creating, Selecting, and
Implementing
Evidence Based Service
Models
David Collins
Assistant Commissioner
Division of Policy, Planning, and Measurement
NYC Children’s Services
April 2014
10
About New York City’s Administration
for Children’s Services (ACS)
•
•
•
•
•
•
Responsible for child protection, child welfare services, juvenile
justice prevention and placement, and early childhood care and
education in New York City.
Approximately 6,000 direct FTEs.
Investigates 54,000 allegations of child maltreatment each year.
25,000+ annual families served in preventive; foster care
population of 11,900 (down from 49,000+ in 1991). Preventive and
foster care systems are privately contracted and use a delegated
case management approach.
Merged with the city’s Department of Juvenile Justice in 2010 and
received authority to directly oversee placement of adjudicated
youth under new legislation in 2012; also provides many evidence
based alternative to detention and placement diversion services.
Oversees the nation’s largest publicly funded child care and early
education system, serving approximately 110,000 children
annually through a mix of contracted slots and vouchers.
11
New York City’s Vision for Evidence
Based Programming
•
Goals: Building on prior experience with EBMs/PPMs, improve outcomes in juvenile justice,
preventive services, foster care and parts of protective services; expand the continuum of services
to better meet the needs of families.
•
Where EBMs and PPMs are being used in Children’s Services:
•
Juvenile Justice – Juvenile Justice Initiative ATP and IPAS; Family Assessment Program
PINS Diversion; Behavior management/change approach in facilities; aftercare services are
all EBMs/PPMs
•
Preventive – 3,185 slots serving up to 8,500 families per year; one third of all NYC child
welfare preventive slots are now EBMs or PPMs, using a total of 11 models. Believed to be
the largest and most diverse municipal EBM initiative currently in existence.
•
Foster Care – ChildSuccessNYC Pilot program combines functional trauma assessments,
caseload reductions and EBM/PPM interventions; IV-E waiver in place to bring similar
reforms online throughout regular family foster care by 2015. Also converted 126 therapeutic
foster care and residential beds to Multidimensional Treatment Family Foster Care (MTFC) in
2012 in order to prevent and/or shorten the need for institutional placements.
•
Protective Services – Solution-Based Casework for child protective staff charged with
monitoring families; also piloting in protective/diagnostic unit
12
Laying the Foundation for the NYC
EBM/PPM Expansion
JJI: Juvenile Justice Initiative; provided to adjudicated juvenile delinquent youth
FAP: Family Assessment Program; provided as an alternative to court to families seeking a Persons In Need of
Supervision (PINS) case
Teen Preventive: Provided to families to prevent a foster care placement
Choosing Models for Each
Program/Procurement Issues


Determined population and contract type (preventive; high needs foster care;
residential aftercare)
For solicitations that listed models, conducted national search to determine best fit

Models rooted in the mental health system

Models rooted in the juvenile justice system

Looked at how model addresses and supports implementation with provider
agencies: what do they assist with, what do they monitor?




Interviewed model developers
EBM open-house for ACS and providers to learn about what fits best
After selection, conducted an analysis of each model to determine
strengths/weaknesses/where we would need to add support
Discovered procurement issues with regard to naming models

Determined ACS’ “Levels of Evidence”

Determined definition of “Model Developer”

Defined what the minimum standards were for a “Model Manual”
14
System Readiness and Capacity
Building




Began with Dr. Allison Metz’s “Listening Tour”

Discovered:

Lack of knowledge of ACS staff;

Perceived misalignment between child welfare policies and the practice of EBMs;

Misalignment in the oversight of model adherence and current ACS evaluation and monitoring;

Redundancy of documentation; and

Limited internal training/technical assistance capacity
Organized 3 Task Teams – Representatives chosen by Deputy Commissioners

ACS Capacity Building

Policy and Practice Alignment

Evaluation and Monitoring
Training

Implementation Institute for ACS staff, facilitated by Dr. Allison Metz

One day overviews of all models

Targeted training for various ACS stakeholders: Family Court attorneys, Evaluation and Case
Review staff, Program Development staff, Preventive Technical Assistance, Office of Placement,
Child Protection staff
Communication – all widely disseminated

One pager on EBM

Desk Guide of all ACS Preventive models

Continuum of ACS Preventive models
15
System Readiness and Capacity
Building

Built Service Connect Instrument

Web-based structured decision making tool
23 questions about specific caregiver, child, and family characteristics.

Automatically determines their overall service level: low, moderate, high, very high.


Standards/Oversight Changes




Lessons Learned From Working With Various Model Developers




Addenda for each evidence-based and promising practice model
Monitoring tools developed for each evidence-based and promising practice model
Evaluation domains reflect what’s needed for a successful EBM (e.g., fidelity tools being used; input
from consultant, etc.)
Models in various stages of development
Focus on fidelity and fidelity instruments widely vary
Different opinions on the length of time developers will be involved vs. self-sustainability

Hard look at ability of government system to take on developer/consultant role
Meetings, meetings, meetings – Importance of “Feedback Loops”




Internal
With providers all together, usually by model
With individual providers
With developers
16
ACS Preventive Continuum
Evidence-Based, Promising Practices and Current ACS models
(See separate handout)
17
Overview of Implementation
Science
18
Why is this
important?
When we implement
efficacious practices, it is
important that we understand
how well the new strategy
is implemented to improve
the likelihood that intended
outcomes will be achieved.
19
Implementation Science
“Children and families
cannot benefit from
interventions they do not
experience”
This is called the
Implementation Gap
20
Illustrations & Conversations
• Perspectives from:
– Preventive service
program leaders
– ACS
– Technical Assistance
provider -- NRCCPS
21
What is “Implementation?”
• A specified set of activities designed to put into
practice an activity or program.
• A strategic, purposeful approach, not a one-time
event, for making a change.
• A process for bridging the gap between “what we
know” and “what we do.”
22
Implementation Science Purpose
Implementation Science provides frameworks for
successful implementation so that:
– Children and families benefit from interventions
and experience positive outcomes
– Workers are supported to learn new skills, manage
change, and identify barriers
– Organizations are responsible for creating
hospitable environments for change and supporting
workers
23
Framework used by ACS
NATIONAL IMPLEMENTATION RESEARCH
NETWORK (NIRN)
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M.
& Wallace, F. (2005). Implementation Research: A
Synthesis of the Literature. Tampa, FL: University of
South Florida, Louis de la Parte Florida Mental Health
Institute, The National Implementation Research
Network (FMHI Publication #231).
Download all or part of the monograph at:
http://www.fpg.unc.edu/~nirn/resources/detail.cfm?res
ourceID=31
24
What works to support successful implementation?
• The combination of:
– Stage-matched implementation activities
– “Drivers” or core components that promote
competency, organizational support, and
leadership
– Teams that provide organized capacity to lead and
support the change effort
25
Stages of Implementation
A 2 – 4 year cycle
Full Implementation
(Sustainability & Effectiveness)
Initial Implementation
Installation
Exploration
26
Exploration & Adoption
Explore the fit and
feasibility of available
interventions to for
preventing child
maltreatment and
placement (Purpose of
Preventive Services)
SAMPLE AREAS to Explore
• Review program
purpose/target population
• Discuss similarities &
differences between current
practice & new practice
• Discuss fidelity criteria
• Discuss installation tailoring
options
• Identify skills needed by staff
• Examine competency building
process
• Understand past research
CONVERSATION
• What factors did you
consider as you went
through the exploration
stage?
• What process was used
to make the decision to
convert to an evidence
based practice?
• Who was involved in
selecting specific
model(s)?
28
Installation
Collaborate on a process to
individualize the
implementation process
SAMPLE AREAS for
development
• Adapt screening criteria based
on target population
• Develop logic model
• Select standardized clinical
assessment measures & arrive
at core outcomes
• Operationalize fidelity criteria
and develop fidelity
assessment procedures
• Decide on model for training
and ongoing
consultation/coaching
A Conversation
• What process was
used to develop a
logic model?
• How were
standardized
assessment
instruments
selected?
30
Areas of Assessment (Instruments)
BASELINE ONLY (used for
assessment but not to measure
change over time)
• DEPRESSIVE SYMPTOMS
(Center for Epidemiologic Studies
– CES-D)
• ALCOHOL/DRUG PROBLEM
SCREENING (CAGE-AID)
• DOMESTIC VIOLENCE
SCREENING (ACS SCREEN)
• READINESS FOR CHANGE
(Readiness to Change Index –
REDI)
• PARENTING ATTITUDES
(Adult-Adolescent Parenting
Inventory – AAPI-2)
• SOCIAL SUPPORT (Support
Functions Scale- SFS)
• FAMILY RESOURCES (Family
Resource Scale – FRS)
• FAMILY FUNCTIONING
(Family Functioning Style ScaleFFSS)
• PARENTING SRESS (Parenting
Stress Index-Short Form – PSI-SF)
• HOME SAFETY-STABILITY
(Developed for FCC)
31
NYC - FCC Logic Model
Inputs
ACS
Funding
Intermediate Outputs
Final
Intake/Outreach/Engagement
Outputs
Emergency Assistance
(initial & ongoing)
Trained
Staff &
Leadership
Teams
Implementation
Planning
Outcome Driven Service Plans
(SMART goals) (6-8 weeks)
Change Focused Intervention
-Minimum 1 hour per week
change focused intervention
-Advocacy/service facilitation
Evaluation of Change/
Case Closure(90 days post Plan)
Long-Term
Outcomes
Increase Protective Factors
Comprehensive Family
Assessment (4-6 weeks)
Eligibility
Criteria and
Referral
Procedures
Short-Term/
Intermediate
Outcomes
1,536
Target
Families
• Parenting Attitudes
• Family Strengths/Functioning
• Social Support
• Family Resources
• Home Safety & Stability
Decrease Risk Factors
•
•
•
•
Parental Stress
Caregiver Risks/Needs (FASP)
Child Risks/Needs (FASP)
Family Risks/Needs (FASP)
Increase child
Safety
(Prevent
Child
Maltreatment)
Achieve
Permanency
(Prevent
Placement)
32
Developing the Plan to Build
Competency
33
Overall Objectives for Building Competency for
NYC Family Connections’ Practice
• To build knowledge
and skills related to
the core components
of Family Connections.
• To practice working
with “sample” families
starting at Intake and
ending at Case
Closure.
34
Overview of Initial Training and Coaching Plan
(N=42)
(N=122)
Initial
Orientation
April 9 & 12
(N=103)
Supervisors
and
Directors
Training
April 22-23 &
29-30
Phase 1 Core
Training
(intake and
engagement)
May 1 or
May 3
(N=103)
(N=104)
Phase 2 Core
Training
Phase 3 Core Training
(family
assessment/service
planning)
(change focused
intervention; evaluating
change; closure decision
making)
May 20-21 or May
30-31
July 9 & 10 or July 11
& 12
_________________________________________________________________________
April
May
June
July
Practicum 1
Practicum 2
Practice
Introducing FC
to Current
Families
5/6 – 5/29
Coaching
As needed
Practice conducting
comprehensive family
assessment including
use of instruments;
practice developing a
FC Case plan.
Onsite Coaching
June
Continued
Practicum
Coaching with
Supervisors –
ongoing on at least
a monthly basis
35
Coaching is unlocking a
person’s potential to maximize their
own performance.
It is helping them to learn
rather than teaching them ”
- John Whitmore, 2002
36
Initial Implementation
(Current Stage)
Support ongoing
implementation with
fidelity, provide additional
training and technical
assistance as directed by
fidelity assessments
SAMPLE AREAS of focus
• Provide monthly consultation,
coaching, technical assistance
• Revise self-assessment fidelity
assessment instruments after
initial fidelity assessment
• Review agency self-assessment
fidelity assessment instruments
(every six months)
• Conduct on-site fidelity reviews (2
per year)
• Tailor ongoing technical
assistance/training based on
results of fidelity reviews
Sample Fidelity
Assessment Tool
38
Full Implementation
What we have to look forward to
• Learning is integrated
– *The project becomes the practice
•
•
•
•
•
•
•
•
Local capacity is built for ongoing training and coaching
Staff feel confident in using the practice with every family
Supervisors continually support case planners
Stakeholders adapted to practice
Procedures/processes are routine
Practice change is observable (effective)
Practice change is now the standard
Implementation drivers are sustained
39
Implementation Drivers
The core components that will increase the effectiveness of implementation
40
From Stages to Drivers
• Drivers are implementation
activities/components that support successful
implementation.
41
Implementation Drivers
Performance Assessment (Fidelity)
Systems Intervention
Coaching (and
Supervision)
Facilitative Administration
Training
Staff
Selection
Integrated
& Compensatory
Decision Support Data
System
LEADERSHIP
© Fixsen & Blase, 2008
42
Implementation Drivers
Competency Drivers
Competency Drivers are
mechanisms that help to
develop, improve, and
sustain one’s ability to
implement an intervention
with fidelity and benefits to
consumers.
Competency Drivers include:
Selection, Training, &
Coaching leading to
performance that meets
fidelity
Performance Assessment
(Fidelity)
Coaching and
Supervision
Training
Staff
Selection
Integrated
&
Compensatory
43
Organization Drivers
Organization Drivers are
mechanisms to create and
sustain hospitable
organizational and systems
environments for effective
services.
Implementation Drivers
Performance Assessment
(Fidelity)
Systems Intervention
Organization Drivers include:
Decision Support Data System,
Facilitative Administration, and
Systems Intervention
*Performance assessment still
included because all of these
influence how well the practice
is implemented
Facilitative
Administration
Integrated
&
Compensatory
Decision Support
Data System
44
NYC FCC Organizational Assessment – Overview and
Purpose
• Purpose of initial NYC FCC Assessment:
1.
2.
Assess the READINESS of agencies and staff across participating agencies to
engage in the implementation of FC; and,
Examine individual and organizational level factors that may SUPPORT or
IMPEDE the successful implementation of FC.
• Results are being used to inform the FCC implementation
leadership team and individual agency teams
– To respond to obvious worries or barriers for making the change
– To ensure everything is in place to support this change in practice
» Using the results is an example of the decision support data system driver
45
Work History from Organizational Assessment
Work History of NYC FCC Respondents
M
SD
Min
Max
Current Position
3.82
4.46
0.25
24.00
With Agency
6.23
7.27
0.33
36.00
Field of Child Welfare
11.75
8.26
0.0
35.17
46
Sample Philosophical Principles and Past Practice
Extent to Which FC Philosophical Principles Drive Current Practice
Helping Alliance
Strengths-Based
Empowerment Approach
Cultural Differences
60.0%
46.3%
47.9%
50.0%
40.4%
40.0%
41.5%
55.8%
47.4%
43.2%
32.6%
30.0%
20.0%
10.0%
1.1%
1.1%
1.1%
1.1%
10.5%
10.6% 10.5%
7.4%
1.1% 1.1%
0.0%
Not at all
Slight extent
Moderate extent
Great extent
Very great extent
47
Building Organizational
Implementation Drivers
Focus on Systems Intervention
48
Definition
• The systems intervention driver includes
strategies for working with external systems to
ensure the availability of the financial,
organizational, and human resources required
to support implementation of the initiative
(Fixsen et al., 2009)
• One element of implementing this driver is
establishing FCC Community Advisory
Committees
49
Purpose of the NYC FCC Community Advisory Committees
• To advise on the implementation of FC programs as
well as to develop support to sustain the program on
a long-term basis if effectiveness is demonstrated.
– Committee members assist in recruitment strategies to
directly identify families, community needs, and resources
for both clients and the program, as well as help to remedy
barriers to program implementation.
– The committee directly participates in activities to build
external stakeholder support and linkages on behalf of
families who receive services from the program (e.g., may
advise on newsletters for families and/or community
providers).
50
51
52
Leadership Drivers
Leadership Drivers are
methods to manage
Technical problems where
there is high levels of
agreement about problems
and high levels of certainty
about solutions and to
constructively deal with
Adaptive challenges
where problems are not
clear and solutions are
elusive
Implementation Drivers
Integrated
&
Compensatory
LEADERSHIP
53
Implementation Drivers
Integrated and
Compensatory
• Not a linear process
• Drivers overlap and
interact with each
other
• Drivers may be more or
less salient at different
stages
• Strength of one driver
can help to compensate
for weakness in
another
Performance Assessment
(Fidelity)
Coaching and
Systems Intervention
Supervision
Facilitative
Administration
Training
Staff
Selection
Integrated
&
Compensatory
Decision Support
Data System
LEADERSHIP
54
Questions/
Reflections?