Using CANS Data to Assist
in the Creation of a Learning
Collaborative to Understand
Program Effectiveness
Representatives of the Maryland Treatment Foster Care Learning
Collaborative:
Chris Kelly, LCSW-C, Child and Family Services Division, Catholic Charities
Steve Herr, PhD, LCSW-C, University of Maryland, Department of Psychiatry,
Division of Service Research
Paul Brylske, LCSW-C, Kennedy Krieger Family Center
Amy Meyerl, LCSW-C, Kennedy Krieger Family Center
Maurice Williams, LCSW, Williams Life Center
Maisha Davis, LGSW, Progressive Life Center
Debbie Marini, LCSW-C, Baptist Family Services
7th Annual National CANS Conference, May 13-15, 2011
Outline and “Takeaways”
I.
II.
III.
IV.
V.
Introduction and Formation of the MD
TFC Learning Collaborative
(MDTFCLC)
Implementation Research Parallels
The Learning Collaborative at Work
Using Data for Decision Making: The
Innovation Stage----Panel Review of
Sample Data
Future Capacity
MD TFC Learning Collaborative
 Baptist
Family and Children Services
 Catholic Charities -Center for Family
Services
 Kennedy Krieger Institute
 Kennedy Krieger Family Center
 Progressive Life Center
 Williams Life Center
 Comparable programs for comparison
MDTFCLC Demographic Profile
(N) = 326
MD TFC Learning Collaborative

Forged around a commitment to outcomes
management - Maryland TFC Coalition
 Belief that outcomes can guide and inform client,
program and system level effectiveness
 July 2009--- five private Treatment Foster Care
providers in partnership with the University
Maryland’s Children’s Outcome Management
Center (COMC) began focus on the use of data for
program and practice improvement in a “learning
collaborative”
 Use of KIDnet and CANS
A Brief Retrospective









A Commitment to Outcomes with Sumone for Kids
(MARFY Initiative - 17+ years ago)
Success and challenges of Sumone for Kids and CAFAS
Federal Science to Service Grant- partnership with local
universities to bring research and outcomes to fields like
child welfare (pilot project)- forged relationship with
COMC
Increased emphasis on accountability through outcomes
Strategic Vision (program level and state wide)
State initiatives with CANS in Group Homes
MARFY TFC Outcomes Committee partnering with
Innovations to foster discussions on Outcomes
State initiatives with CANS in TFC
Formation of MD TFC Learning Collaborative as TFC
programs began adopting the COMC system
Implementation Research






MDTFCLC found parallels in the implementation process and work
of Fixen, Naoom, Blase, Friedman and Wallace in their 2005
publication, Implementation Research: A Synthesis of the Literature
Implementation - “a specified set of activities designed to put into
practice an activity or program”
EBP’s and programs are not effective unless they are fully
implemented
“Performance implementation”---that which is effective is often
elusive
Goal is to get beyond “paper implementation”--- to create
actual change in practices that produces effects benefitting the
clients
The MDTFCLC serves as the vehicle for programs to share,
challenge and vet best practices in implementing KIDnet
Stages of Implementation






Exploration and Adoption: assess match; examine resources; make
determination to adopt (or not)
Program Installation: putting policies in place/hiring
Initial Implementation: process begins (often the stage where inertia
is lost)
Full Implementation: completely operational with all clients
Innovation: experience is gained; desirable changes that improve
program effectiveness are “innovations” and are adopted as
standard practice; fidelity adaptations are made to improve
implementation.
Sustainability: goal is continued implementation in the face of an
ever changing system
•
Fixen et al
Implementation: The Bear Den
Challenge
Implementation of Health IT
Figure 2. Behavior Change Theory for Adoption of HIT
Customized outcomes management tools, process guidelines, reports
Feasible, acceptable, valuable, and effective
enabling strategies
Consumer demand
for services
Financial
incentives
& penalties
Children’s Outcomes
Management Center
KIDnet
Outcomes
Management
Tool
Local peer
influences
Education of
stakeholders
Rules &
regulations
Feedback on
practice patterns
& outcomes
Levers for change:
Factors that affect
mental healthcare
practice
Providers:
Administrators, Program
Directors, QI Officers
Clinicians,
Supervisors
Youth &&
Youth
Families
Families
Payers & Policy
Makers
Favorable attitudes
or
intention to change
Requisite
skills
Absence of
environmental
constraints
Stakeholder groups
Systems change
outcomes: Conditions
favorable to adopting
new behaviors
Adoption
Child &
Family
Outcomes
The Learning Collaborative and
Implementation






Parallels seen in provider experience
Overcoming the Bear Den Challenge: Together, through
joint experience, we were able to anticipate and work
through barriers to counter resistance
MDTFCLC became the vehicle to review and vet
opportunities and challenges in implementation
Through a series of regular monthly meetings, the
MDTFCLC began to discuss the challenges and barriers
in implementation
Solutions and best practices emerged as the group
partnered with COMC
Changes and priorities to the system were made using a
priority and consensus matrix
MDTFC Best Practices



Initial implementation: Champions; User Groups;
Creating buy-in with Leadership; and Development of
Training for TFC parents on the CANS and KIDnet tools--created a remarkable culture shift in many
organizations
Full Implementation: Webinars and use of the priority
and consensus matrix became critical in creating
sustainable buy-in
MDTFCLC is currently in the “Innovation Stage”--making fidelity adaptations and examining data for client
and program level improvement opportunities
The Context of TFC in MD

Focus for youth and families: least restrictive
and permanent environment; access to the
family’s community; limited movement in
placement; reduction of youth entering care; and
reduction of youth in residential care.
 Implications for TFC: average age of placement
increases; increase in range and acuity of
behavior; shorter lengths of stay; greater
potential for disruption of placement.
 Set the stage for examining those youth who exit
our service to higher levels of care
Review of Aggregate Data: The
Starting Point ---ROLES & LOR
Review of Aggregate Data:
Average Length of Stay
EXPLORATORY PROCESS
“Ok---I have good data---now what?” ---common
question among providers
 Our starting point in the exploration process--(1)
What about those youth we are struggling to
serve---the 25% that go to more restrictive
settings? (2) How do we fair in relationship to
state mandated outcome areas?
 First question --- “What can we learn when
looking at CANS domain level discharge
scores?” --- led to further discussions and drill
down

Drilling Down: CANS Domains
Life Functioning Domain







Family
Living Situation
Social Development
Recreational
Developmental
Job Functioning
Legal






Medical
Physical
Sexuality
School Behavior
School Achievement
School Attendance
Chec
k
0
1
2
Check
0
1
2
3
SCHOOL BEHAVIOR Please rate the highest level from the past 30 days
Child is behaving well in school.
Child is behaving adequately in school although some behavior problems exist.
Child is having moderate behavioral problems at school. He/she is disruptive and may
have received sanctions including suspensions.
SCHOOL
Pleasesevere
rate the
highestwith
levelbehavior
from theinpast
30 days
3 BEHAVIOR
Child is having
problems
school.
He/she is frequently or severely
Child is behaving disruptive.
well in school.
School placement may be in jeopardy due to behavior.
Child is behaving adequately in school although some behavior problems exist.
Child is having moderate behavioral problems at school. He/she is disruptive and may
have received sanctions including suspensions.
Child is having severe problems with behavior in school. He/she is frequently or
severely disruptive. School placement may be in jeopardy due to behavior.
Observations at item level: Externalizing behaviors trending toward
area of need---item level score of 2 or 3.
Check
FAMILY Please rate the highest level from the past 30 days
0
1
Child is doing well in relationships with family members.
Child is doing adequately in relationships with family members although some
problems may exist. For example, some family members may have some problems
in their relationships with child.
Child is having moderate problems with parents, siblings and/or other family
members. Frequent arguing, difficulties in maintaining any positive relationship
may be observed.
Child is having severe problems with parents, siblings, and/or other family
members. This would include problems of domestic violence, constant arguing,
etc.
2
3
Child Risk Behavior





Suicide Risk
Self Mutilation
Other Self Harm
Danger to Others
Sexual Aggression





Runaway
Delinquency
Judgement
Fire Setting
Social Behavior
Chec
k
0
1
Check
0
1
2
3
DANGER TO OTHERS Please rate the highest level from the past 30 days
No evidence
History of homicidal ideation, physically harmful aggression or fire setting that has put
self or others in danger of harm.
2
Recent homicidal ideation, physically harmful aggression, or dangerous fire setting but
DANGER TO OTHERS
Please
rate the highest level from the past 30 days
not in past
24 hours.
No evidence
3
Acute homicidal ideation with a plan or physically harmful aggression OR command
History of homicidalhallucinations
ideation, physically
harmful
or fireOr,
setting
putthat
selfplaced others at
that involve
theaggression
harm of others.
childthat
sethas
a fire
or others in danger ofsignificant
harm. risk of harm.
Recent homicidal ideation, physically harmful aggression, or dangerous fire setting but not
in past 24 hours.
Acute homicidal ideation with a plan or physically harmful aggression OR command
hallucinations that involve the harm of others. Or, child set a fire that placed others at
significant risk of harm.
Observations

Both groups tend to be admitted with similar
scoring profiles…
 What is happening in the first few updates
cycles? Why do scores tend to be the same for
both groups until later in service, e.g. 4th update?
 Externalizing behaviors: school behavior,
oppositional behavior, impulsivity, risk activity
are demonstrating areas of need with this “More
Restrictive” cohort.
Observations continued
 ALOS
data suggest program decision
making may be unique at certain points in
the service delivery process. In other
words, there is no defined approach to
working with MR youth.
What’s Next?

Focus on the 25% of youth being discharged to
more restrictive settings
 More refined benchmarking: MDTFCLC
beginning to look at those programs that have
lower values in this area---what are they doing
differently?
 T-Test analysis to identify “means differences”
that are truly statistically significant between
both groups.
 Consider regression analysis to understand the
impact of key variables on discharge disposition,
e.g. age, gender, race, diagnosis.
What’s Next? continued




Consider longitudinal trajectory analysis to understand
the expected trajectory of youth with specific conditions.
Is this data a reflection of what is happening at the state
level---youth traditionally served in residential setting
transitioning to TFC?
Programs with significant wraparound services like PRP,
crisis stabilization, mentoring---are they more
successful?
Are fundamental changes in program preparation (for
example: pre-service training, trauma treatment,
substance abuse treatment and targeted recruitment)
enough to promote stabilization?
Future Capacity and Q&A
 MDTFCLC
meets the third Thursday of
every month from 10:30am - 12:30pm
 To join the mailing list please contact
meyerl@kennedykrieger.org