Maximizing Potential:
Measuring What Matters
Randi Maines Walters, Ph.D., MSW, LCSW-C
Children’s Bureau, ACYF, ACF
NHSTE Symposium, May 22,2013
Greetings & Acknowledgements
Greetings From:
Joe Bock, Acting Associate Commissioner
Jane Morgan, Director of Capacity Building Division
Special Thanks To:
• Nancy Dickinson
• Freda Bernotavicz
• Robin Leake
• Mary McCarthy
5/22/2013
2
Why NHSTE Matters So Much
Measuring Performance: How Do We Know Training Is Making An Impact?
Your Contribution To The Field
Building The Evidence Base For Training That Improves Outcomes For
Children and Families
‘
.”
“THE WAY TO MEASURE THE
VALUE OF SOMETHING IS NOT BY
WHAT YOU GET OUT OF IT, IT’S
WHAT YOU BECOME FROM IT.”
Oliver Wendell Holmes
4
5/3/2013
Maximizing Whose Potential?
• The potential of vulnerable of children and families
• The potential of the people that serve vulnerable children and
families
• The collective potential of the organizations whose missions is to
develop the workforce in order to best serve children and families
“….for you will never be what you ought to be
until they are what they ought to be…”
Martin Luther King
What Matters?
Integrating Safety, Permanency, and Well-Being
Professional Development of Staff
Organizational Culture and Climate
Measuring What Matters: Safety,
Permanency & Well Being
Integrating Safety, Permanency, and Well Being
Focus On Outcomes
Using Data To Drive Outcome Performance in Key Areas
ACYF’s Priority:
INTEGRATING WELLBEING WITH SAFETY
AND PERMANENCY TO
ACHIEVE BETTER
OUTCOMES FOR
CHILDREN, YOUTH,
AND FAMILIES
WELL-BEING
SAFETY
PERMANENCY
SOCIAL AND EMOTIONAL
WELL-BEING
FOR CHILDREN, YOUTH,
AND FAMILIES
Healing and Recovery
Intensive
Intervention
Assessment drives individualized treatment plan with
evidence-based interventions
Targeted Social
and Emotional
Supports
Systematic approaches to teaching coping skills
and social skills
Stress Reducing and
Developmentally
Appropriate Environments
Safe, Supportive, and Responsive
Relationships
Knowledgeable and Effective Workforce
Nurturing environments provide security
and promote positive outcomes
Supportive, responsive
relationships
promote healing and recovery
and reinforce growing social
and emotional skills
Systems and policies promote and
sustain screening, assessment, the
use of evidence-based interventions,
progress monitoring, and continuous
quality improvement
Adapted from the Technical Assistance Center on Social Emotional Intervention for Children and the Center on the Social and Emotional Foundations for Early Learning
TITLE IV-E CHILD ABUSE AND NEGLECT
DEMONSTRATION PROJECTS
TITLE IV-E CHILD WELFARE DEMONSTRATION
PROJECTS
• HHS may waive title IV-E requirements for
States with approved projects, allowing them to
use funds flexibly and reinvest savings
• HHS prioritized well-being and addressing
trauma as the focus of the demonstrations
• States are encouraged to align screening,
assessment, and evidence-based interventions
with the needs and characteristics of the target
population in order to achieve improved wellbeing
INTEGRATING SAFETY, PERMANENCY, AND
WELL-BEING
•
Knowledge building and developing practice
– Training staff and foster parents
– Providing supports to staff to address secondary trauma
•
Validated screening & assessment
– Screening and continual functional assessment that gathers information from multiple
sources
•
Case planning and management
– Requires sensitive and responsive relationship between child and social worker, birth
parents, foster parents, etc.
•
Scaling-up of evidence-informed services
– Skilled mental health providers available
– Increasing capacity to deliver trauma-focused mental health treatment
•
Cross-system partnerships and system collaboration
– Work with Medicaid and mental health respond to trauma-informed needs being
identified
Conradi, L; et al. (2011). Promising practices and strategies for using trauma-informed child welfare practice to improve
foster care placement stability: A breakthrough series collaborative. Child Welfare. 90(6):207.
MATCHING POPULATIONS, OUTCOMES, AND APPROACHES:
IV-E DEMONSTRATION PROJECT EXAMPLES
Population
Screening
& Assessment
EBIs
Outcomes
Trauma-Focused
Cognitive Behavioral
Therapy
- Behavior problems
- PTS symptoms
- Depression
Children, 13-17
- Strengths & Difficulties
Questionnaire
- Child & Adolescent
Needs & Strengths
Multisystemic Therapy
- Delinquency/Drugs
- Peer problems
- Family cohesion
Children, 2-7
- Trauma Symptoms
Checklist for Young
Children
- Infant Toddler
Emotional Assessment
- CBCL
Parent-Child Interaction
Therapy
- Conduct disorders
- Parent distress
- Parent-child interaction
Children, 8-17
- UCLA PTSD Index
- Strengths & Difficulties
Questionnaire
- Child & Adolescent
Needs & Strengths
INTEGRATING SAFETY, PERMANENCY, AND WELLBEING FOR CHILDREN AND FAMILIES MEANS:
1. Focus on child & family level outcomes
2. Monitor progress for reduced symptoms and
improved child/youth functioning
3. Proactive approach to social and emotional needs
4. Developmentally specific approach
5. Promotion of healthy relationships
6. Build capacity to deliver EBPs
15
Measuring What Matters:
Focus On Outcomes
Standardized Risk and Safety Assessments
Caseworker Visits with Parents and Children
Placement Stability
Timeliness of Permanency
Assessment of Need and Services To Meet Those Needs
CB Reading Lists
• Leaps and Bounds with Implementation Science
– Getting To Outcomes With Abe & Co.
– Understanding Stages of Implementation and Drivers
of Change with Dean and Karen
– Illuminating Inner and Outer Context and Seeing
Coaching as a Retention Strategy with Greg
– The Science of Training and Development in
Organizations (Eduardo Salas, Scott Tannebaum,
Kurt Kraiger, and Kimberly Smith-Jentsch, 2012)
5/22/2013
17
Implementation Framework For Strategic
Improvement
• Experimenting with Integrative Workplans
• Theories of Change and Why We Can’t Do
Without Them
• Training: Necessary But Not Sufficient
5/22/2013
18
Measuring What Matters:
Professional Development of
Staff
Broadening The Perspective
From Training To Professional Development
From Training To Blended Learning Systems
From Individual Performance To Organizational Outcomes
Measuring What Matters:
Organizational Improvement
Culture and Climate
The Good, The Bad, The Ugly of The Marriage Between Bureaucracies
and People Caring-People Changing Organizations
Training Evaluation
Research On Training (Salas, et al, 2012)
• Properly Designed Training Works
• The Way Training Is Designed, Delivered and
Implemented Can Greatly Influence Its
Effectiveness
• Training is a systematic process
– What Matters Before
– What Matters During
– What Matters After Training
5/22/2013
22
Six Core Principles of
Improvement
Carnegie Foundation
Improvement Research
http://www.carnegiefoundation.org/improvement-research
Principles of Improvement
• Make the work problem specific and user
centered
• Variation in performance is the core problem to
address
• See the system that produces the current
outcomes
5/3/2013
24
Principles of Improvement
• We can not improve at scale what we cannot
measure
• Anchor practice improvement in disciplined
inquiry
• Accelerate the improvement through networked
communities
“Continuous effort—not strength or
intelligence---is the key to unlocking
potential.”
Winston Churchill
Reading List Referenced During Presentation
Aarons, G.A., Hurlburt, M., & Horwitz, S.M. (2011). Advancing a conceptual model of evidence-based
practice implementation in public service sectors. Adm Policy Ment Health 38:4-23.
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation
research: A synthesis of the literature. National Implementation Research Network
Glisson, C., & Schoenwald, S. K. (2005). The ARC organizational and community intervention strategy
for implementing evidence-based child mental health treatments. Mental Health Services Research , 7
(4), 243-260.
Wandersman, et al. (2008). Bridging the gap between prevention research and practice: The
Interactive Systems Framework for dissemination and implementation. American Journal of
Community Psychology, 41, 171-181.
Wiseman, S., Chinman, M., Ebener, P. A., Hunter, S., Imm, P., & Wandersman, A. (2007). Getting to
Outcomes: 10 steps for achieving results-based accountability. RAND Corporation.