Assessment and Progress
Monitoring in School-Based
Mental Health
A collaboration between Public Health of Seattle-King Country, the Seattle
Public Schools, and the University of Washington School of Medicine/Seattle
Children’s Hospital
Aaron Lyon, PhD (UW/SCH)
Jessica Knaster Wasse, MPH (PHSKC)
Kristy Ludwig, PhD (UW/SCH)
October 16, 2012
Acknowledgments
Collaborators:
•
•
•
•
•
•
•
•
•
Ann Vander Stoep
Elizabeth McCauley
Eric Bruns
Evalynn Romano
Jane Koltracht
Kelly Thompson
Seattle Public Schools
TJ Cosgrove
School-based practitioners!
Funding:
• National Institute of Mental
Health (K08MH095939)
• Institute of Education Sciences
(R305A120128)
• American Psychological
Foundation
• Public Health of Seattle & King
County
• Bill and Melinda Gates
Foundation
Dr. Lyon is an investigator with the Implementation Research Institute (IRI), at the George Warren
Brown School of Social Work, Washington University in St. Louis; through an award from the National
Institute of Mental Health (R25MH080916-01A2) and the Department of Veterans Affairs, Health
Services Research & Development Service, Quality Enhancement Research Initiative (QUERI).
Overview
1. Background & goals of standardized
assessment / progress monitoring initiative
in Seattle’s school-based health centers
(SBHCs)
2. Public Health of Seattle & King County’s
(PHSKC) interest in / support for progress
monitoring in SBHCs
3. Current training approach / activities
4. Future directions
School-Based Health Centers (SBHCs)
• Integrated care clinics situated in schools
• Operate in nearly 2,000 schools in the US (NASBH,
2008)
• Typically provide primary care and mental health
services (Brown & Bolen, 2003)
• Well-substantiated as a mechanism to increase
service accessibility to underserved and
under/uninsured (Gance-Cleveland & Yousey, 2005; Kaplan
et al., 1999; Wade et al., 2008)
Evidence-Based Practice in SBMH
• EBP utilized inconsistently in SBMH (Evans & Weist, 2004)
• Implementation of EBP in SBMH should focus on lowcost, high-yield practice improvement targets
• Disseminate / Implement key competencies instead of full Tx
packages (Beidas et al., 2011; Rotheram-borus et al., 2012)
• Evidence-based assessment may be the “low-hanging
fruit” of EBP implementation
• School-based providers unlikely to use standardized assessment
and progress monitoring routinely
• Previous work has demonstrated good uptake of SA following
training (Lyon et al., 2011)
Evidence-Based Assessment
Evidence-Based Assessment (EBA): “Assessment
methods and processes that are based on empirical
evidence in terms of both their reliability and validity
as well as their clinical usefulness for prescribed
populations and purposes” (Mash & Hunsley, 2005, p.364)
• Evidence-based interventions commonly rely on EBA
Standardized assessment (SA), or the use of
measurement tools with empirical support for their
reliability, validity, etc, is at the core of EBA (JensenDoss & Hawley, 2005)
Value of Standardized Assessment
Initial Assessment
• Rating scales can increase the ease and accuracy of
clinical diagnosis (e.g., Jenkins et al., 2011; Youngstrom et al.,
2005)
• Psychological assessment carries positive, clinically
meaningful effects (Posten & Hanson, 2010)
Progress Monitoring
• Clinicians are often not able to detect client
deterioration (Hannan et al., 2005)
• Providing assessment results to clinicians can result
in improved outcomes (Bickman et al., 2011; Lambert et al.,
2003)
Steps of Standardized Assessment Use
1.
2.
3.
4.
5.
6.
7.
8.
Selection
Administration
Scoring
Interpretation
Feedback to clients
Treatment planning
Monitoring (over time w/ feedback)
Treatment plan adjustment
SA / Progress Monitoring: SBMH v. CMH
SBHC
(n = 16)
CMH
(n = 58)
Percent of Caseload:
0%
1-39%
0%
1-39%
Gave a SA measure in initial 1-2 meetings
47%
27%
16%
32%
Gave a SA measure at termination
63%
36%
52%
23%
Gave a SA measure
38%
44%
42%
42%
Gave feedback about a SA measure
50%
31%
46%
30%
Changed Tx plan based on SA data
69%
31%
64%
32%
Changed indiv. session plan based on SA
56%
38%
64%
27%
Progress Monitoring & Feedback
• Progress monitoring may use SA tools or ideographic
monitoring targets
• Value of ideographic targets (Weisz et al., 2011)…
• Add specificity to ID’d problems
• Focus on client concerns / priorities
• Give clients a voice
• Enhance rapport / alliance
• Provide foci for ongoing assessment
• Provide info about treatment termination
• Easy to fit into everyday practice
• Combination of SA and ideographic is likely optimal
Progress Monitoring & Feedback
• Feedback Intervention Theory (Kluget & DeNisi, 1996): Bx is
regulated by comparisons of feedback to goals
• Two levels (at least!): (1) feedback to the clinician & (2)
feedback to the client
• Most research has focused on feedback to clinicians
• Feedback is associated with higher rapport with provider,
client self-understanding (Allen et al., 2003), and increased
engagement in therapy (Ackerman et al., 2000)
Monitoring School Outcomes in Tx
• Academic outcomes are an essential domain for
youth success, but rarely included in MH
research (Atkins et al., 2010; Franklin et al., 2009; Hoagwood et
al., 2007)
• Youth w/ combined MH and academic risk have
a greater likelihood of long-term negative
outcomes (Roeser et al., 1999)
• School-based clinicians rate school attendance
as the most clinically-useful academic indicator
(Lyon, Ludwig, & McCauly, in prep)
Potential Assessment / Monitoring Targets
Mental Health
• Depression symptoms
• Disruptive behavior
• Self-injurious behavior
School Engagement
• Attendance
• Homework completion
• Class participation
• School connectedness
Academic Outcomes
• Grades
• Credits earned
• Standardized test scores
Social Functioning
• Interpersonal conflicts
• Positive social experiences
• Disciplinary events
Physical Health
• Sleep
• Diet & Exercise
Services
• Satisfaction with treatment
• Engagement in intervention
Current Goals & Activities
1. Engage stakeholders in planning /
adapting a computerized measurement
feedback system (MFS) to support SBMH
providers
2. Provide training to SBMH providers in the
use of standardized and ideographic
assessment / progress monitoring
3. Implement, evaluate, and iterate the MFS,
incorporating the ability to monitor
academic outcomes
Engaging Stakeholders in the
Adaptation of a Measurement
Feedback System for SBHCs
SBHCs in Seattle / King County
Policy/funding context
• Seattle Families & Education Levy
• Property tax levy, renewed for 4th 7 year term starting 2012
• SBHC funding managed by PHSKC
• Levy performance goals are academic in nature
• Successfully lobbied for Levy $ for MH enhancement
•
Goal: Enhance the academic impact of Levy health investments
by improving the quality of school mental health services
• Increase use of evidence-based practice, with focus on
standardized assessment and outcome monitoring
SBHCs in Seattle / King County
Vision: Collaborative stepped care model
• SA data used to support clinical decision making
• Track clinical improvement and identify patients who are not
improving as expected
Aligns with Seattle Public Schools’
Response to Intervention (RtI) strategy
Graphic from the
National Center on
Response to
Intervention
Stakeholder Committee
• “When developing an evaluation system, two of the most
challenging components are getting buy-in from all
partners involved and then selecting the measurement
tools” (Sander et al., 2011, p.15).
• Gates Foundation funding for one-year planning process
• Stakeholder committee comprised of: school-based MH
providers, supervisors, evaluators, program managers,
computer programmers, school district
Stakeholder Committee
Monthly meetings designed to:
• Build initial consensus about data fields and other
•
•
•
•
user-interface issues
Identify initial battery of standardized assessment
measures
Identify SA toolkit for various MH domains
Develop and refine implementation / evaluation plan
Enhance provider buy-in
Stakeholder Committee
Challenges/hurdles
• Differing documentation requirements
• Duplicate data entry EHR and MFS
• Varying degrees of technical knowledge
• Evaluation design: impact of randomization
Stakeholder Committee
Results/Successes
•
•
•
•
Data sharing agreement with Seattle Public Schools
Specifications completed on schedule
Negotiated acceptable evaluation plan
Providers engaged and excited
Training in Standardized
Assessment and Progress
Monitoring
Training in SA & Progress Monitoring
In promoting data-based decision making,
SBMH professionals often struggle to (Kelly,
2011)…
1. Clearly define the reason for collecting the data
2. Ensure that data-collection procedures are
manageable / user-friendly
3. Map out a clear system to use the data for continuous
improvement
Training in SA & Progress Monitoring
• Over the past 2 years, our group has provided
a training and consultation series to local
SBHC MH providers covering…
• Standardized assessment principles, psychometrics,
and normative data
• Initial assessment using standardized tools
• Feedback to students on the basis of assessment
• Identifying and tracking progress monitoring targets
Training in SA & Progress Monitoring
Principles of progress monitoring:
• Monitoring of some type is a good idea for ALL youth
receiving services
• Only track those targets that are meaningful to the
specific case/match treatment goals
• Graphical feedback increases
understanding and makes the
feedback more memorable (Kluger &
DiNisi, 1996; Miller & Watkins, 2010)
• Can be used as a tool to validate changes in a student’s
experience
Training in SA & Progress Monitoring
Moving from the WHAT to WHY…
• Progress monitoring is the first step and gives you the
WHAT
• Next steps include identifying the WHY (i.e., why is a
target changing or not changing?)
• Moving beyond just “success” or “failure”
• The WHY is important to the kid feeling successful
• Kid’s feelings of agency and control (therapy isn’t being done to
them or for them but something they are willfully and actively
participating in)
Assessment Flow Chart
SA
Student
Presentation
Assess
1) Decrease anxiety
symptoms
2) Improved school
engagement
3) Improved sleep
hygiene
Treatment
Goal ID
Ideographic
SA: MASC-10
IG: Anxiety experienced when
arriving at school?
SA: School Connectedness Scale
IG: # on-time arrivals for 1st period
SA: None?
IG: Nights/wk w/o FB or texting
after 8pm
Current Findings and Future
Evaluation Directions
Focus Groups
• Focus groups conducted to evaluate the context in which
the MFS will be implemented.
• Transcribed and evaluated using a conventional content
analysis approach.
• EMERGING THEME: Factors which facilitate or inhibit the
utility of a technology in SBMH (Lyon et al., in prep)
• Accessibility
• Impact on Youth Engagement
• Confidentiality
• Knowledge or Attitudes (Provider)
• Integration of Aspects of Care
• Relevance to MH Care
• Speed of the Technology
Pilot of the Un-adapted MFS
• Anonymous surveys of youth receiving SBMH
services supported by the un-adapted MFS (n = 34)
• 71% of youth respondents stated that answering questionnaires
about emotions made it easier to tell their providers how they
were feeling
• 55% said answering the questionnaires (and receiving feedback)
made the time spent with their SBMH providers either a little or
much easier/better.
• 45% said it did not affect their interactions (no students
reported that it made it worse)
Many Questions Remain…
Selected findings from Key Informant individual
interviews from a related project…
• Who should provide feedback about progress?
• Teachers, students, school teams
• Who should receive / review feedback?
• In addition to clinician, principal, teachers, school staff, the
academic counselor, school teams
• Allow student to determine who will review
• Other challenges
• Unique data systems and data sharing rules
• May require individualized approach district-by-district
• Providers not used to collecting/using data systematically
• Transitioning from health/MH to academic outcomes
Next Steps
1. Complete initial adaption of the MFS
2. Implement the MFS with randomly-selected providers
3. Compare provider behavior (e.g., use of standardized or
ideographic assessment/monitoring) between MFS and
non-MFS groups
4. Conduct independent evaluations of youth outcomes by
recruiting from the caseloads of MFS and non-MFS
providers
Questions and Discussion