Triple P Outcomes in California
Arizona Child Trauma Summit
April 9, 2013
Cricket Mitchell, PhD
Senior Associate, CiMH
1
Summary of Breakout Session
• Overview of California’s Triple P Outcome
Evaluation Data
– Counties supported by CiMH
• Options to consider in developing outcome
evaluation protocols
• Outcome evaluation for Arizona’s Triple P
implementation – facilitated discussion
2
What is CiMH? And How is It
Related to Triple P?
• The California Institute for Mental Health (CiMH) is
a statewide non-profit that provides training,
technical assistance, research, evaluation, and
policy support to publicly-funded agencies
– Supports the dissemination and implementation of 12
evidence-based practices
• Program performance and outcome evaluation is a critical
implementation support
• Triple P was selected for dissemination by CiMH
and promoted to county agencies in 2006
– Some agencies contract with CiMH, and some do not
3
Triple P Implementation Sites Across California
Counties
Mendocino
Alameda
Shasta
Nevada
Sonoma
Marin
San Francisco
Contra Costa
Santa Cruz
Santa Clara
San Joaquin
Ventura
Los Angeles
Riverside
Orange
San Diego
______________
Also Tri-cities Area
4
Overview of California’s Triple P
Outcome Evaluation Data
• Summer 2012 Triple P Data Submission
to CiMH
– Four Counties
•
•
•
•
Los Angeles
Shasta
Sonoma
Ventura
– 74 implementation sites
– 5,292 unique child clients served
5
Overview of California’s Triple P
Outcome Evaluation Data
• Outcome evaluation protocols within each
county vary
– Data elements collected
• Demographics
• Service delivery information
– Outcome measures used
– Applications/software used for data entry
6
Overview of California’s Triple P
Outcome Evaluation Data
• CiMH’s Program Performance and
Outcome Evaluation Reports
– Three primary domains
• Characteristics of clients served
• Description of services provided
• Outcomes achieved
– Two-Pronged Approach to Outcome Measurement
» Target-specific symptoms
» General mental health functioning
7
Overview of California’s Triple P
Outcome Evaluation Data
• Today’s presentation will highlight select
data elements from the Summer 2012
data submission
– Triple P Levels and Types
– Child Client Demographics
• Age, Gender, Ethnicity, Primary Language Spoken
in the Home, and Primary Axis I DSM-IV diagnosis
– Triple P Outcomes
• Eyberg Child Behavior Inventory (ECBI), Parenting
Scale, and Youth Outcome Questionnaire (YOQ)
8
Overview of California’s Triple P Outcome
Evaluation Data – Level and Type of Triple P
1%
12%
6%
Level 3
Level 4 Standard
Level 4 Group
Level 5 Enhanced
Level 5 Pathways
missing
81%
9
Overview of California’s Triple P Outcome
Evaluation Data – Age
• Range: .01 – 26.05 years
– Some counties serve Transition Age Youth (15-26)
• Mean: 7.7
– Standard Deviation: 4.1
• Mode: 4.0
• Frequency distribution is positively skewed
– 25th percentile: 4.6
– 50th percentile: 7.2
– 75th percentile: 10.7
10
Overview of California’s Triple P Outcome
Evaluation Data – Gender
6%
34%
Female
Male
Missing
60%
11
Overview of California’s Triple P Outcome
Evaluation Data – Ethnicity
7%
3%
1%
9%
1%
African American
24%
Asian/Pacific Islander
Caucasian
Hispanic/Latino
Native American
Other
Unknown or Missing
55%
12
Overview of California’s Triple P Outcome
Evaluation Data – Primary Language
7%
1%
English
Spanish
Other
36%
56%
Unknown or Missing
13
Overview of California’s Triple P Outcome
Evaluation Data – Primary Axis I DSM-IV Dx*
14%
31%
6%
Attention Deficit/Hyperactivity Disorders
3%
Disruptive Behavior Disorders
Mood/Anxiety/ Adjustment Disorders
Post-Traumatic Stress Disorder
Other
Missing or Not Reported
26%
20%
*Two of the four
Counties track
mental health dx
14
CiMH Outcome Indicators
• Percent Improvement
– Percent improvement from average pre-score to
average post-score
• Paired t-test conducted to examine whether or not the
difference is likely to be due to chance (p<.01); if not,
the percent change is asterisked (*) to indicate a
statistically significant improvement
• Effect Size Estimate: Cohen’s d
– A standardized measure that estimates the
magnitude, or strength, of the observed change
• Conventional interpretation: .8 ≈ “large” effect;
.5 ≈ “moderate” effect; and, .2-.3 ≈ “small” effect
15
CiMH Outcome Indicators
• Reliable Change
– The amount of change observed in an outcome
measure that can be considered an actual
change, and not likely to be due to the passage
of time or measurement error (p<.05)
• Complex statistical formula that takes the measure’s
reliability into consideration, as well as the variability
observed among scores
– Once the formula is applied, clients can be
grouped into one of three categories: reliable
positive change; reliable negative change; and,
no reliable change
16
Overview of California’s Triple P
Outcome Evaluation Data
• Target-Specific Outcome Measure Focused
on Child Disruptive Behaviors
– Eyberg Child Behavior Inventory (ECBI)
• Parent/Caregiver Report of the Intensity and
Problematic extent of child behavior problems
• 36 items
• Intensity Score Range 36 – 252
– Clinical cutpoint 131 and higher
• Problem Score Range 0 – 36
– Clinical cutpoint 15 and higher
– Used by two of the four counties
17
Overview of California’s Triple P Outcome
Evaluation Data – Outcomes: ECBI
Eyberg Child Behavior Inventory (ECBI)
Percent
Improvement from
the Average PreScore to the
Average PostScore
Effect Size
Estimate
(Cohen’s
d)
Intensity Raw
Score
28.9%*
(n=726)
[pre=135.3]
Problem Raw
Score
51.0%*
(n=744)
[pre=17.7]
Percent of Clients Showing
Reliable Change from Pre- to
PostPositive
Change
No
Change
Negative
Change
1.05
63.4%
(n=460)
31.5%
(n=229)
5.1%
(n=37)
1.14
65.5%
(n=487)
29.5%
(n=220)
5.0%
(n=37)
*A statistically significant
improvement, p < .01
18
Overview of California’s Triple P Outcome
Evaluation Data – Outcomes: ECBI Intensity
236
186
136
135
Pre
Post
96
86
36
ECBI Intensity Raw Score
(n=726)
Solid line
indicates
clinical
cutpoint
19
Overview of California’s Triple P Outcome
Evaluation Data – Outcomes: ECBI Problem
35
30
25
20
18
Pre
15
10
Post
9
5
0
ECBI Problem Raw Score
(n=744)
Solid line
indicates
clinical
cutpoint
20
Overview of California’s Triple P
Outcome Evaluation Data
• Target-Specific Outcome Measure Focused on
Parenting
– Parenting Scale
• Parent/Caregiver Report that assesses parenting and
disciplinary styles that are found to be related to the
development and/or maintenance of child disruptive
behavior problems
• 30 items
• Total Score is a mean item response ranging from 1 – 7
– Clinical cutpoint 2.8 and higher
– Used by two of the four counties
21
Overview of California’s Triple P Outcome
Evaluation Data – Outcomes: Parenting Scale
Parenting Scale
Total Score
Percent
Improvement from
the Average PreScore to the
Average PostScore
Effect Size
Estimate
(Cohen’s
d)
28.0%*
(n=154)
[pre=3.6]
1.25
Percent of Clients Showing
Reliable Change from Pre- to
PostPositive
Change
No
Change
Negative
Change
48.7%
(n=75)
49.4%
(n=76)
1.9%
(n=3)
*A statistically significant
improvement, p < .01
22
Overview of California’s Triple P Outcome
Evaluation Data – Outcomes: Parenting Scale
7.00
6.00
5.00
4.00
3.59
Pre
Post
3.00
2.58
2.00
1.00
Parenting Scale Total
(n=154)
Solid line
indicates
clinical
cutpoint
23
Overview of California’s Triple P
Outcome Evaluation Data
• General Outcome Measure of Mental Health
Functioning
– Youth Outcome Questionnaire
• Parent/Caregiver Report that assesses multiple
dimensions of child/youth mental health functioning
• 64 items
• Total Score Range -16 – 240
– Clinical cutpoint 47 and higher
– Used by one of the four counties
24
Overview of California’s Triple P Outcome
Evaluation Data – Outcomes: YOQ Total
Youth Outcome Questionnaire (YOQ)
Total Score
Percent
Improvement from
the Average PreScore to the
Average PostScore
Effect Size
Estimate
(Cohen’s
d)
36.3%*
(n=638)
[pre=63.9]
.74
Percent of Clients Showing
Reliable Change from Pre- to
PostPositive
Change
No
Change
Negative
Change
57.5%
(n=367)
34.0%
(n=217)
8.5%
(n=54)
*A statistically significant
improvement, p < .01
25
Overview of California’s Triple P Outcome
Evaluation Data – Outcomes: YOQ Total
120
100
80
64
60
Pre
41
40
Post
20
0
YOQ Total Score
(n=638)
Solid line
indicates
clinical
cutpoint
26
Overview of California’s Triple P Outcome
Evaluation Data – Outcomes: Reliable Change
ECBI Intensity
(n=726)
460
ECBI Problem
(n=744)
229
487
37
220
37
Positive Change
No Change
Negative Change
Parenting Scale
(n=154)
75
YOQ Total
(n=638)
76
367
0%
10%
20%
30%
3
217
40%
50%
60%
70%
54
80%
90%
100%
27
Overview of California’s Triple P
Outcome Evaluation Data
• Follow-up analyses of aggregate data
indicate no differences in:
– change in ECBI Intensity Score outcomes;
– change in ECBI Problem Score outcomes;
– change in Parenting Scale outcomes; or,
– change in YOQ Total Score outcomes
by gender or ethnicity
28
Options to Consider in Developing
Outcome Evaluation Protocols
• Data elements to track/collect
– Parsimony
– Utility
• Outcome measures
–
–
–
–
–
Relevance to treatment target/goals
Psychometric characteristics (valid, reliable)
Cost
Time (administration, scoring, data entry)
Training and technical assistance
29
Options to Consider in Developing
Outcome Evaluation Protocols
• Application/software used for data entry
– System already in place that can be modified?
(e.g., EHR, county- or state-level information
system)
– Cost
– Skill level to use/employ
– Utility of data elements for analysis and
reporting
– Training and technical assistance
30
Options to Consider in Developing
Outcome Evaluation Protocols
• Frequency of analysis and reporting
– Multiple stakeholders
• Different reports for different audiences
– Processes for maximizing utility of data
• Clinical utility
• Program improvement
• Systems-level decisions
Feedback is Essential
31
Additional Considerations for
Telling the Whole Story
• Collect minimal data on all clients referred
– Determine entry rate
– Determine additional need (waiting lists)
• Collect completion status (yes/no)
– Determine dropout rate
– May provide the opportunity to examine dose-response
relationships
• Track clients who are served by more than one
Level/Type of Triple P (within and across providers)
• Track population-level indicators (substantiated child
maltreatment cases, out of home placements,
emergency room visits for unexplained child injuries)
32
Evaluation for Arizona’s Triple P
Implementation – Discussion Questions
• Who should be included in decision-making?
• Is there an overarching evaluation framework?
• What data are currently being collected?
– What additional data elements are of interest?
• What outcome measures will be used?
– How will they be obtained, distributed, and used?
– Who will provide training and technical assistance?
• How will data be tracked/collected from individual
Triple P providers?
33
Evaluation for Arizona’s Triple P
Implementation – Discussion Questions
• How will population-level indicators be
tracked? With what frequency?
• Responsibility for and frequency of data
analysis and reporting?
• How will data be used to inform decisions?
– Client-level
– Program-level
– System-level
34
Discussion Summary
35
The End
Contact Information
•Cricket Mitchell, PhD
•Email: cmitchell@cimh.org
•Cell phone: 858-220-6355
36