Outcomes of
Consumer-Operated Service Programs
As Adjuncts to Traditional Services
Gregory B. Teague, Ph.D.
Louis de la Parte Florida Mental Health Institute
College of Behavioral & Community Sciences
University of South Florida
Acknowledgements:
Steven Banks
Crystal R. Blyler
Jean Campbell
Sally Clay
Patrick W. Corrigan
Dianne C. Côté
Nancy Erwin
Susan Essock
Matthew Johnsen
Betsy McDonel Herr
Carolyn Lichtenstein
Jeffrey G. Noel
Ruth O. Ralph
E. Sally Rogers
Joseph Rogers
Mark S. Salzer
L. Joseph Sonnefeld
Tom Summerfelt
Brian Yates
Supported by a grant from the U.S. DHHS, SAMHSA/CMHS # SM-52328
The COSP Study – Purpose & Hypothesis
• Purpose
– Discover extent to which consumer-operated programs are
effective as an adjunct to traditional mental health services in
improving the outcomes of adults with serious mental illness
• Population
– Adults with a diagnosable psychiatric disorder & impairment
– Initially using traditional MH services (TMHS) but not consumeroperated services (COS)
• Primary hypothesis
– Participants offered both TMHS and COS would show greater
improvement in wellbeing over time
• Composite wellbeing measure
– Hope, meaning of life, self-esteem/self-efficacy, recovery
2
The COSP Study - Design
• RCT: Randomly assigned invitation
– To use COSP while continuing to use TMHS (“COSP+”)
vs. not to use COSP (“TMHS”) for 1 year
• Eight sites
– CA, CT, FL, IL, ME, MO, PA, TN
• Common interview protocol across all sites
– Multiple outcome domains
• One-year follow-up period
– Baseline, 4, 8, & 12 mo
• Common measure of program ingredients
– Applied to both COS and TMHS
3
Analytic Approach
• Rigorous “intent-to-treat” (ITT) analysis
– All randomly assigned participants included in comparisons
whether they used the services or not; multi-level model
• Complementary “as-treated” (AT) analysis
– Comparisons among groups of participants defined by use of
services (high and low use, no use)
• Propensity scores used to minimize risk of selection bias
– Exclusion of participants with high or low propensity to use
COSP from AT analyses
• Program fidelity measure used with experimental results to
identify critical program ingredients
4
Participant Characteristics
• N= 1827 ( approximately 1600 in outcome analyses)
• Mean age= 43 (s.d.=10.3)
• Female= 60%
• White = 57%; African American = 16.7%; Hispanic/Latino =
1%; Other/Multiple = 24%
• Education beyond HS= 42%c
• Ever married= 53%
• Psychotic disorder = 50%; Mood disorder = 44.4%; Anxiety
disorder= 4%
• Ever hospitalized = 83%
• Taking psychiatric medication = 96%
5
Intent-To-Treat (ITT) Results: Wellbeing Over
Time by Random Assignment Group
Effect
DF
Time
1, 4073
Time*Group
1, 4073
Time*Site
7, 4073
Time*Group*Site
7, 4073
F
p
38.69 <.0001
4.77
0.029
1.76
0.090
1.47
0.171
.3
.2
.1
.0
TMHS, 797 -.1
COS+, 803
-.2
Base
(N=1600)
4mo
(N=1441)
8mo
(N=1357)
12mo
(N=1272)
6
Well-being Effect Sizes By Site:
Between Conditions & Within COSP+
.6
.4
.2
ES-C
ES
.0
1
2
3
4
5
6
7
8
-.2
-.4
Site
• Unique features of outlying program (site 8)
– Recruitment criteria: high-profile policy aspirations
– Highest outcome scores throughout study
– No intervention during final third of study
7
Adjusted Wellbeing ITT Results
(7 Sites – Outlying Site Omitted)
Effect
DF
F
p
Time
1, 3800 50.54
Time*Group
1, 3800
7.78
0.005
Time*Site
6, 3800
0.63
0.707
Time*Group*Site
6, 3800
1.4
0.210
<.0001
• Overall increase in Wellbeing over time
• Significantly greater increase in Wellbeing for
persons offered use of COSP (p < .01)
• No significant site differences across 7 sites
– Positive trend for both groups
– Consistent positive ES for COSP
– Negligible experimental differences across site
8
As-Treated Analysis (AT)
• Engagement rates/
adherence to assigned
condition low
• Scores calculated for
propensity to use COS
• Analysis limited to
middle third of
propensity
– Lower selection bias
– Smaller N (findings
conservative)
.3
.2
.1
.0
-.1
-.2
Base (N=516)
4mo (N=479)
High, 77
8mo (N=449)
Low, 79
12mo (N=404)
None, 360
• Three-point measure
of intensity of use
9
Overall Findings for Wellbeing
Finding
Analysis
p
ITT/AT
<.001
Significantly greater gains for the group offered
use of COS, despite low engagement rate
ITT
<.05
More consistent and significant gains without one
outlying site
ITT
<.01
Greater gains for participants who actually used
COSP services
AT
<.03
Greatest gains for those who used COSP the most
AT
<.01
Overall increase in Wellbeing over time
Effects were unrelated to amount of TMHS use
10
Other Significant, Positive Study Outcomes
and Interactions
• Other outcomes
– Increase in multi-factorial empowerment (AT, 7-site ITT) and
self-orientation empowerment component (ITT; AT)
– Decrease in psychiatric symptoms (AT)
– Marginal reduction in number and duration of hospitalization
for COSP-invited group (ITT)
– Increase in perception of social inclusion (AT)
• Demographic interactions
– Age: younger participants began with higher levels of
symptoms and experienced greater reduction (AT, p<.01)
– Education was significantly related to gains in Wellbeing ITT
p<.05, AT p<.01) and Empowerment (AT p<.01) and reduction
in Symptoms (AT p<.05)
11
Program Measure: Fidelity Assessment
Common Ingredients Tool (FACIT)
Process
Scales
alpha
Scale
Subscales
Structure
Consumer Ownership,
Responsiveness
.939
Environment
Inclusion, Accessibility
.738
Peer Support
Encouragement;
Self-Expression
.885
Belief Systems
Peer Ideology; Choice &
Respect; Spirituality/
Accountability
.738
(n = 16)
Education
.888
Advocacy
.893
TOTAL
.963
12
FACIT Scale Scores:
Consumer-Operated vs. Traditional Services
100
90
80
70
60
%
TMHS
COSP
50
40
30
20
10
0
STRUCTURE
ENVIRONMENT
PEER
SUPPORT
BELIEF
SYSTEMS
EDUCATION
ADVOCACY
• All COSP overall scores higher than any TMHS
score (p = .004)
• COSP scale scores higher than TMHS (p = 0.00-0.03)
• Individual COS programs higher than respective
TMHS on most subscales
13
Process Scales (combined) & Selected Outcomes:
Between (n=8) & Within Programs (n=16)
1.0
Pearson r
.8
.6
.4
.2
.0
Empow
Wellbeing
COS+ vs T-only
Socializ
Life Sat
All 16 programs over time
14
FACIT Process Scales and Selected Outcomes
1.0
Pearson r
.8
.6
.4
.2
.0
Environment
Empowerment
Peer Support
Belief Systems
Wellbeing/ Socializ/ LifeSat
15
Ingredients Related To COSP Outcomes:
Key Process Subscales
1.0
Pearson r
.8
.6
.4
.2
.0
Environment
Empowerment
Peer Support
Belief Systems
Wellbeing/ Socializ/ LifeSat
• Environment – Inclusion (Low/no cost; protective
program rules; positive social environment; sense of
community; lack of coerciveness)
• Peer Support – Self-Expression (Artistic expression;
participants telling own stories)
• Belief Systems – Choice & Respect (Choices about
participation; acceptance and respect for diversity)
16
Conclusions
• COSPs were effective in producing gains in recovery-related
domains over a 12-month period
– Effect is both incremental and compensatory
• Greater gains for those with more education
• Critical ingredients are related to improvement in wellbeing,
empowerment, and other domains, independent of program
type (COSP, TMHS) or strength of experimental effect
– Gains strongly related to specific program features, stronger in
COSPs but also present in traditional programs
• Policy implications
– Support COS programs
– Foster critical features within TMHS
– Recognize and continue to evaluate benefits of self-help
17