Estimating Mental Illness in an
Ongoing National Survey
Joe Gfroerer, Sarra Hedden, Peggy Barker, Jonaki Bose
Center for Behavioral Health Statistics and Quality, SAMHSA
Jeremy Aldworth
RTI International
COPAFS Meeting
March 16, 2012
Outline of Presentation
• Summary of National Survey on
Drug Use and Health (NSDUH)
• Design of Mental Health
Surveillance Study (MHSS)
• Results
• Methodological issues
National Survey on Drug Use and
Health (NSDUH)
• Sponsor: Substance Abuse and Mental Health
Services Administration (SAMHSA), Center
for Behavioral Health Statistics and Quality
(CBHSQ)
• Purpose: Estimate prevalence, correlates and
trends of substance use in U.S.
• History: Conducted since 1971, annually since
1990
3
NSDUH Design
• Representative nationally and in each state
• Civilian, noninstitutional population, age 12+
• Face-to-face interview using ACASI
• 68,000 respondents each year; oversample age 12-25
• $30 incentive
• Response rates (weighted, 2010):
• 88% of selected households completed screener
• 74% of selected persons completed interview
4
NSDUH Sample Design: Target
Sample Sizes by State and Age Group
• Completed Interviews per State
• Large states (8): 3,600 per year
• Small states (43): 900 per year
• Completed Interviews by Age Group
• 1/3 of sample in each age group
(12-17, 18-25, 26+)
5
NSDUH Questionnaire
• Use of alcohol, tobacco, and illicit drugs
• Substance use disorders (DSM-IV)
• Substance use and mental health treatment
• Health conditions, service utilization
• Demographics
• Mental health (MDE, suicide)
6
NSDUH Mental Health
Surveillance Study (MHSS)
• SAMHSA legislation requires the agency to produce
methods to estimate serious mental illness (SMI)
(and serious emotional disturbance (SED) in children)
• TAG (2006) recommended NSDUH for SMI
(and NHIS for SED)
• MHSS implemented in 2008 NSDUH
7
SAMHSA Definition of Serious
Mental Illness (SMI) among Adults
Any DSM-IV mental disorder
(other than developmental and
substance use disorders)
WITH
serious functional impairment
(both in past year)
Estimating SMI in NSDUH
• A complete diagnostic assessment to determine
SMI is not feasible in NSDUH interview
• Would require too many questions
• Interviewers are not clinicians
• Alternative approach used by SAMHSA:
• Administer clinical interviews on a subsample of NSDUH
respondents, to diagnose SMI
• Include short scales in main NSDUH interview, to be used
as predictors of SMI in a model: K6, WHODAS
• Develop a regression model, based on subsample data, and
apply to main sample data to predict SMI for each
respondent
Kessler 6-item Nonspecific
Psychological Distress Scale (K-6)
• Included in NSDUH and several other large
national surveys
• Developed specifically for use in large surveys
• Discriminates between cases and non-cases in
community samples
• Demonstrates consistency across population groups
• Responses 0-4 for each item; combined score 0-24
Percentage Distribution of K6 Scores
among Persons Aged 18 or Older: 2008
Percentage
25
22.4
20
15
10.710.6
9.4
10
7.9
6.1
5.3
4.0
5
3.3 2.7
2.5 2.0 2.6
1.6 1.4 1.1 1.1 1.0 1.4
0.5 0.5 0.4 0.3 0.2 0.8
0
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
K6 Score
Measuring Impairment in
NSDUH Main Sample: WHODAS
• WHO Disability Assessment Schedule
(WHODAS)
• 16 items assessing functional impairments in
various domains
• Reduced to 8 items for NSDUH based on
IRT analysis
• Responses: 0 to 3 for each item
Clinical Interview Subsample
• At end of NSDUH interview, a request for 2nd
interview on mental health is made to respondents
selected for the clinical followup interview
• $30 incentive
• N=500 to 1500 per year
• Nationally representative, stratified sample
• Interview conducted by a trained clinical interviewer,
by telephone, 2-4 weeks after main interview
13
Clinical Interview Content
• Structured Clinical Interview for DSM-IV
(SCID): 15 specific mental disorders are
covered
• Global Assessment of Functioning scale
(GAF)
14
Estimation Step 1: Determine Best
Weighted Logistic Regression Model
Using Clinical Interview Subsample
Let π = Pr(“true” SMI│X1, X2)
logit(π) = b0 + b1X1 + b2X2
• X1 = recoded K6 score (0-17)
• X2 = recoded WHODAS score (0-8)
Estimation Step 2: Determine MinimumBias Cutpoint from Clinical Interview
Data
1. Based on model, each CI respondent has predicted
Pr(SMI+) = ˆ
2. Based on clinical interview, each CI respondent has
a “true” SMI diagnosis
3. Select cutpoint,  0 , for which false positives equal
false negatives in the CI subsample
- If
- If
ˆ   0 then predicted SMI status = positive
ˆ   0 then predicted SMI status = negative
Final Model Based on 2008
Clinical Interview Data
logit(ˆ) = -4.7500 + 0.2098X1 + 0.3839X2
Where X1 = recoded K6 score (0-17)
X2 = recoded WHODAS score (0-8)
Cutpoint:
0
= 0.26972
17
Estimation Step 3: Apply Model to
Main Sample
1. Based on model, and reported K6 and
WHODAS scores, each NSDUH respondent
has predicted Pr(SMI+) = ˆ
ˆ   0 then SMI status = yes
If ˆ   0 then SMI status = no
2. If
ROC Statistics:
Final SMI Model with K6 and WHODAS
vs. Alternative Model with K6 Only
Model
Predicted False False Sensi- SpeciParameters
Rate
Pos. Neg. tivity ficity
Rate Rate
K6
.046
.029
.028
.387
.971
Area
Under
ROC
Curve
.679
K6 and
WHODAS
.047
.024
.023
.506
.976
.741
19
Levels of Mental Illness
Secondary purpose of the MHSS was to generate estimates of
“any mental illness” and to designate levels of severity:
Level of MI in Past Year
Definition
Low/Mild Mental Illness
(LMI)
Any disorder, and
GAF>59
Moderate Mental Illness
(MMI)
Any disorder, and
GAF 51-59
Serious Mental Illness
(SMI)
Any disorder, and
GAF<51
TOTAL/Any Mental
Illness (AMI)
Any disorder
Estimating Other Levels of
Mental Illness
• Various models were compared
• Result: The SMI model, with
different cutpoints, was found to
predict as well as any other model
AMI/ SMI Prediction Based on Recoded K6
and WHODAS Scores
Recoded WHODAS Score
8
7
SMI
6
5
4
LMI or MMI
3
2
No MI
1
0
0
1
2
3
4
5
6
7
8
9
Recoded K6 Score
10 11 12 13 14 15 16 17
Prevalence of Mental Health Problems
among Adults (18+): 2010
Percent with disorder/problem in past year
25
20
20
15
10
6.8
5
5
3.8
46 mil
11 mil
15 mil
9 mil
0
Any Mental Illness
Serious Mental
Illness
Major Depressive Serious Thoughts of
Episode
Suicide
23
Fig
MH
2.1
Any Mental Illness in the Past Year among Adults
Aged 18 or Older, by Age and Gender: 2010
Percent with Any Mental Illness
(AMI) in the Past Year
35
29.9
30
25
20
23.0
22.1
20.0
16.8
14.3
15
10
5
0
18 or Older
18 to 25
26 to 49
Age Group
50 or Older
Male
Female
Gender
24
Fig
MH
2.2
Serious Mental Illness in the Past Year among Adults
Aged 18 or Older, by Age and Gender: 2010
Percent with Serious Mental Illness
(SMI) in the Past Year
9
7.7
8
7
6.5
5.8
6
5
5.0
4
3.2
3.4
50 or Older
Male
3
2
1
0
18 or Older
18 to 25
26 to 49
Age Group
Female
Gender
25
Fig
MH
2.9
Receipt of Mental Health Services among Adults
Aged 18 or Older, by Level of Mental Illness: 2010
Percent Receiving Mental Health Services
in the Past Year
26
Fig
MH
4.1
Past Year Substance Use among Adults Aged 18 or Older,
by Any Mental Illness: 2010
Percent Using Substance
Marijuana
Illicit Drugs1
1
Cocaine
Psychotherapeutics
Inhalants
Hallucinogens
Heroin
Illicit Drugs include marijuana/hashish, cocaine (including crack), heroin, hallucinogens, inhalants, or prescription-type psychotherapeutics used
nonmedically.
27
Fig
MH
4.2
Past Year Substance Dependence or Abuse and Mental Illness
among Adults Aged 18 or Older: 2010
SUD and
Mental
Illness
SUD,
No Mental
Illness
11.2
9.2
Million Million
20.3 Million
Adults Had SUD
36.7
Million
Mental
Illness,
No SUD
45.9 Million Adults
Had Mental Illness
SUD = substance use disorder.
28
Fig
MH
4.4
Past Year Substance Dependence or Abuse among Adults
Aged 18 or Older, by Level of Mental Illness: 2010
Percent Dependent or Abusing Substance
29
Issue: Trend Measurement
Options:
• Update models, parameters, and/or
cutpoints each year
• Small annual sample
high variance
• Continue to accumulate clinical interview
data and evaluate models; update model
when there is evidence that estimates can
be substantially improved
• Will need to update all prior estimates
Prevalence of Mental Illness
among Adults (18+): 2008 to 2010
Percent in past year
25
20
19.5
19.9
20.0
2008
2009
2010
15
10
4.4
5
4.8
5.0
0
Any Mental Illness
Serious Mental Illness
31
Issue: Nonresponse Bias and Weighting
CI Sample Disposition, 2008-2009:
Unwtd. Unwtd.
N
Pct.
TOTAL
Wtd.
Pct.
3,062
100.0
100.0
2,027
66.2
59.5
Immediate refusal
420
13.7
24.3
Agreed, but noncontact
477
15.6
12.5
Other nonresponse
138
4.5
3.7
Respondents
Nonresponse Bias Assessment:
Rates of Key Measures among Respondents, Refusals, and
Noncontacts: Clinical Interview Sample, 2008-9
Percent in Past Year
20
18
16
14
12
10
8
6
4
2
0
18.2
17.1
15.5
15.0
9.9
4.9
5.6
5.5
5.3
3.7
1.3
1.9
Suicide Thoughts
Perceive MH Tx
Need
Respondent (60%)
Rec'd MH
Treatment
Refusal (24%)
Marijuana Use
Noncontact (13%)
33
Nonresponse Bias Assessment:
Age and Family Income among Respondents, Refusals,
and Noncontacts: Clinical Interview Sample, 2008-9
Percent
35
29.7
30
25
20
19.8
16.8
15
10
12.0
9.2
9.7
Age 18-25
<$20K Income
5
0
Respondent (60%)
Refusal (24%)
Noncontact (13%)
34
Other Issues
• What is the best sample design?
• Optimize for modeling?
• Prevent extreme weights
• What is best estimation method?
• Variance estimation not straightforward
• Estimate prevalences of specific disorders
from the clinical interview sample?
Conclusions
• MHSS provides the only current data on
trends in mental illness and its co-occurrence
with substance use
• Estimates have been widely cited and used in
analyses of the impact of health care reform
• Methods can be replicated in other surveys
• But more work needed to refine the models
and estimation methods