Alternate Strategies for Including
Cell Phone Only Households in Surveys:
Comparing the Use of ABS and
Cell Phone RDD
Samples in Massachusetts
David Dutwin, PhD
State Health Research Policy Interest Group
Academy Health Annual Research Meeting
Seattle, WA
June 11, 2011
Overview
Comparison of alternate sampling strategies to
capture cell phone households


Address-based sample (ABS)
Cell phone random digit dial sample (RDD)
Case study in Massachusetts in 2010

Massachusetts Health Insurance Survey (MHIS)
-- ABS and landline RDD

Massachusetts Health Reform Survey (MHRS)
-- Landline RDD and cell phone RDD
Recommendations
Alternative Approaches in MA
MHIS – ABS & landline RDD

ABS: Random sample of households from the delivery
sequence file used by the United States Postal Service
-- Includes households regardless of phone status


Landline RDD: Random sample of landline numbers
Estimated coverage rate: 98% of households
MHRS – Landline RDD & cell phone RDD



Landline RDD: Random sample of landline numbers
Cell phone RDD: Random sample of cell phone numbers
Estimated coverage rate: 90 to 95% of households
Similarities in MHIS and MHRS

Both fielded every year, including 2010

Similar questionniares focused on health
insurance coverage, access and use




Both draw on SHADAC’s Coordinated State
Coverage Survey (CSCS)
Similar sample sizes
Similar costs
Same organizations involved in both
surveys, including same survey firm
Differences in MHIS and MHRS
Sponsor: state vs. foundation
Sample design
• ABS & landline RDD vs. landline RDD & cell phone RDD
• Size of landline RDD sample
• Oversampling of selected populations
Instrument design
• Data collection mode: Phone, internet & mail vs. phone
• Focus: all members of household vs. random adult
• Respondent selection
Survey fielding
• Time period of the survey: spring vs. fall
• Modes of contact: mail & some phone vs. phone & some mail
• Use of incentives
Response rates & sample coverage
Comparison of Selected Elements
ELEMENT
2010 MHIS
2010 MHRS
Sample Size
4,478: 1,253 landline RDD,
3,040: 2,418 landline RDD,
3,225 ABS
622 cell phone RDD
Contact Method Advance letter w/ call-in and Refusal letter w/ call-in invitation
web invitation; 2 reminders;
(if address available); call-out
hardcopy questionnaire; callout (if phone number
available)
$20 for those without available
Incentives
phone number, plus Red Sox
$10 for all
tickets or $100 sweepstakes
for all
~98%
~90-95%
Estimated
coverage rate
Response rate
40%--49% landline RDD,
37% ABS
35%--42% landline RDD,
31% cell phone RDD
Advantages for Cell Phone Coverage
MHIS--ABS

Very high coverage rates for USPS in Massachusetts

Share of cell phone only households reached was close
to the NHIS estimated proportion of cell phone only
households in Massachusetts

Bonus coverage: Also reached people who were using
voice over internet phones as well as a few nontelephone households
MHRS--Cell phone RDD

A very high percentage of all cell phone numbers are in
cell phone banks that can be sampled

Based on estimates from other sources: Only about
10% of all cell phone owners in MA have non-MA area
codes
Other Advantages
MHIS--ABS
•
•
Able to contact all sample members by mail
Creating survey weights is straightforward using
traditional survey procedures since population totals for
the sample are available
MHRS--Cell phone RDD
•
Single mode for entire sample – All CATI telephone
interviews using the same core instrument
•
Cell phone offers potential to reach respondents who are
seldom at home
Disadvantages
MHIS--ABS

Cannot obtain phone numbers for some households so that
only method of contact was mail, leading to lower response
rates for those households


Some addresses will be vacant or summer homes, etc.
More item non-response on the mail questionnaire (12% of
MHIS ABS surveys from mail)

Greater operational complexity and higher costs
MHRS--Cell phone RDD
•
•
•
Cannot obtain addresses to mail advance letter
Cell phones expire, not in service, business, etc.
Costly to screen out respondents who do not live in
Massachusetts (28% of cell phone RDD sample)
•
Survey weights are challenging since we don’t know the
population totals for the cell phone population
Comparing All Adults in MHIS and MHRS
2010 MHIS ABS
(N=2,103)
2010 MHRS RDD
(N=3,040)
Age 19 to 25
16.4
16.1
Age 26 to 34
17.2
17.2
Non-white
20.1
20.9
Hispanic
7.2
7.9
Female
50.6
50.8
Born in the U.S.
84.2
83.6
Less than high school
6.9
8.0
Employed
70.1
69.3
Married/partner
60.8
61.2
Excellent/Very good health status
62.5
62.7
Fair/Poor health status
12.3
14.0
43.9
44.1
Sample is adults 19 to 64; Estimates are weighted; * p < .05
Greater Boston area
Comparing Cell-phone-only Adults in MHIS and MHRS
MHIS
Target adult
(N=514)
MHRS
Respondent
(N=275)
Age 19 to 25
23.3
29.0
Age 26 to 34
34.6
32.0
Non-white*
22.7
37.9
Hispanic
13.5
15.9
Female
49.0
47.8
Born in the U.S.
82.6
76.1
Less than high school
8.1
13.5
Employed
75.6
67.6
Married/partner
56.8
48.9
Excellent/Very good health status
64.5
60.9
Fair/Poor health status
11.8
14.6
51.7
52.5
Sample is adults 19 to 64; Estimates are weighted; * p < .05
Greater Boston area
Summary
•
Despite their differences, the MHIS and MHRS yielded
similar overall samples of the nonelderly adult population
AND similar samples of cell-phone-only nonelderly adults
in Massachusetts
•
Both ABS and cell phone RDD appear to be credible
strategies for capturing cell-phone only households
•
However, important to remember: Since don’t know
actual population characteristics of the cell-phone-only
population we can’t assess whether missing important
components of the population with both strategies
Recommendations
To states: ABS and cell phone RDD are credible
strategies for including cell-phone households in
surveys
To researchers/survey methodologists:

Explore methods/develop strategies to increase response
rates to surveys across sample frames

Explore methods/develop standards for estimating
consistent response rates and household coverage rates
across sample frames

Explore implications of mode effects for surveys that rely on
multiple modes

Explore methods/develop strategies for combining cell
phone RDD and landline RDD samples
Thanks and Contact Information
Acknowledgement:
The MHIS was funded by the Massachusetts Division of Health
Care Finance and Policy.
The MHRS was funded by the Blue Cross Blue Shield of
Massachusetts Foundation.
We appreciate that they allowed us to use the survey data to
conduct this analysis.
Contacts:
Nicole Buttermore: Research Director, SSRS
David Dutwin: Vice President, SSRS
Timothy Triplett: Survey Methodologist, Urban Institute
Doug Wissoker: Senior Research Associate, Urban Institute
Sharon K. Long: Professor, University of Minnesota