Who Needs RDD?
Combining Directory Listings with
Cell Phone Exchanges
for an Alternative Sampling Frame
Presented at
AAPOR 2008
New Orleans, LA
May 16, 2008
Thomas M. Guterbock
TomG@virginia.edu
James M. Ellis
jme2ce@virginia.edu
Abdoulaye Diop
adiop@virginia.edu
Kien Le
tkl7b@virginia.edu
John Lee Holmes
jlh2r@virginia.edu
CSR—University of Virginia
www.virginia.edu/surveys
Center for Survey Research
University of Virginia
. . . A unit of the Weldon Cooper Center for Public Service
2
The Research Problem:
RDD under threat
Are there good alternatives?
Center for Survey Research
University of Virginia
3
RDD under threat
• Random Digit Dialing involves a certain degree of
inefficiency
• Costs of this extra effort justified by completeness of
coverage (at least until recently)
• Recent trends raising these costs:
– decreasing density of working numbers,
– increasing rates of non-contact,
– and rising rates of refusal
• Advent of cellular phone only households diminishes
completeness of its coverage
Center for Survey Research
University of Virginia
4
A “New Norm?”
• Dual-frame “RDD+Cell” has arisen in response
to these challenges
– traditional list-assisted RDD sample with RDD of
working cellphone exchanges.
• To screen or not to screen?
– And, if not, how to weight?
• But other dual frames may also be worth
exploring . . .
Center for Survey Research
University of Virginia
5
Proposed Alternative: EWP+Cell
• EWP+Cell =
– “Electronic White Pages” + Cell Phone RDD
• Promises considerably greater efficiency and cost
savings over RDD+Cell
– especially for specific, small geographic regions
– or areas not co-extensive with any set of telephone Area
Codes.
• EWP+Cell fails to cover: unlisted landline
households that have no cell phone
– We will examine: How big a problem is that?
Center for Survey Research
University of Virginia
6
Data Source:
2006 National Health Interview Survey
permits estimations of the size of . . .
the non-covered segment
demographic characteristics
health characteristics
degree of coverage bias
Center for Survey Research
University of Virginia
7
What did we find?
A Preview
• Surprisingly little coverage bias to be
expected from EWP+Cell
• Potential cost savings from EWP+Cell
compared to RDD+Cell
Center for Survey Research
University of Virginia
8
A brief review of the research
• Not much literature or research compares directorylisted samples with list-assisted, landline RDD
samples
• Consequently, the degree and nature of the
differences between listed and unlisted households
is not established.
Center for Survey Research
University of Virginia
9
Older studies (before 2002)
• Most found only slight differences in substantive
results between EWP and RDD frame samples
• Some efficiency gains in smaller geographic areas
• Some differences in demographics noticed
• And in 2007, Zogby announced plans to rely on
EWP over RDD phone samples, citing lack of
substantive differences in results.
Center for Survey Research
University of Virginia
10
Recent studies on EWP vs RDD:
Substantial differences shown
Unlisted rates are higher for:
•Blacks, Hispanics
•Lower income
•Renters
•Single people
See: Guterbock, Diop
and Holian (2007)
Center for Survey Research
University of Virginia
11
From 3 segments to 5
Center for Survey Research
University of Virginia
12
The universe of U.S. telephone
households, 2006
RDD samples cover all
landline households, listed or not
RDD
Cell-phoneonly households
are excluded
Cell phone samples include some
that are also in the RDD frame
Landlineonly
households
are excluded
Cell
phones
RDD and Cell samples overlap,
yield complete coverage
RDD
CELL +
LANDLINE
CELL ONLY
16.6%
52.0%
LANDLINE
ONLY
31.4%
Cell
phones
All percentages are from 2006 NHIS data.
We need also to consider: listedness
• Some landlines are listed in the residential
directory or Electronic White Pages [EWP]
– LLL = Listed Landline
• Some landline households are unlisted
– ULL = unlisted landline
• LLL and ULL may or may not also have a cell
phone in the household
• Cell phones are unlisted by definition
• Result: five segments of the telephone universe . .
. Survey Research
Center for
University of Virginia
17
Five telephone segments
2
CELL + ULL
17.7%
1
CELL ONLY
16.6%
4
CELL + LLL
34.3%
All percentages are from 2006 NHIS data.
3
ULL ONLY
14.2%
5
LLL ONLY
17.2%
See table I
Five segments differ,
sometimes sharply
Center for Survey Research
University of Virginia
19
Segments differ on key demographics
Percent
African/
American
Percent
18 – 25
1
Cell
Only
2
Cell +
ULL
3
ULL
Only
4
Cell +
LLL
5
LLL
only
All
13.4
14.2
15.7
8.3
10.6
11.6
31.1
15.7
13.6
11.0
8.0
15.0
All percentages are from 2006 NHIS data.
See table II
Segments differ on key health questions
1
Cell
Only
2
Cell +
ULL
3
ULL
Only
4
Cell +
LLL
5
LLL
only
All
Smokers
(yes)
27.1
18.9
22.1
17.6
20.6
20.5
Diabetes
(yes)
4.5
6.7
9.4
7.8
11.5
7.9
All percentages are from 2006 NHIS data.
See table II
Our analysis deals with
three sampling frames:
1) EWP
2) List-assisted (landline) RDD
3) Cell phone RDD
We examine 4 sampling designs:
• 2 Single frame
designs:
– EWP only
– Landline RDD
We examine 4 sampling designs
• 2 Dual Frame designs:
– EWP+Cell
– RDD+Cell
Center for Survey Research
University of Virginia
Three design contrasts
RDD+Cell is the base for all comparisons
– It includes the full universe of phone HH
We will compute coverage bias for each
contrast:
•
•
•
EWP vs. RDD+Cell
RDD vs. RDD+Cell
EWP+Cell vs. RDD+Cell
Center for Survey Research
University of Virginia
25
Formula for coverage bias
U
YC  Y 
N
(YC  YU )
Ῡ = mean for full population
ῩC = mean for covered cases
ῩU = mean for cases not covered
U = cases not covered
N = all cases
Center for Survey Research
University of Virginia
26
Contrast I:
EWP vs. RDD+Cell
telephone samples
Center for Survey Research
University of Virginia
27
The universe of U.S.
telephone households
Ῡ
EWP sample excludes
unlisted landline and cell-only
1
EXCLUDED
CELL ONLY
16.6%
U/N =
.485
2
EXCLUDED
CELL + ULL
17.7%
ῩU
3
EXCLUDED
ULL ONLY
14.2%
4
CELL + LLL
34.3%
ῩC
5
LLL ONLY
17.2%
EWP
All listed
landline
phones
Coverage bias table:
EWP vs. RDD+Cell
Percent
African/
American
Percent
18 - 25
U/N
ῩC
ῩU
ῩC -ῩU
Ῡ
ῩC -Ῡ
48.5
9.1
14.4
-5.3
11.7
-2.6
48.5
10.0
20.4
-10.4
15.0
-5.0
All percentages are from 2006 NHIS data.
See table IV
Coverage bias table:
EWP vs. RDD+Cell
U/N
ῩC
ῩU
ῩC -ῩU
Ῡ
ῩC -Ῡ
Smokers
(yes)
48.4
18.6
22.6
-4.0
20.5
-1.9
Diabetes
(yes)
48.7
9.0
6.7
2.3
7.9
1.1
All percentages are from 2006 NHIS data.
See table IV
Contrast II:
RDD
vs.
RDD+Cell only
telephone samples
Center for Survey Research
University of Virginia
32
RDD samples cover
all landline households, listed or not
RDD
ῩC
ῩU
RDD fails to cover 16.6%
RDD
CELL +
LANDLINE
CELL ONLY
16.6%
52.0%
LANDLINE
ONLY
U/N =
.166
31.4%
Cell
phones
All percentages are from 2006 NHIS data.
Coverage bias table:
RDD vs. RDD+Cell
Percent
African/
American
Percent
18 - 25
U/N
ῩC
ῩU
ῩC -ῩU
Ῡ
ῩC -Ῡ
16.6
11.3
13.4
-2.1
11.7
-0.4
16.6
11.8
31.1
-19.3
15.0
-3.2
All percentages are from 2006 NHIS data.
See table V
Coverage bias table:
RDD vs. RDD+Cell
U/N
ῩC
ῩU
ῩC -ῩU
Ῡ
ῩC -Ῡ
Smokers
(yes)
16.5
19.3
27.1
-7.8
20.6
-1.3
Diabetes
(yes)
16.7
8.6
4.5
4.1
7.9
0.7
All percentages are from 2006 NHIS data.
See table V
Contrast III:
EWP+Cell
vs.
RDD+Cell
telephone samples
Center for Survey Research
University of Virginia
37
RDD+Cell covers all phone households
RDD
CELL +
LANDLINE
CELL ONLY
16.6%
52.0%
LANDLINE
ONLY
31.4%
Cell
phones
EWP + Cell Sample Design
EXCLUDES
ULLONLY
Cell
EWP
All listed
landline
phones
EWP + Cell excludes ULL-only households
EXCLUDED:
3
ULL ONLY
14.2%
2
CELL + ULL
17.7%
1
CELL ONLY
16.6%
ῩU
4
CELL + LLL
34.3%
5
LLL ONLY
17.2%
Ῡ
C
U/N =
.142
Coverage bias table:
EWP+Cell vs. RDD+Cell
Percent
African/
American
Percent
18 - 25
U/N
ῩC
ῩU
ῩC -ῩU
Ῡ
ῩC -Ῡ
14.2
11.0
15.7
-4.7
11.6
-0.6
14.2
15.3
13.6
1.7
15.0
0.3
All percentages are from 2006 NHIS data.
See table VI
Coverage bias table:
EWP+Cell vs. RDD+Cell
Smokers
(yes)
Diabetes
(yes)
U/N
ῩC
ῩU
ῩC -ῩU
Ῡ
ῩC -Ῡ
14.2
20.3
22.1
-1.8
20.5
-0.2
14.2
7.7
9.4
-1.7
7.9
-0.2
All percentages are from 2006 NHIS data.
See table VI
Summary of 3 contrasts
Center for Survey Research
University of Virginia
43
3 contrasts: 2006 estimates (ῩC )
Variables
Demographics
African American
18-25
Health Related
Questions
Smokers (yes)
Diabetes (yes)
RDD+Cell
EWP
RDD
EWP+Cell
11.6%
15.0%
9.1%
10.0%
11.3%
11.8%
11.0%
15.3%
20.5%
7.9%
18.6%
9.0%
19.3%
8.6%
20.3%
7.7%
See table VII
3 contrasts: 2006 raw bias (ῩC -Ῡ )
Variables
RDD+Cell
EWP
RDD
EWP+Cell
African American
--
-2.5%
-0.3%
-0.6%
18-25
--
-5.0%
-3.2%
0.3%
Smokers (yes)
--
-1.9%
-1.2%
-0.2%
Diabetes (yes)
--
1.1%
0.7%
-0.2%
Demographics
Health Related
Questions
See table VIII
3 contrasts: 2006 percent bias YC  Y  Y 
Variables
RDD+Cell
EWP
RDD
EWP+Cell
African American
--
-21.6%
-2.6%
-5.2%
18-25
Health Related
Questions
--
-33.5% -21.5%
1.8%
Smokers (yes)
--
-9.3%
-5.9%
-1.0%
Diabetes (yes)
--
13.9%
8.9%
-2.5%
Demographics
See table IX
Changes in Coverage
NHIS 2003 - 2006
Center for Survey Research
University of Virginia
Changes in telephone status over time
NHIS data 2003-2006
unlisted landline only
18.0
16.0
16.7
16.6
16.5
15.1
14.2
14.0
cell phone only
percent
12.0
12.6
10.0
9.1
8.0
7.1
6.0
4.0
2.0
0.0
2003
2004
2005
2006
year
Center for Survey Research
University of Virginia
48
Changes in percent bias over time:
RDD vs. RDD+Cell (NHIS data)
GRAPH 2: RDD vs RDD+Cell: 2003-2006
5.0%
0.0% 0.0%
2003
-3.3%
-5.0%
-0.3%
2004
0.3%
-0.9%
2005
0.6%
2006
-3.4%
-5.1%
African American
-7.4%
-7.6%
-8.1%
18-25
-9.7%
-10.0%
-11.2%
HS or Less
Renter
Less than $20,000
-15.0%
-16.5%
-17.7%
-19.7%
-21.3%
-20.0%
-25.0%
Year
49
Changes in percent bias over time:
EWP+Cell vs. RDD+Cell (NHIS data)
GRAPH 3: EWP+Cell vs RDD+Cell: 2003-2005
5.0%
2.0%
0.0%
2003
-5.0%
-10.0%
-15.0%
0.7%
2004
2005
-4.8%
-9.1%
-9.2%
-11.7%
-9.5%
-11.2%
-13.7%
-14.1%
-9.4%
-12.7%
-20.0%
-25.0%
Year
2006
-5.2%
-5.8%
-7.3%
African American
-10.3%
HS or Less
18-25
Renter
Less than $20,000
Changes in percent bias over time:
RDD vs. RDD+Cell (NHIS data)
RDD vs RDD+Cell: 2003-2006
10.0%
8.9%
8.0%
7.4%
6.0%
5.6%
4.0%
3.2%
2.0%
6.6%
5.8%
3.5%
Smokers (yes)
Diabetes (yes)
1.5%
0.0%
2003-1.0%
0.0%
2004
Hypertension (yes)
2005
-1.9%
-2.0%
2006
-1.8%
-2.8%
-4.0%
-3.9%
-5.8%
-6.0%
-8.0%
Year
-6.3%
Asthma (yes)
Changes in percent bias over time:
EWP+Cell vs. RDD+Cell (NHIS data)
EWP+Cell vs RDD+Cell: 2003-2005
10.0%
8.0%
6.0%
4.0%
Smokers (yes)
2.0%
1.6%
1.0%
0.0% 0.0%
2003
-2.0% -1.9%
-4.0%
1.8%
1.0%
0.8%
2004
-2.4%
0.0%
2005-1.2%
-1.9%
-2.6%
Diabetes (yes)
Hypertension (yes)
2006-1.1%
-1.0%
Asthma (yes)
-2.5%
-4.2%
-6.0%
-8.0%
Year
52
Cost comparisons
Center for Survey Research
University of Virginia
Cost comparisons
A
Traditional
Design
B
C
"New Norm"
Proposed
RDD only
RDD + Cell
EWP + Cell
target N
per hour
cost
CPH*
*CPH=
RDD
RDD
Cell
EWP
Cell
1000
800
200
800
200
$32
$32
$32
$32
$32
1.1
1.1
0.55
1.4
0.55
completions per hour
Cost comparisons
(data collection only)
A
B
C
Traditional Design
"New Norm"
Proposed
RDD only
RDD + Cell
EWP + Cell
target N
RDD
RDD
Cell
EWP
Cell
1000
800
200
800
200
cost
$29,091 $23,273 $11,636 $18,286 $11,636
Total
$29,091
$34,909
$29,922
Conclusions
•
EWP+Cell omits ULL–onlies, but:
–
–
–
•
These aren’t particularly untypical
They are not numerous
Their numbers are declining
EWP+Cell includes unlisteds (who have cell
phones) thus avoiding some bias from EWP
EWP+Cell includes cell–onlies, offsetting bias
from omitting unlisted HH
•
–
Unlisted are somewhat similar to the cell–onlies.
Center for Survey Research
University of Virginia
56
Conclusions
• We propose EWP+Cell as a cost effective
sampling solution that appears to offer good
coverage across age, race, most demographics,
and key health indicators
• It offers a significant cost advantage over the ‘new
norm’ (RDD+Cell), especially if:
– geographic area to be studied is small
– target population is hard to find (requiring screener
calls)
Center for Survey Research
University of Virginia
57
Cautions & caveats
• We have not offered a direct, experimental field
test of EWP+Cell sampling in contrast with
RDD+Cell
– But we have several such experiments planned in our
2008 local surveys
• Nobody knows the proper weights for combining
the two sample frames
– but these are also unknown for local studies that use
“the new norm” –RDD+Cell
• 2008 phone segments could differ from 2006 more
than we think
Center for Survey Research
University of Virginia
58
More cautions . . .
• Our analysis assumes that non-response and
measurement errors are the same in realized
samples from each telephone segment
– That is, we have considered coverage error only
• Even if the proportion of excluded cases (ULLonlies) is declining, coverage error may not
decrease, because:
as U changes, so can Y C  Y U .
N
Center for Survey Research
University of Virginia
59
Planned tests
• We are currently conducting a county-wide citizen
survey in Prince William County, VA, that
features
– A 10% cell phone component (unscreened)
– The balance of completions split:
• 45% RDD sample
• 45% EWP sample
• We will be able to compare directly:
– EWP+Cell vs. RDD+Cell
– Coverage bias, productivity, and costs
Center for Survey Research
University of Virginia
60
Who Needs RDD?
Combining Directory Listings with
Cell Phone Exchanges
for an Alternative Sampling Frame
e-mail: TomG@virginia.edu
Presented at
AAPOR 2008
New Orleans, LA
May 16, 2008