ITSEW
2010
Analysis of Factors Related to
Nonresponse for Landline and
Cell Phone Surveys in China
Yan Jiang
School of Statistics, Renmin University of China
Visiting Professor, Iowa State University, Department of
Statistics and Center for Survey Statistics and Methodology
Outline






2
Introduction
Survey Design
Nonresponse Rates
Call Attempt Analysis
Interviewer Analysis
Conclusions
4/9/2015
Introduction
Surveys in China
 High nonresponse rates

Low contact rates (RDD Sampling)
 High refusal rates


Change of coverage
Steady increase in cell phone frame
 Decrease in landline phone frame
since 2006

3
4/9/2015
Amount of telephone users
(Ten thousands)
90000
2009
80000
cell phone users
70000
Cellphone
Users:
747.38 million
landline telephone households
60000
50000
landline telephone users
Landline Users:
697.02 million
40000
Overlap:
unknown
30000
20000
Total :
1335 million
10000
4
2009
2008
2007
2006
2005
2004
2003
2002
2001
2000
1999
1998
1997
1996
1995
1994
1993
1992
1991
1990
0
4/9/2015
Survey Objectives
To evaluate the utility of selected
methods (e.g., increased call effort) in
telephone surveys of adults in China
 To identify factors related to
nonresponse rates (e.g., interviewer
voice quality, call effort)
 To study how the effect of these
factors on nonresponse vary by
sample frame (i.e., landline, cell)

5
4/9/2015
Survey Design


Population: Adults in China (over 18 years old)
Two-phase Sampling Design



Data Collection Mode: CATI

6
Phase 1: Dual frame sample of phone numbers to
sample adults
Phase 2: Sample of nonrespondents
Phase 2: Modified methods
4/9/2015
Survey Design in First
Phase

Sample Size: 17,162 phone numbers



Stratified RDD Sampling



Strata: Provinces
Allocation: proportional to Province HH pop.
Standard call methods


7
Cell phone: 9678
Landline: 7484
Up to 3 call attempts – usually 2
Ring for about 20 secs
4/9/2015
Survey Design in Second
Phase

Population: nonrespondents in first
phase



Effort to improve the response rate (both
cell phone and landline survey)


8
Sample size: 1549
Strata: province, frame source, noncontact
and refusal numbers
Up to 15 call attempts were made to contact
Wait until at least one minute in each dial
process
4/9/2015
Nonresponse Rate
in First Phase Survey
Landline Survey
Number
Percent
Cell phone Survey
Number
Percent
Total
190,680
100%
169,116
100%
Non-Eligible
149,224
78.3%
108,213
64%
41,456
21.7%
60,903
36%
Eligible
Response
7,484
18.0%
9,678
15.9%
Non-Response
33,972
81.9%
51,225
84.1%
Non-Contact
23,446
56.6%
36,716
60.3%
Refusal
10,526
25.4%
14,509
23.8%
Nonresponse Rate
in Second Phase Survey
landline
cellphone
sample size
rate
sample size
Rate
response
763
43.6%
786
45.0%
nonresponse
985
56.4%
961
55.0%
non-contact
443
25.4%
467
26.7%
refuse
542
31.0%
495
28.3%
Number of call attempts in refusal
sample (stratum) of Phase 2


Cumulated Refusal Conversion Rate of landline survey is higher
than cell phone survey when the number of attempts is less than 5.
Cumulated Refusal Conversion Rate increases rapidly up to about 6
call attempts.
Landline
Cell phone
Number of call attempts
11
4/9/2015
logistic model for Refusal Conversion
Dep. Variable=1, refusal unit convert to respondent
landline survey
cell phone survey
Variable
B
Gender
-.014
.604 Gender
-.196
.000
Young
-.791
.000 Young
-1.056
.000
Middle-aged
-.767
.000 Middle-aged
-1.068
.000
Education
-.189
.000 Education
.187
.000
Income
-.066
.000 Income
-.090
.000
City
-.066
.024 City
.156
.000
East
1.236
.000 East
.941
.000
Middle
-.367
.000 Middle
.984
.000
.097
.000
-2.241
.000
Number
of call attempts
Constant
.146
-1.621
Sig.
.000
Variable
Number
of call attempts
.000 Constant
B
Sig.
The relationship between above factors and refusal conversion rates
differs for the landline survey and the cell phone survey.
12
4/9/2015
Number of call attempts in noncontact sample (stratum) of Phase 2

Cumulated Phase 2 Response Rate of landline survey is
lower than cell phone survey in non-contact sample.
Landline
Cell phone
Number of call attempts
13
4/9/2015
Distribution of the number seconds
until the phone is answered

200
150
成
功
手
机
频
数
Cell Phone
100
50
On average,10
additional seconds per
interview are required
for the phone to be
answered in the cell
phone survey relative
to the landline survey.
Me a n = 3 2 .5 0 8 9
S td . De v. = 1 0 .2 6 5 3
N = 786
0
0 .0 0
2 0 .0 0
4 0 .0 0
6 0 .0 0
8 0 .0 0
1 0 0 .0 0
等待时间
250
Land line
200
成
功
固
定
电
话
频
数
150
100
50
0
14
0 .0 0
Me a n = 1 9 .5 4 6 4
S td . De v. = 1 0 .3 0 3 8 7
N = 765
2 0 .0 0
4 0 .0 0
6 0 .0 0
等待时间
8 0 .0 0
1 0 0 .0 0
1 2 0 .0 0
4/9/2015
Phase 1 Interviewer Analysis
Cooperation rate distribution of interviewers
20
Frequency
15
10
Mean = 0.375
5
Std. Dev. = 0.083
N = 108
0
0.20
0.30
0.40
0.50
Cooperation Rate
15
0.60
0.70
4/9/2015
Phase 1 Interviewer Analysis

Which interviewer characteristics affect
cooperation rates?
Interviewer characteristics
 Skill level
 Vocal quality

Control for respondent characteristics


16
Gender, education, career, income,
location of residence, sample frame
4/9/2015
Phase 1 Interviewer Analysis

Hierarchical Generalized Linear Model
 Level 1 model (respondent level)
pij
log(
)  0 j  q  qj X qij
1  pij

Level 2 model : (interviewer level)
 qj   q 0  s  qsWsj  qj
i : respondent; j: interviewer;
X qij : variables of respondent
W : variables of interviewer
sj
qj ~ N (0, qq )
pij
:refusal rate
Variable Definitions
outcome of
interview
respondent
X
y =1, refusal;
y =0, response
pij  prob( yij | Ws )
gender=1, male;
gender=0, female
edu1: low education,
edu2: midium education
career1: student,
career3: clerk,
career2: manager,
career4: self-employee
income>=0
Interviewer
W
18
survey =1, landline ;
survey =0, cell phone
grade: =1, large city;
grade=0, other
W1: interview skill of interviewer
W2: vocal characteristic of interviewer
4/9/2015
Results
Respondent
Null model
Fixed effect
Level 1 model
Coefficients
SE
0.545**
0.060
model
Coefficients
Interviewer
Level 2 model
SE
model
Coefficients
SE
respondent level
0.702**
0.105
0.691**
0.093
career1，  10
-1.974**
0.409
-2.056**
0.426
career2，  20
-2.275**
0.440
-2.194**
0.426
career4，  30
-2.767**
0.463
-2.928**
0.488
income，  40
0.00016*
0.00007
0.00016*
0.00007
grade，  50
1.266**
0.205
1.307**
0.214
W1，  01
-0.28**
0.059
W2，  02
-0.146*
0.059
career4  W2,  32
1.136**
0.134
intercept，  00
Both the demographic
characteristics of
respondents and
interviewers have
significant impacts on
refusal rates.
Survey type does not
have significant effects on
refusal rates in the model
interviewer level
Both the interviewers’
speaking skill and natural
vocal characteristics have
significant impacts on
refusal rates.
Conclusions
1.
2.
3.
4.
5.
Dual frame sampling is necessary for reducing
coverage bias for telephone surveys in China. The
ineligible (e.g., non-working, business) rate in cell
phone surveys appears to be lower than in landline
surveys in China.
High non-response rates can be reduced by effective
voice training of interviewers, more call attempts, and
extending waiting time before giving up on the call.
The demographic characteristics and behaviors of cell
phone users and landline telephone users are different,
which may have implications for nonresponse bias.
Interviewers should wait longer for the respondent to
answer the phone in the cell phone survey relative to
the landline survey.
Refusals occur more quickly in the cell phone survey
than in the landline survey.