Response rates and disposition
coding
PHC 6716
June 8, 2011
Chris McCarty
RDD Telephone Disposition Codes
AAPOR Disposition Description
Completed interview
Partial interview
Strong refusal
Soft refusal
Resp. never available
Answering machine, message
Answering machine, no message
Dead
Physically, mentally unable
Language unable
Miscellaneous unable
Busy
No answer
Answering machine, don’t know if household
Technical phone problem
Temporary phone problem
Fax/data line
Nonworking number
Disconnected number
Number changed
Cell phone
Call forwarding
Business/Government/Other Org.
Institution
Group quarters
No eligible respondent
Quota filled
Callback, Resp. not selected
Callback, Resp. selected
Code
1100
1200
2110
2120
2210
2221
2222
2310
2320
2330
2340
3120
3130
3140
3150
3151
4200
4310
4320
4430
4410
4420
4430
4520
4530
4700
4800
5100
5200
Eligibility
E
E
E
E
E
E
E
E (depends on timing)
E
E
E
E
E
E
E
E
I
I
I
I
I
I
I
I
I
I
I
E
E
I = Complete interview (1.1)
P = Partial interview (1.2)
R = Refusal and break-off (2.10)
NC = Non-contact (2.20)
O = Other (2.30)
UH = Unknown if household/occupied HU
(3.10)
UO = Unknown, other (3.20)
e = Estimated proportion of cases of
unknown eligibility that are eligible
Types of rates
• Response Rate (RR) =
Proportion of cases
interviewed of all eligible
cases
RR1 
• Refusal Rate (REF) =
Proportion of eligible
sample that refused
I
( I  P)  ( R  NC  O)  (UH  UO)
• Cooperation Rate (COOP)=
Proportion of cases
interviewed of eligible cases
contacted
• Contact Rate (CON) =
Proportion of eligible
sample where a household
was reached
What gets reported?
• It is rare to see anything other than the
response rate reported
• Occasionally people will report the
cooperation rate
• This is often reported when the response rate
is low
AAPOR Response Rates
AAPOR Response Rates
I
RR1 
( I  P)  ( R  NC  O)  (UH  UO)
RR2 
I P
( I  P)  ( R  NC  O)  (UH  UO)
I = Complete interview (1.1)
P = Partial interview (1.2)
R = Refusal and break-off
(2.10)
NC = Non-contact (2.20)
I
RR3 
O = Other (2.30)
( I  P)  ( R  NC  O)  e(UH  UO)
I P
RR4 
( I  P)  ( R  NC  O)  e(UH  UO)
I
RR5 
( I  P)  ( R  NC  O)
I P
RR6 
( I  P)  ( R  NC  O)
UH = Unknown if
household/occupied HU (3.10)
UO = Unknown, other (3.20)
e = Estimated proportion of
cases of unknown eligibility that
are eligible
Calculating e
• e is an estimate of the proportion of non-contacts that are
eligible
• Its calculation depends on survey design and execution
• There are a number of ways it can be calculated
• Some people just make assumptions about e
• http://www.aapor.org/pdfs/erate.pdf
Example of calculating e for May 2011 CCI
Comparison of response rates for May
2011 CCI
RR1
11.1
RR2
11.1
RR3
12.9
RR4
12.9
RR5
16.2
RR6
16.2
Exceptions to AAPOR Disposition Codes
Complete
Partial Complete
Strong Refusal
Soft Refusal
Respondent Never Available
Answering Machine, No Message
Answering Machine, Message
Listed Respondent Deceased
Physically/Mentally Unable
Language Unable
Misc Unable
Busy
No Answer
Technical Phone Probs
Fax/Data Line
Non-Working Number
Disconnected Number
Number Changed
Cell Phone
Business/Gov/Other Organization
Institution
Group Quarters
N o one the re with first/ la st na me
Adult not in Me dica id/ H MO
Adult in Me dica id/ H MO, wrong pla n na me
Adult in Me dica id/ H MO, unde r 18
Callback, Resp Not Selected
Callback, Resp Selected
• In some cases AAPOR codes
may not be detailed or
descriptive enough to allow
for exceptional
circumstances with sample
• If possible AAPOR codes
should be used
• If new codes must be used
they should aggregate to
existing AAPOR codes
Last versus Final Dispositions
• Last Disposition is actually the “most recent
disposition”
• Final disposition requires a rule for evaluating call
history
• There are no firm standards for evaluating call
history
• Research suggests that the difference is trivial
Last versus Final Dispositions -Research
RDD 1
RDD 2
RDD 3
Listed 1
Listed 2
Listed 3
Sample released
3500
29174
4450
30084
16268
1100
Call limit
15
12
12
10
10
10
Average attempts per case
5.6
5.1
4.1
5.1
4.1
2.2
Completed interviews
501
8010
1002
7630
3823
402
Response rate using most recent disposition
.28
.41
.32
.43
.49
.40
Response rate using ever eligible
.21
.33
.28
.31
.31
.39
Response rate using all five final disposition rules
.26
.40
.32
.38
.44
.40
Response rate using only Rule 1
.23
.34
.29
.32
.32
.39
Response rate using only Rule 2
.22
.36
.28
.35
.38
.39
Response rate using only Rule 3
.22
.34
.28
.32
.33
.39
Response rate using only Rule 4
.22
.33
.28
.31
.31
.39
Response rate using only Rule 5
.22
.33
.28
.32
.32
.39
Response rate using only Rule 1 and Rule 2
.24
.37
.30
.36
.38
.39
For most recent call:
Rule 1: Business always coded as business
Rule 2: No eligible respondent always coded as no eligible respondent
Rule 3: Disconnected number previously coded as no answer or temporary phone problem always coded as disconnected number
Rule 4: Fax/data line previously coded as no answer or temporary phone problem always coded as Fax/data line
Rule 5: Non-working number previously coded as a temporary phone problem always coded as a non-working number
Council of American Survey Research
Organizations (CASRO) Disposition
Codes and Response Rates
Response rate:
01
(01  02  07  09)
(01  02  07  09) 
x(04  10)
(01  02  07  09)  (03  05  06  08  11)
Where:
01 = Completed interview
02 = Refused interview
03 = Nonworking number
04 = Ring-no-answer
05 = Not a private residence
06 = No eligible respondent at this number
07 = Selected respondent not available during the interview period
08 = Language barrier
09 = Interview terminated within questionnaire
10 = Line busy
11 = Respondent unable to communicate due to physical or mental impairment
Example response rates
• BRFSS Response rates by state (page 32):
ftp://ftp.cdc.gov/pub/Data/Brfss/2010_Summ
ary_Data_Quality_Report.pdf
• Quinnipiac polls:
– http://www.quinnipiac.edu/x271.xml
What affects Response Rates?
•
•
•
•
•
•
•
•
•
Maximum call attempts
Maximum number of refusal callbacks
Fielding time
Rotation of calls across day of the week and time of
day
The population surveyed
Survey length
Sample quality
Listed versus random-digit dial
Salience of survey topic to respondent
Variables in model used to predict
response rate across 205 surveys
Listed Sample (R2=0.42)
RDD Sample (R2=0.67)
Coeff.
Coeff.
Overall
(R2=0.52)
Variable
Constant
Coeff.
Prob>t
Prob>t
Prob>t
0.327
0.000
0.425
0.000
0.096
0.024
Salience level 2
−0.783
0.030
−0.177
0.000
0.019
0.549
Salience level 3
0.066
0.007
−0.002
0.946
0.082
0.018
Number of calls per case
0.001
0.776
0.0008
0.870
0.003
0.643
Minutes per piece of sample
0.022
0.001
0.017
0.041
0.034
0.007
Refusal calls
0.005
0.592
0.0002
0.989
0.013
0.447
Survey length
−0.007
0.000
−0.006
0.000
-0.005
0.059
Fielding time
7.078
0.000
5.198
0.007
17.258
0.005
−23.965
0.003
−16.227
0.067
−147.511
0.042
−0.118
0.019
na
na
na
na
Fielding time squared
Listed versus RDD
What would it take to get a 70%
response rate?
Ways people overstate response rates
• Pre-screening samples incorrectly (e.g.
removing persistently unavailable numbers)
• Unreasonable calculations of e
• Purchasing RDD sample from higher density
banks
Response rates with lists
• When sampling from a list most of these rules
must be modified
• The fundamental question is “Who is
eligible?”
• If being on the list makes them eligible then
they are
Analyze the effects of non-response
• If you have information about all potential
respondents, compare respondents to nonrespondents
• Compare converted refusals to non-refusals
• Compare early responders to late responders