Survey Response Rates:
Trends and Standards
Karen Donelan, ScD
Senior Scientist in Health Policy
Massachusetts General Hospital/Harvard Medical School
AcademyHealth - June 27th, 2006
Overview






What is a response rate?
Why do we care about response rates?
Trends
Does nonresponse = biased response?
How should rates be calculated and reported?
How can we improve?
What is a Response Rate?

The number of complete interviews
with reporting units divided by the
number of eligible reporting units in the
sample
Other Rates

Cooperation rates - The proportion of all
cases interviewed of all eligible units ever
contacted.

Refusal rates - The proportion of all cases in
which a sampled unit or the respondent refuses
to be interviewed, or breaks-off an interview, of
all potentially eligible cases.

Contact rates - The proportion of all cases in
which some responsible sampled unit member
was reached.
Why Do We Care About
Response Rates?

Sources of Error in Surveys


Coverage, Measurement, Nonresponse,
Processing, Sampling
Practical implications




Generalizability
Publishability (often cutoff or major test of
quality)
Credibility
Fundability
What Do We Know About
Response Rate Trends?
Current Population Survey (CPS)
http://www.census.gov/prod/2000pubs/tp63.pdf
Behavioral Risk Factors Survey
(BRFSS) : Telephone Survey
Source: Groves et al. 2004
U Michigan: Survey of Consumers
Source: Groves et al. 2004
Why Does Nonresponse Happen?












Who is in the sample? (demographics, SES, lifestyle)
# Contacts
Schedule of contacts
Mode of contact
Respondent selection
Respondent cooperation (refusal)
Incentives ($, benefit)
Respondent burden (time, boredom, frustration)
Salience
Respondent ability to respond to questions
Sponsorship
Privacy concerns
How Should Rates Be Calculated and
Reported? Disclosure


Units drawn and attempted
 Addresses mailed
 Phone numbers called
 Households approached
 Patients invited
Sample Disposition: what happened?
 Eligible and ineligible respondents
 Bad contact information
 How many attempts, at what intervals
 # Never reached, # actually contacted
 # completed, refused, never got a decision
www.aapor.org
Does Nonresponse = Biased
Response?



Low response rates may not be problematic
Representativeness doesn’t necessarily increase
with response rate
Sample representativeness is a function of the
difference between respondents and
nonrespondents on the statistic(s) of interest
 High response rates can can yield an
unrepresentative sample (high
nonresponse bias)
 Low response rates can still yield a
representative sample

Keeter et al. (2000), Curtin, Presser and Singer
(2000), Merkle and Edelman (2002)
Measuring Nonresponse Bias



With response rates falling, understanding the
impact on study findings is essential
Assess quantitatively if possible, but at least
consider qualitatively
Methods for assessing nonresponse bias:





Response rate comparisons across groups
Follow-up interviews with non-respondents
Comparing early vs. late respondents
Comparisons to similar estimates from other sources
Post survey adjustment for nonresponse
Ways Not to Approach Nonresponse





Ignore it
Obscure it
Omit it
Fail to measure/comment on potential
associated bias
Give up on it
Improving Response Rates








Multiple contacts
Multiple modes of contact
Interviewer training
Advance Notification/Endorsement
Incentives
Reduce respondent burden
Increase relevance to respondent
Time and money to get quality
Reporting Response Rates






Know and apply standards (AAPOR, CASRO)
Engage and manage with vendors
Structure sample disposition
Response and nonresponse analysis
Publish enough information to allow others to do
the calculations
Peer review: build awareness of standards