Response
Rate
in Surveys
Key resource: Dillman, D.A., Smyth, J.D., Christian, L.M., 2009.
Internet, Mail and Mixed-Mode survey: The tailored design
method, 3rd ed. John Wiley & Sons. Chapter 7, pgs 234-299.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
1
Check your understanding
of response rate
Select agree or disagree for each item:
1.Response rate means that everyone answered
the survey who was supposed to
Agree
Disagree
2. There is one standard response rate
Agree
Disagree
3. A higher response helps us be more sure the
findings represent everyone in the study
population
Agree
Disagree
4. People will respond or not; there is little we can
do about it.
Agree
Disagree
5. If your response rate is low, explain your results Agree
relative to those who responded
Disagree
Check answers on next slide
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
2
Answers –
find explanations in the slides that follow
1. Response rate means that everyone
answered the survey who was supposed to
Agree
Disagree
2. There is one standard response rate
Agree
Disagree
3. A higher response helps us be more sure the
findings represent everyone in the study
population
Agree
Disagree
4. People will respond or not; there is little we
can do about it.
Agree
Disagree
5. If your response rate is low, explain your
results relative to those who responded
Agree
Disagree
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
3
Response rate is…
# that answered
# you contacted
The proportion of people who return the
survey questionnaire.
It is calculated by dividing the number of
returned surveys by the total number of
surveys distributed.
Example: If you distribute 250 questionnaires
and you get 85 questionnaires back, your
response rate is 34%.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
4
Low response to our surveys
We often send out surveys and find that few
are returned. Response rates of 30% and
lower are common in Extension. Often, the
number of returned surveys is too small to
aggregate in a meaningful way or make any
comparisons.
Best practice says that a response rate under
70% should be a warning.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
5
Why is response rate important?
It’s the only way to know if your survey
results are representative.
 High response rate promotes
confidence in results.
 Low response rate increases the
likelihood of biased results.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
6
There is no standard response rate
“The higher, the better.”
While it is not actually the % that matters but
WHO responds, a higher response rate
means that you can be more sure that the
answers reflect the population.
So, we want to remove barriers and
motivate as many people as possible to
complete and return the questionnaire.
How can we do that?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
7
Most important things that influence
response rate:
 Salience of the topic – interest in the topic of
the survey
 Personalized request and communications
 Multiple follow-up contacts
 Sponsor of the survey is respected, trusted
 Questionnaire is concise and easy to
complete
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
8
10 Ways to increase response rate
1. Generate positive publicity for your survey.
2. Appeal to people’s helping tendencies –
ask them to help by providing their input.
3. Make the survey topic salient – important
•
•
Ensure that respondents see the value of
the survey and their response.
Point out their personal connection to the
topic
4. Tailor, personalize communications
5. Make the questionnaire attractive and
easy to complete AND easy to return
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
9
10 ways to increase response rate, cont.
6. Provide incentives (token of appreciation)
7. Show positive regard; Say thank you
8. For mail survey, provide 1st class
postage/return postage
9. Make (multiple) follow-up contacts
– by mail, email, telephone, in person…
10. Use a combination of survey modes –
telephone plus mail; internet plus mail.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
10
Personalize communications
Purpose: to motivate people to answer; appeal
to their helping tendencies
• Use name, if appropriate and possible
• Personalize the importance of the survey to
the respondent
• For mail surveys, use of letterhead printed on
quality paper
• Use recognizable logos and graphics
• Personalize salutations as appropriate
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
11
Incentives
In Extension, we may be limited to what we can offer as
incentives, but research shows that incentives do increase
response rates.
Prepaid incentives work better than promising something if the
questionnaire is returned.
Incentives can be cash ($1 as smallest amount) or
nonmonetary, e.g., a tea bag inserted with a mail questionnaire;
a gift card; state park or movie pass; etc.
Or, a lottery, raffle, or prize drawing (e.g., offer to put names of
all completed and return questionnaires into a ‘raffle’ for a $50
gift certificate to the area’s best restaurant); or, offer to donate
$1 for every questionnaire submitted to the local food pantry.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
12
Multiple contacts
• Consider 3-5 contacts
–
–
–
–
–
Brief pre-notification
Questionnaire mailing with cover letter
Thank you follow-up (post card)
Reminder with replacement questionnaire
Final contact
• More important than number of contacts is that
each has a different look and appeal
• Strategically time all contacts
• Use different modes of contact
– different ways of contacting people (email, mail,
individual)
– different delivery of the questionnaire (url, email
attachment, postal mail, in-person)
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
13
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
14
Response rate may be affected by how
the survey is distributed.
In Zoomerang, there are two options:
1. Use email addresses. you need individual names with current
email addresses. Send invitation plus other communications
- reminders, track respondents, and receive detailed
deployment information.
– This is the only option that allows you to calculate the
response rate
2. Post the url in a public place – when you don’t have
individual addresses. E.g., in newsletter, on web site, bulletin
board; or send to an email list (not individual addresses).
– You may not know who sees/received the url – you do not
know the total number of people who received the
questionnaire.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
15
Address low response
 Determine how people who responded are
different from those who didn’t respond.
 Describe your results in terms of who did
respond. Don’t imply that the results apply
to anyone other than those who responded.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
16
When response rate is low…
• Use language that is suggestive rather
than decisive.
Examples:
“The data suggests” vs. “The data prove”;
“It appears” vs. “It shows”
• Don’t generalize findings to the entire
group.
• Indicate who responded, i.e., who the
data represents.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
17
When response rate is low…
Use a summary statement such as:
The response rate was only 45% so care
should be taken when reviewing these
results. The results should be applied to
those who responded, not all participants.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
18
How would you rate your understanding:
Now after reviewing these slides to
Before you started?
NOW
• Very poor
• Poor
• Fair
• Good
• Very good
BEFORE
• Very poor
• Poor
• Fair
• Good
• Very good
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
19
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
20