Surveys
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
1
Survey
A structured way to collect standardized information
from individuals using a questionnaire.
– Surveys may be conducted once; at repeated
intervals, or concurrently with multiple samples
– They may be used to collect information from
a few or many
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
2
This set of slides
draws on the booklet
seen at left and other
resources. We will
cover:
Types of surveys
Pros and cons of surveys
Steps in conducting a survey
Response rate
Cover letter
http://learningstore.uwex.edu/pdf/G3658-10.PDF
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
3
Checking in…
What do you think? Answer YES or NO to
each of the following
1.
2.
3.
4.
5.
6.
A survey is always appropriate
Surveys are one of the most popular ways to collect
information
An email or online survey is better than the old mail or
telephone surveys
Careful planning is necessary
Advance notice to potential respondents helps
increase response rate
A low response rate increases the likelihood of biased
results
Check your answers
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
4
Check your answers
1.
2.
3.
4.
5.
6.
A survey is always appropriate - NO
Surveys are one of the most popular ways to collect
information - YES
An email or online survey is better than the old mail or
telephone surveys – NO, not necessarily
Careful planning is necessary - YES
Advance notice to potential respondents helps
increase response rate - YES
A low response rate increases the likelihood of biased
results - YES
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
5
Surveys are used when…
• You want to collect information from
individuals (vs. a group or collective)
• You want standardized information from
everyone
• Potential respondents can read and write
• You want information from many people
• Privacy is important or independent opinions
and responses are needed
• You have resources to send, track, analyze
and interpret the questionnaires
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
6
Pros and cons of survey
PROS
• Way to collect
information from many
people; dispersed
people
• Person can remain
anonymous
• Provides standardized
information across
respondents
• Allows easy tabulation
CONS
• Results can be easily
biased
• Can miss important
information –
questions and answer
choices are
predetermined
• Requires literacy skills
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
7
Will using a survey be culturally appropriate?
Things to consider:
• Literacy level
• Tradition of reading, writing
• Setting
• Not best choice for people with oral tradition
• Translation (demands more than just literal
translation)
• How cultural traits affect response
• How to sequence the questions
• Pretesting the questionnaire when it may be viewed
as intrusive
• Computer access and use if an electronic survey
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
8
Types of surveys
1.
2.
3.
4.
5.
6.
7.
Hand-out
Mail
Telephone
Face-to-face
Email
Web survey – Online survey
Mixed mode: uses two or more of above
Recommendation: use a mix of modes to ensure that
everyone can and does respond
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
9
Which type of survey should I choose?
“It depends”… upon
– What you want to know – how complex or
sensitive the information is
– Who the respondents are – their
characteristics and which type of survey will
be most appropriate
– Your time line, and
– Available resources
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
10
If you’ve determined that a survey is the best and
most appropriate way to collect information, then
take some time to plan your survey.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
11
Planning a survey: aspects to think about
1.
Determine who should be involved in conducting the
survey - - engage them
Define what information you will collect
2.
–
–
3.
Identify the respondents
–
4.
5.
What do you want to know?
How will use the information?
Determine sampling strategy, if a sample is to be used
Select how the survey will be distributed: telephone,
mail, hand-out, email, web-based
Think about data analysis – what will the end
product/final report include (keep ‘the end in mind’!)
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
12
Survey planning continued…
6.
7.
8.
Develop the questionnaire
Pilot test the questionnaire and other materials
Develop a communication strategy to garner
support for the survey
9. Consider budget, timeline, and management
process
– What resources are available?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
13
Implementing your survey
For tips and examples for ‘getting your survey
done’, see pages 9-15 in the Survey booklet
http://learningstore.uwex.edu/pdf/G3658-10.PDF
See the ‘Questionnaire Design’ section of this web
site for help with developing your survey
questionnaire.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
14
A note on anonymity and confidentiality in
surveys
Anonymous means that NO ONE can identify who
provided the information
– This may be difficult to assure if there is a need to follow-up
with non-respondents or when the survey is administered
online (internet or intranet)…so, don’t promise anonymity!
Confidentiality means that you are able to identify the
person but you guarantee that the information will not
be identified with the person
– This applies to all aspects of data collection, analysis and
reporting
– When reporting and communicating, ensure that no names
or other identifying information is used
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
15
Response rate
The proportion of people who respond:
# that answered = response rate
# you contacted
Example: If you distribute 50 questionnaires
and you get 25 questionnaires back, your
response rate is 50%.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
16
Response rate
 High response rate promotes confidence in
results.
 Lower response rate increases the likelihood
of biased results.
See pages 15-17 in the survey booklet
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
17
There is no standard response rate
“The higher, the better.”
Anything under 60% is a warning.
Why is a high return important?
It’s the only way to know if your
results are representative.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
18
If your response is low, address it!
 Determine how people who responded are
different from those who didn’t respond.
Describe your results in terms of who did
respond. Don’t imply that the results apply to
anyone other than those who responded.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
19
KEYS getting a high response rate
• The survey topic is of interest to the respondents
(called saliency)
• Personalized request and communications
related to the survey
• KISS: Keep It Short and Simple
• Follow-up
• Trust, respect, like the sponsor
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
20
Ways to increase response rate
• Generate positive publicity for your survey.
• Appeal to people’s helping tendencies – ask
them to help.
• Make the topic salient - seem important
– Ensure that respondents see the value of
participating.
– Point out personal connection to the topic
• Tailor, personalize communications
• Make the questionnaire interesting-short and
easy to complete AND easy to return
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
21
How to increase response rate, cont.
•
•
•
•
Provide incentives
Show positive regard; Say thank you
Indicate that opportunities to respond are limited
For mail survey, provide 1st class postage/return
postage.
• Over sample
• Use a combination of survey modes – telephone plus
mail
• Make (multiple) follow-up contacts – by mail, email,
telephone, in person…
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
22
If response rate is low…
• Use language that is suggestive rather than
decisive.
For example: “The data suggests” vs. “The
data prove”; “It appears” vs. “It shows”
• Don’t generalize findings to the entire group.
• Clearly describe who responded, i.e., who the
data represents.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
23
Cover Letters
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
24
A good cover letter includes…
• The purpose of the survey and its importance
• Survey sponsor − use letterhead if a mail
survey
• Why the respondent was selected to
participate
• How the information will be used
• Assurance of anonymity or confidentiality Human Subjects Protection
• Instructions for returning the survey
• Date - when to respond by
• How results will be made available
• Your contact information
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
25
Cover letters − Tips
• Personalize the letter in salutation or
signature
– Connect with the respondent
• Hand-sign the letter
• Express appreciation for their participation
• Keep it short – not wordy
• For mail survey, include pre-addressed,
stamped return envelope
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
26
Cover letters – Pilot-test
Remember to pilot -test your
cover letter just as you pilot
-Test your questionnaire!
 May be a colleague or collaborator or other
stakeholder will check your cover letter for
you.
Is the cover letter likely to motivate the person
to respond?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
27
Checking back in…
Spend a few minutes reflecting
on what we covered in these slides
1.
2.
What, if anything, did you learn that you didn’t
know before?
What is one thing you will do when you develop
a survey?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
28
Resources
• Dillman, D., Smyth, J., Christian, LM. 2009.
Internet, Mail and Mixed-Mode Surveys: The
tailored design method. John Wiley and Sons.
• Fink, A. 2003. The Survey Kit. Sage Publications
• Scheuren, F. What is a survey. American
Statistical Association. Free download at
http://www.whatisasurvey.info/
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
29
• Creative research systems. The survey system,
at
http://www.surveysystem.com/sdesign.htm
• University of WI-Extension, Program
Development and Evaluation
– Collecting Evaluation Data: Surveys
– Questionnaire Design: Asking questions with a
purpose
– Sampling
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
30