EMR 6500:
Survey Research
Dr. Chris L. S. Coryn
Kristin A. Hobson
Spring 2013
Agenda
• Implementation procedures
• When more than one survey mode is
needed
Implementation Procedures
Mail Survey Implementation
Mail Survey Implementation
• When high quality implementation
procedures are used, response rates
of 50%-70% are not uncommon
– In my own work, I have had response
rates as high as 90%
• Implementation must receive
considerable thought and planning
– It should not occur after questionnaire
design, but simultaneously
Mail Survey Implementation
• General guidelines
1. To the extent possible, personalize all
contacts to respondents (even when
names are unavailable)
2. Send a token of appreciation with the
survey request
3. Use multiple contacts, each with a
different look and appeal
4. Carefully and strategically time all
contacts
5. Select all mail-out dates with the
characteristics of the population in mind
Mail Survey Implementation
• General guidelines
6. Place information in the mailing exactly
where it needs to be used
7. Take steps to ensure that mailings will not
be mistaken for junk mail or marketing
materials
8. Evaluate the size and weight of mailing
materials on mailing costs
9. Assemble the mailings in a way that
maximizes the appealing aspects of each
element when the package is opened
Mail Survey Implementation
• General guidelines
10.Ensure that all addresses in the sample
comply with current postal regulations
11.Assign an individual ID number to each
sample member
12.Establish procedures for dealing with
undeliverable mail
13.Establish procedures for dealing with
returned incentives
14.Establish procedures for dealing with
respondent inquiries
15.Evaluate early returns for problems that
can be addressed mid-stride
Prenotice Letter
• Sent to respondents a few days prior
to the questionnaire
• It notes that a questionnaire is for an
important survey that will arrive in a
few days and that the person’s
response will be greatly appreciated
• This letter can increase response
rates by 3%-6%
– No recent research comparing postcards
to letters (I prefer the former)
Questionnaire Mailing
• Includes a detailed cover letter
explaining why a response is
important
• The questionnaire
• A prepaid postage envelope
• A token incentive if one is to be
provided
Thank You Postcard
• Sent a few days to a week after the
questionnaire
• The mailing expresses appreciation
for responding and indicates that if
the completed questionnaire has not
yet been mailed it is hoped that it
will be returned soon
Replacement Questionnaire
• Sent to respondents 2 to 4 weeks
after the previous questionnaire
mailing
• It indicates that the person’s
completed questionnaire has not yet
been received and urges the
recipient to respond
Final Contact
• Sent using a different mode of
delivery 2 to 4 weeks after the
previous mailing
• The different mode of contact
distinguishes each type of final
contact from regular mail delivery
• Special contacts have been shown to
increase response
Web Survey Implementation
Web Survey Implementation
• Major concerns associated with web
surveys
– Mode of initial contact
– Extent to which personalized elements
can be used
– Whether and how incentives are
delivered
– Timing of contacts
Web Survey Implementation
• General guidelines
1. To the extent possible, personalize all
contacts to respondents
2. Send a token of appreciation with the
survey request
3. Use multiple contacts and vary the
message across them
4. Carefully and strategically time all
contacts with the population in mind
5. Consider contacting respondents by
another mode when possible
Web Survey Implementation
• General guidelines
6. Keep e-mail contacts short and to the
point
7. Take steps to ensure that e-mails are not
flagged as spam
8. Carefully select the sender name and
address the subject line text for e-mail
communications
9. Provide clear instructions for how to
access the survey
10.Make obvious connections between the
opening screen and other implementation
features
Web Survey Implementation
• General guidelines
11.Assign each sample member a unique ID
number
12.Know and respect the capabilities and
limits of the web server(s)
13.Establish a procedure for dealing with
bounced e-mails
14.Establish procedures for dealing with
returned incentives
15.Establish procedures for dealing with
respondent inquiries
16.Implement a system for monitoring
progress and evaluating early completes
When More than one Survey
Mode is Needed
Why Consider a Mixed-Mode
Survey Design
•
•
•
•
Lower costs
Improve timeliness
Reduce coverage error
Deliver incentives
Four Types of Mixed-Mode
Surveys
Type I
• Use one mode to contact
respondents and to encourage
response by a different mode
Type II
• Use a second mode to collect
responses from the same
respondents for specific questions
within a questionnaire
Type III
• Use alternative modes for different
respondents in the same survey
period
Type IV
• Use a different mode to survey the
same respondents in a later data
collection period
Mixed-Mode Surveys and their
Implications
Type
Motivation
Limitations
I.
Use one mode to contact
respondents and to encourage
response by a different mode
•
•
Improve response rates
Reduce coverage and
nonresponse error
•
Increased implementation
costs
II.
Use a second mode to collect
responses from the same
respondents for specific questions
within a questionnaire
•
Reduce measurement
error
Reduce social desirability
bias for sensitive
question
•
•
Increased design costs
Increased nonresponse if
respondent must respond by
other mode at a later time
Use alternative modes for different
respondents in a the same survey
period
•
•
Improve response rates
Reduce coverage and
nonresponse error
Reduce survey costs
•
•
Increased design costs
Measurement error from
differences that may be
confounded with differences
among subgroups
Different modes become
available to survey
respondents
Reduce survey costs
•
•
Increased design costs
Measurement error from mode
differences that impact the
ability to measure change over
time
III.
•
•
IV.
Use a different mode to survey the
same respondents in a latter data
collection period
•
•
Why Different Survey Modes
Sometimes Produce Different
Answers to Survey Questions
Presence versus Absence of an
Interviewer
•
•
•
•
Locus of control
Normative question order effects
Social desirability
Acquiescence
Aural versus Visual
Communication Effects
• Primacy/recency effects