How to Design Effective
Web Surveys
2012-2013 Workshop in Methods
October 19, 2012
Kevin Tharp, Lilian Yahng, and Ashley Bowers
Outline
• Overview of Exciting Opportunities and Challenges of
Web Surveys
• Effective Layout and Design of Web Surveys
• Implementation of Web Surveys to Maximize Data
Quality
• Considerations in Selecting Web Survey Tool
Opportunities of Web Surveys
• Improve survey measurement and reduce error
• Eliminates interviewer effects
• Features of computer-assisted instrument
(customized wording, automated skips/routing, edit
checks, randomization)
• Can engage respondent
• Can present rich multi-media information
• Anyone can build even a fairly sophisticated survey
quickly and easily
B6. How important have each of the following sources been in helping you learn how
to perform your job? If you were offered or made aware of a source but did not use it,
please select “Did not use”. If you were not offered or made aware of a source, please
indicate “Not available”.
Very
Important
Moderately Somewhat
Important
Important
Not
Important
At All
Did Not
Use
Not
Available
a. General SRO Orientation
session






b. General Interviewing
Techniques (GIT) training






c. Technical Tools training






d. Initial orientation (formal or
informal) provided by functional
team supervisor






e. Initial orientation (formal or
informal) provided by
administrative staff in your unit
(e.g., Ann, Liza, Nancy B)






f. Ongoing assistance from the
buddy formally assigned to me






g. Ongoing informal help from
peers/co-workers on my
functional team (e.g., TSG,
PDMG, DCO, etc.)






B6a.
[Programmer Note: Please only ask this question if more than three sources in
Question 6 are marked as “Very Important”] You listed the following sources as
being very important to helping you learn how to perform your job well:
ANSWER 1: FILL a-g
ANSWER 2: FILL a-g
ANSWER 3: FILL a-g
ANSWER 4: FILL a-g
ANSWER 5: FILL a-g
…
Of these, which three were the most important in helping you learn how to
perform your job well?
1. [DROP-DOWN MENU WITH ANSWERS 1-5…]
2. [DROP-DOWN MENU WITH ANSWERS 1-5…]
3. [DROP-DOWN MENU WITH ANSWERS 1-5…]
Complicated Skips and Customized Wording
Randomization
Randomization
Ease of Adding Access to Other Information
(THANK YOU SCREEN)
Thank you for your participation in this survey!
We would like to provide you with some resources that
would be valuable to refer to while you are working at
SRO.
• For a list of SRO acronyms: please go to
https://webtrak.isr.umich.edu/sro/
• For general understanding of survey research:
Survey Methodology (2nd edition), by: Robert M.
Groves, et al. and the Encyclopedia of Survey
Research Methods, edited by: Paul Lavrakas.
Completeness Check
http://www.270towin.com/
http://media3.surveycenter.com/FlashDevs/NPD_produ
cts/demo/black/MediaPlayers/analyzer/player.asp
http://media4.surveycenter.com/FlashDevs/WebSite/Em
otionPicker/index.html
Opportunities of Web Surveys (2)
•
•
•
•
Lowers cost
Greater speed
Environmentally friendly
Reduces respondent burden - complete survey at
convenience and over multiple time periods
• Wealth of survey process data for analysis
Challenges of Web Surveys
• Reduce quality of survey measurement
• Lack of interviewer presence
• Constrained format (versus paper)
• Look and feel not under direct control and may affect
measurement
• Browser, connection type and speed, font size
• Type of device used (smartphone, iPad, laptop,
desktop)
• Anyone can easily and quickly put together a survey
http://www.quizrocket.com/pirate-vs-ninja-quiz
Optimized for Device
• Please text TOLUNA8 to the following #: 91318
http://www.polleverywhere
Challenges of Web Surveys (2)
• Poor reporting behavior (person respond multiple times,
straightlining)
• Complexity – more time and money
• Lower response rates
• No comprehensive list of email addresses for US
population
Web Survey Layout and Design
Strategies for
maximizing response and minimizing error
Case study in web survey design:
NSSE (National Survey of Student Engagement)
•
•
•
•
Annual survey of college students
600+ institutions w/ customization
Large sample (1 mil+)
Administered since 2000
– Mostly paper originally
– Change in web design every few years
Scales
• 2008
• 2010
Scales (2012)
Birth year
• 2003
• 2008
Birth year (2012)
Major (2000)
Major (2003)
Major (2008)
Major (2012)
Pagination
•
•
•
•
•
nsse 2000
nsse 2003
nsse 2008
nsse 2010
nsse 2012 (2.0 pilot)
2013?
•
•
•
•
Progress indicator
Submit on click
More paths
Mobile challenges
Web Survey Implementation
Strategies for
maximizing response and minimizing error
“Implementation” – data collection protocols
Implementation Aims:
• Maximize unit response (response rate)
• Minimize “error” (deviations in survey
processes that lead to inaccuracy or bias)
•
With regard to implementation:
nonresponse error
But this is not the only source of error!
High response rates do not guarantee the
absence of response bias. Neither do low
response rates necessitate response bias.
Center for Survey Research
October 19, 2012
TOTAL SURVEY ERROR
Groves R M , Lyberg L Public Opin Q 2010;74:849-879
© The Author 2011. Published by Oxford University Press on behalf of the American Association
for Public Opinion Research. All rights reserved. For permissions, please e-mail:
journals.permissions@oup.com
High response rates reduce the risk of
response bias.
So, how to turn sample members into
respondents?
(without introducing more potential for error)
Center for Survey Research
October 19, 2012
• Think strategically about your target
population
•
•
What would induce them in particular to take the survey?
What would facilitate their giving accurate responses?
• Monitor your data during data collection
for responsive measures
•
•
Look at the survey data: response distributions by subsample, variable
crosstabs, open-response items, etc.
Look at process data: track individuals, break-offs, times of completion
• Construct responsive (possibly corrective) strategies
• Try experimenting, particularly if multiple administrations or years
Center for Survey Research
October 19, 2012
Implementation Planning
Checklist








Sample preparation
Data storage and archiving
Survey distribution, timing, field period
Follow-up and nonresponse protocols
Email message text and signature
Incentives
Leave time for testing
And… do a pilot, if possible!
Center for Survey Research
October 19, 2012
Sample Preparation
 Sample preparation
•
•
•
•
How is the sample to be acquired? How current is the contact
information? How reliable are the email addresses?
Need to assign case id numbers?
Append other (e.g. demographic or subsample) information?
Check the list for blank fields, duplicate emails, conflict of
interest, ineligibles, etc.
 Data management: storage and archiving plan
•
•
Center for Survey Research
What is the level of security needed for the project/grant?
IU Libraries Data Management Resources
October 19, 2012
Email Messages
Write to your audience (target population):
 Content/composition: Survey duration, sponsor,
purpose, study contact info. First person
singular/plural? Length of message? Deadline?
 Branding your survey: Logo? Survey title?
 Signature: Who should sign it? An individual vs. an
entity?
 Survey link: Placement? Masked link?
 Subject and From lines
 Tech considerations: html vs. plain text, images,
emphasis tags, email clients, etc.
Center for Survey Research
October 19, 2012
Survey Distribution
• Method(s) of contact? Via individual email addresses,
list-serv, on a website, postal mail, phone call, text
message, something else?
• Make it convenient for your target population:
 Timing: Month, day, time? Holidays or special
schedules?
 Follow-up reminders: How many, time between
them? Different contact method?
 Field length
Center for Survey Research
October 19, 2012
Incentives
• Do they work? How much of a boost can
be expected?
• What kinds of incentives are most
effective?
• Differential incentives?
• See university or grant policies on prizes
and lotteries
Center for Survey Research
October 19, 2012
Nonresponse
What kind of nonresponse was it?
• Delivery failure (noncontact)
• Refusal to participate (busy, not interested)
• Inability to participate (language, visual
impairment)
• Mistaken perceptions of ineligibility
Compare to other data?
Post-survey interviews?
Center for Survey Research
October 19, 2012
Selecting a Web Survey Tool
Some Considerations and Available Sets of Tools
What Are Top 2-3 Features You Would
Need or Want to Have in Web Survey Tool?
Disclaimer: I do not endorse any web survey tool. This
presentation is only one person’s view. Only a few tools are
mentioned. This is not intended to be a comprehensive
review.
Some Considerations in Selecting Web Survey
Tool
•
•
•
•
•
•
•
•
•
•
•
Survey Complexity
Layout Customization
Multilanguage Capabilities
Ease of Development/Programming
Able to Integrate with Other Modes of Data Collection
Data Output and Analysis
Security
Number of Users/Scalability
Integrated Email and Sample Management Capabilities
Cost
Help and Support
Set #1
• Examples: SurveyMonkey, SurveyExpression,
SurveyGizmo, QuestionPro, eSurveysPro, FluidSurveys,
PollDaddy…………………………many more
• Generally less expensive or even free
• Fair amount of features for designing survey but may be
limited somewhat in customization (customized wording)
• Easy to program/develop
• Less sophisticated in data output and analysis
• More limited in scalability/number of users
• Data stored in company servers with differing security
protocols
Set #2
• Examples: Qualtrics, LimeSurvey, DatStat Illume, Key
Survey
• Generally more expensive than Set #1
• More options for survey design although still not as
flexible and customizable as in-house developed product
• Fairly easy to program/develop
• More sophisticated in data output and analysis
• Data stored in company servers with often stringent
security protocols to meet government or other
standards
• Additional option to consider: Email management tools
with survey add-on (e.g., Constant Contact)
Set #3
•
•
•
•
Examples: In-House Systems, Blaise IS, CASES Web
More expensive
Highly scalable
Security developed by organization so can meet
stringent standards
• High degree of customization and flexibility
• Highly complex instruments in multiple languages
possible
• Possibility of integrating with other modes of data
collection
Sources for Presentation/
Resources
• Couper, Designing Effective Web Surveys
• Dillman, Internet, Mail and Mixed Mode Surveys:
The Tailored Design Method
• http://www.websm.org/
• Survey Geek Blog:
http://regbaker.typepad.com/regs_blog/
• Comparisons of web survey tools, e.g.:
http://www.idealware.org/articles/fgt_online_surv
eys.php
Thank you!
For more information:
Kevin Tharp
Lilian Yahng
Ashley Bowers
kwtharp@indiana.edu lyahng@indiana.edu afbowers@indiana.edu
Indiana University
Center for Survey Research
Center for Survey Research
October 19, 2012