Surveys: What Are They?
Surveys: What Are They Good For?
• A survey is a collection of questions asked
repetitively to a sample of a population to
mathematically derive characteristics of the
total population.
Why is This Cycle Important?
• It’s a framework
• It provides guidelines and reminders as you
work with clients and stake-holders
• You’re likely doing parts of it already
• Those are likely the parts of your process
that work!
Great Survey Design Cycle
The Trifecta: Need, Design & Act
Unit 1: Need
Set Your Goals and
Objectives
Needs: We All Have Them
• Questions to ask:
– What are we trying to figure out?
– What kinds of reports or data do we want or
expect?
– What will we do with this data when we’re done?
– Who is our intended audience or population?
– How are you going to access the target
audience?
Examples of Need
– How well known is my brand?
– Will customers buy this product?
– If we offer X benefit, will our employee happiness
go up?
– Why are my customers not converting?
– Will my product do well in a new market?
Set a Survey Goal
• Setting goals and objectives for a survey
– Define your goal. A goal is not a single learning
point – a goal is what you are going to do with this
data, and why.
• Good goal: grow your company into new
markets. (“A survey will determine
which markets are good for our existing
products.”)
• Bad goal: make more money for your
business.
Learning Objectives
• Determine your learning objectives
– These should all support your overall need and
goal
– A good amount of learning objectives: three
– You should have no more than five!
Brainstorm Your Questions
Selection and Refinement
Eye on the Prize: ROI
• If you are going to spend more time and money
on running the research for this need than the
overall completion of goals would generate, it’s
a waste of time and money
• If there is no ROI measurement, there is no
encouragement to take action
• Without communication from the start
about possible actions to take, survey
results may have no obvious meaning
Unit 2: Design
Organize Brainstorm
Refine Brainstorm Ideas
into Questions
Guide: Writing Questions
• Multiple choice versus open-text questions
– Quantitative versus qualitative
• Phrasing and language use – unclear
language, grammar, ambiguity can all be
issues
– Remember that language can differ
between demographic groups
• Keep your questions:
– Brief
- Simple
– Relevant
- Specific and direct
Qualitative Versus Quantitative
• You may introduce bias into your survey with
every qualitative answer you ask, unless the
resulting answers are discrete.
• This data should never be added to
quantitative data without the information
being entirely clear in all reporting.
Guide: Exploratory Studies
• Open-text questions
• You should never have a required question
that does not have an opt-out option (this
creates bad data)
The Four Horsemen of the Surveypocalypse
Emotional Bias
• Asking loaded questions
• Asking neutral-seeming questions on a
loaded topic
Identity Bias
• “How much do you love SurveyGizmo?”
• Asking “Do you like SurveyGizmo?” with a
SurveyGizmo logo in the corner of the
survey
Option Bias
•
•
•
•
Required, non-applicable questions
Leading or restrictive options
Different types of scales
Option lists of death
Conversational Bias
• Surveys as a conversation
• Respondents giving the answer they think
you want to hear
Lack of Focus
• Covering too many diverse topics
• Additional questions that do not meet the
survey goal
• Questions that are not inline with the
learning objectives
• Questions that do not derive actionable
results
Miscommunication
• Know your audience and the language that
they use and understand
– Avoid technical terms unless it is
appropriate
– Define terms if necessary
• Remember to speak in your company’s voice
• Have a peer review for clarity
Survey fatigue as a cultural trend
• Cultural survey fatigue
– The average
respondent is fatigued
already, just by nature
of:
• Receiving emails
from organizations
• Suggestions on
receipts and from
cashiers
The Wrap-Up: Question Mistakes to Avoid
• Try to avoid…
– Leading questions
– Loaded or suggestive questions (like our
star rankings)
– Fatiguing question types – large tables,
lots of open-text or essay questions
– Sensitive questions
– Highly technical language
Unit 3: Build
How are Design and Build different?
• Design: Involves
thinking about
psychology, emotions
and words. It is the
more abstract phase.
• Build: Involves taking
into account security
walls, logic, combatting
fatigue, bias, and poor
data collection; It is the
more active phase.
Stages of Build
The Usual Suspects
The Radio Button
• Quantitative
– Scale (should be horizontal)
– Categorical (should be vertical)
• “All of the above” is a no-no!
The Scale: In its Natural Environment
Scale questions: The controversy
Neutral
or not?
NPS: Net Promoter Score
Neutral
or not?
The Checkbox: Choose All That Apply
The Checkbox: Beware!
Choosing more than one option changes
statistical reporting a lot!
Multi-Text Questions
Please list the names of phone providers that you have seen or heard advertised.
• Qualitative
• Explorative or
un-aided
response; used
for lists
Essay Questions
• Qualitative and
explorative
• This is a way to
gather unaided
responses for
your survey
3. What is your favorite thing about SurveyGizmo?
Essay Questions: Your Friend? Maybe.
Table Questions
Do NOT use as a space-saver – these are fatiguing!
Table Questions: What’s Totally Okay
Build your survey.
Test It.
Get buy-in from your
stakeholders.
Why Validation Matters:
Test Reports
• Are your questions reporting the way you
expect?
• Are you able to create the reports you
need using the data you are collecting?
• Is the data in the format you need?
Apply and Test Logic
Different Types of Logic
• Fatigue-fighting:
• Page jumping
• Show-when logic
• Percent branching
• Piping (repeating)
• Bias-fighting:
• Randomization
• Disqualifiers
• Survey timing/combatting straight-lining
• Vote protection
Great Survey Design Cycle
Unit 4: Collect
Survey Mode
Mode introduces different forms of bias – so
how the data is collected is important!
Bias of Sample
• The mode of survey you choose
– Example: Percentage of households
with internet capability in the US versus
households with no internet
• Choosing to email (versus telephone) this
survey will create a highly biased sample
Sample
What is Sample? Why is it Important?
• Your options are: survey everyone, or
survey a percentage
• Why?
– Cost
– Survey fatigue
– You will miss certain sections of the
population
– Using a statistically valid sample is just
as effective (or more effective) than
trying to survey your entire population
More on Sample
A sample is statistically valid when every
single person in that population has a equal
chance or probability to be in a sample that
you select.
What is Sample Size?
How many responses do you need for your
survey to be statistically accurate?
– It depends.
• How accurate do you want the data to
be? (margin of error or confidence
interval)
• How repeatable do you want the
results to be?
• How large is your total population?
How to Determine Sample Size
• Estimate 400 responses
• Use a sample calculator!
Sample Calculators: Magic?
Caveats
#1: If you are segmenting your sample data for
comparison, you need to make sure that the
segments that you are using for comparison
are the same as the segments in the
represented population
– Example: When comparing men and
women in the United States, you would
need to make sure that the ratio within
your survey was the same as the ratio
within the larger US population
Caveats, continued
#2: If you are using cross-tabbing for your
data, you need to ensure that the data you
have (per question that you are cross-tabbing)
is statistically valid for representing the larger
population
Where Do You Get Sample?
• Pull a population from your own customer
list.
– Warning: Do NOT use your entire
customer base.
– If everyone has the same chance of being
randomly selected, you are not biasing
your results in any way.
• Panel Companies: A panel company is an
organization that exists to sell anonymous
survey responses to marketers and market
researchers.
Panel Companies: The Issues
Drawbacks:
– Using incentives
– Cannot access market researchers
– Some panel companies will buy from each
other when they cannot provide the
sample needed
– Hard to determine level of bias in sample
• If the panel companies award “points”
for websites like Amazon – helps
reduce sample bias based on incentive
Incentives
• Biases your sample (ex. Toys R Us
gift card as incentive)
• Incentives can jeopardize your data
(because respondents just want to
get to the end)
• Safeguards:
– Survey page timer with
disqualification
– Shorter surveys
– Red herring questions
– Clean data (eliminating straight
liners, Christmas trees, etc)
Unit 5: Report
Clean your data
How to clean data, Step 1
• Look out for:
– Straight liners or Christmas tree behavior
– Unusually quick responses (when using a
timer)
Warning signs
• Look out for:
– Choosing all checkbox options
Warning signs
Look out for:
– Red herring fails or logically inconsistent
answers
Warning signs
• Look out for:
– Nonsense or missing open-ended
answers
How to clean data, Step 2
• Prepare your data for analysis
– Beware of:
• Inconsistent numeric values (How old
are you? Etc.)
• Breaks in validation
• Do not introduce new bias!
– Changing question text
Finding outliers
Analyzing text responses
• How will you deal with your qualitative
data?
– Keyword frequency
– Word clouds
– Positive/negative
Analyzing text responses
Bucketing
– You can use the SurveyGizmo Open Text
Analysis tool.
Run Preliminary Reports
• Your preliminary reports should be focused on your
original learning objectives.
– Did you get your questions answered?
– Is the data in the format you expected?
– Are you seeing the trends that you anticipated?
• Run individual reports for each learning objective.
– Use this process to determine the “highlights” of
data collected as they relate to ultimate actions
so that you can truly understand the most
significant findings of your research.
Running reports: Key factors
• Make sure your data makes sense
• For any overt trends you are finding in the
data, make note of them and ensure that
they are important towards the objectives
that you had set for your survey
Segmenting data for analysis
• Often, your survey will contain demographic
and firmographic questions to create
segments in your survey.
• These segments should remain the same
from start to finish of the survey process.
Good indicators of a trend:
• When you have data that isn’t statistically
sound but is still interesting, you can call it
“directional data”.
– This data gives you an idea of what your
population is saying, thinking or feeling,
but you cannot use statistics to back it up.
Analyze Your Data
Weight your data during design
Some studies will over-sample certain
populations
– Example: Over-sampling females in an
election poll
Report on your Findings
Suggestions for effective reports
Stage 1: Write a summary
– What was the ultimate goal of this survey?
– Who was surveyed?
– Who was the population?
– Who responded?
– Include basic highlights of the survey
audience and your data to introduce the
findings
Suggestions for effective reports
Stage 2: Write a mini-report for each
individual learning objective (ex: 401K
changes).
• The last section for every learning
objective report will include the
recommended actions to take based on the
results of the survey (these should not be a
surprise!)
Suggestions for effective reports
Stage 3 (optional): Interesting and
unexpected trends found
– Good to know, not need-to-know
– Ex. Perhaps you found a new, unintended
segment of your population that could
help you to make good business
decisions moving forward
• This is going the extra mile for your clients!
Suggestions for effective reports
Stage 4: Conclusion
– Recap what actions are going to be taken (if
any) based on your findings.
– Get all of stakeholders to agree to those
actions.
– Create a survey to be sent to stakeholders in
order to gain feedback for the project and put
actions in motion.
– Important for the next stage, Act: ask
stakeholders to provide metrics that can be
used to measure the success of the actions
that will be taken.
Tips for communicating data
• Try to anticipate questions about the
report
• Know the details
• Be honest
Unit 6: Act
Actions: The key to success!
• In order to ensure that your study has a
purpose, it is important to reiterate and
motivate the stakeholders to take action
based on the data collected.
• It can be helpful to establish a reasonable
timeframe in which actionable results
(positive or negative) can be expected.
Monitor Actions
How to get feedback
• This should be in the form of a short
survey that goes out to all stakeholders in
order to gain feedback on the study.
• Ask for any suggestions they may have so
that you can work better together in the
next study and improve the process.
Publish and Share Study
Results
Great Survey Design Cycle