366_8
Survey Design
• Things to consider
– Layout
– Time to complete
– Types of questions
– Question wording
– Sensitive items
– THE RESPONDENT
Survey Design
• THE RESPONDENT
– Does she know what you are asking?
– Does she understand the question?
– Can she recall the needed information?
– Does she have a reason to hide something?
Survey Design
• THE RESPONDENT
– May not be paying close attention
• Make survey easy to read, follow
• Use similar response categories for blocks of questions
where possible
• Sensitive items at the end
Survey Design
• THE RESPONDENT
– Needs to know why the survey is being done
• Why bother to do it?
– Needs to know the responses are anonymous.
•
•
•
•
Phone vs..
Mail vs..
Internet
Handed to them (face-to face)
Types of Questions
• Open ended
– What is the most important issue facing the
country today?
– Can you tell me what it is that makes you like the
Democratic Party?
Types of Questions
• Open ended
– Valuable to measure information
– Might be biased toward people with more
information
Types of Questions
• Open ended
– Very labor intensive to enter data
– Must create categories for responses, code into
spreadsheet
Types of Questions
• Closed ended
– Defined response categories
– Overall, how satisfied are you with how
democracy is working in the United States
•
•
•
•
Very satisfied
Satisfied
Not very satisfied
Not satisfied
Types of Questions
• Closed ended
– Do you want a neutral
category?
• Agree strongly
• Agree
• Neither agree nor
disagree
• Disagree
• Disagree strongly
• Closed ended
– Neutral category might
be easy response default
• Too little
• About right
• Too much
• More
• Same
• Less
Types of Questions
• Closed ended
– Are you creating an ordinal (or even interval)
measure?
•
•
•
•
•
1) a very great deal
2) a great deal
3) quite a bit
4) a fair amount
5) a little
6) Very little
7) None
Types of Questions
• Closed ended
– Thermometer scales
• On a scale from 0 to 100, with 0 being cool, 50 being
neutral, and 100 being warm, tell me how you feel
about [whatever]. Just write a number between 0 and
100 in the space below
– [whatever1] _________
– [whatever2]__________
– [whatever3]__________
Types of Questions
• Closed ended
– Forced choice?
• Should response categories include “don’t know”
• Phone vs.. print surveys
– If phone, we might not offer DK, but code it
– If print, if no DK option, respondent might skip
Closed Ended Question Blocks
Question Blocks
Layout:
Question Wording
• Be specific
– “are you liberal or conservative” vs.:
– When it comes to politics, do you consider
yourself liberal or conservative?
Question Wording
• Avoid biasing responses
– Leading questions
• “Do you favor or oppose dirty coal?”
– Loaded words
• “welfare” vs.. Income support for the less affluent
– Double – barreled questions
• Should we raise taxes and cut spending to lower the
deficit?
Question Wording
• Avoid biasing responses
– Is Clinton an old-style, tax & spend liberal, or a
new style Democrat who who will be careful with
the nation’s money? [ABC] vs.:
– Is Clinton a liberal Democrat or a Conservative
Democrat (or, see slide 14).
Question Wording
• Sensitive questions
– If you must ask, ask indirectly
– Drinking, not voting, rioting
• Occasionally, people drink too much and become
[intoxicated]. In the past year, how often have you
become [intoxicated] while drinking any kind of
alcoholic beverage?
Question Wording
•
•
•
•
•
•
•
•
•
•
•
Sensitive questions [Why didn’t you vote?]
In any election, some people are not able to vote because they are sick or busy or
have some other reason, and others do not want to vote. Which best describes
your situation in the recent local election in [CITY]
Illness or disability (your own, or in your family)
Forgot
Registration problems
Not interested; it doesn't matter
Too busy
Disliked the candidates
Out of town
It was inconvenient
Don’t like the voting rules
•
•
Other (VOL)
Don't know (VOL)
More points about the layout
• Begin with brief intro
– Class project, anonymity, etc.
• Follow with ‘easy’ substantive question
– First we’d like to ask you about...
• Start questions casual, chatty, clear
– In general, do you think_______
– Thinking about _____ ,
– How about, _______, Do you think....
More points about the layout
• Most important
substantive items next
• Qs that test your main
hypotheses
• Demographic questions at
end
• Check established surveys
for coding
–
–
–
–
–
–
–
–
Age
Region of state
Education
Party
Ideology
Religion [ or / and ]
Religious attendance
Family income (last)
More points about the layout
• Whatever layout,
include:
• Are you registered to
vote?
– If no, SKIP to Q X
– If Yes, are you registered
to vote here in Whatcom
County, or elsewhere?
Before you print
• Are you sure everything is in there?
– You have what you need to test your main
hypotheses
– You have questions that measure the attitudes
you are interested in
– You have the questions designed to explain
variation in the attitudes you are interested in
More points about the layout
• Printing
– Single sheet
– Back to back
– Booklet
– Font size
Pre-test
• Once DRAFT is completed, give survey to a few
friends
– How long did it take?
– Ask them about it after they take it
• Any items not make sense?
Monitor refusals
• Is there a pattern to who is not filling out
survey?