Survey Design 201
Getting from “Doing a survey”
to “Doing a survey right.”
Andy Zehner
Office of Assessment
Young Hall, Rm 521
alzehner@purdue.edu
Basic tips
1. Don’t ask what you already know
2. Avoid biased or leading questions
3. Make choice options:
1.
2.
3.
exhaustive
exclusive
balanced
4. Avoid double-barrelled questions
5. Don’t assume knowledge of terms
6.
7.
8.
9.
10.
11.
12.
Avoid jargon
Be brief
Be specific
Do/Don’t include a “neutral” option
Avoid complex negative constructions
Order questions strategically
Save controversial questions for last
Spot the flaw
1. “Rate the quality and variety of the food at Wiley Dining Court”
2. “CCO Career Services are:”
•
Excellent
Outstanding
Very good
Good
Average
3. “Have you participated in any flipped courses during the past year?”
•
Yes
No
4. Were the clerical staff who served you in [office] courteous?
1.
2.
3.
Yes
Neutral
No
5. What portion of the material did you master by the end of the training session?
•
All
Most
Some
Very little
None
New considerations
• Vague quantifiers
• Individual response style
• Dunning-Kruger Effect
Dunning-Kruger Effect
Kruger & Dunning, Journal of Personality and Social Psychology, 1999
What evidence of Dunning Kruger Effect looks like
60%
50%
Freshmen 1
Freshmen 2
40%
Seniors 1
Seniors 2
30%
20%
10%
0%
Very poor
Poor
Fair
Good
Very good
Excellent
2014 Purdue SERU survey
Vague quantifiers
Vague quantifiers
How often have you…
Had a class where the professor knew your name
Communicated with an instructor by email/in person
30%
30%
25%
25%
20%
20%
15%
15%
10%
10%
5%
5%
0%
0%
2014 SERU Survey, Purdue data
Vague quantifiers
•
•
•
•
•
•
•
“Always” almost always means always
“Never” almost always means never
Any other vague quantifier is . . . vague
Vague quantifiers convey relative position only
Meaning varies from one person to another
Meaning varies from one question to another
Grammatical opposites aren’t perceived equally
“Sometimes”
•
•
•
•
“asking questions in class”
“working with other students outside of class”
“making a class presentation”
“participating in a community-based project”
 2 times per week
 1 time per month
 2 times per semester
 1 time per year
How Often is Often Revisited: the Meaning and Linearity of Vague Quantifiers Used on the
National Survey of Student Engagement -- Laird, et al., Indiana University, 2008
Vague quantifiers of behavioral
frequency, Marincic, 2011
“Stay away from using words like
“often,” “usually,” “generally,” etc.
Each person’s thought process is
different and some people may infer
a different meaning.”
-- Smart Survey Design, Survey
Monkey, 2011
Retrieved 3/11/2015
Consider using idiosyncratic terms!
2013 Purdue Student Satisfaction Survey
Response style
Moderate & Extreme Response
Bursar’s drop-box
Registering for classes
90%
90%
80%
80%
70%
70%
60%
60%
50%
50%
40%
40%
30%
30%
20%
20%
10%
10%
0%
0%
Importance
Satisfaction
Importance
Satisfaction
2010 Purdue Student Importance & Satisfaction Survey
“Americans use a lot of hyperbole:
‘It’s super, it’s wonderful!’ That is
not language we use where
I come from.”
“I feel like here in the US
people use stronger language,
like ‘Oh, this is so wonderful.’
But for me, I’m not so willing
to say it.”
Quotes from focus groups, Purdue, 2012-2013
Factors that determine acquiescence
•
•
•
•
Individualism
Power distance
Extraversion
Uncertainty avoidance
-.445
.372
.446
.533
*
*
**
*
Response styles in cross-national survey research:
a 26-country study, Harzing, et al, 2006
Response styles by nationality
30
Extreme Response Style
Mexico
India
25
Turkey
20
Germany
15
China
Japan
10
Holland
5
30
35
40
45
Acquiescence
50
55
60
65
Some words don’t translate well
“good” v.
良い / いい
“normal” v.
нормальный
Does America spend too little, about the right
amount, or too much money on…
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Too little
About right
"Welfare"
Too much
"Assistance to the poor"
General Social Survey, 2014
Again: Does America spend too little, about the
right amount, or too much money on…
Liberal respondents
Conservative respondents
90%
90%
80%
80%
70%
70%
60%
60%
50%
50%
40%
40%
30%
30%
20%
20%
10%
10%
0%
0%
Too little
"Welfare"
About right
Too much
"Assistance to the poor"
Too little
"Welfare"
About right
Too much
"Assistance to the poor"
Order of choice options matters
Purdue’s favorite pizza:
Hot Box / Mad Mushroom / Villa@PMU / Domino’s
Domino’s / Villa@PMU / Mad Mushroom / Hot Box
Biasing Effects of Scale-Checking Style in Response to a Likert Scale,
American Statistical Assn, Friedman, et al., 1994
Numbers on the scale matter,
as much as the words do
0 1 2 3 4 5 6 7 8 9 10
Not at all
Successful
Extremely
successful
-5 -4 -3 -2 -1 0 1 2 3 4 5
Not at all
Successful
Extremely
successful
Rating the Rating Scales, Journal of Marketing Management, Friedman & Amoo, 1999
You can get reliable results with a moderate
sample size or response rate
40%
35%
30%
25%
20%
15%
10%
5%
0%
40%
35%
30%
25%
20%
15%
10%
5%
0%
Strongly
disagree
Disagree
Neutral
True distribution
Agree
Strongly
disagree
Strongly
agree
Sample of 5000
40%
35%
30%
25%
20%
15%
10%
5%
0%
Disagree
Neutral
True distribution
Agree
Strongly
agree
Sample of 1000
40%
35%
30%
25%
20%
15%
10%
5%
0%
Strongly
disagree
Disagree
Neutral
True distribution
Agree
Sample of 500
Strongly
agree
Strongly
disagree
Disagree
Neutral
True distribution
Agree
Sample of 50
Strongly
agree
After the shot
went off, my
training took over.
Resources and links:
A good overall lesson in 106 slides
Vague quantifiers: How Often is Often?
Survey Monkey’s guide
Vague quantifiers: Marincic dissertation
Cross–national response styles
More about importance/performance analysis
Rating the Rating Scales
Tex Grebner’s video (language)