Collective Wisdom
Designing Surveys for Use in Assessment
R. Ternes, October 2011
Defining the Problem

The most important consideration when
creating a good survey is to identify and
define your problem/question/goal.
◦ Vague problems lead to vague surveys which
lead to vague answers.

Chunk the problem
◦ Break broad categories into manageable and
measurable parts.
Examples

Too Broad
◦ “What is our student learning environment
like?”

Break it down.
◦
◦
◦
◦
◦
How many hours do students study?
How many students study in groups?
How many papers do students write?
How many students attend office hours?
etc. etc.
Sampling and Generalizability

Sampling is linked to your goal.
◦ Tackle sampling after defining your goal.

You must sample from the population that
you want to make generalizations about.
◦ If you want to generalize to all your students,
you must sample from all your students.
 Sampling English majors will tell you only about
English majors.
Convenience Sampling

An ideal survey uses random sampling
drawn from the population of interest.

Often, we have to make compromises to
the way we sample.
◦ We might sample our first-year students by
surveying students in their first year writing
course.

Understand the limitations of your choice!
Understanding Limitations

Consider the CLA, CAAP, MAAP etc. as a
Gen Ed student learning measure.
◦ For seniors, sample from everyone.
◦ For freshmen, sample from students that
enroll in your first year writing program
(because it’s convenient).
 If you do so, do you exclude your AP students?
 Aren’t these some of your best students?
Writing Good Survey Questions

Good questions are:
◦
◦
◦
◦
Short
Simple
Clear
And they obviously relate to the goal or
problem!
Writing Good Survey Questions

Binary or Likert scales are good standards
◦ 5, 7, or 10 point scales are standard.
◦ Right Hand Side = high numbers/pos. phrases
◦ Use n/a only if you need to.
 Always make it a separate option.
Writing Good Survey Questions

Use standard phrasing.
◦
◦
◦
◦
Strongly agree, agree, disagree, etc.
Don’t use words like ‘terrific’ or ‘terrible’.
Keep the language simple and plain.
Try to make the scale ‘balanced’.
 One option for a negative weighting and 3 options
for a positive rating ‘anchors’ the results towards the
positive side.
Standard Format

The drinking age should be lowered from
21 to 18.
◦
◦
◦
◦
◦

Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
This format naturally avoids several
common pitfalls in survey design.
Use Non-Standard Formats Sparingly

Try to avoid non-standard question formats
like:
◦ ‘rank these in order of preference’
◦ ‘select all that apply’

Non-standard formats
◦
◦
◦
◦
are more difficult to analyze statistically,
Are often harder to report,
and they can confuse participants.
Sometimes they can be very useful, but these
questions are easy to do poorly.
Bad Question Overview
Leading Questions
 Manipulating the Question Order
 Double-Barrelled
 ‘Either-or’ Fallacies
 Lack of Variability
 Lengthy/Complex Questions

Leading Questions

Leading questions steer respondents
towards a particular answer.
◦ Bad: “Do you think tuition is too high?”
◦ Better: “How would you best classify your
school’s tuition cost?”





Much higher than other comparable schools
Higher than other comparable schools
About the same as other comparable schools
Lower than other comparable schools
Much lower than other comparable schools
Question Order

Question order is important!

If you are polling opinions on government
involvement in health care:
◦ You are likely to illicit more negative
responses if you first ask a dozen questions
about government spending.

Put controversial questions last, if
possible.
Double-Barreled Questions

Do not roll two questions into one.

“Do you like the flavor and texture of the
cereal?”
◦ If you get a ‘no’ response:
 Is it the flavor they did not like?
 Is it the texture they did not like?
 Is it both?
Turn Either-Or Binaries into Likerts

Resist the temptation to create binary
variables out of everything.
◦ Are you a Democrat or a Republican?
 Such a question ignores people that are
Independent, Libertarian, Green Party, Tea Party, etc.
◦ Researchers can lose valuable information by
dichotomizing continuous variables.

Often binary questions can be easily
turned into Likert type scales.
Lack of Variability

Design questions so that they give you
information!
◦ If everyone answers in the same way, then you
haven’t gained much information.

Bad
◦ Do you want more parking on campus?

Better
◦ Do you think the university should create a $5
student fee to build more parking spots?
Avoid Lengthy Questions and
Double Negatives

Don’t use double negatives.

Simplify, Simplify, Simplify.
◦ Change: “At what convenience store were
you or a member of your family most likely to
purchase their gasoline from in the past
year?”
◦ To: “What gas station do you usually buy
gasoline from?”