Developing survey questions
Problems that careful questionnaire
design can alleviate
 Consistency effect – respondents attempt to
make their later answers fit the earlier ones
 Fatigue effect – questions toward the end of
the survey are not carefully answered
 Redundancy effect – all questions in the
same style are answered in the same way
Question types
 There are two major types of questions
 Closed-ended questions
 Open-ended questions
Closed-ended questions
 The respondent is provided with a limited
number of response options
 Each
response option has a numerical value or
code it represents
Closed-ended questions
 Examples:
 Are you:
 18-34 years of age (1)
 35-54 years of age (2)
 55+ years of age (3)
Closed-ended questions
 If you were about to
 1. Magazines
purchase a digital TV,
 2. Advertising
which source of
 3. Salesperson
information, if any,
would you consult first?  4. Product brochure
 5. Friends
 6. Other
 7. Would not
consult information
source
Closed-ended questions
 Please say whether you
 Strongly agree (5),
 agree somewhat (4),
 neither agree nor disagree (3),
 disagree somewhat (2)
 or strongly disagree (1)
with the following statement:
“People who spend over 400 dollars on a cell
phone are vain.”
Open-ended questions
 The question is standardized, but the
respondent answers in her own words.
 The interviewer usually ‘probes’ at least
once for expansion and/or specificity after
the initial response.
 Example:
 “What
features would the ideal science
fiction movie have?”
Closed-ended v. open-ended
 Closed-ended questions take less time to
complete
 Closed-ended questions take less time to
enter into the database
 Closed-ended questions allow the
respondent to interpret her position in
relation to the question
Closed-ended v. open-ended
 Open-ended questions allow the
respondent to elaborate and to make
connections you may not have anticipated
 Answers to open-ended questions must be
‘coded’ in order to be interpreted in a
quantitative manner
 Respondents often prefer open-ended
questions and are bored and frustrated by
too many closed-ended questions
Closed-ended v. open-ended
 Because they take longer and are more
difficult to deal with, open-ended
questions should be used sparingly
 Mix
open-ended questions through the survey
instrument, breaking it up and adding interest
Question wording
 Concrete questions are best
a. Are you happy with the service at Radio
Shack?
b. Better: Were you served within 5 minutes the
last time you went to Radio Shack?
Question wording
 Avoid abbreviations
 In your view, does USC provide a liberal arts
education worth its yearly tuition?
 In your view, does the University of South
Charleston provide a liberal arts education
worth its yearly tuition?
Question wording
 Avoid slang and colloquial expressions
 Should president Obama take his economic
advisers behind the woodshed?
 Avoid jargon and technical terms
 Should a summative evaluation of Head Start
be commissioned by the U.S. government?
 Better: Should the U.S. government
commission a history of Head Start to review
its activities and accomplishments?
Question wording
 Avoid biasing words and phrases
 The left-wing student group protesting U.S. presence
in Sumatra claims that American presence there is
unwarranted. Do you agree??
 An exception: If necessary, use loaded questions
to prompt response, but be careful, respondents
may be offended

Parents get really angry at their children sometimes.
In the past week, have you been really angry at your
son?
Question wording
 Avoid double-barreled questions
 Do you think the University of Kentucky
should subsidize student tuition and staff
benefits?
 Would you say that the decision to increase
President Lee Todd’s salary by 67% will be a
boon to the University of Kentucky or don’t you
care?

You could think it would be a boon but not care
Question wording
 Avoid negative wording
 Double negatives are especially bad
 “Please indicate whether you strongly agree, agree,
neither agree nor disagree, disagree or strongly
disagree with the following statement:
The United Nations should not have more authority to
intervene in a nation’s military affairs.
 Either emphasize NOT when asking the question or else
reword the question in a positive manner

Proper scale construction
 Response options on scales should be
 1) Mutually exclusive
 2) Equivalent
 3) Exhaustive
Mutual exclusivity
 A given respondent or item should not fit
into more than one category
 Example:
 Are
you:
 a) Single; b) Married; c) Divorced;
d) Widowed; e) Separated, or f) Living in a
couple relationship but not married
• In one interview I heard, a 70+ year old woman said
“All of the above”
Category equivalency
 Categories in a scale must differ on a single
dimension
 Example: Would you say that the service in
the shoe department is:
 a)
fast
 b) slow
 c) polite
 d) knowledgeable
Exhaustiveness
 All respondents or items must be
classifiable in one of the categories
 Example: Does your Herald-Leader arrive
on your doorstep by 6 AM:
a) Always
b) Seldom
c) Never
 What about “usually”?
Scales used to measure
knowledge, attitudes, beliefs
 Developing measures of knowledge,
attitudes, and/or beliefs is a challenging
task.
 While anyone can come up with a
measure, it is hard to develop a truly valid
measure of KAB’s
Main types of attitude
measures
 Semantic differentials
 Likert scales
 Thurstone scales
Semantic differential
 Good __ __ __ __ __ __ __ Bad
 Weak __ __ __ __ __ __ __ Strong
 Beautiful __ __ __ __ __ __ __ Ugly
Likert scales
 Example: The Employment Self Esteem
Scale
 INSTRUCTIONS: Please rate how strongly you agree or
disagree with each of the following statements by placing a
check mark in the appropriate box.
 (Strongly Disagree/Somewhat Disagree/ Somewhat
Agree/Strongly Agree)
1. I feel good about my work on the job.
2. On the whole, I get along well with others at work.
3. I am proud of my ability to cope with difficulties at work.
4. When I feel uncomfortable at work, I know how to handle it.
5. I can tell that other people at work are glad to have me there.
6. I know I'll be able to cope with work for as long as I want.
7. I am proud of my relationship with my supervisor at work.
8. I am confident that I can handle my job without constant
assistance.
9. I feel like I make a useful contribution at work.
10. I can tell that my coworkers respect me.
Guttman scale
 Statements are ordered so that they
represent increasing agreement with or
acceptance of one position on a dimension
 If
respondent accepts one statement higher on
the scale, he should also accept those below it
on the scale
A type of Guttman scale:
Bogardus social distance scale
 I am willing to permit immigrants to live in my country
 I am willing to permit immigrants to live in my community
 I am willing to permit immigrants to live in my neighborhood
 I am willing to permit immigrants to live next door to me
 I am willing to have immigrants for friends
 I am willing to have an immigrant marry my son/daughter
 Agreement with item 3 implies agreement with items 1 and
2, and so on
Thurstone scales
 Thurstone scales include a number of
statements that have been developed to
represent their position or magnitude on
some dimension
 The average of the scores of the statements
agreed with by the respondent provides her
placement on the scale
 People with AIDS are like my parents.
 Because AIDS is preventable, we should focus our resources on









prevention instead of curing.
People with AIDS deserve what they got.
Aids affects us all.
People with AIDS should be treated just like everybody else.
AIDS will never happen to me.
It's easy to get AIDS.
AIDS doesn't have a preference, anyone can get it.
AIDS is a disease that anyone can get if they are not careful.
If you have AIDS, you can still lead a normal life.
AIDS is good because it helps control the population.
 I can't get AIDS if I'm in a monogamous relationship.
Response option considerations
 “Don’t know” option
 Researchers debate whether the interviewer
should explicitly ask whether the respondent
doesn’t know an answer
 Fear that the DK option gives the respondent an easy
out without thinking about the question
 If
an explicit “Don’t Know” option is provided,
you’ll get more DKs but your conclusions may
actually be more valid
 “Neutral” option
 Should scales have a middle option (oddnumbered) or no neutral point (even-numbered)?
 Even-numbered scales force respondents to take
a stand
Indexes
 Indexes combine scores from multiple items to
generate a single score for each respondent on
some variable of interest

Thurstone scales are indexes
 Some variables are complex, needing several
items to provide a valid measure
 The items could be organized in a single
dimension (unidimensional) or in several
dimensions (multidimensional)
Unidimensional indexes
 Summated index
 measured by a set of indicators that can be
added together to derive a single, overall score
• Items are related among themselves (correlated), (if
you score high on one item you should score high
on the other ones)
• Make sure the items are not too highly correlated
Multidimensional indexes
 If the construct you are measuring has more
than one significant subconcept, you probably
would gain by using a multidimensional index
 Example -- communicative credibility is
composed of three subconcepts:
Authoritativeness
 Trustworthiness
 Dynamism

 To measure communicative credibility
you would need to ask questions related
to each of the subconcepts
So:
 Questionnaire design is the most important
part of survey analysis
 Developing valid measures of ‘black box’
phenomena is especially difficult
 Writing good questions is probably the most
important part of questionnaire design
A
number of common problems in question
wording exist
 Closed-ended questions are simpler and
faster but open-ended questions allow
respondents to provide more personalized
information and are more interesting for
them
 A number of forms of attitude/belief
measures have been developed over the
years
 Commonly
used types are Likert scales,
semantic differentials, Thurstone scales,
Guttman scales
 Using multiple items to measure a single
construct and then combining them
increases measurement validity