Developing a Questionnaire
Chapter 4
Types of Questions
• Open-ended
– high validity, low manipulative quality
• Closed-ended
– low validity, high manipulative quality
Open-ended
• An open-ended question is one in which
you do not provide any standard answers to
choose from.
1.How old are you? ______ years.
2.What do you like best about your job?
Closed-ended
• A closed-ended question is one in which
you provide the response categories, and
the respondent just chooses one:
What do you like best about your job?
(a) The people
(b) The diversity of skills you need to do it
(c) The pay and/or benefits
(d) Other:
______________________________
Dichotomous Questions
• Dichotomous Question: a question that has
two possible responses
– Could be
• Yes/No
• True/False
• Agree/Disagree
Questions based on Level of Measurement
• Use a nominal question to measure a variable
– Assign a number next to each response that
has no meaning; simply a placeholder.
• Use an ordinal question to measure a variable
– Rank order preferences
– More than 5 – 10 items is difficult
– Does not measure intensity
Interval Level
• Attempt to measure on an interval level
– Likert response scale: ask an opinion question
on a 1-to-5, 1-to-7, etc. bipolar scale
• Bipolar: has a neutral point and scale ends are at
opposite positions of the opinion
– Semantic differential: an object is assessed by
the respondent on a set of bipolar adjective pairs
– Guttman scale: respondent checks each item
with which they agree; constructed as
cumulative, so if you agree to one, you probably
agree to all of the ones above it in the list
Filter/Contingency Questions
• To determine if a respondent is ‘qualified’ to
answer questions, might need a filter or
contingency question (also known as knowledge)
– Limit # of jumps
– If only two levels, use graphic to jump
– If you can't fit the response to a filter on a
single page, it's probably best to be send them
to a page, rather than a question #
How many steps in the response scale?
• Statistical reliability of the data increases
sharply with the number of scale steps up
to about 7 steps
– After 7, it increases slowly, leveling off around
11
– After 20, it decreases sharply
Should there be a middle category?
• Does it make sense to offer it?
• Should not be used as the “don’t know or
no opinion” option.
– The middle option is usually placed between
the positive and negative responses.
– Sometimes it’s last in an interview.
Direct Magnitude Scaling
• Method of obtaining ratio-scaled data
– Idea is to give respondents an anchor point,
and then ask them to answer questions relative
to that
• Example:
– Suppose you are interested in the severity of
crimes.
• Begin by assigning a number to one crime
and then have respondents assign numbers
to the others based upon a ratio.
Filtering "Don't Know"
• Standard format
– No "don't know" option is presented to the respondent,
but is recorded if the respondent volunteers it.
• Quasi filter
– A "don't know" option is included among the possible
responses.
• Full filter
– First the respondent is asked if they have an opinion.
If yes, the question is asked.
Question Placement
• It's a good idea to put difficult, embarrassing or
threatening questions towards the end
– More likely to answer.
– If they get mad and quit, at least you've gotten most of
your questions asked!
• Put related questions together to avoid giving the
impression of lack of meticulousness
• Watch out for questions that influence the answers
to other questions.
Wording of Questions
• Direction of Statements
– Response bias
– Socially desirable
• Always and never
– Avoid this
– Better to phrase as ‘most’, ‘infrequently’
• Language
– Reflect educational level and reading ability
– Need for various languages
Frequency and Quantity
• Consider both frequency and quantity
– Consider number of times
– Consider duration of times
Mutually Exclusive and Exhaustive
• Mutually exclusive: not possible to select
more than one category/value
• Exhaustive: providing all possible
categories/values
Forced Choice
• Choose between 2 choices
– Might not be relevant
– Other choices exist (or at least possible)
– Lesser of two evils
Recalling Behavior
• Can be difficult to remember
• Ask questions that can be answered
• Choose time frames that are reasonable
• Pilot test for time frame issues
Response Bias
• Exaggerating the truth
• Socially desirable answers
• Consider using ‘trap’ questions
– Possibly fictional choice
Sensitive Items
• More comfortable answering in categories
– Minimize missing data
– Might loose statistical power
Evaluating Questions
•
•
•
•
Pre-testing
Cognitive interviewing
Behavior coding
Peer review
• Peer review has shown to be the best
method but it’s the least used.
Validity and Reliability Questions
• Evaluative strategies:
– Analysis of data to evaluate the strength of predictable
relationships among answers and with other
characteristics of respondents.
– Comparisons of data from alternatively worded questions
asked of comparable samples.
– Comparison of answers against records.
– Measuring the consistency of answers of the same
respondents at two points in time.
Coding the Questionnaire
• Create a codebook: reference guide for the
data set
• Code: assigning a value to a response
category
–
–
–
–
Often numeric code
Pre-coding makes it easier
Content analysis on open-ended items
Yes/No often coded as present or not (0 or 1)
Missing Responses
• Why blank?
–
–
–
–
Missed them
Refusal to answer
Didn’t feel it applied
Didn’t know the answer
• To code or not
– Analyze the difference
– If know why, might consider
Piloting the Questionnaire
• Test it on yourself
– Possibly other experts
• Test on people similar to sample
– Don’t reuse (some exceptions)
• Discuss the survey with individuals
– During completion or After
Finding Respondents
• Best Methods of Selection
• Even with a good survey, poorly chosen
sample leads to poor results