Writing Good Questions: Part 1
Slide 1
The goal of this lecture is to provide some basic guidelines and rules of thumb for designing
good questions.
Slide 2
A questionnaire is only as good as the questions it asks. Questionnaire designers must be
careful that their questionnaires are as well designed as possible and that their questions are as
precise and as easy to answer as possible.
Slide 3
This Calvin and Hobbes cartoon illustrates why we should care about the quality of questions
we ask. We could design questions in such a way that any set of responses is possible. As this
cartoon shows, Calvin agrees with many statements about graphic violence until the question is
asked a certain way. Suddenly, it becomes much more difficult for him to answer affirmatively.
The same is true of questionnaires that we would write. Fielding a questionnaire is meant to
reduce decision-related uncertainty. The point is to query respondents and to assess their
attitudes and behaviors accurately; hence, questions should be asked in a way that yields the
most accurate responses. What managers do with those responses is up to them.
Slide 4
This cartoon illustrates why we should care about the preciseness of questions we ask. The
advertising slogan indicates this television is preferred 2-to-1 by service people. Is that good?
Do service people prefer this television because it breaks down repeatedly and thus gives them
multiple opportunities to earn money fixing them, or do they prefer this television for their own
use because they’re relatively reliable? In other words, the statement is ambiguous. We should
strive to avoid this type of ambiguity with our questions.
Slide 5
The first thing we must do to ensure we ask good questions is to plan what we’ll measure. We
must know the required information and the targeted respondents for our questions. Different
respondents will have different abilities to answer different types of questions. Once we
understand the target, then we’ll have a better sense for how to position our questions. Finally,
we must know the data collection method. We can ask far more complex questions with many
more choices in a self-administered mail questionnaire than we can in a telephone-administered
questionnaire, in which the responses choices must be few and straightforward.
Slide 6
Think about open-ended questions being equivalent to essay or short answer questions and
close-ended questions being equivalent to multiple-choice questions.
Slide 7
The advantages of close-ended questions are as follows:

The communication skills required to answer such questions are lower.
Page | 1

Respondents can answer rapidly.

Such questions are easy to answer.

Responses to these questions are easy to code and enter into a computer.

It’s easier to analyze responses to close-ended questions.

Either little or no interviewing skill is needed to administer close-ended questions, which
is why close-ended questions often appear on mail surveys, internet surveys, or other
self-administered surveys.
The disadvantages of close-ended questions are as follows:

It’s impossible to obtain in-depth answers. Respondents merely are looking at several
options and picking the one that most represents their opinions and/or behaviors.

Close-ended questions are poor for providing insights. Such questions assume we
already know the likely responses and we’re merely asking respondents to pick one. Any
response beyond what we’ve provided respondents is beyond the scope of the study
and it’s unlikely that respondents will volunteer insights into unlisted reply options.

It’s harder to write good close-ended questions because they must anticipate all possible
responses.

The provided answers may not fully reflect respondents’ attitudes. We ask respondents
to indicate the choice that’s most reflective, but that most reflective response could be
very close or not very close.

Finally, by providing the possible responses, the interviewer is hinting at the right
answers. Perhaps respondents would’ve answered differently; yet, once they see the
answers, they sense what comprises an appropriate answer and respond accordingly.
Slide 8 (No Audio)
Slide 9
Here’s an example of the problems introduced by hinting at the correct answers. Suppose we
want to ask undergrads how many hours a day they study and watch television. We can provide
two different choice sets for answering these questions. We could provide the choices under
version ‘A’, which range from less than ½ hour daily to more than 2½ hours daily. Alternatively,
we could present the choices under version ‘B’, in which the first five choices from version ‘A’
are lumped into a single 2½ hours or less daily choice and the last choice from version ‘A’ is
split into five choices, which range from 2½ to 3 hours daily to more than 4½ hours daily. The
percentages in the table indicate that choice format strongly influences students’ answers about
their daily studying and television viewing time. If we ask the question as 2½ hours or more
versus less than 2½ hours for studying, as per version ‘A’, then far fewer students report
studying more than 2½ hours daily; compare the 23% for mail surveys, the first number in the
upper-left-hand corner, to the 69%, the third number in the same row. Similarly, self-reported
television viewing varies by choice format. If asked as per version ‘A’, with more of the fewer or
shorter periods, versus version ‘B’, with more of the longer periods, students indicate they watch
fewer hours of television daily.
Page | 2
Slide 10
The relative advantages and disadvantages of open-ended questions are basically the reverse
of the advantages and disadvantages of close-ended questions. The advantages of open-ended
questions are that respondents have an opportunity to provide a wide range of responses. Such
questions encourage people to respond, and because these responses may be unexpected,
they encourage additional probing. In general, open-ended questions encourage additional
probing about respondents’ answers.
However, the disadvantages of such questions are numerous.

To some extent, responses to open-ended questions are weighted unintentionally by
respondent articulateness, as more articulate respondents will say more. Given the way
responses are entered into a computer, more words will count more than fewer words.

Another disadvantage is interviewer bias. Open-ended questions do not lend themselves
to administered questionnaires; they’re best used with a live interviewer, either in person
or via telephone, and interviewers are not all alike. Therefore, inter-interviewer
differences introduce additional response bias.

It’s harder to code and record responses to open-ended questions. Imagine asking
people questions and then writing what they said. Although audio or video recording
seems a great alternative, many people are reluctant to allow themselves to be audio or
video recorded. As a result, the only record of their answers is whatever interviewers can
scribble as quickly as they can scribble it. Often, those scribbles are incomplete and
erroneous.

As for coding inconsistencies and difficulties, any post-data-collection numerical analysis
would require examining every response to every question, developing basic categories
that represent all possible responses, revisiting all the answers, and then assigning a
numeric code to each one. Such an effort requires much additional time and effort. This
is one reason why I urge you, if you’re conducting a survey project for this course, to use
close-ended questions. They may take longer to write, but they’ll make your life easier
for subsequent data analysis.

Open-ended questions are more difficult to use for cross-study comparisons, where
choices and contexts change over time.

Finally, open-ended questions are more costly because their best use requires live—and
expensive—interviewers and additional costly data handling (associated with recording,
coding, entry, and tabulation).
Slide 11
Here are three basic guidelines for writing good questions that relate to research proposals. A
proposal requires you to develop a research problem and related set of research questions. The
three guidelines are as follows: (1) Create specific questions only after you have thoroughly
thought through your research questions, which you’ve formalized in writing. (2) When you’re
working on a questionnaire, constantly refer to your research questions. (3) For each question
you write, understand how responses to that question will help you to answer your research
questions. If you’ve written a question that’s unrelated to your research questions, then that
question should be deleted from your questionnaire.
Page | 3
Slide 12
Now I’ll discuss several examples of why language is critical. This cartoon introduces that idea.
Here, Junkyard Dog refers to a dog that works in the security division of an automobile
wreckage site. That particular definition denotes something vastly different from the historical
use of Junkyard Dog.
Slide 13
One rule for writing good questions is to be certain they are as clear and precise as possible.
Consider these two examples; neither one is great but version ‘A’ is better than version ‘B’.
Version ‘B’ asks how frequently do you drink coffee—extremely often to never? Although
seemingly straightforward, the question is ambiguous because different people may mean very
different things by drinking coffee very often. One person may believe that drinking coffee very
often means drinking a cup every few hours, yet somebody else may believe that drinking
coffee very often means drinking a few cups per week. Obviously, it’s a problem when different
people provide the same answer but refer to vastly different behaviors or attitudes. Version ‘A’
of the question is a bit better because it’s a bit more specific: how many cups of coffee do you
drink in a typical work day? Nonetheless, the question is ambiguous is several ways. What’s a
typical work day? What’s a cup of coffee; an 8 oz. cup or a 32 oz. mug? Version ‘A’ is better
than Version ‘B’, but Version ‘A’ could be made far more precise. In addition, asking people to
model their own behavior is problematic.
Slide 14
This slide hints how version ‘B’ from the previous slide about coffee consumption might be
improved. The quantifiers for the question “How often did you attend religious services during
the past year—never, rarely, occasionally, or regularly?” could be more precise. We’ve replaced
those with not at all, a few times, about once a month, two to three times a month, once a week,
and more than once a week. This form doesn’t ask someone to remember behaviors back more
than roughly one month, and as most people only attend religious services once a week and
such services are relatively memorable, the response set is reasonable.
Slide 15
These next two slides reinforce the point about people differing markedly in the way they
interpret vague quantifiers. The two questions address frequency of movie attendance and beer
consumption. The response categories are quite vague.
Slide 16
This slide shows the extensive range of responses to vague quantifiers. In terms of annual
movie visits, very infrequently was interpreted as 0 to 12 times, infrequently was interpreted as 2
to 52 times, frequently was interpreted as 4 to 104 times, and very frequently was interpreted as
12 to 265 times. In fact, respondents could have said very infrequently and meant the same
thing, if they were thinking 12 times annually, as respondents who answered very frequently.
So, two vastly different answers correspond to the identical set of frequency behaviors. The
category quantifiers for beer are flawed similarly. Hardly any, in terms of 6 packs monthly, is
reasonably consistent with the 0-1 range, but quite a lot is from 1 to 22 6-packs monthly and
very much is from 1 to 40 6-packs monthly. Once again, one person may answer hardly any and
mean one 6-pack monthly, yet someone else could answer very much and also mean one 6pack monthly. Again, it’s a problem when people who behave differently answer similarly about
their behavior.
Page | 4
Slide 17
Good questions provide respondents with categories that are either mutually exclusive or
exhaustive. Exhaustive means all possible answers are present and mutually exclusive means
that one answer precludes all other answers. The choices in the sample question here are not
mutually exclusive, although they are exhaustive because all incomes from 0 to infinity are
represented. The choices aren’t mutually exclusive because there’s no clear answer if the
respondent’s income is $15,000 or $25,000. If it’s $15,000, is the correct answer #2 or #3? If it’s
$25,000, is the correct answer #3 or #4? Again, there is needless ambiguity.
Slide 18
The first of three questions on this slide is an example I hope never to see in any questionnaire
written by a student in this class. The choices are muddled; radio, television, someone at work,
while at home, and while traveling to work, are different kinds of sources and places. Instead of
muddling sources and places, it’s better to ask two separate questions. In the revised version,
question #1 asks “From which one of these sources did you first hear about the tornado?” and
question #2 asks “Where did you first hear: work, home, or traveling somewhere else?”
Slide 19
It’s important that our questions use language that is natural and familiar to respondents. Here’s
an example of ensuring that language is familiar. Remember, a questionnaire is meant to
assess respondent’s attitudes and behaviors; it’s not intended as a vocabulary test. The first
version of the question about intoxication reads: “In the past year, how often did you become
intoxicated while drinking any kind of alcoholic beverage?” Respondents answer relative to a
given response set. Although the choices may be problematic, the major problem with the
question is the word intoxicated. People tend to mean vastly different things by that word. Many
people mean the legal limit, which in New Mexico is 0.08 blood-alcohol level. Other people may
mean being fall down drunk or being totally out of control. There’s a vast difference between
that latter alcohol level and being ever so slightly in excess of the 0.08 level. Rather than
assume (or hope) that all respondents will interpret intoxicated similarly, the solution suggested
at the bottom of the slide is reasonable. Instead, respondents could be told the following:
“Sometimes people drink a little too much beer, wine, or whiskey, so that they act different from
usual. What word do you think we should use to describe people when they get that way so that
you will know what we mean and feel comfortable talking about it?” By describing intoxication in
terms of a modification in behavior, people are more likely to think about the same consumption
level when responding to the question about consumption frequency.
Slide 20
Here’s another example about natural and familiar language. There are regional differences in
the terms that people use to describe the identical thing. For example, here are five different
terms for the type of sandwich you can buy at Subway. In some areas of the country, those
sandwiches are called grinders; in other areas, hoagies. In Chicago, they’re called hero
sandwiches; in New York, they’re called submarine sandwiches; and in Texas and Louisiana,
they’re called poor boys. Although it’s the same sandwich, respondents should receive a version
of the questionnaire with the regionally accepted term for it. Alternatively, the questionnaire
could include a description of the sandwich’s physical characteristics. Regardless, sensitivity to
regional differences and terminology is important.
Page | 5
Slide 21
Another rule to remember is to avoid questions that lead to a certain conclusion. For example,
consider the question “What did you dislike about the product you just tried?” If the respondent
didn’t dislike anything, he or she is forced to think about something that may have been less
than totally satisfactory or to answer the question in the way it was asked. This is a biased
question. Instead, the question should be divided into two parts: (1) Did you dislike any aspect
of the product you just tried, yes or no? and (2) If yes, then what aspect and why is it important?
Another example (although somewhat dated) relates to Johnson & Johnson poisoning episode
in Chicago back in the 1970s. Asking people “Do you think Johnson & Johnson did everything
possible in its handling of the Tylenol poisoning situation?” is not a good way to assess people’s
attitudes towards the firm’s actions. Did they do everything possible? Obviously, the firm could
have done something. Nonetheless, business ethics case studies written about this episode
tend to reinforce the firm’s highly ethical behavior. Asking the question “Did they do everything
possible?” suggests Johnson & Johnson’s behaviors were problematic when they weren’t.
Slide 22
Here’s another example of a leading question and a way to revise it to remove the bias. In the
leading version, respondents are asked “Do you believe private citizens have the right to own
firearms to defend themselves, their families, and property from violent criminal attack?” This is
a rather emotional issue when posed this way. Most people would answer yes to the question
“Do I have the right to protect my family?” As a result, it’s a leading question because it’s
emotionally laden. In the improved version, the question is “Do you believe a ban on private
ownership of firearms would significantly reduce the number of murders and robberies in your
community?” This revised version no longer raises family defense issue, so its emotional
overtone is markedly reduced. It’s less likely that someone would automatically answer a
question of the second type than of the first type.
Page | 6