Survey Contents
Creating an Effective
Questionnaire
Survey Contents

Devote time to design the questionnaire up
front—you cannot change your questions in midsurvey!

Remember that your goal is to create a valid and
reliable instrument, accurately measuring the
concepts in which you are interested.

Remember that your respondents will be “Lazy
Thinkers.”
Survey Contents

When creating questions, consider the
effects of the following elements:

Type
Constraints, if any, placed on responses

Wording
Clarity of questions and response options

Order
Relative placement of questions in the instrument

Content
Topics the questions cover
Survey Contents

Type
Constraints, if any, placed on responses

Open-ended
A survey question to which the respondent replies in
his or her own word, either by writing or by talking

Closed-ended or fixed-choice
A survey question that provides preformatted response
choices for the respondent to circle, check, mark, etc.
Survey Contents

Type
Constraints, if any, placed on responses

Open-ended



Respondents are encouraged to give full reaction
Anticipate amount of space one will need to give a
complete answer
Try to limit to exploratory questions
 Too many will tire subjects
 Lots of work to standardize
 False belief that you got the full answer

Most commonly used are specific such as, “What is
your age in years?”
Survey Contents

Type
Constraints, if any, placed on responses

Closed-ended or fixed-choice


Response options are limited
For example:
 Yes/no
 True/false
 Multiple choice with an “other” option
 Race: White, Black, Hispanic, Asian, Other
Survey Contents

Type
Constraints, if any, placed on responses

Closed-ended or fixed-choice:

Rating Scales
 Ask respondents to rate something like a person, place, thing,
idea, attitude, etc. on a numbered scale, often Likert scales
 Likert Scales, dominant form in GSS
1
2
3
4
5
Strongly Agree Neither Disagree Strongly
Agree
A nor D
Disagree

Bad
Awful
Semantic Differential
a fifth grader is
⊲
⊲
⊲
⊲

⊳
⊳
⊳
Good
⊳ Nice
Survey Contents

Type
Constraints, if any, placed on responses

Closed-ended or fixed-choice:

Ranking Scales
 Force R’s to put a set of items in order
 For example: Place the following animals in order of
their ability to understand humans from lowest ability
to highest ability.
Dog, Cat, Slug

Other types too
Survey Contents

Type
Constraints, if any, placed on responses

Closed-ended or fixed-choice

Special Issue: Coding
 Assigning numbers to response options so computers
can operate with them
 Keep conventional logic and data analysis in mind when
determining your coding schemes for responses
 Code Dummy Variables with 0 = absence and 1 =
presence
 Increasing magnitude should be reflected by increasing
codes
 Remember to think of levels of measurement!!!
 The GSS is horrifically coded!
Survey Contents

Type
Constraints, if any, placed on responses

Closed-ended or fixed-choice

Special Issue: Coding
 If your variable were Religiosity, and you asked, “how
religious do you consider yourself?”
Then you should not have a coding scheme like:
1, very religious
2, somewhat religious
3, barely religious
4, not at all religious
It should be reversed!
Survey Contents

Type
Constraints, if any, placed on responses

Closed-ended or fixed-choice
Special Issue: Coding
From the general social survey codebook:

Do you Feel that you're constantly under stress?
CONTENT
Yes
No
Don't know
Not stated
CODE
1
2
8
9
SAMPLE
3681
6422
51
595
This should be No = 0 and Yes =1 because the idea of the question is
about stress. In binary logic, 1 equals the presence of something.
Survey Contents

Type
Constraints, if any, placed on responses

Closed-ended or fixed-choice
Special Issue: Coding
From the general social survey codebook:

How frequently do you participate in bowling?
CONTENT
CODE
SAMPLE
2-3/month
1
13
1-2/week
2
90
3+/week
3
5
This is actually appropriate, because the higher the code
number, the more the person bowls.
Survey Contents

When creating questions, consider the
effects of the following elements:

Type
Constraints, if any, placed on responses

Wording
Clarity of questions and response options

Order
Relative placement of questions in the instrument

Content
Topics the questions cover
Survey Contents

Wording
Clarity of questions and response options



Respondents should understand your questions
Pre-testing is a very effective way to see if they do
A bad question:

Do you hate taxes?
—What exactly are you measuring?
 The answer will tell you little because our society has many
different taxes, loves to complain about them, but thinks they
are necessary

Bad words:

AIDS prevention researchers learned to avoid words such
as “intercourse,” “homosexual sex,” and “intravenous
injection” because many do not understand these words
Survey Contents

Wording
Clarity of questions and response options

Guidelines for good questions:
1.
2.
3.
4.
Be direct
Maintain simplicity
Be specific
Take the role of your respondent
The following points elaborate on these four main
themes.
Survey Contents

Wording
Clarity of questions and response options
Some things to do to make questions less
unclear and less incomprehensible:

Avoid complex rhetoric, syntax, or disciplinary slang or
jargon.

Do not expect them to learn new words or information
just to answer a question. (Sometimes, you may
establish context with a short paragraph then ask a
series of short, specific questions.)
Survey Contents

Wording
Clarity of questions and response options

Avoid Double-barreled questions. They contain two
questions in one.


Avoid Double-negative questions.


For example, “Do you think that students and Professors
should be given discounts on sports tickets?”
For example, “Do you disagree that professors should not
be required to help students outside of class?”
Avoid hypothetical questions (unless you are studying
the sociology of hypothetical situations).

For example, “If men could have children, would your
husband or boyfriend stay with you?”
Survey Contents

Wording
Clarity of questions and response options

Avoid ambiguous questions.


“Do you teach your children to effectively function?”
“Does your boss engage you in interactive
dialogue?” “Effectively function” and “interactive
dialogue” are subject to interpretation.
Words such as "usually" or "normally" mean
different things to different people. “Do your
customers normally complain?” has a variety of
interpretations.
Survey Contents

Wording
Clarity of questions and response options

Avoid biased questions. Avoid making one
response option look more suitable than the
other, using emotionally loaded terms, or
using unbalanced response categories.


“Don’t you think that suffering terminal cancer patients
should be allowed to be released from their pain by
choosing death?”
SJSU pays professors less than similar universities. SJSU
should: 1. pay them more, 2. keep paying the same, 3.
reduce pay little, 4. reduce pay some, 5. reduce pay a lot,
6. dramatically reduce pay
Survey Contents

Wording
Clarity of questions and response options

Avoid biased questions. Avoid making one
response option look more suitable than the
other, using emotionally loaded terms, or
using unbalanced response categories.
From the GSS:
“What is your opinion about a married person having sexual
relations with someone other than the marriage partner –
is it always wrong, almost always wrong, wrong only
some times, or not wrong at all?”

Always wrong
Almost always wrong
Wrong only sometimes
Not wrong at all
=
=
=
=
1
2
3
4
Survey Contents

When creating questions, consider the
effects of the following elements:

Type
Constraints, if any, placed on responses

Wording
Clarity of questions and response options

Order
Relative placement of questions in the instrument

Content
Topics the questions cover
Survey Contents

Order
Relative placement of questions in the
instrument

Opening questions should be simple and
introduce the topic of the survey.

Try not to mix topics. Put like things into sets
of questions.
Survey Contents

Order
Relative placement of questions in the instrument

Avoid framing later questions with topics that can be linked to
them in previous questions.

For example, one should not ask about attitudes toward crack use
right before asking about attitudes toward the urban poor. This
will invoke stereotypes about the poor.
Survey Contents

Order
Relative placement of questions in the instrument

Sensitive questions should never be at the beginning.
Put in middle. May need opening paragraph for them.

Consider need to transition between types of questions.
Write transitory questions or directions.

Place your most important questions earlier—R’s may
fatigue before answering them if they are later.
Survey Contents

Order
Relative placement of questions in the instrument

Do not put in too many complex format questions (e.g.,
“skipping” or “go to next section”). R’s will get
confused/frustrated.

Closed-ended questions are easier to answer. Put openended questions later in survey. Investment of time up
to that point will likely keep them involved through the
open-ended.
Survey Contents

When creating questions, consider the
effects of the following elements:

Type
Constraints, if any, placed on responses

Wording
Clarity of questions and response options

Order
Relative placement of questions in the instrument

Content
Topics the questions cover
Survey Contents

Content
Topics the questions cover

You should make the topic of the survey clear to R’s. Do not
surprise them with questions about unrelated topics.

If including sensitive questions, provide a rationale for asking
them. Explain how honest answers will be helpful to others.

Avoid phrasing questions to seem too personal or direct when
dealing with culturally sensitive issues. For example, avoid
“Do you abuse your kids?”

Questions relevant to deviance should include normalizing
statements. For example, “Many people use drugs for a
variety of reasons. Have you ever used ecstasy (or x, e,
MDMA, Adam) to feel closer to other people?”
Survey Contents

Content
Topics the questions cover

The more sensitive the topic, the more reassurance of
confidentiality should be stated.

Never open a survey with questions about sensitive
topics, and do not end surveys with negative questions
that will linger in R’s minds.

Continually reassure R’s that there are no right or wrong
answers to questions—just truthful or not.

Questions that require lots of specific details or a good
memory are typically useless. If they are necessary,
employ techniques to prompt recollection.
Survey Contents

When creating questions, consider the effects of
the following elements:

Type
Constraints, if any, placed on responses

Wording
Clarity of questions and response options

Order
Relative placement of questions in the instrument

Content
Topics the questions cover

Now… Some General Advice
Survey Contents

General Advice

Always consult other surveys first




They will give you good ideas for exploring your topic
They may provide good examples of how to ask
questions
You avoid “reinventing the wheel”
Always pre-test your instrument
Survey Contents

General Advice

When constructing and editing your survey,
ask yourself:





Are your questions measuring what you are
interested in?
Is every question integral to your research? (Do not
waste R’s time.)
Do any questions seem to appear out of nowhere?
Is each question general enough to cover the entire
sample’s experiences, but specific enough to get
useful data?
Would you respond to these questions?
Survey Contents
Social Psychology of Survey Response
Consider these issues:

Not everyone will have an opinion on every topic.
Consider “no opinion” as an option.

People want to appear to agree. Consider negative and
positive statements.

R’s may lose track and choose salient options (such as
first in series). Keep questions simple.

Response set problem: In a series of questions with the
same response options, R’s may just quickly check the
same response for all questions.
Survey Contents
Social Psychology of Survey Response
Consider these issues:

Bad memory leads to:





Forward telescoping (reporting that events occurred
more recently than in reality)
Backward telescoping (reporting events further back
than in reality).
Salient events are over-reported
Mundane events are underreported
“Habitual” events will fill in for lost information.
Provide aides to recall such as reference
points, landmark events, etc. Use limited time
frames in questions. Such as: “During the last
week, how often…”
Survey Contents
Social Psychology of Survey Response
Consider these issues:

Fatigue effect. Boredom or tiring. Use
transitions, vary questions or response
options, place easier questions near the end,
or use a “jazzed up” format.

Self bias. People try to assert a vision of
themselves and are the center of their
universe. Try to elicit factual information
rather than impressions by avoiding personal
evaluations and by disguising implications
about persons.
Survey Contents
Survey Questions Assignment—Due next class:
Evaluate your course paper General Social Survey questions
about by responding to each item below. Turn in a copy
of your questions with this assignment.
1.
2.
3.
4.
5.
Are the questions and response options effective at
prompting accurate measures of social phenomena?
How?
How might you change the questions to make them
better, if at all?
How should the questions be recoded, if at all? Why?
Are they sensitive questions that require caution on the
researchers’ part?
Where would they best be placed in a survey?