SURVEY
RESEARCH
Survey Research



a.k.a. “paper-pencil” measures or
“self-report” measures
represents the dominant paradigm for
social science research in the last 30
years
advantages:
 expedience in gathering data
 yields interval level data
Do subjects know their own minds?

problems:



problem of “mindfulness,” verbal
reports of mental processes are
suspect
problem of “non-attitudes,” people
tend to venture an opinion whether
they have one or not
unreliability of self-reports data:
even factual information can be
misreported
Will subjects tell you if they know?
 intentional misreporting of
information.
 social desirability bias
 reasons for non-disclosure
 embarrassment
 fear of negative sanctions
 shame, public humiliation
 privacy
 sometimes helpful to include a
“lie scale”
Questionnaire Design
and Construction



phrasing of questions is
critical
avoid vague, nebulous
questions
 Bad example: “What do
you think about abortion?”
 Bad example: “How many
children do you have?”
questions must be clear,
unambiguous
 Good example: “Which of
the following statements
best represents your
attitude toward elective
abortions in the first
trimester?”
 Conduct a “readability
analysis”

avoid lengthy questions;
keep questions short,
succinct
 Bad example: Sara Palin
would be the first female
Vice President Secretary
of the United States. Do
you think her gender
wuold be a problem in
negotiating with the Arab
world, which is a maledominated culture?
 Good example: Would
Hillary Clinton make a
good Supreme Court
justice?
questionnaire design and construction-continued


avoid double-barreled
questions (compound
questions)
 Bad example: “Do you
favor stricter hand gun
controls and mandatory
minimum sentences for
carjackers?”
avoid loaded language
(push polling)
 Bad example: “Don’t you
think that...?” “Isn’t it
true that...?”
 Bad example: Emotionally
charged words: “ gang
member,” “welfare
mother,” “extremist
groups,” “spin doctor,” etc.


avoid slang, jargon,
abbreviations and
acronyms
 Bad example: “Should
states regulate PETA and
the ALF?”
 Bad example: “Do you
think hip hop is wack?”
avoid or minimize negative
wording
 Bad example: Is the Bush
administration right in not
establishing a firm
deadline for withdrawing
from Iraq?
open-ended versus close-ended
questions

Schuman, Ludwig, & Krosnick (1986): 60% of
respondents selected one of four options in a
close-ended format, but only 2.4% mentioned any
of the same four responses in an open-ended
format.
 open-ended: allows subjects more leeway,
flexibility
 “What is your primary ethnic/cultural
background?”
 requires a content analysis of responses
 close-ended or “forced-choice” ties respondents’
hands somewhat
 easy to code the data
 always include an “other_____________”
category
phrasing of questions is critical



even slight variations in wording can alter
respondents’ answers.
 “occupied territory” versus “contested
territory”
 “assisted suicide” versus “mercy killing”
 “fetus” versus “unborn child”
make questions concrete, come down the
“ladder of abstraction”
Use negatively worded questions or
statements sparingly
 sometimes necessary to include
reverse-valenced items to identify a
“response set”
…more on phrasing questions

Avoid evaluative language


Phrasing of questions should not imply approval,
disapproval
Follow-ups should not suggest surprise, liking,
disliking, etc.
 Bad example: Do you think the Democratic
dominated Congress should lift the harsh
restrictions on stem cell research?
 Bad example: Despite its poor track record in
crisis intervention, do you think the United
Nations should intervene in Darfur, Sudan?
respondents’ abilities

Respondents must be capable and willing
to answer the questions





May be unaware of topic/issue
May be unwilling to disclose
May lie, exaggerate, under-report or over-report
example: If you ask people what nonverbal cues
they leak when they lie, people may not know.
example: if you ask people what they would do in
a hostage crisis, how would they know?
additional considerations

order effects: sequencing of questions can be a
factor
 “halo effect”: carry-over from one item to the
next
 use multiple versions of questionnaire
 “response set”: checking the same response
all the way down a questionnaire
 use “reverse scaling” to detect this
additional considerations




gender of researcher/respondent can influence
responses
ethnicity of researcher/respondent can influence
responses
medium can influence responses
 face-to-face interview, telephone survey, mail survey
 anonymous versus non-anonymous questionnaire
fatigue: minimize length of surveys
Characteristics of types
of surveys
Type of survey
Obtaining
sample
Cooperation
rate
Cost per
respondent
advantages
disadvantages
in person
difficult
medium
high
interviewer
rapport,
nonverbal cues
respondent
apprehension,
expensive
telephone
easy
high
medium
Fast
Limited time,
nonrandom
sample
mall intercept
easy
medium
medium
fast, in-person
Nonrandom
sample
mail
easy
low
low
expensive
Nonrandom
sample,
respondent
errors
computerassisted
easy
high
low
automatic data
entry
Requires
computer
literacy