Survey Research

Survey Research
Prying into Your Personal Life in
Ways Telemarketers Could Never
Dream Of
Appropriate Research
Questions for Surveys
• Self-reported beliefs and behaviors
• Ask things at one time, measure many
variables, and test several
Appropriate Research
Questions for Surveys
– How often do you brush your teeth?
– What is the biggest problem facing the nation today?
– Do you belong to a union?
– Do you plan to buy a new car in the next year?
– Are you liberal or conservative?
– Who was elected mayor in the last election?
Key Limitation
• Surveys ONLY provide data on what
a person or organization says
• This can differ from what they
actually do or think
• Survey research is DEDUCTIVE
Steps in Conducting a
• Step 1:
Develop hypotheses
Decide type of survey
Write survey questions
Decide on response categories
Design layout
Steps in Conducting a
• Step 2:
– Plan how to record data
– Pilot test survey instrument
• Step 3:
Decide on target population
Get sampling frame
Decide on sample size
Select sample
Steps in Conducting a
• Step 4:
– Locate respondents
– Conduct interviews
– Carefully record data
• Step 5:
– Enter data into computers
– Recheck all data
– Perform statistical analysis on data
Steps in Conducting a
• Step 6:
– Describe methods and findings in
research report
– Present findings to others for critique
and evaluation
Constructing the
• Introductory remarks and
• Keep it clear, keep it simple, keep
respondent’s perspective in mind
• Question writing is more of an art
than a science
– Skill, practice, patient, and creativity
Questionnaire Hints
• 1) Avoid jargon, slang, and
– No abbreviations
– Only use this if you’re surveying a
specialized population
– “Did you drown in brew until you were
totally blasted last night?”
Questionnaire Hints
• 2) Avoid ambiguity, confusion, and vagueness
– “What is your income?”
– Confusion causes inconsistencies
• 3) Avoid emotional language
– Implicit connotative as well as explicit denotative
– Use neutral language
– “What do you think about a policy to pay murderous
terrorists who threaten to steal the freedoms of peaceloving people?”
Questionnaire Hints
• 4) Avoid prestige bias
– Avoid associating statements with them
– “Most doctors say that cigarette smoke causes
lung disease for those near a smoker. Do you
• 5) Avoid double-barreled questions
– Each question = ONE QUESTION ONLY
– “Does this company have pension and health
insurance benefitted?”
Questionnaire Hints
• 6) Do not confuse beliefs with reality
– Don’t mix up respondents beliefs with what we
want to measure
– “Do you rate a teacher higher if the teacher
tells many jokes?”
• 7) Avoid leading questions
– Don’t lead them to an answer
– “You don’t smoke, do you?”
Questionnaire Hints
• 8) Avoid asking questions that are
beyond respondents’ capabilities
– Frustrates respondents
– Produces poor-quality responses
– “Two years ago, how many hours did you
watch TV every month?”
Questionnaire Hints
• 9) Avoid false premises
– Can’t put a premise that respondent
could disagree with without risking
alienating them
– “When did you stop beating your
Questionnaire Hints
• 10) Avoid asking about intentions in
the distant future
– Hypotheticals are bad
– Not specific or concrete
– “After you graduate college and have
kids, will you invest in the stock
Questionnaire Hints
• 11) Avoid double negatives
– Grammatically incorrect and confusing
– “Do you disagree with those who do not
want to build a new city swimming pool?”
Questionnaire Hints
• 12) Avoid overlapping or unbalanced
response categories
– Mutually exclusive: categories do not overlap
– Exhaustive: every respondent has a choice
– Balanced: Offer polar opposites at each end of
a continuum
– “Did you find the service at our hotel to be,
Outstanding, Excellent, Superior, or Good?”
Aiding Recall
• Ability to recall accurately declines
over time
• Memory is less trustworthy than
once assumed
• Give respondents special instructions
and extra thinking time to guide
Types of Questions
Threatening Questions
Socially Desirable Questions
Knowledge Questions
Skip or Contingency Questions
• Sensitive issues…people don’t want to talk
about them
• Underreport socially undesirable answers
• Context and wording are how to fix this
• Issue examples:
– Masturbation, sexual intercourse, illicit drug
use, income
Socially Desirable
• Over report socially desirable
answers (Social desirability bias)
– Giving money to charity, voting etc.
• Have to make the norm violation
seem less objectionable then it really
could be
• We can’t answer elementary
questions as Americans… Are You
Smarter than a 5th Grader??
• Pilot-test to make sure they’re at an
appropriate level of difficulty
Skip or Contingency
• Two or more part question
– Answer to one part determines the next
question you’ll receive
– Sometimes called screening questions
Open v. Closed Questions
• Open-ended: Respondent can give any answer they
• Close-ended: Gives respondents fixed answers
from which to choose
• Big surveys use close-ended…quicker and easier
• Can mix both methods
• Open-ended best in early or exploratory stages of
• Partially open questions leave final answer choice
as other
Advantages of Closed
Easy and quick to do
Easy to compare
Easier to code and analyze
Response choices can clarify meaning
of question
• Replication is easier
Disadvantages of Closed
• Suggest ideas that respondent may not
have thought of
• No opinion people can answer anyways
• Frustrating if desired answer is not a
• Simplistic responses to complex ?s
• Force us to make choices we wouldn’t make
in the real world
Advantages of Open
• Unlimited responses
• Answer in detail and clarify
• Unanticipated feelings can be
• Creativity and self-expression
• Reveal logic
Disadvantages of Open
Different degrees of detail in answers
Can be irrelevant
Hard to compare/analyze
Coding is tough
Hard to write verbatim in interviews
More time necessary
Take up lots of space on questionnaire
Nonattitudes and Middle
• Debate whether to include choices for
middle, neutral, or nonattitudes
• Two errors:
– Accept nonattitude when respondent has one
– Force choice when respondent doesn’t have one
• Standard-format v. quasi-format v. fullformat
Rankings, or Ratings?
Best to offer respondents explicit alternatives
Avoid the response set bias (just check agree to
• Respondents can rate several items equally high,
but will put them in hierarchy if asked to
Word Issues
• Need to use simple vocabulary and
grammar to minimize confusion
• Need to watch for effects of specific
words or phrases
– Forbid v. not allow
– Certain words trigger an emotional reaction
– “To help the poor” v. “for welfare”
Length of Survey
• Researchers prefer long because
they’re cost effective
• Phone: 5-20 minutes
• Mail questionnaire: 3-4 pages
• Face-to-face: Around an hour
Order Effects
• Strongest for people who lack strong
views, less educated, elderly
• May not perceive each element of a
survey as isolated and separate
• Influence through content and
Context Effects
• Funnel sequence:
– Start general and get more specific
• Split sample
– Half get questions in one order, other
half different
• More ambiguous the question, the
more context effects matter
Survey Layout
Clear, neat, and easy to follow
Number questions
Identifying info
Cover sheet
Professional appearance
Provide instructions
Survey Layout
Crucial in mail questionnaires
Use letterhead
Always thank them for participation
Circle response v. check box v. fill in
dots v. put an X in a blank
• List answer categories down, not
Survey Layout
• Use arrows and instructions for
contingency questions
• Matrix questions
– Compact way to present a series of
questions with the same response
– Box 7.5
• If not enough people respond, we may
not be able to generalize
• Many people are burnt out with
surveys these days
• Improve eligibility rates through
careful respondent selection
• Decrease refusals through sending
letters before calling, small
incentives, adjusting behavior, using
alternative interviewers, using
alternative interview methods
• Initial contact is HUGE in
determining likelihood for success
• Survey research is research based on
direct or indirect interview methods.
• Many types of surveys:
In person (including interviews)
E-mail and Internet
Group surveys
Mail and Self-Administered
• Advantages:
Wide geographical area
Completed when convenient
Respondent can check personal records
Mail and Self-Administered
• Disadvantages
• Low response rate
• Can’t control conditions under which its
filled out
• Someone other than intended can fill out
• Can’t observe reactions
• Ill suited for those who don’t have good
grasp of English
Mail and Self-Administered
• How to Increase Response
Address to specific person
Careful cover letter on letterhead
Postage-paid, addressed return envelope
Neat, attractive layout
Send two follow-up reminders
Nothing during major holidays
No questions on back page
Include a small monetary inducement
Web Surveys
• Advantages
Very fast
Flexible design
Can use visuals or audio
Web Surveys
• Disadvantages
Coverage (75% have internet)
Multiple e-mail addresses
Protecting privacy and confidentiality
• Encrypt
– Complexity of design depends on software
• Screen-by-screen questions
• Progress indicator
• Needs to be easy to move back and forth across questions
Phone Interviews
• Advantages
95% coverage rate
Can interview about 1500 in a few days
Half the cost of face-to-face interviews
Phone Interviews
• Disadvantages:
– Higher cost
– Limited interview length
– Use of interviewer reduces anonymity
and introduces potential interviewer
• Advantages:
Highest response rates
Permit the longest questionnaires
Can observe surroundings
Ask all types of questions
• Disadvantages:
– High cost
• Training, travel, supervision, personnel costs
– Interviewer bias
Comparing the Methods
• Cost:
– I, M, P, F2F
• Speed:
– I, P, F2F, M
• Length:
– F2F, M, I, P
• Response rate:
– F2F, P, I, M
• Obtain cooperation and build rapport, but
remain neutral and objective
• Interviewer must treat everything in a
matter-of-fact manner
• Pleasant, honest, accurate, mature,
responsible, moderately intelligent, stable,
and motivated
• Poise and tact
Interviewer Bias
• Interview expectations can create
significant bias
• Social setting can affect answers
– Students answer differently at home
and at school
• Visible characteristics can play a
role…such as race and gender
• Computer-assisted telephone
• Read questions from the screen and
enter answers immediately
• Automatically makes calls for you
• Automatically stores answers in a
The Ethical Survey
• Invasion of privacy
– That’s what ANY survey is!!!!
• Voluntary participation
– Informed consent
• Use of push polls—AKA “pseudosurveys”
• Sloppy reporting of surveys by the media
– No quality control standards
• Five important factors influence the
choice of survey type:
Completion rates
Sample-population congruence
Questionnaire length
Data-processing issues
• Cost:
– All surveys share basic expenses for materials.
– Cost of a survey can be quite variable, based on
Professional questionnaire design
Length of the questionnaire
Geographic dispersion of the sample
Callback procedures
Respondent selection rules
Availability of trained staff
Method-specific costs like long distance rates or
• Completion rates:
– Low response rate on a survey limits the
ability to make statistical inferences
about the target population
– Must make an effort to reach those
selected for inclusion in the sample to
maximize the representativeness of the
sample…or risk bias
• Sample-population congruence:
– If those who fail to participate in the
survey share common characteristics,
the sample will be wrought with
systematic sampling error and the
conclusions made with the data could be
• Questionnaire length:
– Keeping respondents’ attention during a survey
helps achieve the best possible response
– A lengthy questionnaire can lead respondents
to lose focus or speed through questions to
finish sooner, resulting in low response quality.
– Survey length ranges between three and four
dozen items, depending on complexity and
• Data-processing issues
– A good deal of time and effort are
required to process data.
– Surveys using more elaborate or
complex designs or a large number of
questions will require more time and
• Each type of survey has advantages
and disadvantages when it comes to
the five factors.