Survey Research
Prying into Your Personal Life in
Ways Telemarketers Could Never
Dream Of
Appropriate Research
Questions for Surveys
• Self-reported beliefs and behaviors
• Ask things at one time, measure many
variables, and test several
hypotheses
Appropriate Research
Questions for Surveys
•
Behavior
– How often do you brush your teeth?
•
Attitudes/Beliefs/Opinions
– What is the biggest problem facing the nation today?
•
Characteristics
– Do you belong to a union?
•
Expectations
– Do you plan to buy a new car in the next year?
•
Self-classification
– Are you liberal or conservative?
•
Knowledge
– Who was elected mayor in the last election?
Key Limitation
• Surveys ONLY provide data on what
a person or organization says
• This can differ from what they
actually do or think
• Survey research is DEDUCTIVE
Steps in Conducting a
Survey
• Step 1:
–
–
–
–
–
Develop hypotheses
Decide type of survey
Write survey questions
Decide on response categories
Design layout
Steps in Conducting a
Survey
• Step 2:
– Plan how to record data
– Pilot test survey instrument
• Step 3:
–
–
–
–
Decide on target population
Get sampling frame
Decide on sample size
Select sample
Steps in Conducting a
Survey
• Step 4:
– Locate respondents
– Conduct interviews
– Carefully record data
• Step 5:
– Enter data into computers
– Recheck all data
– Perform statistical analysis on data
Steps in Conducting a
Survey
• Step 6:
– Describe methods and findings in
research report
– Present findings to others for critique
and evaluation
Constructing the
Questionnaire
• Introductory remarks and
instructions
• Keep it clear, keep it simple, keep
respondent’s perspective in mind
• Question writing is more of an art
than a science
– Skill, practice, patient, and creativity
Questionnaire Hints
• 1) Avoid jargon, slang, and
abbreviations
– No abbreviations
– Only use this if you’re surveying a
specialized population
– “Did you drown in brew until you were
totally blasted last night?”
Questionnaire Hints
• 2) Avoid ambiguity, confusion, and vagueness
– “What is your income?”
– Confusion causes inconsistencies
• 3) Avoid emotional language
– Implicit connotative as well as explicit denotative
– Use neutral language
– “What do you think about a policy to pay murderous
terrorists who threaten to steal the freedoms of peaceloving people?”
Questionnaire Hints
• 4) Avoid prestige bias
– Avoid associating statements with them
– “Most doctors say that cigarette smoke causes
lung disease for those near a smoker. Do you
agree?”
• 5) Avoid double-barreled questions
– Each question = ONE QUESTION ONLY
– “Does this company have pension and health
insurance benefitted?”
Questionnaire Hints
• 6) Do not confuse beliefs with reality
– Don’t mix up respondents beliefs with what we
want to measure
– “Do you rate a teacher higher if the teacher
tells many jokes?”
• 7) Avoid leading questions
– Don’t lead them to an answer
– “You don’t smoke, do you?”
Questionnaire Hints
• 8) Avoid asking questions that are
beyond respondents’ capabilities
– Frustrates respondents
– Produces poor-quality responses
– “Two years ago, how many hours did you
watch TV every month?”
Questionnaire Hints
• 9) Avoid false premises
– Can’t put a premise that respondent
could disagree with without risking
alienating them
– “When did you stop beating your
girl/boyfriend?”
Questionnaire Hints
• 10) Avoid asking about intentions in
the distant future
– Hypotheticals are bad
– Not specific or concrete
– “After you graduate college and have
kids, will you invest in the stock
market?”
Questionnaire Hints
• 11) Avoid double negatives
– Grammatically incorrect and confusing
– “Do you disagree with those who do not
want to build a new city swimming pool?”
Questionnaire Hints
• 12) Avoid overlapping or unbalanced
response categories
– Mutually exclusive: categories do not overlap
– Exhaustive: every respondent has a choice
– Balanced: Offer polar opposites at each end of
a continuum
– “Did you find the service at our hotel to be,
Outstanding, Excellent, Superior, or Good?”
Aiding Recall
• Ability to recall accurately declines
over time
• Memory is less trustworthy than
once assumed
• Give respondents special instructions
and extra thinking time to guide
them
Types of Questions
•
•
•
•
Threatening Questions
Socially Desirable Questions
Knowledge Questions
Skip or Contingency Questions
Threatening
• Sensitive issues…people don’t want to talk
about them
• Underreport socially undesirable answers
• Context and wording are how to fix this
• Issue examples:
– Masturbation, sexual intercourse, illicit drug
use, income
Socially Desirable
• Over report socially desirable
answers (Social desirability bias)
– Giving money to charity, voting etc.
• Have to make the norm violation
seem less objectionable then it really
could be
Knowledge
• We can’t answer elementary
questions as Americans… Are You
Smarter than a 5th Grader??
• Pilot-test to make sure they’re at an
appropriate level of difficulty
Skip or Contingency
• Two or more part question
– Answer to one part determines the next
question you’ll receive
– Sometimes called screening questions
Open v. Closed Questions
• Open-ended: Respondent can give any answer they
want
• Close-ended: Gives respondents fixed answers
from which to choose
• Big surveys use close-ended…quicker and easier
• Can mix both methods
• Open-ended best in early or exploratory stages of
research
• Partially open questions leave final answer choice
•
as other
Advantages of Closed
•
•
•
•
Easy and quick to do
Easy to compare
Easier to code and analyze
Response choices can clarify meaning
of question
• Replication is easier
Disadvantages of Closed
• Suggest ideas that respondent may not
have thought of
• No opinion people can answer anyways
• Frustrating if desired answer is not a
choice
• Simplistic responses to complex ?s
• Force us to make choices we wouldn’t make
in the real world
Advantages of Open
• Unlimited responses
• Answer in detail and clarify
• Unanticipated feelings can be
discovered
• Creativity and self-expression
• Reveal logic
Disadvantages of Open
•
•
•
•
•
•
•
Different degrees of detail in answers
Can be irrelevant
Hard to compare/analyze
Coding is tough
Hard to write verbatim in interviews
More time necessary
Take up lots of space on questionnaire
Nonattitudes and Middle
Positions
• Debate whether to include choices for
middle, neutral, or nonattitudes
• Two errors:
– Accept nonattitude when respondent has one
– Force choice when respondent doesn’t have one
• Standard-format v. quasi-format v. fullformat
Agree/Disagree,
Rankings, or Ratings?
•
•
•
•
•
Agree/Disagree
Rankings
Ratings
Best to offer respondents explicit alternatives
Avoid the response set bias (just check agree to
everything)
• Respondents can rate several items equally high,
but will put them in hierarchy if asked to
Word Issues
• Need to use simple vocabulary and
grammar to minimize confusion
• Need to watch for effects of specific
words or phrases
– Forbid v. not allow
– Certain words trigger an emotional reaction
– “To help the poor” v. “for welfare”
Length of Survey
• Researchers prefer long because
they’re cost effective
• Phone: 5-20 minutes
• Mail questionnaire: 3-4 pages
• Face-to-face: Around an hour
Order Effects
• Strongest for people who lack strong
views, less educated, elderly
• May not perceive each element of a
survey as isolated and separate
• Influence through content and
response
Context Effects
• Funnel sequence:
– Start general and get more specific
• Split sample
– Half get questions in one order, other
half different
• More ambiguous the question, the
more context effects matter
Survey Layout
•
•
•
•
•
•
Clear, neat, and easy to follow
Number questions
Identifying info
Cover sheet
Professional appearance
Provide instructions
Survey Layout
•
•
•
•
Crucial in mail questionnaires
Use letterhead
Always thank them for participation
Circle response v. check box v. fill in
dots v. put an X in a blank
• List answer categories down, not
across
Survey Layout
• Use arrows and instructions for
contingency questions
• Matrix questions
– Compact way to present a series of
questions with the same response
categories
– Box 7.5
Nonresponse
• If not enough people respond, we may
not be able to generalize
• Many people are burnt out with
surveys these days
• Improve eligibility rates through
careful respondent selection
Nonresponse
• Decrease refusals through sending
letters before calling, small
incentives, adjusting behavior, using
alternative interviewers, using
alternative interview methods
• Initial contact is HUGE in
determining likelihood for success
Surveys
• Survey research is research based on
direct or indirect interview methods.
• Many types of surveys:
–
–
–
–
–
In person (including interviews)
Mail
Telephone
E-mail and Internet
Group surveys
Mail and Self-Administered
Questionnaires
• Advantages:
–
–
–
–
–
Cheaper
Wide geographical area
Completed when convenient
Respondent can check personal records
Anonymity
Mail and Self-Administered
Questionnaires
• Disadvantages
• Low response rate
• Can’t control conditions under which its
filled out
• Someone other than intended can fill out
• Can’t observe reactions
• Ill suited for those who don’t have good
grasp of English
Mail and Self-Administered
Questionnaires
• How to Increase Response
–
–
–
–
–
–
–
–
Address to specific person
Careful cover letter on letterhead
Postage-paid, addressed return envelope
Neat, attractive layout
Send two follow-up reminders
Nothing during major holidays
No questions on back page
Include a small monetary inducement
Web Surveys
• Advantages
–
–
–
–
Very fast
Inexpensive
Flexible design
Can use visuals or audio
Web Surveys
• Disadvantages
–
–
–
–
Coverage (75% have internet)
Multiple e-mail addresses
Self-selection
Protecting privacy and confidentiality
• Encrypt
– Complexity of design depends on software
• Screen-by-screen questions
• Progress indicator
• Needs to be easy to move back and forth across questions
Phone Interviews
• Advantages
–
–
–
–
95% coverage rate
Can interview about 1500 in a few days
Half the cost of face-to-face interviews
CATI
Phone Interviews
• Disadvantages:
– Higher cost
– Limited interview length
– Use of interviewer reduces anonymity
and introduces potential interviewer
bias
Face-to-Face
• Advantages:
–
–
–
–
Highest response rates
Permit the longest questionnaires
Can observe surroundings
Ask all types of questions
Face-to-Face
• Disadvantages:
– High cost
• Training, travel, supervision, personnel costs
– Interviewer bias
Comparing the Methods
• Cost:
– I, M, P, F2F
• Speed:
– I, P, F2F, M
• Length:
– F2F, M, I, P
• Response rate:
– F2F, P, I, M
Interviewers
• Obtain cooperation and build rapport, but
remain neutral and objective
• Interviewer must treat everything in a
matter-of-fact manner
• Pleasant, honest, accurate, mature,
responsible, moderately intelligent, stable,
and motivated
• Poise and tact
Interviewer Bias
• Interview expectations can create
significant bias
• Social setting can affect answers
– Students answer differently at home
and at school
• Visible characteristics can play a
role…such as race and gender
CATI
• Computer-assisted telephone
interviewing
• Read questions from the screen and
enter answers immediately
• Automatically makes calls for you
• Automatically stores answers in a
database
The Ethical Survey
• Invasion of privacy
– That’s what ANY survey is!!!!
• Voluntary participation
– Informed consent
• Use of push polls—AKA “pseudosurveys”
• Sloppy reporting of surveys by the media
– No quality control standards
Surveys
• Five important factors influence the
choice of survey type:
–
–
–
–
–
Cost
Completion rates
Sample-population congruence
Questionnaire length
Data-processing issues
Surveys
• Cost:
– All surveys share basic expenses for materials.
– Cost of a survey can be quite variable, based on
•
•
•
•
•
•
•
Professional questionnaire design
Length of the questionnaire
Geographic dispersion of the sample
Callback procedures
Respondent selection rules
Availability of trained staff
Method-specific costs like long distance rates or
postage
Surveys
• Completion rates:
– Low response rate on a survey limits the
ability to make statistical inferences
about the target population
– Must make an effort to reach those
selected for inclusion in the sample to
maximize the representativeness of the
sample…or risk bias
Surveys
• Sample-population congruence:
– If those who fail to participate in the
survey share common characteristics,
the sample will be wrought with
systematic sampling error and the
conclusions made with the data could be
incorrect.
Surveys
• Questionnaire length:
– Keeping respondents’ attention during a survey
helps achieve the best possible response
quality.
– A lengthy questionnaire can lead respondents
to lose focus or speed through questions to
finish sooner, resulting in low response quality.
– Survey length ranges between three and four
dozen items, depending on complexity and
format.
Surveys
• Data-processing issues
– A good deal of time and effort are
required to process data.
– Surveys using more elaborate or
complex designs or a large number of
questions will require more time and
effort.
Surveys
• Each type of survey has advantages
and disadvantages when it comes to
the five factors.