Survey Research
PS 1006 Lecture 1
Sam Cromie
1
Research Methods
• Observational
research
– Unstructured
– Structured
– Participatory
• Experiments
– Group design
• Within-subject
• Between subject
• Surveys
• Qualitative research
–
–
–
–
Interviews
Focus groups
Document analysis
Etc.
• Case studies
– Single-subject design
2
Survey
research
Sampling
Methods
Writing good questions
Data
Collection
Methods
Research Designs
Essential Reading:
Research Methods in
Psychology (5th or 6th
edition)
Shaughnessy, Zechmeister
& Zechmeister
Surveys: Advantage and Disadvantages
• Positives
– Easy to sample a
large number of
people
– Context likely to
retains ecological
validity
– Relatively easy to
administer
• Negatives
– Danger of biased
surveys
– People may not
answer truthfully
– No true IVs - Cannot
infer causality
Types of question
• Open-ended - ‘How do you feel today?’
– Allows wide range or responses - preferable
when researcher has little idea of the type or
response to expect
– Respondents decide how much information
to give
– Lack of structure can make scoring of the
responses difficult
5
Types of question
• Closed questions -
‘On a scale of 1-5 how happy
do you feel today?’
– A fixed set of responses is provided
– Respondent must pick one
– Easy to score
– May miss responses that some participants
wants to give since the scope for answers is
limited
6
Types of question
• Mixed questions
– A fixed set of responses is provided
– An open-ended choice is given to cover any
alternatives that may have been missed
– Still has a limited scope
How do you feel today?
A. Happy
B. Sad
C. Angry
D. Relaxed
E. Other
(specify)_________________________________
7
Response scales
• Categorical questions – non-directional options
How are you happy today?
Happy
Angry
Tired
Sad
• Likert Scales - Labeled points on scale
I am Happy today
Strongly
Agree
Agree
Disagree
Strongly
Disagree
• Percentages
How happy are you on a percentage from 1-100
• Continuous scales
Please mark the line below to express you level of happiness:
Happy
Unhappy
Categorical Questions
• Often used for demographic information:
– Job type, ethnic background, etc.
• Dichotomous questions – only two options:
Yes/no, Male Female
– No indication of response strength
• Can be used as quasi IVs in data analysis
• Can be analysed with other categorical
questions using Chi Squares
Likert Scales
• Labeled points on scale
• Several possibilities
– Strongly agree, Agree,
Disagree, Strongly
disagree
– Scale from 1-5 with ends
(poles) labeled
– Useful to try and define
parts of the scale
(anchoring)
• How many values should
be put on a scale?
– 7±2 options - above this
difficult to use the full range
• When normally
distributed can be
analysed parametrically
10
Percentages
• Gives participants a wider range of
possible responses
• But resposes tend to cluster around
commonly used percentages – 10, 20, 25,
50, 75, 90
11
Continuous scales
Happy
Unhappy
• Line is normally 10cm long
• Score = distance in mm from start of line
to mark
• Get a fuller use of the scale than
percentages
• Allows more sensitive measurement than
Likert
• More challenging to code
Writing good questions
• A survey lives and dies by its questions
– Unclear questions confuse
• Vocabulary should be clear and simple
– Questions should be specific
• Should government do something about health
care?
• Should government pass a law on health care?
• Should government pass a law on health care that
would guarantee health coverage for all?
Writing good questions
• Include conditional information before the
key idea in the question
– If you were the victim of a violent crime, would
you consider buying a gun?
• Consider varying the polarity of the
questions
– Questions should be phrased both negatively
and positively
• Some people just like to say No (or Yes)!
Avoid when writing Qs
• Leading questions - suggesting the ‘right’ answer
– People often recognize the importance of gun control in
the prevention of crime. What is your opinion of gun
control?
• Loaded questions - emotionally charged
– Are you in favor of laws that that allow dangerous
convicted felons out of prison early?
• Double-barreled questions - packing too much in
– Do you favor life sentences for repeat offenders and
stricter sentences for criminals who used guns?
Basic sampling terms
• Population - all members of defined group
– Census - ask entire population
– Survey - ask representative sample
• Sampling frame - specific list of all the members
of the population - not always accurate
– e.g., register of currently enrolled students - what
about students who are abroad or taking year out?
• Element - a member of the population
Basic sampling terms
• Representativeness – dictates how generalisable the results are. Clearly,
biased samples = bad.
• Selection bias –
– procedures used to select sample which result in nonrepresentative sample
17
Sampling methods
• Probability sampling - each element
has same probability of being included
– Simple random sampling
– Stratified random sampling - population
divided into strata and random samples
are drawn from each e.g, make sure
ethnic balance reflects population
• Most sampling is not really random
18
Sampling methods
• Nonprobability sampling - each element
has different probability of being chosen
– Accidental sampling - selection on basis of
availability and willingness to respond e.g.,
television polls. People may participate:
• For money
• Because they are interested in topic
• Because they have a particular view point
– Purposive sampling - elements included on
the basis of their special characteristics
Potential sampling problems
• Flawed surveys often reflect poor samples
– Sampling bias can enter in two places
• At selection of sampling frame
– Use of phone directories for general population
– List of subscribers to a magazine
• At selection of a sample
– Use of mail survey
– Calls made to college students on Friday and Saturday
nights
• Any study in which subjects self-select is
potentially problematic
Survey research designs
• Cross-sectional designs – view of population at
one time period only
21
Example
• Suppose we conduct a survey of the number of
left-handers in the population
• Of inds below 60, left-handers approx = 10%
• Of Inds above 60, left-handers are much less than 10%
– Conclusion: left-handers die younger
– Is this valid
• Need to ensure that timing does not produce a
major confound
Successive independent
samples
• Can be used to assess change over
time
– Must ensure that samples are comparable
• Any change detected should be due to time
and not sample differences
Longitudinal studies
• Follow a single group of individuals over time
– Requires a lot of time and effort
– Subject drop out a problem
• Sometimes mortality
• Most often people drop out for some reason
• Must ensure that there is not a systematic reason for
drop out
Data collection
• There are several
ways to conduct
surveys
–
–
–
–
Through the post
In person
Over the phone
Via internet
• Each of these
methods has its own
advantages and
disadvantages
Postal surveys
• Advantages
– Minimizes effort needed to
collect data
– Good for personal or
embarrassing topics
• Respondent’s replies are
anonymous
• May cut down on the
amount of socially
desirable responding
• Disadvantages
– May not be filled out be
intended participant
– Respondents cannot ask
for clarification
– No follow-up questions can
be asked
– No control over the order in
which completed
– Big potential for response
bias
Self-selection - postal surveys
• Respondents in postal surveys always selfselect
• Controlling response bias in postal surveys
– Try to ensure a high return rate
• Over 50% is considered very good
• Response bias is less severe with return rates over 50%
– Practical steps to improve response rate
• Include a SAE
• Personalize the introduction (at least with name)
• Do follow-up mailings with new copies of the survey to
those who do not respond
Personal interviews
• Advantages
– High response rate (often over
80%, though less in highcrime areas)
– Can control order of
responding
– Control over context of
responding
– Ensure the correct participant
answers question
– Respondent can ask for
clarification
– Interviewer can ask follow-up
questions
• Disadvantages
– There is a potential for
interviewer bias
• May suggest the desired
response (non-verbally)
– People may give socially
desirable responses
• They may not want to
express their true beliefs to
the interviewer
– Difficult to ask about
embarrassing topics
• No anonymity for the
respondent
Telephone surveys
• Advantages
– Better than personal interviews in highcrime areas
• People do answer the phone!
– More anonymity than for personal
interviews
– Can control order of responding
– Respondent can ask for clarification
– Interviewer can ask follow-up questions
Telephone surveys - disadvantages
• People may not want to be bothered over the
phone
– Particularly now that telemarketers often use surveys
as a sales pitch
• No control over surroundings
– Respondent may be doing a number of things at once
• There is the potential for interviewer bias
– May suggest the desired response through nonsemantic content of dialogue
• People may provide socially acceptable
responses
– They may not want to express their true beliefs to the
interviewer
Internet surveys
• More and more widely
used
• Data can be imported
directly into analysis
package
• Quick and efficient
• Cuts down on paper use
• Excellent for international
surveys
• Can use audio-visual
content, etc.
• Lack of control of context
in which it is filled out
• Identity of participant
• Respondents cannot ask
for clarification
• No follow-up questions
can be asked
• No control over the order
in which completed
• Biased against the less
computer literate
• Inappropriate for many
populations