Survey Research
Neuman and Robson Ch. 8
Introduction to Survey Research


Survey research is a very old, and a very
popular, research technique
1880 survey by Karl Marx in France


25000 questionnaires sent to workers
Late 19th century “Chocolate Sociologists”


Rowntree and Cadbury
Community surveys to study poverty
Topics appropriate to survey
research:

descriptive, explanatory, and exploratory
research purposes

best method available to collect original
information about a population

To measure attitudes and orientations
General features of survey research

respondents are chosen through probability
sampling procedures

Systematic questionnaire or interview
procedures

Sophisticated statistical techniques
Strengths and Weaknesses of Survey Research:

Advantages





Reliability
Can describe large population
Flexibility
Standardized
Disadvantages




Validity
Superficiality
Cannot modify questionnaire in field
Artificiality
Public Opinion Polling Canada






No official or mandatory regulation
i.e. election polls
Could influence voters
“bandwagon” phenomenon
“free will” effect
Problems and discrepancies:



20-40% answer “don’t know” or “no answer”
Results often published as if out of 100%
misrepresentation
Mail Surveys and Self-Administered
Questionnaires






Hand-administered to a group or mailed to sample
Mail distribution and return practices
Monitoring returns
Follow-up mailings
Response rates: 50% adequate, 70% very good
Additional factors:


Sponsorship
Inducement to respond
Disadvantages of mail surveys:






Requires simple questions
Cannot observe respondents
No opportunity for probe
No control over conditions or who responds
Not suited for people who are illiterate or for
whom English not first language
Low response rate can be a problem
Face-to-Face Interview
Surveys








Very expensive
Interviewers require training
Use interview schedule
Appearance and demeanour crucial
Need familiarity with questionnaire
Must follow question wording exactly
Must record responses exactly
Also record other events during interview
Advantages and Disadvantages of
Face-to-Face Interview Surveys

Advantages:
 Can probe for responses
 Can keep respondent “on task”

Disadvantages:
 Cost and training
 Interviewer bias
Telephone Surveys




In past, could reach 95% of households
Now, between 20 and 26% of Canadians
have no landline (see next slide)
Use of telephone interview schedules
Selecting a sample for telephone surveys



Telephone directory
Random Digit Dialing (RDD)
Selecting a respondent from a household
Percentage of households in Canada that
use a cell phone only
(Source: Stats Can “The Daily”, 2014)
All households
Households under 35
years of age
Households aged 55 and
older
2008
2010
2013
8
12.8
21
26.1
39.3
60.6
1.9
3.8
6.4
Telephone Surveys

Can achieve higher response rates

Can decrease "don't knows" and "no answers”

Interviewers can help clarify

Telephone surveys have many advantages of faceto-face but without high cost
Advantages and Disadvantages of
telephone surveys

Advantages:




cheaper to carry out than face-to-face interviews
less time and effort
more impersonal
Disadvantages:




large number of tries to achieve a small number of
successes.
less motivation generated among respondents
No visual cues/aids possible
Limited interview length
Computer-assisted telephone
interviewing (CATI)




interviewers sit at a computer terminal with a
screen
answers given are directly entered into the
computer
Very fast
Often used in polling (i.e. Video: Ask a Silly
Question)
Internet (Web) Surveys



In Canada, 95 per cent of Canadians in the highest
income quartile are connected to the Internet, yet
only 62 per cent in the lowest income quartile have
Internet access (2014)
Use same basic design as mail surveys
Advantages:



Fast, cheap
Can use graphic and visual effects
Disadvantages:


Coverage – not all groups use computers
Privacy – need for encryption
Comparison of Survey Methods
Criteria
Interview
Mail
Telephone
Web
_________________________________________________________________
1.
Cost
high
low
moderate
very low
2.
Response rate
high
low
high
moderate
high
low
moderate
low
moderate
high
3.
Level of control of
situation
4.
Applicability to
geographically
dispersed populations
moderate
high
5.
Applicability to
heterogeneous
populations
high
low
high
moderate
6.
Obtaining detailed
information
high
moderate
moderate
low
7.
Speed
slow
slow
fast
fastest
Secondary Analysis of Survey Data



New analysis of data collected for another purpose
Use of shared data and data archives (i.e. Census
and GSS)
Advantages



cheaper and faster
Access to larger datasets
Disadvantages



problems of validity
Questions may not meet your needs
Sample may not be adequate
Questionnaire Construction

usually includes a variety of demographic
questions, and one or more scales and
indexes aimed at collecting data on attitudes,
beliefs, or behaviours

purpose of a survey is to try to collect
information about a sample in order to make
generalizations about the larger population
The general structure of a
good questionnaire:







1. Title
2. A brief introduction and explanation of research.
3. A section of general demographic questions re:
gender, age, marital status, education. Note: more
sensitive demographic questions (i.e. income, home
value, etc.) might be better placed at the end of the
questionnaire.
4. Section of non-threatening or less sensitive
behavioural and attitudinal questions.
5. A section near the end with any sensitive or
threatening questions.
6. Any additional demographic information.
7. Conclusion with brief statement again thanking
respondents for their time and effort
Additional Considerations:





1. Keep questionnaire as short as is reasonably
possible.
2. Appearance of the questionnaire is important.
Don't crowd your questions, and use an easy-toread font. Leave some space between questions.
3. Use bold font and underlining for titles/headings.
4. Use examples and sample questions for clarity,
but be careful not to introduce bias.
5. Include an open-ended question at the end of the
questionnaire asking for any additional information
and/or feedback. This question sometimes provides
some interesting information!
Constructing the questions:




Create questions that use indicators which
will specifically address your hypotheses.
Use existing scales or measures whenever
possible.
You can also modify a scale or index to meet
the needs of your particular study if an
existing scale is not quite appropriate.
The advantage in using an existing scale is
that its reliability and validity are already
known.
Types of questions to use:

a. Simple questions that use a concrete
indicator (i.e. What is your age in years?___).

b. Likert style questions are used for measuring
simple attitudes, beliefs, emotions, or behaviours
(i.e. When you think back to your high school
years, you feel: 1. Very unhappy 2. Somewhat
unhappy 3. Don't feel anything 4. Happy 5. Very
happy)

c. Matrix questions are useful when you have a
number of related items. Matrix questions are
composed of a series of questions that have
identical response categories which are presented
in table format with the response category labels at
the top of the chart.
Types (cont.)

c. Guttman scales: These are useful for measuring
the strength or intensity of an attitude.

d. Rank-ordering: Rank ordering is a method that
works well with a small number of objects, such as
10. For example, instead of rating how interesting
each of a set of subject areas are, you could ask the
respondent to simply sort them in order of most to
least interesting.

e. Paired comparisons: In this method, you
present items two at a time, and ask respondent to
pick which one has more of some attribute. For
example, you can present behavioural problems
and ask which one is more serious.
Closed or open-ended question
responses

A closed response set refers to those
questions which have fixed categories for
answers (uses quantitative analysis)

Open-ended questions allows for free
responses (qualitative analysis)
Filter questions and
contingency questions

Filter questions are useful for sorting
respondents.

Contingency questions allow respondents to
skip portions of the survey that may not be
relevant to them.
Principles of Question-Writing







1. Keep questions as short and concise as possible.
2. Choose your wording carefully.
3. Try not to ask questions beyond a respondent's
capabilities.
4. Avoid emotional language and the use of "loaded"
words.
5. Watch out for prestige bias in your questions.
6. Don't use leading questions like "You don't
smoke, do you?"
7. Avoid ambiguity and vagueness.
Principles (cont.)





8. Don't use double-barrelled questions.
9. Avoid false premises or assumptions.
10. If possible, don't ask respondent about future
intentions. The link between intentions and future
behaviour is tenuous.
11. Try not to use negatives and double negatives in
your questions.
12. Watch for explanatory statements that may bias
the answer to the question. Watch out also for
questions that might influence the answers to
subsequent questions.
Other Issues

Aiding recall



Provide special instructions or extra time
Ask month by month, week by week, etc.
Threatening questions


People under-report illness, disability, deviance,
illegal activity, income/wealth
Create “enhanced questions” or embed activity in
list of more serious activities
Issues (cont.)

Social desirability bias



People tend to over-report socially desirable
behaviours (being cultured, voting, giving to
charity, being good spouses or parents, etc.)
Try to minimize importance of these activities or
present alternatives in questions
Knowledge questions


Make sure questions are appropriate level
Phrase questions so respondents feel
comfortable not knowing the answer