Chapter 4
10/28/2013 12:15
Geog 3250
1
Thoughts on researcheradministered surveys?
 Keep a list of things not to do in a researcher-
administered survey from this tongue-in-cheek video:
10/28/2013 12:15
Geog 3250
2
Structured Interviews
Method
Question Order
Context
Structured interview schedule
interview
strict
usually face-toface
Survey
questionnaire
strict
various
In-depth
Interview
Interview guide, and
the interviewer
flexible to keep it
conversational
usually face-toface
10/28/2013 12:15
Instrument
Geog 3250
3
Structured Interviews: errors and
interviewer effects
Method
“Errors”
Opportunities for
Clarification
Structured minimize
interview
minimize
high: but clarify in
exact same way in all
situations
Survey
minimize
minimize
low: due to social
distance
In-depth
Interview
use to sustain rapport
and keep the
conversation
meaningful
No “errors”, just
opportunities for
deeper
understanding
high: clarification
may be the source of
meaningful
conversation
10/28/2013 12:15
Attitude towards
interviewer effects
Geog 3250
4
Structured Interviews and “errors”
 Variation due to error can come from:
 Intra-interviewer variability: an interviewer is not
consistent in asking questions or recording answers
(with the same respondent or a different one).
 Inter-interviewer variability: lack of consistency in
asking questions or recording answers between different
interviewers.
10/28/2013 12:15
Geog 3250
5
Structured Interviews: responses
Method
Response mode
Data recording
Structured
interview
mostly closed-ended
on the interview schedule
Survey
mostly closed-ended
on the questionnaire
In-depth
Interview
open-ended and conversational
Field notes or audio
recording
Example of close-ended question; data recorded on the document
10/28/2013 12:15
Geog 3250
6
Interview Contexts
 Work with your neighbour to fill out this table - as
many as you can think of per cell
Method of
Administration
Strengths
Limitations
Researcher-Administered
Telephone e.g. with CATI
Face-to-Face
Self-Administered
Drop-off, mail-back
Hand-out, hand-back
Email, 0nline
10/28/2013 12:15
Geog 3250
7
Conducting Structured Interviews:
Basic Points
 Schedule – Interviewers should be familiar with the
interview schedule.
 Introduction – Respondents should be provided with
a credible rationale for the research.
 Rapport – Interviewers should try to develop rapport
with interviewees but refrain from being too chatty
(unlike semi-structured or unstructured)
10/28/2013 12:15
Geog 3250
8
Conducting Interviews: Basic
Points, cont’d.
Asking Questions
 The question should be asked
exactly as stated.
 Small changes to wording can
make a big difference
 Similarly, the answers should be
recorded as exactly as possible.
 Questions should be asked in
the order they are given on the
interview schedule.
10/28/2013 12:15
Geog 3250
9
Conducting Interviews: Basic
Points, cont’d.
 show cards - to provide detailed
information, e.g., Likert scale
options
 appropriate if


long list of possible answers;
same set of answer categories
applies to several questions.
10/28/2013 12:15
Geog 3250
10
Conducting structured interviews:
Basic Points
Leaving the interview
 The interviewer must remain
professional and have a set
response to disengage
respondents so that they may do
more interviews (i.e., if
researcher is motivated to have
large sample size)
10/28/2013 12:15
Geog 3250
Sometimes they want to be done and
move on, sometimes it is the other way
around…
11
Conducting (survey) interviews:
Basic Points
Training
 Good interviewing is a skill
set to be developed. It
usually requires considerable
training and supervision to
become good at it…but you
have to start somewhere!
10/28/2013 12:15
Geog 3250
12
Questionnaires (self-administered)
 Questionnaires are essentially
structured interviews without an
interviewer.
 They involve filling out a form which
is then returned to the researcher,
often by mail or in person.
 Because the respondent has to read
the questionnaire without the aid of
an interviewer, it has to be very clear
and easy to follow – layout matters!!
10/28/2013 12:15
Geog 3250
Example of Statistics Canada selfadministered survey question
13
Questionnaires (self-administered)
 Compared to interviews, questionnaires tend to
 have fewer ‘open’ questions, because ‘closed’ questions
are easier to answer;
 have simple designs, so the respondent can complete
all sections without difficulty;
 be short, to avoid respondent becoming discouraged.
10/28/2013 12:15
Geog 3250
14
Questionnaires (self-administered)
 Advantages over structured interviews:
 Cheaper, quicker, more convenient to administer
 No interviewer effects, including errors in administering
the instrument
 Social desirability bias seems to be reduced.

Respondents are more likely to give genuine answers to
‘sensitive’ questions, e.g., those regarding sexual practices,
drug use, criminal activity, etc.
10/28/2013 12:15
Geog 3250
15
Questionnaires (self-administered)
 Disadvantages
 Researcher cannot explain the question – re: validity.
 Greater risk of missing data, lack supervision.
 Researcher cannot probe.
 Difficult to ask a lot of questions.
 Difficult to ask different kinds of questions, especially
‘open’ ones and questions requiring a filter.
 Order effects hard to control – may answer out of order
 Literacy issues
 Designated respondent may not have completed
questionnaire.
10/28/2013 12:15
Geog 3250
16
Questionnaires (self-administered)
 Online survey
 Growing in popularity
 Finding participant
lists…random?
 Email surveys

Tend to be aimed at smaller,
more homogenous groups
10/28/2013 12:15
Geog 3250
This one is free and can
be restricted to UWO
usernames
17
10/28/2013 12:15
Geog 3250
18
Questionnaires (self-administered)
 Researcher-driven diaries


A form of questionnaire
Participants record their feelings,
perceptions, actions, etc. on a form shortly
after they occur.
 Participants should be given explicit
instructions on how to complete the diary,
the time periods for recording responses,
and the types of experiences to be recorded.
 Used for quantitative or qualitative research.
 ‘structured’

like a regular questionnaire with closed
questions, or
 ‘free text’

experiences written down without fixed
responses, like answers to ‘open’ questions
10/28/2013 12:15
Geog 3250
19
Diaries
 Advantages of diaries over interviews and questionnaires:
 Fairly accurate data, re: frequency or time spent on behaviours
of interest
 Fairly accurate data about sequencing of behaviours
 Good for getting data on sensitive matters, e.g., sexual
practices
 Disadvantages
 More expensive than personal interviews
 Attrition: people may tire of using them
 Details may not be recorded quickly enough.

Errors, omissions
10/28/2013 12:15
Geog 3250
20
Respondent Problems - Structured
Interviews and Questionnaires
 Response sets
 the respondent is not motivated to provide a genuine
response – answer all the same without reading
I am not paying attention to each
question as I respond
10/28/2013 12:15
Geog 3250
21
Respondent Problems - Structured
Interviews and Questionnaires
Acquiescence: trying to please the researcher
 e.g., respondent agrees just to be ‘cooperative’
 partial antidote: mix items so that they have logically
opposite positions
Do you like this image?
10/28/2013 12:15
Geog 3250
22
Respondent Problems - Structured
Interviews and Questionnaires
Social desirability: avoiding
appearance of being
bad/unlikeable
I hope these
are the
answers she is
looking for.
 Answers are not sincere, e.g.
questions on racism or the
environment
10/28/2013 12:15
Geog 3250
23
Respondent Problems - Structured
Interviews and Questionnaires
Laziness or boredom: answers just to get the end of the
process
 E.g., respondent checks all the same answer to get it done
Assume all respondents might be this lazy?
10/28/2013 12:15
Geog 3250
24
Feminist (and other) Critiques
 Structured interviews and
questionnaires - exploitative.
 asymmetrical power relationship
between the researcher and the
respondent.
 Link to Ethics: Increased attention
given to the rights of research
respondents in recent years, e.g.,
privacy rights, the right to end the
interview at any time, etc.
Power imbalances are not always this
overt
 Is this enough?
10/28/2013 12:15
Geog 3250
25
Questionnaire Design
 Identify at least one problem with each question in the following
questionnaire snippet:
10/28/2013 12:15
Geog 3250
26
Open or Closed Questions?
 Closed Questions
 Present the respondent with a set of answers from which to
choose
 E.g., see previous slide
 Open Questions
 Response decisions are left completely to the respondent
 Answers must be coded into to numerical data

Tend to be used in quantitative research
 E.g., “What do you think about using alternatives to
chemical pesticides on your lawn”?
10/28/2013 12:15
Geog 3250
27
Open Questions
 Advantages
 Respondents can answer in their own terms
 Allow for unusual, unanticipated responses
 No suggested answers, so responses may expose
knowledge and be more genuine
 Good for exploring new or changing areas of research
 Answers may lead to fixed-choice responses
10/28/2013 12:15
Geog 3250
28
Open Questions
 Disadvantages
 Time-consuming to record
answers
 Answers have to be coded
 In self-administered
questionnaires, respondents
may balk at the request to
write long answers
 Recording inaccuracies for
verbal answers
10/28/2013 12:15
Geog 3250
Why did your degree take longer
than the funding period?
•Not really sure. I worked away on
it diligently but the analysis just
took longer than expected, I
guess.
• There are worse places to be
than grad school.
• Closer or more effective
supervision was the primary
reason.
• Too much procrastination and
family issues.
What code/categories?
Can one response go in multiple categories?
29
Coding Open Questions
 Post-coding
 Data are gathered, then themes or categories of
behaviour are discerned.
 e.g., an open question on the legalization of marijuana
might have codes ‘very hostile opposition’, ‘indifferent’,
etc.
 Each code would then be assigned a number, and
attributed to respondent
10/28/2013 12:15
Geog 3250
30
Coding Open Questions, cont’d
 Pre-coding
 Themes or categories of behaviour are decided upon
before the data are gathered.
 May be done with fixed-response items (i.e. closedended)

e.g., ‘What is your view on the legalization of marijuana?
Are you strongly opposed, opposed, neither opposed nor in
favour, in favour, or strongly in favour?’
 Data may be gathered freely through open responses
and sorted into pre-set categories afterward. (What if
there are responses that seem to require a new code?)
10/28/2013 12:15
Geog 3250
31
Coding Open Questions, cont’d
 Basic principles:
 The categories must not overlap.
 The list of categories must be exhaustive.


Must cover all possibilities.
‘Other’ category is usually required.
 Coders should be provided with a ‘coding frame’ or
‘coding manual’ that establishes clear rules about how
codes should be applied.
10/28/2013 12:15
Geog 3250
32
Closed Questions
 Advantages
 Easy to process answers
 Standardization allows comparison of answers
 Fixed responses may help clarify what the question
means
 Easier and quicker for the respondent to complete
 Reduced bias in recording answers

No interpretation required by researcher
10/28/2013 12:15
Geog 3250
33
Closed Questions
 Disadvantages
 Loss of spontaneity and
authenticity - relevant answers may
be excluded from choices provided.
 Antidote: ‘Other’ category with
open area to elaborate (but you
may want to code it afterwards!)
 Defining categories that do not
overlap.
 Antidote: Pre-test as open-ended
to establish appropriate and
distinct categories.
 Difficult to make forced-choice
answers exhaustive
10/28/2013 12:15
Geog 3250
24. Why do you oppose the
construction of more wind
turbines in your community?
Bird/bat collisions
Economic costs outweigh
benefits
Health threats
Impacts on farm and domestic
animals
Unfair siting process
Other _____________________
34
Closed Questions, cont’d
 Disadvantages (cont’d)
 Respondents may differ in their interpretation of the
wording of fixed responses.

e.g., the meaning of ‘strongly’ in ‘strongly agree’
 Respondents may not find a fixed response that they feel
applies to them.
 Large numbers of closed questions can reduce rapport in
interviews

Antidote: include open-ended responses even if you may not
code them?
10/28/2013 12:15
Geog 3250
35
Types of Questions
 Personal factual questions
 e.g., age, occupation, number of cars
owned
 Factual questions about others,
events etc?
 Avoid these: makes respondent feel
they are being tested and look
stupid
 Antidote: Questions about
perceptions are better – puts them
in position of being “the expert”.
10/28/2013 12:15
Geog 3250
This “survey” tests, but it is for education,
not research purposes
36
Types of Questions, cont’d.
 Beliefs
 e.g., “Do you believe climate change has
happened globally over the last 50 years?”
 Attitudes
 Very common, subjective, “taste”, or
perception
 use of Likert scales
 E.g., “Do you feel we should do more to
mitigate the effects of global climate change.”
“Gabe Kotter” tests attitudes
where participant is the “expert”
regardless of how they choose
Likert “agree” scale, often used to create an index from multiple items
10/28/2013 12:15
Geog 3250
37
General Rules for Designing
Questions
 Keep the research questions/hypotheses in mind.
 Focus on exactly what you want to know.
 Be specific.
 e.g., “How many children 18 and under are living at
home?” vs. “How many children do you have?”
 Put yourself in the position of the respondent
 How would you answer the question?
10/28/2013 12:15
Geog 3250
38
Specific Rules for Designing
Questions
AVOID:
 ambiguous terms
 Even commonly used terms may be vague, e.g., does ‘income’
include tips? Investments income? Does it mean after tax
income?
 long questions: respondents may lose track of what the
question is asking
 multiple phenomena in the same question
 E.g., “Do you agree your network of closest friends and family
is helpful and supportive?”
10/28/2013 12:15
Geog 3250
39
Specific Rules for Designing
Questions
AVOID:
 several very open-ended questions (do unstructured
interviews instead?)
 e.g., ‘What do you think of the prime minister?’
 leading questions
 e.g., ‘Do you feel the monarchy, which costs taxpayers over
$5 million each year, be abolished?’
10/28/2013 12:15
Geog 3250
40
Specific Rules for Designing
Questions
AVOID:
 questions that include negatives, especially double
negatives.
 e.g., ‘Do you oppose the rule stating that laptops are not
allowed in classrooms?’
 jargon/technical terms. And be sure that respondents
have the knowledge needed to answer the question.
 E.g., ‘How would you rate the level of social capital in your
neighbourhood?’
10/28/2013 12:15
Geog 3250
41
Specific Rules for Designing
Questions
 Ensure symmetry between a closed question and its
answers.
 e.g., avoid:
 Responses should be ‘very satisfied, somewhat satisfied,
…
 Ensure the answers are balanced.
 e.g., avoid:
Don’t let instructors design their own course evaluations?
10/28/2013 12:15
Geog 3250
42
Specific Rules for Designing
Questions
 Memory
 Beware of expecting respondents to recall what they’ve done
or observed in detail.
 e.g., number of hours spent online per week
 ‘Don’t know’ option
 Consider avoiding this(e.g., use “neither agree/disagree”
instead if you do not want to force them to choose)
 Thought to prevent forcing the respondent to choose, but it
actually serves to offer an easy way out.
10/28/2013 12:15
Geog 3250
43
Exercise
 Design a survey question that violates one rule of
questionnaire design identified above
Text 335633 and your question to 37607
Or
Respond at this link
NB You need to be in slideshow mode (full screen) for any of the links to work
10/8/2013 10:15
Baxter Geog 2153
YES peek back
up the notes…
poll results here
44
Specific Rules for Designing
Questions
 Question order
 All respondents should receive questions in the same order
(unless testing for order effects).
 Asking a particular question may affect the responses given to
subsequent questions.
 Early questions should be directly related to the announced
research topic.
 Questions likely to be of interest to respondents should be
asked early in the interview.
 ‘Sensitive’ questions (e.g., ‘Have you ever been a victim of
sexual assault?’) should be asked well into the schedule, but
not at the very end.
10/28/2013 12:15
Geog 3250
45
Specific Rules for Designing
Questions
 Question order, cont’d
 Questions should be grouped logically and into related
sections.
 Within a group of questions, general questions should
precede specific ones.
 Opinion and attitude questions should precede
behaviour questions.
 Even if a respondent provides an answer to a question
before it is asked, the question should be repeated at the
appropriate time.

i.e., there could be question order issues.
10/28/2013 12:15
Geog 3250
46
Designing the Questionnaire
 Provide a clear layout and
presentation
 Choose vertical or horizontal
answers, as appropriate
 Vertical tends to be more
pleasing to the eye
Vertical layout
Horizontal layout
10/28/2013 12:15
Geog 3250
47
Designing the Questionnaire
 Provide clear instructions on how to respond.
 Is more than one response allowed?
 Should the choice of answer be circled? Underlined?
 Keep a question and its answer together on the same
page.
10/28/2013 12:15
Geog 3250
48
Vignette Questions
 Presenting people with one or more
scenarios and asking them how they
would respond
 Anchor the choices in a realistic situation.
 Creates distance between question and
respondent.
 Hopefully more candid response.
 Weakness: how people say they would act
in a particular situation may be very
different from how they would really act.
 Can provide useful information, or at least
Michelle and Philip bought their home
5 years ago and 2 years ago wind
turbines were installed
nearby….Please provide your thoughts
about their situation in the following 5
questions.
a starting point for further research.
10/28/2013 12:15
Geog 3250
49
Pilot Studies
 Used to test whether individual items or the
instrument as a whole operate well.
 Used with open questions to generate closed questions
for subsequent studies.
 Provide interviewers with experience in administering
the instrument.
 Can be used to ensure that there is variability in the
answers given.
10/28/2013 12:15
Geog 3250
50
Pilot Studies
 Can identify questions that are embarrassing,
uninteresting, etc.
 Can identify questions that are difficult to understand.
 Can be used to determine the adequacy of the
instructions.
 Can be used to determine whether the instrument has
satisfactory flow.
10/28/2013 12:15
Geog 3250
51
Using Existing Questions
 If certain questions have worked well for other
researchers, they may be appropriate for your study.
 The original source must be cited.
 Allows results from different studies to be compared.
 Questions may be modified to suit one’s purposes
better.
10/28/2013 12:15
Geog 3250
52