Surveys, questions and other cheap
ways of getting data
How to do stuff
1/14/02
Michael Ramscar
Plan
•
•
•
•
•
•
•
Surveys
What is correlational research?
Sampling a population
Ways of collecting survey data
How to formulate “Good” questions
Scales
Soothing strategies
Correlational Research
• The nature of correlation
– A numerical relationship between two variables
– Predictions:
• Will score A covary with score B?
• Can be either negative or positive
• Presence of correlation allows prediction
– Prediction can be tested on samples
Correlational Research
• Does drinking beer make you appear cooler, funnier
and sexier?
– How do you test this as a hypothesis?
Correlational Research
• 2 ways:
– Experimentally
• Randomly assign people to groups
• Get one to drink no beer; other drink 3 beers/day
• Measure the change in their coolness in relation to pre-test
– Conduct a survey
• Ask people to estimate how much beer they drink
• Ask people to rate them for coolness, sexiness and funniness
• Find the relationship between these two variables
Surveys: Pros and Cons
• Advantages of survey research
– Can sample a large number of individuals
– Do not have to worry that the situation is artificial
– Surveys are often easy to administer
Surveys: Pros and Cons
• Disadvantages of survey research
– Danger of biased surveys
• Sample may not be representative
• Questions may bias responses
– People may not answer truthfully/competently
– Asking questions may not be the right methodology
– Cannot make statements about causality
Correlation and Causality
• Correlation
– A numerical relationship between two variables
– One value need not cause the other
• The third variable problem - “spurious relationships”
– Does drinking beer make you cool?
• Perhaps uncool people are just too intimidated to go into bars
• A Mantra: “Correlation does not equal causation!”
Sampling
The hard part of conducting a survey is getting a representative sample
Population
Sampling Frame
Sample
Potential sampling problems
• Flawed surveys often reflect poor samples
– Sampling bias can enter in two places
• At selection of sampling frame
– Use of phone directories
– List of subscribers to a magazine
• At selection of a sample
– Use of an email survey
– Emails made to students on Friday and Saturday nights
Potential sampling problems
• Suppose you were interested in student attitudes to
beer on campus
– Sampling frame might be the student email directory
– You could randomly sample names from this
• What is good and bad about this frame?
Types of Samples
• Nonprobability samples
– No guarantee that elements of pop can be
included
– Will (almost never) be reflective of the population
• May be helpful for studying ideal cases
• May be helpful for studying specialized populations
– Accidental samples
• Whoever happens to be around...
– Surveys in the street
– Newspaper call-in polls
• Problems with accidental samples
– Built in bias
– Generalizability?
More non-probability samples
• Purposive sampling
– Studying a particular population
– May seek out special individuals
• Survey of types of employee
• May provide insight into what makes good vs. bad
• Generally assume that non-probability samples are
biased
– True for many mail-in surveys
Subject Selection Effects
• Any study in which subjects self-select themselves
for participation are potentially problematic
– Some types of selection are worse than others
•
•
•
•
Those willing to participate for money
Those interested in a particular topic
Those with a particular viewpoint
Those with a particular skill
• Problem with passive net surveys?
What to do?
• Probability samples
– In a probability sample, there is a better chance that the
sample will reflect the underlying population
• Types of probability samples
• Simple random sampling
– Select N individuals from sampling frame
• Stratified random sampling
– Useful if there are key subgroups in data
– Suppose your students were half male and half female
• Divide sampling frame into males and females
– Select N/2 males and N/2 females
• Ensures that this aspect of the sample matches the population
Survey research designs
Cross-sectional designs
•Try to get a view of a population at one time
Example
• Suppose we were interested in the number of lefthanders in the population
– We could conduct a survey
– What if we found that left-handers make up about 10%
of the population up to age 60, and then there is a
continuous decrease in percentage
• Might conclude that left-handers die earlier
• Other factors?
Successive Independent Samples
• Can be used to assess changes in attitude over time
• Must be sure that samples are comparable
• Want results to be due to changes in time, not changes in
population
Longitudinal Studies
• Follow a single group of individuals over time
– Requires a lot of time and effort
– Subject mortality a problem
• Drop out -- sometimes literal mortality
• Most often people drop out for some reason
• Must ensure that there is not a systematic reason for the
mortality
– Left handers again?
Summary So Far...
• Research is only as good as the sample you draw
• Must ensure the most representative sample
possible
• Survey and correlational research cannot determine
causal relationships
• Next - how do we go about it?
Conducting Surveys
• There are many ways to conduct surveys
–
–
–
–
Through the mail
In person via an interview
Over the phone
Via the internet
• some of all of the above can apply to this
• Each of these methods has advantages and
disadvantages
Mail Surveys
• Mail surveys are used frequently
– Minimal effort needed to collect data
– Good for personal / embarrassing topics
• Respondent’s replies are anonymous
• May cut down on the amount of socially desirable responding
• Problems with mail surveys
– Respondents cannot ask for clarification
• A big problem is questions are often poorly worded
– No follow-up questions can be asked
– No control over the order in which a survey is completed
– Big potential for response bias
Self-selection in mail surveys
• Respondents in mail surveys always self-select
– Respondent must decide to do survey and mail it
• Controlling response bias in mail surveys
– Try to ensure a high return rate
• Over 50% is pretty good
• Response bias is less severe with return rates over 50%
– Make survey attractive
• Include a pre-paid reply envelope
• Personal introductions
• Raffle a prize
– But…
Personal Interviews
• Survey can be conducted in person
– Advantages
•
•
•
•
High response rate
Can control order of responding
Respondent can ask for clarification
Interviewer can ask follow-up questions
Personal Interviews
• Survey can be conducted in person
– Disadvantages
• There is a potential for interviewer bias
– May suggest the desired response (nonverbally)
• People may give socially desirable responses
– They may not want to express their true beliefs to the interviewer
• Difficult to ask about embarrassing topics
– No anonymity for the respondent
Telephone Surveys
• Doing surveys over the phone has some advantages
–
–
–
–
More anonymity than for personal interviews
Can control order of responding
Respondent can ask for clarification
Interviewer can ask follow-up questions
Disadvantages of telephone surveys
• People may not want to be bothered over the phone
– Particularly now that telemarketers often use surveys as
a sales pitch (think also email spam)
• No control over surroundings
– Respondent may be doing a number of things at once
• Still the potential for interviewer bias
• People may provide socially acceptable responses
– They may not want to express their true beliefs to the
interviewer
Constructing Surveys
• Kinds of questions
• Open-ended
– Range of allowable responses not determined in advance
– Respondents decide how much information to give
– Open-ended question can be difficult to score
Constructing Surveys
• Kinds of questions
• Closed questions
–
–
–
–
–
A fixed set of responses is provided
Respondent must pick one
Easy to score
May miss responses that respondent wants to give
Limited in the scope of the answer that can be given
More on questions
• Mixed questions
– A fixed set of responses is provided
– An open-ended choice is given to cover any alternatives
that may have been missed
– Still has a limited scope
• What meanings can “bank” have:
–
–
–
–
–
1. A place to keep money
2. The edge of a river
3. A large snail
4. To lean an aeroplane into a turn … etc...
7 Other. _________________
Response Scales
• For closed-ended questions, a list of responses is
needed
– For some questions, the list is a set of options
• What meanings can “bank” have:
– 1. A place to keep money
– 2. The edge of a river
– 3. A large snail … etc.
– Many questions ask for a degree of preference
• Yes/No questions
– Easy response
– No information about strength of preference
Response Scales
• Likert Scales
– Labeled points on scale
• Strongly agree, Agree, Disagree, Strongly disagree
• Variations on these scales
– Scale from 1-5 with ends labeled
• How many values should be put on a scale?
– People are reasonably good with 7±2 options
– With more options, people sub-divide the space
• Ratings are not necessarily more accurate on a 1-100 scale than
they are on a 1-9 scale
– False feeling of accuracy
How many points?
– How many points
• The number of points you use depends on how
much your subjects will be able to discriminate,
and the size of the effect you want to measure.
• If the effect you're trying to measure is very
small, then you'll need more points on the scale
• If you don't think people can meaningfully
discriminate more than 3 levels of a variable, then
you should only have 3 points - having more
points would just add noise to your data.
Even or odd number of points?
– This makes a difference when you're measuring
an effect in the middle of the scale
– An even number of points is good when you
want to force people to make a decision that
commits them to one side of a scale or another
– An odd number of points is good when you're
interested in when people will take a middle of
the road stance
• When will they say “don’t know”
When might you not use a 7point scale?
• 1. Measuring a small effect in the middle of
the scale:
How good at maths are women generally?
1------2------3------4------5------6------7------8------9------10
not at all.............................................................................…extremely
When might you not use a 7point scale?
• 2. Getting more info out of your subjects
than they think they know
Is this picture old or new?
1------2------3------4
I'm sure it's old.........................I'm sure it's new
When might you not use a 7point scale?
• 3. Finding out that people aren't sure
Is this picture old or new?
1------2------3
I'm sure it's old.................I'm sure it's new
When might you not use a 7point scale?
• 4. Another example of forcing a binary
decision
Please rate whether each item below is
masculine or feminine:
violin ..........M or F
hammer.......M or F
chair............M or F
salt..............M or F
Writing Good Questions
• A survey (or test) lives and dies with the questions
– Unclear questions can confuse respondents
– Biased questions can skew results
• Vocabulary in questions should be clear and simple
– Never use two syllables where one will do
Writing Good Questions
• Questions should be clear and specific
– Is something wrong with these sentences?
– How do you feel about the grammar of these sentences?
– Please indicate if one of the following sentences is
grammatical or not, and indicate why.
More on questions
• Edit questions for readability
• Use short questions
– Some words are ambiguous meaning that
people attribute multiple meanings to them.
Given this, do you think or not that more than
one meaning could be given to the following
words?
More on questions
• Do not assume that participants share your
enthusiasm
• Get to the point
• Did I mention use short questions
– Help us understand memory - do this….
• Q-day!
Asking questions
• Participants may not always help you even
when they want to
– Show me how you stop your children from
doing naughty things
Asking questions
• Participants may not always help you even
when they want to
• How to get round this?
Asking questions - stratagems
• Participants may not always help you even
when they want to
– I bet you couldn’t stop little Jimmy from doing
something he really wanted to do…
Asking questions - stratagems
• Participants may not always help you even
when they want to
• Children especially may be too eager to
please / to nervous to perform
Asking questions - stratagems
• Participants may not always help you even
when they want to
• Children especially may be too eager to
please / to nervous to perform
• Deflect the task
– Could you help this really dumb cookie
monster to…
Asking questions - stratagems
• Participants may not always help you even
when they want to
• Do fat people eat more when they are
stressed?
Asking questions - stratagems
• Participants may not always help you even
when they want to
• What happens when people have folk
psychological theories?
– Or when theories are famous?
Things to avoid
• Leading questions
– Questions that suggest the right answer
• People often recognise ambiguities -- or multiple meanings -in sentences. Re-write the following sentences in a way that
uses different words, but preserves their meanings?
Things to avoid
• Loaded questions
– Questions that are emotionally charged
• Competent readers can often spot ambiguities when reading.
Does this sentence have more than one possible reading?
Things to avoid
• Double-barreled questions
– Packing too much into one question
• Say which aspects of the the following sentence are ambiguous
and why you think they are.
Other things
• Include conditional information before the key idea
in the question alter responses?
– If you were reading the following sentence, would you
consider alternative meanings for it?
– Would you consider alternative meanings if you were
reading the following sentence?
Other things
• Consider varying the polarity of the questions
– Some questions should be phrased negatively
– Other questions should be phrased positively
– Controls for a ‘positivity’ or ‘negativity’ bias
• Some people just like to say No (or Yes).