Designing and Using a Survey
February 7, 2011
Objectives
By the end of this meeting, participants should
be able to:
Explain how surveys accomplish the goals of:
measuring attitudes, measuring change over
time, making group comparisons, and analyzing
the causes of behavior.
Goals of Surveys
• Measure the frequency of attitudes, beliefs
and behaviors
• Measure change over time
• Examine differences between groups (race,
class, gender, etc.)
• Analyze causes of attitudes, beliefs, and
behaviors
Measuring Behavior
• Time can be awkward for respondents
• Forward telescoping
• Backward telescoping
• Sensitive subjects
• Bogus pipeline technique (ethical concerns?)
• Random lists
• Randomized response technique (some sensitive, some
not)
• Importance of others’ beliefs
Predicting Behavior
• Election Predictions
•
•
•
•
•
Problems with third parties
Problems with really close elections
Problem with social desirability for turnout
Undecided
Sensitive topics
2008 Presidential Example- NC
Final Results
Obama 2,142,651
McCain 2,128,474
Others 39,726
Total
4,310,851
49.70%
49.38%
0.92%
Source: Office of the Clerk of the US House of Representatives
How did the individual polls do?
FOX News/Rasmussen McCain MofE
Reuters/Zogby
McCain MofE
American Research Group Obama MofE
SurveyUSA
McCain MofE
Mason-Dixon
McCain MofE
Research 2000
Obama MofE
Politico/InsiderAdvantage Tie
MofE
Recall: Margin of Error
• So for example, FOX/Rasmussen’s final poll for North Carolina
showed 50% for McCain and 49% for Obama
• Sample: 1000 likely voters
• Margin of error = ±t {p(1−p)/(n−1)} 1/2 {1 − f}1/2
• But since the population being sampled is large we can ignore
the final {1 − f}1/2
• ±1.96 {50(50)/(1000-1 )} 1/2
• = ±3.1%
• In other words, Rasmussen/FOX was 95% confident that the final
vote for McCain would be between 46.9% and 53.1%. Right on
the mark, pretty much.
• Yet, if they had to make a prediction of the winner, they’d guess
wrong. (Some polls had MO wrong, too.)
Measuring Attitudes
• One of the most common goals of surveys
• Need to be concerned about measuring non-attitudes
(Converse)
• Need to consider attitude strength, respondents may not care
equally about all issues (environment, gun control, etc.)
•
•
•
•
•
How do they feel about the issue personally?
How knowledgeable are they about this issue?
How certain are they of their opinion?
How much thought they have given to the issue?
Should you offer counterarguments?
Measuring attitudes
d) Attitudes especially weakly held attitudes will change
over time (first impression versus considered
opinions)
e) Need to watch disagreements over the meaning of
words
• Example: ideology and Stimson’s work
f) Need to watch for changing frames
• Verbal (example: welfare, next page)
• Contextual (example: sexual harassment)
Example of Verbal Differences- GSS
Support for Welfare Spending
Country Spends
24.1% (2384)
Too Little
Country Spends
34.6% (1999)
About Right
Country Spends
41.3% (2384)
Too Much
Total
100% (5771)
Support for Assistance to the Poor
Country Spends
66.9% (2762)
Too Little
Country Spends
24.4% (1008)
About Right
Country Spends
8.7% (357)
Too Much
Total
100% (4127)
Measuring Change over Time
• Measurement of personal attitude change is
frequently unreliable
• Comparisons of cross sectional surveys are
more common but pose their own issues
•
•
•
•
Similar interviewing methods?
Similar sample sizes?
Similar question wordings?
Does the change exceed sampling variation?
Measuring Change over Time
c) Frequently can be atheoretical
d)Can only measure gross change, not change at
the individual level
e)Insta-polls can allow for immediate reactions
to stimuli but suffers worries about
representativeness and long term impact
Measuring Change over Time
• Preferred method: panel studies
• Repeated studies of the same individuals over time
• Limitations
• Long term studies are prohibitively expensive
• Atrophy of the original sample (and worries about those
that remain)
• Errors in interviewing
Sub-group Comparisons
• Researchers frequently want to compare the
attitudes of various sub-groups of the
population
• One difficulty is the smaller and less accessible
groups may be harder to randomly sample
• Double sample or over sampling
• Pyramiding or combining multiple surveys
What Leads to Change in Attitudes over
Time?
• Surveys generally struggle with this type of
question
• Most people’s answers for why their opinions
changed are post-hoc rationalizations
• Individuals may not be sure of their reasons
for opinion change
• Similar stimulus may affect people differently
• For February 9: Work on research proposals,
you will have time for group work next meeting.
• For February 11: Download the program R from
http://www.r-project.org/ and bring your laptop
to class. You also may want to print-out “Getting
Our Feet Wet with R” from the website.
• For February 14:
– Research proposals due for each group.
– Read WKB Chapter 8
– On p. 189 answer either question 1, 2, or 3.
– Construct a causal model relevant for YOUR research
question. (Turn-in the C.8 items individually.)