SURVEY RESEARCH
Topic #1
[Covered in considerable detail in Weisberg et al. textbook]
Survey Research
• Quantitative study of large human populations
using these basic research operations:
– define the population to be studied;
– select a representative sample;
– measure (by means of interview questions)
characteristics of the respondents in the sample;
– tabulate, crosstabulate, and otherwise analyze the
sample data (using SPSS, etc.) to produce various
sample statistics; and
– make inferences from sample statistics to population
parameters of interest.
Types of Surveys
• Fake/spurious/recreational polls and surveys:
– straw/call-in/write-in or web based polls
– propaganda polls (asking “biased” questions)
sponsored by advocacy or interest groups
– political persuasion disguised as polling (“push polls”)
– commercial selling disguised as polling
Types of Surveys (cont.)
• Legitimate commercial surveys
– Public (results released to the public)
• syndicated polls (starting in late 1930s)
– Gallup, Roper, Harris, etc.
• media polls (starting in the 1960s)
– pre-election tracking polls
– exit polls
– Proprietary (contract research; results not publicly
released)
• market research (starting in 1920s)
• political consulting/polling for candidates, parties, and
advocacy and interest groups (starting in the 1950s)
Types of Surveys (cont.)
• Non-commercial surveys
– Government (“statistical”) survey research
• U.S. Census
• Current Population Survey (CPS)
– Rotating panel survey of ~60,000 households
– Source of labor force size and unemployment data
– Voter registration and turnout (but not vote choices and
opinions)
• Annual Survey of Social Security and Medicare
Beneficiaries
• National Crime Victimization Survey
Types of Surveys (cont.)
• Non-commercial surveys (cont.)
– Academic/scholarly/scientific surveys
• Voting studies (surveys vs. election data)
– “Columbia” (sociological) studies (1940s)
» local samples
» extended panel surveys
» emphasizing groups affiliations
» Index of Political Disposition (IPP)
Types of Surveys (cont.)
• Voting studies (cont.)
– “Michigan” (political/social psychological) studies
»
»
»
»
National samples
Limited (pre- and post-election) panels
Occasional extended panels (four years)
Emphasize opinions, attitudes, and orientation over
group affiliations
» Party Identification (Student Survey Q1 & Q2)
– American National Election Studies (ANES)
– Social Science Data Archives 
» Interuniversity Consortium for Social and Political
Research (ICPSR) at University of Michigan
» European Consortium for Political Research at
University of Essex (UK)
IBM Punch Card
Other Major (Ongoing) Surveys
– General Social Survey (GSS), NORC (University of
Chicago)
– Panel Study of Income Dynamics (PSID), University
of Michigan SRC
– Citizen Participation surveys (Sidney Verba and
others)
– Long-Term Political Socialization Panel Study
• Parent-child pairs interviewed in 1965, 1973, 1982, and 1997
– British, Canadian, German, etc., Election Studies
What Do (Political) Surveys (Purport to) Measure?
• voting intentions (and likelihood of voting)
• attitudes/opinions/preferences
– specific issues
– general orientations
• party ID, ideological placement, political trust/cynicism
– beliefs/perceptions/predictions/salience
• information/knowledge
• reported behavior
–
–
–
–
did you vote? for whom? contribute $? attend meeting?
job-seeking activity
TV watching
crime victimization
• demographics
Preview of Data Collection Methods
• Self-administered [“paper and pencil” or web-based]
questionnaires
– Student Survey
– SCEQ
– Exit polls
• Telephone interviews
– Tracking polls
• Personal interviews
– ANES
– CPS, GSS, PSID, etc.
• Closed-Form vs. Open-Ended Questions
– Multiple-choice test vs. “blue book” test
– SCEQ “scan sheets” vs. “blue sheets”
Closed-Form Questions (cont.)
•
Multiple-choice format: may be
– self-administered,
– telephone, or
– personal (ANES)
Some people don't pay much attention to political campaigns. How about
you? Would you say that you have been VERY MUCH interested,
SOMEWHAT interested or NOT MUCH interested in the political
campaigns so far this year?
1. Very much interested
3. Somewhat interested
5. Not much interested
8. Don't know
9. Refused
0. NA
Closed-Form Questions (cont.)
Agree / Disagree Format (ANES)
“This country would have many fewer problems if there were more emphasis
on traditional family ties.”
Do you AGREE STRONGLY, AGREE SOMEWHAT, NEITHER AGREE NOR
DISAGREE, DISAGREE SOMEWHAT, or DISAGREE STRONGLY with this
statement?
1. Agree strongly
2. Agree somewhat
3. Neither agree nor disagree
4. Disagree somewhat
5. Disagree strongly
8. Don't know
9. Refused
0. NA
Closed-form Questions (cont.)
Rating Scale Format (ANES)
Some people think the government should provide fewer services even in areas
such as health and education in order to reduce spending. Suppose these
people are at one end of a scale, at point 1. Other people feel it is important
for the government to provide many more services even if it means an
increase in spending. Suppose these people are at the other end, at point 7.
And, of course, some other people have opinions somewhere in between, at
points 2, 3, 4, 5 or 6.
Where would you place YOURSELF on this scale, or haven't you thought much
about this?
01. Govt should provide many fewer services
02.
03.
04.
05.
06.
07. Govt should provide many more services
80. Haven't thought much about this
88. Don't know
89. Refused
00. NA
Closed-form Questions (cont.)
Rating Scale Format (cont.)
Where would you place GEORGE W. BUSH on this issue?
DO NOT PROBE DK
1. Government should provide many fewer services
2.
3.
4.
5.
6.
7. Government should provide many more services
8. Don't know
9. Refused
0. NA
Note: Randomize order of administration for Pres candidates
Thermometer Scale
Please look at page 2 of the booklet. [See
Weisberg et al., p. 80]
I'd like to get your feelings toward some of
our political leaders and other people
who are in the news these days.
I'll read the name of a person and I'd like
you to rate that person using something
we call the feeling thermometer.
Ratings between 50 degrees and 100
degrees mean that you feel favorable
and warm toward the person. Ratings
between 0 degrees and 50 degrees
mean that you don't feel favorable
toward the person and that you don't
care too much for that person. You
would rate the person at the 50 degree
mark if you don't feel particularly warm
or cold toward the person.
If we come to a person whose name you
don't recognize, you don't need to rate
that person. Just tell me and we'll move
on to the next one.
Closed-form Questions (cont.)
Indices (composite measures)
Trust in government (ANES)
Index of respondent's trust in government, built from responses to
the following questions: (1) How much of the time do you think
you can trust the government in Washington to do what is right?
(2) Do you think that people in government waste a lot of the
money we pay in taxes, waste some of it, or don't waste very
much of it? (3) Would you say the government is pretty much run
by a few big interests looking out for themselves or that it is run
for the benefit of all the people? (4) Do you think that quite a few
of the people running the government are crooked or that not
very many of them are? Those who rank low on this index were
more likely to select the distrusting position for each of these
items.
1 [Low]
2
3
4
5 [High]
9 [NA]
Experimenting with Closed-Form Questions
(2004 ANES)
IF R SELECTED FOR STANDARD VERSION: In talking to people about
elections, we often find that a lot of people were not able to vote because
they weren't registered, they were sick, or they just didn't have time. How
about you--did you vote in the elections this November?
1. Yes, voted
5. No, didn't vote
8. Don't know
9. Refused
0. NA
IF R SELECTED FOR EXPERIMENTAL VERSION: In talking to people about
elections, we often find that a lot of people were not able to vote because
they weren't registered, they were sick, or they just didn't have time. Which
of the following statements best describes you:
1. I did not vote (in the election this November).
2. I thought about voting this time, but didn't.
3. I usually vote, but didn't this time.
4. I am sure I voted
8. Don't know
9. Refused
0. NA
Closed-Form Questions (cont.)
• Closed-form questions produce pre-coded data.
– This is a major advantage,
– in same way that multiple-choice tests [vs. essay exams] can be
quickly and reliably scored.
– Researchers can move quickly to tabulation and analysis.
• Student Survey (and SCEQ) answer sheets can be sent
directly for scanning, producing an electronic data file.
– See next slide 
• ANES (and other personal) interview sheets must be
transcribed and then entered into electronic format (e.g.,
Excel spreadsheet or SPSS Data Editor screen) to
create an electronic file.
For example, 2009 Student Survey Data
Open-Ended Questions
ANES: Party Likes/Dislikes
I'd like to ask you what you think are the good and bad points about the two
NATIONAL PARTIES. [Note: Randomize order of administration for
parties]
Is there anything in particular that you LIKE about the Democratic party?
IF R SAYS THERE IS SOMETHING R LIKES ABOUT DEMOCRATIC PARTY:
(What is that?)
PROBE: ANYTHING ELSE YOU LIKE ABOUT THE DEMOCRATIC PARTY?
UNTIL R SAYS NO.
Is there anything in particular that you DON'T LIKE about the Democratic
party?
Etc.
Data:
Number of responses
Balance of positive vs. negative responses
Content of responses
Data must be coded for quantitative analysis
Is there anything in particular that you LIKE/DISLIKE about the
Democratic/Republican party?
0101 Traditional Democratic voter: always been a Democrat; just a Democrat; never been a
Republican; just couldn't vote Republican
0102 Traditional Republican voter: always been a Republican; just a Republican; never been a
Democrat; just couldn't vote Democratic
0111 Positive, personal, affective terms applied to party--good/nice people; patriotic; etc.
0112 Negative, personal, affective terms applied to party--bad/lazy people; lack of patriotism; etc.
0121 Can trust them; they keep their promises; you know where they stand
0122 Can't trust them; they break their promises; you don't know where they stand
0131 Party is well-organized, sticks together, is united; members are disciplined; votes party line
0132 Party is poorly-organized/really two parties/divided/factionalized; members not disciplined;
doesn't vote party line
0133 Party is (more) representative/good cross-section of the country; encompasses a wider variety of
views/people; is more at the center of the country's views
ETC
From SETUPS Codebook: Handout #1B in
Course Pack or from Course Website
V30
MOST IMPORTANT NATIONAL PROBLEM “What do you
think is the single most important problem facing this country?”
(Responses have been categorized into the five groups listed
below.)
1
2
3
4
5
9
Economy
Foreign affairs
Social welfare
Crime, public order
Other
NA
• In the actual ANES survey, this is an open-ended question with
hundreds of code categories.
• Data from such open-ended questions usually is not available until
long after data from closed form questions is available.
• Problem with comparable closed-form questions 
2004 EXIT POLL
Similar closed-form
questions can have
serious problems.
Was 2004 election “all
about values,” as the
media claimed?
Exit Poll
Question E
Taxes
Education
Iraq
Terrorism
Economy/Jobs
Moral values
Health care
Omit
5%
4%
15%
19%
20%
22%
8%
7%
2000 EXIT POLL
Question D
World Affairs
Med/Rx
Health Care
Econ/Jobs
Taxes
Education
Soc Security
Omit
12%
7%
8%
18%
14%
14%
14%
13%
Question Design
• See Weisberg et al., Chapter 4
• Question wording can be blatantly biased [“pushing” respondent to a
preferred responses, as in propaganda polls]
• Do you favor murdering innocent babies in the womb?
• Should government officials have the power to interfere with
women’s personal decisions?
Question Design
• Even without manifest bias, phrasing can hugely affect
the distribution of responses, especially when many
respondents are ambivalent about the issue.
– Do you believe that the U.S. should withdraw from South Vietnam?
– Do you believe that the U.S. should withdraw from South Vietnam and
allow the Vietnamese to settle things among themselves?
– Do you believe that the U.S. should withdraw from South Vietnam, even
if this means that the Viet Cong and North Vietnamese [or “the
Communists”] take over?
– If the full House votes to impeach Bill Clinton, then do you think it would
be better for the county if Bill Clinton resigned from office, or not?
• Better to resign: 60%
– If the full House votes to send impeachment articles to the Senate for a
trial, then do you think it would be better for the county if Bill Clinton
resigned from office, or not?
• Better to resign: 43%
Question Design (cont.)
Unbalanced vs. balanced format
Should ILLEGAL immigrants be prosecuted and deported for being in the U.S.
illegally, or shouldn't they?
Should be deported
Should not be deported
Unsure
69%
24%
7%
If you had to choose, what do you think should happen to most illegal
immigrants who have lived and worked in the United States for at least two
years? They should be given a chance to keep their jobs and eventually
apply for legal status. OR, They should be deported back to their native
country.
Chance to keep jobs etc.
Deported
Unsure
62%
33%
5%
CBS News/New York Times Poll. May 18-23, 2007
Question Design (cont.)
• Unbalanced vs. balanced format
– ANES through 1960:
“If cities and towns around the country need help to build more
schools, the government in Washington ought to give them
the money they need. Do you have an opinion on this or
not?”
– ANES from 1964 onwards:
“Some people think that the government in Washington should
help towns and cities provide education for grade and high
school children; others think that this should be handled by
states and local communities. Have you been interested
enough in this to favor one side over the other?”
– Also recall “government services” question.
Question Design (cont.)
• Ambiguity (see Weisberg, p. 85)
– “How many public officials do you think are a little bit
dishonest --- most, some, a few, or none?” [ANES,
earlier version]
• Many respondents answered “few or none,” but evidently
because they though some or most public officials were a
whole lot dishonest.
– “Do you think that that quite a few people running the
government are crooked, not very many are, or do
you think hardly any of them are?” [ANES, current
version]
Question Design (cont.)
Might the Holocaust Have Never Happened?
• Asked in a “wrong” [confusing] way (November 1992)
Question: "Does it seem possible or does it seem impossible to you
that the Nazi extermination of the Jews never happened?"
Possible it never happened
22%
Impossible it never happened
65%
DK
12%
• Asked in a “right” [less confusing way] way (March 1994)
Question: "Does it seem possible to you that the Nazi extermination of
the Jews never happened, or do you feel certain that it happened?"
Possible it never happened
1%
Certain it happened
91%
DK
8%
Source: Surveys by the Roper Organization, reported in Public Perspective
Question Design (cont.)
• Closed-form option order (cf. ballot order)
– Which candidate did you prefer for the 2004 Democratic presidential
nomination?
(1)
Howard Dean
(2)
John Edwards
(3)
Richard Gephardt
(4)
John Kerry
(5)
Other candidate; don’t know; no preference
– Which candidate did you prefer for the 2004 Democratic presidential
nomination?
(1)
John Kerry
(2)
Richard Gephardt
(3)
John Edwards
(4)
Howard Dean
(5)
Other candidate; don’t know; no preference
Question Design (cont.)
• Response bias
• Political Efficacy Index
Index of respondent's sense of “political efficacy,” built from
responses to the following questions:
(1) Do you agree or disagree with the statement, "People like me
don't have any say about what the government does"?
(2) Do you agree or disagree with the statement, "I don't think
public officials care much what people like me think"?
(3) Do you agree or disagree with the statement, "Sometimes
politics and government seem so complicated that a person
like me can't really understand what's going on"?
– Do all the people who “agree” with all these statements
really have a low sense of political efficacy, or are
some of them just “agreeable” (i.e., tend to say they
agree with any statement read to them)?
Questionnaire Design
• These considerations point to the importance of pretesting, survey experiments, and pilot studies:
– Advantage of using standard questions (e.g., POLI Student
Surveys)
– How long will survey take?
• Question order can affect the distribution of responses to
questions (issue preference and Presidential approval in
1984).
• Question context (i.e., the immediately preceding
questions) can affect distribution of responses (e.g., Q4
in PS #1B).
• Early questions should reassure the respondent as to
the legitimate and non-threatening nature of the
questionnaire.
• “Controversial” questions should come last.
– What do you think is the most “controversial” standard question?
• Length of questionnaire.
Survey Data Collection Methods
• Weisberg et.al., Chapter 5
• Self-administered questionnaires (written [or webbased])
• Telephone Surveys (entirely oral)
• Personal interviewing (face-to-face and almost entirely
oral)
Self-Administered Questionnaires
– Varied settings
• controlled (student survey, SCEQ, exit polls)
• distributed and returned later
• Mailed, completed (by whom?), and returned by
mail
– Characteristics
•
•
•
•
•
cheap
low response/completion rate (if returned later)
probably closed form questions only
short questionnaire only
literacy problems
Telephone Surveys
• Increasingly common (especially for pre-election tracking
polls; even academic surveys now use this method more
than in the past)
• Characteristics
– fast and cheap
– computerized sampling (random digit dialing)
– computer-assisted interviewing
– interviews are monitored by supervisors
– limited to simple questions and short questionnaires
– Problems:
• call-back procedures (if no answer)
• diffusion of telephones
• cell phones
Personal Interviewing
• The “gold standard” in scientific surveys, used by
– ANES
– General Social Survey
– Current Population Survey
– and most on-going high quality surveys
• Setting
– Preliminary letter
– Usually in respondent’s home at convenient time
Personal Interviewing (cont.)
• Characteristics
– Expensive
•
•
•
•
Labor intensive
Interviews are unmonitored
Need semi-professional interviewers (almost always women)
Interviewers must be geographically distributed (for national
surveys)
– Flexible
•
•
•
•
Allows more complicated questions
Allows branching questions
Allows open-ended questions
Allows (relatively) long questionnaire
Data Tabulation and Analysis (Preview)
• Coding of responses to open-ended questions
• Data entry and verification
• Cross-sectional analysis
–
–
–
–
Tabulation/frequency distributions
Subgroup differences
Other relationships (crosstabulations)
Causal relationships (control variables)
• Longitudinal analysis
– Cumulated surveys (ANES, Student Survey)
– Panel surveys (some in ANES)
• Primary vs. secondary analysis of survey data
Sampling Methods
• A sample is a [usually small] subset of some larger
population.
• A representative sample is a cross-section of (has the
same distribution of characteristics as) the larger
population from which it is drawn.
• But how can you know that a sample is representative?
After all, you usually don’t know the characteristics of the
larger population. (If you know all about the larger
population, you wouldn’t need to conduct a survey).
– You can make some demographic and other checks,
– but mainly, you need to use proper sampling methodology,
namely, random or probability sampling.
Some “Bad” Sampling Methods That Don’t
Produce Representative Samples
• Self-selected/voluntary samples:
– call-in polls
• American Idol
• after TV debates, etc.
– write-in samples
• Ann Landers once asked her readers, “If you had to do it over again, would
you have children?” She received nearly 10,000 responses, almost 70%
saying “NO!”
• Subsequently, a “scientific” survey (random sample of size = 1373) asked
the same question and 91% responded “Yes.”
“Bad” Sampling Methods (cont.)
• Haphazard/convenience sampling
– POLI 300 Student Survey
– Literary Digest Presidential election polls, 1920-1936
• Interviewer selected samples (go to main street,
shopping mall, bus station, etc.)
– Selection bias
• availability bias
• interviewer bias
– Quota sampling
The “Good” Sampling Method
• Random or Probability Sampling
– leave it to chance who appears in the sample
• Selection Bias:
– Number of children vs. number of siblings
• Over the past thirty years the average American women has about 2.0
children.
• But mothers of POLI 300 in an earlier students had about 2.85 children.
• “Class Size Paradox” See 
– Statistician Abraham Wald & RAF Bomber Command
• Where should extra armor be placed on bombers?
• On to the Handout #2
The “Class Size Paradox”