Page |1
WORKSHEET: PUBLIC OPINION
A Guide to Sample Size and Margin of Error
There are about 209 million adults in America, of every
imaginable background and circumstance. So how can a
survey of only 800 or 1,000 adults reflect what the entire
country is thinking? How can a thousand voices speak for
us all?
Public opinion researchers liken it to making a big pot of
soup — to taste-test the soup, you don't have to eat the
whole pot, or even a whole bowl's worth. You only have
to try a bite. The same is true of public opinion. You don't
have to ask every single person in America to find out
what Americans think; you only need to ask a few to get
the flavor of public opinion.
This fact is reflected by a survey's margin of error, or
sampling error. When public opinion researchers report
the margin of error for their polls (usually expressed as
something like "plus or minus 3 percentage points") they
are stating their confidence in the data they have
collected. The lower the margin of error, the more
accurately the views of those surveyed matches those of
the entire population.
You must also remember that every margin of error has
a "confidence level," usually 95 percent. That means that
if you asked a question from this poll 100 times, 95 of
those times the results would be within 3 percentage
points of the original answer. Of course, this means that
the other five times you ask the question, you may get
answers that are completely off the wall.
For example, if 50 percent of a sample of 1,000 randomly
selected Americans said they favor recycling laws, in 95
cases out of 100, 50 percent of the entire population in
the U.S. would also have given the same response had they been asked, give or take 3 percentage points (i.e.,
the true proportion could be 47 percent or 53 percent).
The bigger the sample, the smaller the margin of error, but once you get past a certain point -- say, a sample
size of 800 or 1,000 — the improvement is very small. The results of a survey of 300 people will likely be
correct within 6 percentage points, while a survey of 1,000 will be correct within 3 percentage points, a lower
margin of error. But that is where the dramatic differences end — when a sample is increased to 2,000
respondents, the margin of error drops only slightly, to 2 percentage points.
Despite this, some surveys have sample sizes much larger than 1,000 people. Usually when a study has a large
sample, it is so certain subgroups — like parents or the elderly — can be teased out and compared to each
other or to the whole. If you want to compare retired people to the general public, for instance, a sample of
1,000 might yield only one or two hundred people who are no longer working, which may not be enough to
Page |2
get a solid grasp on the views of that group. A sample of 2,000, however, will probably yield a larger group of
retired Americans, and provide a more accurate picture of their views.
Sometimes increasing the sample size is not enough, if the subgroup you are examining is rare or particularly
hard to find. Young black men, for example, make up only a small percentage of the U.S. population. In a
standard random sample, you would have to interview an enormous number of people before you had a large
enough subgroup of young black men. In this instance, you would take an "oversample," purposely seeking
out members of the particular group you are interested in, and comparing the results to the main sample.
Of course, in both general samples and oversamples, who is asked is as important as how many are asked.
Reputable survey organizations go to great lengths to make sure their interview sample is random and
representative of whomever they are surveying, be it retired people, young black men, or all Americans.
Substance (or Lack Thereof) of U.S. Public Opinion –
What does America think? Not much.
Do Americans Know Anything? Percentage Unable to Identify the…
Party with Senate majority
Name of Vice President of US
Name of Speaker of the House
Both Senators from their State
Chief Justice of Supreme Court
38%
40%
47%
54%
97%
Where You Stand Depends upon Where You Sit: The Gender Gap
Issue
Women
Men
Poverty and homelessness should be govt priority
Favored child-care funding so welfare parents can work, attend school
Death penalty for murderers
Should criminal justice system be tougher on drunk drivers?
Want stricter gun control
Would not vote for a candidate who is gay
44%
91%
46%
76%
73%
43%
29%
48%
64%
48%
50%
30%
Measurement:Method is Everything
A. Random Sampling.
1. 1948 Election (“Dewey Defeats Truman”).
Problem: non-random sample that correlated w/ income
2. 1996 Election (Bob Smith of New Hampshire) Problem: naturally occurring statistical sampling error
3. 2000 Election (Gore, then Bush, then nobody in Florida)
Problem: early call based upon bad data
B. The Phrasing of the Question
Statement: The country isn’t spending enough $ on assistance to the poor.
Statement (same survey): The country isn’t spending enough $ on welfare.
Agreement: 63%
Agreement: 19%.
Example: Balanced Budget Amendment. With or without Social Security effects?
Example: Stricter pollution control to clean the nation’s rivers? Are job effects included in the question?
Page |3
Polls help public officials (and others interested in public opinion) have a broader and more
scientific understanding of what the public collectively wants. Good polls produce good
information. Bad polls do not. So how do we figure out if a poll is good or bad? To do so, we need
to look at question wording, sampling, and how respondents are contacted.
1) Question wording: The respondent needs to know how the questions are phrased. Bad questions
lead to bad results. There are thousands (or more) bad polls out there. An example of a bad
question might be,
“If the government takes our guns that we use to protect our
families away from us, only criminals will have guns and we will
all be in danger. Are you in favor of placing your family in
greater danger? Yes/No”
2) Sampling: In order for a poll to be reliable, the sample must be taken accurately. The best
method is a scientific random sample of at least (for any population over 100,000) 900
respondents. Such a sample guarantees that each person in the population has the same statistical
chance of being selected.
There are a number of sampling techniques. Some of the techniques are poor and should be
avoided. These include nonstratified sampling, straw polls, and most nonprobability sampling
methods. A more reliable nonprobability method is a quota sample in which a pollster ensures
representativeness using quotas. For example, in a citywide survey, respondents should reflect the
make-up of the city: 30 percent African American, 15 percent Hispanic, and so on.
Most national surveys use stratified sampling. A simple random sample of the American
population would not be a very good predictor of election results, since not everyone votes and
the survey could end up with a sample that excludes women, a minority group, region, or the
like. As we recall from our discussion of political socialization, these things matter.
3) Contacting respondents: The method of contact is important. Since 95 percent of Americans
have a telephone, random phone calling would be a valid method. However, this should not be
the method of choice in Sudan. The rising use of cell phones might cause problems, especially
among younger people, but so far, this has not been a major concern. Some surveys are done in
person, but many worry that the presence of the interviewer causes problems.
In general, you should never trust a poll that
a)does not tell you the question wording,
b) the sampling method, and
c) the ways in which respondents were contacted.
Reputable and reliable pollsters will also inform you of the number of respondents (the “n”) and
the error rate (+ or – 5 %, for example) so that you can determine for yourself whether to believe
the results.
Any poll that tells you to call 555-9712 for “yes” and 555-9713 for “no” is unscientific and
unreliable. The same is true of Internet polls that ask you to register your opinion now. These are
not random samples at all!
Page |4
SOME TERMS & CONCEPTS
Tracking polls: continuous surveys that enable a campaign to chart its daily rise and fall in popularity. These are
small samples and conducted every 24 hours. They are fraught with reliability problems, but may be a decent
measure of trends.
Push polls: try to lead the subject to a specified conclusion and the worst are designed simply to “push” subjects
away from candidates by linking them to negative events or traits in the question.
Exit polls: polls conducted at polling places on Election Day.
Shortcomings of Polling
Bad reporting and bad polling can change political campaigns, hurt careers, and have other bad consequences. There
are large numbers of bad polls out there! It is our job to learn how to consume polls critically so that we ignore the
bad polls and take “good” polls with an understanding of their shortcomings.
Sampling Error: the margin of error. The sampling error is quite small if the sample is carefully selected. All
polls contain some error, and 3 to 5 percent is considered a reasonably small rate of error. A 3 percent error rate
means that the poll is 97 percent accurate! These rates become extremely important if a race is close:
John Kerry
48%
George Bush 52%
Margin of Error: 5%
Do these numbers tell us anything? No. The contestants are only four points apart, and given the error rate, the real
race could look like this:
John Kerry
53% (48% plus 5)
George Bush 47% (52% minus 5)
Limited Respondent Options: Have you ever taken a survey (or a test) and did not like any of the answers? If
the options are not broad enough, you get bad results. This is a common shortcoming of many polls.
Lack of Information: If surveys ask questions about subjects that the respondents don’t understand or don’t
know about, the answers will often be invalid. The use of filter questions is helpful, such as, “have you thought
about...?”
Intensity: Polls do not measure intensity well. We can learn a position on an issue, but not how strong that opinion
might be.
WHY WE FORM AND EXPRESS POLITICAL OPINIONS
Personal Benefits: Conventional wisdom holds that Americans are more “meoriented”
today than ever before. People therefore tend to choose policies that will benefit them. For example, the elderly favor
Social Security. When policies don’t affect us personally, we often have difficulty forming an opinion. Foreign
policy is a prime example, since most Americans know little about the rest of the world. The public good seems to be
a waning commodity.
Political Knowledge: Americans are highly literate and over 82 percent graduate from high
school. We also have access to a wide range of higher education. However, we don’t know much
about politics! In 2002, a Department of Education report found that most high school seniors have a poor grasp of
history. Only 33 percent of people can identify their own representative to Congress
We are also generally geographically illiterate, with most Americans unable to locate the Persian
Gulf or Vietnam on a map. However, most of us have political opinions guided by issues, events,
people, ideology, or something else entirely.
Page |5
Cues from Leaders: Low levels of knowledge make public opinion highly changeable. Rapid opinion shifts are
common when the public does not have much information on an issue or
if the information is bad. Political leaders and the media can often have a large effect on public
opinion, since we are often uninformed and may not care to become knowledgeable about current
issues.
Political Ideology: When people espouse an ideology, even if they don’t fully understand it,
the ideology affects their opinions. Americans tend to assert that they are liberal, conservative, or
moderate. Sometimes these labels can be meaningful. American conservatives generally favor
smaller government and less regulation. American liberals generally believe the government can do a good job
providing for the poor or elderly. But this seems to play out differently today. A person’s conception of what it means
to be a Republican or Democrat can determine their answers to a poll, regardless of what a more thoughtful
consideration of the issue might lead them to believe.
How Polling and Public Opinion Affect Politicians, Politics, and Policy
Now come the most important questions of all: So what? Do polls affect the political
process? If so, how and to what effect? Are they benign ways of measuring the attitudes of a
democratic citizenry, or are they malignant attempts to control and manipulate the people? And if they are
important, that ultimate poll- voting- should matter a lot, no?
PREVIEW: The Paradox of Voting: Why Would Rational People Vote at All?
 Why Vote? The more people I expect to show, the less likely my vote will make a difference.
 “Marginal value” of vote is negligible. So if I care about eventual outcome, I have no incentive
to vote.
 Result: Can’t predict “positive turnout” based on rationality alone. And it happens...
 BUT,…. If I expect no one to show, then I am dictator.
Questions to Ponder:
1. In America, what is the difference between liberalism and conservatism? What is a
communitarian (or populism) and a libertarian? How do these compare on the freedom and
order scales?
2. On the political spectrum, how does the population fall? Is American truly split?
3. What issue stances fall under the respective political ideologies? What do liberals believe?
What do conservatives believe?
4. What is public opinion and how is it measured?
5. What is the role of public opinion in politics?
6. What are the questions one must ask when analyzing a public opinion poll?
7. What are the various demographic groups? How are these sorted into respective ideologies?
What accounts for these groups believing what they do? Why is it difficult to simply place
one group under one ideology?
8. What influences how one thinks about politics and government? Which factors are more
important than others?
Page |6
9. What are some general beliefs Americans have about government? What are some
predominant American values that shape our political beliefs?
10. How do people express themselves politically? What accounts for the different levels of
involvement?
11. What is political efficacy?
12. Why do activists or the political elite tend to be more ideologically consistent than those who
aren’t active? What effect does this ideological consistency have on the difference
ideologically between politicians and voters?
13. Are politicians shaped by public opinion or do they shape public opinion?
Test Your Knowledge!
MULTIPLE CHOICE QUESTIONS
1) The founder of modern-day polling is
a. Louis Harris.
b. George Gallup.
c. Steve Roper. d. Walter Lippman.
2) An unscientific survey used to gauge public opinion on issues and policies is
called a
a. deliberative poll. b. exit poll.
c. straw poll. d. public opinion poll.
3) The popular magazine which from 1920 to 1932 correctly predicted the outcome
of every presidential election based on unscientific surveys of public opinion was
a. The American Voter.
b. The Voice of the People.
c. Public Opinion and American Democracy.
d. Literary Digest.
4) Based on public opinion polling, women tend to be
a. more liberal about issues of social welfare concerns.
b. more negative about war and military intervention.
c. concerned about terrorism and national security at home.
d. All of the above.
5) What percentage of Americans today report that they consider religion an
important part of their lives?
a. 47
b. 76
c. 91
d. 99
6) The single largest predictor of the vote today, following party identification, is
a. age.
b. gender.
c. race.
d. religion.
7) People can be influenced in forming political opinions by ideology, party
affiliation and social grouping. They may also influenced by
a. personal benefits.
b. political knowledge.
c. cues from leaders and opinion-makers.
d. All of the above.
Page |7
8) The process by which individuals acquire their political opinions is most accurately known as:
A)
popular conditioning.
B)
social communication.
C)
socio-economic propaganda.
D)
political socialization.
9) Which of the following poll results are theoretically the most accurate, assuming that all have
a confidence level of .95:
A)
sample size = 2,000; population size - 250 million.
B)
sample size = 1,000; population size - 100,000.
C)
sample size = 1,500; population size - 35 million.
D)
sample size = 500; population size - 25,000.
10) In general, public opinion:
A)
determines government action.
B)
is unrelated to government action.
C)
sets broad limits on government action.
D)
affects government action only on issues that are prominent in elections.
11) Indicators of public opinion in the United States include all except which one of the
following:
A)
election results.
B)
size of crowds at demonstrations.
C)
number of newspapers sold.
D)
letters to editors of newspapers.
12) Discovering how Americans think politically can be important in which respects:
A)
it can provide clues about how public opinion is likely to affect government.
B)
a shared frame of reference can bring citizens together in the pursuit of goals.
C)
it can reveal how truly alienated opinion is from the actions of government
D)
a and b only.
E)
a, b and c.
13) Public opinion assumes more importance in democratic societies than in non-democratic ones
because of the:
A)
president's power as commander in chief.
B)
availability of polls.
C)
legal requirement for a national referendum on presidential war actions.
D)
belief that the source of government is the will of the people.
E)
need to mobilize the populace to action.
14)
Public Opinion is _______________.
A)
B)
C)
D)
E)
an opinion that you talk to your neighbor about daily: gossips, chitchat.
an opinion by you directly spoken to the President of United States.
an opinion consists of people's views concerning political issues.
an opinion consists of people's views concerning any issues.
an opinion which displays objective views regarding any political issues.
Page |8
15)
Typically, views that are expressed as political opinions are acquired through the process
of ________ _________. ; People acquire their political attitudes, such as their party
identification. This also refers to the process of acquiring political values and knowledge.
A.
B.
C.
D.
E.
Political Socialization
Political Trust
Political Analysis
Public Agenda
Divisive Opinion
16)
Political Socialization is all possible because of several agents. Which group does not
have anything to do with Political Socialization?
A.
Family: Children have a strong tendency to adopt the party identification of their parents
B.
Friends tend to reinforce our beliefs since we associate ourselves with like-minded people
as we get older
C.
Media: Impact varies depending on viewing and reading habits
D.
Both A and B are considered as agents, but not C.
E.
All of them are considered as agents for Political Socialization.
17)
Political culture is a set of values and beliefs about government shared by most in society.
Please select the least likely values below:
A.
B.
C.
D.
E.
liberty. equality, and property
community service and personal achievement
support for religion
prosperity, capitalism, and entrepreneurship
All of the choices above are correct
18)
In theory, random probability sampling (or random sample) should provide every
individual in the population with an equal chance of being selected. However, it is too costly and
inefficient. So there are several sampling techniques developed to alleviate the problem.
Among the selections below, what is not the technique to obtain good samples?
A.
Choose a random selection of telephone numbers and interview the respective households.
B.
Choose an area, such as urban area to consider as representative of all urban areas. Then,
randomly select respondents within those area.
C.
Put all voter names into a jar - or a computer - and randomly sample to obtain the data
D.
None of the above are correct.
E.
All of the above are correct.