Polling 101
A Practical Guide for Campaigns
Gaurav Oberoi
CEO, PrecisionPolling.com
gaurav@precisionpolling.com
April 25, 2010
What We’ll Talk About
• Overview
• Methodology
• Campaigns and Polling
Overview
What is public opinion? Who cares?
How can we measure it?
What is Public Opinion?
“The aggregate of individual attitudes or
beliefs held by the adult population.”
- Wikipedia
“… those opinions held by private persons
which government finds it prudent to
heed.”
- V.O. Key Jr.
Who Cares and Why?
• Government
– How well am I doing?
– What else do I need to focus on?
• Candidates
– What do voters want?
– How do I market myself?
– Am I going to get elected?
• Corporations
– How can I leverage public opinion for my own goals?
• Media
– How can I get more pageviews? (or sell dead trees?)
How Can We Measure It?
• Ask everyone what they think
– Haha, that’s good.
• Ask as many people as I know
– But that’s clearly not representative
• Ask a random sample of people that
represent the population at large, and
extrapolate
– Aha! That sounds like science!
Methodology
What does scientific polling look like?
How do we know it’s accurate?
Where does it break down?
5 Steps to a Scientific Poll
1. Define the population to be sampled
– E.g. voters for the upcoming election
2. Construct a sample
– E.g. 20k likely voters with match demographics
3. Write some questions
– Careful to minimize bias in your wording
4. Pick a survey methodology and run it
– Live interviewers, automated, online…
5. Analyze and compute results
– E.g. calculate and report error, weight if necessary…
1: Define Population to Sample
• Usually: those who will vote in the election
• But not always so easy:
– What if unexpected voters show up?
E.g. lots of young voters and Obama
• The answer: a combination of trending + gut
– Who has typically turned up to vote in the past?
– Anything that might materially change that?
2: Construct a Sample
• Randomly pick people who match the
demographics of the population
– E.g. if your population of likely voters are 55% female
and 45% male, your sample should be too
• There will be some sampling error
– This is the +/- margin of error you see on polls
– E.g. sample (error): 100 (11%), 600 (5%), 2k (2%)
• Picking a good sample has other complexities
– E.g. account for non-response (don’t answer phone)
3: Write Questions
• Goal is to gather data while minimizing bias
• Basics:
– Include all options, no leading questions etc.
• More nuanced:
– Randomly rotate names; try to match what will
eventually be on the ballot; identifying the
sponsor may bias answers etc.
4: Pick a Survey Methodology
• Most common: live interviewers on a phone
– CATI (computer assisted telephone interviewing)
– Pros: can ask complex surveys, verify speaker on the
phone, works for long surveys
– Cons: expensive, quality varies across interviewers,
social desirability bias, costs many $$$, slow
• Increasingly common: automated phone polls
– Pros: cheap, fast, removes social desirability bias
(Bradley effect), uniform quality
– Cons: can’t work for long or complex surveys, can’t be
100% sure who’s on the other end
• Not common: online or panel polls
5: Analyze and Compute Results
Typically involves:
• Computing sampling error
• Generating crosstabs by demographic
• Weighting to account for non-response bias
E.g. not enough young people answered my poll
• Scratching heads if there are a lot of undecideds
• Writing analysis with regards to political context
BUT What About These Problems?
• Non-response bias:
– Hang up  low response rate so sample is skewed
– Typically: few young people answer
• Social desirability (Bradley effect):
– “Of course I’m going to vote, I’m a good citizen!”
• Cell phone problem
– “We can’t poll cell phones?! But that’s all I have!”
• Lots of undecideds
– Build models to guess what they will do?
Campaigns and Polling
How do campaigns use polling?
How much does it cost?
What are some practical tips?
How do Campaigns Use Polling?
• Understanding voter sentiment
– “What do they care about most?”
• Testing messages
– “What’s the best way to market my guy/gal?”
• Measuring media spend
– “Did that direct mailing word? If so, let’s do
more.”
• Responding to hot button issues
– “The markets just crashed, how should we
respond?”
How do Campaigns Use Polling? (Cont.)
• Horse race polls
– “Did we get more of the swing vote this week?”
• Micro-targeting
– “With my limited budget, where can I make the
most impact?”
• Influence
– “This poll is super positive, send it to Publicola!”
And How Campaigns Use FAKE Polling
• Fake polls: called “polls” but are mostly ads
– Typically frowned upon as shady
• Push polls:
– One dimensional questions to influence you
– E.g. “Do you want govt. bureaucrats deciding when
you can see your doctor?”
– Don’t confuse this with message testing
• Straw polls:
– Non-scientific/informal poll to a select audience
– E.g. if The Stranger was to ask its readers about
legalization and report on it to gain influence
How Much Does it Cost?
• Live interviewer polls are expensive
– $10k-$35k
– 20 minutes; 20-30 questions; 500-1000 completes
– 1 week to field; 2-3 weeks for results and analysis
• Automated polls are much cheaper
– $1-4k
– 3-7 minutes; 5-15 questions; 500-1000 completes
– Days to setup and get results
– Hours if you use PrecisionPolling.com
Polling in the Campaign Lifecycle
• In the beginning
– Commission a big live interviewer poll ($10-35k)
– Get a lay of the land; strategy and messaging
• Before and after direct mail or TV
– Polls to learn who to target; measure spend after
– Automated polls work well ($1k)
• Last few weeks
– Nightly tracking polls; automated ($1k)
Practical Tips for Polling
• Poll even if your budget is tiny
– Do an automated poll if that’s all you can afford
– Data can focus your ad spend = more value
– Data can help move the out of a deadlock
E.g. Bob & Lisa disagree about the mailer message
– Data can cover your you-know-what
E.g. candidate disagrees, but data backs you up
Practical Tips for Polling (cont.)
• Keep costs low by improving response rates
– Make sure your caller-id is local
– Call in the early-evening (6:30-8:00pm)
– Have a snappy intro
– Use clear and loud voices, or recognizable ones
– Simple sentences; foreign language if it helps
Practical Tips for Polling (cont.)
• Use high quality lists
– Meaning: updated for moving and deaths, scrubbed
for do-not-call, good matches to phone numbers etc.
– Good phone list vendors keep them up-to-date
– Ask around for recommendations
• Leverage your friendships
– Your party may maintain phone lists you can use
– Your consultant may be able to help with basic polling
Practical Tips for Polling (cont.)
• Understand legal issues:
– Disclosure: do I need to say who’s paying for this?
– Do not call list: can I make calls in this state?
– Automated: how about automated ones?
– And more: legal requirements vary based on
audience, message, and funder.