December 1, 2012, 6:01 am
When Internal Polls Mislead, a Whole Campaign May Be to Blame
By NATE SILVER
Pollsters can expect to take their share of blame when their campaigns lose, and this year has been no
exception. Not long after Barack Obama and Democrats had a strong night on Nov. 6, Republicans began
to complain publicly that the polls conducted by their campaigns and by affiliated groups implied
considerably more optimistic outcomes for them than actually occurred.
Perhaps these Republicans shouldn't have been so surprised. When public polls conducted by
independent organizations clash with the internal polls released by campaigns, the public polls usually
prove more reliable.
Take, for example, the gubernatorial recall election in Wisconsin earlier this year. Independent polls had
the Republican incumbent, Scott Walker, favored to retain his office by about six percentage points. A
series of polls conducted for Democratic groups showed a roughly tied race instead.
Mr. Walker in fact won by seven points: the independent polls called the outcome almost exactly, while
the internal polls were far from the mark.
Cases like these are fairly typical. My database of campaign polls released to the public in United States
House races found that they were about six points more favorable to their candidate than independent
surveys on average - and that they were typically less accurate in the end.
The traditional explanation for this phenomenon is that the subset of campaign polls that are released
to the public is subject to a type of selection bias. Campaigns conduct polls all the time, but only
occasionally disclose these results to the public and will be much more inclined to do so when the
numbers are favorable for their candidates (especially in comparison to independent polls). In essence,
the internal polls that filter their way into the public domain may be the outliers.
This is certainly an important part of the story, but my view is that it lets the campaigns off a little too
easily.
Pollsters must make a lot of choices and assumptions about turnout, wording of questions, whether to
include third-party candidates in the polls, how hard to push "leaners" toward their preferred
candidates, among other factors. These can be difficult choices, even if one is operating in good faith in
an effort to be as accurate as possible.
But when campaigns release internal polls to the public, their goal is usually not to provide the most
accurate information. Instead, they are most likely trying to create a favorable news narrative - and they
may fiddle with these assumptions until they get the desired result.
Some reporters make the mistake of assuming that information is valuable simply because it is private
or proprietary. But the information that makes it to the reporter's ears, or into his in-box, may be
something that the campaign wants him to hear or see.
The release of internal campaign polls is not that different from any other form of spin in this respect but that is precisely the point. Internal numbers that a campaign releases to the public should be
thought of less as scientific surveys and more as talking points.
But sometimes, internal polls make their way to the public through leaks that are not authorized by the
campaigns. Or reporters and analysts may see the "real" numbers, or reasonably explicit
characterizations of them, on background, on the condition that they not report them. (I saw some of
the Obama campaign's internal polling in 2008, along with that of some other Republican and
Democratic campaigns in a few instances over the years.)
Perhaps these internal polls are more accurate?
My experience has been that these polls can also exaggerate the standing of their candidate, if perhaps
not by quite as large a margin as those that are authorized for an on-the-record release. An interesting
example of this comes from Noam Scheiber of The New Republic, who says he received data on Mr.
Romney's internal polling in six states from an aide to Mr. Romney's campaign. In addition, Mr.
Romney's chief pollster, Neil Newhouse, disclosed the campaign's polling to Mr. Scheiber in a seventh
state, Ohio.
On average, the polls had Mr. Obama ahead by just one point between the seven states. They had Mr.
Obama trailing in Colorado and New Hampshire and tied with Mr. Romney in Iowa.
In fact, Mr. Obama won all seven states, and by an average margin of 5.7 percentage points based on
the ballots counted so far. (Several of the states have yet to certify their results.) Therefore, the polls
were biased in Mr. Romney's direction by nearly five percentage points, on average.
It should be mentioned that most of the independent polls this year were also slightly biased (in a
statistical sense) toward Mr. Romney. In the same seven states, the final Real Clear Politics averages
overrated Mr. Romney's standing by 2.5 percentage points. The final FiveThirtyEight forecasts were less
biased statistically, in part because our forecast model was designed to respond aggressively to
movement in the polls in the closing days of the campaign, which favored Mr. Obama after Hurricane
Sandy. Still, our forecasts in these seven states had a one-point bias toward Mr. Romney, on average,
compared against the actual results.
Further, the Romney campaign's polls did not have him winning the Electoral College, as they had Mr.
Romney behind in Ohio, Minnesota, Pennsylvania and Wisconsin, according to Mr. Scheiber. Assuming
that the campaign also had Mr. Romney trailing in Nevada and Michigan, but leading in Virginia, Florida
and North Carolina, then Mr. Romney would have been leading in states with 261 electoral votes, but
trailing Mr. Obama in states with 271. Mr. Romney's internal polls implied better chances for him than
the public polls did - but if Mr. Scheiber's reporting is right, they still had him as the underdog.
Nor is it clear which strategic choices Mr. Romney might have made differently had his campaign's polls
been more accurate. It isn't as though, if Mr. Romney felt he had a 10 percent chance of winning instead
of a 50 percent chance, he was simply going to stop campaigning.
Nonetheless, the seeming inaccuracy of Mr. Romney's internal polls ought to present a warning to
future campaigns. The problems with internal polls may run deeper than the tendency for campaigns to
report them to the public in a selective or manipulative way. The campaigns may also be fooling
themselves.
Our self-perceptions are very often more optimistic than the reality; 80 percent of people think they are
above-average drivers, for example.
These problems can be worse when we join together to form businesses or organizations. Honest selfassessment is a challenge for any business, and it is one reason that management consultants are
sometimes engaged at considerable expense to provide a supposedly more objective and unbiased take
on the state of the organization's operations. (Much of Mr. Romney's success in business, of course,
came precisely because he was able to identify companies whose organizational cultures prevented
them from functioning efficiently.)
A pollster working within a campaign may face a variety of perverse incentives that compete with his
ability to produce the most accurate possible results to his candidate. He may worry about harming the
morale of the candidate or the campaign if he delivers bad news. Or he may be worried that the
campaign will no longer be interested in his services if the candidate feels the race is hopeless.
Groupthink and confirmation bias are also risks in any organization, particularly under the stress of the
end stages of a political campaign.
So should the campaigns simply give up on conducting their own polls - and instead base their decisions
on the forecasts issued by Real Clear Politics or FiveThirtyEight?
If internal polls were solely used to diagnose the state of the horse race, then there might be an
argument for this - at least in presidential elections, where the public polling is normally quite reliable.
But the campaigns also use internal polling for message testing and a variety of other purposes. And
they may desire a more granular take on the race than the public polls provide - for example, in an effort
to measure the effectiveness of an advertisement in a particular media market, or among a particular
demographic subgroup. Pollsters like Mr. Newhouse are an important part of campaigns, and most of
them do their jobs well.
Campaigns should foster organizational cultures in which their pollsters are enabled to provide the most
value.
Campaigns might consider how pollsters are compensated; they could tie some of the pollster's
compensation to the accuracy of its final polls, for instance.
Some campaigns have had success with hiring more than one pollster and having them work relatively
autonomously from one another. This can serve as a check against groupthink - and may increase the
likelihood the different assumptions that the pollsters might introduce will be thought over and
debated.
But most important, campaigns would be wise not to have their pollsters serve as public spokesmen or
spin doctors for the campaign. Campaigns have other personnel who specialize in those tasks.
The role of the pollster should be just the opposite of this, in fact: to provide a reality check such that
the campaign does not begin to believe its own spin.
This post has been revised to reflect the following correction:
Correction: December 1, 2012
An earlier version of this post referred to a series of statements made under oath by the Democratic
pollster Harrison Hickman, who testified in the federal case against John Edwards.
The post incorrectly characterized Mr. Hickman as describing the release of internal polls to the news
media as a form of "propaganda." Instead, Mr. Hickman's statements in court referred to public polls
conducted by independent organizations. During the trial, Mr. Hickman said that he was more likely to
circulate public polls that showed more favorable results for his candidate, Mr. Edwards, even if they
contradicted his own views on the race.
The statements did not pertain to internal polling conducted by Mr. Hickman, and Mr. Hickman said in
court that he had told Mr. Edwards that he had little chance to win the Democratic nomination in 2008.
"The prosecutors were in possession of all of the polling we conducted for the Edwards campaign as well
as all other documents we created for them and all of my email to the campaign or sent on behalf of the
campaign," Mr. Hickman said in an email to FiveThirtyEight. "The prosecutors most certainly would have
raised questions about the accuracy of our polling or our characterization of it if there were legitimate
questions that could have been raised. In any case, there were no internal polls involved."
Copyright 2012 The New York Times Company