What is an Opinion Survey or
Poll?
GV917
Opinion Surveys and Polls



An Opinion Survey is a conversation between
people made possible by a relationship of
trust between them
This relationship differs in different survey
modes, but it is always a ‘conversation’ and
so subject to the complexities of human
interaction
There are different types of opinion surveys
and they all have advantages and
disadvantages
The Psychological Roots of Political
Conversations



Interactions in the conversation between
interviewer and interviewee have their origins
in Human Psychology and recent work of this
has drawn attention to very different ways
that people have of reasoning
The Psychologist Daniel Kahneman has an
interesting theory about the nature of
reasoning. He argues that it can take two
forms:
Thinking Fast and Thinking Slow
An Experiment
Read the following column of words and tell me if each
word is printed in Upper Case or Lower Case letters:


upper
lower




LEFT
left

LOWER



upper
UPPER


right
RIGHT
RIGHT
left

lower
LOWER
upper



LEFT
right
Two Systems are at Work



In the experiment it was easier to read out the blue
column with the words upper and lower in it than the red
column with the words left and right in it.
Why? In the first column we were already cued to think
‘upper’ or ‘lower’ because that’s what the words said –
so half of the work had been done for us – system one
can take care of much of the reasoning
In the second column we did not have this cue so we
had to think about upper case and lower case while at
the same time reading the words left and right – this
required more work and so system two was more
involved
Factors influencing responses
to surveys






Relationships between interviewer and
interviewee (interviewer effects)
The type of survey being done (mode effects)
The type of questions asked: factual, attitudinal
– question wording and balance are important
The subject matter of the survey – some things
are easy to ask about, others much less so
The order of the questions in a survey
The type of responses sought – closed or open
questions
An illustration of Complexities of Survey Questions: A
Simple Factual Question from the European Social Survey
2002
F30CARD 56 Using this card, if you add up the income from all sources, which
letter describes your household's total net income? If you don't know the
exact figure, please give an estimate. Use the part of the card that you
know best: weekly, monthly or annual income.
J 01
R 02
C 03
M 04
F 05
S 06
K 07
P 08
D 09
H 10
U 11
N 12
(Refused) 77
(Don’t know) 88
A Simple Factual Question from the European
Social Survey 2002 - (look at Non-Response)
hinctnt Household's total net income, all sources
Valid
Mis sing
Total
1 J
2 R
3 C
4 M
5 F
6 S
7 K
8 P
9 D
10 H
11 U
12 N
Total
77 Refusal
88 Don't know
99 No answer
Total
Frequency
713
1752
2762
4722
4736
4113
3738
3136
4719
1978
554
326
33248
4876
3573
660
9110
42358
Percent
1.7
4.1
6.5
11.1
11.2
9.7
8.8
7.4
11.1
4.7
1.3
.8
78.5
11.5
8.4
1.6
21.5
100.0
Valid Percent
2.1
5.3
8.3
14.2
14.2
12.4
11.2
9.4
14.2
5.9
1.7
1.0
100.0
Cumul ative
Percent
2.1
7.4
15.7
29.9
44.2
56.5
67.8
77.2
91.4
97.4
99.0
100.0
A Simple Attitudinal Question from the European
Social Survey 2002
How interested are you in Politics? are you (READ OUT)
polintr How interested in politics
Valid
Missing
Total
1 Very interested
2 Quite interes ted
3 Hardly interes ted
4 Not at all interested
Total
7 Refusal
8 Don't know
9 No answer
Total
Frequency
4614
15848
14487
7271
42220
31
86
21
138
42358
Percent
10.9
37.4
34.2
17.2
99.7
.1
.2
.1
.3
100.0
Valid Percent
10.9
37.5
34.3
17.2
100.0
Cumulative
Percent
10.9
48.5
82.8
100.0
The Interview ‘Conversation’




Public opinion is dynamic and varied
Some people are well informed and give
comprehensive answers to questions; Others
are not very well informed and give very
sketchy answers
Some people really don’t have an opinion
about some issues and so will say that they
don’t know or alternatively answer randomly
We have to know what polls can do and what
they can’t do
What can Opinion Surveys Do?




Opinion surveys can measure facts about
individuals quite well– their social
characteristics, backgrounds and experiences
They can measure people’s attitudes quite well
in relation to straightforward issues concerning
politics and society and their own behaviour
They can give us a representative picture of
what the nation is thinking about key political
issues like the future of European politics
If they are panels they can give us trend data
about the way people think or behave over time
What Can’t Opinion Surveys Do?



Opinion surveys are not good at getting indepth narratives from people such as their life
histories or detailed accounts of their time use
– these should use qualitative in-depth studies
They are not good at probing really
complicated issues in which people need to
think about things a lot before answering
They have a limited ability to predict what
people will do or say in the future – some
things are predictable but other things not
Question Wording Effects – Rules of
Thumb for drafting questions

Keep questions simple not complicated

“Are you interested in politics?”
NOT
“Are you interested in Parliamentary politics as
opposed to wider issues like Global Warming or
the Debt Crisis?”



Avoid ambiguity

“Have you travelled by air in the last year?”
NOT:
‘Do you ever travel by air and by car?”


Question Wording Effects – Rules of
Thumb for drafting questions

Be specific not vague


‘Do you think the Conservative Government is doing a
good job or a bad job?’
NOT
‘What do you think of the political situation right now?’

Avoid leading questions

‘How likely is that you will vote at the next election?’
NOT
‘What party are you going to vote for at the next
election?’



A Question Wording Experiment- the
Case of Party Identification

The traditional wording of the party identification question in election
studies was inherited from the American National Election Study of the
early 1950s, and used by Butler and Stokes in the first British Election
Study in 1963. It appears in all subsequent surveys

‘Generally speaking, do you think of yourself as Labour, Conservative,
Liberal Democrat (Scottish Nationalist/Plaid Cymru) or what?

If respondents say ‘None’ it is followed by a supplementary question:
‘Do you generally think of yourself as a little closer to one of the parties
than the others?’



This is then followed up with:
Would you call yourself very strong {Party}, fairly strong, or not very
strong?
Criticisms of the Traditional Partisanship
Question


It is too leading – it assumes people are
attached to a political party, when many may
not be. This may have been OK in 1963 but
not any more when fewer people are
attached to political parties.
For analysis purposes responses to the
supplementary question are often added to
the responses to the first question – making
the bias in analysis even bigger.
Revised Party Identification Sequence




‘Some people think of themselves as usually being a
supporter of one political party rather than another.
Do you usually think of yourself as being a supporter
of one particular party or not?’
IF YES
‘Which party is that?’
Note this is much less leading – giving respondents
‘permission’ to say that they don’t support a party
What Difference Does it Make?
Response Category
Traditional wording
Revised Wording
None
17.1
47.8
Labour
35.9
26.9
Conservatives
26.1
18.3
Liberal Democrats
14.6
5.1
Scottish Nationalists
1.4
0.5
Plaid Cymru
0.4
0.3
Greens
0.6
0.5
UKIP
1.0
0.6
BNP
0.2
0.1
Others
1.9
0.4
Question Wording Revisited



Question wording is really important. If one
wants to bias answers, then it is easily done
by phrasing the question in a given way.
However, a biased question can still be
useful. The traditional question has been
used in every election study since 1963 and it
will be included in the next study.
Why? Because it provides a time series.
The Decline of Partisanship in Britain,
1964 to 2005
2.3
64
2.2
66
Mean Strength (0-3 Scale)
70
2.1
2.0
74F
1.9
<--Regression Line
74O
1.8
92
79
1.7
83
1.6
87
97
01
1.5
R2 = .90
05
1.4
1964
2005
Year
Mode Effects






These are associated with different ways of
administering a survey
Face-to-face or ‘in-person’ surveys
Telephone surveys
Internet surveys
Postal Surveys
There are advantages and disadvantages of
different modes
Face-to-Face Surveys - Advantages




We can get national probability samples. This is
very important for inferring the characteristics of
populations from sample
The interviewer can persuade individuals to
participate and thus increase the response rate
The interviewer can respond to individuals who
don’t understand a question and clarify or
explain things
The interviewer can report on how the interview
went – eg.was the respondent attentive or not?
Face-to-Face surveys - Disadvantages






Face-to-Face surveys are very Expensive (e.g., $5500 per
voter in the US National Election Study in 2004)
Because they are expensive the N’s are often small – 792
voters in 2004 ANES
Slooooow Moving – they take an average of 6 weeks to
complete
Limited Ability for Panels – that is interviewing the same
people on more than one occasion
One Study Only! Data Mining, Pre-Test Biases, replication is
very difficult
In the case of election studies there is limited or No InterElection surveys
MODE EFFECTS 1
Reported Party Choice in Face-to-Face and Internet PostElection Surveys and Actual Vote in Britain, 2005 General
Election
50
45
39.6
40
36.1 36.2
Percentage
35
31.7 30.8
33.2
30
24.4
22.6
22.4
25
20
15
10
6.3
8.7 7.9
5
0
Labour
Conservative
In-Person
Liberal Democrat
Internet
Actual Vote
Other Parties
Mode Effect II - Turnout
2005 BSA, CSES, Paper
9.1
2005 BES, CSES, Internet
18.0
2005 BES, Campaign, Internet
21.8
2005 BES, IP
10.6
1964-2001 BES, IP
9.9
2001 BES Campaign, RDD
18.7
2000 CES, RDD
21.5
2004 CES, RDD
25.2
2000 ANES, RDD, rev
27.2
2000 ANES, IP, rev
20.2
2002 ANES, RDD, rev
17.8
2002 ANES, RDD, trad
37.9
2004 ANES, IP, rev
17.3
2004 ANES, IP, trad
24.7
2000 NAES, RDD
24.8
2004 NAES, RDD
34.7
0
5
10
15
20
25
Percent Overreport
30
35
40
Advantages of Internet Surveys




They are cheap – in the election study the
face-to-face survey costs up to ten time more
than an internet survey
They are fast – one can get 5,000
respondents within 48 hours on the internet
while this would take 6 weeks face-to-face
It is possible to do experiments (eg.
Feedback to voters experiment)
BUT – at the moment they are not probability
samples
Conclusions



Opinion Surveys are the core methodology
for empirical social science and are essential
for studying individuals
A lot of aggregate analysis – such as
modelling the relationship between the
economy and behaviour is based on national
data such as GDP per capita and spending
by consumers which is collected by surveys
But users of surveys have to know their
limitations