11-002195-01 HEC/PKC/TM
Technical Report
This technical report, which can be appended to the Electoral Commission’s Post
NAW Referendum 2011 topline, contains a summary of the technical details and
statistical reliability of this quantitative research project.
1. Methodology
For this research Ipsos MORI interviewed a sample of 1,005 adults Wales between
the 5th and 9th March 2011.
Interviews were conducted by telephone using Random Digit Dialling. We
interviewed a representative quota sample. Quotas were set by age, gender,
working status and region. The data were then weighted to the known national
population profile in Wales.
2. Statistical Reliability
The nature of research methodology dictates that respondents to the questionnaire
are samples of the total population of Wales so we cannot be certain that the
figures obtained are exactly those we would have if everybody in Wales had been
interviewed (the ‘true’ values). However, the variation between the sample results
and the ‘true’ values can be predicted from the knowledge of the size of the
samples on which the results are based, the representativeness of the sample
itself and the number of times that a particular answer is given. The confidence
with which this prediction can be made is usually chosen to be 95% - that is, the
chances are 95 in 100 that the ‘true’ value will fall within a specified range.
Given that we have both set quotas during the fieldwork phase as well as weighted
our data to be representative of the profile of Wales, this reduces the ‘effective
base size’ from 1,005 to 7451. All statistical reliability has been calculated using this
effective base size.
Effective base: When data are weighted, it creates this 'design effect' which can
negatively impact on the reliability of the data. Therefore the effective base size
(the base size left when taking this into account) is the base size that can be used
for significance testing.
This is also known as the ‘design effect’, wherein some factors of the research
methodology can negatively impact on the reliability of the data. In this case, this is due to
the weighting used.
1
1
11-002195-01 HEC/PKC/TM
The table below illustrates the predicted ranges for different sample sizes and
percentage results at the ‘95% confidence interval’.
Size of sample on which survey
result is based
100 interviews
150 interviews
200 interviews
300 interviews
400 interviews
500 interviews
600 interviews
700 interviews
1,000 interviews
Approximate sampling tolerances
applicable to percentages at or near
these levels
10% or
30% or
50%
90%
70%
+
+
+
6
5
5
4
3
3
3
2
2
9
8
7
6
5
4
4
3
3
10
8
7
6
5
5
4
4
4
For example, with an effective base size of 600 where 50% give a particular
answer, the chances are 19 in 20 that the ‘true’ value (which would have been
obtained if the whole population had been interviewed) will fall within the range of
+4 percentage points from the sample result (i.e. between 46% and 54% inclusive).
When results are compared between separate groups within a sample, different
results may be obtained. The difference may be ‘real’, or it may occur by chance
(because not everyone in the population has been interviewed). To test if the
difference is a real one – i.e. if it is ‘statistically significant’ – we again have to know
the size of the samples, the percentage giving a certain answer and the degree of
confidence chosen. If we assume ‘95% confidence interval’, the differences
between the results of two separate groups must be greater than the values given
in the table below.
2
11-002195-01 HEC/PKC/TM
Size of samples compared
Differences required for significance
at or near these percentage levels
10% or 90%
50 and 70
100 and 150
100 and 200
100 and 400
200 and 400
300 and 300
300 and 400
300 and 500
300 and 700
400 and 400
400 and 700
500 and 500
200 and 1,000
500 and 1,000
30% or 70%
50%
+
+
+
11
8
8
7
6
5
5
5
5
5
4
4
5
4
17
12
11
10
8
8
7
7
7
7
6
6
7
5
19
13
12
11
9
8
8
8
7
7
7
7
8
6
So, for example, if we were to compare voters and non-voters views in the
Electoral Commission’s Post NAW Referendum Research in Wales, we would be
comparing sub group samples of approximately 624 (voters) and 381 (non-voters).
With these base sizes, where (e.g.) 30% of voters give a particular answer and
(e.g.) 37% of non-voters give the same answer, the chances are 19 in 20 that the
difference in responses between voters and non-voters is a ‘true’ (statistically
significant) difference (because the difference is seven or more).
3. Interpreting data tables
When looking at the data tables for this research, you will notice that in addition to
the percentage figures, there are also some small letters and other indicators
within the text on the tables. We have explained these in full below.
Letters assigned to percentage figures: if you look at the cross breaks (the
columns running across the top of each table, e.g. age, gender etc), each cross
break is assigned a letter (written in lower case directly underneath the column
heading) that it has throughout the tables. For example, ‘total’ is ‘x' , ‘voter’ is 'a',
‘non-voter’ is 'b' and ‘male is 'c'.
If a letter is written alongside a percentage figure within the body of the table, it is
highlighting the fact that the percentage is statistically significantly higher than the
corresponding figure in column 'a' or 'b' or 'c' etc. For example, if the percentage in
the total column (x) has 'b' and 'c' written alongside it, then the total (x) percentage
is significantly higher than that of non-voters (b) and males (c).
3
11-002195-01 HEC/PKC/TM
A practical example of this from the tables can be found in question 4. In the total
column (column x) it is shown that 59% of respondents ‘always vote at elections for
the Welsh Assembly’. As the table shows, this figure is significantly higher than
columns ‘b’ (non-voters 25%), ‘e’ (18-24 year olds 25%) and ‘g’ (35-44 year olds
47%). All other columns either have no significant difference with the total, or are
significantly higher than the total column themselves (which would be shown by
a 'x' next to the percentage in said column).
Small base size: This is a base that is too small to statistically analyse through
significance testing. The small number of respondents means that we cannot
guarantee representativeness and that the findings are therefore indicative. We
would thus advise that you not use these percentages for statistical analysis but
that they are used indicatively with a note to the reader outlining the small base
size and its limitations.
4. Social Grade Definitions
A
Professionals such as doctors, surgeons or dentists; chartered people like
architects; fully qualified people with a large degree of responsibility such
as senior editors, senior civil servants, town clerks, senior business
executives and managers, and high ranking grades of the Services.
B
People with very responsible jobs such as university lecturers, matrons of
hospitals, heads of local government departments; middle management in
business; qualified scientists, bank manages, and upper grades of the
Services, police inspectors.
C1
All others doing non-manual jobs: nurses, technicians, pharmacists,
salesmen, publicans, people in clerical positions and middle ranks of the
Services, police sergeants.
C2
Skilled manual workers/craftsmen who have served apprenticeships;
foremen, manual workers with special qualifications such as long distance
lorry drivers, security officers and lower grades of Services, police
constables.
D
Semi-skilled and unskilled manual workers, including labourers and mates
of occupations in the C2 grade and serving apprenticeships; machine
minders, farm labourers, bus and railway conductors, laboratory assistants,
postmen, waiter/waitress, door-to-door and van salesmen.
E
Those of lowest levels of subsistence including pensioners, casual workers,
the unemployed and other with minimum levels of income.
4
11-002195-01 HEC/PKC/TM
5. Welsh Assembly region definition
5
11-002195-01 HEC/PKC/TM
6. Claimed level of turnout
As part of this research respondents were asked whether or not they had voted in
the referendum. The chart below shows that 60% of respondents said they did
vote, compared to the official turnout of 35%. This finding of overclaimed turnout is
common in this type of research and in 2010 we found that 82% said they had
voted in the General Election.
Claimed turnout
49
Q As you may know, there was a referendum in Wales on Thursday 3rd
March about the powers of the National Assembly for Wales. Many
people have told us they didn’t manage to vote in the referendum on
3rd March. How about you – did you manage to vote in the
referendum?
Yes
2011
Referendum
No
2010
General Election
18%
40%
60%
82%
Base: 1,005 adults, fieldwork dates: 5 th March – 9th March 2011
Source: Ipsos MORI
6