Inter-subject comparability
Dennis Opposs
SQA Seminar, Glasgow
28 February 2013
BUSINESS
FILM
ENGLISH
BIOLOGY
MUSIC
GERMAN
Psychology, law and media studies:
the ‘scandalous’ routes to A-grade success
By Richard Garner, Education Editor, Independent
Thursday, 14 August 2003
It is the "hidden scandal" of A-levels, the leader of one head
teachers' union said last night. Shrewd pupils, aided by their
teachers, are opting for "easier" subjects where they know they
will stand a better chance of getting an A-grade pass.
“It is incredibly worrying’” said Mr Dunford. “We need more maths,
science and modern language students. It is totally bizarre that
these are the hardest subjects at A -level.
“This is causing a vicious circle. It has been shown statistically
that psychology is an easier exam than maths at A-level. Part of
the remit of the Tomlinson committee should be to make sure all
subjects are of a comparable standard.”
Mr Dunford was accused last night of putting a different twist on
the “A-levels are easier” row, denigrating the achievements of the
250,000 or so youngsters who will be receiving results today.
In fact, he was just being honest. He pointed out that universities
do not demand that A-level passes are in the subject to be studied
at degree level – and teachers are therefore giving their students
the best chance of success by suggesting subject options which
they consider easier to pass.
His comments were seized on by the Conservatives. Damian
Green, the shadow Education Secretary, said: “We need to ensure
every A-level subject is regarded as of equal academic value and
equally different”
J.M. Crofts’ view of inter-subject
comparability, 1928
In a large examination is there any reason for thinking that the
candidate who is this 90th percentile in Mathematics has a better
brain, is a more capable scholar, is a better subject for the award
of a scholarship than the candidate who occupies that position in
English, or, for the matter of that, in any other subject? There is
no evidence whatever in favour of this; the case cannot be proved
one way or the other, and it would be unwarrantable to assume
that any difference does exist.
Crofts, J.M. & Caradog Jones, D. (1928). Secondary School Examination Statistics. London:
Longmans, Green and co.
Relation of severity of standard relative to
other JMB A level subjects in 1973 to change
in popularity since 1966
Share down
Severity
Share up
Severity
Stable
Severity
Greek
0.8
English Literature
-0.5
Spanish
0.0
Latin
0.8
Geography
-0.6
History
0.0
German
0.3
Economics
-0.5
Religious
Knowledge
-0.5
French
0.3
British Constitution
-0.5
Physics
0.5
Further Maths
1.1
Geology
-0.8
Chemistry
-0.1
Maths
0.6
Biology
0.2
Art
-1.1
Music
-1.4
Geometrical & Engineering
Drawing
-0.7
Domestic Science (The Home,
the Family and Society)
0.0
Pure Maths
0.5
Pure Maths with Statistics
1.0
Christie, T & Forrest, G.M. (1981). Defining Public Examinations Standards. MacMillan Education Ltd.
16 year olds achieving grades A* - C in
English
2012
Total
English
394,400
Mathematics
410,000
Source: Department for Education
GCSE subject criteria for English language
Grade Description
A
Candidates’ writing shows confident, assured control of a range
of forms and styles appropriate to task and purpose. A wide
range of accurate sentence structures ensures clarity; choices of
vocabulary, punctuation and spelling are ambitious, imaginative
and correct.
F
Candidates’ writing shows some adaptation of form and style for
different tasks and purposes. Paragraphing is straightforward but
effective; the structure of sentences, including some that are
complex, is usually correct. Spelling and basic punctuation are
mostly accurate.
GCSE subject criteria for mathematics
Grade Description
A
Learners use a wide range of mathematical techniques,
terminology, diagrams and symbols consistently, appropriately
and accurately. Learners are able to use different representations
effectively and they recognise equivalent representations; for
example numerical, graphical and algebraic representations.
F
Learners use some mathematical techniques, terminology,
diagrams and symbols from the foundation tier consistently,
appropriately and accurately. Learners use some different
representations effectively and can select information from them.
Comparing English and mathematics
Is English more severely graded?
Is teaching better in mathematics?
Is mathematics more motivating?
Do pupils work harder in mathematics?
Are results in mathematics better due to early entry?
16 year olds achieving grades A* - C in
English
2012
Boys
Girls
Total
English
178,400
215,900
394,400
Mathematics
207,000
203,100
410,000
2011
Boys
Girls
Total
English
196,000
231,000
427,100
Mathematics
198,000
195,500
393,600
Source: Department for Education
11 year olds achieving level 4 or above
86
85
84
83
82
81
80
% at level 4 or above English
79
% at level 4 or above Mathematics
78
77
76
75
74
73
72
2002
2003
2004
2005
2006
2007
2008
Source: Department for Education, England
2009
2010
2011
2012
George Bruce’s view of inter-subject
comparability, 1969
Each subject has its own special characteristics; in Art the entry is
highly selective, in Mathematics the dearth of teachers may be
reflected in the results […] These or some other special features
relate to almost every subject and justify deviations from [the same
percentage pass-rate]. If it seems unfair that candidates should
suffer from teaching deficiencies, it is impossible to see how this
can be avoided.
Bruce, G. (1969). Secondary school examinations.
London: Pergamon Press.
The NFER’s view of inter-subject
comparability, 1974
We […] do not expect an individual candidate to achieve the same
grade in every subject that he takes. However, we can see no
logical reason why, if a large group of candidates representative of
the population took, for example, both English and mathematics,
their average grades should not be the same. […] there is no reason
to suppose that the candidates as a whole would not try equally
hard in both subjects. There is also no reason to suppose […] that
their teachers in one subject are better than their teachers in the
other.
Nuttall, D.L., Backhouse, J.K. & Willmott, A.S. (1974). Comparability of standards between
subjects. London: Schools Council.
Robert Wood’s view of inter-subject
comparability, 1976
[…] the suggestion that standards in different subjects ought also
to be comparable [is a] lunatic idea.
The only way standards might be equated is through some external
statistical intervention except that there is no [valid] statistical
adjustment […]. In any case, I cannot see examiners, candidates or
the public tolerating fudging of grades to make them conform to a
statistical model dreamed up by researchers from the National
Foundation for Educational Research.
Wood, R. (1976). Your chemistry equals my French.
Letter to The Times Educational Supplement.
Available approaches: statistical
Subject pairs analysis
Common examinee methods
Available approaches: statistical
Subject pairs analysis
Common examinee methods
– Kelly’s method
– Average marks scaling
Available approaches: statistical
Subject pairs analysis
Common examinee methods
– Kelly’s method
– Average marks scaling
Rasch method
Available approaches: statistical
Subject pairs analysis
Common examinee methods
– Kelly’s method
– Average marks scaling
Rasch method
Reference tests
Available approaches: statistical
Subject pairs analysis
Common examinee methods
– Kelly’s method
– Average marks scaling
Rasch method
Reference tests
Value added models
Difficulty of A level subjects, using
correction factors
Carol Fitz-Gibbon & Luke Vincent’s view of
inter-subject comparability, 1996
Whilst we cannot rule out a widespread failure of teachers in
mathematics, science and foreign languages, one not shared by
teachers in sociology and other easier subjects, we find it an
offensive proposition to the teaching profession and highly unlikely
that this applies throughout all the datasets we are encountering.
Fitz-Gibbon, C.T. & Vincent, L. (1996). Difficulties regarding subject difficulties: developing
reasonable explanations for observable data.
Oxford Review of Education, 23(3), 291-298.
Issues with statistical approaches
Cyprus
Issues with statistical approaches
Cyprus
Western Australia
Issues with statistical approaches
Cyprus
Western Australia
Singapore
Relative ‘difficulties’ in Singapore and
England
Pollitt, A in Newton et al (2007). Techniques for Monitoring the Comparability of Examination
Standards. London: QCA
Available approaches: qualitative
QCA study
Available approaches: qualitative
QCA study
What action could Ofqual take?
Carry on
Publish tariffs
SQA National Ratings: Higher 2008
Chemistry
-0.29
History
0.08
Physics
-0.27
Geography
0.14
Mathematics
-0.22
French
0.26
English
-0.22
Spanish
0.27
Biology
-0.20
Music
0.72
Latin
-0.19
Art & Design
0.75
Source: Scottish Qualifications Authority
Alton, A & Pearson, S (1996). Statistical approaches to inter-subject comparability.
Report for the Joint forum for the GCSE and GCE
Alton, A & Pearson, S (1996). Statistical approaches to inter-subject comparability.
Report for the Joint forum for the GCSE and GCE
English literature
Before adjustment To align with 0
To align with physics
Grade
AEB
AEB
AEB
A
15.0
10.1
5.0
E
95.7
90.8
79.7
Physics
Before adjustment
To align with 0
Grade
AEB
AEB
A
16.8
26.6
E
74.8
83.9
Alton, A & Pearson, S (1996). Statistical approaches to inter-subject comparability.
Report for the Joint forum for the GCSE and GCE
Raw and revised outcomes for the modelled
AQA A levels
Raw A Level
Revised A Level Difference
A
E
A
Biology
27.5
95.4
30.57
Chemistry
34.6
97.3
Physics
32.1
Mathematics
E
A
E
97.09
3.07
1.69
38.65
98.31
4.05
1.01
95.2
34.07
97.03
1.97
1.83
36.2
97.4
33.17
97.79
-3.03
0.39
French
36.2
98.9
39.5
98.96
3.3
0.06
German
33.9
99
34.58
98.95
0.68
-0.05
Spanish
33.7
98.9
35.01
98.87
1.31
-0.03
English Lit
22.6
98.7
21.62
97.82
-0.98
-0.88
Copyright © 2008 AQA and its licensors. All rights reserved
What action could Ofqual take?
Carry on
Publish tariffs
Make adjustments
Introduce criterion referencing