Oxford University Centre for Educational Assessment
Supporting a Better Transition from Second
Level to Higher Education
Predictability in the Leaving Certificate Examination
Therese N. Hopfenbeck
24th June, 2013
John Hume building, NUI, Maynooth
Concerns about predictability

Concerns about predictability were articulated at a conference
that had been jointly arranged by the National Council for
Curriculum and Assessment (NCCA) and the Higher Education
Authority (HEA) to discuss issues surrounding the transition
from second to third-level education. The report on that
conference recommended that

“the NCCA and the State Examination Commission (SEC) will
develop proposals to address any problematic predictability
identified in an analysis of predictability in the Leaving
Certificate examinations”.
Masters in
Learning &
“A predictable examination is one in which the nature of
the examination paper can be sufficiently accurately
predicted to mean either that the examination is not
testing the full range of content expected or that it is not
assessing the assessment objectives as defined in
the specification [syllabus]. In particular, a highly
predictable examination would tend to reward recall of
knowledge even where it is ostensibly assessing analysis
or evaluation”.
Ofqual in their Predictability Studies Report on GCSE and GCE level examinations (2008)
Research questions

What kinds of learning are the examinations intended to
promote?

How predictable are examination questions in the Leaving
Certificate in Ireland?

Which aspects of this predictability are helpful and which
engender unwanted approaches to learning?

What subject-specific phenomena are associated with
predictability?
The research

Phase A: Literature review

Phase B: Examination material research – analysis of material
in six subjects

Phace C: Teachers and learners research – questionnaires
study of learners and interviews with teachers and learners in
June and September 2013. Fieldwork in 12 schools and
colleges, total of 72 interviews with teachers and 12 small
group interviews with students are planned.

Our final report will be submitted to SEC by the end of October
2013 for review.
Masters in
Learning &
Positively about predictability
In the context of the NCCA and HEA report, released in
September 2011, Ruari Quinn, Education Minister, is quoted in
the Irish Independent (Donnelly, 2011), as follows
“When the exams come out for both the Junior Cert and the
senior Leaving Cert the commentary from education
commentators frequently is, 'It was a good exam, there were no
surprises, it was as predicted',"
A media analysis of allegations of
predictability of the Irish School
Leaving Certificate
Search 1 April 2012 – 15 September 2012
A total of 238 sites were bookmarked, with the most frequent tags
being 'good exam' (68 sites), 'predictability' (58 sites), 'bad exam'
(43 sites), 'students' (40 sites), and 'maths' (35 sites).
Public examinations appear to be of great media interest in
Ireland. During the exam period, The Irish Times, The Irish
Independent, the Irish Examiner and thejournal.ie all publish
articles reviewing each of the major Leaving and Junior Certificate
exams.
Positivity about predictability
In 68 articles from summer 2012 where exams were reviewed as
largely 'good', five mentioned that there were 'no surprises', two
referred to 'few surprises' and one to 'no nasty surprises'. Three
articles presented the exams in a positive light as they were
consistent with past papers. One review praised the conventional
format of the examination. Another article highlighted the
predictable range of questions, whilst other reviewers praised the
inclusion of 'key' or 'popular' topics.
Examinations, but not as we know them

Examinations were criticised in 43 newspaper review articles of
the 2012 summer session. Where reviews were negative,
comparisons were often made with previous years, both
implicitly and explicitly:
 Exams were described as “tricky” (2), “difficult” (4), and “tough”
(2). When exams did not meet expectations, more emotive
language was used. The absence of the expected poets on
English Paper 2 was considered a “betrayal” by a student
writing for a national newspaper, and the Maths papers,
introduced as part of Project Maths, were considered
“disastrous”, “traumatic”, “one of the worst in the history of the
State”, and prompted student “outrage”.
Masters in
Learning &
20
Page 12
January
Presentation
title, edit in
Presentation
title, edit in
Examples of predictability
Fe a t u r e
Predictable
Unpredictable
Curriculum cove rage Know t he topics t hat Do not know how to
prepare for the
will be assessed
exam
M ay n o t n e e d t o
study the breadth of Performance based
upon luck of studied
material intended
curriculum-exam
match
Te a c h e r s m ay
narrow the taught
Te a c h e r s m u s t j u d g e
curriculum
which aspects of the
syllabus to teach
Te s t c o n d i t i o n s
Known in advanced
Va r y a n d s t u d e n t s ’
and can be practised capacity to adapt is
part of the
assessment
Examples of predictability
Fe a t u r e
Scoring
Examination
support materials
Predictable
Unpredictable
How performances are
credited is known openly
Information on rubrics is
not available
It is possible to learn the
scoring criteria rather than
the syllabus materials in an
extreme case
Past papers available
publicly
Wh a t is cred it-worthy
may change in relation to
changes in the
assessment
Little information
publicly available
relating to the
examination questions
Model answers accessible
Advisory materials from the
examination board, such as
examiners’ reports available
Te x t b o o k s c l o s e l y a l i g n e d
with examination questions
Other publicly available
information, such as
newspaper coverage of the
examinations, teacher
publications etc.
Some elements of predictability for teachers and learners are
necessary to ensure that a structured educational experience can
be provided. After all, the Secondary Leaving Certificate is a
curriculum-related examination, unlike, for example Programme
for International Student Assessment tests, which claim to be
curriculum-independent
To be able to learn, practice and perform, students need to know
the topics to study, what performances are expected, how
they will be assessed and what counts as progress in a
subject. Examinations are not intended to be entirely
unpredictable because we want students to be able to prepare for
them, unlike IQ tests, where the items are held securely and
people are not supposed to be able to study for them and improve
their scores.
What counts as valuable learning?
Underlying the debates about the predictability of assessment are
tensions relating to what counts as valuable learning. James
(2006) outlined historical developments in learning theory and
related these to changes in assessment formats over the past
hundred years.

A further element of this is that by making things more
transparent, opportunities for credit-worthy performances
become more accessible and the proportion of students who
attain qualifications can rise:

some stakeholders believing that it is the performances
demonstrated that are most important and others believing that
only a certain proportion of learners can attain the highest
standards (Baird, 2007)
Masters in
Learning &
20
Page 21
January
For discussions:
I.
What specific problems of predictability are there in the Leaving
Certificate examinations? Which subjects? Which
questions/types of questions? What is the difficulty with them?
II.
Given that the Leaving certificate will continue to be externally
assessed, how can we enhance its capacity to encourage good
teaching and learning?
III.
At what point does appropriate examination preparation in a
high stakes environment become undue narrowing of the
curriculum? How can we encourage people to stay on the right
side of this line?
I.
How do we reconcile the need to specify subjects and
examinations clearly enough to be fair to teachers and learners
with the need to encourage and assess adaptability, creativity,
and problem solving/critical thinking?
II.
To what extent does the transparency of the Leaving Certificate
examination system (e.g. availability of past papers, marking
schemes and Chief Examiners’ reports; viewing of scripts)
influence learning and teaching – both positively and negatively?