WJEC Internal Review
Report of Findings
Errors in Assessments
Summer 2011
1. Background
WJEC is extremely disappointed to report that two types of error affected our assessments this
summer. Like all the awarding organisations in a similar position, we greatly regret any distress
caused to students, parents or teachers. During the subsequent awarding processes, we have
taken all possible steps to ensure that candidates affected by the errors are not disadvantaged
in their outcomes.
The first type of error related to enlarged format question papers which are prepared in
response to requests received on behalf of a relatively small number of individual candidates.
These are papers produced to ensure access for visually impaired candidates. In the 2011
summer series, errors affected two questions within the enlarged version of a GCE Mathematics
unit and one question within a GCSE Chemistry paper.
The other issue was a software problem relating to an on-screen GCSE Geology paper. Some
candidates were unable to complete the examination on screen, and were allowed to continue
on paper, with additional time being allowed where necessary.
This report provides a summary of the findings from WJEC’s internal review of these
problematic situations which arose in relation to question papers in the 2011 summer series.
2. Terms of Reference
The detailed terms of reference for our internal review are as follows:
(a) Enlarged format question papers
(i) identify all processes undertaken with modified question papers that have the potential to
introduce errors which are not in the correct standard version of that same question paper; this
will need to include all processes undertaken by internal and external suppliers for the different
types of modified question papers
(ii) identify whether any of these processes can be eliminated by making more direct use of the
correct standard version
(iii) establish what combination of processes need to be undertaken for future production of
modified question papers, including confirmation of what focused checking processes will be
implemented in the 2011-12 production cycle
(iv) identify the impact of the late applications for modified papers made by many centres; the
closing date for submitting requests was 21 February, but many requests come in late [for
GCSE in summer 2011, 574 (out of 2055) applications were made after the closing date; for
A/AS, the corresponding figure was 81 (out of 299)]
(v) having established the impact of late applications, determine whether in the interests of
quality management a different balance needs to be struck between adhering to the closing
date and meeting all requests even when they are very late
(b) on-screen assessments
(i) identify the extent of testing/checking that was undertaken in relation to the GCSE Geology
on-screen assessment in 2011; this to include detail relating to the following:
- the technology platform on which testing was undertaken (including performance,
configuration and capability of communication networks and workstations),
- the extent to which test plans had been produced and followed including the use of multiple
routes though a question paper,
- the extent to which the outcomes of testing/checking were evidenced.
(ii) define a new regime for robust testing of all on-screen assessments, to be implemented for
all on-screen assessments being developed for 2011-12.
3. Main Findings – enlarged format question papers
All processes relating to the large print version of the GCE Mathematics question paper for unit
C3 have been undertaken internally. Processes relating to the GCE Chemistry paper involved
some process stages being undertaken by an external supplier. The review has therefore
needed to consider the process flow for modified question papers where (i) all stages are
undertaken in-house, and also (ii) where some of the processes involve external suppliers.
The process flow for modified question papers includes a stage in which formulae are retyped or
diagrams re-created. This is currently necessary because the digital object in which these are
held is not scalable through an enlargement process. The implication is that the processes
currently undertaken do include a stage which has the potential to introduce errors which are
not in the correct standard version of that same question paper.
Some possible technology solutions have been identified that have the potential to hold items
such as formulae and diagrams in a manner that is scalable through enlargement. If it is
confirmed that a solution of this kind is suitable, it should become possible to eliminate the need
for retyping formulae and re-creating diagrams. This would provide assurance that a correct
formula or diagram in the standard version of the question paper remains correct in the large
print version. Further exploration of these possible solutions is required before an
implementation decision can be made.
Arising from this review, consideration is also being given to other factors which can trigger a
modification requirement when creating an enlarged version of a question paper. One of the
identified factors is font choice, and consideration is being given to the possibility that the use of
a different font choice within the standard version of a question paper would eliminate some
types of text modification that are currently needed.
Alongside the exploration of technology solutions to reduce the extent of retyping and recreating that is necessary when creating an enlarged version of a question paper, proofchecking arrangements for such versions are being reviewed so these are focused on those
elements of the paper that have not been directly transferred from the standard version.
The requests for large print modified versions to the GCE Mathematics question paper for unit
C3 and for the GCSE Chemistry unit were received sufficiently early within the relevant
deadline; as these were the only modified papers which were found to be erroneous in the
summer 2011 series, we have no evidence this year of any direct connection between an
application being received late and the appearance of errors in a modified paper.
As there is no identified impact of late applications for modifications, there is no new information
from the summer 2011 series that can inform the balance we currently have: in the interests of
candidates, we apply our best endeavours in responding to late applications.
4. Main Findings – on-screen assessments
Our technical partner has confirmed that the question type which triggered the problems within
the GCSE Geology on-screen assessment had been developed and tested on a high
specification machine, and had not been subsequently optimised for running on machines which
matched their minimum specification. Their view is that this non-optimised question would have
consumed a disproportionate amount of the processing capacity available within a minimum
specification machine, hence preventing the normal flow through the assessment.
It has also been confirmed that multiple routes through a question paper are not explicitly
tested, although in practice some different routes through a paper will inevitably be followed
incidentally within the testing that is undertaken. Facilitating different routes through a paper is
confirmed as being a feature of the generic functionality of the platform, rather than being
related to individual question papers. The conclusion from the review is, therefore, that the
overall platform is considered to be robust.
For the summer 2011 assessment, there had been no comprehensive reporting of testing
outcomes, nor an indication of the equipment on which testing was undertaken.
5. Actions being taken to deliver improvements
The nature of the problems which led to errors in our summer series assessments are such that
we are able to address the causes ahead of the next cycle. The main actions being taken
forward are:
(i)
for enlarged format question papers:
 technology solutions are being explored that have the potential to reduce the
extent to which formulae are re-typed and diagrams re-created
 proof-checking arrangements for modified versions will be more specifically
focused on those elements of the paper that have not been directly transferred
from the standard version
 these two actions are also being taken forward in relation to those modified
question papers where some of the work is undertaken by an external supplier.
(ii)
for on-screen assessments:
 our technology partner will test all question types on machines which match the
minimum specification which has been communicated to centres, and will make
available all question types in a form that has been optimised for running on that
specification
 subsequent testing/checking work for complete on-screen assessments (within
which a range of different question types may be embedded) will transfer to
WJEC, who will have responsibility for defining the range of checks to be
undertaken at that stage.