Comments from External Examiner on ‘Mark review’ Proposal
In my view as an external you really do not need the process of reviewing individual marks
for borderline cases if you adopted a simple convention. That is to give round up averages. It
is quite ridiculous that a student who achieves 59.9 is adjudged to be less than 60, for
example.
Arithmetic averaging of what are a series of professional judgements (ie the module grades)
cannot be so sound as to be confident that 69.8 or 59.7 are meaningful to one decimal place.
There is plenty of scholarship and research to demonstrate this.
Therefore give exam boards more leeway – allow 59.5’s to be awarded a 2.1 and boards to
exercise their scrutiny in the borderline range above 57.5 and 67.5 etc by looking at the
pattern of marks and determining if there is clear evidence of performance in the higher band.
This is much more in line with approaches in other (Russell Group) universities in the UK.
(CB)
As for the proposed amendment to the Mark Review - I am not clear as to the process for
this. Is the idea that external examiners can propose an amendment to a student's mark at
the actual award board meeting? Or is this a process that would take place prior to the
meeting and then any recommendations ratified or rejected at the meeting?
Regardless of the answer, in principle I think this is a good thing - at the OU we consider all
students whose marks are 1% below each boundary (and often 2% below the pass mark)
and at the EAB we look at all these students' final submissions and decide whether or not to
award more marks or stick with the existing mark. We are not able to consider any of the
continuous assessment marks however, only the End of Course Assessment (which can be
either an exam or dissertation/essay). Although some people consider this unfair because
students whose marks fall outside of these limits are not considered, we are at least applying
a degree of fairness by considering all those with marks 1% below each boundary. If we
reconsidered every single student's mark then this would indicate a lack of faith in our
markers (who use very clear criteria for marking) and would make the whole system
unworkable. I think the only students who might consider this unfair would be those who
have failed, there would be more complaints if we didn't consider all those students who
were just outside boundaries! (CEL)
I would like to comment about the final paragraph regarding the extent to which External
Examiners should have the power/right to make recommendations about the mark being
awarded by the internal markers.
I would be happy to make a recommendation (and no more) about a mark that has been
awarded but as is suggested this can be unfair to students’ work that has not been reviewed.
Sometimes this is because one individual’s (say) dissertation is a “rogue” or unusual one in
some way and any alteration does not impact on the overall accuracy of all the marking
(assuming the external has seen a reasonably wide range).
However, if it is evident there has been a clear pattern identified that say marks in the 60%
to 69% are too generous (or too severe) then that raises a bigger question about the internal
teams standards of marking and this could be a more fundamental issue, which an external
examiner would be duty bound to indicate as it potentially impacts on a number of grades
awarded.
I try to work with the internal teams to agree a way forward by strength of argument and cooperation rather than to impose a particular grade. My experience is that often internal teams
can be over severe rather than over generous with their students. In the case of a “rogue”
dissertation one would hope that the team would ensure the external sees it, simply because
there is some disagreement about how the dissertation should be viewed.
1
In conclusion a single piece of work could be given a strong steer from an external but a
band of work (say 60% to 69%) could open up a can of worms and it may be that the
external can only give a general steer rather than tinker with sets of marks. (MR)
Just a short message expressing my support for the amendment as formulated in the ‘note
to externals’. I refrain from commenting on the other documents that have “taken
approximately 2 years [to draft]and during that time [have] been subject to two University
wide consultations.” (KRSH)
My view regarding “mark review by submitted work” is very clear. It is not an examiners role
to re-mark work (since the examiner will not necessarily read each specific piece in the same
detail as the first marker, they are moderating the collective work of a group). Most
universities have stringent internal moderation procedures (which usually stipulate that the
internal moderator should see all work in each boundary zone) if this quality process is
correctly implemented there should be no reason for the examiner to doubt the integrity of
the awarded mark. Having said this, the proposed amendment appears a sensible solution if
this is deemed a problem for some courses. The final decision however should and must
rest with the exam board. (JP)
I'm external for the ***********. I sympathise with the aim of the proposal to allow Externals to
move marks for candidates in the borderline zone. ********** University have recently been
through the same process of clarifying how this should work. However we decided against
moving marks, because we felt it indicated a certain level of distrust of internal examiners;
who knows whether - had the external seen work earlier in the year - another candidate
might have been moved into the borderline zone. Nor could it recognise how a candidate's
work was developing and improving over the time, so that a stellar dissertation (say) could
show that the candidate by the end of the course had corrected some of the mistakes of their
previous work.
We decided instead that in borderline cases of this sort, numerical marks were insufficiently
subtle. We asked examiners instead to make a recommendation based on reading the entire
run of work to get an idea of 'the candidate's overall profile'. This may be too woolly for you,
but in our case it has meant a pretty even division between candidates whose work is raised
to the higher class, and candidates whose work isn't. (BPH)
Specifically in respect of the role of the external in marginal cases, it has been common
practice now across the sector for externals to comment on overall marking standards and
when these standards appear to be awry, to adjust marks for the module as a whole. This
practice has been in place for 10 years to my knowledge. To revert to a system in which
borderline classification candidates are drawn to the attention to the external seems to me to
be a retrograde move for two reasons:
1. it introduces arbitrary cut-off points (I note here at a mark ending in 8): such cut-off points
simply transfer the potential for inequity to another point in the system; they don't solve this
problem
2. this redraws the role of the external to one of making decisions that internal examiners
have failed to make in respect of the marks given.
In my view the external's role is to ensure standards comparative with other institutions not
to make decisions about individual candidates. It may be that other mechanisms on deciding
overall degree classifications (like for example, the number of modules within a particular
class) are a better way of resolving marginal cases. (SW)
Mark review: I remain of the opinion that individual mark review is inequitable – partly
because it (dis)favours students from the sample against the rest of the cohort whose work
2
has not received this attention; and secondly, because limiting it to borderline marks at the
module level takes no account of aggregation across modules which may at the exam board
reveal a student to have borderline result, which could be related to one or other specific
mark – thus, for me, external examiners should be limited to making recommendations
across the sample/course, and while commenting where their seem to be anomalous grades,
this should only be taken as advice not cause for formal review (except where courses have
sufficiently small cohorts for examiners to see all submissions. (CM)
With regard to the change in practice. I understand the opposition to the change in the rules
and I probably agree with that. However, I can also very well live with the proposed new rule.
(CV)
Being new to the position it is difficult for me to judge where issues may reside. The
regulations you have sent me seem pretty standard and I can't see any problems with them.
The proposed change in the practice of ‘mark review by submitted work’ makes sense to me.
(SB)
In view of my limited time, I focussed particularly on the 'Note to Externals'. Overall the
documentation didn't, so far as I could see, raise any major issues of concern. The process
set out in the aforementioned 'Note' seems reasonable, if a shade cumbersome.(JH)
You explain that "we are considering a change in the practice of 'mark review' by submitted
work'". I am not convinced by the argument that it would be "unfair" to the student whose
work has not been reviewed; this is because the new "mark review" provision will
be available to all students who find themselves in a "boundary zone" situation - and it is
therefore available on a equitable basis. However, I am concerned about how the "mark
review" process would operate in practice, particularly because it will need to operate after
course unit moderation. I think that the practical effect of the new "mark review" practice
should be carefully modelled (e.g. using a sample of last year's results). In particular, it is
important to establish how many new "mark review" situations will be created by this
provision, and to establish how, where and when External Examiner's will undertake the
additional reviews. (RT)
Regarding the issue of external examiners changing marks of borderline candidates - I
would be against this practice, for the reasons stated below. The calculations on which
borderline cases are made are objective and impartial - I feel that the role of the external
should to be to check marking standards for cohorts of students, rather than to intervene to
alter individual students' classification, and that to change marks after moderation could
potentially erode confidence in the judgement of the examiners. (GW)
Broadly, they seem coherent and sensible to me, so no significant remarks to make.
I do have some misgivings about allowing External Examiners to recommend changes in
marks. In my view, this is only really useful and justified when there is a lack of clarity or
agreement between internal markers. But if I'm in the minority here, so be it. (MSR)
Thank you for notifying me of the proposed changes. I am particularly pleased to see that
there is a proposal to allow external examiners to recommend to the Board a change in mark
as I felt this has led to unfairness in the past. (EC)
Marks review: the proposed review for borderline cases only is fine (to change other marks
an external would have to see the entire run for any one module) - but please note my
comments on how this can work in practice, included in last year's report for UG, ********:
3
'The only area in which I feel some action may be appropriate is the way in which externals
are used by the institution when it comes to the consideration of borderline cases.
In subjects such as *******, where the majority of candidates study for joint subject
degrees, the current system at Manchester is for students’ results to be split between a
number of separate boards, before the joint School Board meets. In practice, this means
that while I was able to review all individual borderline marks in *******, I was not in a
position to offer advice in all cases where the overall profile might turn out to be borderline.
In a number of institutions I have worked with, such cases would be flagged up in
advance so that the relevant external(s) could review the full range of each student’s work.
Given that obtaining a first class degree is notoriously difficult in joint honours, students
should perhaps not miss out on this final opportunity for a full review of their performance.
Additionally, the fact that students’ results are split between a variety of boards means
that both the external examiner and departmental staff do not get a clear sense of the full
cohort’s performance.
While smaller, separate boards are a practical solution (both more manageable and less
wasteful in terms of staff time), it might at least be possible for all grids to be seen by the
internal examination secretary as well as by the external examiner, so that any relevant
cases could be identified and reviewed.
Along similar lines, problem cases (e.g. students returning after periods of temporary
withdrawal, who are not necessarily included in the reports produced by the Mitigating
Circumstances Committee) may need to be raised in advance so they can be fully
investigated by the exams officer.' (LP)
Thank you for sending the information about amendments to the Taught Degree Regulations.
Regarding the changes outlined in the “Note to externals”, this seems fine to me and the
increased flexibility it allows for should only be an improvement. (BF)
I recently received in the post a letter with attachments concerning the review of the taught
degree regulations. I realise these will have been sent out to all external examiners for
comment, however the programme I externally examine is not an undergraduate (or
integrated masters) programme. The new regulations should not therefore effect my work.
However, I would like to add that I do not think External Examiners should be able to change
grades of individual students where modules have been processed internally. This said, I
can see where this change is coming from and would not oppose it. (HS)
However, there is a problem with the current procedure of sampling assessments in nonborderline cases (at least in ************). The external examiners example the assignments
and exam papers of selected students, rather than sampling by course units. (It often
happens that we never examine any examples of work from some units.) This leads to the
following bind. It would be invidious to recommend changes (either up or down) to the marks
of those students we have checked out, on the basis that they just happened to be lucky or
unlucky enough to be sampled. But neither are the externals in a position to make any
reliable comments about the overall standard of marking on any unit, because we don't
sample units as such.(AC)
Mark Review. I am not convinced that a review by the External Examiner is the appropriate
procedure in the case of students who are in the boundary zone. Given turnover in externals
this could lead to variations in practice over time. Presumably the borderline situation will
come about in one of two ways: 1) the student just misses the requisite level by a few few
marks in some papers 2) the student achieves exceptionally high marks in a small number of
papers. In the first case I believe the marks should be reviewed by Internal Examiners, with
ratification of any recommendation of the higher level classification by the External. In the
case of the latter, I feel it might be appropriate to refer in the regulations, or
the operationalisation of them, to exceptionally high marks in a specified number of credits,
4
with the Board being allowed to award the higher classification in these exceptional
situations, subject to the agreement of the external examiner.
I accept this may over-complicate things! It all depends really on how many students are
typically likely to fall in the borderline zone. But I think these situations should be nailed
down as far as possible before calling on external examiner discretion. (AP)
I have no comments to make on the proposals other than that I do not agree with externals
having the power to alter marks that are not on the borderline. It is up to the external to
satisfy herself that an appropriate standard is being applied, not to tinker around with such
non-borderline scripts as might happen to find their way into a sample. To allow this would
be to introduce an element of lottery, which would in my view be regrettable. (ND)
I have looked at both sets of the Draft Degree Regulations and I am happy with these.
In relation to the potential for external examiners to make recommendations, I would take
this forward with extreme caution.
I am not sure that a full review of all the boundary students work would be beneficial as it is
difficult to make a comparison with the rest of the cohort marking without reading through all
the other students work submitted too, making this an extremely time intensive process for
the external examiners. The viva may be a better solution but I am aware that some students
will have better presentation or oral skills than others and it may not be fair to judge a
confident speaker against an equally academically clever but very nervous candidate.
I would suggest that the final decision should always lie with the University staff. (JD)
I am responding to the circular to External Examiners concerning the development of your
new Taught Degree Regulations. I have read all the documents, and they seem to me to
have been thoroughly thought through. With regard to the Note to Externals, the
arrangements for mark reviews seem well drafted. I agree with the recommendations to
ensure that ALL borderline candidates are reviewed (the borderline zone being made clear
in the separate ug and pg documentation), and that, wherever possible, internal examiners
are consulted. I note that the dual opportunity for ug review (mark review and viva) is not
replicated at pg level, where the viva route does not feature: was a viva at pg level
considered at all? I'm not necessarily recommending that it should feature, but wondered
whether the grounds for the differentiation could usefully be given somewhere in the
documentation. (MB)
5