Formative thresholded assessment:
Evaluation of a faculty-wide change in
assessment practice
Sally Jordan and Janet Haresnape
Faculty of Science
The Open University, UK
Assessment in Higher Education Conference, June 2013
This is a practice exchange…
• So please interrupt
Our plan
Previous practice
Drivers for change
What do we mean by formative thresholded assessment?
Evaluation and stumbling blocks
Early findings
The UK Open University
•
•
•
•
Founded in 1969;
Supported distance learning;
150,000 students, mostly studying part-time;
Undergraduate modules are completely open entry, so
students have a wide range of previous qualifications;
• Normal age range from 18 to ??
• 10,000 of our students have declared a disability of
some sort;
• 25,000 of our students live outside the UK.
Historic OU Science Faculty
practice
• Tutor-marked assignments (TMAs) and sometimes
interactive computer-marked assignments (iCMAs)
combine together into overall continuous assessment
score (OCAS);
• Examination and/or end-of-module assessment (EMA)
gives overall examination score (OES).
OCAS is integrated and
interactive
Continuous assessment
• Has a useful pacing function;
• Students are concerned about the minutiae of the
grading;
• Even though for most students their final grade is
determined by their performance in the examination or
end-of module assessment;
• A considerable amount of time and effort goes into
producing new TMAs for each presentation of each
module;
• And we don’t always get it right.
Drivers for change
• “when assessments serve both formative and
summative purposes… formative work will always be
threatened due to the dominance of summative
requirements” (Brearley & Cullen 2012 with echoes of
Black & Wiliam 1998, Gibbs 2006, Snyder 1971)
• Lack of alignment of tutor and student understanding of
our assessment strategies.
• Lack of alignment of tutor and student understanding of
the purpose of continuous assessment.
• Saving time and money.
Formative thresholded
assessment – two models
• OCAS is formative, but students have to demonstrate
engagement by scoring more than 30% in x out of y
TMAs and iCMAs; final score is determined by OES
alone.
• Students have to reach threshold (usually 40%) for
OCAS (overall), but their module result is then
determined on the strength of their OES alone.
Risks
• Students will feel insufficiently prepared for the
examination or end-of-module assessment;
• Students will not engage sufficiently in the formative
thresholded OCAS and so will lose valuable formative
opportunity;
• Assignments will be reused, so there may be more
plagiarism. For formative assessment, does this matter?
DISCUSSION POINT – what do you think about our move
to formative thresholded assessment?
Evaluation
• A series of practitioner-led mini projects;
• Strengths: we are practitioners and so close to the
student experience; we will help each other;
• Dangers: we are practitioners so we have considerable
pressures of other work; we won’t deliver: we will have
insufficient expertise.
DISCUSSION POINT: Does anyone have experience of
research or evaluation carried out in this way?
Evaluation
• Quantitative and qualitative;
• Janet – looking at behaviour of students who are taking
two modules, one with summative and one with
formative thresholded assessment;
• Lynda – looking at two new modules with subtly different
assessment strategies;
• iCMA usage can give a ‘signature’ of student
engagement;
• TMA submission rates (before and after the change of
assessment strategy);
• What do students think?
• What do tutors think?
Evaluation - complications
• We need to compare current with historic data; we need
to compare current with historic student perceptions;
• Confounding variables: other changes to assessment
strategy; changing student population; over-committed
students.
TMA submission rates
How many TMAs did students
omit
Correlations between exam score
and omitted TMAs
• 2011J – for students who submitted all TMAs, mean
exam score = 57.5%; for those who did not submit all
TMAs, mean exam score = 41.3%;
• 2012J – for students who submitted all TMAs, mean
exam score = 53.4%; for those who did not submit all
TMAs, mean exam score = 44.4%;
• Should this worry us? (DISCUSSION POINT)
• Not a causal relationship.
Engagement with iCMAs
Engagement with iCMA feedback
– same question, different mode
of use
Similar assignment, different
students
Outcomes from the evaluation
• Overall findings (and thus our plans for the future) – still
too early to say;
• A beneficial side effect is that we are thinking about our
assessment strategy for current and future modules in a
more coherent manner.
Sally Jordan
Senior Lecturer and Staff Tutor
Deputy Associate Dean, Assessment
Faculty of Science
The Open University
sally.jordan@open.ac.uk
blog: http://www.open.ac.uk/blogs/SallyJordan/