Efficiency vs effectiveness
of assessment – the tensions
Chris Rust
Lots of evidence current
practices are not very
effective nor very efficient
 QAA subject reviews
 National Student survey
 “the Achilles’ heel of quality” /Summative assessment
practices “in disarray” (Knight 2002a, p.107; Knight
2002b, p.275)/“Broken” (Race 2003, p. 5)
 “There is considerable scope for professional
development in the area of assessment” (Yorke 2000, p7)
 “No longer fit for purpose” (Burgess 2007, p5)
 Media accusations of dumbing down & grade inflation:
60% gaining 1sts & 2.1s in 2010 cf 30% in 1970
(Sunday Telegraph, 2/1/11, p6)
 “serious grounds for concern” (IUSS 2009, p116)
Danger our assessment
practices encourage a
Surface approach to learning
“The types of assessment we currently use do not
promote conceptual understanding and do not
encourage a deep approach to learning………Our
means of assessing them seems to do little to
encourage them to adopt anything other than a
strategic or mechanical approach to their studies.”
“…students become more interested in the mark and
less interested in the subject over the course of their
studies.” (Newstead 2002, p2)
Many research findings indicate a declining use of
deep and contextual approaches to study as students’
progress through their degree programmes
(Watkins & Hattie, 1985; Gow & Kember, 1990; McKay & Kember,1997;
Richardson, 2000; Zhang & Watkins, 2001; Arum & Roksa, 2011)
Resources and Validity vs
Reliability
[Dependability: one-handed clock (Stobart, 2008)]
Construct validity
(authenticity)
Manageability
(resources)
Reliability
Resources and Validity vs
Reliability
Construct validity
(authenticity)
‘This quest for
reliability tends to skew
assessment towards the
assessment of simple and
unambiguous achievements,
and considerations of cost add
Manageability
to the skew away from
(resources) judgements of complex learning’
(Knight 2002, p278)
Reliability
Resources and Validity vs
Reliability
Construct validity
(authenticity)
assessing
airline pilots
Manageability
(resources)
Reliability
Resources and Validity vs
Reliability
Construct validity
(authenticity)
assessing
airline pilots
Manageability
(resources)
trade-off,
dependant on context
Reliability
Resources and Validity vs
Reliability
Construct validity
(authenticity)
if more purely
formative
assessment,
Manageability
(resources)
could increase
authenticity
with less cost?
Reliability
Resources and Validity vs
Reliability
Construct validity
(authenticity)
if more purely
formative
assessment,
could increase
authenticity
Manageability
(resources)
if less
summative
assessment,
could afford/
should aspire
for it to be
here?
Reliability
Module assessment vs
Programme assessment
Module assessment
 Atomised/disconnected
 Problems with feedback carryover
 Less challenging
 What ‘whole’ do the parts add
up to? (NB aggregation)
 More than necessary to make
summative judgements
 High staff workload
 Problems with bunching
But…
CATS & ? Flexibility/choice
 Fewer, integrated, more
challenging and authentic
assessment possible
 Allows for ‘slow learning’
 Better summative judgements
 Reduced staff workload
But...
logistically very difficult in a
modular system,
especially problems with credit
and flexibility
Equivalence of demand:
choice vs standardisation
 Equivalence of demand should consider total
expected student learning hours, not try to single
out assessment
 Any standardisation should avoid confusing
consistency with conformity, and a reliance on
often ‘numbers-based’ rules (e.g. 1st year essays
3,000 words max, 3rd year essays 5,000 words
max; or, no more than two pieces of assessment
per module) and crude word-equivalence
Improving feedback: a
potential win/win
Staff originated
Involving students
(self and peer-assessment,
marking exercises, etc.)
 High staff workload (especially
with large classes)
 Problems with speed of return
 Frequently not read or
understood
 A monologue
 Can reduce staff workload –
certainly over the programme
 Can be more immediate
 Becomes a dialogue
 Increases students’ assessment
literacy
 Improves quality of students’
work
 Essential graduate attributes