How is evidence-based literature informing
e-Assessment practice: Findings from an
HEA project
Denise Whitelock
d.m.whitelock@open.ac.uk
CALRG 2012 - dmw
The Challenge
 Assessment drives
learning (Rowntree,
1987)
 How does eassessment and
feedback support
student learning?
CALRG 2012 - dmw
I hate marking but want the tasks and
feedback to assist student learning
CALRG 2012 - dmw
HEA funded project to:

Consult the academic community on useful references




Seminar series
Survey
Advisors
Invited contributors

Prioritise evidence-based references

Synthesise main points

For readers:

Academics using technology enhancement for assessment
and feedback


Learning technologists
Managers of academic departments
CALRG 2012 - dmw
Evidence-based literature
 142 references
 Technology-enhanced
methods
 Use for assessment and
feedback
 Type of evidence
 Ease of access (18
could not be retrieved)
CALRG 2012 - dmw
Categories of evidence used
Category
Description
1a
3
Peer-reviewed generalizable study providing effect size
estimates and which includes (i) some form of control
group or treatment (may involve participants acting as
their own control, such as before and after), and / or (ii)
blind or preferably double-blind protocol.
Peer-reviewed generalizable study providing effect size
estimates, or sufficient information to allow estimates of
effect size.
Peer-reviewed ‘generalizable’ study providing quantified
evidence (counts, percentages, etc) short of allowing
estimates of effect sizes.
Peer-reviewed study.
4
Other reputable study providing guidance.
1b
2
Number of references recommended in each
evidence category
Evidence
category
Number of
references
recommended
Cumulative
%
1a
15
12.1%
1b
8
18.5%
2
12
28.2%
3
49
67.7%
4
40
100.00%
Total
124
CALRG 2012 - dmw
MCQ & EVS use with large learning gains,
Draper (2009)
 Assertion – reason questions eg Pauli, Heisenberg,
Planck, de Broglie
 Learners generate reasons for and against each
answer
 Confidence marking, Gardner-Medwin
 Mazur’s method of brain teasing. Role of peers,
cognitive conflict
 Students create MCQs for EVS, reflection on counter
arguments before you hear them
 Students create MCQs for final exam. Will increase
exam performance
CALRG 2012 - dmw
Mobile Technologies and Assessment
 MCQs, PDAs (Valdiva &
Nussbaum, 2009)
 Polls, instant surveys
(Simpson & Oliver,
2007)
 EVS (Draper, 2009)
CALRG 2012 - dmw
CAP peer assessment system, BSc. Network
Management & Security (Intl.), Glamorgan,
Phil Davies
CALRG 2012 - dmw
Peer Assessment and the WebPA Tool
• Loughborough (Loddington et al,
2009)
• Self assess and peer assess with
given criteria
• Group mark awarded by tutor
• Students rated:
− More timely feedback
− Reflection
− Fair rewards for hard work
• Staff rated:
−
−
−
−
Time savings
Administrative gains
Automatic calculation
Students have faith in the
administrative system
CALRG 2012 - dmw
Authentic assessments: e-portfolios
CALRG 2012 - dmw
Candidate Assessment Records section, OCR IT Practitioner,
EAIHFE, Robert Wilsdon
CALRG 2012 - dmw
Building e-portfolios on a chef’s course
Evidence of food preparation
for e-portfolio
Modern Apprenticeship in
Hospitality and Catering,
West Suffolk College,
Mike Mulvihill
CALRG 2012 - dmw
Sharing e-portfolios: The Netfolio concept
 Social constructivism
 Connecting e-portfolios (Barbera, 2009)
 Share and build upon a joint body of evidence
 Trialled with 31 PhD students at a virtual
university
 Control group used but Netfolio group obtained
higher grades
 Greater visibility of revision process and peer
assessment in the Netfolio system
CALRG 2012 - dmw
MCQs: Variation on a theme (1)
An example of a COLA assessment
used at the Reid Kerr College, Paisley.
It is a Multiple Response Question
used in one of their modules.
This question was developed using
Questionmark Perception at the
University of Dundee. It is part a set
of formative assessment for medical
students.
CALRG 2012 - dmw
MCQs: Variation on a theme (2)
Example of LAPT Certainty-Based
Marking, UK cabinet ministers demo
exercise showing feedback,
University College, Tony GardnerMedwin
Drug Chart Errors and Omissions,
Medicines Administration
Assessment,
Chesterfield Royal Hospital
CALRG 2012 - dmw
Self-diagnosis
• Basic IT skills, first year
med students (Sieber,
2009)
• Competency-based testing
• Repeating tests for
revision
• Enables remedial
intervention
CALRG 2012 - dmw
Students want more support with assessment
 More Feedback
 Quicker Feedback
 Full Feedback
 User-friendly Feedback
 And ..................National Students’ Survey
CALRG 2012 - dmw
19
Gains from Interactivity with Feedback:
Formative Assessment
 Mean effect size on standardised tests between 0.4
to 0.7 (Black & Wiliam, 1998)
 Particularly effective for students who have not
done well at school
http://kn.open.ac.uk/document.cfm?docid=10817
 Can keep students to timescale and motivate them
 How can we support our students to become more
reflective learners and enter a digital discourse?
CALRG 2012 - dmw
LISC: Aily Fowler
 Kent University ab-initio Spanish module
 Large student numbers
 Skills-based course
 Provision of sufficient formative assessment
meant unmanageable marking loads
 Impossible to provide immediate feedback
»
leading to fossilisation of errors
CALRG 2012 - dmw
The LISC solution: developed by Ali Fowler
A CALL system designed to enable students to:
 Independently
practise sentence
translation
 Receive immediate
(and robust)
feedback on all
errors
 Attend immediately
to the feedback
(before fossilisation
can occur)
CALRG 2012 - dmw
How is the final mark arrived at in the LISC
System?
The two submissions are unequally weighted
•
•
Best to give more weight to the first
attempt
−
since this ensures that students
give careful consideration to the
construction of their first answer
−
but can improve their mark by
refining the answer
The marks ratio can vary (depending on
assessment/feedback type)
−
the more information given in the
feedback, the lower the weight the
second mark should carry
CALRG 2012 - dmw
Heuristics for the final mark
•
If the ratio is skewed too far in
favour of the first attempt…
. ..students are less inclined to
try hard to correct nonperfect answers
• If the ratio is skewed too far in
favour of the second attempt…
...students exhibit less care over
the construction of their initial
answer
CALRG 2012 - dmw
Free text entry
•
IAT (Jordan & Mitchell, 2009)
•
Open Comment (Whitelock & Watt, 2008)
http://kn.open.ac.uk/public/document.cfm?docid=11638
• McFeSPA system (Kochakornjarupong & Brna, 2010)
− Supports teaching assistants to
mark and give feedback on
undergraduate computer
programming assignments
− Support tool for semi-automated
marking and scaffolding of
feedback
CALRG 2012 - dmw
McFeSPA system

Supports teaching assistants to mark and give feedback on
undergraduate computer programming assignments

Support tool for semi-automated marking and scaffolding of
feedback

Findings that feedback model would be helpful in training
tutors .... Similar to Open Comment findings
CALRG 2012 - dmw
Audio Feedback (Middleton & Nortcliffe, 2010)
1.
Timely and meaningful
2.
Manageable for tutors to produce and the learner to use
3.
Clear in purpose, adequately introduced and pedagogically embedded
4.
Technically reliable and not adversely determined by technical constraints or
difficulties
5.
Targeted at specific students, groups or cohorts, addressing their needs with
relevant points in a structured way
6.
Produced within the context of local assessment strategies and in
combination, if appropriate, with other feedback methods using each medium
to good effect
7.
Brief, engaging and clearly presented, with emphasis on key points that
demand a specified response from the learner
8.
Of adequate technical quality to avoid technical interference in the listener’s
experience
9.
Encouraging, promoting self esteem
10. Formative, challenging and motivational
CALRG 2012 - dmw
Characteristics
Descriptor
Authentic
Involving real-world knowledge and skills
Personalised
Tailored to the knowledge, skills and interests of each student
Negotiated
Agreed between the learner and the teacher
Engaging
Involving the personal interests of the students
Recognises existing skills
Willing to accredit the student’s existing work
Deep
Assessing deep knowledge – not memorization
Problem oriented
Original tasks requiring genuine problem solving skills
Collaboratively produced
Produced in partnership with fellow students
Peer and self assessed
Involving self reflection and peer review
Tool supported
Encouraging the use of ICT
Advice
for
Action
Elliott’s characteristics of Assessment 2.0 activities
CALRG 2012 - dmw
Creating teaching and learning dialogues:
towards guided learning supported by
technology
 Learning to judge
 Providing reassurance
 Providing a variety of
signposted routes to
achieve learning goals
CALRG 2012 - dmw
Key Messages
 Effective regular, online testing can encourage
student learning and improve their performance in
tests (JISC, 2008)
 Automated marking can be more reliable than
human markers and there is no medium effect
between paper and computerized exams (Lee and
Weerakoon, 2001)
 The success of assessment and feedback with
technology-enhancement lies with the pedagogy
rather than the technology itself; technology is an
enabler (Draper, 2009)
CALRG 2012 - dmw
Key Messages 2
 Technology-enhanced assessment is not restricted
to simple questions and clear-cut right and wrong
answers, much more sophisticated questions are
being used as well (Whitelock & Watt, 2008)
 The design of appropriate and constructive
feedback plays a vital role in the success of
assessment, especially assessment for learning
(Beaumont, O’Doherty & Shannon, 2008)
CALRG 2012 - dmw
Key Messages 3
 Staff development essential to the process
(Warburton, 2009)
 Prepare students to take the assessments that use
technology enhancement by practicing with similar
levels of assessment using the same equipment
and methods (Shepherd et al, 2006)
 The reports generated by many technologyenhanced assessment systems are very helpful in
checking the reliability and validity of each test
item and the test as a whole can be checked on
commercial systems (McKenna and Bull, 2000)
CALRG 2012 - dmw
References

Beaumont, C., O’Doherty, M., and Shannon, L. (2008). Staff and student
perceptions of feedback quality in the context of widening participation,
Higher Education Academy. Retrieved May 2010 from:
http://www.heacademy.ac.uk/assets/York/documents/ourwork/research/B
eaumont_Final_Report.pdf.

Draper, S. (2009). Catalytic assessment: understanding how MCQs and
EVS can foster deep learning. British Journal of Educational Technology,
40(2), 285-293.

JISC, HE Academy, and ALT (2008). Exploring Tangible Benefits of eLearning. Retrieved in May 2010 from
http://www.jiscinfonet.ac.uk/publications/info/tangible-benefits-publication.

Lee, G. and Weerakoon, P. (2001). The role of computer-aided
assessment in health professional education: a comparison of student
performance in computer-based and paper-and-pen multiple-choice tests.
Medical Teacher, Vol 23, No. 2, 152 - 157.

McKenna, C. and Bull, J. (2000). Quality assurance of computer-assisted
assessment: practical and strategic issues. Quality Assurance in
Education. 8(1), 24-31.
CALRG 2012 - dmw
References 2

Middleton, A. and Nortcliffe, A. (2010). Audio feedback design: principles and
emerging practice. In D.Whitelock and P.Brna (eds), Special Issue ‘Focusing on
electronic feedback: feasible progress or just unfulfilled promises?’ Int. J.
Continuing Engineering Education and Life-Long Learning, Vol. 20, No. 2, pp.208223.

Shephard, K., Warburton, B., Maier, P. and Warren, A. (2006). Development and
evaluation of computer-assisted assessment in higher education in relation to
BS7988. Assessment & Evaluation in Higher Education, 31: 5, 583 — 595.

Strang, K.D. (2010). Measuring self regulated e-feedback, study approach and
academic outcome of multicultural university students. In D.Whitelock and P.Brna
(eds) Special Issue ‘Focusing on electronic feedback: feasible progress or just
unfulfilled promises?’ Int. J. Continuing Engineering Education and Life-Long
Learning, Vol. 20, No. 2, pp.239-255.

Warburton, B. (2009). Quick win or slow burn: modelling UK HE CAA uptake.
Assessment & Evaluation in Higher Education, 34: 3, 257 — 272.

Whitelock, D. and Watt, S. (2008). Reframing e-assessment: adopting new media
and adapting old frameworks. Learning, Media and Technology, Vol. 33, No. 3,
September 2008, pp.153–156, Routledge, Taylor & Francis Group. ISSN 17439884
CALRG 2012 - dmw
Four Assessment Special Issues

Whitelock, D. and Warburton, W. (In Press). Special Issue of
International Journal of e-Assessment (IJEA) ‘Computer Assisted
Assessment: Supporting Student Learning’

Brna, P. & Whitelock, D. (Eds.) (2010). Special Issue of
International Journal of Continuing Engineering Education and
Life-long Learning, ‘Focusing on electronic feedback: Feasible
progress or just unfulfilled promises?’, Volume 2, No. 2

Whitelock, D. (Ed.) (2009). Special Issue on e-Assessment:
Developing new dialogues for the digital age. Volume 40, No. 2

Whitelock, D. and Watt, S. (Eds.) (2008). Reframing eassessment: adopting new media and adapting old frameworks.
Learning, Media and Technology, Vol. 33, No. 3
CALRG 2012 - dmw