Can a computer-marked exam
improve retention?
Jon Rosewell
Communication & Systems, MCT
CALRG Annual Conference, 19th June 2012
Does
‘one-off’module
 poor retention?
An ‘M-world’
•
•
•
•
•
T180 Living with the Net (2004-2006) as an example
Some studying as ‘one-off’
Students enjoyed and completed course but daunted by EMA
Students who submitted nearly all passed – too late!
Not submitting EMA
 not completing module
 poor retention statistic (51%)
 HEFCE penalty
• Setting up students to fail?
Can a final CMA improve retention?
•
•
•
•
•
Submission of CMA generally high (T184: 87%, n=2158)
iCMA encourages engagement and motivation
Not as daunting as typical EMA
So add final CMA(CME) and reduce EMA
CME is a component of final assessment
 submission = course completion
Apologies for the acronym nightmare!
Can an iCME improve retention?
• Moodle quiz
–
–
–
–
Same format as mid-course iCMA
Open at start of course; single submission by traditional cut-off date
Questions can be revisited
Question level feedback immediately after cut-off date
(cf OU anodyne performance profile 3 months later…)
• Even without adaptive feedback, learner can judge their own
performance
– Builds confidence
– Lowers barrier to pass: relatively low score on EMA programming and
essay questions sufficient to pass
T184 assessment strategy
Original
• 10% – mid-course iCMA
• 90% – final written EMA
– Part I – short answer qns
– Part II – prog & essay qns
New
• 10% – mid-course iCMA
• 30% – final iCME
• 60% – final written EMA
– prog & essay qns
• Threshold:
– 40 marks on EMA
• Thresholds:
– 0 marks on iCME
– 0 marks on EMA
Hypothesis:
• Students will attempt iCME first, before EMA
 They will thus complete the module
• This will improve their confidence
• They will then be motivated to attempt EMA
They will thus pass the module
Did more submit iCME?
EMA
Total
Not submitted Submitted
CME Not submitted
44
1
45 13%
Submitted
19
281
300 87%
Total
63
282
345
18%
82%
T184 2011E & 2011J presentations
Did students submit iCME first?
CME clearly first
214
76%
EMA clearly first
9
3%
58
21%
CME & EMA together (<1hr)
CME slightly earlier
43
15%
EMA slightly earlier
15
5%
Total
T184 2011E & 2011J presentation
281
100%
Leaving it to the last minute:
submitting EMA
200
180
160
140
120
100
EMA submitted
80
60
40
20
10-Dec
03-Dec
26-Nov
19-Nov
12-Nov
05-Nov
29-Oct
22-Oct
15-Oct
08-Oct
01-Oct
0
T184 2011J
Not leaving it to the last minute:
submitting iCME
200
150
iCME started
100
iCME completed
50
10-Dec
03-Dec
26-Nov
19-Nov
12-Nov
05-Nov
29-Oct
22-Oct
15-Oct
08-Oct
01-Oct
0
T184 2011J
T184 completion rates
100%
95%
90%
85%
May
Oct
80%
75%
70%
Introduction of
CME
65%
60%
2004
2005
2006
2007
2008
2009
2010
2011
T184 completion rates
100%
95%
90%
85%
May
Oct
80%
75%
70%
Introduction of
CME
65%
60%
2004
2005
2006
2007
2008
2009
2010
2011
T184 completion rates
100%
95%
90%
85%
May
Oct
80%
75%
70%
Introduction of
CME
65%
60%
2004
2005
2006
2007
2008
2009
2010
2011
T184 pass rates
100%
95%
90%
85%
May
Oct
80%
75%
70%
Introduction of
CME
65%
60%
2004
2005
2006
2007
2008
2009
2010
2011
Interviews with students
I thought the balance of assessment was about the best I’d come across. Interesting
computer-based tests … plus the programming elements … an interesting [EMA]
with a choice of two essays. … The assessment contributed to the enjoyment of the
whole course.
By the time I’d finished the assessment I’d felt I’d learnt more. … I also felt fairly
confident that I’d got enough marks on the computer based test.
Interviews with students
[Early on] get an achievement and get a bit of success, … it was really easy that first
CMA. … So an early success and then you know at the end you’ve got some short
answers in the CMA that you think pick up your score a bit, and then you’ve got the
EMA where you can do even better. I think it’s spot on.
Interviews with students
… as long as they feel it’s rigorous … it didn’t have that kind of ITV quiz thing, you
know ‘what’s the capital of England’ or something. … There was a combination with
‘you’ve done this bit now, and now you need to tell us about it, now you need to
describe it, now you need to do some programming and show us your work’. It felt
like there was real rigour to it so it didn’t feel like ‘this is a quick and dirty kind of
interactive quiz’.
Interviews with students
Student:
I think it was excellent. … The only thing I don’t like is having to wait till
September for my result!
Me:
So that’s another reason for introducing that computer marked exam …
it gives students feedback immediately rather than a long delay.
Student:
Yes, so you know where you’ve gone wrong. That’s really good, and it
cheers you up as well, not an idiot after all!
Interviews with students
“I enjoyed all of it really. I thought the course materials were brilliantly laid out, I
thought it was very clear, concise. It gave you enough about the subject to make you
interested and wanted you to do more.”
“As I say it was a pleasure doing it. I hope I’ve passed, but I just think it was a joy
doing it, it was really fun, you learnt something, it really complemented what I learnt
last year and it’s just made me keen to do other stuff, or try and find out more about
it.”
The end … of T184