Online interactive
computer marked assignments
– active learning or just doing it?
Frances Chetwynd, Chris Dobbyn
and Helen Jefferis
Agenda
• Project background – HEA funded
• interactive Computer Marked Assignments
(iCMAs):
1. Students - attitudes and engagement
2. Authors – some recommendations
3. Teams - predicting student failure
Project background
• Based on Open University distance learning module
– Level 1; 60 credits
– 9 months duration
– Large population
– Local tutors
– Central module team
Project background
• Assessment
• 7 iCMAs; 40% average
• 4 TMAs; 40% average
• End of module assignment; 40% pass mark
• Data gathering
– Student survey: online 175 students
– VLE usage statistics: 2500 students
iCMA
• 20 questions
– 3 tries each question
– Increasing feedback
• Unlimited attempts each quiz
– 5 question variants
– Previous scores wiped
Students – objectives
• Based on student survey (51/175 responded)
– Reviewers (54.9%)
“Used iCMAs as guide to note taking. Didn’t submit until
after completing relevant sections”
– Previewers (23.5%)
“To see what I knew already so I could miss out studying
parts of the text….” (8%)
– Cruisers (21.6%)
“I only completed them in order to pass the module”
Students - VLE Analysis
• Patterns of completion: early and mid-module
iCMA51
iCMA54
SUBMISSION OF BEST ATTEMPT
Students - VLE Analysis
Event
Average mark at Number submitting a
module end (/20) 100% grade as final
version
Number of finally
submitted
iCMA51
16.67
52 (3.09%)
1680
iCMA52 (Sense)
17.64
80 (4.95%)
1617
iCMA53
15.4
13 (0.82%)
1591
iCMA54
15.85
26 (1.65%)
1572
iCMA55
14.93
20 (1.30%)
1544
iCMA56
14.47
19 (1.25%)
1515
iCMA57
14.93
17 (1.09%)
1553
Students - VLE Analysis
• Number of attempts recorded
iCMA56
iCMA51
Tutors – from student survey
Type of
reminder
Tutor-group
online forum
message
Message in
assignment
feedback
SMS message to
nominated
number
Email to
registered
address
Pop-up message
on website
Helpful
Neutral
Not helpful
68%
18%
14%
60%
22%
18%
30%
38%
22%
68%
18%
14%
64%
20%
16%
Authors – student survey
Student views on feedback for each attempt
Type of
feedback
Direction to
module material
Hint at correct
answer
Very useful
Not useful
35%
Sometimes
useful
51%
41%
41%
18%
14%
Authors – student survey
Comparing iCMAs with in text self-assessment
questions (SAQs)
– 70%
– 57%
– 59%
– 14%
iCMA questions made me think more
SAQs more useful when I had no idea
Completing iCMAs is more fun
I didn’t answer any SAQs
Authors – VLE analysis
How daring are we?
Question type
Audio-visual
iCMA51
iCMA52 iCMA53
iCMA54
iCMA55
iCMA56
iCMA57
2
2
Drag & drop
4
Drag & drop
selection
Matching
6
Multiple
choice
Multiple
response
Numerical
Total
1
13
6
6
40
3
2
1
11
5
10
11
8
50
4
1
1
2
3
3
9
6
4
3
1
2
2
3
5
8
4
2
2
3
3
14
4
10
Teams – The Borderliners
• Gather VLE and other data
• Identification of early indicators of failure
– Some students belong to the category of
borderline failures
• Score just below EMA pass mark
• Do not submit EMA despite respectable TMA grades
• Develop software tool to predict failure
Teams – predicting failure
• 165 students – stayed engaged;
• Failed final assessment
iCMA51 – whole cohort
iCMA51 – the ‘Borderliners’
SUBMISSION OF BEST ATTEMPT
Teams – predicting failure
Entire Cohort + Fails
iCMA51
iCMA52
57%
40%
28%
19%
iCMA53
iCMA54
iCMA55
iCMA56
% submitting final
version by:
Block 1 End
Block 2 end
Block 3 End
Block 4 End
Block 5 End
47%
25%
16%
9%
47%
34%
28%
17%
48%
28%
25%
18%
58%
40%
32%
26%
Teams – The Borderliners
• Artificial neural network
Predicted groups –
e.g Borderliners;
clear pass; out
Indicators (22) – e.g. all
assignment scores and
completion; age;
motivation.
Teams – The Borderliners
• Predicted categories passed on to tutors
• Help targeted at Borderliners
• Focus on real-time contact
Summary
• iCMAs new form of assessment
• Questions to answer:
Lack of student engagement – why?
Student motivations?
Best practice for authors?
Data for analytics?
References
• Crisp, B. (2007) 'Is it worth the effort? How feedback influences students'
subsequent submission of assessable work', Assessment & Evaluation in
Higher Education, 32:5, 571 - 581
• Jelfs, A. and Whitelock, D (2000). The notion of presence in virtual learning
environments: what makes the environment "real". British Journal of
Educational Technology, 31(2), pp. 145–15
• Jordan, S. (2011) ‘Using interactive computer‐based assessment to
support beginning distance learners of science’, Open Learning: The
Journal of Open, Distance and e-Learning, 26:2, 147-164
• Roediger, H.L., & Butler, A.C. (2011) ‘The critical role of retrieval practice in
long-term retention’. Trends in Cognitive Sciences’, 15, 20-27
• Timmers, C., Braber-van den Broek, J. and van den Berg, J. (2012)
‘Motivational beliefs, student effort, and feedback behaviour in computerbased formative assessment’, Computers and Education, 60, 25-31