Heriot Watt
University
Breakout Session
MCQ with Confidence
Dr Phil Davies
Division of Computing & Mathematical Sciences
Department of Computing
FAT
University of Glamorgan
E-Assessment can be viewed
from different perspectives
• Commercial / Large Scale Implementations
– Question Mark’s Perception
– MOODLE
– Triads (25 different types of Multiple Choice Type
Questions!)
– TOIA (FREE!) Tool for On-line Interoperable Assessment
– Blackboard Assessment (version!)
– Standards (IMS QTI Specification)
• Try it and see
How easy to do MCQ?
Cost?
•
•
•
•
Question
<normally>
Key =
Distracter One, Distracter Two, ……………..
• Note
– Key(s) must always be correct
– How many students would you expect to get it right for
it to be a GOOD question?
– Distracter One / Two / Three equal # of selections
– Throw away BAD questions?
TOO HARD .. ?
• Too time consuming
• Often needs to be an iterative process
• Does it matter if a question is NOT
perfect?
• Seeking perfection not feasible
• I suggest that assessment MUST be
included within a module, not only at end
> make it embedded within TEACHING &
LEARNING experience
• Formative / Summative / Both
My Entry into CAA
• 1st Attempt TOTAL FAILURE – 1998 OLAL
• 5 minutes all finished apart from one student
• Guessing
– Why should the student care if incorrect?
– How to make it worthwhile – slow students down –
make them think
• Negative marking used +2 and -1 (actually
becomes -3 –> double whammy!)
• Allow TWO PASSES of the test
• By concentrating on first pass
– will get reward on second pass
– Formative
• IMPORTANT – WORTH MARKS
Own Background =
Objective Testing via MCQ
• Four Tests MCQ .. Two passes 50/50
(became graduated)
• Cumulative Tests 5%, 10%, 15%, 20%
• Amateur .. No Research Pedigree (CAA
Conference)
• Improved Class results > not easier –
more directed
• Weaker students improved dramatically
• Assessed the learning outcomes of subject
• “END JUSTIFIES THE MEANS”
What did the students say
about it?
• Excellent way of gaining knowledge
• 2nd Test Allowed us to learn
• Good assistant to learning
• Encourages Revision
• Very good, promotes learning of a
subject early on
• Better than sex!!!!
What decides the QUALITY
of a question?
• In the 2004 World cup qualifier
between England and Greece, who
scored the all important equalizing
goal?
• David Beckham
• Victoria Beckham
• Brooklyn Beckham
• Romeo Beckham
Rewards?
• Know it
• On the tip of my tongue
• If I see the answers I’ll know it
• Deduction
• Guess
• Know it … oh no I’ve got it wrong ..
misinformation
TRIED ONE CONFIDENCE TEST
ONLY SHOWS ANSWERS ONCE CONFIDENCE SELECTED
RELATES TO MARKS +4, +2, +1, -2, -1, 0
Proportion of Questions
Correct
• High Confidence => 84% correct
• Fairly Confident => 58% correct
• Not confident => 47% correct
• 72% of all questions answered with
high confidence
Student thoughts on
Confidence
• “Eliminates guesswork”
• “I was waiting for a leggy blond to
bring on my prize at the end”
• “separated the lucky students from
me”
• “at least I didn’t lose marks for
selecting no confidence”
• “are you testing my confidence or
knowledge”
“Bite The Bullet”
• Use Confidence Testing for all four tests
(10%, 15%, 20% & 25%)
• Real SHOCK to students
• Weighting of Passes (50/50, 60/40,
70/30 & 80/20)
• How to judge if the students had ‘really’
learned?
• Select five questions that were
answered ‘poorest’ in test one & pass
through to test two
• Select five questions that were
answered ‘poorest’ in test two & pass
through to test three ( + original five)
Identifying Quality of a
Question
• Negative on first pass e.g. > -180
• Difference between first and second
pass
• Too big an improvement may just
signify a very poor question e.g.
>500?
• Create heuristic that identifies
degree of difficulty of a question
Passed through to Test Four
• 13 of the ‘hardest’ question
• 6 new questions based upon
additional work between tests 3-4
• 21 questions (cross section of
questions and topics from previous
tests)
Improvement of Results
Through Tests
Questions Through Tests
400
300
Question 5
200
Question 6
Question 10
100
Question 11
Question 14
0
Question 15
1
-100
2
3
4
5
6
7
8
Question 16
Question 19
Question 20
-200
Question 21
Question 26
-300
Question 27
Question 40
-400
-500
Improvement Pass One of
Questions
Pass One Question Improvements
400
300
Question 5
200
Question 6
Question 10
100
Question 11
Question 14
0
Question 15
1
-100
2
3
4
Question 16
Question 19
Question 20
-200
Question 21
Question 26
-300
Question 27
Question 40
-400
-500
Improvement for Question 5
• Stronger Students improved quickly
through the tests
• Became more confident in answering
the questions
• Weaker Students improved less
quickly than stronger through the
test progress, but DID improve
Is it just Confidence or Knowledge
i.e. questions right or wrong?
Category
Average # of Questions
Correct Pass 1
Average # of Questions
Correct Pass 2
Average Total # of
Questions Correct
(1)
131.0
158.0
289.0
75-79 (2)
123.5
150.5
274.0
70-74 (3)
117.6
148.6
266.2
65-69 (4)
113.8
150.7
264.5
60-64 (5)
110.3
145.0
255.3
55-59 (6)
107.6
143.4
251.0
50-54 (7)
100.4
136.2
236.6
45-49 (8)
92.5
132.6
225.1
40-44 (9)
94.0
126.4
220.4
35-39 (10)
87.7
121.6
209.3
30-34 (11)
80.5
117.3
197.8
25-29 (12)
76.7
117.3
194.0
80>
Quality of a multiple choice
question?
• In the 2004 World cup qualifier
between England and Greece, who
scored the all important equalizing
goal?
• Teddy Sheringham
• Michael Owen
• Robbie Fowler
• QUALITY OF QUESTION NOT
DISTRACTERS
Developmental v
Assessment
• Need not be at alternative ends of
the scale
• Student motivation / attendance
increased considerably
• Testing knowledge (not done at
moment for higher order skills)
• Good students have best results i.e.
number of questions correct
COMMENTS?
• HMCQ
• Confidence
• Two Pass
• Formative & Summative
• Graduated Assessment
• Continual Assessment