New results from examinations
based on multiple-choice
questions (MCQs)
Klaus Bothe, Michael Ritzschke
Humboldt-Universität zu Berlin
Department of Computer Science
Software Engineering
14th Workshop “Software Engineering Education and Reverse Engineering”
Sinaia, Romania, 24 – 30 August 2014
Contents





Overview: MCQs over the years at HU
MCQs – advantages for correctors
MCQs – different kinds
Only MCQs?
Conlusions
Sinaia, Romania, 24 – 30 August 2014
2
Software Engineering at HU




Until 2010: SE facultative at Diploma studies (20 – 50
students): oral examinations
Since 2011: SE obligatory at Bachelor studies (90 – 100
students): written exams, correction time-consuming
2014: 5 correctors (staff and student tutors) each 12
hours (1 ½ days)
Three types of questions in written exams:
knowledge, MCQs, skills
3
Types of questions in 2014:
Again more multiple choice than in 2013
Bansko, Bulgaria, 26 – 31 August 2013
4
Development of using
multiple choice questions
Sinaia, Romania, 24 – 30 August 2014
5
Contents





Overview: MCQs over the years at HU
MCQs – advantages for correctors
MCQs – different kinds
Only MCQs?
Conlusions
Sinaia, Romania, 24 – 30 August 2014
6
Advantages of MCQs
Why?


Advantage: MCQs easier and faster to correct
MCQs can be corrected by non-professionals or
automatically
Sinaia, Romania, 24 – 30 August 2014
7
Example (WS 11/12):
Free answer question and
hand-written answers of students
a)
b)
c)
Relation between structured vs. functional testing
What is the difference between functional testing and structured testing?
What are the advantages and the disadvanteges of the two approaches?
How is a shared use of both approaches possible?
Example (WS 11/12):
Free answer question and
hand-written answers of students
a)
b)
c)
Relation between structured vs. functional testing
What is the difference between functional testing and structured testing?
What are the advantages and the disadvanteges of the two approaches?
How is a shared use of both approaches possible?
Example (WS 13/14):
MC question replacing the free answer
question from WS 2011/12
Relation between structured vs. functional testing
Sinaia, Romania, 24 – 30 August 2014
10
Example (WS 13/14):
MC question replacing the free answer
question from WS 2011/12
Relation between structured vs. functional testing
Sinaia, Romania, 24 – 30 August 2014
11
Contents





Overview: MCQs over the years at HU
MCQs – advantages for correctors
MCQs – different kinds
Only MCQs?
Conlusions
Sinaia, Romania, 24 – 30 August 2014
12
MCQs at HU are only binary questions
Relation between structured vs. functional testing
Sinaia, Romania, 24 – 30 August 2014
13
MC: Theory
If we look to the theory of MC we find many different compositions:
MC choice select, MC choose, MC check, MC binary, MC combined …
check
choose
which answers
are accurate?
which answer
are correct?
select
find the 2
accurate answers
Sinaia, Romania, 24 – 30 August 2014
binary
are the answers
accurate?
14
MCQ: combined answers
An example from:
IEEE Computer Society Real-World
Software Engineering Problems: A Self-Study Guide for Today's
Software Professional
J. Fernando Naveda, Stephen B. Seidman
ISBN: 978-0-471-71051-6
328 pages, August 2006, Wiley-IEEE Computer Society Press
15
MCQ: combined answers
IEEE Computer Society Real-World Software Engineering Problems:
A Self-Study Guide for Today's Software Professional
J. Fernando Naveda, Stephen B. Seidman
ISBN: 978-0-471-71051-6
328 pages, August 2006, Wiley-IEEE Computer Society Press
Key problems for the IEEE Computer Society Certified
Software Development Professional (CSDP) Certification Program
IEEE Computer Society Real-World Software Engineering Problems helps prepare
software engineering professionals for the
IEEE Computer Society Certified Software Development Professional (CSDP)
Certification Program. The book offers workable, real-world sample problems
with solutions to help readers solve common problems.
In addition to its role as the definitive preparation guide for the IEEE Computer Society
Certified Software Development Professional (CSDP) Certification Program, this resource also
serves as an appropriate guide for graduate-level courses in software engineering or for
professionals interested in sharpening or refreshing their skills.
The book includes a comprehensive collection of sample problems,
each of which includes the problem's statement, the solution, an explanation, and references.
16
MCQ: Combined answers
Source:
Naveda, Seidman (IEEE)
Question: The use of capture replay tools GUI interface testing suffers from
which of the following problems:
I.
II.
III.
IV.
a)
b)
c)
d)
The execution of the test cases cannot be automated.
Test cases may break if widgets are moved in the interface.
Good test cases must be created and recorded manually.
The test cases cannot be reused if the implementation of the
application ist changed while leaving the interface unchanged.
I and III only
I, II and III only
II and III only
III and IV only
Problem:
higher challenges to students
Sinaia, Romania, 24 – 30 August 2014
17
Contents





Overview: MCQs over the years at HU
MCQs – advantages for correctors
MCQs – different kinds
Only MCQs?
Conlusions
Sinaia, Romania, 24 – 30 August 2014
18
Exams completely based on MCQs
Why?
 Still more efficient
HU in 2014: 5 correctors: together 50 hours



Advantage: MCQs easier and faster to correct
MCQs can be corrected by non-professionals or
automatically
Question: Will an exam completely based on
MCQs properly reflect students performance?
Sinaia, Romania, 24 – 30 August 2014
19
What happens if we use only MCQs?
(Base: 16 questions 2014, same scale as shown for all questions)
only MCQ‘s
overall
The result cannot satisfy – in each case: The mixture of 3
different kinds of questions has proved successful
Sinaia, Romania, 24 – 30 August 2014
20
Contents





Overview
Three types of examination questions
Different assessments for MCQs
Only MCQs?
Conclusions
Sinaia, Romania, 24 – 30 August 2014
21
Conclusions




There are many references about using MCQs
For instance in /2/ we find a good historical overview and
some reflections for writing good multiple-choice tests,
/3/ presents a checklist for writing effective MCQs …
Writing good multiple-choice tests is difficult
So we favour a mixture of different kinds of questions
/1/ Naveda,J.F., Seidman, St.B.: IEEE Computer Society Real Word Software Engineering
Problems. Jon Wiley & Sons, Inc., Hoboken, New Jersey 2006
/2/ Simon: Wrong is a relative concept: part marks for multiple-choice questions. Proceedings of
the 13th Australasian Computing Education Conference (ACE 2011), Perth, Australia, January
2011. CRPIT, Vol. 114, S. 47-53
/3/ Woodfort,K., Bankroft,P.: Multiple Choice Questions Not Considered harmful. Proceedings of
the 7th Australasian Computing Education Conference (ACE 2005), Newcastle, Australia.
CRPIT, Vol. 42, S. 109-115
Sinaia, Romania, 24 – 30 August 2014
22
Conclusions
Thank you for your attention!
Sinaia, Romania, 24 – 30 August 2014
23