Lunelle L Pienaar & Gregory Doyle
Education Development Unit,
Faculty of Health Sciences, University of Cape Town
o
Multiple choice format remain one
method of assessment that is most
commonly used to test theoretical
knowledge in medical education (Palmer
EJ and Devitt PG. 2007, Mavis et al,
2001).
o
Increasingly computer based assessment
is used more within the FHS UCT-student
numbers, drive increase IT use
o
R-type (OBA)

o
Traditionally used more often and is
written with question in the form of a
clinical case scenario with 4 or 5 option,
questions are thought to test recall of
knowledge.
A-type (EMI)

The questions are written with a theme
and questions are constructed around the
focus theme (Beullens et al, 2002, Palmer
and Devitt, 2007).
o
EMIs is considered a better format to
test problem solving and integration
of knowledge( Swanson et al, 2008,
Swanson et al, 2006).
o
EMI has demonstrated better
discrimination between students as
well as more difficulty compared to
OBA (McCoubrie 2004).
o
What is the performance of in house
test administered to medical students
during one of their clinical rotations in
5th year.
o What
is the test item difficulty and
discrimination of EMIs and OBA used in
summative assessment.
o Is there any differences between item
difficulty and discrimination between the
EMI and OBA.
o
The questions were administered to all 5th year
students at the end of their clinical block.
o
We evaluated OBA items and extended matching
items that were developed by the 5th year course
lecturer and evaluated as appropriate for testing
by the course convener(not the researcher).
o
The item difficulty and discrimination for the
EMIs and OBA was determined
o
o
o
Results were from the end of block test with
5 groups of students. The same questions
were administered to students all 5 groups of
students.
The 193 students answered a computer based
test containing five EMI sets with 34 items
and 16 (OBA)-Type MCQs.
The OBA -had 4 options with each of the 5
EMIs sets had 9 options with 9 questions.
80.0%
71.0%
70.0%
60.0%
50.0%
50.0%
31.0%
40.0%
30.0%
OBA
EMI
20.5% 19.0%
20.0%
8.8%
10.0%
0.0%
0.0%
0.0%
>75% Easy
50-75 %
Moderate
20-49 %
Difficult
< 20 Very
difficult
69%
70%
60%
52%
47%
50%
40%
30%
OBA
EMI
25%
20%
6%
10%
0%
0%
< 0.20
0.2-0.45
0
Item difficulty
Item Discrimination
Mean
Std Dev
Mean
Std Dev
OBA (N=16)
0.71
0.22
0.24
0.11
EMI (N=34)
0.79
0.19
0.23
0.12
o
The study did not show any statistical
significance between the 2 types of MCQs –
small sample of test items.
o Possible reason for this may also be that writing OBA
format were used more often in the past.
o
o
o
Both EMI and OBA test items wide range of item
difficulties
The results raises 2 issues
when questions are reused to select the best
performing items.
o evaluation of test post administration is important to
determine how to improve quality of the assessment
items.
o
o
Beullens J, Van Damme B, Jaspaert H, Janssen PJ. 2002. Are extended-matching multiple-choice
items appropriate for a final test in medical education? Medical Teacher 24 (4):390-5.
o
Mavis BE, Cole BL, Hoppe RB. 2001 A survey of student assessment in U.S. medical schools: the
balance of breadth versus fidelity. Teaching and Learning in Medicine, 13 (2), 74-79.
o
McCourbrie P. 2004. improving the fairness of multiple- choice questions: a literature review.
Medical Teacher, vol 26. 8 (709-712)
o
Palmer EJ and Devitt PG. 2007. Assessment of higher order cognitive skills in undergraduate
education: modified essay or multiple choice questions? Research paper. BMC Med Educ; 7: 49.
o
Swanson, D.B., Holtzman, K.Z., Allbee, K. 2008. Measurement characteristics of content-parallel
single-best-answer and extended-matching questions in relation to number and source of options.
Journal of the Association of American Colleges, 83(10 Suppl):S21-4
o
Swanson, D.B., Holtzman, K.Z., Allbee, K., Clauser, B.E. 2006. Psychometric characteristics and
response times for content-parallel extended-matching and one-best-answer items in relation to
number of options. Journal of the Association of American Colleges. Abstract 81(10 Suppl):S52-5
http://edusum.edublogs.org/category/thought
s-on-assessment/