DOES CONTENT ORDER AFFECT PERFORMANCE
ON MULTIPLE CHOICE EXAMS?
1.- Introduction
2.- Theory and hypotheses
3.- Methodology
4.- Results
5.- Conclusion
References
Bresnock, A. E., Graves, P. E. and White, N. 1989. “Multiple-Choice Testing: Question and Response Position.” Journal
of Economic Education, Summer.
Carlson, J. L. and Ostrosky, A. L. 1992. “Item Sequence and Student Performance on Multiple-Choice Exams: Further
Evidence.” Journal of Economic Education, Summer.
Doerner, W. M. and Calhoun, J. P. 2009. “The Impact of the Order of Test Questions in Introductory Economics”.
Economics Educator: Courses, Cases & Teaching, April 20, Economics Research Network (ERN) Working Paper.
Gohmann, S. F. and Spector, L. C. 1989. “Test Scrambling and Student Performance”. Journal of Economic Education,
Summer.
Marín, C. and Rosa-García, A. 2012. "Gender Bias in Risk Aversion: Evidence from Multiple Choice Exams". MPRA
Paper nº 39987, Julio 2012. Online at http://mpra.ub.ini-muenchen.de/39987/."
Sue, D. L. 2006. “The Effect of Test Scrambling on Student Performance.” NBEA Annual ConferenceProceedings.
Sue, D. L. 2009. “The effect of Scrambling test questions on student performance in a small class setting”. Journal for
Economic Educators,vol. 9, n. 1.
Taub, A. J., and Bell, E. B. 1975. “A Bias in Scores on Multiple-Form Exams.” Journal of Economic Education, Fall.
1
DOES CONTENT ORDER AFFECT PERFORMANCE ON MULTIPLE CHOICE EXAMS?
1. INTRODUCTION
• The development of new technologies have facilitated many the roles of
teachers
• It has made possible the realization of test with inmediate feeback and the
ability to shuffle the order of questions and establish different types
• The effects on the student shuffle question has been a hotly debat
• This technique used especially when the number students is high
• The risk behind this technique is the orden in which the questions are
asked
• It necessary to investigate:
• Whether the performance of a test that another can influence
student performance
• What are the variables that can influence more in the grades
2
DOES CONTENT ORDER AFFECT PERFORMANCE ON MULTIPLE CHOICE EXAMS?
2.- THEORY AND HYPOTHESES
•
•
•
•
•
•
•
•
One method of counteracting cheating is to use multiple forms of the same exam
by scrambling test questions
This procedure produces one content-ordere exam that matches the order in
which the material was present in class and one exam that has a random
scrambled order
Previous studes do affirm that students may perform better on a content orderer
exam
Taub and Bell (1975) affirm that students who took the form which questions were
randomly arranged have more test anxiety and they also lose concentration
Other studies showed opposite results (Gohmann and Spector, 1989; Bresnock et
al., 1989)
Marín and Rosa-Garcia evidencing that males tend to answer a higher amount of
questions than females and evidencing a higher degree of risk in males anwer
Hypothesis 1: Student who perform an examination of the type 02 do worse tah
those doing the type 01
Hypothesis 2: The response procedure (right questions, erroneous and white) of
3
student taking the test type 01 differs from those doing the exam type 02
DOES CONTENT ORDER AFFECT PERFORMANCE ON MULTIPLE CHOICE EXAMS?
3.- METHODOLOGY
Sample and data. In order to empirically test the effect of scrambling the content order of multiplechoice questions on a student’s performance and exploring the differences in outputs, we use a
sample of 331 type 01 and 02’s exams done by undergraduate students from the University of Murcia
(Spain) belonging to “Work and Labour Organization” subject of the 2012-13 academic year.
Variables. We have use dependent, independent and moderating variables.
• Dependent variable is student performance, measures in three ways:
1. Quantitative variable in a 0-10 points’ scale, which indicate the score obtained by the student
in the exam;
2. Dummy variable indicating if the students obtain a score below or above average;
3. Qualitative variable indicating the procedure of answer, measurers as the number of correct,
incorrect and blank responses doing by the student in the exam.
• Independent variable is the type of exam, measures through a dummy variable indicating type 01
exam (with ordered questions) and type 02 (with not ordered questions).
• We also have used two segmentations, which lead us to generate two moderating variables:
1. experience of the students, proxies by two variables: the age of students (more or less of 19
years old) and the number of exam’ call student facing (first, second or more);
2. risk taking by the student doing the exam, proxies by the genre of student as a qualitative
variable indicating male or female.
4
DOES CONTENT ORDER AFFECT PERFORMANCE ON MULTIPLE CHOICE EXAMS?
4. RESULTS
Table 1. Analysis of variance (ANOVA) results
Results
Examtype
Right answers
Wrong answers
Blank answers
Scores
01
9,04
7,28
3,68**
3,34
02
01
02
8,82
11,59
11,59
6,76
4,39
4,48
4,42**
4,02
3,92
3,34
5,06
5,05
01
02
6,74
6,24
9,89**
8,87**
3,37***
4,89***
1,78
1,75
01
9,10
7,20
3,70*
3,38
02
01
02
8,93
8,67
8,15
6,79
7,83
6,60
4,28*
3,50
5,25
3,39
3,06
3,01
Firstcall
01
02
9,31
9,16
7,58
6,91
3,11**
3,93**
3,42
3,49
Second call or more
01
02
8,43
8,07
6,60
6,42
4,98
5,51
3,13
2,99
Female
01
02
9,31
8,57
6,82
7,06
3,87
4,37
3,56
3,20
Male
01
02
8,64
9,16
7,96***
6,34***
3,39**
4,5**
3,00
3,52
General comparison
Absolute score 0-10
Best grade
Worse grade
Students experience
A. Students average age
Older students
Younger students
B. Students number of exam’ call facing
Students risk taking by genre
5
DOES CONTENT ORDER AFFECT PERFORMANCE ON MULTIPLE CHOICE EXAMS?
4. RESULTS
Hypothesis 1: Students with a scramble (type 02) exam
have a lower performance than those doing the
unscramble exam (type 01).
HYPOTHESIS 1 -> REJECTED
Hypothesis 2: The response procedure (right questions,
erroneous and white) of students taking the unscramble
(type 02) test differs from those doing the scramble
exam (type 01).
HYPOTHESIS 2 -> ACCEPTED
6
DOES CONTENT ORDER AFFECT PERFORMANCE ON MULTIPLE CHOICE EXAMS?
5. CONCLUSIONS
• There are differences in the procedure of response, but this procedure does
not affect final performance.
•
•
Type 02 exams’ students left more blank answers and have fewer wrong answers.
There aren’t significant differences between the type 01 and 02 exams performance.
• Trying to explain and understand the origin of such differences, we have
segmented the study sample in terms of:
1) Student experience in relation to test examination
•
•
•
In case of older students: type 02 exams’ student left more blank answers.
Second call or more students: No differences in type 01 and 02 exams performance.
First call students: type 02 exams’ students left more blank answers.
2) The student risk taking doing the exam.
•
•
Female group of students: No differences in type 01 and 02 performance exams
Male group of students: In 02 type exams’ students left more blank answers and have
less wrong answers, not affecting final performance.
• General conclusion: older students, first call students, and male student
doing type 02 exams, although they show the same performance than the
others, left more blank answers and have less wrong answers, trying to
secure more their responses, and avoiding penalties (because of wrong
answers), becoming in a more risk adverse students.
7