Writing Multiple Choice Examination Items
Workshop Presented at Western Washington University
April 14, 2008
Douglas A. Bernstein
University of South Florida
Bibliography
Attali, Y. & Bar-Hillel, M. (2003). Guess where: The position of correct answers in multiple-choice test items as a
psychometric variable. Journal of Educational Measurement, 40, 109-128.
Bleske-Rechek, A., Zeug, N., & Webb, R.M. (2007). Discrepant performance on multiple-choice and short answer assessments
and the relation of performance to general scholastic aptitude. Assessment & Evaluation in Higher Education, 32, 89-105.
Bridgeman, B. & Rock, D.A. (1993). Relationships among multiple-choice and open-ended analytical questions. Journal of
Educational Measurement, 30, 313-329.
Bridgeman, B. & Morgan, R. (1996). Success in college for students with discrepancies between performance on multiplechoice and essay tests. Journal of Educational Psychology, 88, 333-340.
Bridgeman, B. & Lewis, C. (1994). The relationship of essay and multiple-choice scores with grades in college courses. Journal
of Educational Measurement, 31, 37-50.
Bruno, J.E., & Dirkzwager, A. (1995). Determining the optimal number of alternatives to a multiple-choice test item: An
information theoretic perspective. Educational and Psychological Measurement, 55, 959-966.
Butler, A.C., Marsh, E.J., Goode, M.K., & Roediger, H.L. III. (2006). when additional multiple-choice lures aid versus hinder
later memory. Applied Cognitive Psychology, 20, 941-956.
Chase, C.I. (1964). Relative length of option and response set in multiple choice items. Educational and Psychological
Measurement, 24, 861-866.
Crehan, K.D., Haladyna, T.M., & Brewer, B.W. (1993). Use of an inclusive option and the optimal number of options for
multiple-choice items. Educational and Psychological Measurement, 53, 241-247.
Delgado, A.R. & Prieto, G. (1998). Further evidence favoring three-option items in multiple-choice tests. European Journal of
Psychological Assessment, 14, 197-201.
Downing, S.M. (2002). Construct-irrelevant variance and flawed test questions: Do multiple-choice item-writing principles
make any difference? Journal of the Association of American Medical Colleges, 77, Supplement, p. S103–S104.
Droegemueller, W., Gant, N., Brekken, A., & Webb, L. (2005). Comedy workshop: An enjoyable way to develop multiplechoice questions. Teaching and Learning in Medicine, 17, 290-291.
Ellsworth, R.A. Dunnell, P., & Duell, O.K. (1990). Multiple-choice test items: What are textbook authors telling teachers?
Journal of Educational Research, 83, 289-293.
Fellenz, M.R. (2004). Using assessment to support higher level learning: The multiple choice item development assignment.
Assessment & Evaluation in Higher Education, 29, 703-719.
Haladyna, T.M. (1999). Developing and validating multiple-choice test items (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
Haladyna, T.M. & Downing, S.M. (1993). How many options is enough for a multiple-choice test item? Educational and
Psychological Measurement, 53, 999-1010.
Haladyna T.M., Downing S.M., & Rodriguez, M.C. (2002). A review of multiple-choice item-writing guidelines. Applied
Measurement in Education, 15, 309–33.
Hancock, G.R. (1994). Cognitive complexity and the comparability of multiple-choice and constructed-response test formats.
Journal of Experimental Education, 62, 143-157.
Harasym, P.H., Doran, M.L., Brant, R., & Lorscheider, F.L. (1993). Negation in stems of single-response multiple-choice items:
An overestimation of student ability. Evaluation & the Health Professions, 16, 342-357.
Huang, Y.M. (2005). The impact of the "all-of-the-above" option and student ability in multiple choice testing. Dissertation
Abstracts International Section A: Humanities and Social Sciences, 65, 3277.
Hughes, H.H. & Trimble, W.E. (1965). The use of complex alternatives in multiple choice items. Educational and
Psychological Measurement, 25, 117-126.
Knowles, S.L. & Welch, C.A. (1992). A meta-analytic review of item discrimination and difficulty in multiple-choice items
using "none-of-the-above." Educational and Psychological Measurement, 52, 571-577.
Landrum, R. E., Cashin, J.R., & Theis, K.S. (1993). More evidence in favor of three-option multiple-choice tests. Educational
and Psychological Measurement, 53, 771-778.
Lukhele, R., Thissen, D., & Wainer, H. (1994). On the relative value of multiple-choice, constructed response, and examineeselected items on two achievement tests. Journal of Educational Measurement, 31, 234-250.
Marsh, E.J., Roediger, H.L. III, Bjork, R.A., & Bjork, E.L. (2007). The memorial consequences of multiple-choice testing.
Psychonomic Bulletin & Review., 14, 194-199.
Neely, D.L., Springston, F.J., & McCann, S.J.H. (1994). Does item order affect performance on multiple choice exams?
Teaching of Psychology, 21, 44-45.
Owen, S.V. & Froman, R.D. (1987). What's wrong with three-option multiple choice items? Educational and Psychological
Measurement, 47, 513-522.
Rodriguez, M.C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research.
Educational Measurement: Issues and Practice 24 , 3–13.
Roediger, H.L. III & Marsh, E.J. (2005). The Positive and Negative Consequences of Multiple-Choice Testing. Journal of
Experimental Psychology: Learning, Memory, and Cognition, 31, 1155-1159.
Sidick, J.T., Barrett, G.V., & Doverspike, D. (1994). Three-alternative multiple choice tests: An attractive option. Personnel
Psychology, 47, 829-835.
Skinner, N.F. (1999). When the going gets tough, the tough get going: Effects of item difficulty on multiple-choice test
performance. North American Journal of Psychology, 1, 79-82.
Stupans, I. (2006). Multiple choice questions: Can they examine application of knowledge? Pharmacy Education, 6, 59-63.
Taylor, A.K. (2005). Violating conventional wisdom in multiple choice test construction. College Student Journal, 39, 141-148.
Traub, R.E. (1993). On the equivalence of the traits assessed by multiple-choice and constructed-response tests. In R.E. Bennett
& W.C. Ward (Eds). (1993). Construction versus choice in cognitive measurement: Issues in constructed response,
performance testing, and portfolio assessment. (pp. 29-44). Hillsdale, NJ: Lawrence Erlbaum.
Trevisan, M.S., Sax, G., & Michael, W.B. (1991). The effects of the number of options per item and student ability on test
validity and reliability. Educational and Psychological Measurement, 51, 829-837.