Mastering the Art of
Writing Objective Test Items
Narrowing the Gulf
Annual Conference 2010
March 2010
Writing Objective Test Items
Mastering the Art of Writing Objective Test Items
March 2010
January
2010
Presenter
 Dr. James Coraggio, Director, Academic
Effectiveness and Assessment
Former Life…
 Director of Test Development , SMT
 Director of Measurement and Test Development,
Pearson
 Taught EDF 4430 Measurement for Teachers,
USF
March 19. 2010
Academic Effectiveness and Assessment
2
Purpose
Mastering the Art of Writing Objective Test Items
March 2010
 Are your students learning the course content
or are they just good test takers?
January 2010
 This presentation will explain how to create
effective multiple choice test questions.
 The presentation will provide item-writing
guidelines as well as best practices to prevent
students from just guessing the correct answers.
March 19. 2010
Academic Effectiveness and Assessment
3
Agenda
Mastering the Art of Writing Objective Test Items







March 2010
Propose of a Test
Advantages of Objective Tests
Types of Objective tests
Writing Multiple Choice Items
The Test-wise Student
Test Instructions
Test Validity
January 2010
March 19. 2010
Academic Effectiveness and Assessment
4
Purpose of a Test
Mastering the Art of Writing Objective Test Items
March 2010
 ‘Clearly delineate between those that know the
content and those that do not.’
 Purpose of an assessment is to determine
whether the student knows the content, not
whether the student is a good test-taker.
 Likewise, confusing and tricky questions
should be avoided to prevent incorrect
responses from students who know (and
understand) the material.
January 2010
March 19. 2010
Academic Effectiveness and Assessment
5
Objective Tests
Mastering the Art of Writing Objective Test Items
March 2010
January
2010
 Measure




several types of learning (also levels)
Wide content, short period of time
Variations for flexibility
Easy to administer, score, and analyze
Scored more reliability and quickly
 What type of learning cannot be measured?
March 19. 2010
Academic Effectiveness and Assessment
6
Types of Objective Tests
Mastering the Art of Writing Objective Test Items
March 2010
 Written-response
January 2010
 Completion (fill-in-the-blank)
 Short answer
 Selected-response




Alternative response (two options)
Matching
Keyed (like matching)
Multiple choice
March 19. 2010
Academic Effectiveness and Assessment
7
Written-response
Mastering the Art of Writing Objective Test Items
March 2010
January
 Single2010
questions/statements or clusters (stimuli)
 Advantages
 Measure several types of learning
 Minimizes guessing
 Points out student misconceptions
 Disadvantages
 Time to score
 Misspelling and writing clarity
 Incomplete answers
 More than one possible correct response (novel answers)
 Subjectivity in grading
March 19. 2010
Academic Effectiveness and Assessment
8
Completion
Mastering the Art of Writing Objective Test Items
March 2010
AJanuary
word2010
that describes a person, place or thing is a
________.
1.
2.
3.
4.
5.
6.
Remove only ‘key’ words
Blanks at end of statement
Avoid multiple correct answers
Eliminate clues
Paraphrase statements
Use answer sheets to simplify scoring
March 19. 2010
Academic Effectiveness and Assessment
9
Short Answer
Mastering the Art of Writing Objective Test Items
March 2010
Briefly describe the term proper noun.
____________________________
January 2010

Terminology – Stimulus and Response
1. Provide an appropriate blank (word (s) or
sentence).
2. Specify the units (inches, dollars)
3. Ensure directions for clusters of items and
appropriate for all items
March 19. 2010
Academic Effectiveness and Assessment
10
Selected-response
Mastering the Art of Writing Objective Test Items
March 2010
Januaryfrom
2010 provided responses
Select
 Advantages
 Measure several types of learning
 Measures ability to make fine distinctions
 Administered quickly
 Cover wide range of material
 Reliably scored
 Multiple scoring options (hand, computer, scanner)
 Disadvantages
 Allows guessing
 Distractors can be difficult to create
 Student misconceptions not revealed
March 19. 2010
Academic Effectiveness and Assessment
11
Alternative Response
Mastering the Art of Writing Objective Test Items
March 2010
January
T F 2010
1. A
noun is a person place or thing.
T F 2. An adverb describes a noun.
1.
2.
3.
4.
5.
6.
Explain judgments to be made
Ensure answers choices match
Explain how to answer
Only one idea to be judged
Positive wording
Avoid trickiness, clues, qualifiers
March 19. 2010
Academic Effectiveness and Assessment
12
Matching Item
Mastering the Art of Writing Objective Test Items
January 2010
Column A
March 2010
Column B
__Person, place, or thing.
__Describes a person, place, or thing.
a. Adjective
b. Noun
Terminology – premises and responses
1. Clear instructions
2. Homogenous premises
3. Homogenous responses (brief and
ordered)
4. Avoid one-to-one
March 19. 2010
Academic Effectiveness and Assessment
13
Keyed Response
Mastering the Art of Writing Objective Test Items
March 2010
January
2010
Responses
a. A noun
b. A pronoun
c. An adjective
d. An adverb
___Person, place, or thing.
___Describes a person, place, or thing.
 Like matching items, more response options
March 19. 2010
Academic Effectiveness and Assessment
14
MC Item Format
Mastering the Art of Writing Objective Test Items
March 2010
January 2010
What is the part of speech that is used to name a person,
place, or thing?
A) A noun*
B) A pronoun
C) An adjective
D) An adverb
March 19. 2010
Academic Effectiveness and Assessment
15
MC Item Terminology
Mastering the Art of Writing Objective Test Items
March 2010
 Stem: Sets the stage for the item; question or
incomplete thought; should contain all the
needed information to select the correct
response.
 Options: Possible responses consisting of one
and only one correct answer.
 Key: correct response
 Distractor: wrong response, plausible, but not
correct, attractive to an under-prepared student
January 2010
March 19. 2010
Academic Effectiveness and Assessment
16
Competency
Mastering the Art of Writing Objective Test Items
March 2010
 Items should test for the appropriate or
adequate level of knowledge, skill, or ability
(KSA) for the students.
 Assessing lower division students on graduate
level material is an ‘unfair’ expectation.
 The competent student should do well on an
assessment, items should not be written for only
the top students in the class.
January 2010
March 19. 2010
Academic Effectiveness and Assessment
17
Clarity
Mastering the Art of Writing Objective Test Items






March 2010
Clear, precise item and instruction
Correct grammar, punctuation, spelling
Address one single issue
Avoid extraneous material (teaching)
One correct or clearly best answer
Legible copies of exam
January 2010
March 19. 2010
Academic Effectiveness and Assessment
18
Bias
Mastering the Art of Writing Objective Test Items
March 2010
 Tests should be free from bias…
January 2010






No stereotyping
No gender bias
No racial bias
No cultural bias
No religious bias
No political bias
March 19. 2010
Academic Effectiveness and Assessment
19
Level of Difficulty
Mastering the Art of Writing Objective Test Items
March 2010
 Ideally, test difficulty should be aimed a
middle level of difficulty. This can not
always be achieved when the subject
matter is based on specific expectations
(i.e, workforce area).
January 2010
March 19. 2010
Academic Effectiveness and Assessment
20
Level of Difficulty
Mastering the Art of Writing Objective Test Items
March 2010
 To make a M/C item more difficult, make
the stem more specific or narrow and the
options more similar.
 To make a M/C item less difficult, make
the stem more general and the options
more varied.
January 2010
March 19. 2010
Academic Effectiveness and Assessment
21
Trivial and Trick Questions
Mastering the Art of Writing Objective Test Items
March 2010
 Avoid trivia and tricks
 Avoid humorous or ludicrous responses
 Items should be straight forward, they
should cleanly delineate those that know
the material from those that do not
 Make sure every item has value and that it
is contributing to the final score
January 2010
March 19. 2010
Academic Effectiveness and Assessment
22
http://www.services.unimelb.edu.au/asu/download/Study-Multiple-ChoiceExams-Flyer.pdf
Test Taking Guidelines
Mastering the Art of Writing Objective Test Items
March 2010
January
When2010
you don’t know the answer
 As with all exams, attempt the questions that are easiest for you
first. Come back and do the hard ones later. Unless you will lose
marks for an incorrect response, never leave a question blank. Make
a calculated guess if you are sure you don’t know the answer. Here
are some tips to help you guess ‘intelligently’.
Use a process of elimination
 Try to narrow your choice as much as possible: which of the options
is most likely to be incorrect? Ask: are options in the right range? Is
the measurement unit correct? Does it sound reasonable?
March 19. 2010
Academic Effectiveness and Assessment
23
http://www.services.unimelb.edu.au/asu/download/Study-Multiple-ChoiceExams-Flyer.pdf
Test Taking Guidelines
Mastering the Art of Writing Objective Test Items
March 2010
January
2010
Look for
grammatical inconsistencies
 In extension-type questions a choice is nearly always wrong if the question
and the answer do not combine to make a grammatically correct sentence.
Also look for repetition of key words from the question in the responses. If
words are repeated, the option is worth considering. e.g.:
 The apparent distance hypothesis explains…
 b) The distance between the two parallel lines appears…
March 19. 2010
Academic Effectiveness and Assessment
24
http://www.services.unimelb.edu.au/asu/download/Study-Multiple-ChoiceExams-Flyer.pdf
Test Taking Guidelines
Mastering the Art of Writing Objective Test Items
March 2010
January
2010
Be wary
of options containing definitive words and generalizations
 Because they can’t tolerate exceptions, options containing words like
‘always’, ‘only’, ‘never’, ‘must’ tend to be incorrect more often. Similarly,
options containing strong generalizations tend to be incorrect more often.
Favor look-alike options
 If two of the alternatives are similar, give them your consideration. e.g.:
A. tourism consultants
B. tourists
C. tourism promoters
D. fairy penguins
March 19. 2010
Academic Effectiveness and Assessment
25
http://www.services.unimelb.edu.au/asu/download/Study-Multiple-ChoiceExams-Flyer.pdf
Test Taking Guidelines
Mastering the Art of Writing Objective Test Items
March 2010
January
2010
Favor numbers
in the mid-range
 If you have no idea what the real answer is, avoid extremes.
Favor more inclusive options
 If in doubt, select the option that encompasses others. e.g.:
A. an adaptive system
B. a closed system
C. an open system
D. a controlled and responsive system
E. an open and adaptive system.
Please note: None of these strategies is foolproof and they do not apply
equally to the different types of multiple choice questions, but they are
worth considering when you would otherwise leave a blank.
March 19. 2010
Academic Effectiveness and Assessment
26
Test-wise Students
Mastering the Art of Writing Objective Test Items






March 2010
Are familiar with item formats
Use informed and educated guessing
Avoid common mistakes
Have testing experience
Use time effectively
Apply various strategies to solve different
problem types
January 2010
March 19. 2010
Academic Effectiveness and Assessment
27
Test-wise Students
Mastering the Art of Writing Objective Test Items
March 2010
 Vary your keys: ‘Always pick option ‘C’ ’
 Avoid ‘all of the above’ and ‘none of the
above’
 Avoid extraneous information: It may
assist in answering another item
 Avoid item ‘bad pairs’ or ‘enemies’
 Avoid clueing with the same word in the
stem and the key
January 2010
March 19. 2010
Academic Effectiveness and Assessment
28
Test-wise Students
Mastering the Art of Writing Objective Test Items
March 2010
 Make options similar in terms of length,
grammar, and sentence structure.
Different options stand out. Avoid ‘clues’.
January 2010
March 19. 2010
Academic Effectiveness and Assessment
29
Item Format Considerations
Mastering the Art of Writing Objective Test Items
March 2010
Information in the stem
Avoid negatively stated stem, qualifiers
Highlight qualifiers if used
Avoid irrelevant symbols (“&”) and jargon
Standard set number of options (Prefer
only four)
 Ideally, you should tie an item to
reference





January 2010
March 19. 2010
Academic Effectiveness and Assessment
30
Test Directions
Mastering the Art of Writing Objective Test Items
March 2010
January 2010 Directions
Highlight
1.
2.
3.
4.
5.
State the skill measured.
Describe any resource materials required.
Describe how students are to respond.
Describe any special conditions.
Time limits, if any
March 19. 2010
Academic Effectiveness and Assessment
31
Ensure Test Validity
Mastering the Art of Writing Objective Test Items
March 2010
2010
January
Congruence
between items and course
objectives
 Congruence between item and student
characteristics
 Clarity of items
 Accuracy of the measures
 Item formatting criteria
 Feasibility-time, resources
March 19. 2010
Academic Effectiveness and Assessment
32
Questions
Mastering the Art of Writing Objective Test Items
March 2010
January 2010
March 19. 2010
Academic Effectiveness and Assessment
33
Mastering the Art of
Writing Objective Test Items
Narrowing the Gulf
Annual Conference 2010
March 2010