Presented to: Majmaah University
Neil Wilkinson

Creating better test items

The FACTS of good test items
C
T
S
F
A
• Face valid
• Accurate
• Consistent with current practice
• Test blueprint representative
• Statistically appropriate

Creating good test items
Right
Expertise
Sound
Processes
Efficient
Tools xxx

Right
Expertise
Sound
Processes
Subject matter expertise
• ensures test is valid
Content development expertise
• ensures test is superiorly crafted
Psychometric expertise
• ensures test is reliable and fair
Efficient
Tools xxx

Right
Expertise
Sound
Processes
Efficient
Tools
Write items
Gap analysis of item bank
Review of field test results
Review & approve items
Field test items xxx

Sound item writing processes
Assignments
•Gap analysis
Training
Review procedures
•Good item writing practices
•Style guidelines
•Cultural sensitivity
•Intellectual property rights
•Confidentiality
•Editorial review
•Content review
•Acceptance criteria
Item performance monitoring
•Feedback mechanisms

Item performance analysis and feedback
Why analyze items?
•
Statistical behavior of “bad” items is fundamentally different from that of “good” items
•
Provides quality control indicating items which should be reviewed by content experts
•
Feedback on how items are performing can be used in future item writing efforts providing continuous quality improvement in the item writing process

Distractor analysis

High-scoring candidates should select the correct option

Look at discrimination for each distractor

Low-scoring candidates should select from among the distractors at random
 Look at p-values for each distractor
Option: A
KEY
B C D E
N 241 9 3 11 70
Mean 219.63
215.11
196.67
210.73
210.03
p-value
PtBis
0.72
+0.22
0.03
-0.02
0.01
-0.10
0.03
-0.06
0.21
-0.19
Total
334

Diagnosing potential problems
Option:
N
A
2
B
2
C
KEY
113
D
23
E
194
Mean 233.00
177.00
215.08
215.48
218.54
p-value 0.01
0.01
0.34
0.07
0.58
PtBis +0.06
-0.16
-0.07
-0.02
+0.09
Total
334

Feedback into the item development process
•
•
Sharing performance statistics to authors of individual items may be problematic due to time between authoring and use, but...
Item performance statistics can still inform the item development process.
•
•
Share and discuss item performance and reasons for poorly performing items.
Show examples of good, average, and poorly performing items.

Right
Expertise
Sound
Processes
Efficient
Tools xxx
Item Bank
A repository of items with their associated metadata & historical record

Right
Expertise
Sound
Processes
Efficient
Tools
Item Bank – Data
• The item itself
• Item author
• Item type
• Content area reference from test blueprint
• Cognitive level
• Revision history
• Item status
• Performance statistics xxx

Right
Expertise
Sound
Processes
Efficient
Tools xxx
Item Bank – Features
Item authoring
• Remote item authoring
• Support of multiple item types
Item banking
• Search capabilities
• Import/export capabilities
• Batch editing capabilities
Test construction
• Test assembly
• Export features
Ancillary features
• Security and access
• Workflow management
• Project tracking

Right
Expertise
Sound
Processes
Efficient
Tools
Effective
Item
Development
Guidelines for an effective item development process
An effective item development process begins well before the first test item is written with development of test blueprints.

Right
Expertise
Sound
Processes
Efficient
Tools
Effective
Item
Development
Guidelines for an effective item development process
Carefully consider the associated information that is required and stored for each item.

Right
Expertise
Sound
Processes
Efficient
Tools
Effective
Item
Development
Guidelines for an effective item development process
Recognize that item development requires more than subject matter expertise.

Right
Expertise
Sound
Processes
Efficient
Tools
Effective
Item
Development
Guidelines for an effective item development process
Item authors require training in item writing.

Right
Expertise
Sound
Processes
Efficient
Tools
Effective
Item
Development
Guidelines for an effective item development process
Item development requires more than authoring, sound review processes are also essential.

Right
Expertise
Sound
Processes
Efficient
Tools
Effective
Item
Development
Guidelines for an effective item development process
Style and grammar guidelines and review processes are necessary to ensure items are testing what they are meant to test.

Right
Expertise
Sound
Processes
Efficient
Tools
Effective
Item
Development
Guidelines for an effective item development process
Use item performance analysis to guide further review of the items and to provide feedback to item authors.

Writing Good Multiple-Choice Test Items

What Makes a Content-Appropriate Item
•
•
•
•
•
•
•
Reflects specific content
Able to be classified on the test blueprint
Does not contain trivial content
Independent from other items
Not a trick question
Vocabulary suitable for candidate population
Not opinion-based
Source: Haladyna, T.M. (1996). Writing Test Items to Evaluate Higher Order Thinking . Boston, USA: Allyn & Bacon.

Anatomy of a Multiple-Choice Item
Which city has been awarded the 2016 Summer Olympics?
Stem
A. Chicago
B. Madrid
C. Rio de Janeiro
D. Tokyo
Key
Answer Options

Not opinion based
•
Avoid the use of opinion based material in items
• Poor Stem:
Which are the best football team in the world?
a. Barcelona b. Manchester United.
c. Bayern Munich d. Athletico Madrid
Better Stem:
Which football team won the European championship in
1999?

Guidelines for Writing the Stem
Place most of the phrasing in the stem.
• Poor Stem:
Type II diabetes is a. b. c. d.
also called juvenile-onset.
characterised by insulin dependency.
primarily seen in adults over 40.
often managed by drug therapy.
• Better Stem:
What is the most frequent age of onset for Type II diabetes?

Guidelines for Writing the Stem
Avoid excessive, unnecessary wording.
• Poor Stem:
Certified individuals are required to undertake continuing education. What is the minimum number of hours required in a two-year period?
a. 10 b. 15 c. 20 d. 25
• Better Stem:
How many hours of continuing education must a certified individual take in a two-year period?

Guidelines for Writing the Stem
•
Clearly describe the problem or what is being asked of the candidate.
•
Poor Stem:
Continuing education is a. required every two years.
b. important to life-long learning.
c. an optional activity.
d. undertaken at certified centers.
Better Stem:
How many hours of continuing education is a certified individual required to undertake in a 5-year cycle?

Guidelines for the Answer Options
•
•
•
•
•
•
•
•
Maintain similarity in the answer options.
Keep the answer options relatively equal in length.
Keep all answer options grammatically correct with the stem and parallel.
Avoid cueing the right answer.
Avoid words such as “always” and “never”.
Avoid repetitive wording in the answer options.
Vary location of key.
Place answer options in logical order.

Number of Answer Options
Four is most common.
• Recent research indicates three may be sufficient if the two distractors are strong and plausible.
• Rarely have three functional distractors.
• “Using more options does little to improve item and test score statistics and typically results in implausible distractors.”*
*Rodriquez, M.C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research.
Educational Measurement: Issues and Practice, 24(2), 3-13 (p. 11).

Ten steps to an Effective Item Writing Process

Ten Steps to an Effective Item Writing Process
Use item bank inventories to direct item writing activities.

Ten Steps to an Effective Item Writing Process
Provide a training program for item writers.

Ten Steps to an Effective Item Writing Process
Develop a style guide .

Ten Steps to an Effective Item Writing Process
Require all items to be classified according to the test blueprint.

Ten Steps to an Effective Item Writing Process
Ensure all items are validated with current references and the validations are recorded.

Ten Steps to an Effective Item Writing Process
Editorially review all items for grammar and style.

Ten Steps to an Effective Item Writing Process
Include an item review process by Subject
Matter Experts (SMEs).

Ten Steps to an Effective Item Writing Process
Share acceptance criteria with item writers and item reviewers.

Ten Steps to an Effective Item Writing Process
Use statistical analysis of items to guide further review.

Ten Steps to an Effective Item Writing Process
Provide feedback to item writers.

Thank You!