Developing Tests and Test
Questions/Items
Why do we evaluate students?
Motivate students
Measure achievement
Identify areas for review
Check effectiveness of materials,
teaching methods, and teacher
Differentiate between different levels of
achievement
Assign grades
Basic Principles of Testing
Validity

Does the test measure what it is
supposed to measure?

Does the test require student
performance as described by a
corresponding objective?
Basic Principles of Testing
Reliability

Would the test generate the same results if
it were given again (assuming no change in
students)?

Is the test affected by misinterpretation?
 Poorly
written directions or instructions
 Vague or tricky questions
 Misleading words
 Not sure of student response
Agenda
1. Aligning evaluation with
level of learning
2. Developing a test blueprint or
“table of specifications”
3. Common pitfalls when writing test
questions
4. Writing questions at various levels
of performance
Defining objectives using
Bloom’s Cognitive Taxonomy
CATEGORY/level
•
Evaluation
•
Synthesis
•
Analysis
SKILL (what students do)
• Judge
• Combine
• Break apart
•
• Apply principles
•
•
Application
Comprehension
Knowledge
• Understanding
• Recall
Sample Verbs for Different
Categories/Levels of Cognition
Knowledge

Cite, define, identify, label, list, match, name,
recognize, reproduce, select, state
Comprehension

Classify, convert, describe, distinguish between,
explain, extend, give examples, illustrate,
interpret, paraphrase, summarize, translate
Application

Apply, arrange, compute, construct,
demonstrate, discover, modify, operate, predict,
prepare, produce, relate, show, solve, use
Sample Verbs continued
Analysis

Analyze, associate, determine, diagram,
differentiate, discriminate, distinguish, estimate,
infer, order, outline, point out, separate, subdivide
Synthesis

Combine, compile, compose, construct, create,
design, develop, devise, formulate, integrate,
modify, organize, plan, propose, rearrange,
reorganize, revise, rewrite, tell, write
Evaluation

Appraise, assess, compare, conclude, contrast,
criticize, discriminate, evaluate, judge, justify,
support, weigh
Agenda
1. Aligning evaluation with level of learning
2. Developing a test blueprint or
“table of specifications”
3. Common pitfalls when writing test
questions
4. Writing questions at various levels of
performance
Developing a Test Blueprint or
a Table of Specifications
Develop a matrix (much like a rubric)
of number of test questions (or
percentage of test) by topic and level
of objective, and by amount of
emphasis on each topic and objective
•
Helps you plan your test
• Do before writing items
Example Test Blueprint or
Table of Specifications
Number of Objectives (Amount of Effort*}
Adjust
Level of Objective Topic A
Topic B Topic C Topic D
Total
Knowledge
1
2
1
1
5 (20%)
Comprehension 2
1
1
2
6 (24%)
Application
2
1
1
1
5 (20%)
Analysis
1
1
2
2
6 (24%)
1
2 (8%)
Synthesis*
1
Evaluation*
Total
*Adjustment factor
1
6
(24%)
6
(24%)
6
(24%)
1 (4%)
7
(28%)
25
(100%)
Levels of performance
Lower levels (knowledge, comprehension, and application)
 Recognize the principle in altered format.
 Match component parts with vocabulary.
 Identify example seen before.
 Recognize an example never seen before.
 Solve a problem using familiar format, but different
specifics.
Higher levels (analysis, synthesis, evaluation)
 Pick out components of a situation.
 Problem solve by combining elements.
 Make judgments of value.
Preparing to write test items
• Create test items while preparing class
•
•
•
•
lessons.
Make note of questions asked frequently by
students.
Make note of misconceptions made often by
students during class or homework.
Invite students to submit items at the end of
each class or at other times.
Adapt questions from formative assessments.
Categories of Test Questions
Test items can roughly be broken into
1.
Objective (“fixed” or single correct response)

True-false
Multiple choice
Matching
Completion or fill-in-the-blank.
2.
Subjective (extended response/multiple correct)






Short answer and/or Essay
Oral test
Case Study
Common pitfalls in writing test
directions—and avoiding them!
Giving unclear directions: “Choose the
best answer.”
•
What is meant by “best”? Funniest?
Simplest? Cheapest? Easiest?
•
“Choose the correct answer” is a much
better statement of directions—and then
ensure that there is a single correct
answer!
Writing GOOD test directions
Indicate carefully and specifically how students
should respond. For example, on a
TRUE/FALSE section on a test:
• Circle “V” for statements that are VERDAD
and “F” for statements that are FALSO.
• Do NOT have students write V or F in a blank.
Give point values for items and sections.
Allow students to EXPLAIN on any fixedresponse (true-false, multiple choice, etc.) item.
Writing True / False Items for
Tests and Exams
Can be used to check for knowledge
and comprehension.
Keep statements simple.
Make entirely true or entirely false.
Avoid double negative items.
In fact, best to avoid negative items.
Common pitfalls in writing
true-false test items
• Using NOT or EXCEPT in the statement
• Using cues such as “ALWAYS” or
“USUALLY”
• Using unclear, vague terms such as
“OFTEN” or “MANY” or “FEW”
• Using double negatives in a statement
• Using compound statements with one
indpendent clause true and one, false.
Writing True / False Test Items
For each of the statements below, circle
“T” if true and circle “F” if false.
T F The length of gestation in swine is 33 days.
For each of the statements below, circle
“T” if true and circle “F” if false. Also,
explain in the blank why it is false.
T F The length of gestation in swine is 33 days.
__________________________________
Writing multiple choice items
for tests and exams
• Use distractors that are plausible
• Do NOT use “all of the above” or “none of
the above” or “only A and B” or “B and C
on alternate Thursdays” as distractors
• Use 4 or 5 distractors, but use the SAME
number of distractors (4 or 5) for all items
• Vary randomly where you put the correct
answer.
• Allow students to explain!
Preparing for students who are
test-wise to some kinds of items
•
•
•
•
•
•
•
•
Longest answer
“C” as correct answer
Process of elimination
Use grammatical cues
Scientific sounding
Avoid simple and obvious
Related word
Guess by figuring out your pattern
Writing Matching Items for
Tests and Exams
Appropriate only for knowledge level
testing
Use a reasonable number of items: 7 – 12
All premises in one list, all responses in
another list
Premises can be long, but try to keep
responses short
Use more responses than premises
Can approximate fill-in-the-blank items
Writing Clear Directions for
Matching Items
Write the LETTER of the correct Response in the
Blank preceding the numbered premise.
____ 1. List premise
____ 2. List premise
____ 3. List Premise
____ 4. List Premise
____ 5. List Premise
____ 6. List Premise
A. Response
B. Response
C. Response
D. Response
E. Response
F. Response
G. Response
Writing Fill-in-the-Blank or
Completion Test Items
Most useful for lower order levels of learning
Use your own words
Try to ensure that only one answer is correct,
but, if that is too hard to do, …
You may consider “matching” using a word
bank
Keep all blanks the same length and at END
of statement
Avoid grammatical clues
Writing Directions for
Completion Test Items
Complete each statement by
writing the correct word(s) in the
blank(s) provided.
1. A thermometer should be rinsed in
_____________ water. Good or bad item?
2. The letters “N-P-K” in fertilizers stand
for, respectively, _____________,
_____________, and _____________.
Extended Response Test Items
Short answer and/or Essay
 Oral test
 Case Study or “Word
Problems/Situations”


These kinds of test questions require
criteria and pre-determined descriptions
of levels of performance to be graded
reliably and fairly. We call that what?
Short Answer and/or Essay
May require many different
behaviors:
 Comparison,
for or against, cause
and effect, explanation, summary,
analysis, describe relationships,
apply rules, or discussion
Oral Test
All students must receive the same
test item stated in a clear, concise
manner designed to elicit correct
answers arranged in logical order.
Determine beforehand how you may
lead the students if they stumble.
Case Study
Problem should be practical and realistic
Requires recall and comprehension of
prior information and, if well-written, may
require application, analysis, synthesis
and/or evaluation of that information
“Word problems” are similar to a case
study.
Which Type of Question to Use?
Select the appropriate technique to test
the student competence at the level of
cognition required by the objective.
Avoid using just one type of test question.
We know students have different
strengths:
Visual
 Audio
 Kinesthetic

Materials in this presentation
developed in part by…
Jason Peake
University of Georgia
Brian Parr
Auburn University
Dawn Zimmaro
University of Texas
Gary Briers
Texas A&M University