CREATING QUALITY
ASSESSMENTS
Guidelines
James Greene
Harlan Independent School District
July, 2012
HISD EXPECTATIONS FOR 2012-13
Continue work on realigning curriculum and
updating curriculum documents
 Continue and refine use of assessment for
learning strategies with emphasis on student
self-assessment and ownership of learning
 Create quality assessments using research to
inform your design
 Use CIITS to create and store assessments and to
track and analyze assessment data

Design Concerns
Classroom Assessment
 Purpose
 Audience
 Content
 Evidence
 Process
 Meaning
of learning
KEYS TO QUALITY
ASSESSMENT
Key No. 1 STANDARDS-BASED
What do we want students to know and
be able to do at the end?
Key No. 2 FAIR
Does the assessment allow students to
show what they know?
Key No. 3 VALIDITY
Does the assessment measure what it’s
supposed to measure?
Key No. 4 MULTIPLE SOURCES
Is there sufficient evidence to judge
whether a student has mastered the
standard?
FAIRNESS MEANS…
Objectivity (no one gets unfair advantage)
 Absence of bias (cultural, gender, socio-economic)
 Opportunity for each student to demonstrate what
he/she knows

ISSUES RELATED TO VALIDITY
 Do
the items form a representative
sample?
 Do the items represent a fresh
experience or are they recycled?
 Do the items rely on skills that students
may lack which are not part of the
assessment?
DO ITEMS FORM A REPRESENTATIVE
SAMPLE?





Items are not aligned with the SFLTs
Items represent teacher convenience rather
than systematic design
Items focus on exceptions rather than key
topics
Items are not appropriately distributed
among topics
Items do not reflect the amount of time
devoted to them during instruction
“The essential characteristic of well-designed
assessments is that the processes used to collect
and interpret data are consistent with the
purpose of the assessment. That match of
purpose and process is achieved through
thoughtful planning…”
—National Science Education Standards
TEST BLUEPRINTS
•Select targets to be assessed
•Select methods to be used
•Create items
•Reasonable number of items per target (based on
type of item)
•Group items by targets
•Assure appropriate balance among items and
among targets
•Consider if redo/retake will be permitted and
what form it will take
Do the items represent a fresh experience?
“Any item may function at the rote learning level if
the concept involved has been taught in the same
way that it is used in the testing situation. In view
of this, we must do our utmost to increase the
likelihood that an item requires reasoning and
understanding.”
—Harry D. Berg
SKILLS ISSUES






Heavy reliance on a skill in which student is
not proficient (e.g., reading level, drawing)
Sentence structure and vocabulary
Extraneous knowledge
Format (complexity; unfamiliar to student)
Student’s ‘testwiseness’
Student test anxiety
PRECISE UNAMBIGUOUS
LANGUAGE
“It is not sufficient to write a question
that can be understood. To paraphrase
the advice of Gen. Douglas MacArthur to
the Corps of Cadets: Write questions that
cannot be misunderstood, not merely
questions that can be understood.”
—John P. Sevenair
" The content of assessments should match
challenging subject matter standards and be
connected to contexts of application. This means
assessing learning based on observations, oral
questioning, significant tasks, projects,
demonstrations, collections of student work, and
students' self-evaluations, and it means that
teachers must engage in systematic analysis of the
available evidence."
—Lorrie A. Shepard
ASSESSMENT METHODS
Personal Communication
Constructed
response
Knowledge
Selected
response
Reasoning
Skills
Product
Observation
Performance
tasks
SELECTED
RESPONSE
True False
Multiple True False (MTF)
Multiple Choice & Alternate Choice
Matching
Keyed Lists
TRUE/FALSE
1. Grass is green.
2. California is the largest state.
3. The U.S. national debt is too high.
TRUE/FALSE
Pitfalls
• Trivia, trickery, and
truisms
• Overemphasis on
recall
• 50% chance of
guessing correctly
•
•
•
Benefits
Ease of use and
coverage of
material
Assessing student
misconceptions
Useful in formative
assessment
WRITING GOOD T/F ITEMS
Focus statements on ONE fact or concept.
 Create statements of relatively equal length.
 Word statements positively using precise,
unambiguous language.
 Use NEW language (don’t lift statements from
textbook or from class).
 Don’t use absolutes (e.g., always, never, all) or
conditional verbs (“can, may”).
 Avoid vague qualifiers like “some,” “usually,”
“typically,” “several,” “many,” “large,” “small”).
 If it is necessary to test an opinion, attribute it to a
source.

WRITING GOOD T/F ITEMS
Lessen the power of guessing by:
 having slightly more “false” items than “true” (when
students don’t know, they tend to opt for “true”)
 mixing up the sequence of responses so that they do
not form patterns
“A major distinction between the [true-false] item and
items in multiple-choice or two-choice formats, is that
the [true-false] item contains no criterion for answering
the item. The criterion is outside of the item, rooted in
the characteristics and experiences of each individual
examinee. Each examinee must ask the question, true
or false with respect to what? It follows that each …
true-false item must be unequivocally true or
unequivocally false.”
https://www.msu.edu/dept/soweb/writitem.html#mcitem
Use T/F to test CONCEPTUAL
UNDERSTANDING
1.
2.
3.
Inflation leads to a lower cost of living.
Physical change results in formation of
a new substance.
Profit is the money a store takes in
when goods are sold.
LESSEN POWER OF GUESSING
•Students can correct entire statement (focuses
on the complete proposition) OR
•They can correct an underlined word or phrase
(allows teacher to focus questions on particular
concepts or facts)
•Correction diminishes the power of guessing
1.
2.
Higher prices are one of the results of
deflation.
The rhythmical succession of single tones
producing a distinct musical phrase or idea is
referred to as harmony.
JUSTIFIED TRUE FALSE
Consider the equation T or F Why I (we) think so
a=12+b (when b≠0)
a is always greater
than b
a and b could be equal
etc.
from Mathematics Formative Assessment by
Page Keeley and Cheryl RoseTobey, ©2011
Corwin Press
MULTIPLE TRUE FALSE (MTF)
Consists of a context or situation and several
statements which the student is to evaluate in
terms of that information.
 Is an alternative way to deal with “All of the
above” multiple choice items

EXAMPLE OF MTF
Which of these are characteristics of a square?
Check (√) all that are:
Sides are equal in length
Sides join to form acute angles
Has four sides
Is a type of polygon
Is a three-dimensional figure
ANOTHER MTF EXAMPLE
Which of these are effects of inflation?
Check (√) all that are:
Shortage of capital to invest in new businesses
Drop in the purchasing power of money
Decline in cost of goods and services
People on fixed incomes can buy less than before
Interest rates favor borrowers rather than lenders
A THIRD MTF EXAMPLE
Place a checkmark (√) by each example of chemical
change:
1.
Butter melting on warm bread
2.
Tarnish on a silver charm
3.
Black edges on overdone toast
4.
Creases in pants from being ironed
5.
Rust on a saw blade
MATCHING
Identify each item on the left by matching it with an item
from the right.
1.
2.
3.
4.
5.
6.
Ship that brought
Pilgrims to America
Indian who helped the
Pilgrims
Name of the Pilgrim
settlement
Important food crop of the
Pilgrims
Key Pilgrim leader
Date Pilgrims arrived in
America
A.
B.
C.
D.
E.
F.
G.
H.
I.
William Bradford
Squanto
Henry Hudson
Mayflower
Jamestown
1620
Plymouth
Rice
Corn
CREATING QUALITY MATCHING
ITEMS
 Target
items should be of the same class
 Directions indicate the relationship between
target items and choices
 Have at least one extra choice
or allow choices to be used more than once
 Sequence choices to make matching easier
(e.g., ABC order)
 Reasonable number of items
BEYOND RECOGNITION/RECALL
What type of figure speech is represented by each quote?
Match the correct term from the right to the quote on the left.
There is one extra term.
1. “It’s better to have it and not need it,
than to need it and not have it.”
2. “He is smart as an owl.”
3. “George has been put out to pasture.”
4. “Henry was a lion in battle.”
5. “This vase will make a good
centralpiece.”
6. “If I don’t make an A, I will just die.”
A.
B.
C.
D.
E.
F.
G.
chiasmus
euphemism
hyperbole
malapropism
metaphor
simile
synedoche
KEYED LIST
Use the key to identify the role the
underlined word plays in the
sentence.
Key
Sentence roles
A.
Subject
B.
Direct object
C. Indirect object
D. Predicate
nominative
1.
2.
3.
4.
5.
6.
Is that your car?
She told him to come in.
Joe is a great student.
Which book do you need?
To whom did you give the
package?
Why didn’t Mr. Howe return
my call?
ANOTHER KEYED LIST
Key
Functions of money
A.
Unit of account
B.
Store of value
C.
Medium of
exchange
Classify each example using the key:





Isabella deposits $25.00 in her savings
account at the bank
The supermarket sign that says
“Strawberries $2.99 per pint”
Mrs. Wallace writes a check to her
beautician for her perm.
The school board receives a bid from Acme
Supply to provide ten cases of copy paper for
$375.
Dr. Greene finds a hundred dollar bill on the
street.
ALTERNATE CHOICE
Two choices
 Useful in formative assessment
 Can highlight particular misconceptions or errors
students are likely to make
 Can be expanded to conventional MC by using
“both of these” and “neither of these” as options

1. If you were making shirts, which of these
would be a capital resource?
A. the person who designed the shirts
B. the sewing machine used in sewing them
2. People who make things for others to use are
called:
A. consumers
B.
producers
MULTIPLE CHOICE
1. The______ lasted from 1861 to 1865.
A. Revolution
B. Civil War
C. Spanish-American War
2. It marked the turning point of the war.
A. Ft. Sumter B. Antietam C. Gettysburg
3. The Civil War
A. Was the bloodiest war in American history
B. Was fought mainly in the South
C. Could have been avoided
EFFECTIVE STEMS
Focus the question through the stem, not the
choices. (Writing the stem as a question
helps do this.)
 Avoid non-functional language.
 Use simple, direct language as much as
possible.
 If stem contains a negative, highlight it in
some manner: e.g., Which of these is NOT
true of paintings by Monet?

Focus the question through the stem:
Seismology
A) Study of human social
organization and
institutions
B) Study of earthquakes
and internal structure
of the earth
C) Study of physical
features of earth and
human response to
them
What does a seismologist
study?
A) Earthquakes and earth’s
internal structure
B) Human social
organization and
institutions
C) Physical features of earth
and human response to
them
(preferred form)
EFFECTIVE CHOICES
Make all choices plausible & attractive
 Use common misconceptions and errors as basis
for distractors
 Keep distractors consistent in content, form, and
length
 Use similar degree of specificity
 Use “new” language rather than phrases from
the text or class
 3-4 good choices are enough
AVOID COMPLEX MULTIPLE CHOICE
ITEMS:
Composers of the Baroque period include:
1.
J.S. Bach
2.
Frederic Chopin
3.
Georg Philipp Telemann
a) 1 & 2 b) 1 & 3 c) 2 & 3 d) 1, 2, 3
•Harder for students to process
•Don’t discriminate better than simple choice
questions
AVOID “ALL OF THE ABOVE” AS
OPTION
•
•
If student finds one exception, then he knows it is
NOT “all of the above”
If he finds at least two choices that are correct,
then he knows it IS “all of the above”
ALTERNATIVES
1.Stem……
A) ------- B) -----C) both of these D) neither of these
2. Use MTF
“NONE OF THE ABOVE” AS CHOICE
•
•
•
•
Useful if the correct response is an absolute
If it is the answer, you can’t be certain that the
student actually knows the information
Better than a weak distractor
Should be the correct answer about half the time
Which of these adds value to Choco-Bars as a product?
A) the elegant blue and gold wrappers in which they are sold
B) the short travel time they have from factory to store
C) the free gift that comes with a purchase of a five pound box
D) none of the above
SUGGESTIONS FOR MANAGING
OPTIONS
•
•
•
•
•
Logical sequencing (e.g., ABC, magnitude)
Random patterns in arrangement of options
Relatively equal use of each option on every
test
Use vertical formatting for longer options
Have students use capital letters rather
than small case if they have to write the
letters
1. Which of these artists is known primarily
for his work as a sculptor?
A) Leonardo da Vinci
B) Auguste Rodin
C) George Bellows
D) Rembrandt van Rijn
2. Which of these is an example of Baroque
sculpture?
A) Donatello’s David
B) Rodin’s The Age of Bronze
C) Bernini’s Apollo and Daphne
D) Miró’s Woman and Bird
1. “Pharaohs were to Egypt as Emperors were to Rome”
is an example of an - - -:
A) simile B) chiasmus C) analogy D) metaphor
2. The poetic meter with a foot consisting of an
unaccented and an accented syllable is known as:
A) lyric
B) epic
C) iambic
D) trochaic
3. Who composed Rhapsody in Blue?
A) Judy Garland
B) Frank Sinatra
C) Cole Porter
D) George Gershwin
GIVEAWAYS






Grammatical clues (e.g., a, an)
Words from the stem repeated in the
correct choice
Correct choice is much longer or more
complicated in wording
Absolutes: always, never
Choices form pairs
Interlocking items (first item answers
second)
Item Analysis
•
•
•
•
Options not selected by anyone should be
replaced as they are not working
An option chosen by almost everyone over
the other options including the right answer
may signal an instructional problem
Items that everyone gets right are not
functioning as discriminators
Items missed by virtually everyone may
signal an assessment problem or an
instructional problem
MULTIPLE CHOICE AND HIGHER
LEVEL THINKING
“Bury the Verb”
Identify the verb you
would like to use
 Change it to a noun
and pair it with a
convergent verb
 You can also use
“which” with the noun

Mike Dickinson

Explain: explanation


Describe: description


Select the best
explanation for ….
Identify the most
accurate description
of…
Solve: solution

Choose the most
viable solution for….
FRAMING QUESTIONS
•
•
•
•
•
•
Identification/recog
nition
Application
Comparison/contra
st
Premise/consequen
ce; cause/effect
Case or scenario
Interpretation
•
•
•
•
•
•
Common factor
Example or
illustration
Problem/solution
Conclusion from
data (e.g. trends)
Explanation of a
sequence
Image-based items
CLOSE DECISION MULTIPLE
CHOICE (MYRON DUECK)
Student may choose a second answer and
provide an explanation
“Lifeline”
Reveals student’s thinking
Reveals flaws in your questions
CONSTRUCTED RESPONSE TYPES
 Short
•
•
response
• Fill in the blank
• Graphic organizers and visuals
• Relationships (e.g., form pairs, which
doesn’t belong, word splashes)
• Exit Cards
• Prompts
Extended response (formerly called “open
response”)
Essay
GRAPHIC ORGANIZERS AND
VISUALS
Simple webs
 Concept maps
 Venn diagrams
 Graphs and data
displays
 Labeling features
in a picture or
diagram

USING GRAPHS
Numbe
r apples
eaten
1
2
3
4
5
6
Kate
Ashley
Ethan
Steve
Ethan ate
number on the graph.
apples. Show this
1. Who ate more apples: the girls or the boys?
2. How many more apples did Ashley eat than Steve?
3. Write a number sentence than shows how many apples Kate and
Ashley ate.
FILL IN THE BLANK
a. ________________ is the current President of the
United States.
b. A good source of protein is __________________.
c. The steps in solving a quadratic equation are
1) ________________ 2) __________________,
3) _________________, 4)
__________________, and 5)
_________________.
Writing Quality Fill in the Blank Items
Item should have only one right answer
 Blank should be placed at or near the end
 Multiple blanks should be avoided

1. The name of the current Governor of Kentucky is
________________________.
2. Based on the data in the graph, the month with the
highest number of absences was
__________________.
SHORT RESPONSE PROMPTS
Focus question on one point.
 Use clear, unambiguous language.
 Assess knowledge and reasoning

Compare the role of producers and consumers in the
economy.
Sam shaded part of this circle. (Diagram shown)
What fraction of the circle is shaded? Explain your answer.
K-PREP sample
Sample K-PREP short response prompts
1. (Based on reading passage)
Tommy learned that watching birds build their nests is
a good way to understand them. How does watching the
birds build their nests help Tommy understand birds
better?
2. A bird called the arctic tern flies about 18,600 miles each
year. What is this distance rounded to the nearest ten
thousand? Show your work or explain how you found your
answer.
3. Draw a closed figure with 5 line segments. Your figure
must contain 2 right angles. Label the right angles with
the correct angle marks.
DIAGNOSTIC QUESTIONS
“The crucial feature of…diagnostic questions is
based on a fundamental asymmetry of teaching;
in general, is better to assume that students do
not know something when they do than it is to
assume they do know something when they don’t.
What makes a question useful as a diagnostic
question, therefore, is that it must be very
unlikely that the student gets the correct answer
for the wrong reason.”
—Dylan Wiliam
DIAGNOSTIC QUESTIONS (DYLAN
WILIAM, PAGE KEELEY)
Ask “Why is…?” rather than “What is…?”
“Why is 17 prime and 15 not?” rather than “What is a
prime number?” or “Is 15 a prime number?”
“Why is a square a rectangle?” rather than “Is a
square a rectangle?”
“What makes the veto an example of checks and
balances?” rather than “What are “checks and
balances?”
Take a straight recall question and transform it by
phrasing it to require explanation or justification.
DIAGNOSTIC QUESTIONS (DYLAN
WILIAM)
Ask for reactions to statements:
“All squares are rectangles.”
“Light travels from the eye to the object.”
Use MTF/MC with multiple correct responses:
“In which of these diagrams is one-fourth of the area
shaded?”
CAVEAT
“Just as there can be no set formulas for producing a good story
or a good painting, so there can be no set of rules that will
guarantee production of good test items Principles can be
established and suggestions offered, but it is the item writer’s
judgment in the application…that determines whether good
items or mediocre ones will be produced.”
—R.L.Ebel