November 17, 2014
Assessment Item Writing
Creating rigorous assessment at the various RBT
and DOK levels
Lesley R. Brickhouse, M. Ed, NBCT
(Created on Feb 2014 revised November 2014)
Introductions
o Who am I?
o Who are you?
1)
2)
3)
Where are you from?
Why are you here?
What is your level of experience with assessment
item writing.
Overview
• Goals for today
– Objective: To instruct teachers on how to
write assessment questions on all DOK
levels and at the higher levels of RBT
– Results: Teachers will be able to create
assessments and leave with several MC
and CR questions for each grade level
Components of Good
Questions
Before we dive into information about
writing questions to a particular DOK
or RBT level, we must first develop
an understanding of the key
components of a good quality
multiple choice question.
Multiple Choice Question Basics
• All multiple-choice items consist of two basic parts: the stem
(or premise) and the choices (distractors). The stem is the
introductory statement or question that elicits the correct or
desired answer. The incorrect choices are called distractors.
• The answer choices are contain 3 distractors which are the
incorrect but plausible choices and the 1 correct choice that is
the answer. The best distractors contain correct information or
be may even be correct in some situations, however one choice
must still be deduced as the best answer.
• The major factor in successful multiple-choice assessment item
writing is the ability to create high-quality distractors that
effectively discriminate between those who have mastered the
content material being assessed and those who haven’t .
Example of good distractors
Example: The word(s) that mean(s) the same or almost the
same as flammable is/are:
a. Fireproof
b. Fire resistant
c. Easily burned
d. Burning
Knowledge of the definition of the word flammable isn’t enough to answer
this question. Knowing that flammable has something to do with fire will not
aid you in answering this question. All of the answers have something to do
with fire. All of the options offer a plausible choice if you know that
flammable is related to fire.
There are two choices with the word fire in them which are there to distract
from the correct answer. If you eliminate them, you are left with only two
options. You must then know that the correct choice is c or you will have a
50/50 chance of selecting the correct option if you must guess.
3 main types of Multiple
Choice questions
• Correct answer items
• Best answer items
• Incomplete sentence items
Correct answer item:
There is only one correct answer among the
distractors
The first effective polio vaccine was developed by:
1. Jonas Salk
2. Louis Pasteur
3. Edward Jenner
4. Robert Koch
Best-answer item:
There are more than one correct answer but one is
better than the others
The major goal of raising the legal drinking age is
to:
1. reduce driving fatalities and injuries.
2. reduce liquor consumption.
3. discourage the habit of drinking by teenagers.
4. reduce roadside litter.
Incomplete sentence item
Item is written as an incomplete sentence. It is best to
draft item as a direct question first and then revise it.
Original question:
What structure is found in both Euglena and Volvox?
A structure found in both Euglena and Volvox is the
______.
A. Chlorophyll
B. Cell walls
C. Flagella
D. Pseudopods
Consider the Readability of
assessment questions
•When writing and reviewing assessment items,
keep in mind that readability levels should be
appropriate for the students being assessed.
•You can use a readability determination site like
the ones listed below.
https://readability-score.com/
http://read-able.com/
Readability
If using MS Word, you can turn on the readability
feature by doing the following:
1.Click the Microsoft Office Button , and then
click Word Options.
2.Click Proofing.
3.Make sure Check grammar with spelling is
selected.
4.Under When correcting grammar in Word,
select the Show readability statistics check box.
5.You then can complete a spell check and the
readability will display in the box after the spell
check is completed.
Let’s check out your
questions on your assessments
Using the assessment you brought
with you, check the readability
• Is it at a level appropriate for your
students?
Check the type of questions
• Is there a good variety of correct
answer, best answer, and
incomplete sentence questions?
Let’s take a look at Cognitive Rigor
There are different ways to examine cognitive
rigor.
• Revised Blooms Taxonomy–What type of
thinking (verbs) is needed to complete the
question or task?
• Webb’s Depth of Knowledge –How deeply
do you have to understand the content to
successfully interact with it? How complex
is the content?
Handouts for RBT/DOK
RBT
DOK
DOK LEVELS
TITLE OF LEVEL
1
Recall and Reproduction
2
Skills and Concepts
3
Short-term Strategic Thinking
4
Extended Thinking
The objective’s or question’s verb(s) alone is/are not
sufficient information to assign a DOK level. Item
writers/reviewer must consider the complexity of the task,
levels of prior knowledge required and the mental processes
used to satisfy the requirements set forth in the objective.
It’s not all about the verb w/DOK
• DOK 3-Describe a model that you might use to
represent the relationships that exist within the rock
cycle. (Requires deep understanding of rock cycle and a
determination of how best to represent it)
• DOK 2-Describe the difference between
metamorphic and igneous rocks. (Requires cognitive
processing to determine the differences in the two rock types)
• DOK 1-Describe three characteristics of
metamorphic rocks. (Simple recall)
Same verb—used at three different DOK
levels
DOK
What does Level 1 look like in assessment items?
Recalling, restating, remembering, or recognizing a fact, term, or
definition.
Listing, describing, identifying, retrieving, naming or locating
information on a chart, graph, diagram or drawing.
Using basic calculations or measuring involving only one step.
Level 1 question stems:
•Can you recall______?
• When did ____ happen?
• Who was ____? Who discovered___?
• How can you recognize____?
• What is____? What is the formula for ___?
• How can you find the meaning of____?
• Can you select____?
• How would you write___? How would you describe____?
• What might you include on a list about___?
• Can you identify___?
DOK
What does Level 2 look like in assessment items?
Contrast or compare people, places, events and concepts.
Require describing or explaining that goes beyond recalled
information; describing or explaining results, relationships, cause
and effect, significance of impact or the why or how something
happens.
Convert information from one form to another; classify or sort
items into meaningful categories, completing multi-step
calculations.
Level 2 question stems:
•Can you classify______? How would you classify ___?
• Which best explains the significance of ____?
• How would you explain___? How would you describe the
relationship between ____?
• Why might you include ____on a list about___?
•How would you convert ___from___to___?
•What is the affect of ____? What is the cause of ____?
•How does __ impact? Which best explains the impact of___?
•How would you apply what you learned about ___ to ___?
DOK
What does Level 3 look like in assessment items?
Involves analysis, evaluation, explanations and support with
evidence, making generalizations and creating.
Solving real-world problems with predictable outcomes and
stating reasoning.
Applying knowledge and skills from multiple subject areas to
carry out processes and reach solutions or conclusions.
Level 3 question stems:
•Which provides evidence that/for ___?
•Which flow chart represents the correct order of___?
• How would you explain when ____ happens?
• How would your organize ___? How would you test ___?
•What would you predict would be the outcome if ____?
•Which graph would represent ____?
•How would you describe the sequence of ___?
•Which facts support that ___? What would happen if ___?
General rules of thumb…
•If there is only one correct answer, it is
probably level DOK 1 or DOK 2
•DOK 1: you either know or you don’t
•DOK 2 (conceptual): apply one concept, then
make a decision before going on applying a
second concept
• If more than one solution/approach,
requiring evidence, it is DOK 3 or 4
•DOK 3: Must provide supporting evidence and
reasoning (not just HOW solved, but WHY –
explain reasoning)
•DOK 4: all of “3” + use of multiple sources or
texts
Which DOK is best?
Questions at lower levels are usually more
appropriate for:
•Evaluating students’ preparation and comprehension
•Diagnosing students’ strengths and weaknesses
•Reviewing and/or summarizing
Usually questions at upper DOK levels are
appropriate for:
•Encouraging students to think deeply and critically
•Problem-solving
•Encouraging discussions
•Stimulating students to seek information on their own
Please take a break and we will
resume the session in 10 minutes.
Let’s see how well you can
identify the DOK levels of
questions.
Let’s play…
What’s my DOK?
Freebies
• Just to give you a little practice,
lets do some together before we
test your knowledge
1. In what situation is the most work
done?
A. Lifting a 50 N box up 2 meters.
B. Lifting a 50 N box up 2 meters and
then down 2 meters.
C. Lifting a 75 N box up 1 meter.
D. Lifting a 75 N box up 1 meter and
then down 1 meter.
2. The equation 2x + 3y + 7=35
represents the total points Frank scored
in last night’s basketball game. In the
equation, x represents field goals scored
and y represents 3-point goals made. If
Frank scored 8 field goals, how many 3point goals did he make?
o4
o8
o9
o 14
3. Which of the following is the correct
ranking of organizational hierarchy of
organisms from simplest to most
complex?
A. cells, organs, tissues, organ
systems, organisms
B. cells, tissues, organs, organ
systems, organisms
C. tissues, cells, organs, organ
systems, organisms
D. tissues, organs, cells, organ
systems, organisms
4. If a parasite invades and irritates the
small intestine of an animal, what effect
will it most likely have on the body?
A. Increase water retention.
B. Disrupt balance and cause
blindness.
C. Decrease white blood cell count.
D. Interfere with nutrient absorption.
5. A cell found in the root of a plant would
most likely lack what cell structure?
A. Cell membrane
B. Cell wall
C. Chloroplasts
D. Mitochondria
6. While conducting careful experimentation, a
chemist discovered a new element. The
international scientific community would most
likely
A. ignore the new data because it did not fit into
the generally accepted periodic table.
B. create a new classification system with the
new information.
C. disagree on how to incorporate the new data
and create many classification systems
based on personal preference.
D. repeat the experiment and modify the current
periodic table to include the new data.
Let the game begin
You need the following handout to
record your answers
What’s my DOK level?
1. José had 64 baseball cards. He gave 12
cards to his sister. Then he divided the
remaining cards equally among his FOUR
friends. How many cards did each of his
friends get?
o
o
o
o
13 cards
16 cards
17 cards
18 cards
2. The illustration below shows the pelvic bone of a modern
day baleen whale and a reconstruction of an extinct whale
based upon fossil evidence.
What is the best explanation for
the presence of the vestigial limbs
in both whale species?
A. Whales have functioning hind
limbs and can walk on land
when necessary.
B. Whales are evolving into a
species that will eventually
have four legs.
C. Whales evolved from a
species that had functioning
hind limbs.
D. Whales lost their hind limbs
because they stopped using
them.
3. Look at the drawing. The numbers
alongside each column and row are the
total of the values of the symbols within
each column and row. What should
replace the question mark?
A. 23
B. 25
C. 28
D. 30
E. 32
5. What is the rule for this
pattern?
2, 1, 3, 2, 4, 3, 5, 4, 6
o
o
o
o
subtract 1, multiply by 3
add 2, add 3
subtract 1, add 2
multiply by 2, divide by 1
6. In a contest, Matt threw a ball 10
yards. Brett threw a ball 400 inches.
How much farther did Brett's throw
travel than Matt's throw?
A. 40 inches
B. 80 inches
C. 100 inches
D. 280 inches
7. The illustrations represent two different cells.
Which statement best identifies Cell X and Cell Y?
A. Cell X is prokaryotic and Cell Y is eukaryotic.
B. Cell X is a nerve cell and Cell Y is nerve tissue.
C. Cell X is a red blood cell and Cell Y is a muscle
cell.
D. Cell X is a plant cell and Cell Y is an animal cell.
8. Look at the two magnets above. If you
push them towards each other as shown,
the magnets will
a)Break into many pieces
b)Turn in opposite directions
c) Be pushed away from each other
d)Be pulled towards each other
9. Greenhouse gases help keep Earth
at a habitable temperature by
A. blocking the cold winds and ices
originating in outer space.
B. serving as essential nutrients for
atmospheric phytoplankton.
C. allowing only infrared light to reach
Earth’s surface.
D. retaining some of the Sun’s energy
in our lower atmosphere.
10. The thermometers are measuring the temperature of
the center of the soil samples.
What factor causes the temperature
difference in the two soils?
A. Conduction within different soil types
B. Condensation within different soil
types
C. Radiation emitted by different soil
types
D. Convection in the air above different
soil types
11. Sara needs to buy a coat for her trip. The table below
describes the properties of four materials often used in coats and
the weather map below shows the predicted weather.
Material
A
B
C
D
Properties of the Material
Waterproof
Warm Lightweight
YES
YES
NO
NO
NO
YES
YES
NO
YES
NO
YES
YES
Her jacket should be made of which material?
A. Material A
B. Material B
C. Material C
D. Material D
12. According to Alfred T. Mahan, the United
States should adopt a foreign policy based on:
a. A strong army and control of the North
American continent.
b. Achieving the best interest of all nations.
c. Isolation from international affairs.
*d. Naval supremacy and undisputed control
of the world’s sea lanes.
13. A newspaper prints a story that criticizes the
current administration’s Policies. The Bill of
Rights allows a government official to respond to
this headline by –
a) Arresting the publisher of the newspaper
b) Closing down the newspaper
c) Demanding that the newspaper print a new
headline
d) Writing a letter of protest to the editor
14. What is the main reason so many
people moved to California in 1849?
a. California land was fertile, plentiful,
and inexpensive.
*b. Gold was discovered in central
California
c. The east was preparing for a civil
war.
d. They wanted to establish religious
settlements.
Under our system of checks and
balances, the Supreme Court can limit
the power of both the Congress and the
President by
15.




impeaching public officials.
vetoing a law.
making appointments.
declaring a law unconstitutional.
16. In a free market economy, the price
and quality of goods and services are
most strongly affected by




advertising.
competition.
borrowing.
regulation.
17. Monopolies cause problems in a
market system because they:
a. Create external costs and imperfect
information.
*b. Lead to higher prices and under
production.
c. Make such large profits.
d. Manufacture products of poor quality.
18. What is the best description of this poster’s purpose?
A. Impartially provide the public with the most current
information
B. Give citizens an accurate description of Franklin D. Roosevelt
C. Influence public opinion by using propaganda techniques
D. Use emotional appeal to change rational opinions
19. Which of the following statements
would the author be most likely to agree
with?
A) He who fears something gives it power
over him.
B) Two minds are better than one.
C) Older means wiser.
D) Great minds think alike.
20. In paragraph 19, Cory became upset
at the thought that
A) Minnie had run away
B) his parents would be upset with him for
not going straight home
C) Elisa was in danger
D) Minnie could drown
21. When Cory found out what had happened to
Minnie, he
A) blamed Elisa for not watching Minnie
B) told Elisa not to try to get Minnie by herself
C) sent Elisa home to get help for Minnie
D) warned Elisa that Minnie might die.
22. During a science experiment, you need to record the
temperature of a liquid on a regular basis. You fill your chart with
the times shown below.
Time (Seconds)
0
15
30
45
?
Temperature (°C)
10
12
15
20
25
At what time should you have recorded the temperature at 25°C, in
seconds?
23. A solid cube was put into a
cylinder containing four liquids with
different densities as shown in the
diagram.
The cube descended through
layer A and B, and stopped upon
reaching layer C. The density of
the cube most likely lies between
A. 1.00 and 1.50 g/cm3.
B. 1.51 and 3.50 g/cm3.
C. 3.51 and 6.00 g/cm3.
D. 6.01 and 9.00 g/cm3.
24. Use the graph to answer the following
question.
How far did the car travel during the first 15
seconds of the trip?
A. 20 m
B. 35 m
C. 150 m
D. 300 m
25. Which statement best explains how the movement of
heat will likely be affected in the Great Basin Desert due
to global climate change?
A. Radiation will transfer more heat from the desert’s
floor to Earth’s atmosphere and conduction will move
more heat out of Earth’s atmosphere into space.
B. Conduction will move more heat from Earth’s surface
to the atmosphere and radiation will move more heat
around in the atmosphere.
C. Radiation will transfer the same amount of heat from
the sun to Earth’s surface and more heat will be
moved around in the atmosphere via convection.
D. Conduction will transfer the same amount of heat from
the sun to Earth’s surface and more heat will move
through the desert’s floor via radiation.
Please take a break and we will
resume the session in 10 minutes.
Tips to writing questions
1. Avoid vocabulary that be favorably or
unfavorably biased or words that are
germane to a particular location, race or
gender.
examples: dirty south, smart asian, lazy mexican, angry
black
2. Avoid using words that may be reinforce
stereotypes.
example: Men are strong, blonds are unintelligent, etc
3. Use gender neutral language.
example: Instead of writing “Who was the first man on the
moon?” write “Who was the first person on the moon?”
If you must use names use names like Pat or Taylor
that could be either a boy or girl’s name.
Writing tips continued…
4. Keep stems as short and succinct as
possible; avoid wordiness and leave out
nonessential information or leading
information in the question.
example: Rose and Paula are best friends who are working
on a homework problem. Mr. Burke provided the following
data regarding the last quiz. 88, 90, 90, 60, 87, 89, 89, 98,
25. Identify the mean, median, mode and outlier?
Writing tips continued…
5. Include words in the stem instead of
repeating them in the distractors.
example: Based on the table above, what can be
concluded about sound waves?
•They travel fastest in solids
•They travel fastest in liquids
•They travel fastest in gasses
•They travel fastest in plasmas
Based on the table above, it can be concluded that
sound waves travel fastest in ____.
•Solids
•Liquids
•Gasses
•Plasmas
Checklist for Reviewing Multiple-Choice Questions/Items
o Has the item been constructed to assess a single written
objective?
o Is the item based on a specific problem stated clearly in the
stem?
o Does the stem include irrelevant material?
o Are the distractors worded clearly and concisely?
o Are the distractors free from clues as to which response is
correct?
o Has the use of distractors including “all of the above” and
“none of the above” been avoided?
o Does the item include one and only one correct or a clearly
best answer?
o Has the answer been randomly assigned to one of the
distractors positions?
o Are the grammar, punctuation, and spelling correct?
o Has unnecessarily difficult vocabulary been avoided?
o If the item has been administered before, has its effectiveness
been analyzed?
Let’s get to it!
• Divide up into content area groups
to write or revise items
Handout
Using standards/objectives to
write items
• Use objectives from CMAPP or
unpacked standards to write
original questions at various
DOK/RBT levels
• Use preexisting questions from
CMAPP, various item banks to
revise questions at various
DOK/RBT levels
Example:
• Which best explains who controls how goods
and services are priced in a free market system?
 The government
 Consumers
 Sellers and the government
 Consumers and sellers
• In a free market economy, the price and quality
of goods and services are most strongly
affected by
 advertising.
 competition.
 borrowing.
 regulation.
Example:
• 6.EE.6 Use variables to represent numbers and write
expressions when solving a real-world or
mathematical problem;
1. Which expression can be used to represent Susan’s
age in three years, when a represents her present
age.
a. 3-a
b. a-3
c. 3a
d. 3/a
Please take a lunch break and we will
resume the session in 60 minutes.
Let’s get back at it!
Let’s get ready to go!
Let’s recap the important
information of the day!
RBT and DOK
Tips to writing quality assessment items
Don’t forget this is a process, it takes time to learn and
become good at it. Rome wasn’t built in a day.
Next Steps
Sign up for future sessions offered from DPI (even if you don’t want
to write items for State assessments, you will still learn more about the process to use in writing
your classroom assessments)
When in doubt….ask (poll your PLT, phone a assessment item writing friend,
check resources)
Work smarter not harder (use your current assessments and question banks to
find questions to revise)
Thank you for your
participation!
Your feedback is appreciated.
1. Please complete the feedback form
and leave it on the front table.
2. Make sure you received your leave
form to take to your Bookkeeper.