Creating Tests That Assess
Higher Order Thinking Skills
Innovative Educators
Dr. Jennifer Hurd
October 13, 2010
Objectives
When you complete this workshop, you
should be able to:
• Write different types of questions that will assess more
than knowledge and comprehension on the Bloom’s
Taxonomy Scale.
• Analyze test items to determine if these test items assess
higher order thinking skills.
• Avoid signaling the correct answer by knowing common
errors in teacher-developed exams.
Bloom’s Taxonomy
Developing Objectives Using
Bloom’s Taxonomy
• Objectives must be measurable
• Objectives must be attainable
• Objectives must have a limiting factor
Objectives must:
• Contain the same key words that we will
discuss for developing questions.
• Map to the mission statement or goals for
the entire course or department
• Demonstrate the skills have been attained
for each level of Bloom’s Taxonomy
Example of an Objective
According to the course material,
85% of the students will be able to
design a business plan for a new
business of their choice.
Example of an Application
Objective
According to the course content, 90%
of the students in this course will be
able to complete a mock interview
with three of the instructors
Learning Outcomes
• Are the primary building blocks of good curriculum
design.
•
Support the goal or learning outcome in that each is a
small step to arriving at what the learner is supposed to
know or be able to do.
•
Are carefully worded with qualifiers to restrict the
conditions and terms under which the learning outcomes
are met.
• Are measurable!
Learning outcomes must
contain verbs that describe
observable, measurable actions.
Learning Outcomes should:
 Provide the basis to select
or design instructional
materials, content or
assessments
 Provide a framework
within which learners can
organize their efforts to
complete the learning task
Planning a test
• Determine the method of assessment
– Demonstrating knowledge and understanding
– Assessing critical thinking skills
– Assessing problem solving skills
– Assessing performance of procedures
– Assessing application skills
– Assessing collaborative learning skills
– Assessing research skills
Planning: Specifying What is to
be Assessed
• Clarify instructional learning outcomes
• Define the purpose of the test
– Pre-instructional (pre-test)
– Tests and quizzes
– Final exams
• Specify what will be tested and at what
level
• Develop a test blueprint
Blueprint for a Test
Content/
Skill
Area
Knowledge/
Understanding
Critical
Thinking
Skills
Skill
Comprehension
Application
Critical
Thinking
Problem
Solving
Performance
of
Procedures/
Demonstrati
ng
Techniques
Collaborative
Learning
Skills
Integrate
Learning
Into
Professional
Practice
Assessing
Research
Skills
Basic Tenants of Test Item
Writing
•
•
•
•
Reading Level
Write items as simply as possible
Test for important details
Avoid direct quotes or verbatim
statements
• Identify any opinions or authority that
are used in test items
Anatomy of an Exam Question
• Should have three parts:
– A key word that tells the students what they are
expected to do
– An object that tells them what general
information to use to perform their task
– A limiting factor or factors that tells what
specific information to include in the answer
Bloom’s Taxonomy
Let’s look at each level of Bloom’s Taxonomy to
see how we can develop questions for each level.
Knowledge Questions
•
•
•
•
•
•
•
remembering;
memorizing;
recognizing;
recalling identification and
recall of information
Who, what, when, where, how ...?
Describe
An example of a knowledge
question
According to the instructor,
define the key words for
knowledge questions.
An example of a knowledge question
According to the instructor, list
the key words for knowledge
questions.
Can you change this question into a M/C
question?
Writing a question
According to the instructor, which of the
following are the key words for knowledge
questions?
A.
B.
C.
D.
E.
list, synthesize, evaluate, define, write
List, define, dramatize, solve
List, define, duplicate, memorize
None of the above
A and B only
Application Question Key Words
•
•
•
•
•
Imagine
Pretend
Make Believe
Visualize
Assume
• Put yourself in the
situation of
• Use your knowledge
to
• Suppose
In groups of three or four write a three-part application
question on any topic or subject area using one of the key
words listed above. Be prepared to share your question.
Imagination Activity
• Imagine that you are creating your own company
• You can hire 5 key people to help you run your
company
• These 5 people can be anyone from the past or
present
• Who would you hire and why?
• You have 3 minutes to decide, then you will
discuss your response with your table partners.
One person from each table will share with the
group.
Evaluation Question Key Words
•
•
•
•
•
•
•
•
Rank
Order
Judge
Grade
Measure
Determine
Convince
persuade
•
•
•
•
•
•
•
Decide
Support
Explain
Critique
Criticize
Recommend
Suggest
Sample Three-part Evaluation
Question
• According to the number of safety-related
features each contains, determine which
compact car is safest to drive.
Key Word: Determine
Object: to find out which compact car is safest to drive
Limiting Factor: cars must be compact
Only safety-related features to be
examined
A/B Visit
Argue Workshop
• This workshop allows students to persuade,
convince, determine, decide, recommend,
suggest, support, and explain parts of course
content.
• Applying course content to other parts of
the course or to their own life makes it even
more meaningful and worthwhile.
A/B Visit
Argue Workshop
• Set up the room for A/B visits, or have
students get into pairs.
• Your topic is to argue whether or not there
should be a student success course for
college credit on your campus.
• The people on the outside (A’s) will take
the pro stance.
• The people on the inside (B’s) will take the
con stance.
Synthesis Question Key Words
•
•
•
•
•
•
•
•
Combine
Integrate
Modify
Revise
Improve
Rearrange
Reconstruct
Hypothesize
•
•
•
•
•
•
•
•
Substitute
Create
Generate
Devise
Design
Compose
Plan
Predict
Synthesis Sample Question
• Rearrange the amendments to the U.S.
Constitution in order from least to most
important in everyday life. As well, devise
any new amendments that you think would
improve everyday life. Justify your
amendments according to your text or your
notes.
Rearrange the amendments to the U.S.
Constitution in order from least to most
important in everyday life. As well, devise
any new ammendments that you think would
improve everyday life.
• Key Words: rearrange and devise
• Object: To discover which constitutional
amendments you consider most important and
which you would like to add to improve everyday
life
• Limiting Factor: deal only with amendments to
the U.S. Constitution.
Other Synthesis Assessments
• Ask students to create
mnemonic devices to
remember course
content.
• Students can create
crossword puzzles that
center on a particular
part of course content
• Create a visual product
to show course
content.
• Example: Create a
company logo for
XXX company.
Analyzing Test Items to
Determine if they Assess Higher
Order Thinking Skills
In the next three questions, use the
chat feature to tell me whether the
question assesses higher order
thinking skills. hy/why not.
According to the text, good money management
skills include
A. Planning, budgeting, and living within your
means
B. Living within your means and using credit
cards to help.
C. Setting goals, budgeting, living within your
means, and using credit cards wisely
D. Living on a budget and using credit cards
when necessary
E. Living on a budget without credit cards.
Use your knowledge from the text and the
class discussions of Hitler’s strategies to
suggest how he lost World War II.
According to the text and class discussions, which of the
following would be a reaction of an insecure/avoidant baby
to the strange situation research technique?
A. The baby would show great upset when the parent
departed and would cling to the parent upon his/her
return.
B. The baby may or may not show great upset when the
parent departed, but would cling to the parent upon
his/her return.
C. The baby would show great upset when the parent
departed, but would ignore the parent upon his/her
return.
D. The baby may or may not show upset when the parent
departed, but would ignore the parent upon his/her
return.
Common Errors Made by
Teachers in Constructing Test
Items
Don’t give the answer away!
Avoid these common errors.
Test Construction Errors to Avoid
• Don’t use “All of the
above” or “None of
the above” only when
they are the correct
answer.
• Avoid distractors that
are a joke or an insult.
Test Construction Errors to Avoid
• Make sure the stem of
the question and the
distractors/answer are
grammatically correct
• Avoid having one
choice that is
extremely longer than
the other choices.
Test Construction Errors to Avoid
• If all the distractors
are numbers, 90% of
the time the correct
answer is not the
highest number or the
lowest number.
• Avoid using absolute
determiners in the
answer choices.
Test Construction Errors to Avoid
• Avoid using more
inclusive options.
• Avoid answer options
that state a reason.
Learning how to play the piano while
singing occurs in the
A. Affective domain
B. Motor domain
C. Cognitive domain
D. Motor and cognitive domains
E. None of the above
According to the text, those who believe in
the work ethic believe that
A. All people will do good work.
B. Hard work is always rewarded.
C. Work is good and important.
D. Only good workers should receive pay.
The normal adult temperature of 98.6
degrees F is equivalent to
A. 18 degrees C
B. 27 degrees C
C. 37 degrees C
D. 42 degrees C
Summary
• Use three-part questions to make sure you are
asking questions that cover all of Bloom’s
Taxonomy
• Write objectives to measure all aspects of Bloom’s
Taxonomy
• Have students create questions and grade
questions
• Use other methods instead of just paper/pencil
tests to assess students in higher order thinking
skills