Tools for Supporting Increased
Rigor & High-Quality Assessment
and Instruction
Karin K. Hess, Ed.D.
www.karin-hess.com
Presentation Overview
• Develop a shared understanding of the concept of
cognitive rigor/DOK
• Dispel some common DOK misconceptions
• Use DOK & the Hess Cognitive Rigor Matrices to:
– Examine what rigor/DOK looks like in action
• Sample videos, assessments/tasks, & rubrics
• Karin’s Coaching Tips: Analyzing assessment
tasks/rubrics & instruction in light of current research
• Provide tools & strategies to plan future work
& support to teachers and students
Track your reflections as we work…
Ways I am refining
my thinking about
DOK/rigor…
•?
•?
Scaffolding strategies
for getting students to
deeper thinking…
•?
•?
Before we begin…
• Take a minute to jot down words/phrases
that come to mind when you think of
“cognitive rigor” as it relates to instruction,
learning, and/or assessment.
Let’s apply your rigor definitions
Your class has just read some version of
Little Red Riding Hood.
• What is a basic comprehension question
you might ask?
• What is a more rigorous question you
might ask?
The Hess Cognitive Rigor Matrix integrates
Bloom + Webb
Different states/schools/teachers use
different models to describe cognitive
rigor. Each addresses something
different.
• Bloom – What type of thinking (verbs) is
needed to complete a task?
• Webb – How deeply do you have to
understand the content to successfully
interact with it? How complex is the
content?
Merging Bloom + Webb:
The thinking behind the
development of the Hess
Cognitive Rigor Matrix …
Bloom’s Taxonomy [1956 ] &
Bloom’s Cognitive Process Dimensions [2001]
Knowledge -- Define, duplicate,
label, list, name, order, recognize,
relate, recall
Remember Retrieve knowledge from
long-term memory, recognize, recall,
locate, identify
Comprehension -- Classify, describe,
discuss, explain, express, identify,
indicate, locate, recognize, report,
review, select, translate
Understand -- Construct meaning,
clarify, paraphrase, represent,
translate, illustrate, give examples,
classify, categorize, summarize,
generalize, predict…
Application -- Apply, choose,
demonstrate, dramatize, employ,
illustrate, interpret, practice, write
Apply -- Carry out or use a procedure
in a given situation; carry out or use
Analysis -- Analyze, appraise, explain
calculate, categorize, compare,
criticize, discriminate, examine
Analyze -- Break into constituent
Synthesis -- Rearrange, assemble,
collect, compose, create, design,
develop, formulate, manage, write
Evaluate -- Make judgments based
on criteria, check, detect
inconsistencies/fallacies, critique
Evaluation -- Appraise, argue,
assess, choose, compare, defend,
estimate, explain, judge, predict, rate,
core, select, support, value
Create -- Put elements together to
form a coherent whole, reorganize
elements into new patterns/
structures
/apply to an unfamiliar task
parts, determine how parts relate
Webb’s Depth-of-Knowledge Levels
• DOK-1 – Recall & Reproduction - Recall of a fact, term, principle,
concept, or perform a routine procedure
• DOK-2 - Basic Application of Skills/Concepts - Use of
information, conceptual knowledge, select appropriate procedures
for a task, two or more steps with decision points along the way,
routine problems applying 2+ concepts, organize/display data,
interpret/use simple graphs
• DOK-3 - Strategic Thinking - Requires reasoning, developing a
plan or sequence of steps to approach problem; requires some
decision making and justification; abstract, complex, or nonroutine; often more than one possible answer or approach
• DOK-4 - Extended Thinking - An original investigation or
application to real world; requires time to research, problem solve,
and process multiple conditions of the problem or task; nonroutine manipulations, across disciplines/content areas/multiple
sources
Turn & talk…
How would you describe some of the
differences between Bloom & Webb?
DOK Misconception #1:
All kids can’t do this; or Kids don’t need
scaffolding to get “up” there.
Engaging in “a complex task” with supports/ scaffolding is
an essential step along the way to proficiency (Vygotsky’s
ZPD)
– Do it with others first; DOK 3 and 4 are not meant to
only be done alone/independently, especially at first
– Oral language & meaningful discourse support deeper
thinking and increase initial exposures to the content
and student engagement. This is NOT cheating!
– One strategy: Plan questioning & formative probes
from DOK 1-2-3-4 over the course of a lesson or unit of
study. Consider all DOK levels in your planning.
Vygotsky: Zone of Proximal Development
(What a child can do with assistance today)
What a child can
do independently
now: “ENTRY”
Actual
Development
Area
The
ZONE
What a child can
do independently
tomorrow/future
Potential
Development
Area
LEARNING PROGRESSIONS ZONE:
Dynamic area
Causes development to move forward
Social interaction essential (scaffolding)
Karin Hess (2008). Using learning progressions as a schema to monitor progress across grades.
DOK Misconception #2:
Webb’s DOK model is a taxonomy
• Bloom’s is a taxonomy, intended to be a
hierarchy
• Primary Weaknesses of Bloom: generic
verbs (void of content); same verbs at
different levels
• Webb’s DOK model is nominative:
– It names how you interact with content; is contentspecific
– It differentiates varying levels of engagement with
content and suggests what tasks might look like
– DOK 4 is not better than DOK 3, or DOK 2, or DOK 1
DOK Misconception #3:
Bloom verbs & levels = Webb DOK
The DOK “Wheel of Misfortune” implies that a DOK level is
indicated by a particular verb or set of verbs.
Norman Webb, “It’s what comes after the verb, that indicates
the complexity of a task.”
DOK Misconception #4:
DOK is about difficulty.
• The intended student learning outcome determines the
DOK level. What mental processing must occur? DOK
= Complexity, not difficulty!
• While verbs may appear to point to a DOK level, it is what
comes after the verb that is the best indicator of the
rigor/DOK level and complexity of the task.
– Describe the information contained in graphics or data tables in
the text; or the rule for rounding a number
– Describe how the two story characters are alike and different.
– Describe the data or text evidence that supports your solution,
reasoning, or conclusions
– Describe varying perspectives on global climate change using
supporting scientific evidence, and identify the most significant
effects it might have on the planet in 100 yrs.
DOK LEVEL 1
Recall and Reproduction
Remember
DOK 2
Skills and
Concepts
Recall, locate
basic facts,
Understand
definitions,
details and
events
Select appropriate
word when
intended meaning
is clear
Explain
relationships
Summarize
Central ideas
Use language
structure or word
relationships
(synonyms/
antonyms)
Use context to find
meaning
Obtain and use
information in text
features
Identify information
in a graphic, table,
visual, etc.
Apply
Compare literary
elements, facts, terms
and events.
Analyze format,
organization & text
structures
“UGs”
Brainstorm ideas,
concepts, problems,
or perspectives
related to a topic
Generate
conjectures or
hypotheses based
on observations or
prior knowledge
DOK LEVEL 3
Reasoning
Analyze
Evaluate
Explain, generalize or connect ideas
using supporting evidence (quote,
text, evidence, data, etc.)
Use concepts to solve non-routine
problems and justify solutions with
evidence
DOK 4
Extended
Thinking
Create
Explain how
concepts relate to
other content
domains
Devise an approach
among alternatives
to research a novel
problem
Analyze or interpret author’s craft (e.g.,
literary devices, viewpoint, or potential
bias) to critique a text
Analyze multiple
sources or texts
Analyze complex
abstract themes.
Cite evidence and develop a logical
argument for conjectures based on
one text or problem.
Evaluate relevancy,
accuracy and
completeness of
information
Develop a complex model or
approach for a given situation
Develop an alternative solution
Synthesize information
across multiple sources
Articulate a new voice,
theme, perspective.
Bloom’s Taxonomy + Webb’s DOK = the Hess CRM
16
The Hess Cognitive Rigor Matrix Applies
Webb’s DOK to Bloom’s Cognitive Process
Dimensions
Depth +
Thinking
Level 1
Recall &
Reproduction
Remember
- Recall, locate basic
facts, details, events
Understand
Level 2
Skills &
Concepts
Level 3
Strategic
Thinking
Level 4
Extended
Thinking
- Select appropriate
words to use when
intended meaning is
clearly evident
- Specify, explain
relationships
- summarize
– identify main ideas
- Explain, generalize,
or connect ideas
using supporting
evidence (quote,
example, data …)
- Explain how
concepts or ideas
specifically relate to
other content
domains or concepts
Apply
- Use language structure
(pre/suffix) or word
relationships
(synonym/antonym) to
determine meaning
– Use context to identify
meaning of word
- Obtain and interpret
information using text
features
- Use concepts to
solve non-routine
problems
- Devise an approach
among many
alternatives to
research a novel
problem
Analyze
- Identify whether
information is contained
in a graph, table, text
feature, etc.
– Compare literary
elements, terms, facts,
events
– analyze format,
organization, & text
structures
- Analyze or interpret
author’s craft
(literary devices,
viewpoint, or
potential bias) to
critique a text
– Analyze multiple
sources
- Analyze
complex/abstract
themes
– Cite evidence and
develop a logical
argument for
conjectures
- Evaluate relevancy,
accuracy, &
completeness of
information
- Synthesize
information within
one source, data set,
or text
- Synthesize
information across
multiple sources or
texts
Evaluate
Create
“UG” = unsubstantiated
generalization
- Brainstorm ideas about
a topic
- Generate conjectures
based on observations or
prior knowledge
DOK Misconception #5:
All DOK levels can be assessed with a
multiple choice question
• That’s just dumb!
• Weak DOK 3 multiple choice items are possible;
but does selecting the best option (e.g., locate
supporting evidence for a theme) provide as
much insight as seeing HOW a student
formulates and reveals thinking?
• By their nature, DOK 3 and 4 questions/tasks
are more open-ended, generally take longer to
respond to/solve, and may have more than one
“appropriate right answer”
2. The DOK
Instruction
& Assessment
Matrix Instructional
Decisions…
Paths
Selected Response
Each standard has an assigned Depth
Performance
of Knowledge.
Constructed Response
Tasks
DOK 2
DOK 1
Skills and
Concepts
Recall and Reproduction
Remember
Understand
The DOK determines the cognitive
DOK 4
level of instruction.
DOK 3
Recall, locate
basic facts,
definitions,
details, events
Extended
Thinking
Reasoning and
Thinking
Select appropriate
words for use when
intended meaning
is clearly evident.
Explain
relationships
Summarize
State central idea
Use context for
word meanings
Use information
using text features
Apply
Analyze
Use concepts to solve
non-routine problems
and justify
Analyze or interpret
author’s craft (e.g.,
literary devices, viewpoint,
or potential bias) to
critique a text
.
Cite evidence and develop
a logical argument for
conjectures based on one
text or problem
Evaluate
Create
Explain, generalize or
connect ideas using
supporting evidence
(quote, evidence, data)
.
Develop a complex model or
approach for a given
situation
Develop an alternative
solution
-Explain how concepts
or ideas specifically
relate to other content
domains.
Devise an approach
among many
alternatives to
research a novel
problem
Analyze multiple
sources or multiple
text
Analyze complex
abstract themes
Evaluate relevancy,
accuracy and
completeness of
information across
texts or sources
Synthesize across multiple
sources/ texts
Articulate a new voice,
theme, or perspective
19
Let’s practice using the CRMback to Little Red Riding Hood
• Your sample questions – a basic and more
rigorous question
• Handout: Linking Research with Practice, Tool
#1: CRM template for Close Reading (page 4)
Depth +
Thinking
Level 1
Recall &
Reproduction
Remember
What color was
Red’s cape?
Who is this story
about?
Understand
Who are the main
characters?
What was the
story’s setting?
Level 3
Retell or summarize the
story in your own words.
What is the author’s
message or theme?
Justify your
interpretation using text
evidence.
Is this a realistic or
fantasy story?
Compare the wolf
character to the
character of Red. How
are they alike-different?
Evaluate
Create
Strategic Thinking/
Reasoning
Level 4
Extended
Thinking
Identify words/phrases
that helped you to
know the sequence of
events in the story.
Apply
Analyze
Level 2
Skills & Concepts
Write text messages
between Red & her
mother explaining the
wolf incident.
Is this a realistic or
fantasy story? Justify
your interpretation
using text evidence.
Are all wolves (in
literature) like the
wolf in this story?
Support your
response using
evidence from this
and other texts.
What is your opinion
about the cleverness of
the wolf? Justify your
opinion using text
evidence.
Which version has
the most satisfying
ending? (establish
criteria first, then
locate evidence)
Some other content examples…
• Your class will be learning about…
– Fractions/decimals; Data use
– Data use; scientific investigation
– Elements of art & principles of design
• Come up with a basic understanding
and more rigorous question you could
pose.
• Use a CRM Template to Plan Units
Depth +
Thinking
Level 1
Recall &
Reproduction
Remember
What is slope?
What is white
space?
Understand
Read, write, and
represent these
fractions
Describe why white
space is used.
Apply
Convert this
fraction to a
decimal
Add these fractions
Analyze
What kind of
graph or model is
this?
Which data point
shows ____?
Find examples…
Evaluate
UG - Which
team is the
best?
Create
How would you
demonstrate each
technique?
Level 2
Skills & Concepts
Level 3
Explain how you
solved this problem.
Why control
variables in the
investigation?
Construct an
argument to show
equivalence using
area, set, and linear
models
Use these data to
graph your
solution
Compare these
methods.
Which graph shows
how the data would be
displayed?
Create a card game
using fractions.
Create scenario
explained by a data
display.
Strategic Thinking/
Reasoning
Level 4
Extended Thinking
Conduct the
investigation, interpret
results, and support
conclusions with data
Interpret what was
happening in the event?
Justify your
interpretation using what
you know about slope.
Analyze more than one
product
(same time period,
medium,
theme drawing from
multiple contexts source
materials for the analyses
How would you rank
these ___? Justify
your rankings using
data that supports
your criteria.
Some say the NFL
settlement for player brain
injury is not adequate.
Evaluate both sides using
data to determine the
validity of this claim.
Integrate multiple
source materials with
intent to develop a
product
DOK Misconception #6:
Higher order thinking = deeper learning
• What we have thought of as “higher order”
(analysis, evaluation, creative thinking) might only
be engaging or fun…and not always deeper
• Many critical thinking examples do not go deep or
get to DOK 3 or 4 (e.g., interpret/solve and justify)
• Shift our thinking from “higher order” to deeper
learning, and that can mean:
– deeper understanding
– deeper application
– deeper analysis, etc.
The Hess CRM illustrates this
shift
Some general rules of thumb…
• If there is one correct answer, it is
probably level DOK 1 or DOK 2
– DOK 1: you either know it (can recall it, locate it, do it) or you
don’t know it
– DOK 2 (conceptual): apply one concept, then make a decision
before going on applying a second concept; express
relationship (if-then; cause-effect)
• If more than one answer/approach,
requiring evidence, it is DOK 3 or 4
– DOK 3: Must interpret, provide supporting evidence and
reasoning (not just HOW solved, but WHY it works– explain
reasoning for each step/decision made)
– DOK 4: all of “3” + use of multiple sources/data/ texts; initiate &
complete an investigation
DOK Misconception #7:
Multi-step or longer tasks, multiple texts, or
complex texts always means deeper thinking
• DOK 2 is not simply more than one step, it’s
applying more than one concept; DOK 2 is still
routine/typical (main idea, word problems, etc.)
• Simply reading more complex texts, but NOT
delving deeply into the text’s meaning/style/etc.,
is likely to still be DOK 1 or 2
• DOK 3 requires some aspect of openendedness and interpretation with justification or
support; DOK 4 = drawing from multiple sources
Where is more-less rigor
implied in any set of
curriculum standards?
Reading Standards
Depth +
Thinking
Level 1
Recall &
Reproduction
Remember
KEY DETAILS
Decode, read orally
Understand
Apply
Analyze
Evaluate
Create
Level 2
Skills &
Concepts
Level 3
Strategic
Thinking
Level 4
Extended
Thinking
KEY DETAILS
WORD MEANINGS- fill
in
CENTRAL IDEAS
SUMMARIZE
predict, infer
REASONING &
SUPPORT – interpret
theme, purpose, pt of
view/perspective
REASONING &
SUPPORT
–multiple texts
WORD STRUCTURE roots, affixes,
structure,
RELATIONSHIPS
synonyms-antonyms
WORD MEANINGSuse in context
USE TEXT
STRUCTURES &
FEATURES
TEXT STRUCTURES &
FEATURES – how is
the message
structured/ presented?
REASONING &
SUPPORT
–multiple texts
-compare
treatments
USE of TEXT
STRUCTURE or
FEATURES
LANGUAGE USEidentify non- literal
usage
ANALYSIS &
REASONING WITHIN
TEXTS
LANGUAGE USEinterpret non- literal
usage
ANALYSIS &
REASONING
ACROSS TEXTS
AUTHOR’s CRAFT
WITHIN A TEXT (e.g.,
language use
impact/intent; bias,
reasoning)
EVALUATE
AUTHOR’s
PURPOSE or
CRAFT ACROSS
TEXTS
Reading & Writing
Depth +
Thinking
Level 1
Recall &
Reproduction
Remember
KEY DETAILS
Understand
Apply
Level 2
Skills &
Concepts
Level 3
Strategic
Thinking
Level 4
Extended
Thinking
KEY DETAILS
WORD MEANINGS- fill
in
CENTRAL IDEAS
SUMMARIZE
predict, infer
REASONING &
SUPPORT – DEVELOP
theme, or pt of view/
perspective topic
REASONING &
SUPPORT
– use multiple texts
- compare or
elaborate
WORD STRUCTURE
RELATIONSHIPS
WORD MEANINGSLANGUAGE USE
USE OF TEXT
STRUCTURES &
FEATURES
INTEGRATE TEXT
STRUCTURES &
FEATURES into
Compositions
REASONING &
SUPPORT
USE of TEXT
STRUCTURES or
FEATURES
LANGUAGE USE
ANALYSIS &
REASONING WITHIN
TEXTS – RESEARCH
ANALYSIS &
REASONING
ACROSS TEXTS
for Writing
Develop reasoning
RESEARCH for
Writing;
Comparing themes
AUTHOR’s CRAFT
WITHIN A TEXT
Evaluate credibility of
sources
EVALUATE
AUTHOR’s
PURPOSE or CRAFT
ACROSS TEXTS
COMPOSE /REVISE
FULL TEXTS
COMPOSE FULL
TEXTS-sources
EDIT/CLARIFY
USE TECHNOLOGY
CITE SOURCES
Analyze
Evaluate
Create
WRITE/EDIT BRIEF
TEXTS
Math Content Standards & Math Practices
Depth +
Thinking
Level 1
Recall &
Reproduction
Remember
Know math facts,
terms
Understand
Attend to precision
Evaluate expressions,
plot point
Level 2
Skills & Concepts
Level 3
Strategic Thinking
(support with data,
equations, models, etc.)
Level 4
Extended
Thinking
(cross
domains)
Model with
mathematics
Construct viable
arguments
Estimate, predict,
observe, explain
relationships
Geometry proof
Integrate
concepts across
domains
Apply
Calculate,
measure, make
conversions
Make sense of
routine problems
Make sense of nonroutine problems
Design & conduct
a project
Analyze
Identify a pattern
Locate information in
table
Use tools
strategically
Classify, organize
data, extend a
pattern
Reason abstractly
Analyze multiple
sources of
evidence
Evaluate
Create
Generalize a pattern
Critique the
reasoning of others
Design a complex
model
For each assessment task (or rubric)…ask
– What is its purpose? (What content/skill is being
assessed? is there a ‘right’ answer?)
– What is the implied/intended rigor? (What mental
processing would you expect students to engage in?
Use the CRM to find descriptors.)
– Which standards does it REALLY assess? (content +
intended rigor)
– Does the scoring guide/rubric match content +
intended rigor?
– What would student responses tell a teacher if
students could/could not do all or part of this task?
(open-ended tasks, reasoning used) – next
instructional decisions are clear
Take-Away Message: Cognitive Rigor &
Some Implications for Assessment
• Begin with daily DOK3 classroom discourse!
• Assessing only at the highest DOK level (the
“ceiling”) will miss opportunities to know what
students do & don’t know – go for a range; end
“high” in selected/prioritized content
• Performance assessments can offer varying
levels of DOK embedded in a larger, more
complex task
• Planned formative assessment strategies and
tools can/should focus on differing DOK levels
Some resources worth exploring…
•
•
•
•
•
•
•
•
www.nms.org – national math science initiative – gr 3 – HS; also has ELA and
SS performance tasks with DOK designations
www.readworks.org – gr k-8; short literary & informational texts with CC Qs
http://www.louisianabelieves.com/resources/library/common-core-statestandards - LA gr 3-HS math & ELA tasks (some math samples are weak and
not all math DOK levels are correct, but generally good examples)
http://ccgpsmathematicsk-5.wikispaces.com/K5+Formative+Assessment+Lessons+%28FALs%29 - GA gr K-5 math
www.exemplars.com – sample math PAs for k-12; science PAs for k-8
Dan Meyer blog – math PAs for MS-HS; kids have to build the problems by
deciding what’s needed to solve them – good strategic thinking required
Hess & Gong (2014). Ready for college and career? Achieving the Common
Core Standards and beyond through deeper, student-centered learning
http://www.nmefoundation.org/resources/scl-2/ready-for-college-and-career
Links and resources on www.karin-hess.com:
–
–
–
Karin’s YouTube video – “Text Complexity Tools” (qualitative text analysis)
Karin’s vimeo DOK video – http://vimeo.com/18281415 (NYC Dept of Ed Common Core Library)
Karin’s YouTube DOK video – excerpt from CT workshop