CELEBRATE THE STRUGGLE:
…it is the Essential Component of RIGOR!
R
Rigor
4
Relevance
Results
Relationships
Making a Difference:
A Vertical Articulation of Thinking
“The difference between who you are and what you want
to be is what you do.”
~Bill Phillips
Begin with the Rigor!
Prepared for the Learning Leadership Network of
HICKMAN MILLS PUBLIC SCHOOL DISTRICT
by Dan Mulligan, Ed. D., flexiblecreativity.com
August 2013
3
TABLE OF CONTENTS
Vertical articulation of Collaboration
2
Rigor: Depth of Thinking
5
Rigor: Type of Thinking
10
Common Core Math Standards: Overview
14
Common Core Math Process Standards
15
Effective Feedback
19
Depth of Knowledge (Universal Design)
23
Depth of Knowledge: Language Arts/Math/Primary
27
Cognitive Rigor Matrix for Comprehensive ELA Assessment
28
Framework for Instructional Planning
31
Notes Page
41
Sample Essential Vocabulary
42
Administrators’ Role in RIGOR
43
Functional Text
47
Habits of Mind
48
RIGOR = TYPE of THINKING + DEPTH of THINKING
DEPTH OF KNOWLEDGE
Depth of THINKING
This body of work offers the Depth of Knowledge (DOK) model employed to analyze the
cognitive expectation demanded by standards, curricular activities, and assessment
tasks (Webb, 1997). The model is based upon the assumption that curricular elements
may all be categorized based upon the cognitive demands required to produce an
acceptable response. Each grouping of tasks reflects a different level of cognitive
expectation, or depth of knowledge, required to complete the task. It should be noted that
the term knowledge, as it is used here, is intended to broadly encompass all forms of
knowledge (i.e. procedural, declarative, etc.). The following table reflects an adapted
version of the model.
DOK Level
Title of Level
1
Recall & Reproduction
2
Basic Skills & Concepts
3
Strategic Thinking & Reasoning
4
Extended Thinking
As an administrator/teacher reflect on the level of thinking experienced by
students by the ‘conclusion’ of most unit learning in your school. Where are
you currently? Where do we want to be? How will you get there?
5
 RECALL AND REPRODUCTION
Curricular elements that fall into this category involve basic tasks that require students to recall or
reproduce knowledge and/or skills. The subject matter content at this particular level usually involves
working with facts, terms and/or properties of objects. It may also involve use of simple procedures
and/or formulas. There is little transformation or extended processing of the target knowledge
required by the tasks that fall into this category. Key words that often denote this particular level
include: list, identify and define. A student answering a Level 1 item either knows the answer or does
not; that is, the answer does not need to be “figured out” or “solved.”
POSSIBLE PRODUCTS
QUIZ
DEFINITION
FACT
WORKSHEET
TEST
LABEL
LIST
COLLECTION
PODCAST
WORKBOOK
EXPLANATION
REPRODUCTION
SHOW AND TELL
CATEGORIZING/
TAGGING
COMMENTING
VOCABULARY QUIZ
OUTLINE
BULLETING
RECITATION
BLOG
HIGHLIGHTING
EXAMPLE
WIKI
SOCIAL
NETWORKING
SOCIAL
BOOKMARKING
SEARCHING
GOOGLING
ROLES
STUDENT
TEACHER
DIRECTS
TELLS
RESPONDS
ABSORBS
SHOWS
EXAMINES
REMEMBERS
RECOGNIZES
QUESTIONS
EVALUATES
MEMORIZES
DESCRIBES
DEMONSTRATES
LISTENS
EXPLAINS
TRANSLATES
COMPARES
CONTRASTS
RESTATES
DEMONSTRATES
EXAMINES
INTERPRETS
NOTES TO SELF:
6
 BASIC SKILLS AND CONCEPTS
This level generally requires students to contrast or compare people, places, events and concepts;
convert information from one form to another; classify or sort items into meaningful categories ;
describe or explain issues and problems, patterns , cause and effect, significance or impact,
relationships, points of view or processes. A Level 2 “describe or explain” would require students to
go beyond a description or explanation of recalled information to describe or explain a result or “how”
or “why.” The learner should make use of information in a context different from the one in which it
was learned.
POSSIBLE PRODUCTS
PHOTOGRAPH
INTERVIEW
LINKING
SOLVE
MIND MAPS
PRESENTATION
BLOG COMMENTING
ILLUSTRATION
CRACKING CODES
REVERSEENGINEERING
BLOG REFLECTING
SIMULATION
PERFORMANCE
MODERATING
SCULPTURE
DEMONSTRATION
EXPLAIN
CATEGORIZE
GENERALIZE
TRANSLATE
ESTIMATE
JOURNAL
ROLES
STUDENT
TEACHER
SHOWS
FACILITATES
SOLVE PROBLEMS
OBSERVES
EVALUATES
CALCULATES
DEMONSTRATES USE OF
KNOWLEDGE
COMPILES
ORGANIZES
QUESTIONS
COMPLETES
ILLUSTRATES
CONSTRUCTS
NOTES TO SELF:
7
 STRATEGIC THINKING & REASONING
Items falling into this category demand a short-term use of higher order thinking processes, such as
analysis and evaluation, to solve real-world problems with predictable outcomes. Stating one’s
reasoning is a key marker of tasks that fall into this particular category. The expectation established
for tasks at this level tends to require coordination of knowledge and skill from multiple subject-matter
areas to carry out processes and reach a solution in a project-based setting. Key processes that
often denote this particular level include: analyze, explain and support with evidence, generalize, and
create.
POSSIBLE PRODUCTS
GRAPH
SPREADSHEET
CHECKLIST
EVALUATING
SURVEY
DEBATE
CONCLUDE
PODCAST
DATABASE
PANEL
PROGRAM
PUBLISHING
REPORT
FILM
CHART
ABSTRACT
ANIMATION
OUTLINE
INVESTIGATE
VIDEO CAST
ROLES
STUDENT
TEACHER
PROBES
GUIDES
DISCUSSES
UNCOVERS
OBSERVES
EVALUATES
DEBATES
THINKS DEEPLY
ACTS AS A RESOURCE
QUESTIONS
EXAMINES
QUESTIONS
ORGANIZES
DISSECTS
JUDGES
DISPUTES
CLARIFIES
ACCEPTS
ASSESSES
DECIDES
JUSTIFIES
TESTS
SELECTS
COMPARES
GUIDES
NOTES TO SELF:
8
 EXTENDED THINKING
Curricular elements assigned to this level demand extended use of higher order thinking processes
such as synthesis, reflection, assessment and adjustment of plans over time. Students are engaged
in conducting investigations to solve real-world problems with unpredictable outcomes. Employing
and sustaining strategic thinking processes over a longer period of time to solve the problem is a key
feature of curricular objectives that are assigned to this level. Key strategic thinking processes that
denote this particular level include: synthesize, reflect, conduct, and manage.
POSSIBLE PRODUCTS
FILM
PLAN
PROJECT
NEW GAME
NEWSPAPER
SONG
MEDIA PRODUCT
STORY
ROLES
STUDENT
TEACHER
FACILITATES
EXTENDS
DESIGNS
FORMULATES
REFLECTS
ANALYZES
TAKES RISKS
MODIFIES
PROPOSES
PLANS
EVALUATES
CREATES
NOTES TO SELF:
9
TYPE of THINKING
11
15
16
17
Rigor-Infused Math Inquiry: FACILITATING MATH-TALK
Questions I currently use to guide and encourage student thinking:
Questions I can add to my toolkit:
What then is effective feedback?
“The most powerful single
moderator that enhances
achievement is feedback”
Quality feedback should:
• focus on the learning intention of the task
John Hattie. University of Auckland. 1999 Inaugural
• occur as the students are doing the learning
• provide information on how and why the student understands and misunderstands
• provides strategies to help the student to improve
• assist the student to understand the goals of the learning
Key Research Findings







When feedback is corrective in nature—that is, it explains where and why students
have made errors--significant increases in student learning occur (Lysakowski &
Walberg, 1981, 1982; Walberg, 1999; Tennenbaum & Goldring, 1989).
Feedback has been shown to be one of the most significant activities a teacher can
engage in to improve student achievement (Hattie, 1992).
Asking students to continue working on a task until it is completed and accurate (until
the standard is met) enhances student achievement (Marzano, Pickering, & Pollock, 2001).
Effective feedback is timely. Delay in providing students feedback diminishes its value
for learning (Banger-Drowns, Kulik, Kulik, & Morgan, 1991).
Administer tests to optimize learning. Giving tests a day after a learning experience is
better than testing immediately after a learning experience (Bangert-Downs, Kulik,
Kulik, & Morgan, 1991).
Rubrics provide students with helpful criteria for success, making desired learning
outcomes clearer to them. Criterion-referenced feedback provides the right kind of
guidance for improving student understanding (Crooks, 1988; Wilburn & Felps, 1983).
Effective learning results from students providing their own feedback, monitoring their
work against established criteria (Trammel, Schloss, & Alper, 1994; Wiggins, 1993).
Additional Resources
RubiStar is a free online tool that teachers can use to make and save rubrics. Developed by the High Plains
Regional Technology in Education Consortium, RubiStar includes a tutorial for new users and a feature that
enables teachers to analyze student data and identify areas for focusing additional instruction.
http://rubistar.4teachers.org/index.php
The National Center for Research on Cultural Diversity and Second Language Learning has published an article
called The Instructional Conversation: Teaching and Learning in Social Activity. The authors discuss the use of
modeling, providing feedback, contingency management, directing, questioning, explaining, and task structuring in
classroom activity settings.http://www.ncela.gwu.edu/pubs/ncrcdsll/rr2.htm
19
20
22
Depth of THINKING
Depth of Knowledge (DOK) Levels for MATHEMATICS
DOK Level 1 (Recall) includes the recall of information such as fact, definition,
term, or a simple procedure, as well as performing a simple algorithm or applying a
formula. That is, in mathematics a one-step, well-defined, and straight algorithmic
procedure should be included at this lowest level. Other key words that signify a Level
1 include "identify," "recall," "recognize," "use," and "measure." Verbs such as
"describe" and "explain" could be classified at different levels depending on what is to
be described and explained.
Level 2 (Skill/Concept) includes the engagement of some mental processing
beyond a habitual response. A Level 2 assessment item requires students to make
some decisions as to how to approach the problem or activity, whereas Level 1
requires students to demonstrate a rote response, perform a well-known algorithm,
follow a set procedure (like a recipe), or perform a clearly defined series of steps.
Keywords that generally distinguish a Level 2 item include "classify," "organize,"
"estimate," "make observations," "collect and display data," and "compare data." These
actions imply more than one step. For example, to compare data requires first
identifying characteristics of the objects or phenomenon and then grouping or ordering
the objects. Some action verbs, such as "explain," "describe," or "interpret" could be
classified at different levels depending on the object of the action. For example, if an
item required students to explain how light affects mass by indicating there is a
relationship between light and heat, this is considered a Level 2. Interpreting
information from a simple graph, requiring reading information from the graph, also is a
Level 2.
Interpreting information from a complex graph that requires some decisions on what
features of the graph need to be considered and how information from the graph can
be aggregated is a Level 3. Caution is warranted in interpreting Level 2 as only skills
because some reviewers will interpret skills very narrowly, as primarily numerical skills,
and such interpretation excludes from this level other skills such as visualization skills
and probability skills, which may be more complex simply because they are less
common. Other Level 2 activities include explaining the purpose and use of
experimental procedures; carrying out experimental procedures; making observations
and collecting data; classifying, organizing, and comparing data; and organizing and
displaying data in tables, graphs, and charts.
24
Level 3 (Strategic Thinking) requires reasoning, planning, using evidence,
and a higher level of thinking than the previous two levels. In most instances, requiring
students to explain their thinking is a Level 3. Activities that require students to make
conjectures are also at this level. The cognitive demands at Level 3 are complex and
abstract. The complexity does not result from the fact that there are multiple answers,
a possibility for both Levels 1 and 2, but because the task requires more demanding
reasoning. An activity, however, that has more than one possible answer and requires
students to justify the response they give would most likely be a Level 3. Other Level 3
activities include drawing conclusions from observations; citing evidence and
developing a logical argument for concepts; explaining phenomena in terms of
concepts; and using concepts to solve problems.
Level 4 (Extended Thinking) requires complex reasoning, planning,
developing, and thinking most likely over an extended period of time. The extended
time period is not a distinguishing factor if the required work is only repetitive and does
not require applying significant conceptual understanding and higher-order thinking.
For example, if a student has to take the water temperature from a river each day for a
month and then construct a graph, this would be classified as a Level 2. However, if
the student is to conduct a river study that requires taking into consideration a number
of variables, this would be a Level 4. At Level 4, the cognitive demands of the task
should be high and the work should be very complex. Students should be required to
make several connections-relate ideas within the content area or among content
areas-and have to select one approach among many alternatives on how the situation
should be solved, in order to be at this highest level. Level 4 activities include
designing and conducting experiments; making connections between a finding and
related concepts and phenomena; combining and synthesizing ideas into new
concepts; and critiquing experimental designs.
25
Mathematics Item Depth of Knowledge Indicators
DOK 1
RECALL
Examples represent, but do not
constitute all Level 1 mathematics
performances:
 Recall or recognize a fact, ,definitions,
or term
 Apply a well-known algorithm
 Apply a formula
 Determine the area or perimeter of
rectangles or triangles given a drawing
and labels
 Identify a plane or three dimensional
figure
 Measure a length
 Perform a specified or routine
procedure
 Evaluate an expression
 Solve a one-step word problem
 Retrieve information from a table or
graph
 Recall, identify, or make conversions
between and among representations or
numbers (fractions, decimals, and
percents), or within and between
customary and metric measures
 Locate numbers on a number line, or
points on a coordinate grid
 Solves linear equations
 Represent math relationships in words,
pictures, or symbols
DOK 2
SKILL/CONCEPTS
DOK 3
STRATEGIC THINKING
DOK 4
EXTENDED THINKING
Examples represent, but do not
Examples represent, but do not
constitute all Level 2 mathematics constitute all Level 3
performances:
mathematics performances:
Examples represent, but do not
constitute all Level 4
mathematics performances:
 Classify plane and three
dimensional figures
 Interpret information from a
simple graph
 Use models to represent
mathematical concepts
 Solve a routine problem
requiring multiple steps, or the
application of multiple
 concepts
 Compare figures or statements
 Compare and contrast figures
 Provide justifications for steps
in a solution process
 Extend a pattern
 Retrieve information from a
table, graph, or figure and use
it solve a problem requiring
multiple steps
 Translate between tables,
graphs, words and symbolic
 notation
 Select a procedure according
to criteria and perform it
 Relate mathematical concepts
to other content areas
 Relate mathematical concepts
to real-world applications in
new situations
 Apply a mathematical model
to illuminate a problem,
situation
 Conduct a project that
specifies a problem, identifies
solution paths, solves the
problem, and reports results
 Design a mathematical model
to inform and solve a practical
or abstract situation
NOTE: Level 4 requires
applying one approach among
many to solve problems.
Involves complex restructuring
of data, establishing and
evaluating criteria to solve
problems.
 Interpret information from a
complex graph
 Explain thinking when more
than one response is possible
 Make and/or justify
conjectures
 Develop logical arguments for
a concept
 Use concepts to solve
problems
 Perform procedure with
multiple steps and multiple
decision points
 Generalize a pattern
 Describe, compare, and
contrast solution methods
 Formulate a mathematical
model for a complex situation
 Provide mathematical
justifications
 Solve a multiple- step
problem, supported with a
mathematical explanation that
justifies the answer
 Formulate an original
problem, given a situation
Using Depth of Knowledge
Following are some questions to consider when analyzing your curriculum tasks for DOK.
1. What level of work are the students most commonly required to perform?
2. What is the complexity of the task rather than the difficulty?
3. What are all the skills and knowledge scaffolding that the students will have already needed to build
to complete the task?
Examples of what Depth of Knowledge ‘sound-like’ in the classroom
Content Area
Elementary
Language Arts
Level 1
Sort known
words as quickly
as possible
Elementary
Math
Collect data on
the number of
teeth lost by
students in one
2nd grade
classroom.
Elementary
Interdisciplinary
List the
ingredients of a
peanut butter
and jelly
sandwich.
Level 2
Find words in text
that illustrate a
defined pattern.
Level 3
Create an open sort
and define the rule
and explain.
Organize these
Using the graph,
data using a grph
predict how many
or chart (e.g., a line teeth would be lost by
plot
all the 2nd grade
classes in the school
and justify your
answer
Collect the
Investigate how many
ingredients for a
people are coming to
peanut butter and
dinner and formulate
jelly sandwich and
the appropriate
write the recipe.
amounts of
ingredients for that
many people.
Level 4
Illustrate through authentic
writing stability in pattern and
content of identified stages.
Create a model to estimate
how many teeth are lost be 2nd
grade students in the United
States in one year. Include the
type of data needed to collect
and explain how your model
works.
Research and design a system
to lift heavy objects using the
conversion of electrical energy
to mechanical energy. Build a
prototype of the system using
materials found in the
classroom.
27
29
30
32
33
34
35
36
37
38
39
40
Personal Reflections
Dan Mulligan
Flexiblecreativity.com
41
42
The Role of the Administrator in RIGOR for ALL:
Few people question the need for America’s schools and classrooms to be more rigorous. But there
is little agreement about what rigor is and what it looks like.
In Rigor is NOT a Four-Letter Word, Barbara Blackburn defined rigor as creating an environment
in which each student is expected to learn at high levels, each student is supported so that he
or she can learn at high levels, and each student demonstrates learning at high levels
(Blackburn, 2008). This three-part approach assures that rigor doesn’t consist of just adding
curriculum requirements or raising grading standards. Integral to the model is providing every student
with high levels of support so that they can thrive and be successful in their classrooms.
Rigor is more than a specific lesson or instructional strategy. It is deeper than what a student
says or does in response to a lesson. Real rigor is the result of weaving together all elements of
schooling to improve the achievement and learning of every student.
Expectations
We’ll start with the first part: rigor is creating an environment in which each student is expected
to learn at high levels. Having high expectations starts with the decision that every
student possesses the potential to be his or her best, no matter what.
Almost everyone we talk with says that they have high expectations for their students. Sometimes
that is evidenced by the behaviors in the school; at other times, actions don’t match the words. When
you visit classrooms and work with your teachers, consider using the following tools to assess the
level of rigor you see.
As you work with teachers to design lessons that incorporate more rigorous opportunities for learning,
you will want to consider the questions that are embedded in the instruction. Higher-level
questioning is an integral part of a rigorous classroom. Look for open-ended questions that are
at the higher levels of Bloom’s Taxonomy (analyze, evaluate, create).
It is also important to look at how teachers respond to student questions. It is not uncommon for
teachers who ask higher-level questions to accept low-level responses from students. In rigorous
classrooms, however, teachers push students to respond at high levels. They ask extending
questions. If a student does not know the answer, the teacher continues to probe and guide the
student to an appropriate answer rather than moving on to the next student.
My thoughts regarding my school:
43
Tool 1: Questions and Responses
You can use this tally tool to chart your observations about questioning techniques and talk with
teachers about questions in class.
QUESTIONS ASKED BY TEACHER
Low-level/rote questions
High-level/application questions
TEACHER RESPONSE TO STUDENT ANSWERS
Low-level student responses
accepted
High-level responses accepted or
probing/extended questions asked
Supports (rigor with nurturing)
High expectations are important, but the most rigorous schools also ensure that each student
is supported so he or she can learn at high levels, the second part of our definition. It is essential
that teachers design lessons that move students to more challenging work while simultaneously
providing ongoing scaffolding to support students learning as they those higher levels.
Providing additional scaffolding throughout lessons is one of the most important ways to support
students. This can occur in a variety of ways, but it requires that teachers ask themselves during
every step of their lesson, “What extra support might my students need?”
44
Tool 2: Scaffolding in Lessons
Examples of scaffolding strategies include:






Asking guiding questions
Chunking information
Color-coding the steps of a project
Writing standards as questions for students to answer
Using visuals and graphic organizers
Providing such tools as interactive reading guides, study guide.
Demonstration
The third component of a rigorous classroom provides each student with opportunities to
demonstrate learning at high levels. We often hear that if teachers provide more challenging
lessons that include extra support, then learning will happen. If we want students to show us that they
understand what they learned at a high level, we also need to provide opportunities for students to
demonstrate that they have truly mastered that learning. One way to accomplish that is through
increased student engagement.
Student engagement is a key aspect of rigor. In too many classrooms, most of the instruction consists
of the teacher-centered large group instruction, perhaps in an interactive lecture or discussion format.
The general practice during these lessons is for the teacher to ask a question, and then call on a
student to respond. Although this provides an opportunity for one student to demonstrate their
understanding, the remaining students don’t have the opportunity to do so. Another option would be
for the teacher to allow all students to respond either through pair-share (talk to your shoulder
partner), writing answers on small whiteboards and sharing their responses, or responding on a
handheld computer that tallies responses. Such activities hold each student accountable for
demonstrating his or her understanding.
Tool 3: Student Engagement
Negative Indicators
Positive Indicators
 One student responds
 Two or three students discuss
content
 Teacher asks students if they
understand, and they answer with a
simple yes or no; there is no
probing.
 All students respond
 All students discuss content in small
groups
 All students write a response in a
journal or exit slip
45
When you talk with your teachers about their instructional practices, you can also ask them about
engagement and how they design lessons to promote positive student engagement and high levels of
student accountability for demonstrating learning.
Tool 4: Talking with Teachers





Talk with me about how you make your lessons engaging. What information do you use to guide
your decisions?
As you design your lessons, what are some of the strategies you use to make sure the lesson is
engaging to students?
While teaching a lesson, how do you monitor the engagement of your students?
As you continue to work on student engagement, what would you identify as the most appropriate
next steps?
What may I do to support you in your work to improve student engagement?
Final Thoughts and Action Planning
Recognizing rigor in classrooms is all about recognizing good instruction. It is important to look
for instructional practices that expect students to learn at very high levels and that also give students
the support to achieve at high levels. It is also essential that teachers expect all students to
demonstrate their learning at high levels.
46
47
48