Oregon Rubric & Quality Review
Training Session: Mathematics
Pilot Review Process
What is a Pilot Review?
• Oregon has specific laws that outlines how state reviews are
carried out and when (ORS 337 & OAR 581-11).
–
–
–
–
30+ year old laws (Basal focused, seven year review cycle)
Review is sustained by publisher fees
State math review has been moved from 2014 to 2016
Will work in the 2015 session to update the state review process
• Districts are able to independently review and adopt at any
time, provided they use the board approved criteria (OAR
581-022-1622)
–
–
–
–
Pilot process to support the local review of materials
Summer 2014: SOESD and Hillsboro “Pilot” review process for math
Fall 2014: share results and supporting documents
Summer 2015: aim to support more regional reviews of math
Session Goals
Use the Oregon quality review process (based on EQuIP and IMET) to determine the
quality and alignment of lessons and units to the Common Core State Standards
(CCSS) in mathematics
During this session, reviewers will:
• Develop their abilities to use Oregon math materials criteria to provide
observations about CCSS-aligned instructional materials and make suggestions for
improvement
• Develop a common understanding of the Oregon quality review process
• Develop a common understanding of the rating scale and descriptors for the four
rubric dimensions and the rating categories and descriptors for overall ratings
• Develop their abilities to use the criteria, rating scales and rating descriptors to
accurately rate instructional materials
Oregon Quality Review:
Principles & Agreements
1. CCSS: Before beginning a review, all members of a review team are familiar with the
CCSS.
2. Inquiry: Review processes emphasize inquiry rather than advocacy and are
organized in steps around a set of guiding questions.
3. Respect & Commitment: Each member of a review team is respected as a valued
colleague and contributor who makes a commitment to the review process.
4. Criteria & Evidence: All observations, judgments, discussions and recommendations
are criterion and evidence based.
5. Constructive: Lessons/units to be reviewed are seen as “works in progress.”
Reviewers are respectful of contributors’ work and make constructive observations
and suggestions based on evidence from the work.
6. Individual to Collective: Each member of a review team independently records
his/her observations prior to discussion. Discussions focus on understanding all
reviewers’ interpretations of the criteria and the evidence they have found.
7. Understanding & Agreement: The goal of the process is to compare and eventually
calibrate judgments to move toward agreement about quality with respect to the
CCSS.
Using the Quality Review Rubric
The Three Shifts in Mathematics
•Focus: Strongly where the
Standards focus
•Coherence: Think across
grades and link to major
topics within grades
•Rigor: Require conceptual
understanding, fluency, and
application
Activity 1:
Finding Focus & Coherence
Looking for Focus
• Taking a closer look at the major work of each
grade
• Two levels of focus:
• What’s in/What’s out
• The shape of the content that is in
Activity 1
Finding Focus in the CCSS
10
HS content
Activity 2a:
Finding Rigor
Rigor: In Major Topics, Pursue Conceptual
Understanding, Procedural Skill and Fluency,
and Application
• The CCSSM require a balance of:
 Conceptual understanding
 Procedural skill and fluency
 Application of skills in problem solving situations
• Pursuit of all three requires equal intensity in
time, activities, and resources.
The three legged stool
How do the Standards signal Rigor?
• Conceptual Understanding:
3.NF.1 Understand a fraction 1/b as the quantity formed by
1 part when a whole is partitioned into b equal parts;
understand a fraction a/b as the quantity formed by a parts
of size 1/b.
• Procedural Skill and Fluency:
5.NBT.5 Fluently multiply multi-digit whole numbers using
the standard algorithm.
• Application:
7.NS.3 Solve real-world and mathematical problems
involving the four operations with rational numbers.
Application
• Real-world problems (single- and multi-step)
• Non-routine problems
• Varied problem types (see Tables in CC.OA
progressions)
• Enhance major work of the grade
• Constructing models (6-12)
Conceptual Understanding
• Problems can (and should sometimes) be brief
• Explaining reasoning is one way to address
conceptual understanding
• Problems and exercises should be grade-level
appropriate
• Connections between representations are
emphasized
Procedural Skill and Fluency
• Purely procedural problems
• Opportunistic strategies; writers are
thoughtful about numbers used
• Repeated practice
• Procedures are built from conceptual
understanding
Activity 2b: Focus & Rigor
19
Activity 2b:
Looking for rigor in texts
20
Implement tasks that promote
reasoning and problem solving
• Dan Meyer: “Math Class Needs a Makeover”
Need to develop patient problem
solvers
• What are characteristics of a task that places:
– A low-level cognitive demand on students?
– A high-level cognitive demand on students?
• What does it mean for students to be
“patient” or “impatient” problem solvers?
– How can task selection and implementation
condition students to be one of these types of
problem solvers?
Layers of a math problem
24
Activity 3
Practice using OR-IMET
25
Using the Quality Review Rubric
OR-IMET
27
Using the Quality Review Rubric
http://tinyurl.com/odemath-omet
For each dimension:
•
Make observations and
suggestions related to criteria
and evidence.
•
Determine a rating for each
dimension based on checked
criteria and observations.
•
Additional comments to
improve the rating of the
material in this section
28
Grouping of Math Practices
Reasoning and Explaining
2. Reason abstractly and quantitatively
3. Construct viable arguments and critique the reasoning of others
Modeling and Using Tools
4. Model with mathematics
5. Use appropriate tools strategically
Seeing Structure and Generalizing
7. Look for and make use of structure
8. Look for and express regularity in repeated reasoning
Overarching Habits of Mind of a Productive Mathematical Thinker
1. Make sense of problems and persevere in solving them
6. Attend to precision
Adapted from (McCallum, 2011)
29
Look for evidence
30
31
Giving Feedback
Writing effective feedback is vital to the Quality Review Process. Below are the
four qualities of effective feedback.
• Criteria-based: Written comments are based on the criteria used for review
in each dimension. No extraneous or personal comments are included.
• Evidence Cited: Written comments suggest that the reviewer looked for
evidence in the lesson or unit that address each criterion of a given
dimension. Examples are provided that cite where and how the criteria are
met or not met.
• Improvement Suggested: When improvements are identified to meet
criteria or strengthen the lesson or unit, specific information is provided
about how and where such improvement should be added to the material.
• Clarity Provided: Written comment are constructed in a manner keeping
with basic grammar, spelling, sentence structure and conventions.
Feedback Example #1: Mathematics
This unit clearly targets three CCSS, which are noted in the overview. The
overview also indicates which Standards for Mathematical Practice are
central to the lesson. The activities throughout the unit present a balance
of mathematical procedures and deeper conceptual understanding of the
standards. The activities reinforce the standards and are well-connected to
the content. I think the activities might be challenging with a large class
with classroom management issues.
Is this feedback criteria-based?
Was evidence cited?
Was there an improvement suggested?
Is clarity provided?
Initial findings
Pilot review SOESD & Hillsboro
Programs Reviewed – Summer 2014
Southern Oregon ESD
• Elementary School
–
–
–
–
–
Bridges (K-5)
Engage NY (K-5)
Math Expressions
My Math
Investigations (incomplete
materials submitted)
• Middle School
–
–
–
–
–
Core Focus
Connected Math 3
Agile Mind
Go Math
Engage NY (6-8)
35
Programs Reviewed – Summer 2014
Southern Oregon ESD
• High School
–
–
–
–
–
–
HMH HS math (unpublished)
Big Ideas
College Prep Math
Core Plus
Pearson Math
Engage NY (attempted incomplete)
Hillsboro Regional Review
• High School
–
–
–
–
–
–
HMH HS math (unpublished)
College Prep Math
Pearson Math
McGraw Hill Math
CK-12
Engage NY (attempted incomplete)
36
Preliminary Results:
Elementary (SUM 14)
Program Name
Publisher
Review Site
“3 or 4”
Count
Percentage
“4”
Bridges
MLC
SOESD
10
50%
Engage NY
OER/Eureka
SOESD
9
0%
Math Expressions
HMH
SOESD
7
0%
My Math
McGraw-Hill
SOESD
2
0%
37
Preliminary Results:
Elementary (SOESD: SUM 14)
Program Name
1
2
3
4
5
6
7
8
9
10
Bridges
3
3
4
3
3
4
4
3
4
4
Engage NY
3
3
3
3
3
3
3
3
2
3
Math
Expressions
3
3
2
3
3
3
2
3
2
3
My Math
3
2
3
2
2
2
2
2
2
2
38
Preliminary Results:
Middle School (SUM 14)
Program Name
Publisher
Review Site
“3 or 4”
Count
Percentage
“4”
Core Focus
SMC
SOESD
10
90%
Connected Math 3
Pearson
SOESD
10
80%
Agile Mind
Agile Mind
SOESD
9
20%
Go Math
HMH
SOESD
6
0%
Engage NY
OER/Eureka
SOESD
2
0%
39
Preliminary Results:
Middle School (SOESD: SUM 14)
Program Name
1
2
3
4
5
6
7
8
9
10
Core Focus
4
4
4
4
4
4
4
3
4
4
Connected Math 3
3
4
4
4
3
4
4
4
4
4
Agile Mind
3
3
4
3
3
3
3
2
4
3
Go Math
3
3
2
2
3
2
2
2
3
3
Engage NY
2
2
2
2
2
2
3
2
3
2
40
Preliminary Results:
High School (SOESD & HSD: SUM 14)
Program Name
Publisher
Review Site
“3 or 4”
Count
Percentage
“4”
Core Plus Math
HMH
SOESD
10
80%
College Prep Math
CPM
SOESD
10
80%
Big Ideas
HMH
SOESD
10
80%
HMH Math
Unpublished
SOESD
10
70%
HMH Math
Unpublished
HSD
10
70%
Pearson Math
Pearson
SOESD
7
10%
College Prep Math
CPM
HSD
6
0%
Pearson Math
Pearson
HSD
5
0%
McGraw Hill
McGraw-Hill
HSD
3
0%
CK-12
OER
HSD
2
0%
Engage NY
SOESD/HSD
Incomplete
Agile Mind
SOESD
Incomplete
41
Preliminary Results:
High School (SOESD & HSD: SUM 14)
Program Name
Review
Site
1
2
3
4
5
6
7
8
9
10
Core Plus Math
SOESD
4
4
4
4
4
4
3
3
4
4
College Prep Math
SOESD
4
4
4
4
3
3
4
4
4
4
Big Ideas
SOESD
4
4
3
4
4
4
3
4
4
4
HMH Math (unpub.)
SOESD
4
4
4
4
4
3
3
4
4
3
HMH Math (unpub.)
HSD
4
4
4
4
4
3
4
4
3
3
Pearson Math
SOESD
3
3
2
2
3
3
2
3
4
3
College Prep Math
HSD
3
3
3
3
2
3
2
3
2
2
Pearson Math
HSD
3
3
3
3
3
2
2
2
2
2
McGraw Hill
HSD
3
3
2
2
3
2
2
2
2
2
CK-12
HSD
3
2
1
1
3
2
2
1
1
2
Engage NY
SOESD
(partial)
2
2
2
2
3
42
Lessons Learned &
Moving Forward
43
Lessons Learned from Sum 14
• Training & Calibration is difficult and non-trivial
– Significant Refinement from SOSED to HSD
– Importance of providing practice with real programs
• Understanding quality criteria valuable regardless if
doing a formal review
– Spill over effect of understanding concepts like focus and rigor in a
new context
– Need to understand quality as materials are organized or created
• Strong interest in this work
– Need for both purchase and creation of materials
– Need to continue in 2015
44
Looking ahead to 2014-15
• Establishment of review cohort
– Provide training to 3-4 leaders from 6-7 regions in the state (~24
statewide)
– Can review materials/facilitate reviews Sum 2015
• PLT conferences
– First day general sessions/Second day breakouts (math)
– Fall training on Finding, evaluating, & modifying resources
• Multi-State OER collaborative
– ~10 states have agreed to support the development of CCSS
OER courses in ELA and Math (including Oregon)
– Coordinated by CCSSO and Creative Commons
– RFP Fall 2014, courses as early as Sum/Fall 2015
45
Questions?
• Mark Freed
Mathematics Education Specialist
Oregon Department of Education
mark.freed@state.or.us
503-947-5610
46