Evaluating Open
Educational Resource
(OER) Objects
An Introduction to the
Achieve OER Rubrics
CC BY Achieve 2013
Open Educational Resources:
[With Webster’s Definitions]
O
Open: Containing an open license - no restrictions on remixing or reusing
E
Educational: Used for teaching and learning
R
Resource: Object used to support an effort or task
[A source of support or aid, especially one that can be readily
[Having no enclosing or confining barrier; not restricted to a particular group
or category of participants]
[Pertaining to the action or process of educating or being
educated]
drawn upon when needed]
2
Open Educational Resources (OER)
What are OER?
• OER are teaching, learning, and research resources that
contain an open license.
• They provide extraordinary opportunities for educators to
freely share knowledge and resources.
• They offer great potential for instructional innovation and
networks for sharing best practices.
• There are millions of OER objects available online.
3
Open Educational Resources (OER)
OER objects may include (but are not limited to):
•
•
•
•
•
•
•
•
•
•
Images
Applets
Games
Worksheets
Lesson plans
Original-source texts
Assessments
Units
Textbooks
Teacher/learner support materials
o NOTE: Any smaller component of a more complex object, that can exist as a stand-alone,
also qualifies as an “object” (e.g. a unit in a textbook, a lesson in a unit, or an activity in a
lesson).
4
The Achieve OER Evaluation Rubrics
How and why were the Achieve OER Rubrics created?
• Educators using the vast system of sharing need a method for filtering OER to meet
the needs of their students.
• The purpose of the rubrics is to provide a structure for evaluating an online resource
in a systematic, purposeful and comprehensive way.
• There are two ways of approaching the evaluation of a resource:
o Holistically evaluating the object with a single rating, (similar to the star-rating
systems used on Netflix, Yelp and Amazon)
o Separately evaluating each of the components of quality, as with the Achieve
OER Rubrics.
• The rubric criteria are based on Achieve protocols used to assist states in alignment
and quality review studies.
• The Achieve OER Evaluation Tool, hosting the rubrics on OERCommons.org, allows
a user to identify, evaluate and sort objects based on the specified essential
elements of quality.
5
Applying the Achieve OER Rubrics
How do the Achieve OER Rubrics work?
• The rubrics represent an evaluation system for objects found within Open
Educational Resources.
• They are hosted as an online evaluation tool on the repository,
OERCommons.org, but can also be used independently.
• They are applied to any content area.
o NOTE: At this stage only Common Core Standards for English Language
Arts/Literacy and Mathematics are available in the Achieve OER Evaluation
Tool. Content standards for History, Social Studies, Sciences, and technical
subjects are not yet available.
• Each rubric is applied independently to an object.
• They rate the potential, not the actual, effectiveness of an object.
• They use a five-point scoring system to describe levels of quality.
6
The Rubrics
There are Seven Rubrics on the Achieve OER Evaluation
Tool
Rubric I. Degree of Alignment to Standards
Rubric II. Quality of Explanation of the Subject Matter
Rubric III. Utility of Materials Designed to Support Teaching
Rubric IV. Quality of Assessment
Rubric V. Quality of Technological Interactivity
Rubric VI. Quality of Instructional Tasks and Practice Exercises
Rubric VII. Opportunities for Deeper Learning
o NOTE: Sometimes one or more of the rubrics may not be relevant to a particular
object. In those cases, a rating of N/A is appropriate.
7
The Rubrics’
Common Scoring Scale
The Detailed Scoring Descriptors are Specific to Each Rubric
3: Superior
2: Strong
1: Limited
0: Very weak / None
N/A: Rubric not applicable to the object
o NOTE: N/A should be used when a particular rubric does not apply to the object
being rated. This is not a pejorative score; it simply means it would be inappropriate
to apply this rubric to the particular object.
8
Rubric I:
Degree of Alignment to Standards
• Before the degree of alignment can be rated, an object must be aligned to
standards. (In the online tool only CCSS Math and ELA/Literacy are
available at this time.)
• Some objects will not have an alignment identified, either in the object or in
the online tool.
• Some objects may provide alignment suggestions or have been aligned by
earlier users in the online tool. However a rater may need to verify those
alignments.
• When using the online tool an alignment should be entered whenever
possible so that Rubric I can be used in the evaluation and so that
accurate data might be collected.
• Aligning objects accurately allows future users to sort by subject, grade
level, and even by a specific standard.
9
Rubric I:
Degree of Alignment to Standards
The following steps are not part of the OER evaluation process but are required for
reviewers who may need to align an object before rating or want to verify the standards
selected by previous raters.
Six Steps in Alignment to the CCSS:
1. Review the object thoroughly, working through tasks and/or exercises.
2. Compare your work with keys/solutions and other instructional support
materials.
3. Identify the content and performance(s) required in the object.
4. Identify the grade level for the object.
5. Match the grade level, content, and performance requirements to the
appropriate CCSS, including the Mathematical Practices and texts for ELA
Literacy.
6. Make a note of any requirements in the object that are not addressed in the
standard(s) and any requirements in the standard that are not addressed in the
object.
10
Rubric I:
Degree of Alignment to Standards
•
This rubric is applied to objects that have suggested alignments to the
standards.
•
It is designed specifically for the Common Core State Standards (may
possibly be used with other standards in the future).
•
Rubric I can be completed only after alignment has been proposed.
• Alignment review involves two major aspects:
o Content
o Performance expectations
• Rubric I is not applicable for an object for which no alignment has been (or
can be) suggested.
11
The Rubrics’
Common Scoring Scale: Rubric I
Rubric I – Superior Degree of Alignment to Standards
3: An object has superior alignment only if both of the following are true:
• All of the content and performance expectations in the identified standard are
completely addressed by the object.
• The content and performance expectations of the identified standard are the
focus of the object. While some objects may cover a range of standards that
could potentially be aligned, for a superior alignment the content and
performance expectations must not be a peripheral part of the object.
[Examples of mathematics and ELA/Literacy alignment ratings can be found at the end of this
section.]
12
The Rubrics’
Common Scoring Scale: Rubric I
Rubric I – Strong Degree of Alignment to Standards
2: An object has strong alignment for either one of two reasons:
• Minor elements of the standard are not addressed in the object.
• The content and performance expectations of the standard align
to a minor part of the object.
[Examples of mathematics and ELA/Literacy alignment ratings can be found at the end
of this section.]
13
The Rubrics’
Common Scoring Scale: Rubric I
Rubric I – Limited Degree of Alignment to Standards
1: An object has limited alignment if a significant part of the
content or performance expectations of the identified standard is
not addressed in the object, as long as there is fidelity to the part
it does cover.
[Examples of mathematics and ELA/Literacy alignment ratings can be found at the end of this
section.]
14
The Rubrics’
Common Scoring Scale: Rubric I
Rubric I – Very Weak Degree of Alignment to Standards
0: An object has very weak alignment for either one of two
reasons:
• The object does not match the intended standards.
• The object matches only to minimally important aspects of a
standard, cluster or domain. These objects will not typically be
useful for instruction of core concepts and performances
covered by the standard.
This rating should be used for any standard that has been misaligned to the object
[Examples of mathematics and ELA/Literacy alignment ratings can be found at the end of this
section.]
15
The Rubrics’
Common Scoring Scale: Rubric I
Rubric I – Degree of Alignment to Standards – Not
Applicable
N/A: This rubric does not apply for an object that has no
suggested standards for alignment.
NOTE: For Rubric I a 0- or 1-rating is preferable to N/A for those misaligned standards
selected by previous raters since only numerical ratings (0, 1, 2, or 3) are averaged
into the overall results for an object. All standards that are aligned should be rated and
an N/A would only be appropriate if it is not possible to align standards to the object or
if there is no intention for the object to align to standard, as in a set of raw data or a
map.
[Examples of mathematics and ELA/Literacy alignment ratings can be found at the end of this
section.]
16
Rubric I:
Degree of Alignment to Standards
MATH ALIGNMENT RATING EXAMPLES: An object has been aligned to CCSS
2.NBT.2, “Count within 1000; skip-count by 5s, 10s, and 100s.” Ratings would be…
3 – If the primary focus of the object is counting to 1000 and skip counting by 5’s, 10’s, and 100’s,
with any other content/performance requirements secondary to this.
For example, in an object that focuses on all these requirements, there may be an interactive game that
requires counting by 2’s, or relating counting to addition and subtraction, or an activity that requires
measurement by counting whole units. These would be peripheral to the overall focus of the object and
would not detract from the 3-rating.
2 – If the object requires counting to 1000 and skip counting by 5’s and 10’s, it aligns with the
majority of the standard’s requirements, with a minor part not addressed (counting by 100’s). OR If
it addresses all of the standard’s content and performance but only a peripheral way. For example,
counting by 100’s might be addressed only in an extension activity, which users may opt not to use
or may not even see.
1 – If the object requires counting with numbers to 500, it would be considered to have limited
alignment: the object aligns very closely with a limited part of the standard.
0 – If the object only addresses skip counting by 10’s, with no clear indication stated as to the
limitation (count to 1000). Or if the object requires only counting within 100.
17
Rubric I:
Degree of Alignment to Standards
ELA ALIGNMENT RATING EXAMPLES: An object is aligned to CCSS RI.9-10.8,
“Delineate and evaluate the argument and specific claims in a text, assessing whether
the reasoning is valid and the evidence is relevant and sufficient; identify false
statements and fallacious reasoning.” Ratings would be…
3 – If the object requires that students evaluate the argument of a text, assessing the validity of
its evidence and reasoning including identifying false statements and reasoning. The text and
instructional material must be constructed to include some false statements and reasoning.
2 – If the object requires that students evaluate the argument of a text, assessing the validity of
its evidence and reasoning, but did not include identifying false statements and reasoning. OR if
the object included all the range of evaluative performances, but the evaluation was a minor part
of the activities.
1 – If the object requires only delineating the argument but does not require students to
evaluate its reasoning or evidence.
0 – If the text of the object includes an argument, but none of the activities require that students
delineate or evaluate it.
18
Rubric II:
Quality of Explanation of Subject
Matter
• Rubric II is used to rate how well the subject matter is explained or
otherwise revealed in the resource.
• Subject explanation could be aimed at either the teacher or a student.
• This rubric could apply to objects used for a whole class, small groups, or
an individual student (e.g. self-tutoring object).
• Sometimes the subject matter explanation is included in teacher support
material and is intended to help a teacher learn or be refreshed on a
specific topic (the teacher becomes the learner).
• Do not use Rubric II to rate the quality of the explanation of how to
present the lesson (planning instructions) for an object.
• Use an N/A-rating only if there is no explanation of subject matter
included in the object or for objects not designed to explain subject
matter (e.g. a formula sheet or a map), even if it might be possible to use
the object to explain a subject.
19
The Rubrics’
Common Scoring Scale: Rubric II
Rubric II – Superior Quality of Explanation of the Subject
Matter
3: An object is rated superior only if all of the following are true:
• The object effectively provides comprehensive information so that the target
audience should be able to understand the subject matter.
• The object connects the subject matter with important associated concepts.
The object does not need to be augmented with additional explanation or
materials.
• The main ideas of the subject matter addressed in the object are clearly
identified for the learner.
For example a lesson on multi-digit addition provides a clear explanation of multiple strategies and
makes connections with place value, rather than simply showing procedure. Or a lesson clearly
explains why and how to analyze how an author develops ideas across extended text would make
connections among the various developmental steps and the various purposes the author has for
the text. These examples would be considered superior for explanation of subject matter.
20
The Rubrics’
Common Scoring Scale: Rubric II
Rubric II – Strong Quality of Explanation of the Subject Matter
2: An object is rated strong if it explains the subject matter in a
way that makes skills, procedures, concepts, and/or information
understandable. It falls short of superior in that it does not make
the connections among important associated concepts within the
content.
For example if an object’s focus is to compare decimals to hundredths, it would be
considered strong if it thoroughly explains how to compare 0.25 and 0.6 by comparing
digits in the same place value position. However it may not make the connection to
number line placement or conversion of the two decimals to fractions with a common
denominator and then comparing the numerators.
21
The Rubrics’
Common Scoring Scale: Rubric II
Rubric II –Limited Quality of Explanation of the Subject
Matter
1: An object is rated limited if it explains the subject matter
correctly but in a limited way.
• This is a cursory treatment of the content that is not developed
enough to serve someone attempting to learn the content for
the first time.
• The explanations are not thorough enough to serve as more
than a review for most learners.
For example an object addressing comparing the mean to the median and mode for a
set of data provides procedure but does not explain how the measures of center are
used or why one is preferable to another in a given situation. This is of limited quality for
explanation of subject matter.
22
The Rubrics’
Common Scoring Scale: Rubric II
Rubric II – Very Weak Quality of Explanation of the Subject
Matter
0: An object is rated very weak or no value if its subject matter
explanations are confusing or contain errors. There is little
likelihood that this object will contribute to understanding.
For example an object designed to teach synthesis of information gathered from
narrative texts, data collections, and simulations found in various science publications, is
missing a clear explanation of the process and elements of synthesis. It is very weak for
quality of explanation of subject matter.
23
The Rubrics’
Common Scoring Scale: Rubric II
Rubric II – Quality of Explanation of the Subject Matter - Not
Applicable
N/A: This rubric is not applicable (N/A) for an object that is not
designed to explain subject matter. It may be possible to apply
the object in some way that aids a learner’s understanding, but
that is beyond any obvious or described purpose of the object.
For example Rubric II would be not applicable for a sheet of mathematical formulae, a
set of raw data, or a map.
24
Rubric III:
Utility of Materials Designed to Support
Teaching
• Applied to objects designed to support teachers in planning or
presenting subject matter.
• Primary user would be the teacher.
• Evaluates the potential utility of an object for the majority of
instructors at the intended grade level.
• Not applicable if the object is not designed to support
teachers in planning and/or presenting subject matter, even
though it might be possible for an educator to find such an
application.
25
The Rubrics’
Common Scoring Scale: Rubric III
Rubric III – Superior Utility of Materials Designed to Support
Teaching
3: An object is rated superior only if all of the following are true:
• The object provides materials that are comprehensive and easy to understand and
use.
• The object includes suggestions for ways to use the materials with a variety of
learners. These suggestions include materials such as “common error analysis tips”
and “precursor skills and knowledge” that go beyond the basic lesson or unit
elements.
• All objects and all components are provided and function as intended and described.
For larger objects like units, materials facilitate the use of a mix of instructional
approaches (direct instruction, group work, investigations, etc.).
For example a superior object for this rubric includes clear descriptions of the lesson’s activities
with suggestions for interventions, extensions, and for use with English language learners. Also the
time needed for lesson planning and presentation appears to be accurately estimated, materials list
is complete, and all explanations make sense.
26
The Rubrics’
Common Scoring Scale: Rubric III
Rubric III – Strong Utility of Materials Designed to Support
Teaching
2: An object is rated strong if it offers materials that are
comprehensive and easy to understand and use but falls short of
“superior” for either one of two reasons:
• The object does not include suggestions for ways to use the
materials with a variety of learners (e.g., error analysis tips).
• Some core components (e.g., directions) are underdeveloped in
the object.
For example an object would be considered strong if it includes clear lesson directions,
with suggestions for students with special needs, but does not include a list of materials
needed or time estimates. Or the object has clear directions for the teacher, including
student materials, but does not provide suggestions for intervention, extension, or for
English language learners students.
27
The Rubrics’
Common Scoring Scale: Rubric III
Rubric III – Limited Utility of Materials Designed to Support
Teaching
1: An object is rated limited if it includes a useful approach or idea to teach an
important topic but falls short of “strong” for either one of two reasons:
• The object is missing important elements (e.g. directions for some parts of a
lesson are not included).
• Important elements do not function as they are intended to (e.g. directions
are unclear). Teachers would need to supplement this object to use it
effectively.
For example an object, a lesson plan for teaching division of fractions has clear directions and
general suggestions for the types of activities for students but has no specific student materials,
assessments, or suggestions for how to adjust the lesson for students with special needs. It is
considered limited since, in order to use the object, the teacher will need to create questions or
problems, student worksheets and/or assessments and will need to make adjustments for
intervention and extension.
28
The Rubrics’
Common Scoring Scale: Rubric III
Rubric III – Very Weak Utility of Materials Designed to
Support Teaching
0: An object is rated very weak or no value for instructional
purposes if it is confusing, contains errors, is missing many
important elements, or is for some other reason simply not useful.
For example an object, a lesson plan for teaching division of fractions, would be
considered very weak if there are some materials included designed to support teaching
but several student support documents are missing and they include only a few of the
needed solution keys, with errors throughout.
29
The Rubrics’
Common Scoring Scale: Rubric III
Rubric III –Utility of Materials Designed to Support Teaching –
Not Applicable
N/A: This rubric is not applicable (N/A) for an object that is not
designed to support teachers in planning and/or presenting
subject matter. It may be possible that an educator could find an
application for such an object during a lesson, but that would not
be the intended use.
For example a game that is designed to be used by students as an intervention or preassessment of skill knowledge would not necessarily need to have teacher support
materials included. In this case Rubric III would be not applicable to the object.
30
Rubric IV:
Quality of Assessments
• Applied to objects which seek to determine what a student
knows.
• Assessments may be used to assess knowledge before,
during, or after a topic is taught.
• Applied to entire sets of assessment items when multiple
items are included in an object.
• Not applicable for an object not designed to have an
assessment component, even when one might imagine
aspects of the object that may be used for assessment
purposes.
31
The Rubrics’
Common Scoring Scale: Rubric IV
Rubric IV –Superior Quality of Assessments
3: An object is rated superior only if all of the following are true:
• All of the skills and knowledge assessed align clearly to the content and performance
expectations intended, as stated or implicit in the object.
• Nothing is assessed that is not included in the scope of intended material unless it is
differentiated as extension material.
• The most important aspects of the expectations are targeted and are given
appropriate weight/attention in the assessment.
• The assessment modes used in the object, such as selected response, long and
short constructed response, or group work require the student to demonstrate
proficiency in the intended concept/skill.
• The level of difficulty is a result of the complexity of the subject-area content and
performance and of the degree of cognitive demand, rather than a result of unrelated
issues (e.g. overly complex vocabulary used in math word problems).
For example a superior object for Rubric IV has appropriately focused and weighted assessments included, of
varying types and modes, with accurate solution keys provided.
32
The Rubrics’
Common Scoring Scale: Rubric IV
Rubric IV –Strong Quality of Assessments
2: An object is rated strong if it assesses all of the content and
performance expectations intended, but the assessment modes
used do not consistently offer the student opportunities to
demonstrate proficiency in the intended concept/skill.
For example an object with an assessment component intended to determine a
student’s ability to write a proof or present an argument uses only multiple choice or
matching assessment items. This object is considered strong for Rubric IV because it
has clear requirements that students know the structure and parts of the presentation of
a proof or argument, but lacks an assessment mode that offers students an opportunity
to fully demonstrate the intended skill/knowledge: writing a proof or argument.
33
The Rubrics’
Common Scoring Scale: Rubric IV
Rubric IV – Limited Quality of Assessments
1: An object is rated limited if it assesses some of the content or
performance expectations intended (as stated or implicit in the
object), but omits some important content or performance
expectations and/or fails to offer the student opportunities to
demonstrate proficiency in the intended content/skills.
For example an object’s stated intention is to teach students to write fractions, with
denominators of 2, 4, 5, 8, 10, and 100, as decimals, but assessment items address
only measurements with the denominators 2, 4 and 8. The object is considered limited in
the information and performance expectations it assesses.
34
The Rubrics’
Common Scoring Scale: Rubric IV
Rubric IV –Very weak Quality of Assessments
0: An object is rated very weak or no value if assessments
contain significant errors, do not assess important content/skills,
are written in a way that is confusing to students, or are unsound
for other reasons.
For example the following are considered very weak for Rubric IV: 1) an object that
includes general suggestions for the type of assessments to be used (formative,
summative, self-, pre-, etc.) but is missing specific questions or even question types; 2)
one that has attached student assessments that address content not included in the
lesson; or 3) one that includes some assessment items with a lesson, but with incorrect
solutions.
NOTE: This list is not meant to be exhaustive. There are other possible examples of 0rated objects for Rubric IV.
35
The Rubrics’
Common Scoring Scale: Rubric IV
Rubric IV – Quality of Assessments – Not Applicable
N/A: This rubric is not applicable (N/A) for an object that is not
designed to have an assessment component.
Even if one might imagine ways an object could be used for
assessment purposes, if it is not its intended use, not applicable is
the appropriate score.
For example an object is an interactive student-directed game that introduces a new
concept. There is no intention by the designer to include assessments so Rubric IV is
not applicable to the object.
36
Rubric V:
Quality of Technological Interactivity
• The rubric does not apply to students interact with each other,
but rather how the technology responds to an individual student
user.
• For interactive objects designed for student use, the rubric is
applied to the entire object.
• This is not a rating for technology in general, but only for
technological interactivity between the user and the object.
• For objects designed for teacher use, for example lesson plans,
which may include interactive components for use by students,
the rubric is applied only to those interactive parts.
• Does not apply to objects where interaction with the object is
limited to, for example, opening a PDF attachment.
37
The Rubrics’
Common Scoring Scale: Rubric V
Rubric V – Superior Quality of Technological Interactivity
3: An object is rated superior only if all of the following are true:
• The object, or interactive component of an object, is responsive to student
input in a way that creates an individualized learning experience. This means
the object adapts to the user based on what s/he does, or the object allows
the user some flexibility or individual control during the learning experience.
• The interactive component is purposeful and directly related to learning.
• The interactive component of the object is well designed and easy to use,
encouraging learner use.
• The interactive component of the object appears to function flawlessly on the
intended platform.
For example an interactive student-directed game is considered superior for Rubric V because it responds to
individual students by increasing the level of challenge when students successfully complete sections of questions
and offers hints when incorrect answers are given and moves the student to intervention questions when they are
unable to give the correct answer.
38
The Rubrics’
Common Scoring Scale: Rubric V
Rubric V – Strong Quality of Technological Interactivity
2: An object, or interactive component of an object, is rated
strong if it has an interactive feature that is purposeful and
directly related to learning, even if it does not provide an
individualized learning experience. Similarly to the superior
objects, strong interactive objects must be well designed, easy to
use, and function flawlessly on the intended platform.
For example an object has a component that reads aloud sections of text, making it
easier for some students to stay focused or to hear the voice in a passage of literature.
Even though this interactive feature is well designed, purposeful, and directly related to
the learning, there is no personal feedback or individualized response. This object is
considered strong for Rubric V.
39
The Rubrics’
Common Scoring Scale: Rubric V
Rubric V – Limited Quality of Technological Interactivity
1: An object, or interactive component of an object, is rated
limited if its interactive element neither relates to nor detracts
from learning.
This kind of interactive element might be designed to increase
student motivation and to build content understanding by
rewarding or entertaining the learner and extending the time the
user engages with the content.
For example even though the interactive element of a student-directed game may
motivate the student to maintain the connection with the object, it is still considered
limited for Rubric V. The game’s interactive feature includes earning points, leading to
achieving levels, as a reward for correct answers. The interactivity component does not
connect directly to the subject matter, provides no opportunity to learn from incorrect
answers, or, in some cases, even to recognize when an incorrect answer was given.
40
The Rubrics’
Common Scoring Scale: Rubric V
Rubric V – Very Weak Quality of Technological Interactivity
0: An object, or interactive component of an object, is rated very
weak or no value if it has interactive features that are poorly
conceived and/or executed. The interactive features might fail to
operate as intended, distract the user, or unnecessarily take up
user time.
For example in a student-directed game a group of clowns tumble across the screen
with each correct answer, distracting the student and slowing their progress. This
interactive feature is considered very weak or of no value.
41
The Rubrics’
Common Scoring Scale: Rubric V
Rubric V – Quality of Technological Interactivity – Not
Applicable
N/A: This rubric is not applicable (N/A) for an object that does
not have an interactive technological element.
Rubric V is not applicable if there is no interactive feature in the object or if technological
interaction with the object is limited to, for example, opening a user-selected pdf.
42
Rubric VI:
Quality of Instructional Tasks and
Practice Exercises
• Applied to objects that contain instructional tasks and/or sets of
practice exercises designed to strengthen skills and knowledge.
• The purpose of these exercises is to deepen understanding and
to routinize foundational skills and procedures. (When
concepts/skills are introduced providing sufficient practice is
critical. When integrating skills into complex tasks, the number of
exercises is less important than their richness.)
• Applied to a group of exercises as a single object.
• Not applicable if the object does not include practice exercises.
Even if one can imagine ways to use the object as practice, if
that is not the intention of the object, N/A is the correct rating.
43
The Rubrics’
Common Scoring Scale: Rubric VI
Rubric VI – Superior Quality of Instructional Tasks and
Practice Exercises
3: An object is rated superior only if all of the following are true:
• The object offers an more than enough exercises needed for the average
student to facilitate mastery of the targeted skills, as stated or implied in the
object. For complex tasks, one or two rich exercises may be considered
more than enough.
• The exercises are clearly written and supported by accurate answer keys as
applicable.
• There are a variety of exercise types and/or the exercises are available in a
variety of formats, as appropriate to the targeted concepts and skills.
For example a superior object for practice is a lesson addressing operations with polynomials includes an
adequate number of practice problems using all four operations with polynomials. In addition there are extra
practice sets designed for warm up, some for students needing remedial practice, and more complex problems for
those needing a greater challenge. All practice sets provide varying formats for the problems and all include
accurate solutions.
44
The Rubrics’
Common Scoring Scale: Rubric VI
Rubric VI – Strong Quality of Instructional Tasks and Practice
Exercises
2: An object is rated strong if it offers only a sufficient number
(for the average student) of well-written exercises to facilitate
mastery of the targeted skills, supported by accurate answer keys
or scoring guidelines, with little variety of exercise types or
formats.
For example a sixth grade object addressing conventions of standard English,
capitalization and punctuation rules, includes a practice worksheet with an adequate
number of exercises to allow for practice for students presently reading at grade level.
Answer keys and scoring suggestions are included. This object is considered strong
rather than superior because even though it offers adequate practice, it does not provide
varying formats or provide for a variety of student learning needs.
45
The Rubrics’
Common Scoring Scale: Rubric VI
Rubric VI – Limited Quality of Instructional Tasks and
Practice Exercises
1: An object is rated limited if it has some, but too few exercises
to facilitate mastery of the targeted skills, is without answer keys,
and provides no variation in format or type.
For example an object is limited for practice in a lesson addressing two-digit
multiplication which has a set of five practice exercises. That is too few to facilitate
mastery of two-digit multiplication but enough to be considered practice.
46
The Rubrics’
Common Scoring Scale: Rubric VI
Rubric VI – Very Weak Quality of Instructional Tasks and
Practice Exercises
0: An object is rated very weak or no value if exercises provided
do not facilitate mastery of the targeted skills, contain errors, or
are unsound for other reasons.
For example an object addressing division of fractions and mixed numbers has
included a set of practice problems. However of the ten problems, none require
that the student convert a mixed number to an improper fraction. In addition the
answer keys for the practice set includes three with incorrect answers and none
show steps in the process. Since the teacher will likely need to create new
practice exercises, this object is very weak for Rubric VI.
47
The Rubrics’
Common Scoring Scale: Rubric VI
Rubric VI – Quality of Instructional Tasks and Practice
Exercises - Not applicable
N/A: This rubric is not applicable (N/A) to an object that does
not include practice exercises.
• Even though one might imagine ways it could be used in this
way, not applicable is the appropriate score if practice is not the
intention of the object.
For example many objects are not intended to be used for practice of a skill but
rather to introduce a concept, teach a subject, assess learning, etc. When
practice is not the intention of the object, Rubric VI is not applicable.
48
Rubric VII:
Opportunities for Deeper Learning
Rubric VII is applied to objects designed to engage learners
in at least one of the following deeper learning skills:
•
•
•
•
•
•
Think critically and solve complex problems
Work collaboratively
Communicate effectively
Learn how to learn
Reason abstractly
Construct viable, logical arguments and critique the reasoning of
others
• Apply discrete knowledge and skills to real-world situations
• Construct, use, or analyze models
o Not applicable when an object does not appear to be designed to provide
the opportunity for deeper learning, even though one might imagine how it
could be used to do this.
49
The Rubrics’
Common Scoring Scale: Rubric VII
Rubric VII – Superior Opportunities for Deeper Learning
3: An object is rated superior only if all of the following are true:
• At least three of the deeper learning skills from the list are
required in the object.
• The object offers a range of cognitive demand that is
appropriate and supportive of the material.
• Appropriate scaffolding and direction are provided.
For example a superior object for deeper learning addresses peer conferencing in the
process of revision of written drafts. It provides adequate scaffolding and direction for
both the teacher and the student in its requirements that students work collaboratively,
communicate their thinking effectively in sharing their work with peer partners, critique
the reasoning of peer partners, and construct a logical argument supporting their final
draft.
50
The Rubrics’
Common Scoring Scale: Rubric VII
Rubric VII – Strong Opportunities for Deeper Learning
2: An object is rated strong if it includes one or two deeper
learning skills identified in this rubric.
For example an eighth grade object is considered strong in its opportunities for deeper
learning since it involves solving a complex mathematical problem that requires critical
thinking and abstract reasoning skills to reach a solution
51
The Rubrics’
Common Scoring Scale: Rubric VII
Rubric VII – Limited Opportunities for Deeper Learning
1: An object is rated limited if it includes one deeper learning skill
identified in the rubric but is missing clear guidance on how to tap
into the various aspects of deeper learning.
For example as part of a fifth grade writing lesson, an object includes a provision for
learners to collaborate, but the process and product are unclear. Since there is a
suggestion to use collaboration, Rubric VII would apply but without clear explanation of
how the collaboration is to take place, the object is limited in its opportunities for deeper
learning.
52
The Rubrics’
Common Scoring Scale: Rubric VII
Rubric VII – Very Weak Opportunities for Deeper Learning
0: An object is rated very weak if it appears to be designed to
provide some of the deeper learning opportunities identified in this
rubric, but it is not useful as presented.
For example an object describes an activity based on poorly formulated problems and
has unclear directions. It is unlikely that this activity will lead to skills like critical thinking,
abstract reasoning, constructing arguments, or modeling. It would be considered very
weak for Rubric VII.
53
The Rubrics’
Common Scoring Scale: Rubric VII
Rubric VII – Opportunities for Deeper Learning - Not
Applicable
N/A: This rubric is not applicable (N/A) to an object that does
not appear to be designed to provide the opportunity for deeper
learning, even if one might imagine how it could be used to do
this.
For example an object focuses on practicing long division with two-digit divisors through
the use of multiple exercises. Because it does not clearly describe the process to be
used in presenting the exercises, the object does not provide clear opportunities for
deeper learning and Rubric VII would be not applicable.
54
Rubric VIII:
Assurance of Accessibility
Rubric VIII – Assurance of Accessibility
• Accessibility is critically important for all learners and should be considered
in the design of all online materials.
• Only those who are qualified to make judgments about an object’s
accessibility should use this rubric.
• Rubric VIII is used to assure materials are accessible to all students,
including students identified as visually impaired or print disabled.
• It was developed to assess compliance with U.S. standards and
requirements, but could be adapted to accommodate differences in other
sets of requirements internationally.
55
Rubric VIII:
Assurance of Accessibility
Rubric VIII – Assurance of Accessibility
• Identification of certain characteristics will assist in determining if materials
will be fully accessible for all students.
• Assurance that materials are compliant with the standards,
recommendations, and guidelines specified assists educators in the
selection and use of accessible versions of materials that can be used with
all students, including those with different kinds of challenges and assistive
devices.
• Rubric VIII does not ask reviewers to make a judgment on the degree of
object quality. Instead, it requests that a determination (yes/no) of
characteristics be made that, together with assurance of specific Standards,
may determine the degree to which the materials are accessible.
56
The Rubrics’
Scoring Guide: Rubric VIII
Rubric VIII – Assurance of Accessibility – Yes/No Rating
Yes: The object displays the characteristic or complies with the
standards, recommendations or guidelines.
No: The object does NOT display the characteristic or comply
with the standards, recommendations or guidelines.
57
Slides developed by Achieve, Inc.:
www.achieve.org
1400 16th Street, NW / Suite 510
Washington, DC 20036