Rubrics & Roll-Ups
By Stephen Brown
SLO Summit, November 6th, 2015
The Challenge of “Rolling-up”
Rubric Scores
Rubrics used to assess SLOs are designed in variety of
different ways as illustrated in the examples on the
next 4 slides taken from existing SLO assessments.
These differences make it difficult to compare and
average rubric scores in a meaningful way for use in
PLO and ILO “roll-up” assessments…
Rubric A
• “acceptable” = 2 out of 4 or 50%
• rubric score = 2.5 out of 4 or 64%
Rubric B
• “acceptable” = 7 out of 10 or 70%
• rubric score = 8.3 out of 10 or 83%
Rubric C
• “acceptable” = 2 out of 3 or 67%
• rubric score = 81%
Rubric D
• “acceptable” (satisfactory) = 3 out of 5 or 60%
• rubric score = 3.9 out of 5 or 77%
Graphical Representation of Rubrics
Percent Rubric Score
100%
75%
70%
Rubric A
Rubric B
4
10
3
83%
7
64%
50%
2
1
0%
0
3
81%
Rubric D
Rubric X
5
3
4
77%
2
3
2
5
3
25%
Rubric C
1
2
1
31%
1
0
“acceptable” as designated by the rubric
NOTE:
Rubric X is a
hypothetical
rubric
Values from Different Rubric Designs
are not Directly Comparable
“Acceptable” in each of the 5 rubrics is assigned a
different numerical value, thus each rubric score is
meaningful only in relation to the same rubric (or a
rubric of the same design). For example:
• 64% in Rubric A is above “acceptable”
• 64% in Rubric B would be below “acceptable”
In addition, “acceptable” relates to the minimum and
maximum possible scores in each rubric differently.
For example:
• “acceptable” in Rubric A is 50% away from the
maximum score and 50% from the minimum score
• “acceptable” in Rubric X is 67% away from the
maximum score and 33% from the minimum score
A 2-Step Normalization Process
1.
Adjust rubric scores so that a common reference point
within each rubric – “acceptable” – is numerically the
same for all rubrics.
2.
Adjust these modified scores in relation to:
•
the maximum possible score (if above “acceptable”)
OR
•
the minimum possible score (if below “acceptable”)
STEP 1 – Adjusting “Acceptable”
Divide the percent rubric score (PRS) by the percent score
for “acceptable” (PRSacc) and subtract 1.00:
• for Rubric A: 64%/50% - 1.00 = 0.28
• for Rubric X: 31%/33% - 1.00 = -0.06
We will call this the adjusted rubric score (ARS) which
indicates its relation to “acceptable” now defined as 0.00.
NOTE: Student performance below “acceptable”
is indicated by a negative value.
Table of ARS Values
Rubric A Rubric B Rubric C Rubric D Rubric X
“Acceptable”
0.00
0.00
0.00
0.00
0.00
ARS
0.28
0.19
0.21
0.28
-0.06
Maximum ARS
(ARSmax)
1.00
0.43
0.50
0.67
2.00
Minimum ARS
(ARSmin)
-1.00
-0.57
-0.50
-0.67
-1.00
Notice that ARSmin and ARSmax still differ among the rubrics…
Adjusted Rubric Score (ARS)
Graph of Adjusted Rubric Scores
2.00
Rubric A
Rubric B
Rubric C
Rubric D
Rubric X
MAX
1.50
1.00
MAX
MAX
0.50
0.28
0.00
MAX
0.19
0.21
0.28
-0.06
-0.50
-1.00
MAX
MIN
MIN
MIN
MIN
MIN
“acceptable” as designated by the rubric
Although “acceptable” is the same value in each case,
the rubric scores are still not normalized:
• for example, Rubric A and Rubric D both have the
same ARS value of 0.28, however they are not
equivalent since 0.28 is much closer to ARSmax in
Rubric D than Rubric A.
Discrepancies such as this will be corrected by carrying
out the second step…
STEP 2 – Adjusting to ARSmax or ARSmin
For positive ARS values (>0.00), divide ARS/ARSmax:
For Rubric A this would be: 0.28/1.00 = 0.28
For Rubric D this would be: 0.28/0.67 = 0.42
For negative ARS values (<0.00), divide ARS/[ARSmin]:
For Rubric X this would be: -0.06/1.00 = -0.06
NOTE: [ARSmin] = the absolute value of ARSmin
Normalizing with a Single Formula
By substituting ARS formulas for the ARS values, we have a
single NRS calculation involving percent rubric scores (PRS)
only. For rubric scores above “acceptable” we would use:
((PRS/PRSacc) – 1)/((100%/PRSacc) – 1) = NRS
NOTE: The maximum possible percent score for all rubrics (PRSmax) is 100%.
For rubric scores below “acceptable” we would use:
((PRS/PRSacc) – 1)/[((PRSmin/PRSacc) – 1)] = NRS
Table of NRS Values
Rubric A Rubric B Rubric C Rubric D Rubric X
Maximum NRS
(NRSmax)
0.00
0.28
1.00
0.00
0.44
1.00
0.00
0.42
1.00
0.00
0.42
1.00
0.00
-0.06
NA
Minimum NRS
(NRSmin)
NA
NA
NA
NA
-1.00
“Acceptable”
NRS
Let’s also look at these numbers on a graph which will show the
rubric scores to be fully normalized…
Normalized Rubric Score (NRS)
Fully Normalized Rubric Scores
1.00
Rubric A
Rubric B
Rubric C
Rubric D
MAX
MAX
MAX
MAX
0.44
0.42
0.42
0.50
Rubric “X”
0.28
0.00
-0.06
-0.50
-1.00
MIN
“acceptable” as designated by the rubric
In Summary
Normalizing rubric scores in this way converts raw
percent rubric scores to normalized (NRS) values that
are directly comparable and averagable.
• the average of NRS values from all 5 rubrics used in
this presentation is 0.30 or 30% above “acceptable”
NOTE: Averaging the raw, un-normalized scores in relation
to an arbitrary 70% benchmark would result in an
average of 67%, i.e., 3% below this benchmark.