Morphing Rubrics to Adapt to
Developmental Student Needs
Aja Henriquez, MFA, EdD Student CSUSB
English Instructor:Crafton Hills College, California Baptist University
What is a Rubric?

Definition of

rubric (ˈruːbrɪk)
1. a title, heading, or initial letter in a book,
manuscript, or section of a legal code, esp
one printed or painted in red ink or in
some similarly distinguishing manner
2. a set of rules of conduct or procedure
3. a set of directions for the conduct of
Christian church services, often printed in
red in a prayer book or missal
4. instructions to a candidate at the head of
the examination paper
5. an obsolete name for red ochre
6. written, printed, or marked in red
From Collins English Dictionary
Why Rubrics?
School Specific Issues


Approximately 31% of incoming students
transfer into the developmental level (English
015) at CHC, while 36% transfer into a
lower, remedial level course (914): Around
33% of students test into college level
English courses (Research Briefs, 2011).
Approximately 52% of students who pass
the “Preparation for College Writing” course
(English 015) persist in the college and
successfully complete a college level writing
course (Student Equity Data, 2011).
What We Can Control




The success rates tell us that there is
something not transferring in our courses.
Many things we cannot control: student
readiness/commitment, family problems,
financial problems, etc.
We can address our rubrics and how we
communicate with the students.
As we know from research, the students
may simply misunderstand what we say/write
when we grade.
What is the Purpose of a Rubric?

We use it to grade for a few reasons:
◦ To assess specific items in a given assignment
◦ To help ensure or increase uniformity of
assessment
It is a tool for instructors to
communicate with students.
 Students must be able to understand the
communication.

Language Disconnect
We may be taking for granted student understanding of our
rubrics.
 Developmental courses often act as an introduction into the
academic discourse community (Bizzell, 1982):
 “students from different social classes come to
school with different abilities to deal with academic
discourse: middle-class students are better suited by
their socialization in language use to deal with
academic discourse's relative formality and
abstraction than working-class students are.This
unequal removal from academic language is, of
course, exacerbated for students whose home
language does not resemble the so called standard
English” (192).

Student Perspective

VS

“It’s like they [the
affluent students] had
their own classes, and
we [the students from
the poor/working
class] had our own
classes. We were, like,
segregated in the same
school. “
--Prep for College
Writing Student
Demographics of Basic Skills
Students

“Crafton Hills College, then, serves a
community where the constituents must
commute to work or school, where the
poverty and unemployment rates exceed
the state average, and where the income
is below the national average.”
From Demographics, CHC webpage:
http://www.craftonhills.edu/About_CHC/D
emographics.aspx
We Should Keep in Mind:
Students may not understand the
language we use on rubrics
 We must ask ourselves if we have written
the rubric for our understanding or
student understanding.

Example: Unexamined Rubric
Excellent
Controlling Idea:
Organization:
Development:
Audience:
Comments:
Good
Adequate
Poor
Needs Work
A Tool That Doesn’t Work
Does a developmental student know
what these terms mean when it comes to
writing?
 If the students don’t understand what the
rubric means, then they can’t use it to
improve.
 How can we make this more
understandable?

Example: Revised Rubric
Other Concerns
Even with some clearer criteria, there are
still some issues with this rubric, which
are less simplistic than just the language
used.
 Validity-does this measure what we want
it to measure?
 Reliability-are the scores consistent?

Messick (1995)
Validity
Content Validity: to ensure we are
measuring what students should know
and not some other thing (personal
preferences), we must match each item
on the rubric to a course objective.
 Substantive Validity: we should make sure
that the items on the rubric connect to
different types of cognitive processes and
are of varying difficulty

Messick (1995)
How to Ensure Content Validity
Review the items on your rubric
and make sure they match to the
course objectives along with any
departmental norms.
 English departments often have
norming sessions for grading,
which these instructors must
take into consideration when
evaluating the content of their
rubrics.
 EXAMPLE partial rationalization
of content validity

 Some things that make it into
our grading that should not be
there: stapling, niceness of
binder or folder, whether it was
accompanied with a latte, etc.

Item one corresponds with
objectives 4, 5, 6, 11, as well as
the grading criteria for writing
(demonstrates skillful use of
vocabulary and syntax; is
generally free from errors in
mechanics, usage, and sentence
structure).
 Item two corresponds with
objectives 7, 8, 9, as well as the
grading criteria for writing
(focuses clearly on the topic and
responds effectively to all aspects
of the assignment; explores the
issues thoughtfully and in depth;
is coherently and logically
organized with a thesis statement
supported by apt reasons and
specific, detailed examples).

Substantive Validity
For explanation of Bloom's Taxonomy,
watch http://youtu.be/qjhKmhKjzsQ
Ensuring Substantive Validity
First of all, we should examine the items
in our rubrics to ensure there is a spread
of difficulty.
 By doing this, we are able to measure the
ability of students who are at different
levels ability (low, mid, and high).

PERSONS - MAP - ITEMS
<more>|<rare>
2
k +
|
T|
|
|
|
m |
|
|
|
|
j |
|
|
S|
1
+
n |
|
b |
f |
|
i |
|
|
e |
h |
M|T Q3
o p |
|S
|
0 c g +M Q2
Q4
| Q1
|S
| Q5
|T
|
|
a |
S|
|
|
|
|
|
|
-1
+
|
|
l |
|
T|
|
d |
|
|
|
|
|
|
|
-2
+
<less>|<frequ>
Substantive Validity



Items do not match
the student ability well.
It is difficult to
establish which items
should be more
difficult, since each
individual will have
personal difficulties.
There should be a
greater spread of
difficulty.
Rasch Model
The previous slide showed an output
table from Winsteps, which is based on
the Rasch mathematical model.
 We don’t all have this program or the
ability to use it.
 There is a simpler way to examine our
student scores using Excel, which most of
us already have.

Steps to Examining Scores
Open an Excel page
 Input the student scores for each item
and their total score.
 You do not need to input student names.
Simply using letters will work.
 Using Excel, array the scores from highest
to lowest.
 Examine the scores.

Step by step instructions linked on
http://www.youtube.com/watch?v=nDez8V
mlN9A
Classical Analysis
Student #
Points
Possible
k
m
j
n
b
f
i
e
h
o
p
c
g
a
l
d
item
variance
Pt. B Cor.
Writing
Conventions
Integration
of
Research
Format
Content/
Development
Organization
12
12
12
10
12
9
6
9
10
9
9
9
12
12
9
6
9
155
16
16
16
16
12
16
14
12
12
16
12
12
4
8
4
0
0
170
12
12
12
12
12
10
10
9
9
9
9
9
3
6
3
4
3
132
48
48
46
46
48
44
48
46
44
40
40
40
44
40
40
40
36
690
12
12
12
12
9
12
12
12
10
9
9
9
12
9
9
6
6
160
3.59
0.41
30.36
0.93
10.69
0.92
12.98
0.84
4.13
0.82
Total
See Traub (1991) for reliability discussion
100
100
98
96
93
91
90
88
85
83
79
79
75
75
65
56
54
Fix Through Metamorphosis
Item one does not have consistent scores.
 In order to fix this, I would expand this
item, Writing Conventions, into its
component parts.
 Adding more items will help pinpoint the
exact issues with writing conventions that
give students trouble, which will in turn
help me differentiate instruction based on
student need.

Morphed Rubric, Item One
Performance
Levels
Dimensions
Exemplary (4)
Accomplished (3)
Developing (2)
Beginning (1)
Score
Spelling
There are no spelling
errors.
Spelling errors are
minimal (1/page)
Spelling errors are a
problem (2-3/page)
Spelling errors are a
problem (4+/page)
/12
Punctuation
There are no
punctuation or
capitalization errors
Punctuation errors
are minimal (1 per
page)
Punctuation errors are a
problem (2-3 per page)
Punctuation errors are
excessive (4+ per page)
/12
Fragments
There are no fragments
in the essay.
Fragments are
minimal (1 total)
Fragments are a problem
(2 total)
Fragments are excessive
(3 or more total)
/12
Run-ons
There are no run-ons in
the essay.
Run-ons are
minimal (1 total)
Run-ons are a problem (2
total)
Run-ons are excessive (3
or more total)
/12
Interpreting the Morph
Because item one was so problematic, I
can interpret that it is not very clear what
the students can do to improve, even with
explicit in-text notes. (As we may know)
 I morphed it by expanding so that both
the students and I could understand what
wasn’t working in their assignments.

◦ They can work on spelling instead of run-ons
◦ I can work on student weaknesses with
instruction
Hidden Issues with Rubrics
The Rasch analysis with Winsteps suggests
that most of the grade scale is not necessary.
 Grading on a 100 point scale suggest the
ability to differentiate an essay in 100 ways
(Dr. Jesunathadas).
 As the Revised Rubric suggests, I would use
a 1-4 point scale for each item and then
report the final score as a percentage for
student understanding.
 The changes to the grading scale will help
keep the scoring effective (no more half
scores that don’t differentiate well).

Item 2
ITEM
NUMBER:
2
Integration
CATEGORY
P
R
1.0
PROBABILITIES:
|
|
A
at
intersections
6666
|
.8
0
0
|
0
|
0
.6
Y
.5
O
.4
4
|
0
4
+
*
|
40
+
|
|
4
4
4
+
P
|
O
|
N
|
.0
4
4
|
.2
4
0
E
|
6
+
R
|
6
|
I
|
66
L
E
measures
00
I
S
Structure
000
0
S
-
6666666666+
+
F
MODES
+0000000000000
B
T
Research
-+---------+---------+---------+---------+---------+---------+-
O
B
of
2
2
2
2
*
*
0
*
26
|
62
+
2
|
2
|
2
6
*
8*2
882200
|
+
6
*8*
0
444
6
6
4
+
6
6
24
2
|
6
4
0
|
6
2
4
0
4
44
22
2
42
0
|
6
4
0
4
6
4
4
4
+
6
4**44
|
2
444
+
2
|
444*
**4488
|
***444
|
+*************************************************************+
-+---------+---------+---------+---------+---------+---------+-3
PERSON
-2
[MINUS]
-1
ITEM
MEASURE
0
1
2
3
Revised Rubric
Assignment Rubric: REVISED ESSAY RUBRIC
Performance
Levels
Dimensions
Exemplary (4)
Accomplished (3)
Developing (2)
Beginning (1)
Score
Spelling
There are no spelling
errors.
Spelling errors are
minimal (1/page)
Spelling errors are a
problem (2-3/page)
Spelling errors are a
problem (4+/page)
/4
Punctuation
There are no punctuation
or capitalization errors
Punctuation errors
are minimal (1 per
page)
Punctuation errors are a
problem (2-3 per page)
Punctuation errors are
excessive (4+ per page)
/4
Fragments
There are no fragments
in the essay.
Fragments are
minimal (1 total)
Fragments are a problem (2
total)
Fragments are excessive
(3 or more total)
/4
Run-ons
There are no run-ons in
the essay.
Run-ons are
minimal (1 total)
Run-ons are a problem (2
total)
Run-ons are excessive (3
or more total)
/4
Integration of
research
Research is integrated to
illustrate and support
main points with specific
details. At least 3
sources used.
Research is
referenced, but
specific details are
sparse. Only 2
sources used.
Research is referenced, but
specific details are not
used. 1 source used.
Research is not discussed
or referenced, and print
out not attached. No
sources used
/4
Proper MLA
format
Proper margins, font,
spacing, header, etc. are
used
Proper formatting is
followed except for
1-2 items
3-4 formatting mistakes
5+ formatting errors
present
/4
Content and
Development
The essay adequately
contends with the subject
matter and offers a clear
and concise discussion
the prompt. Clear thesis
is present.
Essay flows and
transitions well; Logical
paragraphing;
Introduction and
conclusion present
The essay deals
with the subject
matter, but fails to
address entire
prompt. Thesis is
present.
Essay flows
generally well, with
only a few choppy
areas
The essay does not
adequately address the
subject matter or the
prompt. Thesis is weak or
not present
The essay shows a lack of
understanding of the
subject matter or depth of
thought. Attempt at a
thesis is completely
absent.
Essay construction
impedes the purpose and
meaning of the essay
/4
Organization
Ideas in the essay do not
seem to be linked very well.
Essay is difficult to
understand.
Total
/4
/32
*100=
______%
Morphing Your Rubric(s)
Be sure your items are aligned with
objectives and department norms
 Review your rubric for clear language and
explicit (understandable) description of
proficiency levels
 Take one set of assignments and array
them on Excel to find inconsistent items
 Any inconsistent items should be
expanded so you can pinpoint what is not
working for students

Morph Your Instruction
If you notice there is something on the
rubric that students are consistently
unsuccessful at, then you can spend extra
time on that in class instead of on items
they have already mastered.
 The morphed rubric will help you
measure whether your instruction is
helping or if you need to choose another
tactic.

Is it Working?




Student scores on each item should be
consistent with their total scores.
If not, keep morphing the rubric/instruction.
Look for growth in scores over the
semester, if you are using the same rubric
each time. (This also helps students
understand consistent expectations)
Keep in mind student lives will occasionally
impact their scores over the semester, so
don’t freak out if all students don’t show
improvement all the time.
Final Thoughts

Questions?

This presentation is linked at
ajahenriquez.wordpress.com
A screencast of how to manipulate data in excel is
available at
◦ http://www.youtube.com/watch?v=nDez8VmlN9A

References
Bizzell, P. (1982). Review: College Composition:
Initiation into the Academic Discourse Community.
Curriculum Inquiry, 12(2), 191-207.
 Messick, S. (1995). Validity of psychological assessment:
Validation of inferences from persons' responses and
performances as scientific inquiry into score meaning.
American Psychologist. 50(9), 741-749.
 rubric. (n.d.). Collins English Dictionary - Complete &
Unabridged 10th Edition. Retrieved October 20, 2011,
from Dictionary.com website:
http://dictionary.reference.com/browse/rubric
 Traub, R. E.; Rowley, G. L., (1991). Understanding
reliability. Educational Measurement. 10, 37-45.

Related Titles
Bond, T. and Fox, C. (2007). Applying the Rasch
Model: Fundamental Measurement in the
Human Sciences. Lawrence Erlbaum.
 Koretz, D. (2008). Measuring up: what
education testing really tells us. Cambridge, MA:
Harvard University Press.
 Ryan, K., & Shepard, L. A. (2008). The future
of test-based educational accountability. New
York: Routledge.
 Spaulding, D. (2008). Program Evaluation in
Practice: Core Concepts and Examples for
Discussion and Analysis. San Francisco: Wiley.
