Tuned Models of Peer
Assessment in MOOCs
Chris Piech
Jonathan Huang
Stanford
Zhenghao Chen
Chuong Do
Andrew Ng
Daphne Koller
Coursera
A variety of assessments
How can we efficiently grade 10,000 students?
2
The Assessment Spectrum
Multiple
choice
Coding
assignments
Short
Response
Proofs
Essay
questions
Long
Response
Easy to automate
Hard to grade
automatically
Limited ability to ask
expressive questions
or require creativity
Can assign complex
assignments and provide
complex feedback
3
Stanford/Coursera’s HCI course
Video lectures + embedded questions, weekly quizzes, open
ended assignments
Student work
Slide credit: Chinmay Kulkarni
Calibrated peer assessment
[Russell, ’05, Kulkarni et al., ‘13]
✓
staff-graded
1) Calibration
2) Assess 5 Peers
3) Self-Assess
Similar process also used in Mathematical Thinking, Programming
Python, Listening to World Music, Fantasy and Science Fiction,
Sociology, Social network analysis ....
Slide credit: Chinmay Kulkarni (http://hci.stanford.edu/research/assess/)
Image credit: Debbie Morrison (http://onlinelearninginsights.wordpress.com/)
Largest peer grading network to-date
HCI #1 HCI #2
3,607 3,633
# Students
# Assignments
5
5
# Submissions 6,702 7,270
# Peer grades 31,067 32,132
77 “ground truth” submissions
graded by everyone (staff included)
HCI 1, Homework #5
How well did peer grading do?
Black stuff  much
room for improvement!
Percent of experiments
0.1
0.09
0.08
0.07
0.06
0.05
within 5pp
0.04
0.03
within 10pp
0.02
0.01
0
-30 -25 -20 -15 -10 -5
0
5 10 15 20 25 30
Error
Up to 20% students get a grade over 10% from ground truth!
~1400 students
Peer Grading Desiderata
Our work:
• Highly reliable/accurate assessment
– Statistical model for estimating and correcting
for grader reliability/bias
• Reduced workload for both students and
course staff
– A simple method for reducing grader workload
• Scalability (to, say, tens of thousands of
students)
– Scalable estimation algorithm that easily
handles MOOC sized courses
How to decide if a grader is good
Who should
we trust?
100%
30%
Graders
100%
50%
55%
56%
54%
Need to reason with all submissions and
Idea: look at the peer grades jointly!
other submissions
graded by these
graders!
Submissions
Statistical Formalization
Student/Grader
Submission
Average grade
inflation/deflation
Bias
True score
Grading variance
Reliability
Observed
score
Observed
score
Observed
score
Observed
Observedscore
score
Model PG1
Modeling grader bias and reliability
True score
of student
u in literature
Related
models
Crowdsourcing
[Whitehill et al. (‘09), Bachrach et al.
Grader reliability
of student v
(‘12), Kamar et al. (‘12) ]
Anthropology
Grader bias
of student
v (‘88)]
[Batchelder
& Romney
Peer Assessment
[Goldin & Ashley (‘11), Goldin (‘12)]
Student v’s assessment of student u (observed)
Correlating bias variables across
assignments
20
Bias on Assn 5
15
10
5
0
Biases estimated from
assignment T with biases
at assignment T+1
-5
-10
-15
-20
-20
-15
-10
-5
0
5
Bias on Assn 4
10
15
20
Model PG2
Temporal coherence
True score of student u
Grader bias at
homework T depends
on bias at T-1
Grader reliability of student v
Grader bias of student v
Student v’s assessment of student u (observed)
Model PG3
Coupled grader score and reliability
True score of student u
Approximate Inference:
Gibbs sampling (also implemented
Your reliability as a
EM, Variational
for a grader depends on
Grader bias
of studentmethods
v
your ability!
subset of the models)
Running time: ~5 minutes for HCI 1
PG3 cannot
be Gibbs sampled
in “closed form”
Student**v’s
assessment
of student
u (observed)
Incentives
Scoring rules can impact student behavior
Model PG3 gives high
scoring graders more
“sway” in computing a
submission’s final score.
Model PG3 gives higher
homework scores to
students who are accurate
graders!
Improves prediction
accuracy
Encourages students to
grade better
See [Dasgupta & Ghosh, ‘13] for
a theoretical look at this problem
Prediction Accuracy
• 33% reduction in RMSE
• Only 3% of submissions land farther than
10% from ground truth
0.08
0.08
0.06
0.06
0.04
0.04
0.02
0.02
0
0
-30
-25
-20
-15
-10
-5
0
5
10
15
20
25
30
0.1
-30
-25
-20
-15
-10
-5
0
5
10
15
20
25
30
0.1
Error Histogram
Error Histogram
Baseline (median)
prediction accuracy
Model PG3
prediction accuracy
Prediction Accuracy, All models
Despite an improved rubric in HCI2, the simplest model
Just
modelingrubric
bias (constant
reliability)
captures
~95%
An improved
made baseline
grading
in HCI2
(PG1
with justperforms
bias) outperforms
baseline grading on all
PGthe
other models
3 typically
of
improvement
in
RMSE
more accurate than HCI1
metrics.
HCI 1
HCI 2
Meaningful Confidence Estimates
When our model is 90% confident that its prediction is within K%
of the true grade, then over 90% of the time in experiment, we
are indeed within K%. (i.e., our model is conservative)
1
0.95
Actual Pass Rate
Pass Rate
We can use confidence estimates to
0.9
Expected Pass Rate
tell when
a
submission needs to be
0.85
0.8seen by more graders!
0.75
0.7
0.65
0.6
Experiments where
confidence fell
between .90-.95
0.55
0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95
Model Confidence
How many graders do you need?
After
2 rounds
15%
Some submissions
need more graders!
After
3 rounds
10%
Need
more
than 5
rounds
46%
Some grader
assignments can
be reallocated!
After 5
rounds
16%
After
4 rounds
13%
Note: This is quite an overconservative estimate (as in the last slide)
Understanding graders in the
context of the MOOC
Question: What factors influence how well a
student will grade?
1.2
1
0.8
0.6
0.4
0.2
0
-0.2
“Harder”
submissions
to grade
Standard
deviation
Mean
-2.75 -1.75 -0.75 0.25 1.25
Gradee grade (z-score)
Residual (z-score)
Residual (z-score)
“Easiest”
submissions
to grade
Better scoring
graders grade
better
1.4
1.2
Standard
deviation
1
0.8
0.6
0.4
Mean
0.2
0
-0.2 -2.75 -1.75 -0.75 0.25 1.25
Grader grade (z-score)
Gradee grade (z-score)
Residual given grader and gradee scores
The worst students tend to
inflate the best submissions
Grade inflation
Best students tend to
downgrade the worst
submissions
Grade deflation
Grader grade (z-score)
# standard deviations
from mean
How much time should you spend
on grading?
Standard deviation of
residual (z-scores)
1
“sweet spot of grading”:
~ 20 minutes
0.9
0.8
0.7
0.6
0.5
0.4
-0.30
-0.25
-0.20
-0.15
Time Grading (z-score)
-0.10
What your peers say about you!
Best
submissions
Worst
submissions
Commenting styles in HCI
sentiment polarity
160
140
sentiment
polarity
0.6
0.5
120
feedback length (words)
0.9
Students have more to say
about
0.8 weaknesses than
strong points
0.7
100
On average, comments
vary
from neutral to positive, with
few highly negative
80 comments
0.4
feedback
length
0.3
60
0.2
0.1
40
-3
-2
-1
0
residual (z-score)
1
2
Student engagement and peer grading
1
True Positive Rate
0.9
all features
just grade
0.8
0.7
0.6
0.5
0.4
0.3
Task: predict whether a
student will complete
last homework
0.2
0.1
0
0
(AUC = 0.97605)
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False Positive Rate
Takeaways
Peer grading is an easy and practical way to
grade open-ended assignments at scale
Real jointly
worldover
deployment:
Reasoning
all submissionsour
and
accounting
for bias/reliability
canHCI
significantly
system
was used in
3!
improve current peer grading in MOOCs
Grading performance can tell us about other
learning factors such as student engagement
or performance
Gradient descent for linear regression
~40,000 submissions