The Metacognitive Benefits of
Self- and Peer Review
Edward F. Gehringer
Department of Computer Science
North Carolina State University
Our work in peer and self-review
is funded by Google and NSF
http://tinyurl.com/CHEP-SPR
Signup sheet



Name
E-mail address
Institution, if not in your domain name
http://tinyurl.com/CHEP-SPR
Outline
What is metacognition?
 Advantages of peer review?
 Advantages of self-review?
 Self-review
 Peer review
 Combining peer and self-review
 Reviewing vs. being reviewed
 Accuracy of peer grading

Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
What is metacognition?
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
What is metacognition?


“Thinking about one’s thinking”
I am engaging in metacognition if …




I notice I am having more trouble learning A
than B.
it strikes me that I should double-check C
before accepting it as fact.
I realize I don’t understand what the instructor
wants me to do
if I sense that I should write down D so I don’t
forget it
–John Flavell, 1976
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
Advantages of peer review?
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
Some advantages of peer review

Feedback is




Peers may understand
students’ comprehension problems better



more extensive
quicker
scalable
and communicate in language more easily
understood
Can’t blame the reader!
Forces students to think metacognitively
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
Advantages of self-review?
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
Some advantages of self-review



Every student gets
immediate feedback
Judging correctness
of answers deepens
students’ understanding
of material
Students become aware
of own strengths, progress, gaps
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
Self-review: pitfalls

Confounding effects



Early studies asked students to predict how
well they would do in course.
Effort is often included as a criterion.
What is a good
self-assessment?

Is it the same kind of
assessment an instructor
would give?
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
Rubrics

Why use a rubric?




Tell students what to look for
“Fairness” in assessment
Students can help
create the rubric
How detailed?
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
Rubric advice
http://tinyurl.com/CHEP-SPR
Self-review vs. self-grading

Self-grading is more effective on items
requiring lower-order cognitive skills.


Freshmen vs. seniors …
Would students cheat??? …

Wilkowski et al., 2014



70.3% of students in 1st
MOOC awarded selves full
credit
9.9% of these submissions
were blank or nonsense
8.5% were evid. plagiarized
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
Combining self-review & peer review



was the first review system to
incorporate both peer and self-review.
Step 1: Instructor gives students a
writing assignment.
Step 2: Students submit their document
to CPR.
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
Calibration


Step 3: CPR gives the student 3
calibration documents for review.
Basic idea: Training course for reviewers


How they do  how much credence they get
Before students review peers, they get 3
works to review


1 exemplary
Others have known defects
Their agreement with instructor  Reviewer
Competency Index
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
How Does CPR Work?



Step 4: After calibration, CPR gives
student 3 peer documents for review.
Step 5: CPR gives the student his or her
document for self-review.
Step 6: CPR provides a detailed report of
the peer review and the self-review.
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
The Review Process




The review process is the same for
calibration, peer review, and self-review.
The student answers content questions
for each document.
The student answers style questions for
each document.
The student assigns a score to each
document on a scale of 1 to 10.
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
How self-review fits in

Students get full credit for self-reviewing if

their grades agree



within the range set by the instructor
between their self-assessment and
 the weighted value of their peer reviews.
Instructor is notified to check
reviews/self-assessment
when reviewers of a student
did not “train well.”
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
Better to give than to receive?

Would students learn more by …


peer-reviewing, or
being peer-reviewed?
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
What did students appreciate?
Kulkarni et al., 2013, ACM TCHI
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
Other results
Shah-Nelson, 2014, OLC Annual Conference
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
“To give is better than to receive”



“Givers”: reviewed, but received no peer feedback
“Receivers”: got peer feedback, but didn’t review
“Receivers” made more signif. gains in writing,
 especially those at lower proficiency levels.
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
Peer review vs. peer grading



Formative—text feedback
Summative—Likert scale
Should peer review be used summatively?
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
Peer grading—how reliable?

Two studies on Coursera MOOC [2013]



Piech et al.: ≥ 26% of grades ± 5% from
“ground truth.”
Kulkarni et al.: 40% of grades off by 1 letter
grade!
But …
simplistic calibration
 this was, after all, a MOOC

Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
Factors affecting accuracy


Students can only effectively peer-review
what they understand.
This suggests
overweighting
scores assigned by
better-performing
students.
Gehringer, Benefits of self- and peer review
efg@ncsu.edu
http://tinyurl.com/CHEP-SPR
Summary
Metacognition: thinking about thinking
 Peer review: rapid, extensive feedback
 Self-review: deepens understanding
 Self-grading: don’t use by itself!
 Can combine peer & self-review
 Learn more as reviewer than reviewee
 Peer grading: not ready for prime time

Gehringer, Benefits of self- and peer review
efg@ncsu.edu