More Writing But Less Grading:
Calibrated Peer Review™
Share The Future IV - Tempe, AZ
John Wise
The Pennsylvania State University
Penn State’s Engineering
Instructional Services
College of Engineering, University Park PA
 Associate Dean for Undergraduate Studies,
Robert Pangborn
 Support and Training for Faculty and
Teaching Assistants

Engineering Instructional Services
eis@engr.psu.edu
Review of Bloom’s Taxonomy
Knowledge
 Comprehension
 Application
 Analysis
 Synthesis
 Evaluation

Bloom (1956) Taxonomy of educational objectives: the classification of educational goals
Engineering Instructional Services
eis@engr.psu.edu
Problem Statement


Large Classes in Particular:
 Predominantly Lecture
 Multiple Choice / “Scantron” Grading
 Feedback and Writing Difficult
All Classes
 Student “Final” Papers Often First Drafts
 Product vs. Process Orientation
 Concept Misunderstandings
Engineering Instructional Services
eis@engr.psu.edu
History of Calibrated Peer
Review (CPR)
Large Molecular Science Sections @ UCLA
 Science-Based Model
 Research, Write, Peer Review
 Now Can Serve Any Discipline and
Educational Level

Engineering Instructional Services
eis@engr.psu.edu
How It Works
Create (or Select) Assignment
 Set Timing and Standards
 Manage Assignments
 Review Results

Engineering Instructional Services
eis@engr.psu.edu
Create Assignment
Choose Source
 Define Writing Assignment
 Write Exemplar
 Write Moderate / Poor Examples
 Generate Style and Content Questions
 Assess Own Documents for Calibration
(sample)

Engineering Instructional Services
eis@engr.psu.edu
Set Timing and Standards
Control Over Text Entry Times
 Control Over Calibration Standards
 Control Over Scoring Scheme

Engineering Instructional Services
eis@engr.psu.edu
Scoring Template: High Difficulty
Calibrations
To master each calibration students must:
•Answer 3.0 of 4 (75%) STYLE questions correctly, and
•Answer 5.3 of 7 (75%) CONTENT questions correctly, and
•NOT deviate by more than 2.0 points from the RATING of
a calibration text
Reviews
To master reviews students must:
•NOT deviate by more than 2.0 points from the AVERAGE
RATING of the reviewed text
Self-Assessment
To master self-assessment students must:
•NOT deviate by more than 1.0 points from the AVERAGE
RATING of their text to receive full credit, or
•NOT deviate by more than 2.0 points from the AVERAGE
RATING of their text to receive half credit
Directions: Enter the points you want
associated with each assignment stage. All
points must sum to 100. Each point total can
range from 0 to 100.
1.Text quality:
XX points
2.
Calibrations:
XX points over 3
Calibrations
3. Reviews:
XX points over 3
Reviews
4. Selfassessment:
XX points
TOTAL:
must sum to 100
points
Management
Student Reminders
 Special Requests
 Late Submissions
 Technical Difficulties

Engineering Instructional Services
eis@engr.psu.edu
Review Results
Scan Student Results
 Identify “Problem” Scores
 Ability to Randomly Review Some or All
Student Submissions
 Can also participate AS a student!

Engineering Instructional Services
eis@engr.psu.edu
Student
Name
Reviewer
Competency
Index
(RCI)
(1-6)
Complete
Overall
Grade
Text
Score
(Out of 100)
(Out of 10)
[name]
91.81
7.27
6
finished
[name]
94
8
6
finished
[name]
91
7
5
finished
Engineering Instructional Services
eis@engr.psu.edu
Student Point of View
Review Materials
 Write
 Calibrate
 Review three examples, compared to
professor
 Review 3 Random Peers
 Review Self

Engineering Instructional Services
eis@engr.psu.edu
Calibrated Peer Review ™ (CPR)
Available from UCLA
 http://cpr.molsci.ucla.edu
 Contact me for sample assignment
 jwise@engr.psu.edu
 814-865-4020
 Contact UCLA regarding traveling
workshops

Engineering Instructional Services
eis@engr.psu.edu
More Writing But Less Grading:
Calibrated Peer Review™
Share The Future IV - Tempe, AZ
John Wise
The Pennsylvania State University