Expertiza: Improving Course Materials
and Learning Outcomes through Peer
Review of Student Work
Edward F. Gehringer
Dept. of Computer Science
North Carolina State University
Supported by NSF DUE under a CCLI grant
NCSU Center for Teaching and Learning
NCSU LITRE (Learning in a Technology-Rich Env.)
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Outline





Introduction
Expertiza Rationale
Demo
Experiment 1: Improving a Textbook
Experiment 2: Review Wiki Submissions
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Introduction


Electronic peer review is students reviewing
other students’ work over the Web.
Building resources through electronic peer
review




gets students working together to improve others’
learning experiences,
helps them learn, by performing tasks that are similar
to real-world responsibilities,
gives them experience in writing their ideas up for an
audience of their peers,
allows each cohort to “stand on the shoulders” of
students in earlier classes.
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Homework, traditionally …

For students to demonstrate mastery of
the subject.



Every student does the same thing—
redundant effort.
Work is graded and thrown away, never
benefiting anyone but the student who did it.
Now the best work can be published, to
help others learn.
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Outline






Introduction
Expertiza Rationale
Demo
Experiment 1: Improving a Textbook
Experiment 2: Review Wiki Submissions
Conclusion
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
The Rationale



Improve student learning
Improve teaching
Better utilize resources
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Improving student learning

The Expertiza platform improves student
learning in these ways:



Integrates active and cooperative learning
 Active learning allows students to take responsibility
for their own learning.
Extends active learning to out-of-classroom activities
and distance education
 DE has been a roadblock to the use of active learning
… students viewing lectures remotely can work only
by themselves.
Discourages plagiarism
 Multiple deadlines and milestones make it impossible
to submit a finished product obtained from an
external source.
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Improving teaching

The Expertiza platform improves teaching in
these ways:


Aug. 9, 2007
Increases the supply of examples/homework
problems/test questions
 Students are assigned to make up such
examples/questions, and these are peer
reviewed.
Focuses students on explaining/understanding the
concepts that are hardest to master
 “Write an example to clarify the hardest concept
in lecture k.”
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Improving resource utilization

Some work is peer-graded, so teaching
assistants can spend more time working with
students and less time grading.



Having inadequate TA support no longer limits the
amount and kinds (e.g., design problems) of
homework that can be assigned.
Students rely more on their peers for help, less on
the course staff.
Makes teaching large classes an advantage!

Large classes can produce more and better resources.
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Outline






Introduction
Expertiza Rationale
Demo
Experiment 1: Improving a Textbook
Experiment 2: Review Wiki Submissions
Conclusion
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Outline






Introduction
Expertiza Rationale
Demo
Experiment 1: Improving a Textbook
Experiment 2: Review Wiki Submissions
Conclusion
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Improving an OOD Text

Our class




Master’s level course
CS & ECE
Substantial DE enrollment
In Fall 2005, we used a new objectoriented design text for the first time,

Dale Skrien’s, An Intrduction to ObjectOriented Design and Design Patterns Using
Java
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Three Homework Assignments



Improve an explanation of a topic covered in
the text.
Create a new example of a concept covered in
the text.
Write a new exercise for a chapter in the text.
All students did not do these exercises in the
same order.
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
The Review Process:
How It Begins


Classroom discussion of the most
difficult topics
Students select a topic from a list.


Several students are allowed to select the
same topic,
But the number of slots is limited.
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Signing up for a First-Round Topic
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Student Choices
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
The Review Rubric

Students review a submission based on the following
questions (for examples).









Aug. 9, 2007
Does the example fully illustrate the concept being
explained?
Is the example easy to understand, i.e., as clear as it
could be and still illustrate the concept?
Does the example model the real world, i.e., could it be
implemented in practice?
Is the example code elegant?
Does the example use up-to-date Java code or UML?
Does the text that accompanies the example explain it
well?
Is the example more useful than the examples in this
section of the book?
Does the example seem to be original?
Other: How would you rate the submission on factors
not reflected in the scores on other questions?
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
After the Initial Review



Resubmission phase. 2–7 days to revise
work in response to reviewer comments.
Grading phase. 3–7 days to make final
comments and assign scores.
Review of review phase. Students review
each other’s reviews.
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Results
I learned a lot from doing
the peer-reviewed
assignments related to the
text.
I enjoyed doing the peerreviewed assignment
related to the text.
20
20
15
15
10

Strongly Disagree
Disagree
Neutral
Agree
Strongly Disagree
Disagree
0
Neutral
0
Agree
5
Strongly Agree
5
Strongly Agree
10
17 student submissions selected for text!
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Outline






Introduction
Expertiza Rationale
Demo
Experiment 1: Improving a Textbook
Experiment 2: Review Wiki Submissions
Conclusion
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Teaching with Wikis




A wiki is, essentially, a Web site that can
be edited by any user.
Homework done on wikis promotes
collaboration between students.
Problem: How to assess so much writing.
Solution: Peer review.
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Our Approach







Students select from a set of topics for a
weekly wiki assignment.
The students are given class time to work on
the topic in groups.
Then they take a couple of days to finish up
their submission.
Their submission is peer-reviewed by other
students.
They have a chance to revise their submission.
Students rate the contributions of their
partners.
Instructor considers the reviews, revisions, and
partner evals in assigning grades.
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Students Seem to Benefit

Wiki seems to be preferred as a submission
mechanism.



“The wiki was really fantastic for creating pages
because it guarantees a uniform style for everything,
putting the major focus on the content created,
rather than formatting issues. It also helped format
content so that it was more understandable.”
Contributions are more extensive than we have
seen when files are submitted and resubmitted.
Peer review is the only scalable solution.

Expertiza gives a framework for review.
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Other Uses



Lecture annotations
Code reviews: Contributions to OSS
projects
Class “proceedings” in a research course


Survey papers or research papers
FAQ in a service-learning course
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Future Work



“Peer-assisted” review
Total Quality Management
Automated reviews of reviews
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
Conclusion

Students can do good work—




work that will help their classmates learn the
material
work that will help the instructor improve the
class
work that allows each class to “stand on the
shoulders” of earlier classes.
The Expertiza approach enables faculty
to introduce these exercises into their
courses.
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu
For more info …


http://research.csc.ncsu.edu/efg/expertiza
Ed Gehringer, efg@ncsu.edu
Aug. 9, 2007
Gehringer: Improving Course Materials … Through Peer Review … efg@ncsu.edu