Building Resources for Teaching
Computer Architecture Through
Peer Review
Edward F. Gehringer
Dept. of Electrical & Computer Engineering
Dept. of Computer Science
North Carolina State University
efg@ncsu.edu
Peer Review

Long used for technical papers.
 First used in courses in early 1970s.
 Hundreds of studies.
 Much more practical with networked
computing.
 Widely used in writing courses (e.g.,
Daedalus Integrated Writing Environment)
Peer Review on the Web

Can accept submissions in almost any
format.
 Web is a familiar interface.
 Web-creation skills are important to
students.
 Web interface allows use in DE.
 Facilitates the production of Web resources.
Overview
 The
implementation
 The review cycle
 How peer grading has been used
 Student reaction
The Implementation: PG






Students develop homework in the form of one or
more Web pages.
Students submit their pages to the system.
The system copies them to a new Web address,
concealing the submitter’s identity.
Reviewers are assigned semiautomatically.
Reviewers and authors communicate via a shared
Web page.
Reviewers assign grades, and the system averages
them.
The PG Login Page
Choosing an Assignment
Selecting Submission to Review
Submitting and Reviewing

A student logging in has a choice of whether to
submit or review.
 If submitting—
– can use a browser to select a file.
– can submit one file at a time, or
– a whole network of pages in a single Zip file.

If reviewing—
– is presented with a set of pages to review.
– can click on one and type in comments.
Reviewing
a Submission

Review is based on
a rubric.
– A numeric score is
assigned to several
questions.
– Grade is calculated
from these scores.
– Ample opportunity
for comments.
Overview
 The
implementation
 The review cycle
 How peer grading has been used
 Student reaction
Submit-Review-Publish Cycle

Signup phase. A limited number of students allowed to
sign up for each choice.
 Initial feedback phase. Students given 2–7 days to make
initial comments.
 Resubmission phase. 2–7 days to revise work in response
to reviewer comments.
 Grading phase. 3–7 days to make final comments and
assign scores.
 Review of review phase. Students review each other’s
reviews.
 Web-publishing phase. PG creates a Web page with the
“best” assignment in each category.
Overview
 The
implementation
 The review cycle
 How peer review has been used
 Student reaction
How Peer Review Has Been Used

Researching lecture material.
– Find links related to each lecture.

Annotating on-line lecture notes.
 Writing research papers.
 Reviewing papers from the literature.
 Making up homework problems.
 Making up machine-scorable questions.
 Weekly reviews.
Annotating
a Lecture

Students
electronically sign
up to review a
particular lecture,
 then add hyper-links
to instructor’s online notes.
Research
Papers

Can include
hyperlinks to Web
documents.
 Different students
can sign up to write
on different topics.
Madeup Problems
Madeup Problems, cont.
Choosing Assignment Types
Producing Resources
A “divide and conquer” approach to large
projects.
 Lecture annotations for an entire semester.
 Machine-scorable questions for each lecture.
 Madeup problems for the Computer
Architecture Course Database (cf. WCAE
1998, WCAE 2000).

Overview
 The
implementation
 The review cycle
 How peer grading has been used
 Student reaction
Student Reaction

Students have reacted quite positively to
peer review.
– When asked to rate on a scale of 1 to 5, “Peer
review is helpful to the learning process,”
– five classes rated it 3.41 to 4.24, with the
highest score given by one of the classes that
did the most peer-reviewed assignments.
 By a score of 3.9, students said that reviews of
reviews motivated them to do careful reviews.
Conclusion

Electronic peer review is an effective
teaching tool.
 Students do a careful job on their
assignments, and in their review of each
other.
 When students do different work, it can be
aggregated to produce impressive resources
for future classes.