Expertiza: Peer Review and Social
Networking for Co-Producing
Learning Objects
Edward F. Gehringer
Dept. of Computer Science
North Carolina State University
Supported by NSF DUE under a CCLI grant
NCSU Faculty Center for Teaching and Learning
NCSU LITRE (Learning in a Technology-Rich Env.)
Center for Advanced Computing and Communication
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Learning Goals

After this talk, you will …
Have a glimpse of the wide variety of
ways in which peer feedback can be
used,
 Know of various software tools for
gathering peer feedback, and
 Understand how students can assist in
producing course materials using peer
review.

Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Outline





Expertiza Rationale
Co-Production Experiments and Results
Brief Expertiza Demo
Handling Team Assignments
New Features Coming to Expertiza
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Introduction



There are many Web-based systems for peer
review of student writing.
Expertiza takes this one step further—enabling
co-production of the course!
Advantages …




Gets students working together to improve others’
learning experiences,
Helps them learn, by performing tasks that are
similar to real-world responsibilities,
Gives them experience in writing their ideas up for an
audience of their peers,
Allows each cohort to “stand on the shoulders” of
students in earlier classes.
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Homework, traditionally …

For students to demonstrate mastery of
the subject.




Every student does the same thing—
redundant effort.
Work is graded and thrown away, never
benefiting anyone but the student who did it.
In Expertiza, students (or teams) sign up
for different work
Now the best work can be reused, to
help others learn.
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Major advantages

Expertiza discourages plagiarism



Expertiza better utilizes scarce resources



Students are not doing the same work.
Their work must go through multiple review
cycles
Students do most of the reviewing.
Instructor/TA spend less time grading.
With Expertiza, large classes are an
advantage!
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Outline





Expertiza Rationale
Co-Production Experiments and Results
Brief Expertiza Demo
Handling Team Assignments
New Features Coming to Expertiza
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Co-Production Experiments and Results



Exercises for a Textbook
Active-Learning Exercises for a Course
Assessment of Wiki Contributions
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Results: Material for a Textbook

Our class




Master’s level course
CS & ECE
Substantial DE enrollment
In Fall 2005, we used a new
OOD text for the first time,

Dale Skrien’s, An Introduction
to Object-Oriented Design
Using Java
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Three Homework Assignments



Improve an explanation of a topic covered in
the text.
Create a new example of a concept covered in
the text.
Write a new exercise for a chapter in the text.
All students did not do these exercises in the
same order.
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Learning and Enjoyment
I learned a lot from doing
the peer-reviewed
assignments related to the
text.
I enjoyed doing the peerreviewed assignment
related to the text.
20
20
15
15
10

Strongly Disagree
Disagree
Neutral
Strongly Agree
Strongly Disagree
Disagree
0
Neutral
0
Agree
5
Strongly Agree
5
Agree
10
17 student submissions selected for text!
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Results: Active-Learning Exercises

Last spring, I had my CSC 216 students
create active-learning exercises.




They worked in teams.
They created two exercises during the
semester.
One of the exercises was used later in
the class
In the summer, 4 student-generated AL
exercises were used in class.
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
“I learned a lot from doing active-learning exercises in class.”
7
6
5
4
3
2
1
0
SA
Apr. 22, 2009
Ed Gehringer
Agree
efg@ncsu.edu
Neutral
Disagree
http://research.csc.ncsu.edu/efg/expertiza
SD
“I enjoyed doing the AL exercises”
8
7
6
5
4
3
2
1
0
SA
Apr. 22, 2009
Ed Gehringer
Agree
efg@ncsu.edu
Neutral
Disagree
http://research.csc.ncsu.edu/efg/expertiza
SD
“I enjoyed devising an AL exercise”
10
9
8
7
6
5
4
3
2
1
0
SA
Apr. 22, 2009
Ed Gehringer
Agree
efg@ncsu.edu
Neutral
Disagree
http://research.csc.ncsu.edu/efg/expertiza
SD
Comparing Instructor- & StudentGenerated Exercises
Spring
Learning
Enjoyment
Learning
Enjoyment
3.1
3.2
2.8
3.0
3.5
2.9
3.5
3.8
All
Stud.-gen.
No significant difference
Apr. 22, 2009
Summer
Ed Gehringer
efg@ncsu.edu
Significant at
99% level
Significant at
90% level
http://research.csc.ncsu.edu/efg/expertiza
Results: Wiki assessment

We studied ten NCSU classes that used wikis in
2007-08.











CSC 216, Programming Concepts—Java
CSC/ECE 506, Architecture of Parallel Computers
CSC/ECE 517, Object-Oriented Languages & Systems
COM 598W, Gaming and Social Networks
ECI 306, Middle Years Reading
ECI 521, Teaching Literature for Young Adults
ECI 525, Contemp. Approaches in Teaching Social Studies
HI 216, Latin America since 1826.
HI 453/553, U.S. and Latin America.
40 respondents
62 respondents
TOX 415, Environmental Toxicology & Chemistry
Three classes used peer assessment; seven classes
used “expert” assessment.
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Peer Review vs. Other Assessment

Classes that used peer assessment are shown in
gold.











CSC 216, Programming Concepts—Java
CSC/ECE 506, Architecture of Parallel Computers
CSC/ECE 517, Object-Oriented Languages & Systems
COM 598W, Gaming and Social Networks
ECI 306, Middle Years Reading
ECI 521, Teaching Literature for Young Adults
ECI 525, Contemp. Approaches in Teaching Social Studies
HI 216, Latin America since 1826.
HI 453/553, U.S. and Latin America.
TOX 415, Environmental Toxicology & Chemistry
We compared students’ reactions to wiki assignments
and assessment.
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Comparing Peer Review with Expert Review

“The experience of using a wiki helped me to think
critically about the subject matter of this
sassignment.”
95% NPR

2.41
PR
2.00
2.60
PR
2.68
“The feedback I received helped me to improve my
work.”
NPR

2.12
“I received adequate feedback on the quality of my
work.”
NPR

PR
“Using a wiki made it easy to collaborate with other
students.”
95% NPR

2.51
2.44
PR
2.33
All questions rated on a scale of 1=strongly agree to
5=strongly disagree.
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Outline





Expertiza Rationale
Teaching with Wikis
Brief Expertiza Demo
Handling Team Assignments
New Features Coming to Expertiza
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
The Review Process:
How It Begins


One or more times during the semester, a
peer-reviewed project is assigned
Students select a topic from a list.



Several students are allowed to select the same topic,
But the number of slots is limited.
Students are usually allowed to work in teams
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Step 1: Signing up for a Topic
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
After the Initial Review



Resubmission phase. 2–7 days to revise
work in response to reviewer comments.
Final review. 3–7 days to make final
comments and assign scores.
Metareview phase. Students review each
other’s reviews.
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Alternate Tool: Calibrated Peer Review
cpr.molsci.ucla.edu
● For review of writing
● Well developed assgts. for many fields
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Outline





Expertiza Rationale
Co-Production Experiments and Results
Brief Expertiza Demo
Handling Team Assignments
New Features Coming to Expertiza
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Why team assignments?



Teamwork mimics the work environment
Teaming skills are important to success
Teams can tackle larger projects,

which diminishes the effort of integration
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
How does Expertiza support teams?

All team members have access to a
common submission area.


Individuals review teams.



Any member can revise the submission.
This simplifies reviewing, and
provides for more reviews.
Teammates assess each other’s
contribution to the team.

This score can be factored into the student’s
grade.
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
"Virtual" team demo

Team formation






By instructor, from the user interface
By instructor, by importing a file
By students, by issuing invitations
Reviewing
Feedback from author to reviewer
Teammate review
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
The login screen
Let’s log in as an instructor …
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Setting up a team assignment
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Let’s create some teams …
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
In the beginning, there are no teams
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
But let’s create one …
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Team created; let’s add members
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
1st member is added; now for the 2nd …
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Team now has two members
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
"Virtual" team demo

Team formation






By instructor, from the user interface
By instructor, by importing a file
By students, by issuing invitations
Reviewing
Feedback from author to reviewer
Teammate review
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Importing teams
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Here’s the file that’ll be imported
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Here’s how we import it
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Now there are five teams …
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
"Virtual" team demo

Team formation






By instructor, from the user interface
By instructor, by importing a file
By students, by issuing invitations
Reviewing
Feedback from author to reviewer
Teammate review
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Students can create teams too …
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Click on this assignment, Team test 2
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Now, click on “Your team”
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
This user needs to set up a team
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Let’s create a team
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Invite another member to join …
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Invitation has been sent
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Now the invited user logs in
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
… and goes to Team test 2
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Then accepts the invitation
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
… and is placed on the team
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Advantages of various approaches

Instructor-assigned teams



using class rank
using demographics
Student-selected teams
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Alternate Tool: Team-Maker
www.catme.org
Create teams based on …
– demographic characteristics
– student schedules
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
"Virtual" team demo

Team formation






By instructor, from the user interface
By instructor, by importing a file
By students, by issuing invitations
Reviewing―same as above!
Feedback from author to reviewer
Teammate review
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
"Virtual" team demo

Team formation






By instructor, from the user interface
By instructor, by importing a file
By students, by issuing invitations
Reviewing
Feedback from author to reviewer
Teammate review
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Now the reviewee logs in …
… and views the scores
Click on “View” to expand review
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Notice the link down at the bottom
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Use it to give feedback to the reviewer
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Then the reviewer can view the feedback
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
"Virtual" team demo

Team formation






By instructor, from the user interface
By instructor, by importing a file
By students, by issuing invitations
Reviewing
Feedback from author to reviewer
Teammate review
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Review of teammates
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Answer these questions …
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Alternate Tool: CATME
www.catme.org
● Many rubric questions for evaluating
team members
● Works with Team-Maker
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Summary of feedback in Expertiza






Reviews
Feedback to author
Teammate reviews
Metareviews (x reviews y’s review of z)
Surveys
All but surveys can influence the final
grade
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Instructor report
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Additional Types of Feedback: TIDEE

Teamwork assessments





www.tidee.wsu.edu
Team contract assessment
Team-member citizenship assessment
Team processes assessment
Teamwork achieved " "
Design Processes
Assessment



Problem-scoping assmt.
Solution realization " "
Design reflection assmt.
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Outline






Expertiza Rationale
Co-Production Experiments and Results
Expertiza Demo
The Review Process
Results and Conclusion
New Features Coming to Expertiza
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
New Features Coming to Expertiza

Game mechanics …





Message boards


Achievements system
Reputation system
Leaderboards
Levels
Students can ask for help; other students
can answer
Microtasks/micropayments
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Motivating Students: Achievement System


Like in computer games.
Categories in which achievement is tracked






Quality of submitted work
Quality of reviews
Contribution to team
Helpfulness to other students
These factors make up
reputation.
Prompt reviews/answers are more helpful,
so reward for these …
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Recognizing Achievement: Leaderboards

Students can list their achievements for
all to see.
Top 3 reviewers
in CS 210
(by average score)
Chris Jackson
John Smith
Pam Jones
Apr. 22, 2009
97.5
96.9
96.6
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Recognizing Achievement: Leaderboards

Students can list their achievements for
all to see.
Top 3 reviewers
in CS 210
(by average score)
Top 3 in author
feedback in system
(by average score)
Chris Jackson
John Smith
Pam Jones
Terry Fisher
99.8
Brooke Lee
99.7
Brooks Winter 99.5
Apr. 22, 2009
97.5
96.9
96.6
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Recognizing Achievement: Leaderboards

Students can list their achievements for
all to see.
Top 3 reviewers
in CS 210
(by average score)
Top 3 in author
feedback in system
(by average score)
Chris Jackson
John Smith
Pam Jones
Terry Fisher
99.8
Brooke Lee
99.7
Brooks Winter 99.5
Apr. 22, 2009
97.5
96.9
96.6
Ed Gehringer
efg@ncsu.edu
Achievements of
Chris Jackson
Submitted work
86.7
– rank 15 of 38 in class
Author feedback 97.5
– rank 1 of 38 in class
Contrib. to team 95.0
– rank 12 of 38 in class
http://research.csc.ncsu.edu/efg/expertiza
Recognizing Achievement: Leaderboards

Students can list their achievements for
all to see.
Top 3 reviewers
in CS 210
(by average score)
Top 3 in author
feedback in system
(by average score)
Chris Jackson
John Smith
Pam Jones
Terry Fisher
99.8
Brooke Lee
99.7
Brooks Winter 99.5
Apr. 22, 2009
97.5
96.9
96.6
Ed Gehringer
efg@ncsu.edu
Achievements of
Chris Jackson
Achievements of
Terry Fisher
Submitted work
86.7
– rank 15 of 38 in class
Author feedback 97.5
– rank 1 of 38 in class
Contrib. to team 95.0
– rank 12 of 38 in class
Vetted reviews
9
– rank 2 of 38 in class
Weighted reviews 38
– rank 1 of 38 in class
http://research.csc.ncsu.edu/efg/expertiza
Helping Fellow Students: Message Boards




Student a posts a question
Student b responds
Student a rates student b’s response
Others can rate response too.


TAs can also respond
Reliability …
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Challenging Students: Levels


In online games, players can attain new
levels of achievement.
Similarly, we can grant students powers
commensurate with their levels.



Sending a message to the whole class
Moving a post to a more appropriate
message board
Merging message boards
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
Finishing Touches: Microtasks/micropayments

Desired enhancements posted




Along with amount of “payment”
Students can voluntarily perform the
task.
Students can also post tasks.
How students should “pay” for these
tasks
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza
For more info …


http://research.csc.ncsu.edu/efg/expertiza
Ed Gehringer, efg@ncsu.edu
Apr. 22, 2009
Ed Gehringer
efg@ncsu.edu
http://research.csc.ncsu.edu/efg/expertiza