Should Your MOOC Forum
Use a Reputation System?
DERRICK COETZEE, ARMANDO FOX, MARTI A. HEARST, BJÖRN HARTMANN
UNIVERSITY OF CALIFORNIA, BERKELEY
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
1
One-slide summary
• Goal: Improve learning in MOOCs by improving forums
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
2
One-slide summary
• Goal: Improve learning in MOOCs by improving forums
• Reputation systems give points to users who post the most
useful questions and responses
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
3
One-slide summary
• Goal: Improve learning in MOOCs by improving forums
• Reputation systems give points to users who post the most
useful questions and responses
• This work: Controlled experiment in a MOOC where one group
used a forum with a reputation system and one without
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
4
One-slide summary
• Goal: Improve learning in MOOCs by improving forums
• Reputation systems give points to users who post the most
useful questions and responses
• This work: Controlled experiment in a MOOC where one group
used a forum with a reputation system and one without
• Results: Posts in the forum with reputation system received
more responses and quicker responses, but led to no significant
difference in grades/attrition
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
5
Outline
• Motivation and Background
• MOOC Forums, Reputation Systems
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
6
Outline
• Motivation and Background
• MOOC Forums, Reputation Systems
• Experimental Setup
• Randomized controlled study, MOOC integration
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
7
Outline
• Motivation and Background
• MOOC Forums, Reputation Systems
• Experimental Setup
• Randomized controlled study, MOOC integration
• Results
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
8
Outline
• Motivation and Background
• MOOC Forums, Reputation Systems
• Experimental Setup
• Randomized controlled study, MOOC integration
• Results
• Discussion / Follow-up work
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
9
MOOC Forums
• Dominant form of student
interaction and support in
MOOCs today
• “an essential ingredient of
an effective online course”
(Mak et al 2010)
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
10
MOOC Forums
• Dominant form of student
interaction and support in
MOOCs today
• Particularly in technical
classes, used primarily for
answering questions
• Administrative,
particular quiz problems,
concepts etc.
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
11
Reputation Systems
• Used by successful Q&A forums on the web
•StackOverflow
•Yahoo! Answers
• Fast high-quality answers (Mamykina et al, CHI 2011)
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
12
Reputation Systems
• Used by successful Q&A forums on the web
•StackOverflow
•Yahoo! Answers
• Fast high-quality answers (Mamykina et al, CHI 2011)
• Users have a public numeric reputation
• Users receive points when others upvote their questions
and/or responses
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
13
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
14
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
15
Reputation Systems: Gaining Privileges
• As users gain points, they acquire more moderator privileges
on the site (e.g. edit question, retag question, delete question…)
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
16
Experimental Setup
• Deployed in a single software engineering MOOC on edX
• Randomized controlled trial (aka A/B testing)
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
17
Experimental Setup
• Deployed in a single software engineering MOOC on edX
• Randomized controlled trial (aka A/B testing)
• Two independent forums
• Same software, different reputation features enabled
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
18
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
19
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
20
Experimental Setup: edX Integration
• External website linked from edX
• Authenticated using edX OpenID
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
21
Statistics and Data Overview
• Course had 5985
enrolled and 532
passing students
• 1101 study
participants, 548 in
full-featured forum
(with reputation
system) vs. 553 in
basic forum (without)
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
22
Statistics and Data Overview
• Course had 5985
enrolled and 532
passing students
Forum with
reputation
Forum without
reputation
819
587
115
144
• 1101 study
Total posts
participants, 548 in
full-featured forum
Questions
(with reputation
(threads)
system) vs. 553 in
basic forum (without)
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
23
Results: Improved forum metrics
• Faster response times
• Mean of 59 min vs. 2hr20min
• Mann-Whitney U (p < 0.001)
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
24
Results: Improved forum metrics
• Faster response times
• Mean of 59 min vs. 2hr20min
• Mann-Whitney U (p < 0.001)
• More responses per post
• Mean of 3.5 vs. 2.4 responses
• Mann-Whitney U (p < 0.001)
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
25
Results: No significant difference in student outcomes
• Grades
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
26
Results: No significant difference in student outcomes
• Grades
• Retention/Attrition
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
27
Results: No significant difference in student outcomes
• Grades
• Retention/Attrition
• Rovai’s Sense of Community
• 20-question survey measuring
how much student feels like
“I belong to a community that I
can trust and depend on”
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
28
Results: Reputation System Decreases Appeals for Help
• Example of analysis based on post content
• Examples: “Has anyone else experienced something similar?”,
“Any help would be greatly appreciated.”
• Theory: no need to appeal for help if system rewards helpers
• Looked for posts containing appeals for help
•Three raters rated posts as appeal for help or not (κ = 0.80)
• More appeals for help in forum without reputation system
•7.3% of posts vs. 2.9% (Fisher’s exact test, p ≈ 0.0002)
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
29
Do discussion forums affect learning at all?
• Cheng et al, 2011:
•"an online discussion forum was implemented at the beginning
of an undergraduate introductory psychology course [...]
students who participated in the forum tended to have better
performance in the course, and […] participating in the
discussion forum, particularly reading posts on the forum,
slightly improved exam performance“
• Replicate in MOOC setting?
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
30
Subjects – Controlled study
Forum
Forum
with
without
reputation reputation
system
system
(548)
(553)
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
Others
(4884)
31
Subjects – Descriptive/Retrospective study
Visited
any forum
at least
once
(1572)
Non-forum
users
(3221)
Viewed at least
one lecture,
HW, or quiz
(4793)
Others
(1192)
Selfselected
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
32
Results: Forum use predicts better student outcomes
• Grades
Assignment
Homework 1
Forum users
have higher
scores?
Yes
p < 0.001
Homework 2
Homework 3
Homework 4
Quiz 1
Yes
Not shown
Not shown
Yes
p < 0.02
p > 0.1
p > 0.2
p < 0.001
Yes
Yes
p < 0.001
p < 0.008
Quiz 2
Quiz 3
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
Significance
33
Results: Forum use predicts better student outcomes
• Grades
• Retention
• Median 39 of 42 days, vs.
19 of 42
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
34
Discussion
• Main result: Reputation systems improve forum experience
with no evidence of damaging student outcomes, but also no
assurance of improved outcomes
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
35
Discussion
• Main result: Reputation systems improve forum experience
with no evidence of damaging student outcomes, but also no
assurance of improved outcomes
• Possible explanation for lack of significant difference: low
participation
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
36
Discussion: Low participation
• In any given week, 13-17% of
students accessing course
website accessed experimental
forum website
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
37
Discussion: Low participation
• In any given week, 13-17% of
students accessing course
website accessed experimental
forum website
• About half of subjects
accessed 5 or less threads ever
• Significant differences in
outcomes unlikely when most
subjects don’t even use the
feature
• Also found low participation
in work with chats [L@S 2014]
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
38
Limitations
• Single course offering (generalizable?)
• Small number of very active users (see Teo & Johri, this conference)
• Bias due to consent procedure
• Contamination (outside interaction)
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
39/46
Lessons for field study researchers in MOOCs
• Study retention and performance separately
• Student who drops after getting 100% on one quiz may have same final
grade as a student who gets 25% on every quiz
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
40
Lessons for field study researchers in MOOCs
• Study retention and performance separately
• Student who drops after getting 100% on one quiz may have same final
grade as a student who gets 25% on every quiz
• Don’t rely on late surveys – only hits biased survivor group
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
41
Lessons for field study researchers in MOOCs
• Study retention and performance separately
• Student who drops after getting 100% on one quiz may have same final
grade as a student who gets 25% on every quiz
• Don’t rely on late surveys – only hits biased survivor group
• Use established platforms, working around extensibility limitations
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
42
Follow-up work
• “Chatrooms in MOOCs: All Talk and No Action” (Learning @
Scale 2014)
• Another controlled experiment
• Deployed a global synchronous chatroom on every page
• Low participation; no significant effect on student outcomes
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
43
Follow-up work
• “Chatrooms in MOOCs: All Talk and No Action” (Learning @
Scale 2014)
• Another controlled experiment
• Deployed a global synchronous chatroom on every page
• Low participation; no significant effect on student outcomes
• Peer learning chat (in progress)
• Students discuss questions in synchronous chat in small groups
• Early work with Turk simulations shows users enjoy using it
• Planned to be deployed in a MOOC
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
44
Questions?
Derrick Coetzee (dcoetzee@berkeley.edu)
Armando Fox (fox@berkeley.edu)
Marti A. Hearst (hearst@berkeley.edu)
Björn Hartmann (bjoern@berkeley.edu)
Supported by National Science Foundation under award IIS-1149799 and partially
funded by the National Endowment for the Humanities under grant HK-50011
SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?
45