more than 50 hours. There
on the order of onesomewhere
hour and another
(a) Distribution
time spent
by participants
in 6.002x (time axis
is logwas, of in
fact,
noatminimum
between
peak from the transformed);
certificate
earners
we divided
the noncertificate
earners into tranches based
on percentage of assessment activity they attempted (see also Table 1);
somewhere more thanfigure
50 hours.
Theretotal time, and attrition.
2. tranches,
was, in fact, no minimum between
140
4
Final Scores by Question
Developer
Probability
60
40
20
Number of Students
100
80
60
40
0
20
0
Week 2
Week 13
Week 14
Week 1
Week 11
Week 12
Week 10
Number of Students
120
1.0
Number of observed events
Number of observed events
Week 13
Week 14
Week 11
Week 12
Week 1
Week 2
Week 3
Week 4
Week 5
Week 6
Week 7
Week 8
Week 9
Week 10
Week 11
Week 12
Week 13
Week 14
Week 9
Week 10
Number of events/N(day)
Week 7
Week 8
Week 5
Week 6
Week 4
Week 3
Week 1 Week 8
Week 2
Week 9
Week 3
Week 4
Week 10
Week 5
Week 6Week 11
Week 7
Week 8Week 12
Week 9
Week 13
Week 10
Week 11Week 14
Week 12
Week 13
Week 14
Week 7
Week 6
Number of events/N(day)
Total Time
Week 7
Week 8
Week
9
(hrs)/N
(group)
Week 5
Week 6
Week 3
Week 4
Week 1
Week 2
Week 5
May 1
3,17
6/30/13
Week 1
Week 2
Week 3
Week 4
Week 5
Week 6
Week 7
Week 8 Week 1
Week 9
Week 10 Week 2
Week 11
Week 12 Week 3
Week 13 Week 4
Week 14
100 hr
Final
Exam
500 hr
Number of unique users
10 hr
1 hr
10 min
Mar 1
500 hr
1 min
100 hr
Total Time (hrs)/N (group)
500 hr
10 hr
1 hr
16
Jan 1
18
100 hr
Total Time (hrs)/N (group)
Number of participants spending log(t) time
10 min
14
10 hr
12
1 min
10
Course
Start
8
Number of participants spending log(t) time
6
1 hr
4
10 min
2
1500
40
20
0
1000
8
1 min
6
Student
4
0
2
2013
2014
Badge
No Badge
Outside U.S.
Trendlines
500
100
80
60
Score
80
60
Score
40
20
Statistics
100
Final Score vs Time Watched
2013
2014
Badge
No Badge
Outside U.S.
Trendlines
0
Temporal
D
The
algo
0.8
●
●●
●
●
● ● ●●
●
●
●
same
●
●●
●● ●● ● ●●
●
●
● ●●
●●●
●●
●
●●
●
●●●●● ●
● ● ●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●● ●
●
●● ● ●● ●●● ● ●
●
●
● ●● ● ●
● ●
●
●●●
● ●●●●●
●
●
●
●
●
● ●
●
● ●
●
●● ●
●
● ●● ●
● ●●● ●
●
● ●● ●
●
equi
● ●● ●●
●
●
A1
●●
●●
● ● ● ●
●●●
●
● ●●
●
●
●
●
0.6
●
●
●●
●●
● ● ●●●
●●●● ●●
●
●
●
●
●
●
●
●
●
●● ●
●
● ●
●●
●
● ●
●
●●
●
●
●
● ●
A2
prob
●●
●
●
●●
●● ● ●
●
●
●
●
●
●
● ●
●
●●
A3
0.4
one●
●
●
●
●
valu
●
●
●
●
0.2
●
●
our d
No Credit
the b
Partial Credit
0.0
Full Credit
0
5
10
15
20
25
30
gion
Hours Watched
Hours Watched
10/14/13
Observed events by date
videos, textbook, discussion, tutorial, and wiki, showing discussion forums were
to th
1 used
3 more
5 7 9 11 13 15 17 19 21 23 25 27 29 31
1 5 9 13 17 21 25 29 33 37 41 45 49 53 Number
57 61 65 69 of A1 actions
log (t[hours])
Observed events by date
1,000
8%
heavily and with strong periodicity later in the term, similar to graded activities in plot (a),
(a)
(b)
(c)
Question Number
Question Number
solv
5%
60%12%
6 in terms of frequency of accesses.
while
other components lack periodicity and vary greatly
See Figure 1
(Seaton et al., 2014)
See Figure 4
(Anderson et al., 2013)
8
Figure 1:Observed
User events
receives
5%
and
by date the badge after completing 25 A1 acObserv
1,000
the of
tions.
Actions A1 increase towards the badge boundary at Number
6,000,000
Vbj +
4
Discussion
Lecture Video
Activity Types
80,000
6 articles
Book
Homework
Lecture
Question
Lab
Tutorial
Wiki
observed
events
12%
contributed
500
A PR i l 2 01 4 | vO l. 5 7 | NO. 4 | c o m m u n i c At i o n S o f t h e Ac m 61
expense of the other site action A2 (shifting effort within the
badg
Registration
10%
25
Exam
6/30/13
francky.me/mit/moocdb/all/user_certifcate_per_country_normalized_cutoff100.html
25
site)
as well as the life-action A3 (increase in site activity).
2
YouTube
4,500,000
4
5,000
forums is especially noteworthy,
60,000 Two
Percentage of students who got a certificate
Twitteras they observed for supplementary (not explic- was partly responsible. (The wiki and
20
20
were500
neither part of the course sequence itly included in the course sequence) discussion forums had no defined num10%physics ber of resources so are excluded here.)
1
nor did they count for credit. Students e-texts in large introductory
N
and then solving for U (a ) = U (xa ) we have
0
0
15
presumably spent time in discussion courses, though the 6.002x textbook
To better understand the
15
2 middle
3,000,000
40,000 type
forums due to their utility, whether was assigned in the course syllabus. The curves representing
lecture
videos
3,000
pedagogical or social or both. The small course authors were disappointed with and lecture problems, it helps to recall
10
10
θ · x1a · U (xa+e1 ) − g(xa , p)
so th
1
spike in textbook time at the midterm, a the limited use of tutorial videos, sus- that the negative slope of the curve is
U
(a
)
=
0
0
larger peak in the number of accesses, pecting that placing tutorials after the the density of students accessing that
1,500,000
1 − θ(x2a + x3a )
20,000 let n
5
5
(t[hours])
1,000
(they
were fraction of that
as in Figure 3, and the decrease in text- homework and laboratorylog
course component (see
W
sequence Figure 5b and Figure 5c). Interestingly,
book use after the midterm are typical meant to help) in the course (a)
(b)
(c)
Midterm
Final
Midterm
Final
Midterm
Final
0
0
0
of textbook use when online resources
Since we have already computed U (a1 + 1) = U (xa+e1 ), this
affec
figure 4. time on tasks.
are blended with traditional on-campus
0
0
log (t[hours])
becomes-0an
courses. Further studies comparing
3 - 16
3 -0optimization
9 -15 -21 -27 -02 problem
8
4
0 in
6 3 2variables:
8
4
0
6
1
7
3
-1ues
3
3
3
3
3
4
4- 0 4- 1 4- 2 4- 2 5- 0 5- 0 5- 1 5- 2 5- 2 6- 0 6- 0 6- 1
- 01 - 01 2
blended and online textbook use are(a) Certificate earners average time spent, in hours per week,
2- 0 2- 0 2- 0 12- 0 12- 0 12- 0 12- 0 12- 0 12- 0 12- 0 12- 0 12- 0 12- 0 12- 0 12- 0 12- 0 12- 0 12- 0
13 13 013
(b)on each course component;
(c)
60
10
01 201 201 20Bystander
0 20 20 20 20Viewer
0 20 20 Collector
20 20 tions
2
Solver
midterm
and
final
exam
weeks
are
shaded.
2
2
2
20 20 20 20 20 All-rounder
20
also relevant.
1
Date
θ · xa · C − Date
g(xa , p)
(a)
(c)
(b)
Percentage use of course compogram
50
maximize
(Instructure, 2014) nents.
(Börner
and
Polley,
2014)
(Seaton
et
al.,
2014)
(Dernoncourt
et
al.,
2013)
AP Ri l 2 014 | vOl . 57 | NO. 4 | c ommu n i c At i on S of t h e Ac m 61
Along with student time alloca2
3
6.002x:
MIT/MITx
(b)
P (S|F )xa (a)
0.106
0.277
0.408
0.017
1 − θ(x
a + xa ) 0.192
tion, the fractional use of the various
a1 =
40
100%
2 course components continues to be an
100%
3
other
P
(F
|S)
0.050
0.334
0.369
0.894
0.648
other
j
important metric for instructors decidbook
discussion activity increasing
over
of participants effectively drop
out, with
3.5
•
Midterm these extremes, only
Final a noticeable shoul90%
90% n i c At i on S of t h e Ac m profile
AP Ri l 2 014 | vOl . 57 | NO. 4 |1 c ommu
61
30
Legend
ing how to improve
their courses and
subject
to xjaof
≥ interactive
0, j = 1, 2, 3visual
and analytics
xa = 1 showing the n
wikifirst
contributed
articles
Figure 5:
Example
the semester. Lecture question
events
der (see Figure 2a). The intermediate 10but also invested less time in the
3.0
home
exam
•
researchers studying the influence of
80%
80%
Year
decay
early
as
homework
activity
indurations
are
filled
with
attempters
we
few
weeks
than
the
certificate
earn2.5
course structure on student activity and
20
problem
Table 3: How engagement styles are j=1
distributed on the ML3
,
(k
2013
1
learning.
For
fractional
use,
we
plotted
informational
creases.
Textbook
use
peaks
during
exdivided
into
tranches
(in
colors)
on
the
ers.
The
correlation
of
attrition
with
2.0 review earlier homework, or
70%
es, fractional use of those resources,
the distribution for the lecture videos textbook,
70%
tutorial
2014
percentage
of certificate
earners search the discussion forums. We thus and use
10and there is a noticeableindex
•
forum.
P (S|F
) is probability
ofC.engagement
style
given
forum
of resources
during
problem
isthe
distinctly
bimodal:
76% of students
drop in
basis
of how
many
assessment items less time spent in early weeks begs ams,
1.5
where
we’ve
replaced
U
(x
In
the
appendix
of
the
a+e1 ) with
having
accessed
at
least
a
certain
per60%
60%
Among
the more on
significant
accessed over 20% of the videos (or had a unique opportunity to observe solving.
activity after the midterm, as
they
attempted
homework and ex- the
Students
1.0
(k1is−
centage
of resources
in a course
0 question of whether motivating textbook
extended
of theor
paper
we show
how
to solve
problemP (F |S)
presenceversion
(reading
writing
to at
least
one this
thread);
24%
of students
accessed
lesscompothan transitions to all course components findings is that participants who atforum
0 in traditional courses.
780
18
50%
50% to invest more time would
nent (see
5). Homework
is typical
ams:
(gray) attempted
< 5% of 10students
0.5 by students while working
tempted
overbrowsers
5% of the homework
rep20%),
andFigure
33% accessed
over and
80%labs
of accessed
1
2 wiki
3
390
•
currently
available
at http://moocdbfeaturediscovery.csail
0
25
50
75
100
125
150
efficiently.
For
our
purposes,
the
important
point
is
that
the
optimal
10
10
10
(each
15% of
overall
grade) merits
reflect high
exam
probability
of
forum
presence
given
engagement
style.
Time
on
tasks.
Time
represents
the
homework;
tranche
1
(red)
5%–15%
of
resented
only
25%
of
all
participants
the
videos.
This
bimodality
furproblems.
In
previous
studies
of
onincrease
retention
rates.
0.0
1
40%
40%
1
fractional
inflection
in these line problem solving this information but accounted for 92% of the total
ther
study use.
into The
learning
preferences;
#
of
assignment
questions
submitted
can
be
computed
using
the
solution
of
the
distribution
in
state
a
of
this
platform
is
that
it
effectively
democratizes
MOOC
dat
principal
cost
function
for
students,
so
homework;
tranche
2
(orange)
15%–25%
In
the
rest
of
this
article,
we
reSi
lecture
# of quiz attempts
curves
near 80%
been higher
time spent in the course; indeed, 60%
for
example,
domight
somehave
students
learn was simply missing.
Top 5 Countries
30%
1
important to study how students
of homework;
tranche
3 (green) > 25% of strict30%
ourselves to certificate earners, it is
but for
the course
policyexclusively?
of dropping the
+ 1. Since
wedirectly
know xawork
= p with
for allor
states
a access
such thatto specific
state acannot
Figure 6 highlights student transi- of the time
was invested by
the 6% who
from
other
resources
Or
ince
2013
2014
who
either
gain
two
lowest-graded
assignments.
The
U.S. (33.4%)
U.S. (39.5%)
20%
allocate
time
among
available
course
lecture
homework;
and
tranche
4
(cyan)
>25%
of
1
as
they
accounted
for
the
majority
of
did they master the content prior to tions from problems (while solving ultimately received certificates. Par20%
≥ k, we can use
this to compute
the
optimal xa for all a such
a participate
India (6.8%)
India (8.7%)
low course?
proportionate
use of textbook
and them) to other course components, ticipants who left the course invested
(thos
15,19
the
The distribution
of lecFigure 4 shows the
homework and 25% of midterm exam. resource consumption; we also want- components.
to
inconjecture
MOOC
data
science.
U.K. (4.6%)
U.K. (5.1%)
1
be plausible
to
that
the
forum
a place where
10%
tutorials is similar
thebetween
distribution
Figure
8:10%
Number
of handed-in
assignments
(left)
and
quizzes
ture-problem
use istoflat
0% treating homework sets, the midterm, less effort than certificate earners,
Canada (3.7%)
Canada (3.3%)
=
k
−
1,
and
recurse
all
the
way
back to might
a0 , thusbe
solving
that
a
Q
most
time
is
spent
on
lecture
videos;
Certificate
earners
(purple)
attempted
ed
to
study
time
and
resource
use
over
253 students did not identify a country in 2013
Netherlands (3.5%) Spain (2.7%)
and
the
final
exam
as
separate
assesswith
those
investing
the
least
effort
and
80%,
then
rises
sharply,
indicating
figure 5. fractional use of resources.
0%
0%high-achievers.
110 students did not identify a country in 2014
students
engage in problem
back-and-forth
discussions case.
about course consince
three
hours
per week is
oftwo
theweeks
available
mid- thefor
whole
semester.
the
user’s optimization
in the one-dimensional
(right)
the first
tendinghomework,
to
that many students accessed nearly ment types of interest. Figure 6 shows duringmost
US IN to
CNfour
RU DE
PL BR
old b
US IN CN RU DE PL BR
earners The median
them. Along with the fact that the discussion forum is the most fre- leave sooner.
total
duration of the schedterm, Most
andcertificate
final exams.
Frequency (a)
of 6.002x:
accesses.
Figure 3a close to
MIT/MITx
(b)the
Crypto
1: Stanford/Coursera
To illustrate
the effects
by ask
our model,
we compute
the students
Percentage of certificate earners who
accessed
greater than %R
type of course
(a) 6.002x: MIT/MITx
(b)
Crypto
1:of that
Stanford/Coursera
tent,
or a place
wherecaptured
students
questions
that
other
N
See Figure 2 all of(a)resource.
(Dernoncourt
al.,
(Dernoncourt
et al., 2013)
(Meiss)
quent
destination
during
homework invested theet
plurality
of2013)
their time in
time on
drops
Thelecture
density of questions
users is the negative
slope of
the usage
curve. Two points
indicating
total
time
spent
in 1:
the course
for each shows the number of active users per uled videos, students who rewound
(a) 6.002x: MIT/MITxthe
Crypto
Stanford/Coursera
problem
solving,
lecture videos,
though approximately
steadily
in the
first half
bimodality
of lecture
videoof
usethe
are term
plotted:(see
76% of students
accessed
> 20%though
of lecturelecture videos (b)
optimal
policy
on
a
simple
illustrative
instance.
In
this
instance,
rectl
Figure 4: Differences in relative use of resources by students
from
different
countries.
A
student’s
country is
answer,
or
a
place
where
students
weigh
in
one
after
another
on
a
Electorate
and
reviewed
the
videos
must
compentranche
was
0.4
hours,
6.4
hours,
13.1
day
for
certificate
earners,
with
large
videos,
33% of
students accessed
> 80% of lecture
videos. (b)the
Bimodal
distribution
for
consume
most
time. During
exams 25% of the earners watched less than
2),andthis
distribution
suggests
3 Figure
derived from the IP addressOur
s/he commonly
logsisin made
from.
analysis
possible by the granular level
videos accessed (as percentage). And (c) distribution of lecture questions accessed.
12on Sunday deadlines
we place one threshold badge on action A1 with boundary 25 (i.e.,
suggests
the need
for hours,
a
students
not only allocated less time (midterm and final are similar), previ- 20%. This
furth
or of detail
hours,
30.0 hours,
53.0
and 95.1 peaks
for graded sate for those speeding up playback
4
Figure
4:
Left
plot
shows,
via
coloring,
the
ratio
of
certificate
winners
to
the
number
of
registrants
on
a
per
class-related
issue.
Our
goal
here
is
to
develop
an
analysis
frame14 date has been to build community
francky.me/mit/moocdb/all/user_certifcate_per_country_normalized_cutoff100.html
1/1 done homework is the primary
Qs
follow-up
investigation
into the
to them, some abandoned the lecture ously
available
activityomitting
traces on
Stack Overflow . Each individual
ML1
PGM1
Our progress
support,
videos.
hours,
respectively.
In coraddition to these homework and labs
but notin
forthe
lecture
b = (25,to
1)). We then solve the user optimization
problem
de- develop
know
destination, while the book consumes relations between resource use and
problems entirely.
10
As in fact1being used.
100%
country basis for 6.002x offered via edX platform.
Hungary
(16.21%),
Spain (14.55%) and Latvia
(14.40%)
performed
by a user
recorded
and timestamped,
work
that
can
clarify
how
the
forums
are
Theis most
significant
change
over which aftranches,
over 150the
certificate earn- questions.
Thereaction
is a downward
trend
12
ML2
PGM2
100
100%
Finally, just
we highlight
Resources
used when problem solv- the most time. Student 4behavior on learning.
other
as
a
function
of
a
(see
Figfined
above
and
plot
the
optimal
x
concepts
for
open
source
frameworks
for
MOOC
data
science.
In
the
c
a
other
3
Q-votes
the
first
seven
weeks
was
the
apparent
ers
spent
fewer
than
10
hours
in
the
in
the
weeks
between
the
midterm
and
problemsto
thusthe
contrasts
sharply significant popularity of the discusin the
use of exam
fords
ability
observe the complete
sequence of
90%
are the highest. Right plot shows, via coloring, ing.
thePatterns
ratio
ofsequential
certificate
winners
ML3 us the book
PGM3to directly
8
In
particular,
we’d
like
to
address
the
following
questions:
home
2 number of registrants on
10
76%
students
90%
ure
1).
The
user’s
optimal
mixing
probability
gets
progressively
with
behavior
on
homework
problems.
sion
forums
in
spite
of
being
neither
resources
mayof hold
clues
80 by students accessed
A-votes
transfer
of
time
from
lecture
questions
course,
possibly
representing
a
highly
the
final
exam
(shaded
regions).
No
these
frameworks,
seeking
more
community
involvement
and
collab
profile
> 20% of videos
IVMOOC Video Views
actions userswikitake and measure
their progress towards obtaining
1
Note that becauseRussia
homework
was
ag- required Netherlands
nor included in the navigato cognitive
and even
affective state.
a per country basis for the Stanford cryptography
course
offered
via Coursera.
(17.24%),
more deflected
away from his preferred p as he approaches
the
homework, as in Figure 4. Considerskilled tranche seeking certification. homework
or labs were assigned
in the to 80%
80%
0
8 of improving
6
exam Stack Overflow data from the site’s inception on
We therefore
explored the interplay be- gregated, we could not isolate
“refer- tion sequence. If this social learning
badges. We use
file:///C:/Users/francky/Downloads/outpu
ultimate
goal
educational
outcomes through
datainscien
60
0
10 20 30 40 50 60
70 80 90 100
index (see
ing
a
performance-goal
orientation
Similarly,
just
over
250
test
takers
spent
last
two
weeks
before
the
final
exam,
•
Does
the
forum
have
a
more
conversational
structure,
whic
played a significant role
70%
(16.43%) and Germany (12.95%) are highest. tween use of assessment and learning ences to previous assignments” for stu- component
badge
boundary.
The
user
also
increases
his
probability
on
A
%R Resources
1
70%
July
31, 2008
to December
31,
2010.
problem
6
resources by transforming time-series dents doing homework.
in the success
of 6.002x,
ahours
totally asynWeek 1
Figure
5),
it
should
be
noted
that
homefewer
than
10
in
the
course
and
though
the
peaks
persist.
We
plotted
4
40
informational
by offloading
probability
masscontribute
from both other
action
types:
he
60%
a
single
student
may
many
times
to
the
same
thread
data into transition matrices between
chronous alternative might be less apforum
33% of students
60%in events (clicks subject tutorial
work
counted
toward
the
course
grade,
completed
more
than
25%
of
both
exactivity
to
time
Activity around the badge boundary. We first
examine how
4
accessed
> 80%conclusion
of videos
12
wiki
resources. The transition matrix
conpealing, at least for a complex topic
C
is shifting
his effort
within theevolves,
site (moving
mass from structure
2
9
lecture questions
not. But
ams and
butelectronics.
did not earn a certificate.
cutoffs)
student
per day for to whereas
50%
as2 the
conversation
or aprobability
more straight-line
tains all 20individual resource-resource This article’s major contribution
to like circuits
50% per active
exam
users’
propensities
take
different
types of did
actions
vary as they
6
have
transitions
we
aggregated
into
transicourse
analysis
is
showing
how
MOOC
Some
of
these
results
echo
effects
even
on
mastery-oriented
grounds,
stuThe
average
time
spent
in
hours
per
assessment-based
course
components
the other site action A2 ) and also increasing participation on the
3
Coursera course
edX
Course
40%
approach
the
badge
boundary.
We
aim
to
analyze
both
how
users
0
40%
0
in0 which most students contribute just once and don’t return?
tions between
major course compo- data can be analyzed in 0qualitatively seen in
on-campus
studies of how
Week 2
week
for70participants
in each tranche and learning-based
components in Fig- dents might have viewed completion
ther
site overall
(moving probability mass from the life-action A3 ).
0
20
40 6.002x60 different
80 ways100
0
10 20 30
40 structure
50 60 affects
80 resource
90 100 use
The completeness
of the
to address
important
course
0
2 their 4
6
8 onasthe
10
shift
effort
between
actions
site and12
change 14
their overall
Title
Cryptography I nents.
Circuits
and
Electronics
(6.002x)
%R Resources
homework
sufficient
evidence
of
is
shown in
Figure 2c.
−60
−40
−20
0
20
60
ure 3b
sets of 30%
30%and Figure 3c. Homework
%R
– Percentage
of Resources
Accessed
lecture
in Tranches atoutcomes
learning environment
means
students
issues:
attrition/retention, distribu- and performance
•
Does
the
forum
consist
of
high-activity
students
who
initiate
The
authors
acknowledge
discussions
and40support
from
a number
Note
that
unifying
participation
and
shifting
site
effort
in
this
level
of
site
activity.
For
each
user
we
bin
the
number
of
actions
of
lecture
understanding
lecture
content.
The
tempting
fewer
assessment
items
not
introductory
(college)
courses.
This
did
not
have
to
leave
it
to
reference
the
and
the
discussion
forums
account
for
tion
of
students’
time
among
resourcThread20%
length
Instructors
Dan Boneh
Anant Agarwal, Gerald Sussman, Piotr Mitros
20%
Number
days
relative
to badge win
way
does
not
necessarily
make
them students
equivalent
in ourfollow
framework.
of time
in discussion
threads
and of
low-activity
who
up, or
are inst
the
each
typeper
bystudent,
day. Thisprominence
way, changes
in spent
the relative
number of varonly taper off earlier, as the majority the highest rate of
activity
MOOC
space.
The
following
researchers’
contributions
have
been
figure 6. transitions to other components during problem solving on (a) homework, (b) midterm, and (c) final. Arrows are thicker in proportion
University Stanford
MIT
10%
SeeUniversity
Figure
(Seaton
et
al.,
2014) 63
(Anderson
et
al.,
2014)
(Anderson
et al.,to2013)
10%
to overall number of transitions, sorting components from top to bottom; node size represents
total time spent
on that
component.
Week 3 3
, p) could
be chosen
penalThe deviation
penalty
function
g(xacentral
ious types of actions per day indicate a user shifting his effortsand
on building
threads
initiated
by
less
contributors
and
then
picked
community
support:
Sherif
Halawa,
Andreas
Paepcke
Civic Duty
62 commu nicAt ionS of t he Acm | APRi l 20 14 | vOl. 57 0%
4
Length
6 weeks
14 weeks
theB site,
the daily sum0%over Aall actions
measures
his overall
Figure 9:| NO.Number
ofand
a function
ofparize deviating
user’s preference for the life-action more than
10 byfrom
B as C
A
C distinct contributors
up
more
active
students?
Franck
Dernoncourt,
Colin
Taylor,
SherwinQsWu, Elaine Han (MIT),
(a) Homework
(b) Midterm Exam
(c) Final Exam
whe
ticipation
level
(where
an
increase
or
decrease
in
participation
can
(b) Crypto 1: Stanford/Coursera
Platform
Coursera
edX
deviating from his preferences for the various site
actions.
thread length. (a) 6.002x: MIT/MITx
As
be
interpreted
as
steering
away
from
or
towards
the
life-action).
com
•
How
do
stronger
and
weaker
students
interact
on
the
forum?
8
Figure 5: Differences in relative use of resources based on grade cohorts.
Start date
Jan 13th, 2013
March 5, 2012
Multiple badges. So far we considered a caseQ-votes
where there is only
Nodes
Week 4
For each badge, we take the complete set of users who ever
the o
Data Manager
one •badge
on
site. We
now show
that
a similar
algorithm
Registrants
21,744
154,763
Can
wetheidentify
features
in the
content
of the
posts that indiA-votes
Data Miner
achieved that badge and axis-align their activity profiles by letting
6
Designer
can
solve
the
user’s
optimization
problem
when
there
are
multiple
cate which students are likely to continue in the course and
“day 0” denote the day they receive the badge. To eliminate posLibrarian
4. COURSE
ACTIVITY
badges that
the same dimension.
Programmer
sibleFORUM
population effects,
we restrict the set of users to those who
Week 5
4 all target
which
are likely
to leave?
[1]
K.
Veeramachaneni,
F. Dernoncourt,
Project Manager
(kj , 1)} for j C.
∈ Taylor,
{1, . . . , Z.
m}Pardos,
be a and U
Let B = {bj =
Table 1: Courses overview
at least
60 days
before
and after
theypaper,
win the the
badge.
We now movewere
on active
to our
second
main
focus
of the
Usability Expert
standards
for
mooc
data
science.
1st
workshop
on
Massive
Open
set of m2 badges all on the same action A1 , and assume that
Visualization Expert
Figure 3 shows user activity in the days surrounding the awardforums, which provide
a mechanism for students to interact with
The course forums contain many threads of non-trivial length
No Information
ing of the Electorate and Civic Duty badges. Notice how activk1 < k2 < . . . < km without loss of generality. (If kj = kj each other. Because
forums
are cleanly
separated
from
and
we0ask
threadsthese
are
long
because
small
set of
Week 6
ity onCoursera’s
the targeted actions
(Q-votes
for Electorate,
Q-votes
and [2]
A- K. for
Veeramachaneni,
F Dernoncourt,
Taylor,
S
some
j = whether
j , then U.-M
wethese
can O’Reilly,
consider
two badges
as aaC.
sinallows researchers to refine the scripts.
This framework is called MOOCViz. For its current appearance
Edges
−60
−40
−20
0
20
40
60
votes for
Civic Duty)
increases
substantially
before
users
achieve dards
the
course
materials,
students
can
choose
to
consume
the
course
people
are
each
contributing
many
times
to
a
long
conversation,
Direct communication via Twitter
gle
badge
with
value
equal
to
the
sum
of
their
individual
values.)
and systems for mooc data science. MOOCDB Technical or
R
see Figure 6. MOOCViz is currently available at http://moocviz.csail.mit.edu/.
Midterm Score vs Time Watched
2
Researcher
C
Attempted > 25% HW
and > 25% Midterm
(b) percentage of total measured time spent by each tranche; and
(a) Distribution of time spent by participants
in 6.002x (time axis is logCertificate earners
Attempted > 25% HW
Midterm
Scores
12
(c)
average time a student
invested per week.
The shaded
regions by
nearQuestion
Week 8
transformed); we divided the noncertificate earners into tranches
based
Midterm
Final
1
and Week 14 represent the time span for the midterm and final exams.
on percentage of assessment activity they attempted (see also Table 1);
1,500spent by participants in 6.002x (time axis is log10
(b) percentage
of total measured time spent by each tranche; and
(a) Distribution of time
8%
(c) average time a student invested per week. The shaded regions near Week 8
transformed); we divided the noncertificate earners into tranches based
5%
60%
8 14 represent the time span for the midterm and final exams.
and Week
on percentage of assessment activity they attempted (see also5%Table 1);
Browsers
contributed
articles
1,000
Attempted
> 5% HW
6 Attempted > 25% HW
12%
and > 25% Midterm
Attempted > 15% HW
Attempted > 25% HW 4 Certificate earners
Browsers
12
Midterm
Final
figure 3. frequency of accesses.
500
Attempted > 5% HW
10% Attempted > 25% HW
and > 25% Midterm
Attempted > 15% HW
2
Certificate earners
From left to right, number of unique certificate earners N active per day, their average
The shaded regions near Week 8 and Week 14 represent
1,500 Attempted > 25% HW
10
12each day for assessment-based and
No
Creditthe time span for the midterm and final exams.
number of accesses
learning-based course
Midterm
Final
0
0
components. Plot (a) highlights
the periodicity and trends of the certificate earners. Plot
Partial Credit
8%
5% including homework,
(b) is for assessment,
60% lab, and lecture questions, showing number
FullofCredit
8
accesses per active10
users that day. learning-based components
in plot (c) include lecture
1,500
5%
Number of participants spending log(t) time
1
Browsers
Attempted > 5% HW
Teacher
B
figure 2. tranches, total time, and attrition.
Attempted > 15% HW
80
Student
A
(b) percentage of total measured time spent by each tranche; and
(c) average time a student invested per week. The shaded regions near Week 8
and Week 14 represent the time span for the midterm and final exams.
francky.me/mit/moocdb/all/user_certifcate_per_country_normalized_cutoff100.html
# of users
Week 14
Week 12
Lecture Question
Wiki
Week 13
Week 11
Discussion
Tutorial
Week 10
Week 8
Week 9
Homework
Book
Week 7
Week 6
Week 5
Week 4
Week 2
Week 3
Week 1
Total time (hrs)/N (certificate earners)
IVMOOC Video Views
Welcome ]
Course Overview ]
Visualization Framework ]
Workflow Design ]
Introduction ]
Download, Install, and Visualize Data with Sci2 ]
Legend Creation with Inkscape ]
Weekly Tip: Extending Sci2 by adding Plugins ]
Welcome ]
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Burst Detection ]
Introduction ]
Temporal Bar Graph: NSF Funding Profiles ]
Burst Detection in Publication Titles ]
Weekly Tip: Sci2 Log Files ]
Welcome ]
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Color ]
Introduction ]
Choropleth and Proportional Symbol Map ]
Congressional District Geocoder ]
Geocoding NSF Funding with the Generic Geocoder ]
Weekly Tip: Memory Allocation ]
Welcome ]
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Design and Update of a Classifcation System: The UCSD Map of Science ]
Comparison of Text and Linkage−Based Approaches ]
Introduction ]
Mapping Topics and Topic Bursts in PNAS ]
Word Co−Occurrence Networks with Sci2 ]
Weekly Tip: Removing Files from the Data Manager ]
Welcome ]
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Algorithm Comparison ]
Introduction ]
Visualizing Directory Structures ]
Weekly Tip: Create your own TreeML (XML) Files ]
Welcome ]
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Clustering ]
Backbone Identification ]
Error and Attack Tolerance ]
Exhibit Map with Andrea Scharnhorst ]
Introduction ]
Co−Occurrence Networks: NSF Co−Investigators ]
Directed Networks: Paper−Citation Network ]
Bipartite Networks: Mapping CTSA Centers ]
Weekly Tip: How to Use Property Files ]
Welcome ]
Exemplary Visualizations ]
Dynamics ]
Hans Rosling's Gapminder ]
Deployment ]
Color Perception and Reproduction ]
The Making of AcademyScope ]
Introduction ]
Evolving Networks with Gephi ]
Exporting Networks From Gephi with Seadragon (Zoom.it) ]
Plotly Demo ]
TEI with John Walsh pt 1 ]
TEI with John Walsh pt 2 ]
Week 1
Week 2
Lecture Video
Lab
Homework
Book
Tutorial
Lecture Question
Lecture Video
Welcome ]
Course Overview ]
Visualization Framework ]
Workflow Design ]
Week 5
Introduction ]
Download, Install, and Visualize Data with Sci2 ]
Legend Creation with Inkscape ]
Weekly Tip: Extending Sci2 by adding Plugins ]
Week 6
Welcome ]
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Burst Detection ]
Week 7
Introduction ]
Temporal Bar Graph: NSF Funding Profiles ]
New
Burst Detection
in Publication Titles ]
Weekly
Tip:
Sci2 Log Files ]
Video Types
Number of Hits
Welcome ]
Theory 2013
Theory 2014
0
200
400
600
800
Hands−On 2013
Hands-On 2014
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Color ]
Introduction ]
Choropleth and Proportional Symbol Map ]
Congressional District Geocoder ]
Geocoding NSF Funding with the Generic Geocoder ]
Weekly Tip: Memory Allocation ]
Welcome ]
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Design and Update of a Classifcation System: The UCSD Map of Science ]
Comparison of Text and Linkage−Based Approaches ]
Introduction ]
Mapping Topics and Topic Bursts in PNAS ]
Word Co−Occurrence Networks with Sci2 ]
Weekly Tip: Removing Files from the Data Manager ]
Welcome ]
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Algorithm Comparison ]
Introduction ]
Visualizing Directory Structures ]
Weekly Tip: Create your own TreeML (XML) Files ]
Welcome ]
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Clustering ]
Backbone Identification ]
Error and Attack Tolerance ]
Exhibit Map with Andrea Scharnhorst ]
Introduction ]
Co−Occurrence Networks: NSF Co−Investigators ]
Directed Networks: Paper−Citation Network ]
Group membership
Bipartite Networks: Mapping CTSA Centers ]
Weekly Tip: How to Use Property Files ]
Welcome ]
Exemplary Visualizations ]
Dynamics ]
Hans Rosling's Gapminder ]
Deployment ]
Color Perception and Reproduction ]
The Making of AcademyScope ]
Introduction ]
Evolving Networks with Gephi ]
Exporting Networks From Gephi with Seadragon (Zoom.it) ]
Plotly Demo ]
TEI with John Walsh pt 1 ]
TEI with John Walsh pt 2 ]
Week 4
2
Lecture Question
Acknowledgements
18
4,11,19
APR il 2 0 1 4 | vO l . 5 7 | N O. 4 | co m m u n i cAt i o n S o f t he Acm
Discussion
Homework
Homework
Lab
Discussion
Lab
Lecture Video
Lab
Discussion
Lecture Question
Book
Lecture Video
Book
Lecture Video
Book
Tutorial
Lecture Question
Lecture Question
Wiki
Wiki
Wiki
Tutorial
Tutorial
Number of actions per day
Network
5
Week 3
%N Certificate earners
Topical
Figure 4: Left plot shows, via coloring, the ratio of certificate winners to the number of registrants on a per
country basis for 6.002x offered via edX platform. Hungary (16.21%), Spain (14.55%) and Latvia (14.40%)
are the highest. Right plot shows, via coloring, the ratio of certificate winners to the number of registrants on
a per country basis for the Stanford cryptography course offered via Coursera. Russia (17.24%), Netherlands
(16.43%) and Germany (12.95%) are highest.
Number of actions per day
Summary
1/1
# distinct contributors
francky.me/mit/moocdb/all/user_certifcate_per_country_normalized_cutoff100.html
21
%N Certificate earners
4
Lecture Video
Lab
%N – Certificate earners accessing > %R – Resources
Geospatial
3
# of users
Percentage of students who got a certificate
Coursera course
edX Course
References
Title
Cryptography I
Circuits and Electronics (6.002x)
Instructors
Dan Boneh
Anant Agarwal, Gerald Sussman, Piotr Mitros
University Stanford University
MIT
Number of days relative to badge win
the badge,
and other
then almost
immediately
returns
near-baseline
content independently
of the
students,
or they
can to
also
comwhether they
are long because a large number of students are each
levels. Also notice that most of the other site (Ho
actions
are
not adLength
6willweeks
14
MOOCViz, (Börner
when completed,
allow
data analyses
to2014)
be
and Polley,
2014)analytic and visualization scripts for MOOC(Seaton
et al.,weeks
et
al.)
2014)
Figure 3: Number
of actions
per day as a function(Ginda,
of number
of
municate with their
peers.
contributing
roughly
once.
versely affected—the rates of these actions remain relatively stashared, under the understanding that the data being analyzed is organized according to the MOOCDB
days relative the time of obtaining a badge. Notice steering in
Following
our
classification
of
students
into
engagement
styles,
As
a
first
way
to
address
this
question,
we
study
the
mean numble
over
time.
Since
the
four
actions
shown
in
the
Figure
are
the
Platform
Coursera
edX
the sense of increased activity on actions targeted by the badge.
data model. MOOCViz activity
entails developing a set of templates and support for software sharing
main activities on the site, this means users increase their overall
Week 7
64
c o m m un ic At io n S o f t he Ac m
| A P R i l 20 1 4 | vO l . 5 7 | N O. 4
plus developing a means of sharing MOOC analytics results through web-based galleries.
New
our first question is a simple one: which types of students visit
activity level on the site in the days leading up to achieving these
ber of distinct contributors in a thread of length k, as a function of
The first salient feature of the vector field is the gradient from
6