contributed articles
developers
Results
count for this Results
in the future.
The novelty and publicity surrounding
The novelty and publicity surrounding
MOOCs
inattracted
early 2012
MOOCs in
early 2012
a largeattracted a large
Results
number
ofwho
registrants
number of
registrants
were more who were more
The novelty and
publicity
surrounding
curious than
serious.
We still
take par-We still take parcurious
than
serious.
MOOCs in early
2012
attracted
a
large
ticipation in assessment as an indicaticipation
in
as an indication of serious
intent.
Of assessment
the
154,000
number of registrants
who
were
more
tion
of
serious
intent.
the 154,000
registrants
in 6.002x
in spring
2012, Of
Students taking MOOCs need to be extremely or- curious than
Teachers
of
MOOCs
need
effective
means to
Researchers that study human learning are keen
serious.
We
still
take
par46,000 never
accessed the
course,
and in spring 2012,
registrants
in
6.002x
in the
assessment
as
an
indicaganized and disciplined. MOOCs have no weekly ticipationkeep
track
of
and
guide
the activities, progress,
to understand what teaching and learning methmedian time spent by all remain46,000
never
accessed
and
participants
only154,000
one
hour (see the course,
tion of serious
intent.
Ofwas
the
in-class teacher encounters. MOOCs also have
anding
any
problems
encountered
by
thousands
ods work well in a MOOC environment and now
the
time
spent
by
all
remainFigure 2a).
Wemedian
had expected
a bimodal
registrants in 6.002x in spring 2012,
distribution
ofparticipants
total
time spent,
withonly
a understand
much less peer-pressure. Courses might have
of students.
They
need
to
the efhave massive amounts of detailed data with which
ing
was
one
hour
(see
46,000 never large
accessed
the course,
and only
peak of “browsers”
who spent
Figure
2a).
We
had
expected
a bimodal
vastly different schedules, activities such as labs the median
fectiveness
of
materials,
exercises,
and exams
to work. As all student interactions—with learning
on thespent
order ofby
oneall
hour
and another
time
remaindistribution
ofearners
total time
spent,
a to continpeak
from
the certificate
atgoals
and capstone projects, deadlines, and grading
with
respect
to hour
learning
in with
order
materials, teachers, and other students—are reing participants
was
only
one
(see
somewhere
more
than of
50 “browsers”
hours. There who spent only
large
peak
Figure 2a).
Wewas,
had
expected
a
bimodal
rubrics. Effectively using one or more MOOC
uously
improve
course
schedules, activities, and
corded in a MOOC, human learning can be studied
in fact, no minimum between
thespent,
orderwith
of one
of total on
time
a hour and another
platforms can itself be a major learning exercise. distribution
grading
rubrics.
at an extreme level of detail. Many MOOC teachfigure 2.peak
tranches,
total time,
and certificate
attrition.
from
the
earners
at
who spent only
Plus, many students are not used to collaboratinglarge peak of “browsers”
ers double as learning researchers as they are insomewhere
more than 50 hours. There
on the order of one hour and another
(b) percentage of total measured time spentterested
by each tranche; and
(a) Distribution
time spent
by participants
in 6.002x (time axis
is logwas, ofin
fact,
noatminimum
between
with students from different disciplinary backto make their own MOOC course work
peak from the certificate
earners
Analysis Types
vs.
User Needs
B. Teacher
grounds and cultures that speak different native
languages and live in different time zones.
transformed); we divided the noncertificate earners into tranches based
on percentage of assessment activity they attempted (see also Table 1);
somewhere more thanfigure
50 hours.
Theretotal time, and attrition.
2. tranches,
was, in fact, no minimum between
figure 2. tranches, total
D
D
D**
DE
D
D
GA
Activity
Feedback
D = Dashboard
Supports Surveys
DE = Data Export
DE
DE
D
DE
DE
D
D
D
DE
DE
D
DE
GA
GA
X
X
✓
✓
✓
✓
GA = Google Analytics
*If students choose to take optional demographic survey
**If toggled to record
We would like to thank Samuel T. Mills for re-designing the figures in this paper and all
2013 and 2014 IVMOOC students for their feedback and comments, enthusiasm, and
support. All R code and all Sci2 Tool workflows are available and documented at
cns.iu.edu/2015-MOOCVis.
4. Topical
Welcome ]
Course Overview ]
Visualization Framework ]
Workflow Design ]
Introduction ]
Download, Install, and Visualize Data with Sci2 ]
Legend Creation with Inkscape ]
Weekly Tip: Extending Sci2 by adding Plugins ]
Welcome ]
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Burst Detection ]
Introduction ]
Temporal Bar Graph: NSF Funding Profiles ]
Burst Detection in Publication Titles ]
Weekly Tip: Sci2 Log Files ]
Welcome ]
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Color ]
Introduction ]
Choropleth and Proportional Symbol Map ]
Congressional District Geocoder ]
Geocoding NSF Funding with the Generic Geocoder ]
Weekly Tip: Memory Allocation ]
Welcome ]
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Design and Update of a Classifcation System: The UCSD Map of Science ]
Comparison of Text and Linkage−Based Approaches ]
Introduction ]
Mapping Topics and Topic Bursts in PNAS ]
Word Co−Occurrence Networks with Sci2 ]
Weekly Tip: Removing Files from the Data Manager ]
Welcome ]
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Algorithm Comparison ]
Introduction ]
Visualizing Directory Structures ]
Weekly Tip: Create your own TreeML (XML) Files ]
Welcome ]
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Clustering ]
Backbone Identification ]
Error and Attack Tolerance ]
Exhibit Map with Andrea Scharnhorst ]
Introduction ]
Co−Occurrence Networks: NSF Co−Investigators ]
Directed Networks: Paper−Citation Network ]
Bipartite Networks: Mapping CTSA Centers ]
Weekly Tip: How to Use Property Files ]
Welcome ]
Exemplary Visualizations ]
Dynamics ]
Hans Rosling's Gapminder ]
Deployment ]
Color Perception and Reproduction ]
The Making of AcademyScope ]
Introduction ]
Evolving Networks with Gephi ]
Exporting Networks From Gephi with Seadragon (Zoom.it) ]
Plotly Demo ]
TEI with John Walsh pt 1 ]
TEI with John Walsh pt 2 ]
140
Number of observed events
Number of observed events
Summary
1/1
Week 1
Week 2
Lecture Video
Lab
Homework
Book
Tutorial
Lecture Question
Lecture Video
Figure 4: Left plot shows, via coloring, the ratio of certificate winners to the number of registrants on a per
Topical analysis provides an answer country basis for 6.002x offered via edX platform. Hungary (16.21%), Spain (14.55%) and Latvia (14.40%)
to the question of “what” is going on
are the highest. Right plot shows, via coloring, the ratio of certificate winners to the number of registrants on
in a course.
a per country basis for the Stanford cryptography course offered via Coursera. Russia (17.24%), Netherlands
(16.43%) and Germany (12.95%) are highest.
Student cohorts might be created
based on prior expertise, geospatial
region or time zone, access patterns,
project teams, or grades.
Probability
60
40
20
0
Week 14
Week 13
Week 12
Week 11
Week 10
Week 9
Week 1
Week 2
Week 3
Week 4
Week 5
Week 6
Week 7
Week 8
Week 9
Week 10
Week 11
Week 12
Week 13
Week 14
Week 8
Number of events/N(day)
Week 13 Week 6
Week 14 Week 7
Week 5
Week 4
Week 3
Week 2
Week 14
Week 1
Week 13
Week 12
Number of Students
120
100
80
60
40
Week
0 1120
Week
9
Total Time (hrs)/N
(group)
Week 10
Number of Students
Week 1 Week 5
Week 2
Week 3 Week 6
Week 4
Week 5 Week 7
Week 6 Week 8
Week 7
Week 8 Week 9
Week 9
Week 10Week 10
Week 11
Week 11
Week 12
Week 13Week 12
Week 14
Week 7
Week 8
Week 4
Week 2
Week 1
Final
Exam
Week 1
Week 2
Week 3
Week 4
May 5
1
Week
Week 6
Week 7
Week 8
Week 9
Week 10
Week 11
Week 12
Week 13
Week 14
Week
3 of events/N(day)
Number
Week 6
Week 5
Week 4
Week 3
Week 2
Week 1
Total Time (hrs)/N (group)
500 hr
10 hr
100 hr
Number of unique users
1 hr
10 min
Mar 1
500 hr
100 hr
10 hr
500 hr
100 hr
10 hr
1 hr
10 min
1 min
Jan 1
1 min
10 min
500
1000
1500
Number of participants spending log(t) time
Total Time (hrs)/N (group)
Number of participants spending log(t) time
120
100
Score
40
60
80
100
80
60
Score
40
IVMOOC Video Views
5. Network
Acknowledgements
francky.me/mit/moocdb/all/user_certifcate_per_country_normalized_cutoff100.html
Week 3
Welcome ]
Course Overview ]
Visualization Framework ]
Workflow Design ]
Week 5
Introduction ]
Download, Install, and Visualize Data with Sci2 ]
Legend Creation with Inkscape ]
Weekly Tip: Extending Sci2 by adding Plugins ]
Week 6
Welcome ]
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Burst Detection ]
Week 7
Introduction ]
Temporal Bar Graph: NSF Funding Profiles ]
New
Burst Detection
in Publication Titles ]
Weekly Tip: Sci2 Log Files ]
Video Types
Number of Hits
Welcome ]
Theory 2013
Theory 2014
0
200
400
600
800
Hands−On 2013
Hands-On 2014
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Color ]
Introduction ]
Choropleth and Proportional Symbol Map ]
Congressional District Geocoder ]
Geocoding NSF Funding with the Generic Geocoder ]
Weekly Tip: Memory Allocation ]
Welcome ]
Exemplary Visualizations ]
Overview and Terminology ]
Workflow Design ]
Design and Update of a Classifcation System: The UCSD Map of Science ]
Comparison of Text and Linkage−Based Approaches ]
Introduction ]
Mapping Topics and Topic Bursts in PNAS ]
Word Co−Occurrence Networks with Sci2 ]
Weekly Tip: Removing Files from the Data Manager ]
Data Miner
Welcome ]
Exemplary Visualizations ]
Designer
Overview and Terminology ]
Workflow Design ]
Librarian
Algorithm Comparison ]
Programmer
Introduction ]
Visualizing Directory Structures ]
Project Manager
Weekly Tip: Create your own TreeML (XML) Files ]
Welcome ]
Usability Expert
Exemplary Visualizations ]
Overview and Terminology ]
Visualization Expert
Workflow Design ]
No Information
Clustering ]
Backbone Identification ]
Error and Attack Tolerance ]
Exhibit Map with Andrea Scharnhorst ]
Introduction ]
Edges
Co−Occurrence Networks: NSF Co−Investigators ]
Directed Networks:
Paper−Citation Network ]
Direct communication
via Twitter
Bipartite Networks: Mapping CTSA Centers ]
Group membership
Weekly Tip: How to Use Property Files ]
Welcome ]
Exemplary Visualizations ]
Dynamics ]
Hans Rosling's Gapminder ]
Deployment ]
Color Perception and Reproduction ]
The Making of AcademyScope ]
Introduction ]
Evolving Networks with Gephi ]
Exporting Networks From Gephi with Seadragon (Zoom.it) ]
Plotly Demo ]
TEI with John Walsh pt 1 ]
TEI with John Walsh pt 2 ]
Week 4
2
Lecture Question
18
Number of actions per day
D
Acknowledgements
4,11,19
A P R i l 2 0 14 | vO l . 5 7 | N O. 4 | c o m m u n i c At i o n S o f t h e Ac m
Discussion
Homework
Homework
Number of actions per day
Test / Assignment
Completion
Content Usage
Breakdown
Path Through Content
Time-stamped Activity
"Active" Student Count
Student Breakdown
# of users
DE
# distinct contributors
DE
# of users
D
1.0
Week 14
D
21
Week 13
Performance
Final Scores by Question
Lecture Question
Wiki
Week 12
D
D
Week 11
D
D
Discussion
Tutorial
Week 10
D
D
Week 9
D
D
Geospatial data might be examined
at different levels of aggregation: by
address, city, country, or IP address.
Week 8
Class Grades
Quiz / Question
Breakdown
Student Breakdown
3. Geospatial
Homework
Book
Week 7
DE
X
X
GA
X
Week 6
DE
DE
DE
D
D*
Lecture Video
Lab
Week 5
DE
X
X
X
X
4
francky.me/mit/moocdb/all/user_certifcate_per_country_normalized_cutoff100.html
Percentage of students who got a certificate
Week 4
D
D
D
D
D
Email
Gender
Demographics Age / Birth Year
Location
Level Education
6/30/13
Week 3
GCB
Week 2
Coursera
%N Certificate earners
Canvas
3,17
%N Certificate earners
edX
18
Week 1
Data Field
for different types of students.
Browsers
Attempted > 5% HW
Attempted > 25% HW
time, and attrition.
and > 25%
Attempted
>
15%
HW
(a) Distribution of time spent by participants
inMidterm
6.002x
Certificate earners
Attempted > 25% HW
Total time (hrs)/N (certificate earners)
Data Type
0
Student input and feedback. Feedback data allows course providers to learn more about
student learning goals and motivation, intended use of course, and content hoped to
learn. It data also contains information about what students liked or disliked in terms of
course content, structure, grading, and teacher interaction.
Temporal analyses and visualizations
tell when students are active over
the span of a course. Data might be
examined at different levels of aggregation: by minute, hour, day, week, or
semester, by course modules, or before and after a midterm or final.
Student
Feedback Data
2. Temporal
%N – Certificate earners accessing > %R – Resources
How students are using class resources, such as the time and date of watching videos,
reading material, turning in homework, taking quizzes, or using the discussion forum.
Most platforms break down usage by content and media type (i.e. page views, assignment views, textbook views, video views). Path through content via inbound and outbound links is important for understanding learning trajectories.
Number of participants spending log(t) time
Activity Data
20
Student performance based on graded assessments. This is generally collected from
homework, quizzes, and examinations, but it also includes results from pre-course surveys designed to examine student knowledge before they take the course.
0
Performance Data
Line graphs, correlation graphs, and
box-and-whisker plots are all examples of how statistical data can be
rendered visually.
Midterm Score vs Time Watched
20
General student demographics, including age, gender, language, education level, and location. Demographic data is commonly acquired during the registration process, and additional demographic data can be acquired via feedback surveys.
1. Statistics
Platform developers need to design systems that
support effective course design, efficient teaching, and secure but scalable course delivery.
They need to support times of high traffic and resource consumption and schedule maintenances
during low activity times.
(c) average time a student invested per week. The shaded regions near Week 8
and Week 14 represent the time span for the midterm and final exams.
(b) percentage of total measured time spent by each tranche; and
(time axis is logMidterm
Scores
by Question
(c)
average time a student
invested
per week.
The shaded
regions near Week 8
transformed); we divided the noncertificate earners into tranches
based
12
Midterm
Final
1
and Week 14 represent the time span for the midterm and final exams.
on percentage of assessment activity they attempted (see also Table 1);
Final Score vs Time Watched
1,500
(b) percentage
of total measured time spent by each tranche; and
(a) Distribution of time
spent by participants in 6.002x (time axis is log10
●●
●
(c) average time a student invested per week. The shaded regions near Week 8
transformed); we divided the noncertificate earners into tranches based
●
●●●
8%
●● ● ● ●
●● ●
●
●
●●
5%
●
60%
●
●
● ●●
●
●
●●
● ●
●
8 14 represent the time span for the midterm and final exams.
and Week
on percentage of assessment activity they attempted (see also Table 1);
● ●● ●
●
●●●
●
5%Browsers
●
●
●
●● ●
●●●●
●
●
●
1,000
●
contributed
●●
●
●
Attempted > 5%articles
HW
Attempted > 25% HW
●
●
6
12% Attempted > 15% HW
●
●
and > 25% Midterm
Certificate earners
Attempted > 25% HW
Browsers
4
12
Midterm
Final
Attempted > 5% HW
500
figure 3. frequency
of accesses.
Attempted
>
25%
HW
10%
and > 25% Midterm
Attempted > 15% HW
2
● 2013
Certificate
earners
● 2014
Attempted
>
25%
HW
1,500
From left to right, number of unique certificate earners N active per day,
10 their averageNo CreditThe shaded regions near Week 8 and Week 14 represent
● Badge
12
Midterm
Final
number of accesses each day
the time
span for the midterm and final exams.
No Badge
0
0 for assessment-based and learning-based course
Partial
Credit
8%
Trendlines
components. Plot (a) highlights the periodicity
and trends of the certificate earners. Plot
FullofCredit
(b) is for assessment, including5%
homework, lab, and60%
lecture questions, 8
showing number
1,500
0
2
4
6
8
10
12
14
accesses per active users5%
that day.10
learning-based components in plot (c) include lecture
1 used
3 more
5 7 9 11 13 15 17 19 21 23 25 27 29 31
Hours Watched
videos,
textbook,
discussion,
tutorial,
and wiki, showing discussion forums were
log (t[hours])
1,000
8% and with strong periodicity later in the term, similar to graded activities in plot (a),
heavily
Question Number
(a)
(b)
5%
60%12%lack periodicity and vary greatly in (c)
6
while other components
terms
of
frequency
of
accesses.
8
5%
1,000
4
Activity Types
Discussion
Lecture Video
6 articles
contributed
12%
500
Book
Homework
Question
Lab
Tutorial
Wiki
Registration
APRil 2014 | vOl. 57 | NO. 4 | co mmunicAtio
nS o f the Lecture
Acm 61
10%
Exam
25
25
francky.me/mit/moocdb/all/user_certifcate_per_country_normalized_cutoff100.html
YouTube
2
forums is especially noteworthy, as they observed for supplementary (not explic- was partly responsible. (The4wiki and
Twitter
5,000
20
were500
neither part of the course sequence itly included in the course sequence) discussion forums had no defined num20
nor did they count for credit. Students e-texts in large introductory
10%physics ber of resources so are excluded here.)
0
presumably spent time in discussion 0courses,16 though the 6.002x textbook
To better understand the middle
15
2
15
forums due to their utility, whether was assigned in the course syllabus. The curves representing lecture videos
0
Demographic Data
D. Platform Developer
The fact that th
●
algorithm; the
●
●
●
●●
●●
●●● ●
●
●
●
0.8
●
●
●●
●
● ●● ●
●
● ● ●●
same number o
●●
●
●
● ●● ●
●
●●
●
●
●
● ●●
●
●●
●●
●●
● ●
●
●
equidistant from
A1
0.6
●
●
● ●
A2
problem is the
A3
0.4
one-dimensiona
value of each s
● 2013
0.2
● 2014
our dynamic pr
No Credit
● Badge
No Badge
Partial Credit
the badge boun
Trendlines
0.0
Full Credit
0
5
10
15
20
25
30
gion between th
0
2
4
6
8
10/14/13
Observed
events
date37 41 45 49 53 57 61 65 69
1 5 9 13 17
21 25
29by33
Hours Watched
to the one-badg
Number
of
A
1 actions
Observed events by date
Question Number
solved analogo
Figure 1:Observed
User events
receives
and j is identic
by date the badge after completing 25 A1 acevents by date
the of
tions.
Actions A1 increase towards the badge boundary at Number
VObserved
6,000,000
bj + U (xkj ) I
80,000
the events
expense of the other site action A2 (shifting effort within observed
badges to solve
6/30/13
site) as well as the life-action A3 (increase in site activity).
4,500,000
60,000 Two targeted d
Percentage of students who got a certificate
Now we cons
and then solving for U (a1 ) = U (xa ) we have
3,000,000
40,000 types of actions
pedagogical or social or both. The small course authors were disappointed with 3,000
and lecture problems, it helps to recall
10
θ · x1a · U (xa+e1 ) − g(xa , p)
spike in textbook time at the midterm, a the limited use of tutorial videos, sus- that the negative slope of the curve is
10
so there are two
1
0 in the number of accesses, pecting that placing tutorials after the the density of students accessing
0
U (a ) =
larger peak
that
2
3
let n = 2 for co
1
−
θ(x
+
x
(they
were fraction of that course component (see
as in Figure 3, and the decrease in text- homework and laboratorylog
1,500,000
a
a)
(t[hours])
5
20,000
5
1,000
book use after the midterm are typical meant to help) in the course sequence Figure 5b and Figure 5c). Interestingly,
We begin by
(a)
(b)
(c)
of textbook use when online resources
1
Midterm
Final
Midterm
Final
Midterm
Final
0
0
0
figure 4. time on tasks.
Since we have already computed U (a + 1) = U (xa+e1 ), this
are blended with traditional on-campus
affect the optim
0
0
courses. Further studies comparing
log (t[hours])
becomes 0an
optimization problem in 3 variables:
ues in two state
Certificate earners average time spent, in hours per week, on each course component;
blended and online textbook use are
- 13 - 16 - 19 - 22 - 25 - 28
- 3 3- 09 3- 15 3- 21 3- 27 4- 02 4- 08 4- 14 4- 20 4- 26 5- 02 5- 08 5- 14 5- 20 5- 26 6- 01 6- 07 6- 13
3
01 3- 01 3- 01 3- 01 3- 01 3- 01 3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
(a)
(b)
(c)
midterm
and
final
exam
weeks
are
shaded.
2
3
1
1 01 01 01 01 01
also relevant.
12 12 12 12 012 012 012 012 0Viewer
12 012 012 012 Collector
12 12 12 12 12 012
60
10
20 20 tions
2
2 are
2 the
2
2sam
Solver
20 20 20 20Bystander
2
2
2
21 2
2
2
2
20 20 20 20 20 All-rounder
2
Date
Percentage use of course compoθ
·
x
·
C
−
g(x
,
p)
a
(a)
(c)
(b)
a
Date
nents. Along with student time allocagramming table
maximize
APRil 2014 | vOl. 57 | NO. 4 | communicAt
50 ionS of t he Acm 61
3
2
(a)
6.002x:
MIT/MITx
(b)
Crypto 1:2
tion, the fractional use of the various
xa
1
1 − θ(x
P
(S|F
)
0.106
0.277
0.408
0.017
a + xa ) 0.192
=
k1 and a
a
2 course components continues to be an
100%
100%
40
other
3
important metric for instructors decidother
3.5
P
(F
|S)
0.050
0.334
0.369
0.894
0.648 • R: a finite r
Midterm
Final
book
with
discussion
activity
increasing
over
these
extremes,
only
a
noticeable
shoulof
participants
effectively
drop
out,
ing
how
to
improve
their
courses
and
j
j
90%
Legend
APRil
2014
|
vOl.
57
|
NO.
4
|
communicAt
ionS
of
t
he
Acm
61
90%
1
profile
3.0
wiki
subject
to xaof
≥ interactive
0, j = 1, 2, 3visual
and analytics
xa = 1 showing the number of
contributed
articles
researchers studying
the influence of
home
Figure
5:
Example
semester. Lecture question
events
der (see Figure 2a). The intermediate 10but also invested less time in the
first the30
exam
• H: an infin
2.5
Year
80%
course structure on student activity and
80%
j=1
decay
early
as
homework
activity
indurations
are
filled
with
attempters
we
few
weeks
than
the
certificate
earnproblem
2013
20
learning. For fractional use, we plotted
Table 3: How engagement styles are distributed on the ML3
(k1 , k2 − 1) ex
2.0
informational
70%
creases.
Textbook
use
peaks
during
exdivided
into
tranches
(in
colors)
on
the
ers.
The
correlation
of
attrition
with
es,
fractional
use
of
those
resources,
the
distribution
for
the
lecture
videos
textbook,
review
earlier
homework,
or
2014
70%
the percentage of certificate earners
tutorial
index
1.5
use ofmany
resources
during problem
ishaving
distinctly
bimodal:
76%a of
students
thus and
• V : an infin
accessed
at least
certain
per- search the discussion forums. We basis
10
and
there
is
a
noticeable
drop in
of how
assessment
items less time spent in early weeks begs ams,
forum.
P
(S|F
)
is
probability
of
engagement
style
given
forum
where
we’ve
replaced
U
(x
)
with
C.
In
the
appendix
of
the
a+e
1
60%
60%
accessed
20% of
videos
(or had a 1.0
unique opportunity to observe solving. Among the more significant
Students
centage ofover
resources
in athe
course
compothey attempted
on participants
homeworkwho
and
ex- the
question of whether motivating textbook activity after the midterm, as
(k1is− 1, k2 ) ex
780
extended
version
of theor
paper
we show
how
to solve
this
problemP (F |S)
0
findings
is
that
at24%
of
students
accessed
less
than
transitions
to
all
course
components
forum
nent (see Figure 5). Homework and labs
presence
(reading
writing
to
at
least
one
thread);
0.5
0
18
10
50%
wiki
50%
ams: browsers
(gray)
5% of students
to invest more time would is typical 1in traditional courses.
390
tempted over
5% of attempted
the homework<rep20%),
33%
accessed
80%high
of accessed by students while working
(each and
15% of
overall
grade)over
reflect
2
3
• Q: a quadr
currently
available
at http://moocdbfeaturediscovery.csail.mit.ed
0
25
50
75
100
125
150
efficiently.
For
ourforum
purposes,
the important
point
is that the optimal
0.0
exam the
10
10
10
1
only 25%1of(red)
all participants
the
videos. use.
This The
bimodality
merits
fur- problems. In previous studies ofhomework;
on- resentedtranche
Time
on
tasks.
Time
represents
5%–15% of increase
retention
rates.
fractional
inflection
in these
probability
of
presence
given
engagement
style.
40%
40%
1
but accounted for 92% of the total
ther
study
learning
curves
nearinto
80% might
havepreferences;
been higher line problem solving this information
can beitcomputed
usingdemocratizes
the solution of the
distribution
in
state
a
principal cost #
function
for
students,
so
Similarly
to b
homework; tranche 2 (orange) 15%–25% # of In
the
rest
of
this
article,
we
reof
this
platform
is
that
effectively
MOOC data
science
assignment
questions
submitted
lecture
Top 5 Countries
of
quiz
attempts
time spent in the course; indeed, 60%
for
do some
learn
butexample,
for the course
policystudents
of dropping
the was simply missing.
30%
1
30%
of homework;
3 (green)
> who
25% of strict ourselves to certificate earners, it is important to study how students
2013
2014
+ 1. Since
wedirectly
know xawork
= p with
for allor
states
a access
such thatto specific
state acannot
Figure 6 highlights student transiof the timetranche
was invested
by the 6%
from
other resources
exclusively? The
Or
incentive
to dev
two lowest-graded
assignments.
who
either
gain
data.
It
U.S. (33.4%)
U.S. (39.5%)
1
ultimately
Par- of
did
master the
priorand
to tions from problems (while solving
allocate
time among available
course
20%
homework;
andreceived
tranchecertificates.
4 (cyan) >25%
lecture
as they
1
low they
proportionate
usecontent
of textbook
India (6.8%)
India (8.7%)
20% accounted for the majority of
a ≥ k, we can use this to compute the optimal xa for all a such
(those with a ≥
U.K. (4.6%)
U.K. (5.1%)
15,19
ticipants
who25%
left the
course
invested
the
course?
The distribution
of lec- them) to other course components,
tutorials
is similar
to the distribution
Figure
4
shows
the
components.
homework
and
of
midterm
exam.
resource
consumption;
we
also
wantto
participate
in
MOOC
data
science.
1
Canada (3.7%)
Canada (3.3%)
10%
be plausible
conjecture
a place where
ture-problem use is flat between 0% treating homework sets, the midterm, less effort than certificate earners,
10%Number of handed-in assignments
Figure
8:
(left)
and
quizzes
= k − to
1, and
recurse allthat
the the
way forum
back to might
a0 , thusbe
solving
that
a
253 students did not identify a country in 2013
Netherlands (3.5%) Spain (2.7%)
Quadrants H
most
time
is
spent
on
lecture
videos;
Certificate
earners
(purple)
attempted
ed
to
study
time
and
resource
use
over
figure
5.
fractional
use
of
resources.
and
the
final
exam
as
separate
assesswith
those
investing
the
least
effort
and
80%,
then
rises
sharply,
indicating
110 students did not identify a country in 2014
0% three to four hours per week is
0% high-achievers.
students
engage in problem
back-and-forth
discussions case.
about course conthe
user’s optimization
in the one-dimensional
since
most of
thethe
available
homework,
mid- thefor
whole
semester.
during
first two weeks
tending (right)
to
that many students accessed nearly ment types of interest. Figure 6 shows
old badge in on
US IN CN RU DE PL BR
US IN CN RU DE PL BR
the discussion
forum
is the mostterm,
fre- leave
sooner.
Most
certificate
earners
all of
Along
withearners
the fact
that greater
(a) them.
Percentage
of certificate
who accessed
than %R of that
type of course
total duration of the schedand final exams. The median
Frequency (a)
of6.002x:
accesses.
Figure 3a close to(b)the
To illustrate
the effects
captured
by ask
our model,
we compute
the students
MIT/MITx
Crypto 1: Stanford/Coursera
Now we are
Thelecture
density of questions
users is the negative
slope
of
the usage
curve.
Two points
indicating
tent,
or
a
place
where
students
questions
that
other
quent
destination
during
homework invested the plurality of their time in
the resource.
time on
drops
(a) 6.002x: MIT/MITx
(b)
Crypto
1:
Stanford/Coursera
uled
videos,
students
who
rewound
total
time
spent
in
the
course
for
each
bimodality of lecture video use are plotted: 76% of students accessed > 20% of lecture
shows
the
number
of
active
users
per
lecture videos,
though approximately
in the first half of the term (see problem solving, though lecture videos (b)
optimal
policy
on a simple
illustrative
instance.
Inone
this after
instance,
videos, and 33% of students accessed > 80% of lecture videos. (b) Bimodal distribution for
(a) 6.002x: MIT/MITx
Crypto
1: 13.1
Stanford/Coursera
rectly
fill in in
Electorate
3 steadily
Figure
Differences
in relative
use of resources
by students
different
countries.
student’s
country is
and from
reviewed
the
videos A
must
compentranche
0.4earners
hours,
6.4 hours,
answer,
or
a
place
where
students
weigh
in
another
on
a
day4: for
certificate
earners,
with large
the most
time. During exams
25%was
of the
watched
less than
Figure
2),accessed
this (as
distribution
suggests
videos
percentage). And
(c) distributionconsume
of lecture questions
accessed.
analysis
derived from the IP address
s/he
commonly is
logsmade
in from. possible by the granular level of detail
we place one threshold badge on action A1 with boundary 25 (i.e.,
suggests
needand
for 95.1
a
students not only allocated less time (midterm and final are similar), previ12on SundayOur
furthest from th
or
hours,20%.
30.0This
hours,
53.0 the
hours,
peaks
deadlines for graded sate for those speeding up playback
4
14
class-related
issue.
Our
goal
here
is
to
develop
an
analysis
frameQs
ously
done
homework
is
the
primary
follow-up
investigation
into
the
corto
them,
some
abandoned
the
lecture
.
Each
individual
available
in
the
activity
traces
on
Stack
Overflow
Figure
4:
Left
plot
shows,
via
coloring,
the
ratio
of
certificate
winners
to
the
number
of
registrants
on
a
per
francky.me/mit/moocdb/all/user_certifcate_per_country_normalized_cutoff100.html
1/1
ML1
PGM1 omitting videos.
hours,relations
respectively.
In addition to these homework and labs
but not for lecture
Our progress
date
community
support,
b = (25,to
1)).
Wehas
thenbeen
solve to
thebuild
user optimization
problem
de- develop
knowsoftware
from solv
destination,
while the book
between resource use and
problems
entirely.
4 consumes
As
10
100
action
performed
by
a
user
is
recorded
and
timestamped,
which
af1
12
The
most
significant
change
over
tranches,
just
over
150
certificate
earnquestions.
There
is
a
downward
trend
100%
work
that
can
clarify
how
thexforums
are in of
fact
used.In the coming ye
country basis for 6.002x offered via edX platform.
Hungary
(16.21%),
Latvia
(14.40%)
most time. (14.55%)
Student 3behaviorand
on learning.
Finally,
we highlight the
Resources
used when problem
solv- the Spain
ML2
PGM2
a MOOC
function
a being
(see
Figfined
above
and
plot the
optimal
100%
a as
other
concepts
for
open
source
frameworks
for
data
science.
Q-votes
other
sharply
significant
the discusing. Patterns in the sequential
use of exam problems thus contrasts
the first seven
weeks
the apparent
ers spent
fewerpopularity
than 10ofhours
in the in the weeks between
midterm
and
fords
usthethe
ability
to directly
observe
thewas
complete
sequence of
76% of students
80
ML3
PGM3
8
90%
10The user’swe’d
book
Video
Views
are the highest.IVMOOC
Right
plot
shows, via coloring, resources
the ratio
of certificate
winners
the21 number
of
registrants
on
In
like
to address
theA-votes
following
questions:
accessed
> 20%
of videos with
ure
1).particular,
optimal
mixing
probability
gets
progressively
behavior onto
homework
problems.
sion
forums
in
spite
of
being
neither
by students
may hold
clues
home
transfer
of
time
from
lecture
questions
course, possibly representing a highly the 90%
final exam
(shaded
regions).
No
these
frameworks,
seeking
more
community
involvement
and collaborations
actions
users
take
and
measure
their
progress
towards
obtaining
profile
Note that because homework
was ag- required nor included in the navigato cognitive and even affective state.
0
wiki
more deflected
away from his preferred p as he approaches the
to 80%
homework, as in Figure 4. Considera per country basis for the Stanford cryptography
course
offered
via
Russia
Netherlands
tranche
certification.
homework
or badges.
labs were assigned
in
the Overflow
60
8
We therefore
explored
the interplay
be- Coursera.
0 (17.24%),
10skilled
20 30
40
50
60seeking
90
100
gregated, we could
not isolate
“refertion
sequence.
If70this80social
learning
80%
6
U (x
We
use
Stack
data
from
the
site’s
inception
on
file:///C:/Users/francky/Downloads/output
(1)/output/observed_eve
exam
ultimate
goal
of
improving
educational
outcomes
through
data
science.
%R Resources
tween use of assessment and learning ences to previous assignments” forSimilarly,
ing
a
performance-goal
orientation
(see
stu- component
played
atest
significant
role
just
over
250
takers
spent
last
two
weeks
before
the
final
exam,
badge
The user
alsoa more
increases
his probabilitystructure,
on A1 in which
• boundary.
Does
the forum
have
conversational
index
Week 1
70%
(16.43%) and Germany (12.95%) are highest.
July
31,
2008
to
December
31,
2010.
6
70%
40by transforming time-series
resources
dents doing homework.
in
the success
of 6.002x,
a totally
asyn-and
problem
Figure
5),
it
should
be
noted
that
homefewer
than
10
hours
in
the
course
though
the
peaks
persist.
We
plotted
4
33% of students
by offloading
probability
masscontribute
from both other
action
types:
he
data into transition matrices accessed
between
chronous alternative might be less ap> 80% of videos
informational
a
single
student
may
many
times
to
the
same thread
12
60%
work
counted
toward
the
course
grade,
completed
more
than
25%
of
both
exactivity
in
events
(clicks
subject
to
time
forum
tutorial
Activity around
the badge boundary. We firstwikiexamine how
60%
4
resources.
conclusion
pealing, at least for a complex topic
20 The transition matrix con9
Consider a s
is shifting
his
effort
within
the
site
(moving
probability
mass
from
2 per active
whereas
lecture
questions
did
not.
But
ams
notand
earn
a certificate.
cutoffs)
student
per
day
for
6
tains all individual resource-resource This article’s major contribution
to but
likedid
circuits
electronics.
as
the conversation evolves, or a more straight-line structure,
users’ propensities to take50%
different types of actions
50%
examvary as they
3how MOOC
2
have already co
transitions
we
aggregated
into
transicourse
analysis
is
showing
Some
of
these
results
echo
effects
even
on
mastery-oriented
grounds,
stuthe
other
site
action
A
The
average
time
spent
in
hours
per
assessment-based
course
components
0
2 ) and also increasing participation on the
Week 2
0
Coursera course
edX
Course
approach
the badge
boundary.
We aim to analyze both how users
tions between
major20 course40compo-60 data can
be analyzed
in qualitatively
seen40 in 50on-campus
studies
of how
0
in
which
most
students
contribute
just once A
and
don’t return?
40% might have viewed completion
0
80
100
0
10week
20 30
60
70 80
90 100tranche
dents
for
participants
in
each
40%
and
learning-based
components
in
Figther simplify:
).
site
overall
(moving
probability
mass
from
the
life-action
0
nents. The completeness
of
the
6.002x
different
ways to address important course%R
structure
affects resource use
3
Resources
%R – Percentage of Resources
Accessed
0
2
4
6
8
10
12
14
shift
their
effort
between
actions
on
the
site
and
change
their
overall
Title
Cryptography I learning environment
Circuits
and issues:
Electronics
(6.002x)
homework as sufficient evidence of
is shown
in Figure 2c.
Tranchesin at- ure 3b and Figure 3c. Homework sets of 30%
−60
−40
−20
0
20 shifting
40support
60effort
and performance
outcomes
means students
attrition/retention,
distribu30%
lecture
Note
that
unifying
participation
and
site
in this
The
authors
acknowledge
discussions
and
from
a number
•
Does
the
forum
consist
of
high-activity
students
who
initiateof resear
introductory
(college) courses.
Thisnot
did not have to leave it to reference the tion of students’ time among resourclevel
of
site
activity.
For
each
user
we
bin
the
number
understanding
lecture
content. of
Theactions of
tempting
fewer assessment
items
and the discussion
forums
account
for
Thread
length
lecture
Instructors
Dan Boneh
Anant Agarwal, Gerald Sussman, Piotr
Mitros
Number
of days
relative
to badge win
20%
way
does
not
necessarily
make
them students
equivalent
in ourfollow
framework.
20%
prominence
of
time
spent
in
discussion
only
taper
off
earlier,
as
the
majority
the
highest
rate
of
activity
per
student,
MOOC
space.
The
following
researchers’
contributions
have
been
threads
and
low-activity
who
up, or
are instrumenta
the
each
type
by
day.
This
way,
changes
in
the
relative
number
of
varfigure
6.
transitions
to
other
components
during
problem
solving
on
(a)
homework,
(b)
midterm,
and
(c)
final.
Arrows
are
thicker
in
proportion
Week 3
63
from top to bottom; node size represents total time spent on that component.
U (xa
University Stanford University to overall number of transitions, sorting components
MIT
, p) could
be chosen
toand
penalThe deviation
penalty
function
g(xacentral
10% indicate a user shifting his efforts on
10%
ious types of actions per day
and
building
community
support:
Sherif
Halawa,
Andreas
Paepcke
(Stanford
threads
initiated
by
less
contributors
then
picked
Civic
Dutyfor the life-action more than
62 c o m m u nic Atio nS o f the Ac m | A PR il 2014 | vOl. 57 | NO. 4
ize
deviating
from
user’s
preference
Length
6 weeks
14 weeks
the
site,
and
the
daily
sum
over
all
actions
measures
his
overall
parFigure 9:
Number
of
distinct
contributors
as
a
function
of
0%
0%
10
Franck
Dernoncourt,
Colin
Taylor,
Sherwin
Wu,
Elaine Han (MIT),
Piotr
MiU
A
B
C
upfrom
by more
active
students?
A
B
C
(a) Homework
(b) Midterm Exam
(c) Final Exam
where
C
=
j
deviating
his
preferences
for
the
various
site
actions.
Qs
ticipation
level
(where
an
increase
or
decrease
in
participation
can
Platform
Coursera
edX
thread length.(a) 6.002x: MIT/MITx
(b) Crypto 1: Stanford/Coursera
computed. This
As
Week 4
be
interpreted
as
steering
away
from
or
towards
the
life-action).
•
How
do
stronger
and
weaker
students
interact
on
the
forum?
8
Multiple
badges.
So
far
we
considered
a
case
where
there
is
only
Nodes
Start date
Jan 13th, 2013
March 5, 2012
Figure 5: Differences in relative use of resources based on grade cohorts.
Q-votes
the one we had
For each badge, we take the complete set of users who ever
Data Manager
one badge on the site. We now show that a similar algorithm
120
Data
C. Researcher
80
A. Student
Course
Start
Along with these challenges, the sheer volume of data available from MOOCs provides
unprecedented opportunities to study how learning takes place in online courses. This
paper explores the use of data analysis and visualization as a means to empower teachers, students, researchers, and platform developers by making large volumes of data
easy to understand. First, we introduce the insight needs of these four user groups. Second, we review existing MOOC visual analytics studies. Third, we present a framework
for MOOC data types and data analyses to support different types of insight needs.
Fourth, we present exemplary data visualizations that make data accessible and empower teachers, students, developers, and researchers with novel insights. The outlook discusses future MOOC opportunities and challenges.
Scott R. Emmons
Robert P. Light
Katy Börner
developers are considering ways to account
for this in theways
future. to acare
considering
1 min
Massively open online courses (MOOCs) offer instructors the opportunity to reach students in orders of magnitude greater than they could in traditional classroom settings,
while offering students access to free or inexpensive courses taught by world-class educators. However, MOOCs provide major challenges to teachers (keeping track of thousands of students and supporting their learning progress), students (keeping track of
course materials and effectively interacting with teachers and fellow students), researchers (understanding how students interact with materials and each other), and MOOC
platform developers (supporting effective course design and delivery in a scalable way).
MOOC Visual Analytics: Empowering Students, Teachers, Researchers,
and Platform Developers of Massively Open Online Courses
1 hr
Abstract
only direct interactions with the homefigure 1. Screenshot of typical student view in 6.002x.
work are logged with homework recontributed articles
sources.
There
are
clearly
alternatives
to
only direct interactions with the homeAll course
components
accessed from the interface shown below. The left sidebar
figure 1. Screenshot of typical
student
view are
in 6.002x.
this
approach
(such
as
considering
all
defines the course sequence; weekly units include lecture sequences (videos and
work are logged
with
homework
reonly direct
interactions
with
the
homefigure 1. Screenshot of typical
student view
in 6.002x. lab, and tutorials. The header navigation provides access to
questions),
homework,
time alternatives
between opening
sources. Therework
are are
clearly
to re- and answering
logged with homework
21
materials,
including
digital The
textbook,
discussion forums, and wiki.
All coursetime
components
accessed from the
interface
shown below.
left sidebar
). are supplementary
a
problem
as
problem-solving
are clearly alternatives
this approachsources.
(suchThere
as considering
all to defines
The
main
frame
represents
first
lecture
sequence;
the components
course sequence;
weekly
include
(videos
and beige boxes below the header
All course
are accessed
from
theunits
interface
shown lecture
below.the
Thesequences
left sidebar
Our
time-accumulation
algorithm
is
this approach
(such
as considering all
defines the homework,
course sequence;
weekly
units
include
lecture
sequences
(videos and provides access to
indicate
lecture
videos
andnavigation
questions.
questions),
lab,
and
tutorials.
The
header
time betweentime
opening
and
answering
questions), homework, lab, and tutorials. The header navigation provides access to
between
opening
and
answering
partially thwarted
who open
supplementary
materials,
including
digital discussion
textbook,forums,
discussion
supplementary materials,
including
digital textbook,
and wiki.forums, and wiki.
). by21).users
a problem asa problem-solving
time21time
problem as problem-solving
TheThe
main
frame
represents
thelecture
first sequence;
lecture sequence;
beige
main
frameedX
represents
the first
beige boxes below
theboxes
header below the header
multiple
browser
windows
or
tabs;
Our time-accumulation
algorithm
is
Our time-accumulation
algorithm
is
indicate
lecture
videos
and
questions.
indicate
lecture
videos
and questions.
are
considering
ways
to
acpartially developers
thwarted by users
who
open
partially thwarted
by
userswindows
who or
open
multiple count
browserfor
tabs;
the edX
future.
multiple browser windows or this
tabs;inedX
• Can we identify features in the content of the posts that indi21,744
154,763
achieved that badge and axis-align their activity profiles by letting
References
Coursera
course
edX Course
6 the user’s optimization problem when there are multiple
can solve
cate which students are likely to continue in the course and
“day 0”FORUM
denote the day ACTIVITY
they receive the badge. To eliminate pos4. COURSE
badges that all target the same dimension.
sible
population
effects,
we
restrict
the
set
of
users
to
those
who
4
which
are
likely
to ,leave?
Title
Cryptography
I
Circuits
and
Electronics
(6.002x)
[1] K. Veeramachaneni,
F. Dernoncourt,
C.
Taylor,
Z.
Pardos,
and U.-M O’Re
=
(k
1)}
for
j
∈
{1,
.
.
.
,
m}
be
a
Let
B
=
{b
Table 1: Courses overview
We now move
to our
second
of they
the win
paper,
the
wereon
active
at least
60 daysmain
beforefocus
and after
the badge.
maximi
standards
for
mooc
data
science.
1st
workshop
on
Massive
Open
Online
C
,
and
assume
that
set
of
m
badges
all
on
the
same
action
A
Figure
3
shows
user
activity
in
the
days
surrounding
the
award2
forums, whichPiotr
provide aMitros
mechanism for students to interact with
The
course
forums
contain
many
threads (If
of knon-trivial
length,
Instructors
Dan Boneh
Anant Agarwal, Gerald Sussman,
<
k
<
.
.
.
<
k
without
loss
of
generality.
=
k
k
ing of the Electorate and Civic Duty badges. Notice how activeach other. Because
Coursera’s forums are cleanly separated from [2] K.for
and
we0 ask
threadsthese
are
long
because
small
set of
Veeramachaneni,
F Dernoncourt,
Taylor,
S. Halawa
some
j = whether
j , then U.-M
wethese
can O’Reilly,
consider
two badges
as aaC.
sinsubject
ity on the targeted actions (Q-votes for Electorate, Q-votes and Aallows researchers to refine the scripts. This framework is called MOOCViz. For its current appearance
−60
−40
−20
0 data
40 individual
60
University
Stanford
University
MIT
the
course
materials,
students
can
choose
to
consume
the
course
gle
badge
with
value
equal
to the
sum20science.
oftimes
their
people
are
each
contributing
many
to
a
longvalues.)
conversation,
dards
and
systems
for
mooc
MOOCDB
Technical or
Report, C
votes
for
Civic
Duty)
increases
substantially
before
users
achieve
see Figure 6. MOOCViz is currently available at http://moocviz.csail.mit.edu/.
Number
of days
relativea to
badge
win of students are each
the badge,ofand
almost
immediately
returns
near-baseline
content independently
thethen
other
students,
or they
cantoalso
comwhether they
are long
because
large
number
Length
14 weeks
MOOCViz, when completed,6
willweeks
allow analytic and visualization scripts for MOOC data analyses
to be municate with their
levels. peers.
Also notice that most of the other site actions are not adcontributing
roughly
once.
Figure 3: Number
of actions
per day as a function of number of
versely affected—the rates of these actions remain relatively stashared, under the understanding that the data being analyzed is organized according to the MOOCDB
days
relative
the
time
of
obtaining
a badge.
Notice
steering
in mean numFollowing
our
classification
of
students
into
engagement
styles,
As
a
first
way
to
address
this
question,
we
study
the
ble
over
time.
Since
the
four
actions
shown
in
the
Figure
are
the
Platform
Coursera
edX our first question is a simple one: which types of students visit
data model. MOOCViz activity
entails developing a set of templates and support for software sharing
the sense
of increased
activity oninactions
targeted
by the badge.
ber
of distinct
contributors
a thread
of length
k, as a function of
main activities on the site, this means users increase their overall
plus developing a means of sharing MOOC analytics results through web-based galleries.
6
activity
level this,
on thewe
site compute
in the days the
leading
up to achieving
these
the forums? To
answer
distribution
of enk. The
If this
is close
to vector
k, it means
thatgradient
many from
students
are conStart date
Jan 13th, 2013
March 5, 2012
firstnumber
salient feature
of the
field is the
badges.
Thepopulation
one exceptionofis students
the A-votewho
curve read
in theatElectorate
hot to cold if
as ittheis angle
departing
origin varies
between
the
for the
least
tributing;
a constant
or the
a slowly
growing
function
of k, then
Initial design of feature foundry: Another web based platform currently under development will allow gagement styles
badge, which drops in the days leading up to the badge boundary.
two
extremes.
The
fact that theare
color
stays the same
along a given
Registrants
21,744
154,763
the community to propose variables
of interest to be extracted from the data. It is targeted to enable
the one thread on This
ML3is evidence
(shown that
in users
the top
row
of
Table
3).
The
repa
smaller
set
of
students
contributing
repeatedly
to the thread.
are steering their behavior from A-votes
direction starting from the origin is a validation of our modeling
Registrants
Week 5
Week 6
Week 7
New
Video Types
Theory 2013
Hands−On 2013
Theory 2014
Hands-On 2014
Number of Hits
0
200
400
600
800
64
Lab
Discussion
Lab
Lecture Video
Lab
Discussion
Lecture Question
Book
Lecture Video
Book
Lecture Video
Book
Tutorial
Lecture Question
Lecture Question
Wiki
Wiki
Wiki
Tutorial
Tutorial
c o mmu ni c At i o nS o f t he Ac m
| A P R il 20 1 4 | vOl . 5 7 | N O. 4
1
A-votes
j
j
xa
1
2
m
j
j