Using CMS Data as a Force for Good?
Applying Academic Analytics to Teaching
and Learning
Leah P. Macfadyen
Science Centre for Learning and Teaching, UBC, Canada
Shane Dawson
Queensland University of Technology/University of Wollongong,
Australia
Project Foundations
1. Emergence of “Academic Analytics”
2. Increased use of ICTs in teaching and learning
3. Increasing availability and detail of Course
Management System data
4. Increasinging interest in socio-constructivist
learning theory (and its implications)
Teaching and Learning for Engagement
• Socio-constructivist theories of learning
• Importance of engagement and learning
communities
• Increasing use of ICTs
Questions
i. Which web-based tools and activities can promote
student engagement and community online?
ii. How do engagement and sense of community
correlate with student achievement?
CMS data
• CMS usage is now prevalent
(US data 2006: 93% student adoption in average of 2.5
courses; UBC data: >25,000 student users of Bb Vista)
• CMS data is immediate
(can be mined at any time)
• CMS data is non-intrusive
(does not require faculty intervention)
(Bart Collins, Purdue University, 2006)
Bb Vista Tracking Data
Data points available to instructors
Date of first access
# Chat sessions
Date of last access
# Assessments begun
Total # sessions
# Assessments finished
Total time online
Time on Assessments
# Mail messages read
# Assignments read
# Mail messages sent
# Assignments submitted
# Artifacts created
Time on Assignments
# Artifacts saved
# Goals viewed
# Discussion messages read
# Weblinks viewed
# Discussion messages posted
# Content folders viewed
# Viewed Calendar entries
# Files viewed
# Added Calendar entries
# Media library entries viewed
# Media library collections viewed
Project goals
Develop a data interpretation and visualization tool to:
• aid faculty and students in the interpretation of the
vast array of data currently captured by Bb Vista
• permit ongoing formative evaluation of student
engagement in learning activities and allow early
identification of at risk students
• provide administrators and institutions with
benchmarks of activity, usage trends, disciplinary
differences
Case Study: BIOL200
Descriptive Data
BIOL 200 online,
2008
BIOL200, 2007
web-supported
N (completers)
119
1112
Average final grade
60%
65%
Average online sessions/
student
153
77
Average hours
102
41
4589
643
Average messages posted
72
2
Average # files viewed
826
423
online/stud
ent
Average discussion
messages read/student
Summary of correlations (bivariate)
Variable
BIOL200
online
BIOL200 web-supported
R
R2
R
R2
# Disc. messages
posted
.522
.272
.206
.042
# Files viewed
.332
.110
.244
.060
# Online sessions
.402
.162
.231
.053
Total time online
.338
.114
.226
.051
# Assignments
submitted
.256
.066
n/a
n/a
# web links viewed
.251
.063
ns
ns
# Mail messages sent
.282
.080
ns
ns
# Disc. messages read
.290
.084
.183
.033
# Assessments finished
.310
.096
.290
.084
Bivariate Correlations
Categories of variables:
1. Measures of effort
time online, number of sessions online, time on specific activities
2. Engagement and community activities
discussion forums, chat
3. Administrative activities
mail, calendar, announcements, tracking, grades
4. Content-related activities
files, folders, media
5. Assessment activities
assignments, assessments
Predictive modelling
BIOL200 online multiple regression (with variables for tools used)
R
R2
F change
Sig. F change
.660
.435
4.503
.000
BIOL200 web-supported multiple regression (with variables for tools used):
R
R2
F change
Sig. F change
.316
.100
10.119
.000
(Compare to: Morris, Finnegan & Wu (2005): R2 = .310 for online courses)
Discussion participation….
Visualizing student engagement
Instructor
http://www.randomsyntax.com/blackboard-forum-social-network-analysis/
Disconnected students
Instructor
Percentage of total interactions
Institutional tool use
27 Aug 2007
06 Jan 2008
Lessons learned so far…
• Some (but not all) CMS data variables are useful
predictors of eventual student achievement
• Several seem to support theoretical propositions
regarding the importance of community in learning
• Correlation ≠ causality…
• Significance of variables depends on course design