1.
Learning Analytics – an overview
For CAUL Learning and Teaching Advisory Committee
By Ralph Kiel
2012
Introduction
In the last year or so Learning Analytics has been raised as a topic at CAUL meetings
on a number of occasions and has been identified as a topic of interest by the CAUL
Learning and Teaching Advisory Committee. Both EDUCAUSE in its Top-Ten IT Issues:
2012 and NMC along with Griffith University in the Technology Outlook: Australian
Tertiary Education 2012-2017 have identified Learning Analytics as being one of the
major issues to watch, the former at number 6 and the latter number 3.
The aim of this brief report is to give as simple and clear an overview of this complex
issue as possible and to try to identify ways in which learning analytics are of use to
Australian University Libraries. For beginners a good place to start is the EDUCAUSE
paper: 7 things you should know about analytics which can be found on their website at:
http://net.educause.edu/ir/library/pdf/ELI7059.pdf. This paper defines analytics in a
higher education context and identifies how it works, why it is significant, the implications
to teaching and learning, legal and ethical issues and the future of analytics in higher
education.
van Barnegeld et al state that there is confusion in the meaning and understanding of
what analytics means especially in higher education - “the analytics field is now at a
point where clarification and consensus of terms is merited” and in their paper set out a
conceptual framework to establish a common terminology and application in the field of
analytics in the academy” (van Barnegeld 2012).
EDUCAUSE has a working definition for analytics: “Analytics is the use of data,
statistical analysis, and explanatory and predictive models to gain insights and act on
complex issues”. (Bichsel 2012). Bichsel then goes onto say that analytics is a process
that is more than just metrics. This process starts with a strategic question, finds and
collects the data to answer that question, analyses the data “with an eye towards
prediction and insight”, presents the findings and feeds back into the process of
addressing the strategic question and creating new ones. She makes it clear that when it
comes to analytics, things have changed in recent years as data are more extensive and
automated and the process used to get the information is becoming more repeatable.
Also, a broader range of programs have been established to apply the data, analyse it
and then make decisions and develop plans.
In the next section some of the benefits of learning analytics are laid out. But are the
costs in using them worth the effort? The benefits of using analytics can be costed as
described in Campbell and Oblinger: “Retention of students saves institutions the cost of
recruiting students to replace those who withdraw without completing a degree.”
(Campbell and Oblinger, 2007)
2.
Learning / Academic Analytics
Universities in this competitive and accountable age are “ . . adopting practices to
ensure organizational success at all levels by addressing questions about retention,
admissions, fundraising and operational efficiency.” (van Barneveld 2012). Learning
analytical tools can provide statistical evaluation of institutional data sources to
determine patterns that can inform decisions about student recruitment, course offerings
etc. In learning and teaching, Learning Management System (LMS) data can be used to
identify students at risk of not succeeding academically. Data can also be mined to show
student interaction with peers, course content, and teachers, and then measuring and
draw conclusions about these interactions.
From this, reports can be developed that reveal patterns, trends and exceptions.
Analytics can indicate course strengths and weaknesses, align resources with student
need and track student progress. Algorithms which perform calculations to determine a
particular outcome can be used in this analysis and can also set off a trigger sending a
note to a teacher indicating a student is at risk of failure. In this case the goal would be
to improve student performance and to show the accountability of the University.
Legal and ethical issues around privacy, security and ownership involving students and
teachers have been raised especially whether such profiling is intrusive. However it
could be argued that not identifying such knowledge of student underachieving could be
regarded as irresponsible when the teacher or the University has this data at hand. Two
other potential shortcomings could be that algorithms may come to false conclusions
and send the wrong message about a student, or that a student or teacher could
misinterpret what the data is telling them and come to false conclusions. However in
both these cases, as Universities get better at this activity these potential problems are
likely to be ironed out and learning analytics could become standard practice.
van Barneveld et al set out two types of analytics for Universities within an overall
framework based on the Scholarship of Teaching and Learning (SoTL): academic
analytics and learning analytics, and state that analytics can “supplement the
established theory and practice of the field.” (van barneveld 2012)
The Horizon Report defines learning analytics as follows:
“Learning analytics loosely joins a variety of data gathering tools and
analytic techniques to study student engagement, performance, and
progress in practice, with the goal of using what is learned to revise
curricula, teaching, and assessment in real time. Building on the kinds of
information generated by Google Analytics and other similar tools,
learning analytics aims to mobilize the power of data-mining tools in the
service of learning, and embrace the complexity, diversity, and abundance
of information that dynamic learning environments can generate.”
The report goes onto say that learning analytics has generated a lot of attention over the
last few years and is already enjoying growing interest within higher education but that
there is a two to three year horizon over which they expect to see the beginning of
widespread adoptions of learning analytics. The benefit of learning analytics is that it will
enable teachers and universities to tailor educational opportunities to each student’s
level of need.
The Horizon report goes onto say that
“Data are collected from explicit student actions, such as completing
assignments and taking exams, and from tacit actions, including online
social interactions, extracurricular activities, posts on discussion forums,
and other activities that are not directly assessed as part of the student’s
educational progress. …. Learning analytics promises to harness the
power of advances in data mining, interpretation, and modeling to improve
understandings of teaching and learning, and to tailor education to
individual students more effectively.” (Horizon Report 2012)
The bottom line is that academic analytics is all about improving “... one of higher
education’s most important challenges: student success” especially measured through
student retention and graduation. Obtaining data from sources such as a LMS and
student information systems and by using predictive modelling to identify students at risk
and then intervening, analytical tools can be used to improve student retention and
graduation. (Campbell and Oblinger 2007)
This simple diagram from an article by Siemens depicts the process of learning
analytics; however a more complex diagram is also presented in this article. (Siemens,
2010)
Learning analytics is related to similar developments in other areas such as Big Data, escience, web analytics, and educational data mining, with the main, common element
being they rely on large data collections. In all of the above a tool can be used to help
make sense of the data called a dashboard or in this case a ‘learner dashboard’. “For
learners and teachers alike, it can be extremely useful to have a visual overview of their
activities and how they relate to those of their peers or other actors in the learning
experience.” (Duval)
3. Analytics and Australian Universities
Social Networks Adapting Pedagogical Practices (SNAPP) tool developed by the
University of Wollongong which is highlighted in the 2012 Horizon Report uses visual
analysis to show how students interact with discussion forums within commercial and
open source Learning Management Systems. As a diagnostic it allows teachers to
evaluate student patterns of behaviour against learning objectives. Interaction data that
is stored within the discussion forum is interpreted by SNAPP by inferring relationships
from post reply data and develops a social network visualisation.
“The social network diagrams can be used to identify:
1. isolated students
2. facilitator-centric network patterns where a tutor or academic is
central to the network with little interaction occurring between
student participants
3. group malfunction
4. users that bridge smaller clustered networks and serve as
information brokers”
(SNAPP 2012)
At Victoria University the use of data in this way is at an embryonic stage. The annual
course monitoring process uses a limited range of data for trend analysis and faculties
are able to use this data for their own local analysis. However the use of this data is only
partly automated. A project called ‘Flag and Follow’ manually tapped into enrolment data
to determine which students were at risk before classes commenced in 2010. The aim is
to develop a university wide inter-departmental model that is evidence based to inform
retention initiatives. (Brown 2011?) A number of ‘at risk’ factors such as ATAR score,
socio-economic status and distance to campus were used to ‘flag’ students at potential
risk and then these students were ‘followed’ and interventions developed to support their
learning. At risk students were also identified once classes had started by criteria such
as absences from class and failure to hand in the first assignment. Preliminary findings
to date suggest the flagging criteria were successful and that interventions especially in
person as opposed to SMS or telephone were successful.
4.
Analytics and University Libraries
Libraries produce a lot of data through Library Management systems. However results of
a survey conducted by EDUCAUSE, showed that although Libraries are regarded as
one of the top 17 areas in which data is being collected, they are one of the most
laggard areas in using this data to make predictions or take action. In the same survey it
also showed that Libraries were regarded as having one of the lower levels of potential
in using the data and being able to make predictions and take action. (Bichsel 2012)
By having the Library system talk to the University systems, the University of
Wollongong Library with its Library Cube project has used analytics to demonstrate the
contribution the Library is making to student learning and teaching. A correlation
between Library loans gathered from data in the Library Management system and
student achievement in grades from the University systems was shown. That is, the
more a student borrows library books the more likely the student is to achieve at a high
level in academic scores. (Jantti 2011)
This is groundbreaking research and it was cited in an ECAL publication as being a
distinctive example of the use of analytics in an area outside of student recruitment and
retention. (Bichsel 2012) The case could be put that in Australia at least where
University Libraries operate in consistent way and where the measures of library
performance are relatively close, that this outcome would hold for all CAUL Libraries.
This research is important for CAUL Libraries because it can be used to provide
accountability, can be used in marketing of library services to the rest of the University
and can be compared overtime as a quality measure.
References
7 things you should know about analytics, EDUCAUSE, 2012 downloaded 12 April 2012
http://net.educause.edu/ir/library/pdf/ELI7059.pdf
van Barneveld, Angela, Arnold, Kimberly E and Campbell, John P. Analytics in Higher
Education: Establishing a Common Language, EDUCAUSE, 2012, downloaded 13 April
2012 http://www.educause.edu/Resources/AnalyticsinHigherEducationEsta/245405
Bichsel, Jacqueline. Analytics in Higher Education: Benefits, Barriers, Progress and
Recommendations (Research Report). Louisville, CO: EDUCAUSE Center for Applied
Research, August 2012, downloaded 29 August 2012
http://www.educause.edu/library/resources/2012-ecar-study-analytics-higher-education
Cox, Brian and Jantti,Margie. Capturing Business Intelligence Required for Targeted
Marketing, Demonstrating Value, and Driving Process Improvement, 2012, downloaded
13 April 2012
http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1248&context=asdpapers&seiredir=1&referer=http%3A%2F%2Fwww.google.com.au%2Fsearch%3Fsourceid%3Dnav
client%26ie%3DUTF8%26rlz%3D1T4SKPB_enAU383AU384%26q%3Dlibrary%2Bcube%2Bwollongong#sea
rch=%22library%20cube%20wollongong%22
Jantti, Margie H and Cox,Brian. Measuring the value of library resources and student
academic performance through relational datasets, 2011, downloaded from the Internet
13 April 2012
http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1120&context=asdpapers&seiredir=1&referer=http%3A%2F%2Fwww.google.com.au%2Fsearch%3Fsourceid%3Dnav
client%26ie%3DUTF8%26rlz%3D1T4SKPB_enAU383AU384%26q%3Dlibrary%2Bcube%2Bwollongong#sea
rch=%22library%20cube%20wollongong%22
Brown, Claire and Evagelistis, Yolanda. Flag and Follow: a whole of institution approach
to early identification and intervention for students potentially at risk, 2011?, downloaded
from the Internet 26 April 2012
http://www.fyhe.com.au/past_papers/papers11/FYHE-2011/content/pdf/7A.pdf
Campbell, John P. and Oblinger, Diana G. Academic Analytics, EDUCAUSE, October
2007, downloaded from the Internet 9 May 2012
http://net.educause.edu/ir/library/pdf/PUB6101.pdf
Duval, Erik. Attention Please! Learning Analytics for Visualization and Recommendation,
LAK’11:
Proceedings of the 1st International Conference on Learning Analytics and Knowledge,
2011
Horizon Report: 2012 Higher Education Edition, NMC, 2012. downloaded 15 May 2012
http://net.educause.edu/ir/library/pdf/HR2012.pdf
Grajek, Susan, the 2011–2012 EDUCAUSE IT Issues Panel, and Judith A. Pirani. TopTen IT Issues, 2012, EDUCAUSE.
http://www.educause.edu/EDUCAUSE+Review/EDUCAUSEReviewMagazineVolume47/
TopTenITIssues2012/250724
The New Media Consortium (NMC) and Griffith University, 2012, Technology Outlook:
Australian Tertiary Education 2012-2017
http://www.nmc.org/publications/2012-technology-outlook-au
SNAPP: Social Networks Adapting Pedagogical Practice, downloaded 15 May 2012
http://www.snappvis.org/
Siemens, George, What are Learning Analytics? eLearnspace, 25 August 2010.
downloaded 15 May 2012
http://www.elearnspace.org/blog/2010/08/25/what-are-learning-analytics/