USER BEHAVIOUR IN A VIRTUAL LEARNING MANAGEMENT
SYSTEM, AND THE LESSONS TO BE LEARNT
Tay Kwan Ann, Jeffery
National University of Singapore (SINGAPORE)
cittka@nus.edu.sg
Abstract
With increased focus on end user experience, it is extremely important to understand their usage
behaviour and tune our systems to enhance their experience. Capturing usage information not only
helps us understand their behaviour, it benefits other areas as well such as audits, investigation and
capacity sizing.
In 2009, NUS built a user behaviour monitoring module into its own home grown Virtual Learning
Management System (LMS), which allowed us to track the user's movement within the LMS and
understand their usage pattern.
The results gathered busted a fair number of assumptions which we had over students' behaviour. and
when we shared this information with other Institutes of Higher Learning, they were surprised as well.
The paper will showcase the expected as well as unexpected findings from this behaviour module as
well as how we adapted our LMS to be more student centric.
Keywords: Learning management system, user behaviour
1
INTRODUCTION
When the National University of Singapore’s (NUS) home-grown learning management system (LMS)
– Integrated Virtual Learning Environment (IVLE) – was developed in 1998, functionality took
precedence over form. This was de facto for several iterations of IVLE for several years.
In 2008 with the increased adoption of Web 2.0, it seemed we needed to update the IVLE interface
and make it more current by incorporating Web 2.0 features such as re-sizable screens, drag-anddrop menus, menu customisation, RSS and selected collaborative features. This new IVLE Web 2.0
interface developed by the NUS Centre for Instructional Technology (CIT) was launched in July 2008.
After the launch, we began to receive feedback that the old ‘Web 1.0’ interface where everything was
static was better and easier to use. This prompted us to rethink our strategy of changing the IVLE
interface to support Web 2.0 features.
From past experience, CIT found that online surveys usually do not tell the true story, so we decided
to build a user behaviour monitoring module into IVLE to track the user’s movement within the LMS.
This module has several measurement metrics which take into account date and time information for
added correlation capabilities.
2
WHY USER BEHAVIOR INSTEAD OF USER VOICE
CIT felt that instead of basing our requirements on the voice of the students and lecturers, it may be
more practical to gain insight from their behaviour in IVLE and find ways to improve their experience
with IVLE. Additionally users, who gave their time to provide good feedback, usually represent the
minority. There are also the additional benefits which these user logs bring such as auditing, security
investigation and capacity sizing.
Student Behaviour Assumptions From Students’ Voice
2.1
With the experience gathered over the years from speaking to students and from their feedback, CIT
made the following assumptions about the student’s behaviour within IVLE. These assumptions
provide us with the basis to decide if certain features are worthwhile to develop for students.
1. Students prefer to quickly get their information from IVLE and get out as soon as possible.
We estimate students would spend about 20 minutes on IVLE
2. Students tend to access IVLE between 10pm to 2am.
3. Having an eLearning Week generates substantially more load on the system due to the huge
number of students accessing at the same time.
4. Students tend to access IVLE from home or outside of campus rather than from within campus
especially after office hours.
2.2
User Behaviour Measurement Metrics
CIT came up with 7 metrics to measure student behaviour. It is important to note that by themselves
the metrics do not reveal much. More data can be mined by cross-referencing the same metrics during
different time periods, or cross-referencing different metrics in the same time period. E.g. Total
number of active users per hour and Total number of user activities per hour gives us an estimation of
the number of activities each active user is performing during this hour, which in turn translates to
productivity. The higher the number of activities, the more productive they generally are.
Additionally, by correlating these metrics with the academic calendar, we are able to gain more insight
into the behaviour of our users e.g. what is their behaviour when school starts or during exams
The metrics are:
2.2.1
Total number of user logins per hour
This metric measures the total number of successful user logins for each hour of the day. Multiple
successful logins by the same user will be counted as such. E.g. If only user1 logged in successfully 4
times between 2pm to 3pm, the log will show 4 logins between 2pm to 3pm.
The main purpose of this metric is to give an indication of peak login periods along with their origin,
whether it is within the intranet or via the internet.
2.2.2
Total number of user activities per hour
This metric measures the total number of user activities for each hour of the day. A user activity is
defined as a single action performed by the user. E.g. user1 views the forum and decides to post a
thread, this is considered as 2 user activities for the time period.
The main purpose of this metric is to give an indication of the total number of user activities handled
by the system along with their origin (intranet or internet).
2.2.3
Total number of active users per hour
This metric measures the total number of distinct active users for each hour of the day. An active user
is defined as a user who is currently performing an activity on IVLE. E.g. if user1 is viewing the forum
between 2pm to 3pm, user1 is considered an active user.
The main purpose of this metric is to give an indication of the total number of active users.
2.2.4
Average number of activities per user session
This metric measures the average number of activities per user session per day. The higher the value,
the more activities each user does per session.
2.2.5
Average duration per user session
This metric measures the average duration of each user session per day. The higher the value, the
longer they stay within IVLE and use the available tools.
2.2.6
Daily tools usage
This metric measures the usage of the various tools available within IVLE on a daily basis. It provides
us with a more detailed breakdown of the user activity so that we can tell which tool generated the
most activity during the time period.
3
FINDINGS
From the results gathered from the various metrics, it is clear that some assumptions which we held
were no longer valid. Also certain interesting snippets of user behaviour surfaced when looking at the
various charts.
3.1
Students Prefer To Quickly Get Their Information From IVLE But They Do
Not Get Out As Soon As Possible
Previously, it was thought that students try to spend as short a time within IVLE as possible. They will
login to IVLE, perform a few actions e.g. view the forum, download files and then logout. However
from the average duration per user session metric (Figure 1), we realized that on a monthly basis the
average duration per session is about 55 minutes during school term and 30 minutes during vacation
periods. This is slightly more pronounced in December, having an average session duration of 27
minutes.
Cross-referencing this with the average number of activities per user session (Figure 2) metric reveals
that the session duration is independent of the number of activities performed. From April 2010 to
August 2010, even though the number of activities is high, the average session duration fluctuates.
During September 2010 to November 2010, even though the session duration is high, the number of
activities per session is low.
We reason that with the trend of keeping both the internet and personal computer on for longer
periods, students develop the habit of logging onto a site and leaving it on even if they are not using it.
Figure 1 – Average duration per session
Figure 2 – Average activities per session
3.2
Sunday Is The Day When Students Stay The Longest
Taking the average duration per user session metric based on the day of the week revealed that
Sunday is the day when students stay logged onto IVLE the longest (Figure 3). There are minimal
differences between the number of activities done during the various days of the week (Figure 4).
This can be attributed to students wanting to prepare for the coming week so as to maximise their
learning experience. Alternatively, it could be that they are catching up on the previous week’s work
and getting it done before it is due for submission.
Figure 3 – Average duration per session
3.3
Figure 4 – Average activities per session
User Activity Drops As The Term Progresses
Grouping the average number of activities per user session (Figure 5 and Figure 6) by month shows
an interesting picture. The number of user activities drop significantly as the term progresses, by as
much as 50% between the months of August to September.
This can be attributed to students getting used to the system in the initial phase and once they are
used to the system, they are more efficient in getting what they want, resulting in a lower activity rate.
E
L
W
Figure 5 - Average activities per session
3.4
E
L
W
Figure 6 - Average activities per session
ELearning Week Does Not Have Significant Impact On Usage
NUS conducted several eLearning weeks. During the eLearning week, students are supposed to learn
for 1 week using purely online tools. All lectures, tutorials and seminars will be done via online tools.
One of the main applications used during eLearning week is IVLE.
Tool such as assessment, chatroom, forum, media and workbins were used to replace traditional
teaching methods. A total of 260 lectures, 233 seminars and 1049 tutorials were replaced with an
online equivalent for the last eLearning week done from 4th to 8th October 2010. Lecturers were given
free rein on how they want to conduct their lessons. They could for example use the online chatroom
to replace a tutorial session, or use the forum for a more formal replacement for the same tutorial
session. They can also use desktop capturing software to record their lecturers and distribute it via the
IVLE media and the accompanying lecture notes via the workbin.
The eLearning weeks were conducted on
1. 5 to 10 October 2009 for the entire university
2. 8 to 13 March 2010 for the faculty of business, and
3. 4 to 8 October 2010 for the faculty of arts and social sciences.
The eLearning weeks are marked in figure 5 and 6, and we can see that there is no significant
difference between IVLE usage during eLearning week and non e-Learning week periods. From both
charts, the average number of activities done during the month where eLearning week falls (October)
in is not much higher as compared to the non-eLearning week months (September and November)
3.5
Peak Access
Consolidating the Total Number of User
Activities Per Hour (Figure 7) metric, it seems
our students generally access IVLE after 8pm
daily. Additionally, with the exception of
Sunday, they are also active on IVLE between
11am to 5pm. This access stems primarily
within campus and can be attributed to them
accessing IVLE during school hours.
Figure 7 also indicates the type of access,
access within the intranet is marked yellow
and access from the internet is marked blue.
We can see that during school hours, the
activities done are more or less distributed
equally between intranet and internet access.
After school hours, the activities are done
mostly thru the internet i.e outside of campus
3.6
Figure 7 - Peak access by hour
Workbin Is The Most Accessed Tool
From a grouping of the Daily Tools Usage (Figure 8) metric across the whole time period, the workbin
tool is the most heavily used tool amongst all the tools available within IVLE. This is because the
workbin is the main tool used to distribute notes, assignments and projects to students. It is also the
tool for students to submit their assignments and project documents back to the lecturer.
Figure 8 - Consolidated tool usage
3.7
Lesson Plan Is Consistently The Most Accessed Tool Before School
Starts
From the Daily Tools Usage metric, we found out that just before the academic year starts, the Lesson
Plan tool will be the most heavily used. The Lesson Plan contains information about the module’s
lessons during the academic year.
Figure 10 - Tools usage on 18th July 2010
Figure 9 - Tools usage on 5th January 2010
Figure 11 - Tools usage on 1st January 2011
4
ADAPTATIONS TO IVLE
From the various findings highlighted in the previous section, the following actions were taken to make
IVLE more student centric.
4.1
Web 2.0 Interface On Hold
As the number of users wanting a Web 2.0 interface is small, and users are already used to the IVLE
interface and experience, we decided to put the Web 2.0 interface redesign on hold. Additionally, the
Web 2.0 prototype did not garner much interest. However we employed the development technology’s
native Ajax capabilities to build some Ajax functionalities into IVLE.
4.2
Desktop Client For Workbin submissions
Considering that the workbin tool is the most popular tool used within IVLE, CIT worked on a desktop
application which is able to upload files directly to workbin. This is especially useful for students when
they need to submit their assignments to their lecturers via the workbin tool. We are also in the
progress of enhancing this tool so that students can use this application to download the workbin files
as well.
4.3
“Live” Update of New Announcements and Friends Messages
Given that the students are staying within IVLE longer, CIT implemented a live update of new
announcement and messages from friends. With this feature, whenever there is a new announcement
from the modules that the student is taking, or when there is a message from their friend, a small
dialog will appear indicating the number of new announcements and messages from friends.
Moving the mouse over this will show a balloon tip with a worded description of the dialog as shown
below.
4.4
System Procedures
Bearing in mind the usage patterns, CIT changed our system management procedures to minimize
any downtime during high usage periods. These include Sundays, after 8pm on weekdays, and just
before school starts. This effectively reduces our downtime to the university’s vacation.
5
CONCLUSION
From the findings, we can conclude that Web 2.0 interactivity within a LMS is a good to have, however
whether or not it is implemented does not significantly affect the users’ usage of the LMS. It seems
what the student need are ways to be more productive within the LMS, be it in terms of tools which
reduces their dependency on the LMS or the LMS as a service being there for them when they need it
and is always responsive to their requests.