Learning Management System Insights from
the 2016 Faculty Technology Survey
Overview
UNM conducts annual surveys of students and faculty to gauge technology trends/needs and
satisfaction with technology-related services provided by various units at UNM. This document
examines the responses to those questions to inform a decision about whether to proceed with an
upgrade or replacement of UNM Learn.
Survey Specifics
The survey was conducted online during the period March 11 through April 8. 1828 full-time faculty
from main and branch campuses were asked to complete the survey. 387 faculty responded to the
survey (21% response rate) with 328 completing all questions asked. The 2016 response rate is typical
for annual faculty technology surveys. Faculty email invites included web links specific to each faculty
member which will enable survey responses to be cross-referenced to Banner-related data such as
faculty rank, academic unit, and hire date.
The 2016 faculty survey included 27 questions in the following groups:



10 questions - instructional use of technology other than LMSs and related UNM services
5 questions - satisfaction with IT services and use of cloud, email, and calendaring services
12 questions - experience with LMSs including both UNM Learn and other products/services
LMS-Related Questions and Responses
For purposes of this survey, an LMS was operationally defined as follows:
A learning management system (LMS) is a web application that provides an instructor
with a way to create and deliver educational content/experiences, support and monitor
student participation, post and receive student assignments.
LMS Insights - Faculty Tech Survey
Draft - 7/17/2016
Page 1 of 8
LMS-Related Questions
The 12 LMS-related questions are summarized here (numbered 16-27 as they were in the survey):
16. I use an LMS for class sizes of _____ students. (0-9, 10-24, 25-49, 50-99, 100-199, 200+)
17. Please indicate your level of agreement with the following statements about the LMS: The LMS
a. is critical to my teaching
b. is very useful as a tool to enhance my teaching
c. is very useful as a tool to enhance student learning
d. creates efficiencies for my instruction
18. How many years of experience do you have using LMS technology? (0-1, 2, 3, 4, 5-10, 11+)
19. Do you currently use UNM Learn? (Yes, No)
Q19 NO Branch
20. What factors influence your decision not to use UNM Learn (Check all that apply)?
a. instructor ease of use
b. student ease of use
c. ease of maintaining personal web sites
d. training availability
e. support quality
f. LMS tool usefulness
g. LMS features
h. LMS flexibility
i. LME reliability/stability
j. class size
k. lack of 3rd party content integration
l. prefer a different system
Q19 YES Branch
21. Describe how you use an LMS to support your instructional activities (Select all that apply).
a. distribute documents and learning materials to students
b. distribute video to students
c. web conferencing
d. gradebook
e. quiz//exam delivery
f. online assignment submission
g. graded online discussions
h. communication between instructor and student
i. student-to-student messaging
j. group collaboration
k. learning analytics
l. fully online class delivery
m. hybrid class delivery
n. flipped classroom
LMS Insights - Faculty Tech Survey
Draft - 7/17/2016
Page 2 of 8
22. Please indicate your satisfaction with the following aspects of UNM Learn:
a. ease of use in general
b. ease of use from a mobile device
c. supports meaningful interaction with students
d. system availability
e. system response time
f. initial use training
g. ongoing training and professional development
h. posting content
i. managing assignments
j. monitoring/management enrollment
k. entering grades and other progress information
l. reliable assignment receipt
m. integrating 3rd party content
n. integration with other UNM systems
o. overall satisfaction
23. Do you have experience with an LMS that you prefer over UNM Learn? (Yes, No)
Q23 YES Branch
24. What is your preferred LMS? (Angel, Canvas, Desire2Learn/Brightspace, Moodle, WebCT,
Faculty/locally-developed, Other)
25. Why is the above your preferred LMS (Check all that apply)?
a. instructor ease of use
b. student ease of use
c. usefulness of LMS tools
d. features
e. flexibility
f. reliability/stability
g. 3rd-party content integration
26. If my preferred LMS was available for use at UNM, I would invest time in moving my content
to that LMS from UNM Learn. (Yes, No)
27. Please provide additional comments comparing your experience with the other LMS to UNM
Learn.
LMS Insights - Faculty Tech Survey
Draft - 7/17/2016
Page 3 of 8
Survey Respondents
A few observations about respondent characteristics can be gleaned from the survey responses:



43% claim 5 or more years of LMS experience (Q18). Thus, it appears that survey
respondents are more likely than non-respondents to be users of technology in general and
the LMS in particular. If that characterization is true, the good news is that survey
respondents are probably well-informed. The bad news is that they probably don’t
represent the faculty as a whole.
287 respondents (approximately 77%) currently use UNM Learn. This is a fairly large sample
from which to gather responses to the detailed UNM Learn questions. However, there’s no
way to know how (un)representative of the faculty as a whole are those respondents.
53 respondents (approximately 16%) have experience with an LMS that they prefer to UNM
Learn. The sample is large enough to gather some useful information about comparisons
between UNM Learn and other LMSs. But it isn’t large enough to have high confidence in
the accuracy of those comparisons nor whether they would reflect the attitudes of the
entire faculty should UNM switch to a different LMS.
Faculty Use of UNM Learn
Q21 asked UNM Learn users how (which features) they use. The top responses in decreasing order of
frequency are:





Distributing documents and learning materials (95%)
Gradebook (74%)
Online assignment/paper submission (63%)
Quizzes/exams (52%)
Class-related communication between the instructor and students (50%)
The least frequent responses were web conferencing (8%) and student-to-student messaging (14%).
Use of group collaboration, graded discussions, and analytics were also relatively low (17-22%).
Note that Q21 was only asked of faculty members that currently use UNM Learn. Thus, it’s an open
question whether past or present users of other LMSs use those LMS in ways similar to the way other
faculty use UNM Learn.
Responses concerning class delivery modality (fully online, hybrid, and flipped) showed no clear pattern.
Fully online and hybrid had a nearly equal number of responses (15-17%). Flipped classroom was about
half as many responses (9%) but that isn’t surprising given the relative newness of that class delivery
modality. By omission, we might be able to assume that the respondents that didn’t choose any of the
other modalities use UNM learn to teach traditional face-to-face classes (up to 59%).
Q22 measured faculty satisfaction with UNM Learn feature-by-feature and overall. Overall,
approximately 59% of respondents that currently use UNM Learn were satisfied or very satisfied, 21%
were neutral, and 19% were dissatisfied or very dissatisfied. 53 respondents answered yes to Q23 (use
another LMS they prefer over UNM Learn). If those respondents are assumed to be less-than-satisfied
users of UNM Learn then the overall satisfaction for UNM Learn is somewhere between 49% and 59%.
Exactly where in this range is the correct value is unclear since there’s no way to know how many
LMS Insights - Faculty Tech Survey
Draft - 7/17/2016
Page 4 of 8
respondents that answered yes to Q23 also answered Q22 (i.e., they currently use UNM Learn and
currently use or previously used another LMS).
Faculty Non-Use of UNM Learn
88 respondents answered no to Q19 indicating that they do not currently use UNM Learn. Q20 asked
why and provided a set of specific responses and an open-ended response option (choose all that
apply). Of the specific responses, the most common in descending order of frequency were:






Ease of use for faculty (18.6%)
Ease of use for students (10%)
Usefulness (lack thereof) of LMS tools (10%)
Ease of maintaining personal websites (9%)
Availability of training (9%)
Flexibility of the system (9%)
Open-ended responses underscored the above responses with emphasis on poor ease-of-use, system
clunkiness and inflexibility, and the utility of other automated tools such as Dropbox, Google Classroom,
and traditional web sites for supporting instruction.
Comparing the latter point with the responses to Q21 and assuming that users of both UNM Learn and
other LMS use them in similar ways, it’s apparent that many faculty prefer other tools to support the
relatively small number of LMS functions that they employ (e.g., document distribution, gradebook, and
communication with students). A few of the open-ended responses to this question and to Q27 also
noted that utility of using tools such as Drobox that students already know and/or will use after
graduation. Ease-of-use also factors into the reasons to use other tools and LMSs as shown by the large
number of respondents that point out ease-of-use as either a reason not to use UNM Learn or to be
less-than-satisfied with UNM Learn.
Faculty Experience with Other LMSs
49 respondents described other LMSs with which they have experience. The top responses were:


Canvas (16)
Other (14)
Specific descriptions accompanying the “other” responses were all over the map. Other specific
alternatives such as Moodle and Desire2Learn had 5 or fewer responses. The sample sizes for the
various UNM learn alternatives are all too small to draw definitive comparisons to Blackboard or among
the alternatives.
Q25 asked why respondents preferred another LMS to UNM Learn. Similar to responses to earlier
questions, ease-of-use (lack thereof) was the most frequently cited reason. Other frequent responses
included LMS tool usefulness, flexibility, and reliability/stability.
Open-ended responses to Q27 underscored the pre-supplied response alternatives with many
anecdotes describing UNM Learn’s poor ease-of-use and other shortcomings and praising the relative
strengths of a chosen alternative.
LMS Insights - Faculty Tech Survey
Draft - 7/17/2016
Page 5 of 8
Finally Q26 asked if current users of other LMSs would invest the time to transfer content from UNM
Learn to their preferred LM. Of 46 respondents, 37 (80%) said yes.
Putting It All Together
The picture painted by the survey responses is complex. It’s provides neither a strong affirmation of
UNM Learn nor a strong mandate for change. As such, the survey can be mined for pieces of
information that will support one direction or another. But defending either choice will require further
effort to fill in the missing pieces. As such, analysis and discussion of the survey results is only a starting
point for deciding whether/how to improve, upgrade, or replace UNM Learn.
Some key takeaways from the survey include:


UNM Learn is underutilized
Ease-of-use, complexity, and training are problem areas
UNM Learn Under-Utilization
Some UNM faculty don’t use any LMS. Some use an LMS other than UNM Learn. Some faculty cobble
together LMS-like capabilities using tools such as web sites and GoogleDocs. Among faculty that do use
UNM Learn, many use only a handful of its functions.
Whether underutilization is a “problem” is a matter of perspective – faculty, student, or institutional.
Since faculty aren’t mandated to use UNM Learn and since their opinions of its utility vary widely, it’s
little surprise that there’s significant underutilization. Faculty teaching online and those teaching large
classes have obvious motivations to use the LMS though some of those faculty choose to use tools other
than UNM Learn. At the other end of the spectrum, some faculty simply see no need for UNM Learn.
For faculty teaching classes such as those with low enrollments, infrequent assignments, and minimal
document distribution, relatively small benefits (e.g., 7-24 access to documents) accrue only to the
students with the costs of learning to use UNM Learn and maintaining course-related content being
borne by faculty and support staff. Given the number of faculty avoiding UNM Learn or complaining
about ease-of-use, it’s reasonably clear that a significant number of faculty members don’t value the
perceived benefits to themselves or student more highly than the costs that they themselves incur.
The faculty survey doesn’t illuminate the student perspective though a separate student survey with
LMS-related questions to be completed in May will partly fill that gap. There are some obvious benefits
to students from a well-used LMS including accessibility of documents and learning materials, access to
grades, online assignment submission and feedback, collaboration with instructors and peers, and more
efficient learning. But these benefits are only realized if instructors make good and full use of its
features. As the survey results show, many faculty use the LMS as a combination of a static web site,
assignment drop box, and Internet-accessible grade book. Features such as calendar integration, online
quizzes/exams, modularized and programmed learning, and learning and web site analytics are seldom
used. Yet it is these features that LMS-related research points to as having the most significant impact
on learning outcomes.
Another student perspective is that of tool overload. To the extent that faculty use different LMSs,
varying tool sets in place of an LMS, and use UNM learn in widely different ways, students face a
relearning process with each course they take. They must know how to use multiple LMSs/tools and
LMS Insights - Faculty Tech Survey
Draft - 7/17/2016
Page 6 of 8
must adjust to the myriad ways in which LMS/tool use varies from class to class. Some students
negotiate this maze easily but others struggle. It seems obvious that there must be some negative
impact on learning outcomes and grades for some students – though there is no research to back up
that assumption.
LMS underutilization has multiple impacts on UNM. These include:



Inability to tightly and uniformly integrate other systems with the LMS
Loss of learning outcomes that might or could be achieved through better LMS use
Complicating support for faculty and students
Modern organizations are expected to use technology to seamlessly connect various organizational
functions and systems. At its best, this integration improves quality and efficiency and reduces costs.
But achieving those results requires the right technology and widespread or universal use. Lacking those
features, the benefits are poorly achieved. Early alerts through LoboAchieve are a case in point. They
have yet to deliver their promised benefits due to underuse and poor integration with UNM Learn.
Achieving UNM-wide learning improvement through learning analytics is another case in point. Too few
faculty use UNM learn in a standardized enough way to provide sufficient learning data for mining,
analysis, and downstream improvement efforts.
Ease-Of-Use, Training, and Support – A Gordian Knot?
Many current UNM Learn users cite ease-of-use and training and reasons not to use UNM Learn or as
barriers to its effective use. At the same time, support organizations report low utilization of available
support and training services. This apparent paradox has left both sides unhappy and struggling for a
solution.
Training and support complexity is a cost borne throughout UNM though it’s primarily concentrated
within Extended Learning (EL) and, to a lesser extent, IT Services. EL supports faculty and student users
but it struggles to do – chiefly because of high resource requirements, UNM Learn’s complex user
interface and features, and the myriad ways in which it’s used from course to course. In many ways, the
freedom that faculty have to choose their own LMS/tools and decide exactly how to use it creates a
training and support quagmire. Faculty who see significant benefits to LMS use and take advantage of
available training and support services are generally satisfied. But many faculty underutilize available
services.
The issues of ease of use, tool complexity, training, and support are tightly intertwined. Poor ease-ofuse and high tool complexity generates greater need for training and support. But it’s unclear whether
adequate training and support can ever be a sufficiently strong antidote to what some faculty view as a
much-too-complex and clunky LMS. Would upgrading UNM Learn or switching to another LMS solve
this problem? That’s a question that they faculty survey wasn’t design to answer directly.
Where To From Here?
The purported benefits of widespread LMS use by faculty include:
1. Enabling hybrid and online delivery modes
2. Saved time/effort for common instructional tasks
LMS Insights - Faculty Tech Survey
Draft - 7/17/2016
Page 7 of 8
3. Improved instructional outcomes
4. Capturing student performance data to enable downstream capabilities such as alerts, timely
intervention, learning analytics, and data-driven improvement
The faculty technology survey provides some evidence that UNM is achieving the first benefit. The
results paint a noisy picture with respect to the second and third benefits. Some faculty have achieved
those benefits but others haven’t – primarily due to LMS complexity and underutilization. The fourth
benefit is unachievable without much higher and more effective LMS utilization.
LMS Insights - Faculty Tech Survey
Draft - 7/17/2016
Page 8 of 8