Quality of Learning Management Systems towards ELearning Effectiveness: A Study based on
Blackboard 7.1, Moodle 1.9 and Blackboard 9.1
Presented By: Nadeera Ekanayake, Travis Fisher , Brandon Molton
Presented To: Dr. Ahmad Zargari
Department of Applied Engineering and Technology
Morehead State University
March 2013
2
Table of Contents
Chapter 1: General Area of Concern............................................................................................................. 3
Introduction ............................................................................................................................................... 3
Problem Statement .................................................................................................................................... 5
Significance............................................................................................................................................... 6
Assumptions.............................................................................................................................................. 7
Limitations ................................................................................................................................................ 7
Definition of Terms................................................................................................................................... 8
List of Acronyms ...................................................................................................................................... 9
Chapter 2: Literature Review ...................................................................................................................... 10
Background ............................................................................................................................................. 10
ELearning............................................................................................................................................ 10
Learning management systems ........................................................................................................... 14
Blackboard .......................................................................................................................................... 19
Moodle ................................................................................................................................................ 21
Moodle Compared to Blackboard 9.1 ................................................................................................. 22
A brief historical review ......................................................................................................................... 24
Rationale ................................................................................................................................................. 26
Chapter 3: Methodology ............................................................................................................................. 28
Research Design...................................................................................................................................... 28
Instrumentation ....................................................................................................................................... 30
Data Collection Methods ........................................................................................................................ 32
Data Analysis Methods ........................................................................................................................... 32
Appendix A – Sample Survey Questionnaires ............................................................................................ 34
References ................................................................................................................................................... 40
3
Chapter 1: General Area of Concern
Introduction
E-learning programs and utilities have become important mainstays for a large variety of
educational institutions ranging from secondary schools through collegiate and university
programs. E-learning programs help to augment the learning services institutions can provide by
enabling a method for students and faculty to communicate among each other outside of lecture
and discussion, as well as, allowing distance learning for students who are incapable to being
present for traditional personal learning techniques. For many instructors, e-learning tools have
become vital in order to successfully communicate important information to students, including
various texts and media related to the course’s current topic, important dates and times for events
such as exams, and coursework for students to complete and turn-in. Instructors even often have
the option to facilitate class discussion through course forums, and host video seminars. With
these tools, instructors easily have the option to teach courses in a hybrid model, utilizing both
in-class lecture and at-home online studying in order to enhance the learning experience, or even
teach courses completely outside of lecture by providing all of the necessary tools students
require to complete the course so long as all participating students and faculty have access to a
computer and internet access.
Although it is possible for an institution to create their own e-learning system, there
currently exist programs which can be implemented and used across an entire institution which
can automatically setup and maintain all of the background functionality required to establish a
successful e-learning system. Currently, the two most popular programs are Blackboard and
Moodle. Although both are developed and distributed with the same goal in mind (to provide a
direct method of communication between students and instructors outside of the classroom
4
environment) both systems do function somewhat different from both a student’s and a
faculty/instructor viewpoint. Prior studies have also shown that both students and faculty have
shown personal preference and varying levels of success using one system over the other.
Traditionally, Morehead State University relied solely on the Blackboard e-learning
system in order to encourage the application of an online learning environment within its
courses. However, in 2011, Morehead State experimented with a pilot program which permitted
some courses to use Moodle in its courses, in a trial run which would ultimately help to decided
whether the university would continue with Blackboard, or overhaul the e-learning processes
within the university to switch to a Moodle based program. When Morehead started the
evaluation process, it was using Blackboard Learn 7.3. Although, Moodle was successfully
implemented, the decision was made by Morehead State to continue on with the use of
Blackboard. Although, the stint with Moodle was short lived, there still exists the ability to
collect valuable data from those who took part in this pilot program and most closely research
the effects both Blackboard and Moodle have on both students and facilitators.
This proposal will provide background information behind Blackboard and Moodle and
present relevant past research which will demonstrate the importance of continuing the
investigation of the comparison between the effects of Blackboard 7.3, Moodle 1.9 and
Blackboard 9.1 have on Morehead State University’s e-learning programs.
5
Problem Statement
There are various online learning management systems available. As discussed in the
Introduction section Blackboard and Moodle are one of the most discussed and widely used
systems throughout the US universities. Over the past years, MSU online course and hybrid
course offering was increased. And most of the courses were using online components to engage
with the students. Due to this trend MSU had to select or improve existing LMS system to a
newer system, which would improve the e-Learning effectiveness and user satisfaction. In 2011,
MSU decided to switch from old Blackboard 7.3 version to newer Blackboard 9.1 version. This
decision was not one sided, MSU teachers and students were involved in deciding the best LMS
system for them through number of trails between Moodle 1.9 and Blackboard 9.1. This research
will try to identify if the selected (Blackboard 9.1) LMS system quality has improved the online
learning effectiveness.
There are many components in online LMS systems that directly improve the learning/teaching
experience. Each new version of LMS has improved in quality compared with previous versions.
The purpose of this study is to examine how the quality of LMS systems affects the eLearning
effectiveness. Specifically, the study will focus on one primary objective:
To determine how the quality of LMS systems affects e-learning effectiveness.
In addition, this research will try to identify which LMS components have helped students and
instructors to enhance e-learning experience.
6
Hypothesis
Null hypothesis: Quality of LMS does not have an effect on eLearning effectiveness.
Alternative hypothesis: Quality of LMS has an effect on eLearning effectiveness.
Significance
Online education has become widely used within the last 10 years by children as young as preschool age. It’s being implemented into basic curricula throughout the nation, from elementary
schools to college campuses. By using a learning management system (LMS) some schools now
have the ability to better serve their students through an integrated web-based platform of online
education courses. Everything is all there from educational records, documentation of teacher
appraisals, and even personalized content per student. There are webinars, assessments,
discussion forums, and many more components in a LMS system that a student/teacher can take
advantage of.
From MSU point of view, this research will help the university departments to identify if there
has being significant improvement in online course enrollments after moving to new LMS
system from previous version. This research will also help to identify which components are
widely used by teachers to interact with students. Information gathered from this study will help
MSU instructors to identify and incorporate most important LMS components needed for
students when creating online course contents. This will help to maximize the productivity of
online courses.
7
These findings will be helpful for LMS vendors to identify major components that are helpful
for students and what improvements should be integrated to those to improve the quality of
services they are providing.
Assumptions
For this study, the research would have to be based on few assumptions. The following
assumptions are made about this research and the circumstances in which it will be collected:
1. The data and information that will be obtained from the MSU personnel are unbiased and
accurate.
2. The participants in this study, understand the surveyed questions and answers, and will
follow the instructions to complete the survey.
3. Participants have sufficient technological skills to complete the survey.
4. Participants will volunteer to take the survey and will provide the most accurate answers
based on their understanding of the questions.
5. Some participants may not have got the chance to participate in the pilot project for
Moodle.
6. The treatments of all groups (Department, Instructors and Students) will be comparable
in regards of methodology of the study.
Limitations
This study has some limitations. The study is limited to Morehead state university and it will also
be limited to Blackboard Learn 7.3, Moodle 1.9 and Blackboard Learn 9.1, since these are the
only Learning Management Systems used by Morehead state university.
8
Definition of Terms
E-Learning : eLearning - short for electronic learning, is an umbrella term that refers to all types
of training, education and instruction that occurs on a digital medium, like a computer or mobile
phone.
Distance Learning: Educational situation in which the instructor and students are separated by
time, location, or both. Education or training courses are delivered to remote locations via online,
including written correspondence, text, graphics, audio- and videotape.
Learning Management systems (LMS) : Learning Management System (LMS) is a web based
software application that is used to automate the administration of the E-Learning activities such
as managing, tracking and reporting on course events.
Blackboard(Bb) : Virtual learning environment and course management system developed by
Blackboard Inc. It is a Web-based server software which features convenient course
management, customizable open architecture, and accessible design that allows for integration
with student information systems and authentication procedures.
Moodle : Moodle (Modular Object Oriented Developmental Learning Environment) is a
learning management system which allows for facilitators to create online dynamic websites
which can be installed on either an established web server, a personal computer, or a web hosting
company
Online Course: Courses that are taught completely via Internet.
Hybrid Course: Course uses both classroom and online instructional modes and meets face-toface for 25%-50% of the course contact hours prescribed by the course type and units.
9
List of Acronyms
CMS : Course Management System
LMS : Learning Management systems
Bb : Blackboard Learn
Moodle : Modular Object Oriented Developmental Learning Environment
MSU : Morehead State University
10
Chapter 2: Literature Review
Background
ELearning
Electronic learning covers a wide set of applications and processes, such as web-based learning,
computer-based learning, virtual classrooms, and digital collaboration. It includes the delivery of
content via internet, intranet/extranet (LAN/WAN), audio- and videotape, satellite broadcast,
interactive TV, and CD-ROM. Thus, this definition defines eLearning as basically anything that
is electronic.
The global market for corporate eLearning will grow nearly 27%, compounded annually
(McGee, 2004). IDC forecasts that the eLearning market, which was about $6.5 billion in 2003,
will increase to more than $21 billion by 2008, and hit $52.6B by 2010.
Figure 1Projected Corporate e-Learning Growth
11
Almost 3.5 million students were taking at least one online course during the fall 2006 term; a
nearly 10 percent increase over the number reported the previous year (Allen, Seaman, 2007). In
addition, the 9.7 percent growth rate for online enrollments far exceeds the 1.5 percent growth of
the overall higher education student population (nwlink, 2013).
Strengths of Online Learning
There are many reasons why online programs have become a popular form of distance learning
in higher education today. The online environment offers unprecedented opportunities for people
who would otherwise have limited access to education, as well as a new paradigm for educators
in which dynamic courses of the highest quality can be developed. Here is a list of some of the
major benefits of online programs (ION, 2013):
Anywhere….
The main advantage of asynchronous online learning is that it allows students to participate in
high quality learning situations when distance and schedule make on-ground learning difficultto-impossible. Students can participate in classes from anywhere in the world provided they have
a computer and Internet connection. In addition, the online format allows physically challenged
students (and teachers) more freedom to participate in class. Participants access the Virtual
Classroom through their computers instead of having to "go to class" physically.
Anytime, Any Pace….
The Virtual Classroom is accessible 24 hours a day, seven days a week. Time efficiency is
another strength brought by the online learning format. Asynchronous communication through
online conferencing programs allows the professional juggling work, family, and study schedules
12
to participate in class discussions. There is no question about doing the work; just do it at the
times that are more convenient. Students can access their courses at any time of day or night.
Further, they have continuous access to lectures, course materials, and class discussions. This is
particularly convenient for those who may need to reread a lecture or take more time to reflect on
some material before moving on.
Synergy
The online format allows a dynamic interaction between the instructor and students and among
the students themselves. Resources and ideas are shared, and continuous synergy will be
generated through the learning process. Each individual can contribute to the course discussions
and comments on the work of others. The synergy that exists in the student-centered Virtual
Classroom is one of the most unique and vital traits that the online learning format possesses.
High Quality Dialog
Within an online asynchronous discussion structure, the learner may reflect on comments from
others before responding or moving on to the next item. This structure allows students time to
articulate responses with much more depth and forethought than in a traditional face-to-face
discussion situation where the participant must analyze the comment of another on the spot and
formulate a response or otherwise lose the chance to contribute to the discussion.
Level Playing Field
In the online environment, learners have a certain measure of anonymity. Discriminating factors
such as age, dress, physical appearance, disabilities, race and gender are largely absent. Instead,
13
the focus of attention is clearly on the content of the discussion and the individual's ability to
respond and contribute thoughtfully and intelligently to the material at hand.
Access to Resources
It is easy to include distinguished guest experts or students from other institutions in an online
class. Further, today's students have access to resources and materials that may be physically
located anywhere in the world. An instructor can compile a resource section online with links to
scholarly articles, institutions, and other materials relevant to the course topic for students to
access for research, extension, or in depth analysis of course content material.
Weaknesses of Online Learning
Equity and Accessibility to Technology
Lack of access whether it be for economical or logistics reasons will exclude eligible students
from the course.
Computer Literacy
Both students and facilitators must possess a minimum level of computer knowledge in order to
function successfully in an online environment.
Limitations of Technology
User friendly and reliable technology is critical to a successful online program. However, even
the most sophisticated technology is not 100% reliable.
The Students
14
While an online method of education can be a highly effective alternative medium of education
for the mature, self-disciplined student, it is an inappropriate learning environment for more
dependent learners
Lack of Essential Online Qualities
Successful on-ground instruction does not always translate to successful online instruction. If
facilitators are not properly trained in online delivery and methodologies, the success of the
online program will be compromised.
Levels of Synergy
Online learning has its most promising potential in the high synergy represented by active dialog
among the participants, one of the most important sources of learning in a Virtual Classroom.
However, in larger classes (20 or more students), the synergy level starts to shift on the learning
continuum until it eventually becomes independent study to accommodate the large class. At this
point, dialog is limited as well as interaction among participants and the facilitator. The medium
is not being used to its greatest potential.
Learning management systems
A Learning Management System (LMS) is a web based software application that is used to
automate the administration of the E-Learning activities such as managing, tracking and
reporting on course events. The types of learning delivered could be 100% online and a
15
combination of traditional and online learning, which is also called hybrid. Figure 1 illustrates
the learning management system with its typical functionalities.
Security
Administ
ration
Tools
Reportin
g
Assesme
nt
Capabiliti
es
Learning
Management
System (LMS)
Skill
Manage
ments
Content
Accessibi
lity
Content
Develop
ment
Contenet
Integrati
on
Figure 2: Learning Management System and Its Functional Elements (Amirkhanpour, 2010)
The main functional characteristics of a learning management system include administration
tools, content accessibility, content development, content integration, skills management,
assessment capabilities, reporting and security (Amirkhanpour, 2010).
Administration tools: Enables administrator to manage user registration and define roles,
manage content.
Content Accessibility: LMS allows to access course contents through online.
16
Content development: Includes authoring, maintaining and storing learning materials.
Content integration: Provide support to a wide range of external systems such as content
management systems and registration modules.
Skill Management: assessing the competency skills of the E-Learners using peer reviews and
other feedback tools.
Assessment capabilities: Provides assessment capabilities for the delivered learning content.
Examples could be Assignments, Quizzes and Exams.
Reporting: tracks all the learning and training efforts within the LMS and creates customized
reports based on the courses offered, course completion/incompletion status.
Security: Passwords and encryption methods are typical security features of an effective LMS.
Examples could be user authorization, authentication, and data protection
Over the past years LMS has become a powerful tool for higher education institutes. Its impacts
have been felt mostly outside of traditional education institutions.
LMS software is able to add a level of efficiency to higher education institutes’ learning systems,
with a number of other benefits emerging as well, such as:
Highly Flexible and Customizable
Learning management systems are extremely flexible and customizable to meet your specific
needs.
Easily Create Courses
17
The integrated online registration software lets anyone create courses and training programs over
the internet. You can instantly publish the course materials online for people to view and register
for your upcoming classes.
Anytime Access
Since the whole learning experience is created and managed virtually, the system lets an
individual (student/instructor) access the course materials, 24x7. This will allow students to go
through their classes and courses in a more relaxed environment, wherever they wish.
Robust Reporting and Tracking Capabilities
Instructors can easily track and evaluate a course.
The market for Learning Management Systems has grown into a billion dollar industry with
hundreds of offering. 2012 annual Campus Computing Survey held by Campus Computing
ProjectTM confirms the increasingly important role of Learning Management Systems as a core
instructional resource. Overall, the percentage of college courses that use a LMS tool has risen
from 15% in 2000 to more than 75% in 2012 (The National Survey of Computing and
Information Technology, 2012). Figure 3 represents a profile of the LMS Market for Fall 2012
and Table 1 provides information regarding Institutional Demography of LMS providers for Fall
2012.
18
Figure 3: A Profile of the LMS Market, Fall 2012 (The
National Survey of Computing and Information
Technology, 2012)
Table 1 : Institutional Demography of LMS Providers, 2012 (The National Survey of Computing
and Information Technology, 2012)
19
Blackboard
The Blackboard Learning System is a virtual learning environment and course management
system developed by Blackboard Inc. It is a Web-based server software which features
convenient course management, customizable open architecture, and accessible design that
allows for integration with student information systems and authentication procedures. It may be
installed on local servers or hosted by Blackboard ASP Solutions. Its main purposes are to add
online elements to courses traditionally delivered face-to-face and to develop completely online
courses with few or no face-to-face meetings (Education Week, 2013).
The Blackboard Learning System provides users with a platform for communication and sharing
content.
Communication

Announcements: Professors and teachers may post announcements for students to read.
These can be found under the announcement tab, or can be made to pop-up when a
student accesses Blackboard.

Chat: This function allows those students who are online to chat in real time with other
students in their class section.

Discussions: This feature allows students and professors to create a discussion thread and
reply to ones already created.
20

Mail: Blackboard mail allows students and teachers to send mail to one another. This
feature supports mass emailing to students in a course.
Content

Course content: This feature allows teachers to post articles, assignments, videos etc.

Calendar: Teachers can use this function to post due dates for assignments and tests.

Learning modules: This feature is often used for strictly online classes. It allows
professors to post different lessons for students to access.

Assessments: This tab allows teachers to post quizzes and exams and allows students to
access them via the internet

Assignments: This features allows for assignments to be posted and for student to submit
assignments online

Grade Book: Teachers and professors may post grades on Blackboard for students to
view.

Media Library: Videos and other media may be posted under this function.
Students and faculty may benefit from course management systems such as the Blackboard
Learning System. Potential benefits include:

Increased availability

Quick feedback

Improved communication

Tracking
21

Skill building.
Moodle
Currently acting as one of the most popular e-learning systems available for use, Moodle
(Modular Object Oriented Developmental Learning Environment) is a learning management
system which allows for facilitators to create online dynamic websites which can be installed on
either an established web server, a personal computer, or a web hosting company (“About
Moodle”, 2013). Moodle is extremely customizable, as it exists as a free, open source package.
This means that a e-learning instructor, so long as he or she has the necessary computer skills,
can fully customize the utility as necessary. This also means that the software can be scaled to
fit of needs of a variety of institutions, ranging from a single instructor course, to a full 40,000
student university setting (Beatty and Ulasewicz, p.36, 2006).
Moodle comes packaged with activity modules, which are the bases with which to
develop systems such as forums, databases, and wikis which can all be used to construct
collaborative communities within which students and instructors can discuss course material.
These modules also allow instructors to deliver course content to students and provide
coursework and various assessments to students. Figure 1, below, demonstrates an example of
an opening page of a Moodle based learning system. This helps to demonstrate the various types
of activities and content available to students upon beginning a class session using this e-learning
system.
22
Figure 4: Example of a Moodle opening page for a course. This helps to demonstrate the
capabilities of what can be accessed through a Moodle-based e-learning system (Beatty and
Ulasewicz, p.36, 2006).
One major advantage which Moodle has gained over Blackboard is its lack of a licensing
requirement. Currently, in order for any institution to make use Blackboard, they must purchase
a licensing agreement. This makes Moodle a cheaper alternative to Blackboard from a business
standpoint. This does not necessarily make Moodle a “free” program, due to the need to
consider administration and maintenance costs for the system, however compared to
Blackboard’s yearly licensing fees, this is typically much more affordable (Carvahlo et al, 2011).
Moodle Compared to Blackboard 9.1
Moodle and Blackboard are both popular online learning platforms with which educators can
develop complete online course that can include multimedia content.
23
Moodle is an Open Source Learning Management System that is provided freely and can be run
on many operating systems. Its “free to download, change, share, improve, and customize to
whatever you want it to be,” according to the Moodle website. Therefore, any educator can use it
to build or supplement a course.
Blackboard on the other hand is a proprietary Learning Management System and its use is
typically limited to educators at institutions which pay a fee each year to take on a license
agreement for its use. Blackboard Learn 9.1 is only one of many offerings from the software
company Blackboard, Inc.
Improvements in its uses for higher education include course wikis (Moodle improved theirs
too), blogs and journals that stimulate conversation and reflection on a course, and group tools
that make group collaboration and communication easier than the previous version. Its most
notable feature is its Web 2.0 interface, which makes it easy for educators to navigate when
adding content to an online course and for students to navigate when accessing course content.
Features & Functions: Both of these tools have a lot of different functionality available, either
natively, or through add-on types of functionality. If different functions are going to be the
deciding factor in selecting one of these versus the other, this will have to be a decision made by
the institution based on its ultimate needs.
Cost: This is clearly different. As an open source product, Moodle is simply less expensive.
Blackboard is sort of the “Rolls Royce” of today’s LMS, and there are users of the product who
would tell you that if you want the best LMS money can buy, you should make the financial
commitment to Blackboard. On the other hand, if you want a premier product for a much lower
cost, Moodle is really the way to go. Another thing to be aware of is that Blackboard builds
24
substantial annual increases into their pricing model, since they are continually procuring and
integration additional products into their offerings, with the intent of adding value for their users.
Product/vendor model: Moodle and Blackboard are very different products with very different
vendor models. One is open source, and there are many support and service vendors to choose
from, while the other is proprietary and there is just the one company to work with. How that
impacts your decision is up to you and your institution to determine (EmergingEdTech, 2013).
A brief historical review
In 2005 a grant from the US Department of Education funded Boston College (BC) to conduct a
research study with many aggressive goals that would, if met by the ten collaborative state
partnerships, result in a sustainable, systematic delivery of online professional development
(OPD) for teachers. The e-Learning for Educators research efforts focused on assessing the
effects of OPD on three areas: (1) improvements in teacher content knowledge; (2)
improvements in instructional practice; and (3) subsequent improvements in academic
achievement for students. Assessing improvements required the assistance of a team of
researchers and evaluators from inTASC, Boston College’s Technology Assessment and Study
Collaborative Center. Education Development Center (EDC) developed twelve ELA and math
online courses used for the research series and provided the training for facilitators who
delivered the workshops to teachers participating in the research study.
This study took place in the spring of 2007 and continued through the summer of 2009. Out of
the 21 participating states there included 369 teachers and 21,217 students that were subject to
the studies. The scale of participation was large and the quantity of data collected and analyzed
25
was extremely large. For example, more than 21,500 open-ended responses were analyzed in
addition to hundreds of multiple-choice questions. There were no existing systems that could
capture the types and amount of data that this research effort required so Boston College
designed an online data collection system specifically for the e-Learning research effort.
Ultimately, teachers completed pre and post surveys, as well as administered student surveys and
tests over a period of three semesters (Alabama Public Television, 2013).
The collected data were subjected to appropriate statistical tests (ANOVA, ANCOVA, HLM2) to
determine both statistical impact and the practical educational impact (effect size). These
statistical analyses examine whether a measurable effect on teachers or students was
happenstance or is a result of taking the OPD courses.
These study results demonstrate that the e-Learning courses and process can have a positive
effect on teacher knowledge, teacher instructional practices, and, most importantly, student
achievement. Overall, both teachers and students made statistically significant improvements. In
other words, there is some evidence of a cause-and-effect change due to the OPD instruction;
improvements did not occur by chance.
Most of the previous studies on LMS systems are based on comparisons between Blackboard and
Moodle. Generally, these types of studies show a preference to Moodle. In a study in Hong
Kong in 2005, it was reported that out of 84 education students, 45% preferred the use of Moodle
over Blackboard, while 49% showed no preference between the two systems. In this particular
study, Moodle was favored due to it have a superior ability to promote interaction among
students and instructor, however, users tended to prefer Blackboard in terms of organization of
the course’s instructional materials (Carvahlo et al, 2011).
26
This is a stark contrast to another study completed by Machado and Tao (2007) when
course material organization and communication were rated higher in Moodle over Blackboard.
In this particular study, involving 53 students, 71% of the students found that Moodle was easier
to use that Blackboard, with 75% stating they would prefer the use of Moodle as a replacement
for Blackboard in the future (Carvahlo et al, 2011).
Carvahlo et al (2011) conducted further research comparing Blackboard and Moodle
among students in Portuguese university and found that 46.5% of students preferred the use of
Blackboard while only 34.7% preferred Moodle (nearly 20% of the students surveyed had no
preference). However, student assessment of various e-learning features, including the visual
appearance, organization of course materials, user interface, etc, favored Moodle. These results
tended to favor Moodle, more so, when students identified themselves as being familiar with the
use of both systems.
Rationale
Currently, not many studies exist which studies how quality of LMS affects the
eLearning Effectiveness. From a faculty/instructor perspective, there is a need for more direct
research behind preference between Moodle and Blackboard systems. Although there tends to
be a lack of direct information pertaining specifically toward this group of users, some
information has been provided which shows more personal insight into the preferences between
the two systems. Beatty and Ulasewicz (2006) wrote a paper together describing their
experiences with the two programs. According to Beatty, he found the intstructor interface for
Moodle to be “more intuitive and easy to use than the instructor interface in Blackboard” due to
the improved editing tools supplied with Moodle. Ulasewicz agreed that the instructor interface
may have been superior in stating that it was easier for her to instate a gradable forum system
27
than she felt it would have been using a Blackboard system. Beatty found issues with his
students being able to edit posts in discussions, while Ulasewicz found no major issues with the
system on Moodle. Finally, both commented that due to the open source nature of Moodle, there
are times that’s it can be preferred when it pertains to fixing and creating new features needed for
their environment.
Overall, more information is needed to determine if there is indeed a preference among
students and faculty as to each of the features and overall use of Blackboard and Moodle. This
information could be beneficial to institutions, such as Morehead State University, who have
been undergoing changes within their e-learning programs and will empower them to make a
more beneficial decision as to which program to instate.
From MSU point of view, this research will help the university departments to identify if there
has being significant improvement in online course enrollments after moving to new LMS
system from previous version. This research will also help to identify which components are
widely used by teachers to interact with students. Information gathered from this study will help
MSU instructors to identify and incorporate most important LMS components needed for
students when creating online course contents. This will help to maximize the productivity of
online courses.
These findings will be helpful for LMS vendors to identify major components that are helpful
for students and what improvements should be integrated to those to improve the quality of
services they are providing.
28
Chapter 3: Methodology
In the previous section, an introduction to the proposed area of research was presented. This
included a description of the study’s research problem, several research purposes, hypotheses
that will serve to direct the data analysis, and an identification of several terms key to this
research. In addition, a review of relevant literature related to the eLearning, Learning
Management systems, Blackboard and Moodle established a background of support for the
study. Previous related studies have been studied extensively, to identify any related information.
Purpose of this study is to examine how the quality of LMS systems affects the eLearning
effectiveness.
The intent of this section is to describe the methodology that is proposed for this research effort.
Included in this section will be a description of the proposed research design, study sample, and
proposed data collection methods, procedures, and analysis efforts.
Research Design
Research plan will be divided into six main steps including: Research Planning, Survey
development and deployment, Data analysis, findings, Conclusions and Implications.
29
Figure 3: Research Methodology
The first step in this research will start with planning of the research. In this step the study will
involve an extensive literature review. The purpose of the literature will be to study and plan the
research. In the planning phase study setting and populations and sampling plan will be
identified.
Study Setting
The proposed study will be conducted in Morehead State University. The study will consist of
three settings. First setting will focus on Blackboard Learn 7.3, Second will focus on Moodle and
third setting will focus on Blackboard Learn 9.1.
Population and Sampling Plan
Study will be conducted through all the Academic departments in Morehead State University.
Colleges:

Caudill College of Arts, Humanities, & Social Sciences
30

College of Business & Public Affairs

College of Education

College of Science & Technology

University College
1 Department representative, 10 Instructors, and 20 Students from each department will be
selected for the study. A sample of the population of students and instructors at each department
in Morehead State University will be used to draw conclusions about the entire population of
students and instructors.
Instrumentation
Self-administered questionnaires will be used as the instruments in this study. Following three
main categories will be taken in to consideration when creating the questionnaires.
A. Social Issues
: Learner Perspective
a. Learners Attitude toward LMS
b. Learners Computer Anxiety
c. Self-Efficiency
d. Enjoyable Experience
e. Interaction methods with other students and Instructor
: Instructor Attitudes
a. Responsiveness
b. Enjoyable Experience
c. Availability
31
d. Promptness
e. Usefulness
f. Fairness
g. Communication Ability
h. Interaction methods with students
B. System Quality
: Technical Quality
a. Easy to use
b. Security features
c. User Friendly
d. Stable
e. Access speed
f. Help manual availability
: Content Quality
a. Course management
b. Well organized
c. Content Flexibility ( Can move content from one course shell to another)
d. File types supported
e. Availability of editing options.
: Service Quality
a. Tracking options with report generating availability
b. Administration
32
C. Supportive Issues
a. Environment
b. Trend
Questionnaires will be divided in to three main sections based on the respondents.

Department

Instructors

Students
Sample Survey Questionnaire for Departments is attached in the Appendix A. Surveys will be
administrated through free online questionnaire and survey provider, Survey Monkey.
Data Collection Methods
Data will be collected from surveys of departments, faculty and students using online survey tool
Survey Monkey and later collected data will be entered into Excel spreadsheets for review.
Data Analysis Methods
The data analysis will be processes in six general phases. The first phase consists of parsing the
gathered data into Excel spreadsheets for further analysis. Each spreadsheet will be allocated to
Department, Instructor and Student respondent data. The second phase will be filtering the
gathered data for further analysis. This will be filtered based on the criteria: before 2011 and
after 2011.The third phase consists of removing any invalid data and filtering data for further
analysis. The fourth phase consists of grouping the response data into related question categories.
The fifth phase consists of calculating total and sub totals for each categories and calculating the
percentages for collected response data.
33
Final phase will consist of general analysis of the results and to conduct statistical analysis for
determining the acceptance or rejection of the hypothesis of Research question.
34
Appendix A – Sample Survey Questionnaires
Survey Questionnaire for Departments
1. Name of the Department
The next several questions ask about online distance learning courses
2. Which year did your department begin to offer online distance learning courses?
3. What is the estimated percentage of total courses offered by your department before 2011
that were online distance learning?
a. Less than 5%
b. 5-10%
c. 10%-20%
d. More than 20%
4. How did the number of online distance-learning course increase after 2011?
a. Decreased or no change
b. Increase 1-9%
c. Increase 10-24%
d. Increased 25%-50%
e. Increased more than 50%
5. What is the estimated percentage of tenure-track faculty members who taught online
distance-learning courses before 2011?
a. None
b. 1-4%
c. 5-9%
35
d. More than 10%
6. What is the estimated percentage of tenure-track faculty members who taught online
distance-learning courses after 2011?
a. None
b. 1-4%
c. 5-9%
d. More than 10%
7. What is the estimated percentage of students who took online distance-learning courses
before 2011?
a. None
b. 1-9%
c. 10-24%
d. More than 25%
8. What is the estimated percentage of students who took online distance-learning courses
after 2011?
a. None
b. 1-9%
c. 10-24%
d. More than 25%
The next several questions ask about hybrid courses, where the instructor combines the elements
of online distance-learning courses and traditional course to replace some classroom session
with virtual sessions.
36
9. What year did your institution begin to offer hybrid course?
10. What is the estimated percentage of total courses offered by your department before 2011
that were hybrid?
a. Less than 5%
b. 5-10%
c. 10%-20%
d. More than 20%
11. How did the number of hybrid course increased after 2011?
a. Decreased or no change
b. Increase 1-9%
c. Increase 10-24%
d. Increased 25%-50%
e. Increased more than 50%
12. What is the estimated percentage of tenure-track faculty members who taught hybrid
courses before 2011?
a. None
b. 1-4%
c. 5-9%
d. More than 10%
13. What is the estimated percentage of tenure-track faculty members who taught hybrid
courses after 2011?
a. None
b. 1-4%
37
c. 5-9%
d. More than 10%
14. What is the estimated percentage of students who took hybrid courses before 2011?
a. None
b. 1-9%
c. 10-24%
d. More than 25%
15. What is the estimated percentage of students who took hybrid courses after 2011?
a. None
b. 1-9%
c. 10-24%
d. More than 25%
The next few questions ask about students taking online distance courses, hybrid courses,
and traditional courses with LMS activities.
16. How challenging was it for the department to support students' online distance courses,
hybrid courses, and traditional courses with technology activities, in the following areas
before 2011?
Rate each on a scale of 1-5, where 1= not a challenge, 5 = very significant challenge
a. Students lack of knowledge about technology
b. Network access/usage problems
c. Utilizing online course technology
d. Utilizing online course application tools
38
17. How significant a challenge is it for the department to support students' online distance
courses, hybrid courses, and traditional courses with technology activities, in the
following areas after 2011?
Rate each on a scale of 1-5, where 1= not a challenge, 5 = very significant challenge
a. Students lack of knowledge about technology
b. Network access/usage problems
c. Utilizing online course technology
d. Utilizing online course application tools
The next few questions ask about instructors taking online distance courses, hybrid
courses, and traditional courses with LMS activities.
18. How challenging was it for the department to support instructors' online distance courses,
hybrid courses, and traditional courses with technology activities, in the following areas
before 2011?
Rate each on a scale of 1-5, where 1= not a challenge, 5 = very significant challenge
a. Faculty’s lack of knowledge about technology
b. Faculty’s lack of confidence to use technology in teaching environment
c. Unreliable technology: Network/software crashes during teaching session.
d. Instructors’ lack of knowledge about how to design course contents incorporating
LMS.
19. How significant a challenge is it for the department to support instructors' online distance
courses, hybrid courses, and traditional courses with technology activities, in the
following areas after 2011?
39
Rate each on a scale of 1-5, where 1= not a challenge, 5 = very significant challenge
a. Faculty’s lack of knowledge about technology
b. Faculty’s lack of confidence to use technology in teaching environment
c. Unreliable technology: Network/software crashes during teaching session.
d. Instructors’ lack of knowledge about how to design course contents incorporating
LMS.
40
References
Amirkhanpour, M. (2010). Learning Management Systems and E-Learning within Cyprus
Universities.
ION. (2013). Retrieved from
http://www.ion.uillinois.edu/resources/tutorials/overview/weaknesses.asp
nwlink. (2013). Retrieved from http://www.nwlink.com/~donclark/hrd/elearning/tools.html
(2012). The National Survey of Computing and Information Technology. The Campus
Computing Project.
Carvalho, A., Areal, N., & Silva, J. (2011). Students' perceptions of Blackboard and Moodle in a
Portuguese university. British Journal of Educational Technology, 42(5)
Despotovic-Zrakic, M., Markovic, A., Bogdanovic, Z., Barac, D., & Krco, S. (2012). Providing
adaptivity in Moodle LMS courses. Journal of Educational Technology & Society, 15(1),
326-338.
About Moodle. (2013, January 27). Retrieved from https://moodle.org/about/
Beatty, B., & Ulasewicz, C. (2006). Faculty perspectives on moving from Blackboard to the
Moodle learning management system. TechTrends: Linking Research & Practice to
Improve Learning, 50(4)
Alabama Public Television. (2013). Retrieved from
http://www.aptv.org/aptplus/ELearning/impact.asp
Drake University. (2013). Retrieved from http://oit.drake.edu/advantages-blackboard-9/
41
Education Week. (2013). Retrieved from
http://www.edweek.org/dd/articles/2008/06/09/01moodle.h02.html
EmergingEdTech. (2013). Retrieved from http://www.emergingedtech.com/2010/11/moodle-2versus-blackboard-brief-comparison/
The Blackboard Learning System. (n.d.). Retrieved from
http://uupinfo.org/research/working/bradford.pdf