Report of the Moodle Pilot Team, April 2009
Report of the Moodle Pilot Team
North Carolina State University
Last updated on April 23, 2009
Moodle Pilot Team Members
Dr. Donna Petherbridge (DELTA), Chair
James Bossert (CALS)
Dr. Marty Dulberg (DELTA)
Stacy Gant (DELTA)
Lou Harrison (DELTA)
Bill Hicks (DELTA)
Leigh Jay Hicks (CALS)
Greg Kraus (DELTA)
Dr. Greg Robinson (CNR)
Dr. Alan Schueler (CALS)
Bethany Smith (Education)
Jeff Webster (DELTA)
Summary
North Carolina State University has used Learning Management Systems (LMSs) since the
development of WolfWare in 1997 to provide discussion, assignment and file management
options for online course components. Blackboard Vista, a commercial Learning
Management system used on our campus since 2005, has experienced rapid growth since
its implementation, with an estimated 3000 live sections in a given academic year, with our
system, WolfWare, still used heavily for course file storage, media delivery, and class
mailing lists. With an impending redesign of WolfWare needed because of aging server and
codebase issues, and with Blackboard Vista eventually going away as an LMS option for our
campus (a decision made by the Blackboard company to end development of the Vista
platform), NC State University must consider a new LMS for our campus to support the
many course sections that depend on these online technologies.
After evaluating various LMS replacement options, with much consideration of the current
LMS landscape, the maturity of the open source LMS community, and the technical,
training, support and feedback discussions informed by the 2008 Moodle pilot, the Moodle
Pilot Team recommends that NC State University switch to an open source LMS platform
based on Moodle. An open source philosophy, with provisions for the integration of
proprietary products, will enable us to better customize an LMS to meet the needs of the
campus community. As a tactical implementation of this philosophy, we recommend the
development a single LMS platform that ideally will incorporate the best features of both
Moodle and WolfWare, continuing to allow us to use both commercial (e.g. Elluminate,
Respondus) and open source additions to the system that meet the needs of the user
community.
1
Report of the Moodle Pilot Team, April 2009
Table of Contents
Introduction .................................................................................................................................................... 2
Why is NC State University looking at Open Source? ................................................................... 3
What is Moodle? ............................................................................................................................................ 4
What was involved in the Phase I Moodle Pilot at NC State? .................................................... 5
What are our thoughts from a Technical Perspective? ................................................................ 6
What are our thoughts from a Training and Support Perspective? ....................................... 7
How does Moodle Compare to what the Moodle Pilot Team already knows?................... 8
What about Faculty and Student Feedback from the Pilot? ....................................................... 9
What are the recommendations of the Moodle Pilot Team? ..................................................... 12
What are some potential next steps? ................................................................................................... 13
Tables
Table 1.1 Summary of Pilot Participation .......................................................................................... 5
Appendixes
Appendix A: Executive Summary of the TLTC Report (Fall 2008) on
Open Source Course Management Systems ...................................................................................... 14
Appendix B: Summary of the UNC Systems Schools LMS Usage and Plans,
as of March 2009 .......................................................................................................................................... 17
Appendix C: Guiding Principles for LMSs at NC State University ........................................... 22
Appendix D: What else has the Moodle Pilot Team looked at? ............................................... 23
Appendix E: NC State University – Peer Institution’s LMS Usage........................................... 25
Appendix F: LMS Report - Tool Comparison ................................................................................... 30
Appendix G: Assessment/Evaluation Information ....................................................................... 38
Appendix H: Learning Management System (LMS) Committee Structure .......................... 43
Introduction
After two NC State University fall 2007 Open Forums on the future of campus LMSs (see
http://wikis.lib.ncsu.edu/index.php/LMS_Strategy), the Moodle Pilot Team (a
collaboration between Instructional Technology and Administrative staff in CALS, CNR,
DELTA and Education) was charged with implementing an open source pilot using Moodle.
Meeting monthly for most of 2008 and into 2009, this team worked together to implement
the pilot and create the cross-unit collaboration needed to provide the various technical,
support and training elements for the pilot, creating adhoc workgroups to perform specific
pilot tasks and beginning initial planning for a long-term collaborative structure for LMS
implementation at NC State University. This report provides background information for
the Moodle Pilot and describes some of the various areas (toolset comparisons, technical,
training and support, migration, student and faculty feedback, etc.) explored during the
Moodle Pilot. This report also makes recommendations for future LMS planning based on
our experiences implementing Moodle at NC State University.
2
Report of the Moodle Pilot Team, April 2009
Why is NC State University looking at Open Source?
A Systems Perspective. In 2006, UNC President Erskine Bowles commissioned the
President’s Advisory Committee on Efficiency and Effectiveness (PACE). An outcome of
this committee, which was tasked to examine current activities in the UNC system and
make recommendations for improved use of resources, was the recommendation that open
source software for Course/Learning Management Systems (LMSs) be considered for use
when feasible and fully sufficient for university needs.
In response to PACE, the UNC Teaching and Learning with Technology Collaborative
(TLTC) led a pilot investigation comparing Moodle, Sakai and Blackboard (see
http://www.unctlt.org/projects/opensourcecms/index.htm). This investigation is well
documented in the Final Report of the UNC Pembroke Blackboard/Open Source Committee
report (see www.uncp.edu/doit/news/CMS/UNCP_CMS_report_fin.doc) and is also
summarized in Appendix A.
As of March 2009, a number of the UNC Schools are still considering open source options,
with several schools already fully committed to open source LMS software (e.g.
Appalachian State University – Moodle; UNC-Asheville – Moodle). Other pilots (mostly
Moodle pilots with some Sakai pilots) are well underway as the UNC System’s maintenance
license for Blackboard (both Campus Edition and Vista) expires in June 2011 (see Appendix
B). At this time, UNC schools will need to renegotiate a license with Blackboard and
identify the appropriate Blackboard version to meet their campus needs – or select an
alternative option. This is especially true for the campuses that use Blackboard’s Vista
product, as Blackboard Vista is an “end of life” product that will not be continued in an
“upgrade” model, and campuses will need to plan for a move to a new version of
Blackboard or to an alternative solution.
NC State’s Perspective. In 1997, NC State University internally developed and
implemented a Learning Management System (WolfWare) to support both face-to-face
(F2F) and distance education (DE) NC State University courses. When commercial LMSs
arrived on campus in 1999, NC State University selected and implemented WebCT Campus
Edition (CE) to complement and enhance the tools available in WolfWare (see our timeline
at: http://delta.ncsu.edu/projects/timeline/view.php?timeline=43).
Recognizing the need for a more enterprise level product as the usage of WebCT CE
increased to several hundred sections, NC State University conducted a series of campus
focus groups to determine user needs, evaluated vendor presentations, and ultimately
selected WebCT Vista in 2003. In 2004 - 2005, the campus underwent a major LMS
migration from WebCT CE to WebCT Vista, while continuing to use popular WolfWare tools
such as media lockers and file management, mailing lists and assignment submissions.
With the introduction of the more robust Vista toolset and enterprise architecture to
support more course sections, Vista course adoption went from 500+ courses in fall 2005
to 1500+ Vista sections in spring 2009. Although WolfWare usage has generally remained
steady over time (from a high of 2200+ sections in fall 2005 to 1818 sections in spring
2009), a number of faculty members have gradually elected to use only Vista. The NC State
3
Report of the Moodle Pilot Team, April 2009
community can view LMS usage reports at:
http://raleigh.delta.ncsu.edu/lms_reports/index.php (requires Unity ID to login).
The LMS landscape continued to change during this time, with Blackboard acquiring
WebCT in 2005/2006. Our campus, as did other WebCT campuses, adopted the language of
this new reality, referring to our product as “Blackboard Vista” per the product name
change. While a name change was not ultimately that difficult, NC State University had
previously enjoyed relatively good technical support from the WebCT organization,
whereas our initial support experiences from the Blackboard organization after the merger
were not as positive. Unfortunately, a degradation of support was noticeable as NC State
University struggled with a number of software problems in spring 2007 that we were
unable to resolve on our own, resulting in a frustrated user base around peak exam times.
DELTA staff continued to upgrade Vista with the recommended patches; however, new
upgrades (for example, the upgrade from 4.1.2 to 4.2.01 which caused an integration issue
with our registration and records database) were sometimes accompanied by
undocumented changes that resulted in additional technical issues.
Facing the reality of changing LMS markets (5-year cycles of evaluation), an end-of-life
product, a concerned user base, and support concerns with the Blackboard organization in
spring 2007, NC State University decided it was a good time to re-evaluate the LMS
environment for our campus. In two campus forums during fall 2007, the Vice Provost for
Distance Education and Learning Technology Applications (DELTA) outlined guiding
principles for LMS at NC State University (see Appendix C), and received encouragement
from the campus community to conduct a pilot of Moodle (selected as the open source
solution to try because of its “fit” with the campus infrastructure, see Appendix D). In
addition to a number of the UNC System schools using Moodle (see Appendix B), several of
our peer institutions are either using or piloting Moodle (though the use of LMSs at our
peer institutions is quite varied – see Appendix E).
What is Moodle?
Moodle is an open source Learning Management System (LMS) used by over 53,000
organizations in 208 different countries (http://moodle.org/stats/). Moodle allows for the
management, creation, and delivery of online components for courses by using features
similar to all commercial LMSs. For instructors, Moodle provides a "point-and-click"
environment to publish course material, manage a course, and engage with students. For
students, it provides a place to view course documents, submit assignments, take quizzes,
and engage with the instructor and fellow students. For administrators, it allows
management of system wide settings, integration with other on-campus systems, and the
ability to customize the software to meet our needs. Moodle Headquarters in Perth,
Australia, leads the overarching Moodle development, but contributors from all over the
world participate in further developing the open source system.
4
Report of the Moodle Pilot Team, April 2009
What was involved in the Phase I Moodle Pilot at NC State?
During spring 2008, staff members from the College of Agriculture and Life Sciences
(CALS), the College of Natural Resources (CNR), the College of Education (CED - who have
substantial experience in running their own Moodle installs for teacher training and
extension work over the past three years) and DELTA formed the initial Moodle Pilot
Implementation team, getting a Moodle server implemented and supporting a handful of
pilot faculty during spring and summer 2008, with pilot participation expanded during fall
2008. A summary of pilot participation is in Table 1.1.
Table 1.1 Summary of Pilot Participation
Term
Unique1
Number of
Instructors/Faculty Unique
with “active”
Students
2
sections
in “active”
sections
Spring & 11
328
Summer
2008
Number
of “active”
course
sections
Colleges
Participating
Additional
Information
13
CALS, CHASS,
CED and
PAMS
At least 4 course
sections were DE
course sections;
and there were at
least 28
additional noncourse related or
student-less
sections created
during this time
for course
development,
exploration,
testing or other
use, by both
faculty and staff
Courses included
face-to-face (F2F)
classes and
Distance
Education (DE)
course sections.
Additional noncourse sections
for development,
exploration,
testing or other
use by both
faculty and staff.
Courses include
face-to-face (F2F)
classes and
Distance
Fall 2008
29
2232
53
CALS, CHASS,
CED,
Engineering,
PAMS
Spring
2009
46
2482
72
Live courses
CALS
PAMS
Engineering
5
Report of the Moodle Pilot Team, April 2009
Education
Management
CHASS
Textiles
Design
Courses in
development
Vet Med
Education (DE)
course sections.
Additional noncourse sections
for development,
exploration,
testing or other
use by both
faculty and staff.
Unique: non-duplicated, instructors may have taught more than one course and students
may have enrolled in more than one Moodle course
2Active sessions: defined as having updated class rolls via registration and records AND
evidence of student logins and student activity in the course
1
During the Phase I pilot implementation of Moodle (from spring 2008 – spring 2009), the
Moodle Pilot Team considered Moodle from several viewpoints, including technical
considerations, training and support considerations, tool comparisons, and faculty and
student feedback.
What are our thoughts from a Technical Perspective?
From a technical implementation perspective, the Moodle Pilot Team note that Moodle has
run for three semesters at about ten percent of the scale of the other LMSs on campus,
Blackboard Vista and WolfWare. At this scale, Moodle has been robust and readily
modifiable. The Moodle Pilot Team believes that Moodle's infrastructure is better suited to
the campus needs than the alternatives (e.g. Sakai), as Moodle’s Linux, Apache, MySQL and
PHP (LAMP) infrastructures have proven to be resilient, open for modification, fairly easy
to troubleshoot and scalable during the pilot phase, and there is much campus expertise in
LAMP development. Technical issues with Moodle have been relatively minor and readily
addressable, and issues have been resolved internally with support from the wider
community of Moodle administrators world-wide. The Moodle Pilot Team considers
Moodle to be a flexible system which can accommodate NC State University extension and
resource activities as well as traditional academic activities.
Some examples of successful technical integrations during the pilot phase have included:
• Moodle integration with our Registration and Records system to create automated
class-role and student account synchronization.
• Creation of a custom course activity to distribute Library resources and course
tools to students.
• Support added within Moodle for Mediasite, Elluminate and EQUELLA.
• A custom plugin for non-credit certificate courses added.
6
Report of the Moodle Pilot Team, April 2009
What are our thoughts from a Training and Support Perspective?
Both training and supporting faculty in using LMSs are critical to the effective use of these
systems. There are several issues that merit consideration when NC State moves from
Blackboard Vista to Moodle (or any system), including the design philosophy for Moodle
courses, the amount and types of training provided, other resources that are available for
support beyond DELTA and NC State University, and content migration from one system to
the other.
Design & Accessibility Considerations. Blackboard Vista and Moodle both offer many of
the same tools for creating and delivering content, but there are differences in the ways
courses are created and managed. Instructors and designers will notice that selecting and
organizing tools and content is very different from what they are used to in Blackboard
Vista, and students who are experienced Blackboard Vista users will also recognize a
difference in the overall layout and navigational features from their perspective. Because
of the interface differences, user support and training will be important features of any
migration strategy.
Preliminary evaluations of Moodle by an accessibility and usability workgroup reporting to
the Moodle Pilot team, as well as general feedback received from the Moodle user
community, suggest that Moodle has several accessibility features, and is, in general, quite
accessible and usable--much more so than our current version of Blackboard Vista from an
accessibility perspective. The Moodle Accessibility Workgroup is planning on conducting a
formal usability and functional accessibility test of Moodle and its features later this year.
Training and Documentation. Blackboard Vista users will need to be retrained in how to
use Moodle (or any new LMS) effectively. Many of the concepts of Vista are applicable to
other LMSs, but there are many specifics of Moodle that are markedly different, and faculty
will need support in learning about these changes.
Blackboard produces training material for Vista which NC State University can and does
use, such as training documents and online tutorials. In addition, DELTA has also
developed our own in-house support structure through workshops, online documentation,
and Instructional House Calls. Training documentation in the Moodle community is open
source and community created. The main sources are the Moodle Docs wiki to which
anyone can contribute and also the Moodle Teacher Manual, an open source book on how
to use Moodle. There is also a free downloadable book called "Using Moodle."
Creating training resources for Moodle can draw heavily on our current training model for
Blackboard Vista: offering workshops, house calls, and online training. Currently DELTA
teaches a 3 hour introductory workshop to Moodle which is also available online. Several
colleges also offer training to their faculty on Moodle. The greatest challenge will be in
utilizing resources across the university and not just in DELTA to create, share and deliver
these training materials. In creating our own training materials, the NC State University
community can draw on the Moodle community's library of training resources.
7
Report of the Moodle Pilot Team, April 2009
User Support. Support issues for Moodle (or any new LMS) will in some ways be similar to
support issues for Blackboard Vista, which DELTA, OIT and the colleges already have an
infrastructure in place to handle. DELTA’s current support model utilizes the OIT Help
Desk as Tier 1 support and the DELTA LearnTech as Tier 2 support for faculty. Technical
and instructional support resources may also exist at the college level to complement these
central services and are utilized as appropriate.
One of the differences between supporting Blackboard Vista and supporting Moodle will be
the reliance on an open community (vs. a company) for more complex support issues.
There is no "official" Moodle support mechanism as it is an open source product. Instead,
there is an active community (http://www.moodle.org) of developers, trainers, and
support staff from various institutions and the Moodle organization itself that participate in
online discussion forums and software development. Complex issues are often handled by
engaging in these forums or by creating solutions yourself and contributing them back to
the larger community. Blackboard frequently charges for support for complex issues that
do not fall under maintenance contracts. If external support services are needed for
Moodle, there are several companies who do Moodle work-for-hire with a similar but
potentially more competitive model.
Migration. No matter what LMS is adopted after Blackboard Vista’s expiration, there will
need to be a major effort in migrating content from one system to another. Blackboard's
new product, Blackboard NG, does not currently provide a well defined migration path for
automatically moving courses from Blackboard Vista into the new product, and as of early
2009, there was no completely automatic way to easily export content from Blackboard
Vista into Moodle. A well-orchestrated LMS migration strategy will be necessary, and the
Moodle Pilot Team recommends that it be similar in design as the course migration DELTA
completed from WebCT CE to Vista during 2003-2004. (This program was called reVAMP,
the Vista Assisted Migration Program). A key difference will be in the scale of the migration
this time. Through reVAMP, DELTA staff migrated approximately 500 courses. In contrast,
there are over 3,000 sections offered throughout the academic year in Vista (1500+ for
spring 2008). Some of those sections are duplicates, being offered in two or semesters
during the year, but nonetheless, a migration effort will be much greater in size and will
require significant human resources. Although some for-fee tools are available that
facilitate mass migration of content from one system to another, our experience with these
tools has been that a migration will require a degree of manual processing for each course.
DELTA and our partners can draw heavily on our successful and positively-received
migration efforts from CE to Vista to plan a large scale move from Vista to a new system,
but DELTA expects this migration will take three semesters (minimum) and a significant
allocation of resources.
How does Moodle Compare to what the Moodle Pilot Team already knows?
WolfWare. WolfWare is an LMS that is designed primarily for people who want to be able
to publish their course as a website, utilizing other design tools such as Dreamweaver to
create HTML pages. WolfWare does come with tools that allow for discussion boards,
assignment submissions and a grade book. It also allows levels of granularity of access to
8
Report of the Moodle Pilot Team, April 2009
different parts of the site. A WolfWare feature that is unique among our current LMSs (and
also very popular) is the ability to make web pages fully public without requiring a
password. To effectively use WolfWare, the instructor must be comfortable creating HTML
documents and using FTP or WebDAV to transfer files.
Blackboard Vista. Vista is an LMS that is designed as a point-and-click system for
designing and managing online components of classes. Vista offers a wide set of robust
tools for publishing content and promoting classroom interaction, including forums, a
gradebook, a calendar, announcements, assignment submission, and more. Vista is
designed as a system to allow the publishing and managing a course without a great deal of
technical expertise.
Moodle. Like Vista, Moodle does not require an instructor to have extensive technical skills.
Moodle is also very similar to Vista in terms of functionality and toolsets; however, the
tools are not necessarily identical in their implementation. Since Moodle is an open source
product, the campus has a significant flexibility in adding and customizing tools.
See Appendix F for a table comparing tools at a macro level across these systems, as well as
further detail about how specific tools that are heavily utilized (determined per a tool
analysis in 2006/2007) compare across these systems – such as the discussion forums,
quizzes, gradebook, and file storage/management tools. In addition, the report done by
Idaho State University during their evaluation of Moodle, Sakai and WebCT has detailed
rubrics comparing tool functionality and features, and may be worth exploring for those
interested in even more detail. See
http://www.isu.edu/itrc/resources/LMS_FINAL_REPORT_MOODLE.pdf. Louisiana State
University’s final report on their decision to use Moodle also has information that may be
useful. See
http://www.immagic.com/eLibrary/ARCHIVES/GENERAL/LSU_US/L070914R.pdf.
What about Faculty and Student Feedback from the Pilot?
During the 2008 pilot phases, several surveys (IRB approval #246-08-6) were distributed
to both faculty and student participants in the pilot at two different points, one for the
spring and summer pilot faculty and students, and the second time for the fall pilot
students. While the sample populations surveyed were relatively small and responses
cannot be generalized to a larger population, several observations can be made from the
pilot feedback (see Appendix G for more detail and links to more specific assessment
information).
•
Faculty Feedback: The majority of instructors who responded to the survey to the
Moodle pilot survey in both the spring and fall generally indicated that they liked
Moodle better than Blackboard Vista. Although a number of improvements were
suggested for Moodle (e.g. gradebook, forum features, wiki, structure/organization
of course entry page), faculty members expressed positive feedback towards
Moodle’s functionality, reliability, and performance. This position appeared to
shared by both Face-to-Face and Distance Education instructors. The fall pilot
9
Report of the Moodle Pilot Team, April 2009
•
•
•
faculty were asked specifically if NC State University should make the switch to
Moodle; 68% of the fall respondents were in favor of adopting Moodle, while
approximately 22% of faculty indicated NC State University should consider
Blackboard’s new product. Note: Since the participating pilot faculty were a selfselecting population, they may not be representative of the typical NC State
instructor). The total number of faculty that completed the pilot surveys during both
survey periods was 32. All faculty (non-random sample) who participated in the pilot
received the survey. (Response rate varied among faculty surveys between
spring/summer and fall assessment points, see Appendix G for additional information).
Students’ Feedback: Students indicated LMS preferences for Blackboard or Moodle
based on various reasons (e.g. prior experience, software interface
design/navigation, choices the faculty made in setting up the system). In all of the
pilot studies involving students, Moodle was favorably viewed by students overall,
with a larger proportion of students responding favorably than unfavorably
regarding their Moodle experience. In the two general student surveys (one
spring/summer and one fall), around 50% indicated the Moodle was Better or Much
Better when compared to their experiences with Blackboard Vista, and around 30%
of students in the general student survey indicated that, to them, the systems were
about the same. Around 50% of the students indicated it doesn’t matter what the
LMS is at NC State University as long as it is stable and reliable, with an additional
28% indicating NC State University should switch to Moodle. Note: The survey was
sent to all students (non-random sample). The total number of students that
completed the pilot surveys during both survey periods was 428 (Response rate varied
among student surveys between spring/summer and fall assessment points, see
Appendix G for additional information).
Qualitative comments from the pilot respondents indicated that the ease of use and
cross-browser functionality are features that students generally like about Moodle
(though some students indicated that the Moodle home page is cluttered – which
may be related to both the tool itself and the instructor’s design decisions).
Qualitative comments did provide a glimpse of the interplay between the tools
themselves and the instructional design/teaching choices for either system; as one
student expressed: “Moodle to me was a lot like Blackboard, as long as the teachers
posted the assignments then everything is alright.”
Qualitative comments suggested that there are tools and/or design features in both
Blackboard Vista and Moodle that both students and faculty like and dislike.
Opposing comments suggest there will likely be both user likes and dislikes which
will need to be addressed in any LMS adoption at NC State. No LMS will be the
“perfect” choice for everyone, and there are certainly functional issues to address
with Moodle. NC State designers and developers should look at the qualitative
comments for issues that should be addressed with any Moodle implementation and
use as a guide to improve the user experience as part of the implementation. From
the faculty perspective, the gradebook tool was one of the most frequently cited
tools needing improvement (according to the community docs, gradebook
improvements are on their way, see
http://docs.moodle.org/en/Development:Gradebook_improvements_in_Moodle_1.9
10
Report of the Moodle Pilot Team, April 2009
•
•
.5). From the student perspective, the new interface was noted – sometimes
negatively, and sometimes as a plus.
Change management will be an issue for all faculty and students who are used to
using another system (e.g. as one Moodle student noted: “it was a learning curve but
I don't think this would be negative necessarily. Any program that someone has not
used would take a few times using it to become familiar with the system”). This will
be true no matter what LMS is chosen.
These findings are similar to reports presented by other schools:
o Data from Idaho State University’s selection of Moodle in Spring 2007 noted
a slight preference by pilot student respondents (n=108) using Moodle, and
noted that the pilot faculty (n=15) generally liked their experiences with
Moodle; though there were a number of comments that suggested that there
is a learning curve in using Moodle (or any system) when you are already
familiar with another one. See
http://www.isu.edu/itrc/resources/LMS_FINAL_REPORT_MOODLE.pdf.
o A discussion with representatives from Louisiana State University at the
2008 Educause conference revealed similar findings with faculty and student
satisfaction, though a report on faculty/student feedback is not available at
this time.
o UNC System Assessments:


In initial focus groups, the faculty (n=10) surveyed in UNC Charlotte's
Moodle pilot overwhelmingly preferred using Moodle to Blackboard.
In particular, these faculty members felt that Moodle's ease of use and
interface were superior to those in Blackboard. See
http://www.lmseval.uncc.edu/index.php?option=com_frontpage&Ite
mid=1. In additional surveys, students (n=313) indicated a preference
for Moodle and 35 pilot faculty indicated a preference for Moodle. See
http://www.lmseval.uncc.edu/index.php?option=com_docman&task=
cat_view&Itemid=59&gid=43&orderby=dmdatecounter&ascdesc=DE
SC.
UNC Pembroke, UNC Asheville, and Appalachian State University have
reported similar findings. See http://www.uncp.edu/doit/ for
Pembroke’s online report. UNC Asheville and Appalachian State
University have already migrated to Moodle.
11
Report of the Moodle Pilot Team, April 2009
What are the recommendations of the Moodle Pilot Team?
Any recommendation for LMS direction must consider the following:
(1) NC State University has to migrate from Vista to something else. Vista is an end-oflife product. As a campus, we will need to move to something else, either
Blackboard’s new offering (NG), Moodle, or some other LMS. Any new LMS system,
including the completely redesigned Blackboard product, will require faculty and
students to adapt to changes in the user interface.
(2) Vista support from Blackboard will end. We have a support contract for Vista
through the UNC system that runs until June 2011. While Blackboard has not yet
determined an end date for support of Vista, at some point after July 2011, all Vista
support will ultimately end under the current contract.
(3) WolfWare, as we know it now, will stop working. WolfWare is highly dependent on
Sun’s Solaris 8 operating system and the Sybase database. Solaris 8 is incompatible
with any new servers NC State University/DELTA might buy and NC State University
is migrating away from Sybase. WolfWare would require a major revision to
maintain functionality. DELTA does have plans in place to maintain current
WolfWare functionality as an interim solution, but that will only continue to work
for a few more years.
(4) LMS technology evolves. Though the LMS landscape has continuously changed over
the past 10 years, we have enjoyed relative stability with our own system,
WolfWare. As part of the normal evaluation of available tools and campus needs,
DELTA has implemented two commercial LMS’s during this time: WebCT CE starting
in 1999, and a migration to WebCT Vista in 2004-2005. The current examination of
LMS’s, including the pilot of Moodle, follows this cycle.
Investigations into the current LMS landscape have included looking at both proprietary
and open source systems used in the UNC system schools as well as national peer
institutions of North Carolina State University (via conversations with colleagues at other
schools, looking at various products at conferences and/or in-house, etc.). In light of
positive experiences with the Moodle pilot (including technical, user support and training,
and faculty and student feedback), and in consideration of the aforementioned factual
information, the Moodle Pilot Team recommends the following:
a. Follow an open source philosophy for the NC State LMS. Moodle contains the
LMS tools campus uses, has the ability to add additional tools to meet our
needs, and is the open source LMS that best fits our campus LAMP
infrastructure and developer expertise base.
i. Start planning a migration to an NC State branded LMS: WolfWare
(powered by Moodle)
1. Will be a phased approach as systems are integrated.
2. Will change over time. Implementation will be dynamic in
nature, and constructive changes will be readily proposed at all
stages of the process
12
Report of the Moodle Pilot Team, April 2009
3. Will be informed by the LMS committees (see Appendix H) as
the NC State University community works collaboratively to
get to where we want to go – these committees will guide
development and implementation efforts. The committees
will shape what the LMS looks like.
4. Revisit timeline as necessitated by changes to the NC State
branded LMS approved by the LMS committees.
ii. Keep Vista going “as is” through the remainder of our support
agreement in June 2011 (to include needed updates for stability and
reliability) while a migration plan is put into place for getting users to
an NC State branded LMS.
iii. Keep current version of WolfWare going “as is” while DELTA (and
others in the NC State Community) investigates integrations, and
rebrand to WolfWare Classic to differentiate.
b. Use third party tools (open source and commercial) and tools the NC State
Community develops to augment existing functionality (examples now
include Elluminate & Respondus).
c. Continue to keep an eye on the emerging LMS landscape, including an
awareness of what our peer institutions and other institutions are
successfully utilizing. Revisit our decisions as needed, collaboratively.
What are some potential next steps?
Next steps include, but are not limited to, the following:
•
•
•
•
•
•
•
Discuss recommendations as a campus community, identify any problems or "show
stoppers" at this time
Share information (report, recommendations) to a wide campus audience
Incorporate feedback from the DELTA advisory committee
Solicit additional feedback through various campus channels (Teaching and
Learning with Technology Roundtable, other groups as identified on the LMS
Committee Structure/Chart of Influence – see Appendix H)
o Would like as much feedback as possible by June 1, 2009
Announce next steps by July 1, 2009
Dissolve the Moodle Pilot Team, solidify the new LMS Committee structure and kickoff all committees with a clear charge from Steering during Summer 2009
Draft a detailed migration strategy during Summer 2009
Contact information:
For comments, questions, thoughts, etc., please email Dr. Marty Dulberg, Senior
Coordinator, DELTA, and LMS Project Lead: marty_dulberg@ncsu.edu
13
Report of the Moodle Pilot Team, April 2009
Appendix A: Executive Summary of the TLTC Report (Fall 2008) on Open Source
Course Management Systems, written by the TLT Collaborative
(http://www.unctlt.org/).
CMS Exec Summary
Executive Summary of Report on Open Source Course
Management Systems
(as requested in the PACE Report – Information Technology Idea 11)
The purpose of the TLT Collaborative’s (TLTC’s) investigation of open source course
management systems (CMSs) is to determine the viability for instruction and the potential
cost savings for UNC campuses to move to an open source solution (notably Moodle or
Sakai) from the commercial Blackboard CMS (Learning Suite or Vista) currently utilized.
Where the Campuses Are Now: Fall 2008 Production CMSs within UNC
The following are the production CMSs on UNC campuses as of Fall 2008:
• Moodle (ASU, UNCA, NCSSM)
• Blackboard Vista (NCSU, UNCC, UNCW, WCU)
• Blackboard Learning Suite (remaining ten campuses)
Campus Evaluations
In 2007-08, the following campuses formed faculty and staff committees to evaluate open
source solutions:
• ASU elected to migrate to Moodle instead of migrating to Blackboard Vista
• ECU is completing an in-depth study of both Moodle and Sakai
• NCSU is continuing an investigation of Moodle, and planning to make a decision in 200809
• UNC-CH, after an initial study, is piloting Sakai campus-wide in 2008-09
• WCU, after conducting an open source study, has decided to remain with Blackboard
Vista
In 2008-09, the following campuses will run production courses in Moodle and/or Sakai on
pilot production facilities provided by the TLTC: ECSU, NCSA, UNCC, UNCP, UNCW, and one
department at UNCG.
Functional Viability for Instruction
Detailed spreadsheets regarding CMS functional features were completed by vendors,
service providers, and TLTC staff. While there are certain differences in functionality
among the four CMSs examined, analysis of the spreadsheets - as well as the fact that a
number of universities nation-wide are utilizing open source CMSs - demonstrates that
both Moodle and Sakai are viable alternatives to commercial CMSs. The evaluation results
14
Report of the Moodle Pilot Team, April 2009
from UNC faculty members who taught in Moodle or Sakai this year also demonstrate that
open source solutions are a viable alternative to Blackboard.
Operating Costs: Total Cost of Ownership (TCO)
A summary of cost information follows:
• Based on an analysis of data from 12 UNC campuses, current TCO for the Blackboard
CMSs across UNC is approximately $39 per FTE. (This TCO analysis includes all hardware,
software, and staffing costs.)
• The Blackboard CMS software (licensing) costs are approximately $7 per FTE across
UNC, with smaller campuses paying a higher amount per FTE. (Range reported: $2.25 to
$31.)
• The average TCO for six open source universities interviewed for this report is $25 per
FTE (Range reported: $10 to $50.) (Licensing fees, of course, are $0.) To more accurately
determine the cost per FTE, additional data needs to be collected as more universities run
open source solutions in full production.
• Regarding costs of transition from commercial to open source CMSs, the universities
interviewed have not tracked these costs carefully. (The TLTC is collecting this data as UNC
campuses transition.)
Preliminary Conclusions
Remembering that eight UNC campuses will be completing/conducting in-depth open
source investigations in 2008-09, and therefore will be providing extensive and important
additional data which the TLTC will collect, the following preliminary conclusions may be
drawn to date:
• Moodle and Sakai are now viable open source alternatives to Blackboard, while
individual campuses may determine that particular feature differences make them more or
less attractive than Blackboard.
• Initial decisions by UNC campuses (for example, ASU electing to transition to open
source and WCU not) demonstrate that a campus’ selection is based on many campusspecific situations and criteria, including faculty preferences.
• UNC campuses can save, on average, $7/FTE in software costs after transitioning, with
the smaller campuses realizing the larger savings. (Changeover and development costs,
however, may offset part of these savings and are still being investigated.)
• There are major differences in the underlying architecture of Moodle and Sakai that lead
to important support considerations as a campus makes a selection/decision.
2008-09 Investigations
The TLTC will collect additional data from the following studies in 2008-09:
•
•
•
The eight UNC campuses completing/conducting open source studies next year
Common-hosting investigations of open source CMSs
The TLTC is project managing, beginning June 2008, the implementation of a
commonly-hosted production-level instance of Moodle at MCNC for several NCCCS
campuses. This project will provide important data on the cost, viability, and
potential advantages of a large commonly-hosted Moodle installation. (UNC
15
•
Report of the Moodle Pilot Team, April 2009
campuses have expressed interest in “buying-in” to the services of this facility.)
The pilot production instances of Moodle and Sakai that the TLTC is providing to the
six campuses mentioned previously will provide information on cost and operability
for a commonly-hosted instance used by UNC campuses.
16
Report of the Moodle Pilot Team, April 2009
Appendix B: Summary of the UNC Systems Schools LMS Usage and Plans, as of March 2009
Last updated March 5, 2009, dtp
Name of School
Currently Using/
Official Campus LMS
Fully committed to Open Source LMS
Appalachian
Moodle, version 1.9.2+
State University
with custom
modifications and
addition of Zend
Framework.
UNC Asheville
Moodle
Looking at
Notes
Integration of Moodle
with Sakai/
OpenSourcePortfolio
for portfolio and
program assessment
capabilities.
ASU transitioned from WebCT Campus Edition to Moodle (locally branded as
AsULearn) between Summer 2008 and Fall 2008-9, with migration fully completed
in November, 2009 at version 1.9.2+. ASU has successfully and easily integrated
Moodle with Banner (via Luminis Data Interchange) using openly available plugins
and anticipate moving from a hosted Moodle site to a campus-based installation in
summer 2009 with a conversion to the Oracle database in Fall 2009. We have
developed web services based on the Zend framework for auto-enrollment
(external to Banner) of prospective students into math placement exams and for
exporting their scores to Banner, developed a bulk Delete Course tool for cleanup,
modified standard media "filters" for protected (playable only from within
Moodle) external media streams, developed "quicklinks" selectable resource tool,
and developed a faculty tool to download rolls and other student data (e.g. Banner
IDs and other non-confidential profile information). General reception by faculty
and students has been very positive and we now build sites for every course
section (3300 regular and roughly 1700 single-student sites - internships, applied
music, etc.), some 1100-1300 of the regular sections being active courses (with
some dozens cross-listed) and an additional 75 sites for various workgroups,
taskforce, and committee collaborations.
UNCA implemented Moodle completely for the Summer 2008 sessions. UNCA is
being hosted by Remote Learner (current version is 1.9.3+). Since the transition
from WebCT CE to Moodle, LMS use has increased 2.5 fold. Some of that increase
is due to cost saving efforts regarding printing and such, but a large part is due to
the perceived utility of Moodle versus WebCT. The increase in users has come
from two groups: 1) those that tried WebCT and hated it, but now embrace
Moodle enthusiastically, 2) those that never would have considered doing
anything online, but now find it a helpful addition to their face-to-face tool set.
UNCA is looking to add the Mahara ePortfolio later this semester for testing. UNCA
has less than 10 wholly online courses, and their use of Moodle is to supplement
face-to-face instruction with online content, quizzes, forums, wikis, etc. A long
term endeavor is in the direction of creating true hybrid courses that will replace
some face-to-face content/activities with online content/activities to reduce inclass time but not eliminate it all together.
Extensions to Moodle
(e.g. Mahara
ePortfolio)
17
Report of the Moodle Pilot Team, April 2009
Piloting or Partially Using (e.g. for a program or area) an Open Source LMS
At ECU, the official LMS is Blackboard. In addition to Blackboard, ECU has been
East Carolina
Blackboard
Moodle (for faculty
using Moodle since 2006. Usage of Moodle started as pilot project, with an
University
who want to use it).
evaluation done to see if ECU should adopt Moodle as the LMS for ECU. However,
Hope to do another
ECU thinks it is still a little too early to consider adopting Moodle as the official
evaluation in 2010, but LMS. Reasons include:
no current plans to
(1) Campus integration challenges to work out (link with Banner, LDAP
authentication with the AD infrastructure)
“officially” change.
(2) Migration issues (need a conversion tool before considering moving
courses from Blackboard to Moodle)
(3) Local expertise for hosting (ECU has just moved the hosting of Blackboard
back to ECU. Our central ITCS is just getting comfortable with hosting
Blackboard).
(4) Faculty training & support (the faculty are comfortable with Blackboard).
North Carolina
School of
Science and
Math
Residential
program:
Moodle
Online program:
Agilix GoCourse
Outreach IVC
program: faculty
can choose
between either.
No current plans to change.
However, Moodle is still supported for any faculty member who would like to use
it. ECU has a dozen or so "Moodle fans" in that don't want to go back to
Blackboard. ECU will keep running Moodle, along with Blackboard, for now, and
will probably do another formal evaluation in 2010.
NCSSM currently uses Moodle and Agilix GoCourse for LMS solutions. No changes
are anticipated in either LMS system in the near future. The NCSSM residential
program moved from Blackboard to Moodle in 2006 with a one year transition
period when both systems were available to faculty; Blackboard was taken offline
in June, 2007. The decision was made based on the cost of Blackboard not
equaling the difference in benefits/services between Blackboard and an open
source solution, based on the needs of the boarding school program. To
supplement any deficiencies with Moodle, faculty use other free and fee-based
webtools if something within Moodle doesn't meet their needs (such as the testing
tool, wiki, and gradebook). The campus ITS handled installation and
implementation of Moodle, and had to create a custom solution to connect Moodle
to the campus student information system. Because of NCSSM’s small size, in
addition to hosting courses, Moodle is used for faculty committee work,
supplemental academic activities, and software distribution. With the launch of
the online program in 2008, it was determined the needs of the online program
differed from the residential program for a LMS solution. NCSSM Online chose
Agilix GoCourse as an LMS because it offered dedicated support, offline functional
capability, unique assessment capabilities, and self-service content creation tools
not available in Moodle and at a lower cost than Blackboard or Angel. Moodle is
18
Report of the Moodle Pilot Team, April 2009
North Carolina
State
University
Blackboard
Vista 4.2.3
WolfWare
(homegrown
system)
Piloting Moodle as a
possible alternative to
Blackboard Vista.
Blackboard
Vista
Piloting Moodle as a
possible alternative to
Blackboard Vista.
UNC – Chapel
Hill
Blackboard 6.3
Running Sakai 2.5 in Pilot
UNC Greensboro
Blackboard 8.02
UNC - Pembroke
Blackboard 7.x
UNC Wilmington
Blackboard Vista
4.2.3
Piloting Moodle for iSchool
and a small number of other
interested
groups/departments. No
current plans to change
official LMS.
Considering Moodle as the
centrally supported LMS.
Have a few faculty using
Moodle on their own at this
time.
Not planning any changes at
this time but are conducting
a pilot of 3 courses in
Moodle for Spring 09.
UNC - Charlotte
hosted at NCSSM and administered and managed by an engineering faculty
member; Agilix GoCourse is hosted and supported by the vendor.
NC State began piloting Moodle in spring 2008. In spring 2009, NC State has 47
unique faculty teaching 66 course sections in Moodle involving more than 2300
students. The Moodle Pilot Implementation Team is preparing a Phase I pilot
report to share with the campus in April, 2009. A Phase II pilot is expected to
continue at least through the end of 2009 to explore questions not answered in the
Phase I pilot. See http://wikis.lib.ncsu.edu/index.php/LMS_Strategy for
information.
UNCC is continuing to pilot Moodle. UNCC has approximately 30 faculty with
about 1500 students participating in phase 2 of the pilot. Last semester they had
15 faculty with about 800 students. Results have been very positive thus far. An
LMS evaluation committee comprised of faculty, CTL staff and distributed IT
who've been meeting for about a year, are looking at options, surveying, meeting
with various groups on Campus, etc. UNCC’s website is at:
http://www.lmseval.uncc.edu
UNC-CH is piloting Sakai, and is preparing a Fall 2008 interim reporting on their
findings, which will be shared at: http://www.unc.edu/sakaipilot/. Survey
instruments and results will also be shared at this site.
UNCG continues to support Blackboard as the official LMS and has no plans to
move away from it; UNCG is doing a Moodle pilot with their Division of Continual
Learning (iSchool program) and possibly a few others for Fall 09, who have an
interest in exploring Moodle as a possible alternative to Blackboard. Pilot may be
expanded in Spring 10 depending on Fall 09 results.
At UNCP, a committee charged with comparing LMS/CMS has recommended
Moodle over Blackboard. Recently, the Division of Information Technology
responded to this report, and their response can be viewed online at
http://www.uncp.edu/doit/.
UNCW is currently using Blackboard Vista hosted by Blackboard. The campus is
part of a consortium with WCU and UNCC. UNCW faculty members experienced
some issues with Bb Vista in 2006 and since have expressed interest in expanding
online programs and offering more courses in Blackboard each year. The faculty
expresses interest in using Blackboard more each semester. 24/7 support from
19
Report of the Moodle Pilot Team, April 2009
Presidium has been added to our support services to all faculty staff and students
for use with Blackboard or Wimba. 3 experienced Blackboard users are trying
Moodle with courses that they taught previously in Blackboard and are giving
feedback on what their experience has been.
Blackboard Consortium Schools – Evaluate Other Possibilities Later
Elizabeth City
State
University
Blackboard
Learning
Suite 8.0
No current plans to change.
Winston Salem
State
University
Blackboard
Learning Suite
8.0
No current plans to change,
though open source
conversations are on the
table.
North Carolina
Central University
Evaluate Later
Blackboard 7.3
Planned upgrade to
Blackboard 8.0 in May
2009. No formal plan to
change to an open source
LMS; however, several
faculty members are using
Moodle on their own.
ECCU is using Blackboard (via outsourced hosting), after joining a 4 campus
consortium last year with North Carolina Central University, Winston Salem State
University, and Fayetteville State University. Open source discussions are
occurring, perhaps because of the concern around the cost we are paying for
Blackboard. Several issues are of concern related to open source, including:
(1) ECCU is not sure about are how similar Moodle and Sakai are to
Blackboard, and wouldn't want to switch to something that was not
comparable
(2) The learning curve of having to retrain everyone not to mention convert
courses over to a new LMS is a barrier.
(3) Campus integration would be a challenge, as they just integrated
Blackboard with Banner and it is working nicely, and they are not sure
about the issues with integrating the open source options with banner.
(4) Personnel to support open source – they don't have the personnel on our
campus to tweak the code to make those systems do what we want them
to do like some of the larger campuses.
Preliminary discussions have begun with three other schools in a UNC-GA
Blackboard consortium (Winston Salem State University, Fayetteville State
University, North Carolina Central University, and Elizabeth City State University)
to explore the merits of moving toward an open source solution (most likely
MOODLE) within an 18-24 month time frame.
Blackboard has been the official LMS at NCCU since 1999. At this time there are no
formal plans to move away from Blackboard, due to migration issues and the
amount of training required for faculty and staff. However, NCCU has been a part
of preliminary discussions with UNC-GA and other schools in the system (Winston
Salem State University, Fayetteville State University, and Elizabeth City State
University) to explore the possibility of an open source solution in the near future.
Faculty members in BRITE are piloting Moodle as a possible alternative to
Blackboard.
20
Western Carolina
University
Blackboard Vista
Report of the Moodle Pilot Team, April 2009
Waiting . . . will need to
evaluate closer to the Vista
license renewal time.
WCU will be hiring a new CIO and will wait until that person gets on board to
establish a future direction. At this point, Western does not have enough staff to
implement an Open Source LMS. Based on the recommendation of a broadlyrepresentative LMS Task Force working in 2008, WCU faculty generally report
being satisfied with Blackboard Vista. Their migration from WebCT CE to Vista was
very “painful,” and thus their faculty committee indicated migration fatigue was
too much to consider anything else at this time, so no piloting of open source
systems is currently being done.
Waiting on Response
North Carolina
A&T University
North Carolina
School of the
Arts
Fayetteville State
University
Common themes in conversations with the various schools in considering moving (or choosing not to move) to a different
LMS:
(a) Faculty training & support for a new system is a tremendous effort – are we ready for that?
(b) Course migration from any system to another is much work. Some schools are still recovering from their last migration. There is
“change fatigue” in that previous migrations have been tremendous amounts of work, and if things are working, even if they aren’t
perfect, are we ready to change?
(c) Campus integrations require staff/resource availability to implement (e.g. with Banner or other SIS system), and there are
resource concerns.
(d) LMS feature concerns – no one wants to lose functionality and there is concern that Moodle (and perhaps other open source
systems) don’t have the same level of functionality.
21
Report of the Moodle Pilot Team, April 2009
Appendix C: Guiding Principles for LMSs at NC State University
1.
2.
3.
4.
5.
6.
Stability and reliability
Quality and usability
Future growth and flexibility
Control our destiny
Participate with peers as a community of developers and stakeholders
Potential collaboration with other institutions
22
Report of the Moodle Pilot Team, April 2009
Appendix D: What else has the Moodle Pilot Team looked at?
•
•
•
DELTA continually looks at new technologies as they become available
We participated in the UNC-TLT C/LMS Open Source Pilot Project
We have built rubrics to evaluate and compare C/LMSs.
At least two of our peer institutions, Georgia Tech and the University of California-Davis, use
Sakai, and one of our UNC-System colleagues, UNC Chapel Hill, is piloting SAKAI (What does
the Moodle Pilot Team know about SAKAI and how does that influence our thinking?
•
•
•
•
•
•
NC State was one of the founding members of the Open Knowledge Initiative (OKI), a
predecessor of SAKAI.
We participated in the SAKAI conference for approximately three years
SAKAI is an open project (though, using an unusual, pay-to-play model)
SAKAI is written in Java (like Vista), in contrast, Moodle is in PHP. NC State has much
more PHP experience in-house.
SAKAI runs on "big iron" servers (like Vista). NC State has more experience running the
kind of servers Moodle runs on, as they are very similar to the WolfWare server
footprint.
SAKAI might satisfy our desire to use a more open platform, but would not address the
Java issue it has in common with Vista.
At least two of our peer institutions (Michigan State and Penn State) really like ANGEL. What
does the Moodle Pilot Team know about ANGEL and how does that influence our thinking?
•
•
•
•
ANGEL began as a "home brew" solution for Purdue University Indianapolis, and has
since grown into one of the more positively reviewed LMS solutions available. Used by
other large-enrollment universities (like Penn State), ANGEL can clearly handle the
volume we could expect to see here at NC State.
Having partnerships with Elluminate, Turnitin and Wimba means ANGEL already has
integrated support for a number of the other technologies being requested by our
faculty. Its assessment system is currently the industry leader.
Similar to Vista, ANGEL is proprietary software, and runs largely on Java. While it
features open APIs for module creation, the base code itself is, for all intents and
purposes, unchangeable. It runs on a similar infrastructure to what is already in place
for our Vista installation.
While it appears to be a very positive experience for the universities who have switched
to ANGEL, it is still proprietary software that leaves us without the power and range of
motion inherent to an open platform.
23
Report of the Moodle Pilot Team, April 2009
A number of our peer institutions as well as other UNC campuses are using Blackboard
products. What does the Moodle Pilot Team know about Blackboard Vista and Blackboard
Next Generation (NG) and how does that influence our thinking?
•
•
•
•
•
•
•
Blackboard Vista is a terminal product, at some point, it will soon be classified end-oflife (though we don’t know the exact date)
Blackboard NG is their next generation LMS, but is built on the foundation of
Blackboard (not WebCT)
There is not an in-place upgrade path from Vista to NG as of this report that we are
aware of
We must treat NG as a migration to a new product (because it will be)
All licensing will have to be renegotiated. Right now, we have Vista support and
maintenance via a UNC System contract that expires in June 2011.
We have looked at demonstrations of Blackboard’s new product at various conferences.
For example, in July 2008, DELTA sent five instructional faculty (Vista users) as well as
staff members to the Blackboard conference. NC State representatives were also at the
Educause 2008 conference. Currently, the functionality we've seen is through graphical
mock-ups in Flash movies, roadmap discussions and demos, and conference
presentations. We have not gotten our hands on a “real product” for testing.
http://www.blackboard.com/sites/projectng
NC State has successfully used WolfWare for many years. What does the Moodle Pilot Team
know about WolfWare and how does that influence our thinking?
•
•
•
•
•
WolfWare is old, but stable and still used to support courses
Some of the WolfWare tools (e.g. forums/assignment submission) are duplicated in
both open source LMSs such as Moodle and proprietary systems such as Blackboard
Vista. WolfWare tools work but have not been updated to compete with the more
modern tools.
Some of the WolfWare tools (e.g. mailing lists, file storage) can be integrated with
Moodle in multiple ways
WolfWare is written in Perl (similar to, but older than PHP), thus building on a base we
already know and understand
Moodle would be a good vehicle to utilize in building out the next version of WolfWare
In fall 2009, DELTA worked with a consultant from Campus Management to do an analysis of
WolfWare and Moodle. Key points from this analysis include:
• NC State University needs enterprise infrastructure
• NC State University needs to re-brand any open source that we use (e.g. WolfWare)
• NC State University needs public web space
• An incentives program that would encourage faculty participation would be ideal
• About 200 hours of coding would be needed to rewrite the current WolfWare
Gradebook (DELTA should investigate other Moodle gradebook implementations)
• The Campus Management WolfWare/Moodle analysis is available at the LMS strategy
wiki, see http://wikis.lib.ncsu.edu/index.php/LMS_Strategy.
24
Report of the Moodle Pilot Team, April 2009
Appendix E: NC State University – Peer Institution’s LMS Usage
NC State University: Peer Institutions
http://www2.acs.ncsu.edu/UPA/peers/current/ncsu_peers/peerlinks_ncsu.htm
To obtain this information, DELTA staff corresponded with Teaching and Learning with Technology/Information
Technology Centers on our peer institutions campuses. This was last updated in March, 2009. Campus contact
information from the campuses is available upon request (email learntech@ncsu.edu for information).
1. Cornell: Production LMS = Blackboard Campus Edition (Actively piloting Moodle)
2. Virginia Tech: Production LMS = Blackboard Campus Edition (Actively piloting Sakai)
3. University of Maryland: Production LMS = Blackboard Campus Edition
4. University of Wisconsin-Madison: Production LMS = Blackboard Campus Edition (Some Moodle use across colleges as an
alternative)
5. Purdue: Production LMS = Vista (8.2)
6. Iowa State: Production LMS = Vista + Blackboard (8.x) (Extension and some departments use Moodle)
7. Texas A&M: Production LMS = Vista (8.0.1)
8. University of Illinois Urbana/Champaign: Production LMS = Vista
9. University of Florida: Production LMS = Vista (8.2)
10. University of Minnesota: Production LMS = Vista (8.x) and Moodle (1.8)
11. University of Georgia: Production LMS = Vista (8)
12. Georgia Tech: Production LMS = Sakai
13. University of California-Davis: Production LMS = Sakai
14. Michigan State: Production LMS = ANGEL
15. Penn State: Production LMS = ANGEL
16. Ohio State: Production LMS = Desire To Learn
Of these universities, several have "official" Open Source pilots underway on their campuses. Others have some off-model
use of Moodle and Sakai for extension type activities and specific tasks (like online course evaluations.) Universities that
seem to be investigating Moodle, in earnest, are Cornell and Minnesota. Our contact at University of Florida indicated there
is some growing momentum on their campus to move to Open Source. Many of the universities with Vista as their
production LMS talked about needing to develop a plan for the Vista product "end-of-life" in 2011/2012, but it didn't seem
that any of them have gotten real traction as they are so heavily invested in Vista with their integrations, use of the multi-
25
Report of the Moodle Pilot Team, April 2009
institutional capabilities, etc.. Some mentioned scalability of Moodle as a potential concern (see notes Minnesota & Texas
A&M) as well as changes in staffing models and programming expertise (see Purdue & Iowa State).
Peer
Institution
Current Production LMS
(Notes from DELTA staff conversations with campus contacts in Spring, 2009)
Production LMS: Blackboard v 6.2 (they are upgrading to v 7.3 in June 09).
Cornell
University
LMS Re-Eval: They are in the middle of a Moodle pilot with about 40 courses. See info at:
https://confluence.cornell.edu/display/MOODLE/About+the+Pilot
Pilots: Moodle pilot - they also have an e-portfolio pilot using the Sakai OSP tool. They'll be making some recommendations about the
future later this year.
Virginia
Tech
Production LMS: Blackboard 8.x
LMS Re-Eval: Two years ago, VT began a small pilot of Sakai, allowing only project courses (no student data), since the tools used by
an academic course were not ready. They now have 800 sections created, and plan to migrate all academic courses to Sakai by Fall
2010 when they feel that their support staff and faculty, as well as Sakai's tools, will be ready.
Pilots: Sakai, soon to become production environment
Production LMS: Their ELMS (powered by Blackboard) environment is running Blackboard Learning System 8.0.209.9. They also
have the Community System and Content System. Their 5-year review of their LMS will be in fall 2011.
LMS Re-Eval: They are beginning conversations this year - getting feedback from faculty on how things are going, laying the
University groundwork for a formal evaluation process to begin probably towards the end of this year. They will need to go through another
of Maryland formal RFP process.
Pilots and Experiments: Plan to take a brief look at Moodle and Sakai, with potential white paper on Sakai.
Their campus is a founding partner for Kuali Student, they have Kuali Coeus and they are actively looking at implementing Kuali
Financials. So, in terms of overall infrastructure, they may look at Sakai as the potential next "production" LMS on their campus.
Production LMS: Bb Enterprise
University
of
https://learnuw.wisc.edu/
WisconsinMadison
LMS Re-Eval/Pilots: Moodle (or eCOW in the School of Engineering) is a course management system offered by some schools and
26
Report of the Moodle Pilot Team, April 2009
departments as an alternative to Learn@UW. Departments known at this time to support Moodle are Education, Engineering, Letters
& Sciences, and Pharmacy.
Purdue
University
(West
Lafayette)
Source: https://academictech.doit.wisc.edu/technologies
Production LMS: Vista 8.2
LMS Re-Eval: An open-source environment would require a total restructure of staffing to account for the development needs of an
open-source solution. They are watching Blackboard's move to Blackboard Learn.
Pilots and Experiments: None to the TLT staff’s knowledge.
Production LMS: Vista/Blackboard 8.X
Iowa State
University
LMS Re-Eval: Nothing official is being done by central IT or the current Vista/Bb support group. Re-newed Bb license in Dec. for three
more years, and much faculty and staff expertise and custom programming time invested in our current Bb system that it would be
very costly and time consuming to migrate to another system. If there were to be cost savings they would be 2-3 three years down the
road at least. Our users are "OK" with Bb and there is no call for moving systems at this time.
Pilots and Experiments: Our Extension Service has non-credit classes on a Moodle server. The English department runs their own
Moodle server for their on-campus classes but it has no central integration or support. The Human Science College has 1-2 teacher
education courses using Moodle in an instructional lab setting for pre-service K-12 teachers. College of Engineering has a Moodle
server running for testing but does not plan to use it for credit-courses.
Production LMS: Our production LMS is Bb Vista 8.0.1 - Sun Solaris OS and Oracle database, F5 6400 load balancers.
LMS Re-Eval: No current re-evaluation. However, with the expected Vista product end-of-life being 2012, we are planning on doing a
Texas A & M re-evaluation. They anticipate evaluating Bb NG (or whatever the final combined Bb product will be), Sakai and Moodle. They require
University a solution that will be able to scale to 100,000 users. Our campus has 45,000 students and the TAMU System is interested in having a
common LMS and we would likely be the school to host a system-wide LMS. The system currently had 90,000 users.
Pilots and Experiments: No official pilots. The only officially supported LMS is Bb Vista. Will probably look at Moodle and Sakai.
Production LMS: Bb Vista, branded as "Illinois Compass"
University
LMS Re-Eval: Planning to start one this next year
of Illinois Urbana /
Pilots and Experiments: Several colleges run their own learning environments. Centrally they run Bb Vista, Mallard, Drupal, and
Champaign
wiki software
27
Report of the Moodle Pilot Team, April 2009
Other Info: Illinois-Urbana/Champaign is considering changing their model for supporting elearning and the tools used for
instruction, and will re-evaluate not only how they run (or contract for) production systems, but also how they are pursuing
contracts, licenses and other agreements. Their goal is maximum flexibility for the instructor with the fewest barriers to adoption
(technical, legal, or user-centered).
Production LMS: Blackboard Vista 8.2
University
of Florida
LMS Re-Eval: UFstarted a CMS review committee this past summer and are in final data-gathering stages right now. The committee
did market research to determine a short-list of systems of interest followed by a series of “talking” focus groups – small group
interviews to determine the CMS features and functions considered most desirable in the centrally supported CMS. Based on that
feedback, they narrowed the systems of interest to four “finalists:” ANGEL, Blackboard 9, Moodle, and Sakai. They furthermore
conducted a student survey to get student feedback in this process. They then invited vendors (representative support companies for
the open source systems) to present to the UF community. The CMS committee plans to have a recommendation for UF administration
by the end of March.
Pilots and Experiments: The UF College of Education has a hosted instance of Moodle used by roughly half their faculty (the other
half use Vista). There is a feature-limited home-grown system in use by a small number of instructors in the College of Medicine. We
have had an experimental instance of Sakai operational for over a year and a small number of instructors in Engineering and in IFAS
have been using it.
University
of
Minnesota
Production LMS: Moodle 1.8 and Vista 8.x
LMS Re-Eval: Since Vista goes away in 2011, they're working on a formal re-evaluation now. They plan to explore at least 4 CMS's.
He expects the top competitors will be Moodle and Blackboard's self-titled product. Scalability is a major concern, and for the first
time, mobile device access is going to be a factor in the decision.
Pilots: None.
Production LMS: Vista 8 (starting March 19th); migrated from WebCT CE 4.1
University
of Georgia
LMS Re-Eval: No - their evaluation was completed at the beginning of 2009
Pilots and Experiments: The Learning, Design and Technology Dept has a Moodle server, and the University supports Sakai for
collaborative projects (not teaching.)
Other Info:
Production LMS: Sakai
Georgia
Institute of
Technology LMS Re-Eval: No, they are pleased with Sakai, and are participating in product development along with other schools in the Sakai
consortium. Sakai has provided them with a great deal of freedom to support use cases outside of mere coursework. We have study
28
Report of the Moodle Pilot Team, April 2009
groups, organizations, research groups, etc., all with their own collaboration spaces alongside course sites. It really has become a
more diverse support for us, and is more than an LMS
Pilots and Experiments: Not at present.
Production LMS: Sakai
University
LMS Re-Eval: No need currently. During their last evaluation, they looked at Blackboard and WebCT, but neither out-did Sakai, at the
of
time a small-group pilot project on campus.
California Davis
Pilots and Experiments: None currently. Sakai is their main focus, bringing it up to spec with their own homegrown LMS (which it is
replacing)
Michigan
State
University
Production LMS: ANGEL
LMS Re-Eval: No need currently. They really like ANGEL (emphasizing the great support from the company), from staff and students
praising its usability to the support staff handling a simple infrastructure and a professional ANGEL support team that seems to
genuinely care about keeping them up and running.
Pilots and Experiments: There are a couple of departmental installations of Moodle and Sakai, most notably in the College of
Education and College of Engineering.
Penn State
University
Production LMS: ANGEL
LMS Re-Eval: No need currently. They are keeping their ear to the ground in general on what's new in the production and opensource markets. They have a plan of where they want ANGEL to be within 2-5 years. If those goals aren't met, they will revisit the LMS
evaluation issue and consider an open-source solution.
Pilots and Experiments: None.
Ohio State
University
Production LMS: D2L
LMS Re-Eval: They will be launching a re-eval next quarter in conjunction with overall IT strategic planning.
Pilots and Experiments: We have a couple of Moodle installations out in the units, notably the Math Department. The College of
Education has a specific need for an ePortfolio system so have been looking carefully at Tk20.
29
Report of the Moodle Pilot Team, April 2009
Appendix F: LMS Report - Tool Comparison
Meta-Level Tool Comparison
Public Access Allowed
Chat Rooms
Grade Book
Structured Lessons
Announcements
Online Quizzing
Calendar
Whiteboard
Polling
Mail/Messaging
Assignment Submission
Student Groups
Profile Page
Wiki
Glossary
Selective Release
Vista
WolfWare
No
Must have a Unity ID
Yes
Yes, if guest access is enabled
public can access HTML pages for a course
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Point-and-click to retrieve
Yes
Yes
No
Yes
Yes
Moodle
No
Yes
No
Yes
Yes
No
No
No
No
Yes
Yes
FTP access to retrieve
No
No
No
No
No
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Point-and-click to retrieve
Yes
Yes
Yes
Yes
Yes (coming in new releases)
30
Report of the Moodle Pilot Team, April 2009
Tool Comparison: Forums
The discussion board tools in Vista and Moodle (called Forums) are very similar in terms of functionality. Both systems allow
you to create basic threaded discussions where instructors and students can post and reply to each other. Each system has
settings to control how students are able to post and reply to other messages. They both also allow students to do peer
grading for each others' postings and replies. Both systems do have their own unique implementations of a discussion board
which bear noting.
Vista allows you to create two specialized types of discussion boards: blog and journal. The blog is visible to anyone in the
class and operates like a traditional blog. A journal is a private blog that is shared only between the student and the instructor,
with the option of letting other people have access to it.
Moodle has one specialized forum type, the Q&A forum. This forum allows a person to make an initial posting and students
cannot see any replies to the posting until they themselves make a reply to it. Moodle does offer a blog function, but it is a
system wide blog and not tied in with the discussion board functionality. This blog function is tied to the user's profile page
and is public to all members of the Moodle Community and is not class-contained.
Each system does have some unique features that are not present in the other system. In Vista you are able to lock discussions
so they become read-only. In Moodle forums are always unlocked. On the other hand, Moodle will send email notifications
and produce RSS feeds for new discussion posts whereas in Vista you have to log in to Vista to see if there are any new
postings. Instructors can set email notification options such as "force subscribe" where all user's receive an email.
One of the biggest and recurring critiques of the Moodle forums the DELTA faculty help desk has received to date is the way
new posts are displayed to be evaluated. Vista has the ability to display all new postings across all discussion boards in one
place to be evaluated. In Moodle, you can view new postings but it is on a forum by forum basis so each forum must be entered
to see the new postings.
File restrictions are another issue in Moodle forums. Students and instructors are able to upload only one file per forum post.
Although an image embed feature exits, students can only embed images that are already online and have a direct URL.
Instructors, on the other hand, have the ability to upload & embed multiple image files.
31
Tool Comparison: Quizzes
Report of the Moodle Pilot Team, April 2009
Vista
Moodle
Calculated
Yes
Yes
Matching
Yes
Yes
True/False
Essay
Embedded Answers
Multiple Choice (Cloze)
Short Answer
Numerical
Random Selection
Jumbled Sentence
Integrate with Respondus
Yes
Yes
No
Yes
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes (via Cloze)
Yes, but not as elegantly as Vista
Quizzes in Vista (called Assessments) and Moodle are very similar. They both have the same basic question types (although
Moodle does offer 2 additional question types) and similar delivery options. Both systems allow:
•
•
•
•
•
•
•
•
•
instructor feedback for individual questions and the quiz as a whole
setting question display properties
allowing multiple attempts
setting time limits
setting availability dates
setting passwords
limiting quizzes to certain IP addresses
creating question banks
and other basic quiz functionality found in most major LMS systems
32
Report of the Moodle Pilot Team, April 2009
Moodle does offer two additional options for delivering quizzes not found in Vista. First, you can allow student to retake a quiz
and see what their previous answers were. Second, you can allow students to see the results of their answer immediately after
submitting it and attempting the question again if they get it wrong the first time (Adaptive mode). In this mode, a penalty is
applied for each attempt.
Tool Comparison: Gradebook
The grade book tools in Vista, Moodle & WolfWare have similar functionality, but the ways in which users interact with the
grade book vary widely in each system.
Import file
Vista
Moodle
WolfWare
Yes
Yes
Yes
•
Import file types
Export to file
•
Yes
•
•
Export file types
Automatically enter grades
from activities (quiz,
assignment, etc.)
Override automatic grades
Comma separated
values
Tab separated values
Comma separated
values
Tab separated values
•
•
Yes
•
•
•
•
Yes
Yes
Yes
Yes
Comma separated
values
XML
Excel
OpenDocument
spreadsheet
Plain text (comma or
tab separated values)
XML
•
•
•
•
Yes
•
•
•
•
Excel
NExS
Comma separated
values
Tab separated values
Excel
NExS
Comma separated
values
Tab separated values
No
n/a
33
Hide grade book from all
students
Hide grade items from all
students
Hide grade items from select
students
Report of the Moodle Pilot Team, April 2009
Yes
Yes
Yes
Yes
Yes
No
Yes
Grade Types
Yes
•
•
•
•
•
•
•
•
Formula Editing
•
•
Grade item editing
Edit formula for select
students
No
Alphanumeric
Calculated
Grading form
Letter grade
Numeric
Selection list
Text
Yes
•
•
•
•
Point & click interface
to build formulas and
use functions
•
By student
By item
•
•
•
No
None
Value: numeric,
including calculated
Scale: an item in a list
Text
•
•
•
Numeric
Text
Formula
Type in standard
spreadsheet formulas
and/or functions
•
Type in standard
spreadsheet formulas
and/or functions
By student
By item
Master editor (any
item, any student)
•
•
•
Point & click interface
to build formulas and
use functions
By student
By item
Yes
34
Create grade categories
Weighted grade items
Customized letter grade
conversion
Report of the Moodle Pilot Team, April 2009
No
Yes
No
Yes
Yes
Yes
No
Tool Comparison: File Storage
Yes
Yes
Vista
WolfWare
Moodle
SFTP
No
Yes
No, but available via thirdparty tools
WolfCall
No
Yes
No
WebDAV
Web Portal for File
Management
File Space
Streaming Media
Persistent Files
Yes
Yes
Unlimited
No
No, but files can be restored
from a previous semester
Yes
No
1 GB for non-DE courses
16 GB for DE courses
Yes
Yes, depending on where you
store them
No, but available via third
party tools
Yes
Unlimited
No
No, but files can be restored
from a previous semester
The three LMSs at NC State University have significant file storage and maintenance differences between them. These are
outlined in the following sections.
WolfWare
WolfWare offers the greatest flexibility for managing files. Files are managed either through SFTP, WebDAV, or WolfCall. All
three of these methods require some initial setup by the instructor and knowledge of how these technologies work. The
35
Report of the Moodle Pilot Team, April 2009
unique features of WolfWare's file system are the persistence of files, the file size limitations, and the ability to stream files
with the streaming media server.
Persistence of Files. WolfWare has four areas to store files, all with different functionality. The three locations allow
you to:
1.
2.
3.
4.
make files available to the public, but change with each semester (lec folder)
make files private to your class, and change with each semester (lec/wrap folder)
make files available to the public and remain persistent (common folder)
allow files to be streamed from the streaming media server (common/media folder)
The file areas that change from semester to semester can be reloaded with a previous semester's content through a web
interface.
File Size Limitations. By default a WolfWare course is given 200 MB lec, 100MB common, 1GB media of storage divided
between the lec, common, and media folder. Small additional amounts of space can be added to the lec folder, and a non-DE
class can request up to 1 GB in the media folder while a DE class can request up to 16 GB.
Streaming Media Server. WolfWare also provides a space where files can be accessed by the University's streaming media
server. Streaming media files must be stored in specialized online folders. WolfWare's media folder provides this service.
Vista
Vista offers two modes of managing files: a Web-based client, and WebDAV. The web-based client is accessed by logging into
Vista, while the WebDAV client is accessed directly through the local computer's operating system. Vista allows a novice user
to be able to upload and manage course files. The WebDAV feature allows instructors to upload files en masse.
There is no cap on the total file size for a course, but this is actually a design flaw in the software. If people abuse Vista's file
storage space the system's database becomes too large and can cause problems for all Vista users.
36
Moodle
Report of the Moodle Pilot Team, April 2009
Moodle is the least flexible for file maintenance, offering only a web client. Files can only be uploaded one at a time through
the web client. One solution to moving files en masse is to put all of the files to move in a single .zip archive, upload the archive,
then unzip the archive once it is in Moodle. The Moodle community is working on a WebDAV interface for Moodle, but it is not
available in a stable form yet. There are also other third-party solutions DELTA has explored for adding SFTP functionality to
Moodle.
Just as with Vista, there is no cap in place for total file size for a Moodle course. Administrators and Instructors can limit the
size of an individual file upload. Also as with Vista, if people abuse the file storage of Moodle it can cause problems for all the
users of the system.
For WolfWare, Vista and Moodle:
There is no general file storage space for students. They can only upload files when associated with an activity within a course,
such as a wiki or forum.
37
Report of the Moodle Pilot Team, April 2009
Appendix G: Assessment/Evaluation Information
Spring & Summer 2008 – Initial Pilot Assessment Data. Surveys designed to capture both faculty and student feedback about
their experiences with Moodle were created and distributed to the faculty and to three sets of students: (1) students in two
different sections of the same online Instructional Design DE course (EAC 580), one using Blackboard Vista and one using
Moodle, (2) students in a Toxicology course (TOX 401/501), and (3) all other students.
Of the 40 participants listed as instructors during the spring and summer sessions (though only thirteen of these were
teaching “live” courses – the remainder were exploring Moodle and/or designing future courses), ten responded, resulting in
an estimated instructor response rate of 25% if counting all 40 potential instructors. Of the 22 students in both sections of
EAC 580, 17 responded, for a response rate of 77%. Of the 35 students in the Toxicology course, 23 responded, for a response
rate of 66%. Of the 271 students who received an invitation to participate in this survey via their instructor (it was not clear if
all students received the invitation to participate, as all instructors did not confirm passing the email along), 63 students
responded to the survey, resulting in an estimated response rate of 23%. Key assessment information from the spring and
summer 2008 pilot participants includes the following:
•
•
Even though a number of improvements were suggested for Moodle (for example, gradebook, forum features, wiki,
structure/organization of course entry page), the faculty members who responded to the Moodle pilot survey indicated
that they liked Moodle better than Blackboard Vista, with 90% of respondents agreeing that Moodle was easier to use
than Blackboard Vista when rating ease of use (Strongly Disagree, Disagree, About the Same – 10%, Agree – 40%,
Strongly Agree – 50%) and 100% indicating they liked Moodle better than Blackboard Vista when rating likeability
(Much Worse, Worse, About the Same, Better – 30%, Much Better – 70%), on 5-point Likert scales. Caution: As these
faculty chose to participate in the pilot, presumably in some cases because they had concerns about Blackboard Vista, they
may not be representative of the typical NC State instructor or of all the instructors in the pilot. In addition, NC State
University developers/technology support staff will need to pay close attention to the elements that were “not liked” as we
search for improvements to the system beyond “Moodle out of the box.”
Some students indicated they either really liked or didn’t like Moodle or really liked or didn’t like Blackboard Vista, thus
a balanced view of the overall data collected notes that there were student preferences for one LMS or the other for
various reasons (for example, prior experience, software interface design/navigation, choices the faculty made in
setting up the system). Though there were student preferences noted, in all three pilot studies involving students,
Moodle was favorably viewed overall, as follows:
o In TOX 401/501 (n=23), using a 5-point Likert Scale rating to compare Moodle and Blackboard (Much Worse,
Worse, About the Same, Better, Much Better, Don’t Know), 74% of student respondents indicated that Moodle is
38
Report of the Moodle Pilot Team, April 2009
•
•
•
•
Better (39%), or Much Better (35%), than Blackboard Vista. No one compared Moodle negatively to Blackboard,
four students found them about the same, and two did not respond.
o In the EAC 580 DE comparison of a Moodle (n=8) and Blackboard (n=9) section of the same course, using a 5point Likert Scale rating for Likeability (Much Worse, Worse, About the Same, Better, Much Better, Don’t Know),
44% of the students in the Blackboard Vista indicated they could not compare it to other systems, 11% indicated
Blackboard Vista was Better, 33% indicated Blackboard Vista was About the Same, and 11% indicated
Blackboard Vista was Much Worse. All students in the Moodle section had used Blackboard and could thus
compare the systems; 35.7% indicated Moodle was Much Better 12.5%) or Better (25%), 50% of students
indicated Moodle was about the same, and 12.5% indicated Moodle was Worse.
o In the EAC 580 DE comparison of a Moodle (n=8) and Blackboard (n=9) section of the same course, using a 5point Likert Scale rating for Ease of Use(Strongly Disagree, Disagree, About the Same, Agree, Strongly
Agree),44% of the students in the Blackboard Vista indicated they did not know if Blackboard Vista was easier
to use than other LMSs, 33% Agreed Blackboard Vista was easier to use than other systems they had used, 11%
indicated Blackboard Vista was About the Same, and 11% indicated Blackboard Vista Strongly Disagreed that
Blackboard Vista was easier to use. All students in the Moodle section had used Blackboard and could thus
compare the systems; 35.7% Strongly Agreed (12.5%) or Agreed (25%) that Moodle was easier to use that other
systems they had used, 50% of students indicated Moodle was About the Same, 12.5% Disagreed, and 12.5%
Strongly Disagreed that Moodle was easier.
o Caution: The three student assessment points are based on feedback from a total of 103 student respondents, and
may not be representative of all students in the pilot, nor should this be assumed to represent the larger NC State
student populations potential opinion.
The pilot faculty respondents appeared to be more enthusiastic about Moodle than the pilot student respondents, based
on the percentages of respondents who highly favored Moodle across all these assessment points. Students were more
likely to indicate that to them, the systems were about the same.
Qualitative comments from the pilot respondents indicated that the ease of use and cross-browser usability is a feature
that students generally like about Moodle (though some students indicated that the Moodle home page too cluttered).
Qualitative comments also provided a glimpse of the interplay between the tools themselves and the instructional
design/teaching choices; as one student noted “Moodle to me was a lot like Blackboard, as long as the teachers posted
the assignments then everything is alright.”
Qualitative comments indicated that there are tools and/or design features in both Blackboard Vista and Moodle that
neither students nor faculty like. These comments are indicative that there will likely be both user likes and dislikes to
address in any LMS decision at NC State. No LMS will be the “perfect” choice for everyone.
39
Report of the Moodle Pilot Team, April 2009
Fall 2008 – Initial Pilot Assessment Data. Surveys designed to capture both faculty and student feedback about their
experiences with Moodle were created and distributed to the faculty and to two sets of students: (1) students in two different
sections of the same online Instructional Design DE course (EAC 580), one using Blackboard Vista and one using Moodle, (2)
all other students.
As defined by having updated class roles and active student logins, 53 course sections were taught during fall 2008, with
twenty-nine unique instructors, and additional support staff supporting these courses and/or developing additional course
sections on the Moodle server. Courses included face-to-face (F2F) classes and Distance Education (DE) course sections. The
active fall sections included courses in CALS, CHASS, Education, Engineering, PAMS. Twenty-two of the 29 instructors
responded to the fall survey, resulting in an estimated response rate of 75.9%. More than half of the faculty respondents were
using Moodle to supplement a F2F course (57%), whereas the remaining faculty were divided between doing both (18%) or
teaching only a DE course (24%). Of the 31 total students in both sections of EAC 580, 22 students responded, for a response
rate of around 71% (though a big limitation of this survey was the lower response rate for the Blackboard Vista section (n=8 –
the Moodle section had n=14). Of the 2232 students who received an invitation to participate in this survey via their
instructor (it was not clear if all students received the invitation to participate, as all instructors did not confirm passing the
email along), 308 students responded to the survey (though only 303 surveys were complete and thus used in the survey
summary information), resulting in an estimated response rate of 14%. Key assessment information from the fall 2008 pilot
participants includes the following:
•
77% of the faculty respondents had previous experience in teaching using Blackboard Vista.
o Using a 5-point Likert scale, faculty were asked to compare Moodle with Blackboard Vista along several
comparison points, with Moodle favored over Blackboard Vista on all points:
 Around 64% of faculty commenting on Moodle’s ease of use either Agreed or Strongly Agreed (on a 5point Likert Scale) that Moodle was easier to use than Blackboard Vista/WebCT, with two faculty
respondents selecting a Neutral/Not Sure response (9%) and the remaining six indicating they could not
compare the two as they had not used both.
 Around 73% of faculty indicated that the performance of Moodle was Above Average or Excellent,
compared with other LMSs. No faculty respondent felt the reliability and performance of Moodle was
Below Average.
 Around 64% of respondents who had used Blackboard Vista liked Moodle Better (18%) or Much Better
(45.5%) than Blackboard Vista, with seven (32%) indicating they did not know and one respondent
indicating they are about the same.
40
Report of the Moodle Pilot Team, April 2009
•
o From a menu of possible options that included: (a) make the switch to Moodle from Blackboard, (b) look at
Blackboard's new product, Blackboard NG, before making a decision, (c) look at other products on the market,
including Blackboard NG and others (e.g. Angel, Desire to Learn, Sakai), (d) it doesn't matter to me what our
C/LMS is at NC State as long as it is stable and reliable, and (e) other, (see Appendix A), faculty were asked to
make a recommendation to campus technology administrators regarding taking the next step in selecting a
C/LMS for our campus. Around 23% of the faculty respondents recommended that NC State University look at
Blackboard NG and/or other products before making a decision, whereas 68% of the respondents recommended
that we adopt Moodle (with tweaks/considerations elaborated on in the open-ended responses). Keep in mind,
however, that these faculty respondents used Moodle by choice as early adopters in the pilot, which likely
influences their response to this question, and we cannot assume that all faculty members would feel this way.
o 27% indicated they were also using WolfWare, with WolfWare usage including content/file storage (including
media files), mailing list, gradebook, and assignments tool. Considering the continued WolfWare usage even in
conjunction with Moodle (and frankly, with Bb Vista, too), WolfWare should be addressed as part of any
considered next steps in LMS decisions on our campus.
o Similar to the spring and summer faculty survey, open-ended responses indicated that Blackboard Vista has
some features that are liked (for example, robust assignment, discussion and gradebook tools), as well as
disliked (for example, Blackboard was described again by several respondents as clunky and slow). Moodle had
positive comments, with many open-ended responses complimenting the ease of use and flexible layout (with
the RSS feeds mentioned twice as a useful additional tool). On the other hand, the Moodle gradebook was often
mentioned in the responses as a problematic tool within Moodle, with some other concerns cropping up in the
responses (wiki and forum functionality, file management). When asked what features they would like to see in
Moodle, respondents had a number of suggestions, with the gradebook and quizzing tool mentioned several
times for improvement considerations, as well as a number of different, additional tweaks. Technology support
staff should look at the responses regarding the problematic elements and feature requests, and work with the
LMS committees to determine how to address these issues, should NC State University choose to underpin our
LMS with Moodle.
o While overall, the faculty respondents seem positive towards considering Moodle as a (partial) solution for our
campus LMS, as a final comment, it is important to once again note that the twenty-two faculty respondents in this
survey may not be typical of the general faculty population that use LMSs on campus.
In the fall 2009 EAC 580 DE comparison of a Moodle (n=14) and Blackboard (n=8) section of the same course, using a
5-point Likert Scale rating for Ease of Use(Strongly Disagree, Disagree, About the Same, Agree, Strongly Agree),38% of
the students in the Blackboard Vista section indicated they did not know if Blackboard Vista was easier to other
systems, around 38% Agreed Blackboard was easier to use than other systems they had experienced, one student
41
Report of the Moodle Pilot Team, April 2009
•
•
Disagreed that Blackboard Vista was easier to use, and one student was not sure if Blackboard Vista was easier to use
than other LMSs they had used. For students in the course section that used Moodle, around 43% of students Agreed or
Strongly Agreed that Moodle was easier to use than other systems they had used (all with Blackboard Vista experience),
and another 21% indicated they were not sure as they had not used other systems. No students Disagreed or Strongly
Disagreed regarding Moodle’s ease of use.
In the general fall 2008 student survey, around 2232 students could have received the invitation to participate in this
survey (the survey was sent in partnership with OIT using the bulk email service, see Appendix A). There were 303
usable survey responses (5 were excluded from the analysis due to missing or corrupt data – 14%). Key findings
included:
o Around 58% of the students had previous experience in using Blackboard Vista and/or WebCT Vista in their
courses. 82% of students experienced Moodle as a supplement to F2F courses; the remaining students had DE
experiences with Moodle.
o Using a 5-point Likert scale, 49% of the student respondents either Agreed or Strongly Agreed that Moodle was
easier to use than Blackboard Vista/WebCT. Around 29% were Not Sure/Neutral if one was easier than the
other, and around 12% did not find Moodle easier to use.
o Using a 5-point Likert scale, around 68% of students indicated that the performance of Moodle was Above
Average or Excellent. 27% of students indicated the performance was Average, and around 6% found Moodle to
be Below Average.
o Using a 5-point Likert scale, students were asked to rate Moodle as it compares to Blackboard Vista (e.g. is it
much worse, and thus the student would prefer that their classes use something else, or is it much better, and
thus the student would prefer that all their classes use Moodle). Around 50% of the student respondents liked
Moodle Better (30.7%) or Much Better (19.1%) than Blackboard Vista.
o Around 50% of the student respondents indicated that the LMS choice did not matter to them as long as it is
stable and reliable (see Tables 1.6 and 1.7). Around 28% indicated it would be okay to switch to Moodle.
Around 10% indicated that we should look at Blackboard’s next product, and around 12% had additional
recommendations.
Qualitative comments from the fall 2008 student respondents were generally positive regarding the ease of use and
organization of Moodle, though some students found their course layout confusing (at least at first) – similar to the
earlier student responses. Instructor choices may again influence comments about interface and organization; e.g. “I
think my professor over the summer had the moodle site way more organized than my [fall] professor”.
See the LMS strategy page for more detailed assessment summaries and survey instruments
(http://wikis.lib.ncsu.edu/index.php/LMS_Strategy).
42
Report of the Moodle Pilot Team, April 2009
Appendix H: Learning Management System (LMS) Committee Structure
One of DELTA’s goals is to provide an enterprise-level LMS service that meets the needs of university constituents for teaching and
learning within a cost-effective and maintainable structure. In moving forward with the improvement and implementation of enterpriselevel LMS services, DELTA solicits involved participation from our stakeholders, as together we craft and implement the vision and
direction for campus LMS services. With this in mind, we propose the following committees to support our ongoing LMS
recommendation and decision-making process:
The LMS Steering Committee will consider and coordinate the needs and recommendations of the other LMS committees, reconcile
differences (if any), and suggest action plans for services and/or distributed needs to the DELTA Strategy Team. In addition to working
with the "bottom-up" needs and directives of the committees, the steering committee may send issues to particular committees to discuss,
and/or direct the formation of workgroups around specific tasks (e.g. assessment, policy).
The LMS College Needs and Policy Recommendations Committee will build and maintain an awareness of college-level needs related to
LMSs and recommend strategies for collaboratively meeting these needs to the LMS Steering Committee. This group will also discuss and
document the implications of any suggested add-ons and usage requests from a policy perspective, providing recommendations to the
LMS Steering Committee.
The LMS Best Practices and Support Committee will provide suggestions for "best practices" from a user-centered approach to LMSs, from
the way users might utilize a certain tool to what settings we might recommend from a user perspective to optimize the teaching and
learning environment in a pedagogically-sound manner. This group will also recommend training and support structures and strategies
that will help others fully utilize campus LMSs, including identifying proper channels for providing technical support to instructors and
students. Finally, this group will discuss and document the implications of any suggested add-ons to our emerging LMS structure from
best practices, user and training perspectives, providing recommendations to the LMS Steering Committee.
The LMS Technical Concerns Committee will examine the technical implications of decisions and provide an informed technical opinion to
the LMS Steering Committee regarding integrations, changes and additions to our campus LMSs. This committee will explore the technical
feasibility, document appropriate technical methodologies, and implement a collaborative technical workflow for providing ongoing
enterprise-level LMS services at NC State. In this capacity, this committee will also serve as a resource for developers of modules for
campus LMS services.
As needs emerge, committees may direct the formation of workgroups (that may extend beyond committee members) around specific
tasks (for example, a specific assessment, a usability study, or a course rollover flowchart) that need to be accomplished. These
workgroups will report findings to their sponsoring committee as requested, providing needed information for the committee
to move forward in their investigative and decision-making processes.
Both DELTA and various college developers will be collaboratively involved in this project.
The DELTA Advisory Committee, compromised of appointees from every college and a number of campus units, will be actively involved
in decisions regarding investments in LMS technologies.
Last updated April 7, 2009, dtp
43
Report of the Moodle Pilot Team, April 2009
Vice Chancellor
for Information
Technology
Vice Chancellor
for Finance &
Business
Provost
Inform,
Advise,
Coordinate
LMS
Committees
Vice Provost
DELTA
The DELTA
Advisory Council
and the LMS
Steering
Committee and its
subcommittees
formally provide
advice and
recommendations
to DELTA
management
related to Learning
Management
System (LMS)
implementation.
DELTA Strategy Team
Advise
DELTA Advisory
Committee
Report,
Recommend
Inform,
Advise
TLTR
College LMS
Representatives
DE Coordinators
AITD
Library & OIT
Office of Faculty
Development
Faculty &
Student Senates
Solicit Input,
Provide Feedback
LMS Steering Committee
Notify,
Solicit Input
Solicit Direction,
Provide Feedback
College Needs &
Policy
Recommendations
Best Practices
& Support
Workgroups around
tasks
Report, Advise,
Recommend
Technical
DELTA
Tactical
Informal advice will
also be sought
through various
communities of
practice and
partner
organizations.
Developers
44