September 30, 2010
Recommendation – Villanova University’s LMS Replacement
Over the last two years, a faculty pilot group, UNIT staff and student volunteers have actively researched and tested the various LMS
applications. A total cost of ownership model was developed, surveys were completed, collaborative efforts with other schools, and testing of
the vendor/product options were all part of the evaluation/due diligence process used to make a good decision for not only the University as a
whole but for the individual faculty and students with varied teaching and learning needs. It is significant to note that for Villanova, neither
cost nor the core feature sets stood out as the biggest differentiators between the competing LMS products, but the extra features that add
enhancements to our current platform that will provide more diverse tools and capabilities in the learning environment.
During the process there was discussion about the possibility of Villanova moving towards an open-source solution for the next LMS product.
After review by the faculty, it was determined that the current release of the more widely used open-source product (Sakai) does not provide
the broad spectrum of features required by the faculty as a whole. The next release from the Sakai community is expected in early 2012. This
release appears to meet many of the functional requirements cited by the faculty. This could potentially be a longer range strategy to be
considered by the University.
In the final discussion, the main factor considered by faculty was the added functionality to the environment - new innovations without losing
the features of the current LMS. Support for mobile devices, more flexibility with testing, easier to use interface, etc., were some of the
discussion points that lead to the final recommendation. While all the evaluated products did retain most of the core functionality required,
the Blackboard Learn release is the only application that provided enhancements and also supported the integration of the many third party
products that are currently accessed through WebCT and used across campus. Thus the Blackboard Learn system, with its familiar look and
feel and significant improvements, is the recommendation from the faculty as the outcome of this research process.
The attached report will be submitted to the Governance Committees for final approval.
1
The Next Step: Villanova’s Learning Management System
Prepared by UNIT’s Instructional Technologies
This paper summarizes Villanova University’s search for its next Learning Management System (LMS). It describes the needs assessment
process that the University undertook to gather requirements, the survey process for comments and feedback and other associated documents
that resulted, and the process by which an LMS was chosen. In the final phase, three Learning management Systems were considered as
possible alternatives: Blackboard and Desire2Learn, both leading proprietary systems, as well as Sakai, open source initiative. This paper
summarizes all of the activity that occurred during this process and the implications of the decision for Villanova.
Introduction
During the academic year 2008-2009, Instructional Technologies began the process to determine what LMS should replace the current WebCT
system. For a brief history, the University first contracted with WebCT in 1998. The company has proven to be a good partner with its
continual focus on teaching and learning. The University benefited from having an extensible learning platform and predicable small increases
in licensing costs throughout the years. In October of 2005, Blackboard purchased the WebCT company in an attempt to merge the best-ofbreed in learning management systems. Once again this relationship has proven to be beneficial to the University. Unit has been able to
continue to provide a very cost-effective learning management system by leveraging our relationship with the vendor and using our current
infrastructure to deliver an enterprise learning system with a wide variety of 3rd party teaching tools for faculty and student.
In 2008 we renewed our contract and agreed to test a learning adaptor for Blackboard. This agreement enabled us to have a three year freeze
on our license cost while awaiting delivery of a next-generation product as promised by Blackboard. In 2011 our contract with Blackboard for
the WebCT product ends. This end of contract has been coordinated to give the University sufficient time to make a thoughtful and diligent
decision in the selection of our next LM system.
Over the years we have seen a tremendous extension of the WebCT platform from the delivery of thousands of journal articles which are part
of the Library’s eReserve to departmental training of staff and the development of certificate programs. In reviewing usage statistics from year
to year over 9300 students are enrolled in a least one WebCT course each semester; we see usage of faculty as high as 62% with an average of
approximately 55% on a yearly basis. Many faculty deliver video and audio content through WebCT which enables them to have copyrighted
material in a password protected environment thus keeping them in compliance with copyright law. Several faculty are now using an
ePortfolio system which is also integrated with the WebCT environment. In addition to the collaborative, testing and tracking tools which
come packaged as part of WebCT, faculty also routinely use “powerlinks” (3 rd party applications integrated with WebCT). These include
Respondus, a test generator and security testing package; SafeAssign – a plagiarism detection program; iTunes University and Wimba for
synchronous collaboration. Other third party applications have also been added into the environment to address specific needs of individual
faculty or groups of faculty, e.g., Scholar, etc.
As we move forward, several drivers will determine the direction the University will take in pursuit of a new learning management system. The
primary one, as previously mentioned, is the software is at end-of-life. The company will no longer develop new tools for the software
platform. Blackboard will also at the end of year 2012 no longer support the WebCT product. This means that any further updates for product
fixes to the current tools, security patches, etc.., will not be released. As we look at advances in technology over the last few years, such as
mobile computing, improvements in collaborative tools, drop and drag functionalities, etc. , unfortunately faculty and students will not have
access to these tools in the learning environment until we move on to a new platform.
From a central computing perspective, many things have to be considered regarding the administrative and support requirements of a new
system. Will we need additional staff in place to support a new environment? What are the integration requirements for 3 rd party
applications? What will it take to integrate all the third-party applications with respect to financial and staff?
In Appendix 1, you will see a list of functional requirements that have been aggregated from need assessments developed with input from
students, faculty and UNIT’s Instructional Technologies group. What we have attempted to do is to get feedback from all stakeholders and
determine which tools are the most used, the most effective and which tools must to be part of our next LMS. We consider gaps that exist and
how we can address those with 3rd party applications.
Other drivers being considered: what is the health of the vendor in terms of financial stability-is the vendor or community here to stay or are
they subject to the negative effects of the current economic climate? Is the platform scalable --- not only capable of meeting the educational
needs of our classrooms but the broader needs of the University as a whole --- we have many departments across campus that make use of the
LMS system to provide training for their staff, certificate programs to external users and as an environment where online collaboration can
occur within departments and/or among groups. The LMS has to handle many of the University's functional requirements, from educational to
institutional use. These requirements include integration with existing systems, Banner, LDAP, MyNova, etc., and functionality to meet staff
workflow and governmental regulatory standards.
2
Is the company or organization innovative – are they looking at future technology trends and actively updating their application’s tool sets?
This is an important aspect as we move forward with our next learning management system. As we've seen over the last couple of years some
of the tools in the WebCT environment have become outdated since the purchase of the application by Blackboard. When we first acquired
the current version, the toolset was varied and adaptable to the learning environment capable of meeting the needs of a broad spectrum of
faculty users. With very limited development over the last few years, the toolset has become stagnant. It is important that the next company
and product that we select for our new learning management system has the vision to not only put in place the current technologies but also
continue to look at the direction of technology and incorporate tools into their suite to keep it current.
The financial impact in the selection of the next learning management system is also an extremely important consideration. A separate
document provided a total of cost of ownership for the three packages being considered. The cost estimates were extended over a five-year
period. In reviewing the document, a local installation of the two proprietary products is less expensive than the hosted option. However, the
costs to purchase and maintain hardware necessary to support a local installation, the staff resources and the overall environment as a whole
in terms of heating, air-conditioning, physical space, cabling, routers, switches, etc., - all play into the final total cost of ownership. All factors
have to be considered in making a truly diligent decision in terms of our next learning management system. Although the financial impact in
terms of budget allocation is a considerable one, it is not the primary focus or the primary factor in determining the direction of our next
product. The functional use of the product in terms of meeting faculty and student needs in the learning environment is the primary factor in
the selection process.
Process and Input
In 2008 academic year, Unit staff began reviewing the applications that existed in the LMS market. At that time the top five products were:
Blackboard Learn, Sakai, Moodle, Angel and Desire2Learn. This represented three proprietary and two open source products. UNIT actively
reviewed these products in terms of the toolsets inclusive at the time, the cost and the ability to make a change and sustain the application in
our environment.
In conjunction with one of the classes in the Business School, UNIT reviewed the five products previously mentioned in structured process to
isolate and weigh all the parameters within the applications and place a value on the usage of the tools, the cost, and the ability to integrate
into the environment. This process was to quantify a forward direction in the selection process. In Appendix 3, attached to this document you
can review some of the tables that were included in the final report. The final report was based on student input on the usability of the
products and what made sense in terms of going forward with a replacement application. At the time, It was determined that to replace
WebCT with another product would not be a cost-effective initiative. There would be no cost savings or enhancements in toolsets that would
add value sufficient to shift another product. A decision was made to stay with WebCT for the interim until the next generation products
became available. If cost were not a factor at the time, the student preference would have been for the Angel system. Subsequently Angel
was bought by Blackboard. As we went through this process with the students, we were able to glean some valuable information in terms of
student usage of a learning management system. The likes and dislikes of the student groups were tabulated as part of the Project. Student
input was captured as to what they would like to see in any of the next generation products; what tools were considered helpful and what
tools were absolutely necessary.
In the 2009/2010 academic year, UNIT has also reviewed additional applications which have either emerged as new possibilities in the LMS
marketplace or have been bought and rebranded by other companies. These additional applications included CourseCruiser, Edvance360,
Instructure, Web Study and Moodle for the second time. This was done in conjunction with faculty input and with UNIT’s knowledge of the
toolsets that are currently used across campus.
In the ACIT meeting of October 2009, the selection process of the next learning management system was discussed. A recommendation from
the committee was to convene a pilot group of faculty who could research and have input into the selection process. One of the faculty
offered to present the request for faculty participants to the Faculty Senate. Other faculty present at the ACIT meeting were asked to take the
request back to the individual colleges. A core group of faculty representative of all the colleges was convened with the task of reviewing and
researching the top applications in the learning management marketplace in November 2009
At the onset the faculty committee consisted of 40 faculty members representing each college and staff from the Law school, UNIT and the
Falvey Library. The Faculty Committee was asked to review, research and score the top three products in a pilot environment: Blackboard
Learn Suite in a hosted environment; Sakai in a hosted and locally installed environments and Desire2Learn in a hosted environment. Meetings
were scheduled to address the milestones of the project. Information was disseminated that included introductions to the applications being
reviewed; training session schedules, webinars for each of the applications; a cost estimates for each of the applications and score sheets were
provided for each of the application. A needs assessment survey also was made available for faculty to have input regarding desired tools and
functions within the LMS application. The first application that was introduced to the faculty was Sakai. Arrangements were made with a Sakai
reseller, RSmart, to set up a hosted environment for faculty testers for Villanova. We also installed a Sakai instance on Villanova servers. The
3
purpose of this installation was to ascertain whether we as a department could do the installation, the administration and the future support
of an open-source product on campus. The review and research of Sakai ran from December 2009 and continued through the end of the
summer session.
The next application for review began in February 2010. The application was Desired2Learn which was hosted by the vendor. The Faculty
Committee was given access to use sandboxes on the vendor site. Additionally one faculty opted to teach a summer course with the
Desire2Learn product (survey included in Appendix 3). The last application reviewed was Blackboard Learn 9.0 suite of software.
Arrangements were made with the vendor to host a Villanova installation. Again, faculty were given IDs and passwords with the ability to set
up a content, investigate the toolset and were provided a score sheet to rate the application. In summary beginning the first of December
through August 30th, faculty had access to research, review and score the top three applications.
Additionally relevant documents available in the public sector, as well as LMS vendor literature were made available and played a crucial role in
the review process. This literature review, as well as having access to the vendor applications, allowed the University to gain knowledge of the
benefits and drawbacks of each system. Public documentation in the form of surveys of technical requirements, faculty and student
evaluations, as well as vendor responses to RFQs were all very informative,
As we have gone through this process, there have been additional opportunities for faculty to have input into the selection process or at a
minimum to gain knowledgeable about the possibilities of the different alternatives. On April 22 during the Technology Expo, all vendors were
invited to participate. The vendors had the opportunity to have a table with the ability to do a half an hour or hour presentation. Many faculty
took advantage of this opportunity and spent time with the vendors discussing their products and having a demonstration of the user interface
and software functionality. On May 13th during the Teaching and Learning Strategies Day hosted by VITAL and CIT, a presentation was made
describing the learning management selection process and its implications for Villanova over the next year. More than 40 faculty attended the
session and information was collected in regards to faculty input and concerns about the next learning management system at Villanova. On
September 8th the entire day was devoted to vendor presentations by representatives from Blackboard, Desire2Learn and RSmart. All faculty
were invited to the presentations via Campus Currents and multiple emails were sent to the faculty participants of pilot group. Twenty-two of
faculty attended the session and were introduced to the vendor representatives and able to ask questions and have demonstrations of the
individual tools of their learning management systems. Additionally, members of the Northeast eLearning Consortium were invited to provide
some different perspectives.
During the second week of classes (Fall 2010 semester), a survey was generated and sent to all faculty in all the colleges requesting their input
into the selection process. The survey consisted of a series of questions and ratings of the different tools in an attempt to get a broad
indication of the functionality that faculty would like to see in the next platform. A representation of the feedback is included below in Figure
I. This represents all the tools and features that have been indicated by the faculty as required in the next learning management system.
Across-the-board all of the products we are currently reviewing have these features included. The products differentiate themselves by how
well these features actually function within the toolset or if the feature is an add-on application. For example Sakai as an open-source solution
is a series of modules that are integrated in one application. The modules are written by different people at different schools and thus
sometimes have a different look and feel. This is particularly noticeable, for example, when an HTML editor is part of one tool and not part of
another. As a functional example, in the case of a text posting for a Wiki, a faculty member or student would have to know some basic HTML in
order to post formatted text.
4
Figure I – A Model for Proposed System
Wikis
Content Delivery
Respondus
Threaded Discussion
Email, Chat & IM
Blogging
Students Tracking
Course Calendar
Content MGT for
documents & video assets
Online Testing
Online Grade book
Next
LM System
Send/Receive mail
external to LMS
Security
MyNova Integration
SIS Integration
Photos in Grade book
Mobile Component
LDAP Integration
Respondus
Application Sharing
ePortfolios
Respondus
Plagiarism
Detection
Grade Exchange
Drop & Drag
Functionality
Respondus
Appendix 4 and Appendix 5 include summaries of the score sheets and the tool survey sheets that faculty prepared as part of the review process. Appendix 4 is
a tabulation of all the features and tools that faculty would like to see in the next learning management system inclusive in this table are the actual highest
ratings to the lowest ratings of the designated tools. In Appendix 5 is a summary of the score sheets that faculty submitted after using the applications in a
hosted environment.
Financial Costs and Implications
A separate document included in the summary is the total cost of ownership for the individual products. As can be seen the hosted solutions
versus a local installation is significantly different for all products. We have attempted to include all aspects of hosting and self hosting to
adequately reflect the money that would have to be spent for the next-generation product. What is not included in this analysis is the cost that
would be absorbed by the University in terms of the physical requirements for server hosting on campus. A study was done last year with
Facility Management and UnIT to ascertain what improvements would be needed to maintain the data center as it was in the Mendel facility.
The cost for the physical equipment and renovations to the building to include improvements in environmentals such as heat and air
conditioning were substantial enough to consider an option for off site hosting for Villanova’s major computing systems. A decision was made
to move the Data Center in Spring of this year (2010) to an off-site location. To determine a complete and total cost of ownership right for a
local installation of the future LMS product a cost factor of the off-site hosted environment for the LMS would need to be included. The
addition of these costs added to the items already included in the TCO makes the opportunity for a vendor hosted solution appear to be costeffective and fiscally responsible.
Conclusion
The process used to select the next learning management system at Villanova has been laborious for both faculty and staff but necessary in
order to make a good decision for not only the University as a whole but for the individual faculty and students with varied teaching and
learning needs. It is significant to note that for Villanova, neither cost nor the core feature sets stood out as the biggest differentiators
5
between the competing LMS products, but the extra features that added value for small groups with specific needs, e.g., wikis, mobile
computing, selective release capability, etc., seemed to drive the decision. There are two areas that potentially could provide cost savings to
the University in the future. First – ePortfolios are becoming more widely used on campus as individual departments and programs are
realizing their value. Blackboard is the only product with a built-in portfolio that would meet the current needs across campus. We are
currently using a 3rd party application (Taskstream) – our portfolio initiatives over time could potentially be transitioned to the Blackboard
platform thus saving the annual licensing cost on the Taskstream application. The second area where cost savings is a potential would be the
consolidation of licenses. Currently the Law School carries a separate Blackboard license. Future upgrades to the next generation product
(Learn 9.2) would allow for one license for the campus with separate administrative control for each entity.
During the process there was discussion about the possibility of Villanova moving towards an open-source solution for the next LMS product.
After review by the faculty, it was determined that the current release of Sakai does not provide the broad spectrum of features required by
the faculty as a whole. As indicated by the Gartner targeted research, Appendix 6, open source solutions for higher education are expected to
have mainstream adoption within the next two years. The next release from the Sakai community is expected in early 2012. This release
appears to meet many of the functional requirements cited by the faculty. This could potentially be a longer range strategy to be considered
by the University.
In the final discussion, the main factor considered by faculty was the added functionality to the environment - new innovations without losing
the features of the current LMS. Support for mobile devices, more flexibility with testing, easier to use interface, etc., were some of the
discussion points that lead to the final recommendation. While all the evaluated products did retain most of the core functionality required,
the Blackboard Learn release is the only application that provided enhancements and also supported the integration of the many third party
products that are currently accessed through WebCT. Thus the Blackboard Learn system, with its familiar look and feel and significant
improvements, is the recommendation from the faculty as the outcome of this research process.
6
Appendix 1: Functional Requirements
Course Content
Ability to post various types of content
Reporting/Access to updated roster
Require the ability to access detailed downloadable reports of student
progress, grades, attendance, and enrollment based on individual or group
criteria
Security
System must be secure with SSl encryption and password protection for
copyright compliance
Integrated Gradebook
Instructors have the ability to record and release grades and provide feedback
privately to students.
Assignment Submission
An integrated assignment submission area gives instructors more control over
when and how assignments and content is released.
Discussion Board, Wikis, Blogs
Require intuitive interface with multiple forum capability; posts must be
gradable items with html editor
Easily Edit Course Elements
Require the ability to make edits to course structure without disrupting student
progress.
Integration support
System must be integrated w/LDAP/MyNova & Banner as well as 3rd party apps
currently in use
Automated Testing and Scoring
Student Tracking
Ease of Use
The testing tool must be able to create assessments that use the following types
of questions: Multiple Choice, Calculated Answer, and Short Answer/Essay.
Questions can be imported from existing test banks or can be both built with the
tool. An equation editor must be available. The testing tool can support timed test
submission and completion. The automated scoring can score multiple choice and
short answers type questions with optional immediate feedback.
Software must track both student-centered information (first access date, most
recent access date to all parts of course, e.g., readings and contributions, etc)
and content-centered information (number of accesses to each page of content,
average time spent on each page of content, etc).
The interface must intuitive; tools are easy to find and use.
Calendar
Ability to provide a calendar to list course events, requirements and
assignments
Notifications
Notification or Announcement capability
Academic Integrity
Must have a core feature to detect plagiarized submissions or capable of
integration with 3rd party app; must also integrate with Lockdown browser or
other feature to deter cheating
Synchronous collaboration tools
Ability to collaborate with students through chat , whiteboards or 3 rd party
software based integrated conferencing tool
7
Appendix 3: Business Decision Criteria
LMS Alternative Evaluation
BDM Consulting Group
The BDM Consulting Group an organization comprised of students from the Business Decision Making (DIT) class of Spring 2008 at
the Villanova Business School evaluated potential alternatives to the WebCT Learning Management System. To evaluate
alternatives, Analytical Hierarchy Process (AHP) and Decision
Lens software was used.
Assigned Relative
Weights to criteria
Organized Criteria by
topic/function;
Created parent and sub criteria
Segmented criteria into Technical and
Business Hierarchies
Aggregated Similar Criteria, resulting in 75 Criteria
Began with 105 Criteria
After performing the evaluation of potential LMS alternatives it was the
opinion of the BDM Consulting Group that Villanova University should
continue to license the WebCT (Blackboard) product for the duration of
their contract. This recommendation
stems from the results calculated using
the Decision Lens Suite and based on the
technical, functional and cost
comparisons provided by UNIT
A matrix of 105 criteria was considered
during the evaluation. These criteria formed the decision making model
that lead to a LMS recommendation.
A complete copy of the report prepared by BMD Consulting ( Sean Coady,
Marie Lokey, Duane Lacsamana, Matthew Didden, Scott Kosman, Seth
Strohl, Brian Cotroneo) is available.
8
Appendix 4: Survey on importance of Tools in next LMS
Average score out of 5; 114 respondents
Course content posting
5
4.66
4.5
Quizzing
3.66
Gradebook
Assignments/Drop box
3.66
Announcements
3.72
Plagerism detection
Synchronous
3.5
3.46
collaboration
Calendar
3.11
3.08
Whiteboard
Rubrics
2.71
2.71
4
3.5
3
2.5
Discussion board
2.85
Blogs
2.18
Wikis
2.2
ePortfolio
2.2
2
1.5
1
0.5
0
Tool
Average
Score
Tool
Average
Score
4.66
Synchronous
collaboration
3.11
Announcements
3.5
Discussion board
2.85
Assignments/
Drop box
3.72
Calendar
3.08
Blogs
2.18
Quizzing
3.66
ePortfolio
2.2
Wikis
2.2
Grade book
3.66
Plagiarism
detection
3.46
Whiteboard
2.71
Rubrics
2.71
Tool
Average
Score
Course content
posting
9
Appendix 5: Average Scores of Products by Pilot Group
Sakai
Avg score out of 5
Tool
Blackboard
Avg score out of 5
Desire2learn
Avg score out of 5
Discussions
3.2
4.75
4
Assignments/Drop box
3.6
4.5
3.85
Group tools for
presentations/share files
3
4.5
3.75
Chat (in course)
3.5
4.75
3.87
Streaming media (w/out HTML)
2.5
5
4
Whiteboard
3.1
4.3
3.5
Assign students to groups
(instructor)
4
4.75
4.4
Self tests
3.6
4.75
4.6
Chat (system wide)
2.3
3.6
2.3
Assign students to groups (random) 3.4
4.65
4.13
Course Builder (guide for
instructors)
3.1
4.15
4.65
Templates for standard pages
3.2
3.9
4.15
Quiz tool
3.85
4.6
4.27
Selective release
3.5
4.5
4.2
Context-sensitive Help
3
4.8
3.8
Student tracking
4.2
4.8
4.5
Automatically graded assessment
questions
3.9
4.9
4.6
Intuitive interface
3.1
4.5
4.25
Contemporary look and feel
3.2
4.7
4.1
N=
14
8
10
Total Average
3.33
4.55
4.05
10
Appendix 6: Gartner Targeted Research Published in July 2010
11