Learning Management System (LMS) Review Committee Final Report May 5, 2014 Beth Rubin, Ph.D. Assistant Provost for eLearning The Learning Management System (LMS) Review Committee was initiated in January of 2014, with representatives from all Miami divisions. Members include faculty (Jason Abbitt, Mike Brudzinski, Al Cady, Bruce D’Arcus, Michele Dickey, Michele Gingras, Lynette Hudiburgh, Norm Krumpe, Tom Mayes, Mark McBride, Glenn Platt) and staff (Jason Cardoso, Kent Covert, Janet Hurn, Pete Natale, Beth Rubin (chair), Lisa Santucci). The committee established a public Google folder, including all documentation of the process (https://drive.google.com/a/miamioh.edu/?tab=co&urp=https://drive.google.com/a/miamioh.edu/ ?tab%3Dco%26aut#folders/0B0DMdewEr4Yxa1BEajFfVEFXMjQ). Faculty members received copies of evaluations of LMSs conducted by Indiana University. In its initial meeting on February 7, the committee developed a process and finalized a set of criteria to use when evaluating LMS systems. It identified working subcommittees to develop a Request for Proposals (RFP), as well as a set of criteria reflecting essential characteristics to evaluate the proposals, a timeline, set of vendors that should be sent the RFP; and a set of responses to Frequently Asked Questions. The subcommittee drafted an RFP, and received nine responses. The set of essential criteria were turned into an evaluative framework, used to review all proposals. All vendors with 0 or1 from at least two evaluators on any one item were eliminated from consideration, and the numerical assessments were also aggregated. The result was two vendors that passed the evaluation: Desire2Learn (D2L) and Instructure/Canvas. These vendors presented to the university committee on March 17 at the Shriver Center, and projected via WebEx to the university committee; they were also recorded and the recordings placed on the public Google Site. Presentations were publicized via the Miami eReport and the Chairs Email List. Evaluations were distributed to all observers; this survey reflected the full set of criteria identified to evaluate LMSs. Approximately 50 people observed each presentation, with around half watching on WebEx. Twenty-three evaluation forms were completed; Canvas had higher average evaluations on 9 items, D2L on 5 items, although some differences were extremely small (e.g., less than 0.1 on a 1-4 scale). Following the presentations, all faculty and staff were invited to participate in the evaluation of the two LMSs as well as Niihka. A short course was created on each system, and participants were asked to conduct the tasks that they would when teaching, developing or taking a course supported by an LMS (e.g., uploading a syllabus, giving formative feedback). Each task was then assessed using a survey. Invitations were distributed using the Provost’s All Faculty email list and the ALT website; in addition, post cards were sent on April 8 to all full and part-time faculty at all campuses. Committee members were asked to invite their colleagues and students to participate, and a sample slide deck was prepared to facilitate presentations. In addition, a set of 21 testing lab sessions were held across Miami campuses, with staff available to support LMS Review Committee 5/20/14 1 faculty, staff and student testing. A raffle was offered to provide further encouragement for faculty to participate, with one entry for every evaluation submitted. Testing continued for approximately one month, from March 20 through April 21. During this time, two committee members who are “power users” of Niihka responded to the vendor presentations, providing context for the committee by comparing the tools of the new LMSes to their experience of using Niihka. Twenty-four faculty or staff evaluated D2L, 20 evaluated Canvas, and 23 evaluated Niihka; both basic and advanced tasks were evaluated. In addition, three students evaluated the LMSs. Data were aggregated and analyzed into a series of reports: ● Report of Vendor Presentations Surveys ● Report of Faculty Testing Surveys ● Report of Faculty Test Comparisons on Specific Tasks ● Report of Student Testing Surveys Other reports were also produced, including: ● LMS IT Considerations ● LMS Assessment Report ● LMS Migration Report A member of the Miami Accessibility Committee conducted an evaluation of accessibility of the potential new LMSs to people with disabilities, and drafted the LMS Accesibility 04282014 report. This evaluation found that both systems had processes to ensure accessibility, and both systems were generally accessible to people with visual impairments although each has a subsystem that is not yet fully accessible. D2L conducts an annual review to match its annual release cycle. Canvas, after initially having strong accessibility, had developed limitations due to system changes. However, Canvas has remediated the problems and implemented an ongoing review to match its continuous release cycle, including hiring an employee analyst who employs a screen reader. The Miami Accessibility Committee member joined that committee’s communication list. In addition, the committee reviewed Indiana University’s LMS evaluation final report, which was summarized by a committee member. It also reviewed D2L’s pricing proposal, which had been revised a week prior, as well as Canvas’s pricing proposal. The committee calculated that both systems were quite similar in pricing for the same set of options, although D2L had additional optional packages; while D2L’s first year proposal was slightly lower, its cost increased annually, and the average annual cost of Canvas over 5 years was slightly lower. The committee held a two-hour meeting on May 5, and considered all data and reports. After reviewing all evidence, the committee will voted on the following recommendations: ● Miami University should change from Niihka to a new LMS. The result was: o Yes: 12 o Abstain: 1 ● Miami University should adopt D2L or Canvas. The result was: o Canvas: 12 LMS Review Committee 5/20/14 2 o D2L: 1 As a result, the committee recommended to the Provost that Miami University should adopt Canvas as its LMS. As of May 2014, a review of Canvas is being conducted by General Counsel’s office, and financial implications are being assessed by means of a LEAN analysis. An implementation team will be formed as soon as the acquisition is confirmed; this team will oversee the transition. Academic representation will be sought from CoAD. Purchasing will be asked to negotiate a price, and the new system should be initially set up by the end of June, 2014. Instructional Designers will design course formats, and new courses developed in the summer term will be developed on the new LMS. Course materials will be transferred from Niihka to the new system, although automated transitions generally do not produce usable classes as materials need to be organized and uploaded into various system tools. Some faculty teaching online in fall 2014 will be invited to shift to the new LMS, and a phased rollout will be implemented. Trainings will begin in summer of 2014 and continue through the year, and student workers will be employed to re-create courses for faculty who choose to pilot the new system, term by term. Niihka will continue to be available to all faculty and staff through summer of 2015. LMS Review Committee 5/20/14 3