Report: 2014-2015 Canvas Pilot Evaluation
Executive Summary
This report summarizes an evaluation of the teaching and learning experience of Purdue students and
instructors using the Canvas learning management system (LMS) during the fall 2014 and spring 2015
semesters. It also provides comparative information about the Blackboard Learn and Canvas LMSs. The
goal of the Canvas LMS pilot evaluation and this report is to provide data to allow an informed
recommendation by the Canvas Pilot Steering Committee on Canvas as a potential replacement for
Blackboard Learn. Purdue’s current contract with Blackboard ends September 2017.
Potential options for steering committee recommendations include but are not limited to the following:
1) Continue using Blackboard Learn and renew the Blackboard Learn license when it expires in
September 2017.
2) Move to the Canvas LMS as Purdue University’s enterprise-wide LMS when the current license for
Blackboard Learn expires in September 2017.
3) Extend the Blackboard Learn contract an additional two years to 2019 giving both vendors time to
deliver on concerns identified in this report. In Fall 2016 conduct a review of progress LMSs have
made towards their roadmaps.
Data provided in this report:
 Impending Blackboard changes – since the beginning of this pilot, Blackboard has announced that
they are developing a new product called Learn 2015. The information about their new product is
provided in Impending Changes and also in more detail in Appendix L: Blackboard Road Map.
 Peer Institutions – a review of institutions who have recently evaluated Canvas - Of 21 institutions, 7
currently use Blackboard Learn and 8 use Canvas. Of these 21, 8 universities are piloting Canvas
(some of these are not currently using Blackboard). See Options for more details.
 Faculty and Student Evaluation - An assessment of both Blackboard and Canvas was conducted by
the Office of Institutional Assessment as part of the Canvas pilot. In summary, faculty indicated a
slight preference for Canvas and a slight majority believed we should move to Canvas. Some faculty
members pointed out issues which would make Canvas difficult for large classes (see Major Faculty
Issues). The results are summarized below (see Faculty and Student Evaluation). The full, detailed
analysis is provided in a separate report.
 Technical Issues and Considerations – Canvas would require changes in some current faculty
methods and procedures. In addition, a move to Canvas would require considerable application
work estimated at 8,700 hours (see Technical Issues and Considerations and Appendix A: Application
Redevelopment Needed for Canvas).
 Budget - Estimated cost comparison for each option:
NOTE: DETAILS REMOVED FOR POSTING
2014-2015 Canvas Pilot Evaluation
Summary
At the current time, the Canvas pilot has not provided any compelling reasons to change LMSs. Faculty
preferences lean toward Canvas, but not overwhelmingly.
The impending changes in Blackboard’s products point to an industry change that will impact Purdue.
Based on the changes in Blackboard, we predict that Purdue will need to shift to another LMS product
(either the new Blackboard product or a different one) within 5 years.
However, at this time we are unable to predict with any certainty the implications or timing of the
change.
Based on the information contained in this report, the Canvas Pilot Steering Committee recommends
option 3: Extend the Blackboard Learn contract an additional two years to 2019 giving both vendors
time to deliver on concerns identified in this report. In Fall 2016 conduct a review of progress LMSs have
made towards their roadmaps.
Table of Contents
1
Recommendation .................................................................................................................................. 1
2
Impending Changes ............................................................................................................................... 1
3
4
2.1
Furthering the Emphasis on Student-Centered Learning at Purdue ........................................ 1
2.2
LMS Trends................................................................................................................................ 1
2.3
Blackboard Changes .................................................................................................................. 2
2.4
Canvas Changes......................................................................................................................... 2
2.5
Compelling Reasons to Change? ............................................................................................... 2
Background and Goals .......................................................................................................................... 3
3.1
Purdue LMS History................................................................................................................... 3
3.2
OOC recommendation .............................................................................................................. 3
Options .................................................................................................................................................. 4
4.1
5
Pedagogical Considerations ...................................................................................................... 5
Faculty and Student Evaluation ............................................................................................................ 6
5.1
Fall 2014 pilot semester ............................................................................................................ 6
5.2
Spring 2015 pilot semester ....................................................................................................... 7
5.3
Faculty Showcase Comments.................................................................................................. 14
5.4
Major Faculty Issues................................................................................................................ 14
6
Functional Review and Comparison.................................................................................................... 15
7
Technical Issues and Considerations................................................................................................... 18
8
7.1
Architecture ............................................................................................................................ 18
7.2
Availability and Performance .................................................................................................. 18
7.3
Support of Learning Tools Interoperability Standards ............................................................ 19
7.4
Support for Critical Integrations ............................................................................................. 19
7.5
Migration of Blackboard Courses ............................................................................................ 19
7.6
Instructure Administrative and User Support ......................................................................... 19
7.7
Analytics and Reporting .......................................................................................................... 20
Budget ................................................................................................................................................. 20
Appendix ..................................................................................................................................................... 21
Appendix A: Application Redevelopment Needed for Canvas............................................................ 22
Appendix B: Canvas Pilot Project Team & Report Contributor List .................................................... 23
Appendix C: Canvas Pilot Steering Committee ................................................................................... 23
Appendix D: Canvas Communication & Outreach .............................................................................. 24
2014-2015 Canvas Pilot Evaluation
Appendix E: Fall 2014 Canvas Pilot Courses ........................................................................................ 27
Appendix F: Spring 2015 Canvas Courses ........................................................................................... 28
Appendix G: Colleges & Instructors Involved in Spring 2015 Pilot ..................................................... 32
Appendix H: Instructor Tool Usage ..................................................................................................... 34
Appendix I: LMS Evaluation Checklist ................................................................................................. 36
Appendix J: LTI and 3rd party integrations available during the pilot ................................................. 48
Appendix K: Faculty Showcase Summaries of Positives & Negatives ................................................. 49
Appendix L: Blackboard Road Map ..................................................................................................... 51
Appendix M: Cloud Computing ........................................................................................................... 52
Appendix N: Blackboard Learn 9.1 Release Update............................................................................ 53
Appendix O: Faculty and Student Surveys Sorted by Effect Size ........................................................ 55
2014-2015 Canvas Pilot Evaluation
1 RECOMMENDATION
The Steering Committee recommends adoption of option 3: Extend the Blackboard Learn contract an
additional two years to 2019 giving both vendors time to deliver on concerns identified in this report.
Further, the committee recommends that in Fall 2016 Purdue should conduct a review of progress each
LMS has made towards their roadmaps and managing issues with features identified in this report.
Rationale:
1) We currently have no compelling reason to move to Canvas or to move away from Blackboard Learn
9.1
2) Although faculty and students favored many Canvas features over Blackboard, several issues in
Canvas would prove difficult or impossible for some faculty to overcome (Gradebook, SpeedGrader,
and discussion boards, for example). These issues are not all based on pedagogy, but rather on class
size and limitations of these Canvas features.
3) Blackboard is moving to a new product (Learn 2015). Purdue will, within 5 years of this, need to
move to either Learn 2015 or another product. Extending the current contract for two years would
provide time to evaluate options and develop a migration plan.
2 IMPENDING CHANGES
2.1 FURTHERING THE EMPHASIS ON STUDENT-CENTERED LEARNING AT PURDUE
A May 18, 2015 announcement in Purdue Today provides guidance on a move to student-centered
learning at Purdue with the following: “Board members tasked Dutta with incorporating additional
criteria important to student success, as supported by university priorities and findings of the GallupPurdue Index. Changes will include more emphasis on mentorship and engagement with undergraduate
students; advising and mentoring to enhance the academic success of at-risk students; a recognition of
the value of including undergraduates in research; and credit for engaging in innovative pedagogy,
including the transformation of courses and curricula.“
(http://www.purdue.edu/newsroom/purduetoday/releases/2015/Q2/board-requires-amendments-topromotion-and-tenure-approves-two-supporting-documents.html)
2.2 LMS TRENDS
As higher education moves toward a student-centered learning experience, LMS systems are moving
from an instructor-focused system to a student-centric system with incorporation of social media tools
and easier student navigation and management.
In addition, the LMS business, like other technology industries, is undergoing a major shift currently
from on-premise solutions (where the LMS is physically installed and managed by the university) to a
cloud solution (where the LMS is hosted or managed by the LMS provider). Instructure’s Canvas LMS is
an example of a hosted solution. Purdue’s Blackboard Learn 9.1 is on-premise, though Blackboard does
offer a hosted option for Learn 9.1.
6/9/2015
2014-2015 Canvas Pilot Evaluation
2.3 BLACKBOARD CHANGES
2.3.1 Learn 2015/Ultra
Blackboard recently announced that it is developing a new product called Learn 2015. Learn 2015 is
described as a more student-centered product than Learn 9.1, providing new learning experiences which
bring their wide range of products and experiences together in a coherent way. The redesigned solution
will integrate products/services that are focused on “students' educational journey.” Blackboard states
the “New Learning Experience” is “Personal, Intuitive, Pervasive, Connected.”
Learn 2015 provides a new look and feel referred to as Ultra. This Ultra interface is a significant change.
Workflows are improved and navigation is simplified.
Learn 2015 will be offered initially as a hosted-only option this calendar year. Blackboard stated a
commitment to deliver an on-premise option. Blackboard expects to provide access to a hosted test
environment to Learn 2015 this summer. Several features will be missing or incomplete at launch this
summer with an unknown timeline for a full-featured product.
The impact on faculty to move to Learn 2015 will be similar to the shift from Vista to Learn. Technically
the impact is similar to the changes required for adopting Canvas, (estimated at 8,700 hours – see
Appendix A: Application Redevelopment Needed for Canvas).
2.3.2 Learn 9.1
Blackboard stated that they have no plans to eliminate Learn 9.1, although at some point they will stop
supporting their existing system. Blackboard announced “We intend to resume our biannual release
cadence next year with a Learn 9.1 April 2016 Release and a Learn 9.1 October 2016 Release” (See
Appendix N: Blackboard Learn 9.1 Release Update). If we decide to remain with Blackboard and go with
the two year contract extension it would be important to get Blackboards commitment to support
Purdue in Learn 9.1 until summer 2019.
Learn 2015 initial release (cloud only)
Learn 9.1 update (potentially last update)
Purdue Blackboard contract end date
Suspected end of 9.1 support
Summer 2015
October 2016
June 2017
October 2018
2.4 CANVAS CHANGES
Instructure’s Canvas roadmap suggests upcoming changes to the gradebook. (They recently extended
the number of gradebook entries beyond 2,000, but Purdue Instructors felt this was still not sufficient).
Their roadmap/future is based on their customer’s requests for change. Therefore at this time it is
difficult to note trends/changes in the future for Canvas.
2.5 COMPELLING REASONS TO CHANGE?
During presentations to both the Faculty Senate and the Teaching Academy we heard a consistent
message that changing LMSs at this point should only be considered if there were notable differences
and significant gain to be had from such a change. We were asked to make sure the committee
understood the amount of work required to make such a change and not to make a recommendation to
move to Canvas without a compelling reason.
2
2014-2015 Canvas Pilot Evaluation
At the current time, the Canvas pilot has not provided any compelling reasons to change LMSs. Faculty
preferences lean toward Canvas, but not overwhelmingly.
The impending changes in Blackboard’s products points to an industry change that will impact Purdue.
Based on the changes in Blackboard, we predict that Purdue will need to shift to another LMS product
(either the new Blackboard product or a different one) within 5 years.
However, at this time we are unable to predict with any certainty the implications or timing of the
change.
Jonathan Day, associate professor of hospitality and tourism management, summarized it best during a
Q&A for Purdue Today. When asked: “If you had to choose between using Canvas and Blackboard,
which would you choose and why?” he responded “Both Blackboard and Canvas provide solid tools for
learning management and both systems have their strengths and weaknesses. I don’t see a truly
significant difference between the two at this stage.”
3 BACKGROUND AND GOALS
3.1 PURDUE LMS HISTORY
In the late 1990s, the central IT organization at Purdue, PUCC (Purdue University Computing Center)
hosted WebCT at Purdue to provide instructors with the WebCT LMS for offering course content online.
In 2004, Purdue moved to WebCT’s Vista LMS. At this point, Purdue regional campuses (Calumet, North
Central, and IPFW) joined Purdue West Lafayette’s onsite LMS implementation. In 2006, WebCT was
acquired by Blackboard, Inc.; with a subsequent upgrade, WebCT Vista became Blackboard Vista. In
2012, the Purdue campus system moved to Blackboard Learn (with the exception of IPFW, which instead
chose a separate Blackboard-hosted version of Blackboard Learn).
3.2 OOC RECOMMENDATION
Pursuant to a recommendation made by a subcommittee of Purdue’s Operational Oversight Committee
(OOC), in July 2014 a Canvas pilot project team (Appendix B: Canvas Pilot Project Team) based in ITaP’s
Teaching & Learning Technologies group was formed to conduct a pilot of the Canvas LMS. A Canvas
Pilot Steering Committee was also created, chaired by Stephen Beaudoin, interim AVP for Academic
Affairs, and comprised of faculty, staff and students from Purdue’s West Lafayette colleges, key
administrative units, and regional campuses (see Appendix C: Canvas Pilot Steering Committee). The
Steering Committee provided oversight to the Canvas pilot project team and is charged to make a
recommendation to chart the future course of Purdue-wide LMS adoption before July 1, 2015.
The 2014-2015 pilot of the Canvas LMS was designed to provide data to the Steering Committee to
inform their recommendation. Key questions for the pilot included:
 How do the functions and tools of Canvas and Blackboard Learn compare? (see Functional
Review and Comparison)
 What Canvas tools did pilot instructors and students use? (see Instructor Tool Use)
 How does level or type of tool usage affect user perceptions of Canvas? (see Instructor Tool Use)
 What are pilot instructor and student opinions of Canvas vs Blackboard Learn? (see Faculty and
Student Evaluation)
 What challenges would instructors have if Purdue moves to Canvas? (see Major Faculty Issues)
 What development would be required for a move to Canvas? (see Technical Issues and
Considerations)
6/9/2015
2014-2015 Canvas Pilot Evaluation
4 OPTIONS
Based on discussions in the Steering Committee meetings, the following options have been offered as
consideration for the committee:



Continue using Blackboard Learn and renew the Blackboard Learn license when it expires in
September 2017.
Move to the Canvas LMS as Purdue University’s enterprise-wide LMS when the current license for
Blackboard Learn expires in September 2017.
Extend the Blackboard Learn contract an additional two years to 2019 giving both vendors time to
deliver on concerns identified in this report. In Fall 2016 conduct a review of progress LMSs have
made towards their roadmaps.
Much of the impetus behind the original recommendation of the OOC was the growing interest in, and
movement toward, Instructure’s Canvas LMS that has been growing in peer institutions of Purdue
University. Between 2012 and 2015, several have moved from another LMS to Canvas or have
embarked upon pilot projects of the Canvas LMS. The chart (see Figure 1) below provides information
on LMS usage and Canvas interest in Purdue’s peer institutions.
We have no reports of peer institutions using Blackboard who piloted Canvas and decided to remain
with Blackboard.
However we did hear of two non-peer institutes which piloted Canvas and remained with Blackboard:
The University of Southern California reported that they did conduct a pilot of Canvas, but decided not
to use Canvas (Kuzmich, C. 4/28/2015). Seattle Pacific University stated they did a pilot of Canvas but
elected to stay with Blackboard due to cost, and the level of disruption made the change not worth it
(Reidy, R. 4/29/2015).
4
*
*
*
* *
*
*
*
*
Cornell University
X Sakai, Online Degree MOOC w/Udacity
X X
Canvas (moved from Sakai)
X X
Desire2Learn; CourseWeaver, Canvas
Network for MOOCS
*
X
X
X
X
Northwestern University
* Ohio State University
* Pennsylvania State University
Rutgers University
Texas A&M
University of Arizona
*
University of California, Berkley
*
University of California, Davis
*
*
*
*
*
*
*
*
3
*
*
8 7
LMS
X Bb Learn; Canvas used by Medical
College & eCornell
Georgia Institute of Technology
* * Indiana University Bloomington
*
Michigan State University
* *
*
*
*
Big Ten
CIC
Peer
Using Canvas
Unizin
Moved
from
Bb
Learn
to
Canvas
Piloting Canvas
Using Blackboard
2014-2015 Canvas Pilot Evaluation
*
8 7
University of Chicago
University of Illinois
University of Iowa
University of Maryland
University of Michigan
University of Minnesota
University of Nebraska-Lincoln
University of Texas at Austin
University of Wisconsin-Madison
X
X
X
X
X
X
X
X
Canvas
X
Desire2Learn
X X Angel (Blackboard contract ends 2017)
X
Sakai; Moodle; Bb Learn; eCollege
X BbLearn
X Desire2Learn 92%, 7% Blackboard
X Canvas (moved from Sakai)
X Sakai, UC Online using Canvas, Piloting
Canvas, Brightspace
X
Bb Learn
X X Bb Learn; Moodle; LON-CAPA
X
Desire2Learn
X
Canvas
X X Sakai
X
Moodle
X
Bb Learn
X Canvas
X X Desire2Learn; Moodle
21
Figure 1: Current University LMS Use
4.1 PEDAGOGICAL CONSIDERATIONS
Since its launch in 2011, Instructure’s Canvas LMS has attracted more than 1,200 institutions, serving
more than 18 million students. Numerous institutions have cited Canvas' ease of use, by both faculty
and students, as a key factor in their decision to adopt the system.
Specifically, Michigan State University found that courses designed with Canvas "better support student
learning and better support effective teaching”. Further, Indiana University reported "Canvas also shines
when it comes to student-centered pedagogies”. Canvas offers time management tools, including
dynamic notifications, calendars, and contextual information. Across institutions, students consistently
rated the Canvas user interface positively. Consistent with our survey data, students of other institutions
overwhelmingly preferred the Canvas mobile app, available on iOS and Android, to those offered by
other products.
6/9/2015
2014-2015 Canvas Pilot Evaluation
As echoed in our faculty analysis, other Universities found that the integration among course tools in
Canvas significantly reduced the labor involved in building a new course. Faculty at several institutions
cited the SpeedGrader functionality of Canvas as among the most effective functionalities of the LMS,
allowing for easy and efficient grading of student work (see Major Faculty Issues for problems with
SpeedGrader).
5 FACULTY AND STUDENT EVALUATION
To effectively pilot and review Canvas, ITaP Communications worked with the Canvas Pilot Team to
develop a full marketing plan aimed at building student and faculty awareness of the pilot and options
for providing feedback. A copy of the completed plan is available in Canvas Communication & Outreach.
In consultation with the Steering Committee and the project team, an assessment plan for the pilot was
developed by the Office of Institutional Research, Assessment & Effectiveness (OIRAE). A complete copy
of the assessment report is provided as a separate document.
Eighteen colleges participated during the fall and spring pilots. For a list, see Appendices E, F and G.
5.1 FALL 2014 PILOT SEMESTER
Faculty and courses participating in the fall semester Canvas pilot were solicited by the Canvas project
team. Selection aimed for inclusion of a wide variety of departments and targeted faculty characterized
as ‘early adopters.’ Eight faculty taught eleven classes using the Canvas LMS, with a total enrollment of
298 students (for a list of faculty & courses see Appendix E: Fall 2014 Canvas Pilot Courses).
5.1.1 Faculty Evaluations
Data was collected mid-semester from 1 faculty member via interview and from 5 other faculty
members via focus group (details are available in the Assessment Report on page 52).
End of semester data was collected from 4 of the 8 faculty members through an online survey (available
in the Assessment Report on page 64). These results indicate satisfaction with Canvas features.
5.1.2 Student Evaluations
Mid-semester data was collected from students via 1 interview and 3 focus groups:
Interview
1
Survey (Calumet)
6
Focus groups
3
Focus group attendees
52
Received $10 gift cards
19
Overall, students had more positive comments about Canvas than Blackboard. Detailed results of the
interview and focus groups are available in the Assessment Report on page 56.
End of semester data was gathered from students via an online survey. Seventy-three students
completed the survey. Overall, although students were moderately to highly satisfied with each feature
of Canvas, more preferred Blackboard:
6
2014-2015 Canvas Pilot Evaluation
“Which do you like better, Canvas or Blackboard?”
Blackboard
31
Canvas
26
No preference
16
Detailed results are available in the Assessment Report on page 57.
5.2 SPRING 2015 PILOT SEMESTER
Publicity during fall 2014, through word of mouth, email, and Purdue Today articles, solicited interested
instructors to apply to have their courses included in the spring 2015 Canvas pilot. As a result, 49 Purdue
instructors used Canvas to deliver online instruction and materials to approximately 2,500 students
enrolled in 73 online class sections. The Canvas project team included the widest variety of courses
possible from those that applied. The spring 2015 Canvas pilot included several large enrollment (100+)
courses, fully online and hybrid courses, and non-academic sections. See Appendix F: Spring 2015
Canvas Courses for a detailed list.
5.2.1 Faculty Evaluations
Of the 49 faculty participating in the pilot, 36 responded to the end-of-semester survey.
Spring 2015 Canvas Courses
Campus
# Courses
WL - Academic
53
WL - Non Academic
1
CAL - Academic
16
PNC - Academic
5
Total Courses
75
Survey Participation
36
8
4
At the end of the end-of-semester spring 2015 survey faculty were asked which system they preferred.
Preference
Blackboard
Canvas
No preference
No response
Number of faculty
13
21
9
5
Faculty were then asked whether Purdue should move to Canvas or not.
Preference
Yes, move to Canvas
No, do not move to Canvas
Number of faculty
23
20
Faculty were provided a list of LMS features and asked to provide feedback on:
1. Their satisfaction level with Canvas,
2. Their satisfaction level with Blackboard and
3. How important the feature was.
Comparing the 25 most important features, overall, faculty preferred Canvas over Blackboard featureto-feature. The following chart summarizes instructor survey results:
6/9/2015
2014-2015 Canvas Pilot Evaluation
7
Feature
Gradebook
Setting a course up
Organization of the system
Setting up assignments
Uploading course files
Communicating with students/
Announcements
5.16 SpeedGrader
8
9
10
11
12
13
4.98
4.93
4.89
4.89
4.85
4.76
13
15
16
17
18
19
20
21
22
23
4.76
4.69
4.69
4.60
4.57
4.54
4.52
4.39
4.00
3.96
Manipulating course files
Look and feel
Tracking student performance
Giving quizzes
The student view
Helping students keep up with
course material
Groups
Calendar
Customizing the system
Rubrics
Kaltura
Modules
Use of plagiarism detection tools
Discussion boards
Incorporating tools from publishers
Taking Attendance
Meaningful
ness of
difference
5.67
5.34
5.34
5.26
5.26
5.19
Canvas (No.
of
responses)
Mean
import rank
1
2
3
4
4
6
Blackboard
(N0. of
responses)
Importance
Faculty Survey – LMS Features – Blackboard vs Canvas
4.38 (39)
4.29 (38)
4.08 (37)
4.15 (39)
4.08 (39)
4.84 (37)
Advantage
3.95 (40) Blackboard None (d=0.16)
4.75 (40)
Canvas
Small (d=0.22)
4.48 (40)
Canvas
None (d=0.18)
4.90 (40)
Canvas
Small (d=0.38)
4.95 (43)
Canvas
Small (d=0.42)
5.10 (49)
Canvas
None (d=0.16)
3.82 (22)
5.12 (33)
Canvas
3.83 (35)
3.92 (37)
4.14 (28)
4.31 (32)
4.27 (37)
4.11 (27)
4.55 (38)
4.66 (41)
4.42 (31)
4.57 (28)
4.33 (39)
4.71 (28)
Canvas
Canvas
Canvas
Canvas
Canvas
Canvas
3.65 (17)
3.61 (23)
3.93 (28)
4.12 (25)
4.19 (16)
4.13 (23)
4.62 (13)
4.36 (25)
4.14 (7)
2.60 (17)
3.89 (18)
5.21 (24)
3.77 (26)
4.00 (19)
3.83 (12)
4.63 (32)
4.13 (8)
4.33 (21)
4.13 (8)
4.41 (17)
Canvas
Canvas
Blackboard
Blackboard
Blackboard
Canvas
Blackboard
Blackboard
Blackboard
Canvas
Moderate
(d=0.59)
Small (d=0.35)
Small (d=0.34)
None (d=0.14)
None (d=0.03)
None (d=0.13)
None (d=0.14)
Small (d=0.33)
Large (d=0.90)
None (d=0.08)
None (d=0.04)
Small (d=0.39)
None (d=0.19)
Small (d=0.34)
None (d=0.01)
None (d=0.00)
Moderate
(d=0.56)
24 3.86 Scheduler
3.63 (8) 3.85 (13)
Canvas
None (d=0.08)
25 3.14 Learning mastery tool
3.80 (5)
3.80 (5)
Neither
None
Note: Meaningfulness of difference (effect size) range: 0.2 slightly meaningful, 0.5 moderately
meaningful, 0.8 large meaningful (Cohen, 1969)
To see this table sorted by effect size, please see Appendix O:
8
2014-2015 Canvas Pilot Evaluation
This is diagrammed in the following radial:
Detailed results are available in the Assessment Report on pages 9 thru 30.
5.2.2 Instructor Tool Use
To compare instructor evaluations based on their use of Canvas, an educational technologist ranked all
courses based on the time the instructor spent in Canvas, and divided them in 3 groups:
High user
Medium User
Low user
>70 hours of use
Between 25 and 70 hours of use
<25 hours of use
For the courses near the breaking points by +/- 5hrs on High/Medium and +/- 3hrs on Medium/Low. If a
course used 9 or more tools they were put in the higher category (see Appendix H: Instructor Tool Usage
for a list).
Canvas was rated higher in all categories than Blackboard. The difference for high usage courses was not
meaningful, whereas for medium and low usage courses the difference was slightly meaningful.
6/9/2015
2014-2015 Canvas Pilot Evaluation
Average score by usage
6
5
4
3
2
1
High
Medium
Blackboard Average Score
Low
Canvas
Details are available in the Assessment Report on page 12.
5.2.3 Instructor Delivery Method
Delivery type was determined through report by each individual instructor. For face-to-face and
primarily online courses there was a small meaningful difference between Canvas and Blackboard. For
face-to-face courses Canvas was rated higher (M=4.53) than Blackboard (M=3.88), and for primarily
online courses Blackboard (M=4.50) was rated higher than Canvas (M=3.96).
Average rating by course delivery type
6
5
4
3
2
1
0
Face-to-Face
Face-to-Face & online
Blackboard Learn Average Score
Details are available in the Assessment Report on page 13.
10
Primarily Online
Canvas Average Score
2014-2015 Canvas Pilot Evaluation
5.2.4 Sandbox Faculty Evaluations
In addition, all faculty were invited to create a ‘sandbox’ or demo course. Almost 200 sandboxes were
created. Sandbox faculty were asked to complete a survey to provide feedback. Twenty-two instructors
completed the survey.
Sandbox Courses
Campus
# Courses
WL
138
CAL
20
PNC
11
IPFW
14
Total Sandbox Courses
183
Sandbox users were asked which system they preferred. 10 of the 17 prefer Blackboard to Canvas.
Preference
Number of faculty
Blackboard
10
Canvas
4
No preference
3
Faculty with Sandbox courses were then asked whether Purdue should move to Canvas or not, and 13
did not think that Purdue should move to Canvas.
Preference
Number of faculty
Yes, move to Canvas
4
No, do not move to canvas
13
Detailed results are available in the Assessment Report on page 31.
5.2.5 Student Evaluations
Mid-semester data was collected from students via 4 focus groups:
Interview
Mid-semester focus groups
Focus group attendees
Received $5 gift cards
0
4
43
43
In addition, students in spring 2015 pilot courses were surveyed at the end-of-semester. 2,697 students
participated in pilot courses. Of these, 506 (19%) responded to the survey.
Spring 2015 Canvas Courses
Survey Participation
Campus
WL students
431
CAL students
64
PNC students
14
Total Courses
506
Students were provided a list of LMS features and asked to provide feedback on:
1. Their satisfaction level with Canvas,
2. Their satisfaction level with Blackboard and
3. How important the feature was.
6/9/2015
2014-2015 Canvas Pilot Evaluation
Meaningfulne
ss of
difference
Canvas (No.
of responses)
Feature
5.26 Knowing what my current grade is in
the course
2 5.24 Tracking grades on assignments
3 5.17 Accessing files my instructor has put up
4 5.14 Tracking assignments
5 5.10 Taking quizzes / tests
6 4.86 Managing and manipulating my files
7 4.81 Layout
8 4.73 Getting notifications
9 4.61 Interacting with course instructor
10 4.49 Participating in discussions
11 4.38 Participating in group work
12 4.30 Interacting with other students
Blackboard
(No. of
responses)
1
Mean
Importance
Comparing these features, overall, students preferred Canvas over Blackboard feature-to-feature. The
following chart summarizes student survey results:
Student Survey – LMS Features – Blackboard vs Canvas
4.46 (490)
4.72 (503) Canvas
None (d=0.14)
4.62 (488)
4.82 (492)
4.41 (491)
4.71 (454)
4.47 (458)
4.69 (494)
4.29 (480)
4.18 (388)
4.38 (394)
4.08 (375)
4.01 (380)
4.79 (504)
4.53 (505)
4.61 (505)
4.69 (426)
4.52 (466)
4.42 (507)
4.53 (500)
4.43 (396)
4.49 (379)
4.41 (379)
4.36 (382)
None (d=0.09)
None (d=0.18)
None (d=0.12)
None (d=0.01)
None (d=0.03)
None (d=0.13)
None (d=0.11)
None (d=0.12)
None (d=0.05)
None (d=0.16)
None (d=0.17)
Advantage
Canvas
Blackboard
Canvas
Blackboard
Canvas
Blackboard
Canvas
Canvas
Canvas
Canvas
Canvas
Note: Meaningfulness of difference (effect size) range: 0.2 slightly meaningful, 0.5 moderately meaningful, 0.8
large meaningful (Cohen, 1969)
To see this table sorted by effect size, please see Appendix O: This is diagrammed in the following radial:
Detailed results are available in the Assessment Report on pages 35 thru 46.
12
2014-2015 Canvas Pilot Evaluation
Students were also asked to compare the Blackboard and canvas Mobile tools. It should be noted that in
spring 2015 Blackboard began beta testing a free version of their mobile app. Comparing these features,
overall, students preferred Canvas over Blackboard feature-to-feature. The following chart summarizes
student survey results:
Meaningful
ness of
difference
Advantage
Canvas
(Number of
responses)
Blackboard
(Number of
responses)
Feature
Mean
importance
rank
Importance
Student Survey – Mobile App Comparison – Blackboard vs Canvas
1 5.19
Tracking course grade
4.23 (134) 4.68 (165) Canvas Small (d=0.32)
2 5.13
Tracking assignment grades
4.31 (134) 4.63 (169) Canvas Small (d=0.23)
3 5.06
Cost
3.98 (126) 5.02 (147) Canvas Moderate (d=0.67)
4 5.01
Organization
4.26 (136) 4.44 (172) Canvas None (d=0.13)
5 4.99
Layout
4.24 (136) 4.48 (172) Canvas None (d=0.18)
6 4.86
Submitting assignments
4.21 (109) 4.30 (131) Canvas None (d=0.06)
7 4.83
Taking tests / quizzes
4.11 (109) 4.43 (129) Canvas Small (d=0.22)
Note: Meaningfulness of difference (effect size) range: 0.2 slightly meaningful, 0.5 moderately
meaningful, 0.8 large meaningful (Cohen, 1969)
To see this table sorted by effect size, please see Appendix O:
This is diagrammed in the following radial:
Detailed results are available in the Assessment Report on page 47.
6/9/2015
2014-2015 Canvas Pilot Evaluation
5.3 FACULTY SHOWCASE COMMENTS
Several pilot faculty offered showcases where they presented their Canvas findings. The full schedule,
with links to recordings of these is in the following table:
Date/Time
Unique topics in each
session
Video Replay Gradebook limitations
Video Replay Modules, attendance, and a
student perspective
Video Replay Quirks and frustrations
Name
Department
Lynn Hegewald
Stacey Dearing
Engineering Education
English
Jonathon Day
Hospitality and Tourism
Management
Computer Graphics
Technology
Engineering Technology
Video Replay Canvas user interface and a Mihaela Vorvoreanu
course overview
Video Replay Mobile apps and a student
Anne Lucietto
perspective
Video Replay Building course using the
Michelle McMullin
English
calendar
A summary of each presenter’s comments is available in Appendix K: Faculty Showcase Summaries of
Positives & Negatives
5.4 MAJOR FACULTY ISSUES
Based on faculty comments in the assessment report, showcases, etc., the following are the major issues
instructors identified they would have if we moved to Canvas:
1. Gradebook:
a. Although recently increased, the Gradebook in Canvas has a limited number of items. Large
classes would have difficulty using the gradebook for this reason.
b. The Canvas gradebook only allows for one non-gradable column. Some faculty use columns
to indicate student team members, student major, etc.
2. SpeedGrader: although instructors liked this function, it will not work for classes of more than
500 students. In addition, it won’t work if there are 20 students and a large number of
gradebook columns. (2400 cells).
3. Grading Rubrics: The grader’s name appears in the grading rubric. In large classes, this grader
may be a TA.
4. Discussion boards: Peer reviews are not assigned within groups, but across the entire class.
Thus, if the instructor wants 6 peers to discuss their papers amongst themselves, he/she cannot
use peer reviews. This makes it difficult and confusing for students, as they find themselves
reviewing work for peers that are not in their groups and then doing further discussion with
those that are. Discussion cannot be anonymous (ex: Q&A)
5. Import: Canvas did not retain the look and feel of the Blackboard site after migration in some
instances. Some concern was noted from faculty who had spent a considerable amount of time
customizing Blackboard and the rework required to rebuild it in Canvas.
6. Images within assessments: Instructors sometimes choose to create quizzes with images that
students match with descriptions. Canvas does not allow assessments with images that students
match with descriptions. This is an issue for courses such as sign language, where assessment is
totally dependent upon image identification.
14
2014-2015 Canvas Pilot Evaluation
6 FUNCTIONAL REVIEW AND COMPARISON
While most learning management systems (LMSs) contain very similar tools and functionality, there is
some variation across platforms. While these variations may be small in number, they have the
potential to have a significant impact for the LMS user and for the learning process.
Using a template developed by the University of Toronto (available at http://testsoft.ati.utoronto.ca
/wordpress/avi/files/2013/03/LMSEvalChecklist.pdf), educational technologists on the Canvas Project
Team performed an exhaustive review of the tools and functions of the Blackboard and Canvas LMSs.
From the comprehensive list, the project team created the following list of 24 tools and functions that
stand out as important to instruction, and only exist in one of the LMSs. From that list, those that are
thought to be most critical to instructors are highlighted in red and, when necessary, annotated with
explanatory comments. Items highlighted in orange indicate a Canvas feature not available in
Blackboard. Items highlighted in blue indicate a Blackboard feature not available in Canvas.
Canvas vs. Blackboard Learn Function Review – Condensed List
Bb
Canvas
Details
1.1.0
1.4.4
1.5.3
1.5.4
2.0.0
DESIGN
The solution can notify/alert at-risk
students.
Module set-up allows assignments and
due dates to be easily seen in one list.
The solution has ability to enroll
teaching team members in a separate,
private group.
TOOLS
Y
N
N
Y
Y
Y*
2.3.5
The solution has whiteboard
presentation tools (embed PPT).
N
Y
2.3.6
The solution allows the whiteboard to
access/use an equation editor.
N
Y
Y
N
The solution uses a WYSIWYG or HTML
editor in the internal e-mail tool.
The solution has search capabilities in
2.5.2.3
the internal e-mail.
The solution allows users to create
2.5.2.4
folders for e-mails.
The solution allows users to archive
2.5.2.5
incoming and outgoing e-mails.
3.0.0
INTERFACE
The solution allows for searchable
3.2.1
content.
2.5.2.1
6/9/2015
n/a
*
n/a
*
n/a
*
Y
Y
N
Y
N
*Only by an administrator.
*Blackboard email is automatically
forwarded to an external account.
*Blackboard email is automatically
forwarded to an external account.
*Blackboard email is automatically
forwarded to an external account.
2014-2015 Canvas Pilot Evaluation
3.3.4
3.4.8
3.4.9
3.6.1
4.0.2.1
The solution enables authorized users
to send text message alerts.
The solution allows text-to-speech.
The solution has built-in transcription
tools or integration.
The solution allows for the ability to
have anonymous postings to discussion
board.
ASSESS
The solution’s grade book is able to
upload text columns.
The solution’s grade book has ability to
4.0.2.2 lock columns with flexible column
positions.
The solution’s grade book allows for
4.0.2.3
both % and point values.
Bb
Canvas
n/a
Y
N
Y
Details
In Canvas, must be enabled by
each person. Now in Bb, Social
tools disabled on purpose;
particularly important to regional
campuses.
Important for accessibility
N
Y
Important for accessibility
Y
N
Y
*Y/N
Y
N
Y
Y*
The issue: importing letter grades
isn't possible in Canvas UNLESS
grade schemas are enabled in
settings, and a column that
displays grade schema is created
before the upload in Canvas. Six
steps at least. This workaround is
arduous.
*All but quiz tool
*Instructors like/need to be able
to modify this column because it is
calculated - and as a result of
being an unmodifiable calculation,
it may not reflect the students'
final grade.
*Only for matching are images not
available. Work around solution use Respondus.
4.0.2.4
The solution’s grade book allows final
grade column to be changed.
Y
Y*
4.1.1.
The solution’s testing tool allows the
use of images.
Y
Y*
N
Y
Y
N*
*Points only
Y
N
TAs/undergrad graders’ names are
seen by students
Y
N
As ITaP will use CAS / Shibboleth
this is not an issue
The solution has skill assessment and
4.6.4
management capabilities and
competency gaps.
4.7.0.1 Rubrics have points/percentage ranges.
4.7.0.2 Rubrics allow grader anonymity.
5.0.0
ADMINISTRATIVE
5.0.2.4
The solution has the ability to support
authentication using Kerberos.
16
2014-2015 Canvas Pilot Evaluation
The solution has detailed security
elements such as encryption of sensitive
5.0.3.0 data, the enforcement of strong
passwords, the enforcement of
password changes, etc.
The solution has the ability to allow jobs
5.3.2.1
to be monitored from a GUI interface.
The solution has the ability to use
5.7.3
command line to create backups in
batch.
6.0.0
BUSINESS
The source code is freely available and
6.1.0
can be modified.
Bb
Canvas
Details
*We enforce our own password
rules. Unsure of Canvas impact
here. We didn’t have any jobs
running that we depend on for the
pilot
Y
n/a*
Y
N*
*hosted
Y
N*
*hosted
N
Y
Eleven items have been identified as important tools that exist in either Blackboard or Canvas but not in
both systems. It is critical to examine the usefulness of these tools in order to make an informed
decision about which LMS is best for its users. The following is an analysis of these:
11 Important Tools Available in One but not Both LMSs
1.4.4
The solution can notify/alert at-risk students: One advantage of Blackboard over Canvas is
that it can alert at-risk students. A hot trend in online education today, this early warning
system is correlated with higher retention rates and student success.
2.5.2.1
The solution uses a WYSIWYG or HTML editor in the internal e-mail tool: A second feature
that Blackboard contains is an HTML editor in the LMS email tool. The HTML editor allows
users to customize their text (e.g. color, size) and to include bulleted lists, special characters,
and even emoticons that, arguably, increase human connection.
3.3.4
The solution enables authorized users to send text message alerts: If a student would like to
send a copy of a project or paper to another student via the internal email tool, he/she is
able to retain their formatting. This is important because it allows for easier collaboration.
Students don’t have to go outside of the LMS to share work, and that can minimize learner
distractions. Students are sometimes asked to be peer graders or to proofread each other’s
work. When formatting is retained, they are able to do this more accurately. It promotes
team-based learning.
3.4.8
The solution allows text-to-speech: An advantage of Canvas over Blackboard is that is allows
for text-to-speech. This helps users type faster and provides them with flexibility to choose
their typing preference. It also helps people with physical disabilities who might not be able
to type on a keyboard. This can be seen as an important advantage as we seek to minimize
barriers and increase accessibility for learners.
3.4.9
The solution has built-in transcription tools or integration: Digital manuscripts help learners
who are hearing impaired or speak English as a second language. The large population of
international students at Purdue makes this tool especially important. Students can learn
easier and faster if they can both hear and see video contents.
4.0.2.1
The solution’s grade book is able to upload text columns: The text columns allow instructors
to keep track of relevant student data such as major, group number, seat number for exam,
etc.
6/9/2015
2014-2015 Canvas Pilot Evaluation
4.0.2.3
4.0.2.4
4.1.1.
4.7.0.1
4.7.0.2
The solution’s grade book allows for both percentage and point values: Rubrics made inside
of the LMS cannot contain percentages in Canvas. If faculty are using percentages and they
are displayed in the grade book, it confuses the students to see point values for
assignments. It can affect scores and cause confusion for students that can affect their
learning.
The solution’s grade book allows final grade column to be changed: Faculty sometimes
choose to modify final averages to reward students who did well in certain areas. If the new
grade cannot be reflected in the grade book, this can decrease learner motivation.
The solution’s testing tool allows the use of images: This is especially good for visual
learners. Instructors sometimes choose to create quizzes with images that students match
with descriptions, and Canvas does not allow for that type of assessment. This can be seen
as a limitation in taking all learning styles into account. This is also an issue for courses such
as sign language, where assessment is totally dependent upon image identification.
Rubrics have points/percentage ranges: Whereas Canvas requires complicated workarounds
for adding manual grade items and letter grades into the gradebook, Blackboard allows for
this with greater ease. In addition, the Blackboard gradebook allows for quizzes to be
calculated as a percentage or as point values and Canvas does not. Instructors need the
flexibility to be able to choose their aggregation method, and Canvas is limiting in this way.
Rubrics allow grader anonymity: Rubrics reveal the identity of the grader which causes
unrest for some faculty who rely heavily on student graders. It may cause test anxiety for
students if they know who is grading their work, and that could affect their grade and, more
importantly, their learning. This can also cause added stress to the student grader, who may
not feel free to provide candid feedback to fellow students.
The complete LMS functional review can be found in Appendix I: LMS Evaluation Checklist.
7 TECHNICAL ISSUES AND CONSIDERATIONS
7.1 ARCHITECTURE
Canvas is hosted on Amazon Web Services (AWS) as Software as a Service (SaaS) and uses a continuous
delivery model pushing improvements, fixes, and new features on a three-week release cycle without
downtime. This three-week release cycle is much quicker than the release history experienced with
Blackboard Learn and allows for more granular change. Users did not express any concerns with the
frequency or timing of change.
7.2 AVAILABILITY AND PERFORMANCE
Pilot users experienced one unplanned service outage where users were unable to login for
approximately 4 hours and there were two planned outages that occurred during off peak times and
lasted less than 5 minutes.
Storage and processing resources are provisioned and scaled automatically by load demand technology.
Purdue was notified by Canvas several times during the pilot that performance was degraded at some
institutions; however, no problems were reported at Purdue.
18
2014-2015 Canvas Pilot Evaluation
7.3 SUPPORT OF LEARNING TOOLS INTEROPERABILITY STANDARDS
Learning Tools Interoperability (LTI) provides an open, standard way for third-party tools and services to
connect to any LMS, providing an assortment of tools that faculty can incorporate into their courses.
Several LTI integrations were made available during the pilot, as requested by faculty, and all worked as
expected (see Appendix J: LTI and 3rd party integrations available during the pilot). During the course of
the pilot, LTI’s were developed for two of Purdue’s Studio applications which successfully provided
single sign-on with Canvas.
7.4 SUPPORT FOR CRITICAL INTEGRATIONS
Canvas team members were quickly able to configure Canvas with Shibboleth allowing the use of the
Purdue career account login credentials. They were also able to load all faculty, student and staff
accounts, for all campuses, into Canvas without incident or performance issues, resulting in more than
117,000 account creations. Course and enrollment data were manually uploaded daily during the pilot.
Current integration with Purdue’s student information system, Banner, is Blackboard-specific and will
need to be redeveloped for Canvas requiring considerable effort. Purdue currently has three Banner
systems that integrate with Blackboard Learn. Custom middle-ware has been developed by Purdue to
handle data collisions that can occur across three student information systems feeding into the LMS.
Real-time integration using Banner for eLearning (BIEL 8.1.x) leveraging IMS LIS 2.0 specification is the
latest solution available from Ellucian (Banner). However, Canvas will not support the LIS 2.0 standard
until the latter half of 2015. In the short term, API’s are available that would allow us to script
scheduled loads.
Along with integration to Banner, Purdue has developed integrations with other systems (such as
PREMIS and LON-CAPA ) as well as many supporting applications, such as the Grade Submit Tool and
Student Photo Rosters (see Appendix A: Application Redevelopment Needed for Canvas). Moving to
Canvas will require a significant redevelopment effort; these applications were developed over several
years for Blackboard.
7.5 MIGRATION OF BLACKBOARD COURSES
Course content can be exported from Blackboard and imported into Canvas using the IMS Common
Cartridge Specification (CC) which defines a format for moving content from one LMS to another.
Canvas supports this specification and courses migrated fairly well from Blackboard. However, resulting
content organization was affected and some faculty found it easier to rebuild from scratch in Canvas. In
some migration instances Canvas did not retain the look and feel of the Blackboard site after migration.
Some concern was noted from faculty that had spent a considerable amount of time customizing
Blackboard and the rework required to rebuild it in Canvas.
7.6 INSTRUCTURE ADMINISTRATIVE AND USER SUPPORT
Canvas administrative and reporting tools are very limited and configuration options that allow for
customization, such as user roles and permissions, are not as granular as those available with
Blackboard. Supporting a highly complex organization at scale may be challenging.
During the pilot, Purdue provided first tier support to faculty and students. Any issues that couldn’t be
resolved quickly or easily were escalated to Instructure. Instructure provided 24/7 support and no
problems were encountered. Sixty-four (64) tickets were filed over approximately 10 months. Most of
these tickets were “how-to” questions.
6/9/2015
2014-2015 Canvas Pilot Evaluation
7.7 ANALYTICS AND REPORTING
Instructure recently announced a new Hosted Data Service providing a live warehouse of a school's data,
optimized for analytics and reporting and available via any ODBC compliant analytics tool (Tableau, Excel,
R, others) to generate custom Data Visualization and reports. This service became available recently in
beta. Purdue has not tested this new service.
8 BUDGET
NOTE: DETAILS REMOVED FOR POSTING
20
2014-2015 Canvas Pilot Evaluation
APPENDIX
6/9/2015
2014-2015 Canvas Pilot Evaluation
APPENDIX A: APPLICATION REDEVELOPMENT NEEDED FOR CANVAS
Application Redevelopment Needed for Canvas
Application or Building Block
Master Course Creator
Admin Enrollment Status
Change
Brief Description
Allows instructors and support accounts to create a
course to use as a course template to be reused each
semester.
Allows admins to enable and disable enrollments in
courses by course ID along with changing the Source ID
for an enrollment.
Estimate to
Redevelop
(Hours)
400
400
Enrollment Manager
Allows an instructor, blended course Instructor and
support accounts to enroll users as Instructors, TA, and
Graders in their academic courses.
400
Grade Submit
Currently allows instructors of record to push final
grades to the appropriate Banner system.
800
Instructor Self-Serve Course
Merge Tool
Course Email Lists
Photo Roster
VPEC - Respect Boundaries
courses
ICP - Initial Course
Participation for Financial Aid
Allows an instructor to merge sections of his or her
courses appropriately.
Allows an instructor to create an email list for one or
more sections.
Allows instructors to view the photos provided by card
services for each enrollment in his or her section.
800
400
400
Bb courses required for all new students.
600
Record date of last activity for financial aid.
400
Confluence Front End
Create wiki for course and add students & instructor.
200
LON-CAPA Daily Grade Sync
LON-CAPA Daily Roster Sync
Learn Rubric export for ECN
Sync grades LON-CAPA to Canvas
Sync rosters Canvas to LON-CAPA
400
400
400
Digital Desk grade uploader
Scantron grades uploaded to Canvas grade book.
400
SIS integration
Premis Integration
TOTAL
Integrations needed for all 3 Banner systems
Send accounts, courses and enrollments to Canvas.
22
1500
800
8,700
2014-2015 Canvas Pilot Evaluation
APPENDIX B: CANVAS PILOT PROJECT TEAM & REPORT CONTRIBUTOR LIST
Instructional Development Center, TLT, ITaP
Suzanne Ahlersmeyer
Donalee Attardo (Project Director)
Sheree Buikema
Brett Creech
Amy Haston
David Huckleberry
Julie Kercher-Updike (Project Oversight)
Pat Reid (Project Lead)
Debbie Runshe
Eric Yee (Project Coordinator)
IT Application Services, ITaP
Raj Pal
Judy Rantz (Project Manager)
Leo Smith
Pat Wilson
ITaP Communications
Andrea Thomas (Communications Specialist)
Office of Institutional Assessment
Brooke Robertshaw (Assessment Analyst)
APPENDIX C: CANVAS PILOT STEERING COMMITTEE
Name
Steve Beaudoin
Julie Kercher-Updike
Donalee Attardo
Troy Bassett
Robert Louis Bill
Bart Collins
Jennifer Dennis
Aharon Hannan
George Hollich
Ananth Iyer
Logan Jordan
Dawn Laux
George McCabe
David McKinnis
Gail Newton
Anastasia Trekles
Karen VanGorder
Natasha Watkins
Sunnie Watson
Heather Zamojski
6/9/2015
Area
Chair - Interim Associate Provost for Academic Affairs, Faculty, Chemical
Engineering
Co-Chair - AVP, IT Teaching and Learning Technologies & IT Customer
Relations
Director, IT Teaching and Learning Technologies
Faculty, English and Linguistics
Asst Dean Academic Affairs, Basic Medical Sciences
Faculty, Communication (HHSE) & CLA
Faculty, Agriculture
Student Representative (ITaP SAC)
Faculty, Psychological Sciences
Faculty, Purdue NExT
Faculty, Management
Faculty, Computer and Information Technology
Faculty, Statistics (Science) & OVPIT Operational Oversight Committee
Director, Technical Assistance Program & OVPIT Operational Oversight
Committee
Faculty, Pharmacy
Faculty, Education (Purdue North Central)
Continuing Studies, PU Fort Wayne
Faculty, Health and Human Sciences
Faculty, Education
Director, Calumet Academic & Research Computing
2014-2015 Canvas Pilot Evaluation
APPENDIX D: CANVAS COMMUNICATION & OUTREACH
Articles
Date
Published
Headline and link to online story (all articles appeared in Purdue Today and Faculty
Focus)
09/03/2014 Canvas pilot gives faculty, staff, students a chance to explore next-generation learning
management system
10/14/2014 Tech Tuesday to feature demo of Canvas learning management system
01/16/2015 Canvas pilot expands to further evaluate next-generation learning management system
03/25/2015 Faculty invited to attend April information sessions on learning management system
Canvas
04/07/2015 Faculty member weighs in on pros, cons of Canvas learning management system
04/27/2015 Professor discusses experiences using Canvas learning management system
CANVAS INFORMATION SESSIONS FOR FACULTY
(Advertised in Purdue Today, Faculty Focus, Facebook and Twitter)
Instructor
Lynn Hegewald
Stacey Dearing
Department Time
Date
ENGR
2:30-3:30 p.m. Wednesday, April 1
ENGLISH
1:30-2:30 p.m. Thursday, April 2
Deb Pratt
EDUCATION 11-1 p.m. (CST) Wednesday, April 8
Location
Physics, Room 202
Nelson Hall, Room 1215
Technology Building, Room 298
(Purdue North Central)
Beering Hall, Room 1255
Jonathan Day
HTM
3:30-4:30 p.m. Monday, April 13
Mihaela
TECH
1:30-2:30 p.m. Thursday, April 16
Beering Hall, Room 1254
Vorvoreanu
*Anne Lucietto
TECH
2:30-3:30 p.m. Wednesday, April 22 Lilly Hall, Room G420
*Michelle McMullin ENGLISH
9:30-10:30 a.m. Monday, April 27
Beering Hall, Room B206
*These presentations were streamed via WebEx. All events were recorded and published on the Canvas
website.
CANVAS INFORMATION SESSIONS FOR STUDENTS
(Advertised in Purdue Today, Faculty Focus, Facebook, Twitter, Ventfull, Purdue Exponent, Blackboard
login page)
ITaP hosted three Canvas information sessions geared toward students that were led by student trainers.
Those sessions took place as follows:
24

5-5:30 p.m. April 1, Beering Hall, Room 2280

3-3:30 p.m. April 7, Third Street Suites, Room 159

5:30-6 p.m. April 16, Mechanical Engineering Building, Room 1015
2014-2015 Canvas Pilot Evaluation
SOCIAL MEDIA
This overview outlines activity that took place related to the Canvas Session and Canvas Q&A articles,
and the Purdue IT social media accounts for Canvas information sessions. The articles were posted on
March 30, April 1 and April 7 to the ITaP newsroom and Purdue IT social media sites.
The term “views” represents the number of times the ITaP homepage, service alert article or Facebook
post was seen. The terms “Like” and “Favorite” are the different terms used by Facebook and Twitter as
a means of liking the post. The terms “Share” and “Retweet” are the different terms used by Facebook
and Twitter representing individuals distributing the post.
Canvas Session Article Views
March 30:
170
April 1:
53
April 7:
8
Total:
231
Twitter views
March 30:
696
April 1:
392
April 7:
505
Total:
1,593
Shares and Retweets
March 30:
4(F), 3(T)
April 1:
3(F), 1(T)
April 7:
3(F), 2(T)
Total:
10(F), 6(T)
6/9/2015
Canvas Q&A Article Views
March 30:
0
April 1:
0
April 7:
22
Total:
22
Facebook views
March 30:
260
April 1:
225
April 7:
283
Total:
768
Likes and Favorites
March 30:
5(F), 1(T)
April 1:
3(F), 1(T)
April 7:
9(F), 2(T)
Total:
17(F), 4(T)
Total Article views
March 30:
170
April 1:
53
April 7:
30
Total:
253
Total Social Media views
March 30:
956
April 1:
617
April 7:
788
Total:
2,361
2014-2015 Canvas Pilot Evaluation
ADDITIONAL COMMUNICATIONS AND OUTREACH
1. Associate Vice Provost Stephen Beaudoin gave an update on the Canvas LMS Pilot
during the April 20 meeting of the University Senate.
2. ITaP staff provided status reports to the ITaP Student Advisory Committee, including
members of the Purdue Student Government.
3. Associate Vice President Julie Kercher-Updike and Director Donalee Attardo gave an
update to the Teaching Academy.
4. Students, staff and faculty also had the opportunity to enroll in a Canvas demo course here.
26
2014-2015 Canvas Pilot Evaluation
APPENDIX E: FALL 2014 CANVAS PILOT COURSES
Instructor
WL - Academic
Vorvoreanu Mihaela
Collins
Bart
Newby
Tim
Day
Jonathan
Day
Jonathan
Thorlton
Janet
Thorlton
Janet
Lybrook
Dan
Lybrook
Dan
Course
Fall-2014-CGT-51200-001
Fall-2014-COM-40600-001
Fall-2014-EDCI-27000-010
Fall-2014-HTM-21200-002
Fall-2014-HTM-53100-001
Fall-2014-NUR-62600-001
Fall-2014-NUR-63200-001
Fall-2014-OLS-37600-001
Fall-2014-OLS-37600-003
Enrollment
19
16
54
71
13
6
20
17
32
College
WL - College of Technology
WL - College of Liberal Arts
WL - College of Education
WL - College of Health & Human Sci
WL - College of Health & Human Sci
WL - College of Health & Human Sci
WL - College of Health & Human Sci
WL - College of Technology
WL - College of Technology
Department
College of Technology Admin
Communication
Curriculum and Instruction
School of Hosp & Tourism Mgmt
School of Hosp & Tourism Mgmt
School of Nursing
School of Nursing
Org Leadership & Supervision
Org Leadership & Supervision
248
WL - Non Academic
Initial Lean Supply Chain
Proficiency
50 WL - Non-Academic
CAL - Academic
Hixon
Emily
Fall-2014-EDPS-22000-02
27 CAL - College of Education
Educational Studies
IPFW - Academic
Wolf
Mike
Fall-2014-POLY103-04
26 IPFW - College of Arts and Sciences
Political Science
Dunlop
Steve
Total Student Enrollments
6/9/2015
351
27
2014-2015 Canvas Pilot Evaluation
APPENDIX F: SPRING 2015 CANVAS COURSES
Enroll
ment
Instructor
Course
WL - Academic
Dumbacher Daniel
Downey
Scott
Shay
Robin
O'Shea
Kevin
Natt
Jane
Collins
Bart
Spring-2015-AAE-59000-058
Spring-2015-AGEC-33100-001
Spring-2015-ASL-10100-004
Spring-2015-COM-21700-003
Spring-2015-COM-25200-003
Spring-2015-COM-31800-001
30
311
20
16
13
331
Morgan
Melanie
Vemulapalli Tracy
Adams
George
Spring-2015-COM-67400-002
Spring-2015-CPB-35100-002
Spring-2015-CS-25000-LE1
Huston
Richards
Newby
Newby
Grant
Dearing
McMullin
Grant
Haynes
Bay
Hegewald
Hegewald
Hegewald
Lynch
Fehrman
Fehrman
Spring-2015-ECET-12000-001
Spring-2015-ECET-30201-001
Spring-2015-EDCI-27000-001
Spring-2015-EDCI-27000-010
Spring-2015-ENGL-10600-010
Spring-2015-ENGL-10600-257
Spring-2015-ENGL-10600-791
Spring-2015-ENGL-10800-001
Spring-2015-ENGL-30900-001
Spring-2015-ENGL-48800-001
Spring-2015-ENGR-13100-001
Spring-2015-ENGR-13100-002
Spring-2015-ENGR-13100-005
Spring-2015-GRAD-59000-011
Spring-2015-GS-10100-001
Spring-2015-GS-10100-002
0
0 WL - College of Veterinary Medicine
116 WL - College of Science
22 WL - College of Technology
27 WL - College of Technology
77 WL - College of Education
71 WL - College of Education
17 WL - College of Liberal Arts
20 WL - College of Liberal Arts
12 WL - College of Liberal Arts
19 WL - College of Liberal Arts
19 WL - College of Liberal Arts
5 WL - College of Liberal Arts
115 WL - College of Engineering
113 WL - College of Engineering
40 WL - College of Engineering
79 WL - Graduate School
15 WL - College of Liberal Arts
5 WL - College of Liberal Arts
28
Davin
Grant
Tim
Tim
Carrie
Stacey
Michelle
Carrie
Linda
Jennifer
Lynn
Lynn
Lynn
Cyndi
Sarah Elizabeth
Sarah Elizabeth
College
WL - College of Engineering
WL - College of Agriculture
WL - College of Health & Human Sci
WL - College of Liberal Arts
WL - College of Liberal Arts
WL - College of Liberal Arts
WL - College of Liberal Arts
Department
Aeronautics and Astronautics
Agricultural Economics
Speech, Lang and Hearing Sci
Communication
Communication
Communication
Communication
Dept of Comparative Pathobiology
Computer Science
Electrical & Comp Engr Tech
Electrical & Comp Engr Tech
Curriculum and Instruction
Curriculum and Instruction
English
English
English
English
English
English
Engineering Education
Engineering Education
Engineering Education
Graduate School Administration
General Studies
General Studies
6/9/2015
2014-2015 Canvas Pilot Evaluation
WL – Academic, cont.
Fehrman
Sarah Elizabeth
Bush
Deborah Hannah
Climer
Ty C
Moriarty
Stephen William
Albertson Chelsy Jo
Climer
Ty C
Albertson Chelsy Jo
Moriarty
Stephen William
Albertson Chelsy Jo
Moriarty
Stephen William
Baechle
Jennifer N
Bush
Deborah Hannah
Baechle
Jennifer N
Bush
Deborah Hannah
Watkins
Natasha
Fleetham Debbie
Day
Jonathan
Day
Jonathan
Dumbacher Daniel
Berdanier Reid Adam
Ardekani
Arezoo M
Vlachos
Pavlos P
Lucietto
Anne
Richards
Grant
Lucietto
Anne
Spring-2015-GS-10100-004
Spring-2015-GS-10100-008
Spring-2015-GS-10100-010
Spring-2015-GS-10100-013
Spring-2015-GS-10100-015
Spring-2015-GS-10100-016
Spring-2015-GS-10100-017
Spring-2015-GS-10100-018
Spring-2015-GS-10100-019
Spring-2015-GS-10100-020
Spring-2015-GS-10100-021
Spring-2015-GS-10100-022
Spring-2015-GS-10100-023
Spring-2015-GS-10100-X01
Spring-2015-HDFS-34300-001
Spring-2015-HIST-31800-001
Spring-2015-HTM-21200-001
Spring-2015-HTM-39900-001
Spring-2015-IE-59000-007
Spring-2015-ME-30900-001
Spring-2015-ME-30900-002
Spring-2015-ME-30900-160
Spring-2015-MET-22000-001
Spring-2015-MET-28400-001
Spring-2015-MET-31300-001
13
7
4
13
12
10
10
6
6
5
12
11
8
3
22
15
59
20
1
59
55
57
91
92
57
Ye
Lybrook
Lybrook
Spring-2015-MGMT-59000-011
Spring-2015-OLS-37600-001
Spring-2015-OLS-37600-003
0
34
34
6/9/2015
Chen
Dan
Dan
WL - College of Liberal Arts
WL - College of Liberal Arts
WL - College of Liberal Arts
WL - College of Liberal Arts
WL - College of Liberal Arts
WL - College of Liberal Arts
WL - College of Liberal Arts
WL - College of Liberal Arts
WL - College of Liberal Arts
WL - College of Liberal Arts
WL - College of Liberal Arts
WL - College of Liberal Arts
WL - College of Liberal Arts
WL - College of Liberal Arts
WL - College of Health & Human Sci
WL - College of Liberal Arts
WL - College of Health & Human Sci
WL - College of Health & Human Sci
General Studies
General Studies
General Studies
General Studies
General Studies
General Studies
General Studies
General Studies
General Studies
General Studies
General Studies
General Studies
General Studies
General Studies
Human Dev and Family Studies
History
School of Hosp & Tourism Mgmt
School of Hosp & Tourism Mgmt
WL - College of Engineering
WL - College of Technology
WL - College of Technology
WL - College of Technology
WL - College of Technology
WL - College of Technology
WL - College of Technology
WL - College of Business
WL - College of Technology
WL - College of Technology
Industrial Engineering
School of Engineering Tech
School of Engineering Tech
School of Engineering Tech
School of Engineering Tech
School of Engineering Tech
School of Engineering Tech
School of Mgmt Adm & Instr
Org Leadership & Supervision
Org Leadership & Supervision
29
2014-2015 Canvas Pilot Evaluation
WL – Academic, cont.
Tenorio
Cecelia
Tenorio
Cecelia
Tenorio
Cecelia
Vorvoreanu Mihaela
Spring-2015-SPAN-32200-001
Spring-2015-SPAN-32200-002
Spring-2015-SPAN-32200-003
Spring-2015-TECH-69700-011
15
14
16
10
WL - College of Liberal Arts
WL - College of Liberal Arts
WL - College of Liberal Arts
WL - College of Technology
School of Languages & Cultures
School of Languages & Cultures
School of Languages & Cultures
College of Technology Admin
2264
WL - Non Academic
Dunlop
Steve
Calahan
Charles
Alge
Brad
Initial Lean Spply Chn Proficiency
Intercultural Learning 201
OBHR 59000 - Leadership
20
0
20
WL - Non-academic
WL - Non-Acad
WL - Non-Acad
40
CAL - Academic
Ye
Chen
Green
LaVon
Green
LaVon
Green
LaVon
Mick
Michael
Ahmed
Ashfaq
Hixon
Emily
Fewer
Colin
Mabrito
Mark
Mabrito
Mark
St Jean
Wendy
Mich
Claudia
Mich
Claudia
Firlej
Kasia
Tazbir
Janice
Wendell
Dane
Spring-2015-CIS-14000-01
Spring-2015-CIS-20400-02
Spring-2015-CIS-20400-08
Spring-2015-CIS-20400-16
Spring-2015-CIS-46300-01
Spring-2015-ECET-15900-01
Spring-2015-EDPS-22000-02
Spring-2015-ENGL-20100-02
Spring-2015-ENGL-43700-01
Spring-2015-ENGL-43700-02
Spring-2015-HIST-15100-02
Spring-2015-MKG-32400-01
Spring-2015-MKG-32400-03
Spring-2015-MKG-42800-01
Spring-2015-NUR-48701-01
Spring-2015-POL-31500-01
24
36
34
31
26
20
26
14
21
1
36
36
32
13
25
9
CAL - College of Business
CAL - College of Business
CAL - College of Business
CAL - College of Business
CAL - College of Business
CAL - College of Technology
CAL - College of Education
CAL - Coll of Libl Arts & Soc Sciences
CAL - Coll of Libl Arts & Soc Sciences
CAL - Coll of Libl Arts & Soc Sciences
CAL - Coll of Libl Arts & Soc Sciences
Computer Info Services
Computer Info Services
Computer Info Services
Computer Info Services
Computer Info Services
ECET-Electrical&Comp Engr Tech
Educational Studies
English
English
English
History
CAL - College of Business
CAL - College of Business
CAL - College of Business
CAL - College of Nursing
CAL - Coll of Libl Arts & Soc Sciences
Marketing
Marketing
Marketing
Nursing
Political Science
384
30
6/9/2015
2014-2015 Canvas Pilot Evaluation
PNC - Academic
Witulski
Joanna
Shires
Jeff
Pratt
David
Pratt
Debra
Coates
Annette
Spring 2015-ASL-10200-002
Spring 2015-COM-33000-001
Spring 2015-EDPS-23500-002
Spring 2015-EDPS-27500-002
Spring 2015-NUR-49500-001
11
10
14
10
26
PNC - College of Health & Human Sci Speech, Lang and Hearing Sci
PNC - College of Liberal Arts
Communication
PNC - College of Education
Educational Studies
PNC - College of Education
Educational Studies
PNC - College of Health & Human Sci Nursing
71
Total Student Enrollments
2759
(NOTE: Krannert was unable to participate as they were migrating from an internal LMS to Blackboard)
6/9/2015
31
2014-2015 Canvas Pilot Evaluation
APPENDIX G: COLLEGES & INSTRUCTORS INVOLVED IN SPRING 2015 PILOT
Number of Courses by Campus & College
WL - Non-Academic
Courses
WL - Exploratory Studies
Dropped
WL - Graduate School
WL - Honors College
WL - College of Veterinary Medicine
WL - College of Technology
WL - College of Science
WL - College of Pharmacy
WL - Krannert School of Management
WL - College of Liberal Arts
WL - College of Health & Human Sci
WL - College of Engineering
WL - College of Education
WL - College of Agriculture
PNC - College of Liberal Arts
PNC - College of Health & Human Sci
PNC - College of Education
IPFW - College of Arts and Sciences
CAL - College of Technology
CAL - College of Nursing
CAL - College of Liberal Arts and Social Sciences
CAL - College of Education
CAL - College of Business
0
5
10
15
20
25
30
35
(‘Dropped’ indicates courses that indicated they would participate in the Canvas pilot, but dropped out)
(NOTE: Krannert was unable to participate as they were migrating from an internal LMS to Blackboard)
32
6/9/2015
2014-2015 Canvas Pilot Evaluation
Number of Instructors by Campus & College
WL - Non-Acad
WL - Graduate School
WL - College of Technology
WL - College of Science
WL - College of Liberal Arts
WL - College of Health & Human Sci
WL - College of Engineering
WL - College of Education
WL - College of Agriculture
PNC - College of Liberal Arts
PNC - College of Health & Human Sci
PNC - College of Education
IPFW - College of Arts and Sciences
CAL - College of Technology
CAL - College of Nursing
CAL - College of Liberal Arts and Social Sciences
CAL - College of Education
CAL - College of Business
0
2
4
6
8
10
12
14
16
18
(NOTE: Krannert was unable to participate as they were migrating from an internal LMS to Blackboard)
6/9/2015
33
2014-2015 Canvas Pilot Evaluation
APPENDIX H: INSTRUCTOR TOOL USAGE
Instructor Name
Reid Adam Berdanier
Emily Hixon
Kasia Firlej
Carrie Grant
Kevin O'Shea
Carrie Grant
Linda Haynes
Cyndi Lynch
Tim Newby
Anne Lucietto
Debbie Fleetham
Lynn Hegewald
LaVon Green
Jonathan Day
Debra Pratt
Amy Cochran
Mark Mabrito
Anne Lucietto
Jonathan Day
Cecelia Tenorio
Pavlos P Vlachos
Arezoo Ardekani
Deborah Hannah Bush
Tim Newby
Natasha Watkins
Stacey Dearing
Jane Natt
34
Course
Spring-2015-ME-30900-001
Spring-2015-EDPS-22000-02
Spring-2015-MKG-42800-01
Spring-2015-ENGL-10600-010
Spring-2015-COM-21700-003
Spring-2015-ENGL-10800-001
Spring-2015-ENGL-30900-001
Spring-2015-GRAD-59000-011
Spring-2015-EDCI-27000-010
Spring-2015-MET-22000-001
Spring-2015-HIST-31800-001
Spring-2015-ENGR-13100-001
Spring-2015-CIS-20400-02
Spring-2015-HTM-21200-001
Spring 2015-EDPS-27500-002
Spring-2015-AGEC-33100-001
Spring-2015-ENGL-43700-01
Spring-2015-MET-31300-001
Spring-2015-HTM-39900-001
Spanish For Hlth Professions 001
Spring-2015-ME-30900-160
Spring-2015-ME-30900-001
Spring-2015-GS-10100-008
Spring-2015-EDCI-27000-001
Spring-2015-HDFS-34300-001
Spring-2015-ENGL-10600-257
Spring-2015-COM-25200-003
# of Tools
Time in Canvas
10
805:32:40
10
374:23:11
7
284:57:20
10
169:49:50
9
167:49:42
10
160:44:45
12
149:07:58
11
136:39:21
8
133:35:09
10
106:34:22
10
102:37:51
12
88:34:37
8
78:20:26
11
77:30:53
10
76:11:00
9
73:55:00
9
69:37:07
10
66:24:24
10
65:03:37
5
201:07:26
10
491:01:01
10
1018:29:57
7
71:34:10
7
67:52:05
7
65:10:46
10
64:01:14
7
56:19:56
Category Based
on Time
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
MEDIUM
MEDIUM
MEDIUM
HIGH
HIGH
HIGH
HIGH
MEDIUM
MEDIUM
MEDIUM
MEDIUM
Category with Tools >=9
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
MEDIUM
MEDIUM
MEDIUM
MEDIUM
MEDIUM
6/9/2015
2014-2015 Canvas Pilot Evaluation
Instructor Name
Sarah Elizabeth Fehrman
Claudia Mich
Jennifer N Baechle
Mihaela Vorvoreanu
Bart Collins
Ashfaq Ahmed
Chelsy Jo Albertson
Colin Fewer
Stephen William Moriarty
Daniel Dumbacher
David Pratt
Michelle McMullin
Robin Shay
Dane Wendell
Chen Ye
Annette Coates
Wendy St Jean
Jeff Shires
Michael Mick
Joanna Witulski
George Adams
Jennifer Bay
Ty C Climer
Dan Lybrook
Grant Richards
Janice Tazbir
Davin Huston
6/9/2015
Course
Spring-2015-GS-10100-001
Spring-2015-MKG-32400-03
Spring-2015-GS-10100-021
Spring-2015-TECH-69700-011
Spring-2015-COM-31800-001
Spring-2015-ECET-15900-01
Spring-2015-GS-10100-015
Spring-2015-ENGL-20100-02
Spring-2015-GS-10100-013
Spring-2015-IE-59000-007
Spring 2015-EDPS-23500-002
Spring-2015-ENGL-10600-791
Spring-2015-ASL-10100-004
Spring-2015-POL-31500-01
Spring-2015-CIS-14000-01
Spring 2015-NUR-49500-001
Spring-2015-HIST-15100-02
Spring 2015-COM-33000-001
Spring-2015-CIS-46300-01
Spring 2015-ASL-10200-002
Spring-2015-CS-25000-LE1
Spring-2015-ENGL-48800-001
Spring-2015-GS-10100-010
Spring-2015-OLS-37600-001
Spring-2015-ECET-30201-001
Spring-2015-NUR-48701-01
Spring-2015-ECET-12000-001
# of Tools
6
8
7
6
7
9
9
6
5
10
9
6
7
8
7
5
4
8
7
6
3
5
8
5
7
5
2
Time in Canvas
48:56:47
47:32:59
43:07:47
43:05:07
41:52:54
40:32:47
35:37:32
31:20:15
29:55:24
25:16:00
23:55:27
27:39:18
22:30:41
22:19:52
20:16:12
20:08:47
17:53:22
16:46:46
16:39:55
14:29:52
13:49:33
8:19:36
3:47:00
3:05:26
2:34:29
1:59:35
0:50:39
Category Based
on Time
MEDIUM
MEDIUM
MEDIUM
MEDIUM
MEDIUM
MEDIUM
MEDIUM
MEDIUM
MEDIUM
MEDIUM
LOW
MEDIUM
LOW
LOW
LOW
LOW
LOW
LOW
LOW
LOW
LOW
LOW
LOW
LOW
LOW
LOW
LOW
Category with Tools >=9
MEDIUM
MEDIUM
MEDIUM
MEDIUM
MEDIUM
MEDIUM
MEDIUM
MEDIUM
MEDIUM
MEDIUM
MEDIUM
LOW
LOW
LOW
LOW
LOW
LOW
LOW
LOW
LOW
LOW
LOW
LOW
LOW
LOW
LOW
LOW
35
2014-2015 Canvas Pilot Evaluation
APPENDIX I: LMS EVALUATION CHECKLIST
The original, blank checklist was created by ACT@UofT (http://uoft.me/act - ati@utoronto.ca). We made a few additions based on the needs of
our faculty.
It is important to note that some of the responses can vary depending on various factors. For Blackboard, we based our responses on what we
currently have available for campus use. Features available to our campus users are marked with a “Y” (yes). Unavailable features are marked “N.”
Some of the features marked “N” (no) may actually be available on other versions or with different setups.
1.1.0
1.1.1
1.1.2
1.1.3
1.1.4
1.1.5
1.1.6
1.1.7
1.1.8
1.2.1
1.2.2
1.2.3
1.2.4
1.3.1
1.3.2
1.3.3
1.3.4
1.3.5
1.4.1
36
DESIGN
The solution has differentiated delivery methods (e.g., instructor-led,
self- paced).
The solution has adaptive release.
The solution can share content between course instances/courses.
The solution can import and manage content.
The solution includes re-usable learning objects.
The solution includes external hyperlinking.
The solution can create course templates.
The solution can assign ownership of course templates to
individuals/groups.
The solution has integration with library solutions.
The solution can tag content items with metadata.
The solution can use different metadata schemes for tagging content.
The solution can integrate publisher content via e-packs, cartridges, or
packages.
The solution can support a wide range of third-party courseware.
The solution can integrate with plagiarism detection.
The solution can integrate with third-party exam solutions.
The solution can integrate with student response solutions (clickers).
The solution has portfolio solutions through single sign-on.
The solution can map curricular objectives.
Bb
Y
Canvas
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
N*
Y
Y
Y
Y
Y
Y
Available in next release
(Oct. 2014)
6/9/2015
2014-2015 Canvas Pilot Evaluation
1.1.0
1.4.2
DESIGN
The solution can map course requirements.
Bb
N*
Canvas
Y
1.4.3
The solution can map learning outcomes.
N*
Y
1.4.4
1.5.1
1.5.2
The solution can notify/alert at-risk students.
The solution can brand institutional courses.
The solution can accommodate custom banners, headers, footers and
icons.
Module set-up allows assignments and due dates to be easily seen
in one list.
The solution has ability to enroll teaching team members in a separate,
private group.
TOOLS
The solution has discussion forums.
The solution has the ability to use WYSIWYG or HTML editor in the
discussion forums.
The solution has the ability to attach documents in discussion forums.
The solution has the ability to embed links in discussion forums.
The solution has group creation in discussion forums.
The solution has extended group functions/activities.
The solution has the ability for threaded discussions.
The solution has searchable forums.
The solution has private discussions that cannot be viewed by the
teacher.
The solution has a blog tool.
The solution has a WYSIWYG or HTML editor within the blog tool.
The solution has a portfolio tool.
The solution has the ability to view discrete parts of the portfolio
publicly.
The solution has a journal tool.
The solution has a WYSIWYG or HTML editor within the journal tool.
The solution has a wiki tool.
The solution has the ability to do wiki versioning control.
Y
Y
Y*
N
Y
Y*
N
Y
Y
Y*
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
1.5.3
1.5.4
2.0.0.
2.1.1
2.1.1.1
2.1.2
2.1.3
2.1.4
2.1.5
2.1.6
2.1.7
2.1.8
2.2.1
2.2.1.1
2.2.2
2.2.2.1
2.2.3
2.2.3.1
2.2.4
2.2.4.1
6/9/2015
Available in next release
(Oct. 2014)
Available in next release
(Oct. 2014)
*Limited
*Only by admin.
37
2014-2015 Canvas Pilot Evaluation
2.0.0.
2.2.4.2
2.2.5
2.2.6
2.2.7
2.3.1
2.3.1.1
TOOLS
The solution has a WYSIWYG or HTML editor within the wiki tool.
The solution has survey tools.
The solution has the ability to embed videos/images.
The solution has a mashups tool.
The solution has instant messaging (live chat).
The solution allows the user to hide his/her presence in IM tool.
Y
Y
Y
Y
Y
n/a*
Y
Y
Y
Y
Y
n/a*
2.3.2
2.3.3
2.3.4
2.3.5
2.3.6
2.4.1
2.4.2
2.4.3
2.4.4
2.4.5
2.4.6
2.4.7
2.4.8
2.5.1
2.5.2
The solution has IM file sharing.
The solution has IM two-way video.
The solution has a live whiteboard feature.
The solution has whiteboard presentation tools (embed PPT).
The solution allows the whiteboard to access/use an equation editor.
The solution has built-in assistive technologies.
The solution has group assignments.
The solution has linear module creation.
The solution has decision-tree module creation.
The solution has bookmark areas within the course.
The solution allows users to make notations within the course.
The solution has mind mapping tools.
The solution has software coding workspaces.
The solution has internal email.
The solution has rules within the e-mail tool.
N
N
Y
N
N
Y
Y
Y
Y
N
Y
Y
Y
Y
n/a*
N
N
Y
Y
Y
Y
Y
Y
Y
N
Y
Y
Y
Y
n/a
2.5.2.1
The solution uses a WYSIWYG or HTML editor in the internal e-mail
tool.
The solution allows users to forward internal e-mail to an external
account.
The solution has search capabilities in the internal e-mail.
Y
N
Y
Y
n/a*
Y
2.5.2.2
2.5.2.3
38
*We do not allow users to
hide their presence.
*Blackboard email is
automatically forwarded to
an external account.
*Blackboard email is
automatically forwarded to
an external account.
6/9/2015
2014-2015 Canvas Pilot Evaluation
2.0.0.
2.5.2.4
TOOLS
The solution allows users to create folders for e-mails.
n/a*
N
2.5.2.5
The solution allows users to archive incoming and outgoing e-mails.
n/a*
Y
2.6.1
2.6.2
2.6.3
2.6.4
2.6.5
The solution has calendaring tools.
The solution can create reoccurring calendar entries.
The solution can set start and stop time for calendar events.
The solution allows users to subscribe to an external calendar.
The solution allows users to add calendar events from any tool with due
date.
INTERFACE
The solution allows user to export/import course.
The solution allows user to export/import tests/quizzes.
The solution allows user to package a course to import into other
platforms.
The solution allows the copy of specific items to new course.
The solution allows common cartridge export.
The solution allows for searchable content.
The solution has RSS feeds.
The solution allows for moderated discussion forums.
The solution allows logging/saving of chat transcripts
The solution has drag and drop features (e.g., AJAX).
The solution has WYSIWYG editing that allows formatting styles (e.g.,
heading styles, bullet lists).etc.)
The solution allows editor to modify HTML pages created outside
solution.
The solution allows edits after posting.
The solution has spell check.
The solution has adaptive release features.
The solution allows authorized users to change the file size limit on
submitted files.
Y
Y
Y
N
Y
Y
Y
Y
N
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
N
Y
Y
Y
Y
Y
N
N
Y
Y
Y
Y
Y
Y
Y
Y
3.0.0
3.1.1
3.1.2
3.1.3
3.1.4
3.1.5
3.2.1
3.2.2
3.2.3
3.2.4
3.2.5
3.2.6
3.2.6.1
3.2.7
3.2.7.1
3.2.8
3.2.9
6/9/2015
*Blackboard email is
automatically forwarded to
an external account.
*Blackboard email is
automatically forwarded to
an external account.
39
2014-2015 Canvas Pilot Evaluation
3.0.0
3.3.1
3.3.2
3.3.3
3.3.4
INTERFACE
The solution allows for active announcements.
The solution allows for copying of announcements to new courses.
The solution allows users to receive announcements outside solution.
The solution enables authorized users to send text message alerts
3.4.1
The solution is accessibility compliant with most national or state
standards.
The solution allows captioning.
The solution allows captions in assessments.
The solution supports the use of screen readers (e.g., JAWS).
The solution has screen preferences, font size, masking, and color
options.
The solution has screen/cursor magnifiers.
The solution supports screen/document readers.
The solution has a text-only interface.
The solution allows text-to-speech.
The solution has voice recognition/word prediction.
The solution has built-in transcription tools or integration.
The solution allows mobile devices to access course content.
The solution is mobile-ready when using the Web version through
smart phones.
The solution is mobile-ready when using the Web version through
tablets.
The solution ensures that all course tools are available in the mobile
client.
The solution ensures tools in the mobile client have the same
functionality as the web version.
The solution allows for the ability to have anonymous postings.
The solution has a fully functional equation editor.
The solution has warnings when leaving course site.
The solution enables social connections inside solution.
The solution allows for social media third-party integration.
The solution integrates with YouTube.
3.4.2
3.4.2.1
3.4.3
3.4.4
3.4.5
3.4.6
3.4.7
3.4.8
3.4.8.1
3.4.9
3.5.1
3.5.2
3.5.3
3.5.4
3.5.5
3.6.1
3.6.2
3.7.1
3.8.1
3.8.1.1
3.8.2
40
Y
Y
Y
n/a
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
N
N
N
N
Y
Y
Y
Y
N
Y
N
Y
Y
Y
Y
Y
N
N
N
N
Y
Y
N
n/a*
n/a*
Y
N
Y
N
Y
Y
Y
Social tools disabled on
purpose
*Social functions disabled.
*Social functions disabled.
6/9/2015
2014-2015 Canvas Pilot Evaluation
3.0.0
3.8.3
3.8.4
3.8.5
3.9.1
3.9.1.1
3.9.1.2
3.9.1.3
3.9.1.4
3.9.1.5
3.9.2
3.9.3
4.0.0
4.0.1
4.0.2
4.0.2.1
4.0.2.2
4.0.2.3
4.0.2.4
4.0.3
4.0.4
4.0.5
4.0.6
4.0.7
4.0.8
4.0.9
4.1.0
4.1.1.
6/9/2015
INTERFACE
The solution has direct connection to publisher content.
The solution allows for integration with developer extension API.
The solution is compliant with learning content standards (e.g.,
SCORM).
The solution has browser capability checker.
The solution supports Chrome.
The solution supports Firefox.
The solution supports Internet Explorer.
The solution supports Opera.
The solution supports Safari.
The solution has support for alternative languages.
The solution has a built-in translator.
ASSESS
The solution has evaluation, testing, and assessment engines.
The solution has a centralized grade book.
The solution’s grade book is able to upload text columns.
The solution’s grade book has ability to lock columns with flexible
column positions.
The solution’s grade book allows for both % and point values.
The solution’s grade book allows final grade column to be changed.
The solution has multiple assignment tools (e.g., lessons, quizzes, tests).
The solution has a quiz pool/question bank.
The solution has features that enable WYSIWYG authoring in
assessments.
The solution allows browser lockdown techniques to ensure
assessment security.
The solution allows IP restrictions to ensure assessment can only be
taken in certain locations (campus lab).
The solution allows proctor passwords to ensure assessment security.
The solution has the ability to import and export items in QTI (IMS
Question and Test Interoperability).
The solution has peer review tools.
The solution’s testing tool allows the use of images.
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
N
Y
Y
Y
Y
Y
Y
Y
N
Y
Y
Y
Y
Y
Y
Y
Y
Y
N
N
Y
Y
Y
Y
Y
Y*
Y*
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y*
*All but quiz tool
*Not a direct modification
*Work around solution
41
2014-2015 Canvas Pilot Evaluation
4.0.0
4.1.1.1
4.1.1.2
4.1.1.3
4.1.1.4
4.1.1.5
4.1.2
4.1.3
4.1.4
4.1.6
4.1.7
4.1.8
4.1.9
4.2.1
4.2.2
4.2.3
4.2.4
4.2.5
4.2.6
4.2.9
4.4.0
4.4.0.1
4.4.0.2
42
ASSESS
Quiz Question Type: Multiple Choice
Quiz Question Type: Long Answer/Paragraph
Quiz Question Type: Matching
Quiz Question Type: Fill in the Blank/Short Answer
Quiz Question Type: True or False
The solution’s testing tool allows the use of math symbols.
The solution has the ability to add audio/video components to quizzes
or exams.
The solution has the ability to embed videos into assessments.
The solution has the ability to selectively release assessments based on
grade book criteria.
The solution has the ability to selectively release assessments based on
multiple criteria.
The solution’s testing tool can randomize quiz answers.
The solution’s testing tool can randomize quiz questions.
The solution has the ability to selectively release assessments based on
date.
The solution has the ability to selectively release assessments based on
time.
The solution has the ability to selectively release assessments based on
student ID.
The solution has the ability to set conditions (e.g., hide/show, dates) for
multiple assessments at the same time.
The solution has the ability to administer tests based on proficiency
requirements.
The solution has the ability to administer tests based on group
performance.
The solution has ability to set different assessment time frames for
individual students (e.g., students with disabilities who need more
time).
The solution includes multi-source assessment tools.
Assignment Submissions: PDF
Assignment Submissions: Video
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
6/9/2015
2014-2015 Canvas Pilot Evaluation
4.0.0
4.4.0.3
4.4.0.4
4.4.0.5
4.4.1
4.4.2
4.5.0.0
4.5.0.1
4.5.0.2
4.5.0.3
4.5.0.4
4.5.1
4.5.2
4.5.3
4.5.5
4.5.6
4.5.7
4.5.9
4.6.1
4.6.2
6/9/2015
ASSESS
Assignment Submissions: Web Page
Assignment Submissions: Word Doc
Assignment Submissions: Zip File
The solution has the ability for students to submit papers online and
allow instructors to grade and give feedback without downloading the
assignment.
The solution has the ability to accept student Web pages and
collaborative projects with multimedia.
The solution has the ability to export and import from Excel into a
grade book.
The solution has the ability to grade objective quizzes automatically and
reveal results to students.
The solution has the ability to re-grade assessments.
The solution has student tracking.
The solution has the ability for students to see their grades.
The solution has the ability to import/export grades from the LMS using
APIs.
The solution has the ability to support statistical analysis of assessment
results.
The solution has the ability to send grades from LMS to student
information solution.
The solution has the ability to grade discussion postings.
The solution has the ability to see student submission dates for an
assignment in the grade book.
The solution has the ability to create standard and customized reports
on an individual.
The solution has the ability to generate results that are weighted,
averaged, or compared.
The solution has the ability to provide detailed tracking of when
students select an answer (e.g., time stamp of answer choice) within
the assessment tool.
The solution has ability to provide a variety of feedback types (e.g., text,
audio, video) for assignments, discussions, and other student activities.
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
43
2014-2015 Canvas Pilot Evaluation
4.0.0
4.6.3
4.6.4
4.7.0
4.7.0.1
4.7.0.2
4.7.0.3
4.7.3
4.7.4
4.8.1
4.8.2
4.8.3
4.8.4
5.0.0
5.0.1.1
5.0.1.2
5.0.2.0
5.0.2.1
5.0.2.2
5.0.2.3
5.0.2.4
5.0.2.5
5.0.2.6
5.0.2.7
5.0.3.0
44
ASSESS
The solution has the ability to create graded pre- and post-test surveys.
The solution has skill assessment and management capabilities and
competency gaps.
The solution has the ability to integrate course content with rubrics.
Rubrics have points/percentage ranges.
Rubrics allow grader anonymity.
Rubrics have ability to add text to each cell to describe criteria.
The solution has the ability to use rubrics to grade discussion forums.
The solution has the ability to use rubrics to grade essay questions
within an online assessment.
The solution has the ability to share grades between integrated tools
and the LMS.
The solution has assessments that can be exported to another course.
The solution has self-assessment tools.
The solution supports integration with third-party assessment tools
ADMIN
The solution has a database back end.
The solution has the ability to apply institutional branding.
The solution has security measures that typically include passwords and
encryption.
The solution is SSO SAML (Security Assertion Markup Language)
compliant.
The solution has the ability to support SSL.
The solution has the ability to support single sign on.
The solution has the ability to support authentication using Kerberos.
The solution has the ability to support authentication using LDAP.
The solution has the ability to support authentication using Shibboleth.
The solution has the ability to support authentication using Active
Directory.
The solution has detailed security elements such as encryption of
sensitive data, the enforcement of strong passwords, the enforcement
of password changes, etc.
Y
N
Y
Y
Y
Y
Y
Y
Y
Y
Y
N*
N
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
N
Y
Y
Y
Y
n/a*
* Points only
*We enforce our own
password rules
6/9/2015
2014-2015 Canvas Pilot Evaluation
5.0.0
5.0.3.2
5.0.3.3
5.1.1
5.1.2
5.2.0.0
5.2.0.1
5.2.0.2
5.2.0.4
5.2.0.5
5.2.0.6
5.2.1
5.2.2
5.2.3
5.2.4
5.2.5
5.2.6
5.3.0
5.3.0.1
5.3.0.2
5.3.0.3
6/9/2015
ADMIN
The solution has the ability to ensure the highest degree of student
privacy requirements (e.g., FERPA, FIPPA, etc.).
The solution has the ability to protect confidential data and session
activity both within the application and in transit.
The solution has the ability for solution admins to edit active (solution)
announcements.
The solution has the ability for solution admins to edit/delete any role’s
institution announcements.
The solution has the ability to define roles.
The solution has the ability to create an institutional hierarchy.
The solution has the ability to manage user registrations and profiles.
The solution has the ability to assign privileges to course roles (various
instructor levels).
The solution has the ability to assign privileges to solution roles (various
admin levels).
The solution has the ability to integrate with HR solutions.
The solution has the ability to combine sections via the SIS integration.
The solution has the ability to allow automated class creation,
enrollments, withdrawals, etc. via an API.
The solution has the ability to assign course templates via the SIS
integration.
The solution has the ability to assign admins to a selected group of
courses.
The solution has the ability for admins/faculty to combine sections.
The solution has the ability to administratively set students’ LMS email
address to a campus standard solution for all students in all courses.
The solution has a range of reporting options in the LMS.
The solution has detailed student solution tracking available (log in, log
out, last accessed, IP).
The solution has the ability to track attendance in the grade book (e.g.,
last login date).
Detailed student tracking is available for each course tool (e.g., number
of discussion posts read, time spent on each quiz question, etc.).
Y
Y*
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
N
Y
Y
N
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
*Via roles/permissions
45
2014-2015 Canvas Pilot Evaluation
5.0.0
5.3.1
5.3.2.1
5.3.2.2
5.5.1
5.6.1
5.7.1
5.7.2
5.7.3
5.7.4
5.8.1
5.8.2
6.0.0
6.0.0
6.0.1
6.0.2
6.1.0
6.1.1
6.1.2
6.2.1
6.2.3
6.2.4
46
ADMIN
The solution has reporting that allows drilling down to individual and
program level activity.
The solution has the ability to allow jobs to be monitored from a GUI
interface.
The solution has the ability to allow maintenance and solution jobs to
be changed by solution admins.
The solution has the ability for students and faculty to see who is
logged into courses.
The solution has the ability to turn on/off solution settings at various
levels in the hierarchy.
The solution has the ability to support mass notifications for users
during emergencies, including any third party product integrations.
The solution has the ability to restore student artifacts in a course if a
formerly withdrawn student re-enrolls.
The solution has the ability to use command line to create backups in
batch.
The solution has the ability to handle backup and retrieval options at
the solution administrator level.
The solution has the ability to create archives in an open format.
The solution has the ability to maintain complete course and student
records in a locked/proprietary format.
BUSINESS
The solution has a local hosting option.
For hosted solutions, the servers are located in Amazon Web Services.
The solution is directly-run or SaaS.
The source code is freely available and can be modified.
The solution is written in an industry-standard language.
JAVA is required to access multiple aspects of solution.
The solution has the ability to scale the implementation for increased
usage.
The solution has concurrent admin user capabilities.
The solution has flexible and expandable storage parameters per user
type.
Y
Y
Y
N*
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
N*
Y
Y
N
Y
N
Y
Y
Y
Y
N
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
*hosted
*hosted
6/9/2015
2014-2015 Canvas Pilot Evaluation
6.0.0
6.2.5
6.4.1
6.8.1
6.9.1
6.9.2
6.9.3
6.9.4
6.9.5
6.9.6
6.9.9
6/9/2015
BUSINESS
The solution includes a comprehensive redundancy plan.
If the solution includes a native mobile client, then access is standard
and free.
The solution has the ability to integrate with financial transaction
solutions.
Training is offered for institutional support staff.
Training materials are available at no extra cost.
Training materials and documentation are available for use to train
faculty and students.
The solution includes embedded, prerecorded, online training
materials.
Training occurs on campus.
Institution is allowed to modify the documentation to use with own
branding.
The solution provider has user groups or customer advisory councils.
Y
N*
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
*For next release (Bb
student free)
47
2014-2015 Canvas Pilot Evaluation
APPENDIX J: LTI AND 3RD PARTY INTEGRATIONS AVAILABLE DURING THE PILOT
Kaltura streaming video
Boilercast (Echo360)
Turnitin (plagiarism checking)
McGraw-Hill Campus
Cengage
Pearson
Passport
Hotseat
IDP uploads (Scantron)
VoiceThread
iClicker
48
2014-2015 Canvas Pilot Evaluation
APPENDIX K: FACULTY SHOWCASE SUMMARIES OF POSITIVES & NEGATIVES
Instructor A – 3 courses with student numbers ranging from about 40 to 120, face-to-face course format
with online homework
× Issue with peer grading and the grader names show up in the grading rubrics
o (There are new features in Canvas to make peer reviewing anonymous, but were
released late in the semester and not tested)
o Anonymous grading is not an option for rubrics
× Not being able to add more than one non-gradable column in the gradebook to hold additional
text information like major, group names, teaching assistant names, etc.
× The current grade history feature has a size limit on 2,000 items
Instructor B – 1 course with student numbers less than 20, face-to face course format
× Attendance must be manually entered, which would be time-consuming for larger classes
o Shows up as a grade in the gradebook, but doesn’t show dates and confuses students
× Every assignment has a gradebook column regardless of whether graded or not
 Liked the intuitive design, modules for organization, roll call attendance feature, peer review
grading
 SpeedGrader and rubrics have sped up grading significantly
 Visually, prefers Canvas; less of a learning curve than Blackboard
 Students liked the appearance, to-do list, what-if grade analysis, and notification options
o Lot of options, so little confusing finding things
Instructor C – 4 courses with student numbers ranging from about 15 to 70, all face-to-face
× No word wrap around images on pages
× Left-side navigation wording cannot be changed, so instructor needs to think about how to
make navigation easier for students
o It took an ed tech with HTML knowledge to help him make an effective page for
navigation
× Some items save automatically (grades), but other only saved when you push a button
(comments)
× Difficult to find assignments that need grading; there is no notification in the gradebook
× Students disliked the navigation and how the gradebook shows zeroes for items that aren’t
graded
× Had some issues with Canvas Support and received the impression that we don’t matter to them.
 Liked the to-do list feature for both faculty and students
 Easy to link to existing files and items in Canvas
 Kaltura embed is simple and easy to use
 “Fudge points” allows you to easily add or subtract points for late assignments or exceptional
answers
6/9/2015
2014-2015 Canvas Pilot Evaluation
Instructor D – 2 courses with student numbers less than 20, face-to-face
× Canvas does not have all of the powerful options of Blackboard
o Complex grade column calculations
× Sometimes the interface is cumbersome
o In order to add a new assignment to a page, you must stop editing the page and create
an assignment, publish the assignment, and then go back to editing the page
 Really likes the Canvas interface
o Simple, clean, modern
o Fewer clicks than Blackboard
 Assignments are more organized and able to structure into categories
 Syllabus, attendance, and files were all areas that she really liked
 Canvas taught her how to structure her courses better and she’ll apply to Blackboard in the
possible future
Instructor E - 2 courses with student numbers between 50 - 100, face-to-face
 Both students and the instructor really liked the To Do list, making it easy to see which
assignments were coming up or which assignments were available to grade
 Calendar in Canvas is much cleaner and very simple
 Discussion boards were easy to setup and could grade them directly in SpeedGrader
 Two students were invited to speak about their experiences
o Prefer Canvas mobile apps due to the simpler navigation, faster loading times
o Students found the navigation simpler in Canvas
Instructor F - 1 course with student numbers less than 20, face-to-face
× Gradebook doesn’t calculate ungraded assignments and could falsely show a student’s progress
 Really easy to drag and drop assignment due dates onto the Calendar to build a Canvas course
 Easy to organize assignments into groups
o Can weight each category to help with final grade calculations
o Easy to drag and drop assignments into different categories
 When setting up assignments, liked that you can restrict file types (i.e., only accept PowerPoint
files)
 After a paper is graded, students and instructors can exchange comments about the assignment
 Notifications are customizable and modern
 Setup checklist was useful and it only took an hour to set up course
 Files in Canvas are easy to organize into folders
o Easy to drag and drop files into different folders
o Easy to see which files are visible to the students and which are hidden
50
2014-2015 Canvas Pilot Evaluation
APPENDIX L: BLACKBOARD ROAD MAP
Blackboard Learn 2015 Ultra
Ultra, to be released this summer, is a completely new design. Existing courses can use current
structure (Classic Learn 9.1) or new “Ultra” interface. Instructor can choose which interface if we decide
to roll out in this way. Blackboard announced today, 5/26/15, that they will resume their biannual
release cadence next year with a Learn 9.1 April 2016 Release and a Learn 9.1 October 2016 Release.
The new interface will require a substantive training effort.
Our current Learn installation (called Learn 9.1 October 2014 Release) is no longer supported as of June
2017 and moves to limited operational support in January 2017.
Blackboard SaaS (Hosted)
Blackboard Learn is now available as a software-as-a-service (SaaS) offering. With the SaaS offering,
Learn is hosted, delivered, and managed using a modern cloud-computing infrastructure. For the initial
release of the SaaS deployment option, due this summer, Blackboard is using Amazon Web Services as
the deployment environment. As with most SaaS products, all updates are done for you, giving your
users the most recent version of Learn without significant downtime or interruptions in service for the
end users.
3 deployment options:
1. SaaS/public cloud a. SaaS Standard
b. SaaS Premium
2. Managed Hosting 3. Self Hosted
-
Available this summer
In development.
In development
There will be two options for moving from a self-hosted instance of Bb Learn to the new SaaS
deployment options:

Course-based migration.
Recommended if you want to migrate courses selectively.

Full data migration.
Recommended if you want to retain legacy data, settings and customizations. For full
data migration, Purdue will need to export the Learn data base and file system. The
exported database and file system will be provided to Blackboard who will conduct the
necessary conversions to the data and file system structure to restore the data and files
to the new SaaS deployment environment.
Any custom building blocks currently in use will need to be redeveloped. (Similar effort as required for
Canvas).
6/9/2015
2014-2015 Canvas Pilot Evaluation
APPENDIX M: CLOUD COMPUTING
This section is included to provide background information for the steering committee.
Many universities are moving to cloud-based services for a variety of reasons. Figure 2 provides a
comparison. For further information on cloud computing, please see the separate report Cloud Strategy
for Higher Education: Building a Common Solution.
Figure 2: From page 7 of Cloud Strategy for Higher Education: Building a Common Solution
52
2014-2015 Canvas Pilot Evaluation
APPENDIX N: BLACKBOARD LEARN 9.1 RELEASE UPDATE
6/9/2015
2014-2015 Canvas Pilot Evaluation
54
2014-2015 Canvas Pilot Evaluation
APPENDIX O: FACULTY AND STUDENT SURVEYS SORTED BY EFFECT SIZE
Advantage
Calendar
3.61 (23) 5.21 (24) Canvas
Large (d=0.90)
SpeedGrader
3.82 (22) 5.12 (33) Canvas
Moderate (d=0.59)
Taking Attendance
2.60 (17) 4.41 (17) Canvas
Moderate (d=0.56)
Uploading course files
4.08 (39) 4.95 (43) Canvas
Small (d=0.42)
Kaltura
4.19 (16) 3.83 (12) Blackboard Small (d=0.39)
Setting up assignments
4.15 (39) 4.90 (40) Canvas
Small(d=0.38)
Manipulating course files
3.83 (35) 4.55 (38) Canvas
Small (d=0.35)
Look and feel
3.92 (37) 4.66 (41) Canvas
Small (d=0.34)
Use of plagiarism detection tools 4.62 (13) 4.13 (8) Blackboard Small (d=0.34)
Groups
3.65 (17) 3.89 (18) Canvas
Small (d=0.33)
Setting a course up
4.29 (38) 4.75 (40) Canvas
Small (d=0.22)
Modules
4.13 (23) 4.63 (32) Canvas
None (d=0.19)
Organization of the system
4.08 (37) 4.48 (40) Canvas
None (d=0.18)
Gradebook
4.38 (39) 3.95 (40) Blackboard None (d=0.16)
Communicating with students/ 4.84 (37) 5.10 (49) Canvas
None (d=0.16)
Announcements
10 4.89
Tracking student performance
4.14 (28) 4.42 (31) Canvas
None (d=0.14)
13 4.76
Helping students keep up with 4.11 (27) 4.71 (28) Canvas
None (d=0.14)
course material
12 4.85
The student view
4.27 (37) 4.33 (39) Canvas
None (d=0.13)
16 4.69
Customizing the system
3.93 (28) 3.77 (26) Blackboard None (d=0.08)
24 3.86
Scheduler
3.63 (8) 3.85 (13) Canvas
None (d=0.08)
17 4.60
Rubrics
4.12 (25) 4.00 (19) Blackboard None (d=0.04)
11 4.89
Giving quizzes
4.31 (32) 4.57 (28) Canvas
None (d=0.03)
21 4.39
Discussion boards
4.36 (25) 4.33 (21) Blackboard None (d=0.01)
22 4.00
Incorporating tools from
4.14 (7) 4.13 (8) Blackboard None (d=0.00)
publishers
25 3.14
Learning mastery tool
3.80 (5) 3.80 (5) Neither
None
Note: Meaningfulness of difference (effect size) range: 0.2 slightly meaningful, 0.5 moderately
meaningful, 0.8 large meaningful (Cohen, 1969)
6/9/2015
Effect size
Feature
Meaningfulness of
difference
4.69
5.16
3.96
5.26
4.57
5.26
4.98
4.93
4.52
4.76
5.34
4.54
5.34
5.67
5.19
Canvas
(No. of
responses)
Mean
import.
rank
15
7
23
4
18
4
8
9
20
13
2
19
3
1
6
Blackboar
d (No. of
responses)
Import.
Faculty Survey – LMS Features by Effect Size
0.9
0.59
0.56
0.42
0.39
0.38
0.35
0.35
0.34
0.33
0.22
0.19
0.18
0.16
0.16
0.15
0.14
0.13
0.08
0.08
0.04
0.03
0.01
0
0
2014-2015 Canvas Pilot Evaluation
Feature
Advantage
3 5.17 Accessing files my instructor has
4.82 (492) 4.53 (505) Blackboard None (d=0.18)
put up
12 4.30 Interacting with other students
4.01 (380) 4.36 (382) Canvas
None (d=0.17)
11 4.38 Participating in group work
4.08 (375) 4.41 (379) Canvas
None (d=0.16)
1 5.26 Knowing what my current grade is 4.46 (490) 4.72 (503) Canvas
None (d=0.14)
in the course
7 4.81 Layout
4.69 (494) 4.42 (507) Blackboard None (d=0.13)
4 5.14 Tracking assignments
4.41 (491) 4.61 (505) Canvas
None (d=0.12)
9 4.61 Interacting with course instructor 4.18 (388) 4.43 (396) Canvas
None (d=0.12)
8 4.73 Getting notifications
4.29 (480) 4.53 (500) Canvas
None (d=0.11)
2 5.24 Tracking grades on assignments
4.62 (488) 4.79 (504) Canvas
None (d=0.09)
10 4.49 Participating in discussions
4.38 (394) 4.49 (379) Canvas
None (d=0.05)
6 4.86 Managing and manipulating my
4.47 (458) 4.52 (466) Canvas
None (d=0.03)
files
5 5.10 Taking quizzes / tests
4.71 (454) 4.69 (426) Blackboard None (d=0.01)
Note: Meaningfulness of difference (effect size) range: 0.2 slightly meaningful, 0.5 moderately
meaningful, 0.8 large meaningful (Cohen, 1969)
Effect Size
Meaningfulness
of difference
Canvas (No. of
responses)
Blackboard (No.
of responses)
Mean
Importance
Student Survey – LMS Features by Effect Size
0.18
0.17
0.16
0.14
0.13
0.12
0.12
0.11
0.09
0.05
0.03
0.01
3 5.06
Cost
3.98 (126) 5.02 (147) Canvas Moderate (d=0.67)
1 5.19
Tracking course grade
4.23 (134) 4.68 (165) Canvas Small (d=0.32)
2 5.13
Tracking assignment grades 4.31 (134) 4.63 (169) Canvas Small (d=0.23)
7 4.83
Taking tests / quizzes
4.11 (109) 4.43 (129) Canvas Small (d=0.22)
5 4.99
Layout
4.24 (136) 4.48 (172) Canvas None (d=0.18)
4 5.01
Organization
4.26 (136) 4.44 (172) Canvas None (d=0.13)
6 4.86
Submitting assignments
4.21 (109) 4.30 (131) Canvas None (d=0.06)
Note: Meaningfulness of difference (effect size) range: 0.2 slightly meaningful, 0.5 moderately
meaningful, 0.8 large meaningful (Cohen, 1969)
56
Effect size
Meaningful
ness of
difference
Advantage
Canvas
(Number of
responses)
Blackboard
(Number of
responses)
Feature
Mean
importance
rank
Importance
Student Survey – Mobile App Comparison by Effect Size
0.67
0.32
0.23
0.22
0.18
0.13
0.06