Comparing Sakai Users
at a Residential and
Commuter Campus
Steven Lonn
Stephanie D. Teasley
University of Michigan
Email: slonn@umich.edu
1
Session Overview
• Why conduct user surveys?
• Campus descriptions
• Methodology, samples, response rates
• General computer use & attitudes
• Sakai use for courses
• Comparing survey data & log data
• So what?
• Next steps...
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
2
Why Conduct Surveys?
• Level of system usage
• User perceptions of the system
• ease of use, satisfaction, etc.
• Quality of user support / user preferences
• Users' desired improvements
• Pain points / barriers to migration
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
3
Campus Descriptions
•
•
ctools.umich.edu
Residential Campus (Ann Arbor)
•
•
•
•
26,000 undergraduates, 15,000 graduates
5,700 faculty, 1,600 graduate student instructors
Roughly 3,000 courses per term
One LMS: Sakai
Commuter Campus (Dearborn)
•
•
•
•
6,500 undergraduates, 1,900 graduates
500 faculty, 4 graduate student instructors
Roughly 1,800 courses per term
Multiple LMS: Sakai, Blackboard (SOM), VLT (Engin)
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
4
Commuter Campus Course Mode
Q: The majority of courses I teach / take at UM Dearborn are conducted:
1%
2%
4% 1%
18%
13%
79%
82%
Instructors
Students
Face-to-Face
Using a blend of face-to-face and distance education
At a Distance with no face-to-face interaction, other than proctored exams
Other
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
5
Methodology
•
•
Annual user survey - administered online in April
All instructional faculty, all commuter students, and a
random stratified sample of 25% of residential students
Campus
Instructors Students
Residential
n = 1,504
n = 2,469
Commuter
n = 135
n = 1,355
23% average response rate
•
ctools.umich.edu
User event logs from the semester prior to the survey
(January-April, 2008)
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
6
Our Local Instance
•
Survey based on
Sakai 2.4.x
•
For Winter 2008:
•
•
ctools.umich.edu
3,482 course sites
3,028 project sites
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
7
Computer Use
and Attitudes
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
8
Computer Expertise
Q: Rate your expertise with computers
80%
63.8%
66.9%
62.9%
61.9%
60%
40%
28.9%
32.1%
32.1%
24.5%
20%
7.3%
6.0%
8.6%
5.0%
0%
Novice
Intermediate
Res. Instructors
Res. Students
ctools.umich.edu
Advanced
Comm. Instructors
Comm. Students
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
9
Use of IT in Courses
Q: Which of the following best describes your use (preference) of
information technology in your courses?
50%
45.4%
43.4%
38.9%38.5%
37.8%
40%
35.6%
30%
27.2%
25.4%
20%
37.0%
17.0%
16.0%
14.8%
10%
1.7%
3.0%
1.2% 1.6%
0.4%
0%
None
Limited
Moderate
Res. Instructors
Res. Students
ctools.umich.edu
6.9%
6.7%
Extensive
1.6%
Online Only
Comm. Instructors
Comm. Students
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
10
Most Valuable Benefit of Using IT
Q: Which of the following benefits from using information technology in
your courses was the most valuable to you?
Improve
Instructors'
Teaching
Improve
Students'
Learning
Save Time
(Efficiency)
Access
Materials
(Convenience)
Help
Manage
Course
Activities
Improve
Student to
Instructor
Communication
Improve
Instructor
to Student
Communication
Improve
Student to
Student No Benefits
Communication
Residential
Instructors
11% 24% 14% 28% 3% 4% 13% 1% 1%
Commuter
Instructors
11% 35% 4% 26% 4% 2% 12% 2% 1%
Residential
Students
3% 13% 18% 54% 3% 4% 3% 1% 0%
Commuter
Students
3% 14% 20% 49% 4% 4% 3% 1% 1%
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
11
Most Valuable Benefit of Using IT
Q: Which of the following benefits from using information technology in
your courses was the most valuable to you?
Improve
Instructors'
Teaching
Improve
Students'
Learning
Save Time
(Efficiency)
Access
Materials
(Convenience)
Help
Manage
Course
Activities
Improve
Student to
Instructor
Communication
Improve
Instructor
to Student
Communication
Improve
Student to
Student No Benefits
Communication
Residential
Instructors
11% 24% 14% 28% 3% 4% 13% 1% 1%
Commuter
Instructors
11% 35% 4% 26% 4% 2% 12% 2% 1%
Residential
Students
3% 13% 18% 54% 3% 4% 3% 1% 0%
Commuter
Students
3% 14% 20% 49% 4% 4% 3% 1% 1%
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
12
Most Valuable Benefit of Using IT
Q: Which of the following benefits from using information technology in
your courses was the most valuable to you?
Improve
Instructors'
Teaching
Improve
Students'
Learning
Save Time
(Efficiency)
Access
Materials
(Convenience)
Help
Manage
Course
Activities
Improve
Student to
Instructor
Communication
Improve
Instructor
to Student
Communication
Improve
Student to
Student No Benefits
Communication
Residential
Instructors
11% 24% 14% 28% 3% 4% 13% 1% 1%
Commuter
Instructors
11% 35% 4% 26% 4% 2% 12% 2% 1%
Residential
Students
3% 13% 18% 54% 3% 4% 3% 1% 0%
Commuter
Students
3% 14% 20% 49% 4% 4% 3% 1% 1%
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
13
Most Valuable Benefit of Using IT
Q: Which of the following benefits from using information technology in
your courses was the most valuable to you?
Improve
Instructors'
Teaching
Improve
Students'
Learning
Save Time
(Efficiency)
Access
Materials
(Convenience)
Help
Manage
Course
Activities
Improve
Student to
Instructor
Communication
Improve
Instructor
to Student
Communication
Improve
Student to
Student No Benefits
Communication
Residential
Instructors
11% 24% 14% 28% 3% 4% 13% 1% 1%
Commuter
Instructors
11% 35% 4% 26% 4% 2% 12% 2% 1%
Residential
Students
3% 13% 18% 54% 3% 4% 3% 1% 0%
Commuter
Students
3% 14% 20% 49% 4% 4% 3% 1% 1%
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
14
Most Valuable Benefit of Using IT
Q: Which of the following benefits from using information technology in
your courses was the most valuable to you?
Improve
Instructors'
Teaching
Improve
Students'
Learning
Save Time
(Efficiency)
Access
Materials
(Convenience)
Help
Manage
Course
Activities
Improve
Student to
Instructor
Communication
Improve
Instructor
to Student
Communication
Improve
Student to
Student No Benefits
Communication
Residential
Instructors
11% 24% 14% 28% 3% 4% 13% 1% 1%
Commuter
Instructors
11% 35% 4% 26% 4% 2% 12% 2% 1%
Residential
Students
3% 13% 18% 54% 3% 4% 3% 1% 0%
Commuter
Students
3% 14% 20% 49% 4% 4% 3% 1% 1%
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
15
Sakai Use
and Attitudes
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
16
Experience Using Sakai
Q: For how many different courses have you used CTools?
60%
53.1%
47.9%
45%
36.3%
33.3%
32.3%
30%
28.6%
22.3%
15%
0%
14.8%
12.1%
9.6%
11.4%
11.3%
0.3%
None
14.5%
11.0%
3.9%
1.9%
1-2
3-6
Res. Instructors
Res. Students
ctools.umich.edu
20.0%
19.3%
16.1%
7-10
More than 10
Comm. Instructors
Comm. Students
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
17
How Often Users Visit
Q: How often do you visit your CTools sites?
60%
54.8%
48.1%
44.3%
45%
41.0%
36.1%
29.9%
30%
23.5%
17.6%
15%
7.7%
0%
10.6% 11.2%11.1%
9.9%
7.0% 7.4%
1.2%
5.2%
2.7%
Few Times a Semester Few Times a Month
Res. Instructors
Res. Students
ctools.umich.edu
16.5%
14.4%
Once a Week
Few Times a Week
Daily (or More)
Comm. Instructors
Comm. Students
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
18
Materials Management vs. Interactive
Teaching & Learning Activities
Materials Management
Survey Items
Interactive Teaching &
Learning Survey Items
•Post / Access a syllabus online
• Students turn in assignments online
• Send / Receive messages or
• Instructors return assignments to
• Post / Access online readings &
• Students post questions before lecture
notifications from the instructor
students online
supplementary course materials
• Post / Access lecture outline or notes
• Students post questions after lecture
after the lecture
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
19
Materials Management Items
Q: Using CTools is valuable for...
Strongly
Agree
5
4.69 4.68
4.55
4.33
4.57 4.58 4.52
4.44
4.52
4.43 4.35
4.54
4.24
4.34
4.33
3.94
4
3
2
Strongly
Disagree
1
Post / Access online
readings
&
Online
Readings
supplementary
course materials
Post
/ Access a
Syllabus
syllabus online
Res. Instructors
Res. Students
ctools.umich.edu
Send / Receive
messages or
Notifications
notifications from the
instructor
Post / Access lecture
Lecture
Outline
outline or notes
after
the lecture
Comm. Instructors
Comm. Students
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
20
Interactive Teaching & Learning Items
Q: Using CTools is valuable for...
Strongly
Agree
5
4.10 4.13
4.22 4.17
4
4.03
3.76 3.70 3.83
3.67
3.88
3.84
3.50
3.62
3.64
3.54 3.60
3
2
Strongly
Disagree
1
Students turn
in in
Students
turn
assignments online
Students post
questions
after
questions after
lecture
Res. Instructors
Res. Students
ctools.umich.edu
Students post
Instructors return
questions
before Instructors
return
questions before
lecture
assignments online
Comm. Instructors
Comm. Students
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
21
Materials Management vs. Interactive
Teaching & Learning Tools
Materials Management
Tools
Interactive Teaching &
Learning Tools
• Announcements
• Assignments
• Drop Box
• Chat Room
• Gradebook
• Discussion
• Resources
• Email Archive
• Syllabus
• Forums
• Messages
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
22
Materials Management Tools
Q: Within CTools, the following tools are valuable for my course-related activities...
Strongly
Agree
5
4.71 4.67
4.48
4.39
4.62 4.56
4.51 4.50 4.45 4.44
4.48
4.25
4
3.98
3.81
4.15
3.84 3.89
3.78
3.57
3.40
3
2
Strongly
Disagree
1
Announcements Resources
Res. Instructors
Res. Students
ctools.umich.edu
Syllabus
Gradebook
Drop Box
Comm. Instructors
Comm. Students
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
23
Interactive Teaching & Learning Tools
Q: Within CTools, the following tools are valuable for my course-related activities...
Strongly
Agree
5
4.424.494.434.37
4.00
3.97
3.863.80
3.873.873.913.91 3.94
4
3.713.67
3.58
3.64
3.48
3.33
3.21
3.58
3.43
3.20
3.18
3
2
Strongly
Disagree
1
Assignments Chat Room Discussion Email Archive
Res. Instructors
Res. Students
ctools.umich.edu
Forums
Messages
Comm. Instructors
Comm. Students
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
24
Interactive Teaching & Learning Tools
Q: Within CTools, the following tools are valuable for my course-related activities...
Strongly
Agree
5
4.424.494.434.37
4.00
3.97
3.863.80
3.873.873.913.91 3.94
4
3.713.67
3.58
3.64
3.48
3.33
3.21
3.58
3.43
3.20
3.18
3
2
Strongly
Disagree
1
Assignments Chat Room Discussion Email Archive
Res. Instructors
Res. Students
ctools.umich.edu
Forums
Messages
Comm. Instructors
Comm. Students
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
25
Interactive Teaching & Learning Tools
UNDERGRADUATE STUDENTS
Strongly
Agree
5
4.44
4
4.39
4.28
4.23
4.25
4.42
4.30
4.45
4.32
4.40
3.83
3.81
3
2
Strongly
Disagree
1
Assignments Chat Room Discussion Email Archive
Res. Students
ctools.umich.edu
Forums
Messages
Comm. Students
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
26
More Differences of Tool Use?
• Activity & tool analysis from survey indicated
some differences between campus
• particularly between undergraduate students
• How to further analyze differences?
• Log Data
• All W08 activity for course sites of survey
respondents (undergraduate students)
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
27
Log Data: Materials
Management Tools
Tool
Residential
Campus
Commuter
Campus
Mean
Difference
Announcements
2.49%
2.74%
0.25
Gradebook
1.25%
0.9%
0.35
Resources /
Drop Box
83.43%
75.64%
7.79 *
Syllabus
0.42%
0.67%
0.25
* - statistically significant difference
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
28
Log Data: Interactive
Teaching & Learning Tools
Tool
Residential
Campus
Commuter
Campus
Mean
Difference
Assignments
6.00%
9.16%
3.16 *
Chat Room
1.12%
1.36%
0.24
Discussion
1.43%
4.56%
3.13 *
Email Archive
0.75%
0.06%
.69 *
Forums /
Messages
2.16%
4.05%
1.89 *
* - statistically significant difference
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
29
What Do Users Say?
• Asked survey participants to answer the
prompt:
"Do you think using CTools has affected
instructors' use of in class time? If so, how"
• Variety of answers, mostly positive....
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
30
Instructors' Comments
•
•
ctools.umich.edu
More efficient use of class time (20% res., 24% comm.)
•
"I spend less time on reminding the students about assignments and
updating them about their grade."
•
"It assists with the "housekeeping" aspects of teaching. I can direct
the students to CTools for FAQs, etc."
More focussed instruction / discussion (6% res., 9% comm.)
•
"Yes -- it makes provides me a little more freedom to develop a
discussion as the points in the lecture can be revisited online."
•
"I can hit main points harder and refer students to elaborations
available on notes in CTools."
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
31
Students' Comments
•
•
ctools.umich.edu
More efficient use of class time (32% res., 29% comm.)
•
"I think one of my professors can move faster because she knows
that she has the notes for the lecture posted so we can have them in
front of us for the lecture or can review them afterwards if we missed
anything."
•
"Typically it reduces time needed to pass out notes or other
documents, etc., so more time can be used to teach."
More focussed instruction / discussion (6% res., 5% comm.)
•
"We can come in focused and continue a discussion that was already
started online, using CTools."
•
"I think overall it lets my instructors get to what's important and more
difficult in class while letting me learn the easier stuff on my own
time."
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
32
So What?
•
•
•
ctools.umich.edu
There's a noticeable difference between residential and
commuter campus users
•
•
Residential - more materials management activities
Commuter - more teaching & learning activities
What's driving this difference?
•
More reflective of culture?
Needs at different campuses?
Something else?
How does this all impact instructors' teaching and
students' learning?
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
33
Further Research
• How can we best facilitate teaching & learning
activities within LMS technology?
• Examine instructors who use the interactive
tools (chat, discussion, wiki) to better
understand how to design & scaffold innovative
teaching & learning activities within the LMS
• Ways to improve not just whether instructors
use LMS, but how they use the technology
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
34
Thank You
All Survey Results:
ctools.umich.edu
Steven Lonn
Stephanie D. Teasley
University of Michigan
Email: slonn@umich.edu
ctools.umich.edu
Fall 2008 Sakai Regional Conference
Virginia Tech, Blacksburg, VA
Usability, Support, and Evaluation Lab
Digital Media Commons, University of Michigan
35