Using Google Analytics and Event
Logs to Improve Help Documentation
Steve Lonn & Margaret Wagner
University of Michigan
slonn@umich.edu
8th Sakai Conference
mwagner@umich.edu
Newport Beach
4-7 December 2007
Sakai’s Help Documentation
• Out of the box, pretty generic
‣ Covers the tools in the release (not contrib)
‣ Requires customization for institutional branding and
specific help documents that might need to be added
for specific implementations
2
About Michigan’s
Implementation of Sakai: CTools
Fall 2007 Stats:
• Average 18,000 users per day
‣ 7,600 peak concurrent use
• 3,700 course sites
• 5,300 project sites (so far)
2007 Instructor / Student Survey:
• 99% of students have used CTools
• 85% of instructors have used CTools
3
Our Help Docs
• Combination of original documents and modified Sakai
documentation (added as new tools go live)
‣ LOTS of custom pages for our implementation:
• getting help, glossary, known issues, etc.
• Quick Demo
4
Why Collect Data on Help?
• Helps identify tools / functions that users
have difficulty with
• Helps improve training & support
• Helps direct development / improvement efforts
• Helps improve Help docs!
5
Getting Data on Our Help Docs
• Started with a simple hit counter
‣ Wasn’t always accurate
‣ Didn’t have a lot of information (who, where from, etc)
• Also had new web-based help form we wanted to track
• Sakai just started tracking help events in 2.3
• Other sources: User surveys, training sessions, help
requests, etc.
6
Enter Google Analytics
• Google Analytics is a FREE service up to 5,000,000
page views (hits) per month
• TONS of data about users, content, browsers, etc.
‣ Exportable, email, easy web access
• Coupled with Sakai events, makes it easy to track and
pull data about users quickly
7
Setting Up Google Analytics
• Google Analytics uses a short javascript to track
‣
<script src="https://ssl.google-analytics.com/urchin.js" type="text/javascript">
</script>
<script type="text/javascript">
_uacct = "UA-1354622-2";
urchinTracker();
</script>
‣ Help docs owner puts script at bottom of every page
• Data starts showing up after about 24 hours
8
Main Screen
9
Top Content
10
Page Detail - Segmenting
11
Keywords
12
Our Help Form
• Web-based help form -- available within Help Docs and
by clicking “Contact Us” on bottom of CTools pages
13
Tracking “Goals”
• Google Analytics is designed for marketing, but you can
manipulate the system to track other things...
14
Visualizing “Goals”
15
About Your Visitors
16
State
17
City
18
Ann Arbor Network Locations
19
New vs. Returning Users
20
Selected Language
United States English
English (not specified)
Korean
Chinese - Cantonese
French
Not Set
Chinese - Taiwan
English - Great Britain
Japanese
German
21
Average Page Views Per Visit
22
Return Visits / Depth of Visit
23
Browsers
24
Internet Explorer Versions
25
Operating Systems
26
Which OS With Which Browser?
27
Screen Colors
28
Screen Resolution
29
Connection Speed
30
Flash Versions / Java
31
Compare Any Time Range
32
Content Changes
33
Other Comparisons
34
Google Summary
• LOTS of data to sift through
• Can give quick glances at trends, and allow deep drilldowns
• FREE & relatively easy to set up
35
Sakai Event Logs
• Two events:
‣ help.access - Tracks all user clicks to help guide
• REF is either null for menubar link or has tool ref for ? link
‣ help.search - Text of keyword searches from within
help tool
36
One Month for help.access
October
2007
Views
Page
9464
Help Link on Menu Bar
415
Discussion
389
Announcements & MOTD
385
Chat
252
Web Content & Gateway pages
138
Worksite Information on Home Page
132
Resources
117
Assignments
35
Gradebook
17
Syllabus
16
Site Info
14
Schedule
12
Wiki
11
Synoptic Message Center
11
Forums
49
Other
37
Tracking help.access
• 2007: Help link on menu bar
25,000
22,413
20,000
15,167
15,000
9,464
10,000
6,317
5,543
5,262
4,336
5,000
4,140
2,486
2,352
1,813
em
be
r
er
ov
N
O
ct
ob
r
em
be
pt
Se
st
Au
gu
ly
Ju
ne
Ju
ay
M
ril
Ap
M
ar
ry
Fe
br
ua
y
ar
nu
Ja
ch
0
38
One month for help.search
October
2007
Views
30
30
27
24
23
20
17
15
15
13
13
11
10
9
9
9
8
8
8
Search Term
discussion
grades
wiki
upload
drop box
assignments
email
gradebook
chat
groups
webdav
enter search term
forum
copy
delete
grade
assignment
download
schedule
39
Tracking help.search
• 2007: “upload” and “email”
68
70
56
46
40
42
28
24
22
20
17
14
13
11
8
13
10
15
12
12
11
6
4
2
7
2
3
em
be
r
er
ov
N
O
ct
ob
r
em
be
pt
st
Au
upload
Se
email
gu
ly
Ju
Ju
ne
ay
M
ril
Ap
M
ar
ry
Fe
br
ua
y
ar
nu
Ja
ch
0
40
Other Data for Help Docs
• Annual instructor / student survey
‣ Ask how many users get help from docs (“most effective”)
•
12% instructors, 7% students
‣ Comments about help guide
• Feedback from users in Support contact (email,
phone), at training sessions, anecdotally
41
What Do We DO
With All This Data?
• Margaret Wagner
‣ Senior Technical Writer
‣ Writes Sakai & CTools documentation
‣ Writes Sakai Newsletter
42
Docs with the highest usage
numbers 3/07 to 11/07
Most common search terms
8/07
1. Common Questions (viewed 4476 times)
1. books
2. Feedback/Contact Consultants Form
2. delete site
3. Discussions
3. delete
4. Setting Up and Publishing Sites (ranged
from 27 to 276 per week)
4. syllabus
5. tabs
5. Chat Room
6. wiki
6. Resources
7. move
7. Accessibility
8. publish
8. Announcements
9. permissions
9. Gradebook
10. publish site
10. Performing Common Tasks
11. email
11. Known Issues
12. assignments
12. Wiki
13. discussion
13. Setting Up a Site -- Simple Outline
14. textbooks
14. Customizing Home
15. contact
15. Web Content (viewed 982 times)
43
Online Help Section:
“Performing Common Tasks with CTools”
DELETE
• delete a course site or project site
...... delete a tab for an official course site
...... delete a tab for a project or other site
...... delete an entire site so it no longer exists
• delete a discussion entry
• delete a chat entry
UPLOAD
• upload a file
COPY
• copy a file in into another resource folder
• copy a file from CTools to your computer
• duplicate a file in CTools
• copy materials from a site to a new site
• duplicate a site
PASSWORD
• change a password
• create a password
EMAIL
• send email using CTools
• receive email using CTools
• set up an email archive
DOWNLOAD
• download a file
FOLDER (CREATING)
• create a folder in CTools
PRINT
• print a CTools page
• print a resource (file)
REMOVE
- remove me from a site
- remove a course or project tab
- remove a course or project site
44
Getting “Message of the Day” topics from
common search terms and doc usage numbers
Some examples:
Control your tabs
Users can control the tabs used to access course and project worksites. You can
choose to hide particular tabs, and reorder the tabs using the Preferences tool in
your My Workspace...
Can't find your course in CTools?
If you have recently added the course, it takes up to a day for your course
elections to be reflected in CTools...
Minimum requirements
CTools is designed to work fully with the following browsers...
45
Recap: Applications for Data
on Doc Usage
• Prioritize doc efforts and make certain docs are clear
and thorough
• Make sure common search terms are in the sections
• Create new docs that are based on search terms
• Inform support
• Head off confusion with tips in the Message of the Day
46
Questions / Next Steps
• Contact Info:
‣ Steve Lonn - slonn@umich.edu
‣ Margaret Wagner - mwagner@umich.edu
• Next Steps:
‣ Volunteer to participate in sharing your custom help
docs and making Sakai Help better!
• Signup by emailing Jon Hays at Berkeley:
jonhays@media.berkeley.edu
47