USABILITY STUDY OF MY SAC STATE AND SACCT
Corinne Michelle Fielder
B.A., Pitzer College, Claremont, 2002
PROJECT
Submitted in partial satisfaction of
the requirements for the degree of
MASTER OF SCIENCE
in
COUNSELING
(Vocational Rehabilitation Counseling)
at
CALIFORNIA STATE UNIVERSITY, SACRAMENTO
SPRING
2011
©
Corinne Michelle Fielder
ALL RIGHTS RESERVED
ii
USABILITY STUDY OF MY SAC STATE AND SACCT
A Project
by
Corinne Michelle Fielder
Approved by:
_________________________________________, Committee Chair
Guy Deaner, Ph.D.
_________________________________________
Date
iii
Student:
Corinne Michelle Fielder
I certify that this student has met the requirements for format contained in the University
format manual, and that this Project is suitable for shelving in the
Library and credit is to be awarded for the Project.
, Graduate Coordinator
Bruce Ostertag, Ph.D.
Date
Department of Special Education, Rehabilitation,
School Psychology, and Deaf Studies
iv
Abstract
of
USABILITY STUDY OF MY SAC STATE AND SACCT
by
Corinne Michelle Fielder
Oftentimes, web page designers overlook the end-user, such as people with
disabilities, in the development and evaluation of web pages. Sacramento State has not
involved students with disabilities in evaluating the accessibility and usability of
features on the Sacramento State website. In order to ensure that web pages are
accessible and usable by people with disabilities, this population needs to be included
in the evaluation process. The author conducted a usability study that involved students
with disabilities in order to determine accessibility and usability issues on Sacramento
State’s “My Sac State” and “SacCT.” The author provided recommendations for
Sacramento State website designers in an effort to resolve these issues.
_________________________________________, Committee Chair
Guy Deaner, Ph.D.
Date
v
TABLE OF CONTENTS
Page
List of Tables......................................................................................................... viii
Chapter
1. INTRODUCTION...........................................................................................
1
Background of Problem.............................................................................
1
Statement of Problem.................................................................................
4
Definition of Terms...................................................................................
5
Limitations of the Problem........................................................................
8
LITERATURE REVIEW...............................................................................
9
Introduction...............................................................................................
9
Legislation/Regulation/Policy...................................................................
9
California State University System Web Accessibility............................
11
Evaluation of Sacramento State Web Pages.............................................
14
Accessibility of Websites..........................................................................
16
Impact on People with Disabilities...........................................................
21
Usability Testing.......................................................................................
23
METHODOLOGY........................................................................................
26
Review of Sources....................................................................................
26
Method......................................................................................................
26
2.
3.
vi
Participants.................................................................................................
29
Results........................................................................................................
31
SUMMARY AND RECOMMENDATIONS................................................
37
Appendix A. Materials Related to Participation in Research..............................
40
Appendix B. Usability Tasks, Script, and Questions...........................................
47
Appendix C. Participant Responses.....................................................................
53
Appendix D. Demographics and Results.............................................................
74
References.............................................................................................................
77
4.
vii
LIST OF TABLES
Page
1. Sample of Usability Tasks................................................................................
73
2. Participant Demographics................................................................................
75
3. Results of Usability Tasks................................................................................
76
viii
1
Chapter 1
INTRODUCTION
Background of Problem
Accessibility of information becomes a vital component in the advancement of
people with disabilities with the increasing use of the Internet for online courses and
distance education classes. With the increasing complexity of the Internet, including
graphics and auditory features, people with disabilities (e.g., visual, hearing, mobility,
cognitive, and learning disabilities) find it increasingly difficult to access information,
even with the use of assistive technology (Harper & DeWaters, 2008). According to
Kaye, ". . . the lack of access to computers and the Internet limits people with disabilities
from learning essential skills needed to complete college, vocational, and job-related
programs that could lead to more independent lifestyles" (as cited in Wattenberg, 2004,
p. 127).
Most commonly, website accessibility is measured based on the criteria
established by the World Wide Web Consortium's [W3C] Web Accessibility Initiative
[WAI]. WAI developed the Web Content Accessibility Guidelines [WCAG] to ensure
that HyperText Markup Language [HTML] code could be readable by screen readers.
Online web accessibility evaluation tools, such as Bobby, can be used to generate an
accessibility report that informs developers about potential barriers in a web page design
based on WCAG and Section 508 guidelines (Seale, 2006).
However, there are limitations to the accessibility report. According to
Rowland, ". . . pages that pass Bobby's accessibility evaluation may still be inaccessible
2
because the program evaluates only objective content. For example, a page may pass
the Bobby inspection because it contains ALT tags for the graphics” (as cited in Klein et
al., 2003, p. 35). “However, the ALT tag may provide an inappropriate description of
the graphic (e.g., central2.jpg or ‘click here’), which yields the feature inaccessible"
(Klein et al., 2003, p. 39).
According to Kelly, Phipps, and Howell, "A checklist approach can, in fact, be
counterproductive as it encourages developers to prioritise the objective areas which
testing tools can easily report on" (as cited in Brophy & Craven, 2007, p. 960).
Oftentimes, designers become so engrossed in the features of a product that they fail to
include the end-user in their analysis. “The best way to ensure that a website is
accessible and usable is to include the actual users or a representative sample in the
study” (Barophy & Craven, 2007; Federici et al., 2005; Sloan et al., as cited in
Wattenberg, 2004, p. 132).
This author is a graduate student at California State University, Sacramento
[CSUS] (commonly referred to as Sacramento State) where she met with Sacramento
State's Co-Director of the High Tech Center and TRIO SSS Project, and Accessible
Technology Initiative Project Coordinator, Melissa Repa, to discuss accessibility issues
for students with disabilities and viable projects that would serve this segment of the
college population. Repa explained that the California State University system was
required to achieve the goals established by the Accessible Technology Initiative
[ATI] to ensure that their website is in compliance with Section 508 of the
Rehabilitation Act. Sacramento State had recently purchased the software, Morae
3
Academic Version 3.0, for usability testing and user experience research. Usability
testing with Morae was to be used as part of Repa's departmental effort to meet the
Section 508 compliance requirements. However, designing and implementing the
usability test was an endeavor that had not previously been undertaken. Repa agreed to
allow the author to oversee usability testing of students with disabilities in order to meet
the ATI requirements and the author's graduation requirements (personal
communication, May 4, 2009).
The California State University, Office of the Chancellor established the May
15, 2009 deadline, which required ". . . all administrative sites that are critical to
institutional access (as established in the Web Accessibility Implementation Plan)
should, at a minimum, conform to baseline accessibility standards as defined in Section
508" (Reichard & West, 2007, p. 3). A sample of campus websites was evaluated using
automated and manual accessibility evaluation tools in order to provide a guide for
self-evaluation that can be used at all CSU campuses. Sacramento State websites failed
to pass the evaluation (CSUS, 2010a).
The Office of the Chancellor established the May 15, 2012 deadline that requires
that ". . . all websites at the CSU should fully conform to Section 508" (Reichard &
West, 2007, p. 3). To ensure that websites are accessible and usable for people with
disabilities, it is valuable for web page designers to utilize accessibility evaluation tools
in addition to usability tests with actual users. Web page designers are in a better
position to address accessibility and usability of web pages after they gather feedback
from accessibility evaluation tools and actual users, including people with disabilities.
4
Sacramento State has not conducted a usability test that involves students with
disabilities to ensure that their website is accessible and usable by this population.
Statement of Problem
The author volunteered in the High Tech Center at Sacramento State, which
provides assistive technology resources and academic assistance to students with
disabilities. The author became aware of the struggles that students with disabilities
encounter when navigating websites. On numerous occasions, the author observed and
spoke with students with visual impairments who struggled to make sense of web pages
that were cumbersome and cluttered when accessed using a screen reader. A task, such
as accessing an e-mail account, became a problem-solving endeavor that consumed an
hour of staff and student time. The author became aware of the heavy reliance on visual
cues when navigating a website, and how barriers to accessibility and usability can
create significant hurdles for people with disabilities in an academic setting that heavily
relies on the Internet for communicative, informational, and research purposes.
Oftentimes, web page designers create websites with a typical end-user in mind,
which does not factor in the needs of people with disabilities who access the Internet.
They evaluate web pages as a web page designer who is familiar with the features and
navigational aspects of the website, erroneously believing that their perspective
represents all users. Thus, web page designers overlook accessibility and usability
issues encountered by people with disabilities. In order to ensure that these issues are
addressed, people with disabilities need to be included in the evaluation of websites.
Sacramento State has not involved students with disabilities in the evaluation of the
5
accessibility and usability of features on the Sacramento State website. The author is
attempting to help solve this problem by conducting a usability test that involves
students with disabilities.
Definition of Terms
Accessibility
The ability for any person, regardless of disability or assistive technology, to
access information with a similar outcome (Hackett, Parmanto, & Zeng, 2005).
According to Nadler and Furman, an accessible item ". . . can be used in a variety of
ways that do not depend on a single sense or ability" (as cited in Bowman & Jaeger,
2005, p. 70).
Accessible Technology Initiative [ATI]
Developed after Executive Order No. 926 was issued which stated that ". . . it is
the policy of the CSU to make information technology resources and services accessible
to all CSU students, faculty, staff and the general public regardless of disability" (Reed,
2004, n.p.). The goal of the Accessible Technology Initiative is to ensure that federal
and state requirements are met regarding "equal and equitable access" for people with
disabilities, which includes websites and web applications.
ALT tags
Text used to describe an image on a web page. The text description of an image
can be read aloud by screen reader software, which makes this web page element
accessible to people with visual impairments (Evans, 2003).
6
Bobby
A web accessibility evaluation tool that was developed by Watchfire to
determine whether a website meets the accessibility requirements from Section 508 of
the Rehabilitation Act and the Web Content Accessibility Guidelines 1.0. According to
the International Center for Disability Resources on the Internet (2008), “The Bobby
web accessibility checker and online products will no longer be publicly accessible as of
February 1, 2008."
HyperText Markup Language [HTML] code
A text-formatting language that is used to create web pages. HTML code uses
tags that indicate how content on a web page should appear in a web browser (Evans,
2003).
My Sac State
A web portal, which requires a user to login only once, in order to access
numerous applications, such as e-mail, campus news, and SacCT (Biondi, 2009).
SacCT
The name of Sacramento State's WebCT [Course Tools] or Blackboard Learning
System. SacCT is an online virtual learning environment where instructors can manage
course tools, such as assignments, assessments, discussion boards, and live chat (CSUS,
2009).
Section 508 of the Rehabilitation Act
Requires "Federal agencies to make their electronic and information technology
accessible to people with disabilities" in order to remove barriers to information
7
technology (IT Accessibility & Workforce Division, 2009). Section 508 guidelines, as
they relate to web-based technology and information, are based on the World Wide Web
Consortium's Web Content Accessibility Guidelines.
Usability
The user's perspective on accessing information and designing with the user's
perspective in mind (Federici et al., 2005; Seale, 2006). Nielsen (1993) developed five
usability criteria that include efficiency (how quickly a user completes a task), errors
(the number of errors a user commits and how quickly the user recovers from the
errors), satisfaction (user likes how the website functions), memorability (returning
users can retain how to navigate and how the site functions), and learnability (how
quickly new users are productive on the website).
Usability test
Typically involves having ". . . a number of test users (selected to be as
representative as possible of the intended users) use the system to perform a prespecified
set of tasks" (Nielsen, 1993, p. 27). The data obtained from the usability test can be
used in the ". . . creation and maintenance of Web sites in order to bring about the
greatest ease of use, ease of learnability, amount of usefulness, and least amount of
discomfort for the humans who have to use the system" (Pearrow, 2000, p. 12).
Web Content Accessibility Guidelines [WCAG]
Provides information on how to make web content, which includes web
applications, and text, images, and forms on a web page, accessible for people with
disabilities. WCAG 1.0 provides a checklist of guidelines for accessibility. Priority 1
8
checkpoints "must" be satisfied; Priority 2 checkpoints "should" be satisfied; and Priority
3 checkpoints "may" be satisfied. WCAG 2.0 provides 12 guidelines with three
conformance levels based on success criteria, ranging from A (lowest) to AAA (highest).
WCAG 2.0 also provides techniques on how to meet each guideline (Henry, 2008).
Limitations of the Problem
The author has a limited time frame in which to meet the web accessibility
guidelines that are in place now. The study will incorporate the current state of
technology and student expertise, which may change in the future.
The project may be valuable to other departments at Sacramento State and other
California State Universities seeking to replicate this study in an effort to conform to
Section 508 requirements by the deadline mandated by the Office of the Chancellor.
The results of the usability test may be applicable to other California State Universities
if their website is designed with the same format as Sacramento State's website.
The results of the usability study will benefit students with disabilities who are
impacted by issues of accessibility and usability on the Sacramento State website.
Website developers at Sacramento State may incorporate the results of the usability
study in their website design plans in order to comply with Section 508.
The author is a graduate student in the Vocational Rehabilitation Counseling
program at Sacramento State. The author's graduate studies focus on addressing the
needs of people with disabilities. Though the results of the usability test will have
implications for students without disabilities, the author is biased towards the needs of
students with disabilities who struggle with website accessibility and usability issues.
9
Chapter 2
LITERATURE REVIEW
Introduction
The review of literature will cover the following sections: Legislation/
Regulation/Policy; California State University System Web Accessibility; Evaluation of
Sacramento State Web Pages; Accessibility of Websites; Impact on People with
Disabilities; and Usability Testing.
Legislation/Regulation/Policy
The Federal government enacted several laws to ensure that people with
disabilities were allowed full participation and inclusion in employment, transportation,
public accommodations, commercial facilities, state and local government institutions,
education, housing, voting, and telecommunications. One of the laws enacted by the
Federal government is the Rehabilitation Act of 1973, which ". . . prohibits discrimination on the basis of disability in programs conducted by Federal agencies, in programs
receiving Federal financial assistance, in Federal employment, and in the employment
practices of Federal contractors" (U.S. Department of Justice [DOJ], 2005, n.p.).
In 1998, the Rehabilitation Act was amended to remove barriers to electronic
information technology for people with disabilities. Section 508 of the Rehabilitation
Act requires Federal agencies, during the development, procurement, maintenance or
use of electronic and information technology, to ensure that people with disabilities have
". . .access to and use of information and data that is comparable to the access to and use
of the information and data" (United States Access Board, 2010a, Purpose sec., para. 1)
10
by people who do not have disabilities. The Technical Standards in Subpart B, 1194.22
of Section 508 of the Rehabilitation Act describe the requirements for web-based
technology and information. The standards are based on access guidelines developed by
the World Wide Web Consortium's Web Accessibility Initiative, and only apply to ". . .
Federal web sites but not to private sector web sites (unless a site is provided under
contract to a Federal agency, in which case only that web site or portion covered by the
contract would have to comply)" (United States Access Board, 2010a, Web-based
Intranet and Internet Information and Applications sec., para. 8).
Section 508 of the Rehabilitation Act applies to the California State University
system [CSU] because it receives Federal funding as a public post-secondary institution.
In 2003, "California State Legislature amended Government Code 11135 to require CSU
to implement Section 508 and to apply the federal accessibility standards to the
electronic and information technology [EIT] products and services that CSU buys,
creates, uses, and maintains" (Kaplan, 2010, How does Government Code 11135 apply
to CSU? sec., para. 5). In January 2006, the California State University system
implemented the Accessible Technology Initiative [ATI] to meet the provisions of CSU
Executive Order 926, which stated that ". . . it is the policy of the CSU to make
information technology resources and services accessible to all CSU students, faculty,
staff, and the general public regardless of disability" (California State University,
Sacramento [CSUS], 2010a, CSU Policy sec., para. 2).
11
California State University System Web Accessibility
In the Coded Memo AA-2007-04, Reichard and West (2007) established a web
accessibility timeline for the California State University system, which specified
deadlines for goals that were to be met by CSU campuses over the course of 6 years. By
May 17, 2007, all CSU campuses were required to conduct a self-evaluation of their
website accessibility using an automated and manual checking process. The automated
process involved using a computer program, such as HiSoftware AccMonitor, to
determine whether the website conforms to Section 508 guidelines. The automated web
accessibility evaluator, AccMonitor, scans a website and provides of summary of the
items on a web page that passed and failed Section 508 guidelines.
For each failed item, AccMonitor provides a number that references a Section
508 standard and the corresponding section number that the item failed to meet (CSU,
Northridge, 2010). For example, Section 508 Section 1194.22, sub-section (a) requires
that ". . . a text equivalent for every non-text element shall be provided" (United States
Access Board [USAB], 2010a, Web-based intranet and internet information and
applications sec., para. 1). A text equivalent contains a text description of an image that
makes the content accessible for people with visual limitations who use a screen reader,
as well as for people who are accessing the content with images turned off on a mobile
device or at lower bandwidths (W3C, 2009). For example, "Image of the Earth" would
be an equivalent text description for an image of the Earth as opposed to "image1.jpg."
By implementing this Section 508 standard into a website, the web page element is
made accessible to people with visual impairments (Evans, 2003).
12
The Accessible Technology Initiative [ATI] manual checking process is divided
into "must repair" and "best practice" items. "Must repair" items require immediate
compliance with Section 508 guidelines or an alternative means of content delivery, if
the item cannot meet compliance standards immediately. "Best practice" items should
be applied to the website if the website development tools and skills are available. For
example, Checkpoint A of the ATI Section 508 manual evaluation addresses text
equivalents. A "must repair" item asks, "Do images that convey content have equivalent
alt-text?" (fix if "no")" (Jewett, 2007, Checkpoint A, text equivalents sec., para. 1).
This "must repair" item addresses Section 508 Section 1194.22, sub-section (a) as does
the computer program AccMonitor; however, during the manual checking process, a
person peruses each web page to find "must repair" items. A "best practice" for
Checkpoint A requires the inclusion of a title and alt attribute, which identifies the
content of the page (e.g., "A User's Guide to Web Accessibility") and a description of
the image (e.g., "image of a book"), respectively (Berners-Lee, 1998; WebAIM, 2010b).
By June 15, 2007, campuses were originally required to submit their First Year
Web Report, which detailed the results of the HiSoftware automatic accessibility
evaluation report, and a Web Accessibility Implementation Plan, which detailed their
plans for remediating accessibility issues. By September 1, 2007, ". . . new and updated
administrative websites, web applications, and web content produced by the CSU or by
third-party developers should, at a minimum, conform to baseline accessibility standards
as defined in Section 508, Subpart B, and where appropriate, Subpart C" (Reichard &
West, 2007, p. 3). By May 15, 2009, the minimum baseline guidelines were required to
13
be met by all administrative websites ". . . that are critical to institutional access,"
(Reichard & West, 2007, p. 3) which is defined as websites that the campus considers
". . . important for its mission and operation" (Reichard & West, 2007, p. 6). By May
15, 2012, all CSU websites should fully conform to Section 508 guidelines. If
conformity to the guidelines established in Section 508 creates an undue burden, due to
". . . significant difficulty or expense" (Reichard & West, 2007, p. 6), an alternative form
of access that is equally effective must be provided by the CSU campus.
After 2 years of working diligently to meet the goals established in Coded Memo
AA-2007-04, the ATI Department at the Chancellor's Office determined that the original
timelines and milestones were ". . . overly ambitious, and that successful and effective
implementation of the ATI will require more time than originally envisioned"
(Echeverria & Quillian, 2009, p. 2). The deadline for Section 508 conformity by all
administrative websites and instructional websites was extended to June 1, 2010.
Beginning in June, 2011, according to Coded Memo AA-2010-13, Information
Technology Services at the Chancellor's Office will establish baseline requirements for
ATI system-wide implementation based on the goals and success indicators that the
majority of campuses reached in the previous year's annual campus reports. An example
of a web accessibility goal is ". . . identify and repair or replace inaccessible websites,
web applications, and digital content." Success Indicators for this goal could be ". . .
assigned responsibility for the evaluation process to a body (person(s) or business
entity)" and "inventoried all campus administrative websites" (Quillian, 2010, p. 19).
Then, all campuses will be notified of a deadline by which they must comply with
14
baseline requirements for accessibility. The campus reports from Academic Year
2009-2010, which are due in November 2010, will form the initial baseline requirements
for CSU accessibility. Every 3 years, Information Technology Services will revise the
baseline requirements based on the accessibility accomplishments of the majority of
CSU campuses.
Evaluation of Sacramento State Web Pages
During the first year of the web accessibility timeline at the California State
University system, Sacramento State compiled information for their First Year Web
Report and Web Accessibility Implementation Plan. They compiled a cross-section of
20 administrative websites, which ". . . represent approximately 8,000 webpages and are
under the control of as many as 22 web developers with a wide variety of development
experience. Very few of the developers in question have any formal or extended
experience in creating accessible webpages" (CSUS, 2010c, para. 2). The websites
selected were ". . . determined to represent the most frequently accessed areas of our
administrative presence," which included: Campus homepage, Admissions, Associated
Students, Campus Calendar, Catalog, College of Business Administration, College of
Continuing Education, College of Education, Common Management Systems, Financial
Aid, Help Desk, Human Resources, Library, News, Police, Public Affairs, SacLink,
Schedule, Services to the Students with Disabilities, and Transportation and Parking
(CSUS, 2010e, para. 1).
Sacramento State used HiSoftware AccMonitor on all pages in the cross-section
of the 20 administrative websites they selected. The Accessible Technology Initiative
15
then requested that ". . . based on the HiSoftware reports and your evaluation of the
importance of each page, select approximately 15-20 of the cross-section pages as a
‘repair sample’ to manually evaluate and remediate. This sample should include both
pages that have ‘passed’ and pages that have ‘failed’ the automated testing" (CSUS,
2010e, para. 4). Then, Sacramento State was required to completely remediate each
page in the "repair sample" based on the automatic and manual evaluation results. If
they were unable to resolve an accessibility issue immediately, Sacramento State was
required to document how and when the issue would be remediated.
Based on AccMonitor's automatic evaluation of the representative sample of
Sacramento State web pages, ". . . the primary issue causing Sacramento State website to
fail AccMonitor scans is missing alt-text on images. Among the 20 websites on our
cross section, there are 42,166 violations of Section 508 Section 1194.22, sub-section
(a)" (CSUS, 2010e, para. 1) which requires a text description for every non-text element.
The text description of a non-text element, such as an image, enables people with visual
impairments to read the description of an image with a screen reader. Other issues that
were discovered during the manual and automatic evaluation process included
inaccessible forms, simple tables without appropriate headers, and fly-out menus which
present issues for keyboard-only visual users (CSUS, 2010e). Forms allow website
users to complete online transactions, such as registering for a class and making
payment arrangements. The user inputs information into the form element, such as a
text box or by selecting applicable check boxes, and submits the information to
complete a transaction. In order to make a form accessible, each form element, such as
16
a text box or check box, should be assigned a label, which requests or prompts a user for
information. For example, the label for a text box could inform the user that they need
to input their first name in the text box.
Tables are used to arrange information into rows and columns of cells. To make
tables accessible, an appropriate heading is needed at the top of a column to identify
elements in the column (Web Usability, 2004). For example, <th> Semester Courses
</th> would be used to label a column with the heading Semester Courses. Fly-out
menus are sub-menus that appear when a user hovers over the main menu item. In order
to make fly-out menus accessible, Sacramento State recommends providing an alternate
means of accessing web page links that are contained in the sub-menu. In their First
Year Web Report, they noted that ". . . fly-out menus in general seem to cause issues for
users without the hand coordination required to activate sub-menus within menus"
(CSUS, 2010e, para. 5). By November 2010, Sacramento State must submit their
annual report for Academic Year 2009-2010, which will document their progress
towards remediating accessibility issues detailed in their First Year Web Report. The
annual report will also include Sacramento State's progress towards meeting the
Accessible Technology Initiative implementation requirements, which are outlined in
Coded Memo AA-2007-04 and AA-2010-13 (Accessible Technology Initiative, 2010).
Accessibility of Websites
With the increasing complexity of websites, including graphics and auditory
features, people with disabilities find it increasingly difficult to access information, even
with assistive technology (Bowman & Jaeger, 2005). Assistive technology includes
17
". . . any tool or device used by anyone to accomplish a purpose. More specifically,
assistive technology typically focuses on the special needs of people of all ages who
may have a variety of special sensory, motoric, cognitive, and/or linguistic needs"
(King, 1999, p. 13). An example of assistive technology is a screen reader, which
converts text into synthesized speech or Braille characters on a refreshable Braille
device (WebAIM, 2010a). Unfortunately, many web pages were developed without the
end user in mind. Some of the unusable web pages can be attributed to developers
focusing more on the technical aspects of the site than on the user experience; making
assumptions based on their highly technical background that ". . . do not generalize to
the people who will actually be using the site"; and incorporating the latest technological
advances in an effort to surpass their competition, which led to a reduction in the ". . .
actual usability of the Web site" (Pearrow, 2000, pp. 17-18).
With the increasing use of the Internet for online courses and distance education
classes, information accessibility becomes a vital component in the advancement of
people with disabilities. In response to web accessibility issues, the World Wide Web
Consortium [W3C], as part of the Web Accessibility Initiative, published the Web
Content Accessibility Guidelines [WCAG] 1.0 in May 1999. The 14 guidelines, which
are listed below, provide general principles on designing accessible web page content,
including scenarios that describe common accessibility problems encountered by people
with disabilities.
Guideline 1: Provide equivalent alternatives to auditory and visual content.
Guideline 2: Don't rely on color alone.
18
Guideline 3: Use markup and style sheets and do so properly.
Guideline 4: Clarify natural language usage.
Guideline 5: Create tables that transform gracefully.
Guideline 6: Ensure that pages featuring new technologies transform gracefully.
Guideline 7: Ensure user control of time-sensitive content changes.
Guideline 8: Ensure direct accessibility of embedded user interfaces.
Guideline 9: Design for device-independence.
Guideline 10: Use interim solutions.
Guideline 11: Use W3C technologies and guidelines.
Guideline 12: Provide context and orientation information.
Guideline 13: Provide clear navigation mechanisms.
Guideline 14: Ensure that documents are clear and simple.
Each guideline contains checkpoints, which explain how to apply the guideline
to the design of a web page. Checkpoints are assigned a priority level based upon the
impact the design element has on accessibility. Priority 1 checkpoints must be satisfied
to avoid making it "impossible to access information in the document." Priority 2
checkpoints should be satisfied to avoid making it "difficult to access information
in the document." Priority 3 checkpoints may be addressed to avoid making it
"somewhat difficult to access information in the document" (W3C, 1999, Priorities sec.,
para. 1-3). Web content conformance levels are rated based on the checkpoints that are
satisfied: A (satisfied all Priority 1 checkpoints), AA (satisfied all Priority 1 and 2
checkpoints), or AAA (satisfied all Priority 1, 2, and 3 checkpoints).
19
The Web Content Accessibility Guidelines 1.0 still remain in effect. However,
with the publishing of the WCAG 2.0 on December 11, 2009, the World Wide Web
Consortium ". . . recommends that new and updated content use WCAG 2.0. Although
it is possible to conform either to WCAG 1.0 or to WCAG 2.0 (or both)" (W3C, 2008b,
Abstract sec., para. 3). WCAG 2.0 is organized around four design principles:
Perceivable, operable, understandable, and robust.
Perceivable means that users must be able to perceive the information
being presented (it can't be invisible to all of their senses).
Operable means that users must be able to operate the interface.
Understandable means that users must be able to understand the
information as well as the operation of the user interface.
Robust means that users must be able to access the content as
technologies advance. (W3C, 2008a, Understanding the Four Principles of
Accessibility sec., para. 1)
WCAG 2.0 contains 12 guidelines that each contain testable success criteria, which can
be rated at three conformance levels: A (lowest), AA, or AAA (highest). The WCAG
2.0 guidelines are listed below.
Guideline 1.1: Provide text alternatives for any non-text content so that it can be
changed into other forms people need, such as large print, Braille, speech, symbols or
simpler language.
Guideline 1.2: Provide alternatives for time-based media.
20
Guideline 1.3: Create content that can be presented in different ways (for example, a
simpler layout) without losing information or structure.
Guideline 1.4: Make it easier for users to see and hear content including separating
foreground from background.
Guideline 2.1: Make all functionality available from a keyboard.
Guideline 2.2: Provide users enough time to read and use content.
Guideline 2.3: Do not design content in a way that is known to cause seizures.
Guideline 2.4: Provide ways to help users navigate, find content, and determine where
they are.
Guideline 3.1: Make text content readable and understandable.
Guideline 3.2: Make Web pages appear and operate in predictable ways.
Guideline 3.3: Help users avoid and correct mistakes.
Guideline 4.1: Maximize compatibility with current and future user agents, including
assistive technologies.
Further, the World Wide Web Consortium is advocating the international
harmonization of web accessibility standards. At the present time, web developers are
challenged to meet differing technical standards at their national, state, and
organizational levels. By adopting the Web Accessibility Initiative guidelines for web
content, browsers, media players, and authoring tools, one consistent set of technical
standards would be used by all countries. Since the Web is worldwide, standards
harmonization would enable ". . . people with disabilities from other countries, with the
21
same needs for accessibility and using the same kinds of assistive technology" to access
resources of another country (W3C, 2009, Current Situation sec., para. 4).
Impact on People with Disabilities
Though the World Wide Web Consortium [W3C] has been developing protocols
and guidelines for the Internet since October 1994, it was not until the enactment of
Section 508 of the Rehabilitation Act that web developers were legally required to use
protocols and guidelines. Though W3C and Section 508 of the Rehabilitation Act have
advanced the needs of people with disabilities, this population is still adversely affected
by inaccessible websites (Waddell, 1999). Individuals who are unable to access
information technology ". . . will be excluded from learning and using one of the most
essential and important tools for personal, educational, and vocational productivity,
communication, and recreation" (King, 1999, p. 18). The World Wide Web Consortium
describes barriers that people with disabilities may experience when accessing web
pages. For example, people with visual impairments may encounter web pages with
"complex images (e.g., graphs or charts) that are not adequately described," "forms that
cannot be tabbed through in a logical sequence or that are poorly labelled," and "web
pages with absolute font sizes that do not change (enlarge or reduce) easily" (W3C,
2007, Blindness sec., para. 2). People with hearing impairments may encounter "lack of
captions or transcripts of audio on the Web, including webcasts" and "requirements for
voice input on Web sites" (W3C, 2007, Deafness sec., para. 3). People with motor
disabilities affecting the hands or arms may encounter "time-limited response options on
Web pages" and "browsers and authoring tools that do not support keyboard alternatives
22
for mouse commands" (W3C, 2007, Motor disabilities sec., para. 3).
Furthermore, according to Slatin, ". . . users with disabilities are three times less
likely to use the Web for routine tasks, as compared with similarly experienced peers
without disabilities. The trends suggest, therefore, that unequal access to web-based
information may disproportionately hinder persons with disabilities, at school, work,
and home" (as cited in Klein et al., 2003, p. 28). In 2004, the Disability Rights
Commission reported that ". . . 81% of web sites tested failed to meet even the most
basic accessibility criteria." Web pages are difficult to navigate with assistive devices
because they cannot adequately ". . . handle graphics, moving images, frames, tables,
use of scripting languages and streaming audio and video" (as cited in Murphy et al.,
2007, p. 79). Studies have also shown that ". . . guidelines have been found not to cover
more than 55% of accessibility barriers, therefore websites may 'pass' the priority
checkpoints, but there are still issues encountered by real users" (as cited in Murphy et
al., 2007, p. 80).
The Disability Rights Commission recommends that users be involved in the
design and testing process to improve the website's accessibility and usability (Brophy
& Crave, 2007). Further, Adam and Kreps (2006) state that reliance on automatic
accessibility checkers are ". . . insufficient and unable to substitute for a human check.
The existence and proliferation of such software has, in certain respects, actually
hampered the global project of making the web more accessible. False stamps of
accessibility impede the process of genuinely making the web more accessible" (p. 211).
23
Usability Testing
Relying solely on automatic accessibility checkers that evaluate objective
content may overlook accessibility issues. For example, a web page may pass an
inspection because it contains a text-equivalent description of an image. However, the
description may not adequately describe the image, which yields the feature inaccessible
for users (Klein et al., 2003). According to Fukuda et al., a screen reader cannot
recognize which elements of a web page are for layout purposes, and which describe
important content for the user. This becomes a time consuming process for the user to
sort through the audio clutter, even if the web page is fully compliant with the Web
Content Accessibility Guidelines (as cited in Leuthold, Bargas-Avila, & Opwis, 2007).
The best way to ensure that a website is accessible and usable is to include the actual
users or a representative sample in the study (Brophy & Craven, 2007; Federici et al.,
2005; Sloan et al. as cited in Wattenberg, 2004).
Incorporating real users in the evaluation of a website is known as usability. The
users perform real tasks in order to yield qualitative and quantitative information about a
product, such as a web page (Henry, 2007). Pearrow (2000) defines usability as
. . . the broad discipline of applying sound scientific observation,
measurement, and design principles to the creation and maintenance
of Web sites in order to bring about the greatest ease of use, ease of
learnability, amount of usefulness, and least amount of discomfort
for the humans who have to the use the system. (As cited in TechSmith,
2009, p. 12)
24
According to Nielsen (2000), with five users, you are able to discover 85% of the
usability problems. However, you need ". . . at least 15 users to discover all the
usability problems in the design. As you add more and more users, you learn less and
less because you will keep seeing the same things again and again" (Iterative design
sec., para. 1). Nielsen recommends three smaller tests that each involve five users,
using the test results to improve the design before each successive test, instead of
investing in one elaborate test that involves 15 users. In the case of users from disparate
groups, Nielsen recommends ". . . 3-4 users from each category if testing two groups of
users [or] 3 users from each category if testing three or more groups of users (to ensure
that you have covered the diversity of behavior within the group)" (When To Test More
Users sec., para. 4).
Usability testing measures the user's effectiveness, efficiency, and convenience
with the website using usability tasks, questionnaires, and debriefing sessions.
According to King (1999),
Effectiveness is the ability of the device to accomplish the task for
which is intended in conjunction with the user. Efficiency measures
accurate and successful uses versus error uses, plus rate of use.
Convenience is defined as user ease in application of the devicehow much or little effort and discomfort across several domains (physical,
cognitive, linguistic, etc.) is required of the user. (p. 41)
As part of the usability testing process, usability tasks contain a short scripted text that
instructs the user on the task to be performed, criteria to measure success and failure of
25
the task, and a start state for the computer and web session when the task begins. For
example, website developers could be interested in finding out how users use the store
on their website. During the usability test, the participant could be instructed that, "You
have been researching different types of video camera to buy to record family videos
and transfer them on to your computer. You want to use the Web site to find
information about video cameras and purchase a camera based on your need"
(TechSmith, 2009, p. 5).
An example of programs that can be used for usability testing are Morae
Recorder and Manager, which were developed by TechSmith. Morae Recorder is able
to capture audio and video while the user is interacting with software or a website, while
also capturing the activity that is occurring on the computer screen. Morae Manager is
then used to synthesize multiple recordings and analyze the data captured through
Morae Recorder (TechSmith, 2009). The data gathered with Morae can be evaluated
based on multiple criteria, including task success, time on task, satisfaction, and mouse
clicks. Task success measures “whether or not the participant was successful, and to
what degree” when performing a task. Time on task measures “the length of time it
takes the participant to complete a task.” Satisfaction “allows the participant to quantify
and describe their emotional reaction to a product before, during or after a study.”
Mouse clicks “measures the number of clicks that a participant makes while completing
a task” (TechSmith, 2009, p. 7). Analyzing this data can provide crucial information
about usability issues for website developers that can be incorporated into the
improvement of a website's design (Brophy & Craven, 2007).
26
Chapter 3
METHODOLOGY
Review of Sources
The researcher gathered information from books, journals, and articles that
substantiated the need for usability testing that involves the actual users of websites.
The researcher also gathered extensive information from websites.
Method
In Fall 2009, the researcher met with Sacramento State's Co-Director of the High
Tech Center and TRIO SSS Project, and Accessible Technology Initiative Project
Coordinator, Melissa Repa, to identify issues impacting students with disabilities at
Sacramento State. Repa discussed the need to conduct a usability test to address
accessibility issues with the Sacramento State website, including SacCT and My Sac
State. In order to ensure that the website is accessible and usable by the population
served by Services to Students with Disabilities, students with disabilities participated in
the usability test.
The researcher developed a recruitment flier, demographics questionnaire,
consent form, orientation script, and thank you letter, all contained in Appendix A.
The orientation script was read aloud by the researcher to each participant before
commencing the usability test. Developing the orientation script was challenging
because models provided in the literature were of a formal nature. However, given the
anxiety and insecurity that often surrounds tests, an informal orientation script was
27
written because the objective is to make the participants feel at ease. The usability tasks
script, which contains the usability tasks, the qualitative questions, and debriefing script
are contained in Appendix B. They were developed in collaboration with Melissa Repa;
Cryssel Vera, Instructional Materials Accessibility Specialist; and Dave Katten, Web
Accessibility Specialist. In May 2010, the Committee for the Protection of Human
Subjects approved the usability test as minimal risk. The usability test was pilot tested
in Spring 2010 using staff from Sacramento State. Pilot testing allowed the researcher
to ensure that the usability tasks script was comprehensible, the Morae software is
functioning properly, and the time frame allotted for the usability test was adequate.
Participants for the study were volunteers from the population of students who
receive services through Sacramento State's Services to Students with Disabilities.
Recruitment fliers for participation were posted in Sacramento State's High Tech Center
and at the office of Services to Students with Disabilities, which are places frequented
by students with disabilities. Participants contacted the High Tech Center Coordinator,
Carol Houston, to express an interest in participating in the study. Then, they were
scheduled for an hour time slot for the usability test. In order to meet the goal of 10
participants in the study, the researcher recruited additional students who were using the
High Tech Center on the day that the test sessions were held.
The researcher read the orientation script to each participant before asking the
participant to read and sign the consent form. A demographics questionnaire was used
to gather information about the participants. Then, over the course of 1 hour,
participants were seated in front of a computer, and asked to perform a set of 12
28
usability tasks on the website, which are listed below.
Task #1: Please login to My Sac State.
Task #2: You have been added to the Instructional Materials Accessibility SacCT
course. Please login to the course and open the syllabus.
Task #3: Can you tell me what the instructor's office location is on the syllabus?
Task #4: Find the About the Course Learning Module.
Task #5: I have created a quiz in SacCT. Please find and take the quiz.
Task #6: Can you find the Accessible Word Documents Discussion page?
Task #7: Can you post a new message?
Task #8: Go to the Course Content page.
Task #9: Now, go to the Virtual Office Hours.
Task #10: Please open the iMEET session scheduled for today's date to view the
meeting for this course.
Task #11: Using the Course Tools menu, can you go to the Web Links page?
Task #12: Can you open the "Audio Tutorial - Adding Document Language to PDF"
Web Link and listen to the media?
The researcher was present in the room with the participant to read the usability
task script to the participant and to take notes during the session. The participants'
responses and performance on tasks were recorded using the computer software, Morae
Recorder, and a Microsoft web camera. At the conclusion of the usability test,
participants participated in a debriefing session with the researcher. Participants
received a thank you letter from the researcher, and a thank you letter and post-it notes
29
from the Accessible Technology Initiative.
The computer software, Morae Manager, was used to evaluate the audio and
video data captured from the participants during the usability test sessions. The
researcher transcribed the participants' responses to the questions posed during the
session, and developed a summary of recurring themes that posed difficulty for the
participants, which are included in the Results sub-section. In addition, a table was
developed to capture participants' performance on 12 usability tasks using the criteria
completed with ease, completed with difficulty, and failed to complete. Completed with
ease was identified by participants completing the task without needing to be prompted
or redirected by the researcher. Completed with difficulty was identified by participants
needing additional prompts or redirection from the researcher or the participant
exploring or clicking on items that were unrelated to the task. However, the task was
eventually completed. Failed to complete was identified by participants that did not
accomplish the task. The participants’ responses and the table appear in Appendix C.
The results of the usability test guided the development of website improvement
suggestions for the website development team. The website improvement suggestions
were provided in an effort to further comply with the World Wide Web Consortium's
Web Content Accessibility Guidelines and Section 508 of the Rehabilitation Act.
Participants
The demographics questionnaire gathered information about the participants'
highest level of education, functional limitations, computer experience, use of assistive
technology when accessing websites, and frequency of visits to Sacramento State's
30
SacCT and My Sac State. The participant demographics information is presented in
tabular format in Appendix D.
Education
One (10%) participant was a high school graduate, eight (80%) participants
completed some college or earned an associate's degree, and one (10%) participant
earned a bachelor's degree.
Functional limitations
Participants varied in their responses to their functional limitations, which
included mobility impairments (30%) (e.g., arthritis and an inability to move one's
fingers), and learning disabilities (70%) (e.g., difficulties with reading and processing
auditory information).
Level of computer experience
One (10%) participant indicated a beginner level of computer experience, seven
(70%) participants indicated an intermediate level, and two (20%) participants indicated
an advanced level.
Assistive technology
Four (40%) participants do not use assistive technology when accessing
websites, one (10%) participant uses a screen reader (e.g., JAWS), two (20%)
participants use voice activated software (e.g., Dragon NaturallySpeaking), and three
(30%) participants use multiple assistive technologies. Of the three participants who use
multiple assistive technologies when accessing websites, one participant uses a screen
magnifier, screen reader, and voice activated software; one participant uses a screen
31
reader and voice activated software; and one participant uses alternate input devices
(e.g., trackball or touch screen) and voice activated software.
Use of SacCT on the Sacramento State website
One (10%) participant never uses SacCT, three (30%) participants use SacCT
one to three times per week, and six (60%) participants use SacCT seven or more times
per week.
Use of My Sac State on the Sacramento State website
One participant (10%) never uses My Sac State, five (50%) participants use My
Sac State one to three times per week, one (10%) participant uses My Sac State four to
six times per week, and three (30%) participants use My Sac State seven or more times
per week.
Results
After analyzing the audio and video data captured from participants, the
researcher identified recurring themes related to the usability of SacCT and My Sac
State. It is important to note that none of the six participants who indicated on their
demographics questionnaire that they use assistive technology when accessing websites
chose to use assistive technology during the usability testing session. Thus, the data
captured does not assess the usability of assistive technology devices on SacCT and My
Sac State. The themes captured were related to several topics, including obtaining help
on SacCT, logging into SacCT, training on SacCT, accessing documents through
SacCT, syllabi document formats, online quizzes, communicating through discussion
pages, system maintenance, SacLink e-mail, online meeting software, clutter on My Sac
32
State, course tools menu, and internet browsers. In order to capture participants'
performance on the 12 usability tasks, a table was used to quantify how many
participants completed the task with ease, completed the task with difficulty, and failed
to complete the task. The table is included in Appendix D. The author attempted to use
advanced statistical methods, such as the Pearson Chi-square, to determine whether any
correlation existed between participants' demographic information and their performance
on the 12 usability tasks; however, the sample size of participants was too small.
Therefore, qualitative data that was obtained from participants during the usability test
was the most significant source of information about usability issues with SacCT and
My Sac State.
Obtaining Help on SacCT
Three of the participants had difficulty finding the help desk phone number or a
help section on the website.
Logging into SacCT
Three of the participants expressed frustration with having to perform the login
process twice, through My Sac State and SacCT, in order to access SacCT content. The
researcher observed some of participants having difficulty logging into the SacCT
course that was developed for the usability testing sessions. Two of the participants
were unable to identify the correct online course on their SacCT Course List. Two of
the participants clicked on several links, such as Student Center and SacCT Student
Resources, before successfully finding the SacCT login prompt. One of the participants
watched the "How To Login" video tutorial before successfully logging into SacCT.
33
Seven out of the 10 participants did not know an alternate way to login to SacCT.
Training on SacCT
Seven of the participants did not have formal training in the use of SacCT. They
had learned how to navigate SacCT by exploring the Learning Management System
independently or had previous experience with taking an online college class. Two of
the participants would like to improve their navigational abilities on SacCT, and
suggested an online step-by-step tutorial on all of the SacCT features.
Accessing documents through SacCT
Two of the participants expressed frustration with the cumbersome process of
accessing documents through SacCT. In order to access a Word document, participants
had to download the Word document. After downloading the document, they were
returned to the SacCT homepage that listed their online courses. Then, participants had
to enter the course again and return to the webpage where they had previously accessed
the Word document in order to open it and view it.
Two of the participants expressed that it is a time-consuming process to make
documents accessible in Kurzweil, which may involve scanning and formatting
documents. Asking the professor to provide an electronic version of the document was
described as an embarrassing experience.
Syllabi document formats
Six of the participants stated that their professors post their syllabi in PDF
format or Word format. PowerPoint and Excel formats were mentioned by two of the
participants.
34
Online quizzes
Three of the participants expressed the need for additional time when taking
quizzes online. Two of the participants experienced difficulty accessing online quizzes
and keeping track of upcoming quizzes. However, they found the calendar feature on
SacCT to provide a helpful reminder about upcoming quizzes which can be easily
viewed on their SacCT homepage.
Communicating through Discussion pages
There was variability in how participants viewed Discussion pages, ranging from
confusion to ease of use. Three of the participants who found Discussion pages
overwhelming and confusing described difficulty with reading and replying to a plethora
of Discussion posts created by the entire class. However, communicating through
Discussion posts was manageable when the instructors created smaller Discussion
groups, consisting of five students. Four the participants who had favorable responses
described how Discussion pages foster communication among students that may not
occur in the classroom. Also, it is helpful for students to read questions posted by other
students and responses to their questions, which can provide vital information for
students who have the same question. However, the responses may not always be
posted in a timely manner.
System maintenance
Two of the participants disliked that SacCT is unavailable during certain times
for routine system maintenance.
35
SacLink e-mail
Two of the participants did not know how to check SacLink e-mail. Two of the
participants would prefer that SacLink e-mail be sent to their personal e-mail address.
Online meeting software
The researcher observed many participants having difficulty accessing the
iMEET session. Two of the participants were unable to identify the link to join the
scheduled iMEET session. Three of the participants who clicked on the link to join the
scheduled iMEET session did not notice the yellow bar at the top of their screen
prompting them to download files to their computer in order to run the iMEET session.
After being prompted by the researcher, the participants downloaded the files and were
automatically routed to the SacCT homepage. Then, they had to navigate to the
webpage that featured the iMEET session link and click on the link again to join the
iMEET session. One out of 10 participants had previous experience with iMEET, and
described the confusing process of figuring out when it was appropriate to speak and
difficulty with identifying whether a lecture was in progress.
Clutter on My Sac State
One of the participants found My Sac State cluttered with unnecessary features,
such as Sac State News and Academic Calendar.
Course Tools menu
Two of the participants would like the icons on the Course Tools menu to have a
title or description that corresponds to the feature. So, when a participant places the
36
mouse over a Course Tools menu item, a description would appear that informs the
participant of the purpose of the feature.
Internet browsers
One of the participants did not like having to use a specific Internet browser to
access course material on SacCT, such as being instructed to use Mozilla Firefox in
order to avoid accessibility issues.
In Chapter 4, the author outlines recommendations that may be used by website
developers to rectify the usability issues with My Sac State and SacCT that were
identified in this usability study.
37
Chapter 4
SUMMARY AND RECOMMENDATIONS
In order to address the usability issues that were captured in the data analysis
from the usability testing sessions, the researcher recommends the following
14 interventions.
1.
Make the resources for help on My Sac State and SacCT highly visible on the
website, including the service desk's phone number, online tutorials, training
opportunities, and one-on-one assistance.
2.
Advertise the student workshop on "You, Your Computer, and SacCT," which
provides instruction on navigating SacCT, introduction to the most commonly used
tools, and the software needed to effectively access SacCT courses.
3.
Create one centralized login process that provides access to My Sac State
and SacCT.
4.
Inform instructors that they must create electronic versions of all of their course
material as a new standard for universal accessibility.
5.
Inform students that alternative formats of documents that are inaccessible
through SacCT or when using assistive technology can be obtained from the instructor.
6.
Restructure SacCT to enable students to download documents and files without
being re-routed to the SacCT homepage.
7.
Inform instructors on how to obtain the How-To Guide on "Creating
Extended-Time Exams," so that they can allot additional time for students with
disabilities who are taking quizzes on SacCT.
38
8.
Encourage instructors to develop smaller cohorts of students for Discussion
pages in order to make reading and posting messages less cumbersome.
9.
Provide students with instructions on how to configure their personal e-mail
account's Post Office Protocol (POP) server to retrieve SacLink e-mail through their
personal e-mail account if accessibility is an issue.
10.
Make the iMEET session join link highly visible.
11.
Develop guidelines for iMEET etiquette.
12.
Enable students to customize their My Sac State page by being able to hide
information that is not considered useful.
13.
Ensure that students are using the latest versions of Internet Explorer and Mozilla
Firefox Internet browsers, which provide tips on using each Course Tool in SacCT.
14.
Provide students with instructions on how to configure their Internet browser for
accessibility in an online learning environment.
This author recommends that this study be repeated in the future because My Sac
State is undergoing revisions, and usability results need to be generated for the new
website design. Also, future usability tests need to include students with disabilities
who are using assistive technology during the testing sessions. In addition, including
students with visual impairments and hearing impairments may introduce new usability
issues, such as when accessing Discussion pages and iMEET sessions. The author did
not use task completion time to measure participants' task success because the author
wanted to observe participants' process of accomplishing task goals without specifying a
particular path that needed to be taken to reach the desired goal state. However, future
39
research may include time as a measurement of success in order to more objectively
quantify tasks that were completed successfully, completed with difficulty, and tasks
that failed to be completed.
40
APPENDIX A
Materials Related to Participation in Research
41
Recruitment Flier
Vocational Rehabilitation Counseling Program
Sacramento State
PARTICIPANTS NEEDED FOR RESEARCH ON THE
SACRAMENTO STATE WEBSITE
We are looking for volunteers who are students with disabilities
to take part in a study that gathers your feedback on
the usability of features of the Sacramento State website.
As a participant in this study, you will be asked to complete tasks on
Sacramento State's My Sac State and SacCT
and participate in a debriefing session.
Your participation would take approximately 1 hour.
For more information about this study, or to volunteer for this study,
please contact:
Corinne Seaton, Graduate Student
Vocational Rehabilitation Counseling
at
E-mail: xxxxx@saclink.csus.edu
This project has been reviewed and approved by the Human Subjects
Committee, Sacramento State.
42
Demographics Questionnaire
Completion of this demographics questionnaire is voluntary.
The data captured in this questionnaire is for statistical purposes only.
1. What is the highest level of education you have completed? Put an X in front of the
choice that best describes you.
____ High school graduate (includes equivalency)
____ Some college or associate's degree
____ Bachelor's degree
____ Graduate or professional degree
2. What is/are your functional limitation(s)? Write a brief explanation.
_______________________________________________________________________
_______________________________________________________________________
3. How would you describe your level of computer experience? Put an X in front of the
choice that best describes you.
____ Beginner
____ Intermediate
____ Advanced
43
4. Below is a list of assistive technology. Put an X in front of the assistive technology
that you use when accessing websites.
____ None
____ Refreshable braille display
____ Screen magnifier
____ Screen reader (e.g. JAWS)
____ Alternate input devices (e.g. trackball or touch screen)
____ Voice activated software (e.g. Dragon NaturallySpeaking)
5. How often do you use SacCT on the Sacramento State website? Put an X in front of
the choice that best describes you.
____ Never
____ 1 to 3 times per week
____ 4 to 6 times per week
____ 7 or more times per week
6. How often do you use My Sac State on the Sacramento State website? Put an X in
front of the choice that best describes you.
____ Never
____ 1 to 3 times per week
____ 4 to 6 times per week
____ 7 or more times per week
44
Consent Form
Consent to Participate in Research
You are being asked to participate in research which will be conducted by Corinne Seaton, a
graduate student in the Vocational Rehabilitation Counseling program at Sacramento State.
The study will investigate the ease of use of features on Sacramento State's website for students
with disabilities.
You will be asked to complete a demographics questionnaire. Then, I will read instructions
asking you to perform a series of tasks on Sacramento State’s website. You are encouraged to
share your thoughts aloud while performing the tasks. Your computer activity will be
videotaped and audio-taped for later analysis. After completing the tasks, you will participate
in a debriefing session with the researcher. The study should last approximately 1 hour.
You may experience anxiety or discomfort while being videotaped and audio-taped. You may
also experience discomfort when verbalizing your thoughts aloud while performing the
computer tasks. You may experience stress while completing the tasks within the allotted
amount of time.
Your performance on the tasks and your feedback will provide valuable information that will be
used to improve the ease of use of Sacramento State’s website for students with disabilities.
Individual responses captured on videotape and audiotape will not be reported. The intent is for
your responses to be analyzed by the researcher and captured as collective data. Then, they will
be stored in a secure location until they are destroyed no more than a year later. Your personal
information will be kept confidential. Your personal responses will be kept confidential to the
degree permitted by the technology used.
Your participation in this research is entirely voluntary. You may also change your mind and
stop participating in the research at any later time without any consequences, or the researcher
may decide to discontinue your participation in the study at any time.
You will not receive any compensation for participation in this study. However, your
participation will be acknowledged with a letter of thanks from the researcher. You will also
receive a nonmonetary incentive from the Accessible Technology Initiative, which will consist
of a letter of appreciation and a pad of post-it notes with the words "Accessible Technology
Initiative" on them.
If you have any questions about this research, you may contact Corinne Seaton at
xxxxx@saclink.csus.edu or the faculty sponsor, Dr. Guy Deaner, at xxxxx@csus.edu.
___________________________________
Signature of Participant
___________________________
Date
45
Orientation Script
Hello. My name is Corinne Seaton. I am a graduate student in the Vocational
Rehabilitation Counseling program at Sac State. Your participation in this project will
provide valuable feedback to improve the Sac State website, including My Sac State and
SacCT.
I want to stress that we are testing how the website works, not you. We want to know
how you use the website, so we can improve it. You cannot do anything wrong--just do
what makes the most sense to you.
We will start with a demographics questionnaire. Then, we will go through a series of
usability tasks. For each task, I will read instructions, and then you will try to complete
the task. After we finish the tasks, I will ask you to complete a satisfaction
questionnaire. Then, we will have a debriefing session.
During the test, I will be sitting in the room taking notes. Also, you might have noticed
the camera. We like to record the session because it helps us analyze issues later on,
especially for the researchers who cannot be here. The entire process will take
approximately 1 hour.
As you are doing a task, it is very helpful if you could tell me what you are thinking.
Just say whatever comes to your mind. The more that you can say during our session,
the more effective our session will be.
You can ask questions at any time. However, once we start a task, I cannot provide
answers on how to complete that task.
Do you have any questions before you begin?
Please read the Consent Form. If you agree to participate, please sign and date the form.
46
Thank You Letter
(Insert date)
Dear (insert participant name),
Thank you for participating in the usability research as part of my Master's Project in the
Sacramento State, Vocational Rehabilitation Counseling program.
Your feedback will help improve the ease of use of features on the Sacramento State
website.
The time and energy you devoted to this research project are truly appreciated.
Sincerely,
Corinne Seaton, Graduate Student
Vocational Rehabilitation Counseling
47
APPENDIX B
Usability Tasks, Script, and Questions
48
Usability Tasks
Task #1: Please login to My Sac State.
What appears on your screen under MySacState Messages?
Where would you check your email messages?
What do you find in the Library tab of My Sac State?
Where would you get help navigating in My Sac State?
How do you usually login to SacCT?
Task #2: You have been added to the Instructional Materials Accessibility SacCT
course. Please login to the course and open the syllabus.
What are other ways you would login to SacCT?
How else could you have accessed the course syllabus?
Task #3: Can you tell me what the instructor's office location is on the syllabus?
In which document types do your professors usually post their syllabi?
Which document type do you prefer? Why?
Task #4: Find the About the Course Learning Module.
How did you learn to navigate SacCT?
Where do you obtain help on issues with SacCT?
What do you expect to find in a Learning Module?
Task #5: I have created a quiz in SacCT. Please find and take the quiz.
What has been your experience with taking quizzes on SacCT?
Task #6: Can you find the Accessible Word Documents Discussion page?
49
Task #7: Can you post a new message?
Besides iMEET, what other online meeting software have your professors used to
schedule virtual office hours or online class lectures?
Task #8: Go to the Course Content page.
Task #9: Now, go to the Virtual Office Hours.
Task #10: Please open the iMEET session scheduled for today's date to view the
meeting for this course.
What do you expect to find under Course Tools?
What do you expect to find under Web Links in SacCT?
Task #11: Using the Course Tools menu, can you go to the Web Links page?
Task #12: Can you open the "Audio Tutorial - Adding Document Language to PDF"
Web Link and listen to the media?
What Course Tools are most helpful for you when navigating SacCT?
What Course Tools are confusing for you?
How are course materials on SacCT usually arranged? For example, Weekly Folder,
Learning Module, Course Content.
What arrangement of SacCT course materials is the most effective for you?
If you could have one course material or feature available at all times through SacCT,
what course material or feature would you choose?
Do you have anything else to add about your experiences with the My Sac State and
SacCT web sites?
Do you have any feedback about the Sac State webpages that you tested?
Do you have any feedback about your experience with this entire process?
Please log out.
50
Debriefing Script
Before we end the session today, I have a few questions to ask you.
Do you have any feedback about the Sac State web pages that you tested?
Do you have any feedback about your experience with this entire process?
Thank you for participating in this research project.
51
Questions
What appears on your screen under Messages?
Where would you check your email messages?
What do you find in the Library tab of My Sac State?
Where would you get help navigating in My Sac State?
How do you usually login to SacCT?
What are other ways you would log-in to SacCT?
How else could you have accessed the course syllabus?
In which document types do your professors usually post their syllabi?
Which document type do you prefer? Why?
How did you learn to navigate SacCT?
Where do you obtain help on issues with SacCT?
What do you expect to find in a Learning Module?
What has been your experience with taking quizzes on SacCT?
What has been your experience with posting and reading messages on Discussion pages?
Besides iMEET, what other online meeting software have your professors used to
schedule virtual office hours or online class lectures?
What do you expect to find under Course Tools?
What do you expect to find under Web Links in SacCT?
What Course Tools are most helpful for you when navigating SacCT?
What Course Tools are confusing for you?
52
How are course materials on SacCT usually arranged? For example, Weekly Folder,
Learning Module, Course Content.
What arrangement of SacCT course materials is the most effective for you?
If you could have one course material or feature available at all times through SacCT,
what course material or feature would you choose?
Do you have anything else to add about your experiences with the My Sac State and
SacCT web sites?
Do you have any feedback about the Sac State webpages that you tested?
Do you have any feedback about your experience with this entire process?
53
APPENDIX C
Participant Responses
54
Participant Responses to Questions
What appears on your screen under Messages?
Participant 001: You got all these.
Participant 002: Nothing.
Participant 003: All messages. Zero unopened. Four total.
Participant 004: No messages.
Participant 005: Nothing.
Participant 006: Here is something for me. But, I looked at that earlier.
Participant 007: My messages, My SacCT, and then the account.
Participant 008: 8 unopened messages that I have not looked at.
Participant 009: 0 unopened, 8 total. You have no messages.
Participant 010: My inbox, student fees, but I usually go to the Student Center.
Where would you check your email messages?
Participant 001: For Sac State, I rarely go here...at all. Whenever they send me
something here, it also goes to my hotmail.
Participant 002: Just click on the Inbox for SacLink E-mail.
Participant 003: I would down below that and click on Inbox.
Participant 004: Right below that in my SacLink e-mail.
Participant 005: SacLink E-mail.
Participant 006: You know what, I haven't figured that out.
Participant 007: Under my inbox.
55
Participant 008: Right here (mouse arrow is on Inbox)...I would check them right here
on the Inbox...that I don't read really, at all.
Participant 009: Right here...which I have a lot.
Participant 010: Usually, I go back and forth with my inbox or the Student Center or my
hotmail account.
What do you find in the Library tab of My Sac State?
Participant 001: They don't ask us to go there very often. Usually, they will setup a link
to the article that they want us to read from the library.
Participant 002: You find your account...search engine when you want to search for
articles, journals, or book that you're trying to do research for. And, tutorials, guides.
Participant 003: An image, nice sunset. And then just read...whatever grabs my eye.
Participant 004: What I owe, if I owe anything in my account. To look up books. Help
to use the library. The times the library's open. Services they had.
Participant 005: Nothing in my account.
Participant 006: Well, I found out that I don't owe any money. You know what, I
haven't explored it that much, so I can just go to the library from here and type in
searches.
Participant 007: There's a library research, tutorials, and then, the library website.
Participant 008: I had an English class that I had to go through the library to go through
there...all my stuff and advance searching and stuff.
Participant 009: I haven't been there before. Nothing checked out, no money owed.
Participant 010: My account...zero. Library research. I can look up books, articles,
journals. Their hours, their services, announcement.
Where would you get help navigating in My Sac State?
Participant 001: Sometimes the phone number is right on here [on the Sac State
homepage]. I'll call up the help desk. And they will usually help me find what I want
to find. I don't see the number here now, but it's usually the best way to go if you can't
find anything.
56
Participant 002: N/A
Participant 003: I would just look around and see what I needed to do. If I needed help,
I would call that number. I have it written down at home.
Participant 004: The help button at the top left.
Participant 005: You would go all the way to the bottom, and if it says, "Need help?"
You click on "Need help?", and call 'em.
Participant 006: I thought they had like a help section.
Participant 007: They have a...help desk.
Participant 008: I kind of just played around with it. The first year, I asked people what
to do, but now I kinda know where I need to go. Like where I need to get my classes, if
I need to do something or talk to someone.
Participant 009: I'd click on a link. I don't know if there is help, except for maybe
SacCT. That's the only thing I could think of.
Participant 010: Well, I go to...their website where they have this empty box. Just type
in that for help.
How do you usually login to SacCT?
Participant 001: Go home, Sac, Login. Now, that's redundant right there, isn't it?
Because I have to login again.
Participant 002: I just click on My SacCT on this one [Online Courses (SacCT) link on
Sacramento State homepage], I login to My Sac State. But, other times, I just go to new
homepage, and then I just click on this and login, which is pretty a drag sometimes.
Participant 003: I have a bookmark on my computer, so I click that and then I login.
Participant 004: I have never logged into SacCT. I don't even know where it is.
Participant 005: Where it says login to SacCT, and you just login.
Participant 006: I would just type in "Sac State" and then, it would usually take me to
the homepage. From the main homepage, I would just login under here...under My Sac
State on the main Sac State page.
57
Participant 007: I would go under Sac State home, and then...I just learned how to do it.
Yeah, you go under...oh yeah, Student Center. Is that it? Wait, no, that's not it.
Participant 008: Well, on my laptop, I use it as my homepage.
Participant 009: I usually click it. I'm very visual.
Participant 010: With my sac account and my password.
What are other ways you would log-in to SacCT?
Participant 001: I just do it the same way every time.
Participant 002: N/A
Participant 003: That's it..for me.
Participant 004: I have no idea. I listened to the tutorial and I got me to this.
Participant 005: There's other ways? There's no other way that I know of.
Participant 006: Just typing in "SacCT" like on the web browser, and it comes up
automatically.
Participant 007: I don't know. There probably are easier ways.
Participant 008: Probably just going through My Sac too...'cause my thing pops up on
My Sac too.
Participant 009: That's the only way I know to login.
Participant 010: I don't know any other ways to login to SacCT.
How else could you have accessed the course syllabus?
Participant 001: Over here (participant places mouse arrow on Syllabus link in Course
Tools menu.)
Participant 002: Just click on this tab (participant places mouse arrow on Syllabus link in
Course Tools menu.)
Participant 003: I don't know.
58
Participant 004: I don't know.
Participant 005: You could go back to Course Content, where it says Syllabus in Other
Formats, and it shows you two different versions.
Participant 006: If it wasn't on here, I would go to the little side browser thing.
Participant 007: N/A
Participant 008: This is probably like the only way I probably like to. I like doing it this
way.
Participant 009: It's either here or it's usually in Course Content. It's the first item
usually, and it's also listed down here. I don't know why they have duplication, but it is
three times on the page.
Participant 010: I could go research teacher's name, then look up their website.
In which document types do your professors usually post their syllabi?
Participant 001: They vary, but they're usually copyable and pasteable in other places.
Participant 002: Word document or PDF.
Participant 003: I don't know.
Participant 004: It looks like a Word.
Participant 005: I believe there's two different ones. You have Word version or the PDF
version.
Participant 006: Word, PowerPoint or Adobe PDF.
Participant 007: Mostly PDF.
Participant 008: They do a little different. This is not like...this not like what they
usually do. They just print it out in Word document. And they do the little download
that comes down here, do the little click File Download.
Participant 009: It's in Excel, it can be in a Word, it can be in a PDF.
Participant 010: PDF.
59
Which document type do you prefer? Why?
Participant 001: As long as they copy and paste, they are okay with me.
Participant 002: I prefer either or as long as I get like the material itself.
Participant 003: I don't know. I know things say PDF. I think they do it in Word
actually.
Participant 004: Word's fine because it's what I use at home.
Participant 005: Definitely Word version is the worst one 'cause it kicks you out and you
have to go back in again. I like the fact that you have the one where it says regular
syllabus. You could just click in and it pops us. But, the other one with the syllabus,
PDF version, it's also good. So, you have the regular first Course Content one
where...already has the syllabus. So, that one's really good too because it doesn't kick
you out.
Participant 006: I guess Adobe...because it's just easier. It has fewer icons than Word,
but it has enough so I can maneuver through it, and you can scroll a lot easier than a
Word document. And then, PowerPoint, sometimes it's hard to print out, and you have
to always adjust the slides.
Participant 007: Word. It helps me write better.
Participant 008: What I've been growing up with is the paper one, so it's like I'm used to
it now. I like this too...this looks a little more cleaner on this website. Sometimes you
have to download every file, and then you go back, and then when you download it, it'll
start you back right here. So, you have to back into what you just downloaded and find
it again. So...I like it when it just pops up and you don't have to download it.
Participant 009: I prefer Word, but with my disability, if he puts it in PDF, I can usually
go... I have to bring it in to format it. I can usually click behind it and get the outline.
Unless I can talk him into sending it to me.
Participant 010: PDF. It's easier, it's reasonable, it's updated version.
How did you learn to navigate SacCT?
Participant 001: Just years of practice.
Participant 002: I just played around with it.
60
Participant 003: From American River College. Their online thing - D2L and
Blackboard.
Participant 004: Well, I opened up two pages. I guess...it works like any website.
Participant 005: I've had it from a different class...at a different college.
Participant 006: I just figured it out on my own.
Participant 007: From my instructor.
Participant 008: Just practice and questions pretty much. Just asking the teacher.
Usually the teacher will show, most teachers will show the first week how to navigate.
So the teacher has to like show how to work it or the freshman is not going to get it. I
mean it's good update for me too because I didn't use it all summer.
Participant 009: Trial and error.
Participant 010: I don't know. I usually don't really navigate that much.
Where do you obtain help on issues with SacCT?
Participant 001: This year, they really encouraged me to go over here to ARC or upstairs
there. Either place, they say, will know what to do. So, I feel confident about that. I
haven't had to go yet.
Participant 002: I usually just ask the professor, like e-mail them and ask them or I just
e-mail the people, the technician and all that.
Participant 003: I think I have it written down at home...who I would call. I would go to
the instructor probably...ask the instructor. Sometimes they don't know. So, a fellow
student.
Participant 004: I have no idea.
Participant 005: You go to the help desk, which there should be a number. Either it
would be in the syllabus. If not in the syllabus, it would be in Course Content. If it's not
there, it should be somewhere on the site when you login.
Participant 006: There is a little help icon here, and I might have used it one time. But,
usually, I will ask a friend if they are in the room.
Participant 007: They have a help...information.
61
Participant 008: I really don't use it for really help for me. I use the library if I need help
or like I talk to someone about it. Or ask one of my friends. Or talk to someone in the
class that did it already.
Participant 009: I either contact the teacher directly through mail or contact the help
desk, which has been very helpful to me.
Participant 010: Like if I have any issues, I ask people.
What do you expect to find in a Learning Module?
Participant 001: What I've been finding this time around is assignments, things that I
need to be done next.
Participant 002: An overview of the course itself.
Participant 003: I haven't taken an online class here. The assignment, what to do...and
guidance as to the structure and format of the class and what's expected of me.
Participant 004: Homework assignments, quizzes, maybe the notes the teacher posted.
Participant 005: It tells you what you are learning for the week.
Participant 006: It's just activities and material resources probably to teach you how to
use this site.
Participant 007: Information pertaining to different...materials about your classes.
Participant 008: Probably papers that we need to learn on. Some of my classes have
quizzes right after Learning Modules, so quizzes probably. And just tips on that chapter,
study guides pretty much, like short briefs, what was going on in class that day.
Participant 009: I would expect to find the order of things, what date they're due, that
sort of thing. It's kind of an overview so you don't miss something.
Participant 010: Like how to do certain little things...how to format papers or put them,
you know, together.
What has been your experience with taking quizzes on SacCT?
Participant 001: One of them has been, I don't see it, and it's due tonight, midnight. And
nobody's home. That's what usually happens.
62
Participant 002: I've taken a lot of quizzes on SacCT. But, I think it's pretty
convenient...when taking a quiz because most of the questions are on multiple choices.
Participant 003: I haven't.
Participant 004: It's easy.
Participant 005: I've had several. Quizzes-wise, I wish that there's more time because
some teachers don't give you enough time to finish a quiz. I think the maximum is 30
minutes, and when it's a test, it's 2 hours. So I wish professors would give out more.
Participant 006: As long as they're not timed, I think they're pretty good. The only
problem is if you have so many classes, it's kind of hard to remember when you have a
quiz. You have to kind of keep up on it. But some classes are good...they have the
calendar, so it will pop up when you have a quiz coming up.
Participant 007: I never had an experience.
Participant 008: I like 'em online. They are not going to be popped up on you. Quizzes
or any test, when you go up to the front of the website, like right here, tells you
anything.
Participant 009: Some of them time out. This semester, I've liked it a little better. I've
one teacher...he has a test every two to four days, and...what's nice about it is he gives
two attempts and he allows you to go back and find the correct answers, so it kind of
corrects you. I like that. So, I have to admit though if having a class that's more than
50% online sucks...for someone with a disability.
Participant 010: Well, since I have the double time, so it's a lot easier at the same time.
But, it's like a time thing, so you have to watch your time, and I don't like that.
What has been your experience with posting and reading messages on Discussion
pages?
Participant 001: I had a class with 50, 60 people. Everybody posted to the discussion
board, and then you were supposed to go read their post and reply to 'em. And, just like
an article I read about that very topic, it was overwhelming. Just too much information
coming from all over the place. But, now they are starting to put people in groups of 5,
and you know, they're simplifying it more. So, I think they're getting better at using it
now.
Participant 002: It's pretty easy to post or respond to whatever's been posted.
63
Participant 003: I haven't done it here. But, it's okay. I can do it...I can navigate it.
Actually, I started a class last semester, but I had to drop it because it was too confusing.
Participant 004: Works like any blog poster.
Participant 005: I love them! They're amazing because you communicate back and forth
with other students. If you can't have interaction in the classroom, that's probably the
best way to have interaction with other students.
Participant 006: I usually don't do it that often. I read 'em all the time, but I don't usually
post.
Participant 007: I never did that before.
Participant 008: I think it's really good. We have our discussions, and then we have our
groups. So, I'm in this group, this is only for my group, only, that I can read privately
discussions. And then, this is for the whole class (referring to Group Discussion Area).
So I can see what everyone else is talking about, but then if I need to do our group
project, only these kids only see it. Then, we can send our papers, essays, PowerPoints,
what we need to work on, if we had a group project argument we need to work on.
Participant 009: I like it. Once in a while, there'll be a banter between two people.
One'll say, "I can't find it." And the other will say, "Well, check here." And it kind of
brings the whole class up to date. Because if one had the problem, probably 75% had
the problem.
Participant 010: I don't like it 'cause, well, one way you can connect the teacher faster or
your peers, but they usually don't write back, so it's struggling to meet other people the
computer...they're doing everything on their computer. 'Cause one person will be in
other place and don't have computer access to it. That's disadvantage, like group work.
But, it's easier way to just contact the teacher.
Besides iMEET, what other online meeting software have your professors used to
schedule virtual office hours or online class lectures?
Participant 001: Chat. I used chat the other day with a teacher.
Participant 002: This is the first time. I'm pretty new to this. I haven't had any
professors that had that kind of program...virtual office hours.
Participant 003: I do e-mails to the professor.
Participant 004: I've never used any.
64
Participant 005: Nothing.
Participant 006: They usually just like do an announcement or send you an e-mail. I've
never used this before.
Participant 007: I haven't had any.
Participant 008: My math class does the Math Lab...Pearson Math Lab. It's pretty nice.
I like it.
Participant 009: One of my teachers uses something called Page Out, and he's actually
sent us in our e-mail because he doesn't give homework back. But, he actually sends us
the page, so we can see the corrected...which is really nice.
Participant 010: We don't do that.
What do you expect to find under Course Tools?
Participant 001: Is that up there? I don't see it. Oh, calendar, probably.
Participant 002: All these different tabs right here...Announcements, Assessments,
Assignments.
Participant 003: Things which will help me in the course.
Participant 004: I don't know where Course Tools is.
Participant 005: You can find Course Content, Announcements, Assessments,
Assignments, Chat, Discussions, Learning Modules, Mail, Search, which would lead
into the library. Syllabus, which is another way that you could find your course content.
Web Links, Who's Online from your class, so maybe if you need help you can find out if
maybe they might know something that you don't know. And you could find your
grades, your files of everything that you've done, and then your progress of how many
times you've been on and how many hours you've worked throughout the whole week.
Shows you how many times you viewed a message, posted a message, and assessments
that you began and you finished, total time and entry viewed, and collection viewed.
And you can also export it, and then you also have your notes.
Participant 006: My Grades, My Files, My Progress, Notes. I've only looked at my
grades and my progress. I've looked at my files and the notes part, but I didn't really
know what it was for.
Participant 007: All of this. My Grades and stuff.
65
Participant 008: Probably just helping with the course, like websites need to go to,
maybe, all that stuff. I mean, just like, help during the whole course and those are tools
you need to pass the class.
Participant 009: Probably icons, visually, so you can see where to find things, kind of an
overview.
Participant 010: Well, all these...Announcements, Assessments. I usually look up my
grades, but I usually look for my assignments and my assessments. Sometimes
announcement are important too. Those are three I use mostly.
What do you expect to find under Web Links in SacCT?
Participant 001: Classes.
Participant 002: N/A
Participant 003: I can imagine that that's different links...like support links for the class.
Participant 004: Links the teacher wants us to use for our class.
Participant 005: You will find different links that you would be linked to.
Participant 006: Where the professor might have like have an online class...where the
links are for the online video of the teacher.
Participant 007: Different websites.
Participant 008: Websites to go to that allow the teacher...that likes you to go get
information from that so she knows that you're not plagiarizing
Participant 009: Under Web Links, I would expect to see extra reading material to enrich
our learning. Sites to visit.
Participant 010: Sometimes I go in there all I see...not really useful. Like it's just some
extra stuff that I need to look at that aren't really that important.
What Course Tools are most helpful for you when navigating SacCT?
Participant 001: I really don't use [Course Tools]. I use the calendar. I might be able to
like the chat room for office hours, they have virtual office hours. I may get to like that,
but it took us a while to get online the other day 'cause the browser didn't have Java.
66
Participant 002: I'd say the Course Content, the most, because it has like everything, like
the syllabus.
Participant 003: When the instructor says, "Call me anytime or e-mail me anytime."
And meeting with the instructor...that's what's most helpful to me.
Participant 004: Probably the Course Content, the Assignments, and the Assessments.
Participant 005: A little bit of everything.
Participant 006: I don't know. I guess all the icons. I usually just use like the icons.
Participant 007: I haven't really played around a lot with it, so I'm not really sure. All I
do is check...my instructor has notes. I check that, but other than that, I'm really not in
it.
Participant 008: Checking my grades, everything. Pretty much everything right here I
need, especially Assignments, Learning Modules, Assignments, Assessments,
Announcements, Sessions...pretty much all of them. I like them all.
Participant 009: The little home, Course Content. It's the most helpful.
Participant 010: Like I mentioned, the announcement, assessment, and the assignment.
Sometimes like e-mail to my professor, but I usually look for my grades.
What Course Tools are confusing for you?
Participant 001: iMEET was rough, at first. In fact, no, iMEET is still rough...it's very
rough. Getting together with the whole class in iMEET. And voice, you can talk,
everybody can talk at once. So just because of that. Understanding who is to speak and
who is to be quiet. Things like that has taken some time. 'Cause people don't know. "Is
anybody there? Does anybody hear me?" You know, could be right in the middle of a
lecture, they'll say something like that.
Participant 002: I think nothing's confusing.
Participant 003: N/A
Participant 004: I've never used any of 'em, so I wouldn't know.
Participant 005: Actually, they don't seem confusing at all. They're all really easy,
simple...just as long as you, since like when you go over with the mouse, they have the
side where the tools are, there's a little tab that tells you where it is. If you don't know
67
what you are doing, you learn it because there's a tab which will help students with
learning disabilities or anything like that. It shows them where certain things are...it's
easy to access.
Participant 006: Anything that that's not labeled.
Participant 007: I would say...trying to get into SacCT.
Participant 008: All of them are pretty well known now. I use pretty much them on a
weekly basis.
Participant 009: I don't know about this course that you're showing me. But, it's very
confusing to have several different things on several different pages. It would be so
much easier to just see an outline, you know, where you go down rather than having to
flip through your binder every time. It's really confusing.
Participant 010: Like the Web Links. I don't care who's online really because I don't
really chat with them. That shouldn't be on there. Like the search, I don't think that
should be on there. Learning Modules...don't really quite need it. Discussions, like I
said, it's hard to discuss people or even chat with people. But, if I were to e-mail, like I
said, to my professor, would be faster to connect with him.
How are course materials on SacCT usually arranged? For example, Weekly
Folder, Learning Module, Course Content.
Participant 001: They're different, each teacher is different.
Participant 002: Alphabetical order.
Participant 003: Well, the Course Content is there. Weekly Folder...it's there and you
can access it. And if there's a deadline, then that previous one can't be accessed after
that deadline.
Participant 004: Probably on the day they're due.
Participant 005: It should all be in Course Content.
Participant 006: For mine, I usually don't have Learning Modules on here. It's usually
really basic, like Announcements, Assessments, Assignments. They have a chat. The
discussion board. The mail. Do we have a search? I don't know. Syllabus. I guess
they are mostly usually arranged in the icons.
Participant 007: I'm not sure.
68
Participant 008: They're really arranged in a nice orderly way. I think now they have a
nice program setup. I used to have a program setup in high school, and it was just
horrible. This is way, way better...way easier. I like the Learning Modules.
Participant 009: When you click this (referring to Course Content icon), there's like
several different categories, there's no icon. And, you'd have to have your open sheet to
follow it. It's really hard that way.
Participant 010: I think the Course Content, mostly. Course Content shows, I think, all
the stuff that you need.
What arrangement of SacCT course materials is the most effective for you?
Participant 001: Week 5 up here at the top. This is the one we're going to do now.
Course Resources...General Questions...Virtual Office Hours...Previous week course
work. So, that's pretty clear. That one, that one works.
Participant 002: I think the Course Content.
Participant 003: Weekly Folder is good for me.
Participant 004: The day they're due.
Participant 005: I would have to say Messages, the message board...because you get to
communicate more with the professor.
Participant 006: You know what, if it's labeled with a icon and then it tells what it
is...like when you roll over. What would help me if these were labeled to where I know
what they're already saying in the basic terms.
Participant 007: The syllabus because that's where all the work is that I have to do.
Participant 008: Probably the assignments because you can do those stuff a lot. I just
like having the quizzes on there too because the quizzes you can do when it fits your
time.
Participant 009: The one with the icon, the little explanation. This is very, very well
done, very well done. This has got an order. You can just follow it down. Probably, it
should probably have a box up here with a checklist so you don't have to keep
rethinking, "Did I do that or did I not do that?"
69
Participant 010: I don't use that. I use Course Content, but that's it.
If you could have one course material or feature available at all times through
SacCT, what course material or feature would you choose?
Participant 001: Probably iMEET, so you go in there and scream.
Participant 002: I'd say the Course Content and the Grades.
Participant 003: Having someone available right when I need them.
Participant 004: I don't know. I haven't used it.
Participant 005: I'd have to say the iMEET because of the fact that I'm the type of person
who likes learning from the professor himself or herself. Then, I could see the person
and what they're doing and how they could teach it to me through that.
Participant 006: I guess those Learning Modules...because then it shows you kind of
steps. If you were able to do the Modules and then do it at the same time, for me that
would probably be the most helpful.
Participant 007: If they had like a video something pertaining to that class or something
like that.
Participant 008: Discussions too...because you can always talk to people. You need to
always talk to know what's going on, you might have questions. You might not know
anyone in the class, you can just throw something out there, someone will answer back
in the next half hour.
Participant 009: The ability to turn something in where it, like Kurzweil, that it reads to
you, that would be it. And, a lot of the teachers, they're scared to death somebody's
going to change their stuff. So you have to embarrass yourself, go up and ask them,
"Can you send me that?" Because by the time you look at it, and copy it, it takes
volumes of time. They have no idea what we go through trying to learning this. And if
I could just click something, and put my headset on, and have it read to me, that would
be so much better.
Participant 010: Just my Course Content.
70
Do you have anything else to add about your experiences with the My Sac State
and SacCT web sites?
Participant 001: Well, like she [the professor] said about Explorer, Mozilla...use Mozilla
instead of Explorer. Stuff like that drives me nuts, just drives me bats. But, this
semester hasn't been too bad yet, so they must be improving things as we go along.
Participant 002: No.
Participant 003: I wish that when the school is e-mailing me to SacLink, they would also
do it my personal e-mail.
Participant 004: Sometimes My Sac State, when you go to the Student Center,
sometimes you always have to go to, like, if you click into my Schedule, you always
have to go back this way, instead of the back button. Seems a little...like extra work.
And sometimes it's hard to find things from My Sac State.
Participant 005: I don't like the fact how they can kick you off sometimes...like if they
are trying to do maintenance.
Participant 006: I don't think I've had a tutorial on SacCT that tells you, like, where to
find things and step by step how to use stuff.
Participant 007: I don't know. It's pretty good. I know a lot of people use it.
Participant 008: Just overall, I just like having it there because I can go on at anytime
and...I can see my progress at any time.
Participant 009: I don't think I'd ever want to take a class that is totally online again.
Totally, you know, where you don't see the teacher at all...because I actually learn from
hearing the give and take from the other students.
Participant 010: I hate it because there's two different links I have to go connect like I
have to go login My Sac State home or go to my Student Center or go to my teacher
account, which is WebCT. I just think that this should all open up together, so that way
I don't open up again and retyping my password, and just issue like that. 'Cause like I
don't need all this...Sac State News or academic calendar.
Do you have any feedback about the Sac State webpages that you tested?
Participant 001: Well, I would have to run them through Kurzweil to get an
understanding of what exactly it was that you were showing me there. So, it would
mean copying and pasting and hearing what that's all about there. So, even though you
71
ran me through two or three different links, I still don't know why we went there, why
we went there at all.
Participant 002: I think it's pretty good, but I think I navigate pretty well with it.
Participant 003: No, it just always sounds interesting. Everything, how they put it,
sounds interesting. And, I do wish that I could be more knowledgeable and expert at
navigating. And, not so...so confused.
Participant 004: It seemed to run good. SacCT looks like it works pretty good.
Participant 005: No, everything seems really great from what you did. I like the fact that
there's iMEET on there. I wish that regular professors would have that.
Participant 006: I've never seen that [learning modules] before. I think that would help
out a lot.
Participant 007: Sac State is a lot easier to bring up than SacCT.
Participant 008: I love it.
Participant 009: That was the best I'd ever seen. Whoever designed this they could
design for my teacher and I would love it.
Participant 010: No.
Do you have any feedback about your experience with this entire process?
Participant 001: N/A
Participant 002: It's pretty quick...easy.
Participant 003: I appreciate that...you're doing what you need to do for your program.
And that they're looking into making it more friendly...that's good. Here's another
thing...I think the colors could be better...could be more inviting. I realize that they're
school colors...but, there's other colors that are more attractive.
Participant 004: It was easy.
Participant 005: Pretty amazing! I like the way it's setup. This is completely different
from what we have at school, I've seen and what I've used. There's not iMEET on there,
and I would think that would be a lot better to have for all students. The syllabus is
amazing too, how it doesn't kick you out 'cause usually when you have a regular file that
72
when you enter into the syllabus, you have to download it. I don't like the fact that you
have to download it and it kicks you out and you have to go back in.
Participant 006: I don't like the camera, and I noticed it right when I walked in. But, if it
was hidden more and I didn't notice it, I probably wouldn't have had a problem with it.
Participant 007: I have to really focus on that....trying to learn everything on SacCT.
Sometimes I can get in it really fast. But, like today, I don't know why I had a hard time.
Participant 008: Just the...the server maintenance sucks on Tuesday mornings.
Participant 009: It's quite good, especially this year. Since we don't have keys to the
room any more we are forced out on floor with the other people and I've noticed out of
my many friends here, I'm the only one that sits in the ADA identified desk. My other
friends will not sit there and identify themselves to people that they will future network
with. We're like a family, and so if you see somebody looking at you, you know, and
they knock on the door, and say, "Can I have ten minutes of your two hours?" We'll get
up and do it. It's rather frustrating that the disabled are pitted against the regular. The
ADA signs are removed, the guy sitting next to you, he's rocking out with his headphone
on. He's obviously not doing his homework. The scanners are what we really need to
learn. With Kurzweil, you have to scan in and you have to format it. And maybe you
have to format it again to take it home, to put into a Word document, to do your
homework. It's a lengthy process but it's helped my grades.
Participant 010: There's just too much stuff on the website. And you don't really need. I
don't really need these links. I don't really need this...Student fee announcement 'cause
they tell you again when you go to the Student Center...when you have like...the
payment plan. Like if I went to WebCT again, I have to sign up again and login, which I
don't really like that either. And like right here, this is a waste...news, video, this
calendar thing. I just think that it's just a waste of time putting on there.
73
Table 1
Sample of Usability Tasks
Task number
1
2
3
4
5
6
7
8
9
10
11
12
Completed with ease
Completed with
difficulty
Failed to complete
74
APPENDIX D
Demographics and Results
75
Table 2
Participant Demographics
Participant
Education
Functional
limitation
Computer
experience
Assistive
technology
SacCT use
per week
My Sac State
use per week
1
B.S.
LD
Advanced
3
7+
Never
2
H.S.
Mobility
Intermediate
2
7+
7+
3
Some college
Mobility
Beginner
0
1 to 3
1 to 3
4
Some college
LD
Intermediate
0
Never
1 to 3
5
Some college
LD
Intermediate
0
7+
7+
6
Some college
LD
Intermediate
2
7+
1 to 3
7
Some college
Mobility
Intermediate
0
1 to 3
1 to 3
8
Some college
LD
Intermediate
1
7+
7+
9
Some college
LD
Advanced
1
7+
4 to 6
10
Some college
LD
Intermediate
1
1 to 3
1 to 3
Participant: Each number is used to uniquely identify the participant.
Education: B.S. = Bachelor’s degree; H.S. = High school graduate (includes equivalency); Some college =
Some college or associate’s degree.
Functional limitation: LD = Learning disability; Mobility = Mobility impairment.
Computer experience: Participant’s level of computer experience.
Assistive technology: The number of assistive technology devices the participant uses when accessing
websites.
SacCT use per week: The number of times the participant uses SacCT on the Sacramento State website.
My Sac State: The number of times the participant uses My Sac State on the Sacramento State website.
76
Table 3
Results of Usability Tasks
Task number
Completed with ease
Completed with
difficulty
Failed to complete
1
90
10
0
2
50
30
20
3
55.56
22.22
22.22
4
70
20
10
5
70
0
30
6
60
30
10
7
80
0
20
8
100
0
0
9
100
0
0
10
37.50
37.50
25
11
80
0
20
12
60
0
40
%
%
%
Note. Due to technical difficulties, 9 out of 10 participants were given task #3 to complete, and 8 out of 10
participants were given task #10 to complete.
77
REFERENCES
Accessible Technology Initiative. (2010). ATI annual report AY 09/10 instructions.
Retrieved October 6, 2010, from
http://ati.calstate.edu/file.php/1/Annual_Reports_AY_09-10/Instructions_for_
ATI_Annual_Report_AY_09_Final_v1.1_1_.docx
Adam, A., & Kreps, D. (2006). Enabling or disabling technology? A critical approach
to web accessibility. Information Technology & People, 19(3), 203-218.
doi:10.1108/09593840610689822
Berners-Lee, T. (1998). The TITLE element in HTML. Retrieved July 27, 2009, from
http://www.w3/org/Provider/Style/TITLE.html
Biondi, D. (2009). My Sac State 2.0. Information Technology Insights: The newsletter
of information resources & technology (1). Retrieved July 29, 2009, from
http://www.csus.edu/irt/newsletter/issue/1/10.stm
Bowman, C. A., & Jaeger, P. T. (2005). Understanding disability. Westport, CT:
Praeger.
Brophy, P., & Craven, J. (2007). Web accessibility. Library and Information Services
for Visually Impaired People, 55(4), 950-972.
California State University, Northridge [CSUN]. (2010, July 9). How to read an Acc
monitor report: Guide for web accessibility checking tools--California Sta.
Retrieved August 19, 2010, from
http://www.csun.edu/accessibility/training/materials/acccheck/1.html
78
California State University, Sacramento [CSUS]. (2009). SacCT: Introduction to
SacCT. Retrieved July 29, 2009, from
http://www.csus.edu/webct/faculty/intro.pdf
California State University, Sacramento [CSUS]. (2010a). Accessible Technology
Initiative at Sac State. Retrieved August 19, 2010, from
http://www.csus.edu/accessibility/ati.html
California State University, Sacramento [CSUS]. (2010b). First year ATI web
accessibility project workflow guide, Accessible Technology Initiative: Cali.
Retrieved August 19, 2010, from
http://www.calstate.edu/Accessibility/webaccessibility/first-year-workflow.shtml
California State University, Sacramento [CSUS]. (2010c). First year web report.
Retrieved August 19, 2010
fromhttp://www.csus.edu/accessibility/firstyearreport/index.htm
California State University, Sacramento [CSUS]. (2010d). First year web report-Campus-wide trends. Retrieved August 19, 2010, from
http://www.csus.edu/accessibility/firstyearreport/campuswidetrends.htm
California State University, Sacramento [CSUS]. (2010e). First year web report-Cross section. Retrieved August 20, 2010, from
http://www.csus.edu/accessibility/firstyearreport/crosssection.htm
79
Echeverria, J., & Quillian, B. F. (2009, September 21). Revised coded memo-automatic extension of Accessible Technology Initiative requirements.
Retrieved July 12, 2010, from
http://www.calstate.edu/AcadAff/codedmemos/AA-2009-19.pdf
Evans, J. J. (2003). Web design complete course. Hoboken, NJ: Wiley.
Federici, S., Micangeli, A., Ruspantini, I., Borgianni, S., Corradi, F., Pasqualotto, E.,
et al. (2005). Checking an integrated model of web accessibility and usability
evaluation for disabled people. Disability and Rehabilitation, 27(13), 781-790.
doi:10.1080/09638280400014766
Hackett, S., Parmanto, B., & Zeng, X. (2005). A retrospective look at website
accessibility over time. Behaviour & Information Technology, 24(6), 407-417.
doi:10.1080/01449290500066661
Harper, K. A., & DeWaters, J. (2008). A quest for website accessibility in higher
education institutions. Internet and Higher Education, 11, 160-164.
doi:10.1016/j.iheduc.2008.06.007
Henry, S. L. (2007). Usability testing | Accessibility in user-centered design | Just ask:
Integrating accessibility. Retrieved September 22, 2009, from
http://www.uiaccess.com/accessucd/ut_plan.html#characteristic
Henry, S. L. (2008). WCAG overview. (2009.) Retrieved July 29, 2009, from
http://www.w3.org/WAI/intro/wcag.php
80
International Center for Disability Resources of the Internet. (2008, February 13).
Bobby web accessibility checker and online products will no longer be publicly
available as of February 1, 2008. Retrieved July 29, 2009, from
http://www.icdri.org/News/Bobby021208.htm
IT Accessibility & Workforce Division [ITAW]. (2009). Section 508: The road to
accessibility. Retrieved July 29, 2009, from http://www.section508.gov/
Jewett, T. (2007). Accessible Technology Initiative: California State University, Office
of the Chancellor. Retrieved August 20, 2010, from
http://www.calstate.edu/Accessibility/webaccessibility/evaluation/index.shtml
Kaplan, D. (2010). Frequently asked questions about Section 508: Accessible
Technology Initiative: CSU system. Retrieved September 19, 2010, from
http://www.calstate.edu/accessibility/section508/section_508_FAQs.shtml
King, T. W. (1999). Assistive technology essential human factors. Boston: Allyn and
Bacon.
Klein, D., Myhill, W., Hansen, L., Asby, G., Michaelson, S., & Blanck, P. (2003).
Electronic doors to education: Study of high school website accessibility in Iowa.
Behavioral Sciences and the Law, 21, 27-49. doi:10.1002/bsl.521
Leuthold, S., Bargas-Avila, J. A., & Opwis, K. (2007). Beyond web content
accessibility guidelines: Design of enhanced text user interfaces for blind internet
users. International Journal of Human-Computer Studies, 66, 257-270.
doi:10.1016/j.ijhcs.2007.10.006
81
Murphy, E., Kuber, R., McAllister, G., Strain, P., & Yu, W. (2007). An empirical
investigation into the difficulties experienced by visually impaired Internet users.
University Access in the Information Society, 7¸ 79-91.
Nielsen, J. (1993). Usability engineering. New York: AP Professional.
Nielsen, J. (2000, March 19). Usability testing with 5 users (Jakob Nielsen's Alertbox).
Retrieved September 22, 2009, from
http://www.useit.com/alertbox/20000319.html
Pearrow, M. (2000). Web site usability handbook. Rockland, MA: Charles River
Media.
Quillian, B. F. (2010, June 14). Revision of Accessible Technology Initiative coded
memo. Retrieved September 22, 2010, from
http://www.calstate.edu/AcadAff/codedmemos/AA-2010-13.pdf
Reed, C. B. (2004, December 20). Executive Order 926. Retrieved July 29, 2009,
from http://www.calstate.edu/eo/EO-926.html
Reichard, G. W., & West, R. P. (2007, February 9). Access to electronic and
information technology for persons with disabilities. Retrieved September 22,
2009, from http://www.calstate.edu/AcadAff/codedmemos/AA-2007-04.pdf
Seale, J. K. (2006). E-learning and disability in higher education. New York:
Routledge.
TechSmith. (2009). Screen capture, screen recorder, video hosting, and usability
testing software. Retrieved September 22, 2010, from
http://www.techsmith.com/
82
TechSmith. (2010). Usability testing basics: An overview. Retrieved September 22,
2010, from
http://download.techsmith.com/morae/docs/customer-research/Usability-TestingBasics.pdf
United States Access Board. [USAB]. (2010a). Electronic and information technology
accessibility standards (Section 508). Retrieved October 6, 2010, from
http://www.access-board.gov/sec508/standards.htm
United States Access Board [USAB]. (2010b). Summary of final rule. Retrieved
October 6, 2010, from http://www.access-board.gov/sec508/summary.htm
U.S. Department of Justice [DOJ]. (2005, September). A guide to disability rights laws.
Retrieved August 29, 2009, from http://www.ada.gov/cguide.htm#anchor65610
Waddell, C. D. (1999). The growing digital divide in access for people with
disabilities: Overcoming barriers to participation in the digital economy.
Retrieved October 24, 2010, from
http://www.icdri.org/CynthiaW/the_digital_divide.htm
Wattenberg, T. (2004). Beyond legal compliance: Communities of advocacy that
support accessible online learning. Internet and Higher Education, 7, 123-139.
doi:10.1016/j.iheduc.2004.03.002
WebAIM. (2010a). WebAIM: Visual disabilities–blindness. Retrieved September 3,
2010, from http://webaim.org/articles/visual/blind#howblind
WebAIM. (2010b). WebAIM: Web accessibility gone wild. Retrieved September 3,
2010, from http://webaim.org/articles/gonewild/
83
Web Usability. (2004). Web usability: Accessible data tables. Retrieved September
3, 2010, from http://www.usability.com.au/resources/tables.cfm#thead
W3C. (1999). Web content accessibility guidelines 1.0. Retrieved September 22,
2009, from http://www.w3.org/TR/WCAG10/
W3C. (2007). How people with disabilities use the web. Retrieved September 22,
2009, from http://www.w3.org/WAI/EO/Drafts/PWD-Use-Web/#diff
W3C. (2008a). Introduction to understanding WCAG 2.0 | Understanding WCAG 2.0.
Retrieved September 22, 2009, from
http://www.w3.org/TR/UNDERSTANDING-WCAG20/intro.html#introductionfourprincs-head
W3C. (2008b). Web content accessibility guidelines (WCAG) 2.0. Retrieved
September 22, 2009, from http://www.w3.org/TR/WCAG20/
W3C. (2009). Accessibility: W3C. Retrieved September 22, 2009, from
http://www.w3.org/standards/webdesign/accessibility