Final Usability Report
Elbuen.org, the website of El Buen Samaritano
Episcopal Mission
Henry Stokes/Prentiss Riddle
INF 385P
4/21/2005
NOTE: This report was developed for a class project at the School of
Information, UT Austin. It was not developed in consultation with El
Buen Samaritano. Please do not redistribute.
1
STOKES/RIDDLE
INF 385P
2
STOKES/RIDDLE
How this report is structured
The usability findings within this report are offered in three levels of detail. For the reader interested in only the
overall impressions, a one-page Executive Summary is offered on the next page. For the reader interested in
summaries of all the findings, there is the body of the paper, in particular the specific usability findings in
Section 5.3. Finally, all the detail of the methods and the raw data are provided in the Appendices, for only the
reader interested in the smallest detail.
The sections are numbered as follows:
1. Executive Summary
2. Introduction
3. Methodology
4. Results
5. For more information
Appendices:
A. Welcome and Instructions
B. Background Questionnaire
C. Task Descriptions
D. Post-Test Questionnaire
E. Post-Test Questionnaire Data
F. Performance Data
INF 385P
3
STOKES/RIDDLE
1.
Executive Summary
This report describes a study of the usability of elbuen.org, the website of the El Buen Samaritano Episcopal
Mission. The study was conducted by two graduate students in an introductory course in usability at the School
of Information of the University of Texas at Austin in April, 2005.
The study tested the use of the site by five representative users, including both English and Spanish speakers.
The participants were asked to carry out a set of tasks representative of the needs of clients, donors and
volunteers at El Buen Samaritano. Both their comments and test monitors' observations were recorded as they
performed the tasks.
Participants were able to complete most of the tasks and reported that they found the site easy to use, but they
did report some problems, and there were some tasks which they could not complete. Specific problems
include:

Unclear navigational labeling

Unclear placement of information

Confusion regarding services branded in Spanish

Inconsistencies between English and Spanish content
We make recommendations about how to address each of these problems.
INF 385P
4
STOKES/RIDDLE
2.
2.1.
Introduction
Study Purpose
The purpose of this study was to test the usability of the elbuen.org website, using representative users.
2.2.
Study Methods and Context
In this study we employed an “end-user test” method, wherein individual representative users were tested, one
at a time. Test participants were seated at a computer displaying the elbuen.org website, were given some
representative tasks to perform, and were observed as they carried out these tasks. This was a “find-and-fix”
test, more interested in identifying potential usability problems than in establishing baseline performance data
against which later releases could be compared.
2.3.
Study Summary
Five representative users were tested one at a time on the elbuen.org website. We wished to collect
performance data (error rates), satisfaction data (via a questionnaire), and, most importantly, particular areas of
potentially poor usability. Through meetings with the El Buen Samaritano team we agreed on a set of tasks
concerned with locating factual information on the website which would be of interest to the organization's client,
donors and volunteers. Questions we hoped to answer by studying these tasks include:

Will users in all segments be able to navigate the site and find what they are looking for?

Will users in all segments be able to understand the content once they find it?

Will information presented in both languages be consistent?

Will bilingual users be able to switch easily among equivalent content in both languages?
Although we collected general satisfaction data, we did not attempt to measure the website's effect on users'
attitudes toward El Buen Samaritano and its services. We recommend a separate marketing study for that
purpose.
2.4.
User Profile
Members of the El Buen Samaritano community come from the widest possible range of socioeconomic
backgrounds. Some speak only Spanish, others only English, and many speak both with varying degrees of
mastery. Many have only a primary education or no formal education at all. Many have never used a computer
or the Internet and find information online second-hand with the help of friends or family. Some are recent
graduates of El Buen Samaritano's own computer literacy programs.
INF 385P
5
STOKES/RIDDLE
The organizational roles of users of the elbuen.org site are varied as well. For the purposes of this study we
segmented them into clients, donors and volunteers.
Because some categories of users would be hard to recruit in the general population of Austin (in particular, lowincome Spanish speakers of limited education who nevertheless have sufficient computer skills to use the web)
we relied on El Buen Samaritano to recruit participants for the study. We stressed to their team the importance
of finding representative users and to the users the importance of being frank about any problems they found
with the website. In order to ensure that participants represented a cross-section of potential users, our
background questionnaire asked about languages, educational background and computer usage. In order not
to raise privacy concerns we did not inquire in detail about other socioeconomic factors beyond noting
participants' roles with El Buen Samaritano (i.e., clients vs. others).
INF 385P
6
STOKES/RIDDLE
3.
3.1.
Methodology
End-user Test Method
3.1.1.
Participants
As can be seen in figure 1 below, we tested five participants. Two were volunteers, one was a donor and two
were clients of El Buen Samaritano. Both of the clients had grown up bilingual in English and Spanish and
continued to speak both languages at home while using primarily English at work. The remaining participants
were monolingual English speakers. All participants had extensive experience with computers and used the
web on a daily basis. All were college-educated.
Figure 1: Test participants
Participant
Languages
Computer use
Education
no.
role
At home
At work
Growing up
How
long?
At
home?
Web?
P1
volunteer
en
en
en
years
yes
daily
college
P2
volunteer
en
en
en
years
yes
daily
college
P3
donor
en
en
en
years
yes
daily
college
P4
client
en/sp
en
en/sp
years
yes
daily
college
P5
client
en/sp
en
en/sp
years
yes
daily
college
3.1.2.
Procedure
The test was conducted in an office at the El Buen Samaritano facilities, a location chosen to meet the needs of
clients who might have transportation difficulties in reaching a more neutral test location and because it offered
childcare for participants accompanied by their families. The test room consisted of an office with a single
computer. Because no separate observation room was available, the test moderator stayed in the room and
interacted with the test participant as necessary while observing and taking notes.
INF 385P
7
STOKES/RIDDLE
Participants were welcomed and made comfortable in the test room and signed an informed consent form.
They then listened as he read a scripted explanation of the test and administered a short background
questionnaire (Appendices A and B). Because some participants had literacy issues, questionnaires were
administered orally. If there were no questions, they were given the task list (Appendix C) and asked to interact
with the website and carry out the tasks. They were encouraged to think aloud as they worked, to ask for help if
necessary and to feel free to skip to the next question or leave the test at any time if they wished. Both the
explanation of the test and the written task list emphasized that it was the website being tested, not the
participants, and that any problems they discovered would be used to help improve the site and were not a
reflection on the participants or on El Buen Samaritano.
After going through the tasks the participants were given a short post-evaluation questionnaire (Appendix D),
given a gratuity (an El Buen Samaritano t-shirt and, in the case of clients, $40 to cover their time, including time
spent on transportation). They were escorted to their waiting families if needed and thanked for their
participation. The session took less than 60 minutes.
3.1.3.
Task Scenarios
We worked with the El Buen Samaritano team to derive a set of tasks (Appendix C) that would be
representative of users of the website. The tasks consisted of using the site to find specific factual information of
use to clients, donors and volunteers:

Specific services offered (e.g., exercise classes, immunizations).

Times and locations (class schedules, street address).

How to find more information.
INF 385P
8
STOKES/RIDDLE
4.
Results
This section summarizes the data collected during the study sessions. We include three kinds of data here:
1. Performance data: participants' successes and failures at completing the tasks.
2. Satisfaction data: participants' reported enjoyment of the site.
3. Usability findings: usability problems discovered both from participants' comments and from test
moderators' notes, along with our recommendations.
These are summaries. More complete data can be found in the appendices, as noted.
4.1.
Performance Data
For each task we recorded whether the participants' success in one of three possible states: the participant
completed the task in a straightforward fashion, the participant completed the task but reported or displayed
confusion in the process, or the participant was unable to complete the task. We did not record timing data.
Note that of our five participants, one chose the option open to bilingual participants of repeating the tasks in the
second language. Hence the table shows a total of six trials for five participants.
Task
English (4 trials)
Spanish (2 trials)
Completed
task
Completed
task with
confusion
Unable to
complete
task
Completed
task
Completed
task with
confusion
Unable to
complete
task
1
1
3
0
0
1
1
2
3
1
0
1
0
1
3
4
0
0
1
0
1
4
3
1
0
1
0
1
5
4
0
0
2
0
0
6
0
3
1
2
0
0
Total across
tasks
15
8
1
7
1
4
(62.5%)
(33.3%)
(4.2%)
(58.3%)
(8.3%)
(33.3%)
(percentage
within
language)
INF 385P
9
STOKES/RIDDLE
Table 4.1: Performance data.
4.2.
Satisfaction Data
Students answered a post-test questionnaire (Appendix D) at the completion of the last task. The detailed
answers are in Appendix E. Their comments can be summarized as follows:
All participants described the site as "easy" to use, despite some specific difficulties.
All participants who commented on the aesthetics of the design had positive things to say about it ("clean,"
"colors good," etc.)
Negative comments focused on usability issues which we address in section 5.3 below.
4.3.
Usability Findings
As this study was a “find-and-fix” usability study, the findings in this section constitute the most important results
in the study.
4.3.1.
Positive Findings
Finding: The left menu bar was an easy way to navigate. Participants found it familiar and used it immediately
with no difficulty.
Finding: Users had no trouble locating contact information such as the street address. This information was
placed in the footer of every page and in the 'Contact Us' section.
4.3.2.
Problematic Findings
Problem: Unclear labeling
Description: Participants often did not know which category in which to find the specific information they were
looking for. Names such as Health Care, Primary Needs, and Primary Care were difficult to differentiate. At
times, participants stated that they could not discern the association between the desired information and the
category it was labeled with. Trial-and-error "shot-in-the-dark" guessing was sometimes necessary.
In the Spanish site, some navigational labels appeared in English ("Programs/Services") and others used
unfamiliar Spanish terminology ("Portal" for home).
INF 385P
10
STOKES/RIDDLE
Recommendation: Develop better terminology that users are more likely to understand. Pay close attention to
the choosing Spanish labels that will be understood by speakers of multiple varieties of Spanish. Add a search
feature.
Problem: Unclear placement
Description: Information was not necessarily arranged on a page in a way that made sense to users. Important
information required scrolling without cues on the page to that effect. Links were sometimes made small and
hidden within text blocks.
For example, donor and volunteer information appear together on one long page with no internal navigation,
and to make matters worse, motivational information about existing volunteers is in the way of information about
how to volunteer.
Recommendation: Find ways to isolate and clarify information, attract user's eye for minimal searching. Break
up long pages or provide internal links.
Problem: Monolingual branding on a bilingual site
Description: Monolingual English-speaking participants were surprised by the names of the "Vida y Salud" and
"Promotores" programs and, without knowing their meaning, had no way to know what content lay behind them.
Recommendation: Supplement Spanish-"branded" labels with additional labeling. For example: "Vida y Salud
lifestyle classes".
Problem: Distracting content
Description: Participants complained that motivational material such as the staff bios and "Look Who's
Volunteering" were intrusive and confusing, getting in the way of task-oriented use of the site.
Recommendation: Place factual and task-oriented content more prominently. Provide visual and layout cues to
help users distinguish it from background and motivational content.
Problem: Inconsistency between English and Spanish content
Description: Some items on the English site were missing from the Spanish site, such as food pantry
information.
Recommendation: Perform inventory to ensure that all information available in one language is also available in
the other, particularly factual and task-oriented information.
INF 385P
11
STOKES/RIDDLE
Problem: Information about a particular service is scattered
Description: Locations, schedules, and other details about a particular service were often scattered in different
places. For example, participants expected to be able to click on a class such as Citizenship to see its schedule
and were surprised that instead they needed to scroll to the bottom of the page and click on a separate link to a
comprehensive schedule. Similarly, donor information about the food pantry was on one page, whereas
information about its hours and services was on another.
Recommendation: Lump more complete information about a given service in one spot, considering what a
client, donor or volunteer will need to know to use or contribute to the service.
Problem: Inconsistent navigation menus
Description: Some labels on the left-hand navigation menu had popout sub-menus and others did not.
Recommendation: Make the navigation menu consistent, particularly for sections which could benefit from
popouts, such as the donor/volunteer page.
INF 385P
12
STOKES/RIDDLE
5. For More Information
If you have any questions about anything contained in this report, please contact:
Henry Stokes
hstokes@mail.utexas.edu
Prentiss Riddle riddle@ischool.utexas.edu
INF 385P
13
STOKES/RIDDLE
Appendix A – Welcome and Instructions
Thank you very much for coming in today. My name is _________. My partner and I are students at the School of
Information at the University of Texas. We are taking a class on how to evaluate products to see how easy they are to
use. Today we are conducting a study to test the usability of a website.
I will guide you through today's session and interview you about your reactions to the site. Keep in mind, we are not
evaluating you in any way, we are only interested in your experience with the site. Throughout the session, I will
encourage you to freely express your opinions, to comment on what is clear and what is unclear to you, and in particular,
what you find confusing or difficult to understand.
If you are ready, I will take you to the opening page of the website and give you a list of tasks to perform with the
site. Please perform as many of the tasks as you can. Take your time and don't worry if you have problems.
Remember, we are testing the website, not you or your abilities.
As you work, please think aloud. If you run into a problem you can ask me for help. You are also free to skip to the next
question or end this test at any time.
Do you have any questions before we begin?
INF 385P
14
STOKES/RIDDLE
Appendix B - Background Questionnaire
Administered orally to accommodate participants with literacy issues.
Questions given in italic type should be asked as stated. Clarifications may be improvised as necessary. Use the space on the right to record
additional comments by the participant. For checkboxes (), check all that apply.
Name: _____________________
Participant segment:
(Clients will be identified as such by El Buen Samaritano.)
 Client
(For non-clients)
What do you do for El Buen Samaritano?
 Donor
 Volunteer
 Other: __________________________
Languages:
What languages do you usually speak at home?
 English  Spanish Other: _____________
What languages do you usually speak at work?
 English  Spanish Other: _____________
What languages did you speak growing up?
 English  Spanish Other: _____________
Computer literacy:
How long have you used a computer?
Number: ______  Weeks  Months  Years
Do you have a computer at home?
INF 385P
15
STOKES/RIDDLE
 Yes  No
How often do you go online to use the Web?
 Daily  Weekly  Monthly  Less
Education:
How many years did you go to school or what grade did you finish?
(Record as stated by participant)
 Years/grade _________
 Elementary/primaria
 High school/secundaria
 College/university
INF 385P
16
STOKES/RIDDLE
Appendix C – Task Descriptions
Please perform as many of the tasks on the El Buen Samaritano website as you can. Take your time and don't
worry if you have problems. We are testing the website, not you or your abilities.
As you work, please think aloud. If you run into a problem you can ask us for help. You are also free to skip to
the next question or end this test at any time.
1. Find out what day of the week the Citizenship Class meets.
2. Find out whether El Buen Samaritano offers immunizations.
3. Find out who to call to discuss volunteering.
4. Find out what kinds of food are accepted for the food pantry.
5. Find El Buen Samaritano's street address.
6. Find out what kinds of exercise classes are offered.
(Optional) If you speak both English and Spanish, find the same list of exercise classes in Spanish.
INF 385P
17
STOKES/RIDDLE
Appendix D – Post-test Questionnaire
Administered orally in order to accommodate participants with literacy issues.
Name: _____________________
1. Would you say the website was easy or hard to use?
2. Were there parts of it which were particularly hard?
3. Was there information which you couldn't find?
4. Were there particular things about the site which you liked?
5. Were there particular things about the site which you didn't like?
6. Additional probes about specific problems:
7. Is there anything you'd like to add?
INF 385P
18
STOKES/RIDDLE
Appendix E – Post-test Questionnaire Data
Here are the participants' replies to the post-test questionnaire.
1. Would you say the website was easy or hard to use?
P1: Easy for the most part; not much info to have to decide from.
P2: Relatively easy.
P3: Easy, but couldn't find exercise class.
P4: Pretty easy.
P5: Easy but I didn't find everything.
2. Were there parts of it which were particularly hard?
P1: No.
P2: [Trying to locate the] exercise class -- didn't get an immediate impression of [where to look].
P3: No.
P4: What category to use?
P5: No, but a search box would help.
3. Was there information which you couldn't find?
P1: No.
P2: No.
P3: Yes, the exercise class.
P4: The food pantry [in Spanish].
P5: Yes, food items [the food pantry] in Spanish. And I couldn't return home in Spanish.
4. Were there particular things about the site which you liked?
P2: Nice, consistent pages but didn't look exactly alike.
P3: It was clean.
P4: Like photographs, sidebar, switching back and forth between English and Spanish.
P5: The fonts were okay and the colors were good.
5. Were there particular things about the site which you didn't like?
P2: Navigation bar -- not apparent that there is secondary navigation.
P3. Links were hard to see.
INF 385P
19
STOKES/RIDDLE
P4: Too much white space on right.
P5: The inconsistency of information between English and Spanish, broken links, and no search box.
7. Is there anything you'd like to add?
P5: In this community, kids may be web guides for their parents [so the design should keep children in mind].
INF 385P
20
STOKES/RIDDLE
Appendix F – Performance Data
Raw performance data for each participant. Note that participant 5 attempted to perform each task twice, in
English and in Spanish.
Key:
Y
The participant completed the task in a straightforward fashion.
YC
The participant completed the task but reported or displayed confusion in the process.
N
The participant was unable to complete the task.
Task
P1
P2
P3
P4
P5
P5
English
English
English
Spanish
English
Spanish
1
YC
Y
YC
YC
YC
N
2
YC
Y
Y
Y
Y
Y
3
Y
Y
Y
Y
Y
N
4
Y
Y
YC
N
Y
N
5
Y
Y
Y
Y
Y
Y
6
YC
YC
N
Y
YC
Y
Additional notes on tasks:
Task 1: Find out what day of the week the Citizenship Class meets.

Participant expected to be able to click on a class (e.g., Citizenship) to see schedule. Failing that, it was
not obvious to scroll down to find the schedule.

Participants were surprised that all schedules are on one (separate) page and that there is not a
schedule page for each class.

Class schedule missing in Spanish.

"Programs/Services" labeled in English on Spanish site.

Participant confused by testimonial staff bios.
Task 2: Find out whether El Buen Samaritano offers immunizations.

INF 385P
Unclear labeling - participant had difficulty knowing which one to go to ("Health Care" or "Primary
Care"?)
21
STOKES/RIDDLE

Found easier because participant remembered option ("Health Care") from previous task
Task 3: Find out who to call to discuss volunteering.

Surprised by "Look Who's Volunteering" before information on how to volunteer.

Expected secondary sub-menu under Donate/Volunteer in left column.

Participant looked for a link at the top of the Volunteer/Donate page to sections of that page.
Task 4: Find out what kinds of food are accepted for the food pantry.

Would not have expected to find details of food pantry on volunteer page.

Submenu of Donate/Volunteer could include "Money", "Food".

Did not associate "Primary Needs" with food pantry.

Food pantry and list of food items missing from Spanish site.
Task 5: Find El Buen Samaritano's street address.

Answer available on footer of every page AND on Contact page (good thing)

Participant wanted a map
Task 6: Find out what kinds of exercise classes are offered.

INF 385P
English speakers surprised by "Vida y Salud" in Spanish (monolingual branding on a bilingual site)
22