VeraHouse.org
Final Usability Report
Kristin Davis
Erika S.A. Moore
INF385P, School of Information
The University of Texas at Austin
April 21, 2005
VeraHouse.org Usability Report 1
Table of Contents
1.
How this report is structured...................................................................................... 2
2.
Executive Summary................................................................................................... 3
3.
Introduction ................................................................................................................. 4
3.1.
Study Purpose ....................................................................................................... 4
3.2.
Study Methods and Context ................................................................................. 4
3.3.
Study Summary ..................................................................................................... 4
3.4.
User Profile ............................................................................................................ 4
3.5.
What this study is NOT.......................................................................................... 4
4.
Methodology ............................................................................................................... 5
4.1.
End-user Test Method........................................................................................... 5
4.2.
Evaluation Measures ............................................................................................. 6
5.
Results ........................................................................................................................ 7
5.1.
Performance Data ................................................................................................. 7
5.2.
Satisfaction Data.................................................................................................... 7
5.3.
Usability Findings................................................................................................... 9
6.
Next step................................................................................................................... 12
7.
Contact Information.................................................................................................. 12
8.
Appendix A – Welcome and Instructions ............................................................... 13
9.
Appendix B – Task Descriptions ............................................................................. 14
10.
Appendix C – Post-test Questionnaire ................................................................... 15
11.
Appendix D – Performance Data ............................................................................ 16
12.
Appendix E – Post Test Questionnaire Data ......................................................... 17
13.
Appendix F - VeraHouse.org Home Page ............................................................. 18
VeraHouse.org Usability Report 2
1. How this report is structured
The usability findings within this report are offered in three levels of detail. For the reader
interested in only the overall impressions, a one-page Executive Summary is offered on the next
page. For the reader interested in summaries of all the findings, including the performance data
(success rate, time on task), the satisfaction data (the questionnaire results), and, most importantly,
the specific usability findings in Section 5.3, there is the body of the paper. Finally, all the detail of
the methods and the raw data are provided in the Appendices, for only the reader interested in the
smallest detail.
VeraHouse.org Usability Report 3
2. Executive Summary
Vera House, Inc. contracted with usability consultants Kristin Davis and Erika Moore to conduct a
usability evaluation of their existing Web site, VeraHouse.org. Working with Vera House, the
usability consultants designed an end-user test for April, 2005. The test was carried out at the
usability lab of the University of Texas at Austin the week of April 4, 2005
Six representative users were tested -- six graduate students, with a median age group of 20-29.
Two of the six participants were male; four of the participants were female. None had experience
with this Web site.
Overall, the results are promising but also indicative that some substantive redesign work is
needed. The performance data suggest that users can carry out all tested tasks in a reasonable
amount of time. The satisfaction data were positive. Participants generally thought the interface
was functional but that some of the navigation terms were confusing.
The most important data here are the specific usability problems unearthed. There were no Critical
or Major errors found. Section 5.3 details two Moderate problems and three Minor problems, all
with recommended redesigns. The areas that earned criticality ratings include:

Unclear left navigation bar labels and organization

Emergency contact information not given adequate priority

Font size too small and not enough white space between links

Users unable to donate online
The consultants are prepared to work with Vera House to arrive at detailed redesigns.
VeraHouse.org Usability Report 4
3. Introduction
3.1.
Study Purpose
The purpose of this study was to test the usability of the Web site VeraHouse.org (not the
organization or the services themselves), using representative users.
3.2.
Study Methods and Context
In this study we employed an “end-user test” method, wherein individual representative users were
tested, one at a time. Test participants came to a usability lab, were given some representative
tasks to perform, and were observed and measured as they carried out these tasks. This was a
“find-and-fix” test, interested in identifying potential usability problems.
3.3.
Study Summary
Six representative users were tested one at a time on the VeraHouse.org Web site. We wished to
collect performance data (time on task, error rates), satisfaction data (via a questionnaire), and,
most importantly, particular areas of potentially poor usability. Through meetings with Vera House
we agreed on a series of tasks to get at the following questions:
1. Participants begin on the Home: Please identify how an abused woman would seek help.

Designers are curious to know if users access information through the “in crisis”
navigation.
2. Participants begin on the Home: Please identify two ways you would be able to donate time or
money to Vera House.

Designers wish to know if users access information through “events” and “support”
navigation.
3. Participants begin on the Home: Assume that you thought a friend of yours was being abused.
Where would you find general information about abuse?

3.4.
Designers want to know if users access information through “FAQs” navigation.
User Profile
The types of people who might visit the VeraHouse.org Web site are quite varied, including women
in crisis, donors, and volunteers. The typical user tends to be female, though levels of education,
income and computer and Internet experience vary widely. For the purposes of this test we
decided to test graduate school students.
3.5.
What this study is NOT
This study is NOT a baseline test, with crisp, quantitative data like time-on-task, to compare future
performance to. Rather, it is a find-and-fix usability study, with the main goal being to identify
possible usability problems. Further, it is NOT a study of asymptotic behavior (that is,
performance after users get good at using the product). Rather, it is a study of how well people
perform in their first interactions with the VeraHouse.org Web site user interface.
VeraHouse.org Usability Report 5
4. Methodology
4.1.
End-user Test Method
4.1.1.
Participants
As can be seen in Figure 1 below, we tested six participants. Four of the participants were female.
The median age range was 20-29. None had any experience with the VeraHouse.org Web site.
Figure 1: VeraHouse.org Test Participants
Participant #
Gender
Age Range
Level of Education
Experience with
VeraHouse.org?
1
M
20-29
Some graduate school
No
2
F
30-39
Some graduate school
No
3
F
20-29
Some graduate school
No
4
M
20-29
Some graduate school
No
5
F
20-29
Some graduate school
No
6
F
30-39
Some graduate school
No
4.1.2.
Procedure
Test participants were welcomed into the lab at the University of Texas at Austin. They were
seated at a computer workstation in the test room of the lab, and observed by one test moderator,
who stayed in the test room, interacted with the participant, and took notes.
Test participants were shown the observation room, they read and signed an informed consent
form, and then listened and read along as the moderator read the instructions (Appendix A). If
there were no questions, the moderator read the first task description, as the participant read
along, and then the participant attempted to carry out the task. At the completion of the last task the
participant completed the post test questionnaire (Appendix C), was thanked and escorted out of
the lab. The entire session took between 20 and 30 minutes.
4.1.3.
Task Scenarios
The usability consultants worked with Vera House to derive a set of tasks (Appendix B) that would
be representative of a first-time VeraHouse.org user. The tasks included:

Identify how an abused woman would seek help

Identify ways to donate time or money to Vera House

Find general information about abuse
VeraHouse.org Usability Report 6
4.2.
Evaluation Measures
In the next section, “Results,” we will offer summaries of performance data and summaries of the
questionnaire responses. Also in the Results will be a prioritized list of identified usability problems
and recommended fixes. The problems noted by the evaluator will be given a criticality rating per
the following table. The higher the rating of criticality, the more significant the problem is to the
user’s experience or ability to accomplish the task.
Figure 2: Criticality Ratings
CRITICALITY
RATING
CRITICALITY
DESCRIPTION
CRITICALITY DEFINITION
The identified issue is so severe that:
4
3
2
1
Critical
Major

Critical data may be lost

The user may not be able to complete the task

The user may not want to continue using the
application

Users can accomplish the task but only with
considerable frustration and/or performance of
unnecessary steps by participants

Non-critical data may be lost

The user will have great difficulty in circumventing the
problem

Users can overcome the issue only after they have
been shown how to perform the task

Five or more instances of navigational error

The user will be able to complete the task in most
cases, but will undertake some moderate effort in
getting around the problem

The user may need to investigate several links or
pathways through the system to determine which
option will allow them to accomplish the intended task

Users will most likely remember how to perform the
task on subsequent encounters with the system

Three or four instances of navigational error

An irritant

A cosmetic problem

A typographical error

One or two instances of navigational error
Moderate
Minor
VeraHouse.org Usability Report 7
5. Results
This section summarizes the data collected during the study sessions. It is important in usability
studies to address both performance data and preference data, as we are interested in not only
CAN users complete their tasks, but do they like the Web site. Thus, this section consists of three
parts:



Performance of the participants while attempting the tasks (task completion percentages),
Satisfaction of the participants as noted in their post-test questionnaire responses, and
Particular usability findings, compiled from both participants comments and moderator
observations and notes.
Under the rubric of “constructive criticism,” high-level recommendations are offered for all the
identified usability problems. However, Vera House will always best know their own services (and
certainly their possible resource allocations). So, we stand prepared to work with Vera House to
derive more detailed redesigns.
5.1.
Performance Data
We recorded the participants’ success and time on task, for each task. As for errors, we noted the
number of errors as well as their level of criticality. We recorded time on task to the nearest halfminute. Time on task is a valuable data point, particularly taken with the dialog between the test
participant and the test moderator, revealing the logic behind the participants’ choices.
Figure 3: Performance Data
Successful
completion
5.2.
Unsuccessful
completion
Median
Time on
Task in
Minutes
Median
Number
of Errors
Median
Error
Criticality
Task 1
6
0
1
0
0
Task 2
6
0
1
1
1
Task 3
6
0
2
1.5
1
Satisfaction Data
Participants answered a post test questionnaire (Appendix C) at the completion of the last task.
The detailed answers are in Appendix E. The table below summarizes the participants’ responses to
the post test questionnaire
VeraHouse.org Usability Report 8
Figure 4: Satisfaction Data
Question
Responses
Easy
How hard did you think it was to complete
these tasks?
Average
83%
Yes
Were you able to find all the information
you expected to find?
17%
Yes
Would you refer a friend to
VeraHouse.org, if it were a local
organization?
Yes
Do you trust VeraHouse.org?
Yes
33%
No
83%
17%
No
67%
33%
No
83%
If you were a woman in crisis, would you
feel safe getting information from
VeraHouse.org?
Yes
Did the design and content of
VeraHouse.org assure you that it is a
legitimate organization?
Yes
After viewing the site, would you
volunteer for Vera House, if it were a local
organization?
Yes
How would you describe your overall
experience with the Vera House Web
site?
17%
No
100%
0%
No
100%
0%
No
83%
Yes
Would you donate money to Vera House
if it were a local organization?
0
No
67%
If you wanted to give information to a
friend in crisis, would you go to
VeraHouse.org?
Difficult
17%
No
83%
Positive
67%
17%
Neutral
33%
Negative
VeraHouse.org Usability Report 9
5.3.
Usability Findings
As this study was a “find-and-fix” usability study, these specific findings are the most important
findings in the study. For usability studies such as this, it is important to key on the particular
problems identified, and to figure out how to address each. The following findings include a
criticality rating as described in Section 4.3.
The findings below are divided into Moderate and Minor problems. Each has an associated
recommendation.
5.3.1.
Good things
The Web site contains a great deal of information and received positive satisfaction ratings from
participants. One hundred percent of participants said they would feel safe using the site. Eighty
three percent of participants rated the test tasks as easy.
5.3.2.
Critical & Major Findings
There were no problems that rated a criticality score of “critical” or “major”. We believe that the
Web site is functional as it is today. However, there are several moderate and minor findings that,
if not addressed, will lead to user frustration.
VeraHouse.org Usability Report 10
5.3.3.
Moderate Findings
Several problems arose from ambiguity in terms used in the left navigation bar, as well as grouping
and prioritization of these terms. We suggest a redesign of the left navigation as detailed in Figure
5.
Figure 5: Left Navigation Redesign
Current Left Navigation
Proposed Left Navigation
Home
Home
In crisis
In Crisis
Events

Who to Call
Programs

What is Abuse?
Coalition

Services Provided
News
Support Vera House
Support

Upcoming Events
Contacts

Donate to Vera House
FAQs

Volunteer
Links
Partnerships

Domestic Violence Coalition

Rape Crisis Center
Links
(Contacts-removed, on bottom navigation)
Finding Mod1: “FAQs” link not intuitive.
Criticality: 2
Description: Participants expected “FAQs” to provide information about Vera House as an
organization, instead of information about abuse.
Recommendation: Rename link “What is abuse”. Create clear separation in the Web site for
different target audiences, see Figure 5.
VeraHouse.org Usability Report 11
Finding Mod2: “Support” link not intuitive
Criticality: 2
Description: Participants were confused about how to donate money or volunteer time to Vera
House. Some participants expressed that they thought “support” referred to technical support
instead of ways to support Vera House.
Recommendation: These problems would also be addressed by the redesign of the left navigation
bar.
5.3.4.
Minor findings
Finding Min1: Emergency contact information not given adequate priority
Criticality: 1
Description: Participants expressed frustration at having to scroll though a lot of text before
reaching the list of contact numbers.
Recommendation: These numbers, especially those to be used in an emergency, should be
placed at the top of the page in a clear, easy to read font.
Finding Min2: Small font size, overcrowding of text
Criticality: 1
Description: The fonts used throughout the site are inconsistent and often too small. There is not
enough space between lines of links making clicking on links difficult. Participants expressed that
this issue would be especially problematic for a woman in crisis, who is likely to be upset and may
have additional difficulty clicking on small, overcrowded links
Recommendation: Use a larger font and double space text.
Finding Min3: No online donation
Criticality: 1
Description: Participants expressed the desire and expectation to be able to donate money on the
Vera House Web site.
Recommendation: Implement this functionality.
VeraHouse.org Usability Report 12
6. Next step
The next step is to triage these findings and decide which of these identified problems can be
addressed in this release, and in what manner. It is my hope that we will work together to decide
how to redesign the Web site, and then, if time permits, perform one more test.
7. Contact Information
If you have any questions about anything contained in this report, please contact:
Kristin Davis
kmdavis@ischool.utexas.edu
Erika Moore
emoore@ischool.utexas.edu
VeraHouse.org Usability Report 13
8. Appendix A – Welcome and Instructions
Understanding Your Participation
Please read this page carefully.
Vera House is asking you to participate in evaluating their Web site. By participating in this
evaluation, you will help us improve this site and other Vera House Services.
You will be working with a Web site is being redesigned. We’ll observe you and record
information about how you work with the Web site. We will then ask you to answer a few
interview questions. Please think out loud as you go through this process.
If you need a break, just tell us. We expect this process to take approximately 20-30
minutes.
You may withdraw from this evaluation at any time.
If you have any questions, you may ask now or at any time.
Please remember that this is an evaluation of the Web site, not of your abilities.
If you agree with these terms, please indicate your agreement by signing here:
Please print your name
____________________________________________
Signature
____________________________________________
Date
___________________________________________________________
Participant #__________
Test Moderator________
VeraHouse.org Usability Report 14
9. Appendix B – Task Descriptions
Task List for VeraHouse.org
1. Please identify how an abused woman would seek help.
2. Please identify two ways you would be able to donate time or money to
Vera House.
3. Assume that you thought a friend of yours was being abused. Where
would you find general information about abuse?
Please take a few moments to browse the site if you would like.
Participant #__________
Test Moderator________
VeraHouse.org Usability Report 15
10. Appendix C – Post-test Questionnaire
1. Gender
o Female
o Male
2. Which age range do you fall into?
o 20-29
o 30-39
o 40-49
o 50+
3. What is your level of education?
o No High School diploma
o High School diploma, GED or equivalent
o Some college, no degree
o Undergraduate degree
o Some graduate School, no degree
o Graduate degree
4. Have you ever visited VeraHouse.org before?
o Yes
o No
5. How hard did you think it was to complete these tasks?
o Easy
o Average
o Difficult
6. Were you able to find all the information you expected to find?
o Yes
o No
If not, what else did you expect to find? ____________
7. If you wanted to give information to a friend in crisis, would you go to VeraHouse.org?
o Yes
o No
8. Would you refer a friend to VeraHouse.org, if it were a local organization?
o Yes
o No
9. Do you trust VeraHouse.org?
o Yes
o No
10. If you were a woman in crisis, would you feel safe getting information from VeraHouse.org?
o Yes
o No
11. Did the design and content of VeraHouse.org assure you that it is a legitimate organization?
o Yes
o No
12. After using the site, would you volunteer at Vera House if it were a local organization?
o Yes
o No
13. Would you donate money to Vera House if it were a local organization?
o Yes
o No
14. How would you describe your overall experience with the Vera House Web site?
o Positive
o Neutral
o Negative
VeraHouse.org Usability Report 16
11. Appendix D – Performance Data
P1
P2
P3
P4
P5
P6
Median
Mean
Success
Y
Y
Y
Y
Y
Y
Time in minutes
1
1
0.5
0.5
0.5
1
1
1
Number of errors
0
0
0
0
0
0
0
0
Error criticality
0
0
0
0
0
0
0
0
Success
Y
Y
Y
Y
Y
Y
Time in minutes
3
1.5
1
1
1
1.5
1
2
Number of errors
4
1
0
2
0
1
1
1.3
Error criticality
2
1
0
1
0
1
1
0.8
Success
Y
Y
Y
Y
Y
Y
1.5
2
1
2
0.5
3.5
2
2
Number of errors
1
2
0
4
1
4
1.5
2
Error criticality
1
1
0
2
1
2
1
1
Task 1
Identify how an
abused woman
would seek help.
Task 2
Identify two ways you
would be able to
donate time and
money to Vera
House.
Task 3
Find general
information about
abuse.
Time in minutes
VeraHouse.org Usability Report 17
12. Appendix E – Post Test Questionnaire Data
Question
1-Gender
P1
P2
P3
P4
P5
P6
2
1
1
2
1
1
20-29
30-39
20-29
20-29
20-29
30-39
3-What is your level of education?
5
5
5
5
5
5
4-Have you ever visited VeraHouse.org before?
2
2
2
2
2
2
5-How hard did you think it was to complete these tasks?
2
1
1
1
1
1
6-Were you able to find all the information you expected to
find?
2
1
1
2
1
1
7-If you wanted to give information to a friend in crisis, would
you go to VeraHouse.org?
1
1
1
2
1
1
8-Would you refer a friend to VeraHouse.org, if it were a local
organization?
2
1
1
2
1
1
9-Do you trust VeraHouse.org?
1
1
1
2
1
1
10-If you were a woman in crisis, would you feel safe getting
information from VeraHouse.org?
1
1
1
1
1
1
11-Did the design and content of VeraHouse.org assure you
that it is a legitimate organization?
1
1
1
1
1
1
12-After viewing the site, would you volunteer for Vera
House, if it were a local organization?
1
1
1
2
1
1
13-Would you donate money to Vera House if it were a local
organization?
1
1
1
2
1
1
14-How would you describe your overall experience with the
Vera House Web site?
2
1
1
2
1
1
2-Which age range do you fall into?
VeraHouse.org Usability Report 18
13. Appendix F - VeraHouse.org Home Page