Memo
To: Karen Bennett, Manager, User Experience, IHG
cc: Dr. Carol Barnum
From: CBR China: Yufei Duan, Yina Li, Ying Li, Qianying
Liu, Niven Sellars, Michael Somer
Subject: Usability Testing Findings Report for Chinese Holiday
Inn website
Date: April 25, 2008
Included with this memo is team CBR China’s final report for our
research project on the Holiday Inn’s Chinese website. The following
are included within this report:
 Project purpose
 Goals set by Team CBR China
 Processes followed during the project
o Heuristic review
o Test plan
o Recruitment and screening
o Real-time logging
o Analysis
 Categorized results and findings
 Recommendations
In addition to the aforementioned items included in this report, team
CBR China also includes appendices of our test logs, participant
consent forms, screeners, questionnaires, scenarios, and other
materials in support of testing. Other research components include the
team’s heuristic evaluation, personas, and test plan.
Team CBR China would like to extend its gratitude to our sponsor,
Karen Bennett, as well as our instructor Dr. Carol Barnum, who has
assisted us throughout this project. Without both of you we would not
have had the chance to engage in a real world usability experience.
Thank you again,
The members of team CBR China
1
TABLE OF CONTENTS
EXECUTIVE SUMMARY .......................................................................................................................... 3
TEST OBJECTIVE ......................................................................................................................................... 3
OUR PROCESS ............................................................................................................................................. 4
OUR RESULTS ............................................................................................................................................. 4
BACKGROUND ............................................................................................................................................ 7
REPORT OVERVIEW .................................................................................................................................... 8
METHODOLOGY ....................................................................................................................................... 8
OVERVIEW.................................................................................................................................................. 8
USER PROFILES .......................................................................................................................................... 9
USABILITY CRITERIA .................................................................................................................................. 9
TEST OBJECTIVES ......................................................................................................................................10
FINDINGS AND RECOMMENDATIONS ............................... ERROR! BOOKMARK NOT DEFINED.
POSITIVE FINDINGS ...................................................................................................................................16
Positive Impressions ............................................................................................................................16
SEVERITY RATINGS ...................................................................................................................................19
COLLATED FINDINGS, RATED BY SEVERITY ...............................................................................................19
USER IMPRESSIONS ....................................................................................................................................21
General comments ...............................................................................................................................21
Negative Impressions ...........................................................................................................................21
Suggestions ..........................................................................................................................................21
Findings and Recommendations ..........................................................................................................21
OTHER FINDINGS .......................................................................................................................................26
Miscellaneous Findings .......................................................................................................................27
Localization Findings ..........................................................................................................................28
POST-TASK ANALYSIS ...........................................................................................................................30
FUTURE DEVELOPMENT AND TESTING ................................................ERROR! BOOKMARK NOT DEFINED.
CONCLUSION ............................................................................................................................................35
FINDINGS ...................................................................................................................................................35
COMMENTS..........................................................................................ERROR! BOOKMARK NOT DEFINED.
2
Executive Summary
Test Objective
Karen Bennett, Manager of User Experience at IHG, presented her
concerns for the usability study of the China Holiday Inn website as
follows:




Do users connect better with the Holiday Inn or home page or a
competitor site, such as www.elong.com?
What features do the users connect with and like (IHG vs
competitors)?
Does the booking process work for the Chinese user and if not,
why not?
Does a competitor site have a better booking process, and if so,
why?
Our study focused on gathering qualitative and quantitative data to
address IHG’s concerns about the usability of the site.
Based on IHG’s concerns and the heuristic evaluation we conducted,
we assessed the ease and difficulty of the following hotel booking tasks
on www.HolidayInn.com.cn:

The general feeling/layout of the site: Is it easy to find the
information users want to know the first time they visit?

The procedure of booking a hotel online:
 Basic searching: is it easy to use?
 Entering personal information: Does the website require
reasonable and suitable information for Chinese users?
 Do the users understand all information requirements?

Language: Can the users understand the language used on the
Holiday inn website? Do users understand all the phrasings in
the Holiday Inn website? Are there any translation mistakes or
misunderstandings?

Satisfaction with the site: Which aspects do users like with the
site and which aspects do they dislike?
3

Perceived reliability of the site: Do users trust the website? Do
they fill out the personal information readily?

Navigation: Can users find clear and efficient navigation when
booking a hotel or browsing the website?

Compare Holiday Inn website to competitor site: How do users
feel about the elong.com website in comparison to the Holiday
Inn site?

Compare Holiday Inn China website to the Holiday Inn English
website: How do users feel about the English version of the
same site?
Note: Since we concluded our usability testing on the Holiday Inn China
website, the following changes have been made to the website:

There is now a map feature available when search results are
displayed.

The input fields have now been changed from the upper right-hand
corner to the center of the page.
Our Process
The purpose of the Chinese HolidayInn.com.cn usability test was to
collect feedback about how users use the Holiday Inn online hotel
booking site in China, what problems they may encounter using the
site, what improvements they would like to have, and what additional
information they desire. Following industry standard methods of
usability testing, our team undertook an evaluation of the site,
developed user profiles, and developed a six-scenario test plan.
Our Results
The team found many consistent issues among users that were areas
of concern. Our findings have been divided into seven categories:


Aesthetics: The look and feel of the website, and whether or
not it is also functional
Navigation: How well users can find their way around the
website
4




Layout: The arrangement of page elements and their
effectiveness at guiding the user
Feature Requests: Options that users want and expect but do
not find
Brand Identity: How well users recognize Holiday Inn and
other IHG hotels as a brand
Mental Model: The process flow and design users expect to find
and how well the website matches their expectation
Based on the feedback we received from our participants as well as our
analysis of the quantitative data from testing, we identified 14 significant
usability issues with the Holiday Inn China website. The following table lists
these issues and indicates their impact on the usability of the site, along with
the number of participants who experienced each problem.
Usability Problem
Navigation: The ‘start over’ and
‘finish booking’ options on the
confirm booking page confused the
user
# Affected
Severity
1/6
low
Layout: Users found the Priority
Club Login options confusing.
4/6
high
Layout: Users think
advertisements are overwhelming
3/6
medium
Layout: Users want the total price
for their stay to be listed, instead of
a per-night rate only
3/6
high
Layout: Users have trouble finding
the cancellation link to cancel their
reservation
3/6
high
Layout: Error messages were not
seen by the user or they were
misinterpreted
5/6
high
Aesthetics: Search results of
hotels are displayed in a two-color
format, confusing users’ perception
of how many results are actually
being displayed
2/6
low
5
Feature Requests: Users
complain about not seeing
promotional rate information
Feature Requests: Users want to
see a drop-down list of cities
instead of having to type city
names in
Feature Requests: Users want to
see hotel rankings by different
categories (customer experience
and price)
Feature Requests: Users want
more information on hotel
amenities
Mental Model: Users are confused
as to whether or not they are
required to input full Chinese name,
or just family name
Feedback: Loading screen has no
progress bar. This confuses users
as to whether or not their
information is being processed
2/6
low
3/6
high
4/6
medium
4/6
high
5/6
high
6/6
high
6
Background
We are the Chinese Holiday Inn Website usability group, a team of
four Chinese students and two American students in SPSU’s Bachelor
of Science in Technical Communication degree program. Dr. Carol
Barnum, the professor of our usability testing and research course,
suggested the idea of this usability test of the Chinese Holiday Inn
website.
Karen Bennett was interested in having the Chinese Holiday Inn
website evaluated for usability questions and issues. In the initial
meeting, she discussed the scope of the site and its typical users, as
well as competitor sites. She also discussed what concerns she already
had about the website going into testing; for example, she wanted to
make sure that there were not any translation issues.
From the information provided by the sponsor, the project team
learned that the audience for the website is mostly domestic
(mainland) Chinese. They book hotel rooms online for business and/or
leisure purposes. Their ages span from the mid-twenties to the fifties.
Most of them are not comfortable using computers. The majority of
them do not speak or read English.
Purpose
Our team worked with Ms. Bennett’s information and questions to
more fully define:



The scope—How much of the site would be included in the
evaluation
The user base—What type of users work with the site
Use cases—Typical tasks users are expected to perform
We experienced some difficulty in finding participants within the
identified age bracket; therefore we expanded our user’s age range to
include the early twenties as well.
We found our users based on these criteria; however, all of our users
(because they live in the US) speak and read English.
The outcome of this analysis resulted in the creation of two personas,
a heuristic evaluation, and a detailed test plan with a list of suggested
tasks and questions for the test sessions.
7
Overall, our purpose was to collect information about how users
interact with the website and to identify areas in which the site could
be improved to better benefit its users.
Report Overview
The following major sections are included in our report:




Methodology
Post-Task Analysis
Findings and Recommendations
Conclusion
The appendices of this report include all materials that are relevant to
the usability test so that, if needed, subsequent usability tests can be
easily conducted. The appendices include the following:






Appendix A: Heuristic Evaluation—Describes the criteria we
used in our initial evaluation of the website, and identifies our
areas of concern that were later used to create the test plan
Appendix B: Personas Report—Describes the characteristics,
goals, needs, and preferences of our anticipated users.
Appendix C: Test Plan—Describes user types, team roles, test
objectives, and methodology for recruiting participants.
Appendix D: Participant Recruiting Materials—Pre/Post
task/test questionnaires, screener questionnaire
Appendix E: Testing materials—facilitator scripts, consent
form, scenarios
Appendix F: Session Logs—information and notes on each
individual usability test conducted.
Methodology
Overview
This




section of the report provides information on the following:
Typical user profiles for the Holiday Inn China website
Criteria by which we judged the usability of the site
Goals for the test
Profiles of the actual test participants
8



Test location and lab equipment
Tasks presented to participants during testing
Other materials used throughout our testing process
User Profiles
We based our search for users on the two personas we created. These
personas were created based on the information we were given about the
site’s actual users.
The situation of testing a Chinese website within the United States is
somewhat difficult to realistically accomplish. Our users speak and read
English, although many of the Holiday Inn China’s users do not. Because of
this, the test results only represent one of the user groups. The
characteristics of this user group, along with other questions about prior
experience, behaviors, and other criteria, made up a screening questionnaire
(Appendix A) for identifying suitable test participants.
Here is a description of an ideal user: someone who is 45 years old and has
an eight years old son. The family has been in the United States for about 13
years. She works part-time in her husband’s computer shop. The family
income is about $90,000 a year. She is not very comfortable using
computers and internet. She use internet for email and information. She likes
travel and has booked hotel rooms online before. She wants to go back to
China this summer for Olympic Games and visiting her relatives.
To recruit participants similar to the primary persona the team created, we
developed a general description of the user profile characteristics, as follows:
Must be able to speak and read Chinese
Travel purpose: either for business or for pleasure
Age: 20-55
Gender: female and male
Experience booking a hotel room online
Use internet at least 5 hours per week
No experience with the Chinese Holiday Inn website
A screening questionnaire was created for selecting participants from the
target user group and it is included in Appendix D.
Usability Criteria
When we conducted the heuristic evaluation of the website, we applied
Jakob Nielsen’s 10 heuristics adapted for the Web (as cited in Dr. Carol
Barnum’s text, Usability Testing and Research).
9
Detailed information about our heuristic evaluation can be found in
Appendix A: Heuristic Evaluation.
From the heuristic evaluation, we determined the issues we felt would
be problems for users. These became the basis for our scenarios.
Test Objectives
We combined our goals and our sponsor’s concerns to create our test
objectives. We created post-task and post-test questionnaires to assess the
ease or difficulty of the following hotel booking tasks on
www.HolidayInn.com.cn:
 The general feeling/layout of the site: Is it easy to find the information
users want to know the first time they visit?

The procedure of booking a hotel online:

Basic searching: Is it easy to use?

Advanced searching: Can users accomplish their goals on the
advanced search screen?

Entering personal information: Does the website require
reasonable and suitable information for Chinese users? (e.g. U.S.
uses State, China uses Province; Americans have SSN for social
security number, Chinese have IDNO for Identity Card Number ).
Do the users understand all information requirements?

Language: Can users understand the language used on the Holiday
Inn website? Are there any translation mistakes or
misunderstandings? (Such as: “king size bed” translated in Chinese to
a very big size bed.

Logging in to Priority Club: Is it easy for users to find the log-in page?
Does the website offer the benefit information of being a member of
Priority Club? Is the information clear?

Information in confirmation email: How confident are users in
receiving a confirmation email? Does the confirmation email contain
the information users need?

Satisfaction with the site: Which aspects do users like and which
aspects do they dislike?

Perceived reliability of the site: Do users trust the website? Do they
fill out the personal information readily?

Navigation: Can users find the most efficient navigation when
booking a hotel or browsing the website?
10
The test was designed to allow us to extract mostly qualitative data.
Participants were given a pre-test questionnaire to get additional information
about their experiences booking a hotel reservation online, then given
scenarios that directed them to perform specific tasks, and lastly a post-task
questionnaire designed to extract detailed and specific feedback about
www.HolidayInn.com.cn features.
The test was conducted in a full-scale usability lab. Users were given 40
minutes to complete five scenarios. We also designed an extra scenario in
case there was some time left.
Data Collection
The evaluation measures are a combination of quantitative measurements
and subjective observations. The quantitative and qualitative data measures
are listed below.
Quantitative Data
The following data were recorded when we carried out the Holiday Inn web
site usability test:








Time to find an available hotel
Number of participants completing each task
Number of problems encountered during each task
Number of participants that failed a task
The participant’s rating of level of difficulty in booking a hotel room on
the Holiday Inn site
The participant’s rating of level of confidence in booking a hotel onthe
Holiday Inn site
The participant’s level of reaction to the look and appeal of the website
A comparison of satisfaction ratings for using the competitor site
vs.the Holiday Inn website
Qualitative Data


Facial expressions
Verbal comments
o Positive
o Negative
o Questions asked
o Nature of calls for help
Screening Profiles
Including the pilot test, we collected the following data from our six
participants.
Pilot User
1st
11
2nd
3rd
4th
5th
Age Range
Gender
User
User
User
User
User
25-35
25-35
46-55
20-25
20-25
36-45
F
F
M
M
F
F
Accountant
Real
Estate
Electronic
Engineering
student
Architecture
student
Manage
ment
X
X
Computer
Science
Student
Occupation
Never
1. In the past
year, how often
have you
booked a hotel
online?
2. How often do
you use the
internet?
1-3 times
More than 3
times
X
X
X
X
Less than two
hours a day
2-5 hours a
day
More than 5
hours a day
X
X
X
X
X
X
X
X
Less than 5
hours a week
3. What do
you mainly
use the
internet for?
Circle all that
apply
4. How many
nights have
you spent at a
hotel over the
last six
months?
Entertainment
X
Information /
Research
X
X
X
E-Mail
X
X
X
Social
Networking
X
Other
Business
(Stocks)
X
X
7
3
X
X
X
12
2
more than 5
X
X
X
X
X
More than
20 nights
3
5. How many
of those
nights over
the last six
months were
for:
Business
_____
0
0
0
0
Pleasure
_____
7
3
2
5
hotel.com
Hotels.
com;
expedia.
com;
Marriott
and Hilton
6. What
websites have
you used the
most to book
hotel
reservations?
7. Do you
mostly book
hotel
reservations
for yourself, or
does someone
else do it for
you?
8. Finish the
following
sentence:
When it comes
to providing
my credit card
number
online:
a. I do not
mind
b. I will do it,
if absolutely
necessary
c. I don’t give
out credit
card info
online
I booked
most hotel
nights online
by myself. I
travel with
my friends,
sometimes I
did it, and
sometimes
they did.
Yes, I book
hotel for
myself
X
X
Chaoiceh
otel.com
NA/A
1
More than
20 nights
2
www.bestrid
ehotel.com
www.hotel.c
om
Traveloc
ity or
hotel's
own
website
by myself
I do it
myself, and
I oftentimes
booking
online for
my aunt.
For
myself
and for
others
X
X
X
X
Test Environment and Equipment
The test was conducted at the Usability Center at Southern Polytechnic
Usability Center. The usability center includes three rooms:
Evaluation Room
The evaluation room is where participants complete tasks. The room has the
following equipment:
13
SPSU-standard PC configuration (Windows XP, campus network
connection)
Keyboard and mouse
19” Monitor
Cameras
Microphone
Telephone/intercom between control room and evaluation room
Control Room
The control room is where the team observed and recorded the test. There is
a one way mirror between the evaluation room and the control room. The
control room has the following equipment:
A standard PC with Morae logging software
Monitors which display what the cameras are capturing in the evaluation
room
Headphones and microphones for communication among the team
members
Video recording and editing equipment
A picture-in-picture digital mixing board
A phone/intercom for the participant to call the help desk
Lighting controls
Executive Viewing Room
The Executive viewing room provided opportunities for additional observers
watching the test without interrupting the team. The executive viewing room
has the following equipment:
SPSU-standard PC configuration
Monitors
LCD Projector
Test Scenarios/Tasks
Overview/Briefing (10 minutes)
The facilitator welcomed the participant and asked him/her sign the video
consent form. Next, the facilitator explained the facilities, introduced the
team, and asked the participant to think out loud. Also, the facilitator worked
with the participant to complete the pre-test questionnaire. All test materials
are included in Appendix D.
14
Scenarios are included here, as well as in Appendix D, to provide the context
for analyzing the findings, which follow.
Scenario 1: Look and Feel (5 min)
You and your family (if you are not married, it can be your friend) are
thinking of going back to China for the summer Olympics in Beijing.
You heard that the Holiday Inn has several locations in Beijing. You will visit
the Chinese Holiday Inn website (www.holidayinn.com.cn ).
Take a moment to look at the homepage without clicking on anything. While
looking at the homepage, please express what you think of the website and
what you might do to book a hotel room.
Scenario 2: Book A Hotel Room (15 min)
You decide to book a hotel room for yourself and another person coming with
you on the trip to China. Your booking dates are between July 21st and July
25th. Here is the information you may need:
•
•
•
•
•
•
•
First Name: Jing
Last Name: Li
Location: anywhere in Beijing
Check in date: July21st, 2008
Check out date: July 25th, 2008
Credit Card Information: Visa, card number: 4111 1111 1111 1111,
expiration date: 08/2009, card verification number: 635
E-mail Address: CBRholiday@hotmail.com
Please write down the confirmation number when you completed the
reservation.
Scenario 3: Cancel a Hotel Reservation (5 min)
You just received a phone call from your relative who lives in Beijing. She
wants you and the person traveling with you to stay at her house during the
visit.
You accept the invitation and decide to cancel your Holiday Inn reservation.
Go ahead and cancel your reservation.
Scenario 4: Find a Holiday Inn in Changchun (5 min)
You have some relatives in Changchun, Jilin province. You want to go visit
them during your vacation.
15
See if there is a Holiday Inn available in Changchun from July 26th to July
30th.
Scenario 5: Book a Hotel Room at another Site (10 min)
Go to the website www.elong.com.
See if a hotel room is available for you and the person going with you
between July 21st and July 25th in Beijing.
Note: You are not going to actually book a hotel room on this website. Stop
when you get to the last step in completing the booking process.
Scenario 6: Look and Feel of Holiday Inn English Site (Optional)
(5 min)
Go to the Holiday Inn English-language website (www.holidayinn.com).
Take a moment to look at the homepage without clicking on anything.
While looking at the homepage, please express what you think of the website
compared to the Holiday Inn Chinese site and eLong site.
Closing (5-10 minutes)
Participant completes the post-test questionnaire.
Findings/Recommendations
Positive Findings
Below are the positive results that the users wanted to make sure were
known.
Positive Impressions:



“I like the way they set up the website, it is very clear.” (U4)
“This website is easy to understand, it should not take long for me to
work on it.” (U2)
“All the options I need are right here in the center of the page” (U2)
initial reaction to home page
16
Homepage hotel search: Four out of six participants felt that the
homepage’s hotel search box was very useful. They felt that it was
convenient that it was located on the homepage and were glad that they did
not have to search around the site to find this section. Most users said it was
straightforward to use, easy to find, and easy to understand. To quote one of
our users (U4), “I like the design, it is very clear.”
Homepage is easy to navigate and understand.
Information is centered which makes the page easy to use
17
Calendar feature: Three out of six participants mentioned that they were
glad a calendar feature was integrated into the hotel search forms. Most
users found it convenient because rather than having to type in a date and
figuring out what format to put it in, they were able to look at the calendar
and just pick the days they wished to stay and the website would take care
of all the formatting.
Drop-down menu is well-received by the users
Reputation lives up to its name: Two of six participants felt strongly that
because the site was Holiday Inn, they knew they could trust the website and
also knew what kind of standards Holiday Inn has.
Brand variety: Two of six participants were happy to see that other well
known hotels were displayed in addition to just Holiday Inn hotels. One user
chose Crown Plaza while another chose Holiday Inn Express. It allowed the
users to feel that there was a healthy variety of hotel chains to pick and
choose from.
Other well-known hotels that are part
of the IHG brand are displayed in
addition to Holiday Inn
18
Severity Ratings
After our team compiled all of our findings, we assigned a severity rating to
problems that our users encountered. We divided and defined our severity
ratings into the following categories:



High: Findings where the majority of our users experienced the same
problem. Also, high severity ratings were labeled when users
encountered an obstacle or difficulty preventing them from finishing
their task.
Medium: Findings when two to three of our users encountered the
same problem. The medium rating was also applied when elements of
the site frustrated the users but did not keep them from completing
their task.
Low: Findings that either could be problematic or were confusing at
first. The findings were not considered difficult enough to be a
problem, but frustrating enough to deserve attention.
Collated findings, rated by severity
Based on the feedback we received from our participants as well as our
analysis of the quantitative data from testing, we identified 14 significant
usability issues with the Holiday Inn China website. The following table lists
these issues and indicates their impact on the usability of the site, along with
the number of participants who experienced each problem.
Usability Problem
Navigation: The ‘start over’ and
‘finish booking’ options on the
confirm booking page confused the
user
# Affected
Severity
1/6
low
Layout: Users found the Priority
Club Login options confusing.
4/6
high
Layout: Users think
advertisements are overwhelming
3/6
medium
Layout: Users want the total price
for their stay to be listed, instead of
a per-night rate only
3/6
high
19
Layout: Users have trouble finding
the cancellation link to cancel their
reservation
3/6
high
Layout: Error messages were not
seen by the user or they were
misinterpreted
5/6
high
Aesthetics: Search results of
hotels are displayed in a two-color
format, confusing users’ perception
of how many results are actually
being displayed
2/6
low
2/6
low
3/6
high
4/6
medium
4/6
high
5/6
high
6/6
high
Feature Requests: Users
complain about not seeing
promotional rate information
Feature Requests: Users want to
see a drop-down list of cities
instead of having to type city
names in
Feature Requests: Users want to
see hotel rankings by different
categories (customer experience
and price)
Feature Requests: Users want
more information on hotel
amenities
Mental Model: Users are confused
as to whether or not they are
required to input full Chinese name,
or just family name
Feedback: Loading screen has no
progress bar. This confuses users
as to whether or not their
information is being processed
20
User Impressions
Qualitative feedback is indicated by a sample of the general comments we
received from users about their experiences and preferences. The complete
set of comments can be found in the test logs, Appendix F.
General comments:




“Are they sister hotels?”—User 2, reaction to the different hotel logos
at the bottom of the page
“Why is the English name needed?—Pilot user (P), reaction to required
field on reservations page
“What is Priority Club number?”—User 1
“Where is the cancellation button?”—User 1
Negative Impressions:








“I would go to another hotel reservation site at the point.”—User P,
reaction to confusing error messages
“It is not as easy as booking.”—User 4, reaction to cancelling
reservation
“So many advertisements.”—User P, reaction to home page
“Loading page is too slow, too boring.” –User 4, reaction to blank
loading page
“I am waiting again, most websites have a progress bar.”—User 4,
reaction to blank loading page
“So many ads on this website makes me uncomfortable.”—User P
“I don’t want to have to type in the city name, there should be a
dropdown menu.”—User 1
“I am not satisfied at all.”—User 1
Suggestions:




“I think they should use Ming (first name) instead of Mingzi (first and
last name). It’s so confusing.”—User 5, reaction to Chinese name input
field
“I want to know the total price instead of the per night price.”—User 5
“They should show pictures of the actual hotel—both inside and out.”—
User 2
“I want to know what kind of services they provide, such as breakfast,
prices, user ratings…”—User 2
Recommendations
Based on the findings of our usability test, we make the following
recommendations to improve the Holiday Inn China website.
Recommendations are based on the severity ratings for the problems users
encountered.
21
“Sort by” option (feature request)
Four out of six participants showed interest in the competitor
website’s option to arrange search results by price range and/or
by user comments. On the Holiday Inn China website, users do
not have a “sort by” option. User’s verbalized their desire to
have an option that allowed them to sort their search results by
different categories.
Recommendation: Create a “sort by” option that includes at
least price range, and user ratings
The competitor site, elong.com, offers a sort-by option that allows users to
customize their search options
Idle Loading screen (aesthetics)
Five out of six participants complained that the page displayed
while results are being compiled has no progress bar. They are
confused as to whether or not their request is being processed.
Recommendation: Add a progress bar to the loading screen
or display a message that informs the user that his/her request
is being processed.
Error Message (layout)
Error messages were not seen or they were misinterpreted by
the user. When trying to find a hotel in Changchun, Jilin
Province the error message reads “No hotel matches your
requirement,” when in fact there is no Holiday Inn in
Changchun.
On the cancellation page the user is required to enter his/her
last name and first name separately. However, most users
entered their first and last names together. The error message
on the cancellation page says “The name you entered does not
22
match the name you registered with.” This does not help the
user figure out what he/she did wrong.
Recommendation: First, highlight and display the error
message next to the incorrect field. Secondly, provide clear
instructions as to how to fix error.
The error message does not specify exactly what the problem is
Hotel Amenities (feature request)
On the search results page, users complained that there were
no pictures of the interior of the hotels. Five out of six
participants were not aware that they could click on the hotel
pictures to find additional pictures of the hotel. Nor did they see
the link to additional pictures. Users also complained about not
having enough information about the hotel’s services offered.
Recommendation: Make the link that leads to hotel pictures
more visible. List more amenities of the hotel, such as pool and
food service provided.
Input Name Field (mental model)
On the registration page, users are asked to input their English
name and their Chinese name. Most users just put in their
English name because it’s a required field. On the cancellation
page most users input their full name into the field that only
requires a last name. The Chinese audience is not used to
entering their first and last names separately. The phrase used
for “family name” on the website looks similar to the phrase
“first and last name.” This also adds to the user’s confusion.
23
Recommendation: Allow the user to enter his/her first and last
name into one field, instead of requiring the user to enter
his/her first name separately from his/her last name.
The Chinese name input-field is not clearly translated. Users are confused as
to what they should do at the point.
Advertisements (layout)
Three out of six participants found the advertisements
throughout the website to be distracting and irrelevant to the
purpose of the website. Many of them said the advertisements
were overwhelming. When they saw the English Holiday Inn
website, they said the advertisements were appropriate.
Recommendation: Reduce the number of advertisements on
the website and make the pictures relevant.
24
Advertisements are reported to be overwhelming
Drop-down city menu (feature request)
Several users expressed their desire to have a drop-down menu
that provides the names of cities in the region they are
searching in. They communicated that having to type the city
name made them feel that they were wasting time. They were
also concerned that they may have misspelled city names.
Recommendation: Create a drop-down menu that lists city
names so that users have less to type in the input fields.
Cancellation Link (layout)
Users had trouble finding the cancellation link to cancel their
reservations. Most of the users communicated that they would,
under normal circumstances, leave the website and call the
hotel directly.
Recommendation: Put a cancellation button on the main
page.
Price Listing (layout)
Users want the total price for their stay to be listed, instead of a
per-night rate only. Users had trouble finding a total price for
their reservation.
Recommendation: List the total price that guests will pay,
and not just a per-night price.
25
The per-night listing stands out, while users must click another link to view
the total price for their stay

Priority Club Login (layout)
Four out of six users were confused as to whether or not they
had to log n as a Priority Club member to even use the website.
They were not aware of what Priority Club is. On the
cancellation page the Priority Club login option appears first and
confuses users.
Recommendation: Reduce the size of the Priority Club login.
Other Findings
In addition to the findings that were based on our severity rating system, we
had two additional categories:

Miscellaneous Findings: Findings that were labeled as miscellaneous
were for problems that users encountered that we did not expect.
26

Localization Findings: All of our users have been in the United
States for at least two years and upwards of twenty years. While being
here, they have picked up on certain aspects of how American sites
and business operates. Also, they have English speaking skills and can
translate Chinese to English when using the site. In contrast, a native
Chinese person may not have any familiarity with English. Localization
findings have been grouped for potential problems that a native
Chinese person or audience may encounter.
Miscellaneous Findings
Confirmation checks before finalizing hotel reservation: Our second
participant missed the area where a confirmation check was required to
finalize the hotel reservation. The area being mentioned is shown below:
Confusion from “nights” section of the hotel search box: Our second
user initially entered in his stay range with the built-in calendar feature. After
entering in the dates, he then continued to the “nights” dropdown box and
picked the number he felt corresponded with the duration he would be
staying. The choice the user made did not match with the dates he selected
and it changed his date range without him noticing it. After he did notice this,
he thought that the addition of the “nights” stayed drop-box seemed
redundant and confusing. This user was the only one to actually try selecting
the “nights” they would be staying. The rest of our users only used the builtin calendar feature. The following screenshot shows the area where the
problem occurred.
Interim option: Our first participant made a mistake during the
confirmation page and chose “start over” rather than “confirm reservation.”
While this was accidental, we think it might be good to have an additional
option in this area for modifying one’s hotel reservation details. This could
27
potentially be useful for allowing users to go back and make a change to
something rather than having to start over from the beginning of the hotel
search option.
A user can use the back button on the browser to make changes, but
depending on the user’s local settings, the information might not be saved
and the user would have to re-fill out all the information again. The
recommended change is shown below.
Inclusion of a search feature: Our first user felt that it would be nice if a
search feature was added. This could be a benefit to users seeking Holiday
Inn hotel locations in different cities.
Localization Findings
Returning Home: Many Americans are used to clicking on a logo or banner
towards the top of a website to return to the homepage. There was
confusion among several users about how to return to the homepage, but
most tried clicking the logo to return to the homepage. However, one of our
users was uncertain as to how to return to the homepage; rather than
clicking the Holiday Inn logo, the user clicked the “back” button on the
browser until reaching the homepage again.
It might be helpful to include a “Home” or “Return to Homepage” button in
the navigation area for those who are not accustomed to clicking logos to
return to the homepage.
Submit button did not appear clickable: One of our participants was
uncertain as to how to initiate the hotel search from the homepage. The user
eventually found the button (see below) to initiate the search, but it took
significantly longer than the rest of our participants.
28
We do not have a clear recommendation for this issue but think it might
either be worth looking into or modifying the area to make it stand out more.
Trust symbol confusion: One of our users was confused about the “Trust”
image found on the homepage (shown below).. Most native Chinese people
prefer to use cash when making a transaction, so graphics that Americans
might recognize as being “security checks” will not be as easily recognized by
native Chinese residents.
We do not have a recommendation for this but feel that it might be worth
looking into further.
CNY currency confusion: One of our participants was uncertain about what
“CNY” stood for when placed before the numerical value. CNY designation is
known internationally for Chinese yen; however, native residents of China
would be able to identify better with the Chinese symbol for yen.
Our recommendation would be to include the character and/or symbol for
yen in addition to “CNY” so that native users would not be confused about
what currency type is required.
Character
Symbol
29
Post Task Questionnaire Analysis
Four out of our six participants were asked follow-up questions based on the
experience they had with the scenarios we presented to them. Below are
questions we presented to them, in addition to tables that show the answer
sekected by them. The first row of the tables below are the answer selections
presented to the users and the second row identifies which user chose that
answer choice. (e.g., When “#4” is shown under an answer choice, it means
our fourth user selected this answer choice.)
1. Reaction to the look and feel of the website
One of the questions we asked our users was to rate their reaction to the
look and feel of the website. This was asked after first introducing the users
to the Chinese site’s homepage and letting them look around the page. The
responses were gauged based on how visually appealing they found the
website as well as how they thought the “feel” of the site was.
Very
Appealing
Somewhat
Appealing
#4, #5
Neutral (No
opinion)
Somewhat
unappealing
Very
unappealing
#2, #3
2. Interest in continuing the booking process
After we introduced the site to the users, we wanted to find out their interest
in continuing to look for a hotel based on their first impression of the site.
Since first impressions usually will determine if a user wishes to continue
using a website, we felt it would be appropriate to find out if users would
continue trying to book a hotel based on what they have seen so far.
Very
Interested
Somewhat
Interested
Neutral (No
opinion)
#2, #3, #4,
#5
30
Not Very
Interested
No Interest
at all
3. Features users enjoyed
After our users completed the task of booking a hotel room, we asked them
to list any features that they enjoyed during the booking process.




Our second user liked how the website provided information about
if the hotels served breakfast or not.
Our third user said it was easy to tell which hotels had rooms
available and which hotels did not.
Our fourth user liked the way the website was set up. It was very
clear to her.
Our fifth user liked the map that was shown after performing a
hotel search. (This feature was added in later during our testing
time and only the fifth user was able to see this feature.)
4. Features users disliked
In addition to listing the features users liked, we asked them to list any
features they did not like during the booking process.




Our second user felt that the site’s performance was a little slow.
The third user did not like that a picture of the room they booked
was not available for them to see.
Our fourth and fifth users did not see the area saying what the total
price would be for their stay on the confirmation page. User 4
added up the total because it was not clearly visible.
Our fifth user felt misinformed when viewing one of the hotel
choices displayed from her search results. The hotel initially said
that rooms were available that met her criteria, but after continuing
to the booking process it said that no rooms were available.
5. How easy/difficult was cancelling
After users went through our scenario to try cancelling their hotel
reservation, we asked them how easy or difficult the process was for
them.
Very Easy
Somewhat
Easy
#3, #5
#2
Not Easy or
Difficult
Somewhat
Difficult
#4
31
Very
Difficult
6. Level of confidence in being able to cancel
After asking the users about how easy or difficult they found cancelling a
reservation, we continued with a follow-up question to rate their
confidence level of being able to cancel a reservation on their own without
assistance. The rating was based on a scale of 1 to 10, 10 being the most
confident. The top row in the table found below is the numerical
“confidence” value. The row underneath is how many users identified with
number choice shown above.
1
2
3
4
5
6
7
#5
8
9
10
#4
#3
#2
7. Explain confidence rating
After asking our users to choose a numerical value representing their
confidence level, we asked them to explain why they chose that value.
a. The fourth user chose ‘8’ because although she failed to perform
the action online, she was confident that she would just be able
to call in to cancel her reservation.
b. The fifth user said that she would feel very confident about
cancelling a hotel as long an e-mail is sent confirming her
cancellation.
8. How easy/difficult is it to find pertinent information
Next we asked users to rate how easy or difficult it was for them to find
information that was pertinent to them on the website. (This question was
only addressed to two users.)
Very Easy
Somewhat
Easy
Not Easy or
Difficult
32
Somewhat
Difficult
Very
Difficult
#4
#5
9. What information would you like to see on the website?
For users that felt it was difficult to find information on the site, we asked
a follow-up question asking what information they would have liked to see
on the website.
a. Our second user would have liked the search results page to
show cities that have hotels available when a hotel cannot be
found in a city (e.g., In Changchun there are no hotels
available). Our user would have liked the results to say
something along the lines that no hotels are currently available
in Changchun; however (nearest city) has hotels available.
Would you like to view the hotel listings for (nearest city)?
b. Our fifth user encountered an error message that others did not.
Although other users encountered error messages. the fifth
user’s error message appeared in the form of a technical error.
Instead of an error message saying what the user did wrong,
the error message displayed a coding error with no explanatory
information. (The error message was this: U_2008_AN)
10. Reaction to the look and feel of the elong website
We also asked our users to view the www.elong.com website. Just like the
first question where we wanted the users to rate their reaction to the “look
and feel” of the Holiday Inn China site, we asked them to rate their reaction
to the “look and feel” of elong’s website.
Very
Appealing
Somewhat
Appealing
Neutral (No
opinion)
#5
#2, #4
#3
Somewhat
unappealing
Very
unappealing
11. Reaction to the look and feel of the English Holiday Inn website
When time permitted, we asked users to view Holiday Inn’s English-language
website. We asked them to rate their initial impression to the “look and feel”
of the English site. This question was addressed by only two users.
Very
Appealing
Somewhat
Appealing
Neutral (No
opinion)
#4
Somewhat
unappealing
#3
33
Very
unappealing
12. Comments concerning all three websites
For those users who visited the Chinese Holiday Inn site, the English Holiday
Inn site, and elong’s website, we asked them to share any comments they
had concerning the three sites.






Our second user felt that Holiday Inn’s English website would be the
easiest site for him to book a hotel after we had exposed him to the
Chinese site and elong’s website.
Our second user liked that elong gave users a city selection box rather
then having to type in a city name. Another reason the user liked this
is because they did not have to worry about entering in a city name
that may be misspelled, thus not showing the user any results because
of their spelling error. The listing of cities prevents a spelling error
from occurring.
Our third user said that most native Chinese residents are not familiar
with booking a hotel room through an online service.
Our third user did not like how many options were presented to them
on the elong site.
Our fourth user did not like that membership logins were shown on
each page for the Holiday Inn sites.
Our fourth user thought that all the advertisements across the three
websites she saw were distractive.
34
Conclusion
The usability course that our team has taken while conducting this project
has been an extremely informative experience in portraying how usability
testing is performed. While conducting this project, we have not only gone by
guidelines and principles taught during the course, but we have also received
hands-on professional input from our instructor Dr. Carol Barnum.
The team for this project consisted of six undergraduate students pursuing
degrees in the Technical Communication major at Southern Polytechnic State
University. Four of the team members were Chinese exchange students and
the other two members of the team were local American students.
The team conducted six testing sessions in SPSU’s state-of-the-art Usability
Center. Following the completion of testing, the team collectively gathered
data and organized it based on methods taught during the usability course,
as well as methods described in Dr. Carol Barnum’s book, Usability Testing
and Research.
Findings
After analyzing logs and recordings made during our research sessions, we
divided our findings into seven different categories:







Layout
Navigation
Aesthetics
Feature Request
Brand Identity
Mental Model
Feedback
Recommendation
The six users that we received feedback from all have lived in the United
States for at least two years. In the findings section of our report, we
identified issues that could be problems for Chinese language speakers using
the website.
We recommend that further testing be conducted to:

Confirm that recommended changes for English-speaking, U.S.
based Chinese users improve their experience
35

Test the changes with a study of Chinese-speaking users to confirm
that these changes are effective for native Chinese users of the
website and to uncover other areas that need to be address for
these users.
We are confident that the research performed in our usability study will be
helpful for making improvements to the Holiday Inn Chinese website. In
addition, we strongly encourage additional testing for any future
enhancements to the site.
36