Rochester Institute of Technology Mobile Website
Final Testing Report
Usability Testing 4004.749.70,
Rochester Institute of Technology, Rochester, NY
By:
Rahul Edla
Rohan Reen
Pearl Somani
Julian Strothers
Kunya Wongpaisansin
1
Introduction
The usability test was conducted on the RIT Mobile Website. The RIT Mobile Website
consists of a group of various features including Maps, News, Dining Services, Shuttles, Events,
and Lab Hours. Our main focus in this test was to check the user adaptability, navigation,
information architecture of two categories: Dining Services and Shuttle/Bus Schedule.
The test is being conducted to determine design inconsistencies and usability problem
areas within the user interface and content areas. The RIT mobile site has been available since
November 2011 and there has not been any feedback taken from the users since that time. The
objective of the test is to get useful feedback from users. This feedback can be utilized by the
development team to help them implement changes or modifications to improve the overall
functionality and usability of the mobile site. The goal is to find whether users are able to find the
information they want and how easily they are able to find it. We intended to focus on a few
potential sources of error like:
● Navigation errors - failure to locate functions, excessive keystroke to complete a
function, failure to follow the recommended screen flow
● Presentation errors - failure to locate and properly act upon desired information in
screens, selection errors due to labeling ambiguities
● Control usage problems - improper toolbar or entry field usage
Based on the results of the usability test, we have proposed some recommendations whose
implementation, if possible, could result in a more user friendly application.
Test Methodology
Overview
The test was conducted in the Usability Testing lab in the Golisano building at RIT. Each
user was provided with a specific 30-minute time slot. We sent screeners to the users with the
availability of slots. Once the interested users responded, we registered them for different slots.
Our test procedure consisted of the following steps:
1. User greeting & introduction
2. User Consent Form
3. Pre-Test Questionnaire
4. 4 Task Scenarios
a. Present Scenario
b. User Attempts Task
c. Post-Task Evaluation
5. Post-Test Questionnaire
6. Post-Test Debriefing & Interview
7. Thank User
Target Users
The RIT mobile website is intended for a wide range of users, including both faculty and
students. In order to make the testing effective we wanted users from each of the stakeholder
groups. We also wanted users in different age groups in order to study different user
perspectives. Users were recruited with these considerations in mind.
Recruiting
2
User recruitment was done primarily through the distribution of promotional flyers.We
received responses from many students as well as a couple faculty and staff members. After a
study of background and feasibility to the usability test, we narrowed down to ten users.
Pre-Test Questionnaire
Before attempting any tasks, users completed a pre-test questionnaire. The main
purpose of the questionnaire was to gain information about the user demographics and prior
user experience with the mobile site. If anyone has used the mobile website before, questions
about their familiarity with the application were also asked. Questions included in the pre-test
questionnaire were:
● Demographic Information
○ Gender
○ Age Range
○ Position at RIT (Student, Faculty, etc.)
○ Student Type (Full-time, Part-time, etc.)
○ Major
● Exposure to RIT mobile website
○ Experience level with mobile website
○ Features used
○ Most used feature
○ Ease of finding information within used features
Tasks
The four tasks developed for the test were introduced to users within the context of a
certain scenario. This would hopefully help to emulate how a user would utilize the mobile site
in a real-life situation. The first and last tasks pertain to using the dining services feature, while
the other two pertained to using the shuttle(bus) feature. The scenario with integrated tasks are
as follows:
Task 1 Scenario:
You are at the University Commons West bus stop. You are supposed to meet up with your
friends on campus at Brick City Cafeteria.
●
●
●
Task 1
What bus do you need to take to get you close to Brick City Cafeteria?
What time is the next bus scheduled to arrive at the University Commons West stop?
What time will it drop you off at your destination?
Task 2 Scenario:
While on the bus, you want to check Brick City Cafeteria's menu.
●
Task 2
Find Brick City Cafeterias lunch menu for today.
Task 3 Scenario:
After seeing Brick City’s menu and meeting up with your friends, you try to convince them to go
to Beanz cafeteria instead. Your friends, however, don’t even know if Beanz is still open.
Task 3
3
●
Find the weekday operation hours for Beanz cafeteria.
Task 4 Scenario:
You then decide to go shopping with your friends at Marketplace Mall. When you're done
shopping, you decide to head home.
●
Task 4
Find the schedule for buses going to UC West from Marketplace Mall in the next hour.
After each task attempt, the user scored the mobile site from 1 to 5 in four categories:
● Ease of task completion
● Ease of navigation
● Usefulness of information
● Visual presentation
Team members also took notes of any user responses while attempting tasks.
Post-Test Questionnaire & Debriefing
After completing all four tasks, users scored their overall experience with the mobile site from 1
to 5 in five categories:
● Ease of finding information
● Usefulness of information
● Layout of information
● Ease of navigation
● Aesthetic look & color
Users were also asked questions pertaining to the following areas:
● Difficulty with tasks
● Regular use of buses & shuttles
● Presentation & layout of shuttle(bus) feature
● Comparison of mobile site to other public transportation resources
● Presentation & layout of dining services feature
● Confusing or unfamiliar terminology
● General comments about website design
Participants
Ten users were tested in total. Demographic information for these ten users is summarized
below:
Demographics
Gender
Male
8 participants
Female
2 participants
Age Group
21 to 23 years
6 participants
4
24 to 26 years
2 participants
30+ years
2 participants
Position
Undergraduate
2 participants
Graduate
6 participants
Faculty
1 participant
Staff
1 participant
Type
Full -Time
*including faculty & staff
8 participants
Part -Time
2 participants
User Experience
Never Used
8 participants
Once or Twice
1 participant
Sometimes
1 participant
While two users replied that they have used the mobile website before, even they had very little
exposure to the mobile site. This means that we were able to witness the idea of a user’s out-ofthe-box experience.
Results
Task Success Rates
A success rate was calculated for each task. A success was defined as a user being
able to gain the majority of information needed for the task on their own, even if a couple minor
hints or clarifications were provided. Therefore, a failure meant that the user was not able to
gain enough information from the site to help them complete the required task without a large
amount of help from a team member. A chart for the calculated success rates can be found
below:
5
Dining Services (Task 2 & 3)
The success rates for the tasks pertaining to the dining services features were perfect.
Not one user had any problem completing either task. All of the users also completed these
tasks in a very timely manner. Almost no difficulties were encountered when it came to finding
the correct information for the tasks.
Shuttle/Bus (Task 1 & 4)
The success rates for the tasks pertaining to the shuttle feature are significantly lower
when compared to the dining services tasks. Six out of ten users had success in task 1, while
only four out of ten users successfully completed task 4. Also, the majority of users that did
succeed in completing these tasks still needed some amount of assistance or clarification to
help them.
Task Score Averages
Users had the opportunity to score the mobile site in four categories after completing
each task. The average scores for all tasks are summarized in the graph and chart below:
Dining Services (Task 2 & 3)
Nearly all of the scores for the dining services tasks were above a 4.5, with the majority of them
being very close to a perfect 5. This mainly correlates with the 100% success rates that were
6
recorded for these tasks. While still holding high scores, the visual presentation category did do
the worst, scoring a 4.4 for task 2 and a 4.6 for task 3.
Shuttle/Bus (Task 1 & 4)
When compared to the dining services tasks, the shuttle tasks recorded lower scores in every
category. Most of the scores lie in the low 3 to low 4 scoring range. For task 1, the lowest score
was in the ease of task completion category with a 3.4. For task 4, both the ease of task
completion and usefulness of information categories had very low scores with a 2.9.
Post-Test
Score Averages
During the post-test questionnaire, users scored the mobile site in five categories. The
average scores for these categories are charted below:
Overall, the average scores fell within a close range of each other, with all scores being
between 3.5 and 4.5. The lowest score was in the ease of finding information category,
receiving a 3.6.
User Responses
Dining Services
Overall, users were satisfied with the functionality and design of the dining services
feature. While completing the dining tasks, almost all of the users found finding information to be
very simple and straightforward. The interface for dining services was neatly organized and
easily understandable. Most users thought any information they would need about a dining
location was available with very minimal effort on their part.
For the most part, users were satisfied with what they got out of the dining services
feature. A few users did point out the fact that the list of dining locations could only be displayed
in alphabetical order. Many users suggested that the list have the ability to be sorted by type of
cuisine, location, or closing time. Another user wished there was an easier way to determine
which dining locations were open at the time of using the mobile site.
Shuttle (Bus)
The shuttle feature seemed to be the major area of concern in the usability test. While
there were many comments from users about what could be changed or improved, there were
also some positive aspects that were pointed out. A few users praised the convenience of
having RIT and RTS bus schedules available from a mobile device. Users also found navigating
bus routes and scrolling through stops was very simple.
Almost all users, however, were often confused or lost while browsing through the
shuttle bus routes. Many users were not sure which route they should take to get to their
required destination as well as which bus stop would be the closest. For example, there is no
7
chance for the user to be aware of the fact that they have to board the Racquetball shuttle for
heading toward University commons. Some users’ familiarity with the bus schedules on the
official RIT website made for a few difficulties. Many users were used to the official website’s
horizontal layout, which differs from the mobile site’s vertical layout. Due to this difference,
users took a short while getting accustomed. A couple users complained about the large
amount of blank cells in some of the listings. They felt it only made the list unnecessarily long
without adding information. Overall, users seemed to want more context about the bus routes
and stops. The current state of the shuttle functionality has a very informational feel, while users
felt they needed a little more interactivity.
Users were also asked about their past experience with public transportation and the
resources they use to aid in finding correct bus routes and stops. Graphs that show the results
of these questions are below:
Do you ride the bus? If so, how often?
What do you use to help you with bus and shuttle transportation?
Most of the users we tested did use the bus/shuttle in a fairly regular fashion. Almost all
of the users that use the bus regularly use the RIT online schedule as their main resource for
information about bus routes. Along with the contrast in layout from the RIT online schedule, a
few users also referred to how Google Maps handles public transportation, which allows users
to simply enter a start and stop location while the program does the rest.
8
Task Difficulty
Users found Task 4 (Bus from Marketplace Mall to UC West) to be most difficult. This data
reinforces the point made with the success rates and task scores.
Recommendations
Dining Services
One of the primary recommendations for the dining
services feature is to add the ability to sort the list
according to user preference. Users expressed the desire
to sort the list by operation hours, cuisine type, and
location. Users should also be able to easily tell which
locations are open from the list depending upon the time
the mobile site is being used.
Potential look after implementing recommendations
9
Shuttle (Bus)
With all of the user feedback received, there are a few aspects to consider in regards to
the shuttle functionality. It may be useful to re-evaluate the naming and grouping conventions of
the bus route and stop listings in order to add more context for the user. The layout of the table
listings could be altered to better match the expectations of users who are used to looking at the
pdf schedules available on the official website. Most users expressed interest in having a more
dynamic solution to finding where they need to go. The Google Maps trip planner can be used
as a reference. Users only have to enter a start and stop location and the planner figures out
which bus or shuttle routes they need to take. Depending on the implementation, an API
provided by google may be able to be utilized in the mobile site to accomplish this. Another
recommendation would be to implement support for bus reminders or notifications. This way, a
user will have the option to be reminded minutes before their bus is scheduled to arrive at their
stop.
Potential of implemented trip planner
Conclusion
In conclusion, the RIT mobile website usability test ended up being very productive. All
of the tasks developed provided a lot of useful feedback and suggestions from the users. We
were able to get a healthy amount of quantitative and qualitative data, from which we were able
to make a few recommendations in both the dining and shuttles areas of the mobile website.
Our initial goal of conducting the usability test was to find areas within the existing application
which could be improved and ways to make the application more informative and user-friendly.
We feel that we successfully accomplished the requested objectives. If the proposed
recommendations are considered and potentially taken, the RIT mobile website can be a much
better user experience.
10