Jeremy Syn
Group 3
Pilot Usability Study
Introduction
This system targets foreigners who are new to a country or area that’s main language uses
an unknown language to the user. It uses the camera abilities of a phone to take an image
of some form of text, whether they’re signs, menus, etc, recognize the text from the
image converting it to machine-readable format, and then translate that piece of text. By
using this system, users will be able to translate words outside their homes faster and
easier than any other methods out there.
The purpose of the experiment that we are conducting serves to give us, the designers,
useful feedback on our prototype about what works and what doesn’t work. It will help
us to catch fatal flaws in our design as well as minor ones. We seek to discover what
elements of our design are confusing to the user, what are interesting to the user, and how
to better improve on them. Our ultimate goal is to make a feasible product that will used
by many users.
Implementation and Improvements
We spent a lot of time improving our design since the Interactive Prototype presentations.
We noticed we had too much canned functionality and worked to have a less canned
application, based on a more algorithmic functionality. Some of the key things we
changed:

We added and implemented the Select Language options, which allows the user to
change which language he wants to translate from and which language to translate
from. Previously this option was split into two called “From” and “Two”. We
made English as the default ‘To’ language because we assume that the user is an
English speaking person.

We have improved the Crop function of the interface, making it a less canned
functionality. We allow the user to freely crop any portion of the image, and a
Center button click will trigger the image to go into Manage Mode, with the
contents outside of the box grayed out so the user can graphically see and confirm
his selection.

We have successfully implemented the Favorites option, which is available in
every menu option of the interface, from its canned version of the Interactive
Prototype. When the user looks at the available translations of the image, he or
she will get the option to save the image and translations to the Favorites directory.
Upon entering the Favorites screen, they will see a list of the translations that they
had previously saved.

Made the translations mechanism to be less canned for the Usability test so that
the user has more flexibility on what he or she wants to translate. Rather than the
one available translatable sign, we have provided translations for the rest of the
signs, but the actual real translation mechanism is still yet to be implemented.

Improved the cursor for the cropping so that it is more visible and also gives
feedback to the user when pressing the center button to start the cropping

We took away the “Options” menu item in the menu because it was confusing for
users to have options in the menu, so we expanded out all the options in there to
appear on the menu list.
Method
Participant
The user that I tested on is third year Industrial Engineering and Operations Research
student here at UC Berkeley. He is of Chinese descent, and came to the United States
when he was in elementary school. He is a suitable target user because he has traveled to
various places in Asia, including Korea and Japan. He is also interested in learning new
languages, so a fast translator would be helpful to him.
Apparatus
I performed the experiment test at the user’s apartment in order to give him a more
comfortable environment to work in. Feeling at east will help the user to relax and have
less pressures associated with performing the tasks. The equipment I used to perform the
usability test was on my own machine, in which I let him take over and work the
prototype to complete his tasks.
Tasks
1. Easy: The default "Translate from" language is Chinese. However, the signs are in
Korean, so the user must change the "Translate from" language to Korean.
2. Moderate: We want the user to save the image to Favorites. This translation may be
very useful for future reference. Now, pretend that two hours have passed, and the user
needs the translation again: retrieve the saved translation and picture on Favorites.
3. Hard: We want the user to pick any one sign to translate, by cropping. We’re only
interested in what one of these signs has to say.
Procedure
First I started off the experiment by giving my user the Informed Consent form to read
and sign. I explained to him that everything will be kept confidential and his identity will
not be released to the public. I explained to him what our product was and why we are
performing this test. Then we moved over to my machine where I notified him that I will
first present to him a demonstration of the interface that he will be testing on. The simple
task that I performed was to start the program, look at the image in Camera mode, take a
picture by pressing the center button, and then pressing the center button once more in
Manage mode to translate the image. Currently the translation is a messy translation with
all the translations on one line. Then I informed the user that we will now take over and
complete a list of three tasks which we have prepared. Meanwhile during the experiment,
my observer will be jotting down notes for future reference. I then give my user the first
task, which is to set the From language to Korean as described above in Tasks. I then
gave him the next task of cropping the image to translate only a portion of the image.
The next task I assigned him was to save that image and then bring it back up in the
Favorites screen. He asked a few questions along the way and I answered them without
trying to give too much information away. After the three tasks, I asked him if he had
any questions or comments and he gave me feedback on the Manage Dictionary options,
which was confusing to him. He suggested that we name it Configuration or something
along those lines. Afterwards I thanked him for his time and wrapped up the experiment.
Test Measures
The three dependent variables that we measured were the time to complete the task, the
number of times that the user makes mistakes, and the number of questions that the user
asked. The reason why we measured these is because we wanted to know the error rate
behind the user’s ability to use the interface. If they make too many mistakes or takes too
long, then it’s a sign that we need to improve on our interface. If the user asks too many
questions then he or she is in a confused state and needs outside help to continue. We
want our interface to be straightforward and easy to pick up and learn.
Results
When the user was completing the first task of changing the “From” language from
Chinese to Korean, he first asked what all the buttons were. This isn’t really related to
our actual interface, it was a question on how to operate the emulator. Furthermore when
he attempted to complete the tasks, he constantly tried to use the touch-screen capabilities
of the emulator, even though I informed him that the application will not have any sort of
touch-screen capabilities. Other than that, the task was fairly straightforward. When
assigned the second task of cropping a portion of the image, the first time around instead
of selecting a portion to translate, he translated the whole screen again as I had done in
the demonstration at the beginning. What he was thinking was that maybe he translates
the image first and then crop out an image. However the way it’s implemented is the user
needs to crop the image in Manage mode before they click the center button to translate
the image. My user tried to crop it after the image was translated instead. On the third
assigned task of saving the image to Favorites and then pulling up the image again, my
user had no trouble at all saving the image, which took him several seconds to
accomplish. However when looking for the Favorites, he checked in the menu options
“Manage Dictionary” rather than Favorites. Then he asked me what “Manage
Dictionaries” was which I explained to him is where you can add or remove new
languages to your application.
Discussion
What I learned from my Pilot Usability Study is that our interface is far from perfect.
Some of the design choices we made didn’t make the functionality of the interface really
feasible and straightforward. One of the things that we might change is the name menu
item “Manage Dictionaries”. We need to make it clearer what it actually does from the
menu item name such as “Add/Remove Languages”. Another thing that I noticed we
should change is the format of the translation box below the image. It seemed to look
really confusing to the user and so we should work on displaying it in a much clearer
form so that the user can easily browse through them. We also need to think about our
crop functionality, which was the task that elicited the most mistakes. The other tasks
seemed to be straightforward enough that the user accomplished them in a reasonable
amount of time. The confusion in trying to crop the image definitely alerts us that
something may be wrong with it, and we certainly need to look into it. We may change
the order in when we crop the image to when we translate the image, and test to see
which causes less confusion.
Appendices
Attached:

Demo Script

Informed Consent Form

Raw data taken by observer
Demo Script
Me: So first, I am going to walk you through a simple run of the application
<Opens up the application>
Me: This is where our application starts, in camera mode. As you see, there is no real
camera implemented at this point in our prototyping. Just imagine, however, that you are
aiming your camera-phone at this wall of Korean signs that you want to translate. To
take the picture, hit the center button. Now, this is Manage mode. To translate these
signs to English, hit the center button. Oh, look, there are two possible translations!
INFORMED CONSENT FORM
You are invited to participate in a study of user interface design. We hope to learn how
usable and effective our prototype is. You were selected as a possible participant in this
study because you might be a potential user of our application
If you decide to participate, we will have you complete a series of tasks that the
application will support, using our interactive prototype. The entire procedure should
take less than 45 minutes. Physical discomforts may include eye strain because you will
be staring at a computer screen. Emotional discomfort may include slight frustration
from trying to complete the series of tasks. There are no direct benefits, but you may
learn something new from this experiment, and if our product is ever commercialized,
you will have known that you have helped us.
Any information that is obtained in connection with this study and that can be identified
with you will remain confidential and will be disclosed only with your permission. Some
anonymous information will be submitted to the professor, as this is a class
project. Although we will try our best to keep your personal information confidential, we
cannot make any guarantees, and in the event that your personal information is leaked,
we will try to notify you as soon as possible.
Your decision whether or not to participate will not prejudice your future relation with
the University of California at Berkeley. If you decide to participate, you are free to
discontinue participation at any time without prejudice.
If you have any questions, please do not hesitate to contact us. If you have any additional
questions later, please contact Jeremy Syn at jjsyn@berkeley.edu who will be happy to
answer them.
You will be offered a copy of this form to keep.
_______________________________________________________________________
You are making a decision whether or not to participate. Your signature indicates that
you have
read the information provided above and have decided to participate. You may withdraw
at any
time without penalty or loss of benefits to which you may be entitled after signing this
form
should you choose to discontinue participation in this study.
_____________________________________ __________________________
Signature
Date
Raw Data
Time to Complete Task
# of times mistakes
# of questions asked
Task 1
1 min
1
1
Task 2
2 min
3
1
Task 3
10 sec for Saving
1
1
1 min for Favorites
Log of Critical Incidents
Time 8:32
Asked what the buttons were
Tried to use the touch-screen
8:36
Translated whole scene instead of cropping
8:37
Clicked on definition (touch-screen)
Confused how to get to Crop option
8:42
Checked Manage Dictionary instead of Favorites
Asked what Manage Dictionaries was
Questions/Comments

Instead of “Manage Dictionaries”, maybe you could name it “Configuration” or
“Add/Remove” which would make more sense