To: Dr. Guler
From: Mija Isip
Date: October 28, 2015
Subject: Usability Test – Converting Binary Bits Instructional Manual
The purpose of this usability test is to fulfill certain test objectives. This usability test will be used
to further the knowledge in the strength, effectiveness, and ease of my instructional manual. While
conducting this usability test, I will strive to understand several different matters. First, I want to obtain
a better understanding of how inexperienced CS majors absorb the information they are reading. I wish
to value the facial expressions, body posture, and any possible questions asked during this usability test.
This will allow me to calculate just how difficult or helpful the manual truly is. Second, another test
objective I’d like to fulfill is their mood during their test. I’d like to note their reaction to how the
physical look and layout of the test, as well as how easy it is for them to understand the information
given. Finally, I’d like to note the ending results. With the example questions and the questions given
during the usability test, I’d like to take a cautious note on the results of the tests. The successfulness of
the test will be able to tell me just how well my instructional manual works.
The audience of this usability test is somewhat skewed. The instructional manual itself is meant
for elementary educated students (range: K-5), while the secondary audience was meant for non-CS
majors. Although, due to the limitations of my connections, I was unsuccessful in finding an elementary
roommates who aren’t Computer Science majors. My first test subject is majoring in Business, while the
second is studying Social Work. These two test subjects proved to be successful picks due to their lack of
knowledge and experiencing regarding any information taught in my instructional manual (aside from
basic mathematical skills taught in younger years).
After successfully brainstorming all the necessary test objectives needed in order to create the
instructional manual, I then had to draw out the tasks that would be asked during the test. An important
aspect of this usability test was to remember to make note of thanking each test subject for their
voluntary time. The users will be initially handed a copy of the instructional manual and will be asked to
read the entire manual. There will be no time limit upon reading the manual and I will made note to
mention that after reading through, the user will be permitted to keep the instructional manual
throughout the whole test. Next, a pre-test survey will be given to each user. The purpose behind the
pre-test survey is to get a feel behind the purpose of their voluntary act of being a test user. With the
use of the pre-test survey, I will also get a feel of how each user reacted to reading the instructional
manual for the first time. After completing the pre-test survey, I will hand the user a calculator, pencil,
scrap paper, and the questions. There will be six questions which covers one question per conversion
covered inside the manual. For their convenience, I have ordered the questions corresponding order to
the sections being taught within the manual. Because I am not testing the user’s mathematical ability,
I’ve decided to give users a calculator. Users will be permitted to ask questions, take breaks, etc. Also,
during the time of the actual test – users will be encouraged to ask questions and refer back the manual.
Finally, the last task of this usability test will be to take a completely different post-test survey. This
purpose of this second survey is to get their opinion on the difficulty and their understanding of the
questions. This survey will also give me a better understanding of how the users feel after having to
actually implement their read knowledge onto paper. Another important advantage to this survey is that
the results will educate myself in understanding just how well the users had to refer back to their
manual. Consistently referring back to the manual will indicate lack of ease in the instructions.
Based on the results of the usability test, I believe that the manual itself did a fine job in terms
of teaching the methods of binary conversion. Although, that isn’t without saying there are several
adjustments needed to be made. My first test subject was Zoe, the business major. I made that she had
eaten and was comfortable with the atmosphere to avoid any variables that could hinder the
effectiveness of my test subjects. Zoe took about 15-20 minutes to fully read the manual. Based on her
facial expressions while reading the manual, she was a little set back. Because this was new material,
there were times where she would have to stop and ask questions just to clarify that she was correctly
retaining the information. There were times where I could see her flipping back to a few pages. Flipping
back in between pages was a good sign because it seemed as if she was catching on to the pattern of the
conversion methods. After reading the manual, I gave Zoe the first survey. It was clear that she wasn’t
interested in becoming a CS major (this is one of the down falls of having secondary audience members
serve as my test subjects). Although, I was able to gather from her first survey that she felt the manual
wasn’t hard to understand. In regards to the testing, Zoe was able to successfully answer questions 1-4,
although her faults mainly resided within the hexadecimal conversions. She didn’t seem to have any
questions to ask while she was taking the test. Her failure in converting the hexadecimal bits indicates
that it’s necessary to further elaborate on this conversion technique. Raising powers of 16’s with the
inclusion of “A-F symbols,” seemed to prove as unclear. Her post-test survey results proved that
although she feels the manual was in fact very useful and for the most part easy to follow, it was the last
portion of the manual which proved as difficult. In terms of comments, she claimed that the manual was
“Easy to read and something fun and new to learn.” When I asked her if there were any final comments
or feedback she’d like to offer to create a stronger manual she made a few comments:
“Maybe you should add more visuals.”
“There was a point in the second half of the document where I found my mind to be a little exhausted of
all the repetitive work and math.”
“I think adding fun pictures will help younger kids to enjoy this manual, but overall I think you did a great
job!”
After conducting a usability test with Zoe, I proceeded onto my next test subject. My other
roommate Allie is a Social work major. Like Zoe, I began the test by handing her a copy of the
instructional manual. She took about 15 minutes to read the instructional manual. Allie’s pre-test survey
results were much like Zoe’s. Like Zoe, she wasn’t really interested in taking part or being a Computer
Science major. This didn’t prove much of a shock to me, although her mathematical background did. She
had made note to mention in the survey that mathematics wasn’t one of her strong suites. After
completing her survey, I gave Allie the actual test questions. Unlike Zoe, there were several instances in
which Allie would have to stop and ask questions in regards to the questions being asked and how to
solve the questions. I found myself to be a lot more interactive and needed during this test. Allie’s test
score results in her correctly answering 4/6 questions as well. Finally, I asked Allie if she had any
comments/feedback to give. Allie made a few comments:
“I’m not very good at math and I’m a little tired. This may have affected my results.”
“I think you did a good job – the only thing I could say is maybe to make the manual look even happier
or better just because this is kind of a hard topic to sell to kids.”
After analyzing my notes, data, and taking under consideration the comments/feedback given –
I was able to come to a conclusion of the necessary revisions. I believe that the in terms of style and
formatting, lines to separate each section/conversion method being taught is necessary. Because I’m
trying to teach 6 different methods of conversion, I also agree that adding fun visuals after every section
would be a great idea. I believe the fault in both Zoe and Allie’s inability to answer the last two questions
is because of the weakness in instruction and the matter of attention. I believe that because the manual
is heavily math based and consists of several pages, creating “visual breaks,” will benefit the
effectiveness of this manual. Also, I believe that the addition of coloring in words would also help
increase the appeal in my manual. Another thought to consider – is the addition of extra
questions/examples. The inclusion of additional practice problems/examples (especially in the
hexadecimals), will allow my users to practice and better enhance their familiarity with each conversion
set. Although there were times that I could visually notice confusion or distress in each test subject, I’m
overall pleased with the outcome. Because binary conversion is a difficult subject to both introduce and
teach, my results prove that the information being taught isn’t completely out of base with my users.
Although there were a lot of misleading variables such as my inability to have elementary students as
test subjects – I believe that the manual did a fairly good job.