DFS: Diabetes Forecast System
Usability Tests
Overall Summary
The most definitive statement that can be made about usability tests is that they take
more time than expected. Between finishing the prototype and working around
schedules this deliverable took almost three times as long as expected. Tight and solid
scheduling is important because one delay can have a domino effect that leaves the
group with dry periods of waiting followed by late nights spent furiously scampering to
get things done.
We felt we got a lot out of our usability tests, with a number of problems coming to light
that seem blatantly obvious in hindsight. The most common and noticeable problem
was the ambiguity of the icons on our toolbar, which resulted in almost every user either
wasting time pondering over their usage and meaning or wasting clicks navigating down
digital dead-ends. The second most common issue is almost embarrassing: the way we
display our predictions is far too vague to be truly useful. That prediction is the major
piece of innovation with our product and fixing this issue is of the highest importance to
us. We also discovered bugs in our implementation and were offered some very useful
suggestions about how to streamline data entry. We plan on implementing as many of
the suggestions as possible because we found we agreed with the users in most cases.
The actual running of the tests was a learning experience. We were able to some able
to run some very fun tests with the technology at our disposal, including running the
prototype right off a webpage and doing an entire test with someone in another country
thought text messaging and video conferencing! All of the user’s were eager to help, but
it is very important to start getting as much feedback as soon as possible. We found that
as the experiments wore on the users got more and more comfortable expressing their
opinions to us and offering suggestions. If given the opportunity to run the tests again,
we’d add an emphasis to verbal questions and answers at the beginning of the test and
devise some critical thinking or creativity evoking questions just to open communication
lines as much as possible before we move onto questions and tasks that we consider
critical.
Results of Usability testing
Below are the overall results of our testing. They are answers that we received after the
tests were over and we rated them in terms of importance and their complexity
(measured by time it took users to overcome the problems).
“Importance” indicates the overall severity of the problem, and is rated on a scale from 1
(unimportant) to 5 (most important). “Time to solve” is an indication of how long it took
subjects to deal with the listed problem, and is rated on a scale from 1 (not time
consuming) to 5 (very time consuming).
Problems
Suggestions to improve them from users
Importance
Time to solve
Confusing icons.
Color and symbolic meaning are misleading.
Maybe use fewer colors and more standard icons.
Text pop-ups on roll-over
5
5
Buttons that don't work.
Buttons don't have any functionality and
don't give any useful feedback.
5
3
Prediction is vague.
Prediction is not specific and gives no
advice to help prevent a predicted severe BGL.
5
2
Help button
Users had trouble finding the help button and
rarely used it.
We don’t offer helpful instructions on navigation
3
2
Entering food (or new food type) data.
Adding a food to the list of foods or just
recording foods eaten.
4
5
Send users to history screen after data entry.
Currently, users go to home screen after
data entry.
2
3
Eliminate extra steps
Send users to list editing screens from exercise
and diet instead of routing it through settings.
2
2
Test Summaries
**Please note that all names have been changed to protect confidentiality.
James
Jim is a 54 year old High School principal with Type II diabetes. English is his native
language. He’s had diabetes for the past 2.5 years. He says the most difficult part of
dealing with diabetes is the continuous monitoring needed from day to day. Before the
test he showed interest in using a tool that could help him predict his daily BGL. He
averages less than 3 hours of exercise a week.
The test took place at Jim’s house, at his home computer. I was pleasantly surprised to
find that he had difficulty with almost every task. As I watched him perform the tasks I
noticed that each task (or sub-task for larger ones) could be broken down into two
sections. First, navigation. Second, manipulation. Navigation being the process
needed to start the task and manipulation being the task itself. The manipulation portion
of the test went off without a hitch and he expressed pleasure with much of the interface
with regards to data entry and task completion (with a few minor exceptions that were
more the result of an unfinished product than faulty interface design). However, the
navigation portion of the tasks was atrocious. Our main mode of navigation is a toolbar
across the top of the screen. Jim expressed extreme displeasure with this; not with the
idea, but rather with our specific implementation. Whereas before the usability test all
the icons seemed unique and descriptive, afterwards I’m left wondering how we could
not have seen the obvious pitfalls. He felt there is ambiguity between several icons and
not a satisfactory description of what they do. He frequently commented that the icons
needed descriptive text boxes to appear upon rollover, because icons are more prone to
misinterpretation. There is also the case of the “edit list” buttons from the diet and
exercise screens; these buttons take you to settings, from which you have to take one
more step to edit the lists. They should take you straight to the screens to edit the lists.
He also expressed dissatisfaction with the vagueness of the prediction.
Dave
Dave is a 21 year old computer science student with Type I diabetes, and he uses an
insulin pump. English is his native language. He has had diabetes for 18 years. When
first diagnosed he found it most difficult not eating the same things that other kids do.
He is not a vegetarian. He wrote that he knows how to deal with his diabetes "very well."
Dave exercises only a little bit through regular activities. He records his blood glucose
levels (BGL's) a few times per week, and that it is relevant to him to know his BGL. He
uses his own glucometer mostly to transfer readings to his insulin pump, but uses the
BGL memory feature least of all. Dave considers himself "techno-savvy" and would be
interested in using a device that forecasts blood glucose levels over a period of several
hours. His native language is English.
We met at the Student Union on the University of Maryland, College Park campus, in a
study room which was quiet, to do the test. He finished the tasks quickly and without
asking any questions. It took him about five minutes. He is a talented programmer and
familiar with usability testing so I think he understood what he needed to do and was
able to work it out with no problems. He was satisfied by the icons and the text (size and
font) in the interface, and was able to navigate the menus easily. The most useful part
of the device in his opinion was the history feature and the graph of the most recent
BGL's. The least useful feature was the slide bar approach on the insulin screen. Dave
liked the device and gave it a '7' out of a scale from 1 to 10 (10 being "most satisfied"
with the interface). The easiest task for him was getting a BGL and the hardest ones
were entering food that he had eaten and recording exercise. The predictions he found
informative but they don't "suggest what to do" to prevent a high or low reading. He
suggested that the system go right to the history screens once data is entered about
diet, exercise, or insulin doses.
Steve
Steve is a 38 year old man with Type II diabetes, and he has Down Syndrome. English
is his native language. He has had diabetes for 5 or 6 years. When he was first
diagnosed, he said that he had trouble taking off extra weight, and that it was hard to
exercise regularly. Currently, he goes to the gym almost everyday for 30 to 45 minutes.
He is not vegetarian. Steve knows his condition fairly well, saying that the food he eats
turns into sugar, and that he tries not to eat a lot of sugary snacks. When asked if he is
"techno-savvy", he said he's "so-so, not that great" with using computers. When I met
with him, before we started, he was using his computer. I noticed during the test that he
had no problems using the mouse or keyboard to complete the tasks. He said that he
would like to have a device that predicted his BGL's, but only tests his BGL three times a
week, which is typical for someone with Type II diabetes. Despite this, he felt it very
important to know his BGL, because if it was too high, he would be rushed to the
hospital.
I met Steve at his home and we did the test at the table in his dining room. Steve had a
lot of questions. He seemed lost at many points during the testing. I verbally assisted
him with each task. He did have a good memory about the screens after learning what
each one did. I think that he felt he was not doing what he should have been doing
when he attempted to complete a task correctly, but faulty design on the part of the
interface prevented this. For example, in task 7, we request that users enter a new
exercise type, then save it. The 'save' button does not give users any feedback,
however, and it doesn't actually save the exercise to the database. This was an
oversight on the designer's part. Steve finished the testing in a little less than 15
minutes. He liked the device, but felt it was time consuming. The icons were "straightforward" in their meaning but the text was a little small. There were a lot of different
screens in his opinion. He was okay with the feedback presented and thought that he
would be able to fix mistakes if he had made any. His confidence was high about his
ability to remember how to do the tasks we asked him to perform. If the tasks had taken
any longer I think Steve would have become bored and may have asked to stop. It
seemed that he was becoming very impatient toward the end of the task list, which made
me wonder about the usefulness of such a device to someone who has a cognitive
disability.
Julia
Julia is a 26 year old doctoral student in genetics, and she doesn't have diabetes so has
never used a glucometer. English is her native language. She is familiar with
computers, but says that she is not "techno-savvy". On a scale from 1 to 10 (10 being
"most satisfied" with the interface), she rated the system a "6, with high potential. I feel
when all pulled together and all functions working it will be a very nice interface." She
found data entry easy to do and efficient. The graphics and icons seemed appropriate,
except that the icon for the BGL screen was gray, which she took to be a feature that
wasn't available, because it was "grayed out". The prediction was felt to be too vague,
she would like to get a numerical value. She was satisfied with the number of screens
that had to be worked through to get to the right one, but didn't appreciate one screen
which didn't provide any feedback after she tried saving a new type of food to the food
list. Julia couldn't tell whether or not she would be able to easily fix mistakes, because
she didn't make any she told me.
During the test I noticed that she hunted around for the right screens to complete the
tasks. She asked a few questions of me and I responded after letting her look a little
more carefully to discover the answer herself. I think once she learned what the icons
meant, or what screens they led her to, she remembered them well. She spoke to
herself as she worked and I was able to hear that she was looking for the right answer
but was a little stuck. She didn't quite understand the insulin screen, but for someone
who doesn't have diabetes, that response seemed appropriate. Working on our tasks
took her about 10 minutes.
Father Barry
Father Barry is a 31 year old Franciscan priest who has Type I diabetes. He was
diagnosed 23 years ago. He said that he started giving himself injections when he was
8 (many children have their parents do this). His hardest struggle with diabetes is
sticking to a diet low in carbohydrates. English is his native language. He does not
consider himself "techno-savvy". In regards to his knowledge about his own diabetes,
he told me, "I chase a low blood sugar with sugar, and use Humalog [insulin] when
readings are high." Considering that our system makes a BGL prediction, he said that
he would look at the prediction and then do a test. If the test matched the prediction on
several occasions he would probably use it.
He finished the tasks in about 10 minutes and had questions throughout the tasks.
When he began, he tried touching the screen, but the device is only interactive through
the mouse or touchpad on my laptop. He used the touchpad. He found it somewhat
difficult understanding the icons and asked about these. The icons representing the
settings screen and the history screen were confusing to him, the others he found to be
"somewhat ambiguous". I didn't realize that some of the tasks and interface features did
not match up. For example, one task asks the users to press the 'forecast' button, but
the button on the screen that should have been pressed was 'save'. I realized that better
communication across our team would have prevented confusion like this. I wondered if
Fr. Barry was becoming frustrated because of interface - task mismatches. As the
person I tested with the greatest experience with glucometers (23 years), I took his
opinion seriously when he gave the interface a 4 or 5 on a scale from 1 to 10 (10 being
"most satisfied" with the device). He said data entry was complex and time consuming,
and the feedback after saving it was "somewhat unclear". The BGL prediction he found
a little vague. Asked about whether he would be able to save changes, he said that he
wasn't sure because he hadn't made any data entry errors during the testing.
Celia
Celia is a 30 year-old single woman who learned of her condition of type I diabetes not
too long ago. She is originally from El Salvador, however she has been leaving in the
Washington DC area for the past 15 years. Although she speaks English fluently, she
appreciates any sensitive information (related to her health) in Spanish. It was a very
unexpected experience to learn of her condition and she is willing to learn and use
everything she can in order to take care of her body.
Prior to her learning of her condition, Celia enjoyed life doing different things such as
dancing (her preferred hobby) as well as helping others doing community service. As
she started to feel extremely tired very often, she decided to start some tests. Blood
tests gave notice of her condition. At this point she decided to learn as much as she can
about proper and balanced diet. She also tried to work closely with doctors in order to
keep herself in good shape since she has great goals in her life.
Celia is a school teacher for an elementary school in Montgomery County. She enjoys
working with children and would like to keep her condition as secret as possible,
however she understand that things are going to change for her for now on. Although
she find technology tools (computers, PDAs, ipods, etc) fascinating, she does not
consider herself computer savvy. Despite this, she feel very comfortable in using basic
tools commonly found on the internet (such as pressing button, data entry, etc). She find
different cultures very fascinating and enjoys spending time with people from other latin
american countries and learn from them. These has led her to learn about different types
of food. She prefers desserts more than anything and has a great taste when preparing
them. This, however has been a great barrier to cross since she knows she needs to be
more careful now that she needs to eat more balanced from now on.
Celia expressed her impression about the idea of forecasting. She enjoyed very
much the testing process, although she would have hope for complete
functionality. She also found the “diet” record feature very neat (with arrows to
add to your daily intake). She clearly noted that feature as the most userfriendly
and easier to use part of the system.
Monica
Monica is an administrative assistant for a Contruction Company. She handles incoming
calls as well as some book-keeping for the company. She is very familiar with two-three
friends who have type I diabetes. Although she is not personally familiar with diabetes,
she is very mindful of the condition of her friends and at times has conversed with them
to learn more about it. One thing that called her attention was the fact that people with
type I diabetes had to learn to be more disciplined with their diet.
Monica showed that it was a useful tool as long as the user is willing to keep on entering
data constantly. She understood pretty quickly that DFS depends on human interaction
to provide an accurate reading and found that the interface (in general) was pretty well
organized and made a lot of sense.
Jennifer
Jennifer is a college freshman in Taiwan. She was born with type I diabetes. When she
was young, she did not have to worry too much about her BGL and insulin since most of
the issues were taken care of by her parents. After she went to middle school, she
began to learn to use BGL meter herself and calculate how much corresponding insulin
she had to take.
Jennifer seems to be very confident about the device even though it is her first time of
using it. There was no occasion that she needed to seek out help from the device or
from the observer. The flow of her operation over different menus were very smooth.
She did not have any problem performing the assigned tasks. It seems that the user was
satisfied with the interface of the device.
Her case showed that for some users the interface for the DFS system was designed
properly such that it is easy to follow and operate different functions. Even though her
first language is not English, the words chosen for the menu were straightforward and
easy to understand that she did not mistake any content or feedback. Because the user
does not live in America, this usability test was conducted through presenting the
interface online and combine instant messaging program along with a video camera and
microphone to provide the observer chance to monitor the real reaction of the user.