David Nguyen
Alex Pai
Alan Tsang
Margaret Yau
Group 9-PWE
cs160 Fall 2003
Group Assignment 4
Analysis of Heuristic Evaluation of Prototypes
http://kettle.cs.berkeley.edu/cs160-fall-03/uploads/16/group9-a4-he.doc
I. Problem
The interface we are analyzing, HATS, is an interface designed to be used by
Public Works Engineering (PWE) in collecting locations and attributes of street assets,
with an iPAQ Pocket PC and a GPS receiver.
II. Violations Found
1. [User Control and Freedom] [Severity 4]
The login screen has no screen for logging out of the system. Users may accidentally
start the program and wish to exit.
2. [Consistency and Standards] [Severity 3]
The Help button is located in different locations for different screens. This violates the
heuristic by making the user search for the help function.
3. [Aesthetic and Minimalist Design] [Severity 3]
All custom inputs are stored in the pull down menu for future use. This could cause
unnecessary clutter with rare entries.
4. [User control and freedom] [Severity 3]
If the user tries to turn on the GPS when GPS is not available (and therefore the interface
automatically switches to non-GPS mode), it results in the user fighting with the interface
to control the on/off switch.
5. [Consistency and standards] [Severity 3]
The streetlight icon at the top of the input page is different from the icon used to
represent the streetlight asset on the main page.
6. [Recognition rather than recall] [Severity 2]
It is unclear how the user can do to enter the input screen for assets.
7. [Help users recognize, diagnose, and recover from errors] [Severity 2]
Error message may be ambiguous if there are fields that are similar. For example, if there
is a field called “arm length” and another field called “foot length,” then the error
message “length” is unclear.
8. [Visibility of system status] [Severity 2]
The message “asset recorded” on the status bar may be missed by the user if it disappears
too soon, or it may be misinterpreted if it stays on the status bar too long. Currently, it is
unclear how long the duration is.
9. [Error Prevention] [Severity 2]
ID fields are memorized, and are the default value for the next asset entry. This could
cause errors if users forget to modify the field.
10. [Recognition Rather than Recall] [Severity 2]
The icons representing assets and toolbar functions are not labeled. This may be
troublesome if the user forgets what the icons stand for.
11. [Match Between System and Real World] [Severity 2]
The GPS on-off switch automatically switches off when GPS is unavailable. This does
not match the metaphor of a light switch, where you can still turn on the switch if the
bulb is broken.
12. [Match Between System and Real World] [Severity 1]
The control of the switch is confusing. A switch affords to be flipped. In our case, the
switch is pressed (by user clicking the switch).
13. [Error Prevention] [Severity 1]
Users may be confused over the difference of the i icon (information) and the ? icon
(help). This could cause the user to select the incorrect function.
14. [Recognition Rather Than Recall] [Severity 1]
Custom input screens are not labeled to indicate that they permit the user to input a
custom value for the field. The user may be unsure how the given input will be treated
by the system.
15. [User Control and Freedom] [Severity 1]
A user may accidentally close the keyboard. There is not a clear way for the user to bring
back the keyboard.
16. [Visibility of system status] [Severity 1]
On/off icon is unclear and likely misread in glare and sunlight outdoors. More visible
differentiation between on/off states is needed.
17. [Help and documentation] [Severity 0]
The log in screen is missing any way of getting help.
Group 9—Assignment #4: Heuristic Analysis
2
18. [Visibility of system status] [Severity 0]
The GPS signal status is recognizable only by counting the number of bars displayed.
Although this does display system status, it requires more than a quick glance to
determine.
19. [Match between system and the real world] [Severity 0]
Using the return key to move to the next input field does not reflect the standard
conventions commonly used when dealing with forms.
20. [Consistency and standards] [Severity 0]
Use of the return key to enter carriage return in some text input fields, while it serves to
end input in others may cause the user to question which of these functions it will have at
any given time.
21. [Error prevention] [Severity 0]
When the GPS is functioning but the user is required to manually input the location of an
asset there is not a clear way to determine distances from the present location, which
could lead to the user recording an asset at an incorrect position.
22. [Consistency and standards] [Severity 0]
The presence of an arrow next to the name field for the login page is confusing. Scenario
1 describes the process of logging in as using the name as a text field, whereas the
presence of the arrow is standard for a drop-down box.
23. [User control and freedom] [Severity 0]
There are only pre-defined entry fields for data for a specific type of asset. In the case of
the streetlights, the user can only enter in the ID, wattage, arm length, height and notes.
If the user decides to keep some other standard data field on streetlights, there doesn’t
appear to be a way to do so other than entering it every time into the notes section.
24. [User control and freedom] [Severity 0]
There are only pre-defined assets. If the user decides that some other asset of the city is
important, there is no way to add one.
25. [Consistency and standards] [Severity 0]
The system should not indicate the current location without a GPS signal in the same way
it indicates the current location with a GPS signal. If the user thinks the GPS signal is on
but it is not, they might be confused into they are in a different location than they really
are.
26. [Flexibility and efficiency of use] [Severity 0]
It seems like the user would commonly want to input or edit an asset at a location where
they are not currently located. This could happen if the user already knew the location of
an asset they wanted to add or edit, or like in the case of scenario 3, if they couldn’t reach
the location.
Group 9—Assignment #4: Heuristic Analysis
3
27. [Consistency and standards] [Severity 0]
No keyboard is available for login screen when there is one for other inputs.
28. [Aesthetic and minimalist design] [Severity 0]
The decoration beneath the screen on the surface of the device (i.e. the buttons on iPAQ)
looks like buttons. This can confuse the user if they are not functional buttons.
29. [Consistency and standards] [Severity 0]
The scroll bar on the side of the input menu does not have up and down arrow keys.
30. [User control and freedom] [Severity 0]
Login should provide method to get password if it was forgotten or perhaps allow some
guest account.
III. Summary of Violations
Table 1. Number of Violations per Heuristic
Heuristics
Visibility of system status
Match between system and the real world
User control and freedom
Consistency and standards
Error prevention
Recognition rather than recall
Flexibility and efficiency of use
Aesthetic and minimalist design
Help users recognize, diagnose, and recover from errors
Help and documentation
Count
3
3
6
8
2
3
1
2
1
1
Table 2. Number of Violations per Severity Category
Severity Rating
0
1
2
3
4
Count
12
8
5
4
1
Total Number of Violations: 30
Group 9—Assignment #4: Heuristic Analysis
4
IV. Evaluation Statistics
Table 3. Percentage of total violations of each severity category found by the evaluators
severity\evaluator
level 0
level 1
level 2
level 3
level 4
total violation found
evaluator 1
33.33%
12.50%
40.00%
25.00%
100.00%
30.00%
evaluator 2
0.00%
37.50%
0.00%
0.00%
0.00%
10.00%
evaluator 3
25.00%
25.00%
20.00%
0.00%
100.00%
23.33%
evaluator 4
41.67%
0.00%
40.00%
25.00%
0.00%
26.67%
evaluator 5
8.33%
25.00%
60.00%
50.00%
0.00%
26.67%
Table 4. Number and uniqueness of problems found by the evaluators
evaluator #
1
2
3
4
5
problems found
9
3
7
8
8
uniqueness
2 shared with #4, #3
1 shared with #3
3 shared with #1, #3, #4
3 shared with #1, #3, #5
1 shared with #4
Number of Problems
Found
Number of Evaluators vs. Number of Problems
Found
35
30
25
20
15
10
5
0
1
2
3
4
5
Number of Evaluators
Figure 1. Number of evaluators vs. number of total problems found
Group 9—Assignment #4: Heuristic Analysis
5