University of Babylon
College of information Technology
Department of Software
Visual Applications
Lecture 7
Evaluation Techniques
By
Hayder Al-Ghanimi
Mr.hayder.alghanimi@gmail.com
Evaluation Techniques
• Evaluation
Evaluation should occur throughout the design life cycle, with the results feeding
back into medications of the design. A distinction is made between evaluation by
the designer or a usability expert and evaluation that studies actual use of the
system.
So :
• tests usability and functionality of system
• occurs in laboratory, field and/or in collaboration with users
• evaluates both design and implementation
• should be considered at all stages in the design life cycle
Goals of Evaluation
Evaluation has 3 main goals:
• Assess extent of system functionality
• Assess effect of interface on user
• Identify specific problems
Evaluation through expert analysis
[1]Cognitive Walkthrough
Proposed by Polson.
• evaluates design on how well it supports user in learning task
• usually performed by expert in cognitive psychology
• expert ‘walks though’ design to identify potential problems
using psychological principles
• forms used to guide analysis
•
Cognitive Walkthrough is a detailed review of a sequence of actions, in this case, the steps
that an interface will require the user to perform in order to accomplish some known task.
The evaluators go through each step and provide a story about why that step is not good for
new users. To do so, you need four things: a prototype of the system, a description of the
task the user is to perform on the system, a complete written list of the actions needed to
complete the task with the system and an indication of who the users are and what kind of
experience and knowledge the evaluators can assume about them.
Cognitive Walkthrough
• For each task walkthrough considers
• what impact will interaction have on user?
• what learning problems may occur?
Analysis focuses on goals and knowledge:
– Does the design lead the user to generate the
correct goals?
Example: programming a video recorder by
remote control
• Imagine we are designing a remote control for a video recorder (VCR) and are interested in
the task of programming the VCR to do timed recordings. Our initial design is shown in next
Figure. The picture on the left illustrates the handset in normal use, the picture on the right
after the timed record button has been pressed. The VCR allows the user to program up to
three timed recordings in different ‘streams’. The next available stream number is
automatically assigned. We want to know whether our design supports the user’s task. We
begin by identifying a representative task. Program the video to time-record a program
starting at 18.00 and finishing at 19.15 on channel 4 on 24 February 2005.
• We will assume that the user is familiar with VCRs but not with this particular design.
• The next step in the walkthrough is to identify the action sequence for this task. We specify
this
Example: programming a video
recorder by remote control
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
in terms of the user’s action (UA) and the system’s display or response (SD). The initial display is as the left-hand
picture in Figure.
UA 1: Press the ‘timed record’ button
SD 1: Display moves to timer mode. Flashing cursor appears after ‘start:’
UA 2: Press digits 1 8 0 0
SD 2: Each digit is displayed as typed and flashing cursor moves to next position
UA 3: Press the ‘timed record’ button
SD 3: Flashing cursor moves to ‘end:’
UA 4: Press digits 1 9 1 5
SD 4: Each digit is displayed as typed and flashing cursor moves to next position
UA 5: Press the ‘timed record’ button
SD 5: Flashing cursor moves to ‘channel:’
UA 6: Press digit 4
SD 6: Digit is displayed as typed and flashing cursor moves to next position
UA 7: Press the ‘timed record’ button
SD 7: Flashing cursor moves to ‘date:’
UA 8: Press digits 2 4 0 2 0 5
SD 8: Each digit is displayed as typed and flashing cursor moves to next position
UA 9: Press the ‘timed record’ button
SD 9: Stream number in top right-hand corner of display flashes
UA 10: Press the ‘transmit’ button
SD 10: Details are transmitted to video player and display returns to normal mode
Example: programming a video recorder by
remote control
•
•
•
•
•
•
•
•
•
•
•
•
•
Having determined our action list we are in a position to proceed with the walkthrough. For each
action (1–10) we must answer the four questions and tell a story about the usability of the system.
Beginning with UA 1:
UA 1: Press the ‘timed record’ button
Question 1: Is the effect of the action the same as the user’s goal at that point?
The timed record button initiates timer programming. It is reasonable to assume that a user familiar with VCRs would
be trying to do this as his first goal.
Question 2: Will users see that the action is available?
The ‘timed record’ button is visible on the remote control.
Question 3: Once users have found the correct action, will they know it is the one they need?
It is not clear which button is the ‘timed record’ button. The icon of a clock (fourth button down on the right) is a
possible candidate but this could be interpreted as a button to change the time. Other possible candidates might be
the fourth button down on the left or the filled circle (associated with record). In fact, the icon of the clock is the
correct choice but it is quite possible that the user would fail at this point. This identifies a potential usability problem.
Question 4: After the action is taken, will users understand the feedback they get?
Once the action is taken the display changes to the timed record mode and shows familiar headings (start, end,
channel, date). It is reasonable to assume that the user would recognize these as indicating successful completion of
the first action.
So we find we have a potential usability problem relating to the icon used on the ‘timed record’ button. We would
now have to establish whether our target user group could correctly distinguish this icon from others on the remote.
The analysis proceeds in this fashion, with a walkthrough form completed for each action.
[2] Heuristic Evaluation
• Proposed by Nielsen and Molich.
• usability criteria (heuristics) are identified
• design examined by experts to see if these are
valid
• Example heuristics
• system behaviour is predictable
• system behaviour is consistent
• feedback is provided
Others
• [3]Model-based evaluation
– Certain models provide a means of combining
design specifications and evaluation into the same
framework.
• [4]Using previous studies in evaluation
Evaluating through user
Participation
[1] Laboratory studies
• users take part in controlled tests, often in a
specialist usability lab. The advantages the
interruption-free environment. The disadvantage is
the lack of context, which may result in unnatural
situations.
[2]Field Studies
the user is observed using the system in its own
work environment. The advantage is the natural
use of the system that can hardly be achieved in
the lab. However, the interruptions that come
with this natural situation may make the
observations more difficult.
Observational Methods
Think aloud and cooperative
evaluation
Think aloud is a form to observation where the user
is asked to talk through what he is doing as he is
being observed. It has the advantage of simplicity,
but the information provided is often subjective
and may be selective. A variation is cooperative
evaluation, in which the user and evaluator work
together to evaluate the system.
[1]Protocol analysis
Methods for recording user actions include paper and
pencil, audio recording, video recording, computer
logging and user notebooks. In practice, a mixture of
the different methods is used. With recordings, the
problem is transcription.
[2]Post-task walkthroughs
• A walkthrough after the observation reflects
the participants. actions back to them after
the event. The participant is asked to
comment it and to answer questions by the
evaluator in order to collect missing
information.
Query Techniques
Queries provide direct answers from the user about
usability questions, but the information is often subjective.
Interviews provide a direct and structured way of gathering
information and can be varied to suit the situation. They
should be planned in advance with a basic set of questions,
and may then be adapted to the specific user.
Questionnaires are less flexible than interviews: they are
planned in advance. However, it can be used to reach a
wider group and takes less time to administer.
Interviews
• analyst questions user on one-to -one basis
usually based on prepared questions
• informal, subjective and relatively cheap
• Advantages
• can be varied to uniform context
• issues can be explored more fully
• can elicit user views and identify unanticipated problems
• Disadvantages
• very subjective
• time consuming
Questionnaires
• Set of fixed questions given to users
• Advantages
• quick and reaches large user group
• can be analyzed more rigorously
• Disadvantages
• less flexible
• less probing
Questionnaires
• Need careful design
• what information is required?
• how are answers to be analyzed?
• Styles of question
•
•
•
•
general
open-ended
multi-choice
ranked
Physiological methods
The physiological response monitors
receiving currently most attention are eye
tracking and physiological measurement.
Eye tracking
• head or desk mounted equipment tracks the
position of the eye
• eye movement reflects the amount of
cognitive processing a display requires
• measurements include
• fixations: eye maintains stable position. Number and duration indicate
level of difficulty with display
• saccades: rapid eye movement from one point of interest to another
• scan paths: moving straight to a target with a short fixation at the
target is optimal
Physiological measurements
• emotional response linked to physical
changes
• these may help determine a user’s reaction
to an interface
• measurements include:
• heart activity, including blood pressure, volume and pulse.