Mobile Usability Testing
Lecture :: „Mobile Usability Testing“
Inst. f. Softwaretechnik und Interaktive Systeme
qse.ifs.tuwien.ac.at, thurnher@qse.ifs.tuwien.ac.at,
christian.fruehwirth@qse.ifs.tuwien.ac.at
Topics
I.
II.
III.
IV.
V.
VI.
Motivation & Definition
Mobile Devices
Challenges in Mobile Usability Testing
Methods & Approaches
Heuristics & Guidelines
Examples & Ideas
Mobile Usability Testing
Goals :: What do we want to achieve in this lecture?
What we want:
•
Enable you to set up a Mobile Usability Test based on
Quantitative as well as Qualitative Methods.
•
Give you Inspirations & Ideas about what can be done to
make mobile applications more usable.
*vgl.: Schilit and Theimer (1994)
2
Mobile Usability Testing
Methods :: How do we want to achieve our goals?
How we want to do it:
•
Creating an awareness for the challenges of mobileUsability
•
Compare common Usability Testing Methods
•
Overview about possible Heuristics & Guidelines for Mobile
Usability Testing
*vgl.: Schilit and Theimer (1994)
3
I. Motivation & Definition
Definition :: „Mobile Usability Testing“
Considerations for Mobile Testing:
•
Results are strongly influenced by surrounding
environment
•
Results are influenced by devices used for testing
•
Collected Data will be „richer“ (Gesture, Voice, …)
4
Mobile Usability Testing
I. Motivation
I. Motivation & Definition
Costs of late Usability research
For every dollar spent acquiring a customer you will spend $100
dollars reacquiring them after they leave because of poor usability
or bad customer service. (*)
*vgl.: MauroNewMedia (2002)
6
I. Motivation & Definition
Usability Engineering in the Software Development Process
*vgl.: INTERACT 2001 Workshop, Jan Gulliksen, Inger Boivie)
7
Mobile Usability Testing
II. Mobile Devices
II. Mobile Devices
Mobile Devices :: Definition
Are used “on -the-run and for activities that may last only a few
seconds or are highly context dependent” (*)
*vgl.: (Vetere et al., 2003, p.1)
9
II. Mobile Devices
Mobile Devices :: Realistic View 1
„Mobile Devices: One Generation From Useful“(*)
Tighter Integration needed:
•
Devices do not work together well with each other.
 Synchronization with other Applications
•
Different Features packed into one device do not
act as one entity  PDAs with GSM modules
•
Ad-hoc Networking is still far from realisation
 Network coverage & Roaming (WLAN, UMTS, GPRS, GSM)
 Industry Standards (Bluetooth compatibility, vulnerability)
*vgl.: (Jakob Nielsen 2003, Alertbox , Aug. 18th.)
10
II. Mobile Devices
Mobile Devices :: Realistic View 2
Design / Interface Problems still to be solved:
•
Deck-of-Card Size(*) limits the Screen Size
 Higher Screen-Resolutions, better viewing angle.
•
One Dimensional Interfaces (Scroll wheels) are not suitable
for 2D – Screens.
•
Text-input is still a great Problem.
Small Devices  small Buttons.  new Button Alignments?
*vgl.: (Jakob Nielsen 2003, Alertbox , Aug. 18th.)
11
II. Mobile Devices
Mobile Devices :: Realistic View 3
Fundamental Problems:
•
Quality of Service of local Network Providers.
Things that „could be done“ just can‘t because of local ServiceLimitations or lack of network coverage. (Broadband, UMTS, etc.)
•
Online Services must specialize for Mobile use
Much shorter Articles, more use of XML, simplyfied Navigation
•
Reconsider the way email is used
 not just forward every mail to the PDA (Attachments, Executeables)
12
Mobile Usability Testing
III. Challenges
III. Challenges
Challenges :: of Mobile Usability Testing (*)
•
1. Device Proliferation
–
Handling many different Devices, Rendering Methods
•
2. Application Modality
–
Handling simultaneous voice / Data User interactions
•
3. User Mobility
–
Users are likely to be distracted during use
•
4. Data Collection
–
Recording eye-movement and video taping will not work
everywhere
*vgl.: http://www.littlespringsdesign.com/analysis/utest.html
14
III. Challenges
Challenges :: 1 : Device Proliferation (*)
Devices can be
•
•
•
•
As small as possible, optimized for voice communication
Quite large, optimized for data display
Optimized for gaming
Optimized for multimedia
Applications are perceived differently
•
Reading News-Bulletin on a point-matrix phone display
 content is forgotten 3 hours later.
•
Reading News-Bulletin on a java-enabled 19“ CRT Monitor
 Will be stored in long-term memory
*vgl.: http://www.littlespringsdesign.com/analysis/utest.html
15
III. Challenges
Challenges :: 2 : Application Modality(*)
Mobile Applications often combine Graphic / Data and
Voice Elements.
•
Difficult to test in an early stage of development
(Software isn‘t fully functional / not yet written.)
•
Test must be able to provide simultaneous experiences
*vgl.: http://www.littlespringsdesign.com/analysis/utest.html
16
III. Challenges
Challenges :: 3 : User Mobility
Mobile Users are very likely to be distracted.
•
Natural Environment is not always manageable/affordable to be
simulated in a lab
•
Distractions and „Normal anomalies“ (Waiter interrupting you in
a restaurant to take your order) have to be part of the test
17
III. Challenges
Challenges :: 4: Data Collection 1
Acquiring Data outside the Lab
“It is cold and snowing and you do not know from where your bus leaves in 5
minutes. You pick up your WAP phone to check: The mobile user runs to
catch her bus, after her run three researchers with cameras and
microphones..(*)“
•
Eye tracking will hardly work on tiny screens and under mobile
conditions.
•
Recording tools interfere with the users‘ interaction with the device.
(Cameras mounted on a cell phone make the user hold it in an
unnatural position)
*(vgl.: Per-Ola Rasmussen ExarbII 2003)
18
III. Challenges
Challenges :: 4: Data Collection 2
Mobile Users interact not just with the screen and keyboard.
•
Test needs to record, gestures, face-expressions, voice, bodylanguage, etc.
•
 Much richer Data
(People leaning left and right while playing a Formula-1 racing Game
on their Java-enabled Phone.)
•
Record what is the user doing, what is he/she NOT doing.
19
Mobile Usability Testing
IV. Methods & Approaches
IV. Methods & Approaches
Methods :: Approaches to Mobile Usability Testing 1
The common Methods
Heuristic Evaluation
Guidelines vs. Design. Used in early development
(Nielsen & Mack, 1994)
Cognitive Walkthrough
(Rowley&Rhoades, 1992)
How or why a person would react in a certain situation.
Based on assumptions on the user's mental model
*(vgl.: Tomas Lindroth, Stefan Nilsson & Per-Ola Rasmussen, ExarbII – HT2000)
21
IV. Methods & Approaches
Methods :: Approaches to Mobile Usability Testing 2
The common Methods
Feature Inspection
(Nielsen & Mack, 1994)
Consistency Inspection
(Nielsen‚ 1995)
Does the product meet the users needs and demands?
Used in middle stages of development
Checks consistency across multiple products from the
same product – family.
*(vgl.: Tomas Lindroth, Stefan Nilsson & Per-Ola Rasmussen, ExarbII – HT2000)
22
IV. Methods & Approaches
Methods :: Approaches to Mobile Usability Testing 3
The common Methods
Standards Inspection
(Wixon et. Al. 1994, Nielsen 1995)
Guideline Checklist
(Wixon et. Al. 1994, Nielsen 1995)
Ensures compliance with industry standards. Best used
in middle-stages of development
Used in conjunction with other usability methods. The
Checklists give the tester a basis by which to compare
the application
*(vgl.: Tomas Lindroth, Stefan Nilsson & Per-Ola Rasmussen, ExarbII – HT2000)
23
IV. Methods & Approaches
Methods :: Approaches to Mobile Usability Testing 4
The common Methods
Thinking Aloud
(Nielsen‚1994)
Contextual Inquiry
(Holzblatt & Beyer ‚1993)
Lets the evaluator understand how the user views the
system.
Used to get a broad knowledge about the environment
that you are producing for. More a discovery process
than an evaluative process.
*(vgl.: Tomas Lindroth, Stefan Nilsson & Per-Ola Rasmussen, ExarbII – HT2000)
24
IV. Methods & Approaches
Methods :: Lab vs. natural environment
The larger the number of factors that is under control in a test, the more scientific
rigour is emphasized. The more natural like the test setting is, the more
relevant and applicable the results will be.
*(vgl.: Mason 1988, Järvinen, 1999)
25
IV. Methods & Approaches
Methods :: Approaches to Mobile Usability Testing 5
The Ideal Test
•
Natural situation / environment
•
Application is fully functional
•
All possible forms of devices are being tested
•
Users are free to do what they would normally do
Users don't feel „tested“
•
Tester can record every audio / visual / voice / movement / screen data
from the user without affecting the users behavior.
•
Tester sees the Application the way the users see it.
26
IV. Methods & Approaches
Methods :: General Mobile Usability Testing Process 1
Preparation
•
Define goals, methods and tasks/scenarios for the test.
Introduction, Warm-up
•
•
Introduce the test to the user
Start with easier tasks, give time for a short warm-up phase.
Testing
•
Perform the actual test with as less interaction between user and tester
as possible
27
IV. Methods & Approaches
Methods :: General Mobile Usability Testing Process 2
Test situation
•
Give the user time to get out of the test-situation.
•
Then start reviewing his/her opinions, impressions and suggestions.
•
Make sure to discuss special occurrences that may have happened
during the test with the user.
28
IV. Methods & Approaches
Remember :: Challenges to deal with (*)
•
1. Device Proliferation
•
2. Application Modality
•
3. User Mobility
•
4. Data Collection
*vgl.: http://www.littlespringsdesign.com/analysis/utest.html
29
IV. Methods & Approaches
Methods :: Handling User Mobility 1
For informal, problem identifying tests
•
Ask participants to use the application maybe over lunch
•
Offering a compensation helps „motivating“ the participant
•
Don‘t forget to have them sign an informed consent statement
•
Interruptions (waiter, etc.) are welcome
Watch what happens when the users resumes the task and see what
difficulties occur.
30
IV. Methods & Approaches
Methods :: 3 : Handling User Mobility 2
For formal, statistically precise tests
•
Don‘t try to introduce distractions into the test unless you are testing
with a greater number of participants
•
Referring to Nielson, the marginal benefit will decrease if you are
testing with more then 10 Users
Thesis and formula is questioned from many researchers.
31
IV. Methods & Approaches
Methods :: 3 : Simulate the natural environment? 1
Chances(*):
•
•
•
Reproducible conditions
Easier / more complete documentation
Use of more sophisticated tools
Risks
•
•
•
Unusual environment for the user
Restrictions due to simulation
Non recording of the original work surrounding field
(office atmosphere, disturbances, etc.)
*(vgl.: akziv. Requirements from users point of view. 2004)
32
IV. Methods & Approaches
Interactive Example
Which environmental factors are reproduceable in a
laboratory surrounding.
Environmental Conditions BT 041118.xls
33
IV. Methods & Approaches
Methods :: 3 : Simulate the natural environment? 2
„Mobile Devices are build to be mobile so take them out into the field“ (*)
•
Take the lab to the user, not the user to the lab.
*(vgl.: akziv, „wearability“. 2004
34
IV. Methods & Approaches
Methods :: 3 : Lab Test vs. Field Test(*) 1
Example Application: Using SMS Service on a PDA while walking
•
•
In the lab: on a treadmill
In the field: on a pedestrian street
*(vgl.: Jesper Kjeldskov, Aalborg University Denmark)
35
IV. Methods & Approaches
Methods :: 3 : Lab Test vs. Field Test(*) 2
Attention needed to navigate
Body
Motion
None
Conscious
None
1. Sitting at a table
or standing
n/a
Constant
2. Walking on a treadmill
with constant speed
4. Walking at
constant speed on a
changing track
Varying
3. Walking on
a treadmill with varying
speed
5. Walking at
varying speed on a
changing track
*(vgl.: Jesper Kjeldskov, Aalborg University Denmark)
36
IV. Methods & Approaches
Methods :: 3 : Lab Test vs. Field Test(*) 3
Usability problems identified by the test subjects
•
Numbers are basically equal
Techniques
•
Total
Lab 1
Lab 2
Lab 3
Lab 4
Lab 5
Field
Critical
4
4
3
4
3
3
4
Serious
11
11
9
9
9
8
17
Cosmetic
19
8
8
8
6
12
32
Total
34
23
20
21
18
23
53
Notice that while sitting on a desk the cosmetic problems
identified by the users were far more!
*(vgl.: Jesper Kjeldskov, Aalborg University Denmark)
37
IV. Methods & Approaches
Methods :: 3 : Lab Test vs. Field Test(*) 4
Are Lab tests superior?
•
Consider Cost / benefit of different techniques and settings
Time and effort per problem found
Can you afford NOT to find a problem?
Costs of missing Usability:
•
•
•
•
•
•
•
•
•
lost of repurchases
increased calls at helpdesk
lost of repurchases
lost of brand reputation
necessary redesign in late state or next version
law suits
……
*(vgl.: Jesper Kjeldskov, Aalborg University Denmark)
38
IV. Methods & Approaches
Methods :: 4 : Handling Data Collection 1
Mobile devices are extremely personal. Users may pick them up,
gesture, or lean back with them.(*)
•
Record Device Screen + Users Face at the same time
•
Use wireless tracking & recording technology
(WLAN, Bluetooth, small radio cameras, etc.)
•
 Users should not be handicapped by the testing equipment
*(vgl.: Little Springs Inc. 2004)
39
IV. Methods & Approaches
Methods :: 4 : Handling Data Collection 2
Recording a mobile phone‘s screen and the users face
at the same time with two cameras mounted on
the phone(*)
*(vgl.: Little Springs Inc. 2004)
40
IV. Methods & Approaches
Methods :: 4 : Handling Data Collection 3
Recording „Soft Information“
•
Define methods to integrate „soft-information“
E. g.: Users starts shaking the Phone to make it start the application
faster; user‘s thumb tends to cover up parts of the device‘s display)
•
Many qualitative information can be quantized.



Measuring heart-beats / second to determine the stress-level
Recording subconscious hand and leg movements.
Record number of extra-applicational interactions
(user answered 2 phone calls and asked his colleague for help
during the use of the application for 5. min.)
*(vgl.: Little Springs Inc. 2004)
41
Mobile Usability Testing
V. Possible Heuristics & Guidelines
V. Heuristics & Guidelines
Heuristics :: General Guidelines for mobile Applications 1
Highly functional design
•
Don‘t use fancy designs if they don‘t bring a real benefit for the User
Consistent usage of icons / buttons / names and labels
•
Consider the users mental-models when you introduce new functions
or name buttons.
Integrated content navigation
•
•
Help the user tracking it‘s way through the information, always provide
a clear exit – point.
Consider Shortcuts
*(vgl.: Little Springs Inc. 2004)
43
V. Heuristics & Guidelines
Heuristics :: General Guidelines for mobile Applications 2
Reduced HCI interactions
•
•
Especially when done on mobile devices interactions with the user are
often difficult and time-consuming (text-input on a mobile phone)
Reduce Interactions by any means possible (Location based services,
Heuristics, default-values, etc.)
Offer intelligent search-functions
•
Assist the user in finding the information, as any unnecessary
interaction makes the applications less usable for the user.
*(vgl.: Little Springs Inc. 2004)
44
V. Heuristics & Guidelines
Heuristics :: Mobile Games 1
For Mobile Games the Rules are
a little different
•
Navigation Consistency?
•
The User should not feel like
using his/her phone, he/she
should experience the Game
World
*(vgl.: NOKIASeries 60 Developer Platform 2.0: Usability Guidelines For J2ME™ Games
45
V. Heuristics & Guidelines
Heuristics :: Mobile Games 2
Game experience vs. Social acceptable behavior
•
Sound, Light and Vibration enhance the users Game experience
•
 Typically, games are played in locations where it is not suitable or
socially acceptable to have the sound on.
•
During Mobile Usab. Testing consider there are usually other people close
by when the user plays the game.
*(vgl.: NOKIASeries 60 Developer Platform 2.0: Usability Guidelines For J2ME™
Games
46
Mobile Usability Testing
VI. Examples & Ideas
VI. Examples & Ideas
Text Interface :: Projecting the image
A standard-sized Query-Keyboard is projected
by laser on any given surface.
•
The Users input is recognised by a small camera in
the cigarette-pack sized device.
•
Note: Being announced in 1999 the product is still under
development and may never reach market maturity.
(2004)(*)
•
www.virtualdevices.net
*(vgl.: http://www.ibizpda.com)
48
VI. Examples & Ideas
Text Interface :: Breaking the Qwerty Paradigm
Standard-sized keys aligned for one-handed
use.
•
Can be used under mobile conditions
(doesn't need chair + desk environment as
similar fold-up keyboards for PDAs)
*(vgl.: www.frogpad.com/)
49
VI. Examples & Ideas
Remote Controll :: Force Push
Operate household devices with a
gesture of your fingertip.
•
IR-Led points at Device to be controlled
•
Touch & Acceleration Sensors combined
with Software recognise gestures
and execute command
*(vgl.: Koji Tsukada, mobiquitous.com/pub/apchi2002-ubi-finger.pdf/)
50
VI. Examples & Ideas
Natural Interaction :: Stick with the basics
Real „Pick and Drop“
1.
PDA – User picks up a file on his screen by tapping
on it with a digital pen.
2.
He passes on the pen to his colleague who drops
the pen on his PDA-screen.
3.
The file is copied.
*(vgl.: http://news.bbc.co.uk/2/hi/technology/3774747.stm)
51
Mobile Usability Testing
For more information see:
http://qse.ifs.tuwien.ac.at/courses/Usability/VO_Usability_Engineering.htm