Predicting Task Execution Time on
Handheld Devices
Using the Keystroke Level Model
Annie Lu Luo and Bonnie E. John
School of Computer Science
Carnegie Mellon University
CHI’05 – April 6, 2005
Motivation and goals

Keystroke Level Model (KLM)



A priori prediction of expert user task time
Intensively used on desktop computers
Not yet been adapted to handheld devices
• Limited display size
• Input device: stylus, touch-screen, hardware buttons
• Interaction methods: tap, Graffiti, etc.

Investigate KLM on handheld UIs


Applicability of model to novel interface modalities
Accuracy of model predictions
© Annie Luo, Carnegie Mellon University, 2005
slide 2
KLM in brief

Describe a task by placing operators in a sequence









K – keystroke (tap)
P – point with mouse (stylus)
H – homing (move hand from mouse to keyboard) (N/A)
D (takes parameters) – drawing (N/A)
R (takes parameters) – system response time
M – mental preparation
G – Graffiti stroke (580 ms – Fleetwood, et al 2002)
Five heuristic rules to insert candidate Ms into the
sequence
Task execution time = Σ all operators involved
© Annie Luo, Carnegie Mellon University, 2005
slide 3
Handheld task:
Find information about the MET
1
City map
Region map
Street map
Scroll list
2
Soft keyboard
Start
3
Query result
Graffiti
4
Museums list
© Annie Luo, Carnegie Mellon University, 2005
slide 4
Create KLMs


One KLM for each of the four methods
Used CogTool (John, et al 2004)
MacroMedia
DreamWeaver
HTML
mockups
Netscape
HTML
event handler
Interface
event
messages
via
LiveConnect
Behavior
Recorder
ACT-Simple
code
based on
KLM
ACT-R
Environment
KLM Trace
Modeler mocks up
interfaces as HTML
storyboard
Modeler
demonstrates tasks
on the HTML
storyboard
© Annie Luo, Carnegie Mellon University, 2005
ACT-Simple
complies code into
ACT-R production
rules
slide 5
ACT-R
Mozilla Firefox
Behavior Recorder
© Annie Luo, Carnegie Mellon University, 2005
slide 6
User study


10 expert PDA users (Female:Male = 3:7)
At least one year experience using:


Instructed to perform the task on a PalmVx


Using four different methods (within subject design)
Training session before real session


Palm series, pocket PC, or smart cell phone
Repeating each method for 10 times
Data collection


EventLogger: records system events to a log file
Videotaped modeler’s behavior for verification
© Annie Luo, Carnegie Mellon University, 2005
slide 7
New results
since paper published



Latest version of CogTool
Better estimation of system response time
Detailed analysis of model and user traces (140/400 removed)
16.000
-6.9%
14.000
-1.4%
12.000
10.000
8.000
2.3%
8.9%
5.8% -3.7%
7.7%
-9.3%
Old Model Time
Old User Time
New Model Time
6.000
New User Time
4.000
2.000
0.000
Map
SoftKB
© Annie Luo, Carnegie Mellon University, 2005
Graffiti
ScrollBar
slide 8
Conclusion & Future work

KLMs produced with CogTool are effective for
handheld user interfaces:



Produces accurate execution time prediction
Supports new input modalities: Graffiti
Future work:


Detailed analysis of the user pauses (mental time)
Use predictions of pauses to assist energy
management
© Annie Luo, Carnegie Mellon University, 2005
slide 9
Thank you!

Authors’ contact info:
Bonnie John – bej@cs.cmu.edu
 Annie Luo – luluo@cs.cmu.edu


The CogTool project:

http://www.cs.cmu.edu/~bej/cogtool/
© Annie Luo, Carnegie Mellon University, 2005
slide 10
Participants information (backup)
User (gender)
Device owned
How long
1 (M)
Palm Vx
5+ years
2 (M)
Compaq iPAQ
3 years
3 (M)
Palm IIIe
4 years
4 (M)
Handspring Visor
3 years
5 (F)
Handspring visor Pro
2 years
6 (F)
Dell PDA
1 year
7 (M)
iPAQ 3630
4 years
8 (M)
Kyocera 7135
4+ years
9 (M)
Handspring Visor Prism
3 years
10 (F)
Palm VA
3 years
© Annie Luo, Carnegie Mellon University, 2005
slide 11
Results in paper (backup)
Model Time vs. User Time (Average)
16
13.60328
14
12.662 12.8363
12.662
Time (sec)
12
10
9.913
10.29767
9.213 9.0055
Model Time
User Time
8
6
4
2
0
M1-Map
Navigation
M2-Soft Keyboard
M3-Graffiti
M4-Scroll Bar
Tasks
© Annie Luo, Carnegie Mellon University, 2005
slide 12
New results
since paper published



Better measurements of system response time
Removed error trials (140 out of 400)
Latest version of CogTool
12.000
-8.9%
-5.8%
10.000
8.000
-7.7%
9.3%
6.000
4.000
2.000
0.000
Map
SoftKB
Model Time
© Annie Luo, Carnegie Mellon University, 2005
Graffiti
ScrollBar
User Time
slide 13
Interface Widgets:
- Buttons
- Check boxes
- Text fields
- Pull-down lists
- Links
- Menus
- Audio input
- Audio output
© Annie Luo, Carnegie Mellon University, 2005
slide 14
Behavior Recorder
Netscape
ACT-Simple
© Annie Luo, Carnegie Mellon University, 2005
slide 15