Visions for Mobile Devices
Beyond Their Current Role
Brad A. Myers
Carnegie Mellon University
Human Computer Interaction Institute
bam@cs.cmu.edu
http://www.bam.hcii.cmu.edu/
Our Research on Handhelds, 1
Pebbles project
Since 1997
Presented at Faculty Summit 2002, 2003,
2004
http://www.pebbles.hcii.cmu.edu
How handhelds can be used together
with desktop computers
Remote Commander
Shortcutter
SlideShow Commander
Our Research on Handhelds, 2
Pebbles Personal Universal Controller (PUC)
Automatically create remote control panels on
PocketPC and Phone for appliances
Appliance description in XML
Interfaces that are consistent for the user
Automatically combine interfaces for multiple
appliances
Thoughts on Research Topics for
the Future of Handhelds
Data Integration
Multiple antennas in all devices
Cellphone + BlueTooth + WiFi
 Ability to be constantly connected
Not just once a day for synchronization

Can re-think the data model
Integration of all of the user’s data
Small to large pieces of data

Cut-and-paste across devices
Making short notes and short annotations (metanotes) a fundamental part of data that is stored
Intelligent and automatic consistency of data
Never re-enter data more than once anywhere!
Where is the Data?
On desktop, “in the network”, on body,
synchronized everywhere?
Storage is basically free but still have the
synchronization problem
Could put on network, but then loose
reliability, worry about security/theft
Worry if lose mobile device with data
Make control of what is shared with others
easier than emailing
Application Integration
Portable applications across different
form factors
Run same application on handheld as on PC?
Web pages
Word  Pocket Word
Automatic adaptation to I/O capabilities
Even dynamically as application runs
Would help with innovating in hardware since
might just work on the new device
Entering Data
Text entry still hard
Hand-helds have voice recorders, cameras,
styluses for strokes and handwriting
Easier on a phone to take a picture than to type
Rich data types input and output
Need better ability to handle all these types as
easily as text
Handwriting = picture = typed = spoken
Anywhere can use one can use another
Pictures as the answer to questions
Use context, location, etc. so enter less
information
Intelligent interfaces to fill in field with right values?
Use history, etc. to learn from user
Handheld and PC Augment Each
Other When Nearby
Touch screen, camera, voice recorder, phone modem,
as PC’s I/O
PC’s keyboard, mouse, big screen for handheld’s I/O
Opportunistic annexing of anything in the environment
(displays, keyboards, antennas, networks, etc.)
Office, conference room, home, automobile
E.g., Use car’s speakers, microphones, heads-up or navsystem display
http://www.cc.gatech.edu/projects/PIE/projects/oppannex.shtml
UI research issues: where best for:
Control: across appliances/devices, input methods
Output: displays on TV vs. front panel vs. handheld
Form Factors For Handhelds
Fundamental human sizes:
In Hand
Phone, PDA -> smaller, pocket
On Arm
Tablet
On Wrist
Watch
On Tables
Clamshell Windows CE  not successful
New Mini-pcs
Laptop
Wearables
Sensors like BodyMedia
Tradeoffs: portability, battery life, screen size
(readability), text entry
Special Purpose Devices vs.
Convergence
Mobile game device, PDA, phone, pager,
still camera, video camera, music player,
movie player, watch, flash memory, …
Star Trek: Tricorder, Communicator, Phaser,
Medical scanner, portable memories
Or one device that does it all?
Marketing issues – will people create content
(games, apps, movies) for your device?
HCI issues
Good game control UI ≠ good phone control
Larger display ≠ smaller device
Battery life
Convergence vs. Special Purpose, cont.
If many devices, still should
adhere to “enter once” principle
Scattered around environment,
vs. carry around in pocket/on body
Idea: Disassemble the Parts
Each function in separate device
Speaker in ear, microphone, display on
arm, processor in pocket, memory,
antennas
Integrate with watch display, input
E.g., Intel “personal server”
Addresses the one v. many issue?
Specialized I/O mechanisms (sold
separately?) attach to common core
Game controller/game media reader
Camera lens
Other Research Areas for Handhelds
New UIs for Handhelds
Not just shrunk desktops
More uses for cameras
Handicapped access
Digital inclusion using handhelds
Security in general
Summary
Still lots of interesting and important research
topics in the handheld / mobile area!
Challenge: How can handheld devices improve
the user interfaces of everything else, and not
just be another gadget to be learned
6.5 billion people in the world
3,000
1,500
2,000
500
1,000
46
1
et
P
C
s
s
Ta
bl
PD
A
PC
s
es
lP
ho
n
C
el
pl
e
0
Pe
o
Millions
1.5 billion cell phones worldwide, vs.
6,500
500 million PCs (?)
7,000
6,000
vs. only 46 million PDAs
5,000
4,000
vs. 1 million TabletPCs
Visions for Mobile Devices
Beyond Their Current Role
Brad A. Myers
Carnegie Mellon University
Human Computer Interaction Institute
bam@cs.cmu.edu
http://www.bam.hcii.cmu.edu/
-- Extras / Not Used --
Handhelds as Platform When Can’t Afford a PC
Less-developed countries
How use mobile phones for what we would use
a PC for?
What makes sense to try to do?
Already bought a cell phone, computation is
free, what can be done with it?
Computers used for:
Communication, entertainment, business stuff, …
Collaboration through handheld as a way to
bring experts to less-developed countries
Applications in Health care, Education,
Agriculture, etc.
Specialized applications  how create apps.
cheaper? Easier to program/customize
Text Entry on Handhelds Still Hard
Lots of research, still not solved
a b c
z
y
d e f
g h i
j
k
x
!?&@
sp
bksp
w
v
u
t s r
EdgeWrite
QuikWriting
l
Ent
#
$
CAPS %
m
n
q p o
Xnav
Shark
Text Entry Method
Speed
Multitap
10 wpm average, 29 wpm tops
T9
20 wpm tops
Treo-Rim tiny keyboard
30 wpm, up to 60 wpm
Twiddler
45 wpm average, 70 wpm tops (non-looking)
QWERTY keyboard
90 average; 192 wpm world-record
Dvorjak keyboard
90 average, 204 wpm world-record
Normal Speaking
175 wpm
American Sign Language
175 wpm
Court report (chording)
300 wpm (world record)
Dasher
More Uses for Camera
Scanner for text, UPC reader, sign translation
Sensor for movement,
Mouse replacement
Gesture recognition
Where am I? (augment GPS)
Games that use camera as a sensor
Face recognition
Plant recognition, disease recognition, …
Type on table with camera detecting virtual keys
Collaboration support: show the objects
discussed
But needs much better integration  still too hard to
send/receive pictures
New User Interface Metaphor?
Completely new UI metaphor that is
appropriate for phones/mobile devices
Maybe not just applying the 25-year-old
desktop UI to the phone
What could be done?
Haptic interfaces for mobile devices
Besides just vibrate
More appropriate than for desktop
Other Research Issues
Handicapped access
Blind: Braille handhelds?, talking
Muscular difficulties
Cognitive difficulties
Security in general
Safety of data
Trustedness of devices
Transmission of data