Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
Handheld Devices for Control
Brad A. Myers
Jeffrey Nichols, Jacob O. Wobbrock,
Kevin Litwack, Michael Higgins, Joe
Hughes, Thomas K. Harris, Roni
Rosenfeld, Mathilde Pignol
bam@cs.cmu.edu
http://www.cs.cmu.edu/~pebbles
Handheld Devices

Handheld Devices for:




Communication
As a “Personal Information Manager”
As a “Personal Digital Assistant”
As a PC replacement


“PocketPC”
Mobile Devices for Remote Control of other
devices


Remote Control of PCs
Remote Control of appliances
Brad Myers
Carnegie Mellon
2
Problem

Appliances are too complex
Brad Myers
Carnegie Mellon
3
Problem

Too many remotes
Brad Myers
Carnegie Mellon
4
Problem

April 8, 1991
Brad Myers
Carnegie Mellon
5
Approach

Use a personal handheld device as an
Interface to the PC and to the Appliances
Brad Myers
Carnegie Mellon
6
History of Communication

ParcTab – continuous communication

Early “organizers” – no communication

Newton – extra cost for synchronization

Palm – HotSync once a day

WindowsCE/PocketPC –
ActiveSync once a day
Brad Myers
Carnegie Mellon
7
Now, Handhelds will be Communicating

Toshiba e740
802.11 (“Wi-Fi”)
HP iPaq 3870

BlueTooth

Cell-phone network (G3)

(Infrared)

(Wires or cradle)
Microsoft SmartPhone
Brad Myers
Carnegie Mellon
8
Premises of our Research
“With the coming wireless technologies,
connecting the PCs and PDAs together
will no longer be an occasional event
for synchronization. Instead, the
devices will frequently be in
close, interactive communication.”
—
Brad Myers
Brad Myers, “Using Hand-Held Devices and
PCs Together,” Comm. ACM,
Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.
Carnegie Mellon
9
Pebbles Project


Performed as part of the Pebbles Project
Overall goal: Use of multiple devices at the
same time






Palm and desktop PC
Mobile Phone and “Smart Home”
Multiple handhelds in a meeting
Pocket PC and appliances
Multiple users with their devices
Single user with multiple devices
Brad Myers
Carnegie Mellon
10
Pebbles is:
P DAs for
E ntry of
B oth
B ytes and
L ocations from
E xternal
S ources.
http://www.pebbles.hcii.edu/
Brad Myers
Carnegie Mellon
11
Controlling a PC
Remote Commander
 Slide Show Commander
 “Semantic Snarfing”
 Shortcutter

http://www.pebbles.hcii.edu/
Remote Commander

Allow PDAs to control a PC
Can be used with any application
 Uses the standard (single) cursor
 Don’t have to jump up and
grab mouse
 All mouse and keyboard functions

Use PDA like touchpad
 Graffiti or our own pop-up
keyboard


Word prediction and completion
Brad Myers
Carnegie Mellon
13
For People with Muscular Disabilities

Using handhelds as interface to PCs

People with Muscular Dystrophy
have fine-motor control but lose
gross motor control


Difficulties with mouse and keyboard,
but stylus OK
So use Remote Commander
as PC’s keyboard and mouse
Brad Myers
Carnegie Mellon
14
PocketPC version


Get PC’s screen onto PocketPC
Full view, or one-to-one zooming


Scroll with iPaq’s buttons
Control or scribble
Brad Myers
Carnegie Mellon
15
Shared Clipboard

Transfer information among your personal
devices





E.g., between PDA and PC
Also between multiple computers
Connect all clipboards together
Transfer content or reference
Works with all applications
Brad Myers
Carnegie Mellon
16
Power Point Control


Use PC to give the presentation
Use hand-held to control the PC


Two-way communication
Hand-held shows picture of slide,
notes, timer
Brad Myers
Carnegie Mellon
17
SlideShow Commander, cont.





See list of slide titles
Scribble
Tap on on-screen buttons
and links
Preview other
slides
Control
demonstrations
Brad Myers
Carnegie Mellon
18
Control at a Distance

Controlling a PC from
across the room


Use a laser pointer?


Meetings, etc.
Studies
“Semantic Snarfing”

Grab contents to handheld
Brad Myers
Carnegie Mellon
19
Shortcutter


User-created panels of controls
Create custom interfaces and
extensions to PC applications



And then take them with you
Direct manipulation
for edit, then set
properties
PocketPC or Palm
Brad Myers
Carnegie Mellon
20
Shortcutter Widgets

Buttons

Virtual Joy stick

Virtual Knob

Sliders

Mouse pad

Gesture panel
Brad Myers
Carnegie Mellon
21
Scenarios of Use

Lean-back mail reading

Controlling media players

… and many others
Brad Myers
Carnegie Mellon
22
Controlling Appliances

PhD research of Jeffrey Nichols
http://www.pebbles.hcii.edu/puc/
Problem

Too many complex devices, each
with its own idiosyncratic interface








Stereo system
Telephones
ATM
Fax machine
Photocopier
Hotel alarm clock
Increasingly computerized
Low usability
Existing “universal” controls
Pre-programmed at the factory with a subset,
 Or, Laboriously hand-programmed by the user

Brad Myers
Carnegie Mellon
25
Our Approach


Handheld is a “Personal Universal Controller” (PUC)
Two-way communication


Appliance describes its functions
Handheld PUC
 Automatically creates interface
 Controls the appliance
 Displays feedback about appliance status
Specifications
Control
Feedback
Brad Myers
Carnegie Mellon
26
Automatic Generation of UIs

Benefits

All interfaces consistent for a user
With conventions of the handheld
 Even from multiple manufacturers


Addresses hotel alarm clock problem
Can take into account user preferences
 Multiple modalities (GUI + Speech UI)


A Hard Problem

Previous automatic systems have not
generated high quality interfaces
Brad Myers
Carnegie Mellon
27
Research Challenges
1. Automatic Design of Graphical User
Interfaces
2. Automatic Design of Speech User Interfaces
3. Connection with real devices



Through various protocols
X-10, AV/C, HAVi, UPnP, etc.
Also, custom hardware and software
Brad Myers
Carnegie Mellon
28
Hand-Generated Graphical Interfaces

First, Hand-Designed PocketPC interfaces:


AIWA Shelf Stereo (Tape,CD,Tuner)
AT&T Telephone/Answering
Machine

Used Embedded Visual Basic

Ensured quality with heuristic
analysis and think-aloud studies

Compared with manufacturer’s
interfaces
Brad Myers
Carnegie Mellon
29
Results of Comparison

Using PUC, users took 50% less time &
made 50% fewer errors

All differences are significant (p < 0.05)
Brad Myers
Carnegie Mellon
30
Current PUC Specification Language


XML
Full documentation for the
specification language
and protocol:
http://www.pebbles.hcii.cmu.edu/puc/

Contains sample
specification for a stereo
Brad Myers
Carnegie Mellon
31
Properties of PUC Language

State variables & commands

Each can have multiple labels


Typed variables



Useful when not enough room
Base types: Boolean, string,
enumerated, integers,
fixed-point, floating-point, etc.
Optional labels for values
Hierarchical Structure

Groups
Brad Myers
Carnegie Mellon
32
Dependency Information


Crucial for high-quality interfaces
Expressed as <active-if> clauses

Operations:


Combined Logically


Equals, Less-Than,
Greater-Than
AND, OR
Used for:



Dynamic graying out
Layout
Widget selection
Brad Myers
Carnegie Mellon
33
Specifications

Have working specifications for:







Audiophase stereo
X-10 lights control
Sony CamCorder
Windows Media Player
Audio ReQuest hardware MP3 player
WinAmp Media Player
Elevator
Brad Myers
Carnegie Mellon
34
Examples of Generated GUIs

Stereo and X-10
Brad Myers
Carnegie Mellon
35
More Examples

Elevator
Outside
Brad Myers
Inside
Carnegie Mellon
36
Generating Speech Interfaces

“Universal Speech Interface” (USI) project



Creates grammar, language model and
pronunciation dictionary from PUC specification




Prof. Roni Rosenfeld of CMU
http://www.cs.cmu.edu/~usi
Pronunciation from labels using phonetic rules
Can provide other pronunciations as labels for fine-tuning
Will use dependency information to help with
disambiguation and explanation
Supports queries and spoken feedback

Paraphrases as confirmation
Brad Myers
Carnegie Mellon
37
Adaptors

“Adaptors” provide the interface to existing
(and future) appliances
If do not support specification language directly
 AV/C (standard protocol)
Custom hardware






Custom software



Audiophase Stereo
ARQ MP3 player
SIMA MP3 player (future)
HAVi (current work)


X-10


Light switches, etc.
Brad Myers

Sony CamCorder
PlayStation, etc. (future)
Mitsubishi HDTV 65” TV
Mitsubishi HDTV VCR
UPnP (future)

??
Carnegie Mellon
38
Adaptor for AV/C
PUC Java Proxy
Hand-written
spec of AV/C
devices
PUC Java Library
Our Java device code
Multiple device
controllers
Automatic
GUI
generation
Automatic
Speech UI
generation
Our C++ code
Microsoft DirectShow
library
AV/C protocol
Firewire cable
(IEEE 1394)
Brad Myers
Digital Video
Camera or VCR
Carnegie Mellon
39
Demonstration



X-10
Camcorder through AV/C protocol for
IEEE 1394 (Firewire)
Two way communication:

When state changes from appliance, GUI is
updated and can query state with speech
Brad Myers
Carnegie Mellon
40
Future Work on PUC

Controlling User “Experiences”




Consistency for Users


One command for multiple devices
Play a DVD  sets DVD, cable, TV
Setting alarm clock  sets heat, coffee, car
And user preferences
More devices


General Motors automobiles
Etc.
Brad Myers
Carnegie Mellon
41
For More Information

Many papers on the Pebbles web site


On PUC and other work
Most programs available for free
downloading:
http://www.cs.cmu.edu/~pebbles

SlideShow Commander was
licensed for commercial sale

http://www.slideshowcommander.com
Brad Myers
Carnegie Mellon
42
Conclusions

As more and more electronics are
computerized and communicating, mobile
devices can help control them

Handheld devices can improve the user
interfaces of everything else

Not just be another gadget to be learned
Brad Myers
Carnegie Mellon
43
Thanks to Pebbles Students

This research performed by a large number
of students since 1997:
Rishi Bhatnagar
Ben Bostwick
Franklin Chen
Yu Shan Chuang
Karen Cross
Carl Evankovich
Marc Khadpe
Dave Kong
Chun-Kwok Lee
Brad Myers
Joonhwan Lee
Jennifer Li
Leo Lie
Jack Lin
Kevin Litwack
A. Chris Long
Rob Miller
Jeff Nichols
Choon Hong Peck
Mathilde Pignol
Herbert Stiel
Marsha Tjandra
Adrienne Warmack
Jake Wobbrock
Jerry Yang
Sunny Yang
Brian Yeung
Carnegie Mellon
44
Thanks to Our Sponsors!
The Pebbles research project is supported by
grants from:
DARPA
Microsoft
NSF
Pittsburgh Digital Greenhouse
General Motors
And equipment grants from:
Hewlett Packard
Lucent Technologies
Mitsubishi
Palm Computing
Symbol Technologies
Brad Myers
IBM
SMART Technologies
VividLogic
Synergy Solutions
Handango
Carnegie Mellon
45
Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
Handheld Devices for Control
Brad A. Myers
Jeffrey Nichols, Jacob O. Wobbrock,
Kevin Litwack, Michael Higgins, Joe
Hughes, Thomas K. Harris, Roni
Rosenfeld, Mathilde Pignol
bam@cs.cmu.edu
http://www.cs.cmu.edu/~pebbles
Black, followed by OLD
Discussion of Comparison

Our hand designed interfaces
succeeded for several reasons




Good organization
Each button has one function
Good labels
Only available functions are active


Others, hidden on tabs or grayed out
Better feedback and error messages
Brad Myers
Carnegie Mellon
48
PUC Architecture
Brad Myers
Carnegie Mellon
49
Adaptor using Custom Hardware for Stereo



Pretends to send IR codes
Reads LED panel signals to decode state
Created by Pittsburgh company: Maya Design
Brad Myers
Carnegie Mellon
50