Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
Mobile Devices for Control
Brad A. Myers
bam@cs.cmu.edu
http://www.cs.cmu.edu/~pebbles
Mobile Devices




Mobile Devices for communication
As a “Personal Information Manager”
As a “Personal Digital Assistant”
As a PC replacement


“PocketPC”
For Remote Control of other devices


Remote Control of PCs
Remote Control of appliances
Brad Myers
2
Problem

Appliances are too complex
Brad Myers
3
Problem

Too many remotes
Brad Myers
4
Problem

User Interfaces of Mobile Devices
themselves are too hard to use
Brad Myers
5
Problem

PC’s have only a keyboard and mouse as
input devices
Brad Myers
6
Approach

Use a personal Mobile Device as an
Interface to the PC and to the Appliances
Brad Myers
7
History

ParcTab – continuous communication

Early “organizers” – no communication

Newton – extra cost for synchronization

Palm – HotSync once a day

WindowsCE/PocketPC –
ActiveSync once a day
Brad Myers
8
Now, Handhelds will be Communicating
Toshiba e740

802.11 (“Wi-Fi”)

BlueTooth

Cell-phone network (G3)

(Infrared)

(Wires or cradle)
HP iPaq 3870
Microsoft SmartPhone
Brad Myers
9
Premises of our Research
“With the coming wireless technologies,
connecting the PCs and PDAs together
will no longer be an occasional event
for synchronization. Instead, the
devices will frequently be in
close, interactive communication.”
—
Brad Myers
Brad Myers, “Using Hand-Held Devices and
PCs Together,” Comm. ACM,
Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.
10
Pebbles Project


Performed as part of the Pebbles Project
Overall goal: Use of multiple devices at the
same time






Palm and desktop PC
Mobile Phone and “Smart Home”
Multiple handhelds in a meeting
Pocket PC and appliances
Multiple users with their devices
Single user with multiple devices
Brad Myers
11
Pebbles is:
P DAs for
E ntry of
B oth
B ytes and
L ocations from
E xternal
S ources.
http://www.pebbles.hcii.edu/
Brad Myers
12
Controlling a PC
Remote Commander
 Slide Show Commander
 “Semantic Snarfing”
 Scrolling
 Shortcutter

http://www.pebbles.hcii.edu/
Important Work by Others



PARCTab [Want 95], [Weiser, CACM’93]
Abowd: eClass using Handhelds [CHI’98]
Greenberg: public and private data
[Personal Technologies, 3(1), 1999]

Jun Rekimoto: [UIST’97, CHI’98, CHI’99, …]

Pick and drop;
handhelds with
big displays
Brad Myers
14
Remote Commander

Allow PDAs to control a PC
Can be used with any application
 Uses the standard (single) cursor
 Don’t have to jump up and
grab mouse
 All mouse and keyboard functions

Use PDA like touchpad
 Graffiti or our own pop-up
keyboard


Word prediction and completion
Brad Myers
15
For People with Muscular Disabilities

Using handhelds as interface to PCs

People with Muscular Dystrophy
have fine-motor control but lose
gross motor control


Difficulties with mouse and keyboard,
but stylus OK
So use Remote Commander
as PC’s keyboard and mouse
Brad Myers
16
PocketPC version


Get PC’s screen onto PocketPC
Full view, or one-to-one zooming


Scroll with iPaq’s buttons
Control or scribble
Brad Myers
17
Power Point Control


Use PC to give the presentation
Use hand-held to control the PC


Two-way communication
Hand-held shows picture of slide,
notes, timer
Brad Myers
18
SlideShow Commander, cont.





See list of slide titles
Scribble
Tap on on-screen buttons
and links
Preview other
slides
Control
demonstrations
Brad Myers
19
Control at a Distance

Controlling a PC from
across the room


Meetings, etc.
Use a laser pointer?


Studies
Difficult due to:



Jitter of hand (+/- 10 pixels)
No button on the laser pointer
People not accurate at where
beam will go on and off
Brad Myers
20
Idea:“Semantic Snarfing”

Grab contents to handheld


Picture
“Semantic” because
reformat based on the
meaning of the contents



Menus
Text
Re-visualization
Brad Myers
21
Command Post of the Future



Large DARPA funded project
Make commanders more effective
Private drill-down of public information
Brad Myers
22
Classroom Investigations

Investigate improving large lectures with
frequent in-class “concept tests”


Provide immediate feedback to instructor
Easy for instructor
Brad Myers
23
Scrolling Desktop Applications

Scrolling using the non-dominant hand



Parallel and efficient use of both hands
Generates Windows
Time to Scroll 10 Pages
Sec
scrolling events
100
Study showed can be
faster for some tasks
80
60
40
20
0
Trial 2
Brad Myers
ButtonScroller
Mouse
AbsScroller
Trial 3
SlideScroller
Scroll Wheel
RateScroller
24
Shortcutter


User-created panels of controls
Create custom interfaces and
extensions to PC applications



And then take them with you
Direct manipulation
for edit, then set
properties
PocketPC or Palm
Brad Myers
25
Shortcutter Widgets

Buttons

Virtual Joy stick

Virtual Knob

Sliders

Mouse pad

Gesture panel
Brad Myers
26
Shortcutter Actions




Send any keyboard key, mouse button,
scrolling action or string to PC
Open a file or URL
Run an application
Invoke any PC menu or button




Windows message
Recorded
Switch to a different Shortcutter panel
Control the Mouse
Brad Myers
27
Shortcutter Actions, cont.

Control external devices
through PC’s serial port



Macro


Directly (e.g., projectors)
X-10 for electrical devices
Can be multi-application
Application-specific


Same button, different messages
Useful for application sets:
browsers, compilers
Brad Myers
28
More Scenarios of Use

Lean-back mail reading

Controlling media players

… and many others
Brad Myers
29
Controlling Appliances

PhD research of Jeffrey Nichols
http://www.pebbles.hcii.edu/puc/
Problem

Too many complex devices, each
with its own idiosyncratic interface








Stereo system
Telephones
ATM
Fax machine
Photocopier
Hotel alarm clock
Increasingly computerized
Low usability
Problem
Brad Myers
32
Existing “universal” controls
Pre-programmed at the factory with a subset,
 Or, Laboriously hand-programmed by the user

Brad Myers
33
Important Work by Others


Xweb [Olsen Jr. 2000]
INCITS V2 standardization effort: Alternative
Interface Access Protocol (AIAP)
[Zimmermann, CHI’02]

Stanford iRoom, iCrafter
[Ponnekanti, UBICOMP 2001]
Brad Myers
34
Our Approach


Handheld is a “Personal Universal Controller” (PUC)
Two-way communication


Appliance describes its functions
Handheld PUC
 Automatically creates interface
 Controls the appliance
 Displays feedback about appliance status
Specifications
Control
Feedback
Brad Myers
35
Automatic Generation of UIs

Benefits

All interfaces consistent for a user
With conventions of the handheld
 Even from multiple manufacturers


Addresses hotel alarm clock problem
Can take into account user preferences
 Multiple modalities (GUI + Speech UI)


A Hard Problem

Previous automatic systems have not
generated high quality interfaces
Brad Myers
36
Research Challenges
1. Automatic Design of Graphical User
Interfaces
2. Automatic Design of Speech User Interfaces
3. Connection with real devices



Through various protocols
X-10, AV/C, HAVi, UPnP, etc.
Also, custom hardware and software
Brad Myers
37
Hand-Generated Graphical Interfaces

First, Hand-Designed PocketPC interfaces:


AIWA Shelf Stereo (Tape,CD,Tuner)
AT&T Telephone/Answering
Machine

Used Embedded Visual Basic

Ensured quality with heuristic
analysis and think-aloud studies

Compared with manufacturer’s
interfaces
Brad Myers
38
Results of Comparison

Using PUC, users took 50% less time &
made 50% fewer errors

All differences are significant (p < 0.05)
Brad Myers
39
Discussion of Comparison

Our hand designed interfaces
succeeded for several reasons




Good organization
Each button has one function
Good labels
Only available functions are active


Others, hidden on tabs or grayed out
Better feedback and error messages
Brad Myers
40
Current PUC Specification Language


XML
Full documentation for the
specification language
and protocol:
http://www.pebbles.hcii.cmu.edu/puc/

Contains sample
specification for a stereo
Brad Myers
41
Properties of PUC Language

State variables & commands

Each can have multiple labels


Typed variables



Useful when not enough room
Base types: Boolean, string,
enumerated, integers,
fixed-point, floating-point, etc.
Optional labels for values
Hierarchical Structure

Groups
Brad Myers
42
Dependency Information


Crucial for high-quality interfaces
Expressed as <active-if> clauses

Operations:


Combined Logically


Equals, Less-Than,
Greater-Than
AND, OR
Used for:



Dynamic graying out
Layout
Widget selection
Brad Myers
43
Specifications

Have working specifications for:







Audiophase stereo
X-10 lights control
Sony CamCorder
Windows Media Player
Audio ReQuest hardware MP3 player
WinAmp Media Player
Elevator
Brad Myers
44
Examples of Generated GUIs

Stereo and X-10
Brad Myers
45
More Examples

Elevator
Brad Myers
46
Generating Speech Interfaces

“Universal Speech Interface” (USI) project



Creates grammar, language model and
pronunciation dictionary from PUC specification




Prof. Roni Rosenfeld of CMU
http://www.cs.cmu.edu/~usi
Pronunciation from labels using phonetic rules
Can provide other pronunciations as labels for fine-tuning
Will use dependency information to help with
disambiguation and explanation
Supports queries and spoken feedback

Paraphrases as confirmation
Brad Myers
47
PUC Architecture
Brad Myers
48
Adaptors

“Adaptors” provide the interface to existing
(and future) appliances
If do not support specification language directly
 AV/C (standard protocol)
Custom hardware






Custom software



Audiophase Stereo
ARQ MP3 player
SIMA MP3 player (future)
X-10

Light switches, etc.
Brad Myers

HAVi (current work)



Sony CamCorder
PlayStation, etc. (future)
Mitsubishi HDTV 65” TV
Mitsubishi HDTV VCR
UPnP (future)

??
49
Adaptor using Custom Hardware for Stereo



Pretends to send IR codes
Reads LED panel signals to decode state
Created by Pittsburgh company: Maya Design
Brad Myers
50
Adaptor for AV/C
PUC Java Proxy
Hand-written
spec of AV/C
devices
Multiple device
controllers
PUC Java Library
Our Java device code
Automatic
GUI
generation
Automatic
Speech UI
generation
Our C++ code
Microsoft DirectShow
library
AV/C protocol
Firewire cable
(IEEE 1394)
Brad Myers
Digital Video
Camera or VCR
51
Demonstration



X-10
Camcorder through AV/C protocol for
IEEE 1394 (Firewire)
Two way communication:

When state changes from appliance, GUI is
updated and can query state with speech
Brad Myers
52
For More Information

Many papers on the Pebbles web site


On PUC and other work
Most programs available for free
downloading:
http://www.cs.cmu.edu/~pebbles

SlideShow Commander available
commercially
http://www.slideshowcommander.com
Brad Myers
53
Conclusions

Study The Big Picture:

How mobile devices fit into users’ entire
information and control space

As more and more electronics are
computerized, mobiles devices can help
control them

Mobile devices can improve the user
interfaces of everything else

Not just be another gadget to be learned
Brad Myers
54
Thanks to Our Sponsors!
The Pebbles research project is supported by
grants from:
DARPA
Microsoft
NSF
Pittsburgh Digital Greenhouse
General Motors
And equipment grants from:
Hewlett Packard
Lucent Technologies
Mitsubishi
Palm Computing
Symbol Technologies
Brad Myers
IBM
SMART Technologies
VividLogic
Synergy Solutions
Handango
55
Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
Mobile Devices for Control
Brad A. Myers
bam@cs.cmu.edu
http://www.cs.cmu.edu/~pebbles