Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
Mobile Devices for Control of
Ubiquitous Multimedia
Brad A. Myers
bam@cs.cmu.edu
http://www.pebbles.hcii.cmu.edu/
Handheld Devices

Typically, handheld devices for:




Communication
As a “Personal Information Manager”
As a “Personal Digital Assistant”
As a PC replacement



“PocketPC”
As a media player
Handheld Devices for Remote Control of other
devices and media


Remote Control of PCs
Remote Control of appliances
Brad Myers
Carnegie Mellon
2
Problem

Appliances are too complex
Brad Myers
Carnegie Mellon
3
Problem

Too many remotes
Brad Myers
Carnegie Mellon
4
Problem

April 29, 1991
Brad Myers
Carnegie Mellon
5
Approach

Use a personal handheld device as an
Interface to the PC and to the Appliances
Brad Myers
Carnegie Mellon
6
History of Communication

ParcTab – continuous communication

Early “organizers” – no communication

Newton – extra cost for synchronization

Palm – HotSync once a day

WindowsCE/PocketPC –
ActiveSync once a day
Brad Myers
Carnegie Mellon
7
Now, Handhelds are Communicating
Toshiba e740

802.11
HP iPaq 3870

BlueTooth

Cell-phone network (G3)
Microsoft SmartPhone
Brad Myers
Carnegie Mellon
8
Pebbles Project


Performed as part of the Pebbles Project
Overall goal: Use of multiple devices at the
same time






Palm and desktop PC
Mobile Phone and “Smart Home”
Multiple handhelds in a meeting
Pocket PC and appliances
Multiple users with their devices
Single user with multiple devices
Brad Myers
Carnegie Mellon
9
Pebbles is:
P DAs for
E ntry of
B oth
B ytes and
L ocations from
E xternal
S ources.
http://www.pebbles.hcii.edu/
Brad Myers
Carnegie Mellon
10
Pebbles for MultiMedia

How will handheld devices fit into the user’s
overall media space?




Where does media come from?
Where is the media is displayed?
How is the media controlled?
How can handheld devices improve the user
interfaces of other media devices, rather
than just being another gadget to master?
Brad Myers
Carnegie Mellon
11
Controlling a PC
Remote Commander
 Slide Show Commander
 Command Post of the Future
 Shortcutter

http://www.pebbles.hcii.edu/
Remote Commander

Allow PDAs to control a PC
Can be used with any application
 Uses the standard (single) cursor
 Don’t have to jump up and
grab mouse
 All mouse and keyboard functions

Use PDA like touchpad
 Graffiti or our own pop-up
keyboard


Word prediction and completion
Brad Myers
Carnegie Mellon
13
PocketPC version


Get PC’s screen onto PocketPC
Full view, or one-to-one zooming


Scroll with iPaq’s buttons
Control or scribble
Brad Myers
Carnegie Mellon
14
For People with Muscular Disabilities

Using handhelds as interface to PCs

People with Muscular Dystrophy
have fine-motor control but lose
gross motor control


Difficulties with mouse and keyboard,
but stylus OK
So use Remote Commander
as PC’s keyboard and mouse
Brad Myers
Carnegie Mellon
15
Alphabet for Accurate Gestures

Use a square hole in a template to increase
accuracy




Just as easy to learn
as Graffiti™
18% more accurate
for able novices (p<.02)
Patent Pending
Also works with joystick
Brad Myers
Carnegie Mellon
16
Power Point Control


Use PC to give the presentation
Use hand-held to control the PC


Two-way communication
Hand-held shows picture of slide,
notes, timer
Brad Myers
Carnegie Mellon
17
SlideShow Commander, cont.





See list of slide titles
Scribble
Tap on on-screen buttons
and hyper-links
Preview other
slides
Control
demonstrations
Brad Myers
Carnegie Mellon
18
Control at a Distance

Controlling a PC from
across the room


Meetings, etc.
Use a laser pointer?

Studies
Brad Myers
Carnegie Mellon
19
Command Post of the Future

Share media & control across multiple devices


Big displays, speech, gesture
And handhelds
Brad Myers
Carnegie Mellon
20
CPOF, cont.




Provides remote control of main PC
battleboard from handhelds
Also, private drill-down of public information
Small versions of map and data
visualizations on handhelds
Interact locally, and display
changes and annotations
when ready
Brad Myers
Carnegie Mellon
21
Shortcutter


User-created panels of controls
Create custom interfaces and
extensions to PC applications



And then take them with you
Direct manipulation
for edit, then set
properties
PocketPC or Palm
Brad Myers
Carnegie Mellon
22
Shortcutter Widgets

Buttons

Virtual Joy stick

Virtual Knob

Sliders

Mouse pad

Gesture panel
Brad Myers
Carnegie Mellon
23
Scenarios of Use

Lean-back mail reading

Controlling media players

… and many others
Brad Myers
Carnegie Mellon
24
Controlling Appliances

PhD research of Jeffrey Nichols

Assisted by undergrads and MS students
http://www.pebbles.hcii.edu/puc/
Problem

Too many complex devices, each
with its own idiosyncratic interface








Stereo system
Telephones
ATM
Fax machine
Photocopier
Hotel alarm clock
Increasingly computerized
Low usability
Existing “universal” controls
Pre-programmed at the factory with a subset,
 Or, laboriously hand-programmed by the user

Brad Myers
Carnegie Mellon
27
Our Approach


Handheld is a “Personal Universal Controller” (PUC)
Two-way communication


Appliance describes its functions
Handheld PUC
 Automatically creates interface
 Controls the appliance
 Displays feedback about appliance status
Specifications
Control
Feedback
Brad Myers
Carnegie Mellon
28
Automatic Generation of UIs

Benefits

All interfaces consistent for a user
With conventions of the handheld
 Even from multiple manufacturers


Addresses hotel alarm clock problem
Can take into account user preferences
 Multiple modalities (GUI + Speech UI)


A Hard Problem

Previous automatic systems have not
generated high quality interfaces
Brad Myers
Carnegie Mellon
29
Research Challenges
1. Automatic Design of Graphical User
Interfaces
2. Automatic Design of Speech User Interfaces
3. Connection with real devices



Through various protocols
X-10, AV/C, HAVi, UPnP, etc.
Also, custom hardware and software
Brad Myers
Carnegie Mellon
30
Hand-Generated Graphical Interfaces

First, Hand-Designed PocketPC interfaces:


AIWA Shelf Stereo (Tape,CD,Tuner)
AT&T Telephone/Answering
Machine

Used Embedded Visual Basic

Ensured quality with heuristic
analysis and think-aloud studies

Compared with manufacturer’s
interfaces
Brad Myers
Carnegie Mellon
31
Results of Comparison

Using PUC, users took 50% less time &
made 50% fewer errors

All differences are significant (p < 0.05)
Brad Myers
Carnegie Mellon
32
Current PUC Specification Language


XML
Full documentation for the
specification language
and protocol:
http://www.pebbles.hcii.cmu.edu/puc/

Contains sample
specification for a stereo
Brad Myers
Carnegie Mellon
33
Properties of PUC Language

State variables & commands

Each can have multiple labels


Typed variables



Useful when not enough room
Base types: Boolean, string,
enumerated, integers,
fixed-point, floating-point, etc.
Optional labels for values
Hierarchical Structure

Groups
Brad Myers
Carnegie Mellon
34
Dependency Information


Crucial for high-quality interfaces
Expressed as <active-if> clauses

Operations:


Combined Logically


Equals, Less-Than,
Greater-Than
AND, OR
Used for:



Dynamic graying out
Layout
Widget selection
Brad Myers
Carnegie Mellon
35
Specifications

Have working specifications for:









Audiophase stereo
X-10 lights control
Sony CamCorder
Windows Media Player
Audio ReQuest hardware MP3 player
WinAmp Media Player
Elevator
Parts of GMC Yukon Denali SUV
Etc.
Brad Myers
Carnegie Mellon
36
Controller Generators

iPaq PocketPC

SmartPhone

No touchscreen

Desktop (TabletPC)

Speech
Brad Myers
Carnegie Mellon
37
Examples of Generated GUIs
Shelf stereo
Brad Myers
Windows Media Player
Carnegie Mellon
38
Generating Speech Interfaces

“Universal Speech Interface” (USI) project



Creates grammar, language model and
pronunciation dictionary from PUC specification




Prof. Roni Rosenfeld of CMU
http://www.cs.cmu.edu/~usi
Pronunciation from labels using phonetic rules
Can provide other pronunciations as labels for fine-tuning
Will use dependency information to help with
disambiguation and explanation
Supports queries and spoken feedback

Paraphrases as confirmation
Brad Myers
Carnegie Mellon
39
Adaptors

“Adaptors” provide the interface to existing
(and future) appliances
If do not support specification language directly
Custom hardware
 AV/C (standard protocol)
 Sony CamCorder
Custom software
 Lutron
 HAVi
 Windows Media Player
 UPnP
X-10
 Axis Camera





Light switches, etc.
Brad Myers
Carnegie Mellon
40
Demonstration




Sony CamCorder
Windows Media Player
Image library
Two way communication:


When state changes from application or
appliance, GUI is updated
Can query state with speech
Brad Myers
Carnegie Mellon
41
Current Work on PUC

Controlling User “Experiences”




Consistency for Users


One command for multiple devices
Play a DVD  sets DVD, cable, TV
Setting alarm clock  sets heat, coffee, car
And user preferences
More devices


UPnP
Etc.
Brad Myers
Carnegie Mellon
42
For More Information

Many papers on the Pebbles web site


On PUC and other work
Most programs available for free
downloading:
http://www.cs.cmu.edu/~pebbles

SlideShow Commander is
licensed for commercial sale

See handango.com
Brad Myers
Carnegie Mellon
43
Conclusions

As more and more electronics are
computerized and communicating, mobile
devices can help control them

Handheld devices can improve the user
interfaces of everything else

Not just be another gadget to be learned
Brad Myers
Carnegie Mellon
44
Thanks to Pebbles Students

This research performed by a large number
of students since 1997:
Htet Htet Aung
Rishi Bhatnagar
Ben Bostwick
Franklin Chen
Yu Shan Chuang
Karen Cross
Carl Evankovich
Ivan Gonzalez
Marc Khadpe
Dave Kong
Chun-Kwok Lee
Joonhwan Lee
Brad Myers
Jennifer Li
Yuhua Li
Leo Lie
Jack Lin
Kevin Litwack
A. Chris Long
Colin McCabe
Rob Miller
Jeff Nichols
Choon Hong Peck
Mathilde Pignol
Suporn Pongnumkul
Carnegie Mellon
Brandon Rothrock
Rajesh Seenichamy
Pegeen Shen
Herbert Stiel
Jeff Stylos
Claire Tokar
Marsha Tjandra
Adrienne Warmack
Jacob O. Wobbrock
Jerry Yang
Sunny Yang
Brian Yeung
45
Thanks to Our Sponsors!
The Pebbles research is supported by grants from:
DARPA
NSF
Microsoft
General Motors
NEC Foundation
Pittsburgh Digital Greenhouse
And equipment grants from:
Hewlett Packard
Lucent Technologies
Lantronix, Inc.
Lutron, Inc.
Mitsubishi
Palm Computing
Brad Myers
Symbol Technologies
IBM
SMART Technologies
VividLogic
Synergy Solutions
Handango
Carnegie Mellon
46
Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
Mobile Devices for Control of
Ubiquitous Multimedia
Brad A. Myers
bam@cs.cmu.edu
http://www.pebbles.hcii.cmu.edu/