Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
Handheld Devices for Control
Brad A. Myers
bam@cs.cmu.edu
http://www.cs.cmu.edu/~pebbles
Handheld Devices

Handheld Devices for:




Communication
As a “Personal Information Manager”
As a “Personal Digital Assistant”
As a PC replacement


“PocketPC”
Handheld Devices for Remote Control of
other devices


Remote Control of PCs
Remote Control of appliances
Brad Myers
Carnegie Mellon
2
Problem

Appliances are too complex
Brad Myers
Carnegie Mellon
3
Problem

Too many remotes
Brad Myers
Carnegie Mellon
4
Problem

April 8, 1991
Brad Myers
Carnegie Mellon
5
Approach

Use a personal handheld device as an
Interface to the PC and to the Appliances
Brad Myers
Carnegie Mellon
6
History of Communication

ParcTab – continuous communication

Early “organizers” – no communication

Newton – extra cost for synchronization

Palm – HotSync once a day

WindowsCE/PocketPC –
ActiveSync once a day
Brad Myers
Carnegie Mellon
7
Now, Handhelds will be Communicating
Toshiba e740

802.11
HP iPaq 3870

BlueTooth

Cell-phone network (G3)

(Infrared)

(Wires or cradle)
Microsoft SmartPhone
Brad Myers
Carnegie Mellon
8
Premises of our Research
“With the coming wireless technologies,
connecting the PCs and PDAs together
will no longer be an occasional event
for synchronization. Instead, the
devices will frequently be in
close, interactive communication.”
—
Brad Myers
Brad Myers, “Using Hand-Held Devices and
PCs Together,” Comm. ACM,
Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.
Carnegie Mellon
9
Pebbles Project


Performed as part of the Pebbles Project
Overall goal: Use of multiple devices at the
same time






Palm and desktop PC
Mobile Phone and “Smart Home”
Multiple handhelds in a meeting
Pocket PC and appliances
Multiple users with their devices
Single user with multiple devices
Brad Myers
Carnegie Mellon
10
Pebbles is:
P DAs for
E ntry of
B oth
B ytes and
L ocations from
E xternal
S ources.
http://www.pebbles.hcii.edu/
Brad Myers
Carnegie Mellon
11
Controlling a PC
Remote Commander
 Slide Show Commander
 “Semantic Snarfing”
 Shortcutter

http://www.pebbles.hcii.edu/
Remote Commander

Allow PDAs to control a PC
Can be used with any application
 Uses the standard (single) cursor
 Don’t have to jump up and
grab mouse
 All mouse and keyboard functions

Use PDA like touchpad
 Graffiti or our own pop-up
keyboard


Word prediction and completion
Brad Myers
Carnegie Mellon
13
PocketPC version


Get PC’s screen onto PocketPC
Full view, or one-to-one zooming


Scroll with iPaq’s buttons
Control or scribble
Brad Myers
Carnegie Mellon
14
For People with Muscular Disabilities

Using handhelds as interface to PCs

People with Muscular Dystrophy
have fine-motor control but lose
gross motor control


Difficulties with mouse and keyboard,
but stylus OK
So use Remote Commander
as PC’s keyboard and mouse
Brad Myers
Carnegie Mellon
15
NEW!

Alphabet for Accurate Gestures
Use a square hole in a template to increase
accuracy




Just as easy to learn
as Graffiti™
18% more accurate
for able novices (p<.02)
Patent Pending
Also works with joystick
Brad Myers
Carnegie Mellon
16
Shared Clipboard

Transfer information among your personal
devices





E.g., between PDA and PC
Also between multiple computers
Connect all clipboards together
Transfer content or reference
Works with all applications
Brad Myers
Carnegie Mellon
17
Power Point Control


Use PC to give the presentation
Use hand-held to control the PC


Two-way communication
Hand-held shows picture of slide,
notes, timer
Brad Myers
Carnegie Mellon
18
SlideShow Commander, cont.





See list of slide titles
Scribble
Tap on on-screen buttons
and links
Preview other
slides
Control
demonstrations
Brad Myers
Carnegie Mellon
19
Control at a Distance

Controlling a PC from
across the room


Use a laser pointer?


Meetings, etc.
Studies
“Semantic Snarfing”

Grab contents to handheld
Brad Myers
Carnegie Mellon
20
Shortcutter


User-created panels of controls
Create custom interfaces and
extensions to PC applications



And then take them with you
Direct manipulation
for edit, then set
properties
PocketPC or Palm
Brad Myers
Carnegie Mellon
21
Shortcutter Widgets

Buttons

Virtual Joy stick

Virtual Knob

Sliders

Mouse pad

Gesture panel
Brad Myers
Carnegie Mellon
22
Scenarios of Use

Lean-back mail reading

Controlling media players

… and many others
Brad Myers
Carnegie Mellon
23
Controlling Appliances

PhD research of Jeffrey Nichols

Assisted by undergrads and MS students
http://www.pebbles.hcii.edu/puc/
Problem

Too many complex devices, each
with its own idiosyncratic interface








Stereo system
Telephones
ATM
Fax machine
Photocopier
Hotel alarm clock
Increasingly computerized
Low usability
Existing “universal” controls
Pre-programmed at the factory with a subset,
 Or, laboriously hand-programmed by the user

Brad Myers
Carnegie Mellon
26
Our Approach


Handheld is a “Personal Universal Controller” (PUC)
Two-way communication


Appliance describes its functions
Handheld PUC
 Automatically creates interface
 Controls the appliance
 Displays feedback about appliance status
Specifications
Control
Feedback
Brad Myers
Carnegie Mellon
27
Automatic Generation of UIs

Benefits

All interfaces consistent for a user
With conventions of the handheld
 Even from multiple manufacturers


Addresses hotel alarm clock problem
Can take into account user preferences
 Multiple modalities (GUI + Speech UI)


A Hard Problem

Previous automatic systems have not
generated high quality interfaces
Brad Myers
Carnegie Mellon
28
Research Challenges
1. Automatic Design of Graphical User
Interfaces
2. Automatic Design of Speech User Interfaces
3. Connection with real devices



Through various protocols
X-10, AV/C, HAVi, UPnP, etc.
Also, custom hardware and software
Brad Myers
Carnegie Mellon
29
Hand-Generated Graphical Interfaces

First, Hand-Designed PocketPC interfaces:


AIWA Shelf Stereo (Tape,CD,Tuner)
AT&T Telephone/Answering
Machine

Used Embedded Visual Basic

Ensured quality with heuristic
analysis and think-aloud studies

Compared with manufacturer’s
interfaces
Brad Myers
Carnegie Mellon
30
Results of Comparison

Using PUC, users took 50% less time &
made 50% fewer errors

All differences are significant (p < 0.05)
Brad Myers
Carnegie Mellon
31
Current PUC Specification Language


XML
Full documentation for the
specification language
and protocol:
http://www.pebbles.hcii.cmu.edu/puc/

Contains sample
specification for a stereo
Brad Myers
Carnegie Mellon
32
Properties of PUC Language

State variables & commands

Each can have multiple labels


Typed variables



Useful when not enough room
Base types: Boolean, string,
enumerated, integers,
fixed-point, floating-point, etc.
Optional labels for values
Hierarchical Structure

Groups
Brad Myers
Carnegie Mellon
33
Dependency Information


Crucial for high-quality interfaces
Expressed as <active-if> clauses

Operations:


Combined Logically


Equals, Less-Than,
Greater-Than
AND, OR
Used for:



Dynamic graying out
Layout
Widget selection
Brad Myers
Carnegie Mellon
34
Specifications

Have working specifications for:









Audiophase stereo
X-10 lights control
Sony CamCorder
Windows Media Player
Audio ReQuest hardware MP3 player
WinAmp Media Player
Elevator
Parts of GMC Yukon Denali SUV
Etc.
Brad Myers
Carnegie Mellon
35
Controller Generators

iPaq PocketPC

SmartPhone

No touchscreen

Desktop (TabletPC)

Speech
Brad Myers
Carnegie Mellon
36
Examples of Generated GUIs
Shelf stereo
Brad Myers
Windows Media Player
Carnegie Mellon
37
Generating Speech Interfaces

“Universal Speech Interface” (USI) project



Creates grammar, language model and
pronunciation dictionary from PUC specification




Prof. Roni Rosenfeld of CMU
http://www.cs.cmu.edu/~usi
Pronunciation from labels using phonetic rules
Can provide other pronunciations as labels for fine-tuning
Will use dependency information to help with
disambiguation and explanation
Supports queries and spoken feedback

Paraphrases as confirmation
Brad Myers
Carnegie Mellon
38
Adaptors

“Adaptors” provide the interface to existing
(and future) appliances
If do not support specification language directly
Custom hardware
 AV/C (standard protocol)
 Sony CamCorder
Custom software
 Lutron
 HAVi
 Windows Media Player
 UPnP
X-10
 Axis Camera





Light switches, etc.
Brad Myers
Carnegie Mellon
39
Demonstration




Sony CamCorder
Windows Media Player
Simulated automobile
Two way communication:


When state changes from application or
appliance, GUI is updated
Can query state with speech
Brad Myers
Carnegie Mellon
40
Future Work on PUC

Controlling User “Experiences”




Consistency for Users


One command for multiple devices
Play a DVD  sets DVD, cable, TV
Setting alarm clock  sets heat, coffee, car
And user preferences
More devices


UPnP
Etc.
Brad Myers
Carnegie Mellon
41
For More Information

Many papers on the Pebbles web site


On PUC and other work
Most programs available for free
downloading:
http://www.cs.cmu.edu/~pebbles

SlideShow Commander was
licensed for commercial sale
Brad Myers
Carnegie Mellon

42
Conclusions

As more and more electronics are
computerized and communicating, mobile
devices can help control them

Handheld devices can improve the user
interfaces of everything else

Not just be another gadget to be learned
Brad Myers
Carnegie Mellon
43
Thanks to Pebbles Students

This research performed by a large number
of students since 1997:
Htet Htet Aung
Rishi Bhatnagar
Ben Bostwick
Franklin Chen
Yu Shan Chuang
Karen Cross
Carl Evankovich
Marc Khadpe
Dave Kong
Chun-Kwok Lee
Joonhwan Lee
Brad Myers
Jennifer Li
Yuhua Li
Leo Lie
Jack Lin
Kevin Litwack
A. Chris Long
Rob Miller
Jeff Nichols
Choon Hong Peck
Mathilde Pignol
Suporn Pongnumkul
Carnegie Mellon
Rajesh Seenichamy
Pegeen Shen
Herbert Stiel
Jeff Stylos
Claire Tokar
Marsha Tjandra
Adrienne Warmack
Jacob O. Wobbrock
Jerry Yang
Sunny Yang
Brian Yeung
44
Thanks to Our Sponsors!
The Pebbles research is supported by grants from:
DARPA
Microsoft
NSF
General Motors
NEC Foundation
Pittsburgh Digital Greenhouse
And equipment grants from:
Hewlett Packard
Lucent Technologies
Lantronix, Inc.
Lutron, Inc.
Mitsubishi
Palm Computing
Brad Myers
Symbol Technologies
IBM
SMART Technologies
VividLogic
Synergy Solutions
Handango
Carnegie Mellon
45
Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
Handheld Devices for Control
Brad A. Myers
bam@cs.cmu.edu
http://www.cs.cmu.edu/~pebbles
Black, followed by HAVi
Pebbles PUC Project use of the
Home Audio Visual Interface
About HAVi

Home Audio Visual Interface


Protocol for controlling appliances over
IEEE 1394 (i.LINK® or FireWire®) cables


http://www.havi.org
Mostly used for video equipment
Supported by various Mitsubishi and RCA
high-end products
Brad Myers
Carnegie Mellon
49
Our Goals for HAVi

Want to:




Detect HAVi devices on Firewire bus
Automatically create PUC specification from
device capabilities
Control all of the appliance’s features using a PC
as a server (proxy)
HAVi seems to have appropriate features for
these
Brad Myers
Carnegie Mellon
50
HAVi Experience

MERL donated



A 65-inch high-definition
TV (WS-65909)
A high definition digital
VCR (HS-HD2000U)
VividLogic FireBus HAVi
SDK for Linux, v. 1.0.8
Brad Myers
Carnegie Mellon
51
What We’ve Done

Investigated the (several) different ways of
controlling HAVi devices


Made simple applications to control HAVi
devices
But don’t have a working PUC interface

Have put efforts on hold


Limitations of HAVi
Problems with SDK
Brad Myers
Carnegie Mellon
52
Television



Under the HAVi system, acts as the “master
controller”
Refuses to be controlled or to respond to our
requests
Even having the TV and
PC on same bus causes
problems
Brad Myers
Carnegie Mellon
53
VCR

Several different ways to control VCR

VCR FCM



Havlet


“Functional Control Module”
Static list of capabilities
A Java program uploaded and run on TV (or PC)
DDI


“Data-Driven Interaction”
Panel-based interface for arbitrary control
Brad Myers
Carnegie Mellon
54
VCR FCM

A few more features than AV/C:





Play, Record, Fast Forward, Fast Reverse,
Variable Forward, Variable Reverse, Stop,
Record-Pause, Skip, Eject Media, Set Recording
Mode, ClearRTC
Get State, Get Recording Mode, Get Format,
Get Position, Get capability, Get Reject Info
No querying for capabilities
(Apparent) Bugs in SDK on getting status
Have created simple programs to control
VCR using the FCM
Brad Myers
Carnegie Mellon
55
HavLets



Java Applets downloaded to a Java interpreter
No hooks for programmatic control
Very limited functions provided in the VCR’s HavLet


Rewind, Play, Fast Forward, Stop, Pause, Record
Status, Counter, Media Type, Recording Rate of Tape,
Rate of Data Source
Brad Myers
Carnegie Mellon
56
Data-Driven Interaction (DDI)


We hoped this would provide device
querying and control, but…
Only limited features provided in DDI



Same as HavLet interface
Rewind, Play, Fast Forward, Stop, Pause,
Record, and some status requests
Provides each UI element as a separate item


E.g., “Time:
” and “54:35:33” may be two
separate labels placed next to each other
Need to infer relationships and types by layout
Brad Myers
Carnegie Mellon
57
Issues with VividLogic’s SDK




Their engineers have been very helpful
SDK incomplete, poorly documented and
quite buggy
Only runs with a specific desktop Firewire
card, only on Linux (for now)
Slow to come out with new versions

Needed to be “FAV” (Full AV) to use DDI but FAV
support not available until Dec’02
Brad Myers
Carnegie Mellon
58
Demo


Show other devices we have gotten to work
demo
Brad Myers
Carnegie Mellon
59
Black, followed by OLD
Adaptor for AV/C
PUC Java Proxy
Hand-written
spec of AV/C
devices
PUC Java Library
Our Java device code
Multiple device
controllers
Automatic
GUI
generation
Automatic
Speech UI
generation
Our C++ code
Microsoft DirectShow
library
AV/C protocol
Firewire cable
(IEEE 1394)
Brad Myers
Digital Video
Camera or VCR
Carnegie Mellon
61
Adaptor using Custom Hardware for Stereo



Pretends to send IR codes
Reads LED panel signals to decode state
Created by Pittsburgh company: Maya Design
Brad Myers
Carnegie Mellon
62
Driver Information Console

Simulator for the Driver Information Console
of Yukon Denali SUV


All the modes and messages
Full emulation of steering wheel buttons
Brad Myers
Carnegie Mellon
63
Simulator for HVAC
Brad Myers
Carnegie Mellon
64