Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
An Update on the Pebbles Project:
PocketPCs, Smartphones and
TabletPCs for Universal Remote Control
and
A New Input Method for Small Devices
Brad A. Myers
bam@cs.cmu.edu
http://www.cs.cmu.edu/~pebbles
July 30, 2003
Pebbles Project

Use of multiple devices at the same time






Pocket PC and desktop PC
Mobile Phone and “Smart Home”
Multiple handhelds in a meeting
Pocket PC and appliances
Multiple users with their devices
Single user with multiple devices
Brad Myers
2
Handhelds are communicating

802.11

BlueTooth

Cell-phone network (G3)

(Infrared)

(Wires or cradle)
Brad Myers
3
Premises of our Research
“With the coming wireless technologies,
connecting the PCs and PDAs together
will no longer be an occasional event
for synchronization. Instead, the
devices will frequently be in
close, interactive communication.”
—
Brad Myers
Brad Myers, “Using Hand-Held Devices and
PCs Together,” Comm. ACM,
Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.
4
Pebbles is:
P DAs for
E ntry of
B oth
B ytes and
L ocations from
E xternal
S ources.
http://www.pebbles.hcii.cmu.edu/
Brad Myers
5
Previously Reported Here


Microsoft has supported this research since ‘98
Prior presentations here about Pebbles:

For business meetings:



To augment desktop applications:


For individuals
For classrooms


For group work
How laser pointers can be integrated
To make lectures more interactive
For military command posts:

To facilitate communication and collaboration
Brad Myers
6
Example: Slide Show Commander


Released 1999
Remote Control of PowerPoint



Two-way communication
Picture of slide, notes, titles,
other applications, preview
Scribble, click on links
Brad Myers
7
Slide Show Commander, #2


List of applications
Preview of other slides

(Soon: continuous preview ahead)
Brad Myers
8
Download SlideShow Commander


Formerly was for sale, but no longer 
Research version distribution limited to
academics and Pebbles project supporters,
including Microsoft employees:
http://www.pebbles.hcii.cmu.edu/private/sscpocketpc
Brad Myers
9
Example 2: Shortcutter


User-created panels of controls
Create custom interfaces and
extensions to PC applications




Augment desktop applications
Useful for prototyping
Example: two handed interaction:
scrolling with left hand
Direct manipulation for edit, then set
properties
Brad Myers
10
Shortcutter Widgets

Buttons

Virtual Joystick

Virtual Knob

Sliders

Mouse pad

Gesture panel
Brad Myers
11
Scenarios of Use

Lean-back mail reading

Controlling media players

… and many others
Brad Myers
12
Personal Universal Controller

PhD research of Jeffrey Nichols
http://www.pebbles.hcii.cmu.edu/puc/
Today’s Talk

Personal Universal Controller

Last year, reported first steps

New this year:

Tablet PC, Smartphone

New devices: UPnP, WMP, Lutron, etc.

Ported to C# from Java


Portability across devices
Better quality UIs due to “high-level templates”
Brad Myers
14
Problem

Appliances are too complex
Brad Myers
15
Problem

Too many remotes
Brad Myers
16
Problem

April 8, 1991
Brad Myers
17
Approach

Use a personal handheld device as an
interface to the appliances
Brad Myers
18
Why Appliances Are Hard to Use

Too many complex devices, each with its
own idiosyncratic interface







Stereo system
Telephones
ATM
Fax machine
Photocopier
Hotel alarm clock
Increasingly computerized
Brad Myers
19
Existing “Universal” Controls
Pre-programmed at the factory with a subset,
 Or, Laboriously hand-programmed by the user

Brad Myers
20
Our Approach

Two-way communication


Appliance describes its functions
Handheld PUC
 Automatically creates interface
 Controls the appliance
 Displays feedback about appliance status
Specifications
Control
Feedback
Brad Myers
21
Automatic Generation of UIs

Benefits

All interfaces consistent on the controller
With conventions of the handheld
 Even from multiple manufacturers


Addresses hotel alarm clock problem
Can take into account user preferences
 Multiple modalities (GUI + Speech UI)


A Hard Problem

Previous automatic systems have not
generated high quality interfaces
Brad Myers
22
Initial User Study




Compared hand-designed interfaces to
manufacturers’
Stereo and telephone
Using PUC, users took 50% less time &
made 50% fewer errors
All differences are significant (p < 0.05)
Brad Myers
23
Properties of PUC Language


XML
State variables & commands

Each can have multiple labels


Typed variables



Useful when not enough room
Base types: Boolean, string,
enumerated, integers,
fixed-point, floating-point, etc.
Optional labels for values
Hierarchical Structure

Groups
Brad Myers
24
Dependency Information


Crucial for high-quality interfaces
Expressed as <active-if> clauses

Operations:


Combined Logically


Equals, Less-Than,
Greater-Than
AND, OR
Used for:



Layout
Widget selection
Dynamic graying out
Brad Myers
25
Specifications

Have working specifications for:
Windows Media Player
 Simulated Elevator
 Axis Pan-Tilt-Zoom camera through UPnP
 GMC Denali Driver Information Console
 GMC Denali Climate Control System
Key:
 GMC Denali Navigation System
Demonstrate
 Lutron lighting system
today
 Sony CamCorder
 X-10 lights control
Demonstrated
 Audiophase stereo
at Faculty
 Audio ReQuest hardware MP3 player
Summit
 WinAmp Media Player
Demonstrated
 UPnP Light Control
here last year26
Brad Myers

Generating Speech Interfaces

“Universal Speech Interface” (USI) project



Creates grammar, language model and
pronunciation dictionary from PUC specification




Prof. Roni Rosenfeld of CMU
http://www.cs.cmu.edu/~usi
Pronunciation from labels using phonetic rules
Can provide other pronunciations as labels for fine-tuning
Uses dependency information to help with
disambiguation and explanation
Supports queries and spoken feedback

Paraphrases as confirmation
Brad Myers
27
New Work: C#

Ported to C# with .NET Compact Framework



With support from Microsoft – Thanks!
Much faster execution than Java version on
PocketPC
Got portability across PocketPC,
SmartPhone, TabletPC (desktop)



Significant code reuse
90% from PocketPC to TabletPC
60% from PocketPC to SmartPhone

Different input model on SmartPhone requires
different generator rules and different widgets
Brad Myers
28
New Work: High-Level Templates

Templates for “high-level” types




Play control
Time/date
Light on, dim-level, off
Issue: Highly parameterized

E.g., what options available for play group




Tape: play, pause, stop, FF, Rew
CD: play, pause, stop, FF, Rew, next track, prev. track
Live audio feed: play, stop
Answering machine: play, play new, stop, …
Brad Myers
29
High-Level Templates

If PUC generator doesn’t understand
template, then falls back to primitive types

Primitive types also provide parameterization
CD:
Camera:
Brad Myers
30
Adaptors

“Adaptors” provide the interface to existing
(and future) appliances
If do not support specification language directly
Custom hardware
 AV/C (standard protocol)
 Sony CamCorder
Custom software
 Lutron
 HAVi
 Windows Media Player
 UPnP
X-10
 Axis Camera





Light switches, etc.
Brad Myers
31
New Work: New Devices





UPnP interface to Axis Camera
Lutron light interface
General Motors automobile
Windows Media Player
Simulated elevator, etc.
Brad Myers
32
Demonstration

New devices:




Windows Media Player through COM
Axis Camera through UPnP
Simulated Elevator
Simulated GMC Denali systems




Non-driving functions
GUI and speech
Cross platform: PocketPC, SmartPhone, TabletPC
Two way communication:

When state changes from appliance, GUI is updated and
can query state with speech
Brad Myers
33
Example Screens
Axis Camera
Brad Myers
Windows Media Player
34
Future Work

Remote Control for “Experiences”




Combined control of multiple appliances
Better quality generation
New devices?
Automatic creation of specifications for
UPnP devices?
Brad Myers
35
EdgeWrite™

PhD research of Jake Wobbrock
(patent pending)
http://www.pebbles.hcii.cmu.edu/assistive/
Origins


Prior work on using handhelds as input
devices for people with Muscular Dystrophy
Didn’t generalize to other disabilities
Brad Myers
37
Idea: Use Edges for Stability


First design: use edges around entire screen
User study: moving along edges and into
corners much more accurate

Especially for people with disabilities
Brad Myers
38
Initial Design



Place letters around top and sides of screen
Alphabetic layout and by letter frequency
Not effective 




Distances too long
Character sizes too small
Visual search times too large
Requires too much visual
attention
“the woman”
Brad Myers
39
Next Design: Smaller Area



Added custom template with a square hole
Designed alphabet using gestures inside of
hole, along edges and into corners
First implementation: for Palm
Brad Myers
40
Alphabet Properties

Recognizes based on sequence of corners



Users can wiggle during diagonals
Not speed dependent
Easy to customize and
adapt letter forms


Letters “feel” normal



Even user-driven
Use multiple strokes along edge
Presentation (only) uses arcs
No capitalization mode

Capitals end in upper left
Brad Myers
41
EdgeWrite Engineering

Change target area shapes/sizes



Start area vs. moving
Right-handed vs. left-handed
Plastic can’t sit on digitizer
Brad Myers
42
Current Alphabet Design

Also have
alternative
forms


104 unique
characters
288 total
character
forms
Brad Myers
43
Initial User Study

For able-bodied novices:



Just as learnable as Graffiti
Accuracy 18% better
Speed not significantly different
Brad Myers
44
User Study, Cont.

For handicapped users:





Succeeded with EdgeWrite when unable to use
Graffiti
Parkinson’s Disease
Cerebral Palsy
Muscular Dystrophy
For example:
 “The g i gbsiangu% fast” vs:
“The dog is going fast”
Brad Myers
45
Future EdgeWrite Work





Test with Xbox joystick
Test with other devices (watch, phone, etc.)
Test long-term (expert) use
Test on unstable platforms (riding on a bus or car)
Etc….
1
2
4
5
7
*
Brad Myers
8
0
3
12:03
6
9
#
46
For More Information

Many papers on the Pebbles site


On PUC and other work
Shortcutter and Remote Commander
available for free downloading:
http://www.cs.cmu.edu/~pebbles

SlideShow Commander available to
Microsoft employees:
http://www.cs.cmu.edu/~pebbles/private/
Brad Myers
47
Thanks to Our Sponsors!
Supported by grants from:
DARPA
NSF
Microsoft
Pittsburgh Digital Greenhouse
General Motors
NEC Foundation of America
And equipment grants from:
Hewlett Packard
Lucent Technologies
Mitsubishi
Palm Computing
Symbol Technologies
IBM and IBM Canada
SMART Technologies
Brad Myers
VividLogic
Synergy Solutions
TDK Systems Europe, LTD
Lutron, Inc.
Lantronix, Inc.
Handango
48
Thanks to Pebbles Students

This research performed by a large number
of students since 1997:
Rishi Bhatnagar
Ben Bostwick
Franklin Chen
Yu Shan Chuang
Karen Cross
Carl Evankovich
Marc Khadpe
Dave Kong
Chun-Kwok Lee
Joonhwan Lee
Brad Myers
Mathilde Pignol
Suporn Pongnumkul
Jennifer Li
Leo Lie
Jack Lin
Kevin Litwack
A. Chris Long
Rob Miller
Jeff Nichols
Choon Hong Peck
Rajesh Seenichamy
Herbert Stiel
Jeff Stylos
Claire Tokar
Marsha Tjandra
Adrienne Warmack
Jacob O. Wobbrock
Jerry Yang
Sunny Yang
Brian Yeung
49
Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
An Update on the Pebbles Project:
PocketPCs, Smartphones and
TabletPCs for Universal Remote Control
and
A New Input Method for Small Devices
Brad A. Myers
bam@cs.cmu.edu
http://www.cs.cmu.edu/~pebbles
July 30, 2003
Black slide, rest are extra
Discussion of Comparison

Our hand designed interfaces
succeeded for several reasons




Good organization
Each button has one function
Good labels
Only available functions are active


Others, hidden on tabs or grayed out
Better feedback and error messages
Brad Myers
52
Current PUC Specification Language


XML
Full documentation for the
specification language
and protocol:
http://www.pebbles.hcii.cmu.edu/puc/

Contains sample
specification for
Audiophase stereo
Brad Myers
53
Examples of Generated GUIs

Stereo and X-10
Brad Myers
54