Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
The Pebbles Project
General Overview:
Using Hand-Held Computers
and PCs Together
Brad A. Myers
Carnegie Mellon University
bam@cs.cmu.edu
http://www.cs.cmu.edu/~pebbles
SUNY Technology Conference 2002
Multiple Devices

Most of our time spent in places where
there is embedded technology


People have handheld devices


Offices, meeting rooms, classrooms, homes
Palm, PocketPC, cell phone
Often multiple devices will be available:
 Mobile phone and PDA and a PC
 Multiple people’s PDAs
 “Smart Classrooms”
 PDAs in a “Smart Room”
SUNY Technology Conference 2002
Brad Myers
2
Handhelds will be communicating

802.11

Wi-Fi

BlueTooth

Cell-phone network


3G
(Infrared)
SUNY Technology Conference 2002
Brad Myers
3
Premises of our Research
“With the coming wireless technologies,
connecting the PCs and PDAs together
will no longer be an occasional event
for synchronization. Instead, the
devices will frequently be in
close, interactive communication.”
—
SUNY Technology Conference 2002
Brad Myers, “Using Hand-Held Devices and
PCs Together,” Comm. ACM,
Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.
Brad Myers
4
Research Agenda
How can multiple devices
be used effectively together,
at the same time?
How can the user interface
and functionality be spread
across multiple devices?
SUNY Technology Conference 2002
Brad Myers
5
Research Agenda
How
can
multiple
devices
“Multi-Machine
be used effectively together,
User
Interfaces”
at the same time?
How can the user interface
and functionality be spread
across multiple devices?
SUNY Technology Conference 2002
Brad Myers
6
Example: Power Point Control


Use PC to give the presentation
Use hand-held to control the PC




Two-way communication
Hand-held shows picture of slide,
notes, list of titles, timer, etc.
Easy to switch to demo
Easy to click on movies and sounds
SUNY Technology Conference 2002
Brad Myers
7
Pebbles is:
P DAs for
E ntry of
B oth
B ytes and
L ocations from
E xternal
S ources.
http://www.pebbles.hcii.cmu.edu/
SUNY Technology Conference 2002
Brad Myers
8
Handhelds in an Office


Augment desktop applications
Use multiple devices at the same time
SUNY Technology Conference 2002
Brad Myers
9
Handhelds in Meetings



Attendees use handhelds to interact and
annotate presentation
Augment collaboration
Take notes
SUNY Technology Conference 2002
Brad Myers
10
Handhelds in Classrooms


Data projectors for instructor’s slides
Students could have computers for:



Notetaking linked to instructor’s slides
In-class testing
Running simulations
SUNY Technology Conference 2002
Brad Myers
11
Handhelds in Homes

Interact with embedded computation



“Smart homes”
Not just speech and vision as interfaces
Interact with appliances, lights, etc.

“Personal Universal Controller” (PUC)
SUNY Technology Conference 2002
Brad Myers
12
Office Use (For Individuals)

How can handhelds augment desktop
applications?
Applications for Individuals



Extra input and output devices have been
shown to be useful
But can be expensive and hard to configure
People have PDAs and are attached to PC



For example, cradles for recharging
Customizable, extensible
Extend desktop applications
SUNY Technology Conference 2002
Brad Myers
14
Scrolling with the PDA

For scrolling using the non-dominant hand


Studies showed parallel and efficient uses of
both hands together
Generates Windows scrolling events
SUNY Technology Conference 2002
Brad Myers
15
Results of Study of Scrolling with PDA



Scrolling with buttons on PDA was fastest
PDA scrollers similar to mouse speed
Using 2 hands is effective!
Time to Scroll 10 Pages
Sec
100
80
60
40
20
Winner!
0
Trial 2
ButtonScroller
Mouse
AbsScroller
SUNY Technology Conference 2002
Trial 3
SlideScroller
Scroll Wheel
RateScroller
Brad Myers
16
Remote Clipboard





Transfer information between PDA and PC
Connects their clipboards together
Transfer content or reference
Works with all applications
Also between multiple
computers
SUNY Technology Conference 2002
Brad Myers
17
Shortcutter


User-created panels of controls
Create custom interfaces and
extensions to PC applications



And then take them with you
Direct manipulation
for edit, then set
properties
Palm or PocketPC
SUNY Technology Conference 2002
Brad Myers
18
Shortcutter Widgets

Buttons

Sliders

Knobs

Mouse pad

Graffiti Pad (Palm)

Gesture panel
SUNY Technology Conference 2002
Brad Myers
19
Shortcutter Actions




Send any keyboard key, mouse button,
scrolling action or string to PC
Open a file or URL
Run an application
Invoke any PC menu or button




Windows message
Recorded
Switch to a different Shortcutter panel
Control the Mouse
SUNY Technology Conference 2002
Brad Myers
20
Shortcutter Actions, cont.

Control external devices
through PC’s serial port



Macro


Directly (e.g., projectors)
X-10 for electrical devices
Can be multi-application
Application-specific


Same button, different messages
Useful for application sets:
browsers, compilers
SUNY Technology Conference 2002
Brad Myers
21
More Scenarios of Use

Lean-back mail reading

Controlling WinAmp

… and many others
SUNY Technology Conference 2002
Brad Myers
22
Use in Meetings

Enhance group’s collaboration and control
Original Application:
Remote Commander

Allow PDAs to control a PC




Can be used with any application
Uses the standard (single) cursor
Don’t have to jump up and grab
mouse
Perform all mouse and
keyboard functions

Use PDA like touchpad

Graffiti or our own
pop-up keyboard
SUNY Technology Conference 2002
Brad Myers
24
PocketPC version


Get PC’s screen onto PocketPC
Full view, or one-to-one zooming

Scroll with iPaq’s buttons
SUNY Technology Conference 2002
Brad Myers
25
Scribble




Multiple people draw on top of whatever on
PC screen, not just PowerPoint
Each user has own cursor
and color
Save by PrintScreen
Erase by refresh
SUNY Technology Conference 2002
Brad Myers
26
MultiCursor
For special applications that are aware of
multiple inputs
 E.g.: Shared Whiteboard
 Single Display Groupware

Multiple people, one display
 A number of interesting
issues:




Palettes, widgets
Section handles
Undo
SUNY Technology Conference 2002
Brad Myers
27
Chat



Communicate to another PDA user through
the PC
PC serves as a conduit
For side notes and messages


For example, in negotiation
meetings
Send to all or to a specific
person
SUNY Technology Conference 2002
Brad Myers
28
“Semantic Snarfing”


Interacting at a distance
Grab contents to handheld





Picture
Menus
Text
“Re-visualization”
“Private Drill-down
of public information
SUNY Technology Conference 2002
Brad Myers
29
Study of Laser Pointers





Studied properties of laser
pointer interaction techniques
Hand-wiggle +/-8 pixels
Delay until moving average stable +/-1.5 sec
Delay until target acquired +/-1.5 sec
Different devices and poses do not help much
SUNY Technology Conference 2002
Brad Myers
30
Home Use

Personal Universal Controller
Personal Universal Controller

Two-way communication




Appliances describe their functions
Handheld PUC creates interface based on descriptions
Handheld PUC controls the appliance
Appliance sends back status as feedback
Specifications
Control
Feedback of Status
SUNY Technology Conference 2002
Brad Myers
32
Initial Experiments

Use real PocketPC

Pretend that controls devices
SUNY Technology Conference 2002
Brad Myers
33
Initial Study results

For both appliances, users of actual interfaces:



Took about twice as long
Made at least twice as many mistakes as users of
the handheld interfaces
Needed external help five times more often
SUNY Technology Conference 2002
Brad Myers
34
Current Steps



XML specification language and
protocol for describing appliances
Create panels automatically from
the specification
Finding real
appliances which
we can control
SUNY Technology Conference 2002
Specification
Language
<?xml version="1.0" encoding="UTF-8"?>
<spec xmlns="puc.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSc
hema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLS
chema"
name="Audiophase 5 CD Stereo">
<groupings>
<state name="PowerState">
<type name="OnOffType>
<valueSpace>
<boolean/>
</valueSpace>
<valueLabels>
<map value="false">
<label>Off</label>
</map>
<map value="true">
<label>On</label>
</map>
</valueLabels>
</type>
<labels>
<label>Stereo Power</label>
<label>Power</label>
<label>Powr</label>
<label>Pwr</label>
</labels>
<priority>10</priority>
</state>
Brad Myers
35
General Architecture
Brad’s
Palm Pilot
On the PC
Rob’s
Pocket PC
Ben’s
Windows CE
Windows
event
stream
PebblesPC
Serial,
IR or
sockets
Any PC app.
RemoteCmd
MultiCursor
Direct
connection
or sockets
Slideshow
Commander
PebblesDraw
PowerPoint
OLE
Automation
Various PDA
apps
Various dlls
SUNY Technology Conference 2002
Brad Myers
36
Protocols

Can use Pebbles protocols to develop your
own application



Use by:




Libraries for Palm, Windows CE, PC
Independence from communication medium
PalmAmp from IronCreek Software
Intel research
others…
Windows messages or sockets
SUNY Technology Conference 2002
Brad Myers
37
Downloads

Most of this software is available for free
downloading




Also, about 20 published papers
First release in Feb, 1998
Recently released version 5 (Feb, 2002)
Downloaded over 30,000 times

About 200 times a week
http://www.pebbles.hcii.cmu.edu/
SUNY Technology Conference 2002
Brad Myers
38
More in Next Session!


Next session: 11:20 - 12:05
Details of use in classrooms and for the
handicapped:



SlideShow Commander!
Concept Tests
Handhelds as Assistive Technology
SUNY Technology Conference 2002
Brad Myers
39
Conclusions



Handhelds are becoming ubiquitous
Connecting technologies will improve
Important to study how can be used when
connected to computers and each other


“Multi-Machine User Interfaces” will be
increasingly important
People will want to use the most convenient
device for their information and control
SUNY Technology Conference 2002
Brad Myers
40
Thanks to Our Sponsors!
Supported by grants from:
DARPA
Microsoft
Pittsburgh Digital Greenhouse
NSF
General Motors
And equipment grants from:
Hewlett Packard
Lucent Technologies
Palm Computing
Symbol Technologies
SUNY Technology Conference 2002
IBM
SMART Technologies, Inc.
Synergy Solutions, Inc.
Handango
Brad Myers
41
Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
The Pebbles Project
General Overview:
Using Hand-Held Computers
and PCs Together
Brad A. Myers
Carnegie Mellon University
bam@cs.cmu.edu
http://www.cs.cmu.edu/~pebbles
SUNY Technology Conference 2002