Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
Using Hand-Held Computers
and PCs Together:
The Pebbles Project
Brad A. Myers
bam@cs.cmu.edu
http://www.cs.cmu.edu/~pebbles
Multiple Devices

Most of our time spent in places where there
is embedded technology


Offices, meeting rooms, classrooms, homes
Often multiple devices will be available:
 Mobile phone and PDA and a PC
 Multiple people’s PDAs
 PDAs in a “Smart Room”
Brad Myers
2
Handhelds will be communicating

802.11

BlueTooth

Cell-phone network

(Infrared)
Brad Myers
3
Premises of our Research
“With the coming wireless technologies,
connecting the PCs and PDAs together
will no longer be an occasional event
for synchronization. Instead, the
devices will frequently be in
close, interactive communication.”
—
Brad Myers, “Using Hand-Held Devices and
PCs Together,” Comm. ACM,
Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.
Brad Myers
4
Research Agenda
How can multiple devices
be used effectively together,
at the same time?
How can the user interface
and functionality be spread
across multiple devices?
Brad Myers
5
Research Agenda
How
can
multiple
devices
“Multi-Machine
be used effectively together,
User
Interfaces”
at the same time?
How can the user interface
and functionality be spread
across multiple devices?
Brad Myers
6
Example: Power Point Control


Use PC to give the presentation
Use hand-held to control the PC


Two-way communication
Hand-held shows picture of slide,
notes, list of titles, timer, etc.
Brad Myers
7
Pebbles is:
P DAs for
E ntry of
B oth
B ytes and
L ocations from
E xternal
S ources.
http://www.pebbles.hcii.cmu.edu/
Brad Myers
8
Handhelds in an Office


Augment desktop applications
Use multiple devices at the same time
Brad Myers
9
Handhelds in Meetings



Attendees use handhelds to interact and
annotate presentation
Augment collaboration
Take notes
Brad Myers
10
Handhelds in Classrooms


Data projectors for instructor’s slides
Students could have computers for:



Notetaking linked to instructor’s slides
In-class testing
Running simulations
Brad Myers
11
Handhelds in Homes

Interact with embedded computation



“Smart homes”
Not just speech and vision as interfaces
Interact with appliances, lights, etc.

“Personal Universal Controller” (PUC)
Brad Myers
12
Office Use (For Individuals)

How can handhelds augment desktop
applications?
Applications for Individuals



Extra input and output devices have been
shown to be useful
But can be expensive and hard to configure
People have PDAs and are attached to PC



For example, cradles for recharging
Customizable, extensible
Extend desktop applications
Brad Myers
14
Scrolling with the PDA

For scrolling using the non-dominant hand


Studies showed parallel and efficient uses of
both hands together
Generates Windows scrolling events
Brad Myers
15
Results of Study of Scrolling with PDA



Scrolling with buttons on PDA was fastest
PDA scrollers similar to mouse speed
Using 2 hands is effective!
Time to Scroll 10 Pages
Sec
100
80
60
40
20
Winner!
0
Trial 2
ButtonScroller
Mouse
AbsScroller
Trial 3
SlideScroller
Scroll Wheel
RateScroller
Brad Myers
16
Remote Clipboard





Transfer information between PDA and PC
Connects their clipboards together
Transfer content or reference
Works with all applications
Also between multiple
computers
Brad Myers
17
Shortcutter


User-created panels of controls
Create custom interfaces and
extensions to PC applications



And then take them with you
Direct manipulation
for edit, then set
properties
Palm or PocketPC
Brad Myers
18
Shortcutter Widgets

Buttons

Sliders

Knobs

Mouse pad

Graffiti Pad (Palm)

Gesture panel
Brad Myers
19
Shortcutter Actions




Send any keyboard key, mouse button,
scrolling action or string to PC
Open a file or URL
Run an application
Invoke any PC menu or button




Windows message
Recorded
Switch to a different Shortcutter panel
Control the Mouse
Brad Myers
20
Shortcutter Actions, cont.

Control external devices
through PC’s serial port



Macro


Directly (e.g., projectors)
X-10 for electrical devices
Can be multi-application
Application-specific


Same button, different messages
Useful for application sets:
browsers, compilers
Brad Myers
21
More Scenarios of Use

Lean-back mail reading

Controlling WinAmp

… and many others
Brad Myers
22
Study of Individual Use
Time to tap on button depended on size
Few errors
People often didn’t look at PDA



Tap Tim e
(m sec)
Error Rate (%)
Button Task: Tap Time
100
1000
903.68
900
800
700
Button Task: Error Rate
684.74
702.55
683.01
714.43
742.25
779.20
793.82
90
80
70
600
60
500
50
400
40
300
30
200
20
100
10
=5
48
)
0.70
1.05
1.57
2.61
2.27
1.74
2.11
2x2
2x3
3x2
2x4
4x2
3x4
4x3
4.87
0
4x4
4x
4
(n
=5
56
)
4x
3
(n
=5
66
)
3x
4
(n
=5
60
)
4x
2
(n
=5
59
)
2x
4
(n
=5
64
)
3x
2
2x
3
2x
2
Brad Myers
(n
=5
65
)
(n
(n
=5
67
)
0
23
Study of Individual Use

Moving hands to both PDA and mouse
only about 15% slower than just moving to
the mouse
msec
1H Keyboard->Mouse
1H Keyboard->PDA
1H Mouse->Keyboard
1H PDA->Keyboard
Keyboard -> Mouse&PDA
728
744
701
639
838
Mouse&PDA -> Keyboard
791
% slower
15.1%
12.8%
Brad Myers
24
Use in Meetings

Enhance group’s collaboration and control
Original Application:
Remote Commander

Allow PDAs to control a PC




Can be used with any application
Uses the standard (single) cursor
Don’t have to jump up and grab
mouse
Perform all mouse and
keyboard functions

Use PDA like touchpad

Graffiti or our own
pop-up keyboard
Brad Myers
26
PocketPC version


Get PC’s screen onto PocketPC
Full view, or one-to-one zooming

Scroll with iPaq’s buttons
Brad Myers
27
Scribble




Multiple people draw on top of whatever on
PC screen, not just PowerPoint
Each user has own cursor
and color
Save by PrintScreen
Erase by refresh
Brad Myers
28
SlideShow Commander

For PowerPoint


Use PC to give the presentation


Full features of PowerPoint
Use hand-held as “remote control” for PC



PC Only: Windows 95, 98, NT, 2000, ME, XP
Can wander away from the keyboard
Two-way communication
View, Navigate, Highlight, Time
Brad Myers
29
View on Hand-Held: slide

See thumb-nail of current slide

Black and white or color
Brad Myers
30
View on Hand-Held: notes

See the notes of the current slide
Brad Myers
31
View on Hand-Held: titles

See the list of titles
Brad Myers
32
View on handheld: Time



Multi-function timer
View large
And in corner
Brad Myers
33
Control Demonstrations on Palm



Control PC applications and external devices
Uses Shortcutter
Easy demo and resume show
Brad Myers
34
Control Demonstrations on PocketPC



Task list of PC applications
Tap brings one to front
PowerPoint continues
behind – easy to resume
Brad Myers
35
Commercialized

SlideShow Commander
commercialized by:
Synergy Solutions, Inc.
http://www.synsolutions.com/


Available at Office Depot,
CompUSA, etc.
Palm and PocketPC
http://www.slideshowcommander.com/
Brad Myers
36
Studies of Presentations


Summer study of 2 HCI Master’s students
Contextual Inquiry of 9 presentations



Found 220 “breakdowns”



Audience: 10 to hundreds
7 used PowerPoint, 4 used NetMeeting
Most were minor problems
Averaged 8.7 lost minutes per talk (14.5%)
Designed SlideShow Commander to
eliminate some of these problems
Brad Myers
37
MultiCursor
For special applications that are aware of
multiple inputs
 E.g.: Shared Whiteboard
 Single Display Groupware

Multiple people, one display
 A number of interesting
issues:




Palettes, widgets
Section handles
Undo
Brad Myers
38
Chat



Communicate to another PDA user through
the PC
PC serves as a conduit
For side notes and messages


For example, in negotiation
meetings
Send to all or to a specific
person
Brad Myers
39
Command Post of the Future


Large DARPA funded project
Make commanders more effective
Brad Myers
40
Study of Laser Pointers





Studied properties of laser
pointer interaction techniques
Hand-wiggle 8 pixels
Delay until moving average stable 1.5 sec
Delay until target acquired 1.5 sec
Different devices and poses do not help much
Brad Myers
41
Tap Speed Study


Tapping directly on
SmartBoard is fastest
Semantic Snarfing next



But high error rate
Then regular mouse
Laser pointing slowest

Used separate physical
button on handheld
1000
9.00%
8.00%
7.00%
6.00%
5.00%
4.00%
3.00%
2.00%
1.00%
0.00%
800
600
400
200
0
Mouse SmartBoard Laser
Pointer
Movement Time (ms)
Semantic
Snarfing
Error Rate
Brad Myers
42
“Semantic Snarfing”


Interacting at a distance
Grab contents to handheld




Picture
Menus
Text
Re-visualization

“Magic Lenses”
Brad Myers
43
Classroom Use

Using handheld computers in classrooms



For testing
Improve large lectures
Provide immediate feedback to instructor
Hardware

HP donated 110 Jornada 680 and 100 720
computers


Windows CE
Lucent donated Wavelan wireless cards
Brad Myers
45
Context


Collaborator: Prof. David Yaron of Chemistry
Chemistry 09-106: Modern Chemistry II




Spring, 2000 and Spring, 2001
About 90 students
Mostly freshmen
Loaned one Jornada and Wavelan card to
each student for the whole semester
So would get used to using it
 Offset technical difficulties with benefits to
student
Brad Myers

46
Wireless Andrew


CMU has almost complete coverage of
802.11b support
Wavelan



Can use computers wirelessly in all
classrooms and offices


Donations from Lucent, etc.
Funding from state of Pennsylvania, etc.
Most lawns, dorm rooms, etc.
Windows, Mac, Linux, Windows CE support
Brad Myers
47
Pebbles Tests


Create through html pages
Arbitrary html in questions
and answers



Pictures, formatting, links
Use FrontPage, etc. to
author
Embed tags to show
question and answers
Brad Myers
48
Concept Test

Students allowed to answer multiple times



Server stores all answers
Prof. Yaron displayed
questions and answers on
the board
Evidence that significantly
contributes to learning
—
Mazur, E. (1997).
Peer Instruction:
A User's Manual, Prentice Hall.
Brad Myers
49
Instructor’s View

In class, instructor sees a chart of answers



Optionally refreshes every 3 seconds
May be
projected for
whole class to
see
Top displays
instructions for
students
Brad Myers
50
Results



Unfortunately, no data (yet) about effects on
learning
Students thought concept tests were a
valuable part of the class
Students significantly preferred using
handhelds for concept tests.
Raise Hands Hold up cards Handhelds
25%
4%
63%
Brad Myers
51
Home Use


Personal Universal Controller
Help for people with muscular disabilities
Personal Universal Controller

Two-way communication




Appliances describe their functions
Handheld PUC creates interface based on descriptions
Handheld PUC controls the appliance
Appliance sends back status as feedback
Specifications
Control
Feedback of Status
Brad Myers
53
Initial Experiments

Use real PocketPC

Pretend that controls devices
Brad Myers
54
PocketPC results

For both appliances, users of actual interfaces:



Took about twice as long
Made at least twice as many mistakes as users of
the handheld interfaces
Needed external help five times more often
Brad Myers
55
Current Steps



XML specification language and
protocol for describing appliances
Create panels automatically from
the specification
Find real
appliances which
we can control
Specification
Language
<?xml version="1.0" encoding="UTF-8"?>
<spec xmlns="puc.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSc
hema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLS
chema"
name="Audiophase 5 CD Stereo">
<groupings>
<state name="PowerState">
<type name="OnOffType>
<valueSpace>
<boolean/>
</valueSpace>
<valueLabels>
<map value="false">
<label>Off</label>
</map>
<map value="true">
<label>On</label>
</map>
</valueLabels>
</type>
<labels>
<label>Stereo Power</label>
<label>Power</label>
<label>Powr</label>
<label>Pwr</label>
</labels>
<priority>10</priority>
</state>
Brad Myers
56
Current Work: Handhelds for
People with Muscular Disabilities

Using handhelds as interface to PCs

People with Muscular Dystrophy
have fine-motor control but lose
gross motor control


Difficulties with mouse and keyboard,
but stylus OK
Handhelds as interface to other devices

Control room lights, telephone, wheelchair, etc.
Brad Myers
57
Remote Commander Adaptations




Faster Acceleration
More flexible tapping
Turn off key repeat
Multiple Keyboards
Brad Myers
58
General Architecture
Brad’s
Palm Pilot
On the PC
Rob’s
Pocket PC
Ben’s
Windows CE
Windows
event
stream
PebblesPC
Serial,
IR or
sockets
Various PDA
apps
Any PC app.
RemoteCmd
MultiCursor
Direct
connection
or sockets
Slideshow
Commander
PebblesDraw
PowerPoint
OLE
Automation
Various dlls
Brad Myers
59
Protocols

Can use Pebbles protocols to develop your
own application



Use by:




Libraries for Palm, Windows CE, PC
Independence from communication medium
PalmAmp from IronCreek Software
Intel research
others…
Windows messages or sockets
Brad Myers
60
Downloads




Most of this software is available for free
downloading
First release in Feb, 1998
Just released version 5 (Feb, 2002)
Downloaded over 30,000 times

About 200 times a week
http://www.pebbles.hcii.cmu.edu/
Brad Myers
61
Conclusions



Handhelds are becoming ubiquitous
Connecting technologies will improve
Important to study how can be used when
connected to computers and each other


“Multi-Machine User Interfaces” will be
increasingly important
People will want to use the most convenient
device for their information and control
Brad Myers
62
Thanks to Our Sponsors!
Supported by grants from:
DARPA
Microsoft
Pittsburgh Digital Greenhouse
NSF
And equipment grants from:
Hewlett Packard
Lucent Technologies
Palm Computing
Symbol Technologies
IBM
SMART Technologies, Inc.
Synergy Solutions, Inc.
Handango
Brad Myers
63
Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
Using Hand-Held Computers
and PCs Together:
The Pebbles Project
Brad A. Myers
bam@cs.cmu.edu
http://www.cs.cmu.edu/~pebbles