stanford hci group
Paper + Digital
Applications
for mobility and collaboration
Ron B Yeh
14 November 2006
UNIFOR
Augmenting
Notebooks
2
Augmenting
Walls and
Tables
3
Designing &
Building
Paper
Interaction
s
4
Background
Research
5
Fluid Interaction with
Paper
6
Interviews and
Observations
QuickTime™
TIFF
are
needed
(Uncompressed)
toand
seeathis decompressor
picture.
7
Los Tuxtlas Tropical
Rainforest
8
We also observed that…
Transforming
The process of
capturing data
is easy, but…
Access
ing
…the data are
not.
Sharing
9
Clear Difference in
Advantages
Paper Notebooks[Robust,  “Battery”,
…]
Computers [Search, Storage, Sharing,
10
…]
Digitizing Pen
Technology
11
Digitizing Pen
Technology
12
Augmenting
Notebooks
13
14
Automatic Association
notes @ 4:43pm
Notes + Photos associated
by time
photo @
4:44pm
15
Manual Association
Notes + Photos associated by
Inked Gesture (Hotspot Gestures)
16
Video #0
Hotspot Linking
ButterflyNet Browser
18
19
20
21
22
Video #1
Browser &
Spreadsheet
Navigate by Pen
Browse using the Physical
Notebook
24
Longitudinal
Evaluation
Used By:
Field Biologists at
JRBP/Stanford
NASA Researchers
(Death Valley, Alaska Toolik Field Station)
Anthropologists at Intel
Digital Home
25
Augmenting
Walls and
Tables
26
Poster-Sized Displays
The Office of the
27
28
Photo Wall
Download Photographs
29
Video #2
Photo Wall
31
Video #3
Network Monitoring
Map-Based Queries
Search for Photographs
33
Video #4
Twistr Game
Designing &
Building
Paper
Interaction
s
35
Reduce, Recycle,
Reuse (R3)
A Paper Applications
tp://hci.stanford.edu/paper/
Toolkit
36
Goals for the Toolkit
Lower the threshold for
building this class of penbased paper + digital user
interfaces.
Provide a high enough ceiling
that talented
designers/developers can do
interesting research and build
37
useful systems.
Pipeline for building a
Paper + Digital Application
Design and Print the Paper User
Interface
Architect the Application Logic
and Connect the Sheets to Pens
and Other Devices
Write custom GUIs and Event
Handlers
38
Designing and Printing
Create the Paper Interface in
Illustrator (or some other
tool) and Export to PDF
Add Active Regions through Java
Code
or the Interactive Designer
Export to Pattern-Augmented PDF
Print to Laser Printer or Wideformat Inkjet
39
40
41
42
Event Handlers
Add behavior to your Paper UI by
attaching code that will be run
whenever a person clicks on, drags
on, writes on one of your active
regions.
toBeClickedRegion.addEventHandler(
new ClickHandler() {
public void clicked(PenEvent e) {
< do something >
}
}
}
43
Building Blocks
Sheets, Regions, Handlers
Devices, Pens
Applications
Toolkit
44
Powerpoint Advancer
Example
45
Turn Pen Events into
Keystrokes
public
static void main(String[] args) {
Application app = new Application("Advancer");
Sheet s = new Sheet(new Inches(8.5),
new Inches(11));
Region r = new Region("Button", 1, 1, 4, 2);
r.addEventHandler(getMarkHandler());
s.addRegion(r);
app.addSheet(s,
new File("data/Grid/App.patternInfo.xml"));
app.addPen(new Pen("Primary Pen"));
PaperToolkit toolkit = new PaperToolkit();
toolkit.startApplication(app);
}
46
Create an Application
public static void main(String[] args) {
Application app = new Application("Advancer");
Sheet s = new Sheet(new Inches(8.5),
new Inches(11));
Region r = new Region("Button", 1, 1, 4, 2);
r.addEventHandler(getMarkHandler());
s.addRegion(r);
app.addSheet(s,
new File("data/Grid/App.patternInfo.xml"));
app.addPen(new Pen("Primary Pen"));
PaperToolkit toolkit = new PaperToolkit();
toolkit.startApplication(app);
}
47
Add one Sheet
public static void main(String[] args) {
Application app = new Application("Advancer");
Sheet s = new Sheet(new Inches(8.5),
new Inches(11));
Region r = new Region("Button", 1, 1, 4, 2);
r.addEventHandler(getMarkHandler());
s.addRegion(r);
app.addSheet(s,
new File("data/Grid/App.patternInfo.xml"));
app.addPen(new Pen("Primary Pen"));
PaperToolkit toolkit = new PaperToolkit();
toolkit.startApplication(app);
}
48
One Patterned Region
public static void main(String[] args) {
Application app = new Application("Advancer");
Sheet s = new Sheet(new Inches(8.5),
new Inches(11));
Region r = new Region("Button", 1, 1, 4, 2);
r.addEventHandler(getMarkHandler());
s.addRegion(r);
app.addSheet(s,
new File("data/Grid/App.patternInfo.xml"));
app.addPen(new Pen("Primary Pen"));
PaperToolkit toolkit = new PaperToolkit();
toolkit.startApplication(app);
}
49
Use the Marking Gesture
Handler
public
static void main(String[] args) {
Application app = new Application("Advancer");
Sheet s = new Sheet(new Inches(8.5),
new Inches(11));
Region r = new Region("Button", 1, 1, 4, 2);
r.addEventHandler(getMarkHandler());
s.addRegion(r);
app.addSheet(s,
new File("data/Grid/App.patternInfo.xml"));
app.addPen(new Pen("Primary Pen"));
PaperToolkit toolkit = new PaperToolkit();
toolkit.startApplication(app);
}
50
Add the Region & Sheet
public static void main(String[] args) {
Application app = new Application("Advancer");
Sheet s = new Sheet(new Inches(8.5),
new Inches(11));
Region r = new Region("Button", 1, 1, 4, 2);
r.addEventHandler(getMarkHandler());
s.addRegion(r);
app.addSheet(s,
new File("data/Grid/App.patternInfo.xml"));
app.addPen(new Pen("Primary Pen"));
PaperToolkit toolkit = new PaperToolkit();
toolkit.startApplication(app);
}
51
We need at least 1 Pen
public static void main(String[] args) {
Application app = new Application("Advancer");
Sheet s = new Sheet(new Inches(8.5),
new Inches(11));
Region r = new Region("Button", 1, 1, 4, 2);
r.addEventHandler(getMarkHandler());
s.addRegion(r);
app.addSheet(s,
new File("data/Grid/App.patternInfo.xml"));
app.addPen(new Pen("Primary Pen"));
PaperToolkit toolkit = new PaperToolkit();
toolkit.startApplication(app);
}
52
Ask the Toolkit to
public
static void
main(String[]
args) {
Start
the
App
Application app = new Application("Advancer");
Sheet s = new Sheet(new Inches(8.5),
new Inches(11));
Region r = new Region("Button", 1, 1, 4, 2);
r.addEventHandler(getMarkHandler());
s.addRegion(r);
app.addSheet(s,
new File("data/Grid/App.patternInfo.xml"));
app.addPen(new Pen("Primary Pen"));
PaperToolkit toolkit = new PaperToolkit();
toolkit.startApplication(app);
}
53
Marking Gesture
private
static EventHandler getMarkHandler() {
Handler?
return new GestureHandler() {
public void handleMark(PenEvent e,
GestureDirection dir) {
switch (dir) {
default:
RobotAction robot = new RobotAction();
robot.keyType(KeyEvent.VK_RIGHT);
robot.invoke();
break;
}
}
};
}
54
The Event includes the Mark’s
Direction
private static EventHandler getMarkHandler() {
return new GestureHandler() {
public void handleMark(PenEvent e,
GestureDirection dir) {
switch (dir) {
default:
RobotAction robot = new RobotAction();
robot.keyType(KeyEvent.VK_RIGHT);
robot.invoke();
break;
}
}
};
}
55
To Key Stroke
private static EventHandler getMarkHandler() {
return new GestureHandler() {
public void handleMark(PenEvent e,
GestureDirection dir) {
switch (dir) {
default:
RobotAction robot = new RobotAction();
robot.keyType(KeyEvent.VK_RIGHT);
robot.invoke();
break;
}
}
};
}
56
Other Directions
private static EventHandler getMarkHandler() {
return new GestureHandler() {
public void handleMark(PenEvent e,
GestureDirection dir) {
switch (dir) {
case NE: // do something
break;
case E: // do something different
break;
default:
RobotAction robot = new RobotAction();
robot.keyType(KeyEvent.VK_RIGHT);
robot.invoke();
break;
}
} };}
57
Swing Example
58
R3 Example
59
Video #5
Other Paper Applications
AudioGuide Example
We started out with an idea of
making a mobile audio
interface for biologists, and
decided that a tourist map
would be a good first step.
61
The Map
62
Map  Adobe
Illustrator
63
Add Active Regions to
the
PDF
In this case, we used the R3
Acrobat Designer.
You may alternatively choose
the pure programmatic route,
by writing Java code to define
the active
regions.
public
class AudioGuide
extends Application
{
< lots of stuff >
}
64
Add Active Regions to
the class
PDFAudioGuide extends Application
public
{
public static void main(String[] args) {
AudioGuide guide = new AudioGuide();
PaperToolkit p = new PaperToolkit();
p.useApplicationManager(true);
p.loadApplication(guide);
}
< lots of stuff >
}
65
Add Active Regions to
the class
PDFAudioGuide extends Application
public
{
public static void main(String[] args) {
AudioGuide guide = new AudioGuide();
PaperToolkit p = new PaperToolkit();
p.useApplicationManager(true);
p.loadApplication(guide);
}
< lots of stuff >
}
66
67
Conclusions
Audience for the R3 Paper
Toolkit is
HCI Practitioners like
yourself!
Open Source Software
Can evolve based on your
feedback
http://hci.stanford.e
du/paper
68
stanford hci group
Questions?
??
p://graphics.stanford.edu/~ronyeh
ronyeh@cs.stanfor
d.edu