Scriboli: High Performance
Pen Interfaces
Ken Hinckley
Patrick Baudisch
Gonzalo Ramos
Francois Guimbretiere
Microsoft Research
1
Scriboli Design Goals
• Speed without keyboard hotkeys
• Cognitive footprint diminishes with use
• Recall / declarative  procedural skill
• Minimal demands on visual attention
• Repeatable motions for the user
• Predictable & dependable system behavior
• Expressiveness
• Support many command structures & apps
• Economy of design
• only a few things to learn
2
SKETCH – Brown University
• Adding 1 more gesture breaks everything
• hard to learn, gestures specific to 3D editing
3
Recognize Pen Gestures?
• Classic problem: “ink” or “gesture” ?
• Hard problem: any ‘gesture’ could be ink
• Inferred decision comes too late
• real-time interactive feedback difficult
• Make it a physical skill
• All we need to solve this is ONE BIT of info
• physical button on bezel
• habit-forming, deterministic, low
attentional demand
• nonpreferred hand, no trip to “lasso mode”
4
Phrase Structure for Scriboli
Scope
Command
Parameters
Separator
Separator
Separator
Separator
(start)
Scope/Command
Command/Parameters
(Done)
6
Phrase Structure for Scriboli
Scope
Command
Parameters
Separator
Separator
Separator
Separator
(start)
Scope/Command
Command/Parameters
(Done)
Press “Gesture”
Fast and reliable [Sellen 90, 92]
Can be chunked by users
7
Phrase Structure for Scriboli
Scope
Command
Parameters
Separator
Separator
Separator
Separator
(start)
Scope/Command
Command/Parameters
(Done)
Press “Gesture”
Pigtail
Fast, based on muscle “memory”
Reliable and easily chunked by users
8
Phrase Structure for Scriboli
Scope
Command
Parameters
Separator
Separator
Separator
Separator
(start)
Scope/Command
Command/Parameters
(Done)
Press “Gesture”
Pigtail
Crossing boundary
Merge command selection and direct manipulation
FlowMenu [Guimbretiere 00], Control menu [Pook 00]
9
Phrase Structure for Scriboli
Scope
Command
Parameters
Separator
Separator
Separator
Separator
(start)
Scope/Command
Command/Parameters
(Done)
Press “Gesture”
Pigtail
Crossing boundary
One stroke for scope, command, and parameters
Non-modal system (quasi-modal)
10
Delimiters for scope
selection + marking
• A delimiter is
“something different”
• lexical structure of stroke
• e.g. self-crossing
gesture stroke
• DEMO / VIDEO :
delimiter techniques
11
Results – Completion Time
2.5
2.25
1.92
2
1.75
1.5
1.25
1.47
1.17
1.56
1.38
1.26
1.23
0.95
1.01
1
0.84
0.75
0.5
0.25
0
Button
Handle
Pigtail
Pigtail2
Timeout
Button Button-RI Handle Handle-RI Pigtail Pigtail-RI Pigtail2 Pigtail2-RI Tout Tout-RI
• Left bar – main block; right bar – repeated block
• Dotted bars are Pigtail2 design iteration
12
Results – Learning Effects
2.00
Handle
Pigtail
Timeout
Button
1.75
1.50
1.25
1.00
0.75
0.50
0.25
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21 22 23
• For the 24 trials of Repeated Invocation Block
13
24
Results – Error Rate
8.1%
8.0%
7.0%
6.0%
6.0%
5.0%
4.6%
4.0%
4.0%
3.0%
2.7%
2.5%
2.0%
1.0%
0.0%
0.8%
0.0%
0.4% 0.3%
Button Button-RI Handle
Handle-RI Pigtail
Pigtail-RI Pigtail2
Pigtail2-RI Tout
Tout-RI
Button
Handle
Pigtail
Pigtail2
Timeout
• Error rate of selecting wrong marking direction
14
Summary: Grammar for pen input
• Scriboli proposes fundamental building blocks
• Links together object, verb, & indirect object
in fast, fluid, unambiguous command phrases
Punct.
space
capitals
period
comma
Written text
separate words
sentence start
sentence end
delimits clause
object
“The cat
verb
sat
indirect obj. on the mat”
Scriboli Equivalent
Pen up/pen down
Gesture button down
Gesture button up
Pigtail (delimiter)
Scope (lasso, line, …)
Marking menu
Crossing manip. phase
15
Stitching: Pen Gestures
that Span Multiple Displays
Microsoft Research
Ken Hinckley
Gonzalo Ramos
Francois Guimbretiere
Patrick Baudisch
Marc Smith
16
Wireless Network 2004 =
Horseless Carriage 1904?
• Are Wireless Networks really just wired
networks without the wires?
• Or are they something completely different?
17
Wireless Device Soup:
Which links does the user want?
18
Name That Device
• Users need techniques to intuitively form
bridges between devices
• How do users name the devices to connect?
• What is type / purpose of the connection?
• Parameters? (Who copies what, to where?)
19
Stitching
• Pen stroke
that spans
displays
• Move
the pen
• Cross over
bezel
• Finish stroke
on nearby
tablet
• System infers
connection
path taken
by the pen
transferred
pictures
20
VIDEO
• AVI’04
• ACM Advanced Visual Interfaces 2004
21
Establishing a Connection
• Recognizes timing & dynamics of pen trajectory
Tablet 2
Tablet 1
Δt
• There is nothing special about the pen!
• Wireless signal strength determines nearby devices
22
Automatic Screen Calibration
• Fits a line to absolute coords of user’s pen stroke
Direction of Stitching
Device #1
p1
p0
p11
α1
} offset
PA
α0
first half of
gesture
p10
second
half of
gesture
PB
Device #2
α = (α0 + α1) / 2
23
How Do Users Share Physical
Space?
• SociologyProxemics: How people use space
• Invisible bubble that surrounds an individual
24
6. How Do Users Share
Physical Space?
• SociologyProxemics: How people use space
• Invisible bubble that surrounds an individual
• Homework assignment:
• Sit right next to someone at airport
(when it is not necessary to do so)
• Time how long it is before the other
person leaves
25
5. How Do Users Share
Physical Space?
• SociologyProxemics: How people use space
• Invisible bubble that surrounds an individual
26
27
28
Proxemic Lessons
for Stitching
• Don’t require contact : touching is taboo
• “Do they have to be right next to one another?”: 10-40cm
• “wide screen would be nice for collaboration”
• “no two faces trying to peek at only one screen”
29
Proxemic Lessons
for Stitching
• Don’t require contact
• “Do they have to be right next to one another?”: 10-40cm
• “wide screen would be nice for collaboration”
• “no two faces trying to peek at only one screen”
• Gives users flexibility to be involved, or not: mood, task, …
• Intimate Spaces: Combining screens. For close
collaboration with friend or trusted colleague
• Personal Spaces: Tablets can be separated by up to 30”
yet still possible to stitch to give files to colleague, etc.
• Social Spaces: Once connected, “transporter” can be
used to give files to a user beyond arm’s reach
30
User 4
User 2
• Multiple Devices:
Cooperative
Stitching
User 3
Ongoing work
User 1
31
User 4
User 2
• Multiple Devices:
Cooperative
Stitching
User 3
Ongoing work
User 1
32
Questions?
33
Handle vs. Pigtail
• Handle can get in the way
• Adds some visual clutter
• Must check to be sure landed on handle
• Pigtail handles more than one scope
elegantly
• Self-referential gesture
34