Enabling Natural Interaction
Randall Davis
Aaron Adler, Sonya Cates,
Jacob Eisenstein, Tracy Hammond,
Mike Oltmans, Metin Sezgin,
Chen Li, David Pitman
MIT CSAIL
Consider This Device...
Our Model
The designer
sketches with pen and
paper
The observer
interprets the sketch
The observer and
designer interact
Sketches are Ubiquitous
Sketches are Dead
Magic Paper
Demo
More Complex Example
More Complex Example
More Complex Example
A Prevasive Theme
Natural interaction is knowledge-based.
Architecture Overview
Blackboard
Early
processing
Hand-coded
recognizers
Describing New Symbols
Define AndGate
line L1 L2 L3
arc A
semi-circle A1
orientation(A1, 180)
vertical L3
parallel L1 L2
same-horiz-position L1 L2
connected A.p1 L3.p1
connected A.p2 L3.p2
meets L1.p2 L3
meets L2.p2 L3
Learning New Symbols
From hand-drawn example to a shape
description
Define AndGate
line L1 L2 L3
arc A
semi-circle A1
orientation(A1, 180)
vertical L3
parallel L1 L2
same-horiz-position L1 L2
connected A.p1 L3.p1
connected A.p2 L3.p2
meets L1.p2 L3
meets L2.p2 L3
Learning New Symbols
Knowledge about human perception:
Some properties are more important
Experiments by Goldmeier (1936) identify
perceptually relevant properties and relations;
singularities
Architecture Overview
Blackboard
Early
processing
Shape & Seq.
Description
Learn icon
Hand-coded
recognizers
Symbolic
Descr.
Code Gen.
Multi-modal
1 example
New
recognizers
recognizer
Architecture Overview
Descriptions
of Objects
Pixels
Objects
Sketch-Enabled Interaction
Appl’n 2
3
Application
Multi-Modal Interaction
Multi-Modal Interaction
“Three identical
touching
pendulums…”
• Graphical vs verbal communication:
What’s best said, what’s best sketched?
Understanding Gestures
Lexicon: What gestures do people make?
Depends on task.
Our task: explaining how something works.
Understanding Gestures
• 96% of gestures refer to the
diagram.
• Two-handed gestures are
common.
• Deixis is more frequent.
Sketching and Talking:
Laboratory Software
Natural, Unrestricted Input
Domain: Sketching family trees
Lexicon
male
female
child-of married-to divorce
Natural, Unrestricted Input
1
2
10
8
9
7
11
12
4
5
6
3
Task is incremental
Signal is noisy
Styles vary
Segmentation is difficult
The signal is 2-d, non-chronological
Low Level Context Use
Higher Level Context
Lexicon
male
female
Grammar
child-of
married-to divorce
Higher Level Context
Description Matching is Slow
Find Possible Objects Quickly
Find Possible Objects Quickly
Domains
Mechanical engineering
Family trees
Software design
Circuit analysis
Medical applications
Sketching Software
Sketching Software
Sketching Software
Sketching Software
//Source file: c:\\video\\Deck.java
public class Deck extends Hand implements Dealable
{
public Card theCard;
public Game theGame;
/**
* @roseuid 3C21348C0257
*/
public Deck()
{
}
}
Sketching Software
//Source file: c:\\video\\Deck.java
public class Deck extends Hand
implements Dealable
{
public Card theCard;
public Game theGame;
/**
* @roseuid 3C21348C0257
*/
public Deck()
{
}
}
Circuit Analysis
Specifying Shapes
Hardware
Summary
Once the sketch is understood, many
things are possible.
Break down the barriers: eliminate
interfaces
Natural interaction is enabled by
intelligence
About sketching, gesturing, speech
About the domain
About the task