Building Whole Applications
Using Only Programming-byDemonstration
Richard McDaniel
January 14, 1999
Purpose

Allow nonprogrammers to build
applications with original, dynamic
behavior
 Use
programming-by-demonstration
 No written language at all - programs
created solely through demonstration
 Implement in system called Gamut
2
State of the Art

Full-fledged languages and libraries:
 Visual

C++, Java
Simplified languages with syntaxsupporting editor:
 Authorware, Agentsheets

Tools with macro languages:
 Director
/ LINGO, Excel / Visual Basic
3
Outline
Introduction
 Interaction Techniques
 Video
 Inferencing
 User Study
 Conclusion

4
Domain

“Board Games”
 Has
background “board”
 “Pieces” above board show state
 Pieces follow rule-like behavior
 Autonomous behavior controlled by timers

Examples
 Tic-Tac-Toe,
Pacman, Reader Rabbit
5
Example: Pacman
6
Innovations in Gamut

Interaction Techniques
 Allows
developers to specify important
details

Inferencing
 Infer
conditions from sources unaffected by
the behavior
 Infer chains of expressions
7
Inferencing - Conditions

Relationship can be formed between arbitrary
objects
– Relating a toggle switch to decide which way to move
?
Run Away
8
Expression Chain
Expressions form chain of relationships
– In Monopoly, the position to move a piece is, “the square
which is the sum of the dice’s number of squares from the
original position”
GO

Red’s Turn
9
Interaction Techniques

Nudges
 Do
Something, Stop That
 Hint highlighting
Guide Objects
 Temporal Ghosts
 Cards and Decks

10
Nudges
11
Negative Examples

Standard technique for demonstrating
examples: Macro Recorder
 Requires
special controls to make a
negative example

Negative examples in Gamut provided
by Stop That
 No
special set-up required
12
Guides, Ghosts, and Hints
13
Cards and Decks
14
Video
15
Stages of Inferencing
Translation
Matching
Actions
Code
Generation
16
Translation
Read events from undo history list
 Convert events to actions
 Eliminate redundant and
inconsequential events

17
Matching Actions

Matches actions in trace to original
behavior
– Unordered
– Typically not many (three or four)

Considerations
– Degree to which actions match
– Priority of actions in original behavior
– Actions which create new objects
18
Recursive Difference
Propagates changes to descriptions
 Each description contains heuristics for
comparing values
 Builds an And/Or tree result

19
Code Generation

Resolves each value difference by
creating a new description
 Search
for previously defined description
 Use heuristics to generate description from
highlighted objects
 Assemble values into a table and use
Decision Tree learning
20
Decision Trees
Condition For Pacman Monster



Created at places where
Gamut chooses between
different behaviors
Generate attributes
using heuristics and hint
highlighting
Build table of examples
that relate behaviors to
attributes
Is Eaten
Y
N
Run
Home
Is Blue
Y
N
Run
Away
Chase
Pacman
21
User Study

Goal
 Determine
if nonprogrammers can use
Gamut
 Perform an informal study

Participants
 Nonprogrammers,

CMU community
Location
 HCI
Institute’s usability lab
22
Tasks
Safari
Pawn Race
G-bert
23
Expertise with Techniques
techniques
Do Something
Stop That
P1
P2
X
P3
P4
X
X
X
X
Highlights Ghosts
X
Guide Objects
X
X
Cards and Decks
X
X
Player Mouse
Icons
X
24
Problems

Reluctance to create guide objects
 Needed
explicit instructions in tasks
 Once told, participants would create
appropriate objects

Highlighting inappropriate objects
 Some
objects seem too obvious to
highlight, like ghosts
25
Future Work
Apply to other domains
 Feedback for just inferred relationships
 Display for language
 Higher-level inferencing

26
Conclusion
Builds entire applications without a
written programming language
 Infers a broader range of behavior than
prior systems
 Is usable by nonprogrammers

27