Designing Effective Step-By-Step
Assembly Instructions
Maneesh Agrawala, Doantam Phan, Julie Heiser, John Haymaker,
Jeff Klingner, Pat Hanrahan and Barbara Tversky
Microsoft Research and Stanford University
1
2
3
4
5
6
Designing Instructions
Planning
Choose sequence of assembly operations
 Robotics / AI / Mechanical Engineering

[Wolter 89], [de Mello 91], [Wilson 92], [Romney 95]
Presentation
Visually convey assembly operations
 Visualization / Computer Graphics

[Seligmann 91], [Rist 94], [Butz 97], [Strothotte 98]
We jointly optimize plan and presentation
Geometric Analysis [Romney 95]
B blocked by A
B
A
A
both parts free to move
B
A blocked by B
Input Parts
Blocking Graph
A
A
A
B
C
C
B
C
B
A
C
B
A
B
Geometric Assembly Planning
Valid
Valid
Invalid
Many Geometrically Valid Sequences
Valid
Valid
Valid
Valid
Valid
How do we choose the best sequence?
Our Approach
Identify cognitive design principles

How people conceive of 3D assemblies

How people comprehend visual instructions
Encode principles as constraints within
automated design system
Identifying Design Principles
Experiment 1: Assemble and draw instructions
Experiment 2: Rate effectiveness
Experiment 3: Validate effectiveness
Sequence the Assembly Operations
Single exploded view diagram
Step-by-step diagrams
Step-by-step instructions preferable
Illustrate the Assembly Operations
Structural diagrams
Action diagrams
Action diagrams preferable
Ensure Visibility of Parts
Essential
Parts being attached
 Context (show earlier parts)

Less important
All parts in a symmetric group
 Repetitive operations

Automated Instruction Design
Step-by-Step
Action diagrams
Good visibility
TV stand instructions generated by our system
Input
Geometry

Parts in assembled positions
Orientations


Default viewpoint / orientation
Preferred orientation for each part
Groupings

fasteners, significant parts, symmetry,
similar-actions
Ordering constraints

Force one part to attach before another
required
optional
TV Stand Input
required
optional
Geometry: Parts in assembled positions
Orientations: Default viewpoint / orientation
Groupings: fasteners, significant parts, symmetry
All parts
Search
Leftover parts
Best subset
of parts
Structural Diagrams
Sequence Parts
Action Diagrams
Reorientation
Sequence of assembly steps
All parts
Search
Leftover parts
Best subset
of parts
Sequence Parts
Reorientation
Sequence of assembly steps
All parts
Search
All parts
Part subsets
Leftover parts
Best subset
of parts
Sequence Parts
Reorientation
...…
Sequence of assembly steps
All parts
Search
Choose part subset
Search
Interference
Leftover parts
Best subset
of parts
Sequence Parts
Attachment
Ordering
Grouping
Reorientation
Sequence of assembly steps
Visibility
Computing Visibility
Area(P) = # red pixels
Area(P,Q) = # red pixels
Area of top alone
Area of top not occluded by sides
Vis(P,Q) = Area(P,Q) / Area(P)
% pixels that remain visible
Visibility Constraint
Current parts R

Check that each part in current
subset is visible
min ( Vis(r, R-r) )
rR
Attached parts A

Check that context is visible
Unattached parts U

Vis(A, R)
Check that future parts will be
visible
min ( Vis(u, R) )
u
U
Lego Car
Input model
Bookcase
Input model
All parts
Add significant parts one by one
Search
Leftover parts
Best subset
of parts
Sequence Parts

Visibility

Distance to viewer
Add all remaining parts
Omit repetitive operations

Reorientation
Sequence of assembly steps
Skip if 2 similar-action parts
already added
Bookcase – After Sequencing
Bookcase – Omitting Repetition
All parts
Search
Leftover parts
Best subset
of parts
Sequence Parts
Reorientation
Sequence of assembly steps
Set preferred orientation for
significant parts
If visibility of current parts is low
try alternate oblique views
Bookcase – With Reorientation
All parts
Search
Leftover parts
Best subset
of parts
Sequence Parts
Reorientation
Sequence of assembly steps
All parts
Search
Leftover parts
Best subset
of parts
Sequence Parts
Action Diagrams
Choose Direction
Build Stacks
Place Guidelines
Reorientation
Sequence of assembly steps
Sequence of assembly diagrams
Building Stacks
Stack: set of parts that
Share separation direction
 Lie on a stabbing line
 Are in sequential contact

4th stack part
3rd stack part
2nd stack part
1st stack part
Base
Building Stacks
Stack: set of parts that
Share separation direction
 Lie on a stabbing line
 Are in sequential contact

1st stack part
1st stack part
1st stack part
Base
Base
Base
Placing Guidelines
Expand stacks
 Form lines between stack parts

Placing Guidelines
Expand stacks
 Form lines between stack parts

Bookcase
9 parts
Planning: 48s
Table
13 parts
Planning: 28s
Test Object
25 parts
Planning: 53s
Exploded View
Future Work
Experiment 4: Evaluation

Time/errors as they use our instructions
Assemblies with more subparts
Take advantage of hierarchy
 Apply system recursively

Physical and functional constraints
Part size and mass
 Gravitational stability

Summary
Cognitive design principles
Step-by-step
 Action diagrams
 Good visibility

Automated instruction design system
Integrate planning and presentation
Acknowledgements
Boris Yamrom
Christina Vincent
ONR grants N000140210534, N000140110717 and N000140010649