Real-time Acquisition and
Rendering of Large 3D Models
Szymon Rusinkiewicz
Computer Graphics Pipeline
Shape
Rendering
Motion
Lighting
and
Reflectance
• Human time = expensive
• Sensors = cheap
– Computer graphics increasingly relies on
measurements of the real world
3D Scanning
3D Scanning Applications
•
•
•
•
Computer graphics
Product inspection
Robot navigation
As-built floorplans
•
•
•
•
Product design
Archaeology
Clothes fitting
Art history
The Digital Michelangelo Project
• Push state of the art in range scanning and
demonstrate applications in art and art
history
Working in
the museum
Scanning
geometry
Scanning
color
Traditional Range Scanning Pipeline
• High-quality, robust pipeline for producing
3D models:
– Scan object with laser triangulation scanner:
many views from different angles
– Align pieces into single coordinate frame:
initial manual alignment, refined with ICP
– Merge overlapping regions: compute “average”
surface using VRIP [Curless & Levoy 96]
– Display resulting model
3D Scan of David: Statistics
• Over 5 meters tall
• 1/4 mm resolution
• 22 people
• 30 nights of scanning
• Efficiency max : min = 8 : 1
– Needed view planning
• Weight of gantry: 800 kg
• Putting model together:
1000+ man-hours and counting
New 3D Scanning Pipeline
• Need for a fast, inexpensive,
easy-to-use 3D scanning system
• Wave a (small, rigid) object by
hand in front of the scanner
• Automatically align data as
it is acquired
• Let user see partial model as it
is being built – fill holes
Real-Time Model Acquisition
Real-Time 3D Model Acquisition
• Prototype real-time model acquisition system
– 3D scanning of moving objects
– Fast alignment
– Real-time merging and display
Applications of Easy-to-Use
3D Model Acquisition
•
•
•
•
•
Advertising
More capabilities in Photoshop
Movie sets
Augmented reality
User interfaces
3D Scanning Technologies
• Contact-based: touch probes
• Passive: shape from stereo, motion, shading
• Active: time-of-flight, defocus, photometric
stereo, triangulation
• Triangulation systems are inexpensive, robust,
and flexible
– Take advantage of trends in DLP projectors
Laser Triangulation
Object
Laser
Camera
• Project laser stripe onto object
Laser Triangulation
Object
Laser
(x,y)
Camera
• Depth from ray-plane triangulation
Triangulation
• Faster acquisition: project multiple stripes
• Correspondence problem: which stripe
is which?
Triangulation
Multi-stripe
Multi-frame
Single-stripe
Slow, robust
Single-frame
Fast, fragile
Time-Coded Light Patterns
• Assign each stripe a unique illumination code
over time [Posdamer 82]
Time
Space
Gray-Code Patterns
• To minimize effects of quantization error:
each point may be a boundary only once
Time
Space
Structured-Light Assumptions
• Structured-light systems make certain assumptions
about the scene:
• Spatial continuity assumption:
– Assume scene is one object
– Project a grid, pattern of dots, etc.
• Temporal continuity assumption:
– Assume scene is static
– Assign stripes a code over time
Codes for Moving Scenes
• We make a different assumption:
– Object may move
– Velocity low enough to permit tracking
– “Spatio-temporal” continuity
Codes for Moving Scenes
• Code stripe boundaries
instead of stripes
• Perform frame-to-frame
tracking of corresponding
boundaries
– Propagate illumination history
Illumination history = (WB),(BW),(WB)
[Hall-Holt & Rusinkiewicz, ICCV 2001]
Code
New Scanning Pipeline
Project
Code
Capture
Images
Find
Boundaries
Match
Boundaries
Decode
Compute
Range
Designing a Code
Project
Code
Capture
Images
Find
Boundaries
Match
Boundaries
Decode
• Biggest problem is ghosts – WW or BB
“boundaries” that can’t be seen directly
Compute
Range
Designing a Code
• Design a code to make tracking possible:
– Do not allow two spatially adjacent ghosts
– Do not allow two temporally adjacent ghosts
t
Designing a Code
• Graph (for 4 frames):
0000
1101
– Nodes: stripes (over time)
– Edges: boundaries (over time)
1010
1111
0111
0010
0101
1000
1011
0110
0001
0100
Time
1110
Space
1100
1001
0011
Designing a Code
• Graph (for 4 frames):
0000
1101
– Nodes: stripes (over time)
– Edges: boundaries (over time)
Boundary visible at even times
1010
1111
0111
0010
0101
1000
Boundary visible at odd times
1011
0110
0001
0100
1100
1001
• Path with alternating colors: 1110
55 edges in graph 
maximal-length traversal has 110 boundaries (111 stripes)
0011
Image Capture
Project
Code
Capture
Images
Find
Boundaries
Match
Boundaries
Decode
• Standard video camera: fields at 60 Hz
• Genlock camera to projector
Compute
Range
Finding Boundaries
Project
Code
Capture
Images
Find
Boundaries
Match
Boundaries
Decode
Compute
Range
• Standard edge detection problem
• Current solution: find minima and maxima of
intensity, boundary is between them
Matching Stripe Boundaries
Project
Code
Capture
Images
Find
Boundaries
Match
Boundaries
• Even if number of ghosts is
minimized, matching is not easy
?
Decode
Compute
Range
Matching Stripe Boundaries
• Resolve ambiguity by constraining maximum
stripe velocity
• Could accommodate higher speeds by
estimating velocities
• Could take advantage of methods in
tracking literature (e.g., Kalman filters)
Decoding Boundaries
Project
Code
Capture
Images
Find
Boundaries
Match
Boundaries
Decode
Compute
Range
• Propagate illumination history
• Table lookup based on illumination history
and position in four-frame sequence
– Once a stripe has been tracked for at least four frames,
it contributes useful data on every subsequent frame
Computing 3D Position
Project
Code
Capture
Images
Find
Boundaries
Match
Boundaries
• Ray-plane intersection
• Requires calibration of:
– Camera, projector intrinsics
– Relative position and orientation
Decode
Compute
Range
Results
Video
frames
Stripe
boundaries
unknown
known
ghosts
Results
• Single range image of moving object
Top View
Front View
Gray codes, no tracking
Top View
Front View
Boundary codes and tracking
Aligning 3D Data
• This range scanner can be used for any
moving objects
• For rigid objects, range images can be aligned
to each other as object moves
Aligning 3D Data
• If correct correspondences are known,
it is possible to find correct relative
rotation/translation
Aligning 3D Data
• How to find corresponding points?
• Previous systems based on user input,
feature matching, surface signatures, etc.
Aligning 3D Data
• Alternative: assume closest points correspond
to each other, compute the best transform…
Aligning 3D Data
• … and iterate to find alignment
– Iterated Closest Points (ICP) [Besl & McKay 92]
• Converges if starting position “close enough“
ICP Variants
• Classic ICP algorithm not real-time
• To improve speed: examine stages of ICP and
evaluate proposed variants
1.
2.
3.
4.
5.
6.
Selecting source points (from one or both meshes)
Matching to points in the other mesh
Weighting the correspondences
Rejecting certain (outlier) point pairs
Assigning an error metric to the current transform
Minimizing the error metric
[Rusinkiewicz & Levoy, 3DIM 2001]
ICP Variant –
Point-to-Plane Error Metric
• Using point-to-plane distance instead of
point-to-point lets flat regions slide along
each other more easily [Chen & Medioni 91]
Finding Corresponding Points
• Finding closest point is most expensive stage of ICP
– Brute force search – O(n)
– Spatial data structure (e.g., k-d tree) – O(log n)
– Voxel grid – O(1), but large constant, slow preprocessing
Finding Corresponding Points
• For range images, simply project point [Blais 95]
– Constant-time, fast
– Does not require precomputing a spatial data structure
High-Speed ICP Algorithm
• ICP algorithm with projection-based
correspondences, point-to-plane matching
can align meshes in a few tens of ms.
(cf. over 1 sec. with closest-point)
Anchor Scans
• Alignment of consecutive scans leads to
accumulation of ICP errors
• Alternative: align all scans to an “anchor”
scan, only switch anchor when overlap low
• Given anchor scans, restart after failed ICP
becomes easier
Merging and Rendering
• Goal: visualize the model well enough
to be able to see holes
• Cannot display all the scanned data –
accumulates linearly with time
• Standard high-quality merging methods:
processing time ~ 1 minute per scan
Merging and Rendering
• Real-time incremental merging and rendering:
– Quantize samples to a 3D grid
– Maintain average normal of all points
at a grid cell
– Point (splat) rendering
– Can be made hierarchical to conserve memory
Photograph
Real-time Scanning Demo
Postprocessing
• Goal of real-time display is to let user evaluate
coverage, fill holes
– Quality/speed tradeoff
• Offline postprocessing for high-quality models
Merged Result
Photograph
Aligned scans
Merged
Future Work
• Technological improvements:
– Use full resolution of projector
– Higher-resolution cameras
– Ideas from design of single-stripe 3D scanners
• Pipeline improvements:
–
–
–
–
–
Better detection of failed alignment
Better handling of object texture – combine with stereo?
Global registration to eliminate drift
More sophisticated merging
Improve user interaction during scanning
Future Work
• Faster scanning
– Better stripe boundary matching
– Multiple cameras, projectors
– High-speed cameras
• Application in different contexts
– Small, hand-held
– Cart- or shoulder-mounted for digitizing rooms
– Infrared for imperceptibility
Rendering of Large Models
• Range scanners increasingly capable of producing
very large models
– DMich models are 100 million to 1 billion samples
• Challenge: how to allow viewing in real time
– Fast startup, progressive loading
• Traditional answer: triangle meshes, simplification,
hardware-accelerated rendering
– Impractical for such large models
• Alternative: revisit basic data structure
QSplat [Rusinkiewicz & Levoy, SIGGRAPH 00]
QSplat
• Key observation: a single bounding sphere
hierarchy can be used for
– Hierarchical frustum and backface culling
– Level of detail control
– Splat rendering [Westover 89]
QSplat Node Structure
Position
and
Radius
Tree
Structure
13 bits
3 bits
Normal
Width of
Cone of
Normals
Color
(Optional)
14 bits
2 bits
16 bits
6 bytes
QSplat Node Structure
Position
and
Radius
Tree
Structure
13 bits
3 bits
Normal
Width of
Cone of
Normals
Color
(Optional)
14 bits
2 bits
16 bits
• Position and radius encoded
relative to parent node
Center Offset
– Hierarchical coding vs. delta coding
along a path for vertex positions
Radius Ratio
QSplat Node Structure
Position
and
Radius
Tree
Structure
13 bits
3 bits
Uncompressed
Normal
Width of
Cone of
Normals
Color
(Optional)
14 bits
2 bits
16 bits
QSplat Node Structure
Position
and
Radius
Tree
Structure
13 bits
3 bits
Delta Coding
[Deering 96]
Normal
Width of
Cone of
Normals
Color
(Optional)
14 bits
2 bits
16 bits
QSplat Node Structure
Position
and
Radius
Tree
Structure
13 bits
3 bits
Hierarchical
Coding
Normal
Width of
Cone of
Normals
Color
(Optional)
14 bits
2 bits
16 bits
QSplat Rendering Algorithm
• Traverse hierarchy recursively
Hierarchical frustum /
backface culling
if (node not visible)
Skip this branch
Point rendering
else if (leaf node)
Level of detail
control
Draw a splat
else if (size on screen < threshold)
Draw a splat
else
Traverse children
Adjusted to maintain
desired frame rate
Demo – St. Matthew
• 3D scan of 2.7 meter statue
at 0.25 mm
• 102,868,637 points
• File size: 644 MB
• Preprocessing time: 1 hour
Future Work
• Splats as primitive
– Unify rendering of meshes, volumes, point clouds
– Compatible with shading after rasterization
– Hybrid point/polygon systems
• High-level visibility / LOD frameworks
– Store different kinds of data at each node: alpha, BRDF,
scattering function, etc.
– Potentially could be used to unify image-based-rendering
(IBR) techniques
Contributions
• Real-time 3D model acquisition system
–
–
–
–
Video-rate 3D scanner for moving objects
Analysis of ICP variants; real-time algorithm
Real-time merging and rendering
Allows user to see model and fill holes
• QSplat: interactive rendering of large 3D meshes
– Single data structure used for visibility culling,
level-of-detail control, point rendering, compression
– Extension to network streaming [I3D 2001]
Acknowledgments
• Olaf Hall-Holt
• Lucas Pereira
• The Original DMich Gang: Dave Koller, Sean Anderson,
James Davis, Kari Pulli, Matt Ginzton, Jon Shade
• DMich, the next generation: Gary King, Steve Marschner
• Graphics lab
• Advisor: Marc Levoy
• Committee: Pat Hanrahan, Leo Guibas, Mark Horowitz,
Bernd Girod
• Family, friends
• Sponsors: NSF, Interval, Honda, Sony, Intel