Why is 3D scanning hard?
Marc Levoy
Computer Science Department
Stanford University
The Digital Michelangelo Project
Atlas
Awakening
Bearded
Youthful
Dusk
Dawn
Night
Day
St. Matthew
David
Forma Urbis Romae
Eight hard problems
in 3D scanning
1.
2.
3.
4.
5.
6.
7.
8.
•
optically uncooperative materials
scanning in the presence of occlusions
insuring safety for delicate objects
scanning large objects at high resolution
accurate scanning in the field
filling holes in dense polygon models
handling large datasets
creating digital archives of 3D content
Some games with range data
Marc Levoy
Scanners used in the
Digital Michelangelo Project
Cyberware
- for statues
Faro + 3D Scanners
- for tight spots
Cyra
- for architecture
Marc Levoy
Our triangulation-based scanner
4 motorized axes
laser, range camera,
white light, and color camera
truss extensions
for tall statues
Marc Levoy
Scanning St. Matthew
working in
the museum
scanning
geometry
scanning
color
Marc Levoy
single scan of St. Matthew
1 mm
Hard problem #1:
optically uncooperative materials
dark
• fuzzy
• scattering
• transparent
shiny
• moving
• etc.
Marc Levoy
How optically cooperative is marble?
[Godin et al., Optical Measurement 2001]
• systematic bias of 40 microns
• noise of 150 – 250 microns
– worse at oblique angles of incidence
– worse for polished statues
Marc Levoy
Laser scattering noise
on David’s toe
50 micron scan by NRC/Canada
0.1 mm
Marc Levoy
Some marbles
scatter more than others
Michelangelo’s Pietà
Marc Levoy
Hard problem #2:
scanning in the presence of occlusions
< 30°
?
?
20°
?
• occlusions (& self-occlusion) force grazing
scans, which lead to holes
• a scanner with a fixed triangulation angle
cannot circumvent all occlusions
• a hammer & chisel can reach places
a triangulation scanner cannot
Marc Levoy
Some statues may be “unscannable”
(using optical methods)
Laocoon
Marc Levoy
Hard problem #3:
insuring safety for delicate objects
• energy deposition
– not a problem for marble statues
• avoiding collisions
Marc Levoy
Why are collisions hard to avoid?
• to insure safety, scan head should stay
outside the convex hull of the object
• in the worst case, required standoff may
equal diameter of convex hull
• a complicated object cannot be scanned
entirely from outside its convex hull
Marc Levoy
Hard problem #3:
insuring safety for delicate objects
• energy deposition
– not a problem for marble statues
• avoiding collisions
– manual motion controls
– automatic cutoff switches
– one person serves as spotter
– avoid time pressure
– get enough sleep
• surviving collisions
– pad the scan head
Marc Levoy
Hard problem #4:
scanning large objects at high resolution
• David is 5 meters tall
• chisel marks need 1/4mm
• dynamic range of 20,000:1
• 20,0002 = 1 billion polygons
• 14cm wide working stripe
• David was ~30 stripes around
Marc Levoy
Digital Michelangelo Project
reconfigurable robotic gantry
• calibrated motions
– pitch (yellow)
– pan (blue)
– horizontal translation (orange)
• uncalibrated motions
– vertical translation
– remounting the scan head
– moving the entire gantry
Marc Levoy
Scanning St. Matthew
•
•
•
•
•
104 scans
800 million polygons
4,000 color images
15 gigabytes
1 week of scanning
Marc Levoy
Scanning the David
•
•
•
•
•
•
480 individually aimed scans
2 billion polygons
7,000 color images
32 gigabytes
30 nights of scanning
22 people
Marc Levoy
Hard problem #5:
accurate scanning in the field
• rotating scan motion is hard to make accurate
• field reconfigurations are not repeatable
• need a way to recalibrate in the field
Marc Levoy
Digital Michelangelo Project
range processing pipeline
• steps
1. manual initial alignment
- should have tracked gantry
2. ICP to one existing scan
[Besl92]
3. automatic ICP of all overlapping pairs
[Rusinkiewicz01]
4. global relaxation to spread out error
[Pulli99]
5. merging using volumetric method
[Curless96]
Marc Levoy
Digital Michelangelo Project
range processing pipeline
• steps
1. manual initial alignment
- should have tracked gantry
2. ICP to one existing scan
[Besl92]
3. automatic ICP of all overlapping pairs
[Rusinkiewicz01]
4. global relaxation to spread out error
[Pulli99]
5. merging using volumetric method
[Curless96]
Marc Levoy
Digital Michelangelo Project
range processing pipeline
• steps
1.
2.
3.
4.
5.
manual initial alignment
ICP to one existing scan
automatic ICP of all overlapping pairs
global relaxation to spread out error
merging using volumetric method
Marc Levoy
What really happens?
• steps
1.
2.
3.
4.
5.
manual initial alignment
ICP to one existing scan
automatic ICP of all overlapping pairs
global relaxation to spread out error
merging using volumetric method
• ICP is unstable on smooth surfaces
Marc Levoy
What really happens?
• steps
1.
2.
3.
4.
5.
manual initial alignment
ICP to one existing scan
automatic ICP of all overlapping pairs
global relaxation to spread out error
merging using volumetric method
• ICP is unstable on smooth surfaces
• relaxation distributes errors unevenly
Marc Levoy
Digital Michelangelo Project
color processing pipeline
• steps
1.
2.
3.
4.
compensate for ambient illumination
discard shadowed or specular pixels
map onto vertices – one color per vertex
correct for irradiance  diffuse reflectance
Marc Levoy
What really happens?
• steps
1.
2.
3.
4.
•
•
•
•
compensate for ambient illumination
discard shadowed or specular pixels
map onto vertices – one color per vertex
correct for irradiance  diffuse reflectance
we treated reflectance as Lambertian
we used aggregate surface normals
we ignored interreflections
we ignored subsurface scattering
– derived reflectance is incorrect
– renderings are not realistic
Marc Levoy
The importance of
subsurface scattering
BRDF
BSSRDF
[Jensen et al., Siggraph 2000]
Marc Levoy
artificial surface reflectance
estimated diffuse reflectance
photograph
1.0 mm computer model
with diffuse reflectance
Hard problem #6:
filling holes in dense polygon models
signed
distance
function
volumetric
diffusion
process
Marc Levoy
Hard problem #7:
handling large datasets
3mm
mesh
1mm
0.3mm
Marc Levoy
Some solutions we’ve used
• range images instead of polygon meshes
– z(u,v)
– yields 18:1 lossless compression
– multiresolution using (range) image pyramid
• multiresolution viewer for polygon meshes
– 2 billion polygons
– immediate launching
– real-time frame rate when moving
– progressive refinement when idle
– compact representation
– fast pre-processing
Marc Levoy
The Qsplat viewer
[Rusinkiewicz and Levoy, Siggraph 2000]
• hierarchy of bounding spheres with position,
radius, normal vector, normal cone, color
• traversed recursively subject to time limit
• spheres displayed as splats
Marc Levoy
Streaming Qsplat
[Rusinkiewicz and Levoy, I3D 2001]
1 second
10 seconds
60 seconds
Marc Levoy
Hard problem #8:
creating digital libraries of 3D content
•
•
•
•
metadata – data about data
versioning – cvs for data archives
secure viewers for 3D models
robust 3D digital watermarking
• viewing, measuring, extracting data
• indexing and searching 3D content
• insuring longevity for the archive
Marc Levoy
How to think about range data
• range data is measured data
– both the geometry and the topology are noisy
– “geometric signal processing”
• range data is not a cloud of unorganized points
– use connectivity between range samples
– use lines of sight to scanner and camera
• range datasets are large
– representations that are exact only in the limit
– algorithms that are fast and approximate
– implementations that don’t need entire model in memory
Marc Levoy
Games with range data
1. volumetric scan conversion
2. range image synthesis
3. multi-modality scanned models
Marc Levoy
Game #1:
volumetric scan conversion
[Wei01]
•
•
•
•
convert point cloud to volume densities
facilitates volumetric / statistical analyses
new texture synthesis algorithms
volume texture synthesis
Marc Levoy
Game #2:
range image synthesis
(Hertzmann)
• texture synthesis of range data
– in range images
– in merged surface meshes
• application to hole filling
Marc Levoy
Game #3:
multi-modality scanned models
• time-of-flight scans for overall shape
• triangulation scans for range texture
• photography for reflectance
Marc Levoy
Three final questions
1. Can range scanning be fully automated?
2. Can we build a 3D fax machine?
3. Will IBR replace 3D scanning?
Marc Levoy
The Forma Urbis Romae:
a marble map of ancient Rome
Marc Levoy
Can we replace this?
uncrating...
Marc Levoy
Can we replace this?
positioning...
Marc Levoy
Can we replace this?
scanning...
Marc Levoy
Can we replace this?
aligning...
Marc Levoy
Automatic 3D scanning?
• requires 6 DOF robot
• automatic view planning
[Connolly85, Maver93, Pito96, Reed97,…]
• hard to guarantee safety
• use CT scanning instead?
Marc Levoy
Can we build a 3D fax machine?
• quality and cost of rapid
prototyping is improving
• need automatic scanning
• need color printing
• how to match BRDF?
• looking for a killer app
?
[Curless96]
Marc Levoy
Will IBR replace 3D scanning?
…assuming the reason for scanning
a particular object is to fly around it
• Yes
–
–
–
–
–
–
simple, brute force
universal meta-primitive
geometry is implicit
independent of scene complexity
multiresolution is trivial
hardware acceleration
• No
– discards structure of model
– rasterized geometry
– big and slow
Marc Levoy