Mitsubishi Electric Research Laboratories
Raskar 2007
Active Imaging for
3D Photography
Projector
Pos=0
Tags
Pos=255
Ramesh Raskar
Mitsubishi Electric Research Labs (MERL)
Cambridge, MA
Raskar 2007
Mitsubishi Electric Research Laboratories
Active Imaging for 3D Photography
Projector
Pos=0
• Structured Light Scanning
– Fast Stripping
– Global-direct Separation
Tags
Pos=255
• Projector Calibration
– Quadric and Homography Transfer
• Low Cost Scanning
– Multi-Flash Stereo
• Motion
– Coded Exposure
Raskar 2007
Mitsubishi Electric Research Laboratories
Labeling Space
Projector
Pos=0
Pos=255
Each location
Tags
receives a
unique signal
based upon it’s
physical location.
Mitsubishi Electric Research Laboratories
Raskar 2007
High Speed Tracking
Mitsubishi Electric Research Laboratories
Raskar 2007
Optical Capture
• Goal
– Find 2D or 3D location of tags
• Approach
– Each tag has a photosensor
– Label space using zones created by special
high speed projectors
– Find zone in which the tag lies
Raskar 2007
Mitsubishi Electric Research Laboratories
Inside of Projector
Focusing Optics
Condensing Optics
Light Source
Gray code Slide
The Gray code pattern
Mitsubishi Electric Research Laboratories
Inside of Tag
Raskar 2007
Mitsubishi Electric Research Laboratories
Raskar 2007
Solution Overview
• Use inexpensive solid state projectors
to “label space”.
– Solid State lighting (LED’s) can be
modulated extremely quickly (MHz to GHz).
• Each point in space receives a different
code sequence.
• Inexpensive and small tags receive the
codes, and can detect their own
location.
Raskar 2007
Mitsubishi Electric Research Laboratories
Labeling Space
Projector
Pos=0
Pos=255
Each tag
Tags
receives a
unique signal
based upon it’s
physical location.
Raskar 2007
Mitsubishi Electric Research Laboratories
How Labeling Works
Light
source
Optics
GrayCode Mask
Screen
pos=0
pos=15
Light source blink one by one and each position
on the screen has different light pattern.
4 light make 4 bit position resolution
Raskar 2007
Mitsubishi Electric Research Laboratories
Expand to 2D/3D System
2D measurement
X data
Y data
3D measurement
X2 data
X data
Y data
Mitsubishi Electric Research Laboratories
E-Tech Demo
Raskar 2007
Mitsubishi Electric Research Laboratories
Raskar 2007
Key Features
• 500 Hz Tracking
• Id for each Marker Tag
• Capture in Natural Environment
– Visually imperceptible tags
– Photosensing Tag can be hidden under clothes
– Ambient lighting is ok
• Unlimited Number of Tags Allowed
• Base station and tags only a few 10’s $
High Frequency Illumination Pattern
surface
source
i
camera
+
L [c, i]  Ld [c, i]   Lg [c, i]
fraction of activated source elements
Direct and Global Illumination
participating
medium
surface
source
B
D
A
P
E
camera
translucent
surface
C
A : Direct
B : Interrelection
C : Subsurface
D : Volumetric
E : Diffusion
Fast Separation of Direct and Global Images
Using High Frequency Illumination
Shree K. Nayar
Gurunandan G. Krishnan
Columbia University
Michael D. Grossberg
City College of New York
Ramesh Raskar
MERL
High Frequency Illumination Pattern
surface
source
i
camera
+
L [c, i]  Ld [c, i]   Lg [c, i]
fraction of activated source elements
High Frequency Illumination Pattern
surface
source
i
camera
+
L [c, i]  Ld [c, i]   Lg [c, i]
fraction of activated source elements
-
L [c, i]  (1   ) Lg [c, i]
Separation from Two Images
1
 :
2
Ld  Lmax  Lmin , Lg  2Lmin
direct
global
Diffuse
Interreflections
Specular
Interreflections
Diffusion
Volumetric
Scattering
Subsurface
Scattering
Scene
Scene
Direct
Global
Shower Curtain: Diffuser
Mesh
Shadow
Ld  Lmax   Lmin , Lg   Lmin
direct
global
Shower Curtain: Diffuser
Direct
Global
Mitsubishi Electric Research Laboratories
Raskar 2007
Project Topics
• Structured Light Scanning
– Fast Stripping
• Can you build a scanner using very low cost hardware?
• Without full 2D cameras or video projectors?
– Global-direct Separation
• Can you scan difficult (global effect) using direct/global
separation?
Raskar 2007
Mitsubishi Electric Research Laboratories
Active Imaging for 3D Photography
Projector
Pos=0
• Structured Light Scanning
– Fast Stripping
– Global-direct Separation
Tags
Pos=255
• Projector Calibration
– Quadric and Homography Transfer
• Low Cost Scanning
– Multi-Flash Stereo
• Motion
– Coded Exposure
Raskar
Mitsubishi Electric Research Laboratories
Planar projective transfer
(homography)
– Two images of 3D points on a plane are
related by a 3x3 matrix
M
Screen
Defined by 4 or more
corresponding pixels
i
Camera
1
j
Camera
2
Raskar
Mitsubishi Electric Research Laboratories
Planar display surface
Use homography (A3x3)
M
j = Ai
i
A
j
Raskar
Mitsubishi Electric Research Laboratories
Curved projective transfer
Quadric classification
Projectively equivalent to sphere:
sphere
ellipsoid
Ruled quadrics:
hyperboloid paraboloid
of two sheets
hyperboloids
of one sheet
Degenerate ruled quadrics:
cone
two planes
Mitsubishi Electric Research Laboratories
Quardic curved shape Displays
Planetarium
Sim/Viz Center
Raskar, vanBaar, Willwacher, Rao
‘Quadric Transfer for Immersive Curved Displays’,
EuroGraphics 2004
Raskar
Raskar
Mitsubishi Electric Research Laboratories
Quadrics
XT QX  0
For 3D points X on Quadric
X
Q : 4x4 symmetric
Q
Nine d.o.f


matrix, Q  
















In general 9 points in 3D define quadric
Raskar
Mitsubishi Electric Research Laboratories
Parametric Image Transfer
X
X
i
i
j
j
Planar Homography
j  A3x3 i
Quadric Transfer
j  A3x3 i 
 i E i e
T
Raskar
Mitsubishi Electric Research Laboratories
Simplified Quadric Image
Transfer
 T

x'  Ax   x Ex e


17 param warp
X
x
x’
Planar homography:
Quadric transfer:
4 corresponding pixels
9 corresponding pixels
Mitsubishi Electric Research Laboratories
Raskar
Overlap on Quadric Screens
Mitsubishi Electric Research Laboratories
Raskar
Mitsubishi Electric Research Laboratories
Raskar
Mitsubishi Electric Research Laboratories
Raskar
Vertex Shader for Quadric Transfer in Cg
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
vertout main( appin IN,
uniform float4x4 modelViewProj,
uniform float4 constColor, uniform float3x3 A, uniform float3x3 E,
uniform float3 e) {
vertout OUT;
float4 m1 = float4(IN.position.x, IN.position.y, IN.position.z,
1.0f );
float4 m, mi ;
float3 m2,mp; float scale;
m = mul( modelViewProj, m1);
m2.x = m.x/m.w;
m2.y = m.y/m.w;
m2.z = 1;
scale = mul(m2, mul(E,m2));
mp = mul(A,m2) + sqrt(scale)*e;
mi.x = m.w * (mp.x)/(mp.z);
mi.y = m.w * (mp.y)/(mp.z);
mi.zw = m.zw;
OUT.position = mi;
OUT.color0 = IN.color0; // Use the original per-vertex color
specified
return OUT;
}
Mitsubishi Electric Research Laboratories
Raskar 2007
Project Topics
• Structured Light Scanning
– Fast Stripping
• Can you build a scanner using very low cost
hardware?Without full 2D cameras or video projectors?
– Global-direct Separation
• Can you scan difficult (global effect) using direct/global
separation?
• Projector Calibration
– Quadric and Homography Transfer
• Can you calibrate with non-planar surfaces ?
Raskar 2007
Mitsubishi Electric Research Laboratories
Active Imaging for 3D Photography
Projector
Pos=0
• Structured Light Scanning
– Fast Stripping
– Global-direct Separation
Tags
Pos=255
• Projector Calibration
– Quadric and Homography Transfer
• Low Cost Scanning
– Multi-Flash Stereo
• Motion
– Coded Exposure
Depth Edges with Multi-Flash
Raskar, Tan, Feris, Yu, Turk – ACM SIGGRAPH 2004
Bottom Flash
Top Flash
Left Flash
Right Flash
Ratio images and directions of epipolar traversal
Shadow-Free
Depth Edges
Shadow-Free
Depth Edges
Raskar
Mitsubishi Electric Research Laboratories
Photo
Our Method
Result
Canny Intensity
Edge Detection
May 04
Discontinuity Preserving Stereo
with Small Baseline Multi-Flash
Illumination
Rogerio Feris1, Ramesh Raskar2, Longbin Chen1,
Karhan Tan3 and Matthew Turk1
1
University of California, Santa Barbara
2
Mitsubishi Electric Research Labs
3Epson Palo Alto Lab
Sign of Depth Edge
Original
Ratio Left
Ratio Right
Signed Edges
-++-
(+) Foreground (-) Background
Shadow Width
Bottom Flash Image
Ratio Image
Plot Along Scanline
Qualitative Depth
Useful Prior Information for Stereo !
Qualitative Depth
Flash Image
Challenging scenarios for passive stereo:
Complex geometry
Specularities
Flat regions with no texture
Qualitative Depth Map
Uses only the knowledge of what is
foreground and what is background locally
near a depth edge. This image only has
relative (qualitative) depth values but not
actual (quantitative) depth values. Note how
the estimates are sharp near depth edges.
Qualitative Depth
Flash Image
3D Plot
Key problem in stereo matching:
Incorrect matches for partially occluded regions
Partial Occlusion Problem
Occlusion
Partial Occlusion = Region Bounded by
Shadows due to flash around Camera B
Occlusion  Average of Upper/Lower Shadow widths
(Seen by A but not by B)
Object
Object
Multi-flash
Solution
Flash
A
B
Camera
Camera A photo with Flash Around Camera B
A
B
Camera
Estimate of Partial Occlusion using
average of flash shadow widths
Local Stereo Matching
Left View
Depth Edges + Occlusion
Ground Truth
Conventional 31x31
Our Approach 31x31
Depth estimate near depth edges are sharp. But
Mitsubishi Electric Research Laboratories
Raskar 2007
Project Topics
• Structured Light Scanning
– Fast Stripping
• Can you build a scanner using very low cost
hardware?Without full 2D cameras or video projectors?
– Global-direct Separation
• Can you scan difficult (global effect) using direct/global
separation?
• Projector Calibration
– Quadric and Homography Transfer
• Can you calibrate with non-planar surfaces ?
• Low Cost Scanning
– Multi-Flash Stereo
• How far can you push simple flashes for 3D scanning ?
Mitsubishi Electric Research Laboratories
Raskar 2007
Active Imaging for 3D Photography
• Structured Light Scanning
– Fast Stripping
– Global-direct Separation
• Projector Calibration
– Quadric and Homography Transfer
• Low Cost Scanning
– Multi-Flash Stereo
• Motion
– Coded Exposure
Figure 2 results
Input Image
Rectified Image to make motion lines parallel to scan lines.
Approximate cutout. Exact alignment is not required.
Image Deblurred by solving a linear system. No post-processing
Flutter Shutter Camera
Raskar, Agrawal, Tumblin [Siggraph2006]
Ferroelectric LCD shutter in front of the lens is turned
opaque or transparent in a rapid binary sequence
Exposure choices for capturing fast moving objects
Shutter On
Shutter Off
Keep Shutter open for
entire exposure duration
Time
Traditional Camera
The moving object creates smear
Shutter On
Shutter Off
Time
Coded Exposure Camera
Shutter On
Shutter Off Short Exposure
Time
Flutter shutter open and closed
with a psuedo-random binary
sequence within exposure duration
to encode the blur
Keep shutter open for very short
duration. Avoids blur but image is
dark and suffers from noise.
Coded Exposure Photography:
Assisting Motion Deblurring using Fluttered Shutter
Short Exposure
Traditional
MURA
Coded
Shutter
Captured
Single
Photo
Deblurred
Result
Image is dark
and noisy
Result has Banding
Artifacts and some spatial
frequencies are lost
Decoded image is as
good as image of a
static scene
How to Preserve Spatial Frequencies in Blurred Image ?
Traditional Smear = Convolution with temporal box filter
= Multiplication with Sinc function in fourier domain
= Most frequencies are lost !
Coded Smear = Convolution with temporal coded filter
= Multiplication with nearly flat function in fourier domain
= Most frequencies are preserved !
Encoding Blur with a Broadband Temporal Filter
Coded Smear
= Preserves spatial frequencies
= Deconvolution becomes a well-posed problem
= Deconvolution filter frequency response is nearly flat
Encoding Filter (Blurring)
Decoding Filter (Deblurring)
Mitsubishi Electric Research Laboratories
Raskar 2007
Project Topics
• Structured Light Scanning
– Fast Stripping
• Can you build a scanner using very low cost hardware?Without full
2D cameras or video projectors?
– Global-direct Separation
• Can you scan difficult (global effect) using direct/global separation?
• Projector Calibration
– Quadric and Homography Transfer
• Can you calibrate with non-planar surfaces ?
• Low Cost Scanning
– Multi-Flash Stereo
• How far can you push simple flashes for 3D scanning ?
• Motion
– Coded Exposure
• Can one estimate depth in presence of motion ?
Raskar 2007
Mitsubishi Electric Research Laboratories
Active Imaging for 3D Photography
Projector
Pos=0
• Structured Light Scanning
– Fast Stripping
– Global-direct Separation
Tags
Pos=255
• Projector Calibration
– Quadric and Homography Transfer
• Low Cost Scanning
– Multi-Flash Stereo
• Motion
– Coded Exposure
Mitsubishi Electric Research Laboratories
Raskar 2007
Open Problems
• Structured Light Scanning
– Fast Stripping
• Can you build a scanner using very low cost hardware?Without full
2D cameras or video projectors?
– Global-direct Separation
• Can you scan difficult (global effect) using direct/global separation?
• Projector Calibration
– Quadric and Homography Transfer
• Can you calibrate with non-planar surfaces ?
• Low Cost Scanning
– Multi-Flash Stereo
• How far can you push simple flashes for 3D scanning ?
• Motion
– Coded Exposure
• Can one estimate depth in presence of motion ?
Mitsubishi Electric Research Laboratories
Raskar 2007
Active Imaging for 3D Photography
• Sensors
– Multi-flash Camera
– Image Fusion
• Display
– Multiprojector, Display Grid
• Interaction
– Composite RFID
RFIG
– AR with Shader Lamps
Mitsubishi Electric Research Laboratories
Raskar 2007
Active Imaging for 3D Photography
• Structured Light Scanning
– Fast Stripping
– Global-direct Separation
• Projector Calibration
– Quadric and Homography Transfer
• Low Cost Scanning
– Multi-Flash Stereo
• Motion
– Coded Exposure
Mitsubishi Electric Research Laboratories
Raskar 2007
Project Topics
• Structured Light Scanning
– Fast Stripping
• Can you build a scanner using very low cost hardware?Without full
2D cameras or video projectors?
– Global-direct Separation
• Can you scan difficult (global effect) using direct/global separation?
• Projector Calibration
– Quadric and Homography Transfer
• Can you calibrate with non-planar surfaces ?
• Low Cost Scanning
– Multi-Flash Stereo
• How far can you push simple flashes for 3D scanning ?
• Motion
– Coded Exposure
• Can one estimate depth in presence of motion ?