Generalized Mosaics
Yoav Y. Schechner,
Shree Nayar
Department of Computer Science
Columbia University
Mosaics
Processing
Redundant Measurements
Generalized Mosaicing: Yoav Schechner and Shree Nayar
Scanning with Less Redundancy
Mantis Shrimp
C Takata
Different rows =
Different optical characteristics
Generalized Mosaicing
camera
Spatially
varying
filter
Field of View
Brightness
Dynamic Range
Spectrum
Polarization
Depth of Field
Schechner, Nayar, Generalized mosaics
Mosaic + High Dynamic Range
88 - 18,794
Fusion of Measurements
Raw k ( x, y)  I ( x, y)  Mask( x)
Raw k ( x, y)
Mask ( x)
I  I 2 
k
 I k ( x, y)  I k ( x, y)
Ik
I k2


2
I  

 k



1 
I k2 
1
Maximum-Likelihood Solution
Schechner, Nayar, Generalized mosaics
Log of the mask
log 2 M
-2
-4
-6
-8
8 bits (almost)
x
(pixels)
8 bit camera
1
10-6
Dynamic range as 16 bits
=
1
10-2
Generalized Mosaicing: Yoav Schechner and Shree Nayar
Mask Self-Calibration
Unknown filter (vignetting)
Raw frame( x, y)  I (x, y)  Mask(x, y)
Mask ( x) 
  Raw fram e( x, y)
fram es y
Mask( x1)  Raw 2  Mask( x2 )  Raw1  0
M
Consistency, Smoothness
A




min  Mask t At A Mask 
average row
consistency
constraints
Schechner, Nayar, Generalized mosaics
x
Image Registration
2
DIFF total

2
DIFFeach
corresponding pair

all pixels
Minimize:
2
DIFF total

2
DIFFeach
corresponding pair

all pixels
2
DIFFtotal
Schechner, Nayar, Generalized mosaics
Image Registration
Minimize:
2
DIFF total


2
DIFFeach
corresponding pair
all pixels
Raw k ( x, y)  I ( x, y)  Mask( x)
Bias towards “no motion”
Schechner, Nayar, Generalized mosaics
Image Registration
Raw k ( x, y)  I ( x, y)  Mask( x)  Raw
 Mask
Uncertainty
Raw k ( x, y)
Mask ( x)
 I k ( x, y)  I k ( x, y)
2
2
I I 
I I 




2
1 
2
DIFFpixel




pair 
2
2

 I 

I
1 
2 


where
I  I 2 
Ik
2
k I k
,


I 2  

 k



1 
I k2 
1
Schechner, Nayar, Generalized mosaics
Registration: Standard “Coarse to Fine”
motion of
2 pixels
4 pixels
5 pixels
10 pix
9 pix
18 pix
19 pix
However, we need: multiscale UNCERTAINTIES
2
2
I I 
I I 




2
1 
2
DIFFpixel




pair 
2
2

 I 

I
1 
2 


Schechner, Nayar, Generalized mosaics
Maximum-Likelihood Pyramid
At each level L , for each pixel
estimate
I L , I L
I
w( x, y)I ( x, y)

x, y
w( x, y)

x, y
I 2 
w(x, y)

x, y
-1
w(x, y)  a(2x, y)
I (x, y)
Schechner, Nayar, Generalized mosaics
Max-Likelihood Pyramid
Gaussian Pyramid
ML estimation
I  I 2 
I   g ( x) I ( x)
x
x
I ( x)
I 2 ( x)
,


2
I  
 x



1 
I 2 ( x) 

g (x)
I (1) I (2) I (3) I (4) I (5) x
I (1) I (2)
I (3) I (4) I (5) x
I (1) I (2) I (3) I (4) I (5)
( x) 
I  n ( x) I ( x)
x
1
n [ ( x)]
g ( x)
I 2 ( x)

2
I  

 x


( x) 


1
1
Mosaic
Generalized Mosaic
log I
Intensity range
Intensity range
log I
frame
Spatial range
x
Spatial range
x
Schechner, Nayar, Generalized mosaics
Variable Spectral Filter
spectral
y
l
x
Generalized Mosaicing: Yoav Schechner and Shree Nayar
Multi Spectral Mosaic
400
500
600 l 700
400
500
600 l 700
400
500
600 l 700
Generalized Mosaicing: Yoav Schechner and Shree Nayar
Rendering : Any Illumination
sunset
fluorescent
Generalized Mosaicing: Yoav Schechner and Shree Nayar
Illumination
at a Glance
Halogen
y
Fluorescent
y
l
l
x
x
Extra information
 Illuminant spectrum
Spatially Varying Polarizer
Polarization Mosaicing: Yoav Schechner and Shree Nayar
Spatially Varying Polarizer
1
0.8
0.6
polarizance
90
o
polarizing
60
angle
30
0
0.4
0.2
0
o
o
o
-30
o
-60
-90
o
o
Polarization Mosaicing: Yoav Schechner and Shree Nayar
Raw images
Polarization Mosaicing: Yoav Schechner and Shree Nayar
Polarization Mosaic
reflected structure
Schechner
Shamir
Kiryati
JOSA-A
2000
transmitted painting
Spatially Varying Focus
Insert a
prism
behind the lens
y
=
Depth
x
Generalized Mosaicing: Yoav Schechner and Shree Nayar
All Focused Mosaic
… and a Depth Map
Generalized Mosaicing: Yoav Schechner and Shree Nayar
What else?

Multi-dimensional Mosaics - Simultaneously
Dynamic Range & Spectrum & Polarization etc.
Generalized Mosaicing: Yoav Schechner and Shree Nayar
Sampling Criteria
Number of samples per scene point
2
(D / L) 2  (l / B) 2
l
length L
bandwidth B
resolution l
aperture D
 Signal undergoing LPF
 “Band limited” signal
 Minimizing aliasing
1011100101
 Nyquist sampling
rate
# samples
1
1
11111111
1/2
2
11111111?
10111001??
1/4
3
1011100???
1
9
101110????
256
M
1
M
1 log 2 M
Generalized Mosaicing: Yoav Schechner and Shree Nayar
Generalized
Mosaic Mosaic
Wide field of view
Spatially varying
filter
Dynamic range
88 – 18,794
I
Spectrum
l
Polarization
Depth + Focus