Midterm Review
15-463: Computational Photography
Review Topics
•
•
•
•
•
•
•
•
Sampling and Reconstruction
Frequency Domain and Filtering
Blending
Warping
Data-driven Methods
Camera
Homographies
Modeling Light
Review Topics
•
•
•
•
•
•
•
•
Sampling and Reconstruction
Frequency Domain and Filtering
Blending
Warping
Data-driven Methods
Camera
Homographies
Modeling Light
Sampling
Reconstruction
Samplingand
and Reconstruction
ampled representations
Reconstruction
Sampling and Reconstruction
• Making
samples back into a continuous function
How to store and compute with continuous
functions?
[FvDFH fig.14.14b / Wolberg]
– for output (need realizable method)
Common scheme for representation: samples
– for analysis or processing (need mathematical method)
– write down the function’s values at many points
– amounts to “guessing” what the function did in between
© 2006 Steve Marschner • 3
© 2006 Steve Marsch
Mathematically guess what happens in between
Sampling and Reconstruction
Undersampling
•
•
Effects of Undersampling
•
What if we “missed” things between the samples?
informationa sine wave
Simple example:
• Lostundersampling
• High frequency signals get indistinguishable
Aliasing in imagesfrom
– unsurprising result: information is lost
– surprising result: indistinguishable from lower frequency
low frequency ones (aliasing)
© 2006 Steve Marschner • 9
Sampling and Reconstruction
•
•
How to avoid aliasing?
•
•
Sample more often
Low pass filter the signal (anti-aliasing)
Filters work by convolution
Sampling and Reconstruction
•
Examples of filters
•
•
•
Moving average
Weighted moving average
•
•
Equal weights
Gaussian weights
Sobel
Sampling and Reconstruction
•
Gaussian Filters
•
•
Smoothe out images
Convolution of two Gaussians each with
standard deviation σ, gives Gaussian with
standard deviation σ/2
Sampling and Reconstruction
•
•
Matching
•
Use normalized-cross correlation or SSH over patches
Subsampling
•
•
Filter with Gaussian then subsample
Double filter size with every half-sizing
•
Forms image pyramids
Sampling and Reconstruction
Compare with...
1/2
1/4
(2x zoom)
1/8
(4x zoom)
Slide by Steve Seitz
Sampling and Reconstruction
Subsampling with Gaussian pre-filtering
Gaussian 1/2
G 1/4
G 1/8
Slide by Steve Seitz
Review Topics
•
•
•
•
•
•
•
•
Sampling and Reconstruction
Frequency Domain and Filtering
Blending
Warping
Data-driven Methods
Camera
Homographies
Modeling Light
Frequency
Domain
A nice set of basis
Teases away fast vs. slow changes in the image.
What does blurring take away?
smoothed
(5x5 Gaussian)
This change of basis has a special
name…
Decompose signal into different frequencies
Our building block:
Asin("x + ! )
Frequency Domain and Filtering
Add enough of them to get
any signal f(x) you want!
sines
A sum
of
sines
A sum
of sines
Sum
of
of different frequencies
Howsine
manywaves
degrees of
building
block:freedom?
block:
Our Our
building
block:
! ) does each control?
"x + !Asin(
) Asin(
"x "
+x!+) What
nes
enough
of them
to get
of
themAdd
to get
Add
enough
of them
to get
any
signal
f(x) you
want! one encodes the
Which
) you
any want!
signal
f(x) you
want!
ck:
coarse vs. fine structure of
the
signal?
+How
! ) How
many
degrees
of
egrees
of
many degrees of
freedom?
freedom?
hem to get
ou want!
Fourier
What
control? Transform
What
doesdoes
eacheach
control?
ach
control?
We want to understand the frequency ! of our signal. So,
ees of
Which
encodes
let’s
reparametrize the signal by ! instead of x:
Which
one one
encodes
the the
ncodes
the
coarse
vs. fine
structure
vs.offine
structure
of of
e coarse
structure
Fourier
the signal?
the signal?
f(x)
F(!)
control?
Transform
Frequency Domain and Filtering
A nice set of basis
In 2D
Teases away fast vs. slow changes in the image.
What does blurring take away?
smoothed
(5x5 Gaussian)
This change of basis has a special
name…
Frequency Domain and Filtering
Man-made Scene
Low and High Pass filtering
Frequency Domain and Filtering
What does blurring take away?What does blurring take away?High-Pass filter
Why Laplacian?
smoothed – original
original
smoothed (5x5 Gaussian)
Gaussian
delta function
Laplacian of Gaussian
Frequency
Domain
and
Filtering
Lossy Image Compression (JPEG)
Block-based Discrete Cosine Transform (DCT)
Frequency Domain and Filtering
Image gradient
The gradient of an image:
The gradient points in the direction of most rapid change in intensity
The gradient direction is given by:
• how does this relate to the direction of the edge?
The edge strength is given by the gradient magnitude
Frequency Domain and Filtering
Derivative theorem of convolution
Solution: smooth first
This saves us one operation:
Where is the edge? Look for peaks in
Review Topics
•
•
•
•
•
•
•
•
Sampling and Reconstruction
Frequency Domain and Filtering
Blending
Warping
Data-driven Methods
Camera
Homographies
Modeling Light
Blending
Alpha Blending / Feathering
+
1
0
1
0
Iblend = !Ileft + (1-!)Iright
=
•
•
Window size = size of largest feature (to avoid strong seams)
Window size <= 2 * size of smallest feature (to avoid ghosting)
Blending
Pyramid
Octaves in the Spatial Domain
Lowpass
Images
Pyramid
Blending
Blending
Pyramid Blending
1
0
1
0
1
0
Left pyramid
Bandpass Images
blend
Right pyramid
Blending
! !"#$%$&'#$()*+,'%-.(,/
! 0)"1#*1)+$2#-,-3*$%'4-*5 u
Gradient Domain
! !1#"1#*+$2#-,-3*$%'4-*5 f
! 6"-7$+8*3-9$,-3*"$:-2;3$++-,-)7-9*5 <gx, gy)
•
Result image: f
! 6"-7$+8*3-9$,-3*"$:-2;='21-9*5
d
Gradients: fx, fy
! 6"-7$+8*7()9#,'$)#9*.-$4>#9*5
<wx, wy, wd?
•
Want f to ‘look like’ some prespecified d, and
x, gy
fx, fy to ‘look like’
some
prespecified
g
B)-,48*+1)7#$()
%$)***wx<fx 5 gx?@**A*wy<fy 5 gy?@**A*wd<f 5 d?@
•
+
Weights specify per-pixel importance of how much you want f
close to d, fx close to gx, fy close to gy
Blending
Gradient Domain
t
s
f x = s x , fy = s y
f (x, y) − t(x − 1, y) = sx , f (x, y) − t(x, y − 1) = sy
Review Topics
•
•
•
•
•
•
•
•
Sampling and Reconstruction
Frequency Domain and Filtering
Blending
Warping
Data-driven Methods
Camera
Homographies
Modeling Light
Point
Processing
Image Processing
Image Histograms
Change range of image
image filtering: change range of image
g(x) = h(f(x))
f
g
h
Image Histograms
Histogram Equalization
Example: g(x)=f(x)+.3
image warping: change domain of image
f
g(x) = f(h(x))
h
g
Cumulative H
s = T(r
Histogram Equalization
Cumulative Histograms
image filtering: change range of image
g(x) = h(f(x))
f
Warping
g
h
Change
domain
ofofimage
image warping:
change
domain
image
f
g(x) = f(h(x))
h
g
Example: g(x)=f(x/2)
Warping
•
2D Transformations
•
•
•
•
•
•
Translate
Rotate
Scale
Similarity
Affine
Projective
Warping
Change
of
Basis
Linear Transformations as Change of Basis
Linear
Transformations
asas
Change
Linear
Transformations
ChangeofofBasis
Basis
vv=(v
,v,vy) ),v )
=(v
vx=(v
x yx y
j =(0,1)
j =(0,1)
j =(0,1)
ppuvuv uv
p
pijpij ij
p
u=(u
x,u
y) ),u )
u=(u
,u
u=(u
x yx y
ppuvuv==uv
(4,3)
p (4,3)
= (4,3)
ppx=4u
x+3v
x
=4u
x px=4u
x+3v
x x
x+3v
ppy=4u
+3v
y +3v
y⎡
py=4u
y=4u
y y+3v
y u xy
i =(1,0)
i =(1,0)
i =(1,0)
ij
ppij==4u+3v
pij 4u+3v
= 4u+3v
&u&ux &u vvx # &v#4&#4# # &u&xu&u v xv# v# #
ppijij=p=ij$ = x x !x $ x !&=4=#$ = x x !xpxuv
uvuv
p
$
!
$
!
⎤$! !%u y $ v y " !p
v x ⎤ ⎡ 4%⎤uuy $⎡uuvx vy " v%v3$x !3
"
u
v
u
3
y
y
y
yv"y "
%
"
=
uv
%
"
%
y
y
y
%
"
%
"
%
⎥ ⎢ ⎥ ⎢
⎥ p
pij = ⎢
⎣u y v y ⎦ ⎣ 3⎦ ⎣u y v y ⎦
Any
linear
transformation
isisaabasis!!!
Any
linear
transformation
Any linear transformation is basis!!!
a basis!!!
Warping
Change of Basis: Inverse Transform
What’s
the
inverse
transform?
What’sthe
theinverse
inversetransform?
transform?
j =(0,1)
=(0,1)
j =(0,1)
vvv=(v
=(v
x,v
y)y))
=(v
,v
xx,v
y
ppijpij ij
uvuv
pp
uv
u=(u
x,u,u
y) )
u=(u
x y
ii =(1,0)
i=(1,0)
=(1,0)
ij = (5,4) = p u + p v
p
pijpij==(5,4)
(5,4)==pxpxuxu++pypyv
yv
-1
-1-1
&
u
x
&
u
&
u
uv
pppuvuv===$ ⎡ ux xx
$%u$⎢uy
uv
=
p %u
% uy y
⎣
y
•
•
•
•
&
vvxv#x##-1
u
&
#
5
x
&
u
&
u
&
#
5
&
#
5
x
xx
=
⎤
⎡
⎡
⎤
!!x !$$$!5!!==$$$ u x
v
vvvy " ⎥%4⎢4"4 ⎥ =%u⎢uuy
y y"" %% "" %% yy
v ⎦ ⎣ 4⎦ ⎣u
y
y
ppuv
===(p(p
??
uv
x,p,p
y) )= =
puv
(p
xx y
-1-1
-1
vvxv#x##-1
x pij ij
!x!!⎤ ppij
v
vvvy " ⎥ pij
yy""
v ⎦
y
• How
Howcan
canwe
wechange
changefrom
fromany
anybasis
basistoto
toany
anybasis?
basis?
How
can
we
change
from
any
basis
any
basis?
if ifthe
basis
are
orthogonal?
• What
What
the
basis
are
orthogonal?
What if the basis are orthogonal?
Warping
•
Affine Warp
•
Need 3 correspondences
B’
B
?
T(x,y)
A
Source
C
C’
A’
Destination
Warping
•
Many ways to find affine matrix
•
•
Warp Source to [0,0], [1,0], [0,1], and then to Destination
Pose as system of equations in [a;b;c;d;e;f]
B’
B
?
T(x,y)
A
Source
C
C’
A’
Destination
Warping
•
Forward warp
T(x,y)
y
•
y’
x
x’
f(x,y)
Inverse warp
g(x’,y’)
T-1(x,y)
y’
y
x
f(x,y)
x’
g(x’,y’)
Morphing
Review Topics
•
•
•
•
•
•
•
•
Sampling and Reconstruction
Frequency Domain and Filtering
Blending
Warping
Data-driven Methods
Camera
Homographies
Modeling Light
Subspace: convex combin
Data-DrivenLinearMethods
Back to the Subspace
Subspaces Methods (ex: Faces)
Any new image X
obtained as weigh
of stored “basis” im
m
X = !a
Linear Subspace: convex combinations
i =1
Any new image X can be
obtained as weighted sum
of stored “basis” images.
Our old friend, chan
What are the new c
of X?
Write an image as linear combination of
basis images
m
X = ! ai X i
i =1
The Morphable face model
Data-Driven Methods
Again, assuming that we have m such vector pairs in full correspondence
we can form new shapes Smodel and new appearances Tmodel as:
Subspaces Methods (ex: Faces)
m
S model = ! ai S i
i =1
m
Tmodel = ! bi Ti
i =1
If number of basis Shape
faces mand
is large
enough to span
the face subspace then:
Appearance
Models
Any new face can be represented as a pair of vectors
Data-Driven Methods
Subspaces Methods (ex: Faces)
•
How to get basis?
•
•
Principal
Analysis
How
many Component
basis images to
use?
How
get set
images that ,capture
important
variations?
Given to
a point
in an M-dim
space, PCA
• Use PCA (principal component analysis)
• Keep those principal components whose eigenvalues are
finds a basis such that
• coefficients of the point set in that basis are uncorrelated
• first r < M basis vectors provide an approximate basis that
minimizes the mean-squared-error (MSE) in the approximation
(over all bases with dimension r)
above a threshold
x1
x1
2nd principal
component
x0
1st principal
component
x0
Data-Driven Methods
Video Textures
•
•
•
Compute SSD between frames
At frame i, transit either to
frame i+1
• Transition
probabilities
• frame j (if SSD(j, i+1) is small)
•
Probability for transition Pi!j inversely related
Decide to go from i to j or i+1 by tossing a weighted coin.
to cost:
•Pi!j ~ exp ( – Ci!j / "2 )
Data-Driven Methods
Texture
Synthesis
Efros
& Leung
Algorithm
$0$1!-2-3'%2)4
(-3!/)$+
!
5$!6%,)3-+'
"#$%&'()*)$+,-,!).'/
•
•
• Assuming Markov property, compute P(p|N(p))
Search input
image for similar neighborhoods
– Building explicit probability tables infeasible
•
•
– Instead, we
search the input
image
for
all similar
Use Gaussian
weighted
SSD
for
search
to
emphasize
neighborhoods — that’s our pdf for p
central– pixel
To sample from this pdf, just pick one match at
random
Sample one neighborhood at random
Grow texture
Data-Driven Methods
ros & Freeman]
Minimal error boundary
Blocked
Texture
Synthesis
Image Quilting [Efros & Freeman]
(.&'/,00"#1%)/(234
)(+%.'/
!1'#*
#,#-!).)(+%.'/
0)(!1'#*
!
2
"#!$%&'()*+
•
•
!
2
6
"#!$%&'()*+
;6#%7+0'<'#*&)&91,/:
(.&'/,0%&''('
!"#$%&''('%)(*#+,'-
• Observation: neighbor pixels are highly correlated
s are highly correlated
/:
.&'5"2,/%)(*#+,'-
Search input
2$0&)342564277
Grow
! 8/%*)9$,&+:;),(0&*+,<=+)$'')><#+',)<0)$).'1%?)$&)10%+
texture by synthesizing
blocks
+',)<0)$).'1%?)$&)10%+
•
"3+)4 $#'%&,5&06#%7+0'0&8&91,/:
image for
similar
neighborhoods
around
! "#$%&'()&*+),$-+)./&)012)2+)2$0&)342564277
! 61&)&*+),$-+)$,)-/'&<@,%$'+A
Find boundary with minimum error (seam carving)
block
Data-Driven Methods
Lots of Data
•
Ex: Scene completion
•
Search millions of images on the Internet to
find a patch that will complete your image
!"#$%&#'(&)*+,%-&./0'0&1,234056,'&7%6'8&96446,'%&,*&:;,5,8+#3;%-
!"#$#%&'()*#+,'$%-#./*+
!"#$#%&#'"()*+,( Methods
Data-Driven
Scene Completion (GIST descriptor)
!"#$
!"#$#%&#'"()*+,(
Red Car Retrievals (Color histogra
K [h (m) ! h (m)]2
1
j
" 2 (hi , hj ) = ! i
2 m=1 hi (m) + hj (m)
Histogram matching distance
Data-Driven Methods
Multi-scale filter decomposition
Filter bank
Input image
Data-Driven Methods
Lots of Data
•
•
Issues with Data
•
•
Sampling Bias
Photographer Bias
Reduce Bias
•
•
Use autonomous techniques to capture data
StreetView, satellite, webcam etc.
Review Topics
•
•
•
•
•
•
•
•
Sampling and Reconstruction
Frequency Domain and Filtering
Blending
Warping
Data-driven Methods
Camera
Homographies
Modeling Light
eling projection
Camera
Pinhole
Model
Pinhole camera model
Modeling projection
Center of Projection
Image Plane
Focal Length
ction
equations
The coordinate system
Pinhole model:
• We will use the pin-hole model as an approximation
ompute
intersection
with
PP
ray
froma(x,y,z)
totheCOP
• Captures
pencil of
raysProjection
– allof
rays
through
single
point
Perspective
• Put
optical center (Center Of Projection) at the origin
The point
is calledtriangles
Center of Projection
(COP)
erived• using
similar
(on board)
• Put the image plane (Projection Plane) in front of the COP
–
– Why?
• The image is formed on the Image Plane
The camera
• Effective focal length f is distance from COP to •Image
Plane looks down the negative z axis
– we need this if we want right-handed-coordinates
Slide by Steve Seitz
e get the projection by throwing out the last coordinate:
Sl
Camera
Pinhole Model
Perspective Projection
Perspective
Projection (Matrix Representation)
Projection is a matrix multiply using homogeneous
coordinates:
divide by third coordinate
This is known Homogeneous
as perspective projection
• The matrix is the
projection matrix
Coordinates
• Can also formulate as a 4x4
Orthographic Projection
Camera
Special case of perspective projection
• Distance from the COP to the PP is infinite
Image
Pinhole Model
World
Orthographic
Projection (Matrix Representation)
• Also called “parallel projection”
• What’s the projection matrix?
Slide by Steve Seitz
Camera
Pinhole Model
•
•
Pinhole camera aperture
•
•
Large aperture --- blurry image
Small aperture --- not enough light,
diffraction effects
Lenses create sharp images with large
aperture
•
Trade-off: only at a certain focus
Camera
Pinhole Model
•
•
Depth of field
•
Distance over which objects are
in focus
Depth of Field
Field of view
•
Angle of visible region
http://www.cambridgeincolour.com/tutorials/depth-of-field.htm
Review Topics
•
•
•
•
•
•
•
•
Sampling and Reconstruction
Frequency Domain and Filtering
Blending
Warping
Data-driven Methods
Camera
Homographies
Modeling Light
Homographies
•
Panorama
•
•
Reproject images onto a common plane
Images should
have same center of projection
Image reprojection
Mosaic:
Synthetic wide-angle camera
Projective Warp
(Homography)
mosaic PP
Homographies
H
p’
p


��

a
x
 y ��  = d
g
w��
 
b c
x
��
��
x
y
e f  y  , x� = �� , y � = ��
w
w
h 1
1
Homographies

•

��

a
x
 y ��  = d
��
g
w
Expand equations and rewrite as
�
h= a
•
 
b c
x
��
��
x
y
�
�



e f
y , x = �� , y = ��
w
w
h 1
1
Ph = q
b c d e
f
g
Solve using least-squares (h = P\q)
h
�T
Other Projection Models
Cylindrical Projection
Cylindrical projection
• Map 3D point (X,Y,Z) onto cylinder
Y
Z
X
unit cylinder
• Convert to cylindrical coordinates
• Convert to cylindrical image coordinates
unwrapped cylinder
cylindrical image
Other Projection Models
Spherical Projection
Spherical projection
• Map 3D point (X,Y,Z) onto sphere
Y
Z
X
( xˆ , yˆ , zˆ )
1
X 2 Y 2 Z2
( X ,Y , Z )
• Convert to spherical coordinates
(sin T cos I sin I cosT cos I xˆ , yˆ , zˆ )
• Convert to spherical image coordinates
I
unwrapped sphere
spherical image
Review Topics
•
•
•
•
•
•
•
•
Sampling and Reconstruction
Frequency Domain and Filtering
Blending
Warping
Data-driven Methods
Camera
Homographies
Modeling Light
Modeling Light
(The Omnipotent) Plenoptic Function
•
•
Intensity of Light:
•
•
•
•
From all directions: θ, φ
At all wavelengths: λ
At all times: t
Seen from any viewpoint:Vx,Vy,Vz
P(θ, φ, λ, t,Vx,Vy,Vz)
Modeling Light
Lumigraph (Lightfield)
•
Intensity along all lines
•
•
For all views (i.e. s,t), gives intensity at all points (i.e. u,v)
Captures to some extent P(θ, φ,Vx,Vy,Vz)
(s,t)
(u,v)
For all (s,t,u,v)
Modeling Light
Lumigraph / Lightfield
Lumigraph (Lightfield)
Lumigraph / Lightfield
Modeling Light
Acquiring Lightfield
•
•
•
Move camera in known steps over (s,t) using gantry
Move camera anywhere over (s,t) and recover optimal field
Use microlens array after main lens
Good Luck!!