Stanford CS223B Computer Vision, Winter 2007
Stereo
Stereo
Lecture 6
Advanced Stereo
Professors Sebastian Thrun and Jana Košecká
CAs: Vaibhav Vaish and David Stavens
Summary Stereo


Epipolar Geometry
Fundamental/Essential Matrix
p Epl  0
T
r
P
Pl
Pr
Epipolar Plane
Epipolar Lines
p
Ol
p
l
el
er
r
Or
Epipoles
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Correspondence: Where to search?
Search window?
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Stereo Vision 2: Outline





Image Rectification
Correspondence
Active Stereo
Dense and Layered Stereo
Smoothing With Markov Random Fields
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Rectification

Problem: Epipolar lines not parallel to scan lines
P
Pl
Pr
Epipolar Plane
Epipolar Lines
p
Ol
p
l
el
er
r
Or
Epipoles
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Rectification

Problem: Epipolar lines not parallel to scan lines
P
Pl
Pr
Rectified Images
Epipolar Plane
Epipolar Lines
p
p
l
Ol
r
Or
Epipoles at ininity
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Epipolar Rectified Stereo Images
Epipolar line
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Epipolar Rectified Images
Sebastian Thrun and Jana Košecká
Source: A. Fusiello, Verona, 2000]
CS223B Computer Vision, Winter 2007
Example Rectification
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Final Step: Image Normalization



Even when the cameras are identical models, there can
be differences in gain and sensitivity.
The cameras do not see exactly the same surfaces, so
their overall light levels can differ.
For these reasons and more, it is a good idea to
normalize the pixels in each window:
I
I
1
Wm ( x , y )
Wm ( x , y )

 I (u, v)
Average pixel
( u ,v )Wm ( x , y )
2
[
I
(
u
,
v
)]

Window magnitude
( u ,v )Wm ( x , y )
ˆI ( x, y )  I ( x, y )  I
I  I W ( x, y )
Normalized pixel
m
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Stereo Vision 2: Outline





Image Rectification
Correspondence
Active Stereo
Dense and Layered Stereo
Smoothing With Markov Random Fields
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Correspondence
x
pl .1
O1
y
z
P1
P1
x
Phantom points
f
y
O2
z
pr ,1
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Correspondence via Correlation
Left
Right
scanline
SSD error
Rectified images
disparity
(Same as max-correlation / max-cosine for normalized image patch)
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Images as Vectors
Left
Right
wR
wL
Each window is a vector
in an m2 dimensional
vector space.
Normalization makes
them unit length.
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Image Metrics
(Normalized) Sum of Squared Differences
wR (d )
wL
CSSD (d ) 
 [ Iˆ (u, v)  Iˆ
2
(
u

d
,
v
)]
R
L
( u ,v )Wm ( x , y )
 wL  wR (d )
2
Normalized Correlation
CNC (d ) 
 Iˆ (u, v) Iˆ
L
( u ,v )Wm ( x , y )
R
(u  d , v)
 wL  wR (d )  cos
d  arg min d wL  wR (d )  arg max d wL  wR (d )
*
Sebastian Thrun and Jana Košecká
2
CS223B Computer Vision, Winter 2007
Correspondence Using Correlation
Left
Disparity Map
Images courtesy of Point Grey Research
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Correspondence By Features
LEFT IMAGE
corner
line
structure
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Correspondence By Features
RIGHT IMAGE
corner
line
structure

Search in the right image… the disparity (dx, dy) is the
displacement when the similarity measure is maximum
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Stereo Correspondences
Left scan line
…
Sebastian Thrun and Jana Košecká
Right scan line
…
CS223B Computer Vision, Winter 2007
Stereo Correspondences
Left scanline
Right scanline
…
…
Match
Match
Occlusion
Match
Sebastian Thrun and Jana Košecká
Disocclusion
CS223B Computer Vision, Winter 2007
Search Over Correspondences
Occluded Pixels
Left scanline
Right scanline
Disoccluded Pixels
Three cases:
– Sequential – cost of match
– Occluded – cost of no match
– Disoccluded – cost of no match
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Stereo Matching with Dynamic Programming
Occluded Pixels
Left scanline
Right scanline
Dis-occluded Pixels
Scan across grid
computing optimal cost
for each node given its
upper-left neighbors.
Backtrack from the
terminal to get the
optimal path.
Terminal
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Stereo Matching with Dynamic Programming
Occluded Pixels
Left scanline
DP Algorithm:
Right scanline
Dis-occluded Pixels
V[0,0] = 0
V[i,k] = min { V[i-1,k-1] + m(i,j),
c+V[i, k-1],
c+V[i-1,k] }
d[i,k] = argmin { … }
Terminal
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Stereo Matching with Dynamic Programming
Occluded Pixels
Left scanline
DP Algorithm:
Right scanline
Dis-occluded Pixels
V[0,0] = 0
V[i,k] = min { V[i-1,k-1] + m(i,j),
c+V[i, k-1],
c+V[i-1,k] }
d[i,k] = argmin { … }
Terminal
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Stereo Matching with Dynamic Programming
Occluded Pixels
Left scanline
Find Stereo Alignment
Right scanline
Dis-occluded Pixels
D=[X,Y]
repeat until D=[1,1]
add D to alignment
D = d[D]
end
Terminal
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Commercial-Grade Stereo

Tyzx, a leading stereo camera manufacturer

(here strapped on our DARPA Grand Challenge vehicle)
Disparity map
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Dense Stereo Matching: Examples

input
View extrapolation results
depth image
novel view
[Matthies,Szeliski,Kanade’88]
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Dense Stereo Matching

Some other view extrapolation results
input
depth image
Sebastian Thrun and Jana Košecká
novel view
CS223B Computer Vision, Winter 2007
Dense Stereo Matching

Compute certainty map from correlations
input
depth map
Sebastian Thrun and Jana Košecká
certainty map
CS223B Computer Vision, Winter 2007
DP for Correspondence


Does this always work?
When would it fail?
–
–
–
–
Failure Example 1
Failure Example 2
Failure Example 3
Failure Example 4
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Correspondence Problem 1

Ambiguities
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Correspondence Problem 2

Multiple occluding objects
Figure from
Forsyth & Ponce
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Correspondence Problem 3

Correspondence fail for smooth surfaces (edge =
occlusion boundary, poorly localized)

There is currently no good solution to this
correspondence problem
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Correspondence Problem 4



Regions without texture
Highly Specular surfaces
Translucent objects
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Some Stereo Results
Top view
Side view
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
More Stereo Results
Side view
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
A True Challenge!
http://www.well.com/user/jimg/stereo/stereo_list.html
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Stereo Vision 2: Outline





Image Rectification
Correspondence
Active Stereo
Dense and Layered Stereo
Smoothing With Markov Random Fields
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
How can We Improve Stereo?
Space-time stereo scanner
uses unstructured light to aid
in correspondence
Sebastian Thrun and Jana Košecká
Result: Dense 3D mesh (noisy)
CS223B Computer Vision, Winter 2007
Active Stereo: Adding Texture to Scene
By James Davis,
Honda Research,
Now UCSC
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
rectified
Active Stereo (Structured Light)
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Structured Light: 3-D Result
3D Snapshot
By James Davis,
Honda Research
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Time of Flight Sensor: Shutter
http://www.3dvsystems.com
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Time of Flight Sensor: Shutter
http://www.3dvsystems.com
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Time of Flight Sensor: Shutter
http://www.3dvsystems.com
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Time of Flight Sensor: Shutter
http://www.3dvsystems.com
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Time of Flight Sensor: Shutter
http://www.3dvsystems.com
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Scanning Laser Range Finders
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Scanning Laser Results
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Scanning Laser Results
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Stereo Vision 2: Outline





Image Rectification
Correspondence
Active Stereo
Dense and Layered Stereo
Smoothing With Markov Random Fields
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Layered Stereo

Assign pixel to different “layers” (objects, sprites)
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Layered Stereo

Track each layer from frame to frame,
compute plane eqn. and composite mosaic

Re-compute pixel assignment by comparing
original images to sprites
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Layered Stereo

Re-synthesize original or novel images from
collection of sprites
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Layered Stereo

Advantages:
– can represent occluded regions
– can represent transparent and border (mixed) pixels
(sprites have alpha value per pixel)
– works on texture-less interior regions

Limitations:
– fails for high depth-complexity scenes
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Fitting Planar Surfaces (with EM)
*
Sebastian Thrun and Jana Košecká
*
CS223B Computer Vision, Winter 2007
Expectation Maximization

3D Model:
  {1 , 2 ,, J }
Planar surface in 3D
 j   j ,  j  3  
y
surface
normal a
surface
Distance point-surface
z
displacement b
dist( j , zi )   j  zi   j
x
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Mixture Measurement Model

Case 1: Measurement zi caused by plane j
1
p ( zi |  j ) 
2 2
e
1 ( j  zi   j )

2
2
2
§ Case 2: Measurement zi caused by something else
p ( zi |  * ) 
1
z max
Sebastian Thrun and Jana Košecká

1
2 2
e
2
1 z max
 ln
2 2 2
CS223B Computer Vision, Winter 2007
Measurement Model with Correspondences
}
p( zi |  , c1 ,, cJ , c* ) 
( j  zi   j )
z max 2 J
1 
 c* ln

c
j
2
2 2 j 1
2


1
2 2
e
2




correspondence variables C:
c* , c j  {0,1}
J
c*   c j  1
j 1
 p( Z |  , C )  
i 1

1
2
Sebastian Thrun and Jana Košecká
( j  zi   j )
z max 2 J
1 
 ci* ln

c
ij
2
2
2

2
j

1

2
e
2




CS223B Computer Vision, Winter 2007
Expected Log-Likelihood Function
p( Z |  , C )  
i 1
…after some simple math
Ec ln p( Z , C |  )

1
2

( j  zi   j )
z max 2 J
1 
 ci* ln

c
ij
2
2
2

2
j

1

2
e




1


ln


2
 ( J  1) 2



2
  1 E[c ] ln zmax

i  2 i* 2 2


2 
J
(


z


)
j
  1 E[c ] j i


ij
2
 2 j 1




probabilistic
data association
Sebastian Thrun and Jana Košecká
2
mapping with
known data association
CS223B Computer Vision, Winter 2007
The EM Algorithm
Ec ln p(Z , C |  )
J
 const   E[cij ]
i
j 1
( j  zi   j ) 2
2

E-step: given plane params, compute E[cij ]

M-step: given expectations, compute {a j ,  j }
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Choosing the “Right” Number of Planes: AIC
J=0
J=1
J=2
J=3
J=4
J=5
increased data likelihood
increased prior probability
log p( J | d )  const  log p(d | J )  log p( J )
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Determining Number of Surfaces
Add
Firstmodel
model
Prune
E/M
M-step
E-Step
Steps
components
model
component
*
*
Sebastian Thrun and Jana Košecká
J =2
=1
=3
*
CS223B Computer Vision, Winter 2007
Layered Stereo

Resulting sprite collection
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Layered Stereo

Estimated depth map
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Example (here with laser range finder)
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Example (here with laser range finder)

Another Example
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Stereo Vision 2: Outline





Image Rectification
Correspondence
Active Stereo
Dense and Layered Stereo
Smoothing With Markov Random Fields
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Motivation and Goals
James Diebel
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Motivation and Goals
James Diebel
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Network of Constraints (Markov Random Field)
Directions
Vertex Node
Edge Node
Face Node
James Diebel
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
MRF Approach to Smoothing

Potential function: contains a sensor-model term
and a surface prior
   xi  x0i  i xi  x0i    j 1  n1  n2 
T
i


j
The edge potential is important!
Minimize  by conjugate gradient
– Optimize systems with tens of thousands of
parameters in just a couple seconds
– Time to converge is O(N), between 0.7 sec (25,000
nodes in the MRF) and 25 sec (900,000 nodes)
Diebel/Thrun, 2006
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Possible Edge Potential Functions
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Results: Smoothing
James Diebel
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Results: Smoothing
James Diebel
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Results: Smoothing
James Diebel
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Results: Smoothing
James Diebel
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Movies…
Movies in Windows Media Player
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007
Summary





Image Rectification
Correspondence
Active Stereo
Dense and Layered Stereo
Smoothing With Markov Random Fields
Sebastian Thrun and Jana Košecká
CS223B Computer Vision, Winter 2007