Correspondence and
Stereopsis
Introduction
•
•
Disparity
–
Informally: difference between two pictures
–
Allows us to gain a strong sense of depth
Stereopsis
–
•
Ability to perceive depth from disparity
Goal of this chapter
–
Design algorithms that mimic stereopsis
Applications of Stereopsis
•
•
•
•
•
Visual robot navigation
Cartography
Aerial reconnaissance
Close-range photogrammetry
Image segmentation for object recognition
Stereo Vision
•
Two processes
–
Binocular fusion of features observed by the eyes
–
Reconstruction of their three-dimensional preimage
Stereo Vision – Easy Case
•
1 single point being observed
–
The preimage can be found at the intersection of
the rays from the focal points to the image points
Stereo Vision – Hard Case
•
Many points being observed
–
Need some method to establish correct
correspondences
Components of Stereo Vision Systems
•
•
•
•
Camera calibration: previous lectures
Image rectification: simplifies the search for
correspondences
Correspondence: which item in the left image
corresponds to which item in the right image
Reconstruction: recovers 3-D information from
the 2-D correspondences
Epipolar Geometry
•
Epipolar constraint: corresponding points must
lie on conjugated epipolar lines
–
Search for correspondences becomes a 1-D problem
Image Rectification
•
Corresponding epipolar lines become collinear
Image Rectification (cont.)
•
•
Not equivalent to
rotation
The lines through the
centers become
parallel to each other,
and the epipoles move
to infinity
Image Rectification (cont.)
•
Given extrinsic parameters T and R (relative
position and orientation of the two cameras)
–
Rotate the left camera about the projection center so
that the the epipolar lines become parallel to the
horizontal axis
–
Apply the same rotation to the right camera
–
Rotate the right camera by R
–
Adjust the scale in both camera reference frames
Image Rectification (cont.)
•
Formal definition of disparity: d=u'–u
Correspondence
•
•
Given an element in the left image, find the
corresponding element in the right image
Classes of methods
–
Correlation-based
–
Feature-based
Correlation-Based Correspondence
•
•
Input: rectified stereo pair and a point (u,v) in the
first image
Method:
–
Associate a window of size p=(2m+1)(2n+1) centered
in (u,v) and form the vector w(u,v) in Rp
–
For each potential match (u+d,v) in the second image,
compute w' and the normalized correlation between w
and w'
Correlation-Based Correspondence
(cont.)
•
•
Main problem:
–
Implicitly assume that the observed surface is locally
parallel to the two image planes
–
Alleviated by computing an initial disparity and using
it to warp the correlation windows to compensate for
unequal amounts of foreshortening
Other problems:
–
Not robust against noise
–
Similar pixels may not correspond to physical
features
Feature-Based Correspondence
•
•
•
•
Main idea: physically-significant features should
be preferred to matches between raw pixel
intensities
Instead of correlation-like measures, use a
measure of the distance between feature
descriptors
Typical features: points, lines, and corners
Example: Marr-Poggio-Grimson algorithm
Marr-Poggio-Grimson Algorithm
•
•
•
Convolve images with Laplacian of Gaussian
filters with standard deviations s1<s2<s3<s4
Find zero crossings of the Laplacian along
horizontal scanlines of the filtered images
For each s, match zero crossings with same
parity and similar orientations in a [–ws,ws]
disparity range, with
ws  2 2s
Marr-Poggio-Grimson Algorithm
(cont.)
•
Use disparities found at larger scales to control
eye vergence and cause unmatched regions at
smaller scales to come into correspondence
Marr-Poggio-Grimson algorithm
(cont.)
Marr-Poggio-Grimson algorithm
(cont.)
Ordering Constraint
•
The order of matching image features along a pair
of epipolar lines is (usually) the inverse of the
order of the corresponding surface attributes
along the curve where the epipolar plane
intersects the object's boundary
Ordering Constraint (cont.)
•
•
May not be satisfied in real scenes due to
occlusion
Still useful to devise efficient algorithms relying
on dynamic programming to establish stereo
correspondences
Reconstruction
•
•
•
Given pair of image points p and p', and focal
points O and O', find preimage P
In theory: find P by intersecting the rays R=Op
and R'=Op'
In practice: R and R' won't actually intersect due
to calibration and feature localization errors
Reconstruction Approaches
•
Geometric
–
Construct the line segment perpendicular to R and R'
that intersects both rays and take its mid-point
Reconstruction Approaches (cont.)
•
Algebraic (linear)
–
Write down the projection equations
–
The resulting linear system is overconstrained
–
Solve it by linear least-squares
Reconstruction Approaches (cont.)
•
Algebraic (non-linear)
–
Find the point Q that minimizes d2(p,q)+d2(p',q') by
non-linear least-squares
–
Reconstructions obtained by the previous methods
can be used as initial guesses for the optimization