CSE473/573 – Stereo and Multiple View
Geometry
Presented by
Radhakrishna Dasari
Contents
•
Stereo Practical Demo
•
Camera Intrinsic and Extrinsic parameters
•
Essential and Fundamental Matrix
•
Multiple View Geometry
•
Multi-View Applications
Stereo Vision Basics
•
Stereo Correspondence – Epipolar
constraint
•
Rectification
•
Pixel matching
•
Depth from Disparity
Epipolar
C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern
Recognition, 1999.
Stereo Rectification
•
Rectification is the process of transforming stereo images, such that the
corresponding points have the same row coordinates in the two images.
•
It is a useful procedure in stereo vision, as the 2-D stereo correspondence problem is
reduced to a 1-D problem
•
Let’s see the rectification pipeline when we have are two images of the same scene
taken from a camera from different viewpoints
Stereo Input Images
Superposing the two input images on each other and compositing
Matching Feature Points
Eliminating outliers using RANSAC
We can impose geometric constraints while applying RANSAC for eliminating outliers
Estimate Fundamental Matrix using Matched Points
fMatrix = estimateFundamentalMatrix( matchedPtsOut.Location,
matchedPtsIn.Location);
Rectified Input Stereo Images
Depth From Disparity
Rectified Stereo Images as Input
Disparity map using Block Matching
•
There are noisy patches and bad depth estimates, especially on the ceiling.
•
These are caused when no strong image features appear inside of the pixel windows
being compared.
•
The matching process is subject to noise since each pixel chooses its disparity
independently of all the other pixels.
Disparity map using Dynamic Programming –
Simple Example
•
For optimal path we use the underlying block matching metric as the cost function
•
constrain the disparities to only change by a certain amount between adjacent pixels
(Smoothness of disparity) Lets say +/- 3 values of the neighbors
•
We assign a penalty for disparity disagreement between neighbors.
•
Hence most of the noisy blocks will be eliminated. Good matches will be preserved as
block-matching cost function will dominate the penalty assigned for disparity
disagreement
Depth from Disparity and Back-Projection
•
With a stereo depth map and knowledge of the intrinsic parameters (focal length,
image center) of the camera, it is possible to back-project image pixels into 3D points
•
Intrinsic Parameters of a camera are obtained using camera calibration techniques
Camera Intrinsic Parameters
•
Camera Calibration Matrix ‘K’ – 3x3 Upper triangular Matrix
•
Constitutes – Focal length of the camera ‘f’ , Principal Point (u0,v0), aspect ratio
of the pixel ‘γ’ and the skew ‘s’ of the sensor pixel
•
Intrinsic parameters can be estimated using camera calibration techniques
Ideal image sensor
Sensor pixel with skew
Camera Calibration with grid templates
Extrinsic parameters (world-centered)
400
12
18
8 13
310 19
9
2 14
11
5
7
21
20
22
350
300
15
14
6
200
Z
world
250
16
150
17
100
50
0
400
200
0
300
Xw orld
200
100
0
Yw orld
Camera Calibration Toolbox on Matlab
-100
Intrinsic & Extrinsic Parameters
•
The transformation of point ‘pw’ from world is related to the point on image plane ‘x’
through the Projection Matrix ‘P’ which constitutes intrinsic and extrinsic parameters
•
Camera matrix – both intrinsic ‘K’ (focal length, principal point) and extrinsic
parameters (Pose – ‘R’ rotation matrix and ‘t’ translation)
•
Projection Matrix or Camera Matrix ‘P’ is of dimension ‘3x4’
Projection Matrix ‘P’
Special case of perspective projection – Orthographic Projection
Image
Also called “parallel projection”: (x, y, z) → (x, y)
What’s the projection matrix?
World
Projection Matrix ‘P’
In general, for a perspective projection Matrix ‘P’ maps image point ‘x’ into world coordinates ‘X’ as
The Projection Matrix (3x4) can be decomposed into
(3x4)
(3x3)
(3x4)
(4x4)
(4x4)
Pure Rotational Model of Camera - Homography
α,β,γ are angle changes across roll, pitch
and yaw
Homography
Suppose we have two images of a scene captured from a rotating camera
point ‘x1 ’ in Image1 is related to the world point ‘X’ by the equation
x1 = KR1X
which implies
X = R1-1K-1 x1
as
point ‘x2 ’ in Image2 is related to the world point ‘X’ by the equation
x2 = KR2X
= KR2R1-1K-1 * x1
Hence the points in both the images are related to each other by a
transformation of Homography ‘H’
x2 = H x 1
Where
H = KR2R1-1K-1
Rotation of Camera along Pitch, Roll and Yaw
If the camera is only rotating along these axes and there is zero translation, the captured
images can be aligned with each other using Homography estimation
The Homography Matrix ‘H’ (3x3)can be estimated by matching features between two
images
Image Alignment Result - Rotation of Camera
along Pitch Axis
Image Alignment Result- Rotation of Camera along
Roll axis
Image Alignment Result- Rotation of Camera along
Yaw axis
the fundamental matrix is a 3×3 matrix which relates corresponding points in stereo images.
Fundamental and Essential Matrices
Stereo Images have both rotation and translation of camera
the fundamental matrix ‘F’ is a 3×3 matrix which relates corresponding points x and x1 in
stereo images.
It captures the essence of Epipolar constraint in the Stereo images.
Essential Matrix
Where K and K1 are the Intrinsic parameters of the cameras capturing x and x1
respectively
http://en.wikipedia.org/wiki/Eight-point_algorithm
Stereo – Fundamental and Essential Matrices
https://www.youtube.com/watch?v=DgGV3l82NTk
the fundamental matrix is a 3×3 matrix which relates corresponding points in stereo images.
Beyond Two-View Stereo
Third View can be used for verification
Multi-View Video in Dynamic Scenes
Reference link
the fundamental matrix is a 3×3 matrix which relates corresponding points in stereo images.
Multiple-View Geometry
Generic problem formulation: given several images of the same object or scene,
compute a representation of its 3D shape
the fundamental matrix is a 3×3 matrix which relates corresponding points in stereo images.
Multiple-baseline Stereo
Pick a reference image, and slide the corresponding window along the
corresponding epipolar lines of all other images using other images
Remember? disparity
Where B is baseline, f is focal length
and Z is the depth
This equation indicates that for the
same depth the disparity is
proportional to the baseline
M. Okutomi and T. Kanade, “A Multiple-Baseline Stereo System,” IEEE Trans. on
Pattern Analysis and Machine Intelligence, 15(4):353-363 (1993)
the fundamental matrix is a 3×3 matrix which relates corresponding points in stereo images.
Feature Matching to Dense Stereo
1. Extract features
2. Get a sparse set of initial matches
3. Iteratively expand matches to nearby locations Iteratively expand matches to
nearby locations
4. Use visibility constraints to filter out false matches
5. Perform surface reconstruction
the fundamental matrix is a 3×3 matrix which relates corresponding points in stereo images.
View Synthesis
Is it possible to synthesize views from the locations where the cameras are
removed? i.e Can we synthesize view from a virtual camera
the fundamental matrix is a 3×3 matrix which relates corresponding points in stereo images.
View Synthesis - Basics
Problem: Synthesize virtual view of the scene at the mid point of line joining Stereo
camera centers.
Given stereo images, find Stereo correspondence and disparity estimates between
them.
View Synthesis - Basics
Use one of the images and its disparity map to render a view at virtual camera
location. By shifting pixels with half the disparity value
View Synthesis - Basics
Use the information from other image to fill in the holes, by shifting the pixels by
half the disparity
View Synthesis - Basics
Putting both together, we have the intermediary view. We still have holes. Why??
View Synthesis – Problem of Holes
View Synthesis – Problem of Color Variation at
boundaries
Slide Credits
Rob Fergus, S Seitz, Lazebnik
MATLAB Computer Vision Toolbox