Single and Two view summary
Calibrated case

Recover Essential Matrix decompose to R,T




3D reconstruction
Planar case
Recover homography H decompose to R,T,n,d
Uncalibrated case

Recover Fundamental Matrix F



Projective reconstruction
Recover planar homography
(no decomposition possible)
Rotational homography (mosaics)
CS 223b J. Kosecka


Calibration with planar rig
Calibration with 3D rig
Single view

Homography between 3D plane and image plane (rectification)


Partial calibration using vanishing points
Partial calibration and pose recovery using from single view/world homography
2

Problem formulation
Input: Corresponding images (of “features”) in multiple images.
Output: Camera motion , camera calibration , object structure .
CS 223b J. Kosecka 3


Full perspective projection model

Affine camera projection model (in-homogenous coordinates)


Good approximation if the distance from the scene >> scene depth variation
Rigid body motion under affine projection model
CS 223b J. Kosecka 4


Problem – given n correspondences in m views determine and 3D points

Obtained by minimization of the following objective function

Assume that the centroid of the structure is the origin of the coordinate frame and denote the centroids in the following way

Minimize with respect to
CS 223b J. Kosecka 5

The choice of the frame is arbitrary – further assume that
The objective function then becomes
Writing all the constrains in the matrix form
Drop ~ for clarity
CS 223b J. Kosecka 6

Measurement matrix Motion matrix Structure matrix
• Matrix W must have rank 3 (product of two rank 3 matrices)
• In noise case we seek best rank 3 approximation of W matrix
• Given the actual measurement matrix compute SVD
• Best rank 3 approximation is
CS 223b J. Kosecka 7

• Decomposition of is not unique
• Affine ambiguity Q
• The ambiguity can be resolved using rotation matrix constraints where
[Tomasi, Kanade’IJCV 1992] Factorization approach
CS 223b J. Kosecka 8

Measurement matrix Motion matrix Structure matrix
• Similar strategy – expect now the scales are unknown
• We don’t have the matrix W
• Matrix W is a product of two rank 4 matrices – i.e. is rank 4
• How to apply the factorization idea to projective setting ?
- need to compute scales (possible from two view reconstruction)
- or initialize the scales and iterate
CS 223b J. Kosecka 9


Given a set of image points in n views

Compute the projective depths using two view methods or set



Form the measurement matrix W and find the nearest rank
4 approximation using SVD
Decompose into camera matrices and structure
Optionally reproject the and iterate
(i.e. given new motions, we can compute new scales)
CS 223b J. Kosecka 10


Advantages of multiview methods
- more frames – wider baseline – better conditioned algorithms
- additional complexity of matching (establishing correspondences) across multiple views



How are multi-view and two view methods related ?
Can we just run the two view algorithm for each pair of views and get better results ?
What if we are trying to do reconstruction using lines ?


Are there other constraints between multiple views then pairwise epipolar constraints ?
Next – brief tour of multilinear constraints
CS 223b J. Kosecka 11

Traditional multifocal constraints
For images of the same 3-D point :
CS 223b J. Kosecka
(leading to the conventional approach)
Multilinear constraints among 2, 3, 4-wise views
12

Rank conditions for point feature
WLOG, choose camera frame 1 as the reference
Multiple-View Matrix
Lemma [Rank Condition for Point Features]
Let
CS 223b J. Kosecka then and are linearly dependent .
13

Rank conditions vs. multifocal constraints
CS 223b J. Kosecka 14

Rank conditions vs. multi-focal constraints
CS 223b J. Kosecka 15

Rank conditions vs. multifocal constraints
• These constraints are only necessary but NOT sufficient!
• However, there is NO further relationship among quadruple wise views. Quadrilinear constraints hence are redundant!
CS 223b J. Kosecka 16

Point Features – Uniqueness of the pre-image bilinear constraints – coplanarity constraints p p ?
o i o j
Extend to three views o i o i o j o k collinear optical centers
CS 223b J. Kosecka o k o j coplanar optical centers
17

Point Features – Uniqueness of the pre-image trilinear constraints o k o i o j
Given m vectors with respect to m camera frames,
They correspond to a unique point in the 3D space if the rank of the
Matrix M p is 1. If rank is 0, the point is determined up to a line on which all optical centers must lie .
CS 223b J. Kosecka 18

Image of a line feature
Homogeneous representation of a 3-D line
Homogeneous representation of its 2-D co-image
Projection of a 3-D line to an image plane
CS 223b J. Kosecka 19

Multiple-view matrix: line vs. point
Point Features Line Features
CS 223b J. Kosecka 20

Rank conditions: line vs. point
CS 223b J. Kosecka 21

Multiple-view structure and motion recovery
Given images of points:
CS 223b J. Kosecka 22

SVD based 4-step algorithm for SFM
CS 223b J. Kosecka 23

Utilizing all incidence relations
Three edges intersect at each vertex.
CS 223b J. Kosecka
. . .
24

Example: simulations
CS 223b J. Kosecka 25

Example: simulations
CS 223b J. Kosecka 26

Example: experiments
CS 223b J. Kosecka
Errors in all right angles < 1 o
27

Summary
• Incidence relations <=> rank conditions
• Rank conditions => multiple-view factorization
• Rank conditions implies all multi-focal constraints
• Rank conditions for points, lines, planes, and
(symmetric) structures.
CS 223b J. Kosecka 28

Global multiple-view analysis: examples
CS 223b J. Kosecka 29

A family of intersecting lines
CS 223b J. Kosecka each can randomly take the image of any of the lines:
Nonlinear constraints among up to four views
.
.
.
30

Universal rank condition
Theorem [The Universal Rank Condition] for images of a point on a line:
CS 223b J. Kosecka
-Multinonlinear constraints among 3, 4-wise images.
-Multilinear constraints among 2, 3-wise images.
31

Instances with mixed features
Examples:
Case 1: a line reference Case 2: a point reference
• All previously known constraints are the theorem’s instances.
• Degenerate configurations if and only if a drop of rank.
CS 223b J. Kosecka 32

Generalization – restriction to a plane
Homogeneous representation of a 3-D plane
Corollary [Coplanar Features]
Rank conditions on the new extended remain exactly the same!
CS 223b J. Kosecka 33

Generalization – restriction to a plane
Given that a point and line features lie on a plane in 3-D space:
GENERALIZATION – Multiple View Matrix: Coplanar Features
In addition to previous constraints, it simultaneously gives homography:
CS 223b J. Kosecka 34