3D reconstruction
Three questions:
(i)
Correspondence geometry: Given an image point x in
the first image, how does this constrain the position of the
corresponding point x’ in the second image?
(ii) Camera geometry (motion): Given a set of corresponding
image points {xi ↔x’i}, i=1,…,n, what are the cameras P and
P’ for the two views?
(iii) Scene geometry (structure): Given corresponding image
points xi ↔x’i and cameras P, P’, what is the position of
(their pre-image) X in space?
Epipolar geometry
Underlying structure
in set of matches for
rigid scenes
P
L2
C1
m1
lT1 l2
M
L1
l1
e1
m T2 F m1  0
e2
m2
Fundamental matrix
(3x3 rank 2 matrix)
l2
C2
1. Computable from corresponding
points
Canonical representation:
P  [I | 0] P'  [[e' ] F  e' vT | λe' ] 2. Simplifies matching
3. Allows to detect wrong matches
4. Related to calibration
3D reconstruction of cameras and structure
reconstruction problem:
given xi↔x‘i , compute P,P‘ and Xi
x i  PX i
xi  P' X i
for all i
without additional informastion possible
up to projective ambiguity
outline of reconstruction
(i)
(ii)
(iii)
Compute F from correspondences
Compute camera matrices from F
Compute 3D point for each pair of corresponding points
computation of F
use x‘iFxi=0 equations, linear in coeff. F
8 points (linear), 7 points (non-linear), 8+ (least-squares)
(more on this next class)
computation of camera matrices
use
P  [I | 0] P'  [[e' ] F  e' v | λe' ]
triangulation
compute intersection of two backprojected rays
T
Reconstruction Ambiguity
•Without knowledge of a scene placement w.r.t. 3D coordinate frame,
impossible to reconstruct absolute position or orientation of a scene from a
pair of views:
•Scene is determined at best up to a Euclidean transformation , i.e.
rotation and translation w.r.t. world frame
•Overall scale of the scene can also not be determined:
•Example: Coin next to the integrated circuit to give an idea of size
•Scene is determined by the image only up to a similarity transformation,
i.e. rotation, translation and scaling
Reconstruction ambiguity: similarity
Let Hs be a similarity transform
Replace: Xi  Hs Xi
P  P Hs -1
P’  P’ Hs -1
Then xi unchanged;

x i  PX i  PH
-1
S
H X 
S
i
(a)
Let P = K [ Rp | tp ]
for some t’  Multiplying by Hs -1 does not change the intrinsic calibration
matrix K
Upshot: For calibrated cameras, reconstruction possible up to a similarity
This is the only ambiguity of reconstruction ( Longuet and Higgins 1981)
Reconstruction ambiguity: projective
•Reconstruction from uncalibrated
cameras possible up to a projective
transformation.
•Consider H any 4x4 invertible matrix
representing projective transformation
in P3
•Replace Xi  H Xi
P  P H -1
P’  P’ H -1
•Image point xi does not change

x i  PX i  PH
-1
P
H
P
Xi 
Terminology
xi↔x‘i
Original scene Xi
Projective, affine, similarity reconstruction
= reconstruction that is identical to original up to
projective, affine, similarity transformation
Literature: Metric and Euclidean reconstruction
= similarity reconstruction
The projective reconstruction theorem
If a set of point correspondences in two views determine the
fundamental matrix uniquely, then the scene and cameras
may be reconstructed from these correspondences alone,
and any two such reconstructions from these
correspondences are projectively equivalent
key result:
allows reconstruction from pair of uncalibrated images
• points lying on the line joining two camera centers are excluded, since they
cannot be reconstructed uniquely even if camera matrices are determined
•Projective reconstruction answers questions like: At what point does a line
intersect a plane? What is the mapping between two views induced by
planes?
A
B
Relation between Euclidean
and Projective Reconstruction
Stratified reconstruction
(i) Projective reconstruction
(ii) Affine reconstruction
(iii) Metric reconstruction
Projective to affine
remember 2-D case
Projective to affine
•The essence of affine reconstruction is to locate the plane at infinity;
•Suppose we have somehow identified the plane at infinity.
P, P' , X i 
T
T
π    A, B, C , D   0,0,0,1
T
H -T π   0,0,0,1
I | 0

H 
 π 
•
•
•
•
Projective reconstruction of the scene;
(0,0,0,1) is the true coordinates of plane at infinity
find transformation H that maps Π∞ to (0,0,01)
(if D≠0)
Now apply H to all points and to two cameras. Plane at infinity has been correctly placed.
This reconstruction differs from true reconstruction up to projective transformation that fixes the
plane at infinity.
But projective with fixed p∞ is affine transformation
Reconstruction is true up to an affine transformation
•
•
•
Affine reconstruction can be sufficient depending on application,
e.g. mid-point, centroid can be computed
Parallellism: lines constructed parallel to other lines and to planes
Question: how to identify plane at infinity need extra information
Translational motion to find plane at infinity
points at infinity are fixed for a pure translation
 reconstruction of xi↔ xi is on p∞
 Get three points on plane at infinity to reconstruct it.
Example: far away objects such as
moon as we translate
Scene constraints to find plane at infinity
•
•
•
Parallel lines
parallel lines intersect at infinity
reconstruction of corresponding vanishing point yields point on plane at
infinity
3 sets of parallel lines (with different directions) allow to uniquely
determine p∞
remark: in presence of noise determining the intersection of
parallel lines is a delicate problem
Scene constraints
Scene constraints
Distance ratios on a line to find plane at infinity
known distance ratio along a line allow to determine point at infinity
(same as 2D case)
Given two intervals on a line with a known length ratio, the point at
infinity on the line can be found 
from an image of a line on which a world distance ratio is known, for
example that three points are equally spaced, the vanishing point
may be determined.
The infinite homography
•A mapping that transfers points from the P image to the P’ image via
the plane at infinity
~ T
X

X
,0
•Ray corresponding to a point x meets Π∞ at X which can be

projected to point x’ in the other image
∞
 
∞
~
x   MX
~
x'  M' X
•Infinite homography can be computed from an affine reconstruction and Vice versa.
Infinite homography can be computed directly from corresponding image entries
e.g. three vanishing points and F, or vanishing line, vanishing point and F
Affine to metric
•Identify absolute conic; it lies on the plane at infinity
•Transform so that the absolute conic is mapped to the absolute conic in
Euclidean frame
  : X 2  Y 2  Z 2  0, on π 
• Projective transformation relating original and reconstruction is a similarity
transformation
*
•in practice, find image of ∞
•image w∞ back-projects to cone that intersects
p∞ in ∞
w*
Affine to metric
ω Image of the absolute conic as seen by a camera with matrix:
P  [M | m]
possible transformation from affine to metric is
-1

A
0
H
 0 1

AA  M ωM
T
T

1
(cholesky factorisation)
How to find the image of an absolute conic? constraints
Constraints arising from scene Ortogonality to
find image of absolute conic
vanishing points corresponding to orthogonal directions
v1T ωv 2  0
vanishing line and vanishing point corresponding
to plane and normal direction
l  ωv
Constraints arising from known
internal parameters to find image of
absolute conic
ω  K -T K -1
rectangular pixels
s0
ω12  ω 21  0
square pixels
x  y
ω11  ω 22
Constraints arising from Same camera for all
images to find image of absolute conic
•same intrinsics  same image of the absolute conic e.g. moving cameras
•Property of absolute conic: its projection onto an image depends only on the
calibration matrix of the camera, not on the position or orientation of camera
•If P and P‘ have the same calibration matrix, i.e. both images taken with the
same camera at different poses, then ω = ω’
•Since the absolute conic is on the plane at infinity, its image may be
transfered from one view to the other via infinite homogrpahy:
transfer of IAC:
ω'  H  ωH 
•Combine ω = ω’ with the above
-T
-1
ω  H  ωH 
-T
-1
•  four contraints on ω ;
•ω has five dof; use some of the other contraints e.g. orthogonality or
known internal parameters.
a
b
Direct metric reconstruction using w
if ω is known, can directly do metric reconstruction, without stratified recon:
approach :
ω  K -T K -1  K
•Assume ω is known in both images, then can find K and K’ fortwo poses;
•then apply essential matrix to metrically reconstruct:
•Four cases; two are just miroor images; two are twisted pair;
•All solutions but one can be ruled out: points only lie in front of camera
approach 2
•compute projective reconstruction
•back-project w from both images
•Intersection of the two cones defines ∞ and its support plane p∞
•in general two cones intersect in two different plane
conics, each lying in a different support plane two solutions
 Twisted pair ambiguity
Direct reconstruction using ground truth
•use control points XEi with known coordinates to go from projective to
metric
Since projective reconstruction Is related to to
true reconstruction by a Homography:
X Ei  HX i
Each point correspondence provides
3 linearly indep. Eqns on elements of H;
Since H has 15 elements 5 or more points
Alternative: Relate ground control points to image measurements
-1
(2 lin. eq. in H-1 per view,3 for two views;)
i
Ei
x  PH X
a
c
b