Epipolar geometry
Class 5
3D photography course schedule
(tentative)
Lecture
Exercise
Introduction
-
Oct. 3
Geometry & Camera model
Camera calibration
Oct. 10
Single View Metrology
Measuring in images
Oct. 17
Feature Tracking/matching
Correspondence computation
Sept 26
(Friedrich Fraundorfer)
Oct. 24
Epipolar Geometry
F-matrix computation
Oct. 31
Shape-from-Silhouettes
Visual-hull computation
(Li Guan)
Nov. 7
Stereo matching
Project proposals
Nov. 14
Structured light and
active range sensing
Papers
Nov. 21
Structure from motion
Papers
Nov. 28
Multi-view geometry
and self-calibration
Papers
Dec. 5
Shape-from-X
Papers
Dec. 12
3D modeling and registration
Papers
Dec. 19
Appearance modeling and
image-based rendering
Final project presentations
Optical flow
• Brightness constancy assumption
(small motion)
• 1D example
possibility for iterative refinement
Optical flow
• Brightness constancy assumption
(small motion)
• 2D example
the “aperture” problem
(1 constraint)
(2 unknowns)
?
isophote I(t+1)=I
isophote I(t)=I
Optical flow
• How to deal with aperture problem?
(3 constraints if color gradients are different)
Assume neighbors have same displacement
Lucas-Kanade
Assume neighbors have same displacement
least-squares:
Revisiting the small motion assumption
• Is this motion small enough?
• Probably not—it’s much larger than one pixel
(2nd order terms dominate)
• How might we solve this problem?
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Reduce the resolution!
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Coarse-to-fine optical flow
estimation
slides from
Bradsky and Thrun
u=1.25 pixels
u=2.5 pixels
u=5 pixels
image It-1
u=10 pixels
Gaussian pyramid of image It-1
image I
Gaussian pyramid of image I
Coarse-to-fine optical flow
estimation
slides from
Bradsky and Thrun
run iterative L-K
warp & upsample
run iterative L-K
.
.
.
image IJt-1
Gaussian pyramid of image It-1
image I
Gaussian pyramid of image I
Feature tracking
• Identify features and track them
over video
• Small difference between frames
• potential large difference overall
• Standard approach:
KLT (Kanade-Lukas-Tomasi)
Good features to track
• Use same window in feature selection as for
tracking itself
• Compute motion assuming it is small
differentiate:
Affine is also possible, but a bit harder (6x6 in stead of 2x2)
Example
Simple displacement is sufficient between consecutive
frames, but not to compare to reference template
Example
Synthetic example
Good features to keep tracking
Perform affine alignment between first and last frame
Stop tracking features with too large errors
Two-view geometry
Three questions:
(i)
Correspondence geometry: Given an image point x in
the first image, how does this constrain the position of the
corresponding point x’ in the second image?
(ii) Camera geometry (motion): Given a set of corresponding
image points {xi ↔x’i}, i=1,…,n, what are the cameras P and
P’ for the two views?
(iii) Scene geometry (structure): Given corresponding image
points xi ↔x’i and cameras P, P’, what is the position of
(their pre-image) X in space?
The epipolar geometry
C,C’,x,x’ and X are coplanar
The epipolar geometry
What if only C,C’,x are known?
The epipolar geometry
All points on p project on l and l’
The epipolar geometry
Family of planes p and lines l and l’
Intersection in e and e’
The epipolar geometry
epipoles e,e’
= intersection of baseline with image plane
= projection of projection center in other image
= vanishing point of camera motion direction
an epipolar plane = plane containing baseline (1-D family)
an epipolar line = intersection of epipolar plane with image
(always come in corresponding pairs)
Example: converging cameras
Example: motion parallel with image plane
(simple for stereo  rectification)
Example: forward motion
e’
e
The fundamental matrix F
algebraic representation of epipolar geometry
x  l'
we will see that mapping is (singular) correlation
(i.e. projective mapping from points to lines)
represented by the fundamental matrix F
The fundamental matrix F
geometric derivation
x'  H π x
l'  e'x'  e' H π x  Fx
mapping from 2-D to 1-D family (rank 2)
The fundamental matrix F
algebraic derivation
Xλ   P  x  λC

l'  P' C  P' P x
F  e' P' P 
(note: doesn’t work for C=C’  F=0)
PP

P x
Xλ 
I

The fundamental matrix F
correspondence condition
The fundamental matrix satisfies the condition
that for any pair of corresponding points x↔x’ in
the two images
T
T
x' Fx  0
x'
l'  0

The fundamental matrix F
F is the unique 3x3 rank 2 matrix that
satisfies x’TFx=0 for all x↔x’
(i)
(ii)
(iii)
(iv)
(v)
Transpose: if F is fundamental matrix for (P,P’), then
FT is fundamental matrix for (P’,P)
Epipolar lines: l’=Fx & l=FTx’
Epipoles: on all epipolar lines, thus e’TFx=0, x
e’TF=0, similarly Fe=0
F has 7 d.o.f. , i.e. 3x3-1(homogeneous)-1(rank2)
F is a correlation, projective mapping from a point x to
a line l’=Fx (not a proper correlation, i.e. not invertible)
Fundamental matrix for pure translation
Fundamental matrix for pure translation
Fundamental matrix for pure translation
General motion
F  e' P' P 
Pure translation
K


P 
 0 
-1
P  K[I | 0]
P' K[I | t]

e  e 
 0
F  e'   e 0
e 
 e  e 0 


z

y
z
y
x
x
for pure translation F only
has 2 degrees of freedom
The fundamental matrix F
relation to homographies
e' H

π
F
l'  H l
-T
π
e'  H e
valid for all plane homographies
π
The fundamental matrix F
relation to homographies
x
π
l
x
x
x'  H x  l  Fx
π 
π
requires
π
l e'  0
T
π
e.g. H  e' F
e' e'  0
T
Projective transformation and invariance
Derivation based purely on projective concepts
x̂  Hx, x̂'  H' x'  F̂  H' -T FH -1
F invariant to transformations of projective 3-space


x  PX  PH  H -1X  P̂X̂


x'  P' X  P' H  H -1X  P̂' X̂
P, P'  F
F  P, P'
unique
not unique
canonical form
P  [I | 0]
P'  [M | m]
F  m M
F  e' P' P 


Projective ambiguity of cameras given F
previous slide: at least projective ambiguity
this slide: not more!
~~
Show that if F is same for (P,P’) and (P,P’),
there exists a projective transformation H so that
~
~
P=HP and P’=HP’
P  [I | 0] P'  [A | a]
~
~ ~
P  [I | 0] P'  [A | ~
a]
~
~
F  a  A  a  A

~
~
lemma: a  ka and A  k A  av
1
T


rank 2
aF  aa  A  0  ~
aF  ~
a  ka
a  A  ~a  A~  a  kA~ - A  0  kA~ - A  avT




P' H  [A | a] k I 0 
k v k 
~
 [k A - av  | ka ]  P'
(22-15=7, ok)
 k 1 I
H   1 T
k v
0
k 
1
1
1
T
T
The projective reconstruction theorem
If a set of point correspondences in two views determine the
fundamental matrix uniquely, then the scene and cameras
may be reconstructed from these correspondences alone,
and any two such reconstructions from these
correspondences are projectively equivalent
allows reconstruction from pair of uncalibrated images!
Epipolar geometry
Underlying structure
in set of matches for
rigid scenes
p
L2
C1
m1
lT1 l2
mT2 F m1  0
M
l1
e1
e2
m2
Fundamental matrix
(3x3 rank 2 matrix)
L1
l2
C2
1. Computable from corresponding
points
Canonical representation:
P  [I | 0] P'  [[e' ] F  e' vT | λe' ] 2. Simplifies matching
3. Allows to detect wrong matches
4. Related to calibration
Epipolar geometry?
courtesy Frank Dellaert
Other entities besides points?
Lines give no constraint for two view geometry
(but will for three and more views)
Curves and surfaces yield some constraints
related to tangency
(e.g. Sinha et al. CVPR’04)
Computation of F
•
•
•
•
Linear (8-point)
Minimal (7-point)
Robust (RANSAC)
Non-linear refinement (MLE, …)
• Practical approach
Epipolar geometry: basic equation
x' T Fx  0
x' xf11  x' yf12  x' f13  y' xf21  y' yf 22  y' f 23  xf31  yf 32  f 33  0
separate known from unknown
x' x, x' y, x' , y' x, y' y, y' , x, y,1 f11, f12 , f13 , f 21, f 22 , f 23 , f31, f32 , f33 T  0
(data)
 x'1 x1
 
 x ' n xn

x'1 y1

x 'n y n
(unknowns)
(linear)
x'1

x 'n
y '1 x1

y ' n xn
y '1 y1

y 'n yn
Af  0
y '1

y 'n
x1

xn
y1 1
  f  0
yn 1
the NOT normalized 8-point algorithm
 f11 
f 
 12 
 f13 
 x1 x1´ y1 x1´ x1´ x1 y1´ y1 y1´ y1´ x1 y1 1 

f
x x ´ y x ´ x ´ x y ´ y y ´ y ´ x
  21 
y
1
2 2
2
2 2
2 2
2
2
2
 2 2
 f   0
22
 







  

  f 23 
 xn xn ´ y n xn ´ xn ´ xn y n ´ y n y n ´ y n ´ xn y n 1  f 
 31 
~10000
~100
~10000
~100 ~10000 ~10000 ~100 ~100
1 f 
32


Orders of magnitude difference
f
 33 
!
between column of data matrix
 least-squares yields poor results
the normalized 8-point algorithm
Transform image to ~[-1,1]x[-1,1]
(0,500)
(0,0)
(700,500)
 2
 700





0
2
500
(700,0)

 1

 1

1

(-1,1)
(1,1)
(0,0)
(-1,-1)
normalized least squares yields good results
(Hartley, PAMI´97)
(1,-1)
the singularity constraint
e'T F  0
Fe  0 detF  0
rank F  2
SVD from linearly computed F matrix (rank 3)
σ1

 V T  U1σ1V1T  U 2σ 2 V2T  U 3σ 3V3T
F  U
σ2


σ
3

Compute closest rank-2 approximation
min F - F' F
σ1

F'  U 
σ 2  V T  U1σ1V1T  U 2σ 2 V2T


0


the minimum case – 7 point correspondences
 x'1 x1
 
 x'7 x7

x'1 y1

x '7 y 7
x'1

x '7
y '1 x1

y '7 x7
y '1 y1

y '7 y 7
A  U7x7 diag σ1 ,..., σ7 ,0,0V9x9
y '1

y '7
y1 1
  f  0
y7 1
x1

x7
T
 A[V8V9 ]  09x2
x i (F1  λF2 )x i  0, i  1...7
T
one parameter family of solutions
but F1+lF2 not automatically rank 2
e.g.V
T
V8  [000000010 ]T 
the minimum case – impose rank 2
3
(obtain 1 or 3 solutions)
F7pts F
F1
F2
det( F1  λF2 )  a3λ 3  a2 λ 2  a1λ  a0  0
(cubic equation)
det( F1  λF2 )  det F2 det( F2-1F1  λI)  0 detAB  detA.detB
Compute possible l as eigenvalues of F2-1F1
(only real solutions are potential solutions)
Automatic computation of F
Step 1. Extract features
Step 2. Compute a set of potential matches
Step 3. do
RANSAC
Step 3.1 select minimal sample (i.e. 7 matches)
Step 3.2 compute solution(s) for F
Step 3.3 determine inliers (verify hypothesis)

(generate
hypothesis)
until (#inliers,#samples)<95%
Step 4. Compute F based on all inliers
Step 5. Look for additional matches
Step 6. Refine F based on all correct matches
  1  (1 


# inliers 7 # samples
)
# matches
#inliers
90%
80%
70%
60%
50%
#samples
5
13
35
106
382
Finding more matches
restrict search range to neighborhood of epipolar line
(e.g. 1.5 pixels)
relax disparity restriction (along epipolar line)
Issues:
• (Mostly) planar scene (see next slide)
• Absence of sufficient features (no texture)
• Repeated structure ambiguity
• Robust matcher also finds
support for wrong hypothesis
• solution: detect repetition
(Schaffalitzky and Zisserman,
BMVC‘98)
Computing F for quasi-planar scenes
QDEGSAC
#inliers
337 matches on plane, 11 off plane
%inclusion of
out-of-plane inliers
17% success for RANSAC
100% for QDEGSAC
data rank
two-view geometry
geometric relations between two views is fully
described by recovered 3x3 matrix F