Computing F
Epipolar geometry: basic equation
x' T Fx  0
x' xf11  x' yf12  x' f13  y ' xf21  y ' yf 22  y ' f 23  xf31  yf 32  f 33  0
separate known from unknown
x' x, x' y, x' , y' x, y' y, y' , x, y,1 f11, f12 , f13 , f 21, f 22 , f 23 , f31, f32 , f33 T  0
(data)
 x'1 x1
 
 x ' n xn

x'1 y1

x 'n y n
(unknowns)
(linear)
x'1

x 'n
y '1 x1

y ' n xn
y '1 y1

y 'n y n
y '1

y 'n
x1

xn
y1 1
  f  0
yn 1
Af  0
If A has rank 8  soln unique up to a scale factor
For noisy data, rank of A might be larger than 8  least squares soln
Minimize || Af || subject to ||f|| = 1  singular vector for smallest sing. value of A
This is called “8 point” algorithm
the singularity constraint:
e'T F  0
Fe  0 detF  0
rank F  2
SVD from linearly computed F matrix (rank 3)
σ1

 V T  U1σ1V1T  U 2 σ 2 V2T  U 3σ 3 V3T
F  U
σ2


σ
3

Replace F with F‘ that minimizes Forbenius norm || F – F‘ ||F subject to det(F‘) = 0
Compute closest rank-2 approximation
min F - F' F
σ1

F'  U 
σ 2  V T  U1σ1V1T  U 2 σ 2 V2T


0


Two steps: (a) linear soln F obtained from sing. Vect corresp. To smallest
singular value (b) replace F by F’ the closest singular matrix to F under
Forbenius norm
|| A||F = square root of sum of squares of ai,j
α F1 + ( 1 – α ) F2
the minimum case – 7 point correspondences
Assume matrix A has rank 7  7 correspondences
 x'1 x1
 
 x'7 x7

x'1 y1

x '7 y 7
x'1

x '7
y '1 x1

y '7 x7
y '1 y1

y '7 y 7
A  U 7x7 diag σ1 ,..., σ 7 ,0,0V9x9
 A[V8V9 ]  07x2
y '1

y '7
T
y1 1
  f  0
y7 1
x1

x7
A is 7x9 ;
e.g.V
T
V8  [000000010 ]T 
If A rank 7  soln to Af =0 is 2D space of the form α F1 + ( 1 – α ) F2 where
F1 and F2 are matrices corresponding to generators f1 and f2 of the right null
space of A
Det (F) = 0  det (α F1 + ( 1 – α ) F2 ) = 0  cubic polynomial in α 
either one or 3 real solns  either one or 3 F
This is called the “7 point “ algorithm.
the NOT normalized 8-point algorithm
 f11 
f 
 12 
 f13 
 x1 x1´ y1 x1´ x1´ x1 y1´ y1 y1´ y1´ x1 y1 1 

f
x x ´ y x ´ x ´ x y ´ y y ´ y ´ x
  21 
y
1
2 2
2
2 2
2 2
2
2
2
 2 2
 f   0
 






   22 

  f 23 
x
x
´
y
x
´
x
´
x
y
´
y
y
´
y
´
x
y
1
n n
n
n n
n n
n
n
n
 n n
 f 
 31 
~10000
~100
~10000
~100 ~10000 ~10000 ~100 ~100
1 f 
32


Orders of magnitude difference
f
 33 
!
Between column of data matrix
 least-squares yields poor results
the normalized 8-point algorithm
Transform image to ~[-1,1]x[-1,1]
(0,500)
(0,0)
(700,500)
(700,0)
 2
 700





0
2
500

 1

 1

1

(-1,1)
(1,1)
(0,0)
(-1,-1)
(1,-1)
Least squares yields good results (Hartley, PAMI´97)
Translate and scale each image so that the centroid of the reference points
is at the origin of the coordinates and the RMS distance of the points from
the origin is √2
algebraic minimization
Find F‘ with rank 2 directly, i.e. Find singular matrix F‘ that
minimizes || A F‘|| subject to ||F‘|| = 1
• Arbitrary singular matrix F can be written as F = M [e]x where
M is nonsingular and [e]x is any skew symmetrix matrix
• For now, assume e is known
• F = M [e]x  f = Em where f and m are vectors
corresponding to F and M matrices and
• Minimize || A E m || subject to || Em || = 1
• Apply Algorithm A.5.6 from Hartley and Zisserman
Geometric distance
Gold standard
Sampson error
Symmetric epipolar distance
Gold standard
Maximum Likelihood Estimation (= least-squares for Gaussian noise)
xi and x’I measured correspondence;
Find F to minimize
x̂'i
and
x̂ i
true correspondence
T




subject
to
x̂
'
Fx̂  0
d
x
,
x̂

d
x'
,
x̂
'
 i i
i
i
2
2
i
Initialize: normalized 8-point, (P,P‘) from F, reconstruct Xi
Parameterize:
P  [I | 0], P'  [M | t], X i
x̂ i  PXi , x̂'i  P' Xi
(overparametrized)
Minimize cost using Levenberg-Marquardt
(preferably sparse LM, see book)
Symmetric epipolar error
Minimize distance of a point from its projected epipolar line
 d x' , Fx 
2
i
i

 d x i , F x' i
T

2
i


1
1


  x' Fx

2
2
2
2
 x' T F  x' T F
Fx 1  Fx 2 
1
2

T

 

Some experiments: Epipoles far from center of image
Some experiments:
Epipoles close
to image
Epipoles in the image
Some experiments:
Matched points known extremely accurately
Algorithm comparison
•
•
•
•
•
•
Normalized 8 point (alg. 11.1)
Min. Algebraic error while imposing singularlity (Alg. 11.2)
Gold standard algorithm (Alg. 11.3)
N = total number of correspondences
Choose n out of N points randomly 100 times
Find average residual error vs. n


( d x' i , Fx i   d x i , FT x' i ) / N
2
2
• Squared distance between a point’s epipolar line and the
other image averaged over all N
• This is not directly minimized by any of the 3 algorithms
i
Some experiments:
Residual error:

 d x' i , Fx i   d x i , FT x' i
2
i

2
(for all points!)
Recommendations:
1. Do not use unnormalized algorithms
2. Quick and easy to implement: 8-point normalized
(alg. 11.1)
3. Better: enforce rank-2 constraint during minimization
(alg. 11.2)
4. Best: Maximum Likelihood Estimation (alg. 11.3)
(minimal parameterization, sparse
implementation)
Automatic computation of F
(i) Interest points
(ii) Putative correspondences
(iii) RANSAC
(iv) Non-linear re-estimation of F
(v) Guided matching
(repeat (iv) and (v) until stable)
Feature points
• Extract feature points to relate images
• Required properties:
• Well-defined
(i.e. neigboring points should all be different)
• Stable across views
(i.e. same 3D point should be extracted
as feature for neighboring viewpoints)
Feature points
(e.g.Harris&Stephens´88; Shi&Tomasi´94)
Find points that differ as much as possible
from all neighboring points
homogeneous
edge
T  ( x, y)
SSD   M
corner
T
M
 xI 
 I 
W  y 
 


I
x

I
w x, y
y
dxdy
If M has two small eigen value  homogeneous; if one small and one large eigenvalue 
edge; if two large eigenvalues  corner
Feature = local maxima (subpixel) of F(1,  2)
Feature points
Select strongest features (e.g. 1000/image)
Feature matching
Evaluate NCC for all features with
similar coordinates
e.g. x´, y´  x  10w , x  10w   y  10h , y  10h 
Keep mutual best matches
Still many wrong matches!
?
Feature example
3
2
4
1
5
0.96
-0.40
-0.16
-0.39
0.19
-0.05
0.75
-0.47
0.51
0.72
3
2
4
1
-0.18
-0.39
0.73
0.15
-0.75
-0.27
0.49
0.16
0.79
0.21
0.08
0.50
-0.45
0.28
0.99
Gives satisfying results
for small image motions
5
Wide-baseline matching…
• Requirement to cope with larger
variations between images
•
•
•
•

Translation, rotation, scaling
geometric
transformations
Foreshortening
photometric
Non-diffuse reflections
changes
Illumination

Wide-baseline matching…
(Tuytelaars and Van Gool BMVC 2000)
Wide baseline matching for two different
region types
Finding more matches
restrict search range to neighborhood of epipolar line
(1.5 pixels)
relax disparity restriction (along epipolar line)
RANSAC
Step 1. Extract features
Step 2. Compute a set of potential matches
Step 3. do
Step 3.1 select minimal sample (i.e. 7 matches)
Step 3.2 compute solution(s) for F
Step 3.3 determine inliers (verify hypothesis)

(generate
hypothesis)
until (#inliers,#samples)<95%
Step 4. Compute F based on all inliers
Step 5. Look for additional matches
Step 6. Refine F based on all correct matches
  1  (1 


# inliers 7 # samples
)
# matches
#inliers
90%
80%
70%
60%
50%
#samples
5
13
35
106
382
How many samples?
Choose N so that, with probability p, at least one
random sample is free from outliers. e.g. p=0.99
1  1  e 
s N
 1 p

N  log 1  p  / log 1  1  e 
s

proportion of outliers e
s
2
3
4
5
6
7
8
5%
2
3
3
4
4
4
5
10% 20% 25% 30% 40% 50%
3
5
6
7
11
17
4
7
9
11
19
35
5
9
13
17
34
72
6
12
17
26
57
146
7
16
24
37
97
293
8
20
33
54 163 588
9
26
44
78 272 1177
More problems:
• Absence of sufficient features (no texture)
• Repeated structure ambiguity
• Robust matcher also finds
support for wrong hypothesis
• solution: detect repetition
(Schaffalitzky and Zisserman,
BMVC‘98)
two-view geometry
geometric relations between two views is fully
described by recovered 3x3 matrix F
Special case:
Enforce constraints for optimal results:
Pure translation (2dof),
Planar motion (6dof),
Calibrated case (5dof)  compute essential matrix;
but for E, two singular values are equal.