CS 395/495-25: Spring 2004
IBMR:
The Fundamental Matrix
Combines Two Cameras
Jack Tumblin
jet@cs.northwestern.edu
Basic Camera P0: P3P2 (or camera R3)
• Basic Camera P0 is a 3x4 matrix:
xf s px 0 xc
0 yf py 0 yc
0
0
1 0 zc
1
K
(3x3 submatrix)
=
x
y
z
y
zc
P0 X = x
yc
y’
p
C
z
f
xc
[K 0] = P0
• Non-square pixels? change scaling (x, y)
• Parallelogram pixels? set nonzero skew s
K matrix: “(internal) camera calib. matrix”
ONE Camera Matrix P links P3P2:
• Basic camera:
xf s px 0
0 yf py 0
0 0 1 0
x = P0 X where P0 = [K | 0] =
• World-space camera:
~
translate X (origin) to camera location C, then rotate:
~
x = PX = (P0·R·T) X = K·R·[I C] X
• Rewrite as:
~
P = K[R -RC]
Input: X
3D World Space
y
• Redundant notation:
P = [M p4]
M = RK
~
p4 = -K·R·C
Output: x
2D Camera Image
x
X
(world space)
z
ONE Complete Camera Matrix P
• K matrix: “internal camera calib. matrix”
• R·T matrix: “external camera calib. matrix”
~
– T matrix: Translate world to the cam. origin C
– R matrix: 3D rotate world to fit cam. axes
Combine: write
(P0·R·T·X = x
~
or K[R -RC] ·X = x
or
P·X = x
Input: X
3D World Space
Output: x
2D Camera Image
~
C
y
x
X
(world space)
z
TWO Cameras: Epipolar Geometry
Basic idea:
• Given ONLY images from 2 cameras C, C’
• Different views of same objects X, but
we don’t know world-space points X.
X
• If we choose an x,
how can we find x’ ?
• How are x, x’ linked?
x
x’
TWO Cameras: Epipolar Geometry
Basic idea:
• 2 cameras located at C, C’ in world space.
• Find ‘baseline’ through camera centers C, C’
• Baseline hits image planes at ‘epipoles’
Notice baseline and X
form a plane...
(OTHER planes thru
baseline for other X...)
X
x
x’
TWO Cameras: Epipolar Geometry
Basic idea:
• 2 cameras located at C, C’ in world space.
• Find ‘baseline’ through camera centers C, C’
X
• Baseline hits image planes at ‘epipoles’
• Family of planes thru baseline
are all the ‘epipolar planes’
• Image of planes = lines =
x’
x
‘epipolar lines’
baseline
• Lines intersect at
C
epipolar points in both images.
TWO Cameras: Epipolar Geometry
X
epipolar plane 
x’
x
baseline
C
e
e’
C’
Summary:
• Connect cameras C, C’ with a baseline, which
hits image planes at epipoles e, e’.
• Chose any world pt X, then  everything is coplanar!
epipolar plane  includes image points x, x’, and these
connect to epipoles e,e’ by epipolar lines L, L’
Epipolar Geometry
X
epipolar plane 
x’
x
baseline
C
e
e’
C’
Useful properties:
• Every image point x maps to an epipolar line L’
also:
–
–
–
–
Epipoles e,e’ = each camera’s view of the other
All epipolar lines L’ pass through epipole e’
Epipolar Line L’ is (image of the CX ray...)
Epipolar Line L’ links (image of C) to (image of X)
Fundamental Matrix: Fx = L’
X
epipolar plane 
x’
x
baseline
C
e
e’
C’
One Matrix Summarizes ALL of Epipolar Geometry!
Fundamental Matrix F: 3x3, rank 2.
• Maps image point x to image point x’: x’T F x = 0
but F is only Rank 2 –
given only x, F cannot find x’ for you!!
• Maps image point x to epipolar line L’:
F x = L’
Fundamental Matrix: Fx = L’
X
epipolar plane 
x’
x
baseline
C
e
e’
C’
One Matrix Summarizes ALL of Epipolar Geometry!
Fundamental Matrix F: 3x3, rank 2.
• Maps image point x to image point x’: (x’T F) x = 0
(We know x is on line L, thus (LT) x = 0, so
• Maps image point x to epipolar line L’:
F x = L’
Fundamental Matrix: Fx = L’
X
epipolar plane 
x’
x
baseline
C
e’
C’
• 2) What is F if we DON’T know the cameras P, P’,
but we DO know some corresp. point pairs (x, x’)?
– F finds epipolar line L’ from point x : Fx = L’
– (Recall that if (any) point x’ is on line a L’, then x’T L’= 0)
– Substitute Fx for L’: x’TF x = 0
AHA! we can find F using DLT-like method! see Chap. 10
Derive the Fundamental Matrix:
X
epipolar plane 
x’
x
baseline
C
e’
C’
– Recall Pseudo-Inverse: P+ = PT ( P PT)-1 and P P+= I (pg148)
– Write world-space ray XC as: X() = P+x + C
– Other camera’s image of the ray is its epipolar line L’:
L’() = P’X() = P’P+x + P’C
.
– But P’C = e’; it is the epipole of the other camera, so
L’() = P’P+x + e’
Derive the Fundamental Matrix:
epipolar plane 
X
x’
x
baseline
e’
C
C’
– But P’C = e’; it is the epipole of the other camera, so
L’() = P’P+x + e’
– Rewrite epipolar line L’ using points P’P+x and e’
– (Recall: find line between two P2 points with cross product: L= x1  x2)
– To get
L’ = e’ P’ P+ x
or
L’ = F x
.
Derive the Fundamental Matrix:
X
epipolar plane 
x’
x
baseline
C
e’
C’
– But P’C = e’; it is the epipole of the other camera, so
L’() = P’P+x + e’
– Rewrite epipolar line L’ using points P’P+x and e’
.
– (Recall: find line between two P2 points with cross product: L= x1  x2)
– To get
L’ = (e’)  (P’ P+ x)
THUS F = [e’] P’ P+
or
L’ = F x
Hunh? What Is This?!?!
Derive the Fundamental Matrix:
X
epipolar plane 
x’
x
baseline
C
e’
C’
F = [e’] P’ P+ But what’s this? A NEW TRICK:
– Cross Product written as matrix multiply (Zisserman pg. 554)
a  b = a1  b1 = a2b3 – a3b2 = 0 -a3 a2 b1 = [a] · b
a3 0 -a1 b2
a2 b2 a3b1 – a1b3
-a2 a1 0 b3
a3 b3 a1b2 – a2b1
– So write: a  b = -b  a = [a]·b = (aT·[b])T
a ‘skew symmetric’
matrix
Fundamental Matrix: Fx = L’
X
epipolar plane 
x’
x
baseline
C
e’
C’
F = [e’]  P’ P+
• Given a point pair (x,x’) Matrix F is unique!
• Cool! still works even for different cameras!
• F tied DIRECTLY to corresp. point pairs (x, x’):
– F finds epipolar line L’ from point x : Fx = L’
– (Recall that if (any) point x’ is on line a L’, then x’T L’= 0)
– Substitute Fx for L’: x’TF x = 0
Fundamental Matrix: Fx = L’
X
epipolar plane 
x’
x
baseline
C
e’
C’
1) How do we find F?
If we know the camera matrices P and P’
(we almost never do), we showed (Zisserman pg 224)
F = [e’] P’ P+
( Recall: a  b = -b  a = [a]·b = (aT·[b])T )
(Recall: P+ = PT(PPT)-1, the pseudo-inverse)
Fundamental Matrix: Fx = L’
X
epipolar plane 
x’
x
baseline
C
e’
C’
• 2) What is F if we DON’T know the cameras P, P’,
but we DO know some corresp. point pairs (x, x’)?
– F finds epipolar line L’ from point x : Fx = L’
– (Recall that if (any) point x’ is on line a L’, then x’T L’= 0)
– Substitute Fx for L’: x’TF x = 0
AHA! we can find F using DLT-like method! see Chap. 10
Fundamental Matrix Summary
(pg. 226)
F is 3x3 matrix, maps P2P2, rank 2, 7-DOF
• If world space pt X  image space pts. x and x’ then
x’TF x = 0
• Every image pt has epipolar line in the other image:
Fx = L’
FTx’ = L
• Baseline pierces image planes at epipoles e, e’ :
Fe = 0
FTe’ = 0
Fundamental Matrix Summary
(pg. 226)
F is 3x3 matrix, maps P2P2, rank 2, 7-DOF
• Given camera matrices P, P’, find F matrix by:
F = [e’] P’ P+
(recall: e’ is image of C: e’ = P’C)
• F is unaffected by any proj. transforms done
on BOTH cameras
(PH, P’H) has same F matrix as (P, P’) for any full-rank H
(e.g. F measures camera C vs. Camera C’ only,
no matter where you put them)
Fundamental Matrix Uses
Special case: camera translate only (no rotations)
• Camera matrices are P= K[ I | 0 ], P’ = K[I | t ]
– where K is internal calib., t is 3D translation vector tx
• F matrix simplifies to F = [e’]
• Epipolar lines are all parallel to direction t
ty
tz
• x,x’ displacement depends only on t & 3D depth z:
x’ = x + (Kt)(1/z)
Fundamental Matrix Uses
Can we find a camera matrix K from motion + fundamental F?
• Let one camera position define the world’s coords:
P = P0 = K[ I | 0 ], and define the other as
~
P’ = [M | m] = K[R | -RC]
~
where K is internal calib., R is rotation C is world-space
position
~
• F matrix simplifies to F = [m]M
• If we know how we moved the camera (R, C matrices)
then find F by correspondence, and then solve for K. (pg 237)
Fundamental Matrix Uses
General movement?
• Recall: rotations don’t change image content
(camera rotate a P3 homography matrix H)
• ANY cameras, ANY movements can then be
warped to remove rotations, THUS
• Can ALWAYS get parallel epipolar lines!
– Easier to find correspondences
– Easier to find depth values z
– ‘Parallel Epipolar Lines’==‘Rectified Image Pair’
Fundamental Matrix Properties
(pg. 226)
Why bother with F?
• Can find it from image pt. correspondences only
• Works even for mismatched cameras
(example: 100-year time-lapse of Eiffel tower)
• Choose your own world-space coordinate system.
• SVD lets us recover P, P’ camera matrices from F
– (4-way ambiguity; what is front/back of C and C’?) pg 240
– BUT WE DON’T NEED TO!
• Complete 2-camera mapping from worldimage
– 2 images + corresponding point pairs (xi,x’i)F
– Let camera coords == 3D world coords, then (xi,x’i)Xi
Correspondence Problem:
Where Computer Vision, IBMR part ways:
Few Images: (image pairs, trios…)
FIND Fundamental Matrix +
FIND Corresponding point pairs (x,x’)
************Versus**************
Many Images:
GATHER enough for P2-only work:
ignore or simplify the correspondence?
Conclusions
• P2, P3 matrix forms give elegant, principled
notation for ALL image geometry
– Cameras, lights, points, lines, planes, conics,
quadrics, twisted cubics, …
– Matrix form makes everything reversible: 3D
from (2D)*!
– We can find shapes from images,
BUT ONLY if we have enough
reliable point correspondences…
And THAT’s what we usually CAN’T find!
Another way to describe P3 lines!
• Given world-space camera position
C =[a b c 1] and
• a world-space direction D (found from a pixel xd, as shown
on prev. slide),
• even though D is a ‘point at infinity’, we can use it to make
a parametric line in world space:
line(t) = C + Dt =[a b c 1] + [d e f 0]t
• Note that ‘t’ is position along the line, and it’s in worldspace coords!
• If you have two cameras C0, C1, and found two
corresponding image points p0 and p1, you can use
method on previous slide to find a world-space direction
D0, D1 for each image point, then make two parameterized
lines L0(t0), L1(t1) and solve for their world-space
intersection point X. THAT’S how you can do triangulation!.