CS 395/495-25: Spring 2003
IBMR: Week 8A
Chapter 8...
Epipolar Geometry:
Two Cameras:
Jack Tumblin
jet@cs.northwestern.edu
IBMR-Related Seminars
Light Scattering Models
For Rendering Human Hair
Steve Marschner, Cornell University
Friday May 23 3:00pm, Rm 381, CS Dept.
Reminders
• ProjA graded: Good Job! 90,95, 110
• ProjB graded: Good! minor H confusions.
• MidTerm graded: novel solutions
encouraged.
• ProjC due Friday, May 16: many rec’d...
• ProjD posted, due Friday May 30
• (Last Week’s IAC notes revised...)
• Take-Home Final Exam: Thurs June 5, due
June 11
Camera Matrix P links P3P2
• Basic camera:
xf s px 0
0 yf py 0
0 0 1 0
x = P0 X where P0 = [K | 0] =
• World-space camera:
~
translate X to camera location C, then rotate:
x = PX = (P0·R·T) X
• Rewrite as:
~
P = K[R | -RC]
Input: X
3D World Space
y
• Redundant notation:
P = [M | p4]
M = KR ~
p4 = -K R C
Output: x
2D Camera Image
x
X
(world space)
z
Chapter 7: More One-Camera Details
Full 3x4 camera matrix P maps P3world to P2 image
? What does it do to basic 3D world shapes?
Forward Projection:
• Line / Ray in world  Line/Ray in image:
– Ray in P3 is:
– Camera changes to P2:
X() = A + B
x() = PA + PB
yc
A
B
PA
zc
p
f
xc
C
Chapter 7: More One-Camera Details
Full 3x4 camera matrix P maps P3world to P2 image
? What does it do to basic 3D world shapes?
Backward Projection:
• Line L in image  Plane L in world:
– Recall: Line L in P2 (a 3-vector): L = [x1 x2 x3]T
– Plane L in P3 (a 4-vector):
L = PT·L =
p11
p12
p13
p14
p21
p22
p23
p24
p31
p32
p33
p34
x1
x2
x3
yc
L
zc
p
f
xc
C
Cameras as Protractors
• World-Space Direction: D = [xc, yc, zc, 0]T
• Direction from image point x: D = (KR)-1x
• Point C and points x1,x2 form angle : (pg 199)
cos  =
x1T (K-TK-1) x2
(x1T (K-TK-1) x1) (x2T (K-TK-1) x2)
(K-TK-1) =  = ‘Image of Absolute Conic’
Line L makes plane 
with normal n = KT L (in camera coords)
Cameras as Protractors
P () = (K-TK-1). OK. Now what was  again?
• Absolute Conic  =all imaginary points on 
– Satisfies BOTH x12 + x22 + x32 = 0 AND x42 = 0
D1T· ·D2 = 0
– Finds right angles-- if D1  D2, then:
• Dual of absolute conic is Dual Quadric Q*
= all imaginary planes   to , tangent to Q*
– Finds right angles-- if 1  2, then:
1T· Q*·2 = 0
– can write  or Q* as the same matrix:
x1 x2 x3 0 1
0
0
0
0
1
0
0
0
0
1
0
0
0
0
0
x1
x2
x3
0
Cameras as Protractors
Clever vanishing point trick:
• Perpendicular lines in image?
• Find their vanishing pts. by construction:
• Use v1T v2 = 0, stack, solve for  = (K-TK-1)
v3
v1
v2
Cameras as Protractors
P  = (K-TK-1) =  = ‘Image of Absolute Conic’
• Just as has a dual Q*,  has dual * :
* = -1 = K KT
• The dual conic * is the image of Q* , so
* = P (Q*) = P( 1 0 0 0 )
0 1 0 0
0 0 1 0
0 0 0 0
• Vanishing points v1,v2 of 2  world-space lines:
v1T v2 = 0
• Vanishing lines L1, L2 of 2  world-space planes:
L1T* L2 = 0
Movement Detection?
• Can we do it from images only?
– 2D projective transforms often LOOK like 3-D;
– External cam. calib. affects all elements of P
• YES. Camera moved if-&-only-if
Camera-ray points (CxX1,X2, etc…) will
map to LINE (not a point) in the other image
• ‘Epipolar Line’ == l’ = image of L
• ‘Parallax’ == x1’x2’ vector
X2
X1
x
L
x2’
x1’
l’
C
C’
Epipolar Geometry: Chapter 8
Basic idea:
• Given ONLY images from 2 cameras C, C’
• Different views of same objects X, but
we don’t know world-space points X.
X
• If we choose an x,
how can we find x’ ?
• How are x, x’ linked?
x
x’
Epipolar Geometry: Chapter 8
Basic idea:
• 2 cameras located at C, C’ in world space.
• Find ‘baseline’ through camera centers C, C’
• Baseline hits image planes at ‘epipoles’
Notice baseline and X
form a plane...
Many OTHER planes
thru baseline ...
X
x
x’
Epipolar Geometry: Chapter 8
Basic idea:
• 2 cameras located at C, C’ in world space.
• Find ‘baseline’ through camera centers C, C’
• Baseline hits image planes at ‘epipoles’
• Family of planes thru baseline
are all the ‘epipolar planes’
• Image of planes = lines =
x’
x
‘epipolar lines’
• Lines intersect at
C
epipolar points in both images.
Epipolar Geometry: Chapter 8
X epipolar plane 
x’
x
baseline
C
e
e’
C’
Summary:
• Connect cameras C, C’ with a baseline, which
hits image planes at epipoles e, e’.
• Chose any world pt X, then  everything is coplanar!
epipolar plane  includes image points x, x’, and these
connect to epipoles e,e’ by epipolar lines L, L’
Epipolar Geometry
X
epipolar plane 
x’
x
baseline
C
e
e’
C’
Useful properties:
• Every image point x maps to an epipolar line L’
also:
–
–
–
–
Epipoles e,e’ = each camera’s view of the other
All epipolar lines L’ pass through epipole e’
Epipolar Line L’ is (image of the CX ray...)
Epipolar Line L’ links (image of C) to (image of X)
Fundamental Matrix: Fx = L’
X
epipolar plane 
x’
x
baseline
C
e
e’
C’
One Matrix Summarizes ALL of Epipolar Geometry
Fundamental Matrix F: 3x3, rank 2.
• Maps image point x to image point x’: x’T F x = 0
but F is only Rank 2 – given only x, F cannot find x’ for you!!
• Maps image point x to epipolar line L’:
F x = L’
Fundamental Matrix: Fx = L’
X
epipolar plane 
x’
x
baseline
C
e’
C’
1) How do we find F?
If we know the camera matrices P and P’
(we almost never do), book derives: (pg 224)
F = [e’] P’ P+
?!?!What?!?! point e’ cross product with a matrix p’p+ ?!?!
(Recall: P+ = PT(PPT)-1, the pseudo-inverse)
Fundamental Matrix: Fx = L’
X
epipolar plane 
x’
x
baseline
C
e’
C’
F = [e’] P’ P+ But what’s this? NEW TRICK:
– Cross Product written as matrix multiply (pg. 554)
a  b = a1  b1 = a2b3 – a3b2 = 0 -a3 a2 b1 == [a]·b
a3 0 -a1 b2
a2 b2 a3b1 – a1b3
-a2 a1 0 b3
a3 b3 a1b2 – a2b1
– Note: a  b = -b  a = [a]·b = (aT·[b])T
‘skew symmetric’
matrix
Fundamental Matrix: Fx = L’
X
epipolar plane 
x’
x
baseline
C
e’
C’
• 2) What is F if we DON’T know the cameras P, P’,
but we DO know some corresp. point pairs (x, x’)?
– F finds epipolar line L’ from point x : Fx = L’
– (Recall that if (any) point x’ is on line a L’, then x’T L’= 0)
– Substitute Fx for L’: x’TF x = 0
AHA! we can find F using DLT-like method! see Chap. 10
Fundamental Matrix Summary
(pg. 226)
F is 3x3 matrix, maps P2P2, rank 2, 7-DOF
• If world space pt X  image space pts. x and x’ then
x’TF x = 0
• Every image pt has epipolar line in the other image:
Fx = L’
FTx’ = L
• Baseline pierces image planes at epipoles e, e’ :
Fe = 0
FTe’ = 0
Fundamental Matrix Summary
(pg. 226)
F is 3x3 matrix, maps P2P2, rank 2, 7-DOF
• Given camera matrices P, P’, find F matrix by:
F = [e’] P’ P+
(recall: e’ is image of C: e’ = P’C)
• F is unaffected by any proj. transforms done
on BOTH cameras
(PH, P’H) has same F matrix as (P, P’) for any full-rank H
(e.g. F measures camera C vs. Camera C’ only,
no matter where you put them)
Fundamental Matrix Uses
Special case: camera translate only (no rotations)
• Camera matrices are P= K[ I | 0 ], P’ = K[I | t ]
– where K is internal calib., t is 3D translation vector tx
• F matrix simplifies to F = [e’]
• Epipolar lines are all parallel to direction t
ty
tz
• x,x’ displacement depends only on t & 3D depth z:
x’ = x + (Kt)(1/zc)
Fundamental Matrix Uses
Can we find a camera matrix from motion + fundamental F?
• Let one camera position define the world’s coords:
P = P0 = K[ I | 0 ], and other is
P’ = [M | m] = K[R | -RC] ~
where K is internal calib., R is rotation C is position
~
• F matrix simplifies to F = [m]M
• If we know how we moved the camera (R,C matrices)
then find F by correspondence and solve for K. (pg 237)
• No R, C matrices? Use Essential Matrix (pg 238)
Fundamental Matrix Properties
(pg. 226)
Why bother with F?
• Can find it from image pt. correspondences only
• Works even for mismatched cameras
(example: 100-year time-lapse of Eiffel tower)
• Choose your own world-space coordinate system.
• SVD lets us recover P0, P’ camera matrices from F
(See 8.6 ‘The Essential matrix’pg 240)
• Complete 2-camera mapping from worldimage
– 2 images + corresponding point pairs (xi,x’i)F
– Let camera coords == 3D world coords, then (xi,x’i)Xi
Correspondence Problem:
Where Computer Vision, IBMR part ways:
Fundamental Matrix +
Corresponding point pairs (x,x’) 
How can we
‘blunt’ the correspondence problem?
END