Colorado School of Mines
Computer Vision
Professor William Hoff
Dept of Electrical Engineering &Computer Science
Colorado School of Mines
Computer Vision
http://inside.mines.edu/~whoff/
1
Epipolar Geometry and the Essential
Matrix
Colorado School of Mines
Computer Vision
2
Inferring 3D from 2D
• Possible methods to obtain 3D information:
– Model based pose estimation
-> Can determine
the pose of the
model wrt camera
single
(calibrated)
camera
Known model
– Stereo vision
two
(calibrated)
cameras
Arbitrary
scene
-> Can determine
the positions of
points in the
scene
Relative pose between
cameras is also known
Colorado School of Mines
Computer Vision
3
Inferring 3D from 2D
• Another method
– Structure-from-motion (might be better termed
“structure-and-motion from a moving camera”)
One
(calibrated)
moving
camera
Arbitrary
scene
R,t
Relative pose between
camera positions is unknown
-> Can determine
the positions of
points in the
scene, as well as
the motion of the
camera (R,t)
However, we will see that the positions of points and the
translation of the camera have an unknown scale factor
Colorado School of Mines
Computer Vision
4
Outline
• In this lecture
– First we review the geometry and representation of
epipolar lines
– Next we derive the essential matrix and show how it can
predict the locations of epipolar lines
• In the next lecture
– Show how the essential matrix can be estimated from
point correspondences
– Show how to recover rotation and translation
– Show how to recover point positions
Colorado School of Mines
Computer Vision
5
Epipolar Geometry
• We have two views of a scene, taken from different
viewpoints
• We see an image point p in one image, which is the projection
of a 3D point
p1 ?
p0
Given p0 in the first image, where can the
corresponding point p1 in the second image be?
Colorado School of Mines
Computer Vision
6
P
Image plane
Epipolar Line
p0
p1
Z0
C0
X0
X1
Z1
Y0
C1
Epipole
Colorado School of Mines
Computer Vision
Y1
7
P
p0
p1
C0
C1
• The optical centers of the two cameras, a point P, and the
image points p0 and p1 of P all lie in the same plane (epipolar
plane)
• These vectors are co-planar: C0p 0 , C1p1 , C0C1
Colorado School of Mines
Computer Vision
8
p0
p1
C0
C1
• Another way to write the fact they are co-planar is


C0p 0  C0C1  C1p1  0
Colorado School of Mines
Computer Vision
9
• Now, instead of treating p0 as a point, treat it as a 3D direction
vector*
C
 x0 
 
p 0   y0 
1
 
0
We assume “normalized”
image coordinates; ie
effective focal length=1
• p1 is also a direction vector
This is defined with respect
to the coordinate frame of
camera 0
C1
 x1 
 
p1   y1 
1
 
This is defined with respect
to the coordinate frame of
camera 1
• The direction of p1 in camera 0 coordinates is
C0
C1
*A
Namely, we apply the
rotation matrix from the
camera 1 to camera 0 pose
R p1
direction vector is a vector whose starting point (tail) doesn’t matter, just its direction
Colorado School of Mines
Computer Vision
10
p0
C0
p1
C1
t


C0p 0  C0C1  C1p1  0
• So we can write the coplanar
constraint as
p 0  t  Rp1   0
• Where R is the rotation of
camera 1 wrt camera 0
C0
C1
R
• And t is the translation of the
camera 1 origin wrt camera 0
C0
Colorado School of Mines
• Remember that the
pose of camera 1 wrt
camera 0 is
C0

C0
C1 R

C1 H  
 0
C0
t C1org 

1 
t C1org
Computer Vision
11
Cross Product as Matrix Multiplication
• The cross product of a vector a with a vector b, a x b, can be
represented as a 3x3 matrix times the vector b:
– [a] x b, where [a] x is a skew symmetric matrix
• It is easy to show that
 0

a   a3
 a
 2
 a3
0
a1
a2 

 a1 
0 
(Show this!)
Colorado School of Mines
Computer Vision
12
Cross Product as Matrix Multiplication
• The cross product of a vector a with a vector b, a x b, can be
represented as a 3x3 matrix times the vector b:
– [a] x b, where [a] x is a skew symmetric matrix
• It is easy to show that
 0

a   a3
 a
 2
 a3
0
a1
a2 

 a1 
0 
(Show this!)
Colorado School of Mines
Computer Vision
13
Matrix Form of Epipolar Constraint
• We have
p0  t  Rp1   0
• Or
pT0 t  Rp1  0
• Where [t]x is the 3x3 skew
symmetric matrix for t
corresponding to the cross
product operator
• Let E = [t]x R (which is a 3x3 matrix)
• Then
 E11

pT0 E p1  0
x0 y0 1 E21
E
 31
Colorado School of Mines
Computer Vision
E12
E22
E32
E13  x1 
 
E23  y1   0
E33  1 
14
The Essential Matrix
• Is the matrix E, that relates the image of a point in one camera
to its image in the other camera, given a translation and
rotation
P
pT0 E p1  0
p0
p1
Z0
• where
E = [t]x R
Colorado School of Mines
X0
Y0
X1
Z1
Y1
Computer Vision
15
Example – Create a Scene
• Create some points on the face of cube
• Render image from two views
y
z
{M}
x
• Let pose of cube with respect to camera 1
be
ax=120°, ay=0°, az=60°, tx=3, ty=0, tz=0
• Let pose of camera 2 with respect to
camera 1 be
ax=0°, ay=-25°, az=0°, tx=3, ty=0, tz=1
• Assume XYZ fixed angles
Colorado School of Mines
Computer Vision
16
clear all
close all
L = 300;
% size of image in pixels
I = zeros(L,L);
% Define f, u0, v0
f = L;
u0 = L/2;
v0 = L/2;
% Create the matrix of intrinsic camera parameters
Mint = [ f 0 u0;
0 f v0;
0 0
1];
DEG_TO_RAD = pi/180;
% Create some points on
P_M = [
0
0
0
0
0
2
1
0
2
1
0
0
0
-1 -1
1
1
1
1
1
];
NPTS = length(P_M);
Colorado School of Mines
the face of a cube
0
0
-1
1
0
2
-2
1
0
1
-2
1
0
0
-2
1
1
0
0
1
Computer Vision
2
0
0
1
1
0
-1
1
2
0
-1
1
1
0
-2
1
2;
0;
-2;
1;
17
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Define pose of model with respect to camera1
ax = 120 * DEG_TO_RAD;
ay = 0 * DEG_TO_RAD;
az = 60 * DEG_TO_RAD;
Rx = [ 1
0
0;
0
cos(ax)
-sin(ax);
0
sin(ax)
cos(ax) ];
Ry = [ cos(ay)
0
sin(ay);
0
1
0;
-sin(ay)
0
cos(ay) ];
Rz = [ cos(az)
-sin(az)
0;
sin(az)
cos(az)
0;
0
0
1 ];
R_m_c1 = Rx * Ry * Rz;
Pmorg_c1 = [0; 0; 5];
% translation of model wrt camera
M = [ R_m_c1 Pmorg_c1 ];
Colorado School of Mines
% Extrinsic camera parameter matrix
Computer Vision
18
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Render image 1
p1 = M * P_M;
p1(1,:) = p1(1,:) ./ p1(3,:);
p1(2,:) = p1(2,:) ./ p1(3,:);
p1(3,:) = p1(3,:) ./ p1(3,:);
figure(1), imshow(I, []), title('View 1');
% Convert image points from normalized to unnormalized
u = Mint * p1;
for i=1:length(u)
rectangle('Position', [u(1,i)-2 u(2,i)-2 4 4], 'FaceColor', 'r');
end
pause
Colorado School of Mines
Computer Vision
19
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Set up second view.
% Define rotation of camera1 with respect to camera2
ax = 0 * DEG_TO_RAD;
ay = -25 * DEG_TO_RAD;
az = 0;
Rx = [ 1
0
0;
0
cos(ax)
-sin(ax);
0
sin(ax)
cos(ax) ];
Ry = [ cos(ay)
0
sin(ay);
0
1
0;
-sin(ay)
0
cos(ay) ];
Rz = [ cos(az)
-sin(az)
0;
sin(az)
cos(az)
0;
0
0
1 ];
R_c2_c1 = Rx * Ry * Rz;
% Define translation of camera2 with respect to camera1
Pc2org_c1 = [3; 0; 1];
% Figure out pose of model wrt camera 2.
H_m_c1 = [ R_m_c1 Pmorg_c1 ; 0 0 0 1];
H_c2_c1 = [ R_c2_c1 Pc2org_c1 ;
0 0 0 1];
H_c1_c2 = inv(H_c2_c1);
H_m_c2 = H_c1_c2 * H_m_c1;
R_m_c2 = H_m_c2(1:3,1:3);
Pmorg_c2 = H_m_c2(1:3,4);
% Extrinsic camera parameter matrix
M = [ R_m_c2 Pmorg_c2 ];
Colorado School of Mines
Computer Vision
20
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Render image 2
p2 = M * P_M;
p2(1,:) = p2(1,:) ./ p2(3,:);
p2(2,:) = p2(2,:) ./ p2(3,:);
p2(3,:) = p2(3,:) ./ p2(3,:);
figure(2), imshow(I, []), title('View 2');
% Convert image points from normalized to unnormalized
u = Mint * p2;
for i=1:length(u)
rectangle('Position', [u(1,i)-2 u(2,i)-2 4 4], 'FaceColor', 'r');
end
pause
Colorado School of Mines
Computer Vision
21
Example
View 1
R_m_c1 =
0.5000
-0.4330
0.7500
-0.8660
-0.2500
0.4330
0
-0.8660
-0.5000
Pmorg_c1 =
0
0
5
Colorado School of Mines
Computer Vision
22
View 2
R_c2_c1 =
0.9063
0
0.4226
0
1.0000
0
-0.4226
0
0.9063
Pc2org_c1 =
3
0
1
Colorado School of Mines
Computer Vision
23
Essential matrix
• The essential matrix is E = [t]x R
– where
• [t]x is the skew symmetric matrix corresponding to t
• t is the translation of camera 2 with respect to camera 1; i.e.,
c1P
c2org
• R is rotation of camera 2 with respect to camera 1; i.e., c1c2R
% Calculate essential matrix
t = Pc2org_c1;
E = [ 0 -t(3) t(2); t(3) 0 -t(1); -t(2) t(1) 0] * R_c2_c1;
E =
0
-0.3615
0
Colorado School of Mines
-1.0000
0
3.0000
0
-3.1415
0
Computer Vision
24
Representation of a line
• Equation of a line in the (x,y) plane is ax + by + c = 0
• We get the same line with the equation
(ka)x + (kb)y + (kc) = 0, for any non-zero constant k
• So a line may be represented by the homogeneous
coordinates
l = (a,b,c)T
• Note: The point p lies on the line l if and only if pT l = 0
Colorado School of Mines
Computer Vision
25
Epipolar Lines
P
• Recall
pT0 E p1  0
• So E p1 is the epipolar line
corresponding to p1 in the
camera 0 image
p0
C0
• Or, writing another way,
x0
y0
p1
C1
a
 
1 b   ax0  by0  c  0
c
 
where l = (a,b,c)T = E p1 are the parameters of the line
Colorado School of Mines
Computer Vision
26
Visualization
• Pick a point p1 in the second image
• Calculate the corresponding epipolar line in the first image,
l=E p1
– where l=(a,b,c)
– ax0+by0+c=0 is the equation of the line
• Draw the line on the first image
– Find two points (xa,ya) and (xb,yb) on the line, and draw a line
between them
– Let xa = -1, solve for ya
– Let xb = +1, solve for yb
y = (– c – ax)/b
Colorado School of Mines
Computer Vision
27
% Draw epipolar lines
for i=1:length(p2)
figure(1);
% The product l=E*p2 is the equation of the epipolar line corresponding
% to p2, in the first image. Here, l=(a,b,c), and the equation of the
% line is ax + by + c = 0.
l = E * p2(:,i);
% Let's find two points on this line. First set x=-1 and solve
% for y, then set x=1 and solve for y.
pLine0 = [-1; (-l(3)-l(1)*(-1))/l(2); 1];
pLine1 = [1; (-l(3)-l(1))/l(2); 1];
% Convert from normalized to unnormalized coords
pLine0 = Mint * pLine0;
pLine1 = Mint * pLine1;
line([pLine0(1) pLine1(1)], [pLine0(2) pLine1(2)], 'Color', 'r');
pause
end
Colorado School of Mines
Computer Vision
28
Visualization (continued)
•
•
Similarly, can view the corresponding set of epipolar lines on image 1
Find the essential matrix going the other way; E’ = [t]x R
– where
• [t]x is the skew symmetric matrix corresponding to t
• t is the translation of camera 0 with respect to camera 1; i.e., c1Pc0org
• R is rotation of camera 0 with respect to camera 1; i.e., c1c0R
•
Pick a point p in the first image
•
Calculate the corresponding epipolar line in the second image, l=E’p
– where l=(a,b,c)
– ax+by+c=0 is the equation of the line
•
Draw the line on the second image
– Find two points (xa,ya) and (xb,yb) on the line, and draw a line between them
– Let xa = -1, solve for ya
– Let xb = +1, solve for yb
Colorado School of Mines
Computer Vision
29
Results
View 1
View 2
Epipolar lines corresponding to
points in the second image,
projected onto the first image
Colorado School of Mines
Epipolar lines corresponding to points
in the first image, projected onto the
second image
Computer Vision
30