Colorado School of Mines
Computer Vision
Professor William Hoff
Dept of Electrical Engineering &Computer Science
Colorado School of Mines
Computer Vision
http://inside.mines.edu/~whoff/
1
Fundamental Matrix
Colorado School of Mines
Computer Vision
2
Recall the Essential Matrix
• Is the matrix E, that relates the image of a point in one camera to its image in the other camera, given a translation and rotation
P
pT0 E p1  0
p0
p1
Z0
• where
E = [t]x R
• and
X0
Y0
X1
Z1
Y1
– p0, p1 are corresponding points (normalized image coordinates)
Colorado School of Mines
Computer Vision
3
Fundamental Matrix
• To work with the essential matrix we have to know the intrinsic camera parameter matrix K
– We use p0, p1 which are normalized image coordinates (i.e., x = X/Z, y = Y/Z)
– We find normalized image coords using p = K‐1 u, where u
are the un‐normalized image coords
• If we don’t know the intrinsic parameter matrix
– all we have is the un‐normalized image points
– we can still relate the views – We use the fundamental matrix F
Colorado School of Mines
Computer Vision
4
Fundamental Matrix
• We have
• Let
pT0 E p1  0
p1  K 1u1

pT0  K 1u 0

T
 uT0 K T
• Then
• or
uT0 K T E K 1u1  0
uT0 F u1  0
• where F is the fundamental matrix
F  K T E K 1
• Note
– F is defined in terms of pixel coordinates
– You can still reconstruct the epipolar lines using F
Colorado School of Mines
Computer Vision
Also note that
E  KT F K
5
Example – Create a Scene
•
•
Create some points on the face of cube
Render image from two views
•
Let pose of cube with respect to camera 1 be
y
z
{M}
x
ax=120°, ay=0°, az=60°, tx=3, ty=0, tz=0
•
Let pose of camera 2 with respect to camera 1 be
ax=0°, ay=‐25°, az=0°, tx=3, ty=0, tz=1
•
Assume XYZ fixed angles
Colorado School of Mines
Computer Vision
6
•
The Matlab code to create these points is the same as used earlier in the lecture on the essential matrix
% These are the points in image 1
u1 = [
61.4195 102.1798 150.0000
68.3768
124.4290 136.1955 150.0000 167.2490
1.0000
1.0000
1.0000
1.0000
106.2098
181.1490
1.0000
150.0000
197.2377
1.0000
74.3208
203.8325
1.0000
109.6134
219.1146
1.0000
150.0000
236.6025
1.0000
176.0870
127.4080
1.0000
196.1538
110.0296
1.0000
174.0000
170.7846
1.0000
192.8571
150.0000
1.0000
172.2222
207.7350
1.0000
190.0000;
184.6410;
1.0000 ];
% These are the corresponding points in image 2
u2 = [
45.5272
63.4568
86.9447
61.6620
80.2653
126.5989 136.7293 150.0000 165.9997 180.2731
1.0000
1.0000
1.0000
1.0000
1.0000
104.1468
198.5963
1.0000
75.7981
200.5196
1.0000
94.7507
217.7991
1.0000
118.6606
239.5982
1.0000
135.5451
125.7710
1.0000
176.3357
105.4355
1.0000
147.5739
172.3407
1.0000
184.5258
150.0000
1.0000
157.8633
212.1766
1.0000
191.6139;
188.5687;
1.0000 ];
Colorado School of Mines
Computer Vision
7
Ground truth for F
• Calculating the fundamental matrix using the known R, t, K:
E = [ 0 -t(3) t(2); t(3) 0 -t(1); -t(2) t(1) 0] * R_c2_c1;
% Fundamental matrix
F = inv(K)'*E*inv(K);
disp('True F:'); disp(F);
• Results
True F:
0 ‐0.0000 0.0017
‐0.0000 0 ‐0.0099
0.0006 0.0117 ‐0.2696
Colorado School of Mines
Computer Vision
8
Solving for F
• We solve for F using the same methods as we used to solve for E
– Except the corresponding points are in un‐normalized coordinates
• We have
x0
uT0 F u1  0
y0
 F11

1 F21
F
 31
F12
F22
F32
F13  x1 
 
F23  y1   0
F33  1 
• Write as A x = 0, where x = (F11, F12, F13, … , F33)
x0 x1
Colorado School of Mines
x0 y1
x0
y0 x1
Computer Vision
y0 y1
y0
x1
y1
 F11 
 
 F12 
1 F13   0
 
  
F 
 33 
9
Solving for F
• We have A x = 0
– This is a system of homogeneous equations
– We solve using singular value decomposition
• As we did earlier, we will do:
– Preconditioning: We will first translate and scale the data points so they are centered at the origin and the average distance to the origin is √2
– Postconditioning: The values of F are not independent. There are only five independent parameters. F must have rank=2 … we will enforce this
Colorado School of Mines
Computer Vision
10
Complete Code for computing Fundamental Matrix (1)
% Calculate the essential matrix.
Read images and corresponding points
clear all
close all
K = [ 300 0 150;
0 300 150;
0 0
1];
% intrinsic camera parameters
% These are the points in image 1
u1 = [
61.4195 102.1798 150.0000
68.3768
124.4290 136.1955 150.0000 167.2490
1.0000
1.0000
1.0000
1.0000
106.2098
181.1490
1.0000
150.0000
197.2377
1.0000
74.3208
203.8325
1.0000
109.6134
219.1146
1.0000
150.0000
236.6025
1.0000
176.0870
127.4080
1.0000
196.1538
110.0296
1.0000
174.0000
170.7846
1.0000
192.8571
150.0000
1.0000
172.2222
207.7350
1.0000
190.0000;
184.6410;
1.0000 ];
% These are the corresponding points in image 2
u2 = [
45.5272
63.4568
86.9447
61.6620
80.2653
126.5989 136.7293 150.0000 165.9997 180.2731
1.0000
1.0000
1.0000
1.0000
1.0000
104.1468
198.5963
1.0000
75.7981
200.5196
1.0000
94.7507
217.7991
1.0000
118.6606
239.5982
1.0000
135.5451
125.7710
1.0000
176.3357
105.4355
1.0000
147.5739
172.3407
1.0000
184.5258
150.0000
1.0000
157.8633
212.1766
1.0000
191.6139;
188.5687;
1.0000 ];
I1 = imread('I1.tif');
I2 = imread('I2.tif');
% Display points on the images for visualization
imshow(I1, []);
for i=1:length(u1)
x = round(u1(1,i));
y = round(u1(2,i));
rectangle('Position', [x-4 y-4 8 8], 'EdgeColor', 'r');
text(x+4, y+4, sprintf('%d', i), 'Color', 'r');
end
figure, imshow(I2, []);
for i=1:length(u2)
x = round(u2(1,i));
y = round(u2(2,i));
rectangle('Position', [x-4 y-4 8 8], 'EdgeColor', 'r');
text(x+4, y+4, sprintf('%d', i), 'Color', 'r');
end
Colorado School of Mines
Display images and points
Computer Vision
11
Complete Code for computing Fundamental Matrix (2)
% Get unnormalized image points
p1 = u1;
p2 = u2;
Don’t normalize points
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Scale and translate image points so that the centroid of
% the points is at the origin, and the average distance of the points to the
% origin is equal to sqrt(2).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
xn = p1(1:2,:);
% xn is a 2xN matrix
N = size(xn,2);
t = (1/N) * sum(xn,2);
% this is the (x,y) centroid of the points
xnc = xn - t*ones(1,N);
% center the points; xnc is a 2xN matrix
dc = sqrt(sum(xnc.^2));
% dist of each new point to 0,0; dc is 1xN vector
davg = (1/N)*sum(dc);
% average distance to the origin
s = sqrt(2)/davg;
% the scale factor, so that avg dist is sqrt(2)
T1 = [s*eye(2), -s*t ; 0 0 1];
p1s = T1 * p1;
Scale and translate points
xn = p2(1:2,:);
% xn is a 2xN matrix
N = size(xn,2);
t = (1/N) * sum(xn,2);
% this is the (x,y) centroid of the points
xnc = xn - t*ones(1,N);
% center the points; xnc is a 2xN matrix
dc = sqrt(sum(xnc.^2));
% dist of each new point to 0,0; dc is 1xN vector
davg = (1/N)*sum(dc);
% average distance to the origin
s = sqrt(2)/davg;
% the scale factor, so that avg dist is sqrt(2)
T2 = [s*eye(2), -s*t ; 0 0 1];
p2s = T2 * p2;
%
%
%
%
A
Compute fundamental matrix F from point correspondences.
We know that p1s' F p2s = 0, where p1s,p2s are the scaled image coords.
We write out the equations in the unknowns F(i,j)
A x = 0
= [p1s(1,:)'.*p2s(1,:)'
p1s(1,:)'.*p2s(2,:)' p1s(1,:)' ...
p1s(2,:)'.*p2s(1,:)'
p1s(2,:)'.*p2s(2,:)' p1s(2,:)' ...
p2s(1,:)'
p2s(2,:)' ones(length(p1s),1)];
Compute F
% The solution to Ax=0 is the singular vector of A corresponding to the
% smallest singular value; that is, the last column of V in A=UDV'
[U,D,V] = svd(A);
x = V(:,size(V,2));
% get last column of V
% Put unknowns into a 3x3 matrix. Transpose because Matlab's "reshape"
% uses the order F11 F21 F31 F12 ...
Fscale = reshape(x,3,3)';
% Force rank=2
[U,D,V] = svd(Fscale);
Fscale = U*diag([D(1,1) D(2,2) 0])*V';
Force F to have rank 2
% Undo scaling
F = T1' * Fscale * T2;
Undo scaling and translation
disp('Calculated fundamental matrix:');
disp(F);
save F
Colorado School of Mines
Computer Vision
12
Results
•
Run program “fundamental.m”
– This inputs the corresponding points, and calculates the fundamental matrix
•
Verify that calculated fundamental matrix equals the “true” fundamental matrix (to within a scale factor)
True F:
0 ‐0.0000 0.0017
‐0.0000 0 ‐0.0099
0.0006 0.0117 ‐0.2696
Calculated fundamental matrix:
‐0.0000 ‐0.0000 0.0029
‐0.0000 0.0000 ‐0.0172
0.0011 0.0203 ‐0.4703
Colorado School of Mines
Computer Vision
13
Epipolar Lines
• Run program “drawepipolarFund.m”
– This inputs a pair of images, a set of corresponding points, and a fundamental matrix
– It draws epipolar lines in the images
View 2
View 1
Colorado School of Mines
Computer Vision
14
Residual Error
• For each image point , the corresponding point in the other image should ideally lie exactly on the epipolar line • If there is noise, the residual error = distance from the actual point to the epipolar line
to line with • Distance from point parameters is
∗
See http://mathworld.wolfram.com/Point‐LineDistance2‐Dimensional.html
Colorado School of Mines
Computer Vision
15
Matlab code to compute residuals
% Get error residuals for all points, using the final F.
dp = zeros(N,1);
for i=1:N
% The product l=F*p2 is the equation of the epipolar line
corresponding
% to p2, in the first image. Here, l=(a,b,c), and the equation of the
% line is ax + by + c = 0.
x2 = pts2(i,:)';
% Point in second image
l = F * [x2;1];
% Epipolar line in first image
% The equation of the line is ax + by + c = 0.
% The distance from a point p1=(x1,y1,1) to a line with parameters
% l=(a,b,c) is
d = abs(p1' * l)/sqrt( a^2 + b^2 )
% (see
% http://mathworld.wolfram.com/Point-LineDistance2-Dimensional.html)
x1 = pts1(i,:)';
% Point in first image
dp(i) = abs(([x1;1]' * l))/sqrt( l(1)^2 + l(2)^2 );
end
Colorado School of Mines
Computer Vision
16
Reconstruction
• With the essential matrix we could reconstruct the scene points to a scale factor (Euclidean reconstruction)
• We can’t do Euclidean reconstruction with the fundamental matrix; however we can do a projective reconstruction
– Orthogonal lines or planes in the world may not end up being reconstructed as orthogonal
http://www.cse.iitd.ernet.in/~suban/vision/multiple/node11.html
Colorado School of Mines
Computer Vision
17