CSE 6367
Computer Vision
Stereo Reconstruction
Camera Coordinate Transformations
Farhad Kamangar
Computer Science and Engineering Department
The University of Texas at Arlington
“Everything should be made as simple as possible, but not simpler.” Albert Einstein
Stereo Constraints (Review)
p
Given p in left image, find the corresponding point p’
in right image be?
p’ ?
Stereo
P
Epipolar Line
Image plane
Y1
p’
p
Y2
Z1
O1
X1
X2
Focal plane
O2
Epipole
Z2
Stereo
P
Epipolar Line
Image plane
Y1
p’
p
Y2
Z1
O1
X1
X2
Focal plane
O2
Epipole
Z2
All epipolar lines pass through the epipole
Stereo
P
Epipolar Line
Image plane
Y1
p’
p
Y2
Z1
O1
X1
X2
Focal plane
O2
Epipole
Z2
All epipolar lines pass through the epipole
Stereo
P
Epipolar Line
Image plane
Y1
p’
p
Y2
Z1
O1
X1
Focal plane
T
X2
O2
Epipole
Z2
Stereo
P
Epipolar Line
Image plane
Y1
p’
p
Y2
Z1
O1
X1
Focal plane
T
X2
O2
Epipole
Z2
Stereo
P
Epipolar Line
Image plane
Y1
p’
p
Y2
O1
Z1
X1
Focal plane
T=O2-O1
O1 p  O1O2  O2 p '  0
T
T
X2
O2
Epipole
Z2
O1, O2, p, and p’ are on the same plane
Stereo
P
Epipolar Line
Image plane
Y1
p’
p
Y2
Z1
O1
X1
Focal plane
O1 p T  O1O2  O2 p '  0
p  T  Rp'  0
T
T
X2
O2
Epipole
Z2
O1, O2, p, and p’ are on the same plane
Stereo
P
Epipolar Line
Image plane
Y1
p’
p
Y2
Z1
O1
X1
Focal plane
O1 p T  O1O2  O2 p '  0
T
X2
O2
Epipole
Z2
p  T  Rp'  0 p T  E  p '  0 E  T  R
T
E is Essential matrix
The Essential Matrix
p  E  p'  0
T
p  (u, v,1)
U and v are the coordinates of point p in image plane
a 


E  p '  b 
 c 
au  bv  c  0
Equation of epipolar line which corresponds to point p
The Essential Matrix
• Essential Matrix:
• Based on the Relative Geometry of the Cameras
• Cameras are assumed to be calibrated
• Has five independent parameters
Fundamental Matrix
p  E  p'  0
p and p are in camera coordinate system
T
Assuming that the image coordinates of points p and p’ are u and u’
u  C1 p
1
1
pC u
u '  C2 p '
T
T
1
u C
1
2
p'  C u '
1
2
 E  C u'  0
uT  F  u'  0
T
1
F C
F is the fundamental matrix
 E C
1
2
Fundamental Matrix
u  F  u'  0
T
Fundamental Matrix, F, is a 3 by 3 matrix (singular with rank 2)
F has 7 parameters up to scale and can be estimated from 7 point correspondences
Direct Simpler Method requires 8 correspondences
Estimating Fundamental Matrix
uT  F  u'  0
 F11
u v 1 F21
 F31
F12
F22
F32
F13  u
F23   v  0
F33   1 
 F11 
F 
 12 
 F13 
 
 F21 
uu uv u uv vv v u v 1 F22   0
 
 F23 
F 
 31 
 F32 
F 
 33 
The 8-point Algorithm
Camera Parameters
•
Assume a camera (any camera) where the Projection
Reference Point (PRP), is located at:
•
Normal vector to the projection plane (View Plane
Normal VPN) is given as:
•
The View Up (VUP) vector is given as:
 prp x 
 prp 
y
PRP  
 prp z 


1


vpnx 
vpn 
VPN   y 
 vpnz 


1


vupx 
vup 
y
VUP  
 vupz 


 1 
Transform Camera to World
• The general form of the matrix that transforms the camera
coordinate system to the world coordinate system is,
M cw
 r1,1 r1,2

r2,1 r2,2


 r3,1 r3,2

0
 0
r1,3 T1,4 

r2,3 T2,4 
r3,3 T3,4 

0
1 
Two Cameras
•
•
Now assume we have two cameras, left camera and right camera
For left camera:
 prplcx 
 prp 
lcy 
PRPlc  
 prplcz 


1


•
 vpnlcx 
vpn 
VPN lc   lcy 
 vpnlcz 


 1 
 vuplcx 
vup 
VUPlc   lcy 
 vuplcz 


 1 
 vpnrcx 
vpn 
VPN rc   rcy 
 vpnrcz 


 1 
vuprcx 
vup 
VUPrc   rcy 
 vuprcz 


 1 
For right camera:
 prprcx 
 prp 
rcy 
PRPrc  
 prprcz 


1


Point in the World
• Assume a point in the homogenous world coordinate system:
 pwx 
p 
Pw   wy 
 pwz 
 
 1 
• The coordinates of this point in the left camera coordinate
system is:
 plcx 
p 
Plc   lcy 
 plcz 
 
 1 
• The coordinates of this point in the right camera coordinate
system is:
 prcx 
p 
Prc   rcy 
 prcz 


 1 
Left Camera to World
• The matrix that transforms the left camera coordinate system
to the world coordinate system
M lcw
 r1,1lcw
 lcw
r2,1

 lcw
 r3,1

 0
lc  w
r1,2
lc  w
r2,2
lc  w
3,2
r
0
r1,3lc w T1,4lc w 
lc  w
lc w 
r2,3
T2,4 
lc  w
lc w 
r3,3
T3,4

0
1 
Right Camera to World
• The matrix that transforms the right camera coordinate system
to the world coordinate system
M rcw
 r1,1rcw
 rcw
r2,1

 rcw
 r3,1

 0
r1,2rcw
rc  w
r2,2
rc  w
3,2
r
0
r1,3rc w T1,4rc w 
rc  w
rc  w 
r2,3
T2,4 
rc  w
rc  w 
r3,3
T3,4

0
1 
Right Camera to Left Camera
• The matrix that transforms the right camera coordinate system
to the left camera coordinate system
M rclc
 r1,1rclc
 rclc
r2,1

 rclc
 r3,1

 0
r1,2rc lc
rc lc
r2,2
rc lc
3,2
r
0
r1,3rc lc T1,4rc lc 
rc lc
rc lc 
r2,3
T2,4 
rc lc
rc lc 
r3,3
T3,4

0
1 
• Given point P in the world coordinate system, the coordinate
of this point in the left camera coordinate system will be:
M lc  w
 r1,1lc  w
 lc  w
r2,1

 lc  w
 r3,1

 0
lc  w
1,2
lc  w
2,2
lc  w
3,2
r
r
r
Plc   M lc  w  Pw
0
lc  w
1,3
lc  w
2,3
lc  w
3,3
r
r
r
0





1 
lc  w
1,4
lc  w
2,4
lc  w
3,4
T
T
T
• Assuming that the focal length of the camera is f,
• Assume that the image of the point P on the left image plane
is P’. The coordinate of the point P’ in the left camera
coordinate system will be:
P 'lc
 plcx
f

 p 'lcx   plcz
p'   p
lcy
  lcy    f
 p 'lcz   plcz

 
f
 1 


 1










• Assume that the image of the point P on the right image plane
is P’’. The coordinate of the point P’’ in the right camera
coordinate system will be:

f

 p ''rcx  
 p ''  
P ''rc   rcy    f
 p ''rcz  

 
 1 


prcx 
prcz 

prcy 

prcz 
f 

1 
• Problem: given all the camera parameters and only the x and y
coordinates of the points P’ and P’’ , find the coordinates of
the point P in the world coordinate system. In other words;
given the projections of the point P on the left and right image
planes, find the coordinates of the original point P.
Plc  M lc  w Pw
Prc  M rc w Pw
Prc  M rclc Plc
M rclc  M rc  w ( M lc  w ) 1
From Left Camera to Right Camera
M lc rc
 r1,1lc rc
 lc rc
r2,1

 lc rc
 r3,1

 0
 prcx   r1,1lc rc
 p   lc rc
 rcy    r2,1
lc  rc
 prcz   r3,1

 
 1   0
lc  rc
r1,2
lc  rc
r2,2
lc  rc
r3,2
0
r1,3lc rc T1,4lc rc 
lc  rc
lc  rc 
r2,3
T2,4

lc  rc
lc  rc 
r3,3
T3,4

0
1 
lc  rc
r1,2
lc  rc
r2,2
lc  rc
r3,2
0
r1,3lc rc T1,4lc rc   plcx 

lc  rc
lc  rc  
p
r2,3
T2,4   lcy 
lc  rc
lc  rc 
 plcz 
r3,3
T3,4
 
0
1   1 
Calculate Camera Coordinates (1)
lcrc
prcx  r1,1lcrc plcx  r1,2
plcy  r1,3lcrc plcz  T1,4lcrc
lc  rc
lc  rc
lc  rc
lc  rc
prcz  r3,1
plcx  r3,2
plcy  r3,3
plcz  T3,4
p 'lcx
plcx  plcz
f
p 'lcy
plcy  plcz
f
prcz
prcx  prcz
prcy  prcz
p ''rcx
f
p ''rcy
f
p 'lcy
p ''rcx
p 'lcx
lc rc
lc rc
 r1,1 plcz
 r1,2 plcz
 r1,3lcrc plcz  T1,4lc rc
f
f
f
lc rc
3,1
prcz  r
plcz
p 'lcy
p 'lcx
lc rc
lc rc
lc rc
 r3,2 plcz
 r3,3
plcz  T3,4
f
f
Calculate Camera Coordinates (2)
p 'lcy
p 'lcx
lc  rc
lc  rc
lc  rc p ''rcx
(r
plcz
 r3,2 plcz
 r3,3
plcz  T3,4
)

f
f
f
p 'lcy
p 'lcx
lc  rc
lc  rc
r1,1 plcz
 r1,2 plcz
 r1,3lcrc plcz  T1,4lcrc
f
f
lc  rc
3,1
p 'lcy p ''rcx
p 'lcy
p 'lcx p ''rcx
p'
lc  rc
lc  rc p ''rcx
 r3,2
 r3,3
 r1,1lcrc lcx  r1,2lcrc
 r1,3lcrc ) 
f
f
f
f
f
f
f
lc  rc p ''rcx
T1,4lcrc  T3,4
f
lc  rc
plcz (r3,1
Calculate Camera Coordinates (3)
lc rc
1,4
T
plcz 
lc rc
3,4
T
p ''rcx
f
p 'lcy
p 'lcy lc rc
p ''rcx  lcrc p 'lcx
lc rc
lc rc 
lc rc p 'lcx
lc rc
 r3,2
 r3,3   r1,1
 r1,2
 r1,3
 r3,1
f 
f
f
f
f

plcx  plcz
p 'lcx
f
plcy  plcz
p 'lcy
f