Computer Vision, Robert Pless
Lecture 2, Pinhole Camera Model
Last time we talked through the pinhole projection.
• This time we are going to look at the coordinate
systems in the pinhole camera model.
Computer Vision, Robert Pless
Some context
Our goal is to understand the process of stereo vision, which
requires understanding:
• How cameras capture the world
• Representing where cameras are in the world (and
relative to each other)
• Finding similar points on each image
Computer Vision, Robert Pless
“Normalized Camera”
• Pinhole at (0,0,0), point P at (X,Y,Z)
• Virtual film at Z = 1 plane.
• Point X,Y,Z is imaged at intersection of:
– Line from (0,0,0) to (X,Y,Z), and
– the Z = 1 plane
X 
P   Y 
 Z 
• Intersection point (x,y), has coordinates
–
(X/Z, Y/Z, 1)
• This is called the “normalized camera
model”
• What happens when focal length is
not 1?
Computer Vision, Robert Pless
 x
p 
 y
Camera Center of Projection
Pinhole,
(Center of
projection),
at (0,0,0)
(X,Y,Z)
f
Z-axis
Image plane
Computer Vision, Robert Pless
…but the cow is not a sphere
We don’t just want to know where, mathematically, the point ends
up on the image plane… we want to know the pixel coordinates of
the point.
Why might this be hard?!
Computer Vision, Robert Pless
Intrinsic Camera Parameters
The pixels may be rectangular:
x = aX/Z
Y = bY/Z
CCD sensor array
Computer Vision, Robert Pless
Intrinsic Camera Parameters
Chip not centered, or 0,0 may not be center of the chip.
x = aX/Z + x0
Y = bY/Z + y0
CCD sensor array
Computer Vision, Robert Pless
Intrinsic Camera Parameters
cheap CMOS chip
cheap lens
image
cheap
glue
cheap camera
x = aX/Z – a cot(q) Y/Z + x0
y=
b/sin(q) Y/Z + y0
Computer Vision, Robert Pless
Ugly relationship
between (X,Y,Z) and
(x,y).
Homogeneous Coordinates
• Represent 2d point using 3 numbers, instead of the usual 2
numbers.
• Mapping 2d 2d homogeneous:
– (x,y)  (x,y,1)
– (x,y)  (sx, sy, s)
• Mapping 2d homogeneous to 2d
– (a,b,w)  (a/w, b/w)
• The coordinates of a 3D point (X,Y,Z) *already is* the 2D
(but homogeneous) coordinates of its projection.
• Will make some things simpler and linear.
Computer Vision, Robert Pless
Why?!
x = aX/Z – a cot(q) Y/Z + x0
y=
b/sin(q) Y/Z + y0
 x   a
 
 y   0
1  
  0
 a cot (q )
b
sin(q )
0
x0  X 
 
y0  Y 
 
1  Z 
Computer Vision, Robert Pless
 x   fx
  
 y   0
1   0
  
s
fy
0
x0  X 
 
y0  Y 
1  Z 
Homogenous Coordinates!
• Equation of a line on the plane?
• Equation of a plane through the origin in 3D?
Computer Vision, Robert Pless
 x   fx
  
 y   0
1   0
  
s
fy
0
x0  X 
 
y0  Y 
1  Z 
Given enough examples of 3D points and their 2D
projections, we can solve for the 5 intrinsic parameters…
Example calibration object…
Computer Vision, Robert Pless
Practical Linear Algebra 1:
• Given many examples of the
equation at the right, solve
for the matrix of values fx,
fy, s, x0, y0
 x   fx
  
 y   0
1   0
  
• How can we solve for this?
Computer Vision, Robert Pless
s
fy
0
x0  X 
 
y0  Y 
1  Z 
… and all this assumes that we
know X,Y,Z
 x   fx
  
 y   0
1   0
  
s
fy
0
x0  X 
 
y0  Y 
1  Z 
Given enough examples of 3D points and their 2D
projections, we can solve for the 5 intrinsic parameters…
Exactly Known!
(0,0,0)
Computer Vision, Robert Pless
 x   fx
  
 y   0
1   0
  
s
fy
0
x0  X 
 
y0  Y 
1  Z 
More commonly, we can put a collection of points (usually
in a grid) in the world with known relative position, but
with unknown position relative to the camera.
Then we need to solve for the intrinsic calibration
parameters, and the extrinsic parameters (the unknown
relative position of the grid).
How do we represent the unknown relative position of the
grid?
Computer Vision, Robert Pless
The grid defines its own coordinate system
That coordinate system has a
Euclidean transformation that
Z
relates its coordinates to the
camera coordinates… that is,
there is some R, T such that:
Y
X
 X grid 
X


 
 Y   R  Ygrid   T


Z 


 
 Z grid 
With enough example
 X grid 


 X   . . . Tx 
points, could we solve for
  
 Ygrid 
 Y    . R .Ty 

R,T?
Z
 Z   . . . T  grid 
z 
  
1

ComputerVision, Robert Pless
An aside…Representation of Rotation with a Rotation Matrix
Rotation matrix R has two properties:
Property 1:
R is orthonormal, i.e.
In other words,
Property 2:
R RI
T
R T  R 1
Determinant of R is 1.
Therefore, although the following matrix is orthonormal,
it is not a rotation matrix because its determinant is -1.
1 0 0


 0 1 0
0 0 1


this is a reflection matrix
Computer Vision, Robert Pless
Representation of Rotation -- Euler Angles
• Euler angles
• pitch: rotation about x axis :
• yaw:
rotation about y axis:
• roll:
rotation about z axis:
Z



X
~
 X C   cos sin  0  cos 0 sin   1
0
~  


C
 Y     sin  cos 0 
1
0  0 cos
 ~ C  
  sin  0 cos  0  sin
Z
0
0
1



 
Computer Vision, Robert Pless
0  X W 
 W 
sin  Y 
cos  Z W 
Y
Putting it all together…
 x   fx
  
 y   0
1   0
  
 x   fx
  
 y   0
1   0
  
s
fy
0
s
fy
0
x0  X 
 
y0  Y 
1  Z 
+
 X grid 

 X   . . . Tx 
  
 Ygrid 
 Y    . R .Ty 

 Z   . . . T  Z grid 
z 
  
1



 X grid 

x0  . . .Tx 

 Ygrid 
y0  . R .Ty 

Z grid 




1  . . . Tz 
1



 X grid 


x
 
 Ygrid 
 y   P

1 
 Z grid 
 
1



P is a 4X3 matrix, that defines projection of points
(used in graphics). P has 11 degrees of freedom, (the
scale of the matrix doesn’t matter). 5 intrinsic
parameters, 3 rotation, 3 translation
Computer Vision, Robert Pless
• Given many examples of:
– World points (X,Y,Z), and
– Their image points (x,y)
• Solve for P.
• Then
 . . . Tx 


P   . KR .Ty 
. . .T 
z 

Rotation +
intrinsic, all
mixed up.
translation
Computer Vision, Robert Pless
 X world 


x
 
 Yworld 
 y   P

Z
world 
1 

 
1



 . . . Tx 


P   . KR .Ty 
. . .T 
z 

Then take the QR
decomposition of
this part of the
matrix to get the
rotation and the
intrinsic
parameters.
Computer Vision, Robert Pless
(from mathworld).
Computer Vision, Robert Pless
Other common trick, use many planes, then you have to
solve for multiple different R,T, one for each plane…
Counting:
For each grid, we have 3 unknowns (translation),
3 unknowns (rotation).
+ 5 extra unknowns defining the intrinsic calibration.
Method. For a collection of planes, calculate the
homography between the image and the real world
plane.
Computer Vision, Robert Pless
Computer Vision, Robert Pless
Computer Vision, Robert Pless
Computer Vision, Robert Pless
Computer Vision, Robert Pless
Computer Vision, Robert Pless
Corners..
Computer Vision, Robert Pless
Computer Vision, Robert Pless
Computer Vision, Robert Pless
What is left?
Barrel and Pincushion Distortion
wideangle
tele
Computer Vision, Robert Pless
What other calibration is there?
Computer Vision, Robert Pless
Models of Radial Distortion
 x   xd 
2
4
    (1  k1r  k 2 r )
 y   yd 
distance from center
Computer Vision, Robert Pless
More distortion.
Computer Vision, Robert Pless
Distortion Corrected…
Computer Vision, Robert Pless
Applications?
Computer Vision, Robert Pless
Cheap applications
 x   fx
  
 y   0
1   0
  
s
fy
0
 X grid 

x0  . . .Tx 

 Ygrid 
y0  . R .Ty 

Z grid 




1  . . . Tz 
1



 X grid 


x
 
 Ygrid 
 y   P

1 
 Z grid 
 
1



• Can you use this? If you know the 3D coordinates of a virtual
point, then you can draw it on your image…
• Often hard to know the 3D coordinate of a point; but there are
some (profitable) special cases…
Computer Vision, Robert Pless
Applications
First-down line
courtesy of Sportvision
Computer Vision, Robert Pless
Applications
Virtual advertising
courtesy of Princeton Video Image
Computer Vision, Robert Pless
Recap
Three main concepts:
1) Projection,
2) Homogeneous coordinates
3) Camera calibration matrix
Computer Vision, Robert Pless
Computer Vision, Robert Pless
Another Special Case,
the world is a plane.
• Projection from the world to the image:
 x   fx
  
 y   0
1   0
  
s
fy
0
x0 

y0 
1 
R
 X grid 

Tx 
 Ygrid 
Ty 

Z grid 


Tz 
1



0
Irrelevant
• Ignore z-coordinate (it is 0 anyway),
drop the 3rd column of the
P matrix, then you get a mapping between the plane and the
image which is an arbitrary 3 x 3 matrix. This is called a
“homography”.
Computer Vision, Robert Pless
Homography:
(x’,y’,1) ~ H (x,y,1)
• Not a constraint. Unlike the fundamental matrix, tells you exactly
where the point goes.
• Point (x,y) in one frame corresponds to point (x’,y’) in the other
frame. If we need to think about multiple points, we may put
subscripts on them.
• Being careful about the homogenous coordinate, we write:
 x   wx ' 
  
H  y    wy ' 
1   w 
 
Computer Vision, Robert Pless
Homography is a “simple” example of a
3D to 2D transformation
It is also the “most complicated” linear 2D to 2D
transformation.
What other 2D  2D transformations are there?
Computer Vision, Robert Pless
Homography is most general,
encompasses other transformations
Projective
8 dof
Affine
6 dof
Similarity
4 dof
Euclidean
3 dof
 h11 h12
h
 21 h22
 h31 h32
 a11 a12
a
 21 a22
 0
0
h13 
h23 
h33 
tx 
t y 
1 
 sr11 sr12 t x 
 sr

sr
t
22
y
 21
 0
0
1 
 r11 r12 t x 
r

r
t
21
22
y


 0 0 1 
Computer Vision, Robert Pless
Views of a plane from different
viewpoints, any view of a scene
from the same viewpoint.
Images of a “far away” object
under any rotation
Camera looking at an assembly
line w/ zoom.
Camera looking at an assembly
line.
Invariants…
Projective
8 dof
 h11
h
 21
 h31
Affine
6 dof
 a11
a
 21
 0
Similarity
4 dof
 sr11
 sr
 21
 0
Euclidean
3 dof
 r11
r
 21
 0
h12
h22
h32
a12
a22
0
sr12
sr22
0
r12
r22
0
h13 
h23 
h33 
Concurrency, collinearity, order of
contact (intersection, tangency,
inflection, etc.), cross ratio
tx 
t y 
1 
Parallellism, ratio of areas, ratio
of lengths on parallel lines (e.g
midpoints).
tx 
t y 
1 
tx 
t y 
1 
Ratios of any edge lengths, angles.
Absolute lengths, areas.
Computer Vision, Robert Pless
Image registration
Determining the 2d transformation that brings one
image into alignment (registers it) with another.
Computer Vision, Robert Pless
Image Warping
•
What kind of warps are these?
 0.6101  0.1918 34.2411
 0.1680 0.6000 26.7034


  0.0011  0.0010
1 
Computer Vision, Robert Pless
How to solve for these mappings?
Given:
Solve for:
 0.6101  0.1918 34.2411
 0.1680 0.6000 26.7034


  0.0011  0.0010
1 
Computer Vision, Robert Pless
Unwrapping a matrix.
 x   wx ' 
  
H  y    wy ' 
1   w 
 
a
d

 g
b
e
h
c  x   wx ' 
  


f  y    wy ' 




1 1   w 
Write out the lines of this matrix equation.
ax  by  c  wx'
dx  ey  f  wy '
gx  hy  1  w
And remember whichComputer
variables
are unknown.
Vision, Robert Pless
Unwrapping a matrix.
a
d

 g
b
e
h
c  x   wx ' 
  


f  y    wy ' 




w

1  1  

ax  by  c  wx
'
i
dx  ey  f  wy
'
i
gx  hy  1  w
ax  by  c  0d  0e  0 f  0g  0h  wxi'
0a  0b  0c  dx  ey  f  0g  0h  wyi'
0a  0b  0c  0d  0e  0 f  gx  hy  1  w
ax  by  c  0d  0e  0 f  0g  0h  wx'  0
0a  0b  0c  dx  ey  f  0g  0h  wy '  0
0a  0b  0c  0d  0eComputer
 0 fVision,
 gx
hy  w  1
Robert
Pless
a
d

 g
b
e
h
c  x   wx ' 
  


wy
'
f  y   


1 1   w 
ax  by  c  0d  0e  0 f  0g  0h  wx'  0
0a  0b  0c  dx  ey  f  0g  0h  wy '  0
0a  0b  0c  0d  0e  0 f  gx  hy  w  1
a 
Looks like another matrix equation:
b 
 
c 
'  
 x y 1 0 0 0 0 0  xi  d  0 

0 
'  
0
0
0
x
y
1
0
0

y
e

i

   
0 0 0 0 0 0 x y  1   f   1


g 
 
h 
w
Computer Vision, Robert Pless
 
a
d

 g
b
e
h
c  x   wx ' 
  


wy
'
f  y   


1 1   w 
ax  by  c  0d  0e  0 f  0g  0h  wx'  0
0a  0b  0c  dx  ey  f  0g  0h  wy '  0
0a  0b  0c  0d  0e  0 f  gx  hy  w  1
Looks like another matrix equation:
Data from
different
points
x

0
0

x
0

0

x
0

0

x
0

0
0 0 0   a  0 

0 x y 1 0 0  yi' 0 0 0 b  0 
   

0 0 0 0 x y  1 0 0 0 c   1
   
'
1 0 0 0 0 0 0  xi 0 0 d  0 
0 x y 1 0 0 0  yi' 0 0 e  0 
   
0 0 0 0 x y 0  1 0 0  f   1
    
'
0
1 0 0 0 0 0 0 0  xi 0 g
   
0 x y 1 0 0 0 0  yi' 0 h  0 
   
0 0 0 0 x y 0 0  1 0  w   1

1 0 0 0 0 0 0 0 0  xi'   w2 0 
   
'
0 x y 1 0 0 0 0 0  yi   w3  0 

0Computer
0 0Vision,
0 Robert
x yPless0 0 0  1   w4  1
y 1 0 0 0 0 0
0
0
y
0
0
y
0
0
y
0
0
 xi'
a
d

 g
b
e
h
c  x   wx ' 
  


wy
'
f  y   


1 1   w 
Why does this suck?
• Maybe you have error in finding corresponding points, and want
to use many many corresponding points. Then your number of
unknowns keeps growing…
• Is there a better way?
• What is x’?
Computer Vision, Robert Pless
The game of “finding the linear constraint…”
x’ = wx’ / w
ax  by  c  wx
gx  hy  1  w
a
d

 g
'
b
e
h
ax  by  c
x 
gx  hy  1
'
x' ( gx  hy 1)  ax  by  c
Computer Vision, Robert Pless
c  x   wx ' 
  


f  y    wy ' 
1 1   w 
Non-linear
Linear (in a,b,c,g,h)
x' ( gx  hy 1)  ax  by  c
gxx'  hyx'  x '  ax  by  c
gxx  hyx  ax  by  c   x
'
'
'
 ax  by  c  gxx'  hyx'   x '
 ax  by  c  0d  0e  0 f  gxx'  hyx'   x '
0a  0b  0c  dx  ey  f  gxy  hyy   y
'
Computer Vision, Robert Pless
'
'
 ax  by  c  0d  0e  0 f  gxx i'  hyx i'   xi'
0a  0b  0c  dx  ey  f  gxy i'  hyy i'   yi'
0 0 xx'
 x  y  1 0
0
0
0  x  y 1 xy'


yx'
yy'
a 
b 
 
c 
 
 d    x' 
 e   y '
    
f
g 
 
h 
And just add two more rows for each corresponding point
Computer Vision, Robert Pless
0 0 xx'
 x  y  1 0
0
0
0  x  y 1 xy'


yx'
yy'
a 
b 
 
c 
 
 d    x' 
 e   y '
    
f
g 
 
h 
Ax=b
Matlab: x = A\b
Then make your homography matrix by rearranging x into a 3 x 3 matrix
Size of A? b?
Computer Vision, Robert Pless
Computer Vision, Robert Pless
H12
H23
H34
(H12)-1
(H34H23H12)
(H34H23H12)-1= ((H12)-1(H23)-1(H34)-1)
Chaining, inverses, and pre-computation can all be done on the matrices…
Computer Vision, Robert Pless
Computer Vision, Robert Pless
More than Geometry: image processing
• The above gave different ways of transforming the
coordinates of one image into the coordinates of
another.
Graphics question.
– Given one image, the homography H, make the second image.
• (why would you want to do this? One application is image
tweening…).
– Process:
• Take pixel (x,y) of image 1, and put its color at pixel H(x,y,1)
in new image. Repeat.
– Problems?
Computer Vision, Robert Pless
Backward Transformation
• Because of the above, some techniques estimate the
parameters of the “backwards” transformation, H-1:
– From the destination image to the source image
• Remember, you don’t have to work hard to find the
backwards transformation. It’s just the matrix
inverse.
Computer Vision, Robert Pless
Mapping Pixel Values - Going Backwards
p = H-1(p’)
p’
“Source” image, Is
“Destination” image, Id
Id(p’) = Is(H-1(p’))
Intensity at destination pixel p’ = the intensity at the source pixel H-1(p’)
Computer Vision, Robert Pless
If you project backward from a real pixel, you also won’t land
exactly on a pixel??!!?
(but there is a good way of making up a value)
H-1(p’)
Pixel centers…
Computer Vision, Robert Pless
If you project backward from a real pixel, you also won’t land
exactly on a pixel??!!?
(but there is a good way of making up a value)
Computer Vision, Robert Pless
Use linear interpolation along the top and bottom horizontal lines to
determine IA and IB:
IA = (1 – f)•I(xi, yj) + f•I(xi+1, yj)
IB = (1 – f)•I(xi, yj+1)+ f•I(xi+1, yj+1)
Use linear interpolation along the vertical line between IA and IB to
determine I(x,y):
I(x,y) = (1 – g)•zA + g•zB
Or,
I(x,y) =
(1 – f)•(1 – g)•I(xi, yj)
(1 – f)•
g
•I(xi, yj+1)
+
+
f•(1 – g)•I(xi+1, yj) +
f•
g
•I(xi+1, yj+1)
Computer Vision, Robert Pless
Beyond geometry…
Computer Vision, Robert Pless