Camera models
and calibration
Read tutorial chapter 2 and 3.1
http://www.cs.unc.edu/~marc/tutorial/
Szeliski’s book pp.29-73
Schedule
2
(tentative)
#
date
topic
1
Sep.17
Introduction and geometry
2
Sep.24
Camera models and calibration
3
Oct.1
Invariant features
4
Oct.8
Multiple-view geometry
5
Oct.15
Model fitting (RANSAC, EM, …)
6
Oct.22
Stereo Matching
7
Oct.29
Structure from motion
8
Nov.5
Segmentation
9
Nov.12
Shape from X (silhouettes, …)
10
Nov.19
Optical flow
11
Nov.26
Tracking (Kalman, particle filter)
12
Dec.3
Object category recognition
13
Dec.10
Specific object recognition
14
Dec.17
Research overview
Brief geometry reminder
2D line-point coincidence relation: l T x = 0
Line from points:
Point from lines: x = l× l'
3D plane-point coincidence relation:
l = x× x'
πT X = 0
X1T 
 π1T 
 
Point from planes:  π T2  X  0 Plane from points: X T2  π  0
 
X 3T 
 π 3T 
 
 
3D line representation:
(as two planes or two points)
2D Ideal points
(x , x ,0)
T
1
2
 PT 
 T   A B  0 2x 2
Q 
3D Ideal points X 1 , X 2 , X 3 , 0 
T
2D line at infinity l∞ = (0,0,1)T 3D plane at infinity     0, 0, 0,1
3
T
Conics and quadrics
l
Conics
x TCx = 0
lT C* l = 0
C*  C-1
x
C
l=Cx
Quadrics
XT QX = 0
4
π T Q*π = 0
2D projective transformations
Definition:
A projectivity is an invertible mapping h from P2 to itself
such that three points x1,x2,x3 lie on the same line if and
only if h(x1),h(x2),h(x3) do.
Theorem:
A mapping h:P2P2 is a projectivity if and only if there
exist a non-singular 3x3 matrix H such that for any point
in P2 reprented by a vector x it is true that h(x)=Hx
Definition: Projective transformation
 x'1   h11
  
 x'2   h21
 x'  h
 3   31
h12
h22
h32
h13  x1 
 

h23  x2 
h33  x3 
or
x' = H x
8DOF
projectivity=collineation=projective transformation=homography
Transformation of 2D points,
lines and conics
For a point transformation
x' = H x
Transformation for lines
l' = H -T l
Transformation for conics
C' = H-TC H-1
Transformation for dual conics
C'* = HC*HT
Fixed points and lines
He = λe
H ll
-T
(eigenvectors H =fixed points)
(1=2  pointwise fixed line)
(eigenvectors H-T =fixed lines)
Hierarchy of 2D
transformations
Projective
8dof
Affine
6dof
Similarity
4dof
Euclidean
3dof
 h11 h12
h
 21 h22
 h31 h32
 a11 a12
a
 21 a22
 0
0
h13 
h23 
h33 
tx 
t y 
1 
 sr11 sr12 t x 
 sr

sr
t
22
y
 21
 0
0
1 
 r11 r12 t x 
r

r
t
21
22
y


 0 0 1 
transformed
squares
invariants
Concurrency, collinearity,
order of contact (intersection,
tangency, inflection, etc.),
cross ratio
Parallellism, ratio of areas,
ratio of lengths on parallel
lines (e.g midpoints), linear
combinations of vectors
(centroids).
The line at infinity l∞
Ratios of lengths, angles.
The circular points I,J
lengths, areas.
The line at infinity
l  H
 A
l   T T
 t A
T
A 
T
0
0  
  0   l
1  
1
The line at infinity l is a fixed line under a projective
transformation H if and only if H is an affinity
Note: not fixed pointwise
Affine properties from images
projection
1 0 0 
H PA   0 1 0  H A
l1 l2 l3 
rectification
l∞  l1 l2 l3  , l3 ≠0
T
Affine rectification
l∞
v1
l1
l2
l3
v2
v 2 = l3 × l 4
l4
v1  l1  l2
l∞  v1  v2
The circular points
1
 
Ii
0
 
 s cos
I  H S I   s sin 
 0
1
 
J  i
0
 
s sin 
s cos
0
t x  1 
1


 

i
t y  i   se  i   I
0
1  0 
 
The circular points I, J are fixed points under the
projective transformation H iff H is a similarity
The circular points
“circular points”
x12 + x22 + dx1x3 + ex2 x3 + fx32 = 0
x3  0
l∞
x12 + x22 = 0
I  1, i,0
T
J  1,-i,0
T
Algebraically, encodes orthogonal directions
T
T
(
)
(
)
I = 1,0,0 + i 0,1,0
Conic dual to the circular
points
C*∞  IJT  JI T
1 0 0
C*∞  0 1 0
0 0 0
C*∞  HS C*∞HTS
l∞
I
J
The dual conic C*∞ is fixed conic under the
projective transformation H iff H is a similarity
Note: C*∞has 4DOF
l∞ is the nullvector
Angles
(
Euclidean: l = l1 , l2 , l3
cosθ =
Projective: cosq =
)
m = (m , m , m )
T
T
1
2
3
l1m1 + l2 m2
(l
2
1
(l
+ l22 )(m12 + m22 )
l T C*¥ m
T
C*¥ l) ( m T C*¥ m )
lT C*¥ m = 0 (orthogonal)
Transformation of 3D points,
planes and quadrics
For a point transformation
X' = H X
(cfr. 2D equivalent)
(x'= H x)
Transformation for lines
π'= H -T π
(l' = H-T l)
Transformation for quadrics
Q' = H-TQH-1
(C' = H-TCH-1)
Transformation for dual quadrics
Q'* = HQ*HT
(C'* = HC*HT )
Hierarchy of 3D
transformations
Projective
15dof
A
vT

t
v 
Affine
12dof
 A t
0T 1


Similarity
7dof
s R t 
 0T 1


Angles, ratios of length
The absolute conic Ω∞
Euclidean
6dof
 R t
0T 1


Volume
Intersection and tangency
Parallellism of planes,
Volume ratios, centroids,
The plane at infinity π∞
The plane at infinity
 AT
π  H π    T  T
- t A
T
A
0
 
0  0 
    π
1 0
 
1
The plane at infinity π is a fixed plane under a
projective transformation H iff H is an affinity
1.
2.
3.
4.
canonical position π∞ = (0,0,0,1)
contains directions D = (X1 , X 2 , X 3 ,0)T
two planes are parallel  line of intersection in π∞
line // line (or plane)  point of intersection in π∞
T
The absolute conic
The absolute conic Ω∞ is a (point) conic on π.
In a metric frame:
2
2
2
X1  X 2  X 3 
0
X4

or conic for directions:
(with no real points)
X1, X 2 , X 3 IX1, X 2 , X 3 T
The absolute conic Ω∞ is a fixed conic under the
projective transformation H iff H is a similarity
1. Ω∞ is only fixed as a set
2. Circle intersect Ω∞ in two circular points
3. Spheres intersect π∞ in Ω∞
The absolute dual quadric
I
  T
0
*
∞
0
0
The absolute dual quadric Ω*∞ is a fixed conic under
the projective transformation H iff H is a similarity
1. 8 dof
2. plane at infinity π∞ is the nullvector of Ω∞
3. Angles:
π T Ω* π
cos θ =
1
(π
T
1
∞ 2
Ω*∞π1 )(π T2 Ω*∞π 2 )
Camera model
Relation between pixels and rays in
space
?
Pinhole camera
Gemma Frisius, 1544
Distant objects appear smaller
23
Parallel lines meet
• vanishing point
24
Vanishing points
H VPL
VPR
VP2
VP1
25
To different directions
correspond different vanishing points
VP3
Geometric properties of projection
• Points go to points
• Lines go to lines
• Planes go to whole image
or half-plane
• Polygons go to polygons
• Degenerate cases:
– line through focal point yields point
– plane through focal point yields line
26
Pinhole camera model
T
T
( X , Y , Z )  ( fX / Z , fY / Z )
X
   fX   f
Y    
 Z    fY   
   Z  
1
 
f
X
0  
Y 

0  
Z

1 0 
1
linear projection in homogeneous coordinates!
Pinhole camera model
X
X
 fX   f
 10  0 
  
Y 
Y 




f
 fY x  PX

  01Z  0 Z 
 Z  
1  0 1 0 
  
1
1
P  diag( f , f ,1)I | 0
Principal point offset
T
T
( X , Y , Z )  ( fX / Z + px , fY / Z + py )
T
( px , py )
æ
ç
ç
ç
ç
è
X
Y
Z
1
ö
÷
÷
÷
÷
ø
æ fX + Zp
x
ç
ç fY + Zpy
çç
Z
è
principal point
ö é f
÷ ê
÷=ê
÷÷ ê
ø êë
px
f
py
1
æ
ù
0 ç
ú
0 úç
úç
0 úûç
è
X
Y
Z
1
ö
÷
÷
÷
÷
ø
Principal point offset
æ fX + Zp
x
ç
ç fY + Zpy
çç
Z
è
f
K  

æ X ö
ö é f
ù
px 0 ç
÷ ê
ú Y ÷
÷
÷x=ê KI | 0fXcam
py 0 úç
÷÷ ê
úç Z ÷
1 0 úûç 1 ÷
ø êë
è
ø
px 
f p y  calibration matrix
1 
Camera rotation and translation
~
~ ~
(
Xcam = R X - C)
X
~
~  
R - RC Y  R - RC
X cam  
X
   

1 
1  Z   0
0
1
~
 
x  KR
Ccam
X
KI |I0| -X
~
x = PX
P  KR | t 
t  -RC
 
CCD camera
p x f
mx x
KK
my y p y

11
f
px 
p y 
1 
General projective camera
 x s
K  
x

~
P  KR I | C


px 
p y 
1 
11 dof (5+3+3)
non-singular
P  KR | t 
intrinsic camera parameters
extrinsic camera parameters
Radial distortion
• Due to spherical lenses (cheap)
• Model:
R
R
 x
( x, y)  (1  K1 ( x 2  y 2 )  K 2 ( x 2  y 2 )2  ...)  
 y
straight lines are not straight anymore
http://foto.hut.fi/opetus/260/luennot/11/atkinson_6-11_radial_distortion_zoom_lenses.jpg
Camera model
Relation between pixels and rays in
space
?
Projector model
Relation between pixels and rays in
space
(dual of camera)
?
(main geometric difference is vertical principal point
offset to reduce keystone effect)
Meydenbauer camera
vertical lens shift
to allow direct
ortho-photographs
Affine cameras
Action of projective camera on points
and lines
projection of point
x  PX
forward projection of line
Xμ   P(A μB)  PA  μPB  a  μb
back-projection of line
  PTl
 X  l PX
T
T
l x  0; x  PX 
T
Action of projective camera on conics
and quadrics
back-projection to cone
Qco  P CP
T
x T Cx  XT PT CPX  0
x  PX 
projection of quadric
C  PQ P
*
* T
TQ*  lT PQ*PT l  0
  P l 
T
Resectioning
Xi  x i
P?
Direct Linear Transform (DLT)
x i  PXi
x i  PXi
rank-2 matrix
Direct Linear Transform (DLT)
Minimal solution
P has 11 dof, 2 independent eq./points
 5½ correspondences needed (say 6)
Over-determined solution
n  6 points
minimize
Ap subject to constraint
p 1
use SVD
Degenerate configurations
(i)
Points lie on plane or single line passing
through projection center
(ii) Camera and points on a twisted cubic
Data normalization
Scale data to values of order 1
1.
2.
move center of mass to origin
scale to yield order 1 values

2D

3D
Line correspondences
Extend DLT to lines
  P T li
T
li PX1i
(back-project line)
T
li PX2i
(2 independent eq.)
Geometric error
Gold Standard algorithm
Objective
Given n≥6 2D to 3D point correspondences {Xi↔xi’},
determine the Maximum Likelyhood Estimation of P
Algorithm
(i) Linear solution:
~
(a) Normalization: Xi  UXi ~xi  T xi
(b) DLT
(ii) Minimization of geometric error: using the linear
estimate as a starting point minimize the geometric error:
~ ~~
~
(iii) Denormalization: P  T -1PU
Calibration example
(i) Canny edge detection
(ii) Straight line fitting to the detected edges
(iii) Intersecting the lines to obtain the images corners
typically precision <1/10
(H&Z rule of thumb: 5n constraints for n unknowns)
Errors in the image
(standard case)
xˆ  PX
i
i
Errors in the world

x i  PXi
Errors in the image and in the world

Xi
Restricted camera estimation
Find best fit that satisfies
•
skew s is zero
•
pixels are square
•
principal point is known
•
complete camera matrix K is known
Minimize geometric error
impose constraint through parametrization
Minimize algebraic error
assume map from param q  P=K[R|-RC], i.e. p=g(q)
minimize ||Ag(q)||
Restricted camera estimation
Initialization
•
Use general DLT
•
Clamp values to desired values, e.g. s=0, x= y
Note: can sometimes cause big jump in error
Alternative initialization
•
Use general DLT
•
Impose soft constraints
•
gradually increase weights
Image of absolute conic
A simple calibration device
(i)
compute H for each square
(corners  (0,0),(1,0),(0,1),(1,1))
(ii) compute the imaged circular points H(1,±i,0)T
(iii) fit a conic to 6 circular points
(iv) compute K from w through cholesky factorization
(≈ Zhang’s calibration method)
Some typical calibration algorithms
Tsai calibration
Zhangs calibration
http://research.microsoft.com/~zhang/calib/
Z. Zhang. A flexible new technique for camera calibration. IEEE
Transactions on Pattern Analysis and Machine Intelligence,
22(11):1330-1334, 2000.
Z. Zhang. Flexible Camera Calibration By Viewing a Plane From
Unknown Orientations. International Conference on Computer Vision
(ICCV'99), Corfu, Greece, pages 666-673, September 1999.
http://www.vision.caltech.edu/bouguetj/calib_doc/
81
from Szeliski’s book
Next week:
Image features
82