Lecture 5.3
Camera calibration
Thomas Opsahl
Introduction
The image
𝒖
𝑢
𝐶
𝑣
𝑥𝐶
•
•
𝒙
𝑦𝐶
𝑿
𝑧𝐶
𝑊
World frame
𝑧𝐶 = 1
For finite projective cameras, the correspondence between points in the world and points in the image
can be described by the simple model
𝑓𝑢 𝑠 𝑐𝑢
𝑃 = 0 𝑓𝑣 𝑐𝑣 𝑅 𝒕
0 0 1
This camera model is typically not good enough for accurate geometrical computations based on
images
2
Introduction
x
•
Since light enters the camera through a lens
instead of a pinhole, most cameras suffer from
some kind of distortion
Z
y
X
•
Y
No radial distortion
x
This kind of distortion can be modeled and
compensated for
Z
y
•
A radial distortion model can look like this
2
2
2
X
Y
Barrel distortion
2 2
𝑥� = 𝑥 1 + 𝜅1 𝑥 + 𝑦 + 𝜅2 𝑥 + 𝑦
𝑦� = 𝑦 1 + 𝜅1 𝑥 2 + 𝑦 2 + 𝜅2 𝑥 2 + 𝑦 2
x
2
where 𝜅1 and 𝜅2 are the radial distortion
parameters
Z
y
X
Y
Pincushion distortion
3
Image from a non-projective camera
Introduction
•
When we calibrate a camera we typically
estimate the camera calibration matrix 𝐾
together with distortion parameters
•
A model 𝐾 𝑅 𝒕 , using such an estimated 𝐾,
does not in general describe the correspondence
between points in the world and points in the
image
•
Instead it describes the correspondence
between points in the world and points in a
undistorted image – an image where the
distortion effects have been removed
�=𝐾 𝑅
Does not satisfy 𝒖
�
𝑡𝑿
Equivalent projective-camera image
�=𝐾 𝑅
Satisfies 𝒖
�
𝑡𝑿
Images: http://www.robots.ox.ac.uk/~vgg/hzbook/
4
Undistortion
•
So earlier when we estimated homographies between overlapping images, we should really have been
working with undistorted images!
•
How to undistort?
– Matlab
[undist_img,newOrigin] = undistortImage(img,cameraParams);
undistortedPoints = undistortPoints(points,cameraParams);
– OpenCV
cv::undistort(img, undist_img, P, distCoeffs);
cv::undistortPoints(pts,undist_pts,P,distCoeffs);
•
The effect of undistortion is that we get an image or a set of points that satisfy the perspective camera
� much better than the original image or points
� = 𝑃𝑿
model 𝒖
– So we can continue working with the simple model
5
Camera calibration
•
Camera calibration is the process of estimating the matrix 𝐾 together with any distortion parameter that
we use to describe radial/tangential distortion
•
We have seen how 𝐾 can be found directly from the camera matrix 𝑃
•
The estimation of distortion parameters can be baked into this
•
One of the most common calibration algorithms was proposed by Zhegyou Zhang in the paper ’’A
Flexible New Technique for Camera Calibration’’ in 2000
– OpenCV: calibrateCamera
– Matlab: Camera calibration app
•
This calibration algorithm makes use of multiple images of a asymmetric chessboard
6
Zhang’s method in short
𝑍𝑊
𝐶
𝑥𝐶
𝑊
𝑧𝐶
𝑦𝐶
𝑌𝑊
•
Zhang’s method requires that the calibration object is planar
•
Then the 3D-2D relationship is described by a homography
u 
v  = K r
[1
 
 1 
r2
r3
X 
Y 
t ]   = K [ r1
0
 
1
r2
𝑋𝑊
X 
X 
t]Y  = H Y 
 
 
 1 
 1 
7
Zhang’s method in short
•
This observation puts 2 constraints on the intrinsic parameters due to the fact that 𝑅 is orthonormal

r1T r2 = 0 
h1T K −T K −1h2 = 0
⇒  T −T −1
T
T 
T
−T
−1
r1 r1 = r2 r2 
h1 K K h1 = h2 K K h2
•
•
•
Where 𝐻 = 𝒉1 𝑇
𝒉2 𝑇
𝒉3 𝑇 and 𝐾 −𝑇 = 𝐾 𝑇
−1
= 𝐾 −1
𝑇
So for each 3D-2D correspondence we get 2 constraints on the matrix 𝐵 = 𝐾 −𝑇 𝐾 −1
Next step is to isolate the unknown parameters in order to compute 𝐵
8
Zhang’s method in short
•
𝐵 can be computed directly from 𝐾
 b11 b12
B =K −T K −1 = b21 b22

b31 b32
•

1

fu2

b13  
s
b23  =  − 2
 
fu fv
b33  
v s − u f
 0 2 0 y
 fu f v
−
fu2 f v
1
s2
+
fu2 f v2 f v2
s ( v0 s − u0 f y )
2 2
u v
f f


fu2 f v


s ( v0 s − u0 f y ) v0 
− 2
fu2 f v2
fv 

2
( v0 s − u0 f y ) + v02 + 1
fu2 f v2
f v2 
v0 s − u0 f y
s
−
v0
f v2
We see that 𝐵 is symmetric, so that we can represent it by the parameter vector
𝒃 = 𝑏11 𝑏12 𝑏22 𝑏13 𝑏23 𝑏33 𝑇
9
Zhang’s method in short
•
If we denote
•
Thus we have
hi1h j1


h h + h h 
 i1 j 2 i 2 j1 


hi 2 h j 2
T
T
vij = 
 then hi Bh j = vij b
 hi 3 h j1 + hi1h j 3 
 hi 3 h j 2 + hi 2 h j 3 


hi 3 h j 3


 h1T Bh2 = 0 
 v12T 
r1T r2 = 0 
0 
⇒
⇔
=
b

 T

 T
0 
T
T 
=
−
B
B
r1T r1 = r2T r2 
h
h
h
h
v
v
 
2
2
 1 1
 11 22 
10
Zhang’s method in short
•
•
Given 𝑁 images of the planar calibration object we stack the equations to get a homogeneous system
of linear equations which can be solved by SVD when 𝑁 ≥ 3
From the estimated 𝒃 we can recover all the intrinsic parameters
•
The distortion coefficients are then estimated solving a linear least-squares problem
•
Finally all parameters are refined iteratively
•
More details in Zhang’s paper
http://research.microsoft.com/en-us/um/people/zhang/Papers/TR98-71.pdf
11
Camera calibration in practice
•
OpenCV
– Camera calibration tutorial
– We’ll test it out in the lab
•
Matlab
– App: Camera Calibrator
– This opens a new window
12
Camera calibration in practice
•
Add Images and specify the size of the
chessboard squares
– Chessboard detection starts
13
Camera calibration in practice
•
Add Images and specify the size of the
chessboard squares
– Chessboard detection starts
•
Inspect the correctness of detection and
choose what to estimate
– 2/3 radial distortion coeffs?
– Tangential distortion?
– Skew?
•
Calibrate
14
Camera calibration in practice
•
Add Images and specify the size of the
chessboard squares
– Chessboard detection starts
•
Inspect the correctness of detection and
choose what to estimate
– 2/3 radial distortion coeffs?
– Tangential distortion?
– Skew?
•
Calibrate
•
Export to a Matlab variable
15
Camera calibration in practice
•
Add Images and specify the size of the
chessboard squares
– Chessboard detection starts
•
Inspect the correctness of detection and
choose what to estimate
– 2/3 radial distortion coeffs?
– Tangential distortion?
– Skew?
•
Calibrate
•
Export to a Matlab variable
16
Summary
•
•
Calibration
– Zhangs method using planar 3D-points
– Matlab app
– DLT method: Est 𝑃, decompose into 𝐾 𝑅
Undistortion
•
Additional reading
– Szeliski: 6.3
𝒕
•
Optional reading
– A flexible new technique for camera calibration,
by Z. Zhang
– For geometrical computations we work on
undistorted images/feature points
17