Active Calibration of Cameras:
Theory and Implementation
Anup Basu
Sung Huh
CPSC 643 Individual Presentation II
March 4th, 2009
1
Outline
Introduction
 Theoretical Derivation
 Strategies for Active Calibration
 Theoretical Error Analysis
 Experimental Result
 Conclusion and Future Work

2
Outline
Introduction
 Theoretical Derivation
 Strategies for Active Calibration
 Theoretical Error Analysis
 Experimental Result
 Conclusion and Future Work

3
Introduction
Important step for a 2D image to relates to the
3D world
 Involves relating the optical features of a lens to
the sensing device

◦ Pose estimation, 3D motion estimation, automated
assembly

Parameters: image center and focal length
◦ Expressed in terms of image pixels

Linear vs. Nonlinear, Lens distortion
consideration vs. w/o consideration
4
Linear
Nonlinear
Simpler to implement
 Most cannot model camera
distortions


Capable to consider
complicated imaging model
with many parameters
 Computationally expensive
search procedure
 Reasonable good initial
guess for convergence of
the solutions
Technique: Linearity
5
Major Drawback of existing
algorithm

Calibrate with predefined pattern
◦ Relating image projections to the camera
parameters
Recent algorithms suffer from the same
limitation
 New discovery: Active Calibration

6
Active Calibration

Camera capable of panning and tilitng can
automatically calibrate itself
◦ Modeled from eye movement
Active machines can keep track of object
of interest
 Facilitate region-of-interest process

7
Active Calibration – How different?
Does not need a
starting estimate for
focal length and
image center
 Does not need prior
information about
focal length
 Does not need to
match points or
feature b/w images

Reasonably accurate
localization of
contour
 Estimate of center
(Not too far from
true value)

8
Method of Calibration
Using perspective distortion to measure
calibration parameters
 Without using perspective distortion

9
Outline
Introduction
 Theoretical Derivation
 Strategies for Active Calibration
 Theoretical Error Analysis
 Experimental Result
 Conclusion and Future Work

10
Theoretical Derivation
11
Theoretical Derivation
Lemma 1
Camera rotates by R and translate by T
 New image contours

fx
X
y  r31 f x  f x
fy
Z
x
y
Z
r13  r23  r33 
fx
fy
Z
r11 x  r21
xn 
r12
yn 
fx
Y
x  r22 y  r32 f y  f y
fy
Z
x
y
Z
r13  r23  r33 
fx
fy
Z
12
Theoretical Derivation
Lemma 1 - Proof

Use two set of equation
 Xn 
X
 

T 
Y

R
Y
  n
  T
Z 
Z
 n
 
,
X
x  fx
Z
Y
y  fy
Z
13
Theoretical Derivation
Proposition 1
Depth (Z) is larger than ΔX, ΔY, ΔZ
 Camera moves by small tilt angle


y 
xt  x 1  t


fy 


y 
yt   y  t f y   1  t


fy 

14
Theoretical Derivation
Proposition 1 – Proof

Rotation matrix R at small tilt angle
X Y Z
,
,
Z Z Z
1 0
0 1

 0  t
0 
 t 
1 
are negligible
 From Lemma 1


y 
xt  x 1  t


fy 

1

y 
yt   y  t f y  1  t


fy 

1
15
Theoretical Derivation
Proposition 1 – Proof
Expand right side of equation with Taylor
series, because of small θt
 With the same assumption, if camera
moves by small pan angle θp


p
x p   x   p f x  1   p 
fx 


x 
yt  y 1   p 
fx 

16
Outline
Introduction
 Theoretical Derivation
 Strategies for Active Calibration
 Theoretical Error Analysis
 Experimental Result
 Conclusion and Future Work

17
Strategy for Active Calibration
Want – A relation b/w lens parameters
and image information w/ given image
contours before & after camera motion
 Relate focal length to other camera
parameters and the pan/tilt angles

18
Strategy for Active Calibration
Proposition 2
Similar assumption as Proposition 1
 Center of the lens is estimated with a
small error (δx, δy)

fy 
t
 x  x
xy  x
y
 y x

t
19
Strategy for Active Calibration
Proposition 2 – Proof

From Proposition 1
xt  x  t

Estimate image Center with error (δx, δy)
x x  x x 
 xt  x 

xy
fy
t
fy
t
fy
 x x  y  y 
 xy  x
y
 y x 
Ignore δxδy term
20
Strategy for Active Calibration
Proposition 3 (Plan A)

Using tilt (or pan) movement and
considering three independent static
contours, two linear equation in δxδy can
be obtained if negligible terms are ignored
21
Strategy for Active Calibration
Proposition 3 – Proof
Two different contour, C1 and C2
 Point lying on C1 & C2, (x(1),y(1)) and
(x(2),y(2))
 From Proposition 2

fy 
fy 
x
x
t
(1)
t
x
(1)
t
(2)
t
x
(2)
x

x


(1)

(2)
(1)
y (1)  x (1) y  y (1) x
(2)
y (2)  x(2) y  y (2) x
22
Strategy for Active Calibration
Proposition 3 – Proof

Equate right side equations and simplify

 

x (2) y (2)  K1 x (1) y (1)  x (2)  K1 x (1)  y  y (2)  K1 y (1)  x
where

K 
x
x (2) t  x (2)
1
(1)
t
 x (1)
(3)


23
Strategy for Active Calibration
Proposition 3 – Proof
Third contour, C3
 Point on C3, (x(3),y(3))


 

x (3) y (3)  K 2 x (1) y (1)  x (3)  K 2 x (1)  y  y (3)  K 2 y (1)  x (4)
where
K2


x
x (3) t  x (3)
(1)
t
 x (1)


24
Strategy for Active Calibration
Proposition 3 – Proof

Finding fx and fy with estimated center
fx 
fy
y
p




xy  x

 x  x
p
y

(5)

(6)
xy  x y e  y x e
e
t
y
 y x e
t

“e” denote the estimate of a certain
parameter
25
Strategy for Active Calibration
Procedure Summary for Plan A
Estimate δx and δy using (3) and (4) with three
distinct image contour
 Obtain estimate for fx and fy by substituting
resulting estimate into (5) and (6)
 Term  xt  x  and  y p  y  make (5) and (6)
unstable

26
Strategy for Active Calibration
Procedure Summary for Plan A
Variation in x-coord. for any point is due
to change in perspective distortion (tilt)
 Little change in the image y-coord.
corresponding to a given 3-d point (pan)
  xt  x  and  y p  y  are small (few pixel)
 Relative error can be large

◦ presence of noise and inaccuracies in
localization of a contour

Estimate in (5) & (6) are often unreliable
27
Strategy for Active Calibration
Proposition 4 (Plan B)

Using a single contour and pan/tilt camera
movements fx and fy can be obtained if
negligible terms are ignored
28
Strategy for Active Calibration
Proposition 4 – Proof
and δy are non-zero in the second
equation in Proposition 1
 δx


y y  

 yt   y    y   y  t f y  1  t f 
y


Simplify

yt  y 1  

2
t

t y 2
fy
 t f y    
2
y t
t x2
fy

2t y y
fy
The last three terms are negligible even if
δx and δy are large
y  y 1    y
(7)
f 

2
t
t
y
t
2
fy
29
Strategy for Active Calibration
Proposition 4 – Proof

Simplifying eq (7)
y  y 1   
f f
y
2
y
 fx
2
t
t
2
t
y
0
(8)
can be obtained with similar way
x  x 1   
(9)
f f
x 0
2
x
2
p
p
x
2
p
30
Strategy for Active Calibration
Proposition 4 – Corollary

Given two independent contours, pan/tilt
camera movements, and estimate of fx and
e
e
fy given by f x and f y respectively, δx and δy
can be obtained by solving
f
f

e
y
e
y
x
x
(1)
t
 x(1)
t
(2)
t
 x(2)
t

x

x

(10)

(11)
(1)
y (1)  x(1) y  y (1) x
(2)
y (2)  x(2) y  y (2) x
Considering from two independent
contour from Proposition 2
31
Strategy for Active Calibration
Proposition 4 – Proof
Consider (8)
 Most practical system

◦ y < 300, fy > 500
(8) is in form
 A = 1, B < 0, C is small compare to B
Af y2  Bf y  C  0

fy 
fx 

yt  y 1  
2
t

2
p
1
2t

xp  x 1 
2 p


2
1  y 1  t  yt


2 
t



 x 1   xp

2 
p

2
p
2

  4 y2


(12)
2

  4 x2


(13)
32
Strategy for Active Calibration
Procedure Summary for Plan B
Estimate fx and fy from (12) and (13) using
a single image contour
 Solve for δx and δy by substituting
resulting estimates into (10) and (11) and
using another independent contour

33
Strategy for Active Calibration
Proposition 5

When there is error in contour localization
after pan/tilt movements, the ratio of the error
in Plan A compared to Plan B for estimating fx
(fy) is approximately
x  x p  x  xt 


y p  y  yt  y 
34
Strategy for Active Calibration
Proposition 5 – Proof
Introduce similar error term in x  x p and y p  y
in (13) and (5) respectively
2. Simplify the expressions and consider the
approximate magnitude of error in both the
expressions
3. Take the ratio of these two terms
1.
35
Strategy for Active Calibration
Proposition 5 – Implication
Error in Plan A can be as large as 30 times
that of Plan B, for estimating focal lengths
 Plan A is theoretically more precise, but
not reliable for noisy real scenes

36
Outline
Introduction
 Theoretical Derivation
 Strategies for Active Calibration
 Theoretical Error Analysis
 Experimental Result
 Conclusion and Future Work

37
Theoretical Error Analysis

Effect of errors from various sources on
the estimation of different parameters
◦ Errors in measurements of pan/tilt angles
◦ Effect of noise in the extraction of image
contours
38
Theoretical Error Analysis


Remark 1
Error in measurement
of the pan (tilt) angle
generates a
proportional error in
the estimate of fx(fy)



Proof
Consider(5)
fx is proportional to
the pan angle
◦ Any error in the
measurement translates
to a corresponding
error in fx

Any error from tilt
angle generate a
proportional error in
fy
39
Theoretical Error Analysis
Remark 2
 Errors in
measurement of the
pan & tilt angles do
not affect the
estimate of the lens
center

Proof
 Linear equations in δx
and δy are obtained
by equating the right
hand sides of two
equations

◦ independent contours
from the same image
are considered
40
Theoretical Error Analysis
Consider (1) & (2)
 Denote ε1: error in tilt angle
 Contour extracted from same image
 Then (θt+ε1) of (3) cancels out from both
sides

◦ Error in pan/tilt angle do not affect the
estimate of lens center
41
Theoretical Error Analysis
Consider two independent images
generating the contours in (1) & (2)
  

 K1 in (3) modifies to K    
   

is
not
equal
to
1
in
general
   
 Errors in angle can change the estimate of
the lens center if contours from
independent images are considered

t
1
t
2
1
t
1
t
2
42
Theoretical Error Analysis
Remark 3
 The coefficients of
the linear (3)-(6) are
unbiased in the
presence of
uncorrelated noise
with zero mean


Coefficients involve a
linear combination of
terms
x, y, xy

These terms are
unbiased in the
presence of
uncorrelated noise
with zero mean
43
Theoretical Error Analysis

Remark 4

The variance of the
coefficients of (3)-(6) is
inversely proportional to
the number of points on
a contour
◦ Uncorrelated noise with
zero mean is considered
Form of variances
x, y, xy
 Inversely
proportional to the
number of points for
which the averages
were computed

44
Outline
Introduction
 Theoretical Derivation
 Strategies for Active Calibration
 Theoretical Error Analysis
 Experimental Result
 Conclusion and Future Work

45
Experimental Result
Simulation – Validity of Algorithms
Synthetic data used
 Three independent contour represented
by three sets of 3D points
 Points projected onto the image plane
 Values quantized to the nearest integer
 Without noise, A produced more
accurate estimate

◦ Less than 1 percent relative error in focal
length estimate
46
Experimental Result
Simulation – Variation of error in focal length estimate

Change fx and fy from 100 to 1000 with
interval 100
◦ Keep other parameters fixed

Discretization error influence A more
when focal length was small
◦ A is not very robust to noise

Larger the focal length, smaller the error
relative to the focal length
◦ Better estimate production with A
47
Experimental Result
Simulation – Variation of error in focal length estimate
Error estimate from B does not drop off
rapidly
 B is theoretically less accurate than A

Error in estimates with varying focal length
14
Strategy A
Percentage Error
12
Strategy B
10
8
6
4
2
0
100
200
300
400
500
600
Focal Length
700
800
900
1000
48
Experimental Result
Simulation – Gaussian Noise Added

Poor performance with A
◦ 20, 28, and 40 percent error with 3, 4, and 5
noise standard deviation
High robustness with B
Error in focal length estimate with Gaussian noise
8
7
Percentage error

6
5
4
3
2
2
4
6
8
10
12
Standard deviation of noise
14
16
49
Experimental Result
Tracking Contour

Match contours of interest during pan/tilt
◦ For automatic calibration
Edges in the original image was thickened
using the morphological operation of
“dilation”
 Edges after pan/tilt was AND-ed with the
dilated image to extract corresponding
contours after camera rotation

50
A sequence of images for small pan movements
of a camera
51
Corresponding edge images
52
Tracked over the sequence of image
53
Experimental Result
Calibration With Real Images – initial image and its edge
54
Experimental Result
Calibration With Real Images – panned image and its edge
55
Experimental Result
Calibration With Real Images – matching contour
56
Experimental Result
Calibration With Real Images
Plan A
 Estimate of fx and fy,
693 and 981
 With known pattern
and refining initial
estimate by trial and
error, 890 and 1109

Plan B
 Estimate of fx and fy,
917 and 1142
 Produces estimates
fairly close to the
true value

57
Experimental Result
Other Environment
58
Experimental Result
Other Environment
Estimate produced using Plan B, 902 and
1123, 905 and 1099
 Average relative error < 1.5%
 A produced very inaccurate estimate
 B produced stable estimates
 Lens center estimates are not very
accurate for both

59
Outline
Introduction
 Theoretical Derivation
 Strategies for Active Calibration
 Theoretical Error Analysis
 Experimental Result
 Conclusion and Future Work

60
Conclusion

Algorithm do not require unique pattern
◦ Only need scenes with strong and stable edge
A gives almost perfect estimates in an
ideal environment
 B is suitable for noisy synthetic images or
real scenes

61
Future Work
Simplify algorithm further by considering
roll movements of the camera
 Designing a simple method to obtain the
optical center of the lens

62
Question ?
63