Camera Matrix
16-385 Computer Vision (Kris Kitani)
Carnegie Mellon University
2D to 2D Transform
(last session)
3D object
3D to 2D Transform
(today)
2D to 2D Transform
(last session)
A camera is a mapping between
the 3D world
and
a 2D image
A camera is a mapping between
the 3D world and a 2D image
x = PX
2D image
point
camera
matrix
3D world
point
What do you think the dimensions are?
x = PX
2
3
2
X
p1
4 Y 5 = 4 p5
p9
Z
homogeneous
image
3x1
p2
p6
p10
p3
p7
p11
Camera
matrix
3x4
p4
p8
p12
3
2
3
X
6 Y 7
7
56
4 Z 5
1
homogeneous
world point
4x1
The pinhole camera
y
image plane
X
x
x
camera center
principal axis
z=f
principal point
What is the equation for image coordinate x (in terms of X)?
z
y
What is the equation for image coordinate x (in terms of X)?
image plane
X
?
Y
z
f
Z
y
image plane
X
fY
Z
Y
z
f
Z
[X
Y
>
Z] 7! [f X/Z
f Y /Z]
>
Pinhole camera geometry
y
image plane
X
x
x
camera center
principal axis
z=f
principal point
What is the camera matrix P for a pinhole camera model?
x = PX
z
Relationship from similar triangles…
[X
Y
>
Z] 7! [f X/Z
f Y /Z]
>
generic camera model 2
3
3 X
2
3 2
X
p1 p 2 p3 p 4
6 Y 7
7
4 Y 5 = 4 p5 p 6 p7 p 8 5 6
4 Z 5
p9 p10 p11 p12
Z
1
What does the pinhole camera model look like?
2
?
P=4 ?
?
?
?
?
?
?
?
3
?
? 5
?
Relationship from similar triangles…
[X
Y
>
Z] 7! [f X/Z
f Y /Z]
>
generic camera model 2
3
3 X
2
3 2
X
p1 p 2 p3 p 4
6 Y 7
7
4 Y 5 = 4 p5 p 6 p7 p 8 5 6
4 Z 5
p9 p10 p11 p12
Z
1
What does the pinhole camera model look like?
2
f
P=4 0
0
0
f
0
3
0 0
0 0 5
1 0
2
f
P=4 0
0
0
f
0
0
0
1
3
0
0 5
0
Camera origin and image origin might be different
CCD array
p
camera coordinate
system
image coordinate system
CCD array
p
camera coordinate
system
image coordinate system
2
f
P=4 0
0
0
f
0
px
py
1
3
0
0 5
0
Accounts for different origins
In general, the camera and image sensor
have different coordinate systems
X world point
Oimage
x
image point
Ocamera
In general, there are three different coordinate systems…
X world point
Oimage
Oworld
x
image point
Ocamera
so you need the know the transformations between them
Can be decomposed into two matrices
2
f
P=4 0
0
32
0
f
0
(3 x 3)
px
1
py 5 4 0
1
0
3
0 0 0
1 0 0 5
0 1 0
(3 x 4)
P = K[I|0]
2
f
K=4 0
0
0
f
0
3
px
py 5
1
calibration matrix
Assumes that the camera and world
share the same coordinate system
2
f
P=4 0
0
0
f
0
32
px
1
py 5 4 0
1
0
3
0 0 0
1 0 0 5
0 1 0
What if they are different?
How do we align them?
yc
Camera
coordinate
system
yw
xc
xw
zc
World
coordinate
system
zw
Assumes that the camera and world
share the same coordinate system
2
f
P=4 0
0
0
f
0
32
px
1
py 5 4 0
1
0
3
0 0 0
1 0 0 5
0 1 0
What if they are different?
How do we align them?
yc
Camera
coordinate
system
yw
xc
xw
zc
Rt
3R rotation and translation to align axis
World
coordinate
system
zw
yc
Camera
coordinate
system
Xw
yw
xc
xw
zc
World
coordinate
system
zw
yc
Camera
coordinate
system
C
Coordinate of the
camera center in the
world coordinate frame
Xw
yw
xc
xw
zc
World
coordinate
system
zw
Xc
yc
Camera
coordinate
system
Xw
yw
xc
C
Coordinate of the
camera center in the
world coordinate frame
xw
zc
(X w
World
coordinate
system
C)
(X w
C)
Translate
zw
Why aren’t
the points
aligned?
Xc
yc
Camera
coordinate
system
C
Xw
yw
xc
xw
zc
World
coordinate
system
(X w
C)
Translate
zw
What happens to points after alignment?
Xc
Rotate
yc
Camera
coordinate
system
C
R
Xw
yw
xc
xw
zc
World
coordinate
system
R(X w
C)
Rotate Translate
zw
In inhomogeneous coordinates:
X c = R(X w
C)
Optionally in homogeneous coordinates:
2
3
Xc

6 Yc 7
6
7= R
4 Zc 5
0
1
RC
1
(4 ⇥ 4)
2
3
Xw
6 Yw 7
6
7
4 Zw 5
1
General mapping of a pinhole camera
P = KR[I|
C]
Quiz
What is the meaning of each matrix of the camera matrix
decomposition?
P = KR[I|
3x3
intrinsics
C]
Quiz
What is the meaning of each matrix of the camera matrix
decomposition?
P = KR[I|
3x3
intrinsics
3x3
3D rotation
C]
Quiz
What is the meaning of each matrix of the camera matrix
decomposition?
P = KR[I|
3x3
intrinsics
3x3
3D rotation
C]
3x3
identity
Quiz
What is the meaning of each matrix of the camera matrix
decomposition?
P = KR[I|
3x3
intrinsics
3x3
3D rotation
C]
3x3
identity
3x1
3D translation
General mapping of a pinhole camera
P = KR[I|
C]
(translate first then rotate)
Another way to write the mapping
P = K[R|t]
where
t=
RC
(rotate first then translate)
Quiz
The camera matrix relates what two quantities?
Quiz
The camera matrix relates what two quantities?
x = PX
Quiz
The camera matrix relates what two quantities?
x = PX
3D points to 2D image points
Quiz
The camera matrix relates what two quantities?
x = PX
3D points to 2D image points
The camera matrix can be decomposed into?
Quiz
The camera matrix relates what two quantities?
x = PX
3D points to 2D image points
The camera matrix can be decomposed into?
P = K[R|t]
Quiz
The camera matrix relates what two quantities?
x = PX
3D points to 2D image points
The camera matrix can be decomposed into?
P = K[R|t]
intrinsic and extrinsic parameters
Generalized pinhole camera model
P = K[R|t]
2
f
P=4 0
0
0
f
0
32
px
r1
py 5 4 r4
1
r7
intrinsic parameters
2
r1
R = 4 r4
r7
r2
r5
r8
r3
r6
r9
t1
t2 5
t3
extrinsic parameters
3
r3
r6 5
r9
3D rotation
r2
r5
r8
3
2
3
t1
t = 4 t2 5
t3
3D translation
Why do we need P?
to properly relate world points to image points
(by taking into account different coordinate systems)