Image Formation
Jana Kosecka
3-D Euclidean Space - Vectors
A “free” vector is defined by a pair
of points
:
Coordinates of the vector
:
1
3D Rotation of Points – Euler angles
Rotation around the coordinate axes, counter-clockwise:
Y’
y
P
’
γ
P
X’ x
z
Rotation Matrix!
•  Euler theorem – any rotation can be expressed as a sequence of
rotations around different coordinate axes !
•  Different order of rotations yields different final rotation!
•  Rotation multiplication is not commutative!
•  Different ways how to obtain final rotation – rotation around 3 axes
no successive rotations around same axes!
•  XYX, XZX, YXY, YZX, ZXZ, ZYZ – Eulerian involves repetition!
•  Cardanian – no repetitions XYZ, XZY, YZX, YXZ, ZXY, ZYX. !
•  Another widely used convention roll-pitch-yaw!
R = Rx (θ r )Ry (θ p )Rz (θ y )
!
2
Rotation Matrices in 3D
•
•
•
•
3 by 3 matrices
9 parameters – only three degrees of freedom
Representations – either three Euler angles
or axis and angle representation
•  Properties of rotation matrices (constraints between the
elements)
Columns are orthonormal
Rotation Matrix!
•  Problem with 3 angle representations: singularities
•  The mapping between angles and Rotation matrix is unique
•  i.e. given the rotation matrix, compute , ✓,
2
cos cos
4 sin cos
cos ✓ sin sin
cos ✓ sin cos
sin ✓ sin
cos sin + cos ✓ cos sin
sin sin + cos✓ cos cos
sin ✓ cos
3
sin sin ✓
cos sin ✓ 5
cos ✓
•  The inverse mapping between rotation matrix and the angles
sometimes cannot be computed or is not unique
Angle Axis Representation
•  Two coordinates frames of arbitrary orientations can be
related by a single rotation about ‘some’ axis in space and an
angle
3
Rotation Matrix!
Angle Axis Representation
•  Two coordinates frames of arbitrary orientations can be
related by a single rotation about ‘some’ axis in space and an
angle
•  We know that R has 9 entries, but is fully specified by 3
numbers. Hence the rotation matrices have to satisfy some
constraints
T
RR = I
•  Suppose now that R is representing a rigid body pose which
changes as a function of time
Canonical Coordinates for Rotation
Property of R
Taking derivative
Skew symmetric matrix property
By algebra
By solution to ODE
4
3D Rotation (axis & angle)
Solution to the ODE
with
or
Rotation Matrices
Given
How to compute angle and axis
5
3D Translation of Points
Translate by a vector
P
’
Y’
z’
z
t
x’ x
P
y
Rigid Body Motion – Homogeneous
Coordinates
3-D coordinates are related by:
Homogeneous coordinates:
Homogeneous coordinates are related by:
6
Rigid Body Motion – Homogeneous
Coordinates
3-D coordinates are related by:
Homogeneous coordinates:
Homogeneous coordinates are related by:
Properties of Rigid Body Motions
Rigid body motion composition
Rigid body motion inverse
Rigid body motion acting on vectors
Vectors are only affected by rotation – 4th homogeneous coordinate is zero
7
Rigid Body Transformation
Coordinates are related by:
Camera pose is specified by:
Image Formation
•  If the object is our lens the refracted light causes the
images
•  How to integrate the information from all the
rays being reflected from the single point
on the surface ?
•  Depending in their angle of incidence, some are
more refracted then others – refracted rays all
meet at the point – basic principles of lenses
•  Also light from different surface points may hit the same
lens point but they are refracted differently - Kepler’s
retinal theory
8
Let’s design a camera
•  Idea 1: put a piece of film in front of an object
•  Do we get a reasonable image?
Slide by Steve Seitz
Pinhole camera
•  Add a barrier to block off most of the rays
–  This reduces blurring
–  The opening is known as the aperture
Slide by Steve Seitz
9
Pinhole camera model
•  Pinhole model:
–  Captures pencil of rays – all rays through a single point
–  The point is called Center of Projection (focal point)
–  The image is formed on the Image Plane
Slide by Steve Seitz
Camera Obscura
•  Basic principle known to
Mozi (470-390 BCE),
Aristotle (384-322 BCE)
•  Drawing aid for artists:
described by Leonardo da
Vinci (1452-1519)
Gemma Frisius, 1558
Source: A. Efros
10
Home-made pinhole camera
Why so
blurry?
Slide by A. Efros
http://www.debevec.org/Pinhole/
Shrinking the aperture
•  Why not make the aperture as small as possible?
–  Less light gets through
–  Diffraction effects…
Slide by Steve Seitz
11
Shrinking the aperture
Adding a lens
•  A lens focuses light onto the film
–  Thin lens model:
•  Rays passing through the center are not deviated
(pinhole projection model still holds)
Slide by Steve Seitz
12
Adding a lens
•  A lens focuses light onto the film
–  Thin lens model:
•  Rays passing through the center are not deviated
(pinhole projection model still holds)
•  All parallel rays converge to one point on a plane
located at the focal length f
Slide by Steve Seitz
Adding a lens
•  A lens focuses light onto the film
–  There is a specific distance at which objects are “in
focus”
•  other points project to a “circle of confusion” in the
image
Slide by Steve Seitz
13
Thin lens formula
Similar triangles everywhere!
D’
y’/y = D’/D
D
f
y
y’
image
plane
lens
object
Slide by Frédo Durand
Thin lens formula
Similar triangles everywhere!
D’
y’/y = D’/D
y’/y = (D’-f)/f
D
f
y
y’
image
plane
lens
object
Slide by Frédo Durand
14
Thin lens formula
1 +1 =1
D’ D f
D’
Any point satisfying the thin lens equation is in focus. If the focal plane D’ is moved closer, circle of confusion. D Ex: f = 100mm, D = 5m and D’ = 102mm
f
image
plane
lens
object
Slide by Frédo Durand
Depth of Field
•  in real camera lenses, there is a range of points which
are brought into focus at the same distance
•  depth of field of the lens , as Z gets large – z’ approaches f
•  human eye – power of accommodation – changing f
15
How can we control the depth of field?
•  Changing the aperture size affects depth of field
–  A smaller aperture increases the range in which
the object is approximately in focus
–  But small aperture reduces amount of light –
need to increase exposure
Slide by A. Efros
Varying the aperture
Large aperture = small DOF
Small aperture = large DOF
Slide by A. Efros
16
Field of View
Slide by A. Efros
Field of View
What does FOV depend on?
Slide by A. Efros
17
Field of View
f
f
FOV depends on focal length and size of the camera
retina
Smaller FOV = larger Focal Length
Slide by A. Efros
Field of View / Focal Length
Large FOV, small f
Camera close to car
Small FOV, large f
Camera far from the car
Sources: A. Efros, F. Durand
18
Same effect for faces
wide-angle
standard
telephoto
Source: F. Durand
Pinhole Camera Model
Pinhole
Frontal
pinhole
fX
Z
fY
y=
Z
x=
19
More on homogeneous coordinates
Previously homogenous coordinates
[X, Y, Z, 1]T
More generally projective coordinates of a point
T
[W X, W Y, W Z, W ]
For any W, corresponds to the same point, i.e. point is a line
The first coordinates can be obtained from the second by division by W
What if W is zero ?
2
3 2
v1
X1
6 v2 7 6 Y1
7 6
v=6
4 v3 5 = 4 Z1
0
1
3
2
3
X2
6 Y2 7
6
7
4 Z2 5
1
7
7
5
Pinhole Camera Model
•  Image coordinates are nonlinear function of world coordinates
•  Relationship between coordinates in the camera frame and sensor plane
2-D coordinates
Homogeneous coordinates
multiply both sides by Z
2
3 2
3
x
fX
Z 4 y 5 = 4 fY 5
1
Z
Rewrite by separating the camera parameters and the projection
20
Image Coordinates
•  Relationship between coordinates in the sensor plane and image
metric
coordinates
Linear transformation
pixel
coordinates
Calibration Matrix and Camera Model
•  Relationship between coordinates in the camera frame and image
Pinhole camera
Pixel coordinates
Calibration matrix
(intrinsic parameters)
Projection matrix
Camera model
21
Calibration Matrix and Camera Model
•  Relationship between coordinates in the world frame and image
More compactly
Transformation between camera coordinate
Systems and world coordinate system
Radial Distortion
Nonlinear transformation along the radial direction
New coordinates
Distortion correction: make lines straight
Coordinates of distorted points
22
Image of a point
Homogeneous coordinates of a 3-D point
Homogeneous coordinates of its 2-D image
Projection of a 3-D point to an image plane
Image of a line – homogeneous representation
Homogeneous representation of a 3-D line
Homogeneous representation of its 2-D image
Projection of a 3-D line to an image plane
23
Image of a line – 2D representations
Representation of a 3-D line
Projection of a line - line in the image plane
Special cases – parallel to the image plane, perpendicular
When λ -> infinity - vanishing points
In art – 1-point perspective, 2-point perspective, 3-point perspective
Orthographic Projection
•  Special case of perspective projection
–  Distance from center of projection to image plane is
infinite
Image
World
–  Also called “parallel projection”
–  What’s the projection matrix?
Slide by Steve Seitz
24
Perspective distortion
•  Problem for architectural photography: converging
verticals
Source: F. Durand
Perspective distortion
•  Problem for architectural photography: converging
Shifting the lens
verticals
upwards results in a
Tilting the camera
upwards results in
converging verticals
Keeping the camera
level, with an ordinary
lens, captures only the
bottom portion of the
building
picture of the
entire subject
•  Solution: view camera (lens shifted w.r.t. film)
http://en.wikipedia.org/wiki/Perspective_correction_lens
Source: F. Durand
25
Perspective distortion
•  Problem for architectural photography: converging
verticals
•  Result:
Source: F. Durand
Visual Illusions
Impossible Crate
Wrong Perspective
26
Vanishing points
Different sets of parallel lines in a plane intersect
at vanishing points, vanishing points form a horizon line
Ames Room Illusions
27
More Illusions
Which of the two monsters is bigger ?
Approximating an affine camera
Source: Hartley & Zisserman
28
What is happening here ?
Jaws dolly zoom scene:
https://www.youtube.com/watch?v=svEPWBxpYjo
Vertigo Hitchvock (YouTube)
Jaws dolly zoom scene
What is happening here ?
Jaws dolly zoom scene:
https://www.youtube.com/watch?v=svEPWBxpYjo
Zooms while camera moves away.
Vertigo Hitchvock (YouTube)
Jaws dolly zoom scene
29
The dolly zoom
•  Continuously adjusting the focal length while the
camera moves away from (or towards) the subject
•  “The Vertigo shot”
Real Lens Flaws: Chromatic Aberration
•  Lens has different refractive indices for different
wavelengths: causes color fringing
30
Lens flaws: Spherical aberration
•  Spherical lenses don’t focus light perfectly
•  Rays farther from the optical axis focus closer
Lens flaws: Vignetting
•  Reduction of brightness at the periphery
31
Radial Distortion
–  Caused by imperfect lenses
–  Deviations are most noticeable near the edge of the
lens
No distortion
Pin cushion
Barrel
Radial Distortion
Nonlinear transformation along the radial direction
New coordinates
Distortion correction: make lines straight
Coordinates of distorted points
32
Digital camera
•  A digital camera replaces film with a sensor array
–  Each cell in the array is light-sensitive diode that converts photons to electrons
–  Two common types
•  Charge Coupled Device (CCD)
•  Complementary metal oxide semiconductor (CMOS)
–  http://electronics.howstuffworks.com/digital-camera.htm
Slide by Steve Seitz
Color sensing in camera: Color filter array
Bayer grid
Estimate missing
components from
neighboring values
(demosaicing)
Why more green?
Human Luminance Sensitivity Function
Source: Steve Seitz
33