Jan-Michael Frahm, Enrique Dunn
Spring 2013

scene pose transformation: T scene

R
T 
R C
0
T
1

 projection: P
0





1 0 0 0
0 1 0 0
0 0 1 0






I 0
 sensor calibration: K
 


 f x s c
0 f y c y
0 0 1 x




  m

PM , P

KP T
0 scene
=


T  T
K R R C 

2D point
(3x1)
=
Camera to pixel coord. trans. matrix
(3x3)
Perspective projection matrix
(3x4)
World to camera coord. trans. matrix
(4x4)
3D point
(4x1)

• There are undesired effects in real situations o perspective distortion
• Camera artifacts o aperture is not infinitely small o lens o vignetting

o Caused by imperfect lenses o Deviations are most noticeable near the edge of the lens
No distortion Pin cushion Barrel slide: S. Lazebnik

• Brown’s distortion model o accounts for radial distortion o accounts for tangential distortion (distortion caused by lens placement errors)
(x u
, y u
) undistorted image point as in ideal pinhole camera
(x d
,y d
) distorted image point of camera with radial distortion
(x c
,y c
) distortion center
K
P n n n-th radial distortion coefficient n-th tangential distortion coefficient
• typically K
1 is used or K
1
, K
2
, P
1
, P
2

• There are undesired effects in real situations o perspective distortion
• Camera artifacts o aperture is not infinitely small o lens o vignetting, radial distortion

http://www.cambridgeincolour.com/tutorials/depth-of-field.htm
Slide by A. Efros

How can we control the depth of field?
• Changing the aperture size affects depth of field o A smaller aperture increases the range in which the object is approximately in focus o But small aperture reduces amount of light – need to increase exposure
Slide by A. Efros

• f number (f-stop) ratio of focal length to aperture f
number
= focal length aperture diameter

Large aperture = small DOF Small aperture = large DOF
Slide by A. Efros

• There are undesired effects in real situations o perspective distortion
• Camera artifacts o aperture is not infinitely small o lens o vignetting, radial distortion o depth of field

What does FOV depend on?
Slide by A. Efros

f f
FOV depends on focal length and size of the aperture
Smaller FOV = larger Focal Length
Slide by A. Efros

Large FOV, small f
Camera close to car
Small FOV, large f
Camera far from the car
Sources: A. Efros, F. Durand

wide-angle standard telephoto
Source: F. Durand

• Continuously adjusting the focal length while the camera moves away from (or towards) the subject http://en.wikipedia.org/wiki/Dolly_zoom slide: S. Lazebnik

• There are undesired effects in real situations o perspective distortion
• Camera artifacts o aperture is not infinitely small o lens o vignetting, radial distortion o depth of field o field of view

• A digital camera replaces film with a sensor array o Each cell in the array is light-sensitive diode that converts photons to electrons o Two common types
• Charge Coupled Device (CCD)
• Complementary metal oxide semiconductor (CMOS) o http://electronics.howstuffworks.com/digital-camera.htm
Slide by Steve Seitz

Color sensing in camera: Color filter array
Bayer grid
Estimate missing components from neighboring values
(demosaicing)
Why more green?
Human Luminance Sensitivity Function
Source: Steve Seitz

Problem with demosaicing: color moire
Slide by F. Durand

detector
Fine black and white detail in image misinterpreted as color information
Slide by F. Durand

• Requires three chips and precise alignment
• More expensive
CCD(R)
CCD(G)
CCD(B) slide: S. Lazebnik

Color sensing in camera: Foveon X3
• CMOS sensor
• Takes advantage of the fact that red, blue and green light penetrate silicon to different depths http://www.foveon.com/article.php?a=67 http://en.wikipedia.org/wiki/Foveon_X3_sensor better image quality
Source: M. Pollefeys

• There are undesired effects in real situations o perspective distortion
• Camera artifacts o Aperture is not infinitely small o Lens o Vignetting, radial distortion o Depth of field o Field of view o Color sensing

• Many cameras use CMOS sensors (mobile, DLSR, …)
• To save cost these are often rolling shutter cameras o lines are progressively exposed o line by line image reading
• Rolling shutter artifacts image source: Wikipedia

regular camera
(global shutter)
rolling shutter camera

• There are undesired effects in real situations o perspective distortion
• Camera artifacts o Aperture is not infinitely small o Lens o Vignetting, radial distortion o Depth of field o Field of view o Color sensing o Rolling shutter cameras

• Noise
• low light is where you most notice noise
• light sensitivity (ISO) / noise tradeoff
• stuck pixels
• In-camera processing
• oversharpening can produce halos
• Compression
• JPEG artifacts, blocking
• Blooming
• charge overflowing into neighboring pixels
• Smearing o columnwise overexposue
• Color artifacts
• purple fringing from microlenses,
• white balance modified from Steve Seitz

Conventional versus light field camera slide: Marc Levoy

Conventional versus light field camera slide: Marc Levoy

Conventional versus light field camera slide: Marc Levoy

Contax medium format camera Kodak 16-megapixel sensor
Adaptive Optics microlens array 125μ square-sided microlenses
4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens slide: Marc Levoy

slide: Marc Levoy

• stopping down = summing only the central portion of each microlens
Σ
Σ
f / N light field camera, with P × P pixels under each microlens, can produce views as sharp as an f / (N × P) conventional camera slide: Marc Levoy

• refocusing = summing windows extracted from several microlenses
Σ
Σ f/N light field camera can produce views with a shallow depth of field ( f / N ) focused anywhere within the depth of field of an f / (N × P) camera images: Marc Levoy

images: Marc Levoy

Extending the depth of field conventional photograph, main lens at f / 4 conventional photograph, main lens at f / 22 light field, main lens at f / 4, after all-focus algorithm
[Agarwala 2004] images: Marc Levoy

Σ
• moving the observer = moving the window we extract from the microlenses
Σ images: Marc Levoy

slide: Marc Levoy

slide: Marc Levoy

• Pinhole model: Mozi (470-390 BCE),
Aristotle (384-322 BCE)
• Principles of optics (including lenses):
Alhacen (965-1039 CE)
• Camera obscura: Leonardo da Vinci
(1452-1519), Johann Zahn (1631-1707)
• First photo: Joseph Nicephore Niepce (1822)
• Daguerréotypes (1839)
• Photographic film (Eastman, 1889)
• Cinema (Lumière Brothers, 1895)
• Color Photography (Lumière Brothers, 1908)
• Television (Baird, Farnsworth, Zworykin, 1920s)
• First consumer camera with CCD
Sony Mavica (1981)
• First fully digital camera: Kodak DCS100 (1990)
Alhacen’s notes
Niepce, “La Table Servie,” 1822
CCD chip

• Sergey Prokudin-Gorskii (1863-1944)
• Photographs of the Russian empire (1909-
1916)
Lantern projector http://en.wikipedia.org/wiki/Sergei_Mikhailovich_Prokudin-Gorskii http://www.loc.gov/exhibits/empire/

First digitally scanned photograph
• 1957, 176x176 pixels http://listverse.com/history/top-10-incredible-early-firsts-in-photography/