Capturing Light:
Properties of Light and
Geometry of Image Formation
Computer Vision
James Hays
Slides from Derek Hoiem,
Alexei Efros, Steve Seitz, and
David Forsyth
Administrative Stuff
• My Office hours, CoC building 222 (315 later)
– Monday and Wednesday 2-3
• TA Office hours, CoC building picnic benches
– To be announced
•
•
•
•
Staff is growing to support larger enrollment
Piazza should be your first stop for help
T-square is working
Matlab is available for students from OIT
Previous class: Introduction
• Overview of vision, examples of state of art,
preview of projects
Robotics
Computer
Vision
Machine
Learning
Image Processing
Feature Matching
Recognition
Graphics
Computational
Photography
Optics
Human
Computer
Interaction
Medical
Imaging
Neuroscience
What do you need to make a camera from scratch?
Image formation
Let’s design a camera
– Idea 1: put a piece of film in front of an object
– Do we get a reasonable image?
Slide source: Seitz
Pinhole camera
Idea 2: add a barrier to block off most of the rays
– This reduces blurring
– The opening known as the aperture
Slide source: Seitz
Pinhole camera
f
c
f = focal length
c = center of the camera
Figure from Forsyth
Camera obscura: the pre-camera
• Known during classical period in China and Greece
(e.g. Mo-Ti, China, 470BC to 390BC)
Illustration of Camera Obscura
Freestanding camera obscura at UNC Chapel Hill
Photo by Seth Ilys
Camera Obscura used for Tracing
Lens Based Camera Obscura, 1568
Accidental Cameras
Accidental Pinhole and Pinspeck Cameras
Revealing the scene outside the picture.
Antonio Torralba, William T. Freeman
Accidental Cameras
Accidental Pinhole and Pinspeck Cameras
Revealing the scene outside the picture.
Antonio Torralba, William T. Freeman
First Photograph
Oldest surviving photograph
– Took 8 hours on pewter plate
Joseph Niepce, 1826
Photograph of the first photograph
Stored at UT Austin
Niepce later teamed up with Daguerre, who eventually created Daguerrotypes
Image Formation
Digital Camera
Film
The Eye
A photon’s life choices
•
•
•
•
•
•
•
•
•
Absorption
Diffusion
Reflection
Transparency
Refraction
Fluorescence
Subsurface scattering
Phosphorescence
Interreflection
light source
λ
?
A photon’s life choices
•
•
•
•
•
•
•
•
•
Absorption
Diffusion
Reflection
Transparency
Refraction
Fluorescence
Subsurface scattering
Phosphorescence
Interreflection
light source
λ
A photon’s life choices
•
•
•
•
•
•
•
•
•
Absorption
Diffuse Reflection
Reflection
Transparency
Refraction
Fluorescence
Subsurface scattering
Phosphorescence
Interreflection
light source
λ
A photon’s life choices
•
•
•
•
•
•
•
•
•
Absorption
Diffusion
Specular Reflection
Transparency
Refraction
Fluorescence
Subsurface scattering
Phosphorescence
Interreflection
light source
λ
A photon’s life choices
•
•
•
•
•
•
•
•
•
Absorption
Diffusion
Reflection
Transparency
Refraction
Fluorescence
Subsurface scattering
Phosphorescence
Interreflection
light source
λ
A photon’s life choices
•
•
•
•
•
•
•
•
•
Absorption
Diffusion
Reflection
Transparency
Refraction
Fluorescence
Subsurface scattering
Phosphorescence
Interreflection
light source
λ
A photon’s life choices
•
•
•
•
•
•
•
•
•
Absorption
Diffusion
Reflection
Transparency
Refraction
Fluorescence
Subsurface scattering
Phosphorescence
Interreflection
light source
λ1
λ2
A photon’s life choices
•
•
•
•
•
•
•
•
•
Absorption
Diffusion
Reflection
Transparency
Refraction
Fluorescence
Subsurface scattering
Phosphorescence
Interreflection
light source
λ
A photon’s life choices
•
•
•
•
•
•
•
•
•
Absorption
Diffusion
Reflection
Transparency
Refraction
Fluorescence
Subsurface scattering
Phosphorescence
Interreflection
light source
t=1
t=n
A photon’s life choices
•
•
•
•
•
•
•
•
•
Absorption
Diffusion
Reflection
Transparency
Refraction
Fluorescence
Subsurface scattering
Phosphorescence
Interreflection
light source
λ
(Specular Interreflection)
Lambertian Reflectance
• In computer vision, surfaces are often
assumed to be ideal diffuse reflectors with no
dependence on viewing direction.
Digital camera
• A digital camera replaces film with a sensor
array
– Each cell in the array is light-sensitive diode that converts photons to electrons
– Two common types
• Charge Coupled Device (CCD)
• CMOS
– http://electronics.howstuffworks.com/digital-camera.htm
Slide by Steve Seitz
Sensor Array
CMOS sensor
Sampling and Quantization
Interlace vs. progressive scan
http://www.axis.com/products/video/camera/progressive_scan.htm
Slide by Steve Seitz
Progressive scan
http://www.axis.com/products/video/camera/progressive_scan.htm
Slide by Steve Seitz
Interlace
http://www.axis.com/products/video/camera/progressive_scan.htm
Slide by Steve Seitz
Rolling Shutter
The Eye
• The human eye is a camera!
– Iris - colored annulus with radial muscles
– Pupil - the hole (aperture) whose size is controlled by the iris
– What’s the “film”?
– photoreceptor cells (rods and cones) in the retina
Slide by Steve Seitz
The Retina
Cross-section of eye
Cross section of retina
Pigmented
epithelium
Ganglion axons
Ganglion cell layer
Bipolar cell layer
Receptor layer
What humans don’t have: tapetum lucidum
Two types of light-sensitive receptors
Cones
cone-shaped
less sensitive
operate in high light
color vision
Rods
rod-shaped
highly sensitive
operate at night
gray-scale vision
© Stephen E. Palmer, 2002
Rod / Cone sensitivity
Distribution of Rods and Cones
# Receptors/mm2
.
Fovea
150,000
Rods
Blind
Spot
Rods
100,000
50,000
0
Cones
Cones
80 60 40 20 0
20 40 60 80
Visual Angle (degrees from fovea)
Night Sky: why are there more stars off-center?
Averted vision: http://en.wikipedia.org/wiki/Averted_vision
© Stephen E. Palmer, 2002
Eye Movements
• Saccades
•
Can be consciously controlled. Related to perceptual attention.
•
200ms to initiation, 20 to 200ms to carry out. Large amplitude.
• Microsaccades
•
Involuntary. Smaller amplitude. Especially evident during
prolonged fixation. Function debated.
• Ocular microtremor (OMT)
•
involuntary. high frequency (up to 80Hz), small amplitude.
Electromagnetic Spectrum
Human Luminance Sensitivity Function
http://www.yorku.ca/eye/photopik.htm
Visible Light
Why do we see light of these wavelengths?
…because that’s where the
Sun radiates EM energy
© Stephen E. Palmer, 2002
The Physics of Light
Any patch of light can be completely described
physically by its spectrum: the number of photons
(per time unit) at each wavelength 400 - 700 nm.
# Photons
(per ms.)
400 500
600
700
Wavelength (nm.)
© Stephen E. Palmer, 2002
The Physics of Light
Some examples of the spectra of light sources
.
B. Gallium Phosphide Crystal
# Photons
# Photons
A. Ruby Laser
400 500
600
700
400 500
Wavelength (nm.)
700
Wavelength (nm.)
D. Normal Daylight
# Photons
C. Tungsten Lightbulb
# Photons
600
400 500
600
700
400 500
600
700
© Stephen E. Palmer, 2002
The Physics of Light
% Photons Reflected
Some examples of the reflectance spectra of surfaces
Red
400
Yellow
700 400
Blue
700 400
Wavelength (nm)
Purple
700 400
700
© Stephen E. Palmer, 2002
The Psychophysical Correspondence
There is no simple functional description for the perceived
color of all lights under all viewing conditions, but …...
A helpful constraint:
Consider only physical spectra with normal distributions
mean
area
# Photons
400
500
variance
600
700
Wavelength (nm.)
© Stephen E. Palmer, 2002
The Psychophysical Correspondence
# Photons
Mean
blue
Hue
green yellow
Wavelength
© Stephen E. Palmer, 2002
The Psychophysical Correspondence
# Photons
Variance
Saturation
hi. high
med. medium
low
low
Wavelength
© Stephen E. Palmer, 2002
The Psychophysical Correspondence
Area
Brightness
# Photons
B. Area
Lightness
bright
dark
Wavelength
© Stephen E. Palmer, 2002
Physiology of Color Vision
Three kinds of cones:
440
RELATIVE ABSORBANCE (%)
.
530 560 nm.
100
S
M
L
50
400
450
500
550
600 650
WAVELENGTH (nm.)
• Why are M and L cones so close?
• Why are there 3?
© Stephen E. Palmer, 2002
Impossible Colors
Can you make the cones respond in ways that typical
light spectra never would?
http://en.wikipedia.org/wiki/Impossible_colors
Tetrachromatism
Bird cone
responses
• Most birds, and many other animals, have
cones for ultraviolet light.
• Some humans, mostly female, seem to have
slight tetrachromatism.
More Spectra
metamers
Practical Color Sensing: Bayer Grid
• Estimate RGB
at ‘G’ cells from
neighboring
values
Slide by Steve Seitz
Color Image
R
G
B
Images in Matlab
• Images represented as a matrix
• Suppose we have a NxM RGB image called “im”
– im(1,1,1) = top-left pixel value in R-channel
– im(y, x, b) = y pixels down, x pixels to right in the bth channel
– im(N, M, 3) = bottom-right pixel in B-channel
• imread(filename) returns a uint8 image (values 0 to 255)
– Convert to double format (values 0 to 1) with im2double
row
column
0.92
0.95
0.89
0.96
0.71
0.49
0.86
0.96
0.69
0.79
0.91
0.93
0.89
0.72
0.95
0.81
0.62
0.84
0.67
0.49
0.73
0.94
0.94
0.82
0.51
0.92
0.88
0.95
0.81
0.89
0.60
0.96
0.74
0.71
0.54
0.49
0.56
0.86
0.90
0.96
0.89
0.69
0.79
0.91
0.97
0.89
0.55
0.93
0.94
0.89
0.87
0.72
0.58
0.95
0.58
0.81
0.85
0.62
0.66
0.84
0.67
0.67
0.49
0.49
0.73
0.94
0.62
0.56
0.51
0.94
0.56
0.82
0.57
0.51
0.92
0.50
0.88
0.95
0.51
0.81
0.89
0.48
0.60
0.96
0.43
0.74
0.71
0.33
0.54
0.49
0.41
0.56
0.86
0.90
0.96
0.89
0.69
0.79
0.91
0.37
0.31
0.42
0.97
0.46
0.89
0.37
0.55
0.93
0.60
0.94
0.89
0.39
0.87
0.72
0.37
0.58
0.95
0.42
0.58
0.81
0.61
0.85
0.62
0.78
0.66
0.84
0.67
0.67
0.49
0.49
0.73
0.94
0.85
0.75
0.57
0.62
0.91
0.56
0.80
0.51
0.94
0.58
0.56
0.82
0.73
0.57
0.51
0.88
0.50
0.88
0.77
0.51
0.81
0.69
0.48
0.60
0.78
0.43
0.74
0.33
0.54
0.41
0.56
0.90
0.89
0.97
0.92
0.41
0.37
0.87
0.31
0.88
0.42
0.97
0.50
0.46
0.89
0.92
0.37
0.55
0.90
0.60
0.94
0.73
0.39
0.87
0.79
0.37
0.58
0.77
0.42
0.58
0.61
0.85
0.78
0.66
0.67
0.49
0.93
0.81
0.49
0.85
0.90
0.75
0.89
0.57
0.62
0.61
0.91
0.56
0.91
0.80
0.51
0.94
0.58
0.56
0.71
0.73
0.57
0.73
0.88
0.50
0.89
0.77
0.51
0.69
0.48
0.78
0.43
0.33
0.41
0.92
0.95
0.91
0.97
0.97
0.92
0.79
0.41
0.37
0.45
0.87
0.31
0.49
0.88
0.42
0.82
0.50
0.46
0.90
0.92
0.37
0.93
0.90
0.60
0.99
0.73
0.39
0.79
0.37
0.77
0.42
0.61
0.78
0.99
0.91
0.92
0.93
0.95
0.81
0.85
0.49
0.85
0.33
0.90
0.75
0.74
0.89
0.57
0.93
0.61
0.91
0.99
0.91
0.80
0.97
0.94
0.58
0.93
0.71
0.73
0.73
0.88
0.89
0.77
0.69
0.78
R
0.92
0.95
0.91
0.97
0.97
0.92
0.79
0.41
0.45
0.87
0.49
0.88
0.82
0.50
0.90
0.92
0.93
0.90
0.99
0.73
0.79
0.77
0.99
0.91
0.92
0.93
0.95
0.81
0.85
0.49
0.33
0.90
0.74
0.89
0.93
0.61
0.99
0.91
0.97
0.94
0.93
0.71
0.73
0.89
G
0.92
0.95
0.91
0.97
0.79
0.45
0.49
0.82
0.90
0.93
0.99
0.99
0.91
0.92
0.95
0.85
0.33
0.74
0.93
0.99
0.97
0.93
B
Color spaces
• How can we represent color?
http://en.wikipedia.org/wiki/File:RGB_illumination.jpg
Color spaces: RGB
Default color space
0,1,0
R
(G=0,B=0)
G
1,0,0
(R=0,B=0)
0,0,1
Some drawbacks
B
(R=0,G=0)
• Strongly correlated channels
• Non-perceptual
Image from: http://en.wikipedia.org/wiki/File:RGB_color_solid_cube.png
Color spaces: HSV
Intuitive color space
H
(S=1,V=1)
S
(H=1,V=1)
V
(H=1,S=0)
Color spaces: YCbCr
Fast to compute, good for
compression, used by TV
Y=0
Y=0.5
Y
(Cb=0.5,Cr=0.5)
Cr
Cb
Cb
(Y=0.5,Cr=0.5)
Y=1
Cr
(Y=0.5,Cb=05)
Color spaces: L*a*b*
“Perceptually uniform”* color space
L
(a=0,b=0)
a
(L=65,b=0)
b
(L=65,a=0)
If you had to choose, would you rather go
without luminance or chrominance?
If you had to choose, would you rather go
without luminance or chrominance?
Most information in intensity
Only color shown – constant intensity
Most information in intensity
Only intensity shown – constant color
Most information in intensity
Original image
Back to grayscale intensity
0.92
0.95
0.89
0.96
0.71
0.49
0.86
0.96
0.69
0.79
0.91
0.93
0.89
0.72
0.95
0.81
0.62
0.84
0.67
0.49
0.73
0.94
0.94
0.82
0.51
0.88
0.81
0.60
0.74
0.54
0.56
0.90
0.89
0.97
0.89
0.55
0.94
0.87
0.58
0.58
0.85
0.66
0.67
0.49
0.62
0.56
0.51
0.56
0.57
0.50
0.51
0.48
0.43
0.33
0.41
0.37
0.31
0.42
0.46
0.37
0.60
0.39
0.37
0.42
0.61
0.78
0.85
0.75
0.57
0.91
0.80
0.58
0.73
0.88
0.77
0.69
0.78
0.97
0.92
0.41
0.87
0.88
0.50
0.92
0.90
0.73
0.79
0.77
0.93
0.81
0.49
0.90
0.89
0.61
0.91
0.94
0.71
0.73
0.89
0.92
0.95
0.91
0.97
0.79
0.45
0.49
0.82
0.90
0.93
0.99
0.99
0.91
0.92
0.95
0.85
0.33
0.74
0.93
0.99
0.97
0.93
Five minute break
What is wrong with this picture?
The Geometry of Image Formation
Mapping between image and world
coordinates
– Pinhole camera model
– Projective geometry
• Vanishing points and lines
– Projection matrix
Today’s class: Camera and World Geometry
How tall is this woman?
How high is the camera?
What is the camera
rotation?
What is the focal length of
the camera?
Which ball is closer?
Dimensionality Reduction Machine (3D to 2D)
3D world
2D image
Point of observation
Figures © Stephen E. Palmer, 2002
Projection can be tricky…
Slide source: Seitz
Projection can be tricky…
Slide source: Seitz
Projective Geometry
What is lost?
• Length
Who is taller?
Which is closer?
Length is not preserved
A’
C’
B’
Figure by David Forsyth
Projective Geometry
What is lost?
• Length
• Angles
Parallel?
Perpendicular?
Projective Geometry
What is preserved?
• Straight lines are still straight
Vanishing points and lines
Parallel lines in the world intersect in the image at a
“vanishing point”
Vanishing points and lines
Vanishing Point
Vanishing Line
o
Vanishing Point
o
Vanishing points and lines
Vertical vanishing
point
(at infinity)
Vanishing
point
Slide from Efros, Photo from Criminisi
Vanishing
point
Projection: world coordinatesimage coordinates
.
Optical
Center
(u0, v0)
.
.
v
u
u 
p 
v 
f
.
Camera
Center
(tx, ty, tz)
Z
Y
X 
P  Y 
 
 Z 
To be continued