Some figures from Steve Palmer
16-721: Advanced Machine Perception
A. Efros, CMU, Spring 2006

• Name:
• Research area / project / advisor
• What you want to learn in this class?
• When I am not working, I ______________
• Favorite fruit:
Class mailing list…

Film
Digital Camera
The Eye

Monocular Visual Field: 160 deg (w) X 135 deg (h)
Binocular Visual Field: 200 deg (w) X 135 deg (h)

3D world 2D image
Point of observation
Figures © Stephen E. Palmer, 2002

3D world 2D image
Point of observation
Painted backdrop

Figure by Leonard McMillan
Q: What is the set of all things that we can ever see?
A: The Plenoptic Function (Adelson & Bergen)
Let’s start with a stationary person and try to parameterize everything that he can see…

P( q,f
) is intensity of light
• Seen from a single view point
• At a single time
• Averaged over the wavelengths of the visible spectrum
(can also do P(x,y), but spherical coordinate are nicer)

P( q,f,l
) is intensity of light
• Seen from a single view point
• At a single time
• As a function of wavelength

See also: 2003 New Years Eve http://www.panoramas.dk/fullscreen3/f1.html
All light rays through a point form a ponorama
Totally captured in a 2D array -P( q,f
)
Where is the geometry???

P( q,f,l
,t) is intensity of light
• Seen from a single view point
• Over time
• As a function of wavelength

t y x

P( q,f,l
,t,V
X
,V
Y
,V
Z
) is intensity of light
• Seen from ANY viewpoint
• Over time
• As a function of wavelength

P( q,f,l
,t,V
X
,V
Y
,V
Z
)
• Can reconstruct every possible view, at every moment, from every position, at every wavelength
• Contains every photograph, every movie, everything that anyone has ever seen! it completely captures our visual reality! Not bad for a function…

The human eye is a camera!
• Iris colored annulus with radial muscles
• Pupil the hole (aperture) whose size is controlled by the iris
• What’s the “film”?
– photoreceptor cells (rods and cones) in the retina

Cross-section of eye Cross section of retina
Pigmented epithelium
Ganglion axons
Ganglion cell layer
Bipolar cell layer
Receptor layer

Light

Two types of light-sensitive receptors
C on es cone-shaped less sensitive operate in high light color vision
Rods rod-shaped highly sensitive operate at night gray-scale vision
© Stephen E. Palmer, 2002

The famous sockmatching problem…

Distribution of Rods and Cones
Fovea
Blind
Spot
150,000
100,000
Rods
Rods
50,000
Cones Cones
0
80 60 40 20 0 20 40 60 80
Visual Angle (degrees from fovea)
Night Sky: why are there more stars off-center?
© Stephen E. Palmer, 2002

Human Luminance Sensitivity Function http://www.yorku.ca/eye/photopik.htm

Why do we see light of these wavelengths?
…because that’s where the
Sun radiates EM energy
© Stephen E. Palmer, 2002

The Physics of Light
Any patch of light can be completely described physically by its spectrum: the number of photons
(per time unit) at each wavelength 400 - 700 nm.
# Photons
(per ms.)
400 500 600 700
Wavelength (nm.)
© Stephen E. Palmer, 2002

The Physics of Light
Some examples of the spectra of light sources
A. Ruby Laser B. Gallium Phosphide Crystal
400 500 600 700
Wavelength (nm.)
C. Tungsten Lightbulb
400 500 600 700
Wavelength (nm.)
D. Normal Daylight
400 500 600 700 400 500 600 700
© Stephen E. Palmer, 2002

The Physics of Light
Some examples of the reflectance spectra of surfaces
Red
Yellow Blue Purple
400 700 400 700 400 700 400 700
Wavelength (nm)
© Stephen E. Palmer, 2002

The Psychophysical Correspondence
There is no simple functional description for the perceived color of all lights under all viewing conditions, but …...
A helpful constraint:
Consider only physical spectra with normal distributions mean
# Photons
400 area variance
500 600
Wavelength (nm.)
700
© Stephen E. Palmer, 2002

The Psychophysical Correspondence
Mean Hue blue green yellow
Wavelength
© Stephen E. Palmer, 2002

The Psychophysical Correspondence
Variance Saturation hi.
high med.
medium low low
Wavelength
© Stephen E. Palmer, 2002

The Psychophysical Correspondence
Area Brightness
B. Area Lightness bright dark
Wavelength
© Stephen E. Palmer, 2002

Physiology of Color Vision
Three kinds of cones:
440 530 560 nm.
100
S M L
50
400 450 500 550 600 650
WAVELENGTH (nm.)
• Why are M and L cones so close?
© Stephen E. Palmer, 2002

Retinal Processing
© Stephen E. Palmer, 2002

Single Cell Recording
Microelectrode
Amplifier
Electrical response
(action potentials) mV
Time
© Stephen E. Palmer, 2002

Single Cell Recording
© Stephen E. Palmer, 2002

Retinal Receptive Fields
Receptive field structure in ganglion cells :
On-center Off-surround
Response
Time
Stimulus condition Electrical response
© Stephen E. Palmer, 2002

Retinal Receptive Fields
Receptive field structure in ganglion cells :
On-center Off-surround
Response
Time
Stimulus condition Electrical response
© Stephen E. Palmer, 2002

Retinal Receptive Fields
Receptive field structure in ganglion cells :
On-center Off-surround
Response
Time
Stimulus condition Electrical response
© Stephen E. Palmer, 2002

Retinal Receptive Fields
Receptive field structure in ganglion cells :
On-center Off-surround
Response
Time
Stimulus condition Electrical response
© Stephen E. Palmer, 2002

Retinal Receptive Fields
Receptive field structure in ganglion cells :
On-center Off-surround
Response
Time
Stimulus condition Electrical response
© Stephen E. Palmer, 2002

Retinal Receptive Fields
Receptive field structure in ganglion cells :
On-center Off-surround
Response
Time
Stimulus condition Electrical response
© Stephen E. Palmer, 2002

Retinal Receptive Fields
RF of On-center Off-surround cells
Neural Response
Center
Receptive Field
Firing
Rate
Response Profile on-center
Surround off-surround
Horizontal Position
On Off
© Stephen E. Palmer, 2002

Retinal Receptive Fields
RF of Off-center On-surround cells
Neural Response Receptive Field Response Profile
Firing
Rate on-surround
On Off off-center
Horizontal Position
© Stephen E. Palmer, 2002

Retinal Receptive Fields

Retinal Receptive Fields
Receptive field structure in bipolar cells
Light
© Stephen E. Palmer, 2002

Retinal Receptive Fields
Receptive field structure in bipolar cells
LIGHT
Receptors
Direct excitatory
component (D)
Indirect inhibitory component (I)
Horizontal
Cells
Direct Path
Bipolar Cell
Indirect Path
A. WIRING DIAGRAM
D + I
B. RECEPTIVE FIELD PROFILES
© Stephen E. Palmer, 2002

Cortical Area V1 aka:
Primary visual cortex
Striate cortex
Brodman’s area 17
LGN
Thalamus
Parietal visual cortex
Dorsal
Stream Striate cortex
(V1)
Eye Optic nerve Temporal visual cortex
Ventral
Extrastriate cortex
Stream
© Stephen E. Palmer, 2002

Cortical Receptive Fields
Single-cell recording from visual cortex
David Hubel & Thorston Wiesel
© Stephen E. Palmer, 2002

Cortical Receptive Fields
Single-cell recording from visual cortex
Time
© Stephen E. Palmer, 2002

Cortical Receptive Fields
Three classes of cells in V1
Simple cells
Complex cells
Hypercomplex cells
© Stephen E. Palmer, 2002

Cortical Receptive Fields
Simple Cells: “Line Detectors”
B. Dark Line Detector
Firing
Rate
Horizontal Position
© Stephen E. Palmer, 2002

Cortical Receptive Fields
Simple Cells: “Edge Detectors”
C. Dark-to-light Edge Detector D. Light-to-dark Edge Detector
Firing
Rate
Horizontal Position
Firing
Rate
Horizontal Position
© Stephen E. Palmer, 2002

Cortical Receptive Fields
Constructing a line detector
Retina LGN
Receptive Fields
Center-
Surround
Cells
© Stephen E. Palmer, 2002

Cortical Receptive Fields
Complex Cells
STIMULUS
0
0 o
NEURAL RESPONSE
Time
© Stephen E. Palmer, 2002

Cortical Receptive Fields
Complex Cells
STIMULUS
60 o
NEURAL RESPONSE
Time
© Stephen E. Palmer, 2002

Cortical Receptive Fields
Complex Cells
STIMULUS
90 o
NEURAL RESPONSE
Time
© Stephen E. Palmer, 2002

Cortical Receptive Fields
Complex Cells
STIMULUS
120 o
NEURAL RESPONSE
Time
© Stephen E. Palmer, 2002

Cortical Receptive Fields
Constructing a Complex Cell
Retina Cortical Area V1
Receptive Fields Simple Cells
© Stephen E. Palmer, 2002

Cortical Receptive Fields
Hypercomplex Cells
© Stephen E. Palmer, 2002

Cortical Receptive Fields
Hypercomplex Cells
© Stephen E. Palmer, 2002

Cortical Receptive Fields
Hypercomplex Cells
© Stephen E. Palmer, 2002

Cortical Receptive Fields
Hypercomplex Cells
“End-stopped” Cells
© Stephen E. Palmer, 2002

Cortical Receptive Fields
“End-stopped” Simple Cells
© Stephen E. Palmer, 2002

Cortical Receptive Fields
Constructing a Hypercomplex Cell
RETINA CORTICAL AREA V1
Receptive Fields Complex Cell End-stopped Cell
© Stephen E. Palmer, 2002

So, why “edge-like” structures in the Plenoptic Function?

1
The real world is
High dynamic range
1500
25,000
400,000
2,000,000,000

Image pixel ( 312 , 284 ) = 42
42 photos?

Real world
10 -6
Picture
10 -6
High dynamic range 10 6
10 6
0 to 255

Real world
10 -6
Picture
10 -6
High dynamic range 10 6
10 6
0 to 255

The eye has a huge dynamic range
Do we see a true radiance map?

"Every light is a shade, compared to the higher lights, till you come to the sun; and every shade is a light, compared to the deeper shades, till you come to the night."
— John Ruskin, 1879

Campbell-Robson contrast sensitivity curve

Eye is sensitive to changes
(more on this later…)