COGS 300
Notes
February 25, 2014
Today’s “Fun” Example: Cavanagh’s “Vanishing Donut”
video: Vanishing Donut
The video shows a rectangular donut that moves in one direction while the background moves in
the other. The speed of motion never changes. As the movie plays, the relative luminance of the
red and green are varied to cover a range of green with higher luminance than red to red with higher
luminance than green
As the ratio moves through equiluminance, the relative motion becomes less evident and the rectangular donut dissolves into a jelly-like tangle of non-rigid flows
Video credit: Pat Cavanagh
Visit Pat Cavanagh’s Colour Motion Demo Web Page for internet access to today’s video (and for
other isoluminance colour demos).
This Week’s Learning Goals
1. To consider Marr’s “three levels” at which any machine carrying out an information-processing
task must be understood
2. To provide a “fun” overview of human colour vision
— including its deficiencies (compared to some other animals)
3. To explore two state-of-the-art computer vision systems:
(a) UBC’s Curious George: a successful, embodied robot vision system
(b) CMU’s “big data” NEIL (Never Ending Image Learner)
Marr’s Three Levels
Chapter 1 of “Vision” by D. Marr argues that a computational approach to vision occurs on at least
three levels:
Marr’s Three Levels
1. Computational theory. What? / Why solve?
2. Representation/Algorithm. How? (on an abstract level)
3. Mechanism. How? (on a physical level)
1
The level of “computational theory” defines the problem to be solved, given both the needs/goals
of the problem solver and constraints imposed by the (physical) environment in which the problem
solver operates.
The level of “representation/algorithm” deals, at an abstract level, with the “language” used to
represent information and the mechanisms available to compute with that information
The level of “mechanism” deals, at a physical level, with how the requisite computations are implemented on available “hardware.”
Sometimes, understanding a task at the levels of “computational theory” and “representation/algorithm”
leads to new understanding at the level of “mechanism.”
One example: A complete anatomical description of the visual system of musca domestica (the
housefly) was determined in the 1970s.1 Thus, all the visual neurons and their inter-connections
were known for the housefly. The existing model of a biological neuron was that of a “linear
threshold” device. Given this model of (the physical) mechanism, there was no ability for the
visual system of the housefly to perform a multiplication. But, a computational account of what
the visual system of the housefly actually was computing required the ability to perform multiplication. In this case the discovery of a real computational constraint (multiplication) led to further
investigation and revision of the model of an individual neuron. Biological neurons are much more
than linear threshold devices. In particular, an individual neuron can perform the equivalent of a
multiplication of its input signals.
Quiz 1
According to Woodham, what’s the message of colour in aerial perspective?
A)
B)
C)
D)
E)
Distance
Wavelength
Spectral reflectance
The object in view
All of the above
Quiz 2
[According to Marr] what is NOT a level of description for a machine that solves an information
processing problem?
A) Basic component & circuit analysis
B) Study of particular mechanisms
C) That of the algorithm
1
There’s reference to this work in the Marr & Poggio assigned reading.
2
D) Theory of overall computation
E) None of the above
Quiz 3
According to Marr, which of the following is the first step to compose stereoscopic disparity:
A)
B)
C)
D)
E)
Location on surface in scene must be chosen from one image
Cross section image
Close one eye and look side ways
None of the above
A&B
Quiz 4
In Woodham’s article, what are the changes that confound the measurements in spectral classification techniques?
A)
B)
C)
D)
E)
Shape
Colour
Illumination
A&B
A&C
Quiz 5
[Woodham] “problems for colour perception” are:
A)
B)
C)
D)
Colour the medium confounds colour the message
Different spectral reflectances can produce identical responses
Color measurement is independent of spectral characteristics of the illumination
A&B
3
Colour
Colour
• Light is produced in different amounts at different
wavelengths by each light source
• Light is differentially reflected at each wavelength,
which gives objects their natural colour (surface
albedo)
• The sensation of colour is determined by the human
visual system, based on the product of light and reflectance
The visible range is about 400nm to 700nm.
There are significant variations in the relative spectral power of daylight measured at different
times of day and under different conditions of sun and cloud.
Even more significant are the variations in the relative spectral power of daylight and artificial
light.
Relative Spectral Power of Two Standard Illuminants
D65 models sunlight. Illuminant A models incandescent light
Forsyth & Ponce (2nd ed.) Figure 3.4
Relative spectral power plotted against wavelength in nm
Illuminant A models the light from a 100W Tungsten filament light bulb, with colour temperature
2,800K. Illuminant D-65 models daylight. Figure plotted from data available at
http://www.cvrl.org/
4
Relative Spectral Power of Four Lamps
Forsyth & Ponce (2nd ed.) Figure 3.5
Relative spectral power plotted against wavelength in nm
The relative spectral power distribution of four different lamps from the Mitsubishi Electric Corporation. Note the bright, narrow bands that come from the fluorescing phosphors in the fluorescent
lamp.
Spectral Albedo of Natural Surfaces
Forsyth & Ponce (2nd ed.) Figure 3.6
Spectral albedo for several different flowers, with colour names attached. Notice that different
colours typically have different spectral albedo, but that different spectral albedoes may result in
the same perceived colour (compare the two whites). Spectral albedo typically (but not always) is
a smooth function of wavelength.
See Forsyth & Ponce (2nd ed.) Figure 3.7 for additional examples.
5
Colour Appearance
• Reflected light at each wavelength is the product of illumination and surface reflectance at
that wavelength
• Surface reflectance often is modeled as having two components:
— Lambertian reflectance: equal in all directions (diffuse)
— Specular reflectance: mirror reflectance (shiny spots)
Colour Names for Cartoon Spectra
Additive Colour Mixing
Colours combine by adding
the colour spectra
Examples: CRT phosphors,
multiple projectors aimed at
a screen
Red and green make. . .
6
. . . yellow!
Subtractive Colour Mixing
Colours combine by
multiplying the colour spectra
Examples: most
photographic films, paint,
cascaded optical filters,
crayons
Cyan (in crayons, called “blue”) and yellow make. . .
. . . green!
Colour Matching Experiments: I
Forsyth & Ponce (2nd ed.) Figure 3.2
Show a split field to subjects. One side shows the light whose colour one wants to match. The
other a weighted mixture of three primaries (fixed lights).
The human observer sees a test light, T , and can adjust the amount of each of three primaries,
P1 , P2 and P3 , in a mixture that is displayed next to the test light. The observer is asked to
adjust the amounts so that the mixture looks the same as the test light. The mixture of primaries
can be written as ω1 P1 + ω2 P2 + ω3 P3 . If the mixture matches the test light, then we write
T = ω1 P1 + ω2 P2 + ω3 P3 . For most people, three primaries are sufficient to achieve a match
for many colours, and for all colours if we allow subtractive matching (i.e., some amount of some
of the primaries is mixed with the test light to achieve a match). Some people will require fewer
primaries. Furthermore, most people will choose the same mixture weights to match a given test
light.
7
Colour Matching Experiments: I (cont’d)
Figure credit: Brian Wandell, Foundations of Vision, Sinauer Associates, 1995
Sub-figure (A) shows a top view of the experimental apparatus. Sub-figure (B) shows the appearance of the stimuli to the observer.
Example 1: Colour Matching Experiment
Example credit: Bill Freeman
8
Example 1: Colour Matching Experiment (cont’d)
Example credit: Bill Freeman
Example 1: Colour Matching Experiment (cont’d)
Example credit: Bill Freeman
The (final) bar graph on the right show the primary colour amounts needed for a match
9
Example 2: Colour Matching Experiment
Example credit: Bill Freeman
Example 2: Colour Matching Experiment (cont’d)
Example credit: Bill Freeman
10
Example 2: Colour Matching Experiment (cont’d)
Example credit: Bill Freeman
Colour Matching Experiments: II
• Many colours can be represented as a positive weighted sum of A, B, C
• Write
M = aA + bB + cC
where the = sign should be read as “matches”
• This is additive matching
• Defines a colour description system
— two people who agree on A, B, C need only supply (a, b, c)
Colour Matching Experiments: II (cont’d)
• Some colours can’t be matched this way
• Instead, we must write
M + aA = bB + cC
where, again, the = sign should be read as “matches”
• This is subtractive matching
• Interpret this as (−a, b, c)
Problem for designing displays:
Choose phosphors R, G, B so that positive linear combinations match a large set of colours
11
Principle of Trichromacy
Experimental facts:
• Three primaries work for most people, provided we allow subtractive matching
— Exceptional people can match with two or only one primary
— This likely is caused by biological deficiencies
• Most people make the same matches
— There are some anomalous trichromats, who use three primaries but match with different
combinations
Human Cone Sensitivities
Figure credit: Brian Wandell, Foundations of Vision, Sinauer Associates, 1995
The measurements shown are based on a light source at the cornea so that the wavelength loss due
to the cornea, lens and other inert pigments of the eye plays a role in determining overall sensitivity.
Two lights whose spectral power distributions appear identical to most observers are called metamers.
12
Metameric Lights
(A) A tungsten bulb
(B) TV monitor set to match (A)
Figure credit: Brian Wandell, Foundations of Vision, Sinauer Associates, 1995
Standard Photopic Luminosity Function (CIE 1931)
Figure credit: http://en.wikipedia.org/wiki/Luminosity_function
The standard (photopic) luminosity function was established by the Commission Internationale de
l’Éclairage (CIE) to account for the variable sensitivity of the human eye to radiation at different
wavelengths. It converts radiant energy (watts) into luminous energy (lumens).
The standard day time (photopic) luminosity function, shown above, has a peak value of 683
lumens/watt at 555 nm.
For very low levels of light, the night time (scotopic) luminosity function shifts towards the violet
for young eyes (under age 30) with a peak value of 1746 lumens/watt at 507nm.
13
Standard Scotopic Luminosity Function (CIE 1951)
Figure credit: http://hyperphysics.phy-astr.gsu.edu/Hbase/vision/bright.html
This figure adds the standard scotopic luminosity function for comparison purposes. Scotopic
vision is primarily rod vision. Note that the peak value of the scotopic curve is much higher than
that of the photopic curve. Note also that both the photopic and scotopic curves have the same
value of 683 lumens/watt at 555 nm
Grassman’s Laws
• For colour matches:
— symmetry:
U =V ⇔ V =U
— transitivity:
U = V and V = W ⇒ U = W
— proportionality:
U = V ⇔ tU = tV
— additivity: if any two (or more) of the statements
U =V,
W = X,
(U + W ) = (V + X)
are true, then so is the third
• These statements are as true as any biological law. They mean that colour matching in film
colour mode is linear
14
Linear Colour Spaces
• A choice of primaries yields a linear colour space
— the coordinates of a colour are given by the weights of the primaries used to match it
• Choice of primaries is equivalent to choice of colour space
• RGB: Primaries are monochromatic energies, say 645.2 nm, 526.3 nm, 444.4 nm
• CIE XYZ: Primaries are imaginary, but have other convenient properties. Colour coordinates
are (X, Y, Z), where X is the amount of the X primary, etc.
RGB Colour Matching Functions
• Primaries
monochromatic
• Wavelengths
645.2, 526.3 and
444.4 nm
• Negative parts
means some
colours can be
matched only
subtractively
Forsyth & Ponce (2nd ed.) Figure 3.9
CIE XYZ Colour Matching Functions
CIE XYZ: Colour
matching functions are
positive everywhere,
but primaries are
imaginary. Usually
draw x, y, where
x = X/(X + Y + Z)
y = Y /(X + Y + Z)
Forsyth & Ponce (2nd ed.) Figure 3.8
15
Overall brightness is
ignored
Geometry of Colour (CIE)
• White is in the
center, with
saturation
increasing towards
the boundary
• Mixing two
coloured lights
creates colours on a
straight line
• Mixing 3 colours
creates colours
within a triangle
• Curved edge means
there are no 3 actual
lights that can create
all colours that
humans perceive!
The figure provides a qualitative rendering of the CIE xy colour space. The blobby region represents visible colours. There are sets of (x, y) coordinates that don’t represent real colours, because
the primaries are not real lights. Recall: Imaginary primaries are chosen so that the colour matching
functions are positive everywhere.
RGB Colour Space
The sub-space of CIE
colours that can be
displayed on a typical
computer monitor
(phosphor limitations keep
the space quite small)
16
Uniform Colour Spaces
• McAdam ellipses (next slide) demonstrate that differences in x, y are a poor guide to differences in perceived colour
— Each ellipse shows colours perceived to be the same
• Construct colour spaces so that differences in coordinates are a good guide to differences in
perceived colour
McAdam Ellipses
10 times actual size
Actual size
Forsyth & Ponce (2nd ed.) Figure 3.14
This figure shows variations in colour matches in CIE xy colour space. At the center of the ellipse
is the colour of a test light; the size of the ellipse represents the scatter of lights that the human
observers tested would match to the test colour; the boundary shows where the just noticeable
difference is. The ellipses in the figure on the left have been magnified 10× for clarity, and on the
right they are plotted to scale. The ellipses are known as MacAdam ellipses, after their inventor.
Notice that the ellipses at the top are larger than those at the bottom of the figure, and that they
rotate as they move up. This means that the magnitude of the difference in x, y coordinates is a
poor guide to the difference in perceived colour.
Non-Linear Colour Spaces Uniform
• HSV: (Hue, Saturation, Value) are non-linear functions of XYZ.
— because hue relations are naturally expressed in a circle
• Munsell: describes surfaces, rather than lights
— less relevant for graphics
— surfaces must be viewed under fixed comparison light
17
Human Colour Constancy
• Colour constancy: determine hue and saturation under different colours of lighting
• Lightness constancy: grey-level reflectance under differing intensity of lighting
• Humans can perceive
— colour a surface would have under white light
— colour of reflected light (separate surface colour from measured colour)
— colour of illuminant (limited)
It is generally accepted that human vision is mediated by three opponent systems, one for luminance and two for hue. The two hue systems differ in more than just their spectral characteristics.
Differences begin at the photoreceptors themselves and continue in the pathways up to and including visual cortex. Here, we highlight aspect of these differences. Attention then is turned to aquatic
environments.
Here, we do refer to the LMS-cones as the RGB-cones, respectively.
Differences Between B and RG Cones
“In several respects, the B cones resemble rods more than they do cones — in particular, in their
virtual absence from the fovea and in their sluggish temporal responsivity.”
[DeValois & Jacobs, 1984]
Differences Between B and RG Cones
Observations:
1. Many fewer B cones than RG cones
2. Low foveal density of B cones
3. Weber fraction for B cones much higher than for RG cones
4. Peak absolute sensitivity of B cones low relative to RG cones
5. Spatial and temporal contrast sensitivity lower and shifted to lower frequencies for the B
cones
6. B cones involved entirely with the perception of hue, and especially with the perception of
“white”
7. B cones make no contribution to perception of sharp chromatic borders
18
Differences Between B/Y and R/G Systems
“. . . the physiology of the B/Y system is replete with unknowns and disagreements . . . . . . the
nature and organization of the B/Y system is simply not at all understood at the present time.”
[DeValois & Jacobs, 1984]
Differences Between B/Y and R/G Systems
Observations:
1. R/G system differences the output of two cone types that are similar in properties (i.e., spectral response, density and retinal distribution)
2. B/Y system differences the output of entirely different cone types (i.e., distinct spectral response, density and retinal distribution)
3. R/G cells numerous throughout the visual pathway
4. B/Y cells rare (compared to R/G cells) up to and including area 17 of the cortex
5. B/Y cells only occur with B on-center and Y off-surround receptive fields. All 4 combinations of R/G receptive fields occur.
Differences Between Luminance and Hue Systems
Observations:
1. Hue (R/G, B/Y) system has lower spatial resolution than luminance (B/W) system
2. The border enhancement effects characteristic of luminance borders do not appear to have a
hue equivalent
3. Hue system does not contribute to the perception of shadows (and hence of depth from
shadows)
4. Hue system does not contribute to the perception of subjective contours
5. Hue system does not support depth perception via binocular stereo
6. Hue system does not contribute (strongly) to 3-D shape perception. (Cues such as perspective, relative size of objects, shading and texture gradients are colour-blind.)
7. Hue system does not contribute to movement perception
19
Clear Water
Clear Water (adapted from [Levine & MacNichol, Jr., 1982])
0
15
Depth (meters)
30
45
60
75
400
450
500
550
600
650
700
Wavelength (nanometers)
Other Water
Fresh Water (Carrying Green Organic Matter)
Depth (meters)
0
5
10
15
20
25
400
450
500
550
600
650
700
Wavelength (nanometers)
Rivers,Swamps and Marshes
(Carrying Products of Plant and Animal Decay)
0
3
400
450
500
550
600
650
Wavelength (nanometers)
(also adapted from [Levine & MacNichol, Jr., 1982])
Evolution
Question: Is the spectral response of cones immutable
• over evolutionary time?
• among species?
• among individuals within a species?
• over the lifetime of an individual?
20
700
X-Chromosome Inactivation (aka Lyonization)
First described in 1961 by geneticist Mary Lyon2 (hence the synonym lyonization), X-chromosome
inactivation is the phenomenon, in female mammals, by which one X-chromosome (either maternally or paternally derived) is randomly inactivated in early embryonic cells, with fixed inactivation
in all descendant cells.
In mammals, male offspring receive one copy of an X-chromosome from the mother while female
offspring receive two, one each from the mother and the father. At some point, one copy of the
X-chromosome in each female cell is inactivated. This is generally believed to be a necessary
regulatory mechanism to prevent female cells from having twice as many gene products from Xchromosomes as do males.
In placental mammals (including humans), the choice of which X-chromosome is inactivated is
random. By comparison, in marsupials it is always the paternal copy that is inactivated.
Following inactivation, all descendent cells have the same X-chromosome inactivated as the cell
from which they arose. Thus, a female placental mammal is a mosaic composed of two types of
cells, one which expresses only the paternal X-chromosome, and another which expresses only the
maternal X-chromosome.
See http://ghr.nlm.nih.gov/glossary=xchromosomeinactivation
Avner and Heard3 refer to X-chromosome inactivation as, “a process that presents a unique medley
of biological puzzles” including:
• how to inactivate one but not the other X-chromosome in the same nucleus
• how to count the number of X-chromosomes4 and keep only one active
• how to choose which X-chromosome is inactivated
• how to establish inactivation rapidly and efficiently during early development
2
Lyon’s hypothesis became “Lyon’s Law” on July 20–24, 2011, at the European Molecular Biology Organization (EMBO) “50-Years of X-inactivation Research,” workshop, Oxford, UK.
3
P. Avner and E. Heard, “X-chromosome inactivation: counting, choice and initiation,” Nature reviews. Genetics,
2(1)59–67, 2001.
4
In rare cases, there can be more than 2
21