Color-Scale Differential Structure
Color RGB original
Spatial gradient
Illumination spectrum
-invariant gradient
Nuclei of fungus cell Paramecium Caudatum
Geusebroek et al, LNCS 1852, 459-464, 1999
ter Haar Romeny, FEV
The color of an object depends on
•
•
•
•
•
color of the illuminating light
illumination intensity
sensor sensitivity
direction of surface normal
surface reflectance properties
Assumptions:
• Scene is uniformly illuminated
• light source is colored
• surface has Lambertian reflectance
ter Haar Romeny, FEV
What causes color ?
object
color
Lamp
spectral
color
ter Haar Romeny, FEV
350000
300000
250000
200000
150000
100000
50000
400
450
500
550
600
650
700
500
Spectrum reflected from
an arbitrary object
8
hc
1000
hc
5 E kT
1500
2000
1
Emission
spectrum of
black body
radiator
1
Object reflectance function for the observed spectrum for a
resp. 2500K, 6500K and 10,000K light source:
400 450 500 550 600 650 700
400 450 500 550 600 650 700
400 450 500 550 600 650 700
ter Haar Romeny, FEV
Color receptive fields
x

0
1
y
2

x
0
1
2
ter Haar Romeny, FEV
Self-organization:
receptive fields
from Eigenpatches
(12x12 pixels)
ter Haar Romeny, FEV
Blakemore C, Cooper GF (1970),
Development of the brain depends
on the visual environment.
Nature 228, 477 - 478
ter Haar Romeny, FEV
Colour receptive
fields from
Eigenpatches
x

0
1
2
ter Haar Romeny, FEV
GD Field et al. Nature 467, 673-677
(2010) doi:10.1038/nature09424
ter Haar Romeny, FEV
Distribution of cone cells in the fovea of an individual with normal color
vision (left), and a color blind (protanopic) retina. Note that the center of
the fovea holds very few blue-sensitive cones. [Wikipedia]
ter Haar Romeny, FEV
Full functional sampling of cone lattice by four RGC types.
GD Field et al. Nature 467, 673-677 (2010) doi:10.1038/nature09424
ter Haar Romeny, FEV
The color opponency model
Hering E, 1964. Outlines of a Theory
of the Light Sense. Harvard University
Press, Cambridge, Mass.
ter Haar Romeny, FEV
How can we
analyse color
differential structure?
Hering basis
0.3
RF
sensitivity
0.2
0.1
0
-0.1
-0.2
-0.3
-0.4
0
10
20
30
40
50
60
70
80
90
wavelength
Idea Koenderink: Gaussian derivatives of zero, first
and second order in the wavelength domain
ter Haar Romeny, FEV
Cone sensitivity
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
300
M
Taylor color model
L
S
400
500
600
700
800
900
Luminance
Blue-yellowness
Purple-greenness



ter Haar Romeny, FEV
Spatial color

Energy densities cannot be measured at a point, …
… one probes a certain volume
s  
y
x
y
x
Color scale-space starts by probing this space.
ter Haar Romeny, FEV
Reflectance of light
What are invariant
properties?
object
color
Lamp
spectra
l
color
ter Haar Romeny, FEV
Reflectance model
ter Haar Romeny, FEV
Transparent materials
ter Haar Romeny, FEV
The reflected spectrum is:
E ( )  e( ) (1   f (n, s, v)) R ( )
2
v = viewing direction
n = surface patch normal
s = direction of illumination
f = Fresnel front surface reflectance coefficient in v
R = body reflectance
ter Haar Romeny, FEV
Because of projection of the energy distribution on the
image plane the vectors n, s and v will depend on the
position at the imaging plane. So the energy at a point
x is then related to:
E(, x)  e(, x)(1   f ( x)) R (, x)
2
We assume an illumination with a locally constant color:
E(, x)  e( )i( x)(1   f ( x)) R (, x)
2
ter Haar Romeny, FEV
Aim: describe material changes independent
of the illumination.
E(, x)  e( )i( x)(1   f ( x)) R (, x)
2
E
e
2
 i ( x)(1   f ( x)) R ( , x)



2 R
e( )i ( x)(1   f ( x))

Both
equations
have
many
common
terms
ter Haar Romeny, FEV
The normalized differential
1 E
1 e
1
R
ˆ
E


E  e( )  R ( , x) 
determines material changes independent of the
viewpoint, surface orientation, illumination
direction, illumination intensity and illumination
color!
ter Haar Romeny, FEV
The derivative jet to x and  forms a complete family of
geometric invariants:
 Eˆ
n
m
 x
nm
These are observed properties, so we convolve with
Gaussian derivatives
 n Eˆ ˆ
 E  G ( ; 0  515nm;    55nm)
n

ter Haar Romeny, FEV
Color invariants
Color edges can be defined as the thresholding of the
spatial gradient (color-invariant equivalent of Lw):
2
2
ˆ
ˆ
E x  E y  
2
2
ˆ
ˆ
E x  E y  
ter Haar Romeny, FEV
Spatial color model and tracing color edges in microscopy
Influence of illumination color
temperature on edge
strength, scale  is 3.0 px.
Skin tissue section illuminated
by a halogen bulb at 4000 K
(top) and 2600 K (bottom)
color temperature.
ter Haar Romeny, FEV
Color-invariant multi-scale structural operators
1
e
e
ter Haar Romeny, FEV
Total edge strength
ter Haar Romeny, FEV
1
e
[im, ]
Some
color
differential
invariants
e
color invariant 1
e
[im, ]
[im, ]
g [im, ]
g
[im, ]
g [im, ]
e
first wavelength derivative of
2
second wavelength derivative of
2
2
2
x
2
2
2
x
2
red-green edges
y
2
2
x
yellow-blue edges
y
y
2
x
2
2
y
2
total color edge strength
ter Haar Romeny, FEV
Feulgen stain,
red-green edges
Paramecium caudatum, Feulgen and Fast green stain
Color canny, red-green normalized edges, scale 3
ter Haar Romeny, FEV
Hematoxylin eosin stain
Pituitary gland, sheep, adenohypophysis 40x
Cell: E<0, E
Nuclei: E  <0, E
 
 
> 0, scale 1.0
> 0, E  +E

< 0, scale 3.0
additional constraint added to refine selection
ter Haar Romeny, FEV
Safranin O stain
E  > 0, E
 
> 0, scale sigma 1.0
Safranin O stain for proteoglycans (mouse knee joint)
Courtesy of Koen Gijbels and Paul Stoppie
ter Haar Romeny, FEV
Oil red O stain
Oil red O stain of fat emboli in lung
E  > 0, E
 
> 0, scale 1.5
ter Haar Romeny, FEV
PAS stain
Lww > 0, Lvv Lww-Lvw2 > 0, E
 
-E

> 0, scale sigma 2.0
P.A.S. stain for carbohydrates (goblet cells, gut)
carbohydrates stain magenta - elliptic patches
ter Haar Romeny, FEV
Blood smear
Blood smear, Giemsa stain, 100x, JPEG compression
RBC: E

> 0, E
Leucocytes: E


+E
 
> 0, scale 0.5
< 0, scale 12
Leucocyte nuclei: E

< 0, E
 
> 0, scale 3
ter Haar Romeny, FEV
Blue-yellow edges
Note the complete absence of detection of black-white edges.
ter Haar Romeny, FEV
Second order color invariants
Color edges can also be defined as the zero-crossings of
the second order derivative in the spatial gradient
direction (color-invariant equivalent of Lww):
Eˆ ww
2 ˆ
2 ˆ
ˆ
ˆ
ˆ
ˆ
ˆ
Ey Eyy  2 Ey Ex Exy  Ex Exx

0
2
2
Eˆ  Eˆ
x
y
ter Haar Romeny, FEV
Luminance gradient
edge detection
Color invariant
edge detection
ter Haar Romeny, FEV
ter Haar Romeny, FEV
Conclusions
• Color ‘scale-space’ compatible with classical luminance scalespace
• The model enables the design of practical image analysis
‘color reasoning’ solutions, e.g. invariance for illumination
• The color-scale invariant differential operators are building
blocks for a differential geometry on color images
ter Haar Romeny, FEV