Lecture 10 (4.14.07)
Image Representation and
Description
Shahram Ebadollahi
4/15/2008
DIP ELEN E4830
1
Image
Description
Recognition
High-level Image
Representation
4/15/2008
Image
Understanding
2
Lecture Outline
1
Image Description
1
1
1
1
1
4/15/2008
Shape Descriptors
Texture & Texture Descriptors
SIFT
Motion Descriptors
Color Descriptors
3
Shape Description
1
Shape Represented by its Boundary
1
1
1
1
Shape Numbers,
Fourier Descriptors,
Statistical Moments
Shape Represented by its Interior
1
1
4/15/2008
Topological Descriptors
Moment Invariants
4
Boundary Representation:
(Freeman) Chain Code
Boundary representation = 0766666453321212
Original boundary
Chain code for
4-neighborhood
4/15/2008
Sub-sampled boundary
Chain code of boundary
Chain code for
8-neighborhood
5
Chain Code: example
8-directional chain code
2 00006066666666444444242222202202
Starting point normalized chain code 2 00006066666666444444242222202202
Rotation normalized chain code
4/15/2008
2 0006200000006000006260000620626
First difference of chain code
6
Shape Number –
A boundary descriptor
4/15/2008
7
Boundary descriptor – Fourier
s (k ) = x(k ) + jy (k )
K −1
a (u ) = ∑ s (k )e − j 2πuk / K
k = 0,1,2, 1 , K − 1
u = 0,1,2, 1 , K − 1
Fourier Descriptor
k =0
1
s (k ) =
K
4/15/2008
K −1
j 2πuk / K
a
(
u
)
e
∑
k = 0,1,2, 1 , K − 1
u =0
8
Boundary Reconstruction using
Fourier Descriptors
100%
50%
10%
5%
2868
descriptors
Only 8
descriptors
2.5%
4/15/2008
1.25%
0.63%
0.28%
9
Boundary Representation: Signatures
• Represent 2-D boundary shape using 1-D signature signal
4/15/2008
10
Boundary Representation: Signatures
4/15/2008
11
Boundary Description using
Statistical Moments
A−1
µ n (v) = ∑ (vi − m) n p(vi )
n-th moment of v
i =0
A−1
m = ∑ vi p (vi )
i =1
4/15/2008
12
Region Descriptors - Simple
1
1
1
1
1
1
1
Area
Perimeter
Compactness
Circularity Ratio
Mean/Median intensity
Max/Min intensity
Normalized area
A
Rc = 2
P / 4π
(perimeter)2/Area
r
2
r
b
πr
a=2b
C : 4π
Rc :
1
5π
4 ≈ 0.8
5
16
π
4
≈ 0.78
Area of circle with same
perimeter as the shape
4/15/2008
13
4/15/2008
14
Topological Region Descriptors
• Topological properties: Properties of image preserved under
rubber-sheet distortions
H: # holes in the image
H=2, C=1, E=-1
C: # connected components
E = C-H: Euler Number
H=0, C=3, E=3
V −Q + F = C − H = E
H=1, C=1, E=0
4/15/2008
H=2, C=1, E=-1
15
Geometric Moment Invariants
m pq = ∫ ∫ x p y q f ( x, y )dxdy
M −1 N −1
m pq = ∑∑ x p y q f ( x, y )
(p+q)-th 2D geometric moment
Projection of f(x,y) onto monomial
x p yq
x =0 y =0
• Why use moments?
• Geometric moments of different orders represent spatial
characteristics of the image intensity distribution
m00
Total intensity of image. For binary image 2 area
x0 = m10 / m00
Intensity centroid
y0 = m01 / m 00
binary image 2 geometrical center
4/15/2008
16
Central Moments
M −1 N −1
µ 00 = m00
x =0 y =0
µ10 = µ 01 = 0
µ pq = ∑∑ ( x − x0 ) p ( y − y0 ) q f ( x, y )
[Translation invariance]
µ 02 , µ 20
Variance about the centroid
µ11
covariance
Scaled Central Moment
λ pq = µ ' pq /( µ '00 )
( p + q + 2) / 2
Scale and translation invariant
µ ' pq =
µ pq
α p+q+2
Normalized Un-Scaled Central Moment
4/15/2008
η pq = µ pq /( µ00 ) ( p + q + 2 ) / 2
17
Moment Invariants
(translation, scale, mirroring, rotation)
φ1 = η 20 + η 02
φ2 = (η 20 − η 02 ) 2 + 4η112
φ3 = (η30 − η12 ) 2 + (η 21 − η 03 ) 2
φ4 = (η30 + η12 ) 2 + (η 21 + η 03 ) 2
φ5 = 1
ϕ1 = η 20 + η
φ6 = 1
φ7 = 1
4/15/2008
18
Affine Transform & Affine Moment Invariants
m m−r
x' = Tx ( x, y )
y ' = Ty ( x, y )
x' = ∑∑ ark x r y k
r =0 k =0
m m−r
y ' = ∑∑ brk x r y k
r =0 k =0
In practice: bilinear transform
4 pairs of corresponding points
needed to find coefficients
x' = a0 + a1 x + a2 y + a3 xy
y ' = b0 + b1 x + b2 y + b3 xy
x' = a0 + a1 x + a2 y
y ' = b0 + b1 x + b2 y
In practice: affine transform
3 pairs of corresponding points
needed to find coefficients
Rotation:
4/15/2008
x' = x cos φ + y sin φ
y ' = − x sin φ + y cos φ
Scale:
Skew:
x' = ax
y ' = by
x' = x + y tan φ
y' = y
19
Elliptical Shape Descriptors
Principal moment of inertia:
( µ 20 + µ 02 ) + [( µ 20 − µ 02 ) + 4µ ]
2
( µ 20 + µ 02 ) − [( µ 20 − µ 02 ) 2 + 4µ112 ]1/ 2
I2 =
2
I1 =
2
2 1/ 2
11
Image ellipse characterizes
fundamental shape features and
also 2D position and orientation
θ
2
( I1 + I 2 ) / m00
spreadness
( I 2 − I1 ) /( I1 + I 2 )
elongation
θ = 0.5 tan −1 (
2µ11
)
µ 20 − µ 02
a = 2( I1 / µ 00 )1/ 2
4/15/2008
b = 2( I 2 / µ 00 )1/ 2
20
Texture - Definition
4/15/2008
21
Texture – Quantification Methods
1
Statistical: compute local features at
each point in image and derive a set
of statistics from the distribution of
local features
1
1
1st, 2nd, and higher-order statistics
based on how many points are
used to define local features
Structural: texture is considered to
be composed of “texture elements”.
Properties of the “texture elements”
and their spatial placement rules
characterizes the texture
1
4/15/2008
Original texture can be
reconstructed from its structural
description
22
Statistical Texture Analysis
1st order statistics
f → hf
image
histogram
• Obtain statistics of the histogram:
L −1
Mean:
∑ ih(i)
i =0
L −1
Variance:
∑ (i − µ )
: average intensity
2
h(i )
: measure of intensity contrast
i =0
L −1
Skewness:
3
(
i
−
µ
)
h(i )
∑
i =0
L −1
Entropy:
4/15/2008
− ∑ h(i ) log h(i )
i =0
Measure of variability of intensity
23
4/15/2008
24
statistics
histogram
image
Statistical Texture Analysis
1st order statistics
Skewness = 2.08
Skewness = 2.44
Skewness = -0.092
Entropy = 0.88
Entropy = 0.77
Entropy = 0.97
4/15/2008
25
Statistical Texture Analysis
2nd order statistics: Co-occurrence
f (m2 , n2 ) = j
d
θ
P( d ,θ ) (i, j ) ≈ Pr[ f (m1 , n1 ) = i, f (m2 , n2 ) = j ]
f (m1 , n1 ) = i
1 2
1
Joint gray-level histogram of pairs of
pixels
1
j
L
1
2
P( d ,θ ) ( i , j )
2D histogram
i
L
4/15/2008
26
Statistical Texture Analysis
2nd order statistics: Co-occurrence
P( d ,θ =0° ) (i, j ) =| {((k , l ), (m, n)) ∈ ( M × N ) × ( M × N ) :
k − m = 0, | l − n | = d , f (k , l ) = i, f (m, n) = j} |
P( d ,θ = 45° ) (i, j ) =| {((k , l ), (m, n)) ∈ ( M × N ) × ( M × N ) :
(k − m = d , l − n = −d ) ∨ (k = m = −d , l − n = d ), f (k , l ) = i, f (m, n) = j} |
P( d ,θ =90° ) (i, j )
P( d ,θ =135° ) (i, j )
{1}
4/15/2008
135°
90°
45°
k
is set cardinality
0°
j
27
Statistical Texture Analysis
2nd order statistics: Co-occurrence (example)
image
4/15/2008
Co-occurrence matrix
28
Statistical Texture Analysis
2nd order statistics: Co-occurrence (statistics)
L
Angular
2nd
2
P
∑∑ ( d ,θ ) (i, j )
moment (energy):
i =1 j =1
(measure of image homogeneity)
max P( d ,θ ) (i, j )
Maximum Probability:
i, j
L
L
− ∑∑ P( d ,θ ) (i, j ) logP( d ,θ ) (i, j )
Entropy:
i =1 j =1
L
L
λ
−
i
j
P
∑∑
( d ,θ ) (i , j )
Contrast:
κ
i =1 j =1
(measure of local variations)
L
L
∑∑ [ijP
i =1 j =1
Correlation:
(measure of image linearity)
L
4/15/2008
L
L
µ x = ∑ i ∑ P( d ,θ ) (i, j )
i =1
j =1
( d ,θ )
(i, j )] − µ x µ y
σ xσ y
L
σ x = ∑ (i − µ x )
i =1
L
2
∑P
j =1
( d ,θ )
(i, j )
29
4/15/2008
30
Statistical Texture Analysis
2nd order statistics: Difference Statistics
P( d ,θ ) (k ) =
∑P
( d ,θ )
i , j∈{1,1, L}
(i, j )
is a subset of co-occurrence matrix
|i − j | = k
L −1
Angular
2nd
moment (energy):
2
P
∑ (d ,θ ) (k )
k =0
L −1
Mean:
∑ kP
k =0
( d ,θ )
(k )
L −1
Entropy:
− ∑ P( d ,θ ) (k ) log P( d ,θ ) (k )
k =0
L −1
Contrast:
2
k
∑ P(d ,θ ) (k )
k =0
4/15/2008
31
Statistical Texture Analysis
2nd order statistics: Autocorrelation
M − pN −q
C ff ( p, q ) =
MN
( M − p )( N − q )
∑ ∑ f ( k , l ) f ( k + p, l + q )
k =1 l =1
M
N
∑∑
f 2 (k , l )
k =1 l =1
Large texture elements 3 autoccorrelation decreases slowly with increasing distance
Small texture elements 3 autoccorrelation decreases rapidly with increasing distance
Periodic texture elements 3 periodic increase & decrease in autocorrelation value
4/15/2008
32
Statistical Texture Analysis
2nd order statistics: Fourier Power Spectrum
f ( x, y ) ↔ F (u , v)
Power Spectrum
Note:
P(u, v) =| F (u , v) |2
C ff = F −1{| F (u, v) |2 }
v
π
P ( r ) = 2∑ P ( r , θ )
u
θ =0
L/2
P(θ ) = ∑ P(r , θ )
v
r =0
u
4/15/2008
Indicator for size of
dominant texture element
or texture coarseness
Indicator for the
directionality of the texture
33
Law’s Texture Energy Measures
E3 = [−1,0,1]
L3 = [1,2,1]
S3 = [−1,2,−1]
L5 = [1,4,6,4,1]
E5 = [−1,−2,0,2,1]
S5 = [−1,0,2,0,−1]
R5 = [1,−4,6,−4,1]
W5 = [−1,2,0,−2,−1]
4/15/2008
 −1
− 4

T
L5 × S5 = − 6

− 4
 − 1
0 − 1
0 − 4
0 12 0 − 6

0 8 0 − 4
0 2 0 − 1 
0
0
2
8
•Convolute different Law’s
masks with image
• Compute energy statistics
34
4/15/2008
35
4/15/2008
36
4/15/2008
37
4/15/2008
38
4/15/2008
39
Motion – object
Difference image:
d (i, j ) = 0 if
| f1 (i, j ) − f 2 (i, j ) | ≤ ε
object
motion
1 otherwise
No motion direction information !
f1
f2
n
d cum (i, j ) = ∑ ak | f1 (i, j ) − f k (i, j ) |
Cumulative difference image
k =1
Tells us how often the image gray
level was different from gray-level
of reference image
absolute
4/15/2008
positive
negative
40
Motion Field
• A velocity vector is assigned to each pixel in the image
• Velocities due to relative motion between camera and the 3D
scene
• Image change due to motion during a time interval dt
• Velocity field that represents 3-dimensional motion of object
points across 2-dimensional image
Motion field
4/15/2008
41
Optical flow
• Motion of brightness patterns in image sequence
• Assumptions for computing optical flow:
• Observed brightness of any object point is
constant over time
• Nearby points in the image plane move in a
similar manner
f ( x + dx, y + dy, t + dt ) = f ( x, y, t ) +
∂f
∂f
∂f
dx + dy + dt + O(∂ 2 )
∂x
∂y
∂t
= f ( x, y, t ) + f x dx + f y dy + f t dt + O(∂ )
2
f ( x + dx, y + dy, t + dt ) = f ( x, y, t ) ⇒ − f t = f x
c=(
4/15/2008
dx
dy
+ fy
dt
dt
dx dy
, ) = (u, v)
dt dt
Gray-level difference at same
location over time is
equivalent to product of spatial
gray-level difference and
velocity
− f t = f x u + f y v = ∇f .c
known
unknown
42
Optical Flow Constraints
− f t = f xu + f y v
• no spatial change in brightness, induce no temporal
change in brightness 2 no discernible motion
• motion perpendicular to local gradient induce no temporal
change in brightness 2 no discernible motion
• motion in direction of local gradient, induce temporal
change in brightness 2 discernible motion
• only motion in direction of local gradient induces temporal
change in brightness and discernible motion
4/15/2008
43
Optical flow != Motion Field
MF ≠ 0
OF = 0
4/15/2008
MF = 0
OF ≠ 0
44
Which descriptors?
Image Feature Evaluation
Prototype Performance
1.
1
Classify (Segment) image using
different features
Evaluate which feature is optimal
(minimum classification error)
1
Figure of Merit
2.
1
1
4/15/2008
Establish functional distance
measurements between set of
image features (large distance 2
low classification error)
Bhattacharyya distance
45