Digital image processing
Chapter 8 Image analysis and pattern recognition
IMAGE ANALYSIS AND PATTERN RECOGNITION
Introduction
Feature extraction:
- spatial feature extraction
- transform feature extraction
- edge feature extraction; edge detection
Objects representation by their boundaries:
- contour extraction
- contour descriptors
Objects representation by their regions:
- region extraction
- region representation
Shapes and structures for region-based object representation:
- object skeletons
- binary morphology
- shape descriptors (numerical shape descriptors)
Textures; texture analysis
Image segmentation
Grey level based segmentation/color based segmentation
Connected components analysis
Contour-based segmentation
Region-based segmentation
Mixed techniques
Digital image processing
Chapter 8 Image analysis and pattern recognition
Introduction
Input
image
Preprocessing
Data analysis
Feature
extraction
Segmentation
Feature
extraction
Conclusion
of analysis
Classification;
description
Image analysis
Image understanding
Symbolic
representation
Fig. 8.1 Image analysis system – block diagram
Example image analysis tasks:
License plate recognition
Interpretation
& description
Digital image processing
Chapter 8 Image analysis and pattern recognition
Introduction
Example image analysis tasks:
Text detection & recognition or removal
Trafic sign recognition
Calcite localization/assessment
Detect tissue on the slide
Digital image processing
Chapter 8 Image analysis and pattern recognition
Introduction
Example image analysis tasks:
Single object images; defect detection on the object
Ways to approach the image analysis task?
- “straight forward” approach: analyze the content of the whole image => one will
eventually find the entire information, including the information of interest – at too
high computational complexity/cost
- “smart” approach: focus strictly on the information you’re looking for! and extract
only this information from the image  focus on the part of the image containing the
information of interest.
•
How?
1) describe somehow the main characteristics of the information of interest
 describe the discriminative features of the region containing the
information you look for, which are not present in the other regions =>
represent the image in the best feature space, by a feature map
Digital image processing
Chapter 8 Image analysis and pattern recognition
E.g. on the images with apples – only interested in the defect area (if there is), and for this
defect – it appears darker grey than the normal apple coloration, but not black => design a
grey scale slicing to set to black everything else
New feature space
2)
Analyze only the information in the region of interest , using the
previously defined feature space/feature map
e.g. Apples sort: no defects (defect area <2% of the apple); average defects;
large defects…
Digital image processing
Chapter 8 Image analysis and pattern recognition
Feature extraction:
Spatial features extraction:
• Amplitude features: e.g. the brightness levels can identify regions of interest in the image:
Tissue identification
by color coding
(e.g. violet)
Can measure
afterwards the area,
describe the shape,
etc.
•Amplitude features may be discriminative enough if intensity is enough to distinguish wanted
info from the rest of the scene
=> defining the best parameters of the transformation for feature extraction – most difficult
=> amplitude feature space representation is not necessarily binary; just that unwanted
parts of the scenes should be represented uniquely (i.e. black) in the feature space
=> sometimes adaptive thresholding/adaptive grey scale slicing is needed.
Digital image processing
Chapter 8 Image analysis and pattern recognition
• Histogram based features:
Local histogram = a local statistical description of the image;
If u = an image pixel; x=a grey level => pu(x)=the probability of appearance of the grey
level x in the image region = a value in the normalized histogram
=> One can compute: the standard deviation; the entropy; the median; percentiles, of pu(x).
Region
of interest
(ROI)
Measurements
ROI histogram
Tissue of interest
is well discriminated from
the microscopic slide by
the standard deviation
of the local histogram
Digital image processing
Chapter 8 Image analysis and pattern recognition
Transform features extraction
Input
image
Forward unitary
transform
v(k,l)
x
v(k,l)
Inverse unitary
transform
Mask
g(k,l)
Fig.8.2 Transform features extraction
u(m,n)
Digital image processing
Chapter 8 Image analysis and pattern recognition
Edge features extraction. Edge detection
g(m,n) =
(u, H)m,n    h( i, j ) u( i  m, j  n)  u( m, n)  h( m,n)
i
u(m,n)
 g (m,n) = arctg
j
g1(m,n)
h1(-m,-n)
g  g12  g 22
h2(-m,-n)
g 
 g  tan 1  2 
 g1 
g2(m,n)
Magnitude Thresholding
g(m,n)
1
0
Edge
detection
t
Direction
g(m,n)
Fig. 8.3 Edge detection with gradient operators
1
H1   2
1
0
0
0
1
2,
1
 0 1
H1  
,
 1 0
1
H2  0
1
2
0
2
2
1
0
1
1 0 
H2  

0  1
(8.5)
(8.6)
2
g1 (m,n)+ g 2 (m,n)
g 2 (m,n)
g1 (m,n)
Digital image processing
u(m,n)
Chapter 8 Image analysis and pattern recognition
hk(m,n)
gk(m,n)
Max{1 1}
gradient g(m,n)
k
Edge magnitude
thresholding
edge
detection
Fig. 8.4 Edge detection by compass operators
1 1 1
 1 2 1 


1 1 1
5 5 5
3 0 3


3 3 3
a
b
1 1 1
0 0 0


1 1 1
1 2 1
0 0 0


1 2 1
c
d
Fig. 8.5 Compass edge detectors (North direction)
1
1
1


0
0
0
 1  1  1
N
 1  1  1
0 0 0


 1 1 1 
S
1 1
0


1 0  1
0  1  1
NW
  1  1 0
 1 0 1


 0 1 1
SE
1 0  1


1 0  1
1 0  1
W
 1 0 1
 1 0 1


 1 0 1
E
Fig. 8.6 Compass operators on different directions
0  1  1


1 0  1
1 1
0 
SW
 0 1 1
 1 0 1


 1  1 0
NE
Digital image processing
Chapter 8 Image analysis and pattern recognition
gk(m,n) –compass gradient on the direction k0,...,7,  k    k  ,
2
4
The gradient in the spatial position (m,n) is defined as:
g (m, n) = max{ g k (m, n) }
(8.7)
k
Laplace operators and the zero-crossings edge localization method:
2 f 2 f
 f 

 x2  y2
(8.8)
2
0
(1) 1
 0
1
4
1
0   1
 1(2) 1
0   1
8
1
 1  1
 1(3) 2
 1  1
2
4
2
Thresholding
df
dx
f(x)
1
1 
 2 
1 
Double contour
d2 f
dx 2
a
c
b
Fig. 8.7 Edge detection by Laplace operators (the 1-D case)
The Laplacian:
h(m, n) 
1
2 2
 m2  n2
exp 

2 2





Zero
crossing
Digital image processing
Chapter 8 Image analysis and pattern recognition
The Laplacian of Gaussian operator (LoG)

 m2  n 2
(m 2  n 2 ) 
h(m, n)  c1 
exp 


2 2
2 2





Gaussian filter
Discrete implementations:




The derivative of the Gaussian
The 2nd derivative
(Laplacian of Gaussian)
Digital image processing
Chapter 8 Image analysis and pattern recognition
Edge detection by different operators – comparison:
Original image
Roberts
edge detection
Sobel edge detection
LoG edge
detection;
Sigma=10
LoG edge detection; sigma=5
Digital image processing
Chapter 8 Image analysis and pattern recognition
Objects representation by their boundaries:
Contour extraction:
1
C
B
A
A
a)
b)
4
2
3
c)
Fig. 8.8 4-connectivity; 8-connectivity
The Hough transform: s  x  sin  y  cos  s  x  cos  y  sin
y
s
s
θ
x
Φ
Digital image processing
Chapter 8 Image analysis and pattern recognition
The Hough transform of lines/random curves:
s
s
Curve convergence
on horizontal axis=>
Straight line through the origin
s
Approx. convergence =>
approx. straight line
No convergence point =>
The points are not on
a straight line
Digital image processing
Chapter 8 Image analysis and pattern recognition
Applications to contour extraction formed by line segments:
Edge detection
(gradient
operator)
Apply Hough
transform:
+ thresholding
=> Binary edge
map
Input image
Extracting the
contour of the
triangle
Edge map;
Contours not yet extracted/
not yet labeled
Lower left object =
Defined by the 3 line
intersection points
8-connectivity:
Correct labeling
Contour extraction
in 8-connectivity
4-connectivity:
The contour is
“broken” during
labeling
Digital image processing
Chapter 8 Image analysis and pattern recognition
Contour representation/contour descriptors:
• Goal: for a given object, described by its contour, find a compact description, by numerical
attributes, able to:
-Represent the contour with no significant loss of information (regenerative descriptors)
-Generate (by a subset of attributes) descriptions of the contour/shape invariant to: scaling;
rotation; translation; mirroring; projection distortions; small (limited) variations of shape
(among different individual representations of the same shape)
Using the contour descriptors, one can recognize the shape by template matching or shape
classification (contour descriptors classification)
• General assumption: single object contour (all edge pixels connected); 1-pixel width!
Digital image processing
Chapter 8 Image analysis and pattern recognition
Chain codes and polygonal approximations:
D
2
3
C
1
4
B
5
C
3
A
D
4
0
6
2
E
5
7
H
7
6
A
1
F
a)
0
G
E
B
b)
Fig. 8.9 Contour representation by chain codes
Fig. 8.10 Polygonal approximation of the contours
Start point
Encode this contour
by a chain code with
8 directions,
Consider origin= the
upper left point; contour
following direction =
arbitrary; the only
possible positions are
on the grid corners
Original contour
Contour approximation (“quantization”)
on a rectangular grid
Digital image processing
Chapter 8 Image analysis and pattern recognition
Fourier descriptors:
u(n)  x(n)  j  y(n)
u ( n) 
pentru n  0,1,...,N  1
1 N 1
 j 2 kn 
,0  n  N  1
 a (k )exp
N k 0
 N 
  j 2 kn 
a (k )   u (n)exp
,0  n  N  1
N


n 0
N 1
(8.12)
(8.13)
(8.14)
u0  x0  j y 0
(8.15)
Ax  By  C  0
(8.16)
leading to a new contour x'(n) y'(n), given by:
u'( n)  u *( n)e j 2  2
( A  jB)C
 
A2  B 2
where
(8.17)
( A  jB) 2
, exp( j 2  ) 
A2  B 2
(8.18)
Table 8.1
Transform
The transformed contour
Identity
Translation
Scaling
Origin
Rotation
Symmetry towards
a straight line
The Fourier descriptors
u(n)
u'(n)u(n)u0
u'(n)u(n)
u'(n)u(n-n0)
u'(n)u(n)ej0
a(k)
a'(k)a(k) u0 (k)
a'(k)a(k)
a'(k)a(k)e-j2n0k/N
a'(k)a'(k)ej0
u'(n)u*(n)ej202
a'(k)a*(-k)ej22 (k)
Digital image processing
Chapter 8 Image analysis and pattern recognition
Shape reconstruction from its Fourier descriptors:
Original
Reconstruction using
the first 2 descriptors only
Reconstruction using
the first 6 descriptors
…
Reconstruction using
the first 10 descriptors
Reconstruction using
the first 20 descriptors
Digital image processing
Chapter 8 Image analysis and pattern recognition
2
N 1
d ( u0 , , 0 , n0 )  min  u( n)   v( n  n0 )e j 0  u0 
u0 , ,n0 , 0
 n 0

Model
shape
Macthing results: d<Thd
 u(n)   v(n)  0
,
then for a given translation u0, the distance d is minimal when:
u0  0
 
 c( k )cos(
k
k
 k  0 )
 b( k )
2
and
k
tg 0
 c( k )  sin(

 c( k )  cos(
k
 k )
k
 k )
k
k
2

d  min d ( )  min a( k )   b( k ) exp j ( k   0 ) 


 k

where a(k)b*(k)c(k)ejk,  -2n0/N and c(k) is a real valued term.
Shapes dictionary
Digital image processing
Chapter 8 Image analysis and pattern recognition
Objects representation by their regions:
Region extraction: same as for contours (use connected components analysis, in 4- or 8- connectivity)
Region representation:
1 ,if m,n  
u(m,n)  
0 , otherwise
4
1
3
2
G
A
I
A
F
H
E
B
D
C
I
H
G
E
D
C
B
F
4
3
2
1
Fig. 8.11 Quad-tree region representation
=> the quad-tree code : gbgbwwbwgbwwgbwwb ; decoding:
g(bg(bwwb)wg(bwwg(bwwb)))
Digital image processing
Chapter 8 Image analysis and pattern recognition
Shapes and structures for region-based object representation:
 Regions skeletons; medial axis transforms:
uk ( m, n )  u0 ( m, n )  min
 ( m,n ; i , j )
uk 1 ( i, j );( i, j ) : s( m, n; i, j )  1
k  1,2,...
8.23)
u0 ( m, n )  u( m, n )
(m,n) : uk(m,n)  uk(i,j) ,  (m,n;i,j) 1
1
1
1

1
1
1 1 1 1 1 1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1

1 k

1
1
1
1

1 1

1
1
1 1 1 1 1 1
2
2
2
1
2
2
2
1
u0 (m, n)
1

2

2

1

1

  
3 3 3
2
2
2
2
1
2
2
2
1
1
1

2 1

1
1
1 1 1 1 1 1
2
2
2
1
2
3
2
1
u1 (m, n)
1
2
2
2
2
1
1

1 k

1
1

(8.24)
2
3
2
1
2
3
2
1
1

1 k

1
1
2
2
2
1
1
1

 3,4,5 1

1
1
u 2 (m, n)
1 1 1 1 1 1
2
2
2
1
2
3
2
1
2
3
2
1
2
3
2
1
2
2
2
1
1

1 

1
1
u3 , u4 , u5


Schelet
a
Fig. 8.12 Skeleton extraction
P9
1
1
0
0
0
0
1
0
P4 P1
P8
1
P1
1
1
P1
0
0
P1 0
P5 P6
P7
0
0
0
0
0
0
1
1
b.
c
Fig. 8.13 Examples skeletons
P3 P2
a.
b
c.
1
1
d.
Fig. 8.14 2-pass contour thinning: 2 logic rules:
a
Fig. 8.15 Thinning result
R1: P1==1 && N(P1)==1 && 2<=NT(P1)<=6 && P2·P4·P6==0 && P2·P4·P8==0
R2: P1==1 && N(P1)==1 && 2<=NT(P1)<=6 && P2·P6·P8==0 && P4·P6·P8==0
b
Digital image processing

Chapter 8 Image analysis and pattern recognition
Morphological processing. Binary image morphology
Object
Structural
element
Object
Erosion
Structural
element
Dilation
Origin
Origin
Object
Structural
element
Contour
extraction
Obiect
Element
structural
Origin
Obiect
Origine
Element
structural
Inchidere
Obiect
Origine
Obiect
Deschidere
Element
Subtiere
Element
structural
Origine
G
Obiect
Scheletizare
Element
structural
Origine
Origine
structural
Obiect
Element
structural
Ingrosare
Origine
Fig. 8.16 Algoritmi de transformare morfologica
Curatare
Digital image processing

Chapter 8 Image analysis and pattern recognition
Syntactical representation
Primitives
a
b
c
d
e
Object’s
structure
d
b
e
e
a
a
c
a
a
d
d
a
d
d
a
a
Syntactical representation:
adaebeadaddada
c a
Fig. 8.17 Syntactical representation of an object
e
Digital image processing
Chapter 8 Image analysis and pattern recognition
Shape descriptors
Applications: shape recognition; quantitative measurements
Def.: The shape of an object = the profile of the object + its physical structure => “shape descriptors”
Classification:
(1) regenerative descriptors (contours; regions; high order statistics; structural and syntactic descriptors)
(2) geometrical shape descriptors (area, perimeter, max-min radii, eccentricity, corners, roundness, symmetry)
(3) moments
Digital image processing
Chapter 8 Image analysis and pattern recognition
y
Geometrical features (geometrical descriptors):

T 
Perimeter:
2
2
 dx(t )   dy(t ) 

 
 dt
dt
dt

 

t – some contour parameter
dy
- Discrete => T = count of contour pixels
y(t)
y(t+1)
x(t)

Area:
A    dxdy   y( t ) 
R
R
x(t+1)
dx
x
dx ( t )
dy( t )
dt   x( t ) 
dt
dt
dt
R
where: R and R – the object region and the object contour
- Discrete => A = count of pixels inside the object region
 Min-max radii, Rmin and Rmax – the minimum and maximum distances from the center of mass of the object
region to its contour (the Rmax/ Rmin ratio – gives a measure of the eccentricity or elongation of the object)
Rmin
Rmax
Digital image processing
Chapter 8 Image analysis and pattern recognition
Compact
compact

Roundness or compactness:
 
( perim eter) 2
T2

4 ( area)
4 A
For a disc -  is minimum, =1.
 Symmetry: 2 types of shape symmetry: rotational and mirror
Moment-based features:
 Center of mass:
m
1
N
 m ,
n
( m,n )R
1
N
n
( m,n )R
 The (p,q) order central moments:
 (m - m )
 p ,q 
p
( n  n) q
( m,n )R

Orientation = the angle of the axis of the smallest moment of inertia
– found by minimizing:
I ( ) 
   (m, n)  [(n  n)cos  (m  m)sin ]
2
( m,n )R
2
( m,n )R
with respect to :
 


2 1,1
1
 arctg 

2
  2, 0   0, 2 


Non-compact
necompact
Digital image processing
Chapter 8 Image analysis and pattern recognition
Textures
 The texture = the periodic repetition of some basic structures in an image area; the basic image
structure is called texel
Artificial textures
Natural textures
Texture analysis methods: statistical classification; structural classification

Statistical classification techniques:
 The auto-correlation function (ACF): the spatial dimensions of the texels are proportional to the width of the autocorrelation functions::
au,u (k , l )  m2 (k , l ) / m2 (0,0)
where
m2 (k , l ) 
 u(m, n)u(m  k , n  l )
m,n W
 Several measures used to evaluate the distribution of the ACF to describe the texture:
M ( k , l )    ( m   1 ) k ( n   2 ) l r( m, n)
m
n
 1    mr( m, n) ,
m
n
 2    nr( m, n) ,
m
n
Digital image processing
Chapter 8 Image analysis and pattern recognition
ACF for “Sand”
ACF for “Wool”
 Image transforms based approaches:
z i ( k , l )  v ( k , l )  gi ( k , l )
2
(8.41)
2
2
1
1
1
2
2
2
1
1
Fig. 8.18 Various masks in the frequency domain used for texture analysis
1
Digital image processing
Chapter 8 Image analysis and pattern recognition
 The edge density – as texture classification feature:
Sobel
Sobel
Sand
Raffia
 Histogram features for texture analysis: => the co-occurrence histogram:
pu ( x1 , x 2 )  f ( r, ; x1 , x 2 )
=> various features can be extracted from the co-occurrence histogram:
- Inertia:
I ( r, ) 
 x
1
 x2
2
R4, 0
f ( r ,  ; x1 x 2 )
x1 , x 2
- The mean of the distribution:
 ( r; x1 ,x 2 ) 
1
N0


f ( r ,  ; x1 x 2 )
N0 – the total number of possible orientations.
- The variance of the distribution:
1
 2 ( r;x1 ,x 2 ) 
[ f ( r, ;x1 x 2 )   ( r;x1 ,x 2 )]2

N0 
- The spread of the distribution:
( r;x1 ,x 2 )  max{ f ( r, ;x1 ,x 2 )}  min{ f ( r, ;x1 ,x 2 )}

Grass
Digital image processing
Chapter 8 Image analysis and pattern recognition
 Random texture models
u( m, n)  a( m, n)   ( m, n)
ACF
computation
Texture
Feature
extraction
u(m,n)
(8.47)
Texture
features
Decorrelation
(m,n) Histogram
filter
computation
A(x1,x2)
a
white
noise
(m,n)
1
A(x1,x2)
u(m,n)
b
Fig. 8.19 Texture analysis model
Digital image processing
Chapter 8 Image analysis and pattern recognition
Image segmentation
Amplitude thresholding/ grey level window slicing
1) Local maxima detection
Object
2) Local minima detection
Objects are found between the local minima
Component labeling
 Pixel labeling. Region growing
C
A
B
X
D
seed
segmentation with threshold =4
new seed segmented image
Digital image processing
Chapter 8 Image analysis and pattern recognition
Boundary-based image segmentation
Object
Boundarybased
segmentation
Edge
detection
Analysis and
classification
Feature
extraction
Fig. 8.23 Segmentation algorithm based on boundary detection
Region-based segmentation; segmentation based on regions and boundaries
R
R
R
1
5
R
R
2
R
4
1 '=
R
1
 R
2
 R
4
3
R
a
R
5
b
Fig. 8.24 Region merging
3
Digital image processing
Chapter 8 Image analysis and pattern recognition
2
1
R2
A B A B
D C D C Merge
Split
R1
A B
D C
3
4
a
1
A
B
2
D
C
A
3
B C
b
D
A
B C
4
1(A,B,D)
2(A,B,C)
3(B,C,D)
4
1C, 2D, 3A
D
c
Fig. 8.25 Segmentation by split and merge algorithm: a. input
b. region splitting by quad-trees
c. segmented regions
Digital image processing
Chapter 8 Image analysis and pattern recognition
a) Original image
b) Region-based image
segmentation
c) Main objects detected, marked on
the original image by their contour