Image Segmentation
Qualification Exam
Deparment of Electrical Engineering - DEE
Pedro M. Achanccaray Diaz
Advisor: Raul Queiroz Feitosa
Sumário
1. Introduction
2. Spatially blind approaches
3. Spatially guided approaches
2
Sumário
1. Introduction
2. Spatially blind approaches
3. Spatially guided approaches
3
1. Introduction
Digital Image Processing (DIP):
Digital
Image
Real scene
Acquisition
Enhancement /
Pre-Processing
Segmentation
Postprocessing
Feature
Extraction
Recognition
Classification
4
1. Introduction
Digital Image Processing (DIP):
Digitalization errors
Acquisition
Image without noise
Enhancement /
Pre-Processing
Segmentation
Postprocessing
Feature
Extraction
Recognition
Classification
5
1. Introduction
Digital Image Processing (DIP):
Image without noise
Acquisition
Enhancement /
Pre-Processing
Segmentation Outcome
Segmentation
Postprocessing
Feature
Extraction
Recognition
Classification
6
1. Introduction
Digital Image Processing (DIP):
Segmentation Outcome
Acquisition
Enhancement /
Pre-Processing
Segmentation
Improved Segmentation
Postprocessing
Feature
Extraction
Recognition
Classification
7
1. Introduction
Digital Image Processing (DIP):
Improved Segmentation
Features
𝑥1 = 𝑥11 , 𝑥12 , … , 𝑥1𝑑
𝑥2 = 𝑥21 , 𝑥22 , … , 𝑥2𝑑
𝑥3 = 𝑥31 , 𝑥32 , … , 𝑥3𝑑
...
...
Area, Color, Spatial
position, Texture,...
Acquisition
Enhancement /
Pre-Processing
Segmentation
Postprocessing
Feature
Extraction
Recognition
Classification
8
1. Introduction
Digital Image Processing (DIP):
Scene Understanding
Classes
Features
𝑥1 = 𝑥11 , 𝑥12 , … , 𝑥1𝑑
𝑥2 = 𝑥21 , 𝑥22 , … , 𝑥2𝑑
𝑥3 = 𝑥31 , 𝑥32 , … , 𝑥3𝑑
Acquisition
Enhancement /
Pre-Processing
Segmentation
Postprocessing
Feature
Extraction
...
...
...
Area, Color, Spatial
position, Texture,...
Tree
Sky
Road
Labels
Recognition
Classification
9
1. Introduction
Image Segmentation:
“Segmentation subdivides an image into its constituent regions or objects.
The level of detail to which the subdivision is carried depends on the
problem being solved” – Gonzales & Woods, 2008.
Image without noise
Acquisition
Enhancement /
Pre-Processing
Segmentation Outcome
Segmentation
Postprocessing
Feature
Extraction
Recognition
Classification
10
1. Introduction
Image Segmentation:
But...why an image should be segmented?
• Provides a more meaningful representation easier to analyze.
• Reduces complexity of feature extraction and classification steps.
Image Segmentation usually leads to two typical results:
Over segmentation
Under segmentation
11
1. Introduction
Image Segmentation:
Image Segmentation partitions the region 𝑅 in 𝑛 subregions 𝑅1 , 𝑅2 , … , 𝑅𝑛 where:
𝑛
𝑅𝑖 = 𝑅 : each pixel must be in a region.
a)
𝑖=1
b) 𝑅𝑖 is a connected set ∀𝑖 = 1,2, … , 𝑛
c) 𝑅𝑖 ∩ 𝑅𝑗 = ∅
∀𝑖, 𝑗; 𝑖 ≠ 𝑗 regions must be disjoint.
d) 𝑄 𝑅𝑖 =TRUE each pixel has to fulfill certain condition to belong to a segment.
e) 𝑄 𝑅𝑖 ∪ 𝑅𝑗 =FALSE according to the same condition, two regions have to be different.
12
1. Introduction
The following
presentation:
classification
was
considered
for
this
– High-level taxonomy.
– Low-level taxonomy.
Taxonomy: Classification
13
1. Introduction
High-level
taxonomy
Image Type
• Monochromatic
• Color
Human Interaction
Image
Representation
• Supervised
• Unsupervised
• Single scale
• Multiscale
Image Features
• One feature
• Multiple
features
(color,
gradient,
texture,
position, etc.)
Principle of
Operation
• Spatially blind
(color)
• Spatially guided
(spatial position)
14
1. Introduction
Low-level
taxonomy
Spatially
guided
Spatially blind
Clustering
• K-means
• Mean Shift
Histogram
thresholding
• Otsu
segmentation
Region based
• Region
Growing
Energy based
• Graph-based
Based on
Regions and
Contours
• Watersheds
15
Sumário
1. Introduction
2. Spatially blind approaches
3. Spatially guided approaches
16
2. Spatially blind approaches
Clustering : Definitions
– Cluster: Set formed by similar elements in a certain way.
– Clustering: Group data in clusters according to certain similarity criterion.
– Feature: Characteristic representative of a data set. For our case, can be
considered as features: color, area, perimeter, etc.
– Feature Space: 𝑑-dimensional space formed by the extracted features.
Test Image
202 × 302 × 3
17
2. Spatially blind approaches
Clustering : K-Means Segmentation
Group pixels according to the Euclidian distance between each pixel and the cluster’s
centroids.
Then, the main goal is to minimze the following target function:
𝑘
𝑛
𝐽𝐾−𝑚𝑒𝑎𝑛𝑠 𝒗 =
𝑥𝑗 − 𝑣𝑖
2
𝑖=1 𝑗=1
𝒗 = 𝑎𝑟𝑔𝑚𝑖𝑛{𝐽𝐾−𝑚𝑒𝑎𝑛𝑠 𝒗 }
𝒗
Where:
𝑥𝑗 : 𝑑-dimensional feature vector corresponding
to the pixel 𝑗.
𝑥𝑗 = 𝑥𝑗𝑅 , 𝑥𝑗𝐺 , 𝑥𝑗𝐵 in the case of RGB images.
𝑣𝑖 : centroid of cluster 𝑖.
𝑘: number of clusters.
𝑛: number of pixels (𝑤𝑖𝑑𝑡ℎ × ℎ𝑒𝑖𝑔ℎ𝑡)
18
2. Spatially blind approaches
Clustering : K-Means Segmentation – Results
𝑘=3
𝑘 =5
Including spatial information:
𝑥𝑗 =
Spatial
position of
the pixel in
𝑥𝑗𝑅 , 𝑥𝑗𝐺 , 𝑥𝑗𝐵 , 𝑥𝑗𝑋 , 𝑥𝑗𝑌 the image.
The clustering will be spatially guided.
𝑘 =10
𝑘 =20
19
2. Spatially blind approaches
Clustering : Fuzzy C-Means Segmentation
Group pixels according to the Euclidian distance between each pixel and the cluster’s
centroids.
Then, the main goal is to minimze the following target function:
𝑘
𝑛
𝐽𝐶−𝑚𝑒𝑎𝑛𝑠 =
𝜇𝑖𝑗
𝑚
𝑥𝑗 − 𝑣𝑖
𝑖=1 𝑗=1
Where:
𝑥𝑗 : 𝑑-dimensional feature vector corresponding
to the pixel 𝑗.
𝑣𝑖 : centroid of cluster 𝑖.
𝑘: number of clusters.
𝑛: number of pixels (𝑤𝑖𝑑𝑡ℎ × ℎ𝑒𝑖𝑔ℎ𝑡)
𝜇𝑖𝑗 : degree of membership of pixel 𝑗 in the
cluster 𝑖.
𝑚: fuzziness index.
2
Updating centroids:
𝜕𝐽 𝜇𝑖𝑗 , 𝑣𝑘
= 0 → 𝑣𝑖 =
𝜕𝜇𝑖𝑗
𝑛
𝑚
𝑗=1 𝜇𝑖𝑗 . 𝑥𝑗
𝑛
𝑚
𝑗=1 𝜇𝑖𝑗
Updating degrees of membership:
𝜕𝐽 𝜇𝑖𝑗 , 𝑣𝑘
= 0 → 𝜇𝑖𝑗 =
𝜕𝑣𝑘
1
𝑥𝑗 − 𝑣𝑖
𝑘
𝑠=1
𝑚
𝑚−1
1
𝑥𝑗 − 𝑣𝑠
𝑚
𝑚−1
20
2. Spatially blind approaches
Clustering : Fuzzy C-Means Segmentation – Results
𝑘=3
𝑘 =5
𝑘 =10
𝑘 =20
21
2. Spatially blind approaches
Clustering : Mean-Shift Segmentation
Find the local extrema in the density distribution of a data set.
1. Extract features from the image (spatial position and spectral values or color).
2. Choose a search window (shape and probability density function – pdf).
3. Put a search window for each pixel in the image.
4. Compute the window’s center of mass.
5. Move the center of the window to the center of mass.
6. Repeat steps 4 and 5 until the search window stops moving or na sotpping criterion is
reached.
Centro de
Massa
Vetor
Mean shift
Centro de
Massa
nuevo
... Extração de
Atributos
Espaço de
Atributos
(exemplo 2D)
22
2. Spatially blind approaches
Clustering : Mean-Shift Segmentation
The kernel considering the two features is:
𝐾
𝒙; 𝜎𝑠2 , 𝜎𝑟2
=
2𝜋 −𝑑𝑠/2
1 𝒙𝒔 2
exp −
2 𝜎𝑠2
𝜎𝑠2 𝑑𝑠
2𝜋 −𝑑𝑟/2
1 𝒙𝒓 2
exp −
2 𝜎𝑟2
𝜎𝑟2 𝑑𝑟
Spatial position
Color
Where:
𝒙𝒔 , 𝒙𝒓 : Feature vector of dimensionalities 𝑑𝑠 , 𝑑𝑟 related to the spatial position and color respectively.
𝜎𝑠 , 𝜎𝑟 : Standard deviation for each kind of feature.
It is an exception due to the
inclusion of spatial features
23
2. Spatially blind approaches
Clustering : Mean-Shift Segmentation – Results
𝜎𝑠2 = 10, 𝜎𝑟2 = 10
𝜎𝑠2 = 10, 𝜎𝑟2 = 30
𝜎𝑠2
𝜎𝑠2
=
20, 𝜎𝑟2
= 20
𝜎𝑠2 =240, 𝜎𝑟2 = 20
= 40, 𝜎𝑟 = 20
𝜎𝑠2 = 10, 𝜎𝑟2 = 90
𝜎𝑠2 = 90, 𝜎𝑟2 = 20
24
2. Spatially blind approaches
Histogram Thresholding : Definitions
Grayscale
Sky
25
2. Spatially blind approaches
Histogram Thresholding : Otsu Segmentation
Select a threshold that minimizes the intra-group
variance.
1. Select an initial threshold 𝑇 = 1.
2. Threshold the image.
3. Two groups of pixels are created:
 𝐺1 with pixels < 𝑇.
 𝐺2 with pixels ≥ 𝑇.
4. Compute the proportions that represents each
group 𝑟1 , 𝑟2 and the variance of each group
𝜎12 , 𝜎22 .
5. Compute the intra-group variance:
2
𝜎𝑊
𝑇 = 𝑟1 𝑇 𝜎12 𝑇 + 𝑟2 𝑇 𝜎22 𝑇
6. Increase 𝑇 = 𝑇 + 1 and repeat steps from 2 to 5
until it is not possible to obtain two groups.
7. The optimal threshold corresponds to the
minimum intra-group variance.
T
𝑮𝟏
𝑮𝟐
26
2. Spatially blind approaches
Histogram Thresholding : Otsu Segmentation – Results
𝑇 = 169
27
Sumário
1. Introduction
2. Spatially blind approaches
3. Spatially guided approaches
28
3. Spatially guided approaches
Region based: Definitions
– Region Growing:
seeds
– Region Splitting:
– Region Merging:
29
3. Spatially guided approaches
Region based: Baatz & Schäpe Segmentation
Segmentation based on region growing and merging.
1. Initially, each pixel is considerd as one segment.
2. Later, in each step, a pair of segments is merged into one larger segment.
3. Each merge has na associated cost, a merging cost (𝑓).
4. A merging occurs when it has a merging cost inferior to an scale parameter. This merge
will promote the lowest increase in the global heterogeneity.
5. The segmentation procedure ends when no further merging can be performed.
There are two variants based on the heuristic taken to find the best segment for merging:
Local
Best Mutual
Fitting
Best
(BF)Fitting
(LMBF)
Lowestmerging
merging
Lowest
cost(𝑓)
(𝑓)
cost
Lowest merging
cost (𝑓)
30
3. Spatially guided approaches
Region based: Baatz & Schäpe Segmentation
The merging cost is calculated as follows:
𝑓 = 𝜔𝑐𝑜𝑙𝑜𝑟 × ℎ𝑐𝑜𝑙𝑜𝑟 + 1 − 𝜔𝑐𝑜𝑙𝑜𝑟 × ℎ𝑠ℎ𝑎𝑝𝑒
Color
where:
𝜔𝑐𝑜𝑙𝑜𝑟 : color weight.
𝜔𝑐𝑜𝑙𝑜𝑟 ∈ 0,1
Shape
𝐶 ∪𝐶
𝐶
𝐶
1
2
𝜔𝜔𝐿𝐿 𝐴
𝜔𝐶𝑐𝑜𝑚𝑝
×ℎ
𝜎𝐿𝑐𝑜𝑚𝑝
+− 1𝐴−
×𝑐𝑜𝑚𝑝
𝜎𝐿 1 +×𝐴ℎ𝐶𝑠𝑢𝑎𝑣
× 𝜎𝐿 2
𝐶1 𝜔
1 ∪𝐶2
2
ℎ𝑐𝑜𝑙𝑜𝑟
=
𝑠ℎ𝑎𝑝𝑒 =
𝐿𝐿
Onde:
𝑙 𝐶1
𝑙 𝐶1: Segments to be merged.
𝐶
,
𝐶
1
2
𝑆𝑢𝑎𝑣𝐶1 = 𝐶1
𝐶𝑜𝑚𝑝𝐶1 =
𝐶1 ∪𝐴𝐶𝐶21: Resulting segment
merging.
𝑙after
𝑟𝑒
𝐴𝐶1 : Area of segment 𝐶1 .
𝜎 : Standard deviation of pixels in segment 𝐶 .
ℎ𝑐𝑜𝑚𝑝 = 𝐴𝐶𝐶11 ∪𝐶2 × 𝐶𝑜𝑚𝑝𝐶1 ∪𝐶2 − 𝐴𝐶1 × 𝐶𝑜𝑚𝑝𝐶1 + 𝐴𝐶21 × 𝐶𝑜𝑚𝑝𝐶1
𝜔𝐿 : Weight corresponding to band 𝐿.
ℎ𝑠𝑢𝑎𝑣 = 𝐴𝐶1 ∪𝐶2 × 𝑆𝑢𝑎𝑣𝐶1∪𝐶2 − 𝐴𝐶1 × 𝑆𝑢𝑎𝑣𝐶1 + 𝐴𝐶2 × 𝑆𝑢𝑎𝑣𝐶2
𝐶Where:
1 (blue)
, 𝐶2 : Segments to be merged.
𝐶𝐶
2 1(red)
: Resulting segment after merging.
𝐶𝐶
1 1∪∪𝐶𝐶
2 2(green)
𝐴𝐶1 : Area of segment 𝐶1 .
𝑙 𝐶1 : Length of the edge of segment 𝐶1 .
𝐶
𝑙𝑟𝑒1 : Length of the edge of the bounding box of
the segment 𝐶1 .
31
3. Spatially guided approaches
Region based: Baatz & Schäpe Segmentation – Results
𝑠 = 20, 𝜔𝑐𝑜𝑙𝑜𝑟 = 0.9, 𝜔𝑐𝑜𝑚𝑝 = 0.5
𝑠 = 50, 𝜔𝑐𝑜𝑙𝑜𝑟 = 0.9, 𝜔𝑐𝑜𝑚𝑝 = 0.5
𝑠 = 90, 𝜔𝑐𝑜𝑙𝑜𝑟 = 0.9, 𝜔𝑐𝑜𝑚𝑝 = 0.5
𝑠 = 50, 𝜔𝑐𝑜𝑙𝑜𝑟 = 0.1, 𝜔𝑐𝑜𝑚𝑝 = 0.9
𝑠 = 50, 𝜔𝑐𝑜𝑙𝑜𝑟 = 0.5, 𝜔𝑐𝑜𝑚𝑝 = 0.9
𝑠 = 50, 𝜔𝑐𝑜𝑙𝑜𝑟 = 0.9, 𝜔𝑐𝑜𝑚𝑝 = 0.9
32
3. Spatially guided approaches
Energy based: Graph-based Segmentation
An image is modeled as an undirected graph: 𝐺 = 𝑉, 𝐸 ,
where:
𝑉: Vertices
𝐸: Edges
Connectivity Matrix
Each pixel is a vertex 𝑣𝑖 ∈ 𝑉.
Each edge 𝑒𝑖𝑗 has an associated weight 𝜔𝑖𝑗 .
1
𝑊 = 𝜔𝑖𝑗 ; 𝑖, 𝑗 ∈ 𝑉
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
𝜔𝑖𝑗 : Similarity measure
↑ 𝜔𝑖𝑗 → pixels are more similar
9
33
3. Spatially guided approaches
Energy based: Graph-based Segmentation
A cut is:
𝑨
𝑐𝑢𝑡 𝐴, 𝐵 =
𝜔 𝑢, 𝑣
𝑢∈𝐴,𝑣∈𝐵
𝑩
Minimum Cut;
however, it is not
the best one.
Then, the image can be segmented by
looking for minimum cuts.
A normalization based on the size of the
segments is robust to penalization of big
segments.
𝑐𝑢𝑡 𝐴, 𝐵
𝑐𝑢𝑡 𝐴, 𝐵
𝑁𝑐𝑢𝑡 𝐴, 𝐵 =
+
𝑎𝑠𝑠𝑜𝑐 𝐴, 𝑉
𝑎𝑠𝑠𝑜𝑐 𝐵, 𝑉
where:
𝑎𝑠𝑠𝑜𝑐 𝐴, 𝑉 =
𝜔 𝑢, 𝑡
𝑢∈𝐴,𝑡∈𝑉
34
3. Spatially guided approaches
Energy based: Graph-based Segmentation
In a matricial way, a minimum cut after simplification is:
𝒚𝑇 𝑫 − 𝑾 𝒚
min 𝑁𝑐𝑢𝑡 𝒙 = min
𝒙
𝒚
𝒚𝑇 𝑫𝒚
Where: 𝐷 𝑖, 𝑖 =
𝑗𝑊
𝑖, 𝑗 ; 𝐷 𝑖, 𝑗 = 0 (sum of weights of vertex 𝑖).
The solution is given by a problem of generalized eigenvalues:
𝑫 − 𝑾 𝒚 = 𝜆𝑫𝒚
Solved by conversion to a standard eigenvalue problem:
1
𝑫
−2
1
−2
1
2
𝑫 − 𝑾 𝑫 𝒛 = 𝜆𝒛 where: 𝒛 = 𝑫 𝒚
The solution corresponds to the second smallest eigenvector (the first one is zero).
35
3. Spatially guided approaches
Energy based: Graph-based Segmentation – Results
𝑁𝑠𝑒𝑔𝑚𝑒𝑛𝑡𝑠 = 5
𝑁𝑠𝑒𝑔𝑚𝑒𝑛𝑡𝑠 = 10
𝑁𝑠𝑒𝑔𝑚𝑒𝑛𝑡𝑠 = 50
𝑁𝑠𝑒𝑔𝑚𝑒𝑛𝑡𝑠 = 15
𝑁𝑠𝑒𝑔𝑚𝑒𝑛𝑡𝑠 = 20
36
3. Spatially guided approaches
Energy based: Segmentation based on Conditional Random Fields
Related to a different problem: Semantic Segmentation.
Acquisition
Enhancement /
Pre-Processing
Segmentation
Postprocessing
Feature
Extraction
Recognition
Classification
37
3. Spatially guided approaches
Energy based: Segmentation based on Conditional Random Fields
The image is modeled as an undirected graph: 𝐺 = 𝑉, 𝐸 ,
where:
𝑉: Vertices
𝐸: Edges
The goal is to maximize the following energy function:
𝐸 𝒙 =
𝜓𝑖 𝑥𝑖 +
𝑖∈𝑉
𝜓𝑖𝑗 𝑥𝑖 , 𝑥𝑗
𝑖∈𝑉,𝑗∈𝑁𝑖
where:
𝒙: random variable over the pixels to be labeled.
𝑥𝑖 : feature vector of pixel 𝑖.
𝑁𝑖 : neighbor pixels of pixel 𝑖.
𝜓𝑖 : Unary term for the pixel 𝑖.
𝜓𝑖𝑗 : Pairwise term for the edge 𝑖𝑗.
38
3. Spatially guided approaches
Energy based: Segmentation based on Conditional Random Fields
Let’s analyze the energy function:
Unary term
𝐸 𝒙 =
𝜓𝑖 𝑥𝑖 +
𝑖∈𝒱
Position
𝜓𝑖𝑗 𝑥𝑖 , 𝑥𝑗
𝑖∈𝒱,𝑗∈𝑁𝑖
Unary term
Color
Pairwise term
Norm of Difference of
intensity values
Pairwise term
Fourier basis
Histogram of
Gradients
(HoG)
𝑤1 𝜓𝑐𝑜𝑙 𝑥𝑖 + 𝑤2 𝜓𝑝𝑜𝑠 𝑥𝑖 + 𝑤3 𝜓𝑓𝑜𝑢𝑟𝑖𝑒𝑟 𝑥𝑖 + 𝑤4 𝜓ℎ𝑜𝑔 𝑥𝑖
39
3. Spatially guided approaches
Energy based: Segmentation based on Conditional Random Fields – Results
Result
Input Image
Reference
(Ground
Truth)
40
3. Spatially guided approaches
Based on Regions and Contours: Watershed Segmentation
Use region and contour information to segment an image.
The image is visualized as a topographic relief in 3D.
Intensity or Gradient
41
3. Spatially guided approaches
Based on Regions and Contours: Watershed Segmentation
1. Calculate the gradient module of the image (per band).
2. Consider for each pixel, the maximum value of the gradient per band.
3. A hole is punched in each regional minimum and the entire topography is flooded from
below, leaving water rise through the holes at a uniform rate.
4. A threshold 𝑇 can be set to uniformize the regional minimum.
42
3. Spatially guided approaches
Based on Regions and Contours: Watershed Segmentation
𝑇 = 0.1
𝑇 = 0.3
𝑇 = 0.5
𝑇 = 0.9
43
44
Image Segmentation
Qualification Exam
Deparment of Electrical Engineering - DEE
Pedro M. Achanccaray Diaz
Advisor: Raul Queiroz Feitosa