Segmentation from
Examples
By: A’laa Kryeem
Lecturer: Hagit Hel-Or
What is Segmentation from
Examples?
 Segment an image based on one
(or more) correctly segmented
image(s) assumed to be from the
same domain
 Effective when making a
semantic segmentation
Why to use Examples
 The example defines the
granularity of the desired
output
 Give us the ability to
characterize meaningful
parts in the image
 Using example allow us to
use non-parametric model
The example defines the
granularity of the desired output:
Training image
Desired Segmentation
Test Image
Induced Segmentation
Why to use Examples
The example defines the
granularity of the desired output
Give us the ability to
characterize meaningful parts in
the image
Give us the ability to
characterize meaningful parts
in the image
Semantic Segmentation from an
example
 We want to segment an image
into semantically meaningful parts
 Required in various applications
Semantic Segmentation from an
example
 We want to segment an image
into semantically meaningful parts
 Required in various applications
 Problems:
 Meaningful parts are often too
complex
Semantic interpretation is highly
subjective, depending on both
the application, and the user
Meaningful parts are often too complex
Semantic Segmentation from an
example
 We want to segment an image
into semantically meaningful parts
 Required in various applications
 Problems:
 Meaningful parts are often too
complex
Semantic interpretation is highly
subjective, depending on both
the application, and the user
Example of image different
segmentation
Semantic Segmentation from an
example
 So, How to achieve semantic
segmentation
Getting segmented training image(s)
as input
Training set
Semantic Segmentation from an
example
 So, How to achieve semantic
segmentation
Getting segmented training image(s)
as input
Using non-parametric representation
non-parametric model
Each semantic part is represented
by a set of square patches
Semantic Segmentation from an
example
 So, How to achieve semantic
segmentation
Getting segmented training image(s)
as input
Using non-parametric representation
Over-segmenting the Test image into
small fragments
Over segmented image
Semantic Segmentation from an
example
 So, How to achieve semantic
segmentation
Getting segmented training image(s)
as input
Using non-parametric representation
Over-segmenting the Test image into
small fragments
Compute costs for fragment-label
pairs
(fragment,label) cost example
?
Semantic Segmentation from an
example
 So, How to achieve semantic
segmentation
Getting segmented training image(s)
as input
Using non-parametric representation
Over-segmenting the Test image into
small fragments
Compute costs for fragment-label
pairs
Graph-cuts multi-label optimization
Why do we need graph-cuts
 Graph-cuts optimization is used to label each
fragment in a globally optimal manner
Classification
scores
Patch sets
Classification
Training set
Fragments
Test image
Graph-Cuts
optimization
Fragmentation
Result
Over segmenting
 Fragment: small arbitrarily-shaped
and simply-connected pixel clusters
 We assume that small homogeneous
regions always belong to the same
semantic part
Over segmenting
 Fragment: small arbitrarily-shaped
and simply connected pixel clusters
 We assume that small homogeneous
regions always belong to the same
semantic part
 Advantages:
Enforces a locally coherent labeling
Reduces the computational complexity
Graph-cuts multi-label
optimization
 For each fragment we have k cost values,
we need to determine which one is the
optimal
 Using expanded version of the graph-cuts
we saw at Jad’s lecture, where we may
have more than two labels (background ,
object)
Algorithm for semantic
segmentation
1.Pixel labeling costs
2.Fragmentation
3.Fragment labeling costs
4.Graph-cuts optimization
Algorithm for semantic
segmentation
1.Pixel labeling costs
2.Fragmentation
3.Fragment labeling costs
4.Graph-cuts optimization
Pixel labeling costs
Given Itrain and Ltrain
Representing each
label(segment) in Ltrain by a set
of square patches
We get k sets {Sl} l=1,…,k one for
each label
Pixel labeling costs (cont.)
 Next, we define φ (p, l) for each
(pixel,label) pair
The cost of assigning label l to pixel p Itest
φ(p,l)=
𝑠𝑠𝑑(𝑃,𝑃′ )
min
𝑀
P’ Sl
p:pixel at Itest
l : label in Ltrain
ssd(P,P’) is the sum of squared distances
between P,P’
M:mxmx3
P:mXm neighborhood centered at p
P’:mxm patch
Algorithm for semantic
segmentation
1.Pixel labeling costs
2.Fragmentation
3.Fragment labeling costs
4.Graph-cuts optimization
Fragmentation
We partition Itest into small,colorhomogeneous regions using mean shift
segmentation
Fragmentation
We partition Itest into small,colorhomogeneous regions using mean shift
segmentation
Fragment size is adjusted according to
Itest.(fragments are smaller in more detailed
areas of Itest, and larger in more
homogeneous regions)
Fragmentation
We partition Itest into small,colorhomogeneous regions using mean shift
segmentation
Fragment size is adjusted according to
Itest.(fragments are smaller in more detailed
areas of Itest, and larger in more
homogeneous regions)
Fragment boundaries align with edges in
the image
Fragmentation (cont.)
Random colorization
Detailed close-up
Algorithm for semantic
segmentation
1.Pixel labeling costs
2.Fragmentation
3.Fragment labeling costs
4.Graph-cuts optimization
Fragment labeling costs
 Voting scheme in order to compute
labeling costs of each fragment
 For each fragment f Itest we pick a few
representative pixels:
Rep(f)={pi f } i=I,…,Rf
Rf is proportional to |f|
Fragment labeling costs
(cont.)
We talked about the cost of
assigning a pixel into a label φ(p,l),
now we need to adjust φ to work on
fragments
φ(f,l)= median {φ(p,l) | p Rep(f)}
Fragment labeling vs. pixel
labeling
Fragment labeling reducing
complexity
We have n=
𝑓 |𝑓| ,
Assume
|Rep(f)|= |𝑓| ,then we need to
compute costs only for 𝑛 pixels
Fragment labeling vs. pixel labeling
Enforces a locally coherent labeling
Training image
Pixel
labeling
Training seg.
Input image
Fragment
labeling
Algorithm for semantic
segmentation
1.Pixel labeling costs
2.Fragmentation
3.Fragment labeling costs
4.Graph-cuts optimization
Graph-cuts optimization
After calculating labeling cost for all
image fragments we get k images.
Image i describes the cost assigning
each pixel at the test image to label i
fragment labeling costs. Costs range in the interval
[0,1]
Graph-cuts optimization
 Now for each pixel p Itest we have a labeling
cost
 We need to find Ltest the globally optimal
labeling
Requirements:
Minimizes the total labeling cost
Consistent with presence (or absence) of
edges
Graph-cuts optimization
(cont.)
 For each pair of neighboring pixels 𝑝, 𝑞 we
define:
0
Ψ(p,q,L(p),L(q))=
1 − 𝛻 𝑝, 𝑞
𝐿 𝑝 = 𝐿(𝑞)
𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
L(p),L(q) : labels assigned to p,q
𝛻 𝑝, 𝑞 : difference between pixels
p,q (RGB euclidian distance)
Graph-cuts optimization
(cont.)
 In order to force pixels within each fragment to be
labeled the same, and reduce complexity we specify
the energy term E(L) (E(L) the value of a labeling
scheme) in terms of fragments instead of pixels :
𝐸 𝐿 =
𝑓
𝑓 ∙ 𝜑 𝑓, 𝐿 𝑓
+𝛼
𝑓1, 𝑓2
Ψ(𝑓1 , 𝑓2 , 𝐿 𝑓1 , 𝐿 𝑓2 )
𝜑 𝑓, 𝐿 𝑓 : cost of assigning fragment f to label L(f),
weighted by the size of each fragment.
𝑓1, 𝑓2 : neighboring fragments in Itest.
Ψ(𝑓1 , 𝑓2 , 𝐿 𝑓1 , 𝐿 𝑓2 =
𝑝,𝑞 ,𝑝𝜖𝑓1 ,𝑞𝜖𝑓2 Ψ(p,q,L(𝑓1 ),L(𝑓2 ))
𝛼 :controls trade-off between regions and boundaries
Intuition:
Ψ(𝑝, 𝑞, 𝐿 𝑝 , 𝐿 𝑞 )
𝛼
𝑝,𝑞
 Big 𝛼 value:
For pixels p,q even with different colors the
graph-cuts step prefers to connect them
to the same label to have Ψ=0 and
reduce the energy. Instead of 1-𝛻 𝑝, 𝑞
althout 𝛻 𝑝, 𝑞 can be big too.
This mean we prefer continues regions
and not edges.
Intuition:
Ψ(𝑓1 , 𝑓2 , 𝐿 𝑓1 , 𝐿 𝑓2 )
𝛼
𝑓1, 𝑓2
 small 𝛼 value:
For pixels p,q even with similar colors
the graph-cuts step won’t care about
connecting them to different labels
because of small 𝛼 value. Even with
big value at1-𝛻 𝑝, 𝑞 multiplying 𝛼 still
give us small E value.
Favors boundaries, holding out noncontinues regions .
Graph-cuts optimization
(cont.)
Finally Ltest is determined by solving
Ltest=minL E(L)
Fragment labeling
Labeling after GraphCuts Optimization
Multi-label graph-cut
Colored nodes:labels
Squares : fragments
For each
(fragment,label) pair we
have an edge.
Edge weigh according
to φ.
Edges between two
squares weighed
according to Ψ.
Multi-label graph-cut
Induced graph
Each fragment
connected to a
single label.
Multi-label graph-cut is
NP-complete problem!
 Using Isolation Heuristic we can get an
approximation of E(L)
1. For 1≤i ≤ k construct a minimum weight
isolating cut Ei for label Li .
2. Determine h such that Eh has max weight.
3. E=
𝐸𝑖 ≠𝐸ℎ 𝐸𝑖 .
4. Return E.
𝛼 effect
Training image
𝛼=0.1
Training segmentation
Input image
𝛼=5
𝛼=1
Algorithm for semantic
segmentation
1.Pixel labeling costs
2.Fragmentation
3.Fragment labeling costs
4.Graph-cuts optimization
Algorithm results
Training set
a
b
c
Bear results
invariant to the number of
instances of each semantic part
within the image, and insensitive
to the shape of each part.
We can’t separate multiple
objects belonging to the same
label (c).
Algorithm results
Training set
a
b
c
d
Summary
 We saw that giving segmented example from
the same domain of an image can effectively
perform a semantic segmentation
 Using example also defines the granularity of
the desired output
 determining whether an entity belongs to a
particular semantic part is more easily done at
the fragment level, than on a pixel-by-pixel
basis
 Using graph-cuts with multi-label support can
help making global optimization step for
finding optimal labeling
 Only one parameter needed 𝛼, controlling the
trade-off between regions and boundaries
Thank You For Listening
References
 Inducing Semantic Segmentation from an Example,
Yaar Schnitman, Yaron Caspi, Daniel Cohen-Or,
and Dani Lischinski.
 "Segmentation by Example“, Sameer Agarwal and
Serge Belongie.
 Christoudias, C.M., Georgescu, B.: Edge detection
and image segmentation (edison) system.
 Boykov, Y., Veksler, O., Zabih, R.: Fast approximate
energy minimization via graph cuts. IEEE Trans.
Pattern Anal. Mach. Intell. 23 (2001) 1222–1239