Lecture 5
INF 269, 2002, Lecture 5, page 1 of 32
University of Oslo
Department of Informatics
Fritz Albregtsen
Edge Based Segmentation
INF 269, 2002, Lecture 5, page 2 of 32
• In edge based segmentation we find the
basic elements of edges, i.e. edge- pixels
(or even line-pixels, corner-pixels etc),
based on local estimates of gradient.
In the next steps we thin broad edges, join
edge fragments together into edge chains.
Then we have a (partial) region border.
• In region based segmentation we find
those pixels that are similar.
Thresholding splits the image histogram so
that pixels that belong to the same class
get the same label, but these pixels are not
necessarily neighbours.
Region growing and split-and-merge
creates contiguous sets of pixels i.e.
regions.
• We describe two categories of methods,
based on two different principles, namely
similarity and discontinuity.
24/9 2002
• Segmentation creates regions and objects
in images. Remember lecture 1, pp 22-24.
INF 269
Digital Image Analysis
• Segmentation is one of the most important
components of a complete image analysis
system.
SEGMENTERING
INF 269, 2002, Lecture 5, page 3 of 32
• The most common problems of edge
based segmentation, caused by noise,
are an edge presence in locations where
there is no region/object border,
and/or no edge detected where there
actually is a real region/object border.
• May be influenced by variations in lighting,
and severely influenced by local variations
in scene reflectance and scene geometry.
• May be severely influenced by noise.
• Note that these are very local results
of edge detection.
• You have seen several of them!
Edge detection operators
• Thinning
INF 269, 2002, Lecture 5, page 4 of 32
• Morphological "majority black" or median
• Hysteresis thresholding
• Non-maximal suppression
• Simple thresholding
Post-processing of edge images
t
INF 269, 2002, Lecture 5, page 5 of 32
and the probability of false positive
detection
Z ∞
PF (t) =
p(g | no − edge)dg
t
• If we know the a priori probabilities
of edge and no-edge pixels in the image,
and the class conditional probability
distributions of an observed gradient
value, g, given the class, we can express the
probability of correct edge detection, given
a threshold t
Z ∞
p(g | edge)dg
PD (t) =
• What image gradient value is sufficient
to say that an edge has been found?
• Almost no 0-value pixels are present in an
edge image, as there are seldom truly
constant areas in gray level images.
Edge image thresholding
INF 269, 2002, Lecture 5, page 6 of 32
• In practice, finding a suitable global
threshold for gradient magnitude images
can be difficult or even impossible, and
more sophisticated methods must be used.
If the conditions for a solution are present,
but the a priori probabilities and class
conditional probability distributions are
not known, the solution can be found by
one of the methods from lecture 4,
or by trial and error.
P-tile thresholding may be an alternative.
• Note that this expression says nothing
about the shape of the distributions!
∂PE (t)
= 0
∂t
⇓
p(t | edge) · P (edge) = p(t | no − edge) · P (no − edge)
Notice that this is essentially the same
reasoning as you have already seen in
optimal thresholding. So it is a Bayesian
minimum error problem, solved by
PE (t) = [1 − PD (t)] P (edge)+[PF (t)] P (no−edge)
• The rate of classification error is then
Edge image thresholding II
INF 269, 2002, Lecture 5, page 7 of 32
• More sophisticated estimates of edge
direction will require larger filters in edge
direction estimates.
• This may be simplified for 4-connectivity.
3 Erase to zero all marked pixels.
2 For each pixel with g(x, y) 6= 0:
inspect the two opposite pixels indicated
by edge direction of current pixel.
If any of their magnitudes are larger than
that of current pixel, mark current pixel
for deletion.
1 Divide edge pixels into eight directions
(assuming 8-connectivity)
• Solution:
If edges carry directional information,
suppress multiple responses in
neighborhood of single boundaries
through non-maximal suppression.
• Problem:
Simple thresholding of gradient image
gives thick edges.
Non-maximal suppression
INF 269, 2002, Lecture 5, page 8 of 32
• Exercise: compare non-maximal
suppression and hysteresis thresholding,
and try to understand what influence the
two approaches have on the resulting edge
image, and why the results may be
different.
• Hysteresis to filter output of the edge
detector is used to eliminate
noisy/spurious responses in the Canny
edge detector.
• One solution to this is hysteresis
thresholding, provided that suitable
thresholds can be determined.
The p-tile approach is often very good.
• The output of non-maximal suppression
will probably contain noise.
Hysteresis thresholding, once more
INF 269, 2002, Lecture 5, page 9 of 32
• Canny chooses t1/t0 ∈ [2, 3]
• Neighbors may be defined as 4- or
8-connected.
INF 269, 2002, Lecture 5, page 10 of 32
• The method described in the book uses
crack edges, i.e. inter-pixel boundaries.
• An edge pixel (even a strong one)
positioned by itself is highly probable not
to belong to any boundary.
• A pixel with a weak edge magnitude
located between two strong edge pixels is
highly probable to belong to the boundary.
• Relaxation is a matter of making a pixel
more like its neighbours.
2 For each edge pixel with t0 <= g <= t1
If the pixel is the neighbor of a marked
pixel, mark the pixel as edge.
3 Repeat step 2 until no further changes.
• Considering edge properties in a
neighborhood of a pixel can increase the
quality of the resulting border.
— giving gaps in borders
— giving edge pixels where there are no
real borders
1 All edge pixels with magnitude g > t1 are
marked as correct edges.
Hysteresis thresholding, algorithm
• Borders are often affected by noise
Edge Relaxation
INF 269, 2002, Lecture 5, page 11 of 32
• Can you think of any method that
iteratively uses the original image to
omprove the estimate of the boundary?
• Please note that in relaxation, the original
image is only used in the initial step!
• It then iteratively updates this set of initial
probabilities for each pixel, using the
previous generation of class probabilities
in the neighboring pixels together with a
set of a priori compatibility or consistency
coefficients.
• Relaxation segmentation starts with a
preliminary (fuzzy) determination of pixel
class membership.
Relaxation, general principle
INF 269, 2002, Lecture 5, page 12 of 32
— C = +1 => compatible
— C = 0 => "don’t care"
— C = -1 => incompatible
Which gives the probability of pixel k
belonging to class s, when the neighboring
pixel l belongs to class t.
−1 ≤ C(k, s|l, t) ≤ 1
• Relaxation will now update the
probabilities of each pixel, using the
previous probabilities of its neighbors, and
using a compatibility parameter
• For every pixel, we have the probability
p(k, s) that pixel k belongs to class s.
This may be realized as an N -band
probability image.
• Assume that we have an image containing
N classes, e.g. N=2, foreground (F) and
background (B).
Relaxation - I
f (j)(k, s) = 1
INF 269, 2002, Lecture 5, page 13 of 32
• We then iteratively improve (update) the
probabilities of each pixel, using
information about its neighbors.
s=1
N
X
• The sum of the probabilities that a pixel k
belongs to a class s, summed over all N
classes, must be 1:
0 ≤ f (j)(k, s) ≤ 1
• For all states j and all pixels k we have
• Let f 0(k, s) be the initial state, i.e. the initial
estimate of the probability that a pixel k
belongs to class s, for all pixels and all
classes, s ∈ [1, ..., N ].
Relaxation - II
INF 269, 2002, Lecture 5, page 14 of 32
• The product c(k, s; l, t)f (l, t) posesses these
qualities, and gives us the desired
increments of the probability f (k, s)
3: If f (l, t) is low or c(k, s; l, t) ≈ 0
keep f (k, s)
because either the probability of l ∈ t is
small, or this is irrelevant to k ∈ s.
2: If f (l, t) is high and c(k, s; l, t) < 0
decrease f (k, s)
because k ∈ s is in-compatible with
(c < 0)
an event (l ∈ t) having a high probability.
1: If f (l, t) is high and c(k, s; l, t) > 0
increase f (k, s)
because k ∈ s is compatible with (c > 0)
an event (l ∈ t) which has high
probability.
• We have three cases:
• Let f (l, t) be the probability that the
neighbor l belongs to class t, while
c(k, s; l, t) is a measure of the probability
that pixel k belongs to class s while its
neighbor l belongs to class t.
Relaxation - III
0
t=1
l=1
0
c(k, s; l, t)f (j)(l, t) ; cl (k, s) ∈ [−1, 1]
f (k, s) = 1
(j)
INF 269, 2002, Lecture 5, page 15 of 32
• If we multiply f (j)(k, s) by (1 − a(k, s)) we
will surely get a positive result, because
a(k, s) ≥ −1. If we divide every new
estimate by the sum of the estimates, then
the sum of the normalized estimates is 1 :
f (j)(k, s) · (1 + a(k, s))
(j+1)
(k, s) = PN
f
(j)
t=1 f (k, t) · (1 + a(k, t))
s=1
N
X
(j)
0 ≤ f (k, s) ≤ 1
• The mean increment a is then used to
update f j (k, s). Please remember that:
• For each class s we sum over all n
neighbors, and find a mean increment a
n
1X 0
cl (k, s) ; a(k, s) ∈ [−1, 1]
a(k, s) =
n
cl (k, s) =
N
X
• For every neighbor l we sum up the
increment over all N possible classes
Relaxation - IV
INF 269, 2002, Lecture 5, page 16 of 32
— If f (j)(k, s) has become 0 or 1,
then f (k, s) will not change any more.
• Summary:
— If f (j)(k, s) = 1 for a given class s, then
f (j)(k, t) = 0 for all other classes t, and
the denominator in the expression for
f (j+1)(k, s) will be the same as the
nominator, giving f (j+1)(k, s) = 1.
— If f (j)(k, s) = 0,
then we also have f (j+1)(k, s) = 0.
• Note that
• This is performed for all pixels within the
image, and iterated. f (j)(k, s) will converge
to 0 or 1, and the pixel k will be assigned to
the class s where f (j)(k, s) ≈ 1.
• Just repeating the expression to update the
probability f (k, s) that pixel k belongs to
class s :
f (j)(k, s) · (1 + a(k, s))
(j+1)
(k, s) = PN
f
(j)
t=1 f (k, t) · (1 + a(k, t))
Relaxation - V
n
c(k, s; l, t)f (l, t)
j
l=1
1X 0
a(k, s) =
cl (k, s)
n
t=1
N
X
INF 269, 2002, Lecture 5, page 17 of 32
• f j (k, s) will converge to 0 or 1, and the pixel
k is assigned to the class s where
f j (k, s) ≈ 1.
end;
for s := 1 until classes do
f j (k, s) · (1 + a(k, s))
j+1
f (k, s) = PN
j
t=1 f (k, t) · (1 + a(k, t))
end;
end;
end;
end;
cl (k, s) =
0
• New estimates of f j (k, s) are found by :
for j := 0 until iterations - 1 do
for k := 1 until pixels do
for s := 1 until classes do
for l := <all neighboring pixels> do
Relaxation - “program”
INF 269, 2002, Lecture 5, page 18 of 32
• Boundary tracing in gray level images.
• Extended boundary tracing
• Using inner or outer border definition,
two adjacent regions never have a
common border.
— Outer region border is not a subset of
the region.
• Outer boundary tracing
— Inner region border is a subset of the
region.
• Inner boundary tracing
Border tracing
INF 269, 2002, Lecture 5, page 19 of 32
Q: What happens with one pixel wide
regions?
Q: What about borders of region holes?
Q: What is the difference between the
algorithm given above and algorithm 5.8 in
the book?
3 If the new boundary pixel is equal to the
starting pixel P0, then stop, otherwise
repeat step 2.
3 A clean-up may be necessary, as some
pixels may have been tested more than
once, and are therefore listed more than
once in the outer boundary list.
2 Search its 3x3 neighborhood in
anti-clockwise direction.
The first pixel found belonging to the
region is a new boundary pixel Pi.
INF 269, 2002, Lecture 5, page 20 of 32
• Q: How many times may a pixel be
repeated in the outer border list?
1 Trace the inner region boundary in
4-connectivity.
2 The outer boundary consists of all
non-region pixels tested during the
search process.
• Algorithm:
Outer boundary tracing
1 Find a starting pixel P0.
Q: How? A: search image from top left
until a pixel of a new region is found.
This pixel has the minimum column
value of all pixels of that region having
the minimum row value.
• An inner border tracing algorithm for both
4- and 8-connectivity:
• Assume that the image with regions is
either binary or that regions have been
labeled. Remember that an inner region
border is a subset of the region.
Inner boundary tracing
INF 269, 2002, Lecture 5, page 21 of 32
— The border between adjacent regions
has to be traced only once.
— The boundary shape is exactly equal to
the inter-pixel shape, but is shifted 12
pixel down and to the right.
— All the useful properties of the outer
border remain.
• Extended borders are defined as single
common borders between adjacent
regions, still being specified by standard
pixel coordinates.
• Crack edges (inter-pixel boundary) is
common to adjacent regions, but can not
be described by standard pixel
coordinates.
• The inner boundary is always part of a
region, but the outer boundary never is.
If two regions are adjacent, they never have
a common border.
This may cause difficulties in region
description, region merging etc.
Extended boundary
INF 269, 2002, Lecture 5, page 22 of 32
• The Liow 1991 algorithm, based on a look up table of all 12
possible local configurations of 2x2 pixel windows is covered
in the book.
EB(R) = {UPPER outer boundary shifted one pixel down and
to the right}
∪{LEFT outer boundary shifted one pixel to the right}
∪{RIGHT outer boundary shifted one pixel down}
∪{LOWER outer boundary}.
• Construction of EB from outer boundary:
EB(R) = {P |P ∈ LEF T (R)} ∪ {P |P ∈ U P P ER(R)}
∪ {P6 (P )|P ∈ LOW ER(R)} ∪ {P0 (P )|P ∈ RIGHT (R)}
∪ {P7 (P )|P ∈ RIGHT (R)} ∪ {P6 (P )|P ∈ LOW ER(R)}
• RIGHT and LOWER pixels ∈ background.
• LEFT and UPPER pixels ∈ the region.
• EB(R) consists of both region and
boundary pixels, depending on the local
direction of the border.
• All four are subsets of R.
— A LOWER pixel of R if P6 (p) ∈ Q
— A LEFT pixel of R if P4(p) ∈ Q
— A UPPER pixel of R if P2 (p) ∈ Q
— A RIGHT pixel of R if P0 (p) ∈ Q
• Four kinds of inner boundary pixels of a
region R are defined. If Q denotes pixels
outside the region R, then a pixel P ∈ R is
Extended boundary tracing
INF 269, 2002, Lecture 5, page 23 of 32
• Both edge gradient magnitude and
direction are considered in pixels of
probable border continuation.
• The border is constructed iteratively by
adding the next pixel that is in the most
probable direction.
• Tracing is started in a pixel with high
probability of being a border element.
• The border is represented by a path of high
gradient pixels.
Gray level boundary tracing
INF 269, 2002, Lecture 5, page 24 of 32
• The generalized HT can be used if analytic
equations of searched shapes are not
known, but one still needs to define size
range, rotation range etc.
• HT is robust in presence of additional
structures.
• HT is robust in detection of partially
occluded and noisy objects.
• HT can detect straight lines and curves if
their analytic equations are known.
• This problem could be solved by looking
for correlation between the image and a
moving mask. Would have to go through
rotation, zoom, distortions etc.
• Hough transform (HT) based
segmentation is applicable if objects of
known shape are to be detected within the
image. The goal is to find their location
and size.
Hough-transform
INF 269, 2002, Lecture 5, page 25 of 32
in the (ab) parameter domain.
b = −xia + yi
• Every point (xi, yi) in the image domain
corresponds to a straight line given by
• Regard a point (xi, yi)
and a straight line yi = axi + b
• We wish to find sets of pixels that make up
straight lines.
• Thus, we have n pixels
that may partially describe the boundary
of some objects.
• Assume that we have performed some
edge detection, and a thresholding of the
edge magnitude image.
Hough-transform
INF 269, 2002, Lecture 5, page 26 of 32
• This representation does not handle lines
close to the vertical !
• A value A [i, j] = M corresponds to
M points in the x, y-plane being found on
the line
y = ai x + b j
• set all A[a,b] := 0;
for every (x,y)
where g(x,y) > T
begin
for all possible values of a
begin
b = -xa + y;
A[a,b] := A[a,b] + 1;
end
end
• Partition the ab-plane into accumulator
cells
A [a, b] , a ∈ [amin, amax] ; b ∈ [bmin, bmax]
Hough-transform II
INF 269, 2002, Lecture 5, page 27 of 32
• Local maxima => significant lines
will give M curves crossing each other at
(ρi, θj ) in the parameter plane.
x cos θj + sin θj = ρi
• M co-linear points lying on the line
• Each point (xi, yi) in the xy-plane gives a
sinusoid in the ρθ-plane.
x cos θ + y sin θ = ρ
• We utilize the normal representation of
straight lines
Hough-transform III
INF 269, 2002, Lecture 5, page 28 of 32
— vertical lines have θ = 90o, ρ ≥ 0
or θ = −90o, ρ ≤ 0
√
• The range of ρ is ±D 2
where D er the length of the side of the
(square) image.
— horizontal lines have θ = 90o, ρ ≥ 0
• The range of θ is ±90o from the x axis
• Partition the ρθ-plane into accumulator
cells A [ρ, θ] , ρ ∈ [ρmin, ρmax] ; θ ∈ [θmin, θmax]
Hough-transform IV
INF 269, 2002, Lecture 5, page 29 of 32
— Traverse those pixels that correspond to
a maximum in parameter space.
Check that gap < tolerance.
— Local thresholding.
• Solution:
— Low threshold => noise influence.
— High threshold => short segments lost.
• How do we find local maxima in parameter
space, corresponding to line segments in
image space?
• Co-lineare linje segments cannot be
separated.
• The length and the position of a line
segment cannot be determined.
Hough-transform, problems
INF 269, 2002, Lecture 5, page 30 of 32
• Algorithm:
for all g(xi, yi) > T do
ρ := xi cos(φg (x, y)) + yi sin(φg (x, y))
increment A(ρ, φg (x, y))
• Better algorithmif we have both
- the gradient magnitude g(x, y)
- the gradient components
gxand gy
g
φg (x, y) = arctan gxy
• Simple algorithm:
for all g(xi, yi) > T do
for all θ do
ρ := xi cos θ + yi sin θ
increment A(ρ, θ)
• Given a gradient magnitude image g(x, y)
containing a line segment.
Hough-transform, local gradient
INF 269, 2002, Lecture 5, page 31 of 32
• More elegant ways of doing this ?
• Simple procedure :
set all A[a,b,c] := 0;
for every (x,y) where g(x,y) > T
begin
for all a and b
begin
c = sqrt((x-a)**2 + (y-b)**2);
A[a,b,c] := A[a,b,c] + 1;
end;
end;
• So we have a 3D parameter space.
(x − a)2 + (y − b)2 = c2
• A circle in the xy plane is given by
Hough-transform, circles
INF 269, 2002, Lecture 5, page 32 of 32
2 Consider the (scaled) number of
markers a pixel has as the probability of
belonging to a region.
Low probability ⇒ background pixel.
1 For each border pixel:
(a) Search for an opposite border pixel.
(b) If an opposite border pixel is found,
mark each pixel along the straight line as
a potential region pixel.
• Algorithm based on probabilities that
pixels are located inside a region:
• Region forming from partial borders is
possible.
Region construction