Homework #3: Image Segmentation
Due Thursday, November 9th, 2006.
Professor Davi Geiger
CONSIDER THE
IMAGES
(WORK WITH 8 NEIGHBORS, NOT 16, TO MAKE IT SIMPLE)
Fig 1 (a) Test+Noise Images
(b) Butterfly
(d) Bird
(c) House
(e) Face
The goal is to segment these images. To assign to each pixel a label corresponding to a region. The
first step is to choose the labels or levels of grey values that are going to be used for segmentation.
We don’t know a priori how many labels there are, and that must be found as well. After that we apply the Belief Propagation method to assign a label to each pixel..
1. Select the Gray Levels (50 points)

( p )  max  | DIˆ( p ,   / 2, s  5) | to every pixel in the image.
a. Apply the function fˆsmax
5
Select the pixels such that this function is above a threshold T. Make sure to use T high enough
so that only high contrast pixels can be selected. Show the results as images where black pixels
represent that they passed the test. Indicate the threshold used.
b. For each pixel that passed the test, say there are K of them, assign a pair (I1,I2) where I1 and I2

3
, s  5) ), respecare the min and max values from the pair ( Iˆ( x, y,   , s  5) , Iˆ( x, y,  
2
2

tively. Construct the list of all K pairs I K  I 1i , I 2i ; i  1,2,..., K  . Select the smallest set of la1

bels LG={l1, l2, .., l2G-1, l2G} such that for any pair I 1i , I 2i  I K the closest label to I 1i and the
closest label to I 2i (closest in Euclidean distance) are different ones. In this way, we are guar
antee that all pairs I 1i , I 2i  I K will be assigned different labels. One possible, trivial, solution

is to choose the labels to be LG = I K  I 1i , I 2i ; i  1,2,..., K  . However, we want a smaller set
of labels. The algorithm to find a valid label set LG follows

Find_Label _Set ( I K , K) {
P=K;
 
I 0 = IK ;
G=0;
L[2K]=0; /* array that will store the labels */

Loop while I 0 ≠Ø { /* or stop when P=0 */
(x1,x2)=(0,255); /* pair created to become the new label values for 8 bit images */
G=G+1;
/* label index */

Loop for i=1,…,P { /* (i.e., loop for all pairs I 1i , I 2i  I 0 ) */
Q=P;
If (( I 1i ≤ x2) & ( I 2i ≥ x1)) { /* A condition to be part of the group indexed by G,
an intersection with the current interval must exist */
Q=Q-1;
 
I 0 = I 0 - I 1i , I 2i ; /* removing the pair from the list */


i
1
i
2
x1=max(x1, I ); /* extracting the largest value from the left */
x2=min(x2, I ); /* extracting the smallest value from the right */
}
}
L [2G-1]=x1 /* new label */
L [2G]=x2; /* new consecutive label */
P=Q;
If (P=0) break;
}
return G;
}
The set of labels is thus stored in the array L [2K], at L [1], L [2], …, L [2G]. We note that the set of
labels, ordered as pairs (l2k-1, l2k), do not overlap. If they did, they would have been grouped by the
above algorithm. Note that the output is not an ordered set and so one can sort the values and store it
in a list LG =(l1, l2, …, l2k-1, l2k, …, l2G-1, l2G ) so that l2k-1< l2k < l2k+1 for every k=1,…,G. Note that,
from figure below, labels l2k and l2k+1 can be merged to one label, say Lk=1/2 (l2k + l2k+1), except the
first one l1 and the last one l2G , so that effectively one produces G+1 labels, L0, L1 …, Lk,, …, LG-1, LG .
0
255
(
)
(
)
l1
l2
l3
l4
(
)
l2k-1
(
)
l2k l2k+1 l2k+2
2
(
l2G-1
)
l2G
Homework output: For each image, print the list LG =(l1, l2, …, l2k-1, l2k, …, l2G-1, l2G ). Then assign to each pixel the nearest label in LG. Then, combine labels L2k=1/2 (l2k + l2k+1) to reduce the
number of labels and display an image where each pixel is assigned this label (after reduction
by combination).
2. Segmentation via Belief Propagation (50 points)
We are now giving a set LG of 2G labels and we want to apply a segmentation algorithm that not
only assigns labels according to the gray values but also bias neighbor pixels to be at the same state,
thus bonding nearby pixels. Let us index the labels as g=1, 2, …, 2G.
We choose a compatibility function between the data and the labels as
N
   i (l g (i ) | I i )
i 1
i (l g (i ) | I i ) 
e

 I i l g ( i )
G 1
e

2
 I i l g
2
i  1,..., N
,
g (i)  1,...,2G ,
a 1
where the function g(i) assigns a label lg(i) to pixel i. The further away the grey value Ii is from the
label lg(i) the less the likelihood this label will be assigned. We normalize across all labels. The parameter needs to be estimated. If → then all labels are equally likely. If → ∞then only the
best label is set to 1 and the others are set to 0. This is because the best label term will dominate the
denominator sum (all other terms will go to zero much faster than the best label term). Then, when
the numerator is the best label term, it will cancel with the denominator to give 1 and when the numerator is any other term will again loose to the denominator and thus the ratio goes to 0. Note that
the exercise 1, above, provides alabel match solution that maximizes (for any value of ≠). We
now must choose a compatibility function between pixels.
N
    ij (l g (i ) , l g ( j ) )
i 1 jN i
1  
if ( g ( j )  g (i )) or if ( g ( j )  g (i )  1  2[ g (i) mod 2])
 4G
 ij (l g (i ) , l g ( j ) )  


otherwise
4G (G  1) terms
 4G (G  1)
4G terms
,
where needs to be estimated and tested. We have considered labels l2k and l2k+1 to be associated to
the same label “Lk”, k=1, …G, i.e., we bind these labels as if they were the same label. We can’t work
with labels Lk directly because the data driven term, must use the whole set of g=1,…., 2G labels.
The problem is to find the best mapping g(i) for each pixel i=1,…,N. We use Kai Ju Liu’s program
to feed the belief propagation and obtain an estimation to the probability for each pixel to be assigned
a label, P(lg(i)|i). Once this probability are computed we apply a reduction of the labels as in exercise
1, i.e., labels l2k and l2k+1 can be combined to one label, Lk=1/2 (l2k + l2k+1), except the first one, l1,
and the last one, lG+1 , so that effectively one produces G+1 labels. We assign the probability to the
new label Lk to be P(Lk|i) = P(l2k|i)+ P(l2k+1|i) replacing the two labels l2k and l2k+1 . Finally, we can
select the best label for a given pixel as the one that maximizes the new assigned probabilities.
3
Homework output: Display results of the best label for the images we considered and provide
parameters 
4