Colorado School of Mines
Image and Multidimensional Signal
Processing
Professor William Hoff
Dept of Electrical Engineering &Computer Science
Colorado School of Mines
Image and Multidimensional Signal Processing
http://inside.mines.edu/~whoff/
Representation and Description
Colorado School of Mines
Image and Multidimensional Signal Processing
2
Representation and Description
• After segmenting an image to find regions of interest, we
want to represent those regions in a concise form
• This can be used for
– Recognition
– Compression
– Further processing (e.g., joining, simplifying, tracking)
• Two main approaches
– Boundary based methods
– Region based methods
Colorado School of Mines
Image and Multidimensional Signal Processing
3
Boundary Representations
• We want a compact representation of the boundary of a
binary region, to support recognition
• Methods:
– Chain codes
– Fourier descriptors
Colorado School of Mines
Image and Multidimensional Signal Processing
4
Boundary-based Representations
• Chain codes
– Follow the boundary around a region
– Record the direction of travel
– Yields a sequence of numbers that can
be stored concisely, or used for
recognition
4-directional
8-directional
• Problems
– Chain codes can be long
– Small disturbances in the boundary can
cause large changes in the code
• Solution
– We can resample on a larger grid
spacing
Colorado School of Mines
Image and Multidimensional Signal Processing
5
start
Examples
Resulting code:
0033 … 01
Resulting code:
0766 .. 12
We can redefine the
starting point so that
the resulting sequence
of numbers forms an
integer of minimum
magnitude
Colorado School of Mines
Image and Multidimensional Signal Processing
6
Relative Direction
• Can normalize for rotation by looking at the relative
turning direction at each point
• Also the same as taking the first difference of the
chain code
0: Go straight
1: turn left
Colorado School of Mines
3: turn right
Image and Multidimensional Signal Processing
7
Fourier Descriptors
•
Represent the boundary by a sequence of
points (assume clockwise order)
{ (x0,y0), (x1,y1), …, (xK-1,yK-1) }
•
Write each point [x(k),y(k)] as a complex
number
s(k) = x(k) + j y(k)
•
Take 1D Fourier transform of s(k) to get
coefficients a(u)
K 1
a(u )   s(k ) e j 2 uk / K
k 0
• Fourier descriptors are a concise description of
(object) contours
• Can be used for
• Contour processing (filtering, interpolation,
morphing)
• Image analysis (characterizing and recognizing
shapes)
Colorado School of Mines
Image and Multidimensional Signal Processing
8
Fourier Descriptors
• We have Fourier transform coefficients a(u)
K 1
a(u )   s(k ) e j 2 uk / K
What is a(0)?
k 0
• Given coefficients, we can reconstruct boundary
1
s(k ) 
K
K 1
j 2 uk / K
a
(
u
)
e

u 0
• Higher order coefficients can be truncated for a more concise
representation (e.g., low pass filter)
– Other filters: Sharpening, edge extraction, ...
Colorado School of Mines
Image and Multidimensional Signal Processing
9
Matlab Example
• First find boundary points (use
bwtraceboundary)
• Take FFT, and truncate higher order
coefficients
• Take inverse FFT, and plot resulting contour
• (Images from www.lems.brown.edu/~dmc/
Colorado School of Mines
Image and Multidimensional Signal Processing
10
clear all
close all
% Read in a silhouette image
I = imread('Tool088.gif');
imshow(I,[]);
pause
% Find a starting point on the boundary
[rows cols] = find(I~=0);
contour = bwtraceboundary(I, [rows(1), cols(1)], 'N');
% Subsample the boundary points so we have exactly 128, and put them into a
% complex number format (x + jy)
sampleFactor = length(contour)/128;
dist = 1;
for i=1:128
c(i) = contour(round(dist),2) + j*contour(round(dist),1);
dist = dist + sampleFactor;
end
C = fft(c);
% Chop out some of the smaller coefficients (less than umax)
umax = 32;
Capprox = C;
for u=1:128
if u > umax && u < 128-umax
Capprox(u) = 0;
end
end
% Take inverse fft
cApprox = ifft(Capprox);
% Show original boundary and approximated boundary
figure, imshow(imcomplement(bwperim(I)));
hold
on,
plot(cApprox,'r');
Colorado
School
of Mines
Image and Multidimensional Signal Processing
11
Fourier descriptors for recognition
• Need to make Fourier descriptors invariant to common
transformations (translation, changes in scale, rotation)
– Then the contour of a known object can be recognized irrespectively
of its position, size and orientation
• Example application – classify leaves
From: Berlin University of Technology lecture on Fourier descriptors
Colorado School of Mines
Image and Multidimensional Signal Processing
12
Transformations
• Consider these transformations of the contour in the image plane
– Rotation, translation, scaling
– Shifting the starting point of the sequence
• These result in simple transformations of the Fourier transform
–
–
–
–
Rotating the contour is equivalent to multiplying the Fourier transform by ejq
Translating the contour just affects the 0th coefficient
Scaling the contour is equivalent to multiplying the Fourier transform by the same factor
Changing the starting point of the sequence to point k is equivalent to multiplying the
Fourier transform by e-j2ku/N
Colorado School of Mines
Image and Multidimensional Signal Processing
13
Transformations
• Can normalize a Fourier descriptor vector for rotation, translation, scaling,
and starting point
– Set a(0) = 0 => puts centroid at the origin
– Set all a(u)=a(u)/|a(1)| => normalize for scale
•
•
Normalization with respect to rotation and starting point is a little more
complicated, but can be done
A simple way is to just discard the phase information and just take the magnitudes
of the Fourier descriptors (i.e., the spectrum)
–
This isn’t the best way, though, because different shapes can have the same Fourier spectrum
(Information loss, both shapes have the same amplitude spectrum)
Colorado School of Mines
Image and Multidimensional Signal Processing
14
Example
• Rotate and scale one of the images, compare the Fourier descriptors
clear all
close all
I1 = imread('Tool005.gif');
imshow(I1,[]);
Step 1: extract Fourier
descriptors of first image,
normalize for translation
and scale
% Find a starting point on the boundary
[rows cols] = find(I1~=0);
contour = bwtraceboundary(I1, [rows(1), cols(1)], 'N');
% Subsample the boundary points so we have exactly 64, and put them into a
% complex number format (x + jy)
sampleFactor = length(contour)/64;
dist = 1;
for i=1:64
c1(i) = contour(round(dist),2) + j*contour(round(dist),1);
dist = dist + sampleFactor;
end
C1 = fft(c1);
C1(1) = 0;
% Put centroid at the origin
C1 = C1 / abs(C1(2)); % Normalize for scale
Colorado School of Mines
Image and Multidimensional Signal Processing
15
Example
% Make rotated, scaled, and translated version
scale = 1 + (0.5-rand);
I2 = imresize(I1,scale);
ang = 90*(0.5-rand);
I2 = imrotate(I2, ang);
figure, imshow(I2,[]);
Step 2: extract Fourier
descriptors of second
image, normalize for
translation and scale
% Find a starting point on the boundary
[rows cols] = find(I2~=0);
contour = bwtraceboundary(I2, [rows(1), cols(1)], 'N');
% Subsample the boundary points so we have exactly 64, and put them into a
% complex number format (x + jy)
sampleFactor = length(contour)/64;
dist = 1;
for i=1:64
c2(i) = contour(round(dist),2) + j*contour(round(dist),1);
dist = dist + sampleFactor;
end
C2 = fft(c2);
C2(1) = 0;
% Put centroid at the origin
C2 = C2 / abs(C2(2)); % Normalize for scale
Colorado School of Mines
Image and Multidimensional Signal Processing
16
Example
figure, plot(1:64, abs(C1), 1:64, abs(C2));
•
Step 3: compare the Fourier
descriptors for the two images ...
since only the phases are different,
the magnitudes should be the same
Now try comparing the Fourier descriptors for two different images
Colorado School of Mines
Image and Multidimensional Signal Processing
17
Regional Representations
• Describe a segmented region using concise features
– Can use representation for recognition
• We have already looked at describing boundaries using
– Chain codes
– Fourier descriptors
• Now we look at describing the interior
– Statistical moments
– Texture measures
Colorado School of Mines
Image and Multidimensional Signal Processing
18
Statistical Moments
• The (pth,qth) image moment is
m p ,q 

x p y q f ( x, y )
( x , y )R
R
• Note:
– m00 = area
– Centroid is:
 ( x) f ( x, y )
 f ( x, y )
( y ) f ( x, y )


 f ( x, y )
m10
x

m00
m01
y
m00
Colorado School of Mines
Image and Multidimensional Signal Processing
19
Moments
• Central moments (subtract means)
 p,q   x  x   y  y  f ( x, y )
p
x
q
y
• Normalized central moments (divide by area
raised to a power)
 p ,q
 p ,q

0,0 
pq
where  
 1, for p  q  2,3,
2
Colorado School of Mines
Image and Multidimensional Signal Processing
20
Principal Axes
• Major and minor axes are the eigenvectors of
 20 11 

M  
 11 02 
• Eigenvalues l1,l2 are the lengths
 l1 0 

  
 0 l2 
 e11 e12 

E  
 e21 e22 
ê2
ê1
ME  E
• Matlab’s regionprops computes these
Colorado School of Mines
Image and Multidimensional Signal Processing
21
I = imread('Tool088.gif');
[L,n] = bwlabel(I);
stats = regionprops(L, 'all');
bb = stats(1).BoundingBox;
imshow(I, []);
rectangle('Position', bb, 'EdgeColor', 'g');
pause;
See Matlab’s
regionprops
cx = stats(1).Centroid(1);
cy = stats(1).Centroid(2);
major = stats(1).MajorAxisLength/2;
minor = stats(1).MinorAxisLength/2;
ang = -stats(1).Orientation*pi/180;
imshow(I, []);
line([cx-major*cos(ang) cx+major*cos(ang)], ...
[cy-major*sin(ang) cy+major*sin(ang)], 'Color', 'g');
line([cx-minor*cos(ang+pi/2) cx+minor*cos(ang+pi/2)], ...
[cy-minor*sin(ang+pi/2) cy+minor*sin(ang+pi/2)], 'Color', 'y');
pause;
cp = stats(1).ConvexHull;
imshow(I, []);
hold on;
plot(cp(:,1), cp(:,2), 'g');
Colorado School of Mines
Image and Multidimensional Signal Processing
22
Hu’s Invariant Moments
•
•
Combinations of moments are invariant to translation, scale, and rotation
Can be used for recognition
f1 = η20 + η02
f2 = (η20 − η02)2 + (2η11)2
f3 = (η30 − 3η12)2 + (3η21 − η03)2
f4 = (η30 + η12)2 + (η21 + η03)2
f5 = (η30 − 3η12)(η30 + η12)[(η30 + η12)2 − 3(η21 + η03)2] + (3η21 − η03)(η21 +
η03)[3(η30 + η12)2 − (η21 + η03)2]
f6 = (η20 − η02)[(η30 + η12)2 − (η21 + η03)2] + 4η11(η30 + η12)(η21 + η03)
f7 = (3η21 − η03)(η30 + η12)[(η30 + η12)2 − 3(η21 + η03)2] − (η30 − 3η12)(η21 +
η03)[3(η30 + η12)2 − (η21 + η03)2].
Colorado School of Mines
Image and Multidimensional Signal Processing
23
Colorado School of Mines
Image and Multidimensional Signal Processing
24
Texture
• Segment an image based on texture
• Example application: Autonomous road following
Colorado School of Mines
Image and Multidimensional Signal Processing
25
Texture Analysis
• Examples of texture
• Representations: Statistical, structural, spectral
Colorado School of Mines
Image and Multidimensional Signal Processing
Images from the Brodatz
photo album, commonly used
for evaluating texture
recognition algorithms
26
Statistical Descriptions of Texture
• Mean, variance, and higher order moments
• Derived values:
– R-value (is zero for uniform areas, 1 for areas with large variation)
1
R  1
1  2
– Uniformity
L 1
U   p 2 ( zi )
i 0
– Entropy
L 1
e   p( zi ) log 2 p( zi )
i 0
Colorado School of Mines
Image and Multidimensional Signal Processing
27
Colorado School of Mines
Image and Multidimensional Signal Processing
28
Example
• Statistical measures of smooth, coarse, and regular textures
• Notes:
– Third moment indicates skew of histogram to the left or right of the
mean
Colorado School of Mines
Image and Multidimensional Signal Processing
29
Co-occurrence Matrices
• We consider not only the distribution of intensities, but also
their relative positions
• Compute a 2D histogram of pixel pairs (a co-occurrence
matrix), where
– H(a,b) = # occurrences of gray level “a” being at a certain relative
location to gray level “b”
a
r,q
b
Colorado School of Mines
In general, H(a,b; r,q)
Image and Multidimensional Signal Processing
30
Example
• Let relationship = “neighbor immediately to the right”
• gij = # times intensity j is to the right of i
i
j
j
i
Colorado School of Mines
Image and Multidimensional Signal Processing
31
Example
• Compute a co-occurrence matrix for the following image with
gray levels 0,1,2
• Consider a relative position of one pixel to the right and one
pixel down
Co-occurrence matrix G
image
0
0
0
1
2
1
1
0
1
1
2
2
1
0
0
1
1
0
2
0
0
0
1
0
1
Colorado School of Mines
Image and Multidimensional Signal Processing
32
A highly correlated image will yield high values along the diagonal
Colorado School of Mines
Image and Multidimensional Signal Processing
Co-occurrence matrices
(“one pixel to the right”)
33
Interesting Results from Human Vision
• Experiments on what types of texture humans can
discriminate “pre-attentively” (without cognitive processes)
• Julesz conjecture: Textures that have the same first and
second order statistics are indistiguishable
• Statistics:
– First order statistics: measures of single points (mean, variance,
density)
– Second order statistics: measure of pairs of points at different relative
positions (co-occurrence values, co-variance)
Colorado School of Mines
Image and Multidimensional Signal Processing
34
Difference: size & firstorder statistics
Colorado School of Mines
Image and Multidimensional Signal Processing
Difference: orientation &
second-order statistics
35
identical second-order but different third- and higher-order statistics
Colorado School of Mines
Image and Multidimensional Signal Processing
36
Counter Examples
• Some textures with identical 1st and
2nd order statistics can be
discriminated
• These involve conspicuous local
features, called “textons”
• Our visual system can pre-attentively
group these
• This is an example of structural
texture representation
Colorado School of Mines
Image and Multidimensional Signal Processing
37
•
Texton features
–
–
–
Color
Terminator, number of end-of-lines. Ex. Closure, Connectivity
Elongated blobs of different sizes. Ex. Granularity
Colorado School of Mines
Image and Multidimensional Signal Processing
38
Spectral Approaches
• Sum energy in bins corresponding to ranges of
spatial frequencies
• This can detect regular patterns or patterns at certain
orientations
Colorado School of Mines
Image and Multidimensional Signal Processing
39
Example
In this spectra,
the main energy
not associated
with the
background is
along the
horizontal axis,
corresponding to
the strong
vertical edges in
(b)
The periodic
bursts of energy
in both spectra
are due to the
periodic texture
of the coarse
background
Colorado School of Mines
Image and Multidimensional Signal Processing
40

S ( r )   Sq ( r )
q 0
R0
S (q )   Sr (q )
r 1
Colorado School of Mines
Image and Multidimensional Signal Processing
41
f(x,y)
F(u,v)
S(q)
S(r)
Colorado School of Mines
Image and Multidimensional Signal Processing
42
S(q)
Colorado School of Mines
Image and Multidimensional Signal Processing
43
Summary / Questions
• Two methods to represent the boundary of a region
are (1) chain codes, and (2) Fourier descriptors.
• Two methods to represent the region itself are (1)
statistical moments, and (2) texture measures.
• What types of texture measures are there?
Colorado School of Mines
Image and Multidimensional Signal Processing
44