Colorado School of Mines
Image and Multidimensional Signal
Processing
Professor William Hoff
Department of Electrical Engineering and Computer
Science
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Spatial Filtering
2
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Spatial filtering
• Main idea
– Define a neighborhood of a pixel
– Define an operation on pixels in the neighborhood
– Output pixel value = result of the operation
• Topics
– Linear filtering
• Correlation, convolution
• Normalized cross correlation
– Important examples
• Smoothing filters (box average, Gaussian)
• Sharpening filters (Laplacian, Sobel)
– Non-linear filters
• Order-statistics filters (primarily median filter)
3
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Linear Spatial Filtering
• Create a filter or mask w, size m x n
• Apply to image f, size M x N
• Compute the sum of products of mask coefficients with
corresponding pixels under mask
• Slide mask over image, apply at each point
g ( x, y ) 
m /2
n /2
 
w( s, t ) f ( x  s, y  t )
s  m /2 t  n /2
 w( x, y )  f ( x, y )
Also called “crosscorrelation”
• It is a linear operation
w( x, y )   a f ( x, y )   a w( x, y )  f ( x, y )
w( x, y )   f ( x, y )  g ( x, y )   w( x, y )  f ( x, y )  w( x, y )  g ( x, y )
4
Colorado School of Mines
Department of Electrical Engineering and Computer Science
g ( x, y) 
m/ 2
n/2
  w(s, t ) f ( x  s, y  t )
s  m / 2 t  n / 2
Colorado School of Mines
Department of Electrical Engineering and Computer Science
5
Example
•
•
3x3 averaging (“box”) filters
Do manual calculation on a
corner
g ( x, y)  w( x, y)  f ( x, y)
f ( x, y)
0
0
0
0
0
0
0
..
0
0
0
0
0
0
0
..
0
0
0
0
0
0
0
..
0
0
0
9
9
9
9
..
0
0
0
9
9
9
9
..
0
0
0
9
9
9
9
..
0
0
0
9
9
9
9
..
:
:
:
:
:
:
:
..
6
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Example
•
•
3x3 averaging filters
Manual calculation on
corner
7
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Matlab Examples
clear all
close all
% Synthetic image of a white square
I = zeros(200,200);
I(50:150, 50:150) = 1;
imshow(I,[]);
% Apply 3x3 box filter
w = [ 1 1 1; 1 1 1; 1 1 1]/9;
I2 = imfilter(I, w);
imtool(I2,[]);
• fspecial
– useful to create filters
• imfilter
– does cross correlation
% Apply 15x15 box filter
w = fspecial('average', [15 15]);
I3 = imfilter(I, w);
imtool(I3,[]);
% Show that averaging reduces noise
I = I + randn(200,200)*0.5;
imshow(I,[]);
w = fspecial('average', [5 5]);
I4 = imfilter(I, w);
figure, imshow(I4,[]);
8
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Sharpening Spatial Filters – First Derivative
• First derivative
f ( x, y)
f ( x   , y )  f ( x, y )
 lim
 0
x

– In discrete form (can also do central difference)
f ( x, y)
f
 f ( x  1, y)  f ( x, y)
  f ( x  1)  f ( x  1)  / 2
x
x
– As a filter
-1
+1
-.5
0
+.5
• Sobel operators – combine smoothing with derivative
dx filter
-1
0
1
-2
0
2
-1
0
1
dy filter
-1
-2
-1
0
0
0
1
2
1
9
Colorado School of Mines
Department of Electrical Engineering and Computer Science
w( x, y)
-1
0
1
-2
0
2
-1
0
1
Example
g ( x, y)  w( x, y)  f ( x, y)
f ( x, y)
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
0
0
0
0
1
1
1
1
0
0
0
0
1
1
1
1
0
0
0
0
1
1
1
1
0
0
0
0
1
1
1
1
0
0
10
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Example
w( x, y)
-1
0
1
-2
0
2
-1
0
1
g ( x, y)  w( x, y)  f ( x, y)
f ( x, y)
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
0
0
0
0
1
1
1
1
0
0
0
0
1
1
1
1
0
0
0
0
1
1
1
1
0
0
0
0
1
1
1
1
0
0
11
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Gradient
• Compute gradient
components using first
derivative operators
f x 
f  


f

y


• Gradient magnitude
indicates the location of
edges in the image
 f  2  f  2 
f       
 x   y  
• Gradient direction shows
direction of edge
1/ 2
 f y 
  tan 


f

x


1
• We can use the Sobel
operators for df/dx, df/dy
12
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Matlab example
clear all
close all
I = imread('moon.tif');
imshow(I,[]);
% Create Sobel filters
hy = [
% can also do hy = fspecial('sobel');
-1 -2 -1;
0
0
0;
+1 +2 +1];
hx = hy';
Ix = imfilter(I, hx);
Iy = imfilter(I, hy);
figure;
subplot(1,2,1), imshow(Ix,[]);
subplot(1,2,2), imshow(Iy,[]);
• Note - the result is not quite
correct
• We should first convert the
input image to type “double”
before applying “imfilter”
(why?)
13
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Matlab example (continued)
% Compute gradient magnitude (note the use of ".")
Ig = (Ix.^2 + Iy.^2) .^ 0.5;
figure, imshow(Ig, []);
>> help atan2
atan2 Four quadrant inverse tangent.
atan2(Y,X) is the four quadrant arctangent of the real parts of the
elements of X and Y. -pi <= atan2(Y,X) <= pi.
See also atan.
Reference page in Help browser
doc atan2
% Compute gradient direction
Iang = atan2(Iy,Ix);
figure, imshow(Iang, []);
% Only display angles where gradient magnitude is large
Mask = (Ig > 100);
figure, imshow(double(Mask) .* Iang, []);
colormap jet
14
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Sharpening Spatial Filters – 2nd derivative
• Second derivative
 2 f ( x, y)   f ( x, y) 
 
  f ( x  1, y)  2 f ( x, y)  f ( x  1, y)
x 2
x  x 
– As a filter
+1
-2
+1
• Two dimensional – the Laplacian
2 f 2 f
 f  2  2
x
y
2
– As a filter
0
+1
0
+1
-4
+1
0
+1
0
2
x 2
2
x 2
=
0
0
0
+1
-2
+1
0
0
0
+
0
+1
0
0
-2
0
0
+1
0
15
Colorado School of Mines
Department of Electrical Engineering and Computer Science
High Boost Filtering
• We combine original image with results of Laplacian – result is
the original image with enhanced detail
g ( x, y)  f ( x, y)  2 f
– Also called unsharp masking
0
-1
0
-1
5
-1
0
-1
0
• In general
g ( x, y )  Af ( x, y )  f ( x, y )
16
Colorado School of Mines
Department of Electrical Engineering and Computer Science
17
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Vector representation of correlation
m /2
n /2
 
• Correlation is a sum of
products of
corresponding terms
c ( x, y ) 
• We can think of
correlation as a dot
product of vectors w
and f
c  w1 f1  w2 f 2 
w( s, t ) f ( x  s, y  t )
s  m /2 t  n /2
 w( x, y )  f ( x, y )
 wmn f mn
mn
  wk f k  wT f
k 1
• If images w and f are
similar, their vectors
are aligned
18
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Correlation can do matching
• Let w(x,y) be a template subimage
• Try to find an instance of w in
image f(x,y)
• The correlation score c(x,y) is
high where w matches f
m /2
c ( x, y ) 
n /2
 
w( s, t ) f ( x  s, y  t )
s  m /2 t  n /2
 w( x, y )  f ( x, y )
19
Colorado School of Mines
Department of Electrical Engineering and Computer Science
20
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Template Matching (continued)
• Since f is not constant everywhere, we need to normalize
wT f
c
w f
where w  w12  w22  w32    wN2
• The result is between 0..1 (for positive valued images)
• We can get better precision by subtracting off means
c ( x, y ) 
[w(s, t )  w][ f ( x  s, y  t )  f ]
s ,t

2
[
w
(
s
,
t
)

w
]

 s ,t
1/2

[
f
(
x

s
,
y

t
)

f
]


s ,t

2
• This result is between -1 .. +1
• This is the
“normalized
correlation
coefficient”
21
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Normalized Cross-correlation
•
Normalized cross-correlation (values range from -1 .. +1)
c ( x, y ) 
•
[w(s, t )  w][ f ( x  s, y  t )  f ( x, y)]
s ,t

2
 [ w( s, t )  w]
 s ,t
1/2

[
f
(
x

s
,
y

t
)

f
(
x
,
y
)]


s ,t

2
See pg. 870 in
the Gonzalez &
Woods textbook
where
– w(s,t) is a subimage, size mxn
1
f
(
x
,
y
)

 f ( x  s, y  t )
– f(x,y) is a large image, size MxN
mn s ,t
– sums are taken over s=1..m, t=1..n
– f ( x, y ) is the local mean of f, in a mxn neighborhood around (x,y)
w(s,t)
f(x,y)
c(x,y)
Correlation
scores
Size mxn
Size MxN
Colorado School of Mines
Size MxN
Department of Electrical Engineering and Computer Science
22
Matlab Example
•
Crop a small subimage from left image, find its match in the right image
I1 = imread('test000.jpg');
T = imcrop(I1, [66, 213, 86, 233]);
I2 = imread('test012.jpg');
C = normxcorr2(T, I2);
% The scores are in an image that is slightly bigger than the original
% image ... it is expanded by half the size of the template in all
% directions. So we will crop out the center portion.
Csub = imcrop(C, [(size(T,2)-1)/2+1 (size(T,1)-1)/2+1 size(I2,2)-1 size(I2,1)-1]);
cmax = max(Csub(:));
[y x] = find(Csub==cmax);
23
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Doing cross correlation in Matlab without normxcorr2
• The easiest way is to implement the equation directly, using “for”
loops
• You scan through the image ... at every pixel (x,y):
– Extract the subimage fs(x,y) from the image f surrounding x,y
– Subtract off the local image mean value at (x,y)
– Compute the sum of the product of (fs-fmean) and (w-mean2(w); this is the
numerator
– The denominator can be similarly computed
– Dividing the two gives you the value of the normalized cross correlation at pixel
(x,y) in the ouput
• Note - there are faster ways to do this, using Matlab’s ability to
operate on whole matrices
24
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Doing cross correlation in Matlab without normxcorr2
(continued)
•
Numerator
N  [w(s, t )  w][ f ( x  s, y  t )  f ( x, y)]
–
s ,t
where
w is the mean of all pixels in w; it can be computed once. It is a scalar.
f ( x, y ) is the local mean of pixels in f, in a mxn window centered at (x,y).
This can be represented as an image of size MxN.
•
Then
[w(s, t )  w][ f ( x  s, y  t )  f ( x, y)] 
s ,t
  [ w( s, t )  w] f ( x  s, y  t )  f ( x, y ) [ w( s, t )  w]
s ,t
•
s ,t
The first term is the correlation of (w-wmean) and f. The second term is zero
because
[w(s, t )  w]   w(s, t )   w  mnw  mnw  0
s ,t
•
s ,t
s ,t
So in Matlab:
N = imfilter(f,w-mean2(w));
Colorado School of Mines
25
Department of Electrical Engineering and Computer Science
Doing cross correlation in Matlab without normxcorr2
(continued)
•
Denominator
•
In Matlab, you can compute fmean at every pixel by filtering with a mxn rectangle of
1’s (and dividing by mxn):
1/2

2
D   [ w( s, t )  w]
 s ,t
2
[
f
(
x

s
,
y

t
)

f
(
x
,
y
)]


s ,t

[m n] = size(w);
fmean = imfilter(f, ones(m,n))/(m*n);
•
The first term is a scalar, and can be computed once. We rewrite the second term:
 [ f ( x  s, y  t )  f ( x, y)]
2
s ,t
  f ( x  s , y  t ) 2  2 f ( x, y )  f ( x  s , y  t )   f ( x , y ) 2
s ,t
s ,t
s ,t
  f ( x  s, y  t ) 2  2(mn) f ( x, y ) 2  (mn) f ( x, y ) 2
s ,t
  f ( x  s, y  t ) 2  (mn) f ( x, y ) 2
s ,t
Colorado School of Mines
This can be computed by filtering again with a mxn rectangle of 1’s; i.e.,
imfilter(f.^2, ones(m,n))
26
Department of Electrical Engineering and Computer Science
Correlation and Convolution
• Correlation
m /2
n /2
 
g ( x, y ) 
w( s, t ) f ( x  s, y  t )
s  m /2 t  n /2
 w( x, y )  f ( x, y )
• Convolution
m /2
g ( x, y ) 
n /2
 
w( s, t ) f ( x  s, y  t )
s  m /2 t  n /2
 w( x, y )  f ( x, y )
• Convolution is very
similar to
correlation, except
that we flip one of
the functions
• Also a linear
operation
– Note that
m /2
n /2
 
s  m /2 t  n /2
w( s, t ) f ( x  s, y  t ) 
m /2
n /2
 
w(s, t ) f ( x  s, y  t )
s  m /2 t  n /2
– So to implement convolution we can reflect w(x,y) and then do correlation
27
Colorado School of Mines
Department of Electrical Engineering and Computer Science
1D Examples
Figure 3.29
28
Colorado School of Mines
Department of Electrical Engineering and Computer Science
1D Examples
29
Colorado School of Mines
Department of Electrical Engineering and Computer Science
2D Examples
30
Colorado School of Mines
Department of Electrical Engineering and Computer Science
2D Examples
31
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Properties
• Convolution is
– Commutative
f*g=g*f
– Associative
f * (g * h) = (f * g) * h
– Distributive
f * (g + h) = f * g + f * h
• Correlation is distributive but not commutative or associative
• Convolution of a filter h(x,y) with an impulse d(x,y) yields h(x,y)
32
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Gaussian – an important smoothing filter
• 1D
1
Notes:
0.9
0.8
x2
 is the standard
deviation
0.7
 2
1
h( x ) 
e 2
2
0.6
0.5
Values should sum to 1
0.4
Mask size should be at
least ±3
0.3
0.2
0.1
0
-10
• 2D
h ( x, y ) 

1
2
2
e
-8
-6
-4
-2
0
2
4
6
8
10
x2  y 2
2 2
0.01
0.008
0.006
0.004
0.002
0
40
30
40
30
20
20
10
Colorado School of Mines
10
0
0
Department of Electrical Engineering and Computer Science
33
Convolving with two Gaussians
• You can convolve a function with a Gaussian, then convolve again
with a Gaussian. The effect is the same as convolving with a single,
larger Gaussian
• Why? Since convolution is associative
g1   g2  f    g1  g2   f
• and if
x2
x2
 2
 2
1
1
21
g1 
e
, g2 
e 2 2
2 1
2 2
• then
g1  g 2 

1

2   
2
1
2
2

e

x2
2 12  22

i.e., it is a Gaussian
with standard
deviation
 new  12   22
34
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Proof
• You can prove this from the definition of convolution

f g 

f (t ) g (t  x) dt

• And use the idea that
 t 2  t  x 
2
e e
e
2t 2  2tx  x2
e
2 t  x /2
2
e
 x2 /2
where the last term does not depend on t and the first one is just
another Gaussian, but centered around a different point.
35
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Interesting fact
• If you repeatedly convolve an image N times with any
filter, the effect (for large N) is that of convolving with a
Gaussian
• This comes from the Central Limit Theorem: “The sum
of N mutually independent random variables with zero
means and finite variances tends toward the normal
probability distribution”
• And the fact that the pdf of the sum of two independent
random variables is the convolution of their pdf’s
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Separable kernels (filter masks)
• If a 2D correlation (or convolution) kernel can be separated into two
1D kernels, the operation is much faster
– If
h( x, y )  h1 ( x)h2 ( y )
– then
h( x, y)  f ( x, y)  h1 ( x)   h2 ( y)  f ( x, y) 
• Namely, do a 1D correlation with h2(y) along the individual columns
of the input image, then do a 1D correlation with h1(x) along the
rows of the result from the previous step
• If h(x,y) is nxn, and f(x,y) is NxN
– Cost of a 2D correlation?
– Cost of two 1D correlations?
37
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Separable kernels (filter masks)
• If a 2D correlation (or convolution) kernel can be separated into two
1D kernels, the operation is much faster
– If
h( x, y )  h1 ( x)h2 ( y )
– then
h( x, y)  f ( x, y)  h1 ( x)   h2 ( y)  f ( x, y) 
Prove this
(HW2)
• Namely, do a 1D correlation with h2(y) along the individual columns
of the input image, then do a 1D correlation with h1(x) along the
rows of the result from the previous step
• If h(x,y) is nxn, and f(x,y) is NxN
– Cost of a 2D correlation?
– Cost of two 1D correlations?
38
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Nonlinear Filters
Matlab
functions
medfilt1,
medfilt2,
ordfilt2
• Median filter - a non-linear smoothing filter
• Steps:
– Order (sort) pixel values within neighborhood
– Replace center with some value based on the ordering (i.e., median)
• Advantages
– Especially good for reducing impulse, or shot-and-pepper noise
– Doesn’t blur sharp edges in the image
12
12
10
10
8
8
6
6
4
4
2
2
0
-2
0
0
5
10
15
20
25
30
1D noisy step edge
Colorado School of Mines
35
40
45
50
0
5
10
15
20
25
30
35
40
median filtered, filter width=7
Department of Electrical Engineering and Computer Science
45
50
39
Example on ramp edge
25
 x x  1: 4, 6 :10
f ( x)  
x5
20
20
15
10
5
0
-5
0
5
10
15
40
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Example on ramp edge
25
 x x  1: 4, 6 :10
f ( x)  
x5
20
20
15
10
5
0
-5
0
5
10
15
41
Colorado School of Mines
Department of Electrical Engineering and Computer Science
42
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Examples - Matlab
• Read image “eight.tif”
– Add noise: imnoise
– Compare imfilter, medfilt2
clear all
close all
I = imread('eight.tif');
imshow(I,[]);
Inoisy = imnoise(I, 'salt & pepper', 0.05);
imtool(Inoisy,[]);
I1 = imfilter(Inoisy, fspecial('average'));
imshow(I1, []);
I2 = medfilt2(Inoisy);
imshow(I2, []);
43
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Summary
• A spatial filter operates on a neighborhood of a pixel.
The output pixel value is the result of the operation.
• A “linear” spatial filter performs a sum of products of the
neighborhood values with the corresponding filter values.
• The median filter is an example of a “non-linear” filter.
• Filters can be used for smoothing and sharpening.
44
Colorado School of Mines
Department of Electrical Engineering and Computer Science
Questions
• Is the 2D Gaussian filter separable? Why or
why not?
• What is the difference between convolution and
correlation?
• Why might the median filter be preferable for
some situations?
45
Colorado School of Mines
Department of Electrical Engineering and Computer Science