Colorado School of Mines
Image and Multidimensional Signal
Processing
Professor William Hoff
Dept of Electrical Engineering &Computer Science
Colorado School of Mines
Image and Multidimensional Signal Processing
http://inside.mines.edu/~whoff/
Multiresolution Image Processing
Colorado School of Mines
Image and Multidimensional Signal Processing
Multiresolution Processing
• Representing and analyzing signals at more than one
resolution
• One form of this:
– Varying the scale of the LoG or Canny operators
• Important new tools
– Image pyramids
– Wavelets
• Applications
– Feature detection
– Compression
Colorado School of Mines
Image and Multidimensional Signal Processing
3
Observation from human vision
• Analysis is independent of
image scale
– We recognize objects
equally well regardless of
image size
– Recognition speed doesn’t
depend on image size
• typical numbers: ~0.1 sec
for recognition
• ~1 millisec for neuron to fire
• Time for about 100 neuron
firings!
• Obviously not time for linear
propagation across image
Colorado School of Mines
Image and Multidimensional Signal Processing
4
from: http://fias.uni-frankfurt.de/fileadmin/fias/triesch/Object_Recognition_Seminar/SpeedOfRecognition.pdf
Figure: Stimulus specific neural response at different locations according to [Thorpe
and Fabre-Thorpe, 2001]
Thorpe, S. J. and Fabre-Thorpe, M. (2001). Neuroscience. seeking categories in the brain. Science,
291(5502):260–263.
Colorado
School of Mines
Image and Multidimensional Signal Processing
5
• “A lower bound on object
recognition time has been
determined to lie between 80100ms (in monkeys)”
• “An estimate for the processing
speed based on a categorization
approach has been determined
to be around 125ms (in
monkeys)”
• “At 160ms a post-sensory signal
starts, indicating that the lion’s
share of object recognition is
done (in humans)”
from: http://fias.uni-frankfurt.de/fileadmin/fias/triesch/Object_Recognition_Seminar/SpeedOfRecognition.pdf
Colorado School of Mines
Image and Multidimensional Signal Processing
6
Sometimes Objects are Easier to Recognize at Lower Spatial
Frequencies
5
4
3
2
1
0
-1
-2
-3
-4
0
Colorado School of Mines
1
2
3
4
5
6
7
Image and Multidimensional Signal Processing
8
9
10
7
Sometimes Objects are Easier to Recognize at Lower Spatial
Frequencies
5
4
Noisy sine
wave, at
lower
spatial
freq
3
2
1
0
-1
-2
-3
-4
0
Colorado School of Mines
1
2
3
4
5
6
7
Image and Multidimensional Signal Processing
8
9
10
8
Sometimes Objects are
Easier to Recognize at
Lower Spatial
Frequencies
Colorado School of Mines
Image and Multidimensional Signal Processing
9
Multiresolution Pyramids
Construct a series of
subsampled versions of the
same original image
This is called the
“approximation
pyramid”
Colorado School of Mines
Image and Multidimensional Signal Processing
10
Multiresolution Pyramids
• Total size of pyramid is not much larger than that of original image
• Total size for NxN image:
1 1
1  4

Size  N 2 1  1  2    P   N 2
4  3
 4 4
• Advantages:
– Size of objects is smaller in reduced images
– There is a maximum time for linear propagation across the images
– We can process the images in parallel to find the one with the most
appropriate spatial frequency
Colorado School of Mines
Image and Multidimensional Signal Processing
11
Gaussian Pyramids
• Here, we use the Gaussian as a low pass filter before
subsampling
Colorado School of Mines
Image and Multidimensional Signal Processing
12
Downsampling and Upsampling
•
We create a small image from a
large image by downsampling:
– f2↓(n) = f(2n)
•
We create a large image from a
small image by upsampling:
– f2↑(n) = f(n/2) if n even, 0
otherwise
•
The difference between the
approximation and the original
image is the residual
Colorado School of Mines
Image and Multidimensional Signal Processing
13
Definitions
• Let GJ be the image at the Jth level (obtained by low pass filtering and
downsampling)
G1
G2
G0
• Let GJ,K be the expansion of GJ, K times (by upsampling and interpolating)
G1,0
G0,0
Colorado School of Mines
G2,1
G2,0
G3,1
G1,1
Image and Multidimensional Signal Processing
14
Residual Images
• The difference between the J-1th approximation, and the Jth image
-
G0
G1
G2
-
G1,1
=
G2,1
=
G3,1
L0
=
L1
L2
• We can reconstruct the original image from the residuals (and the lowest
2
approximation image)
G0   LJ , J  G3,3
J 0
Colorado School of Mines
Image and Multidimensional Signal Processing
15
Laplacian Pyramid
• The residual images are effectively the result of filtering with
a Laplacian
• Recall that a Laplacian (LoG) is closely approximated by the
difference of two Gaussians (DoG)
• When we convolve a downsampled image with a filter, this is
equivalent to convolving it with a larger filter
Colorado School of Mines
Image and Multidimensional Signal Processing
16
Note histogram of
residual image is
clustered around
zero – it’s easier
to compress
Colorado School of Mines
Image and Multidimensional Signal Processing
17
Matlab example
•
To downsample an image, we apply a Gaussian filter and then subsample:
G0 = double(imread('cameraman.tif'));
% read image
G1 = imfilter(G0,fspecial('gaussian')); % apply Gaussian
G1 = G1(1:2:end, 1:2:end);
% subsample
•
Or, we can use Matlab’s “imresize” function, which applies a low pass filter prior to
subsampling:
G0 = double(imread('cameraman.tif'));
G1 = imresize(G0, 0.5); % apply lowpass, then scale by 0.5
•
To upsample, use Matlab’s “imresize” function with the “nearest neighbor” option
... this will just replicate pixels in 2x2 blocks
G11 = imresize(G1, 2, 'nearest');
% scale up by 2
Also see the “impyramid” function
Colorado School of Mines
Image and Multidimensional Signal Processing
18
G0
G1
G2
G3
=
=
=
=
double(imread('cameraman.tif'));
% read image
imresize(G0, 0.5); % apply lowpass, then scale by 0.5
imresize(G1, 0.5); % apply lowpass, then scale by 0.5
imresize(G2, 0.5); % apply lowpass, then scale by 0.5
figure,
figure,
figure,
figure,
imshow(G0,
imshow(G1,
imshow(G2,
imshow(G3,
[]),
[]),
[]),
[]),
title('G0');
title('G1');
title('G2');
title('G3');
G31 = imresize(G3, 2, 'nearest');
G21 = imresize(G2, 2, 'nearest');
G11 = imresize(G1, 2, 'nearest');
L0 = G0 - G11;
L1 = G1 - G21;
L2 = G2 - G31;
Laplacian
pyramid
% scale up by 2
% scale up by 2
% scale up by 2
% residual image
% residual image
% residual image
figure, imshow(L0, []), title('L0');
figure, imshow(L1, []), title('L1');
figure, imshow(L2, []), title('L2');
L00
L11
L22
G33
=
=
=
=
L0;
imresize(L1, 2, 'nearest');
imresize(L2, 4, 'nearest');
imresize(G3, 8, 'nearest');
G0recon = L00 + L11 + L22 + G33;
figure, imshow(G0recon,[]);
dG = G0 - G0recon;
disp(max(dG(:)));
Colorado School of Mines
% reconstructed image
% Compare original to reconstructed
Image and Multidimensional Signal Processing
19
Applications of Pyramids
• Image Compression
– Histograms of Laplacian images concentrated around zero … can
represent with fewer bits
• Motion Analysis
– Correlation can be use to track objects
– A “coarse-to-fine” strategy keeps cost low
• Texture Analysis
– Estimate local power spectrum within windows of various sizes …
window sizes are kept small
• Image Splining
– Images in a mosaic can be pieced together without discontinuities by
merging their Laplacian pyramids
Colorado School of Mines
Image and Multidimensional Signal Processing
20
Multiresolution Splining
• Image splining: Merging two
images together such that they
have a smooth seam
• Example: image mosaics
• See Burt & Adelson, “A
Multiresolution Spline With
Application to Image Mosaics”,
ACM Transactions on Graphics,
October 1983
Mosaic of
two
images of
San
Francisco
The
images
joined
with a
smooth
spline
Colorado School of Mines
Image and Multidimensional Signal Processing
21
Simple Approach
• Average the two images in a narrow transition zone
• Use a weighting function w(x) which decreases monotonically from left to
right
Iout(x,y) = w(x-x0) Ileft(x,y) + [1-w(x-x0)] Iright(x,y).
x0
w(x)
Ileft
Iright
x
x
Colorado School of Mines
Image and Multidimensional Signal Processing
22
Example
Two synthetic
images of stars
The left image
next to the right
image (with no
splining or
merging)
Colorado School of Mines
Image and Multidimensional Signal Processing
23
Example
Image merged with a
narrow transition zone
(seam is still visible)
Colorado School of Mines
Image merged with a wide
transition zone (features
within zone appear double)
Image and Multidimensional Signal Processing
24
Better Approach
• Spline the images at multiple resolutions
• First construct Laplacian pyramids for each image, LA and LB
• Then create a splined pyramid by averaging pixels along the center line:
• Finally, reconstruct image from the Laplacian pyramid LS
Colorado School of Mines
Image and Multidimensional Signal Processing
25
Example
• Combining two very different images
The left image next to
the right image
Colorado School of Mines
Images merged with the
multiresolution spline
Image and Multidimensional Signal Processing
26
Arbitrary Shaped Regions
Colorado School of Mines
Image and Multidimensional Signal Processing
27
Summary / Questions
• It is useful to represent and analyze images and
signals at more than one resolution.
– The scale of features vary, so features that might go
undetected at one resolution are easy to detect at another.
• Image pyramids and wavelets are two possible tools
to do multiresolution processing.
• What are some applications where multiresolution
processing is useful?
Colorado School of Mines
Image and Multidimensional Signal Processing
28