Image Processing
For Engineers
by Andrew E. Yagle and Fawwaz T. Ulaby
www.dbooks.org
Book companion website:
: ip.eecs.umich.edu
“book” — 2016/3/15 — 6:35 — page iii — #3
IMAGE
PROCESSING
FOR ENGINEERS
Andrew E. Yagle
The University of Michigan
Fawwaz T. Ulaby
The University of Michigan
www.dbooks.org
Copyright  2018 Andrew E. Yagle and Fawwaz T. Ulaby
This book is published by Michigan Publishing under an agreement with the authors.
It is made available free of charge in electronic form to any student or instructor
interested in the subject matter.
Published in the United States of America by
Michigan Publishing
Manufactured in the United States of America
ISBN 978-1-60785-488-3 (hardcover)
ISBN 978-1-60785-489-0 (electronic)
“book” — 2016/3/15 — 6:35 — page v — #5
This book is dedicated to the memories of
Professor Raymond A. Yagle and Mrs. Anne Yagle
www.dbooks.org
Contents
Chapter 4
Preface
Chapter 1
1-1
1-2
1-3
1-4
1-5
1-6
Optical Imagers
Radar Imagers
X-Ray Computed Tomography (CT)
Magnetic Resonance Imaging
Ultrasound Imager
Coming Attractions
Chapter 2
2-1
2-2
2-3
2-4
2-5
2-6
2-7
2-8
2-9
2-10
3-5
3-6
3-7
3-8
3-9
Review of 1-D Signals and Systems
Review of 1-D Continuous-Time Signals
Review of 1-D Continuous-Time Systems
1-D Fourier Transforms
The Sampling Theorem
Review of 1-D Discrete-Time Signals and
Systems
Discrete-Time Fourier Transform (DTFT)
Discrete Fourier Transform (DFT)
Fast Fourier Transform (FFT)
Deconvolution Using the DFT
Computation of Continuous-Time Fourier
Transform (CTFT) Using the DFT
Chapter 3
3-1
3-2
3-3
3-4
Imaging Sensors
2-D Images and Systems
1
3
13
18
19
23
27
38
41
43
47
53
59
66
70
76
80
82
89
Displaying Images
90
2-D Continuous-Space Images
91
Continuous-Space Systems
93
2-D Continuous-Space Fourier Transform
(CSFT)
94
2-D Sampling Theorem
107
2-D Discrete Space
113
2-D Discrete-Space Fourier Transform (DSFT) 118
2-D Discrete Fourier Transform (2-D DFT)
119
Computation of the 2-D DFT Using MATLAB 121
4-1
4-2
4-3
4-4
4-5
4-6
4-7
4-8
4-9
4-10
7-3
7-4
Deterministic Approach to Image
Restoration
Direct and Inverse Problems
Denoising by Lowpass Filtering
Notch Filtering
Image Deconvolution
Median Filtering
Motion-Blur Deconvolution
Chapter 7
7-1
7-2
Image Enhancement
159
Pixel-Value Transformation
160
Unsharp Masking
163
Histogram Equalization
167
Edge Detection
171
Summary of Image Enhancement Techniques 176
Chapter 6
6-1
6-2
6-3
6-4
6-5
6-6
128
Interpolation Using Sinc Functions
129
Upsampling and Downsampling Modalities
130
Upsampling and Interpolation
133
Implementation of Upsampling Using 2-D DFT
in MATLAB
137
Downsampling
140
Antialias Lowpass Filtering
141
B-Splines Interpolation
143
2-D Spline Interpolation
149
Comparison of 2-D Interpolation Methods
150
Examples of Image Interpolation Applications 152
Chapter 5
5-1
5-2
5-3
5-4
5-5
Image Interpolation
Wavelets and Compressed Sensing
Tree-Structured Filter Banks
Expansion of Signals in Orthogonal Basis
Functions
Cyclic Convolution
Haar Wavelet Transform
180
181
183
188
191
194
195
202
203
206
209
213
7-5
7-6
7-7
7-8
7-9
7-10
7-11
7-12
Discrete-Time Wavelet Transforms
Sparsification Using Wavelets of PiecewisePolynomial Signals
2-D Wavelet Transform
Denoising by Thresholding and Shrinking
Compressed Sensing
Computing Solutions to Underdetermined
Equations
Landweber Algorithm
Compressed Sensing Examples
Chapter 8
8-1
8-2
8-3
8-4
8-5
8-6
8-7
8-8
8-9
8-10
8-11
Introduction to Probability
Conditional Probability
Random Variables
Effects of Shifts on Pdfs and Pmfs
Joint Pdfs and Pmfs
Functions of Random Variables
Random Vectors
Gaussian Random Vectors
Random Processes
LTI Filtering of Random Processes
Random Fields
Chapter 9
9-1
9-2
9-3
9-4
9-5
9-6
9-7
9-8
9-9
9-10
9-11
Random Variables, Processes, and
Fields
218
Chapter 10
223
228
232
236
10-1
10-2
10-3
10-4
238
241
242
254
255
259
261
263
265
269
272
275
278
282
285
Stochastic Denoising and
Deconvolution
291
Estimation Methods
Coin-Flip Experiment
1-D Estimation Examples
Least-Squares Estimation
Deterministic versus Stochastic Wiener
Filtering
2-D Estimation
Spectral Estimation
1-D Fractals
2-D Fractals
Markov Random Fields
Application of MRF to Image Segmentation
292
298
300
303
Color Image Processing
Color Systems
Histogram Equalization and Edge Detection
Color-Image Deblurring
Denoising Color Images
Chapter 11
334
335
340
343
346
Image Recognition
353
11-1 Image Classification by Correlation
11-2 Classification by MLE
11-3 Classification by MAP
11-4 Classification of Spatially Shifted Images
11-5 Classification of Spatially Scaled Images
11-6 Classification of Rotated Images
11-7 Color Image Classification
11-8 Unsupervised Learning and Classification
11-9 Unsupervised Learning Examples
11-10 K-Means Clustering Algorithm
354
357
358
360
361
366
367
373
377
380
Chapter 12
Supervised Learning and
Classification
389
Overview of Neural Networks
Training Neural Networks
Derivation of Backpropagation
Neural Network Training Examples
390
396
403
404
12-1
12-2
12-3
12-4
Appendix A
Review of Complex Numbers
411
Appendix B
MATLAB® and MathScript
415
Index
307
309
313
314
320
322
327
www.dbooks.org
421
Preface
“A picture is worth a thousand words.”
This is an image processing textbook with a difference. Instead
of just a picture gallery of before-and-after images, we provide
(on the accompanying website) MATLAB programs (.m files)
and images (.mat files) for each of the examples. These allow
the reader to experiment with various parameters, such as noise
strength, and see their effect on the image processing procedure.
We also provide general MATLAB programs, and Javascript
versions of them, for many of the image processing procedures
presented in this book. We believe studying image processing
without actually performing it is like studying cooking without
turning on an oven.
Designed for a course on image processing (IP) aimed at both
graduate students as well as undergraduates in their senior year,
in any field of engineering, this book starts with an overview
in Chapter 1 of how imaging sensors—from cameras to radars
to MRIs and CAT—form images, and then proceeds to cover a
wide array of image processing topics. The IP topics include:
image interpolation, magnification, thumbnails, and sharpening,
edge detection, noise filtering, de-blurring of blurred images,
supervised and unsupervised learning, and image segmentation,
among many others. As a prelude to the chapters focused on
image processing (Chapters 3–12), the book offers in Chapter 2
a review of 1-D signals and systems, borrowed from our 2018
book Signals and Systems: Theory and Applications, by Ulaby
and Yagle.
Book highlights:
• A section in Chapter 1 called “Coming Attractions,” offering a sampling of the image processing applications
covered in the book.
• MATLAB programs and images (.m and .mat files) on the
book’s website for all examples and problems. All of these
also run on NI LabVIEW Mathscript.
• Coverage of standard image processing techniques, including upsampling and downsampling, rotation and scaling,
histogram equalization, lowpass filtering, classification,
edge detection, and an introduction to color image processing.
• An introduction to discrete wavelets, and application of
wavelet-based denoising algorithms using thresholding and
shrinkage, including examples and problems.
• An introduction to compressed sensing, including examples and problems.
• An introduction to Markov random fields and the ICM
algorithm.
• An introduction to supervised and unsupervised learning
and neural networks.
• Coverage of both deterministic (least-squares) and stochastic (a priori power spectral density) image deconvolution,
and how the latter gives better results.
• Interpolation using B-splines.
• A review of probability, random processes, and MLE,
MAP, and LS estimation.
Book Companion Website: ip.eecs.umich.edu
The book website is a rich resource developed to extend the
educational experience of the student beyond the material covered in the textbook. It contains MATLAB programs, standard
images to which the reader can apply the image processing tools
outlined in the book, and Javascript image processing modules
with selectable parameters. It also contains solutions to “concept
questions” and “exercises,” and, for instructors, solutions to
homework problems.
Acknowledgments: Mr. Richard Carnes—our friend, our piano performance teacher, and our LATEX super compositor—
deserves singular thanks and praise for the execution of this
book. We are truly indebted to him for his meticulous care and
attention. We also thank Ms. Rose Anderson for the elegant design of the cover and for creating the printable Adobe InDesign
version of the book.
A NDREW YAGLE AND FAWWAZ U LABY, 2018
Chapter
1
1
Imaging Sensors
Contents
1-1
1-2
1-3
1-4
1-5
1-6
Overview, 2
Optical Imagers, 3
Radar Imagers, 13
X-Ray Computed Tomography (CT), 18
Magnetic Resonance Imaging, 19
Ultrasound Imager, 12
Coming Attractions, 27
Problems, 36
Lens of diameter D
y
Source s
Objectives
Learn about:
■ How a digital camera forms an image, and what
determines the angular resolution of the camera.
■ How a thermal infrared imager records the distribution of energy emitted by the scene.
■ How a radar can create images with very high
resolution from satellite altitudes.
■ How an X-ray system uses computed tomography
(CT) to generate 3-D images.
■ How magnetic resonance is used to generate 3-D
MRI images.
■ How an ultrasound instrument generates an image of
acoustic reflectivity, much like an imaging radar.
■ The history of image processing.
■ The types of image-processing operations examined
in detail in follow-up chapters.
www.dbooks.org
θ
Object plane
x
Image plane
Image processing has applications in medicine,
robotics, human-computer interface, and manufacturing, among many others. This book is about the
mathematical methods and computational
algorithms used in processing an image from its raw
form—whether generated by a digital camera, an
ultrasound monitor, a high-resolution radar, or any
other 2-D imaging system—into an improved form
suitable for the intended application. As a prelude,
this chapter provides overviews of the image
formation processes associated with several sensors.
2
CHAPTER 1 IMAGING SENSORS
Overview
In today’s world we use two-dimensional (2-D) images generated by a variety of different sensors, from optical cameras
and ultrasound monitors to high-resolution radars and others.
A camera uses light rays and lenses to form an image of the
brightness distribution across the scene observed by the lens,
ultrasound imagers use sound waves and transducers to measure
the reflectivity of the scene or medium exposed to the sound
waves, and radar uses antennas to illuminate a scene with
microwaves and then detect the fraction of energy scattered
back toward the radar. The three image formation processes
are markedly different, yet their output product is similar: a
2-D analog or digital image. An X-ray computed tomography
(CT) scanner measures the attenuation of X-rays along many
directions through a 3-D object, such as a human head, and
then processes the data to generate one or more 2-D cross-
Image
formation
Raw
image
sectional images (called slice) of the attenuation for specific
areas of interest. A rather different process occurs in magnetic
resonance imaging (MRI).
For these and many other sensing processes, the formation of
the 2-D image is only the first step. As depicted in Fig. 1-1, we
call such an image the raw image, because often we subject the
raw image to a sequence of image processing steps designed
to transform the image into a product more suitable for the
intended application (Table 1-1). These steps may serve to filter
out (most of) the noise that may have accompanied the (desired)
signal in the image detection process, rotate or interpolate the
image if called for by the intended application, enhance certain
image features to accentuate recognition of objects of interest,
or compress the number of pixels representing the image so as
to reduce data storage (number of bits), as well as other related
actions.
Image
processing
Sensor
Improved
image
Image display
Image storage/
transmission
Image Formation and Processing
Image analysis
Figure 1-1 After an image is formed by a sensor, image processing tools are applied for many purposes, including changing its scale and
orientation, improving its information content, or reducing its digital size.
1-1 OPTICAL IMAGERS
3
processing techniques covered in future chapters can accomplish.
Table 1-1 Examples of image-processing applications.
• Medicine (radiological diagnoses, microscopy)
• Defense (radar, sonar, infrared, satellites, etc.)
1-1 Optical Imagers
• Human-computer interfaces (face/fingerprint “recognition”
for security, character recognition)
Even though the prime objective of this book is to examine
the various image processing techniques commonly applied to
a raw image (Fig. 1-1) to transform it into an improved image
of specific utility, it will prove helpful to the reader to have a
fundamental understanding of the image formation process that
led to the raw image in the first place. We consider five types
of imaging sensors in this introductory chapter, of which four
are electromagnetic (EM), and the fifth is acoustic. Figure 1-2
depicts the EM spectrum, extending from the gamma-ray region
to the radio region. Optical imagers encompass imaging systems
that operate in the visible, ultraviolet, and infrared segments
of the EM spectrum. In the present section we feature digital
cameras, which record reflected energy in the visible part of the
spectrum, and infrared imagers, which sense thermal radiation
self-emitted by the observed scene.
• Robotics / machine vision (e.g., “intelligent” vehicles)
• Compression for storage, transmission from space probes,
etc.
• Entertainment industry
• Manufacturing (e.g., part inspection)
◮ This book covers the mathematical bases and computational techniques used to realize these image-processing
transformations. ◭
To set the stage for the material covered in future chapters,
this introductory chapter introduces the reader to overviews of
the image formation processes associated with several different
types of sensors. Each of Sections 1-1 through 1-5 sketches the
fundamental physical principles and the terminology commonly
used in connection with that particular imaging sensor. The
chapter concludes with Section 1-6, which provides visual
demonstrations of the various image operations that the image
X-ray imagers
Gamma-ray
10−12
1-1.1 Digital Cameras
In June of 2000, Samsung introduced the first mobile phone
with a built-in digital camera. Since then, cameras have become
integral to most mobile phones, computer tablets, and laptop
computers. And even though cameras may vary widely in terms
of their capabilities, they all share the same imaging process. As
Optical imagers
Radar
X-ray Ultraviolet Visible Infrared
10−10
10−8 0.5×10 −6 10−5
Microwave
10−2
MRI
100
102
Radio
103
Wavelength (m)
3 × 1020
3 × 1018
3 × 1016 3 × 1014
3 × 1012
3 × 108
3 × 104
Frequency (Hz)
10,000,000 K
10,000 K
100 K
1K
Temperature
of object
when
radiationisismost
mostintense
intense
Temperature
of object
when
radiation
Figure 1-2 Electromagnetic spectrum.
www.dbooks.org
4
CHAPTER 1 IMAGING SENSORS
an optical imager, the camera records the spatial distribution of
visible light reflected by a scene due to illumination by the sun
or an artificial light source. In the simplified diagram shown in
Fig. 1-3, the converging lens of the camera serves to focus the
light reflected by the apple to form a sharp image in the image
plane of the camera. To “focus” the image, it is necessary to
adjust the location of the lens so as to satisfy the lens law
1
1
1
+ =
do di
f
Digital
storage
device
Amplifiers
(lens law),
(1.1)
where do and di are the distances between the lens and the object
and image planes, respectively, and f is the focal length of the
lens.
In a traditional analog camera, the image is captured by a
film containing light-sensitive silver halide crystals. The crystals
undergo a chemical change—and an associated darkening—in
proportion to the amount of light absorbed by the crystals.
Modern cameras use arrays of charge-coupled devices
(CCDs) or active pixel sensors (APSs), placed in the image
plane, to capture the image and then transfer the intensity
readings to a data storage device (Fig. 1-4). The CCD relies
on charge transfer in response to incident photons, whereas an
APS uses a photodetector and an amplifier. CCD arrays were
the sensors of choice in the 1970–2000 era, but they have been
Figure 1-4 An active pixel sensor uses a 2-D array of
photodetectors, usually made of CMOS, to detect incident light
in the red, green, and blue bands.
replaced with APS arrays over the past 20 years (Moynihan,
2015) because APS arrays consume less power to operate and
are less expensive to fabricate (but they are more susceptible to
noise than CCDs).
A photodetector uses CMOS (complementary metal-oxide
semiconductor) technology to convert incident photons into an
output voltage. Because both CCD and CMOS are sensitive to
the entire visible spectrum, from about 0.4 µ m to 0.7 µ m, as
well as part of the near-infrared (NIR) spectrum from 0.7 µ m to
Converging lens
of focal length f
2-D detector array
do
di
Image plane
Object plane
Figure 1-3 Camera imaging system.
1-1 OPTICAL IMAGERS
5
Ii (x, y; λ ): continuous intensity image in the image plane of the
camera, with (x, y) denoting the coordinates of the image plane.
Relative spectral sensitivities
1.2
1.0
Blue
Green
V [n1 , m1 ] = {Vred [n1 , m1 ], Vgreen [n1 , m1 ], Vblue [n1 , m1 ]} distribution: discrete 2-D array of the voltage outputs of the CCD or
photo detector array.
0.8
Red
0.6
B[n2 , m2 ] = {Bred [n2 , m2 ], Bgreen [n2 , m2 ], Bblue [n2 , m2 ]} distribution: discrete 2-D array of the brightness across the LCD array.
0.4
0.2
0
0.40 0.45 0.50 0.55 0.60 0.65 0.70
Wavelength λ ( μm)
Figure 1-5 Spectral sensitivity plots for photodetectors.
(Courtesy Nikon Corporation.)
1 µ m, it is necessary to use a filter to block the IR spectrum and
to place red (R), green (G), or blue (B) filters over each pixel
so as to separate the visible spectrum of the incident light into
the three primary colors. Thus, the array elements depicted in
Fig. 1-4 in red respond to red light, and a similar correspondence
applies to those depicted in green and blue. Typical examples
of color sensitivity spectra are shown in Fig. 1-5 for a Nikon
camera.
Regardless of the specific detection mechanism (CCD or
APS), the array output is transferred to a digital storage device
with specific markers denoting the location of each element of
the array and its color code (R, G, or B). Each array consists of
three subarrays, one for red, another for green, and a third for
blue. This information is then used to synchronize the output
of the 2-D detector array with the 2-D pixel arrangement on an
LCD (liquid crystal display) or other electronic displays.
◮ Our notation uses parentheses ( ) with continuous-space
signals and images, as in Io (x′ , y′ ), and square brackets [ ]
with discrete-space images, as in V [n, m]. ◭
The three associated transformations are:
(1) Optical Transformation: from Io (x′ , y′ ; λ ) to Ii (x, y; λ ).
(2) Detection Transformation: from Ii (x, y; λ ) to V [n1 , m1 ].
(3) Display Transformation: from V [n1 , m1 ] to B[n2 , m2 ].
Indices [n1 , m1 ] and [n2 , m2 ] vary over certain ranges of discrete
values, depending on the chosen notation. For a discrete image,
the two common formats are:
A. Continuous and Discrete Images
By the time an image appears on an LCD screen, it will have
undergone a minimum of three transformations, involving a
minimum of three additional images. With λ denoting the
light wavelength and using Fig. 1-6 as a guide, we define the
following images:
Io (x′ , y′ ; λ ): continuous intensity brightness in the object plane,
with (x′ , y′ ) denoting the coordinates of the object plane.
(1) Centered Coordinate System: The central pixel of V [n, m]
is at (n = 0, m = 0), as shown in Fig. 1-7(a), and the image
extends to ±N for n and to ±M for m. The total image
size is (2M + 1) × (2N + 1) pixels. Note that index n varies
horizontally.
(2) Corner Coordinate System: In Fig. 1-7(b), indices n and m
of V [n, m] start at 1 (rather than zero). Image size is M × N.
www.dbooks.org
6
CHAPTER 1 IMAGING SENSORS
Optical transformation
Discretized version
of apple
Detection transformation
Display transformation
,
[n 1
]
m1
V red
Discretized
full screen
Detector array
Io(x′, y′)
Red
filter
Ii(x, y)
[
en
]
m1
n 1,
B[n2, m2]
V gre
Green
filter
Object plane
Image plane
Blue
filter
V blu
]
,m1
[e n 1
Display array
Detector array
Figure 1-6 Io (x′ , y′ ; λ ) and Ii (x, y; λ ) are continuous scene brightness and image intensities, whereas V [n1 , m1 ] and B[n2 , m2 ] are discrete
images of the detected voltage and displayed brightness, respectively.
1-1 OPTICAL IMAGERS
7
Image Notation
N columns
(0, M)
M rows
(0,1)
n
(−1,0) (0,0) (1,0)
(−N,0)
m
(N,0)
• Index n varies horizontally and index m varies vertically.
(0,−1)
• Image notation is the same as matrix notation.
• Image size = # of rows × # of columns = M × N.
(0,−M)
(a) Centered coordinates with
(2M+ 1) × (2N + 1) pixels
The detection and display images may or may not be of
the same size. For example, if image compression is used
to generate a “thumbnail,” then far fewer pixels are used to
represent the imaged object in the display image than in the
detected image. Conversely, if the detected image is to be
enlarged through interpolation, then more pixels are used to
display the object than in the detected image.
1
2
3
B. Point Spread Function
M
m
1
2
3
N
n
(b) Corner coordinates with (M × N ) pixels
Figure 1-7 (a) In the centered coordinate system, index m
extends between −M and +M and index n varies between −N
and +N, whereas (b) in the corner coordinate system, indices n
and m start at 1 and conclude at N and M, respectively.
Consider the scenario depicted in Fig. 1-8(a). An infinitesimally
small source of monochromatic (single wavelength) light, denoted s, is located in the center of the object plane, and the lens
location is adjusted to satisfy Eq. (1.1), thereby producing in the
image plane the best-possible image of the source. We assume
that the lens has no aberrations due to shape or material imperfections. We observe that even though the source is infinitesimal
in spatial extent—essentially like a spatial impulse, its image
is definitely not impulse-like. The image exhibits a circularly
symmetric diffraction pattern caused by the phase interference
of the various rays of light that had emanated from the source
and traveled to the image plane through the lens. The pattern is
called an Airy disc.
Figure 1-8(b) shows a 1-D plot of the image pattern in terms
of the intensity Ii (θ ) as a function of θ , where θ is the angular
deviation from the central horizontal axis (Fig. 1-8(a)). The
expression for Ii (θ ) is
Ii (θ ) = Io
www.dbooks.org
2J1 (γ ) 2
,
γ
(1.2)
8
CHAPTER 1 IMAGING SENSORS
where J1 (γ ) is the first-order Bessel function of the first kind,
and
πD
γ=
sin θ .
(1.3)
λ
Here, λ is the wavelength of the light (assumed to be monochromatic for simplicity) and D is the diameter of the converging
lens. The normalized form of Eq. (1.2) represents the impulse
response h(θ ) of the imaging system,
sin θ =
h(x, y) =
Lens of diameter D
y
(1.5)
Ii (x, y)
2J1 (γ ) 2
=
,
I0
γ
πD
γ=
λ
(1.4)
For a 2-D image, the impulse response is called the point spread
function (PSF).
Detector arrays are arranged in rectangular grids. For a pixel
at (x, y) in the image plane (Fig. 1-9),
x2 + y2
,
x2 + y2 + di2
and Eq. (1.4) can be rewritten as
with
2J1 (γ ) 2
Ii (θ )
=
.
h(θ ) =
Io
γ
s
s
x2 + y2
.
x2 + y2 + di2
(1.6)
(1.7)
The expressions given by Eqs. (1.2) through (1.7) pertain to
coherent monochromatic light. Unless the light source is a laser,
the light source usually is panchromatic, in which case the
diffraction pattern that would be detected by each of the threecolor detector arrays becomes averaged over the wavelength
range of that array. The resultant diffraction pattern maintains
the general shape of the pattern in Fig. 1-8(b), but it exhibits a
gentler variation with θ (with no distinct minima). Here, h(x, y)
denotes the PSF in rectangular coordinates relative to the center
of the image plane.
Source s
x
θ
Object plane
Image plane
(a) Image of impulse source s
y
Ii(θ )
(0,y)
Io
θ
di
γ=
πD
sin θ
λ
−5
5
Along y axis:
γ
(b) 1-D profile of imaged response
Figure 1-8 For an aberration-free lens, the image of a point
source is a diffraction pattern called an Airy disc.
For an image pixel at (x, y):
y
sin θ = q
y2 + di2
s
x2 + y2
sin θ =
2
x + y2 + di2
Figure 1-9 Relating angle θ to pixel at (x, y) in image plane.
1-1 OPTICAL IMAGERS
9
◮ The implication of this PSF is that when the optical
system is used to image a scene, the image it forms in the
image plane is the result of a 2-D convolution (as defined
later in Section 3-3) of the brightness distribution of the
scene in the object plane, Io (x, y), with the PSF given by
Eq. (1.7):
Ii (x, y) = Io (x, y) ∗ ∗ h(x, y).
(1.8)
The convolution effect is then embedded in the discrete
2-D detected image as well as in all subsequent manifestations. ◭
the geometry in Fig. 1-10 with sin θ ≈ θ leads to
∆ymin
λ
≈ 1.22
di
D
(angular resolution).
∆θmin ≈
The angular width ∆θmin is the angular resolution of the
imaging system and ∆ymin is the image spatial resolution. On
the object side of the lens, the scene spatial resolution is
∆y′min = 1.22do ∆θmin = 1.22do
C. Spatial Resolution
Each of the two coherent, monochromatic sources shown in
Fig. 1-10 produces a diffraction pattern. If the two sources are
sufficiently far apart so that their patterns are essentially distinct,
then we should be able to distinguish them from one another. But
as we bring them closer together, their diffraction patterns in the
image plane start to overlap, making it more difficult to discern
their images as those of two distinct sources.
One definition of the spatial resolution capability of the
imaging system along the y′ direction is the separation ∆y′min
between the two point sources (Fig. 1-10) such that the peak of
the diffraction pattern of one of them occurs at the location of
the first null of the diffraction pattern of the other one, and vice
versa. Along the y direction in the image plane, the first null
occurs when [2J1 (γ )/γ ]2 = 0 or, equivalently, γ = 3.832. Use of
(1.9a)
λ
.
D
(1.9b)
(scene spatial resolution)
This is known as the Rayleigh resolution criterion. Because
the lens diameter D is in the denominator, using a larger lens
improves spatial resolution. Thus telescopes are made with very
large lenses and/or mirrors.
These expressions apply to the y and y′ directions at wavelength λ . Since the three-color detector arrays operate over
different wavelength ranges, the associated angular and spatial
resolutions are the smallest at λblue ≈ 0.48 µ m and the largest at
λred ≈ 0.6 µ m (Fig. 1-5). Expressions with identical form apply
along the x and x′ direction (i.e., upon replacing y and y′ with x
and x′ , respectively).
D. Detector Resolution
y
Image pattern of s2
s1
∆θmin
∆ymin
′
s2
∆θmin
∆ymin
The inherent spatial resolution in the image plane is
∆ymin = di λ /D, but the detector array used to record the
image has its own detector resolution ∆p, which is the pixel
size of the active pixel sensor. For a black and white imaging
camera, to fully capture the image details made possible by the
imaging system, the pixel size ∆p should be, at most, equal to
∆ymin . In a color camera, however, the detector pixels of an
individual color are not adjacent to one another (see Fig. 1-4),
so ∆p should be several times smaller than ∆ymin .
do
1-1.2 Thermal IR Imagers
Image pattern of s1
di
Figure 1-10 The separation between s1 and s2 is such that the
peak of the diffraction pattern due to s1 is coincident with the
first null of the diffraction pattern of s2 , and vice versa.
Density slicing is a technique used to convert a parameter
of interest from amplitude to pseudocolor so as to enhance
the visual display of that parameter. An example is shown in
Fig. 1-11, wherein color represents the infrared (IR) temperature of a hot-air balloon measured by a thermal infrared imager.
The vertical scale on the right-hand side provides the color-
www.dbooks.org
10
CHAPTER 1 IMAGING SENSORS
Figure 1-11 IR image of a hot-air balloon (courtesy of Ing.-Buro für Thermografie). The spatial pattern is consistent with the fact that
warm air rises.
1-1 OPTICAL IMAGERS
11
Medium-wave IR
λ = 0.76 μm
2 μm
103 μm
4 μm
Near IR
Gamma
rays
10−6 μm
Long-wave IR
X-rays
Ultraviolet
rays
10−3 μm
Infrared
1 μm
Radio waves
Microwaves
106 μm
103 μm
109 μm
temperature conversion. Unlike the traditional camera—which
measures the reflectance of the observed scene—a thermal IR
imager measures the emission by the scene, without an external
source of illumination. IR imagers are used in many applications
including night vision, surveillance, fire detection, and thermal
insulation in building construction.
A. IR Spectrum
Spectral emittance (Wm−2μm−1)
Figure 1-12 Infrared subbands.
108
106
104
Sun spectrum
5800 K
1000 K
100
The wavelength range of the infrared spectrum extends from the
end of the red part of the visible spectrum at about 0.76 µ m
to the edge of the millimeter-wave band at 1000 µ m (or,
equivalently, λ = 1 mm). For historical reasons, the IR band
has been subdivided into multiple subbands, but these subbands
do not have a standard nomenclature, nor standard definitions
for their wavelength extents. The prevailing practice assigns the
following names and wavelength ranges (Fig. 1-12):
Figure 1-13 The peak of the blackbody radiation spectrum of
(a) The near IR (NIR) extends from λ = 0.76 µ m to
λ = 2 µ m.
the sun is in the visible part of the EM spectrum, whereas the
peak for a terrestrial object is in the IR (at ≈ 10 µ m).
(b) The middle-wave IR (MWIR) extends from λ = 2 µ m to
λ = 4 µ m.
(c) The long-wave IR (LWIR) extends from λ = 4 µ m to
λ = 1000 µ m.
Most sensors operating in the NIR subband are similar to visible
light cameras in that they record light reflectance, but only
in the 0.76–2 µ m range, whereas IR sensors operating at the
longer wavelengths rely on measuring energy self-emitted by the
observed object, which depends, in part, on the temperature of
the object. Hence, such IR sensors are called thermal imagers.
1
0.01
0.1
300 K
Earth spectrum
1
10
Wavelength (μm)
100
The basis for the self-emission is the blackbody radiation
law, which states that all material objects radiate EM energy,
and the spectrum of the radiated energy depends on the physical
temperature of the object, its material composition, and its
surface properties. A blackbody is a perfect emitter and perfect
absorber, and its radiation spectrum is governed by Planck’s law.
Figure 1-13 displays plots of spectral emittance for the sun (at
an effective radiating temperature of 5800 K) and a terrestrial
blackbody at 300 K (27 ◦ C). We observe from Fig. 1-13
that the peak of the terrestrial blackbody is at approximately
www.dbooks.org
12
CHAPTER 1 IMAGING SENSORS
Spectral emissivity
1.0
IR lens
Dectector
array
0.95
0.90
0.85
0.80
Ocean
Vegetation
Desert
Snow/ice
0.75
3.3 μm
5 μm
10 μm 20 μm
50 μm
Wavelength
Data
processing
unit
IR
signal
Optics
Cooler
Figure 1-15 Thermal IR imaging systems often use cryogenic
cooling to improve detection sensitivity.
Figure 1-14 Emissivity spectra for four types of terrain.
(Courtesy the National Academy Press.)
10 µ m, which is towards the short wavelength end of the LWIR
subband. This means that the wavelength range around 10 µ m is
particularly well suited for measuring radiation self-emitted by
objects at temperatures in the range commonly encountered on
Earth. The amount of energy emitted at any specific wavelength
depends not only on the temperature of the object, but also on
its material properties. The emissivity of an object is defined as
the ratio of the amount of energy radiated by that object to the
amount of energy that would have been radiated by the object
had it been an ideal blackbody at the same physical temperature.
By way of an example, Fig. 1-14 displays spectral plots of the
emissivity for four types of terrain: an ocean surface, a desert
surface, a surface covered with snow or ice, and a vegetationcovered surface.
B. Imaging System
The basic configuration of a thermal IR imaging system
(Fig. 1-15) is similar to that of a visible-light camera, but the
lenses and detectors are designed to operate over the intended
IR wavelength range of the system. Two types of detectors
are used, namely uncooled detectors and cooled detectors. By
cooling a semiconductor detector to very low temperatures,
typically in the 50–100 K range, its self-generated thermal noise
is reduced considerably, thereby improving the signal-to-noise
ratio of the detected IR signal emitted by the observed scene.
Cooled detectors exhibit superior sensitivity in comparison with
uncooled detectors, but the cooling arrangement requires the
Visible light
Thermal IR
Figure 1-16 Comparison of black-and-white visible-light
photography with an IR thermal image of the same scene.
availability and use of a cryogenic agent, such as liquid nitrogen,
as well as placing the detectors in a vacuum-sealed container.
Consequently, cooled IR imagers are significantly more expensive to construct and operate than uncooled imagers.
We close this section with two image examples. Figure 1-16
compares the image of a scene recorded by a visible-light blackand-white camera with a thermal IR image of the same scene.
The IR image is in pseudocolor, with red representing high IR
emission and blue representing (comparatively) low IR emission. The two images convey different types of information, but
they also have significantly different spatial resolutions. Today,
digital cameras with 16 megapixel detector arrays are readily
available and fairly inexpensive. In contrast, most standard
1-2 RADAR IMAGERS
13
detector arrays of thermal IR imagers are under 1 megapixel in
size. Consequently, IR images appear “blurry” when compared
with their photographic counterparts.
Our second image, shown in Fig. 1-17, is an IR thermal image
of a person’s head and neck. Such images are finding increased
use in medical diagnostics, particularly for organs close to the
surface [Ring and Ammer, 2012].
Exercise 1-1: An imaging lens used in a digital camera
has a diameter of 25 mm and a focal length of 50 mm.
Considering only the photodetectors responsive to the red
band centered at λ = 0.6 µ m, what is the camera’s spatial
resolution in the image plane, given that the image distance
from the lens is di = 50.25 mm? What is the corresponding
resolution in the object plane?
Answer: ∆ymin = 1.47 µ m; ∆y′min = 0.3 mm.
Exercise 1-2: At λ = 10 µ m, what is the ratio of the
emissivity of a snow-covered surface relative to that of a
sand-covered surface? (See Fig. 1-14.)
Answer: esnow /esand ≈ 0.985/0.9 = 1.09.
1-2 Radar Imagers
Conceptually, a radar can generate an image of the reflectivity of
a scene by scanning its antenna beam across the scene in a rasterlike format, as depicted in Fig. 1-18. Even though the imaging
process is very different from the process used by a lens in a
camera, the radar and the camera share the same fundamental
relationship for angular resolution. In Section 1-1.1, we stated
in Eq. (1.9a) that the angular resolution of a converging lens is
approximately ∆θmin = 1.22λ /D, and the corresponding spatial
resolution is
Figure 1-17 Thermal IR image of a person’s head and neck.
Concept Question 1-1: What is a camera’s point spread
function? What role does it play in the image formation
process?
Concept Question 1-2: How are the image and scene
spatial resolutions related to one another?
Concept Question 1-3:
∆y′min = do ∆θmin = 1.22do
What is the emissivity of an ob-
thermal imager?
Why is an IR imager called a
(camera).
(1.10a)
Here, λ is the wavelength of the light and D is the diameter of
the lens.
Equation (1.10a) is approximately applicable to a microwave
radar with a dish antenna of diameter D (in the camera case,
the scene illumination is external to the camera, so the lens gets
involved in only the receiving process, whereas in the radar case
the antenna is involved in both the transmitting and receiving
processes). In the radar literature, the symbol usually used to
denote the range between the radar antenna and the target is the
symbol R. Hence, upon replacing do with R, we have
∆y′min ≈ R
ject?
Concept Question 1-4:
λ
D
λ
D
(radar).
(1.10b)
It is important to note that λ of visible light is much shorter
than λ in the microwave region. In the middle of the visible
spectrum, λvis ≈ 0.5 µ m, whereas at a typical microwave radar
www.dbooks.org
14
CHAPTER 1 IMAGING SENSORS
∆y′min
R
∆θmin
D
Figure 1-18 Radar imaging of a scene by raster scanning the antenna beam.
frequency of 6 GHz, λmic ≈ 5 cm. The ratio is
5 × 10−2
λmic
=
= 105 !
λvis
0.5 × 10−6
This means that the angular resolution capability of an optical
system is on the order of 100,000 times better than the angular
resolution of a radar, if the lens diameter is the same size as the
antenna diameter.
To fully compensate for the large wavelength ratio, a radar
antenna would need a diameter on the order of 1 km to produce
an image with the same resolution as a camera with a lens 1 cm
in diameter. Clearly, that is totally impractical. In practice, most
radar antennas are on the order of centimeters to meters in size,
but certainly not kilometers. Yet, radar can image the Earth
surface from satellite altitudes with spatial resolutions on the
order of 1 m—equivalent to antenna sizes several kilometers in
extent! How is that possible?
A. Synthetic-Aperture Radar
As we will see shortly, a synthetic-aperture radar (SAR) uses
a synthesized aperture to achieve good resolution in one dimen-
sion and transmits very short pulses to achieve fine resolution in
the orthogonal dimension. The predecessor to SAR is the realaperture side-looking airborne radar (SLAR). A SLAR uses
a rectangular- or cylindrical-shaped antenna that gets mounted
along the longitudinal direction of an airplane, and pointed
partially to the side (Fig. 1-19).
Even though the antenna beam in the elevation direction
is very wide, fine discrimination can be realized along the x
direction in Fig. 1-19 by transmitting a sequence of very short
pulses. At any instant in time, the extent of the pulse along x is
∆x′min =
cτ
2 sin θ
(scene range resolution),
(1.11)
where c is the velocity of light, τ is the pulse width, and θ is
the incidence angle relative to nadir-looking. This represents the
scene spatial resolution capability along the x′ direction. At a
typical angle of θ = 45◦ , the spatial resolution attainable when
transmitting pulses each 5 µ s in width is
∆x′min =
3 × 108 × 5 × 10−9
≈ 1.05 m.
2 sin 45◦
1-2 RADAR IMAGERS
15
Recorder (digital storage)
Transmitte
Tr
terte
rTransmitter-receiver
va
An
Antenna
Idealized
lized elevation
antenna pattern
ly
Short pu
pulse
Trees
Bank edge
Water
Truck
x′′
Shadow
∆y′′
∆
∆y
Start sweep
Truck
Trees
Sloping edge
Shadow
Video
amplitude
End of sweep
Water
Time
(range)
A-scope display
Scrub growth
(brush, grass, bare earth, etc.)
Figure 1-19 Real-aperture SLAR imaging technique. The antenna is mounted along the belly of the aircraft.
Not only is this an excellent spatial resolution along the x′
direction, but it is also independent of range R (distance between
the radar and the surface), which means it is equally applicable
to a satellite-borne radar.
As the aircraft flies along the y direction, the radar beam
sweeps across the terrain, while constantly transmitting pulses,
receiving their echoes, and recording them on an appropriate
medium. The sequential echoes are then stitched together to
form an image.
By designing the antenna to be as long as practicable along
the airplane velocity direction, the antenna pattern exhibits a
relatively narrow beam along that direction (y′ direction in
Fig. 1-19). The shape of the beam of the cylindrical antenna
is illustrated in Fig. 1-20. From range R, the extent of the beam
along the y direction is
λ
λh
R=
,
ly
ly cos θ
(real-aperture azimuth resolution)
∆y′min ≈
(1.12)
where h is the aircraft altitude. This is the spatial resolution
capability of the radar along the flight direction. For a 3 m long
antenna operating at λ = 3 cm from an altitude of 1 km, the
www.dbooks.org
16
CHAPTER 1 IMAGING SENSORS
Along-track resolution
Real-aperture ∆y′min = λR/ly
Synthetic-aperture ∆y′min = ly /2
βxz ≈ λ
lx
ly
R
βyz ≈ λ
ly
Example: λ = 4 cm
spacecraft radar
lx
Length of
synthetic aperture
8 km
Fan beam
Resolution
of synthetic
aperture
1m
Figure 1-20 Radiation pattern of a cylindrical reflector.
400 km
ly = 2 m
Length of
real aperture
resolution ∆y′min at θ = 45◦ is
∆y′min ≈
3 × 10−2
× 103 ≈ 14 m.
3 cos45◦
Ideally, an imaging system should have similar resolution capabilities along both directions of the imaged scene. In the
present case, ∆x′min ≈ 1.05 m, which is highly desirable, but
∆y′min ≈ 14 m, which for most imaging applications is not so
desirable, particularly if the altitude h is much higher than 1 km.
Furthermore, since ∆y′min is directly proportional to the altitude
h of the flying vehicle, whereas ∆x′min is independent of h, the
disparity between ∆x′min and ∆y′min will get even greater when
we consider radars flown at satellite altitudes.
To improve the resolution ∆y′min and simultaneously remove
its dependence on the range R, we artificially create an array
of antennas as depicted in Fig. 1-21. In the example shown in
Fig. 1-21, the real satellite-borne radar antenna is 2 m long and
the synthetic aperture is 8 km long! The latter consists of pulse
returns recorded as the real antenna travels over a distance of
8 km, and then processed later as if they had been received by
an 8 km long array of antennas, each 2 m long, simultaneously.
The net result of the processing is an image with a resolution
along the y direction given by
∆y′min =
ly
2
(SAR azimuth resolution),
(1.13)
where ly is the length of the real antenna. For the present
example, ly = 2 m and ∆y′min = 1 m, which is approximately the
same as ∆x′min . Shortening the antenna length would improve the
azimuth resolution, but considerations of signal-to-noise ratio
would require the transmission of higher power levels.
8 km
Resolution of
real aperture
Figure 1-21 An illustration of how synthetic aperture works.
B. Point Spread Function
The first of our two SAR-image examples displays a large part
of Washington, D.C. (Fig. 1-22). The location information of
a particular pixel in the observed scene is computed, in part,
from the round-trip travel time of the transmitted pulse. Consequently, a target that obscures the ground beneath it, such as the
Washington Monument in Fig. 1-22, ends up generating a radar
shadow because no signal is received from the obscured area.
The radar shadow of the obelisk of the Washington Monument
appears as a dark line projected from the top onto the ground
surface.
Radar shadow is also apparent in the SAR image of the plane
and helicopter of Fig. 1-23.
In Section 1-1.1, we stated that the image formed by the lens
of an optical camera represents the convolution of the reflectivity
of the scene (or the emission distribution in the case of the IR
imager) with the point spread function (PSF) of the imaging
system. The concept applies equally well to the imaging radar
case. For an x–y SAR image with x denoting the side-looking
direction and y denoting the flight direction, the SAR PSF is
1-2 RADAR IMAGERS
17
NRL
N
Capitol
Pentagon
White House
Washington
Monument
Figure 1-22 SAR image collected over Washington, D.C. Right of center is the Washington Monument, though only the shadow of the obelisk
is readily apparent in the image. [Courtesy of Sandia National Laboratories.]
www.dbooks.org
18
CHAPTER 1 IMAGING SENSORS
given by
h(x, y) = hx (x) hy (y),
(1.14)
with hx (x) describing the shape of the transmitted pulse and
hy (y) describing the shape of the synthetic antenna-array pattern. Typically, the pulse shape is like a Gaussian:
2
hx (x) = e−2.77(x/τ ) ,
(1.15a)
where τ is the effective width of the pulse (width between halfpeak points). The synthetic array pattern is sinc-like in shape,
but the sidelobes may be suppressed further by assigning different weights to the processed pulses. For the equally weighted
case,
2 1.8y
,
(1.15b)
hy (y) = sinc
l
where l is the length of the real antenna, and the sinc function is
defined such that sinc(z) = sin(π z)/(π z) for any variable z.
Why is a SAR called a
Concept Question 1-5:
“synthetic”-aperture radar?
Concept Question 1-6:
What system parameters deter-
mine the PSF of a SAR?
Figure 1-23 High-resolution image of an airport runway
with a plane and helicopter. [Courtesy of Sandia National
Laboratories.]
Exercise 1-3: With reference to the diagram in Fig. 1-21,
suppose the length of the real aperture were to be increased
from 2 m to 8 m. What would happen to (a) the antenna
beamwidth, (b) length of the synthetic aperture, and (c) the
SAR azimuth resolution?
Answer: (a) Beamwidth is reduced by a factor of 4, (b)
synthetic aperture length is reduced from 8 km to 2 km, and
(c) SAR resolution changes from 1 m to 4 m.
1-3 X-Ray Computed Tomography
(CT)
Computed tomography, also known as CT scan, is a technique
capable of generating 3-D images of the X-ray attenuation (absorption) properties of an object, such as the human body. The
X-ray absorption coefficient of a material is strongly dependent
on the density of that material. CT has the sensitivity necessary
to image body parts across a wide range of densities, from soft
tissue to blood vessels and bones.
As depicted in Fig. 1-24(a), a CT scanner uses an X-ray
source, with a narrow slit to generate a fan-beam, wide enough
to encompass the extent of the body, but only about 1 mm
thick. The attenuated X-ray beam is captured by an array of
∼ 900 detectors. The X-ray source and the detector array are
mounted on a circular frame that rotates in steps of a fraction
of a degree over a full 360◦ circle around the object or patient,
each time recording an X-ray attenuation profile from a different
angular direction. Typically, on the order of 1000 such profiles
are recorded, each composed of measurements by 900 detectors.
For each horizontal slice of the body, the process is completed
in less than 1 second. CT uses image reconstruction algorithms
to generate a 2-D image of the absorption coefficient of that
horizontal slice. To image an entire part of the body, such as
the chest or head, the process is repeated over multiple slices
(layers).
For each anatomical slice, the CT scanner generates on
the order of 9 × 105 measurements (1000 angular orientations
×900 detectors). In terms of the coordinate system shown in
Fig. 1-24(b), we define α (ξ , η ) as the absorption coefficient
of the object under test at location (ξ , η ). The X-ray beam is
directed along the ξ direction at η = η0 . The X-ray intensity
received by the detector located at ξ = ξ0 and η = η0 is given
by
I(ξ0 , η0 ) = I0 exp −
Z ξ0
0
α (ξ , η0 ) d ξ ,
(1.16)
1-4 MAGNETIC RESONANCE IMAGING
19
where I0 is the X-ray intensity radiated by the source. Outside
the body, α (ξ , η ) = 0. The corresponding logarithmic path
attenuation p(ξ0 , η0 ) is defined as
X-ray
source
Fan beam
of X-rays
p(ξ0 , η0 ) = − log
Detector
array
p(r, θ ) =
(a) CAT scanner
η
η0
I(ξ0,η0)
X-ray detector
Object
0
Absorption
coefficient
α(ξ,η)
I(ξ0,η0) = ξ
0
I0 exp(− ∫ α(ξ,η0) dξ)
0
ξ0
(b) Horizontal path
0
Z ξ0
0
α (ξ , η0 ) d ξ .
(1.17)
The path attenuation p(ξ0 , η0 ) is the integrated absorption
coefficient across the X-ray path.
In the general case, the path traversed by the X-ray source is at
a range r and angle θ in a polar coordinate system, as depicted
in Fig. 1-24(c). The direction of the path is orthogonal to the
direction of r. For a path corresponding to a specific set (r, θ ),
Eq. (1.17) becomes
Computer
and monitor
X-ray
source
I0
I(ξ0 , η0 )
=
I0
ξ
Z ∞Z ∞
−∞ −∞
α (ξ , η ) δ (r − ξ cos θ − η sin θ ) d ξ d η ,
(1.18)
where the Dirac impulse δ (r − ξ cos θ − η sin θ ) dictates that
only those points in the (ξ , η ) plane that fall along the path
specified by fixed values of (r, θ ) are included in the integration.
The relation between p(r, θ ) and α (ξ , η ) is known as the
2-D Radon transform of α (ξ , η ). The goal of CT is to reconstruct α (ξ , η ) from the measured path attenuations p(r, θ ),
by inverting the Radon transform given by Eq. (1.18), which is
accomplished with the help of the Fourier transform.
Concept Question 1-7: What physical attribute of the
imaged body is computed and displayed by a CT scanner?
1-4 Magnetic Resonance Imaging
η
X-ray source
I0
r
θ
α(ξ,η)
I(r,θ)
Detector
ξ
(c) Path at radius r and orientation θ
Figure 1-24 (a) CT scanner, (b) X-ray path along x, and (c)
X-ray path along arbitrary direction.
Since its early demonstration in the 1970s, magnetic resonance
imaging (MRI) has become a highly valuable tool in diagnostic
radiology, primarily because it can generate high-resolution
anatomical images of the human body, without exposing the
patient to ionizing radiation. Like X-ray CT scanners, magnetic
resonance (MR) imagers can generate 3-D images of the body
part of interest, from which 2-D slices can be extracted along
any orientation of interest. The name MRI derives from the fact
that the MRI scanner measures nuclear magnetic resonance
(NMR) signals emitted by the body’s tissues and blood vessels
in response to excitation by a magnetic field introduced by a
radio frequency (RF) system.
1-4.1 Basic System Configuration
The MRI system shown in Fig. 1-25 depicts a human body lying
www.dbooks.org
20
CHAPTER 1 IMAGING SENSORS
The Magnetic Field
Superconducting magnet generates static field B0
Superconducting
magnet
Gradient coils
generate field BG
RF coil excites
nuclei and “listens”
to the response
yˆ
B0
zˆ
Radio frequency
transmitter
Gradient
amplifiers
Radio frequency
receiver
~1.5 T
xˆ
~1 × 10−4 T
Computer
Figure 1-25 Basic diagram of an MRI system.
inside a magnetic core. The magnetic field at a given location
(x, y, z) within the core and at a given instant in time t may
consist of up to three magnetic field contributions:
Figure 1-26 B0 is static and approximately uniform within
the cavity. Inside the cavity, B0 ≈ 1.5 T (teslas), compared with
only 0.1 to 0.5 milliteslas outside.
Magnetic moment
B0
mI = −1/2
B = B0 + BG + BRF ,
where B0 is a static field, BG is the field gradient, and BRF is
the radio frequency (RF) excitation used to solicit a response
from the biological material placed inside the core volume. Each
of these three components plays a critical role in making MRI
possible, so we will discuss them individually.
A. Static Field B0
Field B0 is a strong, static (non–time varying) magnetic field
created by a magnet designed to generate a uniform (constant)
distribution throughout the magnetic core (Fig. 1-26). Usually, a
superconducting magnet is used for this purpose because it can
generate magnetic fields with much higher magnitudes than can
be realized with resistive and permanent magnets. The direction
of B0 is longitudinal (ẑ direction in Fig. 1-26) and its magnitude
is typically on the order of 1.5 teslas (T). The conversion factor
between teslas and gauss is 1 T = 104 gauss. Earth’s magnetic
field is on the order of 0.5 gauss, so B0 inside the MRI core is
on the order of 30,000 times that of Earth’s magnetic field.
Biological tissue is composed of chemical compounds, and
each compound is organized around the nuclei (protons) of the
atoms comprising that compound. Some, but not all, nuclei be-
θ
mI = 1/2
Figure 1-27 Nuclei with spin magnetic number of ±1/2
precessing about B0 at the Larmor angular frequency ω0 .
come magnetized when exposed to a magnetic field. Among the
substances found in a biological material, the hydrogen nucleus
has a strong susceptibility to magnetization, and hydrogen is
highly abundant in biological tissue. For these reasons, a typical
MR image is related to the concentration of hydrogen nuclei.
The strong magnetic field B0 causes the nuclei of the material
inside the core space to temporarily magnetize and to spin
(precess) like a top about the direction of B0 . The precession orientation angle θ , shown in Fig. 1-27, is determined by the spin
quantum number I of the spinning nucleus and the magnetic
1-4 MAGNETIC RESONANCE IMAGING
21
Table 1-2 Gyromagnetic ratio γ– for biological nuclei.
Isotope
Spin I
% Abundance
1H
1/2
1/2
1
5/2
1/2
3/2
1/2
99.985
1.108
99.63
0.037
100
100
100
13 C
14 N
17 O
19 F
23 Na
31 P
γ– MHz/T
42.575
10.71
3.078
5.77
40.08
11.27
17.25
x
z
quantum number mI . A material, such as hydrogen, with a spin
system of I = 1/2 has magnetic quantum numbers mI = ±1/2.
Hence, the nucleus may
p spin along two possible directions
defined by cos θ = mI / I(I + 1), which yields θ = ±54◦ 44′
[Liang and Lauterbur, 2000].
The associated angular frequency of the nuclear precession
is called the Larmor frequency and is given by
ω0 = γ B0
(Larmor angular frequency),
(1.19a)
with ω0 in rad/s, B0 in teslas (T), and γ , the gyromagnetic ratio
of the material, in (rad/s)/T. Alternatively, we can express the
precession in terms of the frequency f0 = ω0 /2π , in which case
Eq. (1.19a) assumes the equivalent form
f0 = γ–B0
(Larmor frequency),
y
z
(1.19b)
where γ– = γ /2π . This fundamental relationship between f0 and
B0 is at the heart of what makes magnetic resonance imaging
possible. Table 1-2 provides a list of nuclei of biological interest
that have nonzero spin quantum numbers, along with their corresponding gyromagnetic ratios. For the hydrogen isotope 1 H,
γ– = 42.575 MHz/T, so the Larmor frequency for hydrogen at
B0 = 1.5 T is f0 = 63.8625 MHz, which places it in the RF part
of the EM spectrum. Since the human body is made up primarily
of water, the most commonly imaged nucleus is hydrogen.
B. Gradient Field BG
The MRI system includes three current-activated gradient coils
(Fig. 1-28) designed to generate magnetic fields pointed along
the ẑ direction—the same as B0 , but whose magnitudes exhibit
linear spatial variations along the x̂, ŷ, and ẑ directions. That
is why they are called gradient fields. The three coils can be
activated singly or in combination. The primary purpose of the
x
y
Figure 1-28 Magnetic coils used to generate magnetic fields
along three orthogonal directions. All three gradient fields BG
point along ẑ, but their intensities vary linearly along x, y, and z.
gradient magnetic field is localization (in addition to other information that can be extracted about the tissue material contained
in the core volume during the activation and deactivation cycles
of the gradient fields). Ideally, the gradient fields assume the
following spatial variation inside the core volume:
BG = (Gx x + Gy y + Gz z)ẑ,
with the center of the (x, y, z) coordinate system placed at the
center of the core volume. The gradient coefficients Gx , Gy ,
and Gz are on the order of 10 mT/m, and they are controlled
individually by the three gradient coils.
Let us consider an example in which Gx = Gz = 0 and Gy = 10
mT/m, and let us assume that the vertical dimension of the core
volume is 1 m. If B0 = 1.5 T, the combined field will vary from
(
1.495 T @ y = − 12 m, to
B = B 0 + Gy y =
1.505 T @ y = 12 m,
as depicted in Fig. 1-29. By Eq. (1.19b), the corresponding
Larmor frequency for hydrogen will vary from 63.650 MHz for
hydrogen nuclei residing in the plane at y = −0.5 m to 64.075
MHz for nuclei residing in the plane at y = +0.5 m. As we will
explain shortly, when an RF signal at a particular frequency fRF
is introduced inside the core volume, those nuclei whose Larmor
frequency f0 is the same as fRF will resonate by absorbing part
of the RF energy and then reemitting it at the same frequency (or
slightly shifted in the case of certain chemical reactions). The
strength of the emitted response is proportional to the density
of nuclei. By varying the total magnetic field B linearly along
www.dbooks.org
22
CHAPTER 1 IMAGING SENSORS
intended application [Liang and Lauterbur, 2000]. The magnetic
field of the transmitted energy causes the exposed biological
tissue to resonate at its Larmor frequency. With the transmitter
off, the receiver picks up the resonant signals emitted by the
biological tissue. The received signals are Fourier transformed
so as to establish a one-to-one correspondence to the locations
of the voxels responsible for the emission. For each voxel, the
strength of the associated emission is related to the density of 1 H
nuclei in that voxel as well as to other parameters that depend
on the tissue properties and pulse timing.
Gradient field intensity
y
Volume
Image slice
Figure 1-29 Imposing a gradient field that varies linearly with
y allows stratification into thin slices, each characterized by its
own Larmor frequency.
the vertical direction, the total core volume can be discretized
into horizontal layers called slices, each corresponding to a
different value of f0 (Fig. 1-29). This way, the RF signal can
communicate with each slice separately by selecting the RF
frequency to match f0 of that slice. In practice, instead of
sending a sequence of RF signals at different frequencies, the RF
transmitter sends out a short pulse whose frequency spectrum
covers the frequency range of interest for all the slices in the
volume, and then a Fourier transformation is applied to the
response from the biological tissue to separate the responses
from the individual slices.
The gradient magnetic field along the ŷ direction allows discretization of the volume into x–y slices. A similar process can
be applied to generate x–z and y–z slices, and the combination is
used to divide the total volume into a three-dimensional matrix
of voxels (volume pixels). The voxel size defines the spatial
resolution capability of the MRI system.
C. RF System
The combination of the strong static field B0 and the gradient
field BG (whose amplitude is on the order of less than 1%
of B0 ) defines a specific Larmor frequency for the nuclei of
every isotope within each voxel. As we noted earlier through
Table 1-1, at B0 intensities in the 1 T range, the Larmor frequencies of common isotopes are in the MHz range. The RF system
consists of a transmitter and a receiver connected to separate
coils, or the same coil can be used for both functions. The
transmitter generates a burst of narrow RF pulses. In practice,
many different pulse configurations are used, depending on the
1-4.2 Point Spread Function
Generating the MR image involves applying the discrete form
of the Fourier transform. Accordingly, the point spread function
of the MR image is given by a discrete form of the sinc function,
namely [Liang and Lauterbur, 2000]:
hx (x) = ∆k
sin(π N ∆k x)
,
sin(π ∆k x)
(1.20)
where x is one of the two MR image coordinates, k is a spatial
frequency, ∆k is the sampling interval in k space, and N is the
total number of Fourier samples. A similar expression applies
to hy (y). The spatial resolution of the MR image is equal to the
equivalent width of hx (x), which can be computed as follows:
1
∆xmin =
h(0)
Z 1/(1 ∆k)
−1/(1 ∆k)
hx (x) dx =
1
.
N ∆k
(1.21)
The integration was performed over one period (1/∆k) of hx (x).
According to Eq. (1.21), the image resolution is inversely
proportional to the product N ∆k. The choices of values for N
and ∆k are associated with signal-to-noise ratio and scan time
considerations.
1-4.3 MRI-Derived Information
Generally speaking, MRI can provide three types of information
about the imaged tissue:
(a) The magnetic characteristics of tissues, which are related
to biological attributes and blood vessel conditions.
(b) Blood flow, made possible through special time-dependent
gradient excitations.
(c) Chemical properties discerned from measurements of
small shifts in the Larmor frequency.
1-5 ULTRASOUND IMAGER
23
Figure 1-30 MR image.
An example of an MR image is shown in Fig. 1-30.
Concept Question 1-8: An MRI system uses three different types of magnetic fields. For what purpose?
Concept Question 1-9: What determines the Larmor frequency of a particular biological material?
receive switch. Thus, the array serves to both launch acoustic
waves in response to electrical excitation as well as to receive
the consequent acoustic echoes and convert them back into
electrical signals. The echoes are reflections from organs and
tissue underneath the skin of the body part getting imaged by
the ultrasound imager (Fig. 1-32).
The transmitter unit in Fig. 1-31, often called the pulser,
generates a high-voltage short-duration pulse (on the order of
a few microseconds in duration) and sends it to the transmit
beamforming unit, which applies individual time delays to
the pulse before passing it on to the transducers through the
transmit/receive switch. The choice of time delays determines
the range at which the acoustic waves emitted by the four
transducers interfere constructively, as well as the direction
of that location relative to the axis of the array. The range
operation is called focusing and the directional operation is
called steering. Reciprocal operations are performed by the
receive beamforming unit; it applies the necessary time delays
to the individual signals made available by the transducers and
then combines them together coherently to generate the receive
echo. The focusing and steering operations are the subject of the
next subsection.
1-5 Ultrasound Imager
1-5.2 Beam Focusing and Steering
Human hearing extends up to 20 kHz. Ultrasound is defined
as sound at frequencies above that range. Ultrasound imaging
systems, which operate in the 2 to 20 MHz range, have numerous industrial and medical applications, and the latter include
both diagnosis and therapy. Fundamentally, ultrasound imagers
are similar to radar imagers in that both sensors employ phase
shifting (or, equivalently, time delaying) to focus and steer their
beams at the desired distances and along the desired directions.
Imaging radars use 1-D or 2-D arrays of antennas, and likewise,
ultrasound imagers use 1-D or 2-D arrays of transducers. However, electromagnetic waves and sound waves have different
propagation properties, so the focusing and steering techniques
are not quite identical.
The focusing operation is illustrated by the diagram in Fig. 1-33
using eight transducers. In response to the electrical stimulations
introduced by the beamforming unit, all of the transducers
generate outward-going acoustic waves that are identical in
every respect except for their phases (time delays). The specific
distribution of the time delays shown in Fig. 1-33(a) causes
the eight acoustic waves to interfere constructively at the point
labeled Focus 1 at range Rf1 . The time-delay distribution is
symmetrical relative to the center of the array, so the direction
of Focus 1 is broadside to the array axis. Changing the delay
shifts between adjacent elements, while keeping the distribution
symmetrical, as in Fig. 1-33(b), causes the focal point to move
to Focus 2 at range Rf2 . If no time delay is applied to any of the
eight transducer signals, the focal point moves to infinity.
1-5.1 Ultrasound System Architecture
Ultrasound imagers use both 1-D and 2-D transducer arrays
(with some as large as 2000 × 8000 elements and each on the
order of 5 µ m × 5 µ m in size), but for the sake of simplicity,
we show in Fig. 1-31 only a 1-D array with four elements. The
system has a transmitting unit and a receiving unit, with the
transducer array connected to the two units through a transmit/
◮ The combined beam of the transducer array can be
focused as a function of depth by varying the incremental
time delay between adjacent elements in a symmetrical
time-delay distribution. ◭
www.dbooks.org
24
CHAPTER 1 IMAGING SENSORS
Transmit pulse
Transmitter
unit
Display
τ
Transmit
beamforming
(time delay
generator)
System
processor
Transducers
T/R
switch
Receive echo
Data
acquisition
unit
Receive
beamforming
(time delay
generator)
Figure 1-31 Block diagram of an ultrasound system with a 4-transducer array.
The image displayed in Fig. 1-34 is a simulation of acoustic
energy across two dimensions, the lateral dimension parallel to
the array axis and the axial dimension along the range direction.
The array consists of 96 elements extending over a length of
20 mm, and the beam is focused at a range R = 40 mm.
The delay-time distribution shown in Fig. 1-35 is symmetrical
relative to the broadside direction. By shifting the axis of
symmetry to another direction, the focal point moves to a new
direction. This is called steering the beam, and is illustrated in
Fig. 1-35 and simulated in part (b) of Fig. 1-34.
With a 2-D array of transducers, the steering can be realized
along two orthogonal directions, so the combination of focusing
and steering ends up concentrating the acoustic energy radiated
by the transducer array into a small voxel within the body
getting imaged by the ultrasound probe. Similar operations are
performed by the receive beamforming unit so as to focus and
steer the beam of the transducer array to receive the echo from
the same voxel.
Figure 1-32 Ultrasound imaging of the thyroid gland.
1-5 ULTRASOUND IMAGER
25
Beamforming unit
Beamforming unit
Array
axis
Array axis
Rf1
Rf2
Focus 1
Focus 2
Broadside direction
(a) Focal point at Rf1
Broadside direction
(b) Focal point at Rf2
Figure 1-33 Changing the inter-element time delay across a symmetrical time-delay distribution shifts the location of the focal point in
the range direction.
1-5.3 Spatial Resolution
For a 2-D transducer array of size (Lx × Ly ) and focused at
range Rf , as shown in Fig. 1-36 (side Ly is not shown in
the figure), the size of the resolution voxel is given by an
axial resolution ∆Rmin along the range direction and by lateral
resolutions ∆xmin and ∆ymin along the two lateral directions. The
axial resolution is given by
λN
∆Rmin =
2
cycles in the pulse. The wavelength is related to the signal
frequency by
v
λ= ,
(1.23)
f
where v is the wave velocity and f is the frequency. In biological
tissue, v ≈ 1540 m/s. For an ultrasound system operating at
f = 5 MHz and generating pulses with N = 2 cycles per pulse,
∆Rmin =
(axial resolution),
(1.22)
where λ is the wavelength of the pulse in the material in which
the acoustic waves are propagating and N is the number of
www.dbooks.org
1540 × 2
vN
≈ 0.3 mm.
=
2f
2 × 5 × 106
CHAPTER 1 IMAGING SENSORS
Lateral distance (mm)
26
90
90
70
70
50
50
30
30
10
10
10
10
30
30
50
50
70
70
90
0
Lx
∆ Rmin
Rf
90
0 20 40 60 80
20 40 60 80 100
Axial distance (mm)
(a) Focused beam
with no steering
∆ xmin
Figure 1-36 Axial resolution ∆Rmin and lateral resolution
∆xmin for a transducer array of length Lx focused at range Rf .
(b) Focused beam
with steering
by 45o
The lateral resolution ∆xmin is given by
Figure 1-34 Simulations of acoustic energy distribution for
(a) a beam focused at Rf = 40 mm by a 96-element array and
(b) a beam focused and steered by 45◦ .
∆xmin = Rf
λ
Rf v
=
Lx
Lx f
(lateral resolution),
(1.24)
where Rf is the focal length (range at which the beam is
focused). If the beam is focused at Rf = 5 cm and the array
length Lx = 4 cm and f = 5 MHz, then
∆xmin =
(a) Uniform distribution
(b) Linear shift
(c) Non-uniform
symmetrical
distribution
5 × 10−2 × 1540
≈ 0.4 mm,
4 × 10−2 × 5 × 106
(d) Nonlinear shift
and non-uniform
distribution
Figure 1-35 Beam focusing and steering are realized by shaping the time-delay distribution.
1-6 COMING ATTRACTIONS
27
which is comparable with the magnitude of the axial resolution
∆Rmin . The resolution along the orthogonal lateral direction,
∆ymin , is given by Eq. (1.24) with Lx replaced with Ly . The size
of the resolvable voxel is
∆V = ∆Rmin × ∆xmin × ∆ymin.
(1.25)
Figure 1-37 displays an ultrasound image of a fetus.
1-6 Coming Attractions
Through examples of image processing products, this section
presents images extracted from various sections in the book.
In each case, we present a transformed image, along with a
reference to its location within the text.
1-6.1 Image Warping by Interpolation
Section 4-10 demonstrates how an image can be warped by
nonlinear shifts of its pixel locations and then interpolating the
results to generate a smooth image. An example is shown in
Fig. 1-38.
0
199
0
199
(a) Original clown image
Figure 1-37 Ultrasound image of a fetus.
0
50
100
Concept Question 1-10: How does an ultrasound imager
focus its beam in the range direction and in the lateral
direction? There are two orthogonal lateral directions, so
how is that managed?
150
200
250
300
Exercise 1-4: A 6 MHz ultrasound system generates pulses
with 2 cycles per pulse using a 5 cm × 5 cm 2-D transducer
array. What are the dimensions of its resolvable voxel when
focused at a range of 8 cm in a biological material?
Answer: ∆V = ∆Rmin × ∆xmin × ∆ymin
= 0.26 mm × 0.41 mm × 0.41 mm.
350
399
0
50
100
150
200
250
300
350
399
(b) Warped image product
Figure 1-38 Original clown image and nonlinearly warped
product. [Extracted from Figs. 4-14 and 4-17.]
www.dbooks.org
28
CHAPTER 1 IMAGING SENSORS
1-6.2 Image Sharpening by Highpass Filtering
1-6.3 Brightening by Histogram Equalization
Section 5-2 illustrates how an image can be sharpened by spatial
highpass filtering, and how this amplifies noise in the image. The
original image of an electronic circuit and its highpass-filtered
version are displayed in Fig. 1-39.
Histogram equalization (nonlinear transformation of pixel values) can be used to brighten an image, as illustrated by the pair
of images in Fig. 1-40.
0
200
0
200
(a) Dark clown image
(a) Original image
0
200
0
200
(b) Brightened clown image
(b) Sharpened image
Figure 1-39 Image of electronic circuit before and after
application of a highpass sharpening filter. [Extracted from
Fig. 5-6.]
Figure 1-40 Application of histogram equalization to the dark
image in (a) leads to the brighter image in (b). [Extracted from
Fig. 5-8.]
1-6 COMING ATTRACTIONS
29
1-6.4 Edge Detection
1-6.5 Notch Filtering
Edges can be enhanced in an image by applying edge detection
algorithms, such as the Canny edge detector that was applied to
the image in Fig. 1-41(a).
Notch filtering can be used to remove sinusoidal interference
from an image. The original Mariner space probe image and its
notch-filtered version are shown in Fig. 1-42.
0
0
20
40
60
80
100
120
659
140
0
799
(a) Original Mariner image
160
180
0
200
0
20
40
60
80
100 120 140 160 180 200
(a) Original clown image
0
20
40
60
80
659
100
0
799
(b) Notch-filtered Mariner image
120
Figure 1-42 The horizontal lines in (a) are due to sinusoidal
interference in the recorded image. The lines were removed by
applying notch filtering. [Extracted from Fig. 6-7.]
140
160
180
200
0
20
40
60
80
100 120 140 160 180 200
(b) Edge-detected image
Figure 1-41 Original clown image and its Canny edgedetected version. [Extracted from Fig. 5-16.]
www.dbooks.org
30
CHAPTER 1 IMAGING SENSORS
1-6.6 Motion-Blur Deconvolution
1-6.7 Denoising of Images Using Wavelets
Section 6-6 shows how to deblur a motion-blurred image. A
blurred image and its motion-deblurred version are shown in
Fig. 1-43.
One method used to denoise an image is by thresholding
and shrinking its wavelet transform. An example is shown in
Fig. 1-44.
0
20
40
60
80
100
120
140
224
0
274
(a) Original motion-blurred image caused by
taking a photograph in a moving car
160
180
200
20
40
60
80 100 120 140 160 180 200
(a) A noisy clown image
0
20
40
60
80
100
120
140
160
274
0
274
(b) Motion deblurred image
180
200
20
40
60
80 100 120 140 160 180 200
(b) Wavelet-denoised clown image
Figure 1-43 Motion blurring is removed. [Extracted from
Fig. 6-11.]
Figure 1-44 Image denoising. [Extracted from Fig. 7-21.]
1-6 COMING ATTRACTIONS
31
1-6.8 Image Inpainting
The wavelet transform of an image can be used to “inpaint”
(restore missing pixels in an image). An image with deleted
pixels and its inpainted version are shown in Fig. 1-45.
20
1-6.9 Deconvolution Using Deterministic and
Stochastic Image Models
Chapters 8 and 9 provide reviews of probability and estimation.
The reason for these reviews is that using a stochastic image
model, in which the 2-D power spectral density is modeled
as that of a fractal image, can yield much better results in
refocusing an image than is possible with deterministic models.
An example is shown in Fig. 1-46.
40
60
80
100
120
140
160
180
200
20
40
60
(a) Unfocused MRI image
80 100 120 140 160 180 200
(a) Image with missing pixels
20
40
60
80
100
(b) Deterministically refocused MRI image
120
140
160
180
200
20
40
60
80 100 120 140 160 180 200
(b) Inpainted image
Figure 1-45 The image in (b) was created by “filling in”
values for the missing pixels in the upper image. [Extracted
from Fig. 7-23.]
(c) Stochastically refocused MRI image
Figure 1-46 The images in (b) and (c) demonstrate two
methods used for refocusing an unfocused image. [Extracted
from Fig. 9-4.]
www.dbooks.org
32
CHAPTER 1 IMAGING SENSORS
1-6.10 Markov Random Fields for Image
Segmentation
In a Markov random field (MRF) image model, the value of each
pixel is stochastically related to its surrounding values. This
is useful in segmenting images, as presented in Section 9-11.
Figure 1-47 illustrates how an MRF image model can improve
the segmentation of an X-ray image of a foot into tissue and
bone.
1-6.11 Motion-Deblurring of a Color Image
Chapters 1–9 consider grayscale (black-and-white) images,
since color images consist of three (red, green, blue) images.
Motion deblurring of a color image is presented in Section 10-3,
an example of which is shown in Fig. 1-48.
(a) Motion-blurred Christmas tree
(a) Noisy image
(b) Segmented image
Figure 1-47 Segmenting a noisy image into two distinct
classes: bone and tissue. [Extracted from Fig. 9-14.]
(b) Deblurred Christmas tree
Figure 1-48 Deblurring a motion-blurred color image. [Extracted from Fig. 10-11.]
1-6 COMING ATTRACTIONS
33
1-6.12 Wavelet-Based Denoising of a Color
Image
1-6.13 Histogram Equalization (Brightening) of
a Color Image
Wavelet-based denoising can be used on each color component
of a color image, as presented in Section 10-4. An illustration is
given in Fig. 1-49.
Color images can be brightened using histogram equalization, as
presented in Section 10-5. Figure 1-50(a) displays a dark image
of a toucan, and part (b) of the same figure displays the result of
applying histogram equalization to each color.
(a) Noisy flag image
(a) Original dark toucan image
(b) Denoised image
Figure 1-49 Denoising an image of the American flag using
wavelet-based denoising. [Extracted from Fig. 10-12.]
(b) Brightened image
Figure 1-50 Application of histogram equalization to a color
image. [Extracted from Figs. 10-13 and 10-14.]
www.dbooks.org
34
CHAPTER 1 IMAGING SENSORS
1-6.14 Unsupervised Learning
1-6.15 Supervised Learning
In unsupervised learning, a set of training images is used to
determine a set of reference images, which are then used to
classify an observed image. The training images are mapped to a
subspace spanned by the most significant singular vectors of the
singular value decomposition of a training matrix. In Fig. 1-51,
the training images, depicted by blue “ ” symbols, cluster into
different image classes.
Supervised learning by neural networks is presented in Chapter
12. An example of a neural network is shown in Fig. 1-52.
0
1
2
3
4
5
6
7
8
9
u2
784 terminals
1.5
1
Class 1
0.5
0
−0.5
Class 2
Class 3
−1
−1.5
1
1.5
2
2.5
3
3.5
u1
Input terminals
Hidden layer
Output layer
Figure 1-52 A multilayer neural network.
Figure 1-51 Depiction of training images in 2-D subspace.
[Extracted from Fig. 11-13.]
1-6 COMING ATTRACTIONS
35
Summary
Concepts
• Color images are actually triplets of red, green, and blue
images, displayed together.
• The effect of an image acquisition system on an image
can usually be modelled as 2-D convolution with the
point spread function of the system (see below).
• The resolution of an image acquisition system can be
computed using various formulae (see below).
• Images may be formed using any of these imaging
modalities: optical, infrared, radar, x-rays, ultrasound,
and magnetic resonance imaging.
• Image processing is needed to process a raw image,
formed directly from data, into a final image, which has
been deblurred, denoised, interpolated, or enhanced, all
of which are subjects of this book.
Mathematical Formulae
Lens law
1
1
1
+ =
d0 di
f
X-ray tomography path attenuation
Optical point spread function
πD
2J1 (γ ) 2
, γ=
h(θ ) =
sin θ
γ
λ
SAR point spread function
2
h(x, y) = e−2.77(x/τ ) sinc2
MRI point spread function
1.8y
l
hx (x) = ∆k
2-D convolution
Ii (x, y) = Io (x, y) ∗ ∗ h(x, y) =
Z ∞Z ∞
−∞ −∞
p(r, θ ) =
Z ∞Z ∞
−∞ −∞
Optical resolution
Radar resolution
Ultrasound resolution
a(ξ , η ) δ (r − ξ cos θ − η sin θ ) d ξ d η
λ
∆θmin ≈ 1.22
D
λ
∆y′min ≈ R
D
λN
∆Rmin =
2
sin(π N ∆k x)
sin(π ∆k x)
Io (x − x′ , y − y′) h(x′ , y′ ) dx′ dy′
Important Terms
Provide definitions or explain the meaning of the following terms:
active pixel sensor
beamforming
charge-coupled device
infrared imaging
liquid crystal display
magnetic resonance imaging (MRI)
optical imaging
point spread function
radar
resolution
synthetic-aperture radar
ultrasound imaging
www.dbooks.org
X-ray computed tomography
36
PROBLEMS
Section 1-1: Optical Imagers
1.1 An imaging lens in a digital camera has a focal length of
6 cm. How far should the lens be from the camera’s CCD array
to focus on an object
(a) 12 cm in front of the lens?
(b) 15 cm in front of the lens?
1.2 An imaging lens in a digital camera has a focal length of
4 cm. How far should the lens be from the camera’s CCD array
to focus on an object
(a) 12 cm in front of the lens?
(b) 8 cm in front of the lens?
CHAPTER 1 IMAGING SENSORS
1.6 The following program loads an image stored in
sar.mat as Io (x, y), passes it through an imaging system with
the PSF given by Eq. (1.15), and displays Io (x, y) and Ii (x, y).
Parameters ∆, τ and l are specified in the program’s first line.
clear;Delta=0.1;l=5;tau=1;I=[-15:15];
z=pi*1.8*Delta*I/l;load sar.mat;
hy=sin(pi*z)./(pi*z);hy(16)=1;hy=hy.*hy;
hx=exp(-2.77*Delta*Delta*I.*I/tau/tau);
H=hy’*hx;Y=conv2(X,H);
figure,imagesc(X),axis off,colormap(gray),
figure,imagesc(Y),axis off,colormap(gray)
Run the program and display Io (x, y) (input) and Ii (x, y)
(output).
1.3 The following program loads an image stored in
clown.mat as Io (x, y), passes it through an imaging system Section 1-3: X-Ray Computed Tomography (CT)
with the PSF given by Eq. (1.6), and displays Io (x, y) and
Ii (x, y). Parameters ∆, D, di , and λ (all in mm) are specified in 1.7 (This problem assumes prior knowledge of the 1-D Fourier
the program’s first line.
transform (FT)). The basic CT problem is to reconstruct α (ξ , η )
clear;Delta=0.0002;D=0.03;
in Eq. (1.18) from p(r, θ ). One way to do this is as follows:
lambda=0.0000005;di=0.003;
(a) Take the FT of Eq. (1.18), transforming r to f . Define
T=round(0.01/Delta);
p(−r, θ ) = p(r, θ + π ).
for I=1:T;for J=1:T;
(b) Define and substitute µ = f cos θ and ν = f sin θ in this
x2y2(I,J)=(I-T/2).*(I-T/2)+(J-T/2).
FT.
(J-T/2);end;end;
*
gamma=pi*D/lambda*
(c) Show that the result defines 2 FTs, transforming ξ to µ and
sqrt(x2y2./(x2y2+di*di/Delta/Delta));
η to ν , and that A(µ , ν ) = P( f , θ ). Hence, α (ξ , η ) is the
h=2*besselj(1,gamma)./gamma;
inverse FT of P( f , θ ).
h(T/2,T/2)=(h(T/2+1,T/2)+h(T/2-1,T/2)
+h(T/2,T/2+1)+h(T/2,T/2-1))/4;
h=h.*h;H=h(T/2-5:T/2+5,T/2-5:T/2+5);
Section 1-4: Magnetic Resonance Imaging
load clown.mat;Y=conv2(X,H);
figure,imagesc(X),axis off,colormap(gray),
figure,imagesc(Y),axis off,colormap(gray) 1.8 The following program loads an image stored in
mri.mat as Io (x, y), passes it through an imaging system with
Run the program and display Io (x, y) (input) and Ii (x, y) the PSF given by Eq. (1.20), and displays Io (x, y) and Ii (x, y).
Parameters ∆, N, and dk are specified in the program’s first line.
(output).
clear;N=16;Delta=0.01;dk=1;
I=[-60:60];load mri.mat;
Section 1-2: Radar Imagers
h=dk*sin(pi*N*dk*I*Delta)./sin(pi*dk*I*Delta);
h(61)=N;H=h’*h;Y=conv2(X,H);
1.4 Compare the azimuth resolution of a real-aperture radar
figure,imagesc(X),axis off,colormap(gray),
with that of a synthetic-aperture radar, with both pointed at the
figure,imagesc(Y),axis off,colormap(gray)
ground from an aircraft at a range R = 5 km. Both systems
operate at λ = 3 cm and utilize a 2-m-long antenna.
Run the program and display Io (x, y) (input) and Ii (x, y)
1.5 A 2-m-long antenna is used to form a synthetic-aperture (output).
radar from a range of 100 km. What is the length of the synthetic
aperture?
PROBLEMS
37
Section 1-5: Ultrasound Imager
1.9 This problem shows how beamforming works on a linear
array of transducers, as illustrated in Fig. 1-35, in a medium
with a wave speed of 1540 m/s. We are given a linear array
of transducers located 1.54 cm apart along the x axis, with
the nth transducer located at x = 1.54n cm. Outputs {yn (t)}
from the transducers are delayed and summed to produce the
signal y(t) = ∑n yn (t − 0.05n). In what direction (angle from
perpendicular to the array) is the array focused?
www.dbooks.org
Chapter
2
2
Review of 1-D Signals
and Systems
Contents
x(t)
1.0
Trumpet signal
0.8
2-1
2-2
2-3
2-4
2-5
2-6
2-7
2-8
2-9
2-10
Overview, 39
Review of 1-D Continuous-Time Signals, 41
Review of 1-D Continuous-Time Systems, 43
1-D Fourier Transforms, 47
The Sampling Theorem, 53
Review of 1-D Discrete-Time Signals
and Systems, 59
Discrete-Time Fourier Transform (DTFT), 66
Discrete Fourier Transform (DFT), 70
Fast Fourier Transform (FFT), 76
Deconvolution Using the DFT, 80
Computation of Continuous-Time Fourier
Transform (CTFT) Using the DFT, 82
Problems, 86
Objectives
Learn to:
■ Compute the response of an LTI system to a given
input using convolution.
■ Compute the frequency response (response to a
sinusoidal input) of an LTI system.
■ Compute the continuous-time Fourier transform of a
signal or impulse response.
■ Use the sampling theorem to convert a continuoustime signal to a discrete-time signal.
■ Perform the three tasks listed above for continuoustime signals on discrete-time signals.
■ Use the discrete Fourier transform (DFT) to denoise,
filter, and deconvolve signals.
0.6
0.4
0.2
0
t (ms)
−0.2
−0.4
−0.6
−0.8
−1
0
1
2
3
4
(a) x(t)
5
6
7
|X( f )|
0.35
Magnitude spectrum of trumpet signal
0.30
0.25
0.20
0.15
0.10
0.05
0
0 500
1500
2500
3500
(b) |X( f )|
4500
f (Hz)
Many techniques and transforms in image processing are direct generalizations of techniques and
transforms in 1-D signal processing. Reviewing
these 1-D concepts enhances the understanding of
their 2-D counterparts. These include: linear
time-invariant (LTI) systems, convolution, frequency response, filtering, Fourier transforms for
continuous and discrete-time signals, and the
sampling theorem. This chapter reviews these 1-D
concepts for generalization to their 2-D counterparts
in Chapter 3.
Overview
Some topics and concepts in 1-D signals and systems generalize
directly to 2-D. This chapter provides quick reviews of those
topics and concepts in 1-D so as to simplify their repeat presentation in 2-D in future chapters. We assume the reader is already
familiar with 1-D signals and systems∗ —in both continuous and
discrete time, so the presentation in this chapter is more in the
form of a refresher than an extensive treatment. Moreover, we
limit the coverage to topics that generalize directly from 1-D
to 2-D. These topics include those listed in the box below.
Some topics that do not generalize readily from 1-D to 2-D
include: causality; differential and difference equations; transfer
functions; poles and zeros; Laplace and z-transforms. Hence,
these topics will not be covered in this book.
1-D Signals and Systems
2-D Signals and Systems
(1) Linear time-invariant (LTI) 1-D systems
(2) Frequency response of LTI systems
(3) Impulse response of 1-D systems
(4) 1-D filtering and convolution
(5) Sampling theorem in 1-D
(6) Discrete-time Fourier transform (DTFT)
Linear shift-invariant (LSI) 2-D systems
Spatial frequency response of LSI systems
Point-spread function of LSI systems
2-D filtering and convolution
Sampling theorem in 2-D
Discrete-space Fourier transform (DSFT)
∗ For a review, see Engineering Signals and Systems in Continuous and
Discrete Time, Ulaby and Yagle, NTS Press, 2016.
39
www.dbooks.org
40
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
For the sake of clarity, we start this chapter with a synopsis of
the terminology and associated symbols used to represent 1-D
continuous-time and discrete-time signals and 2-D continuousspace and discrete-space signals, and their associated spectra.
1-D Signals
Continuous Time
FT
x(t)
Signal in
time domain
X( f )
Spectrum in
frequency domain
Discrete Time
DTFT
x[n]
Signal at
discrete times
t = n∆
DFT
X(Ω)
Spectrum at
continuous
frequency Ω
X[k]
Spectrum at
2πk
discrete frequencies Ω =
N
0≤k≤N−1
2-D Images
Continuous Space
CSFT
f (x,y)
Image in
spatial domain
F(μ,ν)
Spectrum in
frequency domain
Discrete Space
DSFT
f [n,m]
Image in
discrete space
2-D DFT
F(Ω1,Ω2)
order N
Spectrum in
continuous
frequency domain
F[k1,k2]
Spectrum in
discrete frequency domain:
2πk1
2πk2
Ω1 =
, Ω2 =
N
N
2-1 REVIEW OF 1-D CONTINUOUS-TIME SIGNALS
41
2-1 Review of 1-D Continuous-Time
Signals
A continuous-time signal is a physical quantity, such as voltage
or acoustic pressure, that varies with time t, where t is a real
number having units of time (usually seconds). Mathematically
a continuous-time signal is a function x(t) of t. Although t can
also be a spatial variable, we will refer to t as time, to avoid
confusion with the spatial variables used for images.
The rectangle function rect(t) is defined as
(
1
for − 1/2 < t < 1/2,
rect(t) =
0
otherwise.
(2.3a)
The pulse x(t) defined in Eq. (2.2) can be written in terms of
rect(t) as
t − t0
x(t) = rect
(2.3b)
.
T
Rectangle Pulse
2-1.1 Fundamental 1-D Signals
In Table 2-1, we consider three types of fundamental,
continuous-time signals.
C. Impulses
An impulse δ (t) is defined as a function that has the sifting
property
A. Eternal Sinusoids
An (eternal) sinusoid with amplitude A, phase angle θ (radians),
and frequency f0 (Hz), is described by the function
x(t) = A cos(2π f0t + θ ),
−∞ < t < ∞.
Z ∞
−∞
(2.1)
The period of x(t) is T = 1/ f0 .
Even though an eternal sinusoid cannot exist physically (since
it would extend from before the beginning of the universe until
after its end), it is used nevertheless to mathematically describe
periodic signals in terms of their Fourier series. Also, another
useful aspect of eternal sinusoids is that the response of a linear
time-invariant LTI system (defined in Section 2-2.1) to an eternal sinusoid is another eternal sinusoid at the same frequency as
that of the input sinusoid, but with possibly different amplitude
and phase. Despite the fact that the sinusoids are eternal (and
therefore, unrealistic), they can be used to compute the response
of real systems to real signals. Consider, for example, a sinusoid
that starts at t = 0 as the input to a stable and causal LTI system.
The output consists of a transient response that decays to zero,
plus a sinusoid that starts at t = 0. The output sinusoid has the
same amplitude, phase, and frequency as the response that the
system would have had to an eternal input sinusoid.
B. Pulses
A (rectangular) pulse of duration T centered at time t0 (Table
2-1) is defined as
(
1
for (t0 − T /2) < t < (t0 + T /2),
x(t) =
(2.2)
0
otherwise.
x(t) δ (t − t0 ) dt = x(t0 ).
(2.4)
Sifting Property
◮ Multiplying a function x(t) that is continuous at t = t0 by
a delayed impulse δ (t −t0 ) and integrating over t “sifts out”
the value x(t0 ). ◭
Setting x(t) = 1 in Eq. (2.4) shows that an impulse has an area
of unity.
An impulse can be thought of (non-rigorously) as the limiting
case of a pulse of width T = 2ε multiplied by an amplitude
A = 1/(2ε ) as ε → 0:
t 1
.
rect
ε →0 2ε
2ε
δ (t) = lim
(2.5)
Because the width of δ (t) is the reciprocal of its amplitude
(Fig. 2-1), the area of δ (t) remains 1 as ε → 0.
The limit is undefined, but it is useful to think of an impulse
as the limiting case of a short duration and high pulse with unit
area. Also, the pulse shape need not be rectangular; a Gaussian
or sinc function can also be used.
Changing variables from t to t ′ = at yields the time scaling
property (Table 2-1) of impulses:
www.dbooks.org
δ (at) = δ (t)/|a|
(2.6)
42
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
Table 2-1 Types of signals and signal properties.
Types of Signals
x(t)
A cos (2πt /T )
A
Eternal sinusoid
x(t) = A cos(2π f0t + θ ), −∞ < t < ∞
−T
Pulse (rectangle)
Impulse δ (t)
t − t0
x(t) = rect
T
(
1
for (t0 − T /2) < t < (t0 + T /2),
=
0
otherwise
Z ∞
−∞
−T/2
0
T/2
rect
( )
T
1
t − t0
T
t (s)
t0
0
t
T
δ(t − t0)
1
x(t) δ (t − t0 ) dt = x(t0 )
0
t
t0
Properties
x(t)
Causal
x(t) = 0 for t < 0 (starts at or after t = 0)
t
0
6
x(t + 10)
Time delay by t0
x(t)
x(t − t0 )
2
0
−10 −6 0 4
10
Time scaling by a
Signal energy
x(t)
E=
x(t)
4
x(t − 10)
t (s)
10 14
y1(t) = x(2t)
x(t)
20
y2(t) = x(t / 2)
x(at)
Z ∞
−∞
1
|x(t)|2 dt
2
3
30
4
5
t
2-2 REVIEW OF 1-D CONTINUOUS-TIME SYSTEMS
The scaling property can be interpreted using Eq. (2.5) as
follows. For a > 1 the width of the pulse in Eq. (2.5) is
compressed by |a|, reducing its area by a factor of |a|, but its
height is unaltered. Hence the area under the pulse is reduced to
1/|a|.
Impulses are important tools used in defining the impulse
responses of 1-D systems and the point-spread functions of 2-D
spatial systems (such as a camera or an ultrasound), as well as
for deriving the sampling theorem.
43
A nonzero signal x(t) that is zero-valued outside the interval
[a, b] = {t : a ≤ t ≤ b},
(i.e., x(t) = 0 for t ∈
/ [a, b]), has support [a, b] and duration b − a.
Why does scaling time in an im-
Concept Question 2-1:
pulse also scale its area?
R∞
Exercise 2-1: Compute the value of −∞ δ (3t − 6) t 2 dt.
2-1.2 Properties of 1-D Signals
Answer: 43 . (See IP )
A. Time Delay
Delaying signal x(t) by t0 generates signal x(t −t0 ). If t0 > 0, the
waveform of x(t) is shifted to the right by t0 , and if t0 < 0, the
waveform of x(t) is shifted to the left by |t0 |. This is illustrated
by the time-delay figure in Table 2-1.
B. Time Scaling
A signal x(t) time-scaled by a becomes x(at). If a > 1, the
waveform of x(t) is compressed in time by a factor of a. If
0 < a < 1, the waveform of x(t) is expanded in time by a factor
of 1/a, as illustrated by the scaling figure in Table 2-1. If a < 0,
the waveform of x(t) is compressed by |a| or expanded by 1/|a|,
and then time-reversed.
Exercise 2-2: Compute
the energy of the pulse defined by
.
x(t) = 5 rect t−2
6
Answer: 150. (See IP )
2-2 Review of 1-D Continuous-Time
Systems
A continuous-time system is a device or mathematical model
that accepts a signal x(t) as its input and produces another signal
y(t) at its output. Symbolically, the input-output relationship is
expressed as
C. Signal Energy
x(t)
The energy E of a signal x(t) is
E=
Z ∞
−∞
|x(t)|2 dt.
(2.7)
SYSTEM
y(t)
Table 2-2 provides a list of important system types and properties.
2-2.1 Linear and Time-Invariant Systems
δ(t)
Systems are classified on the basis of two independent properties: (a) linearity and (b) time invariance, which leads to four
possible classes:
Area = 1
1
2ε
(1) Linear (L), but not time-invariant.
(2) Linear and time-invariant (LTI).
−ε
0
2ε
ε
t
Figure 2-1 Rectangular pulse model for δ (t).
(3) Nonlinear, but time-invariant (TI).
(4) Nonlinear and not time-invariant.
Most practical systems (including 2-D imaging systems such
a camera, ultrasound, radar, etc.) belong to one of the first two
www.dbooks.org
44
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
Table 2-2 Types of systems and associated properties.
Property
Definition
N
If xi (t)
Linear System (L)
L
N
yi (t), then ∑ ci xi (t)
∑ ci yi (t)
L
i=1
Time-Invariant (TI)
If x(t)
TI
Linear Time-Invariant (LTI)
If xi (t)
LTI
y(t), then x(t − τ )
i=1
TI
y(t − τ )
N
N
yi (t), then ∑ ci xi (t − τi )
LTI
i=1
δ (t − τ )
Impulse Response of LTI System
LTI
of the above four classes. If a system is moderately nonlinear,
it can be approximated by a linear model, and if it is highly
nonlinear, it may be possible to divide its input-output response
into a series of quasi-linear regions. In this book, we limit our
treatment to linear (L) and linear time-invariant (LTI) systems.
∑ ci yi (t − τi)
i=1
y(t) = h(t − τ )
exactly the same direction. That is, if
x(t)
TI
y(t),
x(t − τ )
TI
y(t − τ ).
then it follows that
A. Linear Systems
A system is linear (L) if its response to a linear combination
of input signals acting simultaneously is the same as the linear
combination of the responses to each of the input signals acting
alone. That is:
If
xi (t)
L
yi (t),
(2.8a)
(2.9)
for any input signal x(t) and constant time shift τ .
◮ Systems that are both linear and time-invariant are termed
linear time-invariant (LTI). ◭
then for any N inputs {xi (t), i = 1 . . . N} and any N constants
{ci , i = 1 . . . N},
N
∑ ci xi (t)
2-2.2 Impulse Response
N
L
i=1
∑ ci yi (t).
(2.8b)
i=1
Under mild assumptions (regularity) about the system, the finite
sum can be extended to infinite sums and integrals.
Linearity also is called the superposition property.
B. Time-Invariant System
A system is time-invariant (TI) if time shifting (delaying) the
input, time shifts the output by exactly the same amount and in
In general, we use the symbols x(t) and y(t) to denote, respectively, the input signal into a system and the resultant output
response. The term impulse response is used to denote the output response for the specific case when the input is an impulse.
For non–time-invariant systems, the impulse response depends
on the time at which the impulse is nonzero. The response to
the impulse δ (t) delayed by τ , which is δ (t − τ ), is denoted as
h(t; τ ). If the system is time-invariant, then delaying the impulse
merely delays the impulse response, so h(t; τ ) = h(t − τ ), where
h(t) is the response to the impulse δ (t). This can be summarized
2-2 REVIEW OF 1-D CONTINUOUS-TIME SYSTEMS
which is known as the convolution integral. Often, the convolution integral is represented symbolically by
in the following two equations:
δ (t − τ )
SYSTEM
h(t; τ ),
(2.10a)
and for a time-invariant system,
δ (t − τ )
45
x(t) ∗ h(t) =
h(t − τ ).
TI
LTI
x(t)
x(τ ) δ (t − τ ) d τ = x(t).
(2.11)
Next, if we multiply both sides of Eq. (2.10a) by x(τ ) and then
integrate τ over the limits (−∞, ∞), we obtain
−∞
x(τ ) δ (t − τ ) d τ
y(t) =
L
Z ∞
−∞
x(τ ) h(t; τ ) d τ .
(2.12)
Upon using Eq. (2.11) to replace the left-hand side of Eq. (2.12)
with x(t), Eq. (2.12) becomes
x(t)
y(t) =
L
(2.15)
y(t) = x(t) ∗ h(t).
(2.16)
The steps leading to Eq. (2.16) are summarized in Fig. 2-2.
The convolution in Eq. (2.15) is realized by time-shifting the
impulse response h(t). Changing variables from τ to t − τ shows
that convolution has the commutative property:
Upon interchanging t and t0 in the sifting property given by
Eq. (2.4) and replacing t0 with τ , we obtain the relationship
Z ∞
x(τ ) h(t − τ ) d τ .
(2.10b)
A. Linear System
−∞
−∞
Combining the previous results leads to the symbolic form
2-2.3 Convolution
Z ∞
Z ∞
Z ∞
−∞
x(τ ) h(t; τ ) d τ .
x(t) ∗ h(t) = h(t) ∗ x(t).
(2.17)
The expression for y(t) can also be derived by time-shifting the
input signal x(t) instead, in which case the result would be
y(t) = h(t) ∗ x(t) =
Z ∞
−∞
x(t − τ ) h(τ ) d τ .
(2.18)
Table 2-3 provides a summary of key properties of convolution
that are extendable to 2-D, and Fig. 2-3 offers a graphical
representation of how two of those properties—the associative
and distributive properties—are used to characterize the overall
impulse responses of systems composed of multiple systems,
when connected in series or in parallel, in terms of the impulse
responses of the individual systems.
(2.13)
Concept Question 2-2: What is the significance of a system being linear time-invariant?
This integral is called the superposition integral.
Concept Question 2-3: Why does delaying either of two
signals delay their convolution?
B. LTI System
For an LTI system, h(t; τ ) = h(t − τ ), in which case the expression for y(t) in Eq. (2.13) becomes
y(t) =
Z ∞
−∞
x(τ ) h(t − τ ) d τ ,
Convolution Integral
Exercise 2-3: Is the following system linear, time-invariant,
both, or neither?
dy
= 2x(t − 1) + 3tx(t + 1)
dt
(2.14)
Answer: System is linear but not time-invariant. (See IP )
www.dbooks.org
46
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
LTI System with Zero Initial Conditions
1.
δ (t)
LTI
y(t) = h(t)
2.
δ (t − τ )
LTI
y(t) = h(t − τ )
3.
x(τ ) δ (t − τ )
LTI
y(t) = x(τ ) h(t − τ )
x(τ ) δ (t − τ ) d τ
LTI
y(t) =
Z ∞
x(τ ) h(t − τ ) d τ
x(t)
LTI
y(t) =
Z ∞
x(τ ) h(t − τ ) d τ = x(t) ∗ h(t)
4.
Z ∞
−∞
5.
−∞
−∞
Figure 2-2 Derivation of the convolution integral for a linear time-invariant system.
Table 2-3 Convolution properties.
Convolution Integral
• Causal Systems and Signals:
y(t) = h(t) ∗ x(t) =
Z ∞
−∞
y(t) = h(t) ∗ x(t) = u(t)
h(τ ) x(t − τ ) d τ
Z t
0
h(τ ) x(t − τ ) d τ
Property
Description
1. Commutative
x(t) ∗ h(t) = h(t) ∗ x(t)
(2.19a)
2. Associative
[g(t) ∗ h(t)] ∗ x(t) = g(t) ∗ [h(t) ∗ x(t)]
(2.19b)
3. Distributive
x(t) ∗ [h1 (t) + · · · + hN (t)] = x(t) ∗ h1 (t) + · · · + x(t) ∗ hN (t)
(2.19c)
Z t
h(τ ) x(t − τ ) d τ
4. Causal ∗ Causal = Causal
y(t) = u(t)
5. Time-shift
h(t − T1 ) ∗ x(t − T2 ) = y(t − T1 − T2 )
(2.19e)
6. Convolution with Impulse
x(t) ∗ δ (t − T ) = x(t − T )
(2.19f)
0
(2.19d)
2-3 1-D FOURIER TRANSFORMS
h1(t)
x(t)
h2(t)
47
hN(t)
y(t)
Answer:
y(t) =
x(t)
h1(t) ∗ h2(t) ∗ … ∗ hN(t)
y(t)
(See
(
e−2t − e−3t
0
for t > 0,
for t < 0.
IP )
(a) In series
2-3 1-D Fourier Transforms
The continuous-time Fourier transform (CTFT) is a powerful
tool for
h1(t)
x(t)
h2(t)
y(t)
hN(t)
• computing the spectra of signals, and
• analyzing the frequency responses of LTI systems.
2-3.1 Definition of Fourier Transform
The 1-D Fourier transform X( f ) of x(t) and the inverse 1-D
Fourier transform x(t) of X( f ) are defined by the transformations
x(t)
h1(t) + h2(t) + … + hN(t)
y(t)
X( f ) = F {x(t)} =
(b) In parallel
and
Figure 2-3 (a) The overall impulse response of a system composed of multiple LTI systems connected in series is equivalent
to the cumulative convolution of the impulse responses of the
individual systems. (b) For LTI systems connected in parallel,
the overall impulse response is equal to the sum of the impulse
responses of the individual systems.
Z ∞
−∞
x(t) = F −1 {X( f )} =
x(t) e− j2π f t dt
Z ∞
−∞
X( f ) e j2π f t d f .
(2.20a)
(2.20b)
Throughout this book, variables written in boldface (e.g., X( f ))
denote vectors or complex-valued quantities.
A. Alternative Definitions of the Fourier
Transform
Exercise 2-4: Compute the output y(t) of an LTI system
with impulse response h(t) to input x(t), where
(
e−3t
for t > 0,
h(t) =
0
for t < 0,
Note that Eq. (2.20) differs slightly from the usual electrical
engineering definition of the Fourier-transform pair:
Xω (ω ) =
and
x(t) =
and
(
e−2t
x(t) =
0
for t > 0,
for t < 0.
1
2π
Z ∞
x(t) e− jω t dt
(2.21a)
Xω (ω ) e jω t d ω .
(2.21b)
−∞
Z ∞
−∞
Whereas Eq. (2.20) uses the oscillation frequency f (in Hz),
the definition given by Eq. (2.21) uses ω (in rad/s) instead,
where ω = 2π f . Using Hz makes interpretation of the Fourier
www.dbooks.org
48
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
transform as a spectrum—as well as the presentation of the
sampling theorem in Section 2-4—easier.
The definition of the Fourier transform used by mathematicians has a different sign for ω than the definition used by
electrical engineers:
Shifting:
X−ω (ω ) =
and
1
x(t) =
2π
Z ∞
−∞
Z ∞
x(t) e jω t dt
x(t − τ ) =
=
(2.22a)
Z ∞
−∞
Z ∞
−∞
−1
=F
−∞
X( f ) e j2π f (t−τ ) d f
X( f ) e j2π f t e− j2π f τ d f
{X( f ) e− j2π f τ }.
(2.24)
Modulation:
X−ω (ω ) e
− jω t
dω .
(2.22b)
e j2π f0t x(t) =
Geophysicists use different sign conventions for time and space!
In addition, some computer programs,√such as Mathematica,
split the 1/(2π ) factor into factors of 1/ 2π in both the forward
and inverse transforms.
=
Z ∞
X( f ) e j2π f t e j2π f0t d f
−∞
Z ∞
−∞
−1
=F
X( f ) e j2π ( f + f0 )t d f
{X( f − f0 )}.
(2.25)
Derivative:
◮ In this book, we use the definition of the Fourier transform given by Eq. (2.20) exclusively. ◭
d
dx(t)
=
dt
dt
=
B. Fourier Transform Notation
Throughout this book, we use Eq. (2.20) as the definition of the
Fourier transform, we denote the individual transformations by
F {x(t)} = X( f )
and
Z ∞
−∞
−1
=F
−∞
{( j2π f ) X( f )}.
X( f ).
2-3.2 Fourier Transform Properties
The major properties of the Fourier transform are summarized
in Table 2-4, and derived next.
x(at) =
−∞
X( f ) e
j2π f (at)
Z ∞
x(t) dt.
−∞
df
Z ∞
Z ∞
−∞
X( f ) d f .
The other properties in Table 2-4 follow readily from the
definition of the Fourier transform given by Eq. (2.20), except
for the convolution property, which requires a few extra steps of
algebra.
Parseval’s theorem states that
E=
X( f ) e j2π ( f a)t d(a f )/a
−∞
1
f
−1
.
X
=F
|a|
a
=
(2.26)
Zero frequency:
Setting f = 0 in Eq. (2.20a) leads to
x(0) =
Z ∞
df
Zero time:
Similarly, setting t = 0 in Eq. (2.20b) leads to
and we denote the combined bilateral pair by
Scaling:
For a 6= 0,
X( f ) e
j2π f t
X( f ) ( j2π f ) e j2π f t d f
X(0) =
F −1 {X( f )} = x(t),
x(t)
Z ∞
Z ∞
−∞
x(t) y∗ (t) dt =
Z ∞
−∞
X( f ) Y∗ ( f ) d f .
(2.27)
Setting y(t) = x(t) gives Rayleigh’s theorem (also commonly
known as Parseval’s theorem), which states that the energies of
(2.23)
2-3 1-D FOURIER TRANSFORMS
49
Table 2-4 Major properties of the Fourier transform.
Property
X( f ) = F [x(t)] =
x(t)
∑ ci xi (t)
1. Linearity
2. Time scaling
x(at)
3. Time shift
x(t − τ )
4. Frequency shift (modulation)
1
X
|a|
x(t) e− j2π f t dt
f
a
e− j2π f τ X( f )
X( f − f0 )
dx
dt
j2π f X( f )
6. Reversal
x(−t)
X(− f )
7. Conjugation
x∗ (t)
X∗ (− f )
8. Convolution in t
x(t) ∗ y(t)
X( f ) Y( f )
9. Convolution in f (multiplication in t)
x(t) y(t)
X( f ) ∗ Y( f )
10. Duality
−∞
∑ ci Xi ( f )
e j2π f0 t x(t)
x′ =
5. Time derivative
Z ∞
x(− f )
X(t)
Special FT Relationships
X(0) =
11. Zero frequency
x(0) =
12. Zero time
Z ∞
13. Parseval’s theorem
−∞
Z ∞
−∞
|x(t)|2 dt =
Z ∞
−∞
−∞
Z ∞
−∞
x(t) dt
X( f ) d f
x(t) y∗ (t) dt =
x(t) and X( f ) are equal:
E=
Z ∞
Z ∞
−∞
X( f ) Y∗ ( f ) d f
where the even component xe (t) and the odd component xo (t)
are formed from their parent signal x(t) as follows:
|X( f )|2 d f .
(2.28)
and
A. Even and Odd Parts of Signals
A signal x(t) can be decomposed into even xe (t) and odd x0 (t)
components:
x(t) = xe (t) + x0(t),
(2.29)
xe (t) = [x(t) + x∗ (−t)]/2
(2.30a)
xo (t) = [x(t) − x∗(−t)]/2.
(2.30b)
A signal is said to have even symmetry if x(t) = x∗ (−t), in
which case x(t) = xe (t) and xo (t) = 0. Similarly, a signal has
odd symmetry if x(t) = −x∗ (−t), in which case x(t) = xo (t) and
xe (t) = 0.
www.dbooks.org
50
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
B. Conjugate Symmetry
If x(t) is real-valued, then the following conjugate symmetry
relations hold:
X(− f ) = X∗ ( f ),
(2.31a)
and a frequency response
(
where X∗ ( f ) is the complex conjugate of X( f ),
eliminates all frequency components of X( f ) above fc .
X( f ) = − X(− f ) (phase is an odd function), (2.31b)
|X( f )| = |X(− f )| (magnitude is even),
(2.31c)
(2.31d)
real(X( f )) = real(X(−f)) (real part is even),
imag(X( f )) = −imag(X(− f )) (imaginary part is odd).
(2.31e)
The real and imaginary parts of the Fourier transform of a realvalued signal x(t) are the Fourier transforms of the even and
odd parts of x(t), respectively. So the Fourier transform of a
real-valued and even function is real-valued, and the Fourier
transform of a real-valued and odd function is purely imaginary:
x(t) is even
x(t) is odd
x(t) is real and even
x(t) is real and odd
X( f ) is real,
X( f ) is imaginary,
X( f ) is real and even,
X( f ) is imaginary and odd.
HLP ( f ) =
for | f | < fc ,
for | f | > fc ,
1
0
(2.33)
D. Sinc Functions
The impulse response of the ideal lowpass filter characterized
by Eq. (2.33) is
hLP (t) = F −1 {HLP ( f )}
=
Z fc
− fc
1e
j2π f t
df =
(
sin(2π fct)/π t
2 fc
for t 6= 0,
for t = 0.
(2.34)
The scientific literature contains two different, but both commonly used definitions for the sinc function:
(1) sinc(x) = sin(x)/x, and
(2) sinc(x) = sin(π x)/π x.
With either definition, sinc(0) = 1 since sin(x) ≈ x for x ≪ 1.
C. Filtering and Frequency Response
The following is a very important property of the Fourier
transform:
◮ The Fourier transform of a convolution of two functions
is equal to the product of their Fourier transforms:
x(t)
LTI
y(t) = h(t) ∗ x(t)
Hence, per the definition given by Eq. (2.35), the impulse
response of the ideal lowpass filter is given by
hLP (t) = 2 fc sinc(2 fc t).
implies that
Y( f ) = H( f ) X( f ).
◮ Throughout this book, we use the sinc function definition
(
sin(π x)/π x
for x 6= 0,
sinc(x) =
(2.35)
1
for x = 0.
(2.32)
The function H( f ) = F {h(t)} is called the frequency response
of the system. The relationship described by Eq. (2.32) defines
the frequency filtering process performed by the system. At a
given frequency f0 , frequency component X( f0 ) of the input is
multiplied by H( f0 ) to obtain the frequency component Y( f0 )
of the output.
For example, an ideal lowpass filter with cutoff frequency fc
(2.36)
2-3.3 Fourier Transform Pairs
Commonly encountered Fourier transform pairs are listed in
Table 2-5. Note the duality between entries #1 and #2, #4 and
#5, and #6 and itself.
2-3.4 Interpretation of the Fourier Transform
A Fourier transform can be interpreted in three ways:
(1) as the frequency response of an LTI system,
2-3 1-D FOURIER TRANSFORMS
51
Table 2-5 Examples of Fourier transform pairs. Note that constant a ≥ 0.
|x(t)|
X( f ) = F [x(t)]
|X( f )|
BASIC FUNCTIONS
δ(t)
1
1a.
δ (t)
1
1
t
f
1
1b.
δ (t − τ )
t
τ
1
2.
1
1
e− j2π f τ
f
δ(f)
1
t
f
1
3.
e−a|t| , a > 0
t
2/a
2a
(2π f )2 + a2
f
T
1
4.
−T
2
T
2
t
rect(t/T )
T sinc( f T )
−1
T
T
5.
−1
f0
6.
1
1
f0
f
1
T
1
f
f0 sinc( f0t)
e−π t
2
f
− 0
2
rect( f / f0 )
e−π f
f0
2
f
1
2
f
t
B. Spectrum
(2) as the spectrum of a signal, and
(3) as the energy spectral density of a signal.
The spectrum of x(t) is X( f ). Consider, for example, the eternal
sinusoid defined by Eq. (2.1):
A. Frequency Response
x(t) = A cos(2π f0t + θ )
A
A
= e jθ e j2π f0t + e− jθ e− j2π f0t ,
2
2
The frequency response H( f ) of an LTI system is the frequency
domain equivalent of the system’s impulse response h(t):
H( f )
h(t).
(2.37)
where we used the relation cos(x) = (e jx + e− jx )/2. From the
properties and pairs listed in Tables 2-4 and 2-5, we have
X( f ) =
www.dbooks.org
A jθ
A
e δ ( f − f0 ) + e− jθ δ ( f + f0 ).
2
2
(2.38)
52
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
Strictly speaking, the Fourier transform of an eternal sinusoid is
undefined, since an eternal sinusoid is not absolutely integrable.
Nevertheless, this example provides a convenient illustration
that the spectrum of a sinusoid at f0 is concentrated entirely
at ± f0 . By extension, the spectrum of a constant signal is
concentrated entirely at f = 0.
Real signals are more complicated than simple sinusoids, as
are their corresponding spectra. Figure 2-4(a) displays 7 ms
of a trumpet playing note B, and part (b) of the same figure
displays the corresponding spectrum. The spectrum is concentrated around narrow spectral lines located at 491 Hz and the
next 6 harmonics. In contrast, speech exhibits a much broader
spectrum, as illustrated by the examples in Fig. 2-5.
x(t)
1.0
Trumpet signal
0.8
0.6
0.4
0.2
t (ms)
0
−0.2
−0.4
C. Energy Spectral Density
A more rigorous interpretation of X( f ) that avoids impulses in
frequency uses the concept of energy spectral density. Let the
spectrum of a signal x(t) be
(
constant
for f0 < f < f0 + ε ,
X( f ) =
0
otherwise,
(
X( f0 )
for f0 < f < f0 + ε ,
(2.39)
≈
0
otherwise.
The approximation becomes exact in the limit as ε → 0, provided X( f ) is continuous at f = f0 .
Using Rayleigh’s theorem, the energy of x(t) is
E=
Z ∞
−∞
|X( f )|2 d f ≈ |X( f0 )|2 ε .
(2.40)
The energy of x(t) in the interval f0 < f < f0 + ε is |X( f0 )|2 ε .
The energy spectral density at frequency f = f0 (analogous to
probability density or to mass density of a physical object) is
|X( f0 )|2 .
Note that a real-valued x(t) will, by conjugate symmetry, also
have nonzero X( f ) in the interval − f0 > f > − f0 − δ . So the
bilateral energy spectral density of x(t) at f0 is 2|X( f0 )|2 .
Concept Question 2-4:
Provide three applications of the
Fourier transform.
Concept Question 2-5:
sinc function.
Provide an application of the
−0.6
−0.8
−1
0
1
2
3
4
(a) x(t)
5
6
7
|X( f )|
0.35
Magnitude spectrum of trumpet signal
0.30
0.25
0.20
0.15
0.10
0.05
0
f (Hz)
0 500
1500
2500
3500
(b) |X( f )|
4500
Figure 2-4 Trumpet signal (note B) and its magnitude spectrum.
2-4 THE SAMPLING THEOREM
53
d
Exercise 2-6: Compute the Fourier transform of dt
[sinc(t)].
Sound magnitude
Answer:
“oo” as in “cool”
F
(See
1
(
d
j2π f
sinc(t) =
dt
0
for | f | < 0.5,
for | f | > 0.5.
IP )
f (kHz)
2
3
(a) “oo” spectrum
2-4 The Sampling Theorem
The sampling theorem is an operational cornerstone of both
discrete-time 1-D signal processing and discrete-space 2-D
image processing.
Sound magnitude
2-4.1 Sampling Theorem Statement
“ah” as in “Bach”
The samples {x(n∆)} of a signal x(t) sampled every ∆ seconds
are
{x(n∆), n = . . . , −2, −1, 0, 1, 2, . . .}.
(2.41)
1
f (kHz)
2
3
(b) “ah” spectrum
Figure 2-5 Spectra of two vowel sounds.
The inverse of the sampling interval ∆ is the sampling rate
S = 1/∆ samples per second. The sampling rate has the same
dimension as Hz, and is often expressed in “Hz.” For example,
the standard sampling rate for CDs is 44100 samples/second,
often stated as 44100 Hz. The corresponding sampling interval
is ∆ = 1/44100 s = 22.676 µ s.
A signal x(t) is bandlimited to a maximum frequency of
B (extending from −B to B), measured in Hz, if its Fourier
transform X( f ) = 0 for | f | > B. Although real-world signals
are seldom truly bandlimited, their spectra are often negligible
above some frequency B.
◮ The sampling theorem states that if
Exercise 2-5: A square wave x(t) has the Fourier series
expansion
1
1
1
x(t) = sin(t) + sin(3t) + sin(5t) + sin(7t) + · · ·
3
5
7
Compute output y(t) if
x(t)
h(t) = 0.4 sinc(0.4t)
Answer: y(t) = sin(t). (See IP )
X( f ) = 0 for | f | > B,
and if
x(t) is sampled at a sampling rate of S samples/s,
then
y(t).
x(t) can be reconstructed exactly from {x(n∆),
n = . . . , −2, −1, 0, 1, 2, . . .}, provided S > 2B.
The sampling rate must exceed double the maximum frequency
in the spectrum X( f ) of x(t). The minimum (actually an infi-
www.dbooks.org
54
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
mum) sampling rate 2B samples/second is called the Nyquist
rate, and the frequency 2B is called the Nyquist frequency.
Dividing by ∆ and recalling that S = 1/∆ gives
∞
S ∑ X( f − kS).
xs (t)
(2.47)
k=−∞
2-4.2 Sampling Theorem Derivation
A. The Sampled Signal xs (t)
Given a signal x(t), we construct the sampled signal xs (t) by
multiplying x(t) by the impulse train
The spectrum Xs ( f ) of xs (t) consists of a superposition of copies
of the spectrum X( f ) of x(t), repeated every S = 1/∆ and
multiplied by S. If these copies do not overlap in frequency,
we may then recover X( f ) from Xs ( f ) using a lowpass filter,
provided S > 2B [see Fig. 2-6(a)].
X s( f )
∞
δs (t) =
∑ δ (t − n∆).
(2.42)
n=−∞
−B − S
Copy
That is,
B−S
−B
0
S X(0)
B
S−B
(a) S > 2B
Copy
S+B
f
∞
xs (t) = x(t) δs (t) =
∑ x(t) δ (t − n∆)
X s( f )
n=−∞
∞
=
∑ x(n∆) δ (t − n∆).
(2.43)
−B − S
n=−∞
B − S0S − B
−B
S+B
f
B
(b) S < 2B
B. Spectrum of the sampled signal xs (t)
Using Fourier series, it can be shown that the Fourier transform
of the impulse train δs (t) is itself an impulse train in frequency:
∞
Figure 2-6 Sampling a signal x(t) with maximum frequency B at
a rate of S makes X( f ) change amplitude to S X( f ) and to repeat
in f with period S. These copies (a) do not overlap if S > 2B, but
(b) they do if S < 2B.
∞
∆ ∑ δ (t − n∆)
n=−∞
∑ δ ( f − k/∆).
(2.44)
k=−∞
2-4.3 Aliasing
This result can be interpreted as follows. A periodic signal has
a discrete spectrum (zero except at specific frequencies) given
by the signal’s Fourier series expansion. By Fourier duality, a
discrete signal (zero except at specific times) such as xs (t) has
a periodic spectrum. So a discrete and periodic signal such as
δs (t) has a spectrum that is both discrete and periodic.
Multiplying Eq. (2.44) by x(t), using the definition for xs (t)
given by Eq. (2.43), and applying property #9 in Table 2-4 leads
to
If the sampling rate S does not exceed 2B, the copies of X( f )
will overlap one another, as shown in Fig. 2-6(b). This is called
an aliased condition, the consequence of which is that the
reconstructed signal will no longer match the original signal
x(t).
Example 2-1:
Two Sinusoids and Aliasing
∞
xs (t) ∆
X( f ) ∗ ∑ δ ( f − k/∆),
(2.45)
k=−∞
Two signals, a 2 Hz sinusoid and a 12 Hz sinusoid:
which, using property #6 of Table 2-3, simplifies to
x1 (t) = cos(4π t)
∞
xs (t) ∆
∑ X( f − k/∆).
k=−∞
(2.46)
and
x2 (t) = cos(24π t),
2-4 THE SAMPLING THEOREM
55
B. Sinc Interpolation Formula
were sampled at 20 samples/s. Generate plots for
(a) x1 (t) and its sampled version x1s (t),
(b) x2 (t) and its sampled version x2s (t),
(c) Spectra X1 ( f ) and X1s ( f ), and
(d) Spectra X2 ( f ) and X2s ( f ).
Solution: (a) Figure 2-7(a) displays x1 (t) and x1s (t), with the
latter generated by sampling x1 (t) at S = 20 samples/s. The
applicable bandwidth of x1 (t) is B1 = 2 Hz. Since S > 2B1 , it
should be possible to reconstruct x1 (t) from x1s (t), which we
demonstrate in a later subsection.
Similar plots are displayed in Fig. 2-7(b) for the 12 Hz
sinusoid. In this latter case, B = 12 Hz and S = 20 samples/s.
Hence, S < 2B.
(b) Spectrum X1 ( f ) of x1 (t) consists of two impulses at
±2 Hz, as shown in Fig. 2-8(a). The spectrum of the sampled
version consists of the same spectrum X1 ( f ) of x1 (t), scaled by
the factor S, plus additional copies repeated every ±S = 20 Hz
(Fig. 2-8(b). Note that the central spectrum in (Fig. 2-8(b),
corresponding to S X1 ( f ), does not overlap with the neighboring
copies.
(c) Spectra X2 ( f ) and X2s ( f ) are shown in Fig. 2-9. Because
S < 2B, the central spectrum overlaps with its two neighbors.
Mathematically, we can use an ideal lowpass filter with a cutoff
frequency anywhere between B and S − B. It is customary to use
S/2 as the cutoff frequency, since it is halfway between B and
S − B, so as to provide a safety margin for avoiding aliasing if
the actual maximum frequency of X( f ) exceeds B but is less
than S/2. The frequency response of this ideal lowpass filter is,
from Eq. (2.48),
(
1/S
for | f | < S/2,
H( f ) =
(2.49)
0
for | f | > S/2.
Setting f0 = S in entry #4 of Table 2-5, the impulse response is
found to be
h(t) = (1/S)[S sinc(St)] = sinc(St).
(2.50)
Using the convolution property x(t) ∗ δ (t − τ ) = x(t − τ ) [see
property #6 in Table 2-3], we can derive the following sinc
interpolation formula:
x(t) = xs (t) ∗ h(t)
∞
=
∑ x(n∆) δ (t − n∆) ∗ h(t)
n=−∞
∞
=
∑ x(n∆) sinc(S(t − n∆)).
(2.51)
n=−∞
2-4.4 Sampling Theorem Implementation
A. Physical Lowpass Filter
If S > 2B, the original signal x(t) can be recovered from
the sampled signal xs (t) by subjecting the latter to a lowpass
filter that passes frequencies below B with a gain of 1/S (to
compensate for the factor of S induced by sampling (as noted
in Eq. (2.47)) and rejects frequencies greater than (S − B) Hz:
(
1/S
for | f | < B,
H( f ) =
(2.48)
0
for | f | > S − B.
This type of filter must be implemented using a physical
circuit. For example, a Butterworth filter can be constructed
by connecting op-amps, capacitors and resistors in a series of
Sallen-Key configurations.† This is clearly impractical for image
processing.
In principle, this formula can be used to reconstruct x(t) for
any time t from its samples {x(n∆)}. But since it requires an
infinite number of samples {x(n∆)} to reconstruct x(t), it is of
theoretical interest only.
C. Reconstruction of X( f ) from Samples {x(n∆)}
According to Eq. (2.43)), the sampled signal xs (t) is given by
∞
xs (t) =
∑ x(n∆) δ (t − n∆).
(2.52)
n=−∞
Application of property #6 in Table 2-5 yields
† Ulaby and Yagle, Signals and Systems: Theory and Applications, pp. 296–
297.
∞
Xs ( f ) =
∑ x(n∆) e− j2π f n∆.
n=−∞
www.dbooks.org
(2.53)
56
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
1
x1(t)
0.8
0.6
x1s(t)
0.4
0.2
t (s)
0
−0.2
−0.4
−0.6
−0.8
−1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
(a) 2 Hz sinusoid x1(t) and its sampled version x1s(t)
1
x2(t)
0.8
0.6
0.4
x2s(t)
0.2
t (s)
0
−0.2
−0.4
−0.6
−0.8
−1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
(b) 12 Hz sinusoid x2(t) and its sampled version x2s(t)
Figure 2-7 Plots of (a) x1 (t) and x1s (t) and (b) x2 (t) and x2s (t). Sampling rate S = 20 samples/s.
2-4 THE SAMPLING THEOREM
57
X1( f )
X1( f ) of 2 Hz sinusoid
0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
−30
−20
−10
0
10
20
30
f (Hz)
(a) X1( f ) of 2 Hz sinusoid
X1s( f )
10
X1s( f ) for 2 Hz sinusoid sampled at 20 Hz. Note no aliasing.
S X1( f )
9
Copy of S X1( f )
at −20 Hz
8
Copy of S X1( f )
at +20 Hz
7
6
5
4
3
2
1
0
−30
−20
−10
0
S = 20 Hz
10
20
30
f (Hz)
S = 20 Hz
(b) Spectrum X1s( f ) of sampled signal x1s(t)
Figure 2-8 Spectra (a) X1 ( f ) and (b) X1s ( f ) of the 2 Hz sinusoid and its sampled version, respectively. The spectrum X1s ( f ) consists of
X( f ) scaled by S = 20, plus copies thereof at integer multiples of ±20 Hz. The vertical axes denote areas under the impulses.
www.dbooks.org
58
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
X2( f )
X2( f ) of 12 Hz sinusoid
0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
−30
−20
−10
0
10
20
30
f (Hz)
(a) Spectrum X2( f ) of 12 Hz sinusoid
X2s( f )
10
X2s( f ) for 12 Hz sinusoid sampled at 20 Hz. Note the presence of aliasing.
9
8
Copy of S X2( f )
at −20 Hz
7
6
Copy of S X2( f )
at +20 Hz
S X2( f )
5
Overlap
4
Overlap
3
2
1
0
−30
−20
−10
S = 20 Hz
0
10
S = 20 Hz
20
30
f (Hz)
(b) Spectrum X2s( f ) of sampled signal x2s(t)
Figure 2-9 Spectra (a) X2 ( f ) and (b) X2s ( f ) of the 12 Hz sinusoid and its sampled version. Note the overlap in (b) between the spectrum
of X2s ( f ) and its neighboring copies. The vertical axes denote areas under the impulses.
2-5 REVIEW OF 1-D DISCRETE-TIME SIGNALS AND SYSTEMS
Note that Xs ( f ) is periodic in f with period 1/∆, as it should be.
In the absence of aliasing,
1
X( f ) = Xs ( f )
S
59
reconstruction x̂1 (t), we apply Eq. (2.57) with ∆ = 1/S = 0.05 s:
X̂1 ( f ) = X1s ( f ) ∆ sinc(∆ f ).
for | f | < S/2.
(2.54)
The relationship given by Eq. (2.53)) still requires an infinite
number of samples {x(n∆)} to reconstruct X( f ) at each frequency f .
D. Nearest-Neighbor (NN) Interpolation
A common procedure for computing an approximation to x(t)
from its samples {x(n∆)} is the nearest neighbor interpolation.
The signal x(t) is approximated by x̂(t):


x(n∆)
x̂(t) = x((n + 1)∆)

 ..
.
for (n − 0.5)∆ < t < (n + 0.5)∆,
for (n + 0.5)∆ < t < (n + 1.5)∆,
..
.
(2.55)
So x̂(t) is a piecewise-constant approximation to x(t), and it is
related to the sampled signal xs (t) by
x̂(t) = xs (t) ∗ rect(t/∆).
(2.56)
Using the Fourier transform of a rectangle function (entry #4 in
Table 2-5), the spectrum X̂( f ) of x̂(t) is
X̂( f ) = Xs ( f ) ∆ sinc(∆ f ),
(2.57)
where Xs ( f ) is the spectrum of the sampled signal. The zerocrossings of the sinc function occur at frequencies f = k/∆ = kS
for integers k. These are also the centers of the copies of the
original spectrum X( f ) induced by sampling. So these copies
are attenuated if the maximum frequency B of X( f ) is such that
B ≪ S. The factor ∆ in Eq. (2.57) cancels the factor S = 1/∆ in
Eq. (2.47).
Example 2-2: Reconstruction of 2 Hz Sinusoid
For the 2 Hz sinusoid of Example 2-1: (a) plot spectrum X̂1 ( f )
of the approximated reconstruction x̂1 (t), and (b) apply nearestneighbor interpolation to generate x̂1 (t).
Solution: (a) Spectrum X1s ( f ) of the sampled version of the
2 Hz sinusoid was generated earlier in Example 2-1 and displayed in Fig. 2-8(b). To obtain the spectrum of the approximate
The sinc function is displayed in Fig. 2-10(a) in red and uses the
vertical scale on the right-hand side, and the spectrum X̂1 ( f ) is
displayed in blue using the vertical scale on the left-hand side.
The sinc function preserves the spectral components at ±2 Hz,
but attenuates the components centered at ±20 Hz by a factor
of 10 (approximately).
(b) Application of Eq. (2.56) to x1 (n∆) = cos(4π n∆) with
∆ = 1/20 s yields plot x̂1 (t) shown in Fig. 2-10(b).
Concept Question 2-6: Why must the sampling rate of a
signal exceed double its maximum frequency, if it is to be
reconstructed from its samples?
Concept Question 2-7: Why does nearest-neighbor interpolation work as well as it does?
Exercise 2-7: What is the Nyquist sampling rate for a signal
bandlimited to 4 kHz?
Answer: 8000 samples/s. (See IP )
Exercise 2-8: A 500 Hz sinusoid is sampled at 900
samples/s. No anti-alias filter is being used. What is the
frequency of the reconstructed continuous-time sinusoid?
Answer: 400 Hz. (See IP )
2-5 Review of 1-D Discrete-Time
Signals and Systems
Through direct generalizations of the 1-D continuous-time definitions and properties of signals and systems presented earlier,
we now extend our review to their discrete counterparts.
2-5.1 Discrete-Time Notation
A discrete-time signal is a physical quantity—such as voltage or
acoustic pressure—that varies with discrete time n, where n is a
dimensionless integer. Mathematically, a discrete-time signal is
a function x[n] of discrete time n.
www.dbooks.org
60
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
ˆ 1( f )
X
sinc(0.05 f )
0.5
1.0
0.45
0.9
0.4
0.8
0.35
sinc(0.05 f )
ˆ 1( f )
X
0.3
0.7
0.6
Right scale
Left scale
0.25
0.5
0.2
0.4
0.15
0.3
0.1
0.2
0.05
0.1
0
0
−30
−20
−10
0
10
20
30
f (Hz)
(a) Spectrum Xˆ 1( f ) of 2 Hz signal x1(t). The sinc function is shown in red.
1
0.8
0.6
0.4
0.2
0
−0.2
−0.4
−0.6
−0.8
−1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
(b) 2 Hz sinusoidal signal reconstructed from samples at 20 Hz,
using nearest neighbor interpolation. The 2 Hz sinusoid is shown in red.
Figure 2-10 Plots of Example 2-2.
1
2-5 REVIEW OF 1-D DISCRETE-TIME SIGNALS AND SYSTEMS
61
property of impulses still holds, with a summation replacing the
integral:
4
3
2
1
∞
∑ x[i] δ [n − i] = x[n].
(2.61)
i=−∞
−2
−1
0
1
2
3
2-5.2 Discrete-Time Eternal Sinusoids
n
A discrete-time eternal sinusoid is defined as
Figure 2-11 Stem plot representation of x[n].
x[n] = A cos(Ω0 n + θ ),
• Discrete-time signals x[n] use square brackets,
whereas continuous-time signals x(t) use parentheses.
• t has units of seconds, while n is dimensionless.
can be depicted using either the bracket notation
x[n] = {3, 2, 0, 4},
(2.59)
where the underlined value is the value at time n = 0, or in the
form of the stem plot shown in Fig. 2-11.
The support of this x[n] is the interval [−1, 2], and its
duration is 2 − (−1) + 1 = 4. In general, a discrete-time signal
with support [a, b] has duration b − a + 1.
A discrete-time (Kronecker) impulse δ [n] is defined as
(
1
for n = 0,
δ [n] = {1} =
(2.60)
0
for n 6= 0.
−∞ < t < ∞,
(2.63)
it is apparent that a discrete-time sinusoid can be viewed as a
continuous-time sinsoid sampled every ∆ seconds, at a sampling
rate of S = 1/∆ samples/s. Thus,
for n = {. . . , −2, −1, 0, 1, 2, . . .}.
Discrete-time signals are often represented using bracket notation, and plotted using stem plots. For example, the discrete-time
signal x[n] defined by


3
for n = −1,


2
for n = 0,
x[n] =
(2.58)

4
for
n = 2,



0
for all other n,
(2.62)
where Ω0 is the discrete-time frequency with units of radians
per sample, so it is dimensionless.
Comparing the discrete-time eternal sinusoid to the
continuous-time eternal sinusoid given by Eq. (2.1), which we
repeat here as
x(t) = A cos(2π f0t + θ ),
Discrete-time signals x[n] usually result from sampling a
continuous-time signal x(t) at integer multiples of a sampling
interval of ∆ seconds. That is,
• x[n] = x(n∆)
−∞ < n < ∞,
Ω0 = 2π f0 ∆ = 2π f0 /S,
(2.64)
which confirms that Ω0 , like n, is dimensionless. However,
almost all discrete-time eternal sinusoids are nonperiodic! In
fact, x[n] is periodic only if
2π
N
= ,
Ω0
D
(2.65)
with N/D being a rational number. In such a case, the fundamental period of the sinusoid is N, provided N/D has been
reduced to lowest terms.
Example 2-3:
Discrete Sinusoid
Compute the fundamental period of
x[n] = 3 cos(0.3π n + 2).
Solution: From the expression for x[n], we deduce that
Ω0 = 0.3π . Hence,
Unlike the continuous-time impulse, the discrete-time impulse
has no issues about infinite height and zero width. The sifting
www.dbooks.org
2π
2π
20
=
=
Ω0
0.3π
3
62
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
reduced to lowest terms. Therefore, the period is N = 20.
convolution
Another important property of discrete-time eternal sinusoids
is that the discrete-time frequency Ω0 is periodic, which is not
true for continuous-time sinusoids. For any integer k, Eq. (2.62)
can be rewritten as
−∞ < n < ∞
x[n] = A cos(Ω0 n + θ ),
= A cos((Ω0 + 2π k)n + θ ),
= A cos(Ω′0 n + θ ),
with
−∞ < n < ∞,
Ω′0 = Ω0 + 2π k.
(2.66)
(2.67)
Also, the nature of the variation of x[n] with n has a peculiar
dependence on Ω0 . Consider, for example, the sinusoid
−∞ < n < ∞,
x[n] = cos(Ω0 n),
(2.68)
where, for simplicity, we assigned it an amplitude of 1 and
a phase angle θ = 0. Next, let us examine what happens as
we increase Ω0 from a value slightly greater than zero to 2π .
Initially, as Ω0 is increased, x[n] oscillates faster and faster, until
x[n] reaches a maximum rate of oscillation at Ω0 = π , namely
n
x[n] = cos(π n) = (−1)
at (Ω0 = π ).
(2.69)
At Ω0 = π , x[n] oscillates as a function of n between (−1) and
(+1). As Ω0 is increased beyond π , oscillation slows down and
then stops altogether when Ω0 reaches 2π :
x[n] = cos(2π n) = 1
at (Ω0 = 2π ).
(2.70)
Beyond Ω0 = 2π , the oscillatory behavior starts to increase
again, and so on. This behavior has no equivalence in the world
of continuous-time sinusoids.
∞
y[n] = h[n] ∗ x[n] = ∑ h[i] x[n − i].
Discrete-time convolution
Most of the continuous-time properties of convolution also
apply in discrete time.
Real-world signals and filters are defined over specified
ranges of n (and set to zero outside those ranges). If h[n] has
support in the interval [n1 , n2 ], Eq. (2.71a) becomes
n2
y[n] = h[n] ∗ x[n] = ∑ h[i] x[n − i].
Reversing the sequence of h[n] and x[n] leads to the same
outcome. That is, if x[n] has support in the interval [n3 , n4 ], then
n4
y[n] = h[n] ∗ x[n] = ∑ x[i] h[n − i].
SYSTEM
y[n].
The definition of LTI for discrete-time systems is identical to
the definition of LTI for continuous-time systems. If a discretetime system has impulse response h[n], then the output y[n]
can be computed from the input x[n] using the discrete-time
(2.71c)
i=n3
◮ The duration of the convolution of two signals of durations N1 and N2 is Nc = N1 + N2 − 1, not N1 + N2 . Since
h[n] is of length N1 = n2 − n1 + 1 and x[n] is of length
N2 = n4 − n3 + 1, the length of the convolution y[n] is
Nc = N1 + N2 − 1 = (n2 − n1) + (n4 − n3 ) + 1. ◭
For causal signals (x[n] and h[n] equal to zero for n < 0), y[n]
assumes the form
n
y[n] = ∑ x[i] h[n − i],
n ≥ 0.
(2.71d)
{1, 2} ∗ {3, 4} = {3, 10, 8}.
(2.72)
Causal
2-5.3 1-D Discrete-Time Systems
x[n]
(2.71b)
i=n1
i=0
A 1-D discrete-time system accepts an input x[n] and produces
an output y[n]:
(2.71a)
i=−∞
For example,
The duration of the output is 2 + 2 − 1 = 3.
2-5.4 Discrete-Time Convolution Properties
With one notable difference, the properties of the discrete-time
convolution are the same as those for continuous time. If (t)
2-5 REVIEW OF 1-D DISCRETE-TIME SIGNALS AND SYSTEMS
63
Table 2-6 Comparison of convolution properties for continuous-time and discrete-time signals.
Property
Continuous Time
y(t) = h(t) ∗ x(t) =
Definition
Z ∞
−∞
Discrete Time
∞
y[n] = h[n] ∗ x[n] = ∑ h[i] x[n − i]
h(τ ) x(t − τ ) d τ
i=−∞
1. Commutative
x(t) ∗ h(t) = h(t) ∗ x(t)
x[n] ∗ h[n] = h[n] ∗ x[n]
2. Associative
[g(t) ∗ h(t)] ∗ x(t) = g(t) ∗ [h(t) ∗ x(t)]
[g[n] ∗ h[n]]∗ x[n] = g[n] ∗ [h[n] ∗ x[n]]
3. Distributive
x(t) ∗ [h1(t) + · · · + hN (t)] =
x(t) ∗ h1(t) + · · · + x(t) ∗ hN (t)
x[n] ∗ [h1 [n] + · · · + hN [n]] =
x[n] ∗ h1[n] + · · · + x[n] ∗ hN [n]
Z t
n
y[n] = u[n] ∑ h[i] x[n − i]
h(τ ) x(t − τ ) d τ
4. Causal ∗ Causal = Causal
y(t) = u(t)
5. Time-Shift
h(t − T1 ) ∗ x(t − T2 ) = y(t − T1 − T2 )
h[n − a] ∗ x[n − b] = y[n − a − b]
6. Sampling
x(t) ∗ δ (t − T ) = x(t − T )
x[n] ∗ δ [n − a] = x[n − a]
7. Width
width y(t) = width x(t) + width h(t)
width y[n] =
width x[n] + width h[n] − 1
!
0
i=0
∞
∑ y[n] =
area of y(t) = area of x(t) × area of h(t)
8. Area
9. Convolution with Step
y(t) = x(t) ∗ u(t) =
Z t
−∞
is replaced with [n] and integrals are replaced with sums, the
convolution properties listed in Table 2-3 lead to those listed in
Table 2-6.
The notable difference is associated with property #7. In
discrete time, the width (duration) of a signal that is zero-valued
outside interval [a, b] is b − a + 1, not b − a. Consider two
signals, h[n] and x[n], defined as follows:
Signal
From
To
Duration
h[n]
x[n]
y[n]
a
c
a+c
b
d
b+d
b−a+1
d−c+1
(b + d) − (a + c) + 1
where y[n] = h[n] ∗ x[n]. Note that the duration of y[n] is
(b + d) − (a + c) + 1 = (b − a + 1) + (d − c + 1) − 1
= duration h[n] + duration x[n] − 1.
n=−∞
∞
∑ h[n]
n=−∞
∞
!
∑ x[n]
n=−∞
n
x[n] ∗ u[n] = ∑ x[i]
x(τ ) d τ
i=−∞
2-5.5 Delayed-Impulses Computation Method
For finite-duration signals, computation of the convolution sum
can be facilitated by expressing one of the signals as a linear
combination of delayed impulses. The process is enabled by the
sampling property (#6 in Table 2-6).
Consider, for example, the convolution sum of the two signals
x[n] = {2, 3, 4} and h[n] = {5, 6, 7}, namely
y[n] = x[n] ∗ h[n] = {2, 3, 4} ∗ {5, 6, 7}.
The sampling property allows us to express x[n] in terms of
impulses,
x[n] = 2δ [n] + 3δ [n − 1] + 4δ [n − 2],
which leads to
www.dbooks.org
y[n] = (2δ [n] + 3δ [n − 1] + 4δ [n − 2])∗ h[n]
= 2h[n] + 3h[n − 1] + 4h[n − 2].
64
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
2
Given that both x[n] and h[n] are of duration = 3, the duration
of their sum is 3 + 3 − 1 = 5, and it extends from n = 0
to n = 4. Computing y[0] using the delayed-impulses method
(while keeping in mind that h[i] has a non-zero value for only
i = 0, 1, and 2) leads to
y[3] = ∑ x[i] h[3 − i]
i=1
= x[1] h[2] + x[2] h[1] = 3 × 7 + 4 × 6 = 45,
2
y[4] = ∑ x[i] h[4 − i] = x[2] h[2] = 4 × 7 = 28,
y[0] = 2h[0] + 3h[−1] + 4h[−2]
= 2 × 5 + 3 × 0 + 4 × 0 = 10.
The process can then be repeated to obtain the values of y[n] for
n = 1, 2, 3, and 4.
i=2
y[n] = 0, otherwise.
Hence,
y[n] = {10, 27, 52, 45, 28}.
(b) The convolution sum can be computed graphically
through a four-step process.
Example 2-4: Discrete-Time Convolution
Given x[n] = {2, 3, 4} and h[n] = {5, 6, 7}, compute
y[n] = x[n] ∗ h[n]
by (a) applying the sum definition and (b) graphically.
Solution: (a) Both signals have a length of 3 and start at time
zero. That is, x[0] = 2, x[1] = 3, x[2] = 4, and x[i] = 0 for all
other values of i. Similarly, h[0] = 5, h[1] = 6, h[2] = 7, and
h[i] = 0 for all other values of i.
By Eq. (2.71d), the convolution sum of x[n] and h[n] is
Step 1: Replace index n with index i and plot x[i] and h[−i],
as shown in Fig. 2-12(a). Signal h[−i] is obtained from h[i] by
reflecting it about the vertical axis.
Step 2: Superimpose x[i] and h[−i], as in Fig. 2-12(b), and
multiply and sum them. Their product is 10.
Step 3: Shift h[−i] to the right by 1 to obtain h[1 − i], as
shown in Fig. 2-12(c). Multiplication and summation of x[i] by
h[1 − i] generates y[1] = 27. Shift h[1 − i] by one more unit to
the right to obtain h[2 − i], and then repeat the multiplication
and summation process to obtain y[2]. Continue the shifting and
multiplication and summation processes until the two signals no
longer overlap.
n
y[n] = x[n] ∗ h[n] = ∑ x[i] h[n − i].
Step 4: Use the values of y[n] obtained in step 3 to generate a
plot of y[n], as shown in Fig. 2-12(g);
Since h[i] = 0 for all values of i except i = 0, 1, and 2, it follows
that h[n − i] = 0 for all values of i except for i = n, n − 1, and
n − 2. With this constraint in mind, we can apply Eq. (2.71d) at
discrete values of n, starting at n = 0:
y[n] = {10, 27, 52, 45, 28}.
i=0
0
y[0] = ∑ x[i] h[0 − i] = x[0] h[0] = 2 × 5 = 10,
i=0
1
y[1] = ∑ x[i] h[1 − i]
i=0
= x[0] h[1] + x[1] h[0] = 2 × 6 + 3 × 5 = 27,
2
y[2] = ∑ x[i] h[2 − i]
i=0
= x[0] h[2] + x[1] h[1] + x[2] h[0]
= 2 × 7 + 3 × 6 + 4 × 5 = 52,
Concept Question 2-8:
Why are most discrete-time si-
nusoids not periodic?
Concept Question 2-9: Why is the length of the convolution of two discrete-time signals not equal to the sum of
the lengths of the two signals?
Exercise 2-9: A 28 Hz sinusoid is sampled at 100 samples/s. What is Ω0 for the resulting discrete-time sinusoid?
What is the period of the resulting discrete-time sinusoid?
Answer: Ω0 = 0.56π ; N = 25. (See IP )
2-5 REVIEW OF 1-D DISCRETE-TIME SIGNALS AND SYSTEMS
8
8
n=0
6
−3
−2
−1
3
2
0
1
2
3
4
5
i
6
−3
(a) x[i] and h[−i]
−2
−1
h[−i]
n=0
0
1
3
4
h[1 − i]
5
6
i
n=1
6
4
4
x[i]
2
−2
−1
0
1
2
3
x[i]
2
4
5
6
i
−3
−2
−1
0
1
(b)
2
3
4
5
6
i
(c)
y[2] = 2 × 7 + 3 × 6 + 4 × 5 = 52
8
h[2 − i]
y[3] = 3 × 7 + 4 × 6 = 45
8
n=2
6
h[3 − i]
n=3
6
4
4
x[i]
2
−3
2
y[1] = 2 × 6 + 3 × 5 = 27
8
6
−3
h[−i]
4
4
y[0] = 2 × 5 = 10
8
n=0
6
5
4
2
7
6
x[i]
2
65
−2
−1
0
1
2
2
3
4
5
6
i
−3
x[i]
−2
−1
0
1
(d)
3
4
6
i
y[n]
n=4
6
60
h[4 − i]
4
52
45
40
x[i]
2
5
(e)
y[4] = 4 × 7 = 28
8
2
28
27
20
10
−3
−2
−1
0
1
2
3
4
5
6
i
−1
(f )
0
1
2
(g) y[n]
Figure 2-12 Graphical computation of convolution sum.
www.dbooks.org
3
4
n
66
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
Exercise 2-10: Compute the output y[n] of a discrete-time
LTI system with impulse response h[n] and input x[n], where
h[n] = { 3, 1 } and x[n] = { 1, 2, 3, 4 }.
(a) m 6= n
Evaluation of the integral in Eq. (2.74) leads to
Answer: { 3, 7, 11, 15, 4 }. (See IP )
1
2π
Z π
−π
e jΩ(m−n) dΩ =
=
2-6 Discrete-Time Fourier Transform
(DTFT)
e jΩ(m−n)
2π j(m − n)
π
−π
π
(m−n)
−
j
− e jπ (m−n)
e
j2π (m − n)
(−1)m−n − (−1)m−n
=
j2π (m − n)
= 0,
(m 6= n).
The discrete-time Fourier transform (DTFT) is the discretetime counterpart to the Fourier transform. It has the same two
functions: (1) to compute spectra of signals and (2) to analyze
the frequency responses of LTI systems.
(2.75)
(b) m = n
2-6.1 Definition of the DTFT
If m = n, the integral reduces to
The DTFT of x[n], denoted X(Ω), and its inverse are defined as
1
2π
∞
X(Ω) =
∑ x[n] e− jΩn
(2.73a)
Z π
(2.73b)
n=−∞
and
x[n] =
1
2π
−π
X(Ω) e jΩn dΩ.
Readers familiar with the Fourier series will recognize that
the DTFT X(Ω) is a Fourier series expansion with x[n] as the
coefficients of the Fourier series. The inverse DTFT is simply
the formula used for computing the coefficients x[n] of the
Fourier series expansion of the periodic function X(Ω).
We note that the DTFT definition given by Eq. (2.73a) is
the same as the formula given by Eq. (2.53) for computing the
spectrum Xs ( f ) of a continuous-time signal x(t) directly from
its samples {x(n∆)}, with Ω = 2π f ∆.
The inverse DTFT given by Eq. (2.73b) can be derived as
follows. First, we introduce the orthogonality property
1
2π
Z π
−π
e jΩ(m−n) dΩ = δ [m − n].
(2.74)
To establish the validity of this property, we consider two cases,
namely: (1) when m 6= n and (2) when m = n.
Z π
−π
e jΩ(n−n) dΩ =
1
2π
Z π
−π
1 dΩ = 1.
(2.76)
The results given by Eqs. (2.75) and (2.76) can be combined into
the definition of the orthogonality property given by Eq. (2.74).
Having verified the validity of the orthogonality property, we
now use it to derive Eq. (2.73b). Upon multiplying the definition
for the DTFT given by Eq. (2.73a) by 21π e jΩm and integrating
over Ω, we have
1
2π
Z π
−π
X(Ω) e jΩm dΩ =
1
2π
Z π
∞
∑ x[n] e jΩ(m−n) dΩ
−π
n=−∞
∞
1
=
x[n]
∑
2π n=−∞
∞
=
Z π
−π
e jΩ(m−n) dΩ
∑ x[n] δ [m − n] = x[m]. (2.77)
n=−∞
Equation (2.74) was used in the final step leading to Eq. (2.77).
Exchanging the order of integration and summation in Eq. (2.77)
is acceptable if the summand is absolutely summable; i.e., if the
DTFT is defined. Finally, replacing the index m with n in the top
left-hand side and bottom right-hand side of Eq. (2.77) yields
the inverse DTFT expression given by Eq. (2.73b).
2-6 DISCRETE-TIME FOURIER TRANSFORM (DTFT)
2-6.2 Properties of the DTFT
◮ The DTFT X(Ω) is periodic with period 2π . ◭
The DTFT can be regarded as the Fourier transform of the
sampled signal xs (t) with a sampling interval ∆ = 1:
)
(
∞
∑ x[n] δ (t − n) .
X(Ω) = F
67
(2.78)
We also note the following special relationships between x[n]
and X(Ω):
n=−∞
This statement can be verified by subjecting Eq. (2.73a) to the
time-shift property of the Fourier transform (#3 in Table 2-4).
Consequently, most (but not all) of the Fourier transform properties listed in Table 2-4 extend directly to the DTFT with 2π f
replaced with Ω, which we list here in Table 2-7. The exceptions
mostly involve the following property of the DTFT:
∞
∑ x[n],
X(0) =
x[0] =
1
2π
and
n=−∞
Z π
(2.79a)
X(Ω) dΩ,
(2.79b)
∑ (−1)n x[n].
(2.79c)
−π
∞
X(±π ) =
n=−∞
If x[n] is real-valued, then conjugate symmetry holds:
Table 2-7 Properties of the DTFT.
Property
X(Ω)∗ = X(−Ω).
X(Ω)
x[n]
1. Linearity
∑ ci xi [n]
∑ ci Xi (Ω)
2. Time shift
x[n − n0]
X(Ω) e− jn0 Ω
3. Modulation
x[n] e jΩ0 n
X(Ω − Ω0)
4. Time reversal
x[−n]
X(−Ω)
5. Conjugation
x∗ [n]
X∗ (−Ω)
6. Time
convolution
h[n] ∗ x[n]
Special DTFT Relationships
X∗ (Ω) = X(−Ω)
8. Zero
frequency
X(0) = ∑∞
n=−∞ x[n]
x[0] =
9. Zero time
10. Ω = ±π
X(±π ) =
∞
11. Rayleigh’s
(often called
Parseval’s)
theorem
1
2π
Z π
−π
∞
n
∑ (−1) x[n]
n=−∞
1
∑ |x[n]|2 = 2π −π |X(Ω)|2 dΩ
n=−∞
∑ |x[n]|2 =
1
2π
Z π
−π
|X(Ω)|2 dΩ
(2.80)
The energy spectral density is now 21π |X(Ω)|2 .
Finally, by analogy to continuous time, a discrete-time ideal
lowpass filter with cutoff frequency Ω0 has the frequency
response for |Ω| < π (recall that H(Ω) is periodic with period π )
(
1
for |Ω| < Ω0 ,
H(Ω) =
(2.81)
0
for Ω0 < |Ω| ≤ π ,
which eliminates frequency components of x[n] that lie in the
range Ω0 < |Ω| ≤ π .
X(Ω) dΩ
Z π
∞
n=−∞
H(Ω) X(Ω)
7. Conjugate
symmetry
Parseval’s theorem for the DTFT states that the energy of x[n]
is identical, whether computed in the discrete-time domain n or
in the frequency domain Ω:
2-6.3 Important DTFT Pairs
For easy access, several DTFT pairs are provided in Table 2-8.
In all cases, the expressions for X(Ω) are periodic with period
2π , as they should be.
Entries #7 and #8 of Table 2-8 deserve more discussion,
which we now present.
www.dbooks.org
68
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
Table 2-8 Discrete-time Fourier transform (DTFT) pairs.
x[n]
X(Ω)
1
1.
δ [n]
1a.
δ [n − m]
2.
1
Condition
e− jmΩ
m = integer
∞
2π
∑ δ (Ω − 2π k)
k=−∞
∞
3.
e jΩ0 n
2π
∑ δ (Ω − Ω0 − 2π k)
k=−∞
∞
4.
cos(Ω0 n)
π
∑ [δ (Ω − Ω0 − 2π k) + δ (Ω + Ω0 − 2π k)]
k=−∞
5.
sin(Ω0 n)
6.
an cos(Ω0 n + θ ) u[n]
7.
8.
rect
hni
N
Ω0
sin(Ω0 n)
Ω0
sinc
n =
π
π
πn
π ∞
∑ [δ (Ω − Ω0 − 2π k) − δ (Ω + Ω0 − 2π k)]
j k=−∞
e j2Ω cos θ − ae jΩ cos(Ω0 − θ )
e j2Ω − 2ae jΩ cos Ω0 + a2
sin Ω N + 21
sin Ω2
∞
Ω − 2π k
∑ rect 2Ω0
k=−∞
A. Discrete-Time Sinc Functions
|a| < 1
Ω 6= 2π k
by
hFIR [n] = h[n] hHam [n]

π n Ω0
Ω0


sinc
n
,
0.54
+
0.46
cos

π
π
N }
|
{z
{z
}
|
=
hHam [n]

h[n]


0,
The impulse response of an ideal lowpass filter is
Z Ω0
Ω0
Ω0
sinc
n .
1e jΩn dΩ =
h[n] =
π
π
−Ω0
(2.82)
This is called a discrete-time sinc function. A discrete-time sinc
function h[n] with Ω0 = π /4 is displayed in Fig. 2-13(a), along
with its frequency response H(Ω) in Fig. 2-13(b). Such a filter
is impractical for real-world applications, because it is unstable
and it has infinite duration. To override these limitations, we
can multiply h[n] by a window function, such as a Hamming
window. The modified impulse response hFIR [n] is then given
|n| ≤ N,
|n| > N.
(2.83)
As can be seen in Fig. 2-13(c) and (d), hFIR [n] with N = 10
provides a good approximation to an ideal lowpass filter, with
a finite duration. The Hamming-windowed filter belongs to a
group of filters called finite-impulse response (FIR) filters. FIR
filters can also be designed using a minimax criterion, resulting
in an equiripple filter. This and other FIR filter design procedures are discussed in discrete-time signal processing textbooks.
2-6 DISCRETE-TIME FOURIER TRANSFORM (DTFT)
69
h[n]
H(Ω)
Impulse response
0.25
0.20
Frequency response
1.5
1
0.10
0
n
−0.10
−20 −15 −10 −5
0
5
10
15
20
0
−π −Ω00 Ω0
−2π
Ω
π
2π
(b) Ideal lowpass filter spectrum H(Ω) with Ω0 = π/4
(a) Impulse response h[n]
hFIR[n]
Impulse response
0.25
0.20
0.5
HFIR(Ω)
Frequency response
1.5
1
0.10
0
−0.05
−10 −5
0.5
n
0
5
0
−π
10
(c) Impulse response of Hamming-windowed filter
π
0
(d) Spectrum HFIR(Ω) of Hamming-windowed filter
Figure 2-13 Parts (a) and (b) are for an ideal lowpass filter with Ω0 = π /4, and parts (c) and (d) are for the same filter after multiplying its
impulse response with a Hamming window of length N = 10.
with r = e jΩ , the summation in Eq. (2.85) becomes
B. Discrete Sinc Functions
We note that rect Nn has duration 2N + 1. This differs from the
continuous-time rect
rect(t/T ), which has duration T .
function
The DTFT of rect Nn is obtained from Eq. (2.73a) by setting
x[n] = 1 and limiting the summation to the range (−N, N):
N
2N
n=−N
n=0
∑ e− jΩn = e− jΩN ∑ e jΩn
A discrete-time rectangle function rect Nn is defined as
(
hni
1
for |n| ≤ N,
rect
=
(2.84)
N
0
for |n| > N.
1 − e jΩ(2N+1)
1 − e jΩ
sin((2N + 1)Ω/2)
=
.
sin(Ω/2)
= e− jΩN
(2.87)
This is called a discrete (or periodic) sinc function. A rectangular pulse with N = 10 is shown in Fig. 2-14 along with its DTFT.
N
DTFT{rect[n/N]} =
∑ e− jΩn.
(2.85)
n=−N
Why does the DTFT share so
many properties with the CTFT?
Concept Question 2-10:
Using the formula
N
∑ rk =
k=0
1 − rN+1
1−r
Concept Question 2-11:
(2.86)
frequency?
www.dbooks.org
Why is the DTFT periodic in
70
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
x[n]
X(Ω) and X[k]
1.5
1
0.5
0
−15 −10
−5
0
5
10
(a) x[n] = rect(n/10)
15
n
14
12
10
8
6
4
2
X[k]
X(Ω)
0
1
2
3
4
5
6
Ω
Figure 2-15 The DFT X[k] is a sampled version of the DTFT
X(Ω).
X(Ω)
21
20
15
10
5
0
Exercise 2-12: Compute the inverse DTFT of
4 cos(2Ω) + 6 cos(Ω) + j8 sin(2Ω) + j2 sin(Ω).
Ω
−π
−2π
0
π
2π
Answer:
Figure 2-14 Discrete-time rectangle function with N = 10 and
DTFT−1 [4 cos(2Ω) + 6 cos(Ω) + j8 sin(2Ω) + j2 sin(Ω)]
= { 6, 4, 0, 2, −2 }.
its DTFT.
(See
(b) X(Ω)
IP )
2-7 Discrete Fourier Transform (DFT)
The discrete Fourier transform (DFT) is the numerical bridge
between the DTFT and the fast Fourier transform (FFT). For a
signal {x[n], n = 0, . . . , M − 1} of duration M, its DFT of order
N is X[k], where X[k] is X(Ω) sampled at the N frequencies
Ω = {2π k/N, k = 0, . . . , N − 1}:
Exercise 2-11: Compute the DTFT of 4 cos(0.15π n + 1).
Answer:
DTFT[4 cos(0.15π n + 1)]
∞
=
∑ 4π e j1[δ (Ω − 0.15 − 2kπ )]
k=−∞
∞
+ ∑ 4π e− j1[δ (Ω + 0.15 − 2kπ )].
k=−∞
(See IP )
X[k] = X(Ω = 2π k/N),
k = 0, . . . , N − 1.
(2.88)
An example is shown in Fig. 2-15. Usually N = M; i.e., the order
N of the DFT is equal to the duration M of x[n]. However, in
some situations, it is desirable to select N to be larger than M.
Fro example, to compute and plot the DTFT of a short-duration
signal such as x[n] = { 3, 1, 4 }, for which M = 3, we may choose
N to be 256 or 512 so as to produce a smooth plot, such as the
blue plot in Fig. 2-15. Choosing N > M will also allow the use of
the FFT to compute convolutions quickly (see Section 2-7.2C).
As a result, the properties of the DFT follow those of the
DTFT, with some exceptions, such as time reversal and cyclic
convolution (discussed later in Section 2-7.2C).
2-7 DISCRETE FOURIER TRANSFORM (DFT)
71
◮ To avoid confusion between the DTFT and the DFT, the
DFT, being a discrete function of integer k, uses square
brackets, as in X[k], while the DTFT, being a continuous
and periodic function of real numbers Ω, uses round parentheses, as in X(Ω). ◭
by N1 e j2π mk/N and summing over k gives
1 N−1 M−1
1 N−1
j2π mk/N
=
X[k]
e
∑
∑ ∑ x[n] e j2π (m−n)k/N
N k=0
N k=0
n=0
=
N−1
1 M−1
x[n]
∑ e j2π (m−n)k/N
∑
N n=0
k=0
M−1
= ∑ x[n] δ [m − n]
n=0
=
(
x[m]
0
for 0 ≤ m ≤ M − 1,
for M ≤ m ≤ N − 1.
(2.91)
Upon changing index m to n on the left-hand side of Eq. (2.91)
and in the right-hand side in x[m], we obtain the formal definition
of the inverse DFT given by Eq. (2.89b).
The main use of the DFT is to compute spectra of signals and
frequency responses of LTI systems. All plots of spectra in this
book (and all other books on signal and image processing) were
made by computing them using the DFT and plotting the results.
2-7.1 Definition of the DFT
The N-point (or Nth-order) DFT of {x[n], n = 0, . . . , M − 1},
denoted X[k], and the inverse DFT of X[k], namely x[n], are
defined as
M−1
X[k] = ∑ x[n] e− j2π nk/N ,
n=0
k = 0, . . . , N − 1,
(2.89a)
n = 0, . . . , M − 1.
(2.89b)
Example 2-5:
DFT of Periodic Sinusoids
and
x[n] =
1 N−1
∑ X[k] e j2π nk/N ,
N k=0
The definition for X[k] given by Eq. (2.89a) is obtained by
applying Eq. (2.88) to Eq. (2.73a), namely by replacing Ω with
2π k/N and limiting the range of summation over n to [0, M − 1].
For the inverse DFT, the definition given by Eq. (2.89b) can
be derived by following a process similar to that we presented
earlier in Section 2-6.1 in connection with the inverse DTFT.
Specifically, we start with the discrete equivalent of the orthogonality property given by Eq. (2.74):
1 N−1 j2π (m−n)k/N
= δ [m − n].
∑e
N k=0
Compute the N-point DFT of the segment of a periodic discretetime sinusoid
x[n] = A cos(2π (k0 /N)n + θ ),
(2.92)
with k0 a fixed integer.
Solution: We start by rewriting x[n] as the sum of two exponentials:
A jθ j2π k0 n/N A − jθ − j2π k0 n/N
+ e e
e e
2
2
A jθ j2π k0 n/N A − jθ j2π (N−k0 )n/N
= e e
+ e e
,
2
2
x[n] =
(2.90)
Next, multiplying the definition of the DFT given by Eq. (2.89a)
0 ≤ n ≤ N − 1,
(2.93)
where we have multiplied the second term in the first step by
e j2π N/N = 1.
www.dbooks.org
72
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
Inserting Eq. (2.93) into Eq. (2.89a) with M = N, we have
N−1
A jθ j2π k0 n/N − j2π nk/N
e
e e
n=0 2
X[k] = ∑
Table 2-9
Properties of the DFT. In the time-shift and
modulation properties, (n − n0 ) and (k − k0 ) must be reduced
mod(N).
Property
N−1
A − jθ j2π (N−k0 )n/N − j2π nk/N
e
e e
n=0 2
+∑
N−1
=
A jθ
e ∑ e j2π n(k0 −k)/N
2
n=0
+
A − jθ N−1 j2π n(N−k0 −k)/N
.
e
∑e
2
n=0
A jθ
A
e N δ [k − k0] + e− jθ N δ [N − k − k0],
2
2
which can be restated as

N
jθ

 2 Ae
X[k] = N2 Ae− jθ

0
∑ ci xi [n]
∑ ci Xi [k]
2. Time shift
x[n − n0 ]
e− j2π n0 k/N X[k]
e j2π k0 n/N x[n]
X[k − k0 ]
x[N − n]
X[N − k]
5. Conjugation
x∗ [n]
6. Convolution
h[n] c x[n]
H[k] X[k]
4. Time reversal
In view of the orthogonality property given by Eq. (2.90), the
summations simplify to impulses, resulting in
X[k] =
1. Linearity
3. Modulation
(2.94a)
X[k]
x[n]
X∗ [N − k]
Special DFT Relationships
7. Conjugate
symmetry
X∗ [k] = X[N − k]
8. Zero frequency
X[0] = ∑ x[n]
9. Zero time
1 N−1
x[0] =
∑ X[k]
N n=0
10. k = N/2
X[N/2] = ∑ (−1)n x[n]
(2.94b)
N−1
for k = k0 ,
for k = N − k0 ,
otherwise.
n=0
(2.94c)
N−1
Thus, the DFT of a segment of a periodic sinusoid with
Ω0 = 2π k0/N consists of two discrete-time impulses, at indices
k = k0 and k = N − k0 .
n=0
N−1
11. Parseval’s
theorem
2
1 N−1
∑ |x[n]| = N ∑ |X[k]|2
n=0
k=0
2-7.2 Properties of the DFT
Table 2-9 provides a summary of the salient properties of the
DFT, as well as some of the special relationships between x[n]
and X[k]. Of particular note are the three relationships
A. Conjugate Symmetry Property of the DFT
If x[n] is real-valued, then conjugate symmetry holds for the
DFT, which takes the form
N−1
X[0] = ∑ x[n],
(2.95)
1
∑ X[k],
N n=0
(2.96)
n=0
N−1
x[0] =
and
n=0
k = 1, . . . , N − 1.
For example, the 4-point DFT of x[n] = { 1, 2, 3, 4 } is
X[k] = { 10, −2 + j2, −2, −2 − j2 }
and
N−1
X[N/2] = ∑ (−1)n x[n] for N even.
X∗ [k] = X[N − k],
(2.97)
X∗ [1] = −2 − j2 = X[4 − 1] = X[3] = −2 − j2.
Similarly,
X∗ [2] = X[4 − 2] = X[2] = −2,
(2.98)
2-7 DISCRETE FOURIER TRANSFORM (DFT)
73
which is real-valued. This conjugate-symmetry property follows
from the definition of the DFT given by Eq. (2.89a):
N−1
X∗ [k] = ∑ x[n] e j2π nk/N
where (n − n1 )N means (n − n1 ) reduced mod N (i.e., reduced
by the largest integer multiple of N without (n − n1 ) becoming
negative).
(2.99)
n=0
and
N−1
N−1
n=0
n=0
X[N − k] = ∑ x[n] e− j2π n(N−k)/N = ∑ x[n] e− j2π n e j2π k/N .
(2.100)
Since n is an integer, e− j2π n = 1 and Eq. (2.100) reduces to
N−1
X[N − k] = ∑ e j2π nk/N = X∗ [k].
n=0
B. Use of DFT for Convolution
The convolution property of the DTFT extends to the DFT after
some modifications. Consider two signals, x1 [n] and x2 [n], with
N-point DFTs X1 [k] and X2 [k]. From Eq. (2.89b), the inverse
DFT of their product is
2π
1 N−1
(X1 [k] X2 [k])e jk N n
DFT−1 (X1 [k] X2 [k]) =
∑
N k=0
"
#
2π
1 N−1 jk 2π n N−1
−
jk
n
1
=
∑ e N ∑ x1 [n1] e N
N k=0
n1 =0
#
"
N−1
·
2π
∑ x2[n2 ] e− jk N n2 .
(2.101)
C. DFT and Cyclic Convolution
Because of the mod N reduction cycle, the expression on the
right-hand side of Eq. (2.103) is called the cyclic or circular
convolution of signals x1 [n] and x2 [n]. The terminology helps
distinguish it from the traditional linear convolution of two
nonperiodic signals.
The symbol commonly used to denote cyclic convolution is c .
Combining Eqs. (2.101) and (2.103) leads to
N−1
yc [n] = x1 [n] c x2 [n] = ∑ x1 [n1 ] x2 [(n − n1)N ]
n1 =0
= DFT−1 (X1 [k] X2 [k])
=
y[n] = x1 [n] ∗ x2[n]
Rearranging the order of the summations gives
(2.102)
z[0] = y[0] + y[0 + N]
z[1] = y[1] + y[1 + N]
..
.
z[N − 2] = y[N − 2] + y[2N − 2]
z[N − 1] = y[N − 1].
In view of the orthogonality property given by Eq. (2.90),
Eq. (2.102) reduces to
DFT−1 (X1 [k] X2 [k])
1 N−1 N−1
∑ x1 [n1] x2[n2 ] N δ [(n − n1 − n2)N ]
N n∑
1 =0 n2 =0
N−1
= ∑ x1 [n1 ] x2 [(n − n1)N ],
(2.105)
is of duration 2N − 1, extending from n = 0 to n = 2N − 2.
Aliasing y[n] means defining z[n], the aliased version of y[n],
as
DFT−1 (X1 [k] X2 [k]) =
=
(2.104)
The cyclic convolution yc [n] can certainly by computed by
applying Eq. (2.104), but it can also be computed from the
linear convolution x1 [n]∗ x2[n] by aliasing the latter. To illustrate,
suppose x1 [n] and x2 [n] are both of duration N. The linear
convolution of the two signals
n2 =0
N−1
2π
1 N−1 N−1
x1 [n1 ] x2 [n2 ] ∑ e jk N (n−n1 −n2 ) .
∑
∑
N n1 =0 n2 =0
k=0
2π
1 N−1
∑ X1 [k] X2 [k] e jk N n .
N k=0
(2.106)
The aliasing process leads to the result that z[n] is the cyclic
convolution of x1 [n] and x2 [n]:
(2.103)
n1 =0
www.dbooks.org
yc [n] = z[n] = x1 [n] c x2 [n].
(2.107)
74
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
Example 2-6: Cyclic Convolution
Hence, by Eq. (2.107),
x1 [n] c x2 [n] = z[n] = { 28, 21, 30, 31 },
Given the two signals
which is the same answer obtained in part (a).
x1 [n] = { 2, 1, 4, 3 },
x2 [n] = { 5, 3, 2, 1 },
compute the cyclic convolution of the two signals by
(a) applying the DFT method;
(b) applying the aliasing of the linear convolution method.
Solution:
(a) With N = 4, application of Eq. (2.89a) to x1 [n] and x2 [n]
leads to
X1 [k] = { 10, −2 + j2, 2, −2 − j2 },
X2 [k] = { 11, 3 − j2, 3, 3 + j2 }.
2-7.3 DFT and Linear Convolution
In the preceding subsection, we examined how the DFT can
be used to compute the cyclic convolution of two discretetime signals (Eq. (2.104)). The same method can be applied to
compute the linear convolution of the two signals, provided a
preparatory step of zero-padding the two signals is applied first.
Let us suppose that signal x1 [n] is of duration N1 and signal
x2 [n] is of duration N2 , and we are interested in computing their
linear convolution
y[n] = x1 [n] ∗ x2[n].
The duration of y[n] is
Nc = N1 + N2 − 1.
The point-by-point product of X1 [k] and X2 [k] is
X1 [k] X2 [k]
= { 10 × 11, (−2 + j2)(3 − j2), 2 × 3, (−2 − j2)(3 + j2) }
= { 110, −2 + j10, 6, −2 − j10 }.
Application of Eq. (2.104) leads to
x1 [n] c x2 [n] = { 28, 21, 30, 31 }.
Next, we zero-pad x1 [n] and x2 [n] so that their durations are
equal to or greater than Nc . As we will see in Section 2-8 on how
the fast Fourier transform (FFT) is used to compute the DFT, it
is advantageous to choose the total length of the zero-padded
signals to be M such that M ≥ Nc , and simultaneously M is a
power of 2.
The zero-padded signals are defined as
x′1 [n] = {x1 [n], 0, . . . , 0},
|{z} | {z }
(b) Per Eq. (2.71d), the linear convolution of
N1
x1 [n] = { 2, 1, 4, 3 }
N2
is
y[n] = x1 [n] ∗ x2[n]
3
= ∑ x1 [i] x2 [n − i]
i=0
= { 10, 11, 27, 31, 18, 10, 3 }.
Per Eq. (2.106),
z[n] = { y[0] + y[4], y[1] + y[5], y[2] + y[6], y[3] }
= { 10 + 18, 11 + 10, 27 + 3, 31 } = { 28, 21, 30, 31 }.
(2.109a)
M−N1
x′2 [n] = {x2 [n], 0, . . . , 0},
|{z} | {z }
and
x2 [n] = { 5, 3, 2, 1 }
(2.108)
(2.109b)
M−N2
and their M-point DFTs are X′1 [k] and X′2 [k], respectively. The
linear convolution y[n] can now be computed by a modified
version of Eq. (2.104), namely
y[n] = x′1 [n] ∗ x′2[n]
= DFT−1 { X′1 [k] X′2 [k] }
=
1 M−1 ′
∑ X1[k] X′2 [k] e j2π nk/M .
M k=0
(2.110)
Note that the DFTs can be computed using M-point DFTs of
x′1 [n] and x′2 [n], since using an M-point DFT performs the zeropadding automatically.
2-7 DISCRETE FOURIER TRANSFORM (DFT)
75
and
Example 2-7: DFT Convolution
X′2 [3] = −2 + j2.
Multiplication of corresponding pairs gives
Given signals x1 [n] = {4, 5} and x2 [n] = {1, 2, 3}, (a) compute
their convolution in discrete time, and (b) compare the result
with the DFT relation given by Eq. (2.110).
X′1 [0] X′2 [0] = 9 × 6 = 54,
X′1 [1] X′2 [1] = (4 − j5)(−2 − j2) = −18 + j2,
X′1 [2] X′2 [2] = −1 × 2 = −2,
Solution:
(a) Application of Eq. (2.71a) gives
and
3
x1 [n] ∗ x2[n] = ∑ x1 [i] x2 [n − i] = {4, 13, 22, 15}.
i=0
(b) Since x1 [n] is of length N1 = 2 and x2 [n] is of length
N2 = 3, their convolution is of length
Nc = N1 + N2 − 1 = 2 + 3 − 1 = 4.
X′1 [3] X′2 [3] = (4 + j5)(−2 + j2) = −18 − j2.
Application of Eq. (2.110) gives
y[n] = x′1 [n] ∗ x′2[n] =
=
Hence, we need to zero-pad x1 [n] and x2 [n] as
x′1 [n] = {4, 5, 0, 0}
and
1 3 ′
∑ X1 [k] X′2 [k] e jkπ n/2.
4 k=0
Evaluating the summation for n = 0, 1, 2 and 3 leads to
y[n] = x′1 [n] ∗ x′2[n] = {4, 13, 22, 15},
x′2 [n] = {1, 2, 3, 0}.
From Eq. (2.89a) with N = Nc = 4, the 4-point DFT of
x′1 [n] = {4, 5, 0, 0} is
3
X1 [k] = ∑ x′1 [n] e− jkπ n/2 ,
1 Nc −1 ′
∑ X1[k] X′2 [k] e j2π nk/Nc
Nc k=0
k = 0, 1, 2, 3,
n=0
which is identical to the answer obtained earlier in part (a).
For simple signals like those in this example, the DFT method
involves many more steps than does the straightforward convolution method of part (a), but for the type of signals used in
practice, the DFT method is computationally superior.
which gives
X′1 [0] = 4(1) + 5( j) + 0(1) + 0( j) = 9,
X′1 [1] = 4(1) + 5(− j) + 0(−1) + 0( j) = 4 − j5,
X′1 [2] = 4(1) + 5(−1) + 0(1) + 0(−1) = −1,
and
X′1 [3] = 4(1) + 5( j) + 0(−1) + 0(− j) = 4 + j5.
Similarly, the 4-point DFT of x′2 [n] = {1, 2, 3, 0} gives
X′2 [0] = 6,
X′2 [1] = −2 − j2,
X′2 [2] = 2,
www.dbooks.org
76
◮ To summarize, the linear convolution y[n] = h[n] ∗ x[n]
defined in Eq. (2.71) computes the response (output) y[n]
to the input x[n] for an LTI system with impulse response
h[n]. The cyclic convolution yc [n] = h[n] c x[n] defined in
Eq. (2.104) is what the DFT maps to products, so a cyclic
convolution can be computed very quickly using the FFT
algorithm to compute the DFTs H[k] of h[n] and X[k] of
x[n], and the inverse DFT of H[k] X[k]. Fortunately, a linear
convolution can be zero-padded to a cyclic convolution, as
presented in Subsection 2-7.3, so linear convolutions can
also be computed quickly using the FFT algorithm. Cyclic
convolutions will also be used in Chapter 7 for computing
wavelet transforms, because the cyclic convolution of a
signal x[n] of duration N with an impulse response h[n] (that
has been zero-padded to length N) gives an output yc [n]
of the same length as that of the input x[n]. For wavelets,
the cyclic convolution approach is superior to the linear
convolution approach because the latter results in an output
longer than the input. ◭
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
Contrast these large number of multiplications and additions
(MADs) with the number required using the FFT algorithm:
for N large, the number of complex multiplications is reduced
from N 2 to approximately (N/2) log2 N, which is only 2304
complex multiplications for N = 512. For complex additions,
the number is reduced from N(N − 1) to N log2 N, or 4608
for N = 512. These reductions, thanks to the efficiency of the
FFT algorithm, are on the order of 100 for multiplications and
on the order of 50 for addition. The reduction ratios become
increasingly more impressive at larger values of N (Table 2-10).
The computational efficiency of the FFT algorithm relies on a
“divide and conquer” concept. An N-point DFT is decomposed
(divided) into two (N/2)-point DFTs. Each of the (N/2)-point
DFTs is decomposed further into two (N/4)-point DFTs. The
decomposition process, which is continued until it reaches the
2-point DFT level, is illustrated in the next subsections.
2-8.1 2-Point DFT
For notational efficiency, we introduce the symbols
Exercise 2-13: Compute the 4-point DFT of {4, 3, 2, 1}.
Answer: {10, (2 − j2), 2, (2 + j2)}. (See IP )
2-8 Fast Fourier Transform (FFT)
◮ The fast Fourier transform (FFT) is a computational
algorithm used to compute the discrete Fourier transforms
(DFT) of discrete signals. Strictly speaking, the FFT is
not a transform, but rather an algorithm for computing the
transform. ◭
As was mentioned earlier, the fast Fourier transform (FFT) is
a highly efficient algorithm for computing the DFT of discrete
time signals. An N-point DFT performs a linear transformation
from an N-long discrete-time vector, namely x[n], into an Nlong frequency domain vector X[k] for k = 0, 1, . . . , N − 1.
Computation of each X[k] involves N complex multiplications,
so the total number of multiplications required to perform the
DFT for all X[k] is N 2 . This is in addition to N(N − 1) complex
additions. For N = 512, for example, direct implementation of
the DFT operation requires 262,144 multiplications and 261,632
complex additions. For small N, these numbers are smaller,
since multiplication by any of { 1, −1, j, − j } does not count as
a true multiplication.
WN = e− j2π /N ,
(2.111a)
WNnk = e− j2π nk/N ,
(2.111b)
WN−nk = e j2π nk/N .
(2.111c)
and
Using this shorthand notation, the summations for the DFT, and
its inverse given by Eq. (2.89), assume the form
N−1
X[k] = ∑ x[n] WNnk ,
n=0
k = 0, 1, . . . , N − 1,
(2.112a)
n = 0, 1, . . . , N − 1.
(2.112b)
and
x[n] =
1 N−1
∑ X[k]WN−nk ,
N k=0
In this form, the N-long vector X[k] is given in terms of the
N-long vector x[n], and vice versa, with WNnk and WN−nk acting
as weighting coefficients.
For a 2-point DFT,
N = 2,
W20k = e− j0 = 1,
and
W21k = e− jkπ = (−1)k .
2-8 FAST FOURIER TRANSFORM (FFT)
77
Table 2-10 Comparison of number of complex computations required by a standard DFT and an FFT using the formulas in the
bottom row.
Multiplication
N
x[0]
Standard DFT
FFT
Standard DFT
FFT
2
4
8
16
..
.
512
1,024
2,048
4
16
64
256
..
.
262,144
1,048,576
4,194,304
1
4
12
32
..
.
2,304
5,120
11,264
2
12
56
240
..
.
261,632
1,047,552
4,192,256
2
8
24
64
..
.
4,608
10,240
22,528
N
N2
N
log2 N
2
N(N − 1)
N log2 N
2-8.2 4-Point DFT
1
X[0]
1
1
x[1]
Additions
For a 4-point DFT, N = 4 and
X[1]
−1
WNnk = W4nk = e− jnkπ /2 = (− j)nk .
Figure 2-16 Signal flow graph for a 2-point DFT.
(2.115)
From Eq. (2.112a), we have
3
Hence, Eq. (2.112a) yields the following expressions for X[0]
and X[1]:
X[0] = x[0] + x[1]
(2.113a)
and
X[1] = x[0] − x[1],
(2.113b)
which can be combined into the compact form
k
X[k] = x[0] + (−1) x[1],
k = 0, 1.
(2.114)
The equations for X[0] and X[1] can be represented by the signal
flow graph shown in Fig. 2-16, which is often called a butterfly
diagram.
X[k] = ∑ x[n] W4nk
n=0
= x[0] + x[1] W41k + x[2] W42k + x[3] W43k ,
k = 0, 1, 2, 3.
(2.116)
Upon evaluating W41k , W42k , and W43k and the relationships
between them, Eq. (2.116) can be cast in the form
X[k] = [x[0] + (−1)k x[2]] + W41k [x[1] + (−1)k x[3]],
|
{z
}
|
{z
}
2-point DFT
(2.117)
2-point DFT
which consists of two 2-point DFTs: one that includes values of
x[n] for even values of n, and another for odd values of n. At this
point, it is convenient to define xe [n] and xo [n] as x[n] at even and
odd times:
xe [n] = x[2n],
xo [n] = x[2n + 1],
www.dbooks.org
n = 0, 1,
n = 0, 1.
(2.118a)
(2.118b)
78
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
xe[0] = x[0]
xe[1] = x[2]
Xe[0]
1
1
1
X[0]
1
1
Xe[1]
1
−1
2-point DFT
divided into four 4-point DFTs, which are just additions and
subtractions. This conquers the 16-point DFT by dividing it into
4-point DFTs and additional MADs.
X[1]
W14
A. Dividing a 16-Point DFT
1
xo[0] = x[1]
xo[1] = x[3]
1 Xo[0]
1
1
X[2]
−1
1
−W14
Xo[1]
−1
2-point DFT
X[3]
Recomposition
We now show that the 16-point DFT can be computed for
even values of k using an 8-point DFT of (x[n] + x[n + 8]) and
for odd values of k using an 8-point DFT of the modulated
signal (x[n] − x[n + 8])e− j2π n/16. Thus, the 16-point DFT can
be computed as an 8-point DFT (for even values of k) and as a
modulated 8-point DFT (for odd values of k).
Figure 2-17 Signal flow graph for a 4-point DFT. Weighting
coefficient W41 = − j. Note that summations occur only at red
intersection points.
Thus, xe [0] = x[0] and xe [1] = x[2] and, similarly, xo [0] = x[1]
and xo [1] = x[3]. When expressed in terms of xe [n] and xo [n],
Eq. (2.117) becomes
B. Computation at Even Indices
We consider even and odd indices k separately.
For even values of k, we can write k = 2k′ and split the
16-point DFT summation into two summations:
15
k
X[k] = [xe [0] + (−1) xe [1]]
|
{z
}
n=0
7
2-point DFT of xe [n]
+ W41k [xo [0] + (−1)k xo [1]],
|
{z
2-point DFT of xo [n]
k = 0, 1, 2, 3.
′
X[2k′ ] = ∑ x[n] e− j2π (2k /16)n
}
(2.119)
The FFT computes the 4-point DFT by computing the two
2-point DFTs, followed by a recomposition step that involves
multiplying the even 2-point DFT by W41k and then adding it to
the odd 2-point DFT. The entire process is depicted by the signal
flow graph shown in Fig. 2-17. In the graph, Fourier coefficients
Xe [0] and Xe [1] represent the outputs of the even 2-point DFT,
and similarly, Xo [0] and Xo [1] represent the outputs of the odd
2-point DFT.
2-8.3 16-Point DFT
We now show how to compute a 16-point DFT using two 8-point
DFTs and 8 multiplications and additions (MADs). This divides
the 16-point DFT into two 8-point DFTs, which in turn can be
15
′
′
= ∑ x[n] e− j2π (2k /16)n + ∑ x[n] e− j2π (2k /16)n .
n=0
n=8
(2.120)
Changing variables from n to n′ = n − 8 in the second summation, and recognizing 2k′ /16 = k′ /8, gives
7
′
7
′
′
X[2k′ ] = ∑ x[n] e− j2π (k /8)n + ∑ x[n′ + 8] e− j2π (k /8)(n +8)
n=0
7
n′ =0
′
= ∑ (x[n] + x[n + 8])e− j2π (k /8)n
n=0
= DFT({x[n] + x[n + 8], n = 0, . . . , 7}).
(2.121)
So for even values of k, the 16-point DFT of x[n] is the 8-point
DFT of { x[n] + x[n + 8], n = 0, . . . , 7 }.
2-8 FAST FOURIER TRANSFORM (FFT)
79
C. Computation at Odd Indices
• For even values of index k, we have
For odd values of k, we can write k = 2k′ + 1 and split the 16point DFT summation into two summations:
15
′
For odd index values, we need twiddle mults. The twiddle
factors are given
by { e− j2π n/8√} for n = 0, 1, 2, and 3, which
√
reduce to { 1, 22 (1 − j), − j, 22 (−1 − j) }.
X[2k′ + 1] = ∑ x[n] e− j2π (2k +1)/16n
n=0
7
′
= ∑ x[n] e− j2π (2k +1)/16n
n=0
15
+ ∑ x[n] e
− j2π (2k′ +1)/16n
X[0, 2, 4, 6] = DFT({ 7 + 8, 1 + 5, 4 + 3, 2 + 6 })
= { 36, 8 + j2, 8, 8 − j2 }.
• Implementing the twiddle mults gives
.
(2.122)
{ 7 − 8, 1 − 5, 4 − 3, 2 − 6 }
)
( √
√
2
2
(1 − j), − j,
(−1 − j)
× 1,
2
2
√
√
= { −1, 2 2(−1 + j), − j, 2 2(1 + j) }.
n=8
Changing variables from n to n′ = n − 8 in the second summation, and recognizing that e− j2π 8/16 = −1 and
2k′ + 1 k′
1
= +
,
16
8 16
• For odd values of index k, we have
√
√
X[1, 3, 5, 7] = DFT({ −1, 2 2(−1 + j), − j, 2 2(1 + j) })
= { −1 + j4.66, −1 + j6.66, −1 − j6.66, −1 − j4.66 }.
gives
7
′
X[2k′ + 1] = ∑ (x[n] e− j2π (1/16)n )e− j2π (k /8)n
n=0
7
′
′
′
+ ∑ (x[n′ + 8] e− j2π (1/16)(n +8) )e− j2π (k /8)(n +8)
• Combining these results for even and odd k gives
n′ =0
7
= ∑ e− j2π (1/16)n (x[n] − x[n + 8])e
DFT({ 7, 1, 4, 2, 8, 5, 3, 6 })
= { 36, −1 + j4.7, 8 + j2, −1 + j6.7, 8,
− 1 − j6.7, 8 − j2, −1 − j4.7 }.
− j2π (k′/8)n
n=0
= DFT({e− j2π (1/16)n (x[n] − x[n + 8]), n = 0, . . . , 7}).
(2.123)
So for odd values of k, the 16-point DFT of x[n] is the 8-point
DFT of { e− j(2π /16)n(x[n] − x[n + 8]), n = 0, . . . , 7 }. The signal
{ x[n] − x[n + 8], n = 0, . . . , 7 } has been modulated through
multiplication by e− j(2π /16)n. The multiplications by e− j(2π /16)n
are known as twiddle multiplications (mults) by the twiddle
factors e− j(2π /16)n.
Example 2-8:
Note the conjugate symmetry in the second and third lines:
X[7] = X∗ [1], X[6] = X∗ [2], and X[5] = X∗ [3].
• This result agrees with direct MATLAB computation using
fft([7 1 4 2 8 5 3 6]).
2-8.4 Dividing Up a 2N-Point DFT
We now generalize the procedure to a 2N-point DFT by dividing
it into two N-point DFTs and N twiddle mults.
(1) For even indices k = 2k′ we have:
Dividing an 8-Point DFT
into Two 4-Point DFTs
N−1
Divide the 8-point DFT of { 7, 1, 4, 2, 8, 5, 3, 6 } into two 4-point
DFTs and twiddle mults.
′
X[2k′ ] = ∑ (x[n] + x[n + N])e− j2π (k /N)n
Solution:
www.dbooks.org
n=0
= DFT{ x[n] + x[n + N], n = 0, 1, . . . , N − 1 }. (2.124)
80
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
(2) For odd indices k = 2k′ + 1 we have:
N−1
2-9.1 Deconvolution Procedure
′
X[2k′ + 1] = ∑ e− j2π (1/(2N))n(x[n] − x[n + N])e− j2π (k /N)n
n=0
= DFT{e− j2π (1/(2N))n(x[n] − x[n + N])}.
(2.125)
e
h[n] = {h[n], 0, . . . , 0 },
| {z }
(2.127a)
N−L zeros
Thus, a 2N-point DFT can be divided into
• Two N-point DFTs,
xe[n] = {x[n], 0, . . . , 0 }.
| {z }
(2.127b)
N−M zeros
• N multiplications by twiddle factors e− j2π (1/2N)n, and
• 2N additions and subtractions.
2-8.5 Dividing and Conquering
Now suppose N is a power of two; e.g., N = 1024 = 210 . In
that case, we can apply the algorithm of the previous subsection
recursively to divide an N-point DFT into two N/2-point DFTs,
then into four N/4-point DFTs, then into eight N/8-point DFTs,
and so on until we reach the following 4-point DFTs:
X[0] = x[0] + x[1] + x[2] + x[3],
X[1] = x[0] − jx[1] − x[2] + jx[3],
X[2] = x[0] − x[1] + x[2] − x[3],
X[3] = x[0] + jx[1] − x[2] − jx[3].
If x[n] has duration M and h[n] has duration L, then y[n] has
duration N = L+ M − 1. Let us define the zero-padded functions
(2.126)
At each stage, half of the DFTs are modulated, requiring N/2
multiplications. So if N is a power of 2, then an N-point
DFT computed using the FFT will require approximately
(N/2) log2 (N) multiplications and N log2 (N) additions. These
can be reduced slightly by recognizing that some multiplications
are simply multiplications by ±1 and ± j.
To illustrate the computational significance of the FFT, suppose we wish to compute a 32768-point DFT. Direct computation using Eq. (2.89a) would require (32768)2 ≈ 1.1 × 109
MADs. In contrast, computation using the FFT would require
less than 32768
2 log2 (32768) ≈ 250,000 MADs, representing a
computational saving of a factor of 4000!
2-9 Deconvolution Using the DFT
Recall that the objective of deconvolution is to reconstruct the
input x[n] of a system from measurements of its output y[n] and
knowledge of its impulse response h[n]. That is, we seek to solve
y[n] = h[n] ∗ x[n] for x[n], given y[n] and h[n].
(1) With xe[n], e
h[n], and y[n] all now of duration N, we can obtain
e
e
their respective N-point DFTs, X[k],
H[k],
and Y[k], which are
interrelated by
e X[k].
e
Y[k] = H[k]
(2.128)
e and taking an N-point inverse DFT, we
Upon dividing by H[k]
have
e
xe[n] = DFT−1 {X[k]}
)
(
Y[k]
−1
= DFT
e
H[k]
)
(
DFT{y[n]}
.
= DFT−1
DFT{e
h[n]}
(2.129)
(2) Discarding the (N − M) final zeros in xe[n] gives x[n]. The
zero-padding and unpadding processes allow us to perform the
e is nonzero
deconvolution problem for any system, provided H[k]
for all 0 ≤ k ≤ N − 1.
2-9.2 FFT Implementation Issues
(a) To use the FFT algorithm (Section 2-8) to compute the three
DFTs, N should be rounded up to the next power of 2 because
the FFT can be computed more rapidly.
e may be zero, which
(b) In some cases, some of the values of H[k]
e
is problematic because the computation of Y[k]/H[k]
would
involve dividing by zero. A possible solution to the division-bye = 0 for
zero problem is to change the value of N. Suppose H[k]
some value of index k, such as k = 3. This corresponds to H(Ω)
having a zero at 2π k/N for k = 3, because by definition, the DFT
is the DTFT sampled at Ω = 2π k/N for integers k. Changing N
to, say, N + 1 (or some other suitable integer) means that the
DFT is now the DTFT H(Ω) sampled at Ω = 2π k/(N + 1), so
the zero at k = 3 when the order was N may now get missed with
the sampling at the new order N + 1. Changing N to N + 1 may
2-9 DECONVOLUTION USING THE DFT
81
e
avoid one or more zeros in H[k],
but it may also introduce new
ones. It may be necessary to try multiple values of N to satisfy
e 6= 0 for all k.
the condition that H[k]
and
Y[3] = 6(1) + 19( j) + 32(−1) + 21(− j) = −26 − j2.
The 4-point DFT of x[ñ] is, therefore,
e = Y[0] = 78 = 13,
X[0]
e
6
H[0]
e = Y[1] = −26 + j2 = 6 − j7,
X[1]
e
−2 − j2
H[1]
Example 2-9: DFT Deconvolution
In response to an input x[n], an LTI system with an impulse
response h[n] = {1, 2, 3} generated an output
y[n] = {6, 19, 32, 21}.
e = Y[2] = −2 = −1,
X[2]
e
2
H[2]
and
Determine x[n], given that it is of finite duration.
e = Y[3] = −26 − j2 = 6 + j7.
X[3]
e
−2 + j2
H[3]
Solution: The output is of duration N = 4, so we should zeropad h[n] to the same duration by defining
e
h[n] = {1, 2, 3, 0}.
(2.130)
From Eq. (2.89a), the 4-point DFT of e
h[n] is
3
e = ∑e
h[n] e− j2π kn/4,
H[k]
k = 0, 1, 2, 3,
(2.131)
e is
By Eq. (2.89b), the inverse DFT of X[k]
xe[n] =
which yields
1 3 e
∑ X[k]e j2π kn/4,
4 k=0
which yields
e = 1(1) + 2(1) + 3(1) + 0(1) = 6,
H[0]
e = 1(1) + 2(− j) + 3(−1) + 0( j) = −2 − j2,
H[1]
e = 1(1) + 2(−1) + 3(1) + 0(−1) = 2,
H[2]
Similarly, the 4-point DFT of y[n] = {6, 19, 32, 21} is
Y[k] = ∑ y[n] e
− j2π kn/4
,
Given that y[n] is of duration N = 4 and h[n] is of duration L = 3, it follows that x[n] must be of duration
M = N − L + 1 = 4 − 3 + 1 = 2, if its duration is finite. Deletion
of the zero-pads from xe[n] leads to
x[n] = {6, 7},
(2.133)
whose duration is indeed 2.
e = 1(1) + 2( j) + 3(−1) + 0(− j) = −2 + j2.
H[3]
3
(2.132)
xe[n] = {6, 7, 0, 0}.
n=0
and
n = 0, 1, 2, 3,
Example 2-10:
Removal of Periodic Interference
We are given the signal of two actual trumpets playing simultaneously notes A and B. The goal is to use the DFT to
eliminate the trumpet playing note A, while preserving the
trumpet playing note B. We only need to know that note B is
at a higher frequency than note A.
k = 0, 1, 2, 3,
n=0
which yields
Y[0] = 6(1) + 19(1) + 32(1) + 21(1) = 78,
Y[1] = 6(1) + 19(− j) + 32(−1) + 21( j) = −26 + j2,
Y[2] = 6(1) + 19(−1) + 32(1) + 21(−1) = −2,
Solution: The two-trumpets signal time-waveform is shown
in Fig. 2-18(a), and the corresponding spectrum is shown in
Fig. 2-18(b). We note that the spectral lines occur in pairs of
harmonics with the lower harmonic of each pair associated with
www.dbooks.org
82
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
x(t)
Waveform of two-trumpet signal
0.8
0.6
0.4
0.2
0
−0.2
−0.4
−0.6
(a)
23
24
25
26
27
t (ms)
X[k]
Spectrum of two-trumpet signal
0.10
0.08
0.06
0.04
0.02
0
(b) 0
Note A (440 Hz)
Note B (491 Hz)
k
2-10 Computation of Continuous-Time
Fourier Transform (CTFT) Using
the DFT
Spectrum of filtered two-trumpet signal
Let us consider a continuous-time signal x(t) with support
[−T /2, T /2], which means that
1000
2000
3000
4000
X[k] (after filtering)
0.10
0.08
0.06
0.04
0.02
0
(c) 0
x(t) = 0
1000
xf (t)
2000
3000
k
4000
Waveform of filtered two-trumpet signal
0.6
0.4
0.2
0
−0.2
−0.4
(d)
nent of each pair of spectral lines to zero. The modified spectrum
is shown in Fig. 2-18(c). The inverse DFT of this spectrum,
followed by reconstruction to continuous time, is shown in
Fig. 2-18(d).
The filtering process eliminated the signal due to the trumpet
playing note A, while preserving the signal due to note B, almost
completely. This can be confirmed by listening to the signals
before and after filtering.
Whereas it is easy to distinguish between the harmonics of
note A and those of note B at lower frequencies, this is not
the case at higher frequencies, particularly when they overlap.
Hence, neither note can be eliminated without affecting the other
slightly. Fortunately, the overlapping high-frequency harmonics
contain very little power compared with the non-overlapping,
low-frequency harmonics, and therefore, their role is quite
insignificant.
for |t| >
T
.
2
Also, let us assume (for the time being) that the signal spectrum
X( f ) is bandwidth-limited to a maximum frequency F/2 in Hz,
which means that
X( f ) = 0
for | f | >
F
.
2
Our goal is to compute samples {X(k∆ f )} of X( f ) at a
frequency spacing ∆ f from signal samples { x(n∆t ) } of x(t)
recorded at time interval ∆t , and to do so using the DFT.
23
24
25
26
27
t (ms)
Figure 2-18 Removing the spectrum of note A.
note A and the higher harmonic of each pair associated with
note B.
Since we wish to eliminate note A, we set the lower compo-
2-10.1 Derivation Using Sampling Theorem
Twice
Whereas it is impossible for a signal to be simultaneously
bandlimited and time-limited, as presumed earlier, many realworld signals are approximately band- and time-limited.
According to the sampling theorem (Section 2-4), x(t) can
be reconstructed from its samples { x(n∆t ) } if the sampling rate
St = 1/∆t > 2 F2 = F. Applying the sampling theorem with t
and f exchanged, X( f ) can be reconstructed from its samples
2-10 COMPUTATION OF CONTINUOUS-TIME FOURIER TRANSFORM (CTFT) USING THE DFT
{ X(k∆ f ) } if its sampling rate S f = 1/∆ f > 2 T2 = T .
In the sequel, we use the minimum sampling intervals
∆t = 1/F and ∆ f = 1/T . Finer discretization can be achieved
by simply increasing F and/or T . In practice, F and/or T is (are)
increased slightly so that N = FT is an odd integer, which makes
the factor M = (N − 1)/2 also an integer (but not necessarily an
odd integer). The factor M is related to the order of the DFT,
which has to be an integer.
The Fourier transform of the synthetic sampled signal xs (t),
defined in Eq. (2.43) and repeated here as
∞
xs (t) =
∑ x(n∆t ) δ (t − n∆t ),
(2.134)
n=−∞
83
In the exponent of Eq. (2.136), the expression becomes
M
Xs (k∆ f ) =
∑ x(n∆t ) e− j2π (k∆ f )(n∆t ) ,
n=−M
|k| ≤ M
M
=
∑ x(n∆t ) e− j2π nk/(2M+1),
n=−M
|k| ≤ M. (2.140)
This expression looks like a DFT of order 2M + 1. Recall from
the statement in connection with Eq. (2.47) that the spectrum
Xs ( f ) of the sampled signal includes the spectrum X( f ) of the
continuous-time signal (multiplied by the sampling rate St ) plus
additional copies repeated every ±St along the frequency axis.
With St = 1/∆t = F in the present case,
was computed in Eq. (2.53), and also repeated here as
Xs ( f ) = FX( f ),
∞
∑ x(n∆t ) e− j2π f n∆t .
Xs ( f ) =
(2.135)
n=−∞
from which we deduce that
X(k∆ f ) = Xs ( f ) ∆t =
∑ x(n∆t ) e− j2π (k∆ f )(n∆t ) .
(2.136)
n=−∞
Noting that x(t) = 0 for |t| > T2 and X( f ) = 0 for | f | > F2 , we
restrict the ranges of n and k to
|n| ≤
T /2 FT
N
=
=
∆t
2
2
(2.137a)
|k| ≤
N
F/2 FT
=
= .
∆f
2
2
(2.137b)
and
∑ x(n∆t ) e− j2π nk/(2M+1)∆t ,
n=−M
∞
Next, we introduce factor M defined as
M=
N −1
,
2
1 1
1
1
1
=
= =
.
FT
FT
N
2M + 1
|k| ≤ M.
(2.142)
Ironically, this is the same result that would be obtained by
simply discretizing the definition of the continuous-time Fourier
transform! But this derivation shows that discretization gives the
exact result if x(t) is time- and bandlimited.
Example 2-11:
Computing CTFT by DFT
Use the DFT to compute the Fourier transform of the continuous
Gaussian signal
1
2
x(t) = √ e−t /2 .
2π
(2.138)
and we note that if N is an odd integer, M is guaranteed to be
an integer. In view of Eq. (2.137), the ranges of n and k become
n, k = −M, . . . , M. Upon substituting
∆t ∆ f =
(2.141)
M
Setting f = k∆ f gives
Xs (k∆ f ) =
for | f | < F,
Solution: Our first task is to assign realistic values for the
signal duration T and the width of its spectrum F. It is an
“educated” trial-and-error process. At t = 4, x(t) = 0.00013,
so we will assume that x(t) ≈ 0 for |t| > 4. Since x(t) is
symmetrical with respect to the vertical axis, we assign
(2.139)
T = 2 × 4 = 8 s.
2 2
The Fourier transform of x(t) is X( f ) = e−2π f . By trial and
error, we determine that F = 1.2 Hz is sufficient to characterize
X( f ). The combination gives
N = T F = 8 × 1.2 = 9.6.
www.dbooks.org
84
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
X[k]
1
0.8
0.6
0.4
0.2
−4
−3
−2
−1
0
1
2
3
4
k
Figure 2-19 Comparison of exact (blue circles) and DFT-computed (red crosses) of the continuous-time Fourier transform of a Gaussian
signal.
To increase the value of N to an odd integer, we increase F to
1.375 Hz, which results in N = 11 and M = 5. In Fig. 2-19
computed values of the discretized spectrum of x(t) are compared with exact values based on evaluating the analytical
2 2
expression for X( f ) = e−2π f . The comparison provides an
excellent demonstration of the power of the sampling theorem;
representing x(t) by only 11 equally spaced samples is sufficient
to capture its information content and generate its Fourier
transform with high fidelity.
2-10.2 Practical Computation of X( f ) Using the
DFT
Example 2-11 shows that simple discretization of the
continuous-time Fourier transform works very well, provided
that the discretization lengths in time and frequency are chosen
properly. The integer N was odd for clarity of derivation. In
practice, we would increase T and F so that N = T F is a power
of two, permitting the fast Fourier transform (FFT) algorithm to
be used to compute the DFT quickly.
Concept Question 2-12:
Fourier transform (DFT)?
Why do we need a discrete
The DFT is often used to compute the CTFT numerically. Why does this often work as
well as it does?
Concept Question 2-13:
2-10 COMPUTATION OF CONTINUOUS-TIME FOURIER TRANSFORM (CTFT) USING THE DFT
85
Summary
Concepts
• Many 2-D concepts can be understood more easily by
reviewing their 1-D counterparts. These include: LTI systems, convolution, sampling, continuous-time; discretetime; and discrete Fourier transforms.
• The DTFT is periodic with period 2π .
• Continuous-time signals can be sampled to discrete-time
signals, on which discrete-time signal processing can be
performed.
• The response of an LTI system with impulse response
h(t) to input x(t) is output y(t) = h(t)∗ x(t), and similarly
in discrete time.
• The response of an LTI system with impulse response
h(t) to input A cos(2π f0t + θ ) is
A|H( f )| cos(2π f0t + θ + H( f )),
where H( f ) is the Fourier transform of h(t), and
similarly in discrete time.
Mathematical Formulae
Impulse
Sinc interpolation formula
t 1
δ (x) = lim
rect
ε →0 2ε
2ε
∞
n=−∞
Energy of x(t)
E=
Z ∞
−∞
Discrete-time Fourier transform (DTFT)
∞
2
|x(t)| dt
X(Ω) =
y(t) = h(t) ∗ x(t) =
Convolution
Z ∞
−∞
Inverse DTFT
Z
1 π
x[n] =
X(Ω) e jΩn dΩ
2 π −π
h(τ )x(t − τ ) d τ
Discrete-time sinc
Ω0
Ω0 n
h[n] =
sinc
π
π
∞
y[n] = h[n] ∗ x[n] = ∑ h[i] x[n − i]
i=−∞
Fourier transform
Z ∞
−∞
Discrete sinc
sin((2N + 1)Ω/2)
X(Ω) =
sin(Ω/2)
x(t) e− j2π f t dt
Inverse Fourier transform
x(t) =
Z ∞
−∞
X( f ) e
j2π f t
Discrete Fourier Transform (DFT)
M−1
df
X[k] = ∑ x[n] e− j2π nk/N
n=0
Sinc function
sin(π x)
sinc(x) =
πx
Inverse DFT
1 N−1
x[n] =
∑ X[k] e j2π nk/N
N k=0
Ideal lowpass filter impulse response
h(t) = 2 fc sinc(2 fc t)
Cyclic convolution
N−1
Sampling theorem
Sampling rate S =
∑ x[n] e− jΩn
n=−∞
Convolution
X( f ) =
∑ x(n∆) sinc(S(t − n∆))
x(t) =
1
> 2B if X( f ) = 0 for | f | > B
∆
yc [n] = x1 [n] c x2 [n] = ∑ x1 [n1 ] x2 [(n − n1)N ]
www.dbooks.org
n1 =0
86
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
Important Terms
Provide definitions or explain the meaning of the following terms:
aliasing
cyclic convolution
deconvolution
DFT
DTFT
FFT
Fourier transform
frequency response function
impulse response convolution
linear time-invariant (LTI)
PROBLEMS
Parseval’s theorem
Rayleigh’s theorem
sampled signal
sampling theorem
sinc function
2.8 Compute a Nyquist sampling rate for reconstructing signal
Section 2-2: Review of Continuous-Time Systems
2.1 Compute the following convolutions:
(a) e−t u(t) ∗ e−2t u(t)
(b) e−2t u(t) ∗ e−3t u(t)
(c) e−3t u(t) ∗ e−3t u(t)
Section 2-3: 1-D Fourier Transforms
2.2 Show that the spectrum of
sin(20π t) sin(10π t)
πt
πt
is zero for | f | > 15 Hz.
2.3 Using only Fourier transform properties, show that
sin(30π t)
sin(10π t)
[1 + 2 cos(20π t)] =
.
πt
πt
2.4 If x(t) = sin(2t)/(π t), compute the energy of d 2 x/dt 2 .
2.5
Compute the energy of e−t u(t) ∗ sin(t)/(π t).
2.6 Show that
spectrum
zero padding
Z ∞
a
sin2 (at)
dt =
2
π
−∞ (π t)
if a > 0.
x(t) =
sin(40π t) sin(60π t)
π 2t 2
from its samples.
2.9 Signal
x(t) =
sin(2π t)
[1 + 2 cos(4π t)]
πt
is sampled every 1/6 second. What is the spectrum of the
sampled signal?
2.10 Signal x(t) = cos(14π t) − cos(18π t) is sampled at 16
sample/s. The result is passed through an ideal brick-wall
lowpass filter with a cutoff frequency of 8 Hz. What is the
spectrum of the output signal?
2.11 Signal x(t) = sin(30π t) + sin(70π t) is sampled at 50
sample/s. The result is passed through an ideal brick-wall
lowpass filter with a cutoff frequency of 25 Hz. What is the
spectrum of the output signal?
Section 2-5: Review of Discrete-Time Signals and
Systems
2.12 Compute the following convolutions:
(a) {1, 2} ∗ {3, 4, 5}
(b) {1, 2, 3} ∗ {4, 5, 6}
(c) {2, 1, 4} ∗ {3, 6, 5}
Section 2-4: The Sampling Theorem
2.13 If {1, 2, 3} ∗ x[n] = {5, 16, 34, 32, 21}, compute x[n].
2.7 The spectrum of the trumpet signal for note G (784 Hz)
is negligible above its ninth harmonic. What is the Nyquist
sampling rate required for reconstructing the trumpet signal
from its samples?
2.14 Given the two systems connected in series as
x[n]
h1 [n]
w[n] = 3x[n] − 2x[n − 1],
PROBLEMS
87
Section 2-6: Discrete-Time Fourier Transform
(DTFT)
and
h2 [n]
w[n]
y[n] = 5w[n] − 4w[n − 1],
2.20 Compute the DTFTs of the following signals (simplify
answers to sums of sines and cosines).
(a) {1, 1, 1, 1, 1}
(b) {3, 2, 1}
compute the overall impulse response.
2.15 The two systems
2.21 Compute the inverse DTFT of
y[n] = 3x[n] − 2x[n − 1]
X(Ω) = [3 + 2 cos(Ω) + 4 cos(2Ω)] + j[6 sin(Ω) + 8 sin(2Ω)].
and
y[n] = 5x[n] − 4x[n − 1]
are connected in parallel. Compute the overall impulse response.
Section 2-6: Discrete-Time Frequency Response
π n
2
y[n] = x[n] + 0.5x[n − 1] + x[n − 2]
y[n],
(a) Compute the frequency response H(Ω).
(b) Compute the output y[n].
2.23 Compute the DFTs of each of the following signals:
(a) {12, 8, 4, 8}
(b) {16, 8, 12, 4}
2.24 Determine the DFT of a single period of each of the
following signals:
(a) cos( π4 n)
2.17 Given
π cos
n
2
X(Ω) = [7 + 5 cos(Ω) + 3 cos(2Ω)] + j[sin(Ω) + sin(2Ω)].
Section 2-7: Discrete Fourier Transform (DFT)
2.16 Given
cos
2.22 Compute the inverse DTFT of
(b) 14 sin( 34π n)
y[n] = 8x[n] + 3x[n − 1] + 4x[n − 2]
(a) Compute the frequency response H(Ω).
(b) Compute the output y[n].
y[n], 2.25 Compute the inverse DFTs of the following:
(a) {0, 0, 3, 0, 4, 0, 3, 0}
(b) {0, 3 + j4, 0, 0, 0, 0, 0, 3 − j4}
Section 2-9: Deconvolution Using the DFT
2.18 If input x[n] = cos( π2 n) + cos(π n), and
2.26 Use DFTs to compute the convolution
x[n]
y[n] = x[n] + x[n − 1] + x[n − 2] + x[n − 3]
y[n],
by hand.
(a) Compute the frequency response H(Ω).
(b) Compute the output y[n].
2.19
x[n]
If input x[n] = 1 + 2 cos( π2 n) + 3 cos(π n), and
y[n] = x[n] + 4x[n − 1] + 3x[n − 3]
(a) Compute the frequency response H(Ω).
(b) Compute the output y[n].
{1, 3, 5} ∗ {7, 9}
y[n],
2.27 Solve each of the following deconvolution problems for
input x[n]. Use MATLAB.
(a) x[n] ∗ {1, 2, 3} = {7, 15, 27, 13, 24, 27, 34, 15}.
(b) x[n] ∗ {1, 3, 5} = {3, 10, 22, 18, 28, 29, 52, 45}.
(c) x[n] ∗ {1, 4, 2, 6, 5, 3} =
{2, 9, 11, 31, 48, 67, 76, 78, 69, 38, 12}.
2.28 Solve each of the following deconvolution problems for
input x[n]. Use MATLAB.
www.dbooks.org
88
(a) x[n] ∗ {3, 1, 4, 2} = {6, 23, 18, 57, 35, 37, 28, 6}.
(b) x[n] ∗ {1, 7, 3, 2} = {2, 20, 53, 60, 53, 54, 21, 10}.
(c) x[n] ∗ {2, 2, 3, 6} =
{12, 30, 42, 71, 73, 43, 32, 45, 42}.
Section 2-10: Computation of CTFT Using the
DFT
2.29 Use a 40-point DFT to compute the inverse Fourier
transform of
sin(π f ) 2
.
X( f ) =
πf
Assume that X( f ) ≈ 0 for | f | > 10 Hz and x(t) ≈ 0 for |t| > 1 s.
Plot the actual and computed inverse Fourier transforms on the
same plot to show the close agreement between them.
2.30 Use an 80-point DFT to compute the inverse Fourier
transform of
sin(π f ) 2
X( f ) H( f ) =
(1 + e−2 jπ f ).
πf
Assume that X( f ) ≈ 0 for | f | > 10 Hz and x(t) ≈ 0 for |t| > 2 s.
Plot the actual and computed inverse Fourier transforms on the
same plot to show the close agreement between them.
CHAPTER 2 REVIEW OF 1-D SIGNALS AND SYSTEMS
Chapter
3
3
2-D Images and Systems
Contents
3-1
3-2
3-3
3-4
3-5
3-6
3-7
3-8
3-9
Ω2
Overview, 90
Displaying Images, 90
2-D Continuous-Space Images, 91
Continuous-Space Systems, 93
2-D Continuous-Space Fourier Transform
(CSFT), 94
2-D Sampling Theorem, 107
2-D Discrete Space, 113
2-D Discrete-Space Fourier Transform
(DSFT), 118
2-D Discrete Fourier Transform (2-D DFT), 119
Computation of the 2-D DFT Using
MATLAB, 126
Problems, 86
Objectives
Learn to:
■ Compute the output image from an LSI system to a
given input image using convolution.
■ Compute the continuous-space Fourier transform of
an image or point-spread function.
■ Use the 2-D sampling theorem to convert a continuous-space image to a discrete-space image.
■ Perform the two tasks listed above for continuousspace images on discrete-space images.
www.dbooks.org
4π
Ω1
−4π
−4π
4π
This chapter presents the 2-D versions, suitable for
image processing, of the 1-D concepts presented in
Chapter 2. These include: linear shift-invariant
(LSI) systems, 2-D convolution, spatial frequency
response, filtering, Fourier transforms for continuous and discrete-space images, and the 2-D sampling theorem. It also covers concepts that do not
arise in 1-D, such as separability, image scaling and
rotation, and representation of discrete-space
images using various coordinate systems.
Overview
A 2-D image is a signal that varies as a function of the two
spatial dimensions, x and y, instead of time t. A 3-D image, such
as a CT scan (Fig. 1-24), is a signal that varies as a function of
(x, y, z).
In 1-D, it is common practice to assign the symbol x(t) to
represent the signal at the input to a system, and to assign y(t)
to the output signal [or x[n] and y[n] if the signals are discrete].
Because x, y, and z are the standard symbols of the Cartesian
coordinate system, a different symbolic representation is used
with 2-D images:
3.5
3
2.5
2
1.5
1
0.5
0
19
x 10
10 y
19 0
◮ Image intensity is represented by f (x, y), where (x, y) are
two orthogonal spatial dimensions. ◭
(a) Mesh plot format
0
1
This chapter extends the 1-D definitions, properties, and transformations covered in the previous chapter into their 2-D
equivalents. It also presents certain 2-D properties that have no
counterparts in 1-D.
3
5
7
x
9
3-1 Displaying Images
11
In 1-D, a continuous-time signal x(t) is displayed by plotting
x(t) versus t. A discrete-time signal x[n] is displayed using a
stem plot of x[n] versus n. Clearly such plots are not applicable
for 2-D images.
Image intensity f (x, y) of a 2-D image can be displayed either
as a 3-D mesh plot (Fig. 3-1(a)), which hides some features of
the image and is difficult to create and interpret, or as a grayscale
image (Fig. 3-1(b)). In a grayscale image, the image intensity is
scaled so that the minimum value of f (x, y) is depicted in black
and the maximum value of f (x, y) is depicted in white. If the
image is non-negative ( f (x, y) ≥ 0), as is often the case, black in
the grayscale image denotes zero values of f (x, y). If the image
is not non-negative, zero values of f (x, y) appear as a shade of
gray.
MATLAB’s imagesc(X),colormap(gray) displays
the 2-D array X as a grayscale image in which black depicts
the minimum value of X and white depicts the maximum value
of X.
It is also possible to display an image f (x, y) as a false-color
image, in which case different colors denote different values
of f (x, y). The relation between color and values of f (x, y) is
denoted using a colorbar, to the side or bottom of the image. An
example of a false-color display was shown earlier in Fig. 1-11,
15
13
17
19
0 1
3
5
7
9
11 13 15 17 19
y
(b) Grayscale format
Figure 3-1 An image displayed in (a) mesh plot format and (b)
grayscale format.
depicting the infrared intensity emitted by a hot air balloon.
We should not confuse a false-color image with a true-color
image. Whereas a false-color image is a single grayscale image
(1 channel) displayed in color, a true-color image actually is a
set of three images (3 channels):
{ fred (x, y), fgreen (x, y), fblue (x, y)},
representing (here) the three primary colors: red, green and
blue. Other triplets of colors, such as yellow, cyan and magenta,
can also be used. Hence, image processing of color images
90
3-2 2-D CONTINUOUS-SPACE IMAGES
91
encompasses 3-channel image processing (see Chapter 10).
A grayscale image can be regarded as a still of a black-andwhite TV image, while a color image can be regarded as a still
of a color TV image. Before flat-panel displays were invented,
color images on TVs and monitors were created from three
separate signals using three different electron guns in picture
tubes, while black-and-white TV images were created using a
single electron gun in a picture tube. Modern solid-state color
display, such as liquid crystal displays (LCDs), are composed
of three interleaved displays (similar to the APS display in
Fig. 1-4), each driven by one of the three signal channels.
Concept Question 3-1: What is the difference between a
true-color image and a false-color image?
Exercise 3-1: How can you tell whether an image displayed
in color is a color image or a false-color image?
False-color images should have colorbars to
identify the numerical value associated with each color.
Answer:
B. Box Image
The box image fBox (x, y) is the 2-D pulse-equivalent of the
rectangular function rect(t) and is defined as
fBox (x, y) = rect(x) rect(y)
(
1
for |x| < 1/2 and |y| < 1/2,
=
0
otherwise.
By extension, a box image of widths (ℓx , ℓy ) and centered at
(x0 , y0 ) is defined as
x − x0 y − y0
x − x0
y − y0
fBox
,
= rect
rect
ℓx
ℓy
ℓx
ℓy
(
1
for |x − x0| < ℓx /2 & |y − y0| < ℓy /2,
=
(3.2)
0
otherwise,
and shown in Fig. 3-2(a).
3-2 2-D Continuous-Space Images
A continuous-space image is a physical quantity, such as temperature or pressure, that varies with spatial position in 2-D.
Mathematically a continuous-space image is a function f (x, y)
of spatial position (x, y), where x and y have units of length
(meters).
y
ℓx
ℓy
y0
x0
3-2.1 Fundamental 2-D Images
(a) f Box x − x0 , y − y0 = rect x − x0 rect y − y0
ℓy
ℓx
ℓy
ℓx
(
A. Impulse
)
A 2-D impulse δ (x, y) is simply
δ (x − ξ , y − η ) = δ (x − ξ ) δ (y − η ).
x0
(3.1)
x
(b) f Disk x − x0 , y − y0
a
a
(
The sifting property generalizes directly from 1-D to 2-D. In
2-D
f (ξ , η ) δ (x − ξ , y − η ) d ξ d η = f (x, y).
a/2
y0
and a 2-D impulse shifted by ξ along x and by η along y is
−∞ −∞
( ) ( )
y
δ (x, y) = δ (x) δ (y),
Z ∞Z ∞
x
)
Figure 3-2 (a) Box image of widths (ℓx , ℓy ) and centered at
(x0 , y0 ) and (b) disk image of radius a/2 and centered at (x0 , y0 ).
www.dbooks.org
92
CHAPTER 3 2-D IMAGES AND SYSTEMS
C. Disk Image
Being rectangular in shape, the box-image function is suitable
for applications involving Cartesian coordinates, such as shifting
the box sideways or up and down across the image. Some
applications, however, require the use of polar coordinates, in
which case the disk-image function is more suitable. The disk
image fDisk (x, y) of radius 1/2 is defined as
(
p
1
for px2 + y2 < 1/2,
fDisk (x, y) =
(3.3)
0
for x2 + y2 > 1/2.
The expression given by Eq. (3.3) pertains to a circular disk
centered at the origin and of radius 1/2. For the more general
case of a circular disk of radius a/2 and centered at (x0 , y0 ),
x − x0 y − y0
fDisk
=
,
a
a
(
p
1
for p(x − x0)2 + (y − y0)2 < a/2,
(3.4)
0
for (x − x0)2 + (y − y0)2 > a/2.
x
(0,0)
y
(a) Origin at top left and y axis downward
y
x
(0,0)
(b) Origin at bottom left and y axis upward
y
An example is displayed in Fig. 3-2(b).
(0,0)
x
3-2.2 Properties of Images
A. Generalizations of 1-D Properties
(c) Origin at center of image
1. Spatial shift
When shifted spatially by (x0 , y0 ), image f (x, y) becomes
f (x − x0 , y − y0). Assuming the x axis is pointing to the right,
image f (x − x0 , y − y0 ) is shifted to the right by x0 , relative to
f (x, y), if x0 is positive, and to the left by the same amount if x0
is negative.
Whereas it is customary to define the direction of the x axis as
pointing to the right, there is less uniformity with regard to the
definition of the y axis; sometimes the y axis is defined along the
upward direction, and in other cases it is defined to be along the
downward direction. Hence, if y0 is positive, f (x − x0 , y − y0 )
is shifted by y0 either upwards or downwards, depending on the
direction of the y axis.
As to the location of the origin (x, y) = (0, 0), it is usually
defined to be at any one of the following three locations:
upper left corner, lower left corner, or the center of the image
(Fig. 3-3).
In this book:
• Images f (x, y) are usually displayed with the origin (0, 0)
at the upper-left corner, as in Fig. 3-3(a).
Figure 3-3 Three commonly used image coordinate systems.
• Point spread functions (PSFs)—introduced in Section
3-3.2—are usually displayed with the origin at the center,
as in Fig. 3-3(c).
• Image spectra, defined in Section 3-4, are usually displayed as in Fig. 3-3(c).
2. Spatial scaling
When spatially scaled by (ax , ay ), image f (x, y) becomes
f (ax x, ay y). If ax > 1, the image is shrunk in the x direction by
a factor ax , and if 0 < ax < 1, the image is magnified in size
by 1/ax . So ax represents a shrinkage factor. If ax < 0, the image
is reversed in the x direction, in addition to being shrunk by a
factor of |ax |. The same comments apply to ay .
3-3 CONTINUOUS-SPACE SYSTEMS
93
3. Image energy
y
Extending the expression for signal energy given by Eq. (3.7)
from 1-D to 2-D leads to
E=
Z ∞Z ∞
−∞ −∞
| f (x, y)|2 dx dy.
y′
x′
y
x′
θ
(3.5)
y′
x
x′ = x cos θ + y sin θ
y′ = −x sin θ + y cos θ
x
4. Even-odd decomposition
A real-valued image f (x, y) can be decomposed into its even
fe (x, y) and odd fo (x, y) components:
f (x, y) = fe (x, y) + fo (x, y),
Figure 3-4 Rotation of coordinate system (x, y) by angle θ to
coordinate system (x′ , y′ ).
(3.6)
where
These can be combined into
fe (x, y) = [ f (x, y) + f (−x, −y)]/2,
fo (x, y) = [ f (x, y) − f (−x, −y)]/2.
′ cos θ
x
=
y′
− sin θ
(3.7a)
(3.7b)
sin θ
cos θ
x
.
y
(3.11)
B. Non-Generalizations of 1-D Properties
Hence, after rotation by angle θ , image f (x, y) becomes transformed into a new image g(x, y) given by
We now introduce two new properties of images, separability
and rotation, neither one of which has a 1-D counterpart.
g(x, y) = f (x′ , y′ ) = f (x cos θ + y sin θ , y cos θ − x sin θ ) (3.12)
1. Separability
Note that rotating an image usually assumes that the image has
been shifted so that the center of the image is at the origin (0, 0),
as in Fig. 3-3(c).
An image f (x, y) is separable if it can be written as a product of
separate functions f1 (x) and f2 (y):
f (x, y) = f1 (x) f2 (y).
(3.8)
As we will see later, the 2-D Fourier transform of a separable image can be computed as a product of the 1-D Fourier transforms
of f1 (x) and f2 (y). 2-D impulses and box images are separable,
whereas disk images are not.
(a) 2-D impulse; (b) box; (c) disk?
Answer: 2-D impulse and box are separable; disk is not
separable.
Exercise 3-3: Which of the following images is invariant
to rotation: (a) 2-D impulse; (b) box; (c) disk with center at
the origin (0, 0)?
2. Rotation
To rotate an image by an angle θ , we define rectangular
coordinates (x′ , y′ ) as the rectangular coordinates (x, y) rotated
by angle θ . Using the sine and cosine addition formulae, the
rotated coordinates (x′ , y′ ) are related to coordinates (x, y) by
(Fig. 3-4)
(3.9)
x′ = x cos θ + y sin θ
and
Exercise 3-2: Which of the following images is separable:
y′ = −x sin θ + y cos θ .
(3.10)
Answer: 2-D impulse and disk are invariant to rotation;
box is not invariant to rotation.
3-3 Continuous-Space Systems
A continuous-space system is a device or mathematical model
that accepts as an input an image f (x, y) and produces as an
www.dbooks.org
94
CHAPTER 3 2-D IMAGES AND SYSTEMS
output an image g(x, y).
f (x, y)
and if the system also is shift-invariant, the 2-D superposition
integral simplifies to the 2-D convolution given by
SYSTEM
g(x, y).
g(x, y) =
The image rotation transformation described by Eq. (3.12) is a
good example of such a 2-D system.
Z ∞Z ∞
−∞ −∞
f (ξ , η ) h(x − ξ , y − η ) d ξ d η
= f (x, y) ∗ ∗h(x, y),
(3.15a)
where the “double star” in f (x, y) ∗ ∗h(x, y) denotes the 2-D
convolution of the PSF h(x, y) with the input image f (x, y). In
symbolic form, the 2-D convolution is written as
3-3.1 Linear and Shift-Invariant (LSI) Systems
The definition of the linearity property of 1-D systems (Section
2-2.1) extends directly to 2-D spatial systems, as does the
definition for the invariance, except that time invariance in 1-D
systems becomes shift invariance in 2-D systems. Systems
that are both linear and shift-invariant are termed linear shiftinvariant (LSI).
3-3.2 Point Spread Function (PSF)
The point spread function (PSF) of a 2-D system is essentially
its 2-D impulse response. The PSF h(x, y; x0 , y0 ) of an image
system is its response to a 2-D impulse δ (x, y) shifted by
(x0 , y0 ):
δ (x − x0 , y − y0 )
SYSTEM
h(x, y; x0 , y0 ). (3.13a)
If the system is also shift-invariant (SI), Eq. (3.13a) becomes
δ (x − x0 , y − y0)
SI
h(x − x0, y − y0).
(3.13b)
3-3.3 2-D Convolution
For a linear system, extending the 1-D superposition integral
expression given by Eq. (2.13) to 2-D gives
f (x, y)
L
g(x, y) =
Z ∞Z ∞
−∞ −∞
f (ξ , η ) h(x, y; ξ , η ) d ξ d η , (3.14)
f (x, y)
LSI
g(x, y) = f (x, y) ∗ ∗h(x, y).
(3.15b)
◮ A 2-D convolution consists of a convolution in the x
direction, followed by a convolution in the y direction, or
vice versa. Consequently, the 1-D convolution properties
listed in Table 2-3 generalize to 2-D. ◭
Concept Question 3-2: Why do so many 1-D convolution properties generalize directly to 2-D?
Exercise 3-4: The operation of a 2-D mirror is described by
g(x, y) = f (−x, −y). Find the PSF of the mirror.
Answer: h(x, y; ξ , η ) = δ (x + ξ ; y + η ).
Exercise 3-5: If g(x, y) = h(x, y) ∗ ∗ f (x, y), what is
4h(x, y) ∗ ∗ f (x − 3, y − 2) in terms of g(x, y)?
Answer: 4g(x − 3, y − 2), using the shift and scaling
properties of 1-D convolutions.
3-4 2-D Continuous-Space Fourier
Transform (CSFT)
The 2-D continuous-space Fourier transform (CSFT) operates
between the spatial domain (x, y) and the spatial frequency
domain (µ , v), where µ and v are called spatial frequencies or
wavenumbers, with units of cycles/meter, analogous to the units
of cycles/s (i.e., Hz) for the time frequency f .
The 2-D CSFT is denoted F(µ , v) and it is related to f (x, y)
3-4 2-D CONTINUOUS-SPACE FOURIER TRANSFORM (CSFT)
by
F(µ , v) =
Z ∞Z ∞
−∞ −∞
f (x, y) e− j2π (µ x+vy) dx dy.
(3.16a)
95
◮ The spectrum F(µ , ν ) of an image f (x, y) is its 2-D
CSFT. The spatial frequency response H(µ , ν ) of an LSI
2-D system is the 2-D CSFT of its PSF h(x, y). ◭
The inverse operation is given by
f (x, y) =
Z ∞Z ∞
−∞ −∞
F(µ , v) e j2π (µ x+vy) d µ dv,
3-4.1 Notable 2-D CSFT Pairs and Properties
(3.16b)
and the combination of the two operations is represented symbolically by
F(µ , v).
f (x, y)
In the 1-D continuous-time domain, we call the output of
an LTI system its impulse response h(t) when the input is an
impulse:
δ (t)
LTI
h(t),
and we call the Fourier transform of h(t) the frequency response
of the system, H( f ):
The 2-D CTFT of a real-valued image f (x, y) obeys the conjugate symmetry property
F∗ (µ , v) = F(−µ , −v),
which states that the 2-D Fourier transform F(µ , v), must be
reflected across both the µ and v axes to produce its complex
conjugate.
B. Separable Images
The analogous relationships for a 2-D LSI system are
δ (x, y)
h(x, y)
LSI
h(x, y),
H(µ , v).
f1 (x) f2 (y)
(3.17a)
(3.17b)
The CSFT H(µ , v) is called the spatial frequency response of
the LSI system.
◮ As in 1-D, the 2-D Fourier transform of a convolution
of two functions is equal to the product of their Fourier
transforms:
LTI
g(x, y) = h(x, y) ∗ ∗ f (x, y)
F1 (µ ) F2 (v).
(3.19)
This assertion follows directly from the definition of the CSFT
given by Eq. (3.16a):
F(µ , v) =
=
f (x, y)
(3.18)
◮ The CSFT of a separable image f (x, y) = f1 (x) f2 (y) is
itself separable in the spatial frequency domain:
H( f ).
h(t)
A. Conjugate Symmetry
Z ∞Z ∞
−∞ −∞
Z ∞
−∞
f (x, y) e− j2π (µ x+vy) dx dy
f1 (x) e− j2π µ x dx
= F1 (µ ) F2 (v).
Z ∞
−∞
f2 (y) e− j2π vy dy
(3.20)
◮ The CSFT pairs listed in Table 3-1 are all separable
functions, and can be obtained by applying Eq. (3.20) to the
1-D Fourier transform pairs listed in Table 2-5. CSFT pairs
for non-separable functions are listed later in Table 3-2. ◭
implies that
C. Sinusoidal Image
G(µ , v) = H(µ , v) F(µ , v).
Consider the sinusoidal image described by
All of the 1-D Fourier transform properties listed in Table 2-4
and all of the 1-D transform pairs listed in Table 2-5 generalize
readily to 2-D. The 2-D version of the two tables is available in
Table 3-1.
f (x, y) = cos(2π µ0 x) cos(2π v0 y),
where µ0 = 1.9 cycles/cm and v0 = 0.9 cycles/cm are the
frequencies of the spatial variations along x and y in the spatial
www.dbooks.org
96
CHAPTER 3 2-D IMAGES AND SYSTEMS
Table 3-1 2-D Continuous-space Fourier transform (CSFT).
Selected Properties
1. Linearity
∑ ci fi (x, y)
2. Spatial scaling
f (ax x, ay y)
3. Spatial shift
f (x − x0 , y − y0 )
∑ ci Fi (µ , v)
1
µ v
F
,
|ax ay |
ax ay
e− j2π µ x0 e− j2π vy0 F(µ , v)
f (−x, −y)
F(−µ , −v)
5. Conjugation
f ∗ (x, y)
F∗ (−µ , −v)
6. Convolution
in space
f (x, y) ∗ ∗h(x, y)
F(µ , v) H(µ , v)
7. Convolution
in frequency
f (x, y) h(x, y)
F(µ , v) ∗ ∗H(µ , v)
4. Reversal
CSFT Pairs
8.
δ (x, y)
9.
δ (x − x0 , y − y0 )
e− j2π µ x0 e− j2π vy0
10.
e j2π µ0 x e j2π v0 y
x
y
rect
rect
ℓx
ℓy
δ (µ − µ0 , v − v0 )
11.
12.
µ0 v0 sinc(µ0 x) sinc(v0 y)
13.
e−π x e−π y
14.
cos(2π µ0 x) cos(2π v0 y)
2
domain, respectively. A grayscale image-display of f (x, y) is
shown in Fig. 3-5(a), with pure black representing f (x, y) = −1
and pure white representing f (x, y) = +1. As expected, the
image exhibits a repetitive pattern along both x and y, with 19
cycles in 10 cm in the x direction and 9 cycles in 10 cm in
the y direction, corresponding to spatial frequencies of µ = 1.9
cycles/cm and v = 0.9 cycles/cm, respectively.
1
ℓx ℓy sinc(ℓx µ ) sinc(ℓy v)
µ
v
rect
rect
µ0
v0
2
2
e−π µ e−π v
2
1
4 [δ ( µ ± µ0 ) δ (v ± v0 )]
By Eq. (3.20), the CSFT of f (x, y) is
F(µ , v) = F1 (µ ) F2 (v)
= F { cos(2π µ0 x) } F { cos(2π v0y) }
1
= [δ (µ − µ0 ) + δ (µ + µ0 )]
4
× [δ (v − v0) + δ (v + v0)],
(3.21)
where we derived entry #14 in Table 3-1. The CSFT
consists of four impulses at spatial frequency locations
{ µ , v } = { ±µ0 , ±v0 }, as is also shown in Fig. 3-5(b).
We should note that the images displayed in Fig. 3-5 are
3-4 2-D CONTINUOUS-SPACE FOURIER TRANSFORM (CSFT)
0
97
f (t)
Signal
1
0
−T
2
5 cm
T
2
t
(a) Rectangular pulse
|F( μ)|
Magnitude
spectrum
10 cm
y
0
5 cm
T
x
10 cm
(a) 2-D sinusoidal image f (x, y)
−3
T
v
13
−2
T
−1
T
0
1
T
2
T
3
T
μ
(b) Magnitude spectrum
ϕ( μ)
Phase
spectrum
180o
μ
0
−3
T
−13
−13
0
(b) F( μ,v), with axes in cycles/cm
−2
T
−1
T
0
1
T
2
T
3
T
μ
(c) Phase spectrum
13
Figure 3-6 (a) Rectangular pulse, and corresponding (b) magnitude spectrum and (c) phase spectrum.
Figure 3-5 (a) Sinusoidal image f (x, y)=cos(2π µ0 x) cos(2π v0 y)
with µ0 = 1.9 cycles/cm and v0 = 0.9 cycles/cm, and (b) the
corresponding Fourier transform F(µ , v), which consists of four
impulses (4 white dots) at { ±µ0 , ±v0 }.
not truly continuous functions; function f (x, y) was discretized
into 256 × 256 pixels and then a discrete form of the Fourier
transform called the DFT (Section 3-8) was used to generate
F(µ , v), also in the form of a 256 × 256 image.
D. Box Image
As a prelude to presenting the CSFT of a 2-D box image, let us
examine the 1-D case of the rectangular pulse f (t) = rect(t/T )
shown in Fig. 3-6(a). The pulse is centered at the origin
and extends between −T /2 and +T /2, and the corresponding
Fourier transform is, from entry #3 in Table 2-5,
F(µ ) = T sinc(µ T ),
www.dbooks.org
(3.22)
98
CHAPTER 3 2-D IMAGES AND SYSTEMS
where sinc(θ ) is the sinc function defined by Eq. (2.35) as
sinc θ = [sin(πθ )]/(πθ ). By defining F(µ ) as
F(µ ) = |F(µ )|e j φ (µ ) ,
y
(3.23)
we determine that the phase spectrum φ (µ ) can be ascertained
from
sinc(µ T )
F(µ )
=
.
(3.24)
e j φ (µ ) =
|F(µ )| | sinc(µ T )|
The quantity on the right-hand side of Eq. (3.24) is always equal
to +1 or −1. Hence, φ (µ ) = 0◦ when sinc(µ T ) is positive
and 180◦ when sinc(µ T ) is negative. The magnitude and phase
spectra of the rectangular pulse are displayed in Figs. 3-6(b)
and (c), respectively.
Next, let us consider the white square shown in Fig. 3-7(a).
If we assign an amplitude of 1 to the white part of the image
and 0 to the black part, the variation across the image along the
x direction is analogous to that representing the time-domain
pulse of Fig. 3-6(a), and the same is true along y. Hence, the
white square represents the product of two pulses, one along x
and another along y, and is given by
x
y
f (x, y) = rect
rect
,
(3.25)
ℓ
ℓ
x
l
(a) White square image
v
μ
where ℓ is the length of the square sides. In analogy with
Eq. (3.22),
F(µ , v) = ℓ2 sinc(µ ℓ) sinc(vℓ).
(3.26)
The magnitude and phase spectra associated with the expression given by Eq. (3.26) are displayed in grayscale format in
Fig. 3-7(b) and (c), respectively. For the magnitude spectrum,
white represents the peak value of |FBox (µ , v)| and black represents |FBox (µ , v)| = 0. The phase spectrum φ (µ , v) varies between 0◦ and 180◦ , so the grayscale was defined such that white
corresponds to +180◦ and black to 0◦ . The tonal variations along
µ and v are equivalent to the patterns depicted in Figs. 3-6(b)
and (c) for the rectangular pulse.
In the general case of a box image of widths ℓx along x and ℓy
along y, and centered at (x0 , y0 ),
x − x0
y − y0
f (x, y) = rect
rect
.
(3.27)
ℓx
ℓy
(b) Magnitude image |F( μ,v)|
v
μ
In view of properties #3 and 11 in Table 3-1, the corresponding
CSFT is
F(µ , v) = ℓx e− j2π µ x0 sinc(µ ℓx )ℓy e− j2π vy0 sinc(vℓy ),
(3.28)
(c) Phase image ϕ( μ,v)
Figure 3-7 (a) Grayscale image of a white square in a black
background, (b) magnitude spectrum, and (c) phase spectrum.
3-4 2-D CONTINUOUS-SPACE FOURIER TRANSFORM (CSFT)
99
and the associated magnitude and phase spectra are given by
|F(µ , v)|
(3.29a)
(3.29b)
and
φ (µ , v) = tan
−1
Im[F(µ , v)]
.
Re[F(µ , v)]
y
A visual example is shown in Fig. 3-8(a) for a square box of
sides ℓx = ℓy = ℓ, and shifted to the right by L and also downward
by L. Inserting x0 = L, y0 = −L, and ℓx = ℓy = ℓ in Eq. (3.28)
leads to
F(µ , v) = ℓ2 e− j2π µ L sinc(µ ℓ) e j2π vL sinc(vℓ).
x
L
l
(3.30)
L
(a) Box image f (x,y)
The magnitude and phase spectra associated with the CSFT of
the box image defined by Eq. (3.30) are displayed in Fig. 3-8(b)
and (c), respectively. The magnitude spectrum of the shifted box
is similar to that of the unshifted box (Fig. 3-7), but the phase
spectra of the two boxes are considerably different.
v
E. 2-D Ideal Brickwall Lowpass Filter
An ideal lowpass filter is characterized by a spatial frequency
response HLP (µ , v) with a specified cutoff frequency µ0 along
both the µ and v axes:
(
1
for 0 ≤ |µ |, |v| ≤ µ0 ,
(3.31)
HLP (µ , v) =
0
otherwise.
μ
Mathematically, HLP (µ , v) can be expressed in terms of rectangle functions centered at the origin and of width 2 µ0 :
µ
HLP (µ , v) = rect
2 µ0
v
rect
2 µ0
(b) |F( μ,v)|
v
.
(3.32)
The inverse 2-D Fourier transfer of HLP (µ , v) is the PSF
hLP (x, y). Application of property #12 in Table 3-1 yields
hLP (x, y) =
μ
sin(2π xµ0 ) sin(2π yv0 )
πx
πy
= 4 µ02 sinc(2xµ0 ) sinc(2yv0 ).
ωx
(3.33)
F. Example: Lowpass-Filtering of Clown Image
Figure 3-9(a) displays an image of a clown’s face. Our goal is to
lowpass-filter the clown image using an ideal lowpass filter with
a cutoff frequency µ0 = 0.44 cycles/mm. To do so, we perform
(c) ϕ( μ,v)
Figure 3-8 (a) Image of a box image of dimension ℓ and centered
at (L, −L), (b) magnitude spectrum, and (c) phase spectrum.
www.dbooks.org
100
CHAPTER 3 2-D IMAGES AND SYSTEMS
(x, y) Domain
( μ,v) Domain
v
FT
(a) Clown face image f (x,y)
μ
(b) Magnitude spectrum of clown image F( μ,v)
×v
IFT
(d) Spatial frequency response of 2-D LPF, HLP( μ,v)
=
(c) Magnified PSF h(x,y) of 2-D LPF
with μ0 = 0.44 cycles/mm
μ
v
IFT
μ
(e) Lowpass-filtered clown image g(x,y)
(f ) Magnitude spectrum of filtered image G( μ,v)
Figure 3-9 Lowpass filtering the clown image in (a) to generate the image in (e). Image f (x, y) is 40 mm × 40 mm and the magnitude
spectra extend between −2.5 cycles/mm and +2.5 cycles/mm in both directions.
3-4 2-D CONTINUOUS-SPACE FOURIER TRANSFORM (CSFT)
the following steps:
(1) We denote the intensity distribution of the clown image
as f (x, y), and we apply the 2-D Fourier transform to obtain the
spectrum F(µ , v), whose magnitude is displayed in Fig. 3-9(b).
(2) The spatial frequency response of the lowpass filter, shown
in Fig. 3-9(d), consists of a white square representing the
passband of the filter. Its functional form is given by Eq. (3.32)
with µ0 = 0.44 cycles/mm. The corresponding PSF given by
Eq. (3.33) is displayed in Fig. 3-9(c).
(3) Multiplication of F(µ , v) by HLP (µ , v) yields the spectrum
of the filtered image, G(µ , v):
G(µ , v) = F(µ , v) HLP (µ , v).
(3.34)
The magnitude of the result is displayed in Fig. 3-9(f). Upon
performing an inverse Fourier transform on G(µ , v), we obtain
g(x, y), the lowpass-filtered image of the clown face shown in
Fig. 3-9(e). Image g(x, y) looks like a blurred version of the
original image f (x, y) because the lowpass filtering smooths out
rapid variations in the image.
Alternatively, we could have obtained g(x, y) directly by
performing a convolution in the spatial domain:
g(x, y) = f (x, y) ∗ ∗hLP(x, y).
101
y
x′
y′
μ′
v′
θ
θ
x
μ
(a) Spatial domain
(b) Frequency domain
Figure 3-10 Rotation of axes by θ in (a) spatial domain causes
rotation by the same angle in the (b) spatial frequency domain.
of the rotated image g(x, y) = f (x′ , y′ ), and Rθ is the rotation
matrix relating (x, y) to (x′ , y′ ):
(3.35)
Even though the convolution approach is direct and conceptually
straightforward, it is computationally much easier to perform the
filtering by transforming to the angular frequency domain, multiplying the two spectra, and then inverse transforming back to
the spatial domain. The actual computation was performed using
discretized (pixelated) images and the Fourier transformations
were realized using the 2-D DFT introduced later in Section 3-8.
v
Rθ =
sin θ
.
cos θ
cos θ
− sin θ
(3.38)
The inverse relationship between (x′ , y′ ) and (x, y) is given in
terms of the inverse of matrix Rθ :
′ x
cos θ − sin θ x′
−1 x
= Rθ
=
.
(3.39)
y′
y′
y
sin θ
cos θ
The 2-D Fourier transform of g(x, y) is given by
3-4.2 Image Rotation
G(µ , v) =
◮ Rotating an image by angle θ (Fig. 3-10) in the 2-D
spatial domain (x, y) causes its Fourier transform to also
rotate by the same angle in the frequency domain (µ , v). ◭
To demonstrate the validity of the assertion, we start with the
relationships given by Eqs. (3.11) and (3.12):
g(x, y) = f (x′ , y′ )
and
′
x
x
=
R
θ y ,
y′
=
Z ∞Z ∞
−∞ −∞
Z ∞Z ∞
−∞ −∞
g(x, y) e− j2π (µ x+vy) dx dy
f (x′ , y′ ) e− j2π (µ x+vy) dx dy.
(3.40)
Using the relationships between (x, y) and (x′ , y′ ) defined by
Eq. (3.39), while also recognizing that dx dy = dx′ dy′ because
a differential element of area is the same in either coordinate
system, Eq. (3.40) becomes
(3.36)
G(µ , v)
(3.37)
where f (x, y) is the original image, (x′ , y′ ) are the coordinates
=
=
Z ∞Z ∞
−∞ −∞
Z ∞Z ∞
−∞ −∞
www.dbooks.org
′
′
′
′
f (x′ , y′ ) e− j2π [µ (x cos θ −y sin θ )+v(x sin θ +y cos θ )] dx′ dy′
′ ′
′ ′
f (x′ , y′ ) e− j2π [µ x +v y ] dx′ dy′ ,
(3.41)
102
CHAPTER 3 2-D IMAGES AND SYSTEMS
y
coordinates (ρ , φ ), with
µ = ρ cos φ
v = ρ sin φ
v
(x,y)
y
(μ,v)
v
ρ
r
θ
x
μ
μ
F(µ , v) =
(a) Spatial domain
(b) Frequency domain
Figure 3-11 Relationships between Cartesian and polar coordinates in (a) spatial domain and (b) spatial frequency domain.
where we define
′ cos θ
µ
=
v′
− sin θ
sin θ
cos θ
p
ρ = µ 2 + v2
.
φ = tan−1 (v/µ )
(3.45)
The Fourier transform of f (x, y) is given by Eq. (3.16a) as
ϕ
x
µ
µ
= Rθ
.
v
v
(3.42)
The newly defined spatial-frequency coordinates (µ ′ , v′ ) are
related to the original frequency coordinates (µ , v) by exactly
the same rotation matrix Rθ that was used to rotate image f (x, y)
to g(x, y). The consequence of using Eq. (3.42) is that Eq. (3.38)
now assumes the standard form for the definition of the Fourier
transform for f (x′ , y′ ):
G(µ , v) = F(µ ′ , v′ ).
(3.43)
In conclusion, we have demonstrated that rotation of image
f (x, y) by angle θ in the (x, y) plane leads to rotation of F(µ , v)
by exactly the same angle in the spatial frequency domain.
3-4.3 2-D Fourier Transform in Polar
Coordinates
In the spatial domain, the location of a point can be specified by its (x, y) coordinates in a Cartesian coordinate system
or by its (r, θ ) in the corresponding polar coordinate system
(Fig. 3-11(a). The two pairs of variables are related by
p
x = r cos θ
r = x2 + y2
.
(3.44)
y = r sin θ
θ = tan−1 (y/x)
Similarly, in the spatial frequency domain (Fig. 3-11(b)), we
can use Cartesian coordinates (µ , v) or their corresponding polar
Z ∞Z ∞
−∞ −∞
f (x, y) e− j2π (µ x+vy) dx dy.
(3.46)
We wish to transform F(µ , v) into polar coordinates so we
may apply it to circularly symmetric images or to use it in
filtering applications where the filter’s frequency response is
defined in terms of polar coordinates. To that end, we convert
the differential area dx dy in Eq. (3.46) to r dr d θ , and we use
the relations given by Eqs. (3.44) and (3.45) to transform the
exponent in Eq. (3.46):
µ x + vy = (ρ cos φ )(r cos θ ) + (ρ sin φ )(r sin θ )
= ρ r[cos φ cos θ + sin φ sin θ ]
(3.47)
= ρ r cos(φ − θ ).
The cosine addition formula was used in the last step. Conversion to polar coordinates leads to
F(ρ , φ ) =
Z ∞ Z 2π
r=0 θ =0
f (r, θ ) e− j2πρ r cos(φ −θ ) r dr d θ . (3.48a)
The inverse transform is given by
f (r, θ ) =
Z ∞ Z 2π
ρ =0 φ =0
F(ρ , φ ) e j2πρ r cos(φ −θ ) ρ d ρ d φ . (3.48b)
3-4.4 Rotationally Invariant Images
A rotationally invariant image is a circularly symmetric image,
which means that f (r, θ ) is a function of r only. According to
Section 3-4.2, rotation of f (r, θ ) by a fixed angle θ0 causes
the transform F(ρ , φ ) to rotate by exactly the same angle θ0 .
Hence, if f (r, θ ) is independent of θ , it follows that F(ρ , φ ) is
independent of φ , in which case Eq. (3.48a) can be rewritten as
F(ρ ) =
=
Z ∞ Z 2π
f (r) e− j2πρ r cos(φ −θ ) r dr d θ
Z 2 π
− j2πρ r cos(φ −θ )
d θ dr.
r f (r)
e
r=0 θ =0
Z ∞
r=0
θ =0
(3.49)
3-4 2-D CONTINUOUS-SPACE FOURIER TRANSFORM (CSFT)
103
J0(z)
1
y
3 cm
0.5
0
−0.5
z
0
2
4
6
8
10
12
14
16
18
0
x
20
Figure 3-12 Plot of J0 (z) the Bessel function of order zero, as a
function of z.
−3 cm
−3 cm
0
3 cm
(a) Ring impulse
v
Because the integration over θ extends over the range (0, 2π ),
the integrated value is the same for any fixed value of φ . Hence,
for simplicity we set φ = 0, in which case Eq. (3.49) simplifies
to
Z 2 π
Z ∞
− j2πρ r cos θ
r f (r)
d θ dr
e
F(ρ ) =
θ =0
r=0
= 2π
Z ∞
r=0
21
r f (r) J0 (2πρ r) dr,
μ
0
(3.50)
where J0 (z) is the Bessel function of order zero:
1
J0 (z) =
2π
Z 2π
0
e− jz cos θ d θ .
(3.51)
A plot of J0 (z) versus z is shown in Fig. 3-12.
The integral expression on the right-hand side of Eq. (3.50)
is known as the Hankel transform of order zero. Hence, the
Fourier transform of a circularly symmetric image f (r) is given
by its Hankel transform of order zero. An example is the ring
impulse
f (r) = δ (r − a),
(3.52a)
−21
−21
0
(b) Fourier transform of ring impulse
21
Figure 3-13 (a) Image of ring impulse of radius a = 1 cm and
(b) the logarithm of its 2-D CTFT. [In display mode, the images
are 256 × 256 pixels.]
which defines a unit-intensity circle of radius a (Fig. 3-13(a)) in
the spatial coordinate system (r, θ ). The corresponding Fourier
www.dbooks.org
104
CHAPTER 3 2-D IMAGES AND SYSTEMS
Table 3-2 2-D Fourier transforms of rotationally invariant
images.
f (r)
F(ρ )
δ (r)
πr
1
rect(r)
J1 (π r)
2r
1
r
2
e− π r
δ (r − r0 )
J1 (πρ )
2ρ
rect(ρ )
1
ρ
2
e−πρ
(a) Letters image f (x,y)
2π r0 J0 (2π r0 ρ )
transform is
F(ρ ) = 2π
Z ∞
r=0
r δ (r − a) J0 (2πρ r) dr = 2π a J0 (2πρ a).
(3.52b)
The image in Fig. 3-13(b) displays the variation of F(ρ ) as a
function of ρ in the spatial frequency domain for a ring with
a = 1 cm (image size is 6 cm × 6 cm).
Table 3-2 provides a list of Fourier transform pairs of rotationally symmetric images.
3-4.5 Image Examples
A. Scaling
Figure 3-14 compares image f (x, y), representing an image of
letters, to a scaled-down version f (x′ , y′ ) with x′ = ax, y′ = ay,
and a = 4. The area of the scaled-down image is 1/16 of the area
of the original. To enlarge the image, the value of a should be
smaller than 1.
B. Image Rotation
The image displayed in Fig. 3-15(a) is a sinusoidal image that
oscillates along only the y direction. Its 2-D spectrum consists
of two impulse functions along the v direction, as shown in
Fig. 3-15(b). Rotating the sinusoidal image by 45◦ to the image
in Fig. 3-15(c) leads to a corresponding rotation of the spectrum,
as shown in Fig. 3-15(d).
(b) Letters image f (x′,y′) with x′ = ax and y′ = ay,,
spatially scaled by a = 4
Figure 3-14 (a) Letters image and (b) a scaled version.
C. Gaussian Image
A 2-D Gaussian image is characterized by
2
2
f (x, y) = e−π (x +y ) .
Gaussian image
(3.53a)
Since x2 + y2 = r2 , f (x, y) is rotationally invariant, so we can
rewrite it as
2
f (r) = e−π r .
(3.53b)
To obtain the Fourier transform F(ρ ), we can apply Eq. (3.50),
the Fourier transform for a rotationally invariant image:
F(ρ ) = 2π
= 2π
Z ∞
Z0 ∞
0
r f (r) J0 (2πρ r) dr
2
re−π r J0 (2πρ r) dr.
(3.54)
3-4 2-D CONTINUOUS-SPACE FOURIER TRANSFORM (CSFT)
105
v
μ
x
y
(a) Sinusoidal image
(b) Spectrum of image in (a)
v
μ
x
y
(c) Sinusoidal image rotated by 45◦
(d) Spectrum of rotated sinusoidal image
Figure 3-15 (a) Sinusoidal image and (b) its 2-D spectrum; (c) rotated image and (d) its rotated spectrum.
From standard tables of integrals, we borrow the following
identity for any real variable t:
Z ∞
0
2 2
te−a t J0 (bt) dt =
1 −b2 /4a2
e
,
2a2
The integrals in Eq. (3.54) and Eq. (3.55) become identical if we
set t = r, a2 = π , and b = 2πρ , which leads to
for a2 > 0. (3.55)
www.dbooks.org
2
F(ρ ) = e−πρ .
Gaussian spectrum
(3.56)
106
CHAPTER 3 2-D IMAGES AND SYSTEMS
◮ Hence, the Fourier transform of a 2-D Gaussian image is
itself 2-D Gaussian. ◭
D. Disk Image
A disk image has a value of 1 inside the disk area and zero
outside it. A disk image centered at the origin and of radius 1/2
is characterized by Eq. (3.3) in (x, y) coordinates. Conversion to
polar coordinates gives
(
1
for 0 ≤ r < 1/2,
fDisk (r) = rect(r) =
(3.57)
0
otherwise.
After much algebra, it can be shown that the corresponding
Fourier transform is given by
FDisk (ρ ) =
J1 (πρ )
= jinc(ρ ),
2ρ
(3.58)
where J1 (x) is the Bessel function of order 1, and
jinc(x) = J1 (π x)/(2x) is called the jinc function, which comes
from its resemblance in both waveform and purpose to the
sinc function defined by Eq. (2.35), except that the numerator
changes from a sine to a Bessel function. Figure 3-16 displays
a plot of jinc(ρ ), as well as a plot of the sinc function sinc(ρ ),
included here for comparison. In one dimension, the Fourier
transform of rect(x) is sinc(µ ); in two dimensions, the Fourier
transform of a disk image fDisk (r) = rect(r) is given by jinc(ρ ),
which resembles the variation exhibited by the sinc function.
1
sinc( ρ) =
0.5
sin(πρ)
πρ
jinc( ρ) =
By Fourier duality, or equivalently by application of the inverse
transformation given by Eq. (3.48b), we obtain the PSF of the
lowpass filter as
hLP (r) =
J1 (2πρ0r)(2ρ0 )
= (2ρ0 )2 jinc(2ρ0 r).
2r
(3.59)
◮ We should always remember the scaling property of the
Fourier transform, namely if
f (r)
F(ρ ),
then for any real-valued scaling factor a,
ρ 1
F
.
f (ar)
|a|2
a
The scaling property allows us to use available expressions,
such as those in Eqs. (3.58) and (3.59), and to easily
convert them into the expressions appropriate (for example)
to disks of different sizes or filters with different cutoff
frequencies. ◭
Concept Question 3-3: Why do so many 1-D Fourier
transform properties generalize directly to 2-D?
J1(πρ)
2ρ
ρ
0
−0.5
0
extends between 0 and ρ0 , and is given by
(
ρ
1
for 0 < |ρ | < ρ0 ,
=
HLP (ρ ) = rect
2ρ0
0
otherwise.
Concept Question 3-4:
Where does the jinc function get
its name?
1
2
3
4
5
6
7
8
9
10
Figure 3-16 Sinc function (blue) and jinc function (red).
Exercise 3-6: Why do f (x, y) and f (x − x0 , y − y0 ) have the
same magnitude spectrum |F(µ , ν )|?
Answer: Let g(x, y) = f (x − x0 , y − y0 ). Then, from entry
#3 in Table 3-1,
E. PSF of Radial Brickwall Lowpass Filter
In the spatial frequency domain, the frequency response of a
radial brickwall lowpass filter with cutoff spatial frequency ρ0
|G(µ , ν )| = |e− j2π µ x0 e− j2πν y0 F(µ , ν )|
= |e− j2π µ x0 e− j2πν y0 ||F(µ , ν )| = |F(µ , ν )|.
3-5 2-D SAMPLING THEOREM
107
2
Exercise 3-7: Compute the 2-D CSFT of f (x, y) = e−π r ,
where r2 = x2 + y2 , without using Bessel functions. Hint:
f (x, y) is separable.
−π r 2
−π x2
y
−π y2
Answer: f (x, y) = e
=e
e
is separable, so
Eq. (3.19) and entry #5 of Table 2-5 (see also entry #13
of Table 3-1) give
2
2
2
2
2
F(µ , ν ) = e−π µ e−πν = e−π (µ +ν ) = e−πρ .
Exercise 3-8: The 1-D phase spectrum φ (µ ) in Fig. 3-6(c)
is either 0 or 180◦ for all µ . Yet the phase of the 1-D
CTFT of a real-valued function must be an odd function
of frequency. How can these two statements be reconciled?
x
Figure 3-17 The “bed of nails” function
∑ ∑ δ (x − n∆) δ (y − m∆).
Replacing 180◦ with −180◦ for µ < 0 in Fig. 3-6(c) makes
the phase φ (µ ) an odd function of µ .
3-5 2-D Sampling Theorem
The sampling theorem generalizes directly from 1-D to 2-D
using rectangular sampling:
f [n, m] = f (n∆, m∆) = f (n/S, m/S)
(3.60)
where ∆ is the sampling length (instead of interval) and
Sx = 1/∆ is the sampling rate in samples/meter.
If the spectrum of image f (x, y) is bandlimited to B—that is,
F(µ , v) = 0 outside the square region defined by
{ (µ , v) : 0 ≤ |µ |, |v| ≤ B },
then the image f (x, y) can be reconstructed from its samples
f [m, n], provided the sampling rate is such that S > 2B. As in
1-D, 2B is called the Nyquist sampling rate, although the units
are now samples/meter instead of samples/second.
The sampled signal xs (t) defined by Eq. (2.43) generalizes
directly to the sampled image:
∞
fs (x, y) =
∞
∑ ∑ f (n∆, m∆)
n=−∞ m=−∞
× [δ (x − n∆) δ (y − m∆)].
(3.61)
The term inside the square brackets (product of two impulse
trains) is called the bed of nails function, because it consists
of a 2-D array of impulses, as shown in Fig. 3-17.
∞
∞
Answer: A phase of 180◦ is equivalent to a phase of −180◦ .
n=−∞ m=−∞
Conceptually, image f (x, y) can be reconstructed from its
discretized version f (n∆, m∆) by applying the 2-D version of
the sinc interpolation formula. Generalizing Eq. (2.51) to 2-D
gives
∞
f (x, y) =
∞
∑ ∑ f (n∆, m∆)
n=−∞ m=−∞
×
sin(π S(x − n∆)) sin(π S(y − m∆))
.
π S(x − n∆)
π S(y − m∆)
(3.62)
As noted earlier in Section 2-4.4 in connection with Eq. (2.51),
accurate reconstruction using the sinc interpolation formula
is not practical because it requires summations over infinite
number of samples.
3-5.1 Sampling/Reconstruction Examples
The following image examples are designed to illustrate the important role of the Nyquist rate when sampling an image f (x, y)
(for storage or digital transmission) and then reconstructing it
from its sampled version fs (x, y). We will use the term image
reconstruction fidelity as a qualitative measure of how well
the reconstructed image frec (x, y) resembles the original image
f (x, y).
Reconstruction of frec (x, y) from the sampled image fs (x, y)
can be accomplished through either of two approaches:
(a) Application of nearest-neighbor (NN) interpolation
(which is a 2-D version of the 1-D nearest-neighbor interpola-
www.dbooks.org
108
CHAPTER 3 2-D IMAGES AND SYSTEMS
tion), implemented directly on image fs (x, y).
(b) Transforming image fs (x, y) to the frequency domain,
applying 2-D lowpass filtering (LPF) to simultaneously preserve
the central spectrum of f (x, y) and remove all copies thereof
(generated by the sampling process), and then inverse transforming to the spatial domain.
Both approaches will be demonstrated in the examples that
follow, and in each case we will compare an image reconstructed
from an image sampled at the Nyquist rate with an aliased image
reconstructed from an image sampled at a rate well below the
Nyquist rate. In all cases, the following parameters apply:
• Size of original (clown) image f (x, y) and reconstructed
image frec (x, y): 40 mm × 40 mm
• Sampling interval ∆ (and corresponding sampling rate
S = 1/∆) and number of samples N:
• Nyquist-sampled version: ∆ = 0.2 mm, S = 5 samples/mm, N = 200 × 200
• Sub–Nyquist-sampled version: ∆ = 0.4 mm, S = 2.5
sample/mm, N = 100 × 100
• Spectrum of original image f (x, y) is bandlimited to
B = 2.5 cycles/mm
• Display
• Images f (x, y), fs (x, y), frec (x, y): Linear scale
• Image magnitude spectra: Logarithmic scale (for
easier viewing; magnitude spectra extend over a wide
range).
Reconstruction Example 1:
Image Sampled at Nyquist Rate
Our first step is to create a bandlimited image f (x, y). This was
done by transforming an available clown image to the spatial
frequency domain and then applying a lowpass filter with a
cutoff frequency of 2.5 cycles/mm. The resultant image and its
corresponding spectrum are displayed in Figs. 3-18(a) and (b),
respectively.
A. LPF Reconstruction
Given that image f (x, y) is bandlimited to B = 2.5 cycles/mm,
the Nyquist rate is 2B = 5 cycles/mm. Figure 3-18(c) displays
fs (x, y), a sampled version of f (x, y), sampled at the Nyquist
rate, so it should be possible to reconstruct the original image with good fidelity. The spectrum of fs (x, y) is displayed
in part (d). The spectrum of the sampled image contains
the spectrum of the original image (namely, the spectrum in
Fig. 3-18(b)), plus periodic copies spaced at an interval S along
both directions in the spatial frequency domain. To preserve the
central spectrum and simultaneously remove all of the copies,
a lowpass filter is applied in step (f) of Fig. 3-18. Finally.
application of the 2-D inverse Fourier transform to the spectrum
in part (f) leads to the reconstructed image frec (x, y) in part (e).
We note that the process yields a reconstructed image with highfidelity resemblance to the original image f (x, y).
B. NN Reconstruction
Figure 3-19 displays image f (x, y), sampled image fs (x, y), and
the NN reconstructed image f (x, y). The last step was realized
using a 2-D version of the nearest-neighbor interpolation technique described in Section 2-4.4D. NN reconstruction provides
image
fˆ(x, y) = fs (x, y) ∗ ∗ rect(x/∆) rect(y/∆),
(3.63)
which is a 2-D convolution of the 2-D sampled image fs (x, y)
with a box function. The spectrum of the NN interpolated signal
is
sin(π ∆µ ) sin(π ∆v)
.
(3.64)
F̂(µ , v) = F(µ , v)
πµ
πv
As in 1-D, the zero crossings of the 2-D sinc functions coincide
with the centers of the copies of F(µ , v) induced by sampling.
Consequently, the 2-D sinc functions act like lowpass filters
along µ and v, serving to eliminate the copies almost completely.
Comparison of the NN-interpolated image in Fig. 3-19(c)
with the original image in part (a) of the figure leads to the
conclusion that the NN technique works quite well for images
sampled at or above the Nyquist rate.
Reconstruction Example 2:
Image Sampled below the Nyquist Rate
A. LPF Reconstruction
The sequence in this example (Fig. 3-20) is identical with that
described earlier in Example 1A, except for one very important
difference: in the present case the sampling rate is S = 2.5
cycles/mm, which is one-half of the Nyquist rate. Consequently,
the final reconstructed image in Fig. 3-20(e) bears a poor
resemblance to the original image in part (a) of the figure.
3-5 2-D SAMPLING THEOREM
109
v
μ
(a) Bandlimited image f (x,y)
(b) Spectrum F( μ,v) of image f (x,y)
Sampling at S = 5 samples/mm
v
FT
μ
(c) Nyquist-sampled image fs(x,y)
(d) Spectrum Fs( μ,v) of sampled image
Filtering
v
IFT
μ
(e) LPF reconstructed image frec(x,y)
(f ) Lowpass-filtered spectrum
Figure 3-18 Reconstruction Example 1A: After sampling image f (x, y) in (a) to generate fs (x, y) in (c), the sampled image is Fourier
transformed [(c) to (d)], then lowpass-filtered [(d) to (f)] to remove copies of the central spectrum) and inverse Fourier transform [(f) to (e)]
to generate the reconstructed image frec (x, y). All spectra are displayed in log scale.
www.dbooks.org
110
CHAPTER 3 2-D IMAGES AND SYSTEMS
B. NN Reconstruction
The sequence in Fig. 3-21 parallels the sequence in Fig. 3-19,
except that in the present case we are working with the subNyquist sampled image. As expected, the NN interpolation
technique generates a poor-fidelity reconstruction, just like the
LPF reconstructed version.
3-5.2 Hexagonal Sampling
(a) Bandlimited image f (x,y)
Sampled at
S = 5 samples/mm
(b) Nyquist-sampled image fs(x,y)
NN
(c) NN reconstructed image frec(x,y)
Figure 3-19 Reconstruction Example 1B: Nearest-neighbor
(NN) interpolation for Nyquist-sampled image. Sampling rate is
S = 5 samples/mm.
The transformation defined by Eq. (3.48a) converts image
f (r, θ )—expressed in terms of spatial polar coordinates (r, θ )—
to its spectrum F(ρ , φ )—expressed in terms of radial frequency
ρ and associated azimuth angle φ . Spectrum F(ρ , φ ) is said to
be radially bandlimited to radial frequency ρ0 if:
F(ρ , φ ) = 0
for ρ > ρ0 .
If f (r, θ ) is sampled along a rectangular grid—the same as
when sampling f (x, y) in rectangular coordinates—at a sampling spacing ∆rect (Fig. 3-22(a)) and corresponding sampling
rate S = 1/∆rec such that S ≥ 2ρ0 (to satisfy the Nyquist rate),
then the spectrum Fs (ρ , φ ) of the sampled image fs (r, θ ) would
consist of a central disk of radius ρ0 , as shown in Fig. 3-22(b),
plus additional copies at a spacing S along both the µ and v
directions. The term commonly used to describe the sampling in
(x, y) space is tiling; the image space in Fig. 3-22(a) is tiled with
square pixels.
Square tiling is not the only type of tiling used to sample
2-D images. A more efficient arrangement in terms of data rate
(or total number of samples per image) is to tile the image
space using hexagons instead of squares. Such an arrangement
is shown in Fig. 3-23(a) and is called hexagonal sampling.
The image space is tiled with hexagons. The spacing along y is
unchanged (Fig. 3-23(a)), but the spacing along x has changed
to
2
∆hex = √ ∆rect = 1.15∆rect .
3
The modest wider spacing along x translates into fewer samples
needed to tile the image, and more efficient utilization of the
spatial frequency space (Fig. 3-23(b)).
Hexagonal sampling is integral to how the human vision system functions, in part because our photoreceptors are arranged
along a hexagonal lattice. The same is true for other mammals
as well.
Reconstruction of f (r, θ ) from its hexagonal samples entails
the application of a radial lowpass filter with cutoff frequency
ρ0 to Fs (ρ , φ ), followed by an inverse Fourier transformation
(using Eq. (3.48b)) to the (r, θ ) domain. A clown image ex-
3-5 2-D SAMPLING THEOREM
111
v
μ
(a) Bandlimited image f (x,y)
(b) Spectrum F( μ,v) of image f (x,y)
Sampling at S = 2.5 sample/mm
v
FT
μ
(c) Sub-Nyquist sampled image fs(x,y)
(d) Spectrum Fs( μ,v) of sampled image
Filtering
v
IFT
μ
(e) LPF reconstructed image frec(x,y)
(f ) Lowpass-filtered spectrum
Figure 3-20 Reconstruction Example 2A: Image f (x, y) is sampled at half the Nyquist rate (S = 2.5 sample/mm compared with 2B = 5
samples/mm). Consequently, the reconstructed image in (e) bears a poor resemblance to the original image in (a). All spectra are displayed in
log scale.
www.dbooks.org
112
CHAPTER 3 2-D IMAGES AND SYSTEMS
∆rec
∆rec
(a) Bandlimited image f (x,y)
Sampled at
S = 2.5 samples/mm
x
y
(a) Square tiling of f ( x,y)
ν
(b) Sub-Nyquist sampled image fs(x,y)
NN
ρ0
0
μ
(c) NN reconstructed image frec(x,y)
Figure 3-21 Reconstruction Example 2B: Nearest-neighbor
interpolation for sub-Nyquist-sampled image.
(b) Spectrum Fs( ρ,ϕ)
Figure 3-22 (a) Square tiling at a spacing ∆rect and a sampling
rate S = 1/∆rect ≥ 2ρ0 , and (b) corresponding spectrum Fs (ρ , φ )
for an image radially bandlimited to spatial frequency ρ0 .
3-6 2-D DISCRETE SPACE
113
ample with hexagonal sampling at the Nyquist rate is shown
in Fig. 3-24. Note that the clown image has been antialiased
(lowpass-filtered prior to hexagonal sampling) so that the copies
of the spectrum created by hexagonal sampling do not overlap in
Fig. 3-24(d). This is why the clown image looks blurred, but the
reconstructed clown image matches the original blurred image.
∆rec
Concept Question 3-5: Why is the sampling theorem
important in image processing?
Exercise 3-9: An image is spatially bandlimited to 10
cycles/mm in both the x and y directions. What is the
minimum sampling length ∆s required in order to avoid
aliasing?
∆hex
y
x
1
1
= 20
mm.
Answer: ∆s < 2B
Exercise 3-10: The 2-D CSFT of a 2-D impulse is 1, so it
is not spatially bandlimited. Why is it possible to sample an
impulse?
(a) Hexagonal tiling of f ( x,y)
Answer: In general, it isn’t. Using a sampling interval of
∆s , the impulse δ (x − x0 , y − y0 ) will be missed unless x0
and y0 are both integer multiples of ∆s .
ν
3-6 2-D Discrete Space
3-6.1 Discrete-Space Images
ρ0
0
μ
A discrete-space image represents a physical quantity that
varies with discrete space [n, m], where n and m are dimensionless integers. Such an image usually is generated by sampling a
continuous-space image f (x, y) at a spatial interval ∆s along the
x and y directions. The sampled image is defined by
f [n, m] = f (n∆s , m∆s ).
(3.65)
The spatial sampling rate is Ss = 1/∆s , [n, m] denotes the
location of a pixel (picture element), and f [n, m] denotes the
value (such as image intensity) of that pixel.
A. Image Axes
(b) Spectrum Fs( ρ,ϕ)
Figure 3-23 (a) Hexagonal tiling and (b) corresponding spectrum Fs (ρ , φ ).
As noted earlier in connection with continuous-space images,
multiple different formats are used in both continuous- and
discrete-space to define image coordinates. We illustrate the
most common of these formats in Fig. 3-25. In the top of the
figure, we show pixel values for a 10 × 10 array. In parts (a)
www.dbooks.org
114
CHAPTER 3 2-D IMAGES AND SYSTEMS
v
μ
(a) Radially bandlimited image f (r,θ)
(b) Spectrum F( ρ,ϕ) of image f (r,θ)
Sampling at 2ρ0
v
FT
μ
(c) Hexagonally sampled image fs(r,θ)
(d) Spectrum Fs( ρ,ϕ) of sampled image
Filtering
v
IFT
μ
(e) Radial LPF reconstructed image frec(r,θ)
(f ) Radial lowpass-filtered spectrum
Figure 3-24 Hexagonal sampling and reconstruction example.
3-6 2-D DISCRETE SPACE
115
0 0 0 0 0 0 0 0 0 0
0 3 5 7 9 10 11 12 13 14
0 5 10 14 17 20 23 25 27 29
0 8 15 21 26 30 34 37 40 43
0 10 20 27 34 40 45 50 54 57
0 10 20 27 34 40 45 50 54 57
0 8 15 21 26 30 34 37 40 43
0 5 10 14 17 20 23 25 27 29
0 3 5 7 9 10 11 12 13 14
0 0 0 0 0 0 0 0 0 0
Pixel values
Origin
m
0
1
2
3
4
5
6
7
8
9
9
8
7
6
5
4
3
2
1
0
m
0 1 2 3 4 5 6 7 8 9
n
(a) Top-left corner format
0 1 2 3 4 5 6 7 8 9
n
(b) Bottom-left corner format
Origin
m
4
3
2
1
0
−1
−2
−3
−4
−5
n
−5 −4 −3 −2 −1
0 1 2 3 4
1
2
3
4
5
6
7
8
9
10
1 2 3 4 5 6 7 8 9 10
n′
m′
(c) Center-of-image format
(d) MATLAB format
Figure 3-25 The four color images are identical in pixel values, but they use different formats for the location of the origin and for coordinate
directions for image f [n, m]. The MATLAB format in (d) represents X(m′ , n′ ).
www.dbooks.org
116
CHAPTER 3 2-D IMAGES AND SYSTEMS
through (d), we display the same color maps corresponding to
the pixel array, except that the [n, m] coordinates are defined
differently, namely:
MATLAB format
In MATLAB, an (M × N) image is defined as
{ X(m′ , n′ ), 1 ≤ m′ ≤ M, 1 ≤ n′ ≤ N }.
Top-left corner format
Figure 3-25(a): [n, m] starts at [0, 0] and both integers extend
to 9, the origin is located at the upper left-hand corner, m increases downward, and n increases to the right. For a (M × N)
image, f [n, m] is defined as
{ f [n, m], 0 ≤ n ≤ N − 1, 0 ≤ m ≤ M − 1 }
(3.66a)
MATLAB uses the top-left corner format, except that its
indices n′ and m′ start at 1 instead of 0. Thus, the top-left corner
is (1, 1) instead of [0, 0]. Also, m′ , the first index in X(m′ , n′ ),
represents the vertical axis and the second index, n′ , represents
the horizontal axis, which is the reverse of the index notation
represented in f [n, m]. The two notations are related as follows:
or, equivalently,
f [n, m] =

f [0, 0]
 f [0, 1]

..

.
f [0, M − 1]
f [n, m] = X(m′ , n′ ),
(3.69a)
m′ = m + 1,
(3.69b)
with
f [1, 0] . . . f [N − 1, 0]
f [1, 1] . . . f [N − 1, 1]
..
.


.

(3.66b)
′
n = n + 1.
(3.69c)
f [1, M − 1] . . . f [N − 1, M − 1]
Bottom-left corner format
Common-Image vs. MATLAB Format
Figure 3-25(b): [n, m] starts at [0, 0] and both integers extend
to 9, the origin is located at the bottom left-hand corner, m increases upward, and n increases to the right. This is a vertically
flipped version of the image in Fig. 3-25(a), and f [n, m] has the
same definition given by Eq. (3.66a).
Center-of-image format
Figure 3-25(c): The axes directions are the same as in the image
of Fig. 3-25(b), except that the origin of the coordinate system
is now located at the center of the image. The ranges of n and
m depend on whether M and N are odd or even integers. If pixel
[0, 0] is to be located in the center in the [n, m] coordinate system,
then the range of m is
−
(3.68)
M−1
M−1
≤m≤
,
2
2
M
M
− ≤ m ≤ −1 ,
2
2
The format used by MATLAB to store images is different
from the conventional matrix format given in Eq. (3.66b) in
two important ways:
(1) Whereas the pixel at the upper left-hand corner of image
f [n, m] is f [0, 0], at location (0, 0), that pixel is denoted
X(1, 1) in MATLAB.
if M is odd,
(3.67a)
(2) In f [n, m], the value of n denotes the column that f [n, m]
resides within (relative to the most-left column, which is
denoted by n = 0), and m denotes the row that f [n, m]
resides within (relative to the top row, which is denoted by
m = 0). In MATLAB, the notation is reversed: the first index
of X(m′ , n′ ) denotes the row and the second index denotes
the column. Hence,
if M is even.
(3.67b)
f [n, m] = X(m′ , n′ )
A similar definition applies to index n.
with m′ and n′ related to n and m by Eq. (3.69). To
distinguish between the two formats, f [n, m] uses square
brackets whereas X(m′ , n′ ) uses curved brackets.
3-6 2-D DISCRETE SPACE
117
◮ From here on forward, the top-left-corner format will be
used for images, the center-of-image format will be used
for image spectra and PSFs, and the MATLAB format will
be used in MATLAB arrays. ◭
Symbolically, the 2-D convolution is represented by
f [n, m]
h[n, m]
g[n, m].
The properties of 1-D convolution are equally applicable in 2-D
discrete-space. Convolution of images of sizes (L1 × L2 ) and
(M1 × M2 ) yields an image of size (N1 × N2 ), where
B. Impulses and Shifts
In 2-D discrete-space, impulse δ [n, m] is defined as
(
1
if n = m = 0,
δ [n, m] = δ [n] δ [m] =
0
otherwise.
(3.70)
N1 = L1 + M1 − 1,
N2 = L2 + M2 − 1.
(3.72a)
(3.72b)
The 2-D convolution process is illustrated next through a simple
example.
In upper-left-corner image format, shifting an image
f [n, m] by m0 downward and n0 rightward generates image
f [n − n0, m − m0]. An example is shown in Fig. 3-26.
Example 3-1:
2-D Convolution
Image Size
N columns
Compute the 2-D convolution
5
1 2
∗∗
7
3 4
M rows
(M × N) image
m
n
As noted earlier in Chapter 1, the size of an image is denoted
by (# of rows × # of columns) = M × N.
3-6.2 Discrete-Space Systems
For discrete-space systems, linearity and shift invariance follow
analogously from their continuous-space counterparts. When a
linear shift-invariant (LSI) system characterized by a point
spread function h[n, m] is subjected to an input f [n, m], it
generates an output g[n, m] given by the 2-D convolution of
f [n, m] and h[n, m]:
= ∑
∞
∑ h[i, j] f [n − i, m − j].
i=−∞ j=−∞
Solution: By using entries in the first image as weights, we
have








5 6 0
0 5 6
0 0 0
0 0 0
1 7 8 0 + 2 0 7 8 + 3 5 6 0 + 4 0 5 6
0 0 0
0 0 0
7 8 0
0 7 8


5 16 12
= 22 60 40 .
21 52 32
Concept Question 3-6: Why do we bother studying
discrete-space images and systems, when almost all realworld systems and images are defined in continuous space?
Exercise 3-11: If g[n, m] = h[n, m] ∗ ∗ f [n, m], what is
4h[n, m] ∗ ∗ f [n − 3, m − 2] in terms of g[n, m]?
g[n, m] = h[n, m] ∗ ∗ f [n, m]
∞
6
.
8
(3.71)
4g[n − 3, m − 2], using the shift and scaling
properties of 1-D convolutions.
Answer:
www.dbooks.org
118
CHAPTER 3 2-D IMAGES AND SYSTEMS
Column 3 of f [n,m]
n
Image f [n,m]
Image f [n − 2, m − 1]
Row 5 of f [n,m]
m
Figure 3-26 Image f [n − 2, m − 1] is image f [n, m] shifted down by 1 and to the right by 2.
Exercise 3-12: Compute

1 2
4
1 2
Answer: 2

1
2 .
1
1
1
1
1
∗∗
1
1
Conjugate symmetry for real-valued images f [m, n] implies that
1
.
1
F∗ (Ω1 , Ω2 ) = F(−Ω1 , −Ω2 ).
As in the 2-D continuous-space Fourier transform, F(Ω1 , Ω2 )
must be reflected across both spatial frequency axes to produce
its complex conjugate.
The DSFT is doubly periodic in (Ω1 , Ω2 ) with periods 2π
along each axis, as demonstrated by Example 3-2.
3-7 2-D Discrete-Space Fourier
Transform (DSFT)
The 2-D discrete-space Fourier transform (DSFT) is obtained
via direct generalization of the 1-D DTFT (Section 2-6) to
2-D. The DSFT consists of a DTFT applied first along m
and then along n, or vice versa. By extending the 1-D DTFT
definition given by Eq. (2.73a) (as well as the properties listed
in Table 2-7) to 2-D, we obtain the following definition for the
DSFT F(Ω1 , Ω2 ) and its inverse f [n, m]:
∞
F(Ω1 , Ω2 ) =
f [n, m] =
∞
∑ ∑ f [n, m] e− j(Ω1n+Ω2 m) ,
n=−∞ m=−∞
Z π Z π
1
4π 2
×e
(3.74)
dΩ1 dΩ2 .
Example 3-2:
DSFT of Clown Image
(3.73a)
Use MATLAB to obtain the magnitude image of the DSFT of
the clown image.
(3.73b)
Solution: The magnitude part of the DSFT is displayed in
Fig. 3-27. As expected, the spectrum is periodic with period 2π
along both Ω1 and Ω2 .
F(Ω1 , Ω2 )
−π −π
j(Ω1 n+Ω2 m)
◮ The spectrum of an image f [n, m] is its DSFT F(Ω1 , Ω2 ).
The discrete-space frequency response H(Ω1 , Ω2 ) of an
LSI system is the DSFT of its point spread function (PSF)
h[n, m]. ◭
The properties of the DSFT are direct 2-D generalizations of
the properties of the DTFT, and discrete-time generalizations of
the properties of the 2-D continuous-space Fourier transform.
Concept Question 3-7:
three applications.
What is the DSFT used for? Give
3-8 2-D DISCRETE FOURIER TRANSFORM (2-D DFT)
119
format. That conversion is called the discrete Fourier transform
(DFT). With the DFT, both f [n, m] and its 2-D Fourier transform
operate in discrete domains. For an (M × N) image f [n, m],
generalizing Eq. (2.89) to 2-D leads to the 2-D DFT of order
(K2 × K1 ):
Ω2
4π
N−1 M−1
nk1 mk2
F[k1 , k2 ] = ∑ ∑ f [n, m] exp − j2π
+
,
K1
K2
n=0 m=0
Ω1
k1 = { 0, . . . , K1 − 1 },
k2 = { 0, . . . , K2 − 1 },
(3.75)
where we have converted (Ω1 , Ω2 ) into discrete indices (k1 , k2 )
by setting
2π
k1
Ω1 =
K1
and
−4π
−4π
4π
Ω2 =
Figure 3-27 DSFT magnitude of clown image (log scale).

1
h[n, m] = 2
1
2
4
2
Array F[k1 , k2 ] is given by the (K2 × K1 ) array


F[k1 , k2 ] = 

Exercise 3-13: An LSI system has a PSF given by

1
2 .
1
Compute its spatial frequency response H(Ω1 , Ω2 ). Hint:
h[n, m] is separable.
Answer: The DTFT was defined in Eq. (2.73). Recog-
nizing that h[n, m] = h1 [n] h1 [m] with h1 [n] = {1, 2, 1}, the
DSFT is the product of two 1-D DTFTs, each of the form
2π
k2 .
K2

F[1, 0] . . . F[K1 − 1, 0]
F[1, 1] . . . F[K1 − 1, 1]
..
.
F[0, 0]
F[0, 1]
..
.

.

F[0, K2 − 1] F[1, K2 − 1] . . .F[K1 − 1, K2 − 1]
(3.76)
Note that the indexing is the same as that used for f [n, m]. The
inverse DFT is
1 K1 −1 K2 −1
nk1 mk2
f [n, m] =
∑ F[k1 , k2 ] exp j2π K1 + K2 ,
K1 K2 k∑
=0 k =0
1
2
n = { 0, . . . , N − 1 },
m = { 0, . . . , M − 1 }.
(3.77)
◮ Note that if N < K1 and M < K2 , then the reconstructed image f [n, m] = 0 for N ≤ n ≤ K1 − 1 and
M ≤ m ≤ K2 − 1. ◭
H1 (Ω) = e jΩ + 2 + e− jΩ = 2 + 2 cos(Ω),
and therefore
H(Ω1 , Ω2 ) = [2 + 2 cos(Ω1 )][2 + 2 cos(Ω2 )].
3-8.1 Properties of the 2-D DFT
3-8 2-D Discrete Fourier Transform
(2-D DFT)
According to Eq. (3.73), while f [n, m] is a discrete function,
its DSFT F(Ω1 , Ω2 ) is a continuous function of Ω1 and Ω2 .
Numerical computation using the fast Fourier transform (FFT)
requires an initial step of converting F(Ω1 , Ω2 ) into discrete
The 2-D DFT, like the 2-D CSFT and 2-D DSFT, consists of a
1-D transform along either the horizontal or vertical direction,
followed by another 1-D transform along the other direction.
Accordingly, the 2-D DFT has the following properties listed
in Table 3-3, which are direct generalizations of the 1-D DFT
properties listed earlier in Table 2-9.
The cyclic convolution h[n] c x[n] was defined in Eq. (2.104).
The 2-D DFT maps 2-D cyclic convolutions to products, and
www.dbooks.org
120
CHAPTER 3 2-D IMAGES AND SYSTEMS
Table 3-3 Properties of the (K2 × K1 ) 2-D DFT. In the time-shift and modulation properties, (k1 − k1′ ) and (n − n0 ) must be reduced
mod(K 1 ), and (k2 − k2′ ) and (m − m0 ) must be reduced mod(K 2 ).
Selected Properties
∑ ci fi [n, m]
1. Linearity
2. Shift
3. Modulation
e
4. Reversal
5. Convolution
∑ ci F[k1 , k2 ]
f [(n − n0 ), (m − m0 )]
e− j2π k1 n0 /K1 e− j2π k2 m0 /K2 F[k1 , k2 ]
f [(N − n), (M − m)]
F[(K 1 − k1 ), (K 2 − k2 )]
j2π k1′ n/K1
e
j2π k2′ m/K2
f [n, m]
h[n, m] c c f [n, m]
F[(k1 − k1′ ), (k2 − k2′ )]
H[k1 , k2 ] F[k1 , k2 ]
Special DFT Relationships
6. Conjugate Symmetry for f [n, m] real
F∗ [k1 , k2 ] = F[(K 1 − k1 ), (K 2 − k2 )]
7. Zero spatial frequency
F[0, 0] = ∑ ∑ f [n, m]
N−1 M−1
8. Spatial origin
f [0, 0] =
N−1 M−1
9. Rayleigh’s theorem
1
K1 K2
∑ ∑ F[k1 , k2 ]
k1 =0 k2 =0
1
K1 −1 K2 −1
∑ ∑ | f [n, m]|2 = K1 K2 ∑ ∑ |F[k1, k2 ]|2
n=0 m=0
linear 2-D convolutions h[n, m] ∗ ∗ f [n, m] can be zero-padded to
cyclic convolutions, just as in 1-D.
n=0 m=0
K1 −1 K2 −1
k1 =0 k2 =0
3-8.2 Conjugate Symmetry for the 2-D DFT
Concept Question 3-8: Why is the 2-D DFT defined
only over a finite region, while the DSFT is defined over
all spatial frequency space?
Exercise 3-14:
 Compute
 an expression for the (256 × 256)
1 2
2-D DFT of 2 4
1 2
1
2. Use the result of Exercise 3-11.
1
The 2-D DFT is the DSFT sampled at
Ωi = 2π ki /256 for i = 1, 2. Substituting in the answer to
Exercise 3-11 gives
k1
k2
X[k1 , k2 ] = 2 + 2 cos 2π
2 + 2 cos 2π
,
256
256
0 ≤ k1 , k2 ≤ 255.
Answer:
Given an (M × N) image f [n, m], the expression given by
Eq. (3.75) allows us to compute the 2-D DFT of f [n, m] for any
order (K2 × K1 ). If f [n, m] is real, then conjugate symmetry
holds:
F∗ [k1 , k2 ] = F[K1 − k1 , K2 − k2 ],
1 ≤ k1 ≤ K1 − 1; 1 ≤ k2 ≤ K2 − 1.
(3.78)
[Compare this statement with the conjugate symmetry of the
1-D DFT X[k] of a real-valued signal x[n], as given by
Eq. (2.98).]
The conjugate-symmetry relation given by Eq. (3.78) states
that an array element F[k1 , k2 ] is equal to the complex conjugate
of array element F[K1 − k1 , K2 − k2 ], and vice versa. To
demonstrate the validity of conjugate symmetry, we start by
rewriting Eq. (3.75) with k1 and k2 replaced with (K1 − k1 ) and
3-9 COMPUTATION OF THE 2-D DFT USING MATLAB
(K2 − k2), respectively:
F[K1 − k1 , K2 − k2 ]
N−1 M−1
n(K1 − k1 ) m(K2 − k2)
+
= ∑ ∑ f [n, m] exp − j2π
K1
K2
n=0 m=0
N−1 M−1
nk1 mk2
= ∑ ∑ f [n, m] exp j2π
+
K1
K2
n=0 m=0
nK1 mK2
× exp − j2π
+
K1
K2
N−1 M−1
nk1 mk2
+
= ∑ ∑ f [n, m] exp j2π
K1
K2
n=0 m=0
× e− j2π (n+m)
N−1 M−1
= ∑ ∑ f [n, m] exp j2π
n=0 m=0
nk1 mk2
+
K1
K2
,
(3.79)
where we used e− j2π (n+m) = 1 because n and m are integers. The
expression on the right-hand side of Eq. (3.79) is identical to the
expression for F[k1 , k2 ] given by Eq. (3.75) except for the minus
sign ahead of j. Hence, for a real-valued image f [n, m],
F∗ [k1 , k2 ] = F[K1 − k1 , K2 − k2 ],
1 ≤ k1 ≤ K1 − 1; 1 ≤ k2 ≤ K2 − 1,
(3.80)
where (K2 × K1 ) is the order of the 2-D DFT.
3-8.3 Special Cases
A. f [n, m] is real
If f [n, m] is a real-valued image, the following special cases
hold:
M−1
(1) F[0, 0] = ∑N−1
n=0 ∑m=0 f [n, m] is real-valued.
− j(2π /K )mk
N−1
2
2,
(2) F[0, k2 ] = ∑M−1
m=0 ∑n=0 f [n, m] e
N−1
which is the K2 -point 1-D DFT of ∑n=0 f [n, m].
− j(2π /K )nk
M−1
1
1,
(3) F[k1 , 0] = ∑N−1
n=0 ∑m=0 f [n, m] e
M−1
which is the K1 -point 1-D DFT of ∑m=0 f [n, m].
B. f [n, m] is real and K1 and K2 are even
If also K1 and K2 are even, then the following relations apply:
121
N−1
n − j(2π /K2 )mk2 ,
(4) F[K1 /2, k2 ] = ∑M−1
m=0 ∑n=0 f [n, m] (−1) e
n
which is the K2 -point 1-D DFT of ∑N−1
n=0 f [n, m] (−1)
−
j(2
π
/
K1 )n(K1 /2)
j
π
n
n
because e
= e = (−1) .
M−1
m − j(2π /K1 )nk1
,
e
(5) F[k1 , K2 /2] = ∑N−1
n=0 ∑m=0 f [n, m] (−1)
M−1
which is the K1 -point 1-D DFT of ∑m=0 f [n, m] (−1)m
because e− j(2π /K2)m(K2 /2) = e jπ m = (−1)m .
M−1
m+n .
(6) F[K1 /2, K2 /2] = ∑N−1
n=0 ∑m=0 f [n, m] (−1)
3-9 Computation of the 2-D DFT Using
MATLAB
We remind the reader that the notation used in this book
represents images f [n, m] defined in Cartesian coordinates, with
the origin at the upper left corner, the first element n of the
coordinates [n, m] increasing horizontally rightward from the
origin, and the second element m of the coordinates [n, m]
increasing vertically downward from the origin. To illustrate
with an example, let us consider the (3 × 3) image given by

 

f [0, 0] f [1, 0] f [2, 0]
3 1 4
f [n, m] =  f [0, 1] f [1, 1] f [2, 1] = 1 5 9 . (3.81)
f [0, 2] f [1, 2] f [2, 2]
2 6 5
When stored in MATLAB as array X(m′ , n′ ), the content remains
the same, but the indices swap roles and their values start at
(1, 1):

 

X(1, 1) X(1, 2) X(1, 3)
3 1 4
X(m′ , n′ ) = X(2, 1) X(2, 2) X(2, 3) = 1 5 9 .
X(3, 1) X(3, 2) X(3, 3)
2 6 5
(3.82)
Arrays f [n, m] and X(m′ , n′ ) are displayed in Fig. 3-28.
Application of Eq. (3.75) with N = M = 3 and K1 = K2 = 3
to the 3 × 3 image defined by Eq. (3.81) leads to


36
−9 + j5.2 −9 − j5.2
F[k1 , k2 ] = −6 − j1.7 9 + j3.5 1.5 + j0.9 . (3.83)
−6 + j1.7 1.5 − j0.9 9 − j3.5
◮ In MATLAB, the command FX=fft2(X,M,N) computes the (M × N) 2-D DFT of array X and stores it in array
FX. ◭
www.dbooks.org
122
CHAPTER 3 2-D IMAGES AND SYSTEMS
3 × 3 Image

3
1
2
1
5
6

4
9
5
Common Image Format
MATLAB Format
n′
n
m

f [0, 0] = 3
f [n, m] =  f [0, 1] = 1
f [0, 2] = 2
f [1, 0] = 1
f [1, 1] = 5
f [1, 2] = 6

f [2, 0] = 4
f [2, 1] = 9
f [2, 2] = 5
m′

X(1, 1) = 3
X(m′ , n′ ) = X(2, 1) = 1
X(3, 1) = 2
DFT
F[k1 , k2 ] =

F[0, 0] = 36
 F[0, 1] = −6 − j1.7
F[0, 2] = −6 + j1.7
F[1, 0] = −9 + j5.2
F[1, 1] = 9 + j3.5
F[1, 2] = 1.5 − j0.9
9 − j3.5
Fc [k1c , k2c ] = −9 − j5.2
1.5 + j0.9
−6 + j1.7
36
−6 − j1.7

X(1, 3) = 4
X(2, 3) = 9
X(3, 3) = 5
fft2(X)

F[2, 0] = −9 − j5.2
F[2, 1] = 1.5 + j0.9 
F[2, 2] = 9 − j3.5
Shift to Center

X(1, 2) = 1
X(2, 2) = 5
X(3, 2) = 6
FX(k2′ , k1′ ) =

FX(1, 1) = 36
 FX(2, 1) = −6 − j1.7
FX(3, 1) = −6 + j1.7
FX(1, 2) = −9 + j5.2
FX(2, 2) = 9 + j3.5
FX(3, 2) = 1.5 − j0.9

FX(1, 3) = −9 − j5.2
FX(2, 3) = 1.5 + j0.9 
FX(3, 3) = 9 − j3.5
fftshift(fft2(X))

1.5 − j0.9
−9 + j5.2
9 + j3.5

9 − j3.5
′
′
FXC(k2c
, k1c
) = −9 − j5.2
1.5 + j0.9
−6 + j1.7
36
−6 − j1.7

1.5 − j0.9
−9 + j5.2
9 + j3.5
Figure 3-28 In common-image format, application of the 2-D DFT to image f [n, m] generates F[k1 , k2 ]. Upon shifting F[k1 , k2 ] along k1 and k2 to
center the image, we obtain the center-of-image format represented by Fc [k1c , k2c ]. The corresponding sequence in MATLAB starts with X(m′ , n′ ) and
′ , k′ ).
concludes with FXC(k2c
1c
The corresponding array in MATLAB, designated FX(k2′ , k1′ )
and displayed in Fig. 3-28, has the same content but with
MATLAB indices (k2′ , k1′ ). Also, k2′ increases downward and
k1′ increases horizontally. The relationships between MATLAB
indices (k2′ , k1′ ) and common-image format indices (k1 , k2 ) are
identical in form to those given by Eq. (3.69), namely
k2′ = k2 + 1,
(3.84)
k1′ = k1 + 1.
(3.85)
3-9.1 Center-of-Image Format
In some applications, it is more convenient to work with the
2-D DFT array when arranged in a center-of-image format
(Fig. 3-25(c) but with the vertical axis pointing downward) than
in the top-left corner format. To convert array F[k1 , k2 ] to a
center-of-image format, we need to shift the array elements to
the right and downward by an appropriate number of steps so as
to locate F[0, 0] in the center of the array. If we denote the 2-D
3-9 COMPUTATION OF THE 2-D DFT USING MATLAB
DFT in the center-of-image format as Fc [k1c , k2c ], then its index
k1c extends over the range
Ki − 1
Ki − 1
−
≤ k1c ≤
,
for Ki = odd, (3.86)
2
2
123
with k1 = K1 − k1c , k2 = K2 − k2c ,
(d) Fourth Quadrant
Fc [k1c , −k2c ] = F[k1 , k2 ],
and
−
Ki
Ki
≤ k1c ≤
− 1,
2
2
for Ki = even.
(3.87)
To obtain Fc [k1c , k2c ] from F[k1 , k2 ] for the array given by
Eq. (3.83), we circularly shift the array by one unit to the right
and one unit downward, which yields
Fc [k1c ,k2c ] =

Fc [−1,1] = 9 − j3.5
 Fc [−1,0] = −9 − j5.2
Fc [−1,−1] = 1.5 + j0.9
Fc [0,1] = −6 + j1.7
Fc [0,0] = 36
Fc [0,−1] = −6 − j1.7

Fc [1,1] = 1.5 − j0.9
Fc [1,0] = −9 + j5.2  .
Fc [1,−1] = 9 + j3.5
(3.88)
with k1 = k1c , k2 = K2 − k2c .
To demonstrate the recipe, we use a numerical example with
K1 = K2 = 3, and again for K1 = K2 = 4 because the recipe is
different for odd and even integers.
3-9.2 Odd and Even Image Examples
A. N = M and K1 = K2 = odd
◮ In MATLAB, the command
FXC=fftshift(fft2(FX))
shifts array FX to center-image format and stores it in array
FXC. ◭
In the general case for any integers K1 and K2 , transforming the 2-D DFT F[k1 , k2 ] into the center-of-image format
Fc [k1c , k2c ] entails the following recipe:
For
(
Ki /2 − 1
if Ki is even,
′
Ki =
(3.89)
(Ki − 1)/2
if Ki is odd,
The (3 × 3) image shown in Fig. 3-28 provides an example of an
(M × M) image with M being an odd integer. As noted earlier,
when the 2-D DFT is displayed in the center-of-image format,
the conjugate symmetry about the center of the array becomes
readily apparent.
A. N = M and K1 = K2 = even
Let us consider the (4 × 4) image

1
2
f [n, m] = 
3
4
and 0 ≤ k1c , k2c ≤ K′i :
(a) First Quadrant
Fc [k1c , k2c ] = F[k1 , k2 ],
(3.90a)
with k1 = k1c and k2 = k2c ,
(b) Second Quadrant
Fc [−k1c , k2c ] = F[k1 , k2 ],
(3.90b)
with k1 = K1 − k1c , k2 = k2c ,
(c) Third Quadrant
Fc [−k1c , −k2c ] = F[k1 , k2 ],
(3.90d)
(3.90c)
2
4
4
3
3
5
6
2

4
3
.
2
1
(3.91)
The (4 × 4) 2-D DFT F[k1 , k2 ] of f [n, m], displayed in the upperleft corner format, is


F[0, 0] F[1, 0] F[2, 0] F[3, 0]
F[0, 1] F[1, 1] F[2, 1] F[3, 1]
F=
F[0, 2] F[1, 2] F[2, 2] F[3, 2]
F[0, 3] F[1, 3] F[2, 3] F[3, 3]


49
−6 − j3
3
−6 + j3
j 
−5 − j4 2 + j9 −5 + j2
=
.
(3.92)
1
−4 + j3
−1
−4 − j3
−5 + j4
−j
−5 − j2 2 − j9
www.dbooks.org
124
Application of the recipe given by Eq. (3.90) leads to
Fc (k1c , k2c )
 ′

F [−2, −2] F′ [−1, −2] F′ [0, −2] F′ [1, −2]
′
′
′
′
 F [−2, −1] F [−1, −1] F [0, −1] F [1, −1] 
= ′
F [−2, 0]
F′ [−1, 0]
F′ [0, 0]
F′ [1, 0] 
′
′
′
F [−2, −1] F [−1, 1]
F [0, 1]
F′ [1, 1]


−1
−4 − j3
1
−4 + j3
−j 
−5 − j2 2 − j9 −5 + j4
=
.
(3.93)
3
−6 + j3
49
−6 − j3
−5 + j2
j
−5 − j4 2 + j9
Now conjugate symmetry Fc [−k1c , −k2c ] = F∗c [k1c , k2c ] applies,
but only after omitting the first row and column of Fc [k1c , k2c ].
This is because the dc value (49) is not at the center of the array.
Indeed, there is no center pixel in an M × M array if M is even.
It is customary in center-of-image depictions to place the origin
at array coordinates [ M2 + 1, M2 + 1]. The first row and column
are not part of the mirror symmetry about the origin. This is not
noticeable in M × M arrays if M is even and large.
Conjugate symmetry applies within the first row and within
the first column, since these are 1-D DFTs with M2 = 2.
Exercise 3-15: Why did this book not spend more space on
computing the DSFT?
Answer: Because in practice, the DSFT is computed using
the 2-D DFT.
CHAPTER 3 2-D IMAGES AND SYSTEMS
3-9 COMPUTATION OF THE 2-D DFT USING MATLAB
125
Summary
Concepts
• Many 2-D concepts are generalizations of 1-D counterparts. These include: LSI systems, convolution, sampling, 2-D continuous-space; 2-D discrete-space; and
2-D discrete Fourier transforms.
• The 2-D DSFT is doubly periodic in Ω1 and Ω2 with
periods 2π .
• Rotating an image rotates its 2-D continuous-space
Fourier transform (CSFT). The CSFT of a radially symmetric image is radially symmetric.
• Continuous-space images can be sampled to discrete-
space images, on which discrete-space image processing
can be performed.
• Nearest-neighbor interpolation often works well for interpolating sampled images to continuous space. Hexagonal sampling can also be used.
• Discrete-space images can be displayed in several different formats (the location of the origin differs).
• The response of an LSI system with point spread
function h(x, y) to image f (x, y) is output g(x, y) =
h(x, y) ∗ ∗ f (x, y), and similarly in discrete space.
Mathematical Formulae
Impulse
Ideal radial lowpass filter PSF
h(r) = 4ρ02 jinc(2ρ0 r)
δ (r)
δ (x, y) = δ (x) δ (y) =
πr
Energy of f (x, y)
E=
Z ∞Z ∞
−∞ −∞
2-D Sampling
Sampling interval
| f (x, y)|2 dx dy
2-D Sinc interpolation formula
f (x, y) =
Convolution
h(x, y) ∗ f (x, y) =
Z ∞Z ∞
−∞ −∞
∞
∑ h[i, j] f [n − i, m − j]
Fourier transform
Z Z (CSFT)
∞
Inverse CSFT
Z Z
∞
f (x, y) =
∞
−∞ −∞
y − m∆
sinc
∆
F(Ω1 , Ω2 ) =
∑ ∑ f [n, m] e− j(Ω1 n+Ω2 m)
n=−∞ m=−∞
(K2 × K1 ) 2-D DFT of (M × N) image
N−1 M−1
∞
−∞ −∞
∞
∞
i=−∞ j=−∞
Discrete-space Fourier transform (DSFT)
∞
∞
h[n, m] ∗ ∗ f [n, m] = ∑
F(µ , ν ) =
∞
x − n∆
∑ ∑ f (n∆, m∆) sinc ∆
n=−∞ m=−∞
f (ξ , η ) h(x − ξ , y − η ) d ξ d η
Convolution
1
> 2B if F(µ , ν ) = 0 for |µ |, |ν | > B
∆
f (x, y) e
− j2π ( µ x+ν y)
dx dy
F(µ , ν ) e j2π (µ x+ν y) d µ d ν
F[k1 , k2 ] = ∑ ∑ f [n, m] e− j2π (nk1/K1 +mk2 /K2 )
n=0 m=0
Inverse 2-D DFT
1 K1 −1 K2 −1
f [n, m] =
∑ F[k1 , k2 ] e j2π (nk1/K1 +mk2 /K2)
K1 K2 k∑
=0 k =0
1
2
Ideal square lowpass filter PSF
h(x, y) = 4µ02 sinc(2 µ0 x) sinc(2ν0 y)
Important Terms
aliasing
convolution
CSFT
DFT
Provide definitions or explain the meaning of the following terms:
DSFT
FFT
linear shift-invariant (LSI)
nearest-neighbor interpolation
www.dbooks.org
point spread function
sampled image
sampling theorem
sinc function
126
CHAPTER 3 2-D IMAGES AND SYSTEMS
PROBLEMS
Section 3-2: 2-D Continuous-Space Images
3.1 Let f (x, y) be an annulus (ring) with inner radius 3 and
outer radius 5, with center at the point (2,4). Express f (x, y) in
terms of fDisk (x, y).
Section 3-3: Continuous-Space Systems
3.2 Compute the autocorrelation
r(x, y) = fBox (x, y) ∗ ∗ fBox(−x, −y)
its samples using NN interpolation, but displays spectra at each
stage. Explain why the image reconstructed from its samples
matches the clown image.
Section 3-6: 2-D Discrete Space
3.10 Compute the 2-D convolution
5 9
3 1
∗∗
2 6
4 1
by hand. Check your answer using MATLAB’s conv2.
of fBox (x, y).
3.11 An LTI system is described by the equation
Section 3-4: 2-D Continuous-Space Fourier
Transform (CSFT)
g[n, m] =
3.3 Compute the 2-D CSFT F(µ , ν ) of a 10 × 6 ellipse f (x, y).
3.4 A 2-D Gaussian function has the form
2
fg (r) =
1
1
1
f [n − 1, m − 1] + f [n − 1, m] + f [n − 1, m + 1]
9
9
9
1
1
1
+ f [n, m − 1] + f [n, m] + f [n, m + 1]
9
9
9
1
1
1
+ f [n + 1, m − 1] + f [n + 1, m] + f [n + 1, m + 1].
9
9
9
2
e−r /(2σ )
.
2πσ 2
What is the PSF h[n, m] of the system? Describe in words what
it does to its input.
Compute the 2-D CSFT fg (ρ ) of fg (r) using the scaling property
of the 2-D CSFT.
3.12 An LTI system is described by the equation
3.5 Compute the CSFT of an annulus (ring) f (x, y) with inner
radius 3 and outer radius 5, with center at:
(a) the origin (0,0);
(b) the point (2,4).
g[n, m] = 9 f [n − 1, m − 1] + 8 f [n − 1, m] + 7 f [n − 1, m + 1]
+ 6 f [n, m − 1] + 5 f [n, m] + 4 f [n, m + 1]
+ 3 f [n + 1, m − 1] + 2 f [n + 1, m] + f [n + 1, m + 1].
Section 3-5: 2-D Sampling Theorem
3.6 f (x, y) = cos(2π 3x) cos(2π 4y) is sampled every ∆ = 0.2.
What is reconstructed by a brick-wall lowpass filter with cutoff
= 5 in µ and ν and passband gain ∆ = 0.2?
3.7 f (x, y) = cos(2π 12x) cos(2π 15y) is sampled every
∆ = 0.1. What is reconstructed by a brick-wall lowpass filter
with cutoff = 10 in µ and ν and passband gain ∆ = 0.1?
What is the PSF h[n, m] of the system? Use center-of-image
notation (Fig. 3-25(c)).
Section 3-7: Discrete-Space Fourier Transform
(DSFT)
3.13 Prove the shift property of the DSFT:
f [n − a, m − b]
e− j(aΩ1 +bΩ2 ) F(Ω1 , Ω2 ).
3.8 Antialias filtering: Run MATLAB program P38.m. This
lowpass-filters the clown image before sampling it below its
Nyquist rate. Explain why the image reconstructed from its
samples has no aliasing.
3.14 Compute the spatial frequency response of the system in
Problem 3.11.
3.9 Nearest-Neighbor interpolation: Run MATLAB program
P39.m. This samples the clown image and reconstructs from
3.15 Compute the spatial frequency response of the system in
Problem 3.12.
PROBLEMS
127
3.16 Show that the 2-D spatial frequency response of the PSF


1 2 1
h[m, n] = 2 4 2
1 2 1
using (2 × 2) 2-D DFTs.
is close to circularly symmetric, making it a circularly symmetric lowpass filter, by:
(a) displaying the spatial frequency response as an image with
dc at the center;
(b) using
Ω2 Ω4
cos(Ω) = 1 −
+
− ···
2!
4!
and neglecting all terms of degree four or higher.
Section 3-8: 2-D Discrete Fourier Transform
(DFT)
3.17 Compute by hand the (2 × 2) 2-D DFT of
1 2
.
F = f [n, m] =
3 4
Check your answer using MATLAB using FX=fft(F,2,2).
3.18 Prove that the (M × M) 2-D DFT of a separable image
f [n, m] = f1 [n] f2 [m] is the product of the 1-D M-point DFTs of
f1 [n] and f2 [n]: F[k1 , k2 ] = f1 [k1 ] f2 [k2 ].
3.19 Show that we can extend the definition of the (M × M)
2-D DFT to negative values of k1 and k2 using
F[−k1 , −k2 ] = F[M − k1 , M − k2 ]
for indices 0 < k1 , k2 < M. Conjugate symmetry for real f [n, m]
is then
F[−k1 , −k2 ] = F[M − k1 , M − k2 ] = F∗ [k1 , k2 ].
3.20 Compute the 2-D cyclic convolution
y[n, m] = x1 [n, m] c c x2 [n, m],
where
x1 [n, m] =
1
3
2
4
x2 [n, m] =
5
7
6
8
and
www.dbooks.org
Chapter
4
Image Interpolation
Contents
4-1
4-2
4-3
4-4
4-5
4-6
4-7
4-8
4-9
4-10
4
Overview, 129
Interpolation Using Sinc Functions, 129
Upsampling and Downsampling Modalities, 130
Upsampling and Interpolation, 133
Implementation of Upsampling Using 2-D DFT
in MATLAB, 137
Downsampling, 140
Antialias Lowpass Filtering, 141
B-Splines Interpolation, 143
2-D Spline Interpolation, 149
Comparison of 2-D Interpolation Methods, 150
Examples of Image Interpolation
Applications, 152
Problems, 156
Objectives
Learn to:
■ Use sinc and Lanczos functions to interpolate a
bandlimited image.
■ Perform upsampling and interpolation using the 2-D
DFT and MATLAB.
■ Perform downsampling for thumbnails using the 2-D
DFT and MATLAB.
■ Use B-spline functions of various orders to interpolate non-bandlimited images.
■ Rotate, magnify, and morph images using interpolation.
0
50
100
150
200
250
300
350
399
0
50
100
150
200
250
300
350
399
In many image processing applications, such as
magnification, thumbnails, rotation, morphing, and
reconstruction from samples, it is necessary to
interpolate (roughly, fill in gaps between given
samples). This chapter presents three approaches to
interpolation: using sinc or Lanczos functions;
upsampling using the 2-D DFT; and use of
B-splines. These methods are compared and used on
each of the applications listed above.
Overview
Suppose some unknown signal x(t) had been sampled at a
sampling rate S to generate a sampled signal x[n] = { x(n∆),
n = . . . , −1, 0, 1, . . . }, sampled at times t = n∆, where ∆ = 1/S is
the sampling interval. Interpolation entails the use of a recipe
or formula to compute a continuous-time interpolated version
xint (t) that takes on the given values { x(n∆) } and interpolates
between them. In 1-D, interpolation is akin to connecting
the dots, represented here by { x(n∆) }, to obtain xint (t). The
degree to which the interpolated signal is identical to or a close
rendition of the original signal x(t) depends on two factors:
(1) Whether or not the sampling rate S used to generate xint (t)
from x(t) satisfies the Nyquist criterion, namely S > 2B,
where B is the maximum frequency in the spectrum of
signal x(t), and
(2) the specific interpolation method used to obtain xint (t).
If the spectrum X( f ) of x(t) is bandlimited to B and S > 2B,
it should be possible to use the sinc interpolation formula given
by Eq. (2.51) to reconstruct x(t) exactly. A simple example is
illustrated by the finite-duration sinusoid shown in Fig. 4-1.
In practice, however, it is computationally more efficient to
perform the interpolation in the frequency domain by lowpassfiltering the spectrum of the sampled signal.
Oftentimes, the sampling rate does not satisfy the Nyquist
criterion, as a consequence of which the signal obtained by
applying the sinc interpolation formula may be an aliased
version of x(t). In such cases, different interpolation methods
should be used, several of which are examined in this chapter.
In 2-D, interpolation seeks to fill in the gaps between the
image values f [n, m] = { f (n∆, m∆) } defined at discrete locations (x = n∆, y = m∆) governed by the spatial sampling
rate S = Sx = Sy = 1/∆. In analogy with the 1-D case, if the
spectrum F(µ , v) of the original image f (x, y) is bandlimited
to Bx = By = B and if the sampling rate satisfies the Nyquist
criterion (i.e., S > 2B), then it should be possible to interpolate
f [n, m] = f (n∆, m∆) to generate f (x, y) exactly.
This chapter explores several types of recipes for interpolating
a sampled image f [n, m] = { f (n∆, m∆) } into a continuousspace image fint (x, y). Some of these recipes are extensions of
the sinc interpolation formula, while others rely on the use of
B-spline functions. As discussed later in Section 4-7, B-splines
are polynomial functions that can be designed to perform
nearest-neighbor, linear, quadratic, and cubic interpolation of
images and signals. We will also explore how interpolation is
used to realize image zooming (magnification), rotation, and
warping.
4-1 Interpolation Using Sinc Functions
4-1.1 Sinc Interpolation Formula
An image f (x, y) is said to be bandlimited to a maximum spatial
frequency B if its spectrum F(µ , v) is such that
F(µ , ν ) = 0
1
0.5
t (ms)
0
−0.5
If such an image is sampled uniformly along x and y at sampling
rates Sx = Sy = S = 1/∆, and the sampling interval ∆ satisfies the
Nyquist rate, namely
1
,
∆<
2B
then we know from the 2-D sampling theorem (Section
3-5) that f (x, y) can be reconstructed from its samples
f [n, m] = { f (n∆, m∆) } using the sinc interpolation formula
given by Eq. (3.62), which we repeat here as
0
0.5
1
1.5
2
2.5
3
3.5
Figure 4-1 1-D interpolation of samples of a sinusoid using the
sinc interpolation formula.
x
y
−
n
sinc
−
m
,
∑ ∑
∆
∆
n=−∞ m=−∞
(4.1)
where for any argument z, the sinc function is defined as
∞
f (x, y) =
−1
for |µ |, |v| ≥ B.
∞
f (n∆, m∆) sinc
sinc(z) =
129
www.dbooks.org
sin(π z)
.
πz
(4.2)
130
CHAPTER 4 IMAGE INTERPOLATION
In reality, an image is finite in size, and so is the number of
samples { f (n∆, m∆) }. Consequently, for a square image, the
ranges of indices n and m are limited to finite lengths M, in
which case the infinite sums in Eq. (4.1) become finite sums:
x
y
− n sinc
−m .
∆
∆
n=0 m=0
(4.3)
The summations start at n = 0 and m = 0, consistent with the
image display format shown in Fig. 3-3(a), wherein location
(0, 0) is at the top-left corner of the image.
The sampled image consists of M × M values—denoted here
by f (n∆, m∆)—each of which is multiplied by the product of
two sinc functions, one along x and another along y. The value
fsinc (x, y) at a specified location (x, y) on the image consists of
the sum of M × M terms. In the limit for an image of infinite
size, and correspondingly an infinite number of samples along
x and y, the infinite summations lead to fsinc (x, y) = f (x, y),
where f (x, y) is the original image. That is, the interpolated
image is identical to the original image, assuming all along that
the sampled image is in compliance with the Nyquist criterion
of the sampling theorem. When applying the sinc interpolation
formula to a finite-size image, the interpolated image should be
a good match to the original image, but it is computationally
inefficient when compared with the spatial-frequency domain
interpolation technique described later in Section 4-3.2.
M−1 M−1
fsinc (x, y) = ∑ ∑ f (n∆, m∆) sinc
4-1.2 Lanczos Interpolation
To reduce the number of terms involved in the computation of
the interpolated image, the sinc function in Eq. (4.3) can be
modified by truncating the sinc pattern along x and y so that
it is zero beyond a certain multiple of ∆, such as |x| = 2∆
and |y| = 2∆. Such a truncation is offered by the Lanczos
interpolation formula which replaces each of the sinc functions
in Eq. (4.3) by windowed sinc functions, thereby assuming the
form
M−1 M−1
fLanczos (x, y) = ∑ ∑ f (n∆, m∆)
n=0 m=0
× sinc
x
x
x
− n sinc
− n rect
a∆
2ay y
× sinc
,
− m sinc
− m rect
∆
a∆
2a
(4.4)
∆
y
where the rectangle function, defined earlier in Eq. (2.2), is given
by
(
x
1
for − a < x < a,
=
rect
2a
0
otherwise.
Parameter a usually is assigned a value of 2 or 3. In MATLAB,
the Lanczos interpolation formula can be exercised using the
command imresize and selecting Lanczos from the menu.
In addition to the plot for sinc(x), Fig. 4-2 also contains plots
of the sinc function multiplied by the windowed sinc function,
with a = 2 and also with a = 3. The rectangle function is zero
beyond |x| = a, so the windowed function stops at |x| = 2 for
a = 2 and at |x| = 3 for a = 3. The Lanczos interpolation method
provides significant computational improvement over the simple
sinc interpolation formula.
Concept Question 4-1: What is the advantage of Lanczos interpolation over sinc interpolation?
Exercise 4-1: If sinc interpolation is applied to the samples
{x(0.1n) = cos(2π (6)0.1n)}, what will be the result?
Answer: {x(0.1n)} are samples of a 6 Hz cosine sampled at a rate of 10 samples/second, which is below the
Nyquist frequency of 12 Hz. The sinc interpolation will
be a cosine with frequency aliased to (10 − 6) = 4 Hz (see
Section 2-4.3).
4-2 Upsampling and Downsampling
Modalities
As a prelude to the material presented in forthcoming sections,
we present here four examples of image upsampling and downsampling applications. Figure 4-3 depicts five image configurations, each consisting of a square array of square pixels. The
central image is the initial image from which the other four were
generated. We define this initial image as the discrete version of
a continuous image f (x, y), sampled at (M × M) locations at a
sampling interval ∆o along both dimensions:
f [n, m] = { f (n∆o , m∆o ), 0 ≤ n, m ≤ M − 1 }.
(4.5)
We assume that the sampling rate S is such that ∆o = 1/S
satisfies the Nyquist rate S > 2B or, equivalently, ∆o < 1/2B,
where B is the maximum spatial frequency of f (x, y).
When displayed on a computer screen, the initial (central
4-2 UPSAMPLING AND DOWNSAMPLING MODALITIES
131
1
sinc(x)
sinc(x) sinc(x/2) rect(x/4)
sinc(x) sinc(x/3) rect(x/6)
0.8
0.6
0.4
0.2
0
−0.2
−3
−2
−1
0
1
3
2
x
Figure 4-2 Sinc function (in black) and Lanczos windowed sinc functions (a = 2 in blue and a = 3 in red).
(Mu × Mu ) image g[n, m], with Mu > Mo and
image is characterized by four parameters:
Mo × Mo = M 2 = image array size,
T′ =
2
T × T = T = image physical size on computer screen,
w × w = w2 = image pixel area,
2
∆o × ∆o = ∆o = true resolution area.
If the displayed image bears a one-to-one correspondence to the
array f [n, m], then the brightness of a given pixel corresponds
to the magnitude f [n, m] of the corresponding element in the
array, and the pixel dimensions w × w are directly proportional
to ∆o × ∆o . For simplicity, we set w = ∆o , which means that the
image displayed on the computer screen has the same physical
dimensions as the original image f (x, y). The area ∆2o is the true
resolution area of the image.
and ∆u =
Mo
∆o .
Mu
(4.6)
B. Increasing Image Array Size While Keeping
Physical Size Unchanged
The image displayed in Fig. 4-3(b) is an upsampled version of
f [n, m] in which the array size was increased from (M × M) to
(Mu × Mu ) and the physical size of the image remained the same,
but the pixel size is smaller. In the image g[n, m],
4-2.1 Upsampling Modalities
The image configuration in part (a) of Fig. 4-3 depicts what
happens when the central image f [n, m] is enlarged in size from
(T × T ) to (T ′ × T ′ ) while keeping the pixel size the same. The
process, accomplished by an upsampling operation, leads to an
w′ = w,
Since the sampling interval ∆u in the enlarged upsampled image
is shorter than the sampling interval in the initial image, ∆o , the
Nyquist requirement continues to be satisfied.
T = T ′,
A. Enlarging Physical Size While Keeping Pixel
Size Unchanged
Mu
T,
Mo
w′ = w
Mo
,
Mu
and ∆u =
Mo
∆o .
Mu
(4.7)
◮ The two images in Fig. 4-3(a) and (b) have identical arrays g[n, m], but they are artificially displayed on computer
screens with different pixel sizes. ◭
4-2.2 Downsampling Modalities
www.dbooks.org
132
CHAPTER 4 IMAGE INTERPOLATION
T′
g[n,m]
T′ = T
Upsampling
Δu = Δo /2
Upsampling
w′
Δu = Δo /2
(b) Image with Mu = 2Mo,
T ′ = T, and w′ = w /2
f [n,m]
T
w′
(a) Image with w′ = w, Mu = 2Mo,
and T ′ = 2T
Downsampling
T′
Δd = 2Δo
Downsampling
w
Original image
Mo × Mo
T′ = T
Δd = 2Δo
w′
w′
(c) Thumbnail image with w′ = w,
Md = Mo /2 and T ′ = T/2
(d) Image with w′ = 2w, T ′ = T,
and Md = Mo /2
Figure 4-3 Examples of image upsampling and downsampling.
A. Thumbnail Image
If we apply downsampling to reduce the array size from
(Mo × Mo ) to (Md × Md ), we end up with the downsampled
images depicted in Figs. 4-3(c) and (d). In the thumbnail image
shown in Fig. 4-3(c), the pixel size of the computer display is
the same as that of the original image. Hence,
Md
T,
T =
Mo
′
′
w = w,
Mo
and ∆d =
∆o .
Md
B. Reduce Array Size While Keeping Physical
Size Unchanged
The final of the transformed images, shown in Fig. 4-3(d), has
the same identical content of the thumbnail image, but the pixel
size on the computer screen has been enlarged so that
T = T ′,
(4.8)
∆d =
Mo
∆o ,
Md
and w′ = w
Mo
.
Md
(4.9)
4-3 UPSAMPLING AND INTERPOLATION
133
4-3 Upsampling and Interpolation
Let f (x, y) be a continuous-space image of size T (meters) by T
(meters) whose 2-D Fourier transform F(µ , v) is bandlimited to
B (cycles/m). That is,
F(µ , v) = 0
for |µ |, |ν | > B.
(4.10)
Image f (x, y) is not available to us, but an (Mo × Mo ) sampled
version of f (x, y) is available. We define it as the original
sampled image
f [n, m] = { f (n∆o , m∆o ), 0 ≤ n, m ≤ Mo − 1 },
(4.11)
where ∆o is the associated sampling interval. Moreover, the
sampling had been performed at a rate exceeding the Nyquist
rate, which requires the choice of ∆o to satisfy the condition
∆o <
1
.
2B
(4.12)
Given that the sampled image is T × T in size, the number of
samples Mo along each direction is
Mo =
T
.
∆o
illustrated in Fig. 4-3(a), rather than to decrease the sampling
interval.
Upsampling image f [n, m] to image g[n′ , m′ ] can be accomplished either directly in the discrete spatial domain or indirectly
in the spatial frequency domain. We examine both approaches in
the subsections that follow.
(4.13)
Next, we introduce a new (yet to be created) higher-density
sampled image g[n, m], also T × T in physical dimensions but
containing Mu × Mu samples—instead of Mo × Mo samples—
with Mu > Mo (which corresponds to the scenario depicted in
Fig. 4-3(b)). We call g[n′ , m′ ] the upsampled version of f [n, m].
The goal of upsampling and interpolation, which usually is
abbreviated to just “upsampling,” is to compute g[n′ , m′ ] from
f [n, m]. Since Mu > Mo , g[n′ , m′ ] is more finely discretized than
f [n, m], and the narrower sampling interval ∆u of the upsampled
image is
T
Mo
∆u =
=
∆o .
(4.14)
Mu
Mu
4-3.1 Upsampling in the Spatial Domain
In practice, image upsampling is performed using the 2-D
DFT in the spatial frequency domain because it is much faster
and easier computationally than performing the upsampling
directly in the spatial domain. Nevertheless, for the sake of
completeness, we now provide a succinct presentation of how
upsampling is performed in the spatial domain using the sinc
interpolation formula.
We start by repeating Eq. (4.3) after replacing ∆ with ∆o and
M with Mo :
Mo −1 Mo −1
y
x
− n sinc
−m .
f (x, y) = ∑ ∑ f [n, m] sinc
∆o
∆o
n=0 m=0
(4.16)
Here, f [n, m] is the original (Mo × Mo ) sampled image available
to us, and the goal is to upsample it to an (Mu × Mu ) image
g[n′ , m′ ], with Mu = LMo , where L is an upsampling factor. In
the upsampled image, the sampling interval ∆u is related to the
sampling interval ∆o of the original sampled image by
∆u =
′
′
′
′
′
g[n , m ] = { f (n ∆u , m ∆u ), 0 ≤ n , m ≤ Mu − 1 }.
g[n′ , m′ ] = f (n′ ∆u , m′ ∆u )
Mu −1 Mu −1
= ∑
(0 ≤ n′ , m′ ≤ Mu − 1)
∑ f [n, m]
n=0 m=0
′ ∆u
′ ∆u
− n sinc m
−m
× sinc n
∆o
∆o
Mu −1 Mu −1
= ∑
∑ f [n, m]
n=0 m=0
(4.15)
In a later section of this chapter (Section 4-6) we demonstrate
how the finer discretization provided by upsampling is used to
compute a rotated or warped version of image f [n, m]. Another
application is image magnification, but in that case the primary
goal is to increase the image size (from T1 × T1 to T2 × T2 ), as
(4.17)
To obtain g[n′ , m′ ], we sample f (x, y) at x = n′ ∆u and y = m′ ∆u :
Since ∆u < ∆o , it follows that the sampling rate associated with
g[n′ , m′ ] also satisfies the Nyquist rate.
The upsampled image is given by
′
∆o
Mo
∆o =
.
Mu
L
′
′
m
n
− n sinc
−m .
× sinc
L
L
(4.18)
If greater truncation is desired, we can replace the product of
sinc functions with the product of the Lanczos functions defined
in Eq. (4.4). In either case, application of Eq. (4.18) generates
www.dbooks.org
134
CHAPTER 4 IMAGE INTERPOLATION
the upsampled version g[n′ , m′ ] directly from the original sampled version f [n, m]. If ∆u = ∆o /L and L is an integer, then the
process preserves the values of f [n, m] while adding new ones in
between them. To demonstrate that the upsampling process does
indeed preserve f [n, m], let us consider the expression given by
Eq. (4.15) for the specific case where n′ = Ln and m′ = Lm:
g[Ln, Lm] = { f (n′ ∆u , m′ ∆u ), 0 ≤ n′ , m′ ≤ Mu − 1 }
= { f (Ln∆u , Lm∆u ), 0 ≤ n, m ≤ Mo − 1 }
= { f (n∆o , m∆o ), 0 ≤ n, m ≤ Mo − 1 } = f [n, m],
where we used the relationships given by Eqs. (4.11) and (4.14).
Hence, upsampling by an integer L using the sinc interpolation formula does indeed preserve the existing values of f [n, m],
in addition to adding interpolated values between them.
4-3.2 Upsampling in the Spatial Frequency
Domain
Instead of using the sinc interpolation formula given by
Eq. (4.18), upsampling can be performed much more easily,
and with less computation, using the 2-D DFT in the spatial
frequency domain. From Eqs. (4.11) and (4.18), the (Mo × Mo )
original f [n, m] image and the (Mu × Mu ) upsampled image
g[n′ , m′ ] are defined as
f [n, m] = { f (n∆o , m∆o ), 0 ≤ n, m ≤ Mo − 1 },
(4.19a)
g[n′ , m′ ] = { g(n′ ∆u , m′ ∆u ), 0 ≤ n′ , m′ ≤ Mu − 1 }.
F[k1 , k2 ] = ∑
∑ f [n, m] e
− j(2π /Mo )(nk1 +mk2 )
Mo −1 Mo −1
fo (x, y) = ∑
∑ f (n∆o , m∆o ) δ (x − n∆o) δ (y − m∆o)
n=0 m=0
Mo −1 Mo −1
= ∑
∑ f [n, m] δ (x − n∆o) δ (y − m∆o),
(4.21)
n=0 m=0
where we used the definition for f [n, m] given by Eq. (4.19a).
Using entry #9 in Table 3-1, the 2-D CSFT Fo (µ , ν ) of
fo (x, y) can be written as
Fo (µ , ν ) = F { fo (x, y)}
Mo −1 Mo −1
= ∑
∑ f [n, m] F {δ (x − n∆o) δ (y − m∆o)}
= ∑
∑ f [n, m] e− j2π µ n∆o e− j2πν m∆o .
n=0 m=0
Mo −1 Mo −1
,
0 ≤ k1 , k2 ≤ Mo − 1,
(4.20a)
Mu −1 Mu −1
∑ f [n, m] e− j(2π /Mu)(nk1 +mk2 ) ,
n=0 m=0
0 ≤ k1 , k2 ≤ Mu − 1.
Let us start with the sampled image fo (x, y) of continuous
image f (x, y) sampled at a sampling interval ∆o and resulting
in (Mo × Mo ) samples. Per Eq. (3.61), adapted to a finite sum
that starts at (0, 0) and ends at (Mo − 1, Mo − 1),
(4.22)
n=0 m=0
n=0 m=0
G[k1 , k2 ] = ∑
A. Original Image
(4.19b)
Note that whereas in the earlier section it proved convenient to
distinguish the indices of the upsampled image from those of
the original image—so we used [n, m] for the original image and
[n′ , m′ ] for the upsampled image—the distinction is no longer
needed in the present section, so we will now use indices [n, m]
for both images.
From Eq. (3.75), the 2-D DFT of f [n, m] of order (Mo × Mo )
and the 2-D DFT of g[n, m] of order (Mu × Mu ) are given by
Mo −1 Mo −1
tion for F[k1 , k2 ] extends to (Mo − 1) whereas the summation for
G[k1 , k2 ] extends to (Mu − 1).
The goal is to compute g[n, m] by (1) transforming f [n, m] to
obtain F[k1 , k2 ], (2) transforming F[k1 , k2 ] to G[k1 , k2 ], and (3)
then transforming G[k1 , k2 ] back to the spatial domain to form
g[n, m]. Despite the seeming complexity of having to execute a
three-step process, the process is computationally more efficient
than performing upsampling entirely in the spatial domain.
Upsampling in the discrete frequency domain [k1 , k2 ] entails increasing the number of discrete frequency components
from (Mo × Mo ) for F[k1 , k2 ] to (Mu × Mu ) for G[k1 , k2 ], with
Mu > Mo . As we will demonstrate shortly, G[k1 , k2 ] includes all
of the elements of F[k1 , k2 ], but it also includes some additional
rows and columns filled with zeros.
The spectrum Fo (µ , ν ) of sampled image fo (x, y) is doubly
periodic in µ and ν with period 1/∆o , as expected.
Extending the relations expressed by Eqs. (2.47) and (2.54)
from 1-D to 2-D, the spectrum Fo (µ , ν ) of the sampled image is
related to the spectrum F(µ , ν ) of continuous image f (x, y) by
(4.20b)
The summations are identical in form, except that the summa-
Fo ( µ , ν ) =
∞
1 ∞
F(µ − k1 /∆o , ν − k2 /∆o ). (4.23)
∑
∑
2
∆o k =−∞ k =−∞
1
2
4-3 UPSAMPLING AND INTERPOLATION
135
The spectrum Fo (µ , ν ) of the sampled image consists of copies
of the spectrum F(µ , ν ) of f (x, y) repeated every 1/∆o in both
µ and ν , and also scaled by 1/∆2o.
In (µ , ν ) space, µ and ν can be both positive or negative. The
relation between the spectrum Fo (µ , ν ) of the sampled image
fo (x, y) and the spectrum F(µ , ν ) of the original continuous
image f (x, y) assumes different forms for the four quadrants of
(µ , ν ) space.
For ease of presentation, let Mo be odd. If Mo is even, then
simply replace (Mo − 1)/2 with Mo /2 (see Section 4-4.2).
Next, we sample µ and ν by setting them to
k2
,
Mo ∆o
Mo − 1
0 ≤ |k1 |, |k2 | ≤
.
2
µ=
k1
Mo ∆o
and ν =
(4.24)
k1 , k2 ≥ 0 and redefining µ as µ = −k1 /(Mo ∆o ) leads to
Mo − 1
k2
−k1
0 ≤ k1 , k2 ≤
,
Fo
Mo ∆o Mo ∆o
2
Mo −1 Mo −1
= ∑
∑ f [n, m] e− j2π n(−k1)/Mo e− j2π mk2 /Mo
= ∑
∑ f [n, m] e− j2π n(Mo−k1 )/Mo e− j2π mk2/Mo
n=0 m=0
Mo −1 Mo −1
n=0 m=0
= F[Mo − k1 , k2 ],
0 ≤ k1 , k2 ≤
Mo − 1
,
2
(4.26)
3. Quadrant 3: µ ≤ 0 and ν ≤ 0
Redefining µ as µ = −k1 /(Mo ∆o ) and ν as ν = −k2 /(Mo ∆o )
leads to
Mo − 1
−k1 −k2
,
0 ≤ k1 , k2 ≤
,
Fo
Mo ∆o Mo ∆o
2
Mo −1 Mo −1
1. Quadrant 1: µ ≥ 0 and ν ≥ 0
∑ f [n, m] e− j2π n(Mo−k1 )/Mo e− j2π m(Mo −k2 )/Mo
= F[Mo − k1, Mo − k2 ],
0 ≤ k1 , k2 ≤
Mo − 1
.
2
(4.27)
4. Quadrant 4: µ ≥ 0 and ν ≤ 0
∑ f [n, m] e− j2π nk1/Mo e− j2π mk2 /Mo
n=0 m=0
= F[k1 , k2 ],
= ∑
n=0 m=0
Mo −1 Mo −1
Mo − 1
0 ≤ k1 , k2 ≤
.
2
∑ f [n, m] e− j2π n(−k1)/Mo e− j2π m(−k2)/Mo
n=0 m=0
Mo −1 Mo −1
At these values of µ and ν , Fo (µ , ν ) becomes
k2
k1
,
Fo
Mo ∆o Mo ∆o
= ∑
= ∑
(4.25)
Upon defining µ as in Eq. (4.24) and redefining ν as
ν = −k2 /(Mo ∆o ),
Mo − 1
−k2
k1
, 0 ≤ k1 , k2 ≤
,
Fo
Mo ∆o Mo ∆o
2
Mo −1 Mo −1
= ∑
∑ f [n, m] e− j2π nk1/Mo e− j2π m(−k2)/Mo
2. Quadrant 2: µ ≤ 0 and ν ≥ 0
= ∑
∑ f [n, m] e− j2π nk1/Mo e− j2π m(Mo −k2 )/Mo
In quadrants 2–4, we make use of the relation
= F[k1 , Mo − k2],
e
− j2π n(−k1 )/Mo
=e
n=0 m=0
Mo −1 Mo −1
n=0 m=0
− j2π nMo /Mo − j2π n(−k1 )/Mo
e
= e− j2π n(Mo −k1 )/Mo ,
where we used e− j2π nMo /Mo = 1. A similar relation applies to k2 .
In quadrant 2, µ is negative and ν is positive, so keeping
0 ≤ k1 , k2 ≤
Mo − 1
,
2
(4.28)
The result given by Eqs. (4.25)–(4.28) states that the 2-D
CSFT Fo (µ , ν ) of the sampled image fo (x, y)—when sampled
at the discrete spatial frequency values defined by Eq. (4.24)—
is the 2-D DFT of the sampled image f [n, m]. Also, the spectrum Fo (µ , ν ) of the sampled image consists of copies of the
www.dbooks.org
136
CHAPTER 4 IMAGE INTERPOLATION
spectrum F(µ , ν ) repeated every 1/∆o in both µ and ν , and also
scaled by 1/∆2o. Hence, generalizing Eq. (2.54) from 1-D to 2-D
gives
k1
k1
k2
k2
1
Fo
,
,
= 2F
,
(4.29)
Mo ∆o Mo ∆o
∆o
Mo ∆o Mo ∆o
upon sampling Fu (µ , ν ) at the rates defined by Eq. (4.24), we
obtain
k2
k1
,
Fu
Mo ∆o Mo ∆o
Mu −1 Mu −1
= ∑
where F(µ , ν ) is the 2-D CSFT of the continuous-space image
f (x, y) and 0 ≤ |k1 |, |k2 | ≤ (Mo − 1)/2.
A. Upsampled Image
n=0 m=0
= G[k1 , k2 ],
Fu
Mo − 1
,
2
(4.32)
−k1
k2
,
Mo ∆o Mo ∆o
Mu −1 Mu −1
0 ≤ k1 , k2 ≤
Mo − 1
2
= ∑
∑ g[n, m] e− j2π n(−k1)/Mu e− j2π mk2 /Mu
∑ f (n∆u , m∆u ) δ (x − n∆u) δ (y − m∆u)
= ∑
∑ g[n, m] e− j2π n(Mu−k1 )/Mu e− j2π mk2/Mu
∑ g[n, m] δ (x − n∆u) δ (y − m∆u),
= G[Mu − k1, k2 ],
n=0 m=0
Mu −1 Mu −1
Mu −1 Mu −1
n=0 m=0
Mu −1 Mu −1
= ∑
0 ≤ k1 , k2 ≤
2. Quadrant 2: µ ≤ 0 and ν ≥ 0
Now we repeat this entire derivation using a sampling interval
∆u instead of ∆o , and replacing Mo with Mu , but keeping the form
of the relations given by Eq. (4.24) the same. Since ∆u < ∆o , the
sampled image is now (Mu × Mu ) instead of (Mo × Mo ). Hence,
the (Mu × Mu ) sampled image is
fu (x, y) = ∑
∑ g[n, m] e− j2π nk1/Mu e− j2π mk2/Mu
n=0 m=0
(4.30)
n=0 m=0
0 ≤ k1 , k2 ≤
Mo − 1
,
2
(4.33)
with g[n, m] as defined in Eq. (4.19b). The associated 2-D CSFT
of the sampled image fu (x, y) is
Fu (µ , ν ) = F { fu (x, y)}
3. Quadrant 3: µ ≤ 0 and ν ≤ 0
Mu −1 Mu −1
= ∑
∑ g[n, m] F {δ (x − n∆u) δ (y − m∆u)}
n=0 m=0
Mu −1 Mu −1
= ∑
∑ g[n, m] e− j2π µ n∆u e− j2πν m∆u .
n=0 m=0
Fu
(4.31)
−k1 −k2
,
Mo ∆o Mo ∆o
Mu −1 Mu −1
,
0 ≤ k1 , k2 ≤
Mo − 1
2
= ∑
∑ g[n, m] e− j2π n(−k1)/Mu e− j2π m(−k2)/Mu
= ∑
∑ g[n, m] e− j2π n(Mu−k1 )/Mu e− j2π m(Mu −k2 )/Mu
n=0 m=0
Mu −1 Mu −1
n=0 m=0
1. Quadrant 1: µ ≥ 0 and ν ≥ 0
In view of the relationship (from Eq. (4.14))
∆u
∆u
1
=
=
,
Mo ∆o
Mu ∆u
Mu
= G[Mu − k1, Mu − k2 ],
0 ≤ k1 , k2 ≤
Mo − 1
.
2
(4.34)
4-4 IMPLEMENTATION OF UPSAMPLING USING 2-D DFT IN MATLAB
4. Quadrant 4: µ ≥ 0 and ν ≤ 0
Fu
k1
−k2
,
Mo ∆o Mo ∆o
Mu −1 Mu −1
,
0 ≤ k1 , k2 ≤
137
sampled signal is zero, since sampling the original signal f (x, y)
at above its Nyquist rate separates the copies of its spectrum,
leaving bands of zero between copies. Thus
Mo − 1
2
G[k1 , k2 ] = 0,
= ∑
∑ g[n, m] e− j2π nk1/Mu e− j2π m(−k2)/Mu
= ∑
∑ g[n, m] e− j2π nk1/Mu e− j2π m(Mu −k2 )/Mu
Mo ≤ k1 , k2 ≤ Mu − 1.
(4.39)
n=0 m=0
Mu −1 Mu −1
n=0 m=0
= G[k1 , Mu − k2 ],
Mo − 1
0 ≤ k1 , k2 ≤
,
2
(4.35)
The result given by Eqs. (4.32)–(4.35) states that the 2-D
CSFT Fu (µ , ν ) of the upsampled image fu (x, y)—when sampled
at the discrete spatial frequency values defined by Eq. (4.24)—is
the 2-D DFT of the sampled image g[n, m]. Also, the spectrum
Fu (µ , ν ) of the sampled image consists of copies of the spectrum F(µ , ν ) of f (x, y) repeated every 1/∆u in both µ and ν ,
and also scaled by 1/∆2u. Thus,
k1
1
k2
k2
k1
Fu
= 2F
,
(4.36)
,
,
Mo ∆o Mo ∆o
∆u
Mo ∆o Mo ∆o
From Eq. (4.14), Mo ∆o = Mu ∆u . Combining Eq. (4.29) and
Eq. (4.36) shows that
k1
k1
M2
k2
k2
Fu
= u2 Fo
.
(4.37)
,
,
Mo ∆o Mo ∆o
Mo
Mo ∆o Mo ∆o
4-4 Implementation of Upsampling
Using 2-D DFT in MATLAB
In MATLAB, both the image f [n, m] and its 2-D DFT are stored
and displayed using the format shown in Fig. 3-25(d), wherein
the origin is at the upper left-hand corner of the image, and the
indices of the corner pixel are (1, 1).
Image and 2-D DFT Notation
To avoid confusion between the common-image format
(CIF) and the MATLAB format, we provide the following
list of symbols and definitions:
Original image
Upsampled image
2-D DFT of f [n, m]
2-D DFT of g[n, m]
Hence, the (Mo × Mo ) 2-D DFT F[k1 , k2 ] of f [n, m] and the
(Mu × Mu ) 2-D DFT G[k1 , k2 ] of g[n, m] are related by
G[k1 , k2 ] =
G[Mu − k1 , k2 ] =
G[k1 , Mu − k1 ] =
CIF
MATLAB
f [n, m]
g[n, m]
F[k1 , k2 ]
G[k1 , k2 ]
X(m′ , n′ )
Y(m′ , n′ )
FX(k2′ , k1′ )
FY(k2′ , k1′ )
Mu2
F[k1 , k2 ],
Mo2
Mu2
F[Mo − k1 , k2 ],
Mo2
M2
u
Mo2
As noted earlier in Section 3-9, when an image f [n, m] is stored
in MATLAB as array X(m′ , n′ ), the two sets of indices are related
by
F[k1 , Mo − k2 ],
Mu2
F[Mo − k1 , Mo − k2 ],
Mo2
Mo − 1
0 ≤ k1 , k2 ≤
.
(4.38)
2
m′ = m + 1,
G[Mu − k1 , Mu − k2 ] =
This leaves G[k1 , k2 ] for Mo ≤ k1 , k2 ≤ Mu − 1 to be determined.
But Eq. (4.38) shows that these values of G[k1 , k2 ] are samples
of Fu (µ , ν ) at values of µ , ν for which this spectrum of the
′
n = n + 1.
(4.40a)
(4.40b)
The indices get interchanged in orientation (n represents row
number, whereas n′ represents column number) and are shifted
by 1. For example, f [0, 0] = X(1, 1), and f [0, 1] = X(2, 1). While
the indices of the two formats are different, the contents of
array X(m′ , n′ ) is identical with that of f [n, m]. That is, for an
www.dbooks.org
138
CHAPTER 4 IMAGE INTERPOLATION
(Mo × Mo ) image
zeros in the “middle” of the array FX. The result is
X(m′ , n′ ) = f [n, m] =

f [0, 0]
f [1, 0]
f
[0,
1]
f
[1, 1]


.
..

..
.
f [0, Mo − 1]
···
···
..
.
f [1, Mo − 1] · · ·

f [Mo − 1, 0]
f [Mo − 1, 1]
..
.

.

f [Mo − 1, Mo − 1]
(4.41)
The MATLAB command FX = fft(X, Mo , Mo ) computes the
2-D DFT F[k1 , k2 ] and stores it in array FX(k2′ , k1′ ):
FX(k2′ , k1′ ) = F[k1 , k2 ] =

F[0, 0]
F[1, 0]
F[1, 1]
 F[0, 1]

..
..

.
.
···
···
..
.
F[Mo − 1, 0]
F[Mo − 1, 1]
..
.


.

F[0, Mo − 1] F[1, Mo − 1] · · · F[Mo − 1, Mo − 1]
(4.42)
The goal of upsampling using the spatial-frequency domain
is to compute g[n, m] from f [n, m] by computing G[k1 , k2 ] from
F[k1 , k2 ] and then applying the inverse DFT to obtain g[n, m].
The details of the procedure are somewhat different depending
on whether the array size parameter M is an odd integer or an
even integer. Hence, we consider the two cases separately.
4-4.1 Mo = Odd Integer
The recipe for upsampling using the 2-D DFT is as follows:
1. Given: image { f [n, m], 0 ≤ n, m ≤ Mo −1 }, as represented
by Eq. (4.41).
2. Compute: the 2-D DFT F[k1 , k2 ] of f [n, m] using
Eq. (4.20a) to obtain the array represented by Eq. (4.42).
3. Create: an upsampled (Mu × Mu ) array G[k1 , k2 ], and then
set its entries per the rules of Eq. (4.38).
4. Compute: the (Mu × Mu ) 2-D inverse DFT of G[k1 , k2 ] to
obtain g[n, m].
In MATLAB, the array FY containing the 2-D DFT G[k1 , k2 ]
is obtained from the array FX given by Eq. (4.42) by inserting
(Mu − Mo ) rows of zeros and an equal number of columns of
FY =

Mu2
×
Mo2
h
i
Mo −1
2 ,0

F[0,0]... F


..


.h
 h
i
i

F 0, Mo2−1 ... F Mo2−1 , Mo2−1


0.


..

{
0.


.

.

 h
0
i
h
i

F 0, Mo +1 ... F Mo −1 , Mo +1

2
2
2


..

.h

i
F [0,Mo −1]... F Mo2−1 ,Mo −1
(Mu −Mo ) columns
z }| {
0 ... 0 ... 0
.. .. ..
. . .
..
.
.
0 ..
.. .. ..
. . .
0| {z
0 0}
h
i
Mo +1
2 ,0




..


.
h
i
h
i
Mo +1 Mo −1
Mo −1 
F
... F Mo −1, 2 
2 , 2


0.


..

0.
}


.

.

0
h
i
h
i

Mo +1 Mo +1
Mo +1 
F
... F Mo −1, 2 
2 , 2


..

.

i
h
Mo +1
... F [Mo −1,Mo −1]
F Mo −1, 2
F
... F[Mo −1,0]
(4.43)
◮ Note that the (Mu − Mo ) columns of zeros start after entry
F[(Mo − 1)/2, 0], and similarly the (Mu − Mo ) rows of zeros
start after F[0, (Mo − 1)/2]. ◭
Once array FY has been established, the corresponding upsampled image g[n, m] is obtained by applying the MATLAB
command Y=real(ifft2(FY,N,N)), where N = Mu and
the “real” is needed to eliminate the imaginary part of Y, which
may exist because of round-off error in the ifft2.
As a simple example, consider the (3 × 3) array


F[0, 0] F[1, 0] F[2, 0]
FX = F[0, 1] F[1, 1] F[2, 1] .
(4.44a)
F[0, 2] F[1, 2] F[2, 2]
To generate a 5 × 5 array FY we insert Mu − Mo = 5 − 3 = 2
columns of zeros after element F[(Mo − 1)/2, 0] = F[1, 0], and
also 2 rows of zeros after F[0, (Mo − 1)/2] = F[0, 1]. The result
is


F[0, 0] F[1, 0] 0 0 F[2, 0]
F[0, 1] F[1, 1] 0 0 F[2, 1]
52 


0
0 0
0 .
(4.44b)
FY = 2  0
3  0
0
0 0
0 
F[0, 2] F[1, 2] 0 0 F[2, 2]
Application of the inverse 2-D DFT to FY generates array Y in
MATLAB, which is equivalent in content to image g[n, m] in
common-image format.
4-4 IMPLEMENTATION OF UPSAMPLING USING 2-D DFT IN MATLAB
4-4.2 Mo = Even Integer
For a real-valued (Mo × Mo ) image f [n, m] with Mo = an odd
integer, conjugate symmetry is automatically satisfied for both
F[k1 , k2 ], the 2-D DFT of the original image, as well as for
G[k1 , k2 ], the 2-D DFT of the upsampled image. However, if
Mo is an even integer, application of the recipe outlined in the
preceding subsection will violate conjugate symmetry, so we
need to modify it. Recall from Eq. (3.80) that for a real-valued
image f [n, m], conjugate symmetry requires that
F∗ [k1 , k2 ] = F[Mo − k1 , Mo − k2 ],
1 ≤ k1 , k2 ≤ Mo − 1.
(4.45)
Additionally, in view of the definition for F[k1 , k2 ] given by
Eq. (4.20a), the following two conditions should be satisfied:
Mo −1 Mo −1
F[0, 0] = ∑
∑ f [n, m] = real-valued,
(4.46a)
n=0 m=0
Mo −1 Mo −1
Mo Mo
= ∑ ∑ (−1)n+m f [n, m] = real-valued.
,
F
2 2
n=0 m=0
(4.46b)
one column of zeros and multiplying by (52 /32 ) would generate


49
−6 − j3 0
3
−6 + j3
−5 − j4 2 + j9 0 −5 + j2
j 
52 


0
0
0
0 .
G[k1 , k2 ] = 2  0
3  1
−4 + j3 0
−1
−4 − j3
−5 + j4
−j
0 −5 − j2 2 − j9
(4.48)
This G[k1 , k2 ] array does not satisfy conjugate symmetry. Applying the inverse 2-D DFT to G[k1 , k2 ] generates the upsampled
image g[n, m]:
g[n,m] = (52 /32 )×

0.64
1.05 + j0.19
1.05 + j0.19
2.01 + j0.22

2.38 − j0.44
 1.64 − j0.3
2.39 + j0.30 2.37 − j0.066
2.27 − j0.19
1.89 − j0.25
2
4
4
3
3
5
6
2

4
3
.
2
1
The (4 × 4) 2-D DFT F[k1 , k2 ] of f [n, m] is

1.64 − j0.30
2.85 − j0.20
3.9 + j0.46
2.88 + j0.36
1.33 + j0.26
G[k1 , k2 ] = (62 /42 )×

F[0, 0]
F[1, 0] F[2, 0]/2
 F[0, 1]
F[1, 1] F[2, 1]/2

F[0, 2]/2 F[1, 2]/2 F[2, 2]/4

0
0
0

F[0, 2]/2 F[1, 2]/2 F[2, 2]/4
F[0, 3]
F[1, 3] F[2, 3]/2

49
−6 − j3
1.5
−5 − j4
2 + j9
−2.5 + j1

0.5
−2 + j1.5
−0.25
(4.47a) 
 0
0
0

 0.5
−2 + j1.5
−0.25
−5 − j4
−j
− j0.5

49
−6 − j3
3
−6 + j3
j 
−5 − j4 2 + j9 −5 + j2
F[k1 , k2 ] = 
.
1
−4 + j3
−1
−4 − j3
−5 + j4
−j
−5 − j2 2 − j9
(4.47b)
This F[k1 , k2 ] has conjugate symmetry.
Let us suppose that we wish to upsample the (4 × 4) image
f [n, m] to a (5 × 5) image g[n, m]. Inserting one row of zeros and
2.39 + j0.30
2.8 − j0.13
3.16 + j0.08
2.04 − j0.9
0.83 + j0.08

2.27 − j0.19
1.82 − j0.19

1.31 + j0.39 ,
0.84 + j0.12
1.18 + j0.22
which is clearly incorrect; all of its elements should be realvalued because the original image f [n, m] is real-valued. Obviously, the upsampling recipe needs to be modified.
A simple solution is to split row F[k1 , Mo /2] into 2 rows and
to split column F[Mo /2, k2 ] into 2 columns, which also means
that F[Mo /2, Mo /2] gets split into 4 entries. The recipe preserves
conjugate symmetry in G[k1 , k2 ].
When applied to F[k1 , k2 ], the recipe yields the (6 × 6) array
In the preceding subsection, we inserted the appropriate number
of rows and columns of zeros to obtain G[k1 , k2 ] from F[k1 , k2 ].
For Mo equal to an odd integer, the conditions represented
by Eqs. (4.45) and (4.46) are satisfied for both F[k1 , k2 ] and
G[k1 , k2 ], but they are not satisfied for G[k1 , k2 ] when Mo is
an even integer. To demonstrate why the simple zero-insertion
procedure is problematic, let us consider the (4 × 4) image

1
2
f [n, m] = 
3
4
139
0
0
0
0
0
0
0
0
0
0
0
0

F[2, 0]/2 F[3, 0]
F[2, 1]/2 F[3, 1] 

F[2, 2]/4 F[3, 2]/2
=
0
0

F[2, 2]/4 F[3, 2]/2
F[2, 3]/2 F[3, 3]

1.5
−6 + j3

−2.5 + j1
j

−0.25
−2 − j1.5
.
0
0

−0.25
−2 − j1.5
−2.5 − j1
2 − j9
(4.49)
Application of the inverse 2-D DFT to G[k1 , k2 ] yields the
www.dbooks.org
140
CHAPTER 4 IMAGE INTERPOLATION
Answer: The 2-point DFT X[k] = {8 + 4, 8 − 4} = {12, 4}.
Since Mo = 2 is even, we split 4 and insert a zero in the
middle, and multiply by 4/2 to get Y[k] = {12, 2, 0, 2}. The
inverse 4-point DFT is
(a) Original 4 × 4 image
(b) Upsampled 6 × 6 image
Figure 4-4 Comparison of original and upsampled images.
2
y[0] = (12 + 2 + 0 + 2) = 8,
4
2
y[1] = (12 + 2 j − 0 − 2 j) = 6,
4
2
y[2] = (12 − 2 + 0 − 2) = 4,
4
2
y[3] = (12 − 2 j + 0 + 2 j) = 6.
4
y[n] = {8, 6, 4, 6}.
4-5 Downsampling
upsampled image


0.44 0.61 1.06 1.33 1.83 1.38
0.61 1.09 1.73 1.91 1.85 1.21

62 
1.06 1.55 2.31 2.58 1.66 0.90
g[n, m] = 2 

4 1.33 1.55 2.22 2.67 1.45 0.78
1.83 1.72 1.52 1.42 0.54 0.73
1.38 1.25 0.94 0.76 0.72 1.04


1.0 1.37 2.39 3.0 4.12 3.11
1.37 2.46 3.90 4.30 4.16 2.72


2.39 3.49 5.20 5.81 3.74 2.03
=
 , (4.50)
 3.0 3.49 5.00 6.01 3.26 1.76
4.12 3.87 3.42 3.20 1.22 1.64
3.11 2.81 2.12 1.71 1.62 2.34
which is entirely real-valued, as it should be. The original (4×4)
image given by Eq. (4.47a) and the upsampled (6 × 6) image
given by Eq. (4.50) are displayed in Fig. 4-4. The two images,
which bear a close resemblance, have the same physical size but
different-sized pixels.
Downsampling is the inverse operation to upsampling. The
objective of downsampling is to reduce the array size f [n, m]
of an image from (Mo × Mo ) samples down to (Md × Md ), with
Md < Mo . The original image f [n, m] and the downsampled
image g[n, m] are defined as:
Original image f [n, m] = { f (n∆o , m∆o ), 0 ≤ n, m ≤ Mo − 1},
Downsampled image g[n, m] = { f (n∆d , m∆d ), 0 ≤ n, m ≤ Md − 1}.
Both images are sampled versions of some continuous-space
image f (x, y), with image f [n, m] sampled at a sampling interval
∆o that satisfies the Nyquist rate, and the downsampled image
g[n, m] is sampled at ∆d , with ∆d > ∆o , so it is unlikely that
g[n, m] satisfies the Nyquist rate.
The goal of downsampling is to compute g[n, m] from f [n, m].
That is, we compute a coarser-discretized Md × Md image g[n, m]
from the finer-discretized Mo × Mo image f [n, m]. Applications
of downsampling include computation of “thumbnail” versions
of images, as demonstrated in Example 4-1, and shrinking
images to fit into a prescribed space, such as columns in a
textbook.
4-5.1 Aliasing
Exercise 4-2: Upsample the length-2 signal x[n] = {8, 4} to
a length-4 signal y[n].
It might seem that downsampling by, say, two, meaning that
Md = Mo /2 (assuming Mo is even), could be easily accomplished
by simply deleting every even-indexed (or odd-indexed) row
and column of f [n, m]. Deleting every other row and column of
4-6 ANTIALIAS LOWPASS FILTERING
141
f [n, m] is called decimation by two. Decimation by two would
give the result of sampling f (x, y) every 2∆o instead of every
∆o . But if sampling at S = 1/2∆o is below the Nyquist rate, the
decimated image g[n, m] is aliased. The effect of aliasing on the
spectrum of g[n, m] can be understood in 1-D from Fig. 2-6(b).
The copies of the spectrum of f (x, y) produced by sampling
overlap one another, so the high-frequency parts of the signal
become distorted. Example 4-1 gives an illustration of aliasing
in 2-D.
Example 4-1: Aliasing
The 200 × 200 clown image shown in Fig. 4-5(a) was decimated
to the 25 × 25 image shown in Fig. 4-5(b). The decimated image
is a poor replica of the original image and could not function as
a thumbnail image.
4-6 Antialias Lowpass Filtering
Clearly decimation alone is not sufficient to obtain a downsampled image that looks like a demagnified original image. To
avoid aliasing, it is necessary to lowpass filter the image before
decimation, eliminating the high spatial frequency components
of the image, so that when the filtered image is decimated, the
copies of the spectra do not overlap. In 1-D, in Fig. 2-6(b),
had the spectrum been previously lowpass filtered with cutoff
frequency S/2 Hz, the high-frequency parts of the spectrum
would no longer overlap after sampling. This is called antialias
filtering.
The same concept applies in discrete space (and time). The
periodicity of spectra induced by sampling becomes the periodicity of the DSFT and DTFT with periods of 2π in (Ω1 , Ω2 )
and Ω, respectively. Lowpass filtering can be accomplished by
setting certain high-spatial-frequency portions of the 2-D DFT
to zero. The purpose of this lowpass filtering is now to eliminate
the high-spatial-frequency parts of the discrete-space spectrum,
prior to decimation. This eliminates aliasing, as demonstrated in
Example 4-2.
Example 4-2:
Antialiasing
The 200 × 200 clown image shown in Fig. 4-5(a) was first
lowpass-filtered to the image shown in Fig. 4-6(a), with the
spectrum shown in Fig. 4-6(b), then decimated to the 25 × 25
image shown in Fig. 4-6(c). The decimated image is now a good
replica of the original image and could function as a thumbnail
image. The decimated image pixel values in Fig. 4-6(c) are all
equal to certain pixel values in the lowpass-filtered image in
Fig. 4-6(a).
(a) Original
◮ A MATLAB code for this example is available on the
book website. ◭
4-6.1 Downsampling in the 2-D DFT Domain
(b) Decimated 25 × 25 version
Figure 4-5 Clown image: (a) original 200 × 200 and (b)
decimated 25 × 25 version.
p
The antialiasing approach works well when downsampling by
an integer factor L, since decimation can be easily performed
by keeping only every Lth row and column of the antialiasedfiltered image. But downsampling by a non-integer factor
Md /Mo must be performed entirely in the 2-D DFT domain, as
follows.
Antialias lowpass filtering is performed by setting to zero
the Mo − Md center rows and columns of the 2-D DFT, in
www.dbooks.org
142
CHAPTER 4 IMAGE INTERPOLATION
MATLAB depiction. Decimation is then performed by deleting
those (Mo − Md ) center rows and columns of the 2-D DFT. Of
course, there is no reason to set to zero rows and columns that
will be deleted anyways, so in practice the first step need not be
performed. By “center rows and columns” we mean those rows
and columns with DFT indices k (horizontal or vertical; add one
to all DFT indices to get MATLAB indices) in the (Mo × Mo )
2-D DFT of the (M × M) original image f [n, m]:
• For Md odd: (Md + 1)/2 ≤ k ≤ Mo − (Md + 1)/2.
• For Md even: Md /2 ≤ k ≤ Mo − Md /2, then insert a row or
column of zeros at index k = Md /2.
Note that there is no need to subdivide some values of F[k1 , k2 ]
to preserve conjugate symmetry.
The procedure is best illustrated by an example.
(a) Lowpass-filtered 200 × 200 image
Example 4-3:
Downsampling
The goal is to downsample the MATLAB (6 × 6) image array:


3 1 4 1 5 9
2 6 5 3 5 8


9 7 9 3 2 3
X(m, n) = 
(4.51)

8 4 6 2 6 4
3 3 8 3 2 7
9 5 0 2 8 8
The corresponding magnitudes of the (6 × 6) 2-D DFT of this
array in MATLAB is
(b) Spectrum of lowpass-filtered image
(c) The decimated 25 × 25 image is not aliased
Figure 4-6 Clown image: (a) lowpass-filtered 200 × 200 image, (b) spectrum of image in (a), and (c) unaliased, decimated
version of (a).
|FX(k1 , k2 )| =

173.00 23.81
 6.93 25.00

 11.14 19.98
 9.00 16.09

 11.14 4.58
6.93 21.93
20.66
7.55
16.09
19.98
14.93
7.94
15.00
14.00
15.10
1.00
15.10
14.00
20.66
7.94
14.93
19.98
16.09
7.55

23.81
21.93

4.58 
. (4.52)
16.09


19.98
25.00
Setting to zero the middle three columns of the 2-D DFT
magnitudes in MATLAB depiction gives the magnitudes


173.00 23.81 0 0 0 23.81
 6.93 25.00 0 0 0 21.93


0
0 0 0
0 
 0
|FG(k1 , k2 )| = 
. (4.53)
0
0 0 0
0 
 0

 0

0
0 0 0
0
6.93 21.93 0 0 0 25.00
4-7 B-SPLINES INTERPOLATION
143
Deleting the zero-valued rows and columns in the 2-D DFT
magnitudes in MATLAB depiction gives the magnitudes


173.00 23.82 23.82
|FG(k1 , k2 )| =  6.93 25.00 21.93 .
(4.54)
6.93 21.93 25.00
Multiplying by Md2 /Mo2 = 32 /62 = 1/4 and taking the inverse
2-D DFT gives


5.83 1.58 6.00
g[m, n] = 6.08 6.33 3.00 .
(4.55)
6.25 3.5 4.67
This is the result we would have gotten by decimating the
antialiased lowpass-filtered original image, but it was performed
entirely in the 2-D DFT domain.
Photoshop R and through the command imresize.
4-7.1 B-Splines
Splines are piecewise-polynomial functions whose polynomial
coefficients change at half-integer or integer values of the
independent variable, called knots, so that the function and some
of its derivatives are continuous at each knot. In 1-D, a B-spline
βN (t) of order N is a piecewise polynomial of degree N, centered
at t = 0. The support of βN (t), which is the interval outside of
which βN (t) = 0, extends between −(N + 1)/2 and +(N + 1)/2.
◮ Hence, the duration of βN (t) is (N + 1). ◭
Formally, the B-spline function βN (t) is defined as
◮ A MATLAB code for this example is available on the
book website. ◭
Why must we delete rows and
columns of the 2-D DFT array to perform downsampling?
Concept Question 4-2:
4-7 B-Splines Interpolation
In the preceding sections we examined several different image
interpolation methods, some of which perform the interpolation
directly in the spatial domain, and others that perform the interpolation in the spatial frequency domain. Now, we introduce yet
another method, known as the B-splines interpolation method,
with the distinguishing feature that it is the method most commonly used for image interpolation. Unlike with downsampling,
B-spline interpolation has no aliasing issues when the sampling
interval ∆ is too large. Moreover, unlike with upsampling,
B-spline interpolation need not result in blurred images.
B-splines are a family of piecewise polynomial functions,
with each polynomial piece having a degree N, where N is
a non-negative integer. As we will observe later on in this
section, a B-spline of order zero is equivalent to the nearestneighbor interpolation method of Section 3-5.1, but it is simpler
to implement than the sinc interpolation formula. Interpolation
with B-splines of order N = 1 generates linear interpolation,
which is used in computer graphics. Another popular member
of the B-spline interpolation family is cubic interpolation, corresponding to N = 3. Cubic spline interpolation is used in Adobe R
βN (t) =
Z ∞
−∞
sin(π f ) N+1 j2π f t
d f,
e
πf
(4.56)
which is equivalent to the inverse Fourier transform of
sincN+1 ( f ). Recognizing that (a) the inverse Fourier transform
of sinc(t) is a rectangle function and (b) multiplication in
the frequency domain is equivalent to convolution in the time
domain, it follows that
βN (t) = rect(t) ∗ · · · ∗ rect(t),
|
{z
}
(4.57)
N+1 times
with
rect(t) =
(
1
0
for |t| < 1/2,
for |t| > 1/2.
(4.58)
Application of Eq. (4.57) for N = 0, 1, 2, and 3 leads to:
(
1
for |t| < 1/2,
(4.59)
β0 (t) = rect(t) =
0
for |t| > 1/2,
(
1 − |t|
for |t| < 1,
β1 (t) = β0 (t) ∗ β0(t) =
(4.60)
0
for |t| > 1,
β2 (t) = β1 (t) ∗ β0(t)

2

3/4 − t
1
= 2 (3/2 − |t|)2

0
www.dbooks.org
for 0 ≤ |t| ≤ 1/2,
for 1/2 ≤ |t| ≤ 3/2,
for |t| > 3/2,
(4.61)
144
CHAPTER 4 IMAGE INTERPOLATION
β3 (t) = β2 (t) ∗ β0(t)

2
3

2/3 − t + |t| /2
3
= (2 − |t|) /6

0
for |t| ≤ 1,
for 1 ≤ |t| ≤ 2,
for |t| > 2.
1
(4.62)
Note that in all cases, βN (t) is continuous over its full duration.
For N ≥ 1, the B-spline function βN (t) is continuous and
differentiable (N − 1) times at all times t. For β2 (t), the function
is continuous across its full interval (−3/2, 3/2), including at
t = 1/2. Similarly, β3 (t) is continuous over its interval (−2, 2),
including at t = 1.
Plots of the B-splines of order N = 0, 1, 2, and 3 are displayed
in Fig. 4-7. From the central limit theorem in the field of
probability, we know that convolving a function with itself
repeatedly makes the function resemble a Gaussian. This is
evident in the present case as well.
From the standpoint of 1-D and 2-D interpolation of signals
and images, the significance of B-splines is in how we can use
them to express a signal or image. To guide us through the
process, let us assume we have 6 samples x(n∆), as shown in
Fig. 4-8, extending between t = 0 and t = 5∆. Our objective
is to interpolate between these 6 points so as to obtain a
continuous function x(t). An important constraint is to ensure
that x(t) = x(n∆) at the 6 discrete times n∆.
For a B-spline of a specified order N, the interpolation is
realized by expressing the desired interpolated signal x(t) as a
linear combination of time-shifted B-splines, all of order N:
∞
x(t) =
t
∑ c[m] βN ∆ − m .
m=−∞
−0.5 0
(a) β0(t)
1
−1
0
t
1
(b) β1(t)
0.8
0.4
(4.63)
Here, βN ∆t − m is the B-spline function βN (t), with t scaled by
the sampling interval ∆ and delayed by a scaled time integer m.
◮ The support of βN ∆t − m is
t
N +1
N+1
< < m+
.
m−
2
∆
2
That is, βN ∆t − m = 0 outside that interval. ◭
t
0.5
−2
−1
0
1
2
1
2
t
(c) β2(t)
0.7
0.6
(4.64)
Associated with each value of m is a constant coefficient c[m]
whose value is related to the sampled values x(n∆) and the order
N of the B-spline. More specifically, the values of c[m] have to
be chosen such that the aforementioned constraint requiring that
x(t) = x(n∆) at discrete times t = n∆ is satisfied. The process is
0.4
0.2
t
−3
−2
−1
0
3
(d) β3(t)
Figure 4-7 Plots of βN (t) for N = 0, 1, 2, and 3.
4-7 B-SPLINES INTERPOLATION
145
4
4
3
3
2
2
1
1
0
0
1
2
3
4
5
t/Δ
0
x(nΔ)
x(t)
0
1
Figure 4-8 Samples x(n∆) to be interpolated into x(t).
described in forthcoming subsections.
Since each B-spline is a piecewise polynomial of order N,
continuous and differentiable (N − 1) times, any linear combination of time-shifted B-splines also constitutes a piecewise
polynomial of order N, and will also be continuous and differentiable. Thus, the B-splines form a basis—hence, the “B” in
their name—and where the basis is used to express x(t), as in
Eq. (4.63), x(t) is continuous and differentiable (N − 1) times at
the knots t = m∆ if N is odd and at the knots t = (m + 1/2)∆
if N is even. This feature of B-splines makes them suitable for
interpolation, as well as for general representation of signals and
images.
Exercise 4-3: Show that the area under βN (t) is 1.
R∞
Answer: −∞ βN (t) dt = B(0) by entry #11 in Table 2-4.
Set f = 0 in sincN ( f ) and use sinc(0) = 1.
Exercise 4-4: Why does βN (t) look like a Gaussian func-
tion for N ≥ 3?
Answer: Because βN (t) is rect(t) convolved with itself
(N + 1) times. By the central limit theorem of probability, convolving any square-integrable function with itself
results in a function resembling a Gaussian.
Exercise 4-5: Show that the support of βN (t) is
−(N + 1)/2 < t < (N + 1)/2.
2
3
4
5
t/Δ
Figure 4-9 B-spline interpolation for N = 0.
4-7.2 N = 0: Nearest-Neighbor Interpolation
For N = 0, Eqs. (4.63) and (4.64) lead to
t
∑ c[m] rect
−m
∞
∑ c[m] β0 ∆ − m
x(t) =
m=−∞
∞
=
m=−∞
∞
=
(
1
∑ c[m] 0
m=−∞
t
∆
for m − 21 < ∆t < m + 12 ,
otherwise.
(4.65)
The expression given in Eq. (4.65) consists of a series of adjoin-
ing, but not overlapping, rectangle functions. For m = 0, rect ∆t
1
t
1
is centered at ∆t = 0 and extends over
the range − 2 t < ∆ < 2 .
t
Similarly, for m = 1, rect ∆ − 1 is centered at ∆ = 1 and
extends over 21 < ∆t < 32 . Hence, to satisfy the constraint that
x(t) = x(n∆) at the sampled locations t = n∆, we need to set
m = n and c[m] = x(n∆):
x(t) = x(n∆) rect
t
∆
−n ,
n−
1
t
1
< < n + . (4.66)
2 ∆
2
The interpolated function x(t) is shown in Fig. 4-9, along with
the 6 samples { x(n∆) }. The B-spline representation given by
Eq. (4.66) is a nearest-neighbor (NN) interpolation: x(t) in
the interval { n∆ ≤ t ≤ (n + 1)∆ } is set to the closer (in time)
of x(n∆) and x((n + 1)∆). The resulting x(t) is a piecewise
constant, as shown in Fig. 4-9.
Answer: The support of βN (t) is the interval outside of
which βN (t) = 0. The support of rect(t) is −1/2 < t < 1/2.
βN (t) is rect(t) convolved with itself N + 1 times, which has
duration N + 1 centered at t = 0.
4-7.3 Linear Interpolation
For N = 1, the B-spline β1 ∆t − m assumes the shape of a
triangle (Fig. 4-7(b)) centered at t/∆ = m. Figure 4-10(b)
displays the triangles centered at t/∆ = m for m = 0 through 5.
Also displayed in the same figure are the values of x(n∆). To
www.dbooks.org
146
CHAPTER 4 IMAGE INTERPOLATION
4
3
2
1
0
t/Δ
0
1
2
3
4
β1(t/Δ − 3)
β1(t/Δ − 4)
5
(a) x(nΔ)
β1(t/Δ − 1)
β1(t/Δ − 2)
0
1
2
0
1
2
β1(t/Δ) 1
0
−1
β1(t/Δ − 5)
3
(b) β1(t/Δ − m)
4
5
3
4
5
4
3
2
1
0
(c) x(t)
Figure 4-10 B-spline linear interpolation (N = 1).
6
t/Δ
4-7 B-SPLINES INTERPOLATION
147
satisfy Eq. (4.63) for N = 1, namely
∞
x(t) =
∑
c[m] β1
m=−∞
3
t
−m ,
∆
(4.67)
(N = 1).
Consequently, for any integer n, x(t) at time t between n∆ and
(n + 1)∆ is a weighted average given by
t
t
+ x((n + 1)∆)
− n . (4.68a)
x(t) = x(n∆) (n + 1) −
∆
∆
The associated duration is
n∆ ≤ t ≤ (n + 1)∆.
2
1.5
as well as meet the condition that x(t) = x(n∆) at the 6 given
points, we should set m = n and select
c[m] = c[n] = x(∆n)
m=0
2.5
m=2
m = −2
1
m=1
m = −1
0.5
0
−4
−3
−2
−1
0
1
3
4
t
Δ
Figure 4-11 B-splines β2 (t/∆ − m) overlap in time.
samples { x(n∆) } can be derived by starting with Eq. (4.63),
∞
x(t) =
(4.68b)
Application of the B-spline linear interpolation to the given
samples x(n∆) leads to the continuous function x(t) shown in
Fig. 4-10(c). The linear interpolation amounts to setting { x(t),
n∆ ≤ t ≤ (n + 1)∆ } to lie on the straight line connecting x(n∆)
to x((n + 1)∆).
2
t
∑ c[m] βN ∆ − m ,
m=−∞
(4.69)
and then setting t = n∆, which gives
∞
x(n∆) =
∑ c[m] βN (n − m) = c[n] ∗ βN (n),
(4.70)
m=−∞
where use was made of the discrete-time convolution relation
given by Eq. (2.71a).
For N = 2, Eq. (4.61) indicates that β2 (n) 6= 0 only for integers
n = { −1, 0, 1 }. Hence, the discrete-time convolution given by
Eq. (4.70) simplifies to
Exercise 4-6: Given the samples
{ x(0), x(∆), x(2∆), x(3∆) } = { 7, 4, 3, 2 },
compute x(∆/3) by interpolation using: (a) nearest neighbor; (b) linear.
x(n∆) = c[n − 1] β2 (1) + c[n] β2 (0) + c[n + 1] β2 (−1)
= 81 c[n − 1] + 43 c[n] + 81 c[n + 1]
Answer: (a) ∆/3 is closer to 0 than to ∆, so
x(∆/3) = x(0) = 7.
(b)
x(∆/3) = (2/3)x(0)+(1/3)x(∆) = (2/3)(7)+(1/3)(4) = 6.
(N = 2). (4.71)
In the second step, the constant coefficients were computed
using Eq. (4.61) for β2 (t). The sum truncates because β2 (t) = 0
for |t| ≥ 3/2, so only three basis functions overlap at any specific
time t, as is evident in Fig. 4-11.
Similarly, for N = 3, β3 (t) = 0 for |t| ≥ 2, which also leads
to the sum of three terms:
x(n∆) = c[n − 1] β3 (1) + c[n] β2 (0) + c[n + 1] β3 (−1)
4-7.4 Quadratic Interpolation
For N ≥ 2, interpolation using B-splines becomes more complicated than for N = 0 and 1 because the supports of the basis
functions { βN ∆t − m } overlap in time for different values
of m, as shown in Fig. 4-11 for N = 2. Unlike the cases N = 0
and N = 1, wherein we set c[m] = x(m∆), now c[m] is related to
values of more than one of the discrete samples { x(n∆) }.
For N ≥ 2, the relationships between coefficients { c[m] } and
= 61 c[n − 1] + 64 c[n] + 61 c[n + 1]
(N = 3). (4.72)
If N is increased beyond 3, the number of terms increases, but
the proposed method of solution to determine the values of c[n]
remains the same. Specifically, we offer the following recipe:
(1) Delay { x(n∆) } to { xe(n∆) } to make it causal. Compute
the No th-order DFT X[k] of { xe(n∆) }, where No is the number of
samples { x(n∆) }.
www.dbooks.org
148
CHAPTER 4 IMAGE INTERPOLATION
(2) Delay { β2 (−1), β2 (0), β2 (1) } by 1 to { βe2 (t) } to make it
causal. Compute the No th-order DFT B[k] of
1 3 1
{ β2 (−1), β2 (0), β2 (1) } =
.
, ,
8 4 8
(3) Compute the No th-order inverse DFT to determine c[n]:
−1 X[k]
.
(4.73)
c[n] = DFT
B[k]
30
25
20
15
10
5
Example 4-4: Quadratic Spline Interpolation
0
−5
−4
−3
−2
−1
x(n) = { x(−3), x(−2), x(−1), x(0), x(1), x(2) }
= { 3, 19, 11, 17, 26, 4 }.
1
2
3
4
5
2
3
4
5
t
(a) x(n)
Figure 4-12(a) displays samples { x(n∆) } with ∆ = 1. Obtain
an interpolated version x(t) using quadratic splines.
Solution: From Fig. 4-12(a), we deduce that x(n) has 6
nonzero samples and is given by
0
30
25
20
15
As noted earlier,
β2 (n) = { β2 (−1), β2 (0), β2 (1) } =
1 3 1
.
, ,
8 4 8
Inserting x(n) and β (n) in Eq. (4.70) establishes the convolution
problem
1 3 1
{ 3, 19, 11, 17, 26, 4 } =
∗ c[n].
, ,
8 4 8
Following the solution recipe outlined earlier—and demonstrated in Section 2-9—the deconvolution solution is
10
5
0
−5
−4
−3
−2
−1
0
1
t
(b) x(t) interpolated using quadratic splines
Figure 4-12 (a) Original samples x(n) and (b) interpolated
function x(t).
c[n] = { 24, 8, 16, 32 },
and, therefore, the interpolated continuous function is
x(t) = 24β2(t + 2) + 8β2(t + 1) + 16β2(t) + 32β2(t − 1).
A plot of x(t) is shown in Fig. 4-12(b). It is evident that x(t) has
the same values as x(n) at t = { −3, −2, −1, 0, 1, 2 }.
◮ Note: The MATLAB code for solving Example 4-4 is
available on the book website. ◭
Concept Question 4-3: What is the difference between
zero-order-spline interpolation and nearest-neighbor interpolation?
Concept Question 4-4: Why use cubic interpolation,
when quadratic interpolation produces smooth curves?
4-8 2-D SPLINE INTERPOLATION
149
Why do quadratic and cubic interpolation require computation of coefficients, while linear
interpolation does not?
Concept Question 4-5:
four neighbors:
{ f (n∆, m∆), f ((n + 1)∆, m∆), f (n∆, (m + 1)∆),
f ((n + 1)∆, (m + 1)∆) }.
(4.77)
Exercise 4-7: In Eq. (4.63), why isn’t c[n] = x(n∆) for
N ≥ 2?
Answer: Because three of the basis functions βN ∆t − m
overlap for any t.
4-8 2-D Spline Interpolation
1-D interpolation using splines generalizes directly to 2-D. The
task now is to obtain a continuous function f (x, y) from samples
{ f (n∆, m∆) }.
The 2-D spline functions are separable products of the 1-D
spline functions:
βN (x, y) = βN (x) βN (y).
For example, for N = 1 we have
(
(1 − |x|)(1 − |y|)
β1 (x, y) =
0
(4.74)
for 0 ≤ |x|, |y| ≤ 1
otherwise,
(4.75)
which is pyramidal in shape. Interpolation using β1 (x, y) is
called bilinear interpolation, which is a misnomer because
β1 (x, y) includes a product term, |x||y|.
In 2-D, Eq. (4.63) becomes
∞
f (x, y) =
∞
x
y
∑ ∑ c[n, m] βN ∆ − n βN ∆ − m . (4.76)
m=−∞ n=−∞
For N = 0 and N = 1, c[n, m] = f (n∆, m∆), but for N ≥ 2, c[n, m]
is computed from { f (n∆, m∆) } using a 2-D version of the DFT
recipe outlined in Section 4-7.4.
4-8.1 Nearest-Neighbor (NN) image
Interpolation
NN interpolation of images works in the same way as NN
interpolation of 1-D signals. After locating the four samples
surrounding a location (x0 , y0 )—thereby specifying the applicable values of n and m, the value assigned to f (x, y) at location
(x0 , y0 ) is the value of the nearest-location neighbor among those
4-8.2 Bilinear Image Interpolation
Linear interpolation is performed as follows:
(1) Each image location (x0 , y0 ) has four nearest sampled
values given by Eq. (4.77), with a unique set of values for n
and m.
(2) Compute:
x0 f (x0 , m∆) = f (n∆, m∆) n + 1 −
x∆
0
−n ,
(4.78a)
+ f ((n + 1)∆, m∆)
∆
x0
f (x0 , (m + 1)∆) = f (n∆, (m + 1)∆) n + 1 −
x∆
0
+ f ((n + 1)∆, (m + 1)∆)
− n , (4.78b)
∆
and then combine them to find
y0 f (x0 , y0 ) = f (x0 , m∆) m + 1 −
y∆
0
+ f (x0 , (m + 1)∆)
−m .
∆
(4.79)
The preceding computation linearly interpolates in x for y = m∆,
and again for y = (m + 1)∆, and then linearly interpolates in y
for x = x0 .
4-8.3 Cubic Spline Interpolation
We describe the cubic spline interpolation procedure through
an example. Figure 4-13(a) shows a synthetic-aperture radar
(SAR) image of a metropolitan area, and part (b) of the figure
shows a magnified version of the central part of the original
image using cubic-spline interpolation. The magnification factor
is 3 along each direction. The interpolation uses
x[n, m] = c[n, m] ∗ ∗(β3[n] β3 [m])
(4.80)
to compute x[n, m] at x = n/3 and y = m/3 for integers n
and m. The values of c[n, m] are determined using the DFT
recipe outlined in Section 4-7.4 in combination with the product
of cubic spline functions given by Eq. (4.62) evaluated at
www.dbooks.org
150
CHAPTER 4 IMAGE INTERPOLATION
(−1, 0, 1):


β3 (−1)  β3 (0)  β3 (−1) β3 (0) β3 (1)
β3 (1)
1


6
1 1 4 1
41 4 1
4 16 4 .
= 6 6 6 6 =
36 1 4 1
1
0
100
200
300
400
(4.81)
6
500
1 2
is interpolated using
3 4
bilinear interpolation. What is the interpolated value at the .
center of the image?
600
Exercise 4-8: The “image”
700
800
Answer: 14 (1 + 2 + 3 + 4) = 2.5
900
1000
0
200
400
600
800
1000
(a) Original SAR image to be magnified
4-9 Comparison of 2-D Interpolation
Methods
0
100
200
300
400
500
599
0
100
200
300
400
500
599
(b) SAR image magnified by 3 using cubic splines
Figure 4-13 The image in (b) is the (200 × 200) central part
of the synthetic-aperture radar (SAR) image in (a) magnified by a
factor of 3 along each direction using cubic-spline interpolation.
In this chapter, we have discussed three image interpolation
methods: The sinc interpolation formula in Section 4-1.1, the
Lanczos interpolation formula in Section 4-1.2, and the Bspline interpolation method in Section 4-8. To compare the
effectiveness of the different methods, we chose the original
clown image shown in Fig. 4-14(a) and then downsampled it
by a factor of 9 by retaining only 1/9 of the original pixels (1/3
along each direction). The locations of the downsampled pixels
and the resulting (67 × 67) downsampled image are shown in
parts (b) and (c) of Fig. 4-14, respectively.
Next, we applied the various interpolation methods listed
earlier to the downsampled clown image in Fig. 4-14(c) so as
to generate an interpolated version of the original clown image.
This is equivalent to magnifying the (67 × 67) downsampled
clown image in Fig. 4-14(c) by a factor of 3 in each dimension. The results, displayed in Fig. 4-15, deserve the following
commentary:
(a) Relative to the original clown image, of the first three interpolated images, the Lanczos with a = 3 is slightly better than
that with a = 2, and both are better than the sinc-interpolated
image.
(b) Among the B-spline images, significant improvement is
realized in image quality as N is increased from N = 0 (nearest
neighbor) to N = 1 (bilinear) and then to N = 3 (cubic). MATLAB code for this example is available on the book website.
4-9 COMPARISON OF 2-D INTERPOLATION METHODS
151
0
0
20
20
40
40
60
60
80
80
100
100
120
120
140
140
160
160
180
180
199
0
40
80
120
160
199
199
0
40
(a) Sinc interpolation
0
0
20
20
40
40
60
60
80
80
100
100
120
120
140
140
160
160
180
180
199
0
40
80
120
160
199
199
0
(b) Lanczos interpolation with a = 2
0
0
20
40
40
60
60
80
80
100
100
120
120
140
140
160
160
180
180
0
40
80
120
160
120
160
199
199
(c) Lanczos interpolation with a = 3
40
80
120
160
199
(e) B-spline linear interpolation
20
199
80
(d) B-spline NN interpolation
199
0
40
80
120
160
199
(f ) B-spline cubic-spline interpolation
Figure 4-15 Comparison of three interpolation methods: (a) sinc interpolation; (b) and (c) Lanczos interpolation with a = 2 and a = 3,
respectively; and (d) to (f) B-spline with N = 0, N = 1, and N = 3, respectively.
www.dbooks.org
152
CHAPTER 4 IMAGE INTERPOLATION
4-10 Examples of Image Interpolation
Applications
0
Example 4-5:
Image Rotation
Recall from Eq. (3.12), that rotating an image f (x, y) by an
angle θ leads to a rotated image g(x, y) given by
g(x, y) = f (x cos θ + y sin θ , y cos θ − x sin θ ).
(4.82)
Sampling g(x, y) at x = n∆ and y = m∆ gives
199
0
(a) Original clown image
199
g(n∆, m∆) =
f (n∆ cos θ + m∆ sin θ , m∆ cos θ − n∆ sin θ ),
(4.83)
which clearly requires interpolation of f (x, y) at the required
points from its given samples f (n∆, m∆). In practice, nearest
neighbor (NN) interpolation is usually sufficient to realize the
necessary interpolation.
Figure 4-16(a) displays a zero-padded clown image, and
part (b) displays the image after rotation by 45◦ using NN
interpolation. The rotated image bears a very good resemblance
to the rotated original. The MATLAB code for this figure is on
the book website.
0
Example 4-6:
Exponential Image Warping
Image warping or morphing entails creating a new image g(x, y)
from the original image f (x, y) where
199
0
(b) Samples to be interpolated
199
(4.84)
for some 1-D transformations Tx (x) and Ty (y). Image shifting
by (xo , yo ) can be implemented using
0
20
Tx (x) = x − xo
40
60
0
g(x, y) = f (Tx (x), Ty (y))
and
Ty (y) = y − yo.
Magnification by a factor of a can be implemented using
20
40
60
(c) Downsampled image
Figure 4-14 The original clown image in (a) was downsampled
to the image in (c) by sampling only 1/9 of the pixels of the
original image.
Tx (x) =
x
a
and
Ty (y) =
y
.
a
More interesting warping of images can be performed using
nonlinear transformations, as demonstrated by the following
illustrations.
4-10 EXAMPLES OF IMAGE INTERPOLATION APPLICATIONS
153
(a) Warped Clown Image
After shifting the clown image so that the origin [n, m] = [0, 0]
is at the center of the image, the image was warped using
Tn (n) = ne−|n|/300 and Tm (m) = me−|m|/300 and NN interpolation. The warped image is shown in Fig. 4-17(a). Repeating the
process with the space constant 300 replaced with 200 leads to
greater warping, as shown in Fig. 4-17(b).
0
50
100
150
Example 4-7:
200
Square-Root and Inverse Image
Warping
250
Another form of image warping is realized by applying a
square-root function of the form
p
p
m |m|
n |n|
and
Tm (m) =
.
Tn (n) =
25
25
300
350
399
0
50
100
150
200
250
300
350
399
(a) Zero-padded clown image
0
Repetition of the steps described in the previous example, but
using the square-root transformation instead, leads to the images
in Fig. 4-18(a). The MATLAB code for this figure is available
on the book website.
Another transformation is the inverse function given by
50
Tn (n) =
100
n|n|
a
Tm (m) =
and
m|m|
.
a
The result of warping the clown image with a = 300 is shown in
Fig. 4-18(b). The MATLAB code for this figure is available on
the book website.
150
200
Concept Question 4-6:
250
300
350
399
Provide four applications of in-
terpolation.
0
50
100
150
200
250
300
350
399
(b) Clown image rotated by 45˚
Figure 4-16 Clown image before and after rotation by 45◦ .
1 2
Exercise 4-9: The “image”
is rotated counter3 4
◦
clockwise 90 . What is the result?
2 4
Answer:
1 3
4 8
is magnified by a
12 16
factor of three. What is the result, using: (a) NN; (b) bilinear
interpolation?
Exercise 4-10: The “image”
www.dbooks.org
154
CHAPTER 4 IMAGE INTERPOLATION
0
0
50
50
100
100
150
150
200
200
250
250
300
300
350
350
399
0
50
100
150
200
250
300
350
399
−|n|/
300
−|m|/
300
(a) Tn(n) = ne
and Tm(m) = me
399
0
50
50
100
100
150
150
200
200
250
250
300
300
350
350
0
50
100
150
200
250
300
350
399
−|n|/
200
−|m|/
200
(b) Tn(n) = ne
and Tm(m) = me
Figure 4-17 Nonlinear image warping with space constants of
(a) 300 and (b) 200.
50
100
150
200
250
300
350
399
(a) Tn(n) = n√|n|/ 25 and Tm(m) = m√|n|/ 25
0
399
0
399
0
50
100
150
200
250
300
350
399
(b) Tn(n) = n|n|/ 300 and Tm(m) = m|m|/ 300
Figure 4-18 Clown image warped with (a) square-root transformation and (b) inverse transformation.
4-10 EXAMPLES OF IMAGE INTERPOLATION APPLICATIONS
Answer:
(a)
(b)


4 4 4 8 8 8
4 4 4 8 8 8


4 4 4 8 8 8
12 12 12 16 16 16


12 12 12 16 16 16
12 12 12 16 16 16

1
2

1

3
6
3
2
4
2
6
12
6
1
2
1
3
6
3
2
4
2
4
8
4
4
8
4
8
16
8

2
4

2

4
8
4
www.dbooks.org
155
156
CHAPTER 4 IMAGE INTERPOLATION
Summary
Concepts
• Interpolation is “connecting the dots” of 1-D samples,
and “filling in the gaps” of 2-D samples.
• Interpolation can be used to rotate and to warp or
“morph” images.
• Upsampling an image can be performed by inserting
rows and columns of zeros in the 2-D DFT of the image.
Care must be taken to preserve conjugate symmetry in
the 2-D DFT.
• Downsampling an image can be performed by deleting
rows and columns in the 2-D DFT of the image. Care
must be taken to preserve conjugate symmetry in the
2-D DFT. Deleting rows and columns performs lowpass
filtering so that the downsampled image is not aliased.
• B-splines are piecewise polynomial functions that can be
used to interpolate samples in 1-D and 2-D.
• For N ≥ 2, computation of the coefficients {c[m]} from
samples {x(m∆)} can be formulated as a deconvolution
problem.
• 2-D interpolation using B-splines is a generalization of
1-D interpolation using B-splines.
Mathematical Formulae
B-splines
βN (t) = rect(t) ∗ · · · ∗ rect(t)
|
{z
}
B-spline 
2
3

2/3 − t + |t| /2
3
β3 (t) = (2 − |t|) /6


0
N+1 times
B-spline
β0 (t) = rect(t) =
B-spline (
β1 (t) =
(
1 − |t|
0
for |t| < 1/2,
for |t| > 1/2
1
0
Important Terms
B-spline
B-spline 1-D interpolation
t
∞
x(t) = ∑ c[m] βN
−m
∆
m=−∞
for |t| ≤ 1,
for |t| ≥ 1
B-spline 
2

3/4 − t
β2 (t) = (3/2 − |t|)2/2


0
for |t| ≤ 1,
for 1 ≤ |t| ≤ 2,
for |t| ≥ 2
Nearest-neighbor 1-D interpolation
x(t) = x(n∆) for |t − n∆| < ∆/2
Linear 1-D interpolation
t
t
+ x((n + 1)∆)
−n
∆
∆
for n∆ ≤ t ≤ (n + 1)∆
x(t) = x(n∆) (n + 1) −
for 0 ≤ |t| ≤ 1/2,
for 1/2 ≤ |t| ≤ 3/2,
for |t| ≥ 3/2
Provide definitions or explain the meaning of the following terms:
downsampling
interpolation
Lanczos function
PROBLEMS
Section 4-3: Upsampling and Interpolation
4.1 Write a MATLAB program that loads the 50 × 50 image in
tinyclown.mat and magnifies it by four using upsampling.
Note that 50 is an even number.
nearest-neighbor
thumbnail image
upsampling
4.2 Write a MATLAB program that loads the 50 × 50 image
in tinyclown.mat, deletes the last row and column to make
it 49 × 49, and magnifies it by four using upsampling. This is
easier than Problem 4.1 since 49 is an odd number.
4.3 Write a MATLAB program that loads the 64 × 64 image
in tinyletters.mat and magnifies it by four using upsam-
PROBLEMS
157
pling. Note that 64 is an even number.
4.4 Write a MATLAB program that loads the 64 × 64 image in
tinyletters.mat, deletes the last row and column to make
it 63 × 63, and magnifies it by four using upsampling. This is
easier than Problem 4.3 since 63 is an odd number.
4.14 Another way to derive the formula for linear interpolation is as follows: The goal is to interpolate the
four points { f (0, 0), f (1, 0), f (0, 1), f (1, 1)} using a formula
f (x, y) = f0 + f1 x + f2 y + f3 xy, where { f0 , f2 , f2 , f3 } are found
from the given points. This extends to
{ f (n, m), f (n + 1, ), f (n, m + 1), f (n + 1, m + 1)}
Section 4-5: Downsampling
4.5 Write a MATLAB program that loads the 200 × 200 image
in clown.mat, antialias lowpass filters it, and demagnifies
it by four using downsampling. (This is how the image in
tinyclown.mat was created.)
4.6 Repeat Problem 4.5, but skip the antialias lowpass filter.
4.7 Write a MATLAB program that loads the 256 × 256 image
in letters.mat, antialias lowpass filters it, and demagnifies
it by four using downsampling. (This is how the image in
tinyletters.mat was created.)
for any integers n, m.
(a) Set up a linear system of equations with unknowns
{ f0 , f1 , f2 , f3 } and knowns
{ f (0, 0), f (1, 0), f (0, 1), f (1, 1)}.
(b) Solve the system to obtain a closed-form expression for
f (x, y) as a function of { f (0, 0), f (0, 1), f (1, 0), f (1, 1)}.
4.15 The image
a
d
g
4.8 Repeat Problem 4.7, but skip the antialias lowpass filter.
4.9 Show that if the sinc interpolation formula is used to
upsample an M × M image f [n, m] to an N × N image g[n, m] by
an integer factor L (so that N = ML), then g[nL, mL] = f [n, m],
so that the values of f [n, m] are preserved after upsampling.
4.11 Write a MATLAB program that loads the 64 × 64 image
in tinyletters.mat and magnifies it by four using nearestneighbor interpolation.
4.12 NN interpolation is used to magnify


a b c
d e f 
g h i
by three. What is the result?
4.13 Linear interpolation is used to magnify


a b c
d e f 
g h i
by two. What is the result?
b
e
h

c
f
i
is rotated 90◦ clockwise. What is the result?
4.16 The image

a
d
g
Section 4-8: 2-D Spline Interpolation
4.10 Write a MATLAB program that loads the 50 × 50 image
in tinyclown.mat and magnifies it by four using nearestneighbor interpolation.

b
e
h

c
f
i
is rotated 45◦ clockwise and magnified by
interpolation. What is the result?
√
2 using linear
4.17 Recall from Eq. (3.12) that rotating an image f (x, y) by
θ to get g(x, y) is implemented by
g(x, y) = f (x′ , y′ ) = f (x cos θ + y sin θ , y cos θ − x sin θ ),
where from Eq. (3.11)
′ x
cos θ
=
− sin θ
y′
sin θ
cos θ
x
.
y
This is point-by-point, and so it is very slow. A faster way to
rotate an image by transforming it first in y and then in x is
as follows: (1) Let h(x, y) = f (x, (y/ cos θ − x tan θ )); (2) then
g(x, y) = h(cos θ + y sin θ , y)).
(a) Show that this computes g(x, y) from f (x, y).
(b) Show that this amounts to the matrix factorization
sin θ
cos θ
1 0 ∗ ∗
=
− sin θ cos θ
∗ ∗ 0 1
www.dbooks.org
158
CHAPTER 4 IMAGE INTERPOLATION
for some elements ∗.
4.18 Write a MATLAB program that loads the 50 × 50 image
in tinyclown.mat and magnifies it by four using cubic
spline interpolation.
4.19 Write a MATLAB program that loads the 50 × 50 image
in tinyletters.mat and magnifies it by four using cubic
spline interpolation.
4.20 This problem is for readers who have some familiarity
with 1-D DSP. Use of 1-D quadratic splines on N interpolation
points {x(n∆)} requires solving
x(n∆) = c[n − 1] β2 (1) + c[n] β2 (0) + c[n + 1] β2 (1)
(Eq. (4.64)) for {c[n]} from {x(n∆)}. In Section 4-7.4 this was
solved using the DFT, which requires (N/2) log2 N multiplications. This problem gives a faster method, requiring only
2N < (N/2) log2 N multiplications.
Let
h[n] = {β2 (−1), β2 (0), β2 (1)} = β2 (1){1, r, 1},
where r = β2 (0)/β2 (1).
(a) Show that h[n] can be written as
h[n] = β2 (1){1, 1/ρ } ∗ {1, ρ }
for some constant ρ . Determine ρ .
(b) Show that h[n] can be implemented by the following two
systems connected in series:
y1 [n] = h1 [n] ∗ x1[n]
= x1 [n] + ρ x1[n − 1]; y2[n]
= h2 [n] ∗ x2[n]
1
= x2 [n + 1] + x2 [n].
ρ
(c) Show that for each of these systems, xi [n] can be computed
recusively and stably from yi [n] using
x1 [n] + ρ x1[n − 1] = y1 [n]; x2 [n] + ρ x2[n + 1] = ρ y2 [n].
The latter system must be run backwards in time n.
(d) Determine a recipe for computing {c[n]} from {x(n∆)}
using these concepts.
4.21 Repeat Problem 4.20 for cubic splines.
Chapter
5
5
Image Enhancement
0
Contents
5-1
5-2
5-3
5-4
5-5
20
40
Overview, 160
Pixel Value Transformation, 160
Unsharp Masking, 163
Histogram Equalization, 167
Edge Detection, 171
Summary of Image Enhancement
Techniques, 176
Problems, 178
60
80
100
120
140
160
180
Objectives
200
0
20
40
60
80
100 120 140 160 180 200
Learn to:
■ Use linear or gamma transformation to alter pixel
values to bring out image features.
■ Use unsharp masking or the Laplacian to sharpen an
image.
■ Use histogram equalization to brighten an image.
■ Use Sobel or Canny edge detection to produce an
edge image of a given image.
www.dbooks.org
This chapter covers various types of image
enhancement, in which the goal is to deliberately
alter the image to brighten it, increase its contrast,
sharpen it, or enhance features such as edges.
Unsharp masking sharpens an image using a
high-pass filter, but this also makes the image
noisier. Histogram equalization nonlinearly alters
pixel values to spread them out more evenly over
the display range of the image. Edge enhancement
produces an edge image of just the edges of the
image, which can be useful in image recognition in
computer vision.
Overview
5-1.1 Linear Transformation of Pixel Values
Image enhancement is an operation that transforms an image
f [n, m] to another image g[n, m] in which features of f [n, m],
such as edges or contrasts between different pixel values, are
emphasized. It is not the same as image restoration, which
includes denoising (removing noise from an image), deblurring
(refocusing an out-of-focus image), and the more general case
of deconvolution (undoing the effect of a PSF on an image). In
these three types of operations, the goal is to recover the true
image f [n, m] from its noisy or blurred version g[n, m]. Image
restoration is covered in Chapter 6.
Image enhancement techniques covered in this chapter include: linear and nonlinear transformations of pixel values for
displaying images more clearly; unsharp masking, a technique
originally developed for sharpening images in film-based photography; histogram equalization for brightening images; and
edge detection for identifying edges in images.
In general, image f [n, m] may have both positive and negative
pixel values. A linear transformation linearly transforms the
individual pixel values from f [n, m] to g[n, m], with 0 displayed
as pure black and gmax displayed as pure white. Functionally,
the linear transformation is given by
f [n, m] − fmin
g[n, m] = gmax
.
(5.1)
fmax − fmin
Usually, g[n, m] is normalized so that gmax = 1.
Without the linear transformation, a display device would
display all negative values of f [n, m] as black and would display
all values larger than gmax as white. In MATLAB, the command imagesc(X),colormap(gray) applies the linear
transformation given by Eq. (5.1), with gmin = 0 and gmax = 1,
prior to displaying an image, thereby ensuring that the full range
of values of array X are displayed properly, including negative
values.
5-1 Pixel-Value Transformation
The image dynamic range Ri of an image f [n, m] is defined
as the range of pixel values contained in the image, extending
between a minimum value fmin and a maximum value fmax :
Ri
fmin
fmax
For a display device, such as a printed page or a computer
monitor, the display dynamic range Rd of the display intensity
g[n, m] is the full range available for displaying an image,
extending from a minimum of zero to a maximum gmax :
5-1.2 Logarithmic Transformation of Pixel
Values
If coherent light is used to illuminate a circular opening, as
depicted by the diagram in Fig. 5-1, the light diffracted by the
opening generates an interference pattern in the image plane,
consisting of a “ring-like” structure. The 1-D image intensity
along any direction in the image plane is given by
I(θ ) = I0 sinc2 (aθ ),
(5.2)
Rd
0
gmax
Ideally, the display device should display an image such that the
information content of the image is conveyed to the user most
optimally. This is accomplished by applying a preprocessing
transformation of pixel values. If Ri extends over a narrow range
of Rd , a transformation can be used to expand Ri to take full
advantage of the available extent of Rd . Conversely, if Ri extends
over several orders of magnitude, displaying the image over the
limited linear range Rd would lead to pixel-value truncation. To
avoid the truncation issue, a nonlinear transformation is needed
so as to convert the dynamic range Ri into a range that is more
compatible with the dynamic range Rd of the display device. We
now explore both types of transformations.
160
Light illumination
y
θ
x
Image plane
Figure 5-1 Image generated by coherent light diffracted by a
circular opening.
5-1 PIXEL-VALUE TRANSFORMATION
161
representation of the normalized intensity in the range between
the first pair of nulls, but it is difficult to examine the plot
quantitatively outside the range between the two nulls.
Figure 5-2(b) provides an alternative format for displaying
the normalized intensity, namely as
A(θ)
1
0.9
0.8
0.7
AdB (θ ) = 10 log10 A(θ ) = 10 log10 [sinc2 (aθ )].
0.6
0.5
The use of the logarithm in the decibel (dB) scale serves to
compress the dynamic range; the range between 1 and 10−3,
for example, gets converted to between 0 and −30 dB. Consequently, very small values of A(θ ) become more prominent,
while values close to 1 get compressed. This is evident in the
plot shown in Fig. 5-2(b), where it is now much easier to “read”
the values of the second and third lobes, compared with doing so
using the linear scale shown in Fig. 5-2(a). We should note that
at angles θ where A(θ ) = 0, the corresponding value of AdB (θ )
is −∞. In plots like the one in Fig. 5-2(b), the lower limit along
the vertical axis is truncated to some finite value, in this case
−35 dB.
The decibel scale also is used in displaying image spectra, an
example of which is shown in Fig. 5-3, as well as in multiple
fields of science and engineering, including:
0.4
0.3
0.2
0.1
0
−40
−20
0
20
40
θ (degrees)
(a) A(θ) in natural units
AdB
0
−5 dB
−10 dB
−15 dB
• Acoustics:
−20 dB
−25 dB
PdB = 20 log10
−30 dB
−35 dB
−40
−20
0
20
40
θ (degrees)
(b) Decibel scale, AdB(θ) = 10 log10 A(θ)
Figure 5-2 Plot of the normalized intensity as a function of
angle θ .
where I0 and a are constants related to the wavelength λ the
diameter of the opening, and the overall imaging geometry.
A plot of the normalized intensity
A(θ ) =
I(θ )
= sinc2 (aθ )
I0
(5.4)
(5.3)
with a = 0.1 is displayed in Fig. 5-2(a) as a function of angle θ ,
with θ expressed in degrees. The peak value of A(θ ) is 1,
and the sinc2 (aθ ) function exhibits sidelobes that decrease in
intensity with increasing value of |θ |. The plot provides a good
pressure
,
20 µ pascals
(5.5)
where 20 µ pascals is the smallest acoustic pressure that can
create an audible sound. On this PdB scale, a whisper is about
30 dB and the sound intensity of a jet plane taking off is about
130 dB, making the latter 100 dB (or, equivalently, 100,000)
times louder than a whisper.
• Richter Scale for Earthquakes:
displacement
DdB = log10
,
1 µm
(5.6)
where “displacement” is defined as the horizontal ground displacement at a location 100 km from the earthquake’s epicenter.
For an earthquake of Richter magnitude 6 the associated ground
motion is 1 m. In contrast, the ground displacement associated
with a Richter magnitude 3 earthquake is only 1 mm.
• Stellar Magnitude (of stars viewed from Earth):
star brightness
.
SdB = −2.512 log10
brightness of Vega
www.dbooks.org
(5.7)
162
CHAPTER 5
IMAGE ENHANCEMENT
pixel values is given by the functional form
ν
g[n, m] = a log10 ( f [n, m] + b),
μ
(5.8)
where constant a is chosen so that the maximum value of g[n, m]
is equal to gmax , and constant b is chosen so that the smallest
value of g[n, m] is zero, which corresponds to setting the smallest
value of ( f [n, m] + b) equal to 1. The logarithmic transformation
of pixel values also is used when displaying image spectra
F(Ω1 , Ω2 ), because the range of values of |F(Ω1 , Ω2 )| usually
is very large.
5-1.3 Gamma Transformation of Pixel Values
(a) Linear spectrum
ν
For most display devices—including cathode-ray tubes, printers, and scanners, the intensity I of the displayed image is related
to the signal voltage V by a power-law relation of the form
I[n, m] = aV b [n, m],
μ
(b) Logarithmic spectrum
Figure 5-3 Spectrum F(µ , ν ) in (a) linear scale and (b) logarithmic scale.
(5.9)
where a and b are constants, and b having a value in the range
0.4 ≤ b ≤ 0.55. Because b is on the order of 0.5, the dynamic
range of the intensity I is much smaller than that of V . Hence,
for a display device with a fixed dynamic range, the displayed
intensity is a “compressed” version of what a display of V would
have looked like. To correct for this compression, the image
can be preprocessed prior to displaying it by applying a gamma
transformation of pixel values given by
f [n, m] − fmin γ
,
(5.10)
g[n, m] = gmax
fmax − fmin
where γ is an application-dependent constant with the same
range as 1/b of Eq. (5.9), namely
1.8 ≤ γ ≤ 2.5.
A first-magnitude star, such as Spica in the constellation Virgo,
has a stellar magnitude of approximately 1. Stars of magnitude
6 are barely visible to the naked eye (depending on viewing
conditions) and are 100 times less bright than a first-magnitude
star. The factor of −2.512 was chosen so that a first-magnitude
star has a brightness equal to 40% of that of the star Vega, and
Vega was chosen as the star with a reference brightness of zero
magnitude.
The dB scale also is used in voltage and power ratios, and in
defining signal-to-noise ratio.
When applied to images, the logarithmic transformation of
One role of the gamma transformation is to correct for the
power-law relationship given by Eq. (5.9), thereby generating an
image display of the true signal f [n, m]. Here, f [n, m] is the true
pixel value, g[n, m] is the output of the preprocessing step, which
makes it the input to the display device. That is, g[n, m] = V [n, m]
and
I[n, m] = aV b [n, m] = a gb [n, m]
b
f [n, m] − fmin γ
.
= a gmax
fmax − fmin
(5.11)
5-2 UNSHARP MASKING
163
If γ of the preprocessor is chosen such that it is approximately
equal to 1/b of the display device, then Eq. (5.11) simplifies to
f [m, n] − fmin
,
(5.12)
I[n, m] = a′
fmax − fmin
where a′ = agbmax .
Figure 5-4(a) is an image of the planet Saturn, displayed
with no preprocessing. By comparison, the image in part (b) of
the figure had been subjected to a preprocessing step using a
gamma transformation with γ = 3 (the value that seemed to best
enhance the image). The cloud bands are much more apparent
in the transformed image than in the original.
0
20
40
60
80
100
Why do we not simply use pixel
values directly as numerical measures of image intensities?
Concept Question 5-1:
Concept Question 5-2:
120
0
Why are logarithmic scales so
useful in so many fields?
120
0
Exercise 5-1: A star of magnitude 6 is barely visible to
the naked eye in a dark sky. How much fainter is a star of
magnitude 6 than Vega, whose magnitude is 0?
20
From Eq. (5.7), 106/2.512 = 244.6, so a sixth
magnitude star is 245 times fainter than Vega.
40
Exercise 5-2: What is another name for gamma transfor-
60
Answer:
20
40
60
80
100
(a) Original Saturn image
mation with γ = 1?
Answer: Linear transformation. Compare Eq. (5.10) and
80
Eq. (5.1).
100
5-2 Unsharp Masking
120
5-2.1 Film Photography Version of Unsharp
Masking
Unsharp masking is a contrast-enhancement procedure commonly used in both photographic and electronic displays. The
procedure acts like a high-pass spatial filter that enhances high
spatial-frequency (fast varying) image components.
◮ Thus, unsharp masking enhances edges and fast-varying
parts of the image, and is a standard tool available in
Adobe R Photoshop R . ◭
0
20
40
60
80
100
120
(b) Result of Gamma transform with γ = 3
Figure 5-4 Image of Saturn (a) before and (b) after gamma
transformation.
The “unsharp” and “masking” parts of the name are associated
with a technique in which a blurred (or unsharp) image is used
to create a corrective “mask” for removing the blurriness from
the image. The audio equivalent of unsharp masking is turning
www.dbooks.org
164
CHAPTER 5
up the treble.
In its original form, unsharp masking was developed for
film photography to deal with blurring caused by the printing
process, which involves the passage of the light image through
a sheet of glass. If the original image is f (x, y) and the blurred
printed version is fblur (x, y), the difference is called fmask (x, y),
fmask (x, y) = f (x, y) − fblur (x, y).
fmask (x, y) = f (x, y) − fblur (x, y)
represents a high-pass filtered version of f (x, y). A highpass
spatial filter emphasizes the presence of edges; hence the name
“mask.”
By photographically adding the mask image to the original
image, we obtain a sharpened image fsh (x, y) in which high
spatial-frequency components of f (x, y) are boosted relative to
low spatial-frequency components:
(5.14)
In digital image processing, high spatial-frequency components can also be boosted by applying the discrete form of the
Laplacian operator to f (x, y).
5-2.2 Laplacian Operator in Continuous Space
In continuous space, the Laplacian g(x, y) of a 2-D image f (x, y)
is defined as
g(x, y) = ∇2 f (x, y) =
∂2 f ∂2 f
+
.
∂ x2 ∂ y2
(5.15)
The spatial frequency response of the Laplacian can be obtained
by computing the 2-D CSFT of g(x, y). Application of property
#5 in Table 2-4 leads to
G(µ , ν ) = −4π 2(µ 2 + ν 2 ) F(µ , ν ).
G(µ , ν ) = HLaplace (µ , ν ) F(µ , ν ).
(5.17)
Hence,
HLaplace (µ , ν ) = −4π 2(µ 2 + ν 2 ).
(5.18a)
Similarly, in polar coordinates
HLaplace (ρ , φ ) = −4πρ 2.
(5.18b)
It is evident from the definitions given by Eqs. (5.18a and b) that
the Laplacian emphasizes high spatial-frequency components
(proportional to ρ 2 ) of the input image f (x, y). It is equally
evident that all frequency components of HLaplace (µ , ν ) and
HLaplace (ρ , φ ) have negative values.
5-2.3 Laplacian in Discrete Space
Derivatives in continuous space are approximated as differences
in discrete space. Hence, in discrete space, the Laplacian g[n, m]
of a 2-D image f [n, m] is defined as
g[n, m] = f [n + 1, m] + f [n − 1, m]
+ f [n, m + 1] + f [n, m − 1] − 4 f [n, m].
(5.19)
This operation is equivalent to the convolution
g[n, m] = f [n, m] ∗ ∗ hLaplace [n, m],
(5.20)
where hLaplace [n, m] is the point-spread-function (PSF) of the
Laplacian operator, and is given by


0 1 0
(5.21)
hLaplace [n, m] = 1 −4 1 .
0 1 0
The DSFT F(Ω1 , Ω2 ) of image f [n, m] was defined by
Eq. (3.73a) as
∞
(5.16a)
F(Ω1 , Ω2 ) =
∞
∑ ∑ f [n, m] e− j(Ω1 n+Ω2 m) ,
(5.22)
n=−∞ m=−∞
Conversion to polar coordinates (ρ , φ ) in the spatial frequency
domain leads to
G(ρ , φ ) = −4π 2 ρ 2 F(ρ , φ ).
In the spatial frequency domain G(µ , ν ) is the product of
the spatial frequency response of the Laplacian operator,
HLaplace (µ , ν ), and the spectrum F(µ , ν ):
(5.13)
Image fmask (x, y) can be formed in a darkroom by adding a
“negative” version of fblur (x, y) to f (x, y). The blurring process
caused by the imperfect printing process is, in effect, a lowpassfiltering process. Hence, fblur (x, y) represents a lowpass-filtered
version of f (x, y), and the “mask” image
fsh (x, y) = f (x, y) + fmask (x, y)
= f (x, y) + [ f (x, y) − fblur (x, y)].
IMAGE ENHANCEMENT
(5.16b)
and the properties of the DSFT are direct 2-D generalizations of
the properties of the 1-D DTFT given in Table 2-7. Of particular
interest is the time-shift property in Table 2-7, which when
5-2 UNSHARP MASKING
165
fact, in close proximity to the origin, such that |Ω1 |, |Ω2 | ≪ 1,
expansion of the cosine functions in Eq. (5.26) in a Taylor series
gives
|HLaplace(Ω1,0)|
8
Eq. (5.26) [Exact]
Eq. (5.27) [Approx.]
6
Ω21
+ ··· ,
2
Ω2
cos Ω2 ≈ 1 − 2 + · · · ,
2
cos Ω1 ≈ 1 −
4
2
0
−π
−π/2
0
π/2
π
Ω1
which when used in Eq. (5.26), the latter simplifies to
HLaplace (Ω1 , Ω2 ) ≈ −Ω21 − Ω22 = −R 2 ,
Figure 5-5 |HLaplace (Ω1 , Ω2 )| versus Ω1 at Ω2 = 0 (in red) and
the approximation for small values of Ω1 and Ω2 blue.
extended to 2-D, leads to
f [n − n0, m − m0]
F(Ω1 , Ω2 ) e− j(n0 Ω1 +m0 Ω2 ) .
(5.23)
Application of Eq. (5.23) to Eq. (5.19) leads to
G(Ω1 , Ω2 ) = F(Ω1 , Ω2 ) e− jΩ1 + F(Ω1 , Ω2 ) e jΩ1
+ F(Ω1 , Ω2 ) e− jΩ2 + F(Ω1 , Ω2 ) e jΩ2
− 4F(Ω1, Ω2 )
= F(Ω1 , Ω2 ) [(e− jΩ1 + e jΩ1 ) + (e− jΩ2 + e jΩ2 ) − 4]
= F(Ω1 , Ω2 ) [2 cos Ω1 + 2 cosΩ2 − 4].
(5.24)
The spectrum G(Ω1 , Ω2 ) is the product of the spectrum of
the original image, F(Ω1 , Ω2 ) and the Laplacian’s spatial frequency response HLaplace (Ω1 , Ω2 ):
G(Ω1 , Ω2 ) = F(Ω1 , Ω2 ) HLaplace (Ω1 , Ω2 ).
(5.25)
(5.27)
This frequency dependence of the discrete-space Laplacian is
analogous to the response given by Eq. (5.18b) for the frequency
response of the continuous-space Laplacian; both have negative
signs and both vary as the square of the spatial frequency
(ρ and R).
The blue plot in Fig. 5-5 represents the approximate expression for |HLaplace (Ω1 , Ω2 )| given by Eq. (5.27). It confirms that
the approximation is valid not only for |Ω1 |, |Ω2 | ≪ 1, but also
up to |Ω1 |, |Ω2 | ≈ 1.
In Fig. 5-5, the plots for the exact and approximate expressions of |HLaplace (Ω1 , 0)| are displayed over the range
−π < Ω1 < π . They are in close agreement over approximately
the central one-third of the spectral range, and they deviate
significantly as |Ω1 | exceeds 1 (or |Ω2 | exceeds 1), or more
generally, as the radial frequency R exceeds 1. In most images,
the bulk of the image “energy” is contained within this central
region.
This last statement deserves further elaboration. To do so, we
refer the reader to Eq. (2.64), which relates the frequency Ω0 in
discrete time to the frequency f0 in continuous time, namely
Equating Eqs. (5.24) and (5.25) leads to
HLaplace (Ω1 , Ω2 ) = 2[cosΩ1 + cosΩ2 − 2].
Ω0 = 2π f0 ∆,
(5.26)
In Fig. 5-5, we display a plot of |HLaplace (Ω1 , Ω2 )| as a function
of Ω1 , with (for simplicity) Ω2 set equal to zero. The plot
would look the same as a function of the radial discrete-space
frequency
R = [Ω21 + Ω22 ]1/2 .
The frequency response |HLaplace (Ω1 , Ω2 )| exhibits a shape similar to that of a high-frequency filter, with |HLaplace (Ω1 , Ω2 )| = 0
at the origin and then increasing rapidly with increasing R. In
for R ≪ 1.
(5.28)
where ∆ is the sampling interval in seconds. Extending the
relationship to 2-D provides the connections
Ω1 = 2 π µ ∆
(5.29a)
Ω2 = 2πν ∆,
(5.29b)
and
where now Ω1 and Ω2 are continuous spatial frequencies associated with the discrete image f [n, m], µ and ν are spatial
frequencies (in cycles/m) associated with the continuous image
f (x, y), and ∆ is the sampling length in meters/sample. Since 2π
www.dbooks.org
166
CHAPTER 5
represents the amount in radians per a single complete cycle, the
units of Ω1 and Ω2 are radians/sample.
If an image f (x, y) is sampled at the Nyquist rate such
that ∆ = 1/2B, where B is the maximum spatial frequency of
the image spectrum F(µ , ν ), then the maximum discrete-space
frequency is
Ω1 (max) = 2π × B ×
1
= π,
2B
(5.30)
and the same conclusion applies to Ω2 and R. This is why the
plots in Fig. 5-5 extend over the range −π ≤ Ω1 ≤ π .
Most images are sampled at rates greater than the Nyquist
rate. If an image is sampled at three times the Nyquist rate
(i.e., at ∆ = 1/6B), then Ω1 (max) = π /3 ≈ 1. In such a case,
the approximation given by Eq. (5.27) becomes valid over the
complete relevant ranges of Ω1 , Ω2 , and R.
5-2.4 Image Sharpening
An image f [n, m] can be sharpened into image gsharp [n, m] by
subtracting g[n, m] of Eq. (5.20) from the original image:
gsharp [n, m] = f [n, m] − f [n, m] ∗ ∗ hLaplace [n, m]
= f [n, m] ∗ ∗ hsharp [n, m],
(5.31)
where hsharp [n, m] is an image sharpening filter with PSF
hsharp [n, m] = δ [n] δ [m] − hLaplace [n, m].
(5.32)
This operation is analogous to Eq. (5.14) for film photography,
except that in the present case we used a minus sign (rather than
a plus sign) in the first step of Eq. (5.31) because
F(Ω1 , Ω2 ) HLaplace (Ω1 , Ω2 ),
(5.33)
and HLaplace (Ω1 , Ω2 ) is always negative.
Use of Eq. (5.21) in Eq. (5.32) leads to


0 −1 0
(5.34)
hsharp [n, m] = −1 5 −1 .
0 −1 0
f [n, m] ∗ ∗ hLaplace [n, m]
Since hsharp [n, m] is only 3 × 3, it is faster to compute the 2-D
convolution given by Eq. (5.31) in the spatial [n, m] domain than
by multiplying zero-padded 2-D DFTs.
In a later part of this section, we will compare sharpened
images to their original versions, but we should note that:
IMAGE ENHANCEMENT
◮ A common detractor of all image sharpening algorithms
is that because they emphasize high spatial frequencies,
they also tend to emphasize high spatial-frequency noise.
So in general, sharpened images tend to be noisy. ◭
5-2.5 Valid Convolution
When convolving an image f [n, m] with a filter characterized
by a PSF h[n, m], it is important that edge effects are dealt with
appropriately. If the image size is (M × M) and the filter size is
(L × L), the convolved image
y[n, m] = f [n, m] ∗ ∗ h[n, m]
(5.35)
is (N × N), with N = M + L − 1. However, some parts of y[n, m]
are not “valid” because they include pixels that are a result
of convolving h[n, m] with pixels outside of the boundaries of
f [n, m]. The “valid” part of y[n, m], which we call yvalid [n, m], is
given by
yvalid [n, m] = { y[n, m], 0 ≤ n, m ≤ Nvalid },
(5.36)
where Nvalid = M − L + 1. To illustrate with an example, let us
consider the image given by


4 8 12
f [n, m] = 16 20 24 ,
(5.37a)
28 32 36
and let us assume that we wish to perform local averaging by
sliding a 2 × 2 window across the image, both horizontally and
vertically. Such a filter has a PSF given by
1 1 1
.
(5.37b)
h[n, m] =
4 1 1
Upon performing the convolution given by Eq. (5.35) onto the
arrays given in Eqs. (5.37a and b), we obtain


1 3 5 3
 5 12 16 9 
y[n, m] = 
.
(5.38)
11 24 28 15
7 15 17 9
The border rows and columns are not the average values of
4 neighboring pixels, but of only 1 neighboring pixel and 3
zeros or 2 neighboring pixels and 2 zeros. These are invalid
entries. The more realistic valid output is yvalid [n, m], obtained
via Eq. (5.33) or equivalently, by removing the top and bottom
5-3 HISTOGRAM EQUALIZATION
167
rows and the columns at the far left and far right. Either approach
leads to
12 16
yvalid [n, m] =
.
(5.39)
24 28
Since f [n, m] is M × M = 3 × 3 and h[n, m] is L × L = 2 × 2,
yvalid [n, m] is Nvalid × Nvalid with
Nvalid = M − L + 1 = 3 − 2 + 1 = 2.
We now use yvalid [n, m] as the filtered image in the following
two examples:
(1) Image of an electronic circuit, before and after applying
the sharpening algorithm: Fig. 5-6.
(2) Image of US coins, before and after sharpening: Fig. 5-7.
Concept Question 5-3: Why does image sharpening not
work well on noisy images?
Concept Question 5-4: Why is the Laplacian a common
choice for a high-pass filter?
(a) Original image
Exercise 5-3: Show that the valid convolution of a Laplacian with a constant image f [n, m] = c is zero.
Answer: The valid convolution (defined in Section 5-2.5)
is the usual convolution with edge effects omitted.


0 1 0
f [n, m] ∗ ∗ hLaplace [n, m] = c ∑ ∑ 1 −4 1 = 0.
n=−1 m=−1 0
1 0
1
1
5-3 Histogram Equalization
Consider the two clown images shown in parts (a) and (b) of
Fig. 5-8. The first one, labeled “original image,” is rather dark,
making it difficult to discern some of the features in the clown’s
face. In contrast, the image labeled “histogram-equalized” exhibits a broader range of intensities, thereby allowing the viewer
to see details that are difficult to see in the original image. What
is a histogram-equalized image? That is the topic of the present
section.
Given an image f [n, m] with pixel values that extend over the
input dynamic range Ri , with
Ri = { fmin ≤ f [n, m] ≤ fmax },
(b) Sharpened image
Figure 5-6 Image of electronic circuit before and after application of the sharpening filter.
(5.40a)
www.dbooks.org
168
CHAPTER 5
IMAGE ENHANCEMENT
the objective of histogram equalization is to convert the pixel
values f [n, m] into a new set g[n, m] so that they become more
evenly distributed across the dynamic range of the display
device Rd , with
Rd = { 0 ≤ g[n, m] ≤ gmax }.
(a) Original image
(5.40b)
In Section 5-1, the conversion from f [n, m] to g[n, m] was
accomplished by “stretching” Ri to fit Rd , but now we explore
a different approach that relies on converting the histogram
of f [n, m] into a new histogram associated with the converted
image g[n, m]. The histogram of an image (or signal) is simply
a bar graph of the number of times that each pixel value occurs
in the image. The histograms associated with the original and
histogram-equalized images shown in Figs. 5-8(a) and (b) are
displayed in Figs. 5-8(c) and (d). In both histograms, the horizontal axis represents pixel value and the vertical axis represents
the number of times that a specific pixel value occurs in the
image. For image f [n, m], its continuous original f (x, y) had
been sampled to 200 × 200 discrete locations [n, m], and its nonnegative values had been quantized using 8-bit precision, which
means that the pixel value f0 can take on any integer value
between 0 and 255:
f [n, m] ∈ { 0, 1, 2, . . ., 255 },
for { 0 ≤ n, m ≤ 199 }.
The horizontal axis in a histogram of f [n, m] represents the pixel
value f0 , which may range between 0 and 255, and the vertical
axis is p f [ f0 ], which is the number of times that pixel value
f0 occurs in the image. We refer to p f [ f0 ] as the distribution
function, or histogram, of f [n, m].
A related quantity is the cumulative distribution function
(CDF), Pf [ f0 ], which is the cumulative sum of p f [ f0 ]:
f0
Pf [ f0 ] = ∑ p f [ f ′ ].
(5.41)
f ′ =0
(b) Sharpened image
Figure 5-7 Coins image, before and after sharpening.
The CDF Pf [ f0 ] is a non-decreasing function that jumps upward
in value at the values of f0 for which p f [ f0 ] 6= 0. Figure 5-9(a)
displays the histogram of the original clown image, p f [ f0 ], and
a plot of its associated CDF, Pf [ f0 ]. We observe that Pf [ f0 ]
increases rapidly with f0 between f0 = 0 and f0 = 100 and at
a lower rate at higher values of f0 . Thus, the majority of the
image pixels are concentrated in the range corresponding to low
image intensity, thereby giving the image its dark appearance.
The pixel-value transformation converts each pixel value
f [n, m] into a new pixel value g[n, m], based on the CDF Pf [ f0 ],
5-3 HISTOGRAM EQUALIZATION
169
0
200
0
0
200
200
0
(a) Original image f [n,m]
200
(b) Histogram-equalized g[n,m]
pf ( f0)
pg(g0)
0
255
f0
0
(c) Histogram pf [ f0] of original image
255
g0
(d) Histogram pg[g0] of g[n,m]
Figure 5-8 Clown image, before and after application of histogram equalization, and associated histograms. Histogram pg (g0 ) was
generated by nonlinearly transforming (redistributing) p f ( f0 ), but the total number of pixels at each new value (the g0 axis is a nonlinear
transformation of the f0 axis) remains the same.
evaluated at f0 = f [n, m]. That is,
g[n, m] = Pf [ f0 ] f = f [n,m] .
0
(5.42)
Such a transformation leads to a histogram pg [g0 ] that is more
uniformly spread out over the range 0 to 255 than the histogram
of the original image, p f [ f0 ]. The associated CDF, Pg [g0 ],
approximates a straight line that starts at coordinates (0, 0) and
concludes at (255, M 2 ), where M 2 is the total number of pixels.
These attributes are evident in Fig. 5-10 for the histogramequalized clown image.
www.dbooks.org
170
CHAPTER 5
pf ( f0)
IMAGE ENHANCEMENT
Pf [ f0]
M2
0
50
100
150
200
255
0
f0
(a) Histogram pf [ f0] of f [n,m]
50
100
150
200
255
200
255
f0
(b) CDF Pf [ f0]
Pg[g0]
M2
Pf [ f0]
M2
0
0
50
100
150
200
255
f0
(b) CDF Pf [ f0]
Figure 5-9 Histogram and associated CDF of original clown
image f [n, m]. Image size is (M × M).
50
100
150
g0
(b) CDF Pg[g0]
Figure 5-10 CDF for an (M × M) image before and after
application of the histogram-equalization algorithm given by
Eq. (5.42).
Answer: The histogram of the image is: 0.2 occurring
Concept Question 5-5:
What is the purpose for using
histogram equalization?
Concept Question 5-6: Why is histogram equalization
often described as a nonlinear warping transformation of
pixel values?
Exercise 5-4: Perform
histogram
equalization on the
(2 × 2) “image”
0.3 0.2
.
0.2 0.3
twice and 0.3 occurring twice. The CDF is


for 0 ≤ f0 < 0.2,
0
Pf ( f0 ) = 2
for 0.2 ≤ f0 < 0.3,

4
for 0.3 ≤ f0 < 1.
Pixel value 0.2 is mapped to Pf (0.2) = 2 and pixel value
0.3 is mapped
to Pf (0.3) = 4. The histogram-equalized
4 2
, which has a wider range of values than
“image” is
2 4
the original “image.”
5-4 EDGE DETECTION
171
5-4 Edge Detection
x[n]
An edge in an image is a sharp boundary between two different
regions of an image. Here “sharp” means a width of at most a
few pixels, and a “boundary” means that significant differences
exist in the pixel values between the two sides of the edge.
“Significant” is not clearly defined; its definition depends on
the characteristics of the image and the reason why edges are
of interest. This is a nebulous definition, but there is no uniform
definition for an edge.
The goal of edge detection is to determine the locations [n, m]
of edges in an image f [n, m]. Edge detection is used to segment
an image into different regions, or to determine the boundaries
of a region of interest. For example, a medical image may
consist of different human organs. Interpretation of the image
is easier if (say) the region of the image corresponding to the
pancreas is identified separately from the rest of the image.
Identification of a face is easier if the eyes in an image of the
face are identified as a region separate from the rest of the image.
Ideally, an edge is a contour that encloses a region of the image
whose values differ significantly from the values around it. Edge
detection also is important in computer vision.
6
4
2
0
0
5
10
15
20
25
30
n
(a) x[n]
y[n]
4
2
n
0
−2
0
5
10
15
20
25
30
20
25
30
(b) d[n]
z[n]
1
0.5
0
0
5
10
15
n
(c) z[n] with ∆ = 1
5-4.1 1-D Edge Detection
We start by examining a simple 1-D edge-detection method,
as it forms the basis for a commonly used 2-D edge-detection
algorithm.
An obvious approach to detecting the locations of sharp
changes in a 1-D signal x(t) is to compute its derivative
x′ (t) = dx/dt. Rapid changes of x(t) with t generate derivatives
with large magnitudes, and slow changes generate derivatives
with small magnitudes. The times t0 at which |x′ (t0 )| is large
represent potential edges of x(t). The threshold for “large” has
to be defined in the context of the signal x(t) itself.
For a 1-D discrete-time signal x[n], the discrete-time counterpart to the derivative is the difference operator
d[n] = x[n + 1] − x[n].
(5.43)
The difference d[n] is large when x[n] changes rapidly with n,
making it possible to easily pinpoint the time n0 of an edge. As
simple as it is, computing the difference d[n] and thresholding
|d[n]| is a very effective method for detecting 1-D edges. If the
threshold is set at a value ∆, the edge-detection algorithm can be
cast as
(
1
for |d[n]| > ∆,
z[n] =
(5.44)
0
for |d[n]| < ∆.
Figure 5-11 Edge detection by thresholding absolute values
of differences: (a) original signal x[n], (b) d[n] = x[n + 1] − x[n],
(c) z[n] with ∆ = 1.
The times ni at which z[ni ] = 1 denote the edges of x[n].
Specification of the threshold level ∆ depends on the character
of x[n]. In practice, for a particular class of signals, the algorithm
is tested for several values of ∆ so as to determine the value that
provides the best results for the intended application.
For the signal x[n] displayed in Fig. 5-11(a), the difference
operator d[n] was computed using Eq. (5.44) and then plotted in
Fig. 5-11(b). It is evident that |d[n]| exhibits significant values
at n = 5, 14, and 21. Setting ∆ = 1 would detect all three edges,
as shown in part (c) of the figure, but had we chosen ∆ to be 2,
for example, only the edge at n = 15 would have been detected.
The choice depends on the intended application.
5-4.2 2-D Edge Detection
The 1-D edge detection method can be extended to edge detection in 2-D images. Let us define a vertical edge (VE) as a
vertical line at n = n0 , extending from m = m1 to m = m2 , as
www.dbooks.org
172
CHAPTER 5
IMAGE ENHANCEMENT
0
Pixel [n,m]
(n0,m1)
−1 0
−2 0
−1 0
VE
@ n0
1
2
1
(n0,m2)
n
m
0
m
n0
n
Figure 5-12 Vertical edge VE at n = n0 extends from m = m1
to m = m2 , and its length is (m2 − m1 + 1).
−1 −2 −1
0 0 0
1 2 1
shown in Fig. 5-12. That is,
VE = { [n, m]: n = n0 ; m1 ≤ m ≤ m2 }.
(5.45)
The total length of VE is (m2 − m1 + 1).
One way to detect a vertical edge is to apply the difference
operator given by Eq. (5.43) to each row of the image. In 2-D,
the difference operator for row m is given by
d[n, m] = f [n + 1, m] − f [n, m].
(a) dH[n,m]
(5.46)
If d[n, m] satisfies a specified threshold for a group of continuous
pixels (all at n = n0 ) extending between m = m1 and m2 , then we
call the group a vertical edge. In real images, we may encounter
situations where d[n, m] may exhibit a large magnitude, but it is
associated with a local variation in tone, not an edge. A vertical
edge at n = n0 requires that not only d[n0 , m] at row m be large,
but also that d[n0 , m + 1] at the row above m and d[n0 , m − 1]
at the row below row m be large as well. All three differences
should be large and of the same polarity in order for the three
pixels to qualify as a vertical edge.
This requirement suggests that a vertical edge detector should
not only compute horizontal differences, but also vertical sums
of the differences. The magnitude of a vertical sum becomes
an indicator of the presence of a true vertical edge. A relatively simple edge operator is illustrated in Fig. 5-13 for
both a horizontal-direction vertical-edge detector dH [n, m]
and a vertical-direction horizontal-edge detector dV [n, m]. Each
n
m
(b) dV[n,m]
Figure 5-13 Point spread functions dH [n, m] (in red) and
dV [n, m] (in blue), displayed in center-of-image format.
detector consists of a 3 × 3 window centered at the pixel of
interest. Detector dH [n, m] computes the difference between the
values of pixel [n + 1, m] and pixel [n − 1, m], whose positions
are to the left and right of pixel [n, m], respectively. Similar
differences are performed for the row above and the row below
row m. Then, the three differences are added up together, with
the middle difference assigned twice the weight of the two other
differences. The net result is
dH [n, m] = f [n + 1, m + 1] − f [n − 1, m + 1]
+ 2 f [n + 1, m] − 2 f [n − 1, m]
+ f [n + 1, m − 1] − f [n − 1, m − 1].
(5.47)
The coefficients of the six terms of Eq. (5.47) are the nonzero
weights shown in Fig. 5-13(a).
5-4 EDGE DETECTION
173
Computing dH [n, m] for every pixel is equivalent to validly
convolving (see Section 5-2.5) image f [n, m] with the window’s
point spread function hH [n, m] along the horizontal direction.
That is,
dH [n, m] = f [n, m] ∗ ∗ hH [n, m],
with

where, again, ∆ is a prescribed gradient threshold. In the image,
pixels for which z[n, m] = 1 are shown in white, and those with
z[n, m] = 0 are shown in black. Usually, the value of ∆ is selected
empirically by examining a histogram of g[n, m] or through
repeated trials.
(5.48)
5-4.3 Sobel Edge Detector Examples

−1 0 1
hH [n, m] = −2 0 2 .
−1 0 1
(5.49)
dV [n, m] = f [n, m] ∗ ∗ hV [n, m],
(5.50)
To compute dH [n, m] for all pixels [n, m] in the image, it is
necessary to add an extra row above of and identical with the
top row, and a similar add-on is needed at the bottom end of
the image. The decision as to whether or not a given pixel is
part of a vertical edge is made by comparing the magnitude of
dH [n, m] with a predefined gradient threshold ∆ whose value is
selected heuristically (based on practical experience for the class
of images under consideration).
Horizontal edges can be detected by a vertical-direction edge
detector dV [n, m] given by
where hV [n, m] is the point spread function (PSF) for a pixel
(n, m). By exchanging the roles of the rows and column in
Eq. (5.49), we have


−1 −2 −1
0
0 .
hV [n, m] =  0
(5.51)
1
2
1
Of course, most edges are neither purely horizontal nor purely
vertical, so an edge at an angle different from 0◦ or 90◦ (with
0◦ denoting the horizontal dimension of the image) should
have edge components along both the horizontal and vertical
directions. Hence, the following edge-detection gradient is
often used:
q
g[n, m] = dH2 [n, m] + dV2 [n, m] .
(5.52)
For each pixel [n, m], we define the edge indicator z[n, m] as
(
1
if g[n, m] > ∆,
z[n, m] =
(5.53)
0
if g[n, m] < ∆,
The gradient algorithm given by Eq. (5.53) is known as the
Sobel edge detector, named after Irwin Sobel, who developed
it in 1968, when computer-based image processing was in its
infancy and only simple algorithms could be used. Application
of the Sobel edge detector to the letters image in part (a) of
Fig. 5-14 leads to the image in part (b). Through repeated
applications using different values of ∆, it was determined that
∆ = 200 provided an image with clear edges, including diagonal
and curved edges. The value specified for ∆ depends in part on
the values assigned to black and white tones in the image.
The Sobel edge detector does not always capture all of the
major edges contained in an image. When applied to the clown
image of Fig. 5-15(a), the edge detector identified some parts of
continuous edges, but failed to identify others, which suggests
the need for a detector that can track edges and complete edge
contours as needed. Such a capability is provided by the Canny
edge detector, the subject of the next subsection.
5-4.4 Canny Edge Detector
The Canny edge detector is a commonly used algorithm that
extends the capabilities of the Sobel detector by applying
preprocessing and postprocessing steps. The preprocessing step
involves the use of a 2-D Gaussian PSF to reduce image noise
and to filter out isolated image features that are not edges.
After computing the Sobel operator given by Eq. (5.52), the
Canny algorithm performs an edge thinning step, separating
detected edges into different candidate categories, and then
applies certain criteria to decide whether or not the candidate
edges should be connected together. The five-step process of the
Canny detection algorithm are:
Step 1: Image f [n, m] is blurred (filtered) by convolving it with
a truncated Gaussian point spread function. An example of a
practical function that can perform the desired operation is the
www.dbooks.org
174
CHAPTER 5
IMAGE ENHANCEMENT
0
0
20
50
40
60
100
80
150
100
120
200
140
250
0
50
m
100
150
200
250
n
160
180
(a) Letters image
200
0
0
20
40
m
60
80
100 120 140 160 180 200
n
(a) Clown image
50
0
20
100
40
150
60
80
200
100
250
0
m
50
100
150
200
250
n
120
140
(b) Edge-detected image
160
Figure 5-14 Application of the Sobel edge detector to the
image in (a) with ∆ = 200 led to the image in (b).
180
200
0
m
5 × 5 PSF given by

2 4 5 4 2
4 9 12 9 4
1 


hG [n, m] =
5 12 15 12 5 .
159 4 9 12 9 4
2 4 5 4 2

(5.54)
The standard deviation of the truncated Gaussian function is 1.4.
Application of hG [n, m] to image f [n, m] generates a filtered
image f1 [n, m] given by
f1 [n, m] = hG [n, m] ∗ ∗ f [n, m].
(5.55)
20
40
60
80
100 120 140 160 180 200
n
(b) Sobel edge-detected image
Figure 5-15 Application of the Sobel edge detector to the
image in (a) captures some of the edges in the image, but also
misses others.
Step 2: For image f1 [n, m], compute the horizontal and vertical
edge detectors given by Eqs. (5.48) and (5.50).
5-4 EDGE DETECTION
175
Step 3: Compute the gradient magnitude and orientation:
q
g[n, m] = dH2 [n, m] + dV2 [n, m]
(5.56a)
and
θ [n, m] = tan−1
dV [n, m]
.
dH [n, m]
(5.56b)
0
20
40
60
80
For a vertical edge, dV [n, m] = 0, and therefore θ [n, m] = 0.
Similarly, for a horizontal edge, dH [n, m] = 0 and θ = 90◦ .
100
Step 4: At each pixel [n, m], round θ [n, m] to the nearest of
{ 0◦ , 45◦ , 90◦ , 135◦ }. Next, determine whether to keep the value
of g[n, m] of pixel [n, m] as is or to replace it with zero. The
decision logic is as follows:
(a) For a pixel [n, m] with θ [n, m] = 0◦ , compare the value of
g[n, m] to the values of g[n + 1, m] and g[n − 1, m], corresponding to the pixels at the immediate right and left of pixel [n, m]. If
g[n, m] is the largest of the three gradients, keep its value as is;
otherwise, set it to zero.
(b) For a pixel [n, m] with θ = 45◦ , compare the value of
g[n, m] to the values of g[n − 1, m + 1] and g[n + 1, m − 1],
corresponding to the pixel neighbors along the 45◦ diagonal. If
g[n, m] is the largest of the three gradients, keep its value as is;
otherwise, set it to zero.
(c) For a pixel [n, m] with θ = 90◦ , compare the value of
g[n, m] to the values of g[n, m − 1] and g[n, m + 1], corresponding to pixels immediately above and below pixel [n, m].
If g[n, m] is the largest of the three gradients, keep its value as is;
otherwise, set it to zero.
(d) For a pixel [n, m] with θ = 135◦, compare the value of
g[n, m] to the values of g[n − 1, m − 1] and g[n + 1, m + 1]. If
g[n, m] is the largest of the three gradients, keep its value as is;
otherwise, set it to zero.
The foregoing operation is called edge thinning, as it avoids
making an edge wider than necessary in order to indicate its
presence.
140
Step 5: Replace the edge indicator algorithm given by Eq. (5.53)
with a double-threshold algorithm given by


if g[n, m] > ∆2 ,
2
z[n, m] = 1
(5.57)
if ∆1 < g[n, m] < ∆2 ,

0
if g[n, m] < ∆1 .
The edge indicator z[n, m] may assume one of three values,
indicating the presence of an edge (z[n, m] = 2), the possible
presence of an edge (z[n, m] = 1), and the absence of an edge
(z[n, m] = 0). The middle category requires resolution into one
120
160
180
200
0
20
40
60
80
100 120 140 160 180 200
(a) Clown image
0
20
40
60
80
100
120
140
160
180
200
0
20
40
60
80
100 120 140 160 180 200
(b) Canny edge-detected image
Figure 5-16 The Canny edge detector provides better edgedetection performance than the Sobel detector in Fig. 5-15.
of the other two categories. This is accomplished by converting
pixel [n, m] with z[n, m] = 1 into an edge if any one of its nearest
8 neighbors is a confirmed edge. That is, pixel [n, m] is an edge
location only if it adjoins another edge location.
The values assigned to thresholds ∆1 and ∆2 are selected
through multiple trials. For example, the clown edge-image
shown in Fig. 5-16 was obtained by applying the Canny algo-
www.dbooks.org
176
rithm to the clown image with ∆1 = 0.05 and ∆2 = 0.125. This
particular combination provides an edge-image that successfully
captures the contours that segment the clown image.
◮ Edge detection can be implemented in MATLAB’s Image
Processing Toolbox using the commands
E=edge(X,’sobel’,T1) for Sobel and
E=edge(X,’canny’,T1,T2) for Canny.
The image is stored in array X, the edge image is stored
in array E and T1 and T2 are the thresholds. MATLAB
assigns default values to the thresholds, computed from the
image, if they are not specified. ◭
Concept Question 5-7: Why does edge detection not
work well on noisy images?
Concept Question 5-8: In edge detection, why do we not
simply take differences, as in Eq. (5.46), instead of taking
differences in one direction and sums in the other, as in
Eq. (5.47)?
Exercise 5-5: Show that edge detection applied to a constant image f [n, m] = c gives no edges.
Answer: The valid convolution (defined in Section 5-2.5)
is the usual convolution with edge effects omitted. The
two valid convolutions (hH [n, m] and hV [n, m] are defined
in Eq. (5.49) and Eq. (5.51))


−1 0 1
1
1
f [n, m] ∗ ∗ hH [n, m] = c ∑ ∑ −2 0 2 = 0,
n=−1 m=−1 −1 0 1


−1 −1 1
1
1
0 0 = 0,
f [n, m] ∗ ∗ hV [n, m] = c ∑ ∑  0
n=−1 m=−1
1
2 1
Hence, the edge-direction gradient g[n, m] defined in
Eq. (5.52) is zero.
CHAPTER 5
IMAGE ENHANCEMENT
5-5 Summary of Image Enhancement
Techniques
• To increase contrast in an image by altering the range
of pixel values, use a gamma transformation. Simply try
different values of γ and choose the one that gives the best
visual result.
• To compress the range of values; e.g., in a spectrum, use a
log transformation.
• To sharpen an image, use unsharp masking, bearing in mind
that this also makes the image noisier.
• To make an image brighter, use histogram equalization.
• To detect edges, or create an image of edges, use Canny
edge detection (if available), otherwise use Sobel edge
detection.
5-5 SUMMARY OF IMAGE ENHANCEMENT TECHNIQUES
177
Summary
Concepts
• Image enhancement transforms a given image into another image in which image features such as edges or
contrast has been enhanced to make them more apparent.
• Linear, logarithmic, and gamma transformations alter the
range of pixel values so that they fit the range of the
display.
• Unsharp masking and Laplacians sharpen an image, but
increase noise.
• Histogram equalization nonlinearly alters pixel values to
brighten images.
• Edge detection produces an image consisting entirely of
edges of the image.
Mathematical Formulae
Linear transformation
f [n, m] − fmin
g[n, m] = gmax
fmax − fmin
Laplacian
g[n, m] = f [n + 1, m] + f [n − 1, m] + f [n, m + 1]
+ f [n, m − 1] − 4 f [n, m]
Logarithmic transformation
g[n, m] = a log10 ( f [n, m] + b)
Cumulative distribution
f0
Pf [ f0 ] = ∑ p f [ f ′ ]
Gamma transformation
f [n, m] − fmin γ
g[n, m] = gmax
fmax − fmin
f ′ =1
Histogram equalization
g[n, m] = Pf [ f0 ] f0 = f [n,m]
Unsharp masking
g(x, y) = f (x, y) + f (x, y) − fblur (x, y)
{z
}
|
Horizontal and verticaledge detectors

−1 0 1
dH [n, m] = f [n, m] ∗ ∗ −2 0 2
−1 0 1


−1 −2 −1
0
0
dV [n, m] = f [n, m] ∗ ∗  0
1
2
1
Sobel edge detector
(
p
1
if dh [n, m]2 + dv [n, m]2 > ∆,
p
z[n, m] =
0
if dh [n, m]2 + dv [n, m]2 < ∆
f mask (x,y)
Unsharp masking
g[n, m] = f [n, m] − f [n, m] ∗ ∗ hLaplace [n, m]
Laplacian
g(x, y) = ∇2 f (x, y) =
Important Terms
Canny edge detector
gamma transformation
∂2 f ∂2 f
+
∂ x2 ∂ y2
Provide definitions or explain the meaning of the following terms:
histogram equalization
Laplacian
logarithmic transformation
Sobel edge detector
www.dbooks.org
unsharp masking
178
CHAPTER 5
PROBLEMS
Section 5-1: Pixel Value Transformations
5.1 Explain why, in the gamma transformation (Eq. (5.10)),
γ > 1 tends to darken images, while γ < 1 tends to lighten
images.
5.2 Use gamma transformation with γ = 3 to darken the image
in coins1.mat.
5.3 Use gamma transformation with γ = 3 to darken the image
in coins2.mat.
Section 5-2: Unsharp Masking
5.4 Use the sharpening filter Eq. (5.32) to sharpen the two
images in the files (a) plane.mat and (b) coins2.mat.
5.5 Use the sharpening filter Eq. (5.32) to sharpen the two
images in the files (a) quarter.mat and (b) rice.mat.
5.6 Use sharpening filter Eq. (5.32) to sharpen the two images
in the files (a) moon.mat and (b) unsharp.mat.
5.7 Unsharp masking was originally based on Eq. (5.14),
which in discrete space is
g[n, m] = f [n, m] + ( f [n, m] − fblur [n, m]).
fblur [n, m] is a lowpass version of f [n, m]. If fblur [n, m] is the
average of { f [n + 1, m], f [n − 1, m], f [n, m + 1], f [n, m − 1]},
show that
g[n, m] = f [n, m] − f [n, m] ∗ ∗
1
hLaplacian [n, m],
4
similar to Eq. (5.32).
5.8 Unsharp masking was originally based on Eq. (5.14),
which is
g(x, y) = f (x, y) + ( f (x, y) − fblur (x, y)).
fblur (x, y) is a lowpass version of f (x, y). Adobe R Photoshop R
uses the following form of unsharp masking:
p
fblur (x, y) = f (x, y) ∗ ∗ fg ( x2 + y2),
where
2
2
2
p
e−(x +y )/(2σ )
2
2
fg ( x + y ) =
.
2πσ 2
IMAGE ENHANCEMENT
Plot a cross-section (with ν = 0) of the spatial frequency
response of this form of unsharp masking with that of the
continuous-space version of Eq. (5.32), which is
hsharpen (x, y) = δ (x) δ (y) − ∇2 f (x, y).
2 2 2
Hint: Use the result of Problem 3.4, which is Fg (ρ ) = e−2σ π ρ .
Use σ 2 = 2.
5.9 Use unsharp masking as defined in Problem 5.8 to
sharpen the two images in the files (a) circuit.mat and (b)
quarter.mat. Use unsharp.m.
5.10 Use unsharp masking as defined in Problem 5.8 to
sharpen the two images in the files (a) tire.mat and (b)
coins2.mat. Use unsharp.m.
Section 5-3: Histogram Equalization
5.11 This problem applies histogram equalization to a tiny
(3 × 3) image. The goal is for the reader to work the problem
entirely by hand, thereby aiding understanding. The (3 × 3)
image is


1 2 1
f [n, m] = 2 3 9 .
3 2 9
(a) Plot the histogram of the image.
(b) List its distribution and CDF in a table.
(c) List values of f [n, m] and values of the histogram-equalized
image g[n, m] in a table.
(d) Depict g[n, m] as a 3 × 3 matrix, similar to the depiction of
f [n, m].
(e) Depict f [n, m] and g[n, m] as images, and plot their respective histograms and CDF’s.
5.12 Use the program hist.m to apply histogram equalization to the image in circuit.mat. Print out the images,
histograms, and CDFs of the original and equalized images.
5.13 Use the program hist.m to apply histogram equalization to the image in pout.mat. Print out the images,
histograms, and CDFs of the original and equalized images.
5.14 Use the program hist.m to apply histogram equalization to the image in tire.mat. Print out the images,
histograms, and CDFs of the original and equalized images.
5.15 Use the program hist.m to apply histogram equalization to the image in coins.mat. Print out the images,
histograms, and CDFs of the original and equalized images.
PROBLEMS
179
Section 5-4: Edge Detection
5.16 Sobel edge detection works by convolving the image
f [n, m] with the two PSFs


1 0 −1
hH [n, m] = 2 0 −2
1 0 −1
and


1
2
1
0
0 .
hV [n, m] =  0
−1 −2 −1
Compute the discrete-space frequency response H(Ω1 , Ω2 ) of
each of these PSFs.
5.17 Sobel edge detection works by convolving the image
f [n, m] with the two PSFs


1 0 −1
hH [n, m] = 2 0 −2
1 0 −1
and


1
2
1
0
0 .
hV [n, m] =  0
−1 −2 −1
Show how to implement these two convolutions using
(a) 16N 2 additions and subtractions since doubling is two
additions;
(b) 10N 2 additions and subtractions since hH [n, m] and hV [n, m]
are separable.
5.18 Apply (a) Sobel edge detection and (b) Canny edge
detection to the image in plane.mat using the programs
sobel.m and canny.m. Compare results.
5.19 Apply (a) Sobel edge detection and (b) Canny edge
detection to the image in quarter.mat using the programs
sobel.m and canny.m. Compare results.
5.20 Apply (a) Sobel edge detection and (b) Canny edge detection to the image in moon.mat using the programs sobel.m
and canny.m. Compare results.
5.21 Apply (a) Sobel edge detection and (b) Canny edge
detection to the image in saturn.mat using the programs
sobel.m and canny.m. Compare results.
www.dbooks.org
Chapter
6
Deterministic Approach to
Image Restoration
6
0
Contents
6-1
6-2
6-3
6-4
6-5
6-6
Overview, 181
Direct and Inverse Problems, 181
Denoising by Lowpass Filtering, 183
Notch Filtering, 188
Image Deconvolution, 191
Median Filtering, 194
Motion-Blur Deconvolution, 195
Problems, 199
Objectives
Learn to:
■ Denoise a noisy image using the 2-D DFT and a
Hamming-windowed lowpass filter.
224
0
274
(a) Image g[n,m]: motion-blurred highway sign
0
224
0
274
(b) Image g′[n,m]: motion-blurred highway sign
with additive noise
0
■ Notch-filter an image with sinusoidal interference
added to it.
■ Use median filtering to denoise an image with
salt-and-pepper noise added to it.
■ Use Tikhonov regularization in a deconvolution
problem.
■ Deconvolve an image blurred with a known
point-spread function.
■ Deblur a motion-blurred image.
274
0
274
(c) Reconstructed highway sign
This chapter covers image restoration from a noisy
or blurred version of it, where the blur and noise
were introduced by the imaging system. Denoising
can be performed by lowpass-filtering the noisy
image using the 2-D DFT (a Hamming-windowed
filter works better than a brick-wall filter). Deblurring (deconvolution) can be performed using the
2-D DFT, although Tikhonov regularization is
usually required. Motion-blur deblurring is a
common application; refocusing an out-of-focus
image is also common.
Overview
The goal in image restoration is to recover the true image
f [n, m]—or a close version of it—from its noisy or blurred
version g[n, m]. Examples include:
(1) Denoising: removing noise that had been added by the
imaging system.
(2) Removing interference: subtracting an unwanted image
that had been added to f [n, m].
(3) Deblurring: undoing the effect of the convolution of a
system PSF h[n, m] with image f [n, m], where h[n, m] is the
system response due to motion blur or defocusing.
Image restoration methods are categorized into two groups:
deterministic and probabilistic. Deterministic methods apply
algorithms—such as lowpass filtering—that do not incorporate
knowledge of probability and random processes associated with
the image or the image formation process. The present chapter
deals exclusively with deterministic restoration methods applied
to discrete-space images. Image restoration methods using the
probabilistic approach are treated in Chapter 9.
6-1 Direct and Inverse Problems
In image processing, we encounter two interrelated operations
commonly called the direct and inverse problems, and our goal
is to obtain solutions to both problems.
The solution to the direct problem consists of a mathematical
model that correctly accounts for the two forms of distortions
commonly encountered in the generation of an image: (1) blurring by the imaging system and (2) the introduction of additive
noise. As noted in Chapter 1, an imaging sensor—be it our
eye’s pupil, a camera, an MRI system, or any other 2-D imageforming configuration—has a non-zero “beam” described by
a point spread function h(x, y). The image formation process
is equivalent to convolving a filter h(x, y) with the true image
f (x, y), which results in blurring. The second form of distortion
involves the addition of random noise, which may be contributed
entirely by the electronics of the imaging and recording systems,
or it may also include a component due to peripheral sources in
the imaged scene.
Mathematically, the direct problem is modeled as
where
To illustrate the impact of each type of distortion separately, let us start with a noise-free imaging process by setting
υ (x, y) = 0. In Fig. 6-1, we show a high-quality MRI image
labeled f (x, y). We will treat it as if it were a true image, and
then through simulations, we will subject f (x, y) to convolution
with hi (x, y) to generate image gi (x, y):
gi (x, y) = hi (x, y) ∗ ∗ f (x, y),
(6.1)
(6.2)
with index i = 1, 2, or 3 referring to Gaussian-shaped filters
hi (x, y) of different effective radii. Image g1 (x, y) is the result of
convolving f (x, y) with a narrow PSF h1 (x, y). Because h1 (x, y)
is narrow, the blurring is visually unnoticeable. Image g2 (x, y)
and g3 (x, y) are the result of convolving the same original
image with a medium-wide PSF h2 (x, y) and wide PSF h3 (x, y),
respectively. Not surprisingly, increasing the filter’s effective
width leads to more image blurring.
Next we simulate the impact that additive random noise
imparts onto the appearance of an image. The noise is added
to f (x, y) after convolving it with the narrow filter h1 (x, y), so
the applicable model is
g1 j (x, y) = h1 (x, y) ∗ ∗ f (x, y) + υ j (x, y),
6-1.1 The Direct Problem
g(x, y) = h(x, y) ∗ ∗ f (x, y) + υ (x, y),
g(x, y) = recorded image,
f (x, y) = true image of the scene,
h(x, y) = PSF of the imaging system,
υ (x, y) = additive noise.
(6.3)
where j = 1, 2, or 3 refers to three noise-addition simulations,
characterized by different signal-to-noise ratios. All three output
images in Fig. 6-2 were generated by adding noise randomly
to different segments of the convolved image, but in g11 (x, y),
the average power content of the added noise is much smaller
than the average power of the noise-free image, whereas the
signal (image) and noise powers are comparable to one another
in g12 (x, y), and the noise power is much larger than the signal
power in g13 (x, y). Noise distorts an image by changing its
amplitude, whereas the PSF distorts it through spatial averaging.
Most images include both types of distortions.
The solution to the direct problem entails computing g(x, y)
from f (x, y) using Eq. (6.1). Doing so requires knowledge of the
PSF of the imaging system, h(x, y), and the statistical nature of
the added noise υ (x, y). The PSF usually is determined through
calibration tests in which the output g(x, y) is measured in
response to a strong point-like target placed at the center of the
scene, as illustrated in Fig. 1-8. The strong target allows us to
ignore the noise υ (x, y), and its point-like spatial extent makes
it equivalent to a 2-D impulse δ (x, y). Setting υ (x, y) = 0 and
181
www.dbooks.org
182
CHAPTER 6
DETERMINISTIC APPROACH TO IMAGE RESTORATION
f (x,y)
h1(r)
h1(x,y)
h2(r)
r
h1(x,y) ** f (x,y)
g1(x,y)
h2(x,y)
h3(r)
r
h2(x,y) ** f (x,y)
g2(x,y)
h3(x,y)
r
h3(x,y) ** f (x,y)
g3(x,y)
Figure 6-1 Simulation of image blurring: the original noise-free image atpthe top is convolved with Gaussian-shaped point spread
functions of different effective widths. The variable r is the radial distance r = x2 + y2 .
f (x, y) = δ (x, y) in Eq. (6.1) gives
g(x, y) = h(x, y) ∗ ∗δ (x, y) = h(x, y).
(6.4)
In most imaging systems, the noise υ (x, y) is random in nature
and usually modeled as a zero-mean Gaussian random variable
(Chapter 8). Accordingly, υ (x, y) is described by a probability
density function (pdf ) that contains a single parameter, the
noise variance σv2 . The pdf and associated variance can be measured experimentally by recording the output g(x, y) for many
locations (x, y), while having no signal as input ( f (x, y) = 0).
For a camera, this is equivalent to imaging a perfectly dark
object. The recorded image in that case is the noise added by
the camera.
Once h(x, y) has been characterized and υ (x, y) has been
modeled appropriately, image g(x, y) can be readily computed
using Eq. (6.1), thereby providing a possible solution of the
direct problem. Because υ (x, y) is random in nature, each
simulation of Eq. (6.1) will result in a statistically different, but
comparable image g(x, y).
6-1.2 The Inverse Problem
Whereas the solution of the direct problem seeks to generate
image g(x, y) from image f (x, y), the solution of the inverse
problem seeks to do the exact opposite, namely to extract the
true image f (x, y)—or a close facsimile of f (x, y)—from the
blurred and noisy image g(x, y). The process involves (a) denoising g(x, y) by filtering out υ (x, y)—or at least most of it—and
(b) deconvolution of g(x, y) to generate a close approximation
of the true image f (x, y). The denoising and deconvolution steps
of the inversion algorithm are performed using deterministic
6-2 DENOISING BY LOWPASS FILTERING
183
f (x,y)
h1(x,y)
g1(x,y) = h1(x,y) ** f (x,y)
υ(x,y)
SNR = 20 dB
g11(x,y)
υ(x,y)
SNR = 0 dB
υ(x,y)
SNR = −10 dB
g12(x,y)
g13(x,y)
Figure 6-2 The image in the upper center, g1 (x, y), had been convolved with the narrow filter before noise was added to it. SNR = 20 dB
corresponds to average signal power/average noise power = 100, so g11 (x, y) is essentially noise-free. In contrast, SNR = 0 dB in g12 (x, y),
which means that the average signal and noise powers are equal, and in g13 (x, y) the noise power is 10× the signal power..
methods, as demonstrated in later sections of the present chapter,
or they are performed using stochastic (probabilistic) methods,
which we cover later in Chapter 9. As a “heads up”, we note that
the stochastic approach usually outperforms the deterministic
approach.
6-2 Denoising by Lowpass Filtering
In Section 3-7, we introduced and defined the 2-D discretespace Fourier transform (DSFT) F(Ω1 , Ω2 ) of discrete-space
image f [n, m]. Here, Ω1 and Ω2 are continuous spatial frequen-
www.dbooks.org
184
CHAPTER 6
Application of the lowpass filter to the spectrum G(Ω1 , Ω2 ) of
noisy image g[n, m] generates spectrum
cies one period of which is over the range
− π ≤ Ω1 , Ω2 ≤ π .
We will refer to this continuous-frequency domain as the
discrete-space spatial frequency (DSSF) domain.
In the DSSF domain, most of the energy in the spectra
of typical images is concentrated in a small central region
surrounding the origin (Ω1 = 0, Ω2 = 0). In contrast, additive
noise may be distributed over a wide range of frequencies Ω1
and Ω2 . If we denote G(Ω1 , Ω2 ) as the spectrum of noisy image
g[n, m], the rationale behind lowpass filtering is to remove highfrequency noise from G(Ω1 , Ω2 ) while preserving (as much as
possible) the spectrum of the original image F(Ω1 , Ω2 ). The
disadvantage of lowpass filtering is that the high-DSSF regions
of an image may represent features of interest, such as edges.
◮ Straightforward lowpass filtering—the subject of the
present section—eliminates high-frequency noise, but also
eliminates edges and sharp variations in the image. If
preserving edges is important, alternative methods should
be used, such as the wavelet-denoising approach described
in Chapter 7. ◭
6-2.1 Brickwall Lowpass Filtering
Signal-to-noise (SNR) is a measure of how significant (or
insignificant) the presence of noise is and the degree to which
it is likely to distort the image. If the noisy image g[n, m] is
composed of the true image f [n, m] plus additive noise υ [n, m],
g[n, m] = f [n, m] + υ [n, m],
then the SNR in dB is defined as
∑ ∑ f 2 [n, m]
,
SNR = 10 log10
∑ ∑ υ 2 [n, m]
DETERMINISTIC APPROACH TO IMAGE RESTORATION
(6.5)
(6.6)
where the summations are performed over all image pixels.
A brickwall lowpass filter passes all frequency components
below a specified cutoff frequency Ωc (along both Ω1 and Ω2 )
and removes all components at frequencies above Ωc . The
DSSF response of the brickwall lowpass filter over 1 period of
(Ω1 , Ω2 ) is
(
1
for 0 ≤ Ω1 , Ω2 ≤ Ωc ,
(6.7)
Hbrick (Ω1 , Ω2 ) =
0
for Ωc < Ω1 , Ω2 ≤ π .
Gbrick (Ω1 , Ω2 ) = Hbrick (Ω1 , Ω2 ) G(Ω1 , Ω2 ).
(6.8)
The operation given by Eq. (6.8) can be performed in the
(N × N) 2-D DFT domain (Section 3-8) by defining a 2-D DFT
cutoff index K such that
K=
Ωc N
,
2π
(6.9)
and then setting to zero those elements of the 2-D DFT of
G[k1 , k2 ] that fall in the range
K ≤ k1 , k2 ≤ N + 2 − K.
(6.10)
We note that the operation preserves conjugate symmetry in the
filtered spectrum Gbrick [k1 , k2 ]. Consequently, the inverse 2-D
DFT gbrick [n, m] of Gbrick [k1 , k2 ] is real-valued.
To illustrate the trade-off between noise reduction and preservation of fast-varying (with position) image features, we consider a noisy letters image, which characteristically has many
edges. Starting with a noise-free image f [n, m], we synthesized
a noisy image g[n, m] by adding random amounts of noise to the
image pixels. The noisy image, consisting of (256 × 256) pixels,
is shown in Fig. 6-3(a). The values f [n, m] of the original image
ranged between 0 (representing black) and 255 (representing
white). Thus,
0 ≤ f [n, m] ≤ 255.
Intentionally, the amount of noise that was added to the original
image was much greater in total energy than the energy of the
original image itself, thereby producing a very noisy looking
image. The associated SNR is −12.8 dB, which means that the
total energy of f [n, m] is only 5.25% of that of the total noise
energy. To preserve non-negativity of the noisy image g[n, m],
the noise image υ [n, m] was assigned the range 0 to 500:
0 ≤ υ [n, m] ≤ 500.
Consequently, the range of pixel values in noisy image g[n, m] is
0 ≤ g[n, m] ≤ 755.
• Figure 6-3(b) displays spectrum 10 log10 [G(Ω1 , Ω2 )] of
g[n, m], which extends between −π and π along both Ω1
and Ω2 .
• Multiplication of the spectrum in Fig. 6-3(b) by a brickwall
6-2 DENOISING BY LOWPASS FILTERING
185
−π
π
0
Ω2
π
DSFT
Ω1
0
255
0
255
n
(b) Spectrum 10 log10[G(Ω1,Ω2)]
of noisy image
−π
m (a) Noisy image g[n,m]
−π
π
0
Ω2
π
Inverse DSFT
Ω1
0
255
0
n
255
m (c) Filtered image gbrick[n,m] with Ωc1 = 75π/128
0
(d) Ωc1 = 75π/128
−π
−π
π
Ω2
π
Inverse DSFT
Ω1
0
255
0
n
255
m (e) Filtered image gbrick[n,m] with Ωc2 = 50π/128
0
−π
π
Inverse DSFT
0
255
0
n
255
m (g) Filtered image gbrick[n,m] with Ωc3 = 25π/128
(f ) Ωc2 = 50π/128
−π
−π
Ω2
π
Ω1
(h) Ωc3 = 25π/128
Figure 6-3 Image denoising by three lowpass filters with different cutoff wavenumbers.
www.dbooks.org
186
CHAPTER 6
DETERMINISTIC APPROACH TO IMAGE RESTORATION
lowpass filter with Ωc = 75π /128 leads to the spectrum
in Fig. 6-3(d). The two spectra are identical within the
square defined by |Ω1 |, |Ω2 | ≤ Ωc1 . The fractional size of
the square is (75/128)2 = 34% of the spectrum of the
original image.
Inverse transforming the spectrum in Fig. 6-3(d) produces
the lowpass-filtered image in Fig. 6-3(c). We observe that
the noise is reduced, albeit only slightly, but the letters are
hardly distorted.
The fundamental frequency of x(t) is f0 = 1/T = 1/2π . We
can apply the equivalent of a brickwall lowpass filter with cutoff
frequency kc f0 by truncating the Fourier series at k = kc . For
example, if we select kc = 21, we obtain a brickwall lowpassfiltered version of x(t) given by
• Narrowing the filtered spectrum down to a box with
Ωc2 = 50π /128 leads to the spectrum in Fig. 6-3(f). The associated filtered image gbrick [n, m] is shown in Fig. 6-3(e).
In this case, only (50/128)2 = 15% of the spectrum of
the original noisy image is retained. The filtered image
contains less noise, but the letters are distorted slightly.
The truncated summation contains 11 nonzero terms. The plot of
ybrick (t) displayed in Fig. 6-4(b) resembles the original square
wave, except that it also exhibits small oscillations; i.e., the
ringing effect we referred to earlier.
• Repeating the process, but limiting the spectrum to
Ωc3 = 25π /128—in which case, only (25/128)2 = 4% of
the spectrum is retained—leads to the image in Fig. 6-3(g).
The noise is greatly reduced, but the edges of the letters are
fuzzy.
◮ This example illustrates the trade-off inherent in Fourierbased lowpass filtering: noise can be reduced, but at the expense of distorting the high-frequency content of the image.
As noted earlier, in Chapter 7 we show how to avoid this
trade-off using wavelets instead of Fourier transforms. ◭
6-2.2 Tapered Lowpass Filtering
Even though it is not apparent in the filtered images of Fig. 6-3,
at lower cutoff DSSFs, some of what appears to be noise is
actually “ringing” caused by the abrupt “brickwall” filtering of
the spectrum of the noisy image. Fortunately, the ringing effect
can be reduced significantly by modifying the brickwall filter
into a tapered filter. We will examine both the problem and the
proposed solution for 2-D images, but before we do so, it will
be instructive to consider the case of a periodic 1-D signal.
A. 1-D brickwall lowpass-filtered signal
Signal x(t), shown in Fig. 6-4(a) is a square wave with period
T = 2π and amplitude A = 1. Its Fourier series expansion is
given by
∞
4
x(t) = ∑
sin(kt).
(6.11)
k=1 kπ
k=odd
21
ybrick (t) =
4
∑ kπ sin(kt).
(6.12)
k=1
k=odd
B. 1-D tapered lowpass-filtered signal
The ringing in the lowpass-filtered signal, which is associated
with the sharp cutoff characteristic of the brickwall filter, can be
reduced significantly by multiplying the terms in Eq. (6.12) by
a decreasing sequence of weights, thereby tapering those terms
gradually to zero. Several tapering formats are available, one of
which is the Hamming window defined by Eq. (2.83). Adapting
the expression for the Hamming window to the square wave
leads to
21
π (k − 1)
4
yHam (t) = ∑
sin(kt) 0.54 + 0.46 cos
.
20
k=1 kπ
k=odd
(6.13)
A plot of the tapered signal yHam (t) is shown in Fig. 6-4(c).
Even though the tapered signal includes the same number of frequency harmonics as before, the oscillations have disappeared
and the transitions at t = integer values of T /2 = π are relatively
smooth.
C. 2-D brickwall lowpass-filtered image
The ringing observed in 1-D signals also manifests itself in 2-D
images whenever the image spectrum is lowpass-filtered by a
sharp filter. The DSSF response Hbrick (Ω1 , Ω2 ) of a brickwall
lowpass filter with cutoff frequency Ωc along both Ω1 and Ω2 is
given by Eq. (6.7). The corresponding inverse DSFT is
hbrick [n, m] = hbrick [n] hbrick [m]
2
Ωc
Ωc n
Ωc m
=
sinc
sinc
.
π
π
π
(6.14)
6-2 DENOISING BY LOWPASS FILTERING
187
1
0.5
0
−0.5
−1
0
1
2
3
π
4
5
6
2π
7
8
9
7
8
9
7
8
9
t
(a) Square wave x(t)
1
0.5
0
−0.5
−1
0
1
2
3
4
5
6
t
(b) Brickwall lowpass-filtered signal ybrick(t)
1
0.5
0
−0.5
−1
0
1
2
3
4
5
6
t
(c) Tapered Fourier series signal yHam(t)
Figure 6-4 (a) Square wave x(t), (b) brickwall lowpass-filtered version, and (c) Hamming-windowed version.
www.dbooks.org
188
CHAPTER 6
Per the definition of the sinc function given by Eq. (2.35),
namely sinc(x) = [sin(π x)]/(π x),
π Ωc n
sin
Ωc n
sin(Ωc n)
π
=
sinc
,
(6.15)
= π Ωc n
π
Ωc n
π
and a similar definition applies to sinc(Ωc m/π ).
For an image g[n, m] with a corresponding DSFT G(Ω1 , Ω2 ),
the filtered image spectrum is
Gbrick (Ω1 , Ω2 ) = Hbrick (Ω1 , Ω2 ) G(Ω1 , Ω2 ),
(6.16a)
0.25
0.20
hHam[n]
Impulse response
0.10
n
0
−0.05
−10 −5 0
5 10
(a) Impulse response of Hamming-windowed filter
1.5
HHam(Ω)
Frequency response
1
and the corresponding spatial-domain relationship is
gbrick [n, m] = hbrick [n, m] ∗ ∗g[n, m].
DETERMINISTIC APPROACH TO IMAGE RESTORATION
(6.16b)
The impact of the brickwall lowpass-filtering process on a 2-D
image was illustrated earlier through Fig. 6-3.
0.5
0
−π
π
0
(b) Spectrum HHam(Ω) of Hamming-windowed filter
D. 2-D tapered lowpass-filtered image
Figure 6-5 Hamming window of length N = 10: (a) impulse
Figure 6-5 displays the spatial and frequency domain responses
of a Hamming window with N = 10, adapted from Fig. 2-13.
For a 2-D image g[n, m], lowpass filtering its spectrum with a
Hamming window of length N is equivalent to performing the
convolution
response, and (b) spectrum.
gHam [n, m] = hHam [n, m] ∗ ∗g[n, m]
(6.17a)
with
hHam [n, m] = hHam [n] hHam [m] =
 2
Ωc

sinc Ωπc n 0.54 + 0.46 cos πNn

 π
Ωc m
×
sinc
0.54 + 0.46 cos πNm

π


0
In contrast, the image in Fig. 6-6(e)—which was generated by
applying a Hamming windowed filter to the original image—
exhibits no “ringing.” Note that the spectrum of the Hammingwindowed spectrum in Fig. 6-6(f) tapers gradually from the
center outward. It is this tapering profile that eliminates the
ringing effect.
Concept Question 6-1: For lowpass filtering, why would
we use a Hamming-windowed filter instead of a brick-wall
filter?
for |n|, |m| ≤ N,
for |n|, |m| > N. 6-3 Notch Filtering
(6.17b)
Occasionally, an image may contain a 2-D sinusoidal interTo illustrate the presence of “ringing” when a sharp-edged filter ference contributed by an electromagnetic source, such as the
like a brickwall is used, and its absence when a Hamming win- ac power cable in a camera. In 1-D discrete-time signals,
dowed filter is used instead, we refer the reader to Fig. 6-6. In sinusoidal interference can be eliminated by subjecting the
part (a), we show a noiseless letters image, and its corresponding signal’s spectrum to a notch filter, which amounts to setting the
spectrum is displayed in part (b). Application of a brickwall spectrum at that specific frequency to zero. A similar process
lowpass filter (with the impulse response given by Eq. (6.14)) can be applied to a 2-D image. To illustrate, let us consider the
leads to the image in Fig. 6-6(c). The “ringing” in the image is example portrayed in Fig. 6-7. In part (a) of the figure, we have a
visible in the form of whorls that resemble a giant thumbprint. 660 × 800 image of the planet Mars recorded by a Mariner space
6-3 NOTCH FILTERING
189
−π
π
0
Ω2
π
DSFT
128
255
Ω1
0
0
128
−π
255
(a) Noiseless letters image
(b) Log of spectrum of noiseless letters image
−π
π
0
Ω2
π
DSFT
128
255
0
128
255
(c) Brickwall lowpass-filtered image
−π
(d) Log of spectrum of brickwall lowpass-filtered image
−π
π
−10
128
265
Ω1
0
−10
128
265
(e) Hamming lowpass-filtered image
DSFT
0
Ω2
π
Ω1
−π
(f) Log of spectrum of Hamming-windowed
lowpass-filtered image
Figure 6-6 Letters image and its spectrum in (a) and (b); brickwall lowpass-filtered version in (c) and (d), and brickwall lowpass-filtered
version with a Hamming window in (e) and (f). The logarithmic scale enhances small values of the spectrum.
www.dbooks.org
190
CHAPTER 6
DETERMINISTIC APPROACH TO IMAGE RESTORATION
Ω2
π
0
DSFT
659
0
Ω1
0
−π
−π
π
(b) Log of magnitude of spectrum of original image
799
(a) Original image with scan lines
Ω2
π
0
DSFT
659
799
0
(c) Image of vertically periodic horizontal lines
Ω1
0
−π
−π
π
(d) Magnitude of spectrum of vertically
periodic horizontal lines
Ω2
π
0
Inverse DSFT
0
659
0
799
(e) Notch-filtered image
Ω1
−π
−π
π
(f ) Log of magnitude of spectrum of notch-filtered image
Figure 6-7 Process for notch-filtering horizontal scan lines. Fig. 6-7(a) courtesy of NASA.
6-4 IMAGE DECONVOLUTION
191
probe. In addition to the image of the planet, the image contains
near-horizontal scan lines that we wish to eliminate by applying
notch filtering.
Let us designate the image containing only the nearhorizontal lines as f ′ (x, y), which we display in Fig. 6-7(c),
along with its spectrum in Fig. 6-7(d). Along the vertical
direction, the image contains (2M + 1) lines, with one line
passing through the center and M lines each above and below
the center line. The separation between adjacent lines is ∆. For
the present, we will treat the near-horizontal lines as if they were
perfectly horizontal (correction for skewness will be applied
later). Accordingly, f ′ (x, y) is given by
M
f ′ (x, y) =
∑ δ (y − m∆),
(6.18)
m=−M
and (using entry #3 in Table 2-4 and property #2 in Table 2-5)
the associated spectrum of f ′ (x, y) is
F′ ( µ , ν ) =
M
∑ e− j2πν m∆ δ (µ ).
(6.19)
m=−M
The sum in Eq. (6.19) is similar in form to the sum in Eq. (2.87),
M
∑ e− jΩm =
m=−M
sin((2M + 1)Ω/2)
,
sin(Ω/2)
(6.20)
thereby allowing us to rewrite Eq. (6.19) as
F′ ( µ , ν ) =
sin((2M + 1)πν ∆)
δ (µ ).
sin(πν ∆)
(6.21)
The spectrum, which is a discrete sinc function along the ν
(vertical spatial frequency) direction and an impulse function
along the µ (horizontal spatial frequency) direction, is displayed
in Fig. 6-7(d). Only the peaks of the discrete sinc are visible in
the image.
Recall from Section 3-4.2 that rotating an image causes its
spectrum to rotate by the same angle. Hence, the interfering
vertically periodic near-horizontal lines in Fig. 6-7(a) should
appear in its spectrum (Fig. 6-7(b)) as a near-vertical line rotated
slightly counterclockwise.
The spectrum shown in Fig. 6-7(f) is the spectrum of the
original image after setting the spectrum associated with the
interfering lines to zero. Application of the inverse transform
leads to the filtered image shown in Fig. 6-7(e). The interfering
horizontal lines have been eliminated, with minor degradation
to the rest of the image.
Concept Question 6-2:
From where does a notch filter
get its name?
Concept Question 6-3:
What is notch filtering used for?
6-4 Image Deconvolution
When our eyes view a scene, they form an approximate image of
the scene, because the optical imaging process performed by the
eyes distorts the true scene, with the degree of distortion being
dependent on the imaging properties of the eyes’ lenses. The
same is true when imaging with a camera, a medical imaging
system, and an optical or radio telescope. Distortion can also be
caused by the intervening medium between the imaged scene
and the imaging sensor. Examples include the atmosphere when
a telescope is used to image a distant object, or body tissue
when a medical ultrasound sensor is used to image body organs.
In all cases, the imaging process involves the convolution of a
true image scene f [n, m] with a point spread function h[n, m]
representing the imaging sensor (and possibly the intervening
medium). The recorded (sensed) image g[n, m] is, therefore,
given by
g[n, m] = h[n, m] ∗ ∗ f [n, m].
(6.22)
The goal of image deconvolution is to deconvolve the recorded
image so as to extract the true image f [n, m], or a close
approximation of it. Doing so requires knowledge of the PSF
h[n, m]. In terms of size:
• f [n, m] is the unknown true image, with size (M × M).
• h[n, m] is the known PSF, with size (L × L).
• g[n, m] is the known recorded image, with size
(L + M − 1) × (L + M − 1).
As noted earlier in Chapter 1, the PSF of the imaging sensor
can be established by imaging a small object representing a 2-D
impulse.
Since convolution in the discrete-time domain translates into
multiplication in the frequency domain, the DSFT-equivalent of
Eq. (6.22) is given by
G(Ω1 , Ω2 ) = H(Ω1 , Ω2 ) F(Ω1 , Ω2 ).
(6.23)
Before we perform the frequency transformation of g[n, m], we
should round up (L+ M − 1) to N, where N is the smallest power
of 2 greater than (L + M − 1). The rounding-up step allows us
www.dbooks.org
192
CHAPTER 6
DETERMINISTIC APPROACH TO IMAGE RESTORATION
to use the fast radix-2 2-D FFT to compute 2-D DFTs of order
(N × N). Alternatively, the Cooley-Tukey FFT can be used, in
which case N should be an integer with a large number of small
factors.
Sampling the DSFT at Ω1 = 2π k1/N and Ω2 = 2π k2 /N for
k1 = 0, 1, . . . , N − 1 and k2 = 0, 1, . . . , N − 1 provides the DFT
complex coefficients G[k1 , k2 ].
A similar procedure can be applied to h[n, m] to obtain
coefficients H[k1 , k2 ], after zero-padding h[n, m] so that it also
is of size (N × N). The DFT equivalent of Eq. (6.23) is then
given by
G[k1 , k2 ] = H[k1 , k2 ] F[k1 , k2 ].
(6.24)
(a) Letters image f [n,m]
The objective of deconvolution is to compute the DFT coefficients F[k1 , k2 ], given the DFT coefficients G[k1 , k2 ] and
H[k1 , k2 ].
6-4.1 Nonzero H[k1, k2 ] Coefficients
In the ideal case where none of the DFT coefficients H[k1 , k2 ]
are zero, the DFT coefficients F[k1 , k2 ] of the unknown image
can be obtained through simple division,
F[k1 , k2 ] =
G[k1 , k2 ]
.
H[k1 , k2 ]
(6.25)
Exercising the process for all possible values of k1 and k2 leads
to an (N × N) 2-D DFT for F[k1 , k2 ], whereupon application of
an inverse 2-D DFT process yields a zero-padded version of
f [n, m]. Upon discarding the zeros, we obtain the true image
f [n, m]. The deconvolution procedure is straightforward, but
it hinges on a critical assumption, namely that none of the
DFT coefficients of the imaging system’s transfer function is
zero. Otherwise, division by zero in Eq. (6.25) would lead to
undeterminable values for F[k1 , k2 ].
(b) Blurred image g[n,m]
6-4.2 Image Deconvolution Example
To demonstrate the performance of the deconvolution process,
we used a noise-free version of the letters image, shown in
Fig. 6-8(a), which we denote f [n, m]. Then, we convolved it with
a truncated 2-D Gaussian PSF (a common test PSF) given by
2
2
h[n, m] = e−(m +n )/20 ,
−10 ≤ n, m ≤ 10.
(6.26)
The PSF represents the imaging system. The convolution process generated the blurred image shown in Fig. 6-8(b), which
(c) Deconvolved image f [n,m]
Figure 6-8 The blurred image in (b) was generated by convolving f [n, m] with a Gaussian PSF, and the image in (c) was
recovered through deconvolution of g[n, m].
6-4 IMAGE DECONVOLUTION
193
regularization, which seeks to minimize the cost function
we label g[n, m]. The image sizes are:
• Original letters image f [n, m]: 256 × 256
N−1 N−1
ˆ m])2
e = ∑ ∑ [(g[n, m] − h[n, m] ∗ ∗ f[n,
• Gaussian PSF h[n, m]: 21 × 21
m=0 n=0
+ (λ fˆ[n, m])2 ],
• Blurred image g[n, m]: 276 × 276, where
276 = 256 + 21 − 1.
After zero-padding all three images to 280 × 280, the 2-D FFT
was applied to all three images. Then, Eq. (6.25) was applied
to find coefficients F[k1 , k2 ], which ultimately led to the deconvolved image f [n, m] displayed in Fig. 6-8(c). The deconvolved
image matches the original image shown in Fig. 6-8(a). The
process was successful because none of the H[k1 , k2 ] coefficients
had zero values and G[k1 , k2 ] was noise-free. To avoid division
by H[k1 , k2 ] when computing F[k1 , k2 ], we use image regularization and Wiener filtering, as discussed in the next subsections.
6-4.3 Tikhonov Image Regularization
All electronic imaging systems generate some noise of their
own. The same is true for the eye-brain system. Hence,
Eq. (6.24) should be modified to
G[k1 , k2 ] = H[k1 , k2 ] F[k1 , k2 ] + V[k1, k2 ],
(6.29)
where zero-padding to size N × N has been implemented so that
all quantities in Eq. (6.29), except for λ , are of the same order.
The parameter λ is non-negative and it is called a regularization
parameter. The second term on the right-hand side of Eq. (6.29)
represents the bias error associated with fˆ[n, m] and the first
term represents the variance. Setting λ = 0 reduces Eq. (6.29)
to the unregularized state we dealt with earlier in Section 6-4.1,
wherein the measurement process was assumed to be noise-free.
For realistic imaging processes, λ should be greater than zero,
but there is no simple method for specifying its value, so usually
its value is selected heuristically (by trial and error).
The estimation process may be performed iteratively in the
discrete-time domain by selecting an initial estimate fˆ[n, m]
and then recursively iterating the estimate until the error e
approaches a minimum level. Alternatively, the process can be
performed in the frequency domain using a Wiener filter, as
discussed next.
(6.27)
where V[k1 , k2 ] represents the spectrum of the additive noise
contributed by the imaging system. The known quantities are the
measured image G[k1 , k2 ] and the PSF of the system, H[k1 , k2 ],
and the sought-out quantity is the true image F[k1 , k2 ]. Dividing
both sides of Eq. (6.27) by H[k1 , k2 ] and solving for F[k1 , k2 ]
gives
G[k1 , k2 ] V[k1 , k2 ]
F[k1 , k2 ] =
−
.
(6.28)
H[k1 , k2 ] H[k1 , k2 ]
In many practical applications, H[k1 , k2 ] may assume very small
values for large values of [k1 , k2 ]. Consequently, the second term
in Eq. (6.28) may end up amplifying the noise component and
may drown out the first term. To avoid the noise-amplification
problem, the deconvolution can be converted into a regularized
estimation process. Regularization involves the use of a cost
function that trades off estimation accuracy (of f [n, m]) against
measurement precision. The process generates an estimate
fˆ[n, m] of the true image f [n, m]. Accuracy refers to a bias
associated with all pixel values of the reconstructed image
fˆ[n, m] relative to f [n, m]. Precision refers to the ± uncertainty
associated with each individual pixel value due to noise.
A commonly used regularization model is the Tikhonov
6-4.4 Wiener Filter
Using Rayleigh’s theorem (entry #9 in Table 3-3), the frequency
domain DFT equivalent of the Tikhonov cost function given by
Eq. (6.29) is
E=
1 N−1 N−1
∑ [|G[k1 , k2 ] − H[k1, k2 ] F̂[k1, k2 ]|2
N 2 k∑
1 =0 k2 =0
+ λ 2 |X[k1 , k2 ]|2 ].
(6.30)
The error can be minimized separately for each (k1 , k2 ) combination. The process can be shown (see Problem 6-11) to lead to
the solution
F̂[k1 , k2 ] = G[k1 , k2 ]
H∗ [k1 , k2 ]
,
|H[k1 , k2 ]|2 + λ 2
(6.31)
where H∗ [k1 , k2 ] is the complex conjugate of H[k1 , k2 ]. The
quantity multiplying G[k1 , k2 ] is called a Wiener filter W[k1 , k2 ].
That is,
F̂[k1 , k2 ] = G[k1 , k2 ] W[k1 , k2 ],
(6.32a)
www.dbooks.org
194
CHAPTER 6
with
W[k1 , k2 ] =
H∗ [k1 , k2 ]
.
|H[k1 , k2 ]|2 + λ 2
DETERMINISTIC APPROACH TO IMAGE RESTORATION
Concept Question 6-4: What does the Wiener filter
given by Eq. (6.31) reduce to when λ = 0?
(6.32b)
Why is Tikhonov regularization
Concept Question 6-5:
needed in deconvolution?
The operation of the Wiener filter is summarized as follows:
(a) For values of (k1 , k2 ) such that |H[k1 , k2 ]| ≫ λ , the Wiener
filter implementation leads to
F̂[k1 , k2 ] ≈ G[k1 , k2 ]
G[k1 , k2 ]
H∗ [k1 , k2 ]
=
,
2
|H[k1 , k2 ]|
H[k1 , k2 ]
(6.33a)
which is the same as Eq. (6.25).
(b) For values of (k1 , k2 ) such that |H[k1 , k2 ]| ≪ λ , the Wiener
filter implementation leads to
F̂[k1 , k2 ] ≈ G[k1 , k2 ]
H∗ [k1 , k2 ]
.
λ2
(6.33b)
In this case, the Wiener filter avoids the noise amplification
problem that would have occurred with the use of the unregularized deconvolution given by Eq. (6.25).
6-4.5 Wiener Filter Deconvolution Example
To demonstrate the capabilities of the Wiener filter, we compare
image deconvolution performed with and without regularization. The demonstration process involves images at various
stages, namely:
• f [n, m]: true letters image (Fig. 6-9(a)).
• g[n, m] = h[n, m] ∗ ∗ f [n, m] + v[n, m]: the imaging process
not only distorts the image (through the PSF), but also adds
random noise v[n, m]. The result, displayed in Fig. 6-9(b),
is an image with signal-to-noise ratio of 10.8 dB, which
means that the random noise energy is only about 8% of
that of the signal.
• fˆ1 [n, m]: estimate of f [n, m] obtained without regularization (i.e., using Eq. (6.25)). Image fˆ1 [n, m], displayed in
Fig. 6-9(c), does not show any of the letters present in the
original image, despite the fact that the noise level is small
relative to the signal.
• fˆ2 [n, m]: estimate of f [n, m] obtained using the Wiener
filter of Eq. (6.31) with λ 2 = 5. The deconvolved image
(Fig. 6-9(d)) displays all of the letters contained in the
original image, but some high wavenumber noise also is
present.
Exercise 6-1: Apply Tikhonov regularization with λ = 0.01
to the 1-D deconvolution problem
{x[0], x[1]} ∗ {h[0], h[1], h[2]} = {2, −5, 4, −1},
where h[0] = h[2] = 1 and h[1] = −2.
Y(Ω)
Answer: H(0) = 1 − 2 + 1 = 0 so X(Ω) = H(Ω) will not
work at Ω = 0. But
X(Ω) =
H∗ (Ω)
Y(Ω)
|H(Ω)|2 + λ 2
does work. Using 4-point DFTs (computable by hand) gives
x[n] = {1.75, −1.25, −0.25, −0.25}, which is close to the
actual x[n] = {2, −1}. MATLAB code:
h=[1 -2 1];x=[2 -1];y=conv(x,h);
H=fft(h,4);Y=fft(y);
Z=conj(H).*Y./(abs(H).*abs(H)+0.0001);
z=real(ifft2(Z)) provides the estimated x[n].
6-5 Median Filtering
Median filtering is used to remove salt-and-pepper noise, often
due to bit errors or shot noise associated with electronic devices.
The concept of median filtering is very straightforward:
◮ A median filter of order L replaces each pixel with the
median value of the L2 pixels in the L × L block centered on
that pixel. ◭
For example, a median filter of order L = 3 replaces each pixel
[n, m] with the median value of the 3 × 3 = 9 pixels centered at
[n, m]. Figure 6-10(a) shows an image corrupted with salt-andpepper noise, and part (b) of the same figure shows the image
after the application of a median filter of order L = 5.
Concept Question 6-6:
When is median filtering useful?
6-6 MOTION-BLUR DECONVOLUTION
195
(a) f [n,m]
(b) g[n,m] = h[n,m] ** f [n,m] + v[n,m]
(c) fˆ1[n,m] without regularization
(d) fˆ2[n,m] with Wiener filter
Figure 6-9 (a) Original noise-free undistorted letters image f [n, m], (b) blurred image due to imaging system PSF and addition of random
noise v[n, m], (c) deconvolution using Eq. (6.25), and (d) deconvolution using Eq. (6.31) with λ 2 = 5.
6-6 Motion-Blur Deconvolution
6-6.1 Continuous Space
If, during the recording time for generating a still image of an
object or scene, the imaged object or scene is in motion relative
to the imaging system, the recorded image will exhibit a streak-
ing pattern known as motion blur. An example is the simulated
photograph of the highway sign shown in Fig. 6-11(a), taken
from a moving car. Often, the direction and duration of the blur
can be discerned from the blurred image.
To describe motion blur mathematically, we start by making
the following assumptions:
(1) The blurred image has been appropriately rotated so that
www.dbooks.org
196
CHAPTER 6
DETERMINISTIC APPROACH TO IMAGE RESTORATION
0
(a) Noisy image
(b) After median filtering
Figure 6-10 Median filtering example: (a) letters image corrupted by salt-and-pepper noise, and (b) image after application
of median filtering using a 5 × 5 window.
the direction of motion is aligned along the x axis.
(2) Image recording is T (seconds) in duration, with instantaneous start and stop actions, thereby allowing us to represent
the process in terms of a rectangle function.
(3) The motion of the imager, relative to the scene, is linear
(along the x axis) and at a constant speed s (meters/s).
In terms of the unblurred image f (x, y), corresponding to
no-motion conditions, the blurred image g(x, y) consists of a
superposition of copies of f (x, y) shifted along the x axis by
distance x′ = st:
g(x, y) =
Z T
0
f (x − x′ , y) dt.
(6.34)
224
0
274
(a) Image g[n,m]: motion-blurred highway sign
0
224
0
274
(b) Image g′[n,m]: motion-blurred highway sign
with additive noise
0
Upon replacing dt with dx′ /s and changing the upper integration
limit to D = sT , we have
g(x, y) =
1
s
Z D
0
f (x − x′ , y) dx′ .
(6.35)
The spatial shift occurs along x only, and its length is D, which
is equivalent to defining g(x, y) as the convolution of f (x, y) with
a point spread function h(x, y) composed of a rectangle function
of spatial duration D = sT and centered at D/2:
g(x, y) = f (x, y) ∗ ∗h(x, y),
with
h(x, y) =
1
x − D/2
rect
s
D
(6.36)
274
0
274
(c) Reconstructed highway sign
δ (y).
(6.37)
The spatial frequency response H(µ , ν ) is the 2-D Fourier
transform of h(x, y), which separates into two 1-D Fourier
Figure 6-11 Highway sign: (a) motion-blurred, (b) with combination of blurring and noise, and (c) after reconstruction.
6-6 MOTION-BLUR DECONVOLUTION
197
transforms. Using entries #1a and #4 in Table 2-5 gives
δ (y)
x − D/2
H(µ , ν ) = F x→µ rect
F y→ν
D
s
D
= sinc(µ D) e− jπ µ D .
(6.38)
s
The convolution in the spatial domain given by Eq. (6.36)
becomes a product in the spatial frequency domain:
G(µ , ν ) = F(µ , ν ) H(µ , ν ).
To recover the unblurred image f (x, y), we need to:
(a) divide Eq. (6.39) by H(µ , ν ) to obtain
1
G(µ , ν ),
F(µ , ν ) =
H(µ , ν )
(6.39)
(6.40)
where G(µ , ν ) is the spatial frequency spectrum of the blurred
image, and then
(b) perform an inverse transform on F(µ , ν ).
However, in view of the definition of the sinc function,
sinc(µ D) =
sin(π µ D)
,
π µD
(6.41)
it follows that the spatial frequency response H(µ , ν ) = 0 for
integer values of µ D. Consequently, the inverse filter 1/H(µ , ν )
is undefined for nonzero integer values of µ D, thereby requiring
the use of regularization (Section 6-4.3).
6-6.2 Motion Blur after Sampling
To convert image representation from the continuous-space case
of the previous subsection to the sampled-space case, we start by
sampling unblurred (still) image f (x, y) and the motion-blurred
image g(x, y) at x = n∆ and y = m∆:
f [n, m] = f (x = n∆, y = m∆),
g[n, m] = g(x = n∆, y = m∆).
(6.42a)
(6.42b)
We also discretize time t as
t = i∆t ,
(6.43)
where ∆t is the time interval associated with the movement of
the imager (relative to the scene) by a distance ∆:
∆t =
∆
,
s
where s is the relative speed of the imager (along the x axis).
The total number of time shifts N that occur during the total
recording time T is
T
.
(6.45)
N=
∆t
In terms of these new quantities, the discrete-case analogues to
Eqs. (6.36) and (6.37) are
N
g[n, m] = ∑ f [n − i, m] ∆t = f [n, m] ∗ ∗h[n, m],
(6.46)
i=0
where the discrete-space PSF h[n, m] is
n − N/2
δ [m] ∆t .
h[n, m] = rect
N/2
(6.47)
The rectangle function is of duration (N + 1), extending from
n = 0 to n = N, and centered at N/2. We assume that N is an
even integer. As with the continuous-space case, the deblurring
operation (to retrieve f [n, m] from g[n, m]) is performed in the
spatial frequency domain, wherein the assumption that N is an
even integer is not relevant, so the assumption is mathematically
convenient, but not critical.
The spatial frequency domain analogue of Eq. (6.46) is
G(Ω1 , Ω2 ) = F(Ω1 , Ω2 ) H(Ω1 , Ω2 ).
(6.48)
Here, G(Ω1 , Ω2 ) is the 2-D spectrum of the recorded blurred
image, and H(Ω1 , Ω2 ) is the discrete-space spatial frequency
response function (DSSF) response of h[n, m]. From entry #7
in Table 2-8, and noting that ∆t = T /N, the DSSF response
function corresponding to Eq. (6.47) is given by
H(Ω1 , Ω2 ) = DSFT{h[n, m]}
n − N/2
= DTFTn→Ω1 rect
N/2
T
× DTFTm→Ω2 {δ [m]}
N
T sin Ω1 N+1
2
e− jΩ1 N/2 .
=
N sin(Ω1 /2)
(6.49)
The sinc function dictates that H(Ω1 , Ω2 ) = 0 for
Ω1 (N + 1)/2 = kπ for nonzero integer values of k.
Consequently, the inverse filter 1/H(Ω1, Ω2 ) is undefined
at these values of Ω1 , thereby requiring regularization.
Regularization can be accomplished using the Wiener filter
(6.44)
www.dbooks.org
198
CHAPTER 6
(Section 6-4.4), which leads to
G(Ω1 , Ω2 ) H∗ (Ω1 , Ω2 )
F(Ω1 , Ω2 ) =
|H(Ω1 , Ω2 )|2 + λ 2
T sin Ω1 N+1
2
e jΩ1 N/2
G(Ω1 , Ω2 )
N sin(Ω1 /2)
. (6.50)
=
2
2
sin Ω1 N+1
T
2
2
+λ
N
sin(Ω1 /2)
The image deblurring process is illustrated by the three images
shown in Fig. 6-11.
(a) A (still) (225 × 225) image f [n, m] has been motionblurred into a (225 × 275) image g[n, m]. The image is a
simulated highway sign taken from a moving vehicle. The length
(N + 1) of the blur is 51.
(b) To further distort the blurred image, noise was added to
image g[n, m] to produce
g′ [n, m] = g[n, m] + υ [n, m],
DETERMINISTIC APPROACH TO IMAGE RESTORATION
Concept Question 6-8: How does one determine the
value of λ to use in Tikhonov regularization?
Exercise 6-2: For motion blur with a PSF of length
N + 1 = 60 in the direction of motion, at what spatial
frequencies Ω will the spatial frequency response be zero?
Answer: From Eq. (6.50), the numerator of the spatial
frequency response is zero when Ω = ±kπ /30 for any
nonzero integer k.
(6.51)
with υ [n, m] being a zero-mean random variable with a variance
of 100. The consequent signal-to-noise ratio is 5.35 dB.
(c) Application of the Wiener filter recipe given by Eq. (6.50)
and then inverting to the spatial domain to obtain f [n, m] leads
to the reconstructed image shown in Fig. 6-11(c). The implementation involved the use of 2-D DFTs of order (275 × 275).
Reconstructed images were generated for different values of λ
(in Eq. (6.50)); the value of λ = 1 provided the best result
visually.
Concept Question 6-7: How does one determine the
length N + 1 of the PSF for motion blur from the spectrum
of the blurred image?
Summary
Concepts
• Image restoration is about reconstructing an image from
its blurred, noisy, or interference-corrupted version.
• Lowpass filtering reduces noise, but it blurs edges and
fine-scale image features. Wavelet-based denoising (in
Chapter 7) reduces noise while preserving edges.
• A Hamming-windowed PSF reduces “ringing” in the
filtered image.
• Notch filtering reduces sinusoidal interference caused by
AC interference.
• Motion blur deconvolution undoes the blur caused by
camera motion.
PROBLEMS
199
Mathematical Formulae
1-D Hamming-windowed lowpass filter
hFIR [n] =
h

π n i
 Ωc sinc Ωc n
0.54 + 0.46 cos
π
π
N

0
Tikhonov regularization criterion
N−1 N−1
|n| ≤ N,
|n| > N
2-D Hamming-windowed lowpass filter
hFIR [n, m] = hFIR [n] hFIR [m]
e = ∑ ∑ [(g[n, m] − h[n, m] ∗ ∗ fˆ[n, m])2 + λ 2 fˆ[n, m]2 ]
n=0 m=0
Wiener filter
F̂[k1 , k2 ] = G[k1 , k2 ]
H∗ [k1 , k2 ]
|H[k1 , k2 ]|2 + λ 2
Deconvolution formulation
g[n, m] = h[n, m] ∗ ∗ f [n, m] + v[n, m]
Motion blur PSF
x − D/2
1
h(x, y) = rect
δ (y)
s
D
Deconvolution implementation by 2-D DFT
G[k1 , k2 ]
F[k1 , k2 ] =
H[k1 , k2 ]
Motion blur PSF
n − N/2
h[n, m] = rect
δ [m] ∆t
N/2
Important Terms
deconvolution
Provide definitions or explain the meaning of the following terms:
Hamming window
motion blur
PROBLEMS
Section 6-2: Denoising by Lowpass Filtering
6.1 The usual procedure for lowpass filtering an (N × N)
image f [n, m] is to set its 2-D DFT F[k1 , k2 ] = 0 for
K ≤ k1 , k2 ≤ N + 2 − K for some index K. Specify two problems
with using this brick-wall filtering approach.
6.2 Explain how a Hamming-windowed filter solves the problems in Problem 6.1.
6.3 This problem investigates denoising images by 2-D brickwall lowpass filtering. The program adds noise to the “clown”
image, then 2-D brick-wall lowpass filters it:
load clown.mat;
Y=X+0.2*randn(200,200);FY=fft2(Y);
FZ=FY;L=??;FZ(L:202-L,L:202-L)=0;
Z=real(ifft2(FZ));imagesc(Z),colormap(gray)
(a) Run this for L = 101, 20, and 10. Display the filtered
images.
(b) Discuss the tradeoffs involved in varying the cutoff frequency.
notch filter
Tikhonov criterion
Wiener filter
6.4 This problem denoises by 2-D lowpass filtering with a
separable 2-D lowpass filter h[m, n] = h[m] h[n], where h[n] is an
FIR lowpass filter designed by windowing the impulse response
of a brick-wall lowpass filter, which suppresses “ringing.”
(a) Design a 1-D lowpass filter h[n] of duration 31 by using a
Hamming window on the impulse response of a brick-wall
lowpass filter with cutoff frequency Ω0 = π3 .
(b) Filter the “letters” image by 2-D convolution with
h[m] h[n]. Display the result.
(c) Try varying the filter duration and cutoff frequency. See if
this improves the result.
Section 6-3: Notch Filtering
6.5 We derive a system for performing notch filtering of
the 2-D sinusoidal signal f [n, m] = cos((Ω1 )0 n) cos((Ω2 )0 m),
which is to be eliminated from an image. Let the PSF


1
0
1
h[n, m] = 0 h[0, 0] 0
1
0
1
where h[0, 0] = −4 cos((Ω1 )0 ) cos((Ω2 )0 ).
www.dbooks.org
200
CHAPTER 6
DETERMINISTIC APPROACH TO IMAGE RESTORATION
(a) Compute the spatial frequency response H(Ω1 , Ω2 ) of this
system.
(b) Show that it does indeed eliminate f [n, m].
(c) Specify a problem with using this LSI system as a notch
filter.
Add and subtract |GH∗ |2 , divide by (HH∗ + λ 2 ), and complete
the square.
6.6 Download file P66.mat. Use notch filtering to eliminate
stripes in the clown image. Print out the striped image, its
spectrum, and the notch-filtered image and its spectrum. Hint:
Let x[n] have length N and its N-point DFT X[k] have a peak
at k = k0 . From Eq. (2.88), the peak represents a sinusoid
with frequency Ω0 = 2π k0 /N. The sinusoid will repeat about
Ω/(2π )N = k0 times over the length N of x[n].
6.7 Download file P67.mat. Use notch filtering to eliminate
stripes in the head. Print out the striped image, its spectrum, and
the notch-filtered image and its spectrum. Hint: Let x[n] have
length N and its N-point DFT X[k] have a peak at k = k0 . From
Eq. (2.88), the peak represents a sinusoid with frequency Ω0 =
2π k0/N. The sinusoid will repeat about Ω/(2π )N = k0 times
over the length N of x[n].
6.8 Download file P68.mat. Use notch filtering to eliminate
two sets of lines. Note that there are horizontal lines on top of the
image, and vertical lines in the image. Use the procedure used in
Section 6-2, but in both horizontal and vertical directions. Print
out the original image, its spectrum, and the notch-filtered image
and its spectrum.
6.9 Download file P69.mat. Use notch filtering to eliminate
two sets of lines. Note that there are horizontal lines on top of the
image, and vertical lines in the image. Use the procedure used in
Section 6-2, but in both horizontal and vertical directions. Print
out the original image, its spectrum, and the notch-filtered image
and its spectrum.
Section 6-4: Image Deconvolution
6.10 Derive the Wiener filter by showing that the fˆ[n, m]
minimizing the Tikhonov functional
T = ∑ ∑[(g[n, m] − h[n, m] ∗ ∗ fˆ[n, m])2 + λ 2 ( fˆ[n, m])2 ]
n m
has 2-D DFT
F̂[k1 , k2 ] = G[k1 , k2 ]
H[k1 , k2 ]∗
.
|H[k1 , k2 ]|2 + λ 2
Hints: Use Parseval’s theorem and
|a + b|2 = aa∗ + ab∗ + ba∗ + bb∗.
6.11 This is an introductory image deconvolution problem
using a Wiener filter. A crude lowpass filter is equivalent to
convolution with PSF
(
1/L2
for 0 ≤ n, m ≤ L − 1,
h[n, m] =
0
otherwise.
This problem undoes this crude lowpass filter using a Wiener
filter with λ = 0.01.
(a) Blur the clown image with h[n, m] for L = 11 using:
clear;load clown;
H=ones(11,11)/121;Y=conv2(X,H);
(b) Deblur the blurred image using a Wiener filter using:
imagesc(Y),colormap(gray);
FY=fft2(Y);FH=fft2(H,210,210);
FZ=FY.*conj(FH)./(abs(FH).
*abs(FH)+.0001);
Z=real(ifft2(FZ));
figure,imagesc(Z),colormap(gray)
6.12 This is an introductory image deconvolution problem
using a Wiener filter. A crude lowpass filter is equivalent to
convolution with PSF
(
1/L2
for 0 ≤ n, m ≤ L − 1,
h[n, m] =
0
otherwise.
This problem undoes this crude lowpass filter using a Wiener
filter with λ = 0.01.
(a) Blur the letters image with h[n, m] for L = 15 using:
clear;load letters;H=ones(15,15)/225;
Y=conv2(X,H);
(b) Deblur the blurred image using a Wiener filter using:
imagesc(Y),colormap(gray);FY=fft2(Y);
FH=fft2(H,270,270);
FZ=FY.*conj(FH)./(abs(FH).
*abs(FH)+.0001);Z=real(ifft2(FZ));
figure,imagesc(Z),colormap(gray)
6.13 Deblurring due to an out-of-focus camera can be modelled crudely as a 2-D convolution with a disk-shaped pointspread function
(
1
for n2 + m2 < R2 ,
h[n, m] =
0
for n2 + m2 > R2 .
PROBLEMS
201
This problem deblurs an out-of-focus image in the (unrealistic)
absence of noise.
(a) Blur the letter image with an (approximate) disk PSF using
H(25,25)=0;for I=1:25;for J=1:25;
if((I-13)*(I-13)+(J-13)*(J-13)<145);
H(I,J)=1;end;end;end;
load letters;Y=conv2(X,H);
subplot(221),imagesc(Y),colormap(gray)
(b) Deblur this out-of-focus image using the command
Z=real(ifft2(fft2(Y)./fft2(H,280,280)));
subplot(222),imagesc(Z),colormap(gray)
Note that the size of the blurred image is 256 + 25 − 1
= 280.
(c) Explain why this approach will not work in the real world
(i.e., in the presence of noise).
6.14 Repeat Problem 6.13, only now add noise to the blurred
image:
(a) Add noise to the blurred image using
Y=Y+100*randn(280,280);
(b) Deblur the image as in Problem 6.13. You should get noise!
(c) Deblur the image using a Wiener filter, using
FH=fft2(H,280,280);
W=real(ifft2(fft(Y).*conj(FH)./
(abs(FH).*abs(FH)+10)));
subplot(221),imagesc(Z),colormap(gray)
subplot(222),imagesc(W),colormap(gray)
6.15 Repeat Problem 6.13 using the clown image. Note that
the size of the blurred image is now 200 + 25 − 1 = 224.
6.16 Repeat Problem 6.14 using the clown image. Note that
the size of the blurred image is now 200 + 25 − 1 = 224.
Add noise using Y=Y+randn(224,224); and use λ 2 = 100,
since the clown pixel values have a maximum value of only 1,
while the letters pixel values have a maximum value of 255.
(b) followed by, for an additional Ty − Tx s (for Tx < t < Ty ),
(c) vertical, in increasing y, at speed ry cm/s for Ty − Tx s.
Compute the spatial frequency response of the camera motion.
6.19 Download file P619.mat. The goal is to deconvolve the
motion blur.
(a) Compute and display the spectrum of the blurred image.
What causes the vertical bands of zeros (VBZ)? Hint: See
the numerator of Eq. (6.45).
(b) From the spacing between the VBZ, compute N in
Eq. (6.45).
(c) Deconvolve the image using the Wiener filter Eq. (6.46).
Let T = 1 and λ = 0.01.
6.20 Download file P620.mat. The goal is to deconvolve the
motion blur.
(a) Compute and display the spectrum of the blurred image.
What causes the vertical bands of zeros (VBZ)? Hint: See
the numerator of Eq. (6.45).
(b) From the spacing between the VBZ, compute N in
Eq. (6.45).
(c) Deconvolve the image using the Wiener filter Eq. (6.46).
Let T = 1 and λ = 0.01.
6.21 Download file P621.mat. The goal is to deconvolve the
motion blur. The blurred image in this problem is the SAR image
from Chapter 4.
(a) Compute and display the spectrum of the blurred image.
What causes the vertical bands of zeros (VBZ)? Hint: See
the numerator of Eq. (6.45).
(b) From the spacing between the VBZ, compute N in
Eq. (6.45).
(c) Deconvolve the image using the Wiener filter Eq. (6.46).
Let T = 1 and λ = 0.01.
Section 6-6: Motion Blur Deconvolution
6.17 The motion sensor in a Steadicam camera records its
motion as being in increasing x and y at an angle θ from the
horizontal (x) at speed r cm/s for T s for a diagonal distance
of D = rT cm. Compute the spatial frequency response of the
camera motion.
6.18 The motion sensor in a Steadicam camera records its
motion as being:
(a) horizontal, in increasing x, at speed rx cm/s for Tx s,
www.dbooks.org
Chapter
Wavelets and Compressed
Sensing
7
Contents
7-1
7-2
7-3
7-4
7-5
7-6
7-7
7-8
7-9
7-10
7-11
7-12
7
Overview, 203
Tree-Structured Filter Banks, 203
Expansion of Signals in Orthogonal Basis
Functions, 206
Cyclic Convolution, 209
Haar Wavelet Transform, 213
Discrete-Time Wavelet Transforms, 218
Sparsification Using Wavelets of PiecewisePolynomial Signals, 223
2-D Wavelet Transform, 228
Denoising by Thresholding and Shrinking, 232
Compressed Sensing, 236
Computing Solutions to Underdetermined
Equations, 238
Landweber Algorithm, 241
Compressed Sensing Examples, 242
Problems, 251
20
40
60
80
100
120
140
160
180
200
20
40
60
80 100 120 140 160 180 200
(a) 200 × 200 clown image
20
40
60
80
100
120
140
Objectives
Learn to:
■ Compute wavelet transforms of 1-D signals and 2-D
images.
■ Design discrete-time Daubechies wavelets of various
orders.
■ Use wavelet transforms to compress signals and
images.
■ Use thresholding and shrinkage of its wavelet
transform to denoise an image.
■ Use compressed sensing to reconstruct an image
from a reduced set of measurements.
160
180
200
20
40
60
80 100 120 140 160 180 200
(b) 2-D D3 Daubechies wavelet transform
of the clown image
The wavelet transform is an important tool. Its
applications in image processing include: denoising
while preserving edges, compression, and compressed sensing, which is reconstruction of an
image from a reduced set of linear measurements of
it. After a review of the 1-D discrete-time Haar and
Daubechies wavelet transforms, we present applications of them to denoising, compression, and compressed sensing, including image inpainting and
X-ray tomography (CAT).
Overview
7-1 Tree-Structured Filter Banks
The wavelet transform is an important signal processing tool
for representing signals or images consisting mostly of slowly
varying regions, but containing a few fast-varying regions. Like
the discrete Fourier transform (DFT), it represents signals or
images as a linear combination of basis functions.
One characteristic of the DFT is that images, even with only
a few fast-varying segments such as edges, require high spatial
frequency complex exponentials to represent them. Hence, most
or all of the DFT values X[k1 , k2 ] are nonzero. In contrast, the basis functions used in wavelet transforms are localized in time and
frequency. This means that a signal that is mostly slowly-varying
but has a few localized fast-varying regions requires only a few
low-resolution basis functions to represent the slowly-varying
regions, and a few high-resolution basis functions to represent
just the localized fast-varying regions. Many of the wavelet
transform values are thus zero (or near zero). This feature leads
to the following three major applications of wavelet transforms:
Wavelet transforms can be viewed as a generalization of treestructured filter banks (TSFBs) and subband coding. A filter
bank is a set of bandpass filters connected in parallel; each
bandpass filter passes a different range of frequencies. So the
signal input into the filter bank is separated into different
components, each of which consists of a different part of the
spectrum of the input signal.
Filter banks are used in audio signal processing. In human
hearing, some frequencies cannot be heard as well as others.
Also a large component at one frequency can mask a component
at another frequency. So it makes sense to keep only the
frequency bands that humans can hear. The basic idea behind
coding of signals is to omit the frequency bands that contribute
little to the perception of the signal, and the process is called
subband coding. The mp3 coding of music uses this idea (and
many others).
An efficient tree-like filter structure for separating a 1-D
sampled signal into different frequency bands (subband decomposition) is shown in Fig. 7-1, in which g[n] is a lowpass filter
with cutoff frequency Ωc = π /2 and h[n] is a highpass filter with
the same cutoff frequency. The concept is easily extendable to
2-D images.
Filter-bank diagrams may involve five types of operations:
• Compression of signals and images: they are represented
in the wavelet transform domain by many fewer numbers
than in the original signal or image. The JPEG-2000 image
compression standard uses the wavelet transform.
• Compressed sensing of signals and images: since the signal
or image in the wavelet transform domain requires many
fewer numbers to represent it, it can be reconstructed
from many fewer observations than would be required to
reconstruct the original signal or image. An introduction to
compressed sensing is presented later in this chapter.
• Filtering of signals and images: since the signal or image
in the wavelet transform domain requires many fewer numbers to represent it, thresholding small values of the wavelet
transform of a noisy signal or image to zero reduces the
noise in the original signal or image. We will show that the
combination of thresholding and shrinkage gives results
far superior to using the 2-D DFT for noise reduction.
After this Overview section, we present the Haar wavelet
transform, which is the simplest wavelet transform, and yet
illustrates many features of the family of wavelet transforms.
We then present quadrature mirror filters (QMFs) and derive
the Smith-Barnwell condition for perfect reconstruction of
the original signal from its wavelet transform. We conclude
our treatment of wavelets by deriving the Daubechies wavelet
function, which is the most commonly used wavelet function
because it sparsifies many real-world signals. Finally, examples
of image compression, denoising, and compressed sensing are
provided.
(1) Duplication:
x[n]
x[n]
x[n]
(2) Addition:
y1[n]
z[n] = y1[n] + y2[n]
y2[n]
(3) Downsampling (decimation):
x[n]
yd [n] = x[2n]
2
Discarding every other sample in x[n].
(4) Upsampling (zero-stuffing):
x[n]
203
www.dbooks.org
2
yu [n] =
(
x[n/2]
0
for n even
for n odd
204
CHAPTER 7
g[n]
xL[n]
2
xLD[n]
WAVELETS AND COMPRESSED SENSING
g[n]
2
xLDLD[n]
h[n]
2
xLDHD[n]
g[n]
2
xHDLD[n]
h[n]
2
xHDHD[n]
x[n]
h[n]
xH[n]
2
xHD[n]
Figure 7-1 Tree-like filter structure for subband decomposition. The green boxes denote lowpass and highpass frequency filters, realized
through cyclic convolution.
Inserting a zero between successive values of x[n].
(5) Convolution:
x[n]
h[n]
y[n] = h[n] ∗ x[n]
In Fig. 7-1, the input signal x[n] with spectrum (DTFT)
X(Ω) is separated into a low-frequency-band signal xL [n] whose
spectrum is roughly
(
X(Ω)
for 0 ≤ |Ω| < π /2,
(7.1)
XL (Ω) =
0
for π /2 < |Ω| ≤ π ,
and a high-frequency-band signal xH [n] whose spectrum is
roughly
(
0
for 0 ≤ |Ω| < π /2,
XH (Ω) =
(7.2)
X(Ω)
for π /2 < |Ω| ≤ π .
Each signal can be downsampled by 2 without aliasing, resulting
in xLD [n] = xL [2n] and xHD [n] = xH [2n]. There are now two
different signals, each of which is sampled only half as often
as x[n], so the total number of samples is unaltered, and each
represents a different frequency band of the original signal.
This same decomposition can then be applied to each of
the two downsampled signals xLD and xHD , which results in
four signals, each of which is sampled only one fourth as
often as x[n], so the total number of samples is the same, and
each represents a different frequency band of bandwidth π /4.
Repeating this decomposition N times, x[n] can be decomposed
into 2N signals, each of which represents a different frequency
band of bandwidth π /2N and is sampled only 1/2N as often as
x[n]. Use of N = 5, resulting in 25 = 32 subbands, is a common
choice.
In Fig. 7-1:
• xLD [n] is the lowpass part, 0 ≤ |Ω| ≤ π2 , of x[n].
• xHD [n] is the highpass part, π2 ≤ |Ω| ≤ π , of x[n].
The signals at the second stage of the filter bank have spectra
that are roughly as follows:
• xLDLD [n] is the lowpass part, 0 ≤ |Ω| ≤ π2 , of xLD [n], which
is equivalent to the lowpass part, 0 ≤ |Ω| ≤ π4 , of x[n].
• xLDHD [n] is the highpass part, π2 ≤ |Ω| ≤ π , of xLD [n],
which is equivalent to the bandpass part π4 ≤ |Ω| ≤ π2 of
x[n].
If we were to extend the filter bank in Fig. 7-1 to another stage,
the signals at the third stage would have spectra that are roughly:
xLDLDLD [n] is the lowpass part 0 ≤ |Ω| ≤ π2 , of xLDLD [n], which
is equivalent to the lowpass part, 0 ≤ |Ω| ≤ π8 , of x[n].
xLDLDHD [n] is the highpass part, π2 ≤ |Ω| ≤ π , of xLDLD [n],
which is equivalent to the bandpass part π8 ≤ |Ω| ≤ π4 of x[n].
At each stage, decimation (halving the sampling rate) expands
the spectrum of each signal to the full range 0 ≤ |Ω| < π (see
Section 7-1.1).
7-1.1 Octave-Based Filter Banks
The tree structure in Fig. 7-1 can be replaced with the simpler
structure shown in Fig. 7-2. As we show later in Section 7-5,
7-1 TREE-STRUCTURED FILTER BANKS
g[n]
xL[n]
205
2
xLD[n]
xLDL[n]
g[n]
xLDH[n]
h[n]
2
xLDLD[n]
2
xLDHD[n]
x[n]
h[n]
xH[n]
2
xHD[n]
Figure 7-2 Octave-based filter bank structure for subband decomposition. Note that only the upper half (lowpass) of each stage is
decomposed further.
the wavelet transform is implemented using this simpler tree
structure.
This form of filter bank decomposes the spectrum X(Ω) of
x[n] into octaves. An octave is a frequency band fmin < f < fmax
in which fmax = 2 fmin , using either continuous-time frequency
f in Hz or discrete-time frequency Ω. We use Ω in the sequel, since the signals are all in discrete time. The octave
π /2K ≤ Ω ≤ π /(2K−1) is represented by a single signal. Since
the width of this band is π /2K , the sampling rate can be reduced
by a factor of 2K using decimation by 2 at each of K stages. The
lowest band 0 ≤ Ω ≤ π /2K is represented by a single signal, as
shown in Fig. 7-3 for K = 2.
xLDLD[n]
band
0
xLDHD[n]
band
π/4
π/2
xHD[n] band
π
denoted as
x[n]
2
yu [n] =
(
x[n/2]
0
for n even,
for n odd,
(7.3)
which gives
yu [n] = { . . . , x[0], 0, x[1], 0, x[2], 0, x[3], . . . },
followed by interpolation using a filter g[−n] or h[−n] to replace
the zeros introduced by zero-stuffing with the correct values of
the signal input into that filter-bank stage. For octave-based filter
banks, the time reversals are unnecessary, but time reversals are
necessary when using this same structure for the inverse wavelet
transform introduced later in Section 7-5.
Zero-stuffing is presented in more detail in Subsection 7-4.3.
Ω
Figure 7-3 Partitioning of signal spectrum by octave-based
filter bank for K = 2.
7-1.3 Significance of Wavelets
7-1.2 Reconstruction Using Octave-Based Filter
Banks
The original signal x[n] can be recovered from its octave-based
frequency decomposition using the reconstruction filter bank
shown in Fig. 7-4. The inverse wavelet transform in Section 7-5
is implemented using this same structure. Each filter bank stage
consists of zero-stuffing (inserting zeros between samples),
The wavelet transform differs from this subband coding in that
x[n] is not recursively decomposed explicitly into lower and
higher frequency bands, although the decomposition is still
roughly into lower and higher frequency bands. Instead, the
lower frequency bands are replaced with signals that represent
slowly-varying parts of x[n], and the higher frequency bands
are replaced with signals that represent fast-varying parts of
x[n]. The latter signals are mostly zero-valued if x[n] is slowlyvarying most of the time.
www.dbooks.org
206
CHAPTER 7
xLDLD[n]
2
g[−n]
xLD[n]
xLDHD[n]
2
WAVELETS AND COMPRESSED SENSING
2
g[−n]
x[n]
h[−n]
xHD[n]
2
h[−n]
Figure 7-4 Octave-based filter bank structure for subband reconstruction.
◮ Real-world signals and images do tend to consist of
mostly slowly-varying regions, containing a few localized
regions in which they are fast-varying. Wavelets are good
at representing such signals with wavelet transforms that
are mostly zero-valued. ◭
To see why representing a signal using only a few wavelet
transform components is useful, consider periodic signals. A
periodic signal x(t) with period T0 and maximum frequency B
(in Hz) can be represented in the frequency domain using only
BT0 frequencies, since its spectrum consists of harmonics at
frequencies k/T0 Hz for integers k. The maximum frequency B
must equal N/T0 for some integer N, or equivalently, N = BT0
frequencies. Instead of storing x(t), we can generate it using BT0
sinusoidal generators, each of which requires only an amplitude
and phase. So x(t) can be compressed into 2BT0 (plus a dc term,
if present) numbers.
If noise had been added to x(t), most of the noise can be
eliminated because any part of the spectrum of the noisy x(t) that
is not at a harmonic k/T0 in Hz is noise and can be filtered out.
We will perform similar actions on signals and images that are
not periodic, but which have wavelet transforms that are mostly
zero-valued. This includes many real-world signals and images.
Concept Question 7-1: Why is it useful to represent a
signal or image using a few wavelet coefficients?
Exercise 7-1: (a) An input signal of duration N is fed
into a tree-based filter bank of five stages. What is the
combined total duration of the output signals? Repeat for
(b) an octave-based filter bank.
(a) 25 N = 32N. (b) N, because lowerwavenumber bands can be sampled less often.
Answer:
7-2 Expansion of Signals in Orthogonal
Basis Functions
7-2.1 Signal Expansion
The general form of the expansion of a signal x[n] into a linear
φk [n] } with coefficients { xk }
combination of basis functions {φ
(often called an orthogonal expansion of x[n]) is
∞
x[n] = ∑ xk φ k [n].
(7.4)
k=1
The coefficients {xk } constitute the transform of x[n] using
φk [n] }. For wavelet transforms, the basis
the basis functions {φ
φk [n] } and coefficients { xk } are real-valued, but for
functions {φ
many other orthogonal expansions, such as the DFT and Fourier
series, they are complex-valued. We assume here that the signal
x[n] to be expanded is real-valued.
φk [n] } are chosen to be orthogonal, which
Basis functions {φ
means that, for some constant C,
∞
∑ φ k1 [n] φ ∗k2 [n] = Cδ [k1 − k2].
n=−∞
(7.5)
7-2 EXPANSION OF SIGNALS IN ORTHOGONAL BASIS FUNCTIONS
An important consequence of the orthogonality property is that
coefficients xk can be computed from x[n] using
xk =
1 ∞
∑ x[n] φ ∗k [n].
C n=−∞
(7.6)
The other part of the significance is Rayleigh’s theorem (Section
7-2.3).
207
function given by Eq. (7.4) and its inverse in Eq. (7.6). Table 7-1
compares attributes of the generic orthogonal expansion basis
function with those of the DFT and the continuous-time Fourier
series.
◮ If C = 1 in Eq. (7.6), the orthogonal basis functions are
said to be orthonormal. The wavelet transform (Section
7-5) uses orthonormal basis functions. ◭
7-2.2 Expansion Coefficients
7-2.3 Parseval’s Theorem and its Significance
Equation (7.6) can be derived as follows:
1. In Eq. (7.4), change index k to index k2 , giving
∞
x[n] = ∑ xk2 φ k2 [n].
(7.7)
k2 =1
2. Upon multiplying both sides of Eq. (7.7) by φ ∗k1 [n] and
summing over index n, we have
∞
∞
Z ∞
−∞
(7.8)
E=
n=−∞ k2 =1
n=−∞
3. Interchanging the order of summations in the right side of
Eq. (7.8) leads to
∞
∞
∞
n=−∞
k2 =1
n=−∞
∑ x[n] φ ∗k1 [n] = ∑ xk2 ∑ φ k2 [n] φ ∗k1 [n].
∑
n=−∞
x[n] φ ∗k1 [n] =
(7.9)
Z ∞
−∞
∞
n=−∞
(7.10)
k2 =1
X[k] = ∑ x[n] e
,
n=0
1 N−1
x[n] =
∑ X[k] e j2π nk/N ,
N k=0
k = 0, . . . , N − 1,
Z ∞
−∞
|X( f )|2 d f ,
1
2π
Z π
−π
|X(Ω)|2 dΩ,
∞
∞
n=−∞
k=−∞
∑ |x[n]|2 = C ∑ |xk |2
A good example of an orthogonal expansion is the 1-D DFT
and its inverse defined in Eq. (2.89) for a finite-length signal
x[n]:
− j2π nk/N
X( f ) Y∗ ( f ) d f .
(7.12a)
(7.12b)
(7.13)
where X(Ω) is the DTFT of x[n].
The statements given by Eqs. (7.12) and (7.13) can be
generalized to the generic orthogonal basis function expressed
in Eq. (7.4):
Dividing by C and replacing k1 with k gives Eq. (7.6).
M−1
−∞
|x(t)|2 dt =
∑ |x[n]|2 =
∞
∑ xk2 C δ [k2 − k1] = C xk1 .
Z ∞
which states that the energies of x(t) and X( f ) are equal.
Similarly, Rayleigh’s theorem for a discrete-time signal x[n]
is given by Eq. (2.80) as
4. Using the orthogonality property given by Eq. (7.5) gives
∞
x(t) y∗ (t) dt =
The special case wherein x(t) = y(t) is known as Rayleigh’s
theorem:
∞
∑ x[n] φ ∗k1 [n] = ∑ ∑ xk2 φ k2 [n] φ∗k1 [n].
For two continuous-time signals x(t) and y(t) and their associated Fourier transforms X( f ) and Y( f ), Parseval’s theorem is
stated in Eq. (2.27) as
(7.14a)
(Rayleigh’s theorem),
∞
∞
n=−∞
k=−∞
∑ x[n] y[n]∗ = C ∑ xk y∗k
(7.11a)
(7.14b)
(Parseval’s theorem),
n = 0, . . . , M − 1.
(7.11b)
The ranges in the summations in Eqs. (7.11) are particular to the
DFT and differ from those in the generic definition of the basis
where x[n] and y[n] are any two discrete-time functions. Our
interest in this book is in real-valued 2-D images, so the complex
conjugation on x[n] and y[n] in Eq. (7.14a) is irrelevant, but we
have decided to retain it for the sake of completeness.
www.dbooks.org
208
CHAPTER 7
WAVELETS AND COMPRESSED SENSING
Table 7-1 1-D DFT and Fourier series compared with generic orthogonal expansion function.
Generic Orthogonal Function
DFT
Fourier Series
φ k [n]
e j2π kn/N
e jnω0 t
Basis function
∞
x[n] =
k=1
xk =
Coefficients
∞
1 N−1
∑ X[k] e j2π nk/N ,
N k=0
n = 0, . . . , M − 1
x[n] = ∑ xk φ k [n]
Expansion
1 ∞
∑ x[n] φ∗k [n]
C n=−∞
Orthogonality property
M−1
X[k] = ∑ x[n] e− j2π nk/N ,
xn =
n=0
∑ φk [n] φ∗k [n] = Cδ [k1 − k2 ]
1
n=−∞
2
y[n] = x[n] + ε [n].
Z T0
N−1
1
∑ e j2π (m−n)k/N = δ [m − n]
N k=0
For an orthonormal set of basis functions with C = 1, Rayleigh’s theorem states that the energy of x[n], summed over all n,
is equal to the energy of coefficients { xk }, summed over all k.
The statement is equally applicable to small perturbations in
total energy. Consider, for example, a small perturbation ε [n]
from signal x[n]. The perturbed signal is
0
n=−∞
k = 0, . . . , N − 1
∞
∑ xn e jnω t
x(t) =
0
1
T0
Z T0
0
x(t) e− jnω0 t dt
e jmω0 t e− jnω0 t dt = T0 δ [m − n]
equal to the energy of the transform coefficients { ε k }.
(7.15)
Signals x[n] and y[n] can each be expanded as:
∞
x[n] = ∑ xk φk [n],
(7.16a)
k=1
∞
y[n] = ∑ yk φk [n].
◮ An orthonormal transformation does not amplify perturbations, a property that will prove highly significant to the
application of wavelets for denoising images. ◭
(7.16b)
k=1
Use of Eqs. (7.16a) and (7.16b) in Eq. (7.15) leads to
∞
ε [n] = y[n] − x[n] = ∑ (yk − xk ) φk [n] = ∑ ε k φk [n], (7.17)
k=1
k=1
Concept Question 7-2:
What is the significance of Ray-
leigh’s theorem?
where
ε k = yk − xk .
Application of Rayleigh’s theorem, as stated by Eq. (7.14b), to
sampled signal ε [n] leads to
∞
∞
n=∞
k=∞
∑ |ε [n]|2 = ∑ |ε k |2 ,
(7.18)
which confirms that the energy of the perturbation ε [n] of x[n] is
Exercise 7-2: A square wave x(t) has the Fourier series
1
expansion x(t) = ∑∞
k=1 k sin(2π kt). If x(t) is passed through
a brick-wall lowpass filter with a cutoff frequency of 2.5 Hz.
What is the ratio of the average power of the output signal
to the average power of x(t)? Hint:
∞
1
π2
∑ k2 = 6 .
k=1
7-3 CYCLIC CONVOLUTION
209
and
h[n] = { h[0], h[1], . . . , h[N2 ] }.
Answer: By Rayleigh’s theorem, the average power of x(t)
is
∞
1
(7.19b)
Their linear convolution is
π2
∑ k2 = 6 ,
∞
y[n] = h[n] ∗ x[n] = ∑ h[n] x[n − i].
k=1
and the average power of the output signal is
(7.20)
i=∞
If x[n] has
2
1
∑ k2 = 1.25.
k=1
This is because the lowpass filter sets the Fourier series
coefficients for k ≥ 3 to zero.
1.25
= 0.76.
π 2 /6
support: Nxℓ ≤ n ≤ Nxu , and
duration: Nx = Nxu − Nxℓ + 1,
where second subscripts ℓ and u refer to the lower and upper
values of Nx , and if h[n] has
support: Nhℓ ≤ n ≤ Nhu , and
duration: Nh = Nhu − Nhℓ + 1,
7-3 Cyclic Convolution
(7.21a)
(7.21b)
then their linear convolution y[n] has
support: Nyℓ ≤ n ≤ Nyu , and
duration: Nyu − Nyℓ + 1 = Nx + Nh − 1,
7-3.1 Why Use Cyclic Convolutions?
In Section 2-7.2, we introduced the concept of cyclic convolution x1 [n] c x2 [n] between two signals x1 [n] and x2 [n], and
we showed how it can be computed from the traditional linear
convolution x1 [n] ∗ x2 [n], as demonstrated in Example 2-6, or by
applying the DFT method.
The wavelet transform—the prime topic of this chapter—
employs convolution, decimation, and zero-stuffing. If we use
linear convolutions in computing the wavelet transform, the
total length of the decimated signal will be longer than that of
the original signal. At each stage in Fig. 7-1 or Fig. 7-2, for
example, the linear convolution with h[n] or g[n] would generate
a new signal longer than that of the input signal by the length
of h[n] or g[n], respectively. The advantage of cyclic convolution
is that the new signal remains at the same length as that of the
input signal. This property limits the computational storage to
the same storage required for the original signal.
As noted, the cyclic convolution can be computed directly
from the linear convolution, or indirectly by applying the DFT
method. In preparation for the material presented in forthcoming
sections, we present reviews of both computational approaches.
(7.21c)
(7.21d)
where
Nyℓ = Nxℓ + Nhℓ,
Nyu = Nxu + Nhu .
(7.21e)
(7.21f)
Graphically, these supports and associated durations are:
x[n]:
Nx = Nxu − Nxl + 1
Nxl
Nh = Nhu − Nhl + 1
h[n]:
y[n]:
Nxu
Nhl
Nhu
Ny = Nx + Nh − 1
Nyl = Nxl + Nhl
Nyu = Nxu + Nhu
A. Causal * Causal
7-3.2 Computing Linear Convolution
Suppose we are given a causal signal x[n] and a causal filter (or
another signal) h[n] defined as
x[n] = { x[0], x[1], . . . , x[N1 ] },
If x[n] and h[n] are both causal signals, with Nxℓ = 0 and Nhℓ = 0,
then Eq. (7.20) simplifies to
(7.19a)
www.dbooks.org
n
y[n] = ∑ h[i] x[n − i],
i=0
0 ≤ n ≤ Nyu .
(7.22)
210
CHAPTER 7
Nx = Nxu + 1
x[n]:
(2.106) and (2.107) as
Nxu
0
yc [n] = h[n] c x[n] = y[n] + y[(n)N ],
n = 0, 1 . . . , N − 1,
Nh = Nhu + 1
h[n]:
Nhu
0
Nxu + Nhu
0
B. Anticausal * Causal
If x[n] is causal with Nxℓ = 0 and h[n] is anticausal with its upper
limit Nhu = 0, then Eq. (7.20) simplifies to
A. Causal * Causal
If both h[n] and x[n] are causal, so that
Nxd = Nhd = 0,
n
y[n] = ∑ h[i] x[n − i],
i=Nhℓ
y[n]:
(7.23)
then Eq. (7.25) simplifies to
Nx = Nxu + 1
x[n]:
h[n]:
Nhℓ ≤ n ≤ Nxu .
yc [n] = y[n] + y[n + N],
Nxu
0
0
Ny = Nx + Nh − 1 = Nxu + |Nhl| + 1
Nyl = Nhl
Nyu = Nxu
An easy way to compute the convolution given by Eq. (7.23) is
to use the time-shift property of convolution (# 5 in Table 2-6):
(7.26)
B. Anticausal * Causal
If x[n] is causal and h[n] is anticausal, so that Nxℓ = 0 and
Nhu = 0, then Eq. (7.25) simplifies to
h[n] ∗ x[n] = y[n]
h[n − n1] ∗ x[n − n2] = h[n − n1 − n2],
n = 0, 1, . . . , N − 1,
where, by definition, y[n + N] = 0 for n > Nyu − N.
The values {y[N], y[N + 1], . . . , y[Nyu ]} of y[n] for n ≥ N
get added point-by-point to the given values {y[0], y[1], . . .,
y[Nyu − N]}. The process is illustrated graphically in Fig. 7-5(a).
If N > Nyu , the cyclic convolution equals the linear convolution.
Nh = |Nhl| + 1
Nhl
(7.25)
where y[n] is the linear convolution of x[n] and h[n], and y[(n)N ]
is y[n] at values of n outside the range n = 0, 1, . . . , N − 1,
with those values of n reduced mod(N); i.e., the remainders
after reducing n by the largest multiple of N without becoming
negative.
Ny = Nx + Nh − 1
y[n]:
WAVELETS AND COMPRESSED SENSING
(7.24)
for any two integers n1 and n2 . The procedure involves the
following steps:
(1) Keeping in mind that Nhℓ is a negative integer, delay
the anticausal signal h[n] by |Nhℓ | to obtain the causal signal
h[n − |Nhℓ|].
(2) Compute h[n − |Nhℓ |] ∗ x[n] using Eq. (7.22), the convolution expression for two causal signals.
(3) Advance h[n − |Nhℓ|] ∗ x[n] by |Nhℓ |.
yc [n] = y[n] + y[n − N],
n = 0, 1, . . . , N − 1,
where we set y[n − N] = 0 for n < Nyℓ + N and we assume
N ≥ Nyℓ . If N < Nyℓ , we must also add the values of {y[n],
N ≤ n ≤ Nyℓ } to y[n] as described in the causal*causal subsection. This situation seldom happens.
The anticausal values {y[Nyℓ ], . . . , y[−1]} of y[n] for n < 0
get added point-by-point to the given values {y[N − |Nyℓ |], . . . ,
y[N − 1]}. The process is illustrated graphically in Fig. 7-5(b).
Example 7-1:
Cyclic Convolution
7-3.3 Computing Cyclic Convolution
The cyclic convolution yc [n] of order N ≥ Nx , Nh of signals x[n]
and h[n] as specified by Eq. (7.19), was defined in Eqs. (2.104),
(7.27)
Given
x[n] = {3, 4, 5, 6, 7, 8},
7-3 CYCLIC CONVOLUTION
211
First N terms
Additional terms if N < Nyu
Linear
y[n] = y[0] y[1] ... y[ ] ... y[N − 1] y[N] y[N + 1]
+
+
+
Cyclic
yc[n] = y[0] + y[N] y[1] + y[N + 1] ... y[N − 1]
... y[Nyu]
(a) Causal * causal
Anticausal terms
Causal terms
Linear
y[n] = y[Nyl] ... y[−2] y[−1] y[0] ... y[ ] ... y[Nyu − 1] y[Nyu]
+
+
+
Causal
yc[n] = y[0] y[1] ... y[Nyu − 1] + y[−2] y[Nyu] + y[−1]
(b) Anticausal * causal
Figure 7-5 Graphical representation of obtaining cyclic convolution of order N from linear convolution.
h1 [n] = {1, 2, 3},
h2 [n] = h1 [−n] = {3, 2, 1},
and N = 6, compute (a) yc1 = h1 [n] c x[n] and (b)
yc2 = h2 [n] c x[n].
yc1 [n] = h1 [n] c x[n]
= y1 [n] + y1[n + N]
= {3 + 37, 10 + 24, 22, 28, 34, 40}
= {40, 34, 22, 28, 34, 40}.
The same result can be obtained by computing the cyclic
convolution with the DFT (see Eq. (2.104)):
Solution: (a)
y1 [n] = h1 [n] ∗ x[n]


1×3 = 3



1 × 4 + 2 × 3 = 10
= 1 × 5 + 2 × 4 + 3 × 3 = 22




 ...
for n = 0,
for n = 1,
for n = 2
ifft(fft([3,4,5,6,7,8]).*fft([1,2,3],6))
= {3, 10, 22, 28, 34, 40, 37, 24},
www.dbooks.org
212
CHAPTER 7
WAVELETS AND COMPRESSED SENSING
which is equivalent to multiplying x[n] by 21 (1 + (−1)n):
(b)
y2 [n] = h2 [n] ∗ x[n]


3×3 = 9



3 × 4 + 2 × 3 = 18
= 3 × 5 + 2 × 4 + 1 × 3 = 26




 ...
x[n]
for n = −2,
for n = −1,
for n = 0,
= {9, 18, 26, 32, 38, 44, 23, 8}.
This is {3, 2, 1} ∗ {3, 4, 5, 6, 7, 8} advanced in time by 2.
yc2 [n] = h2 [n] c x[n]
= y1 [n] + y1[n − N]
= {26, 32, 38, 44, 23 + 9, 8 + 18}
= {26, 32, 38, 44, 32, 26}.
Again, the same result can be obtained by computing the cyclic
convolution with the DFT method:
ifft(fft([3,4,5,6,7,8]).*fft([1,0,0,0,3,2]))
2
1
(1 + (−1)n) x[n].
2
2
(7.30)
7-3.5 A Useful Convolution Relation
Recall from Eq. (7.20) that the discrete-time linear convolution
is defined as
∞
y[n] = h[n] ∗ x[n] = ∑ h[i] x[n − i].
(7.31)
i=−∞
Let us consider the modified functions:
h′ [n] = (−1)n h[n],
(7.32a)
′
(7.32b)
n
x [n] = (−1) x[n].
The linear convolution of the modified functions is
∞
y′ [n] = h′ [n] ∗ x′ [n] = ∑ h′ [i] x′ [n − i]
i=−∞
∞
= ∑ (−1)i h[i] (−1)n−i x[n − i]
7-3.4 Decimating and Zero-Stuffing
Decimating 1-D signals by 2, and zero-stuffing 1-D signals
by 2, are essential parts of computing discrete-time wavelet
transforms. Hence, we present this quick review of these two
concepts.
Decimating a signal x[n] by two means deleting every other
value of x[n]:
yd [n] = x[2n] = { . . . , x[0], x[2], x[4], . . . }.
(7.28)
Zero-stuffing a signal x[n] by two means inserting zeros between
successive values of x[n]:
(
x[n/2]
for n even,
yu [n] =
x[n]
2
0
for n odd,
x[n]
2
= { . . . , x[0], 0, x[1], 0, x[2], 0, x[3], . . . }.
(7.29)
Decimating by 2, followed by zero-stuffing by 2, replaces x[n]
with zeros for odd times n:
x[n]
2
2
{ . . . , x[0], 0, x[2], 0, x[4], 0, x[6], . . . },
i=−∞
∞
= ∑ (−1)n h[i] x[n − i]
i=−∞
= (−1)n
∞
∑ h[i] x[n − i]
i=−∞
n
= (−1) y[n].
(7.33)
Combining the result given by Eq. (7.33) with the definition of
convolution given in Eq. (2.71a) leads to the conclusion:
(−1)n { h[n] ∗ x[n] } = { (−1)n h[n] } ∗ { (−1)n x[n] }.
(7.34)
7-3.6 Wavelet Applications
For applications involving wavelets, the following conditions
apply:
• Batch processing is used almost exclusively. This is because the entire original signal is known before processing
begins, so the use of non-causal filters is not a problem.
• The order N of the cyclic convolution is the same as the
duration of the signal x[n], which usually is very large.
7-4 HAAR WAVELET TRANSFORM
213
• Filtering x[n] with a filter h[n] will henceforth mean computing the cyclic convolution h[n] c x[n]. The duration L
of filter h[n] is much smaller than N (L ≪ N), so h[n] gets
zero-padded (see Section 2-7.3) with (N − L) zeros. The
result of the cyclic convolution is the same as h[n] ∗ x[n],
except for the first (L − 1) values, which are aliased, and
the final (L − 1) values, which are no longer present, but
added to the first (L − 1) values.
• Filtering x[n] with the non-causal filter h[−n] gives the
same result as the linear convolution h[−n] ∗ x[n], except
that the non-causal part of the latter will alias the final
(L − 1) places of the cyclic convolution.
7-4 Haar Wavelet Transform
The Haar transform is by far the simplest wavelet transform,
and yet it illustrates many of the concepts of how the wavelet
transform works.
7-4.1 Single-Stage Decomposition
Consider the finite-duration signal x[n]
x[n] = { a, b, c, d, e, f , g, h }.
Define the lowpass and highpass filters with impulse responses
ghaar [n] and hhaar [n], respectively, as
• Zero-padding does not increase the computation, since
multiplication by zero is known to give zero, so it need not
be computed.
• For two filters g[n] and h[n], both of length L, g[n] c h[n]
consists of g[n] ∗ h[n] followed by N − (2L − 1) zeros.
• As long as the final result has length N, linear convolutions
may be replaced with cyclic convolutions and the final
result will be the same.
Exercise 7-3: Compute the cyclic convolution of {1, 2} and
{3, 4} for N = 2.
Answer: {1, 2} ∗ {3, 4} = {3, 10, 8}. Aliasing the output
gives {8 + 3, 10} = {11, 10}.
Exercise 7-4: x[n]
2
Express y[n] in terms of x[n].
2
y[n].
Answer: y[n] = x[n]. Zero stuffing inserts zeros between
consecutive values of x[n] and decimation removes those
zeros, thereby restoring the original x[n].
Exercise 7-5: If y[n] = h[n] ∗ x[n], what is
h[n](−1)n ∗ x[n](−1)n
(7.35)
1
ghaar [n] = √ { 1, 1 },
2
1
hhaar [n] = √ { 1, −1 }.
2
(7.36a)
(7.36b)
The frequency responses of these filters are the DTFTs given by
√
1
Ω − jΩ/2
− jΩ
√
e
, (7.37a)
(1 + e
) = 2 cos
Ghaar (Ω) =
2
2
√
1
Ω
Hhaar (Ω) = √ (1 − e− jΩ) = 2 sin
je− jΩ/2 , (7.37b)
2
2
which have lowpass and highpass frequency responses, respectively (Fig. 7-6).
Define the average (lowpass) signal xL [n] as
xL [n] = x[n] c ghaar [n]
(7.38a)
1
= √ {a + h, b + a, c + b, d + c, e + d . . .}
2
1.414
1.2
1.0
0.8
0.6
0.4
0.2
0
0
|H(Ω)|
|G(Ω)|
0.5
1
1.5
2
2.5
3
Ω
in terms of y[n]?
Answer: y[n](−1)n . See Eq. (7.34).
Figure 7-6 |G(Ω)| (in blue) and |H(Ω)| (in red) for the Haar
wavelet transform. This (quadrature-mirror filters) QMF pair
has symmetry about the Ω = π /2 axis.
www.dbooks.org
214
CHAPTER 7
and the detail (highpass) signal xH [n]
Note that downsampling by 2 followed by upsampling by 2
replaces values of x[n] with zeros for odd times n.
2. Next, filter xLDU [n] and xHDU [n] with filters ghaar [−n] and
hhaar [−n], respectively.
As noted earlier in Section 7-1.3, the term “filter” in the
context of the wavelet transform means “cyclic convolution.”
Filters ghaar [n] and hhaar [n] are called analysis filters, because
they are used to compute the Haar wavelet transform. Their time
reversals ghaar [−n] and hhaar [−n] are called synthesis filters,
because they are used to compute the inverse Haar wavelet
transform (that is, to reconstruct the signal from its Haar wavelet
transform). The reason for using time reversals here is explained
below.
The cyclic convolutions of xLDU [n] with ghaar [−n] and
xHDU [n] with hhaar [−n] yield
xH [n] = x[n] c hhaar [n]
(7.38b)
1
= √ {a − h, b − a, c − b, d − c, e − d . . .}.
2
Next, define the downsampled average signal xLD [n] as
1
xLD [n] = xL [2n] = √ {a + h, c + b, e + d, g + f }
2
(7.39a)
and the downsampled detail signal xHD [n] as
1
xHD [n] = xH [2n] = √ {a − h, c − b, e − d, g − f }. (7.39b)
2
The signal x[n] of duration 8 has been replaced by the two
signals xLD [n] and xHD [n], each of durations 4, so no information
about x[n] has been lost. We use cyclic convolutions instead
of linear convolutions so that the cumulative length of the
downsampled signals equals the length of the original signal.
Using linear convolutions, each convolution with ghaar [n] or
hhaar [n] would lengthen the signal unnecessarily. As we shall
see, using cyclic convolutions instead of linear convolutions is
sufficient to recover the original signal from its Haar wavelet
transform.
WAVELETS AND COMPRESSED SENSING
1
{a+h, c+b, c+b, . . . , a+h},
2
(7.41a)
1
xHDU [n] c hhaar [−n] = {a − h, b − c, c − b, . . ., h − a}.
2
(7.41b)
xLDU [n] c ghaar [−n] =
3. Adding the outcomes of the two cyclic convolutions gives
x[n]:
x[n] = xLDU [n] c ghaar [−n] + xHDU [n] c hhaar [−n].
7-4.2 Single-Stage Reconstruction
The single-stage Haar decomposition and reconstruction processes are depicted in Fig. 7-7. The signal x[n] can be reconstructed from xLD [n] and xHD [n] as follows:
1. Define the upsampled (zero-stuffed) signal xLDU [n] as
(
xLD [n/2]
for n even
xLDU [n] =
0
for n odd
1
= √ {a + h, 0, c + b, 0, e + d, 0, g + f , 0},
2
(7.40a)
and the upsampled (zero-stuffed) signal xHDU [n] as
(
xHD [n/2]
for n even
xHDU [n] =
0
for n odd
1
= √ {a − h, 0, c − b, 0, e − d, 0, g − f , 0}.
2
(7.40b)
(7.42)
It is still not evident why this is worth doing. The following
example provides a partial answer.
Consider the finite-duration (N = 16) signal x[n]:


4
for 0 ≤ n ≤ 4


1
for 5 ≤ n ≤ 9
x[n] =
(7.43)

3
for 10 ≤ n ≤ 14



4
for n = 15.
The Haar-transformed signals are
1
xLD [n] = xL [2n] = √ {8, 8, 8, 2, 2, 4, 6, 6},
2
1
xHD [n] = xH [2n] = √ {0, 0, 0, 0, 0, 2, 0, 0}.
2
These can be derived as follows. We have
x[n] = {4, 4, 4, 4, 4, 1, 1, 1, 1, 1, 3, 3, 3, 3, 3, 4},
(7.44)
7-4 HAAR WAVELET TRANSFORM
ghaar[n]
215
xL[n]
2
xLD[n]
2
xLDU[n]
ghaar[−n]
x[n]
x[n]
hhaar[n]
xH[n]
2
xHD[n]
Decomposition
2
xHDU[n]
hhaar[−n]
Reconstruction
Figure 7-7 Single-stage Haar decomposition and reconstruction of x[n].
1
xL [n] = x[n] c √ {1, 1}
2
1
= √ {8, 8, 8, 8, 8, 5, 2, 2, 2, 2, 4, 6, 6, 6, 6, 7},
2
1
xH [n] = x[n] c √ {1, −1}
2
1
= √ {0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 2, 0, 0, 0, 0, 1},
2
1
xLD [n] = xL [2n] = √ {8, 8, 8, 2, 2, 4, 6, 6},
2
1
xHD [n] = xH [2n] = √ {0, 0, 0, 0, 0, 2, 0, 0}.
2
The original signal x[n] can be recovered from xLD [n] and xHD [n]
by
1
xLDU [n] = √ {8, 0, 8, 0, 8, 0, 2, 0, 2, 0, 4, 0, 6, 0, 6, 0},
2
1
xHDU [n] = √ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0},
2
1
xLDU [n] c √ {1, 1}
2
= {4, 4, 4, 4, 4, 1, 1, 1, 1, 2, 2, 3, 3, 3, 3, 4},
1
xHDU [n] c √ {−1, 1}
2
= {0, 0, 0, 0, 0, 0, 0, 0, 0, −1, 1, 0, 0, 0, 0, 0},
1
1
xLDU [n] c √ {1, 1} + xHDU [n] c √ {−1, 1}
2
2
= {4, 4, 4, 4, 4, 1, 1, 1, 1, 1, 3, 3, 3, 3, 3, 4} = x[n].
We observe that the outcome of the second cyclic convolution
is sparse (mostly zero-valued). The Haar transform allows x[n],
which has duration 16, to be represented using the eight values
of xLD [n] and the single nonzero value (and its location n = 5)
of xHD [n]. This saves almost half of the storage required for x[n].
Hence, x[n] has been compressed by 43%.
Even though x[n] is not sparse, it was transformed, using
the Haar transform, into a sparse representation with the same
number of samples, meaning that most of the values of the Haartransformed signal are zero-valued. This reduces the amount of
memory required to store x[n], because only the times at which
nonzero values occur (as well as the values themselves) need
be stored. The few bits (0 or 1) required to store locations
of nonzero values are considered to be negligible in number
compared with the many bits required to store the actual nonzero
values. Since the Haar transform is orthogonal, x[n] can be
recovered perfectly from its Haar-transformed values.
7-4.3 Multistage Decomposition and
Reconstruction
In the simple example used in the preceding subsection, only 1
element of the Haar-transformed signal xHD [n] is nonzero, but all
8 elements of xLD [n] are nonzero. We can reduce the number of
nonzero elements of xLD [n] by applying a second Haar transform
stage to it. That is, xLD [n] can be transformed into the two signals
xLDLD [n] and xLDHD [n] by applying the steps outlined in Fig. 7-8.
www.dbooks.org
216
CHAPTER 7
ghaar[n]
xL[n]
2
xLD[n]
WAVELETS AND COMPRESSED SENSING
xLDL[n]
ghaar[n]
xLDH[n]
hhaar[n]
2
xLDLD[n]
2
xLDHD[n]
x[n]
hhaar[n]
xH[n]
2
xHD[n]
Figure 7-8 Two-stage Haar analysis filter bank. Note that only the upper half of the first stage is decomposed further.
Decomposition
Thus,
1
xLD [n] = √ {8, 8, 8, 2, 2, 4, 6, 6},
2
1
xLDL [n] = xLD [n] c √ {1, 1}
2
= {7, 8, 8, 5, 2, 3, 5, 6},
1
xLDH [n] = xLD [n] c √ {1, −1}
2
= {1, 0, 0, −3, 0, 1, 1, 0},
xLDLD [n] = xLDL [2n] = {7, 8, 2, 5},
xLDHD [n] = xLDH [2n] = {1, 0, 0, 1}.
(7.45)
Signal xLDHD [n] is again sparse: only two of its four values are
nonzero. So x[n] can now be represented by the four values
of xLDLD [n], the two nonzero values of xLDHD [n], and the one
nonzero value of xHD [n]. This reduces the storage required for
x[n] by 57%.
The average signal xLDLD [n] can in turn be decomposed even
further. The result is an analysis filter bank that computes
the Haar wavelet transform of x[n]. This analysis filter bank
consists of a series of sections like the left half of Fig. 7-7,
connected as in Fig. 7-8, except that each average signal is
decomposed further. The signals computed at the right end of
this analysis filter bank constitute the Haar wavelet transform of
x[n]. Reconstruction of x[n] is shown in Fig. 7-9.
A signal x[n] of duration N = 2K (with K an integer) can be
represented by the Haar wavelet transform through a K-stage
decomposition process involving cyclic convolutions with filters
ghaar [n] and hhaar [n], as defined by Eq. (7.36). The signal x[n]
can be zero-padded so that its length is a power of 2, if that is
not already the case, just as is done for the FFT. The sequential
process is:
Stage 1:
x[n]
hhaar[n]
2
x[n]
ghaar[n]
2
Xe1 [n] = xLD [n].
hhaar[n]
2
ghaar[n]
2
xe2 [n] = xLDHD [n],
Stage 2:
Xe1 [n]
Xe1 [n]
..
.
xe1 [n] = xHD [n],
Xe2 [n] = xLDLD [n].
Stage K:
7-4.4 Haar Wavelet Transform Filter Banks
XeK−1 [n]
XeK−1 [n]
hhaar[n]
2
ghaar[n]
2
xeK [n],
XeK [n].
7-4 HAAR WAVELET TRANSFORM
xLDLD[n]
2
217
ghaar[−n]
xLD[n]
xLDHD[n]
2
ghaar[−n]
2
x[n]
hhaar[−n]
xHD[n]
hhaar[−n]
2
Figure 7-9 Reconstruction by a two-stage Haar synthesis filter bank.
The Haar transform of x[n] consists of the combination of
K + 1 signals:
Duration
{ xe1 [n], xe2 [n], xe3 [n], . . . , xeK [n], XeK [n] }.
|{z} |{z} |{z}
| {z } | {z }
N/2
N/4
N/8
N/2K
Stage 1:
(7.46)
XeK [n]
N/2K
To represent x[n], we need to retain the “high-frequency” outputs
of all K stages (i.e., { xe1 [n], xe2 [n], . . . , xeK [n] }), but only the final
output of the “low-frequency” sequence, namely XeK [n]. The
total duration of all of the (K + 1) Haar transform signals is
N
N
N N N
+ + + · · · + K + K = N,
2
4
8
2
2
(7.47)
which equals the duration N of x[n]. We use cyclic convolutions
instead of linear convolutions so that the total lengths of the
downsampled signals equals the length of the original signal.
Were we to use linear convolutions, each convolution with
ghaar [n] or hhaar [n] would lengthen the signal unnecessarily.
• The xek [n] for k = 1, 2, . . . , K are called detail signals.
• The XeK [n] is called the average signal.
xeK [n]
Stage 2:
2
ghaar [−n]
AK−1 [n],
2
hhaar [−n]
BK−1 [n],
XeK−1 [n] = AK−1 [n] + BK−1[n].
XeK−1 [n]
xeK−1 [n]
2
ghaar [−n]
AK−2 [n],
2
hhaar [−n]
BK−2 [n],
XeK−2 [n] = AK−2 [n] + BK−2[n].
..
.
Stage K:
Reconstruction
The inverse Haar wavelet transform can be computed in reverse
order, starting with { XeK [n], xeK [n], xeK−1 [n], . . . }.
www.dbooks.org
Xe1 [n]
xe1 [n]
2
ghaar [−n]
A0 [n],
2
hhaar [−n]
B0 [n],
x[n] = A0 [n] + B0[n].
218
CHAPTER 7
7-4.5 Haar Wavelet Transform in the Frequency
Domain
As noted in Section 7-1, the original objective of subband
coding was to decompose a signal into different frequency
bands. Wavelets are more general than simple subbanding in
that decomposition of the spectrum of the input into different
bands is not the explicit purpose of the wavelet transform.
Nevertheless, the decomposition does approximately allocate
the signal into different frequency bands, so it is helpful to track
and understand the approximate decomposition.
Recall from Eq. (7.37) that Ghaar (Ω) is approximately a
lowpass filter and Hhaar (Ω) is approximately a highpass filter.
Hence, at the output of the first stage of the Haar wavelet
transform:
• Xe1 [n] is the lowpass-frequency part of x[n], covering the
approximate range 0 ≤ |Ω| ≤ π /2, and
• xe1 [n] is the highpass-frequency part of x[n], covering the
approximate range π /2 ≤ |Ω| ≤ π .
At each stage, downsampling expands the spectrum of each
signal to the full range 0 ≤ |Ω| ≤ π . Hence, at the output of
the second stage,
• Xe2 [n] covers the frequency range 0 ≤ |Ω| ≤ π /2 of Xe1 [n],
which corresponds to the range 0 ≤ |Ω| ≤ π /4 of the
spectrum of x[n].
• xe2 [n] covers the frequency range π /2 ≤ |Ω| ≤ π of Xe1 [n],
which corresponds to the range π /4 ≤ |Ω| ≤ π /2 of the
spectrum of x[n].
The Haar wavelet transform decomposes the spectrum of X(Ω)
into octaves, with
Octave
π
π
≤ |Ω| ≤ k−1 represented by xek [n].
k
2
2
Since the width of this band is π /2k , the sampling rate can be
reduced by a factor of 2k using downsampling.
For a signal of duration N = 2K , the Haar wavelet transform
decomposes x[n] into (K + 1) components, namely XeK [n] and
{ xe1 [n], xe2 [n], . . . , xeK [n] }, with each component representing a
frequency octave. The spectrum decomposition is illustrated
in Fig. 7-10 for K = 3. Because the different components
cover different octaves, they can be sampled at different rates.
Furthermore, if the signal or image consists of slowly varying
segments with occasional fast-varying features, as many realworld signals and images do, then xek [n] for small k will be
sparse, requiring few samples to represent it.
WAVELETS AND COMPRESSED SENSING
~
X3[n] x˜ 3[n]
band band
0
π/8
x˜ 2[n]
band
π/4
x˜ 1[n]
band
π/2
π
Ω
Figure 7-10 Approximate frequency-band coverage by components of the Haar wavelet transform for K = 3.
Concept Question 7-3: Why do we use cyclic convolutions instead of linear convolutions?
Exercise
√ 7-6: Compute the one-stage Haar transform of
x[n] =
2{4, 3, 3, 4}.
Answer:
x[n] c √12 [1, 1] = {8, 7, 6, 7}. Hence, xLD [n] = {8, 6}.
x[n] c √12 [1, −1] = {0, −1, 0, 1}. Hence, xHD [n] = {0, 0}.
7-5 Discrete-Time Wavelet Transforms
The two Haar wavelet filters hhaar [n] and ghaar [n] introduced
in the preceding section and defined by Eq. (7.36) are very
simple in structure and short in duration, and yet they provide a
powerful tool for decomposing signals (so they may be stored or
transmitted more efficiently) and reconstructing them later. We
now extend our presentation to the general case of wavelet transform filters h[n] and g[n], which also use the octave-based filterbank decomposition structure shown in Fig. 7-2 and the octavebased filter-bank reconstruction structure shown in Fig. 7-4,
but the main difference is that g[n] and h[n] are no longer
explicitly lowpass and highpass filters, although their frequency
responses are still approximately lowpass and highpass. Now
h[n] is designed to sparsify (make mostly zero-valued) all of
the octave-based filter bank outputs except the lowest frequency
band output. Filters g[n] and h[n] are now called the scaling and
wavelet functions, respectively.
In this and future sections of this chapter, we examine the
structure, the properties, and some of the applications of the
wavelet transform. We start by addressing the following two
questions:
(1) What conditions must g[n] and h[n] satisfy so that the
output of the octave-based reconstruction filter bank is the
same as the input to the octave-based decomposition filter
7-5 DISCRETE-TIME WAVELET TRANSFORMS
219
bank?
and
(2) How should h[n] be chosen so that the outputs of all of the
octave-based decomposition filter banks are sparse, except
for the output of the lowest frequency band?
(−1)n g[n] ∗ g[−n] + (−1)n h[n] ∗ h[−n] = 0.
We now derive more general conditions on g[n] and h[n] so
that the output of the octave-based reconstruction filter bank is
identical to the input of the octave-based decomposition filter
bank. If this occurs at each stage, then it will occur at all stages.
Accordingly we consider the single-stage decomposition and
reconstruction shown in Fig. 7-11.
To determine the necessary conditions that g[n] and h[n]
should satisfy in order for the input/output relationship in
Fig. 7-11 to be true, we replicate Fig. 7-11 in equation form,
but we also replace the combined downsampling/upsampling
operations in the figure with the equivalent functional form
given by Eq. (7.30):
1 + (−1)n
(x[n] ∗ g[n])
∗ g[−n]
2
1 + (−1)n
∗ h[−n] = x[n].
(7.48)
+ (x[n] ∗ h[n])
2
Expanding Eq. (7.48) and using the convolution relation given
by Eq. (7.34) gives
7-5.2 Quadrature Mirror Filters
There are many possible solutions that satisfy Eq. (7.51). A
sensible approach is to find a pair of filters g[n] and h[n] that automatically satisfy Eq. (7.51b), and then use them in Eq. (7.51a).
One such pair is the quadrature mirror filter (QMF), which is
based on the QMF relation (where L is an odd integer):
g[n] = −(−1)n h[L − n] = (−1)n−L h[L − n].
g[−n] = −(−1)−n h[L + n] = (−1)−(n+L) h[L + n],
(7.52b)
where h[L − n] is h[−n] shifted in time by L, with L chosen to
be an odd integer and its length is such as to make g[n] causal.
Thus, if h[n] and g[n] are each of length N, L should be equal to
or greater than (N − 1).
Using Eq. (7.52), the first term in Eq. (7.51b) becomes
(−1)n g[n] ∗ g[−n] = (−1)2n−L h[L − n] ∗ (−1)−(n+L) h[L + n].
In view of the time-shift property (#5 in Table 2-6)
x1 [n − L] ∗ x2[n + L] = x1 [n] ∗ x2[n],
(7.49)
(7.53)
it follows that
(−1)n g[n] ∗ g[−n] = −(−1)n h[n] ∗ h[−n],
which satisfies the condition stated by Eq. (7.51b).
(7.50)
The expression given by Eq. (7.50) is satisfied for any input
x[n] if and only if both of the following perfect-reconstruction
conditions are satisfied:
(1)
g[n] ∗ g[−n] + h[n] ∗ h[−n] = 2δ [n]
(7.52a)
Replacing n with −n implies:
Collecting terms convolved with x[n] and (−1)n x[n] separately
leads to
x[n] ∗ {g[n] ∗ g[−n] + h[n] ∗ h[−n]}
+ (−1)n x[n] ∗ {(−1)n g[n] ∗ g[−n]
+ (−1)n h[n] ∗ h[−n]} = 2x[n].
(7.51b)
Next, we examine how to relate g[n] to h[n] so as to satisfy the
two parts of Eq. (7.51).
7-5.1 Conditions for Filter Bank Perfect
Reconstruction
x[n] ∗ g[n] ∗ g[−n]
+ {(−1)n x[n]} ∗ {(−1)n g[n]} ∗ g[−n]
+ x[n] ∗ h[n] ∗ h[−n]
+ {(−1)n x[n]} ∗ {(−1)n h[n]} ∗ h[−n] = 2x[n].
(2)
(7.51a)
◮ We conclude that the QMF relations between g[n] and
h[n], as specified in Eq. (7.52), do indeed satisfy one of the
two conditions required for perfect reconstruction. ◭
The significance of the odd-valued time shift L is best illustrated
by a simple example.
www.dbooks.org
220
CHAPTER 7
h[n]
xH[n]
2
xHD[n]
2
xHDU[n]
WAVELETS AND COMPRESSED SENSING
h[−n]
xin[n]
xout[n]
g[n]
xL[n]
2
xLD[n]
Decomposition
2
xLDU[n]
g[−n]
Reconstruction
Figure 7-11 Single-stage decomposition of x[n] to xLD [n] and xHD [n], and reconstruction of x[n] from xLD [n] and xHD [n].
The two sequences in Eqs. (7.54b) and (7.54d) are identical in
value, but opposite in sign. Hence, their sum adds up to zero.
Example 7-2: QMF
Show that for h[n] = { 1, 2, 3, 4 }, the QMF relations given
by Eq. (7.52) satisfy the reconstruction condition given by
Eq. (7.51b).
Solution: Since h[n] is of length 4, we select the time shift L to
be 3. The second term in Eq. (7.51b) is
(−1)n h[n] ∗ h[−n] = { 1, −2, 3, −4 } ∗ { 4, 3, 2, 1}
= { 4, −5, 8, −10, −8, −5, −4 }.
(7.54a)
(7.54b)
For L = 3 and the QMF relation defined by Eq. (7.52a), g[n] is
g[n] = −(−1)n h[3 − n],
which yields
g[0] = −h[3] = −4,
g[1] = h[2] = 3,
g[2] = −h[1] = −2,
g[3] = h[0] = 1.
Hence,
g[n] = { −4, 3, −2, 1 }.
The first term of Eq. (7.51b) is then given by
(−1)n g[n] ∗ g[−n] = { −4, −3, −2, −1 } ∗ { 1, −2, 3, −4}
(7.54c)
= −{ 4, −5, 8, −10, −8, −5, −4 }. (7.54d)
7-5.3 Smith-Barnwell Condition
We now turn our attention to the first of the two perfectreconstruction conditions, namely the condition defined by
Eq. (7.51a), and we do so in concert with the QMF relations
given by Eq. (7.52).
Using the QMF relations, the first term in Eq. (7.51a) becomes
g[n] ∗ g[−n] = {(−1)n−L h[−(n − L)]} ∗ {(−1)−(n+L) h[n + L]}
= {(−1)n h[−n]} ∗ {(−1)n h[n]},
(7.55)
where we used the time-shift property given by Eq. (7.53).
Replacing the first term in Eq. (7.51a) with Eq. (7.55) and
employing the commutativity property of convolution yields
the Smith-Barnwell condition for perfect reconstruction using
QMF filters:
{(−1)n h[n]} ∗ {(−1)n h[−n]} + h[n] ∗ h[−n] = 2δ [n]. (7.56)
Finally, upon taking advantage of the “useful convolution relation” encapsulated by Eq. (7.34), we obtain the result
(h[n] ∗ h[−n])(−1)n + h[n] ∗ h[−n] = 2δ [n].
(7.57)
The usual form of the Smith-Barnwell condition uses the DTFT.
Recall that the DTFT maps convolutions to products (entry #6 of
Table 2-7) and modulation (entry #3 of Table 2-7) to frequency
7-5 DISCRETE-TIME WAVELET TRANSFORMS
shift:
e jΩ0 n h[n]
221
H(Ω − Ω0).
(7.58)
H(Ω − π ).
(7.59)
Setting Ω0 = π and recognizing that e jπ n = (−1)n gives
(−1)n h[n]
relation in Eq. (7.52a), does also. Consequently, g[n] also is
orthonormal to even translations of itself. Using Eq. (7.52a),
it can be shown that g[n] and h[n] are also orthogonal to even
translations of each other.
So the wavelet transform constitutes an orthonormal expansion of x[n] as
Using the properties of the DTFT, the DTFT of the SmithBarnwell condition given by Eq. (7.57) is
2
2
|H(Ω)| + |H(Ω − π )| = 2.
(7.60)
(7.61)
h[n] ∗ h[−n] = δ [n],
for n even.
(7.62)
Writing out Eq. (7.62) for n even gives
L
∑ h2[i] = 1,
(7.63a)
∑ h[i] h[i − 2] = 0,
(7.63b)
∑ h[i] h[i − 4] = 0,
(7.63c)
i=0
L
i=2
L
n=4
i=4
..
.
∑ h[i] h[i − (L − 1)] = 0.
Xe1 [n] = ∑ g[2n − i] x[i]
(7.64)
(7.65a)
∞
xe1 [n] = ∑ h[2n − i] x[i].
(7.65b)
i=−∞
Computation of Xe1 [n] and xe1 [n] in Eq. (7.65) is implemented
using the wavelet analysis filter bank shown in Fig. 7-12(a).
Computation of x[n] from Xe1 [n] and xe1 [n] in Eq. (7.64) is
implemented using the wavelet synthesis filter bank shown in
Fig. 7-12(b).
The decimation in Fig. 7-12(a) manifests itself in Eq. (7.65)
as the 2n in g[2n − i] and h[2n − i]. The zero-stuffing and time
reversals in Fig. 7-12(b) manifest themselves in Eq. (7.64) as
the 2i in g[2i − n] and h[2i − n].
The average signal Xe1 [n] can in turn be decomposed similarly,
as in the wavelet analysis filter bank shown in Fig. 7-12(a). So
the first term in Eq. (7.64) can be decomposed further, resulting
in the K-stage decomposition
∞
L
n = L−1
i=−∞
i=−∞
and
n=2
i=−∞
∞
for odd n,
for even n,
Eq. (7.57) holds for odd n for any h[n], and for n even, it
simplifies to
n=0
∞
where average signal Xe1 [n] and detail signal xe1 [n] are defined as
Recognizing that
(
0
1 + (−1)n =
2
∞
x[n] = ∑ g[2i − n] Xe1 [i] + ∑ h[2i − n] xe1 [i],
x[n] = ∑ XeK [i] g(K) [2K i − n]
(7.63d)
i=−∞
i=L−1
∞
K
+ ∑ ∑ xek [i] h(k) [2k i − n],
Recall that since L is odd, L − 1 is even.
(7.66)
k=1 i=−∞
◮ The Smith-Barnwell condition is equivalent to stating
that the autocorrelation rh [n] = h[n] ∗ h[−n] of h[n] is zero
for even, nonzero, n and 1 for n = 0. This means that h[n] is
orthonormal to even-valued translations of itself. ◭
where average signal XeK [n] and detail signal {e
xk [n],
k = 1, . . . , K} are computed using
∞
Xek [n] = ∑ g(k) [2k n − i] x[i]
i=−∞
7-5.4 Wavelets as Basis Functions
and
It is easy to show that if h[n] satisfies the Smith-Barnwell
condition given by Eq. (7.57), then g[n], as defined by the QMF
www.dbooks.org
∞
xek [n] = ∑ h(k) [2k n − i] x[i],
i=−∞
(7.67)
222
CHAPTER 7
h[n]
WAVELETS AND COMPRESSED SENSING
x˜ 1[n]
2
x[n]
g[n]
2
X˜ 1[n]
h[n]
2
x˜ 2[n]
g[n]
2
X˜ 2[n]
(a) Wavelet analysis filter bank
X˜ 2[n]
2
g[−n]
g[−n]
2
x˜ 2[n]
2
x[n]
h[−n]
x˜1[n]
h[−n]
2
(b) Wavelet synthesis filter bank
Figure 7-12 (a) Wavelet analysis filter bank and (b) wavelet synthesis filter bank.
and the basis functions g(k) [n] and h(k) [n] can be computed
recursively offline. We will not bother with explicit formulae for
these, since the decomposition and reconstruction are performed
much more easily using filter banks. The wavelet analysis and
synthesis filter banks shown respectively in Fig. 7-12(a) and
(b) are identical in form to the octave-band filter banks shown
in Figs. 7-2 and 7-4, except for the following nomenclature
changes:
xe1 [n] = xHD [n],
xe2 [n] = xLDHD [n],
Xe1 [n] = xLD [n],
Xe2 [n] = xLDLD [n].
(7.68)
If x[n] has duration N = 2K , the components of the wavelet
transform of x[n] have durations
{e
x1 [n], xe2 [n], xe3 [n], . . . , xeK [n], XeK [n]}.
|{z} |{z} |{z}
| {z } | {z }
N/2
N/4
N/8
N/2K
(7.69)
N/2K
7-5.5 Amount of Computation
The total duration of all of the wavelet transform signals together
is
N N N
N
N
(7.70)
+ + + · · · + K + K = N,
2
4
8
2
2
which equals the duration N of x[n].
Again, we use cyclic convolutions instead of linear convolutions so that the total lengths of the decimated signals equals the
length of the original signal.
The total amount of computation required to compute the
wavelet transform of a signal x[n] of duration N can be computed
as follows. Let the durations of g[n] and h[n] be the even integer
7-6 SPARSIFICATION USING WAVELETS OF PIECEWISE-POLYNOMIAL SIGNALS
(L + 1) where L is the odd integer in Eq. (7.52). Convolving
x[n] with both g[n] and h[n] requires 2(2(L + 1))N = 4(L + 1)N
multiplications-and-additions (MADs). But since the results will
be decimated by two, only half of the convolution outputs must
be computed, halving this to 2(L + 1)N.
At each successive decomposition, g[n] and h[n] are convolved with the average signal from the previous stage, and the
result is decimated by two. So g[n] and h[n] are each convolved
with the signals
{ x[n] , Xe1 [n], Xe2 [n], . . . , XeK [n]}.
| {z }
|{z} | {z } | {z }
N
N/2
N/4
N/2K
The total number of MADs required is thus
N N
N
2(L + 1) N + + + · · · + K < 4(L + 1)N.
2
4
2
(7.71)
The additional computation for computing more decompositions (i.e., increasing K) is thus minimal.
Since L is small (usually 1 ≤ L ≤ 5), this is comparable to
the amount of computation N2 log2 (N) required to compute the
DFT, using the FFT, of a signal x[n] of duration N. But the DFT
requires complex-valued multiplications and additions, while
the wavelet transform uses only real-valued multiplications and
additions.
Concept Question 7-4:
What is the Smith-Barnwell con-
dition?
1
Exercise 7-7: If h[n] = √
{6, 2, h[2], 3}, find h[2] so that
5 2
223
in Eq. (7.67) and computed using the analysis filter bank shown
in Fig. 7-12(a) are sparse (mostly zero-valued). In particular, we
design g[n] and h[n] so that the wavelet transform detail signals
are sparse when the input signal x[n] is piecewise polynomial,
which we define next. The g[n] and h[n] filters are then used in
the Daubechies wavelet transform.
7-6.1 Definition of Piecewise-Polynomial Signals
A signal x[n] is defined to be piecewise-Mth-degree polynomial
if it has the form

M


 ∑ a0,k nk
for − ∞ < n ≤ N0 ,



k=0


M

 a nk

for N0 < n ≤ N1 ,
∑ 1,k
(7.72)
x[n] = k=0

M



for N1 < n ≤ N2 .

∑ a2,k nk



k=0



 ..
.
This x[n] can be segmented into intervals, and in each interval
x[n] is a polynomial in time n of degree M. The times Ni at which
the coefficients {ai,k } change values are sparse, meaning that
they are scattered over time n. In continuous time, such a signal
would be a spline (see Section 4-7), except that in the case of a
spline, the derivatives of the signal must match at the knots (the
times where coefficients {ai,k } change values). The idea here is
that the coefficients {ai,k } can change completely at the times Ni ;
there is no “smoothness” requirement. Indeed, these times Ni
constitute the edges of x[n].
h[n] satisfies the Smith-Barnwell condition.
According to Eq. (7.63), the Smith-Barnwell
condition requires the autocorrelation of h[n] to be 1 for
n = 0 and to be 0 for even n 6= 0. For h[n] = {a, b, c, d},
these conditions give a2 + b2 + c2 + d 2 = 1, ac + bd = 0,
and h[2] = −1.
Answer:
7-6 Sparsification Using Wavelets of
Piecewise-Polynomial Signals
The wavelet filters g[n] and h[n] are respectively called scaling
and wavelet functions. In this section, we show how to design
these functions such that they satisfy the Smith-Barnwell condition for perfect reconstruction given by Eq. (7.56), form a QMF
pair, and have the property that the detail signals x̃k [n] defined
A. Piecewise-Constant Signals
First, let x[n] be piecewise constant (M = 0 in Eq. (7.72)), so that
x[n] is of the form


a0
for − ∞ < n ≤ N0 ,



a 1
for N0 < n ≤ N1 ,
x[n] = a
(7.73)
for N1 < n ≤ N2 .
2




.
 ..
The value of x[n] changes only at a few scattered times. The
amount by which x[n] changes at n = Ni is the jump ai+1 − ai :
(
0
for n 6= Ni ,
x[n + 1] − x[n] =
(7.74)
ai+1 − ai
for n = Ni .
www.dbooks.org
224
CHAPTER 7
This can be restated as
In practice, q[n] = q[0] δ [n] has duration = 1, so x[n] ∗ h[n] is still
mostly zero-valued.
x[n + 1] − x[n] = ∑(ai+1 − ai ) δ [n − Ni ].
WAVELETS AND COMPRESSED SENSING
(7.75)
i
Taking differences sparsifies a piecewise-constant signal, as
illustrated in Fig. 7-13.
Now let the wavelet function h[n] have the form, for some
signal q[n] that is yet to be determined,
h[n] = q[n] ∗ { 1, −1}.
(7.76)
Since from Fig. 2-3 the overall impulse response of two systems
connected in series is the convolution of their impulse responses,
h[n] can be implemented by two systems connected in series:
{ 1, −1 }
x[n]
x[n] ∗ h[n].
q[n]
Convolution with h[n] sparsifies a piecewise-constant input x[n],
since (using the time-shift property of convolution)
x[n] ∗ h[n] = x[n] ∗ {1, −1} ∗ q[n]
= (x[n + 1] − x[n]) ∗ q[n]
B. Piecewise-Linear Signals
Next, let x[n] be piecewise linear (M = 1 in Eq. (7.72)), so that
x[n] is of the form


a0,1 n + a0,0,
−∞ < n ≤ N0 ,



a1,1 n + a1,0,
N0 < n ≤ N1 ,
x[n] = a n + a ,
(7.78)
N
2,1
2,0
1 < n ≤ N2 ,




 ...
Proceeding as in the case of piecewise-constant x[n], taking
differences, and then taking differences of the differences, will
sparsify a piecewise-linear signal. The process is illustrated in
Fig. 7-14.
The bottom signal in Fig. 7-14 is in turn convolved with q[n],
4
2
= ∑(ai+1 − ai) δ [n − Ni ] ∗ q[n]
0
i
= ∑(ai+1 − ai) q[n − Ni].
(7.77)
−2
0
5
10
15
i
20
25
30
35
20
25
30
35
25
30
(a) x[n]
1
0.5
6
4
0
2
−0.5
0
0
0
5
10
15
20
25
5
10
(b) w1[n] = x[n + 1] − x[n]
30
(a) x[n]
3
2
1
0
−1
0
15
1
0
−1
−2
5
10
15
20
25
30
0
5
10
15
20
(b) x[n + 1] − x[n]
(c) w2[n] = w1[n + 1] − w1[n]
Figure 7-13 A piecewise-constant signal is compressed by
taking differences. (a) a piecewise-constant signal, (b) differences of the signal.
Figure 7-14 A piecewise-linear signal is compressed by
taking successive differences. (a) A piecewise-linear signal, (b)
differences of the top signal, (c) differences of the middle signal.
7-6 SPARSIFICATION USING WAVELETS OF PIECEWISE-POLYNOMIAL SIGNALS
resulting in a series of scaled and delayed versions of q[n]. In
practice, q[n] has length two, so x[n] ∗ h[n] is still mostly zerovalued.
Now let the wavelet function h[n] have the form
h[n] = q[n] ∗ {1, −1} ∗ {1, −1}.
Table 7-2 Daubechies wavelet notation.
K
dbK
D(2K)
Number of differences
Duration = 2K
Degree sparsified = M
(7.79)
Convolution with h[n] sparsifies a piecewise-linear input x[n],
since
x[n] ∗ h[n] = x[n] ∗ {1, −1} ∗ {1, −1} ∗ q[n]
= (x[n + 1] − x[n]) ∗ {1, −1} ∗ q[n]
= (x[n + 2] − 2x[n + 1] + x[n]) ∗ q[n].
(7.80)
The output consists of a set of scaled and delayed versions of
q[n]. In practice, q[n] is very short (length two), so x[n] ∗ h[n] is
still mostly zero-valued.
C. Piecewise-Polynomial Signals
Finally, let x[n] be piecewise polynomial, meaning that

k

−∞ < n ≤ N0 ,
∑M

k=0 a0,k n ,


k
∑M
N0 < n ≤ N1 ,
k=0 a1,k n ,
x[n] =
M
k,
a
n
N
∑
1 < n ≤ N2 ,
 k=0 2,k



..

.
(7.81)
Taking (M + 1) sets of successive differences will sparsify x[n],
since each difference will reduce the degrees of the polynomial
segments by one. The continuous-time analogue of this is
y(t) =
d M+1 x
=0
dt M+1
(7.82)
for any Mth-degree polynomial x(t).
Now let the wavelet function h[n] have the form
h[n] = q[n] ∗ {1, −1} ∗ {1, −1} . . . {1, −1} .
{z
}
|
(7.83)
M+1 differences
The output consists of a set of scaled and delayed versions of
q[n]. In practice, q[n] is still relatively short (length M + 1), so
x[n] ∗ h[n] is still mostly zero-valued.
225
1
1
2
1
2
0
2
2
4
2
4
1
3
3
6
3
6
2
7-6.2 Design of Wavelet Functions h[n] of
Desired Form
We now show how to design wavelet functions of the form given
by Eq. (7.83). These wavelet functions compress piecewiseMth-degree polynomials to sparse signals xe[n]. Since many realworld signals can be so modeled, their wavelet transforms will
always be sparse.
The functions h[n] given by Eq. (7.83) are called Daubechies
wavelet functions. The order of Daubechies wavelet function
has several different definitions. The Daubechies wavelet function that compresses piecewise-(K − 1)th-degree polynomials
to sparse signals is termed “dbK Daubechies wavelet functions,” where “K” is a reference to the number of differences
that need to be taken, “db” is short for Daubechies, and
K = M + 1. Daubechies wavelet functions having duration N
are termed “DN Daubechies wavelet functions.” We shall see
that dbK Daubechies wavelet functions have duration 2K, so a
dbK Daubechies wavelet function is also a D(2K) Daubechies
wavelet function. The DN notation is inefficient because N is
always even. For that reason, we will use “dbK” to refer to
the order of a Daubechies wavelet function. The Haar functions
introduced in Section 7-4 are db1 and D2 wavelets. To confuse
matters further, DN was used in the past to mean both DN and
db(N/2). The notation is summarized in Table 7-2.
To make h[n] causal, we delay each of the differences in
Eq. (7.83) to get
h[n] = q[n] ∗ {1, −1} ∗ {1, −1} ∗ · · · ∗ {1, −1} .
|
{z
}
(7.84)
M+1 differences
A degrees-of-freedom analysis shows that q[n] must have duration M + 1, so h[n] has duration (2M + 2) = 2K. Function q[n]
is determined by inserting the form Eq. (7.84) into the SmithBarnwell condition given by Eq. (7.56). The scaling function
g[n] is then determined by the QMF formula in Eq. (7.52a).
www.dbooks.org
226
CHAPTER 7
3
A. Piecewise-Constant Signals
1 = ∑ h[n]2 .
To compress piecewise-constant signals, we set M = 0; a constant is a polynomial of degree zero. We wish to design a
db1 = D2 order Daubechies wavelet, which has only 1 difference because M + 1 = 0 + 1 = 1, and a duration = 2(1) = 2, so
L = 1. The expression in Eq. (7.84) becomes
h[n] = (q[0] δ [n]) ∗ {1, −1} = {q[0], −q[0]}.
| {z } | {z } |
{z
}
duration=1
1 difference
WAVELETS AND COMPRESSED SENSING
(7.85)
duration=2
(7.90b)
n=0
The first equation states that h[n] must be orthogonal to its
even-valued translations, and the second equation simply states
that the energy of h[n] must be one, which is a normalization
requirement that should be imposed after q[0] and q[1] have been
determined.
Substituting the elements of h[n] given in Eq. (7.89) into
Eq. (7.90a) leads to
Inserting this result into the Smith-Barnwell condition of
Eq. (7.63a) gives
q[0]2 + q[0]2 = 1.
(7.86)
0 = (q[0] − 2q[1]) q[0] + (q[1] − 2q[0]) q[1]
The requirement given in Eq. (7.63b) is satisfied automatically,
since h[n] has duration = 2. Hence, q[0] = √12 and
Since the scale factor will be set by ∑ h[n]2 = 1, we can, without
loss of generality, set q[0]=1. This gives
1
h[n] = √ {1, −1},
2
q[1]2 − 4q[1] + 1 = 0,
(7.92)
√
which
√ has the two roots q[1] = {2 ± 3}. Using the smaller root,
2 − 3, in Eq. (7.89) gives
√
√
√
h1 [n] = q1 [0]{1, − 3, 2 3 − 3, 2 − 3}.
(7.93)
(7.87)
and the scaling function g[n] is, from Eq. (7.52a), given by
1
g[n] = −(−1)n h[1 − n] = √ {1, 1}.
2
(7.88)
We recognize this as the Haar transform. It compresses piecewise constant signals. Making the alternate choice in the solu−1
, simply changes the signs
tion of Eq. (7.86), namely q[0] = √
2
of g[n] and h[n].
B. Piecewise-Linear Signals
To compress piecewise-linear signals, we set M = 1. We wish
to design a db2 = D4 order Daubechies wavelet, which has
M + 1 = 2 differences and a duration = 2(2) = 4. Function h[n]
in Eq. (7.84) becomes a filter of duration 4:
h[n] = {q[0], q[1]} ∗ {1, −1} ∗ {1, −1}
{z
}
| {z } |
M+1=2 differences
duration=2
= {q[0], q[1]} ∗ {1, −2, 1}
= {q[0], q[1] − 2q[0], q[0] − 2q[1], q[1]}.
(7.89)
The Smith-Barnwell condition for n = 0 and n = 2, as stated in
Eqs. (7.63b) and (7.63a), gives two equations:
3
0 = ∑ h[n] h[n − 2],
n=2
(7.90a)
= q[0]2 − 4q[0] q[1] + q[1]2.
(7.91)
Finally, the energy normalization ∑ h1 [n]2 = 1 requires
q1 [0] = ±0.4830. Hence,
h1 [n] = ±{0.4830, −0.8365, 0.2241, 0.1294}.
(7.94)
The length of h[n] is N = 2(M + 1) = 2K = 4. To satisfy
Eq. (7.52), the time shift L should be an odd integer and equal to
or greater than (N − 1). Since N − 1 = 4 − 1 = 3, we set L = 3.
Accordingly, the scaling function g1 [n] is, from Eq. (7.52a),
given by
g1 [n] = −(−1)n h[3 − n]
= ±{0.1294, −0.2241, −0.8365, −0.4830}.
(7.95)
√
Using the larger root of Eq. (7.92), namely (2 + 3), in
Eq. (7.89) gives
√
√
√
(7.96)
h2 [n] = q2 [0] {1, 3, −2 3 − 3, 2 + 3},
and the energy normalization ∑ h[n]2 = 1 leads to
q2 [0] = ±0.1294.
7-6 SPARSIFICATION USING WAVELETS OF PIECEWISE-POLYNOMIAL SIGNALS
Hence,
227
3
h2 [n] = ±{0.1294, 0.2241, −0.8365, 0.4830},
(7.97)
2
1
and the scaling function g[n] is then
0
g2 [n] = −(−1)n h[3 − n]
= ±{0.4830, 0.8365, 0.2241, −0.1294}.
−1
−2
5
(7.98)
The two choices of h[n] are time reversals of each other, and so
are the choices of g[n]. The signs of g[n] and h[n] can be changed
due to the (plus/minus) signs in Eqs. (7.94), (7.95), (7.97), and
(7.98).
These g[n] and h[n] functions are called the Daubechies db2
or D4 scaling and wavelet functions. The Daubechies db3 or
D6 wavelet transform sparsifies piecewise-quadratic signals, as
there are three differences (M = 2 and, hence, K = 3) in h[n].
◮ The Daubechies scaling functions g[n], with g[0] > 0, are
listed in Table 7-3 for various values of K. ◭
10
15
n
20
25
30
10
12
14
16
n
10
12
14
16
n
5
6
7
8
n
5
6
7
8
n
(a) x[n]
6
4
2
0
−2
−4
0
2
4
6
8
~
(b) X1[n]
0
−0.2
−0.4
Table 7-3 Daubechies scaling functions.
−0.6
2
g[n]
g[0]
g[1]
g[2]
g[3]
g[4]
g[5]
g[6]
g[7]
K=1
K=2
K=3
K=4
db1
.7071
.7071
0
0
0
0
0
0
db2
.4830
.8365
.2241
–.1294
0
0
0
0
db3
.3327
.8069
.4599
–.1350
–.0854
.0352
0
0
db4
.2304
.7148
.6309
–.0280
–.1870
.0308
.0329
–.0106
4
6
8
(c) x˜ 1[n]
6
4
2
0
−2
−4
1
2
3
4
~
(d) X [n]
2
0.5
0
−0.5
Example 7-3: db2 Wavelet Transform
−1
1
Compute the db2 (or equivalently, D4) Daubechies wavelet
transform of the piecewise-linear signal x[n] shown in
Fig. 7-15(a).
Solution: The two-stage procedure involves the computation
of average signals Xe1 [n] and Xe2 [n], and detail signals xe1 [n] and
2
3
4
(e) x˜2[n]
Figure 7-15 (a) Piecewise linear signal x[n], (b) stage-1
average signal Xe1 [n], (c) stage-1 detail signal xe1 [n], (d) stage2 average signal Xe2 [n], and (e) stage-2 detail signal xe2 [n].
www.dbooks.org
228
CHAPTER 7
xe2 [n], using Eqs. (7.67), (7.94), and (7.95). The results are
displayed in parts (b) to (e) of Fig. 7-15. We note that
signals to 2-D images. Downsampling in 2-D involves downsampling in both directions. For example:
• Average signals Xe1 [n] and Xe2 [n] are low-resolution versions
of x[n].
• Detail signals xe1 [n] and xe2 [n] are sparse (mostly zero), and
their nonzero values are small in magnitude.
• The db2 wavelet transform of the given x[n] consists of
xe1 [n], xe2 [n], and Xe2 [n].
These patterns explain the terms “average” and “detail.”
Concept Question 7-5: How is it possible that the
wavelet transform requires less computation than the FFT?
Exercise 7-8: What are the finest-detail signals of the db3
wavelet transform of x[n] = 3n2 ?
Answer: Zero, because db3 wavelet basis function h[n]
eliminates quadratic signals by construction.
Exercise 7-9: Show by direct computation that the db2
scaling function listed in Table 7-3 satisfies the SmithBarnwell condition.
Answer: The Smith-Barnwell condition requires the au-
tocorrelation of g[n] to be 1 for n = 0 and to be 0 for
even n 6= 0. For g[n] = {a, b, c, d}, these conditions give
a2 + b2 + c2 + d 2 = 1 and ac + bd = 0. The db2 g[n] listed
in Table 7-3 is g[n] = {.4830, .8365, .2241, −.1294}. It is
easily verified that the sum of the squares of these numbers
is 1, and that (.4830) × (.2241) + (.8365) × (−.1294) = 0.
7-7 2-D Wavelet Transform
The real power of the wavelet transform becomes apparent when
it is applied to 2-D images. A 512 × 512 image has more than
a quarter-million pixel values. Storing a sparse representation
of an image, rather than the image itself, saves a huge amount
of memory. Compressed sensing (covered later in Section 7-9)
becomes very powerful when applied to images.
7-7.1 Decimation and Zero-Stuffing of Images
The concepts of downsampling (decimation) and upsampling
(zero-stuffing) and interpolation generalize directly from 1-D
WAVELETS AND COMPRESSED SENSING
x[n, m]
(3,2)
x[3n, 2m].
The downsampling factor in this case is 3 along the horizontal
direction and 2 along the vertical direction. To illustrate the
process, we apply it to a 5 × 7 image:


1 2 3 4 5 6 7


1 4 7
 8 9 10 11 12 13 14


15 18 21 .
(3,2)
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 32 35
29 30 31 32 33 34 35
Upsampling in 2-D involves upsampling in both directions.
Upsampling a signal x[n, m] by a factor 3 along the horizontal
and by a factor 2 along the vertical, for example, is denoted
symbolically as
x[n, m]
(3,2)
 n m

x 3 , 2

0
for n = integer multiple of 3
and m = integer multiple of 2,
otherwise.
Applying this upsampling operation to a simple 2 × 2 image
yields


1 0 0 2
1 2
0 0 0 0
(3,2)
3 0 0 4 .
3 4
0
0 0
0
Downsampling and upsampling are used extensively in image
processing operations.
7-7.2 Image Analysis Filter Bank
The generalization of the wavelet transform from signals to
images is straightforward if separable scaling and wavelet
functions are used. In this book we restrict our attention to
separable functions; i.e., the 2-D filter g2 [n, m] = g[n] g[m]. With
g[n] and h[n] denoting scaling and wavelet functions, such as
the Haar or Daubechies functions, the image analysis filter bank
performs the following operations:
7-7 2-D WAVELET TRANSFORM
229
(1) Stage-1 decomposition
up to the largest (in size) three detail images:
(1)
x[n, m]
g[n] g[m]
(1)
xeLL [n, m]
(2,2)
x[n, m]
g[n] h[m]
(2,2)
x[n, m]
h[n] g[m]
(2,2)
x[n, m]
h[n] h[m]
(2,2)
(7.99a)
(1)
xeLH [n, m]
(7.99b)
(1)
xeHL [n, m]
(7.99c)
(1)
xeHH [n, m]
(7.99d)
(1)
(1)
{ xeLH [n, m], xeHL [n, m], xeHH [n, m] }.
(k)
The average images xeLL [n, m] are analogous to the average
signals Xek [m] of a signal, except that they are low-resolution
versions of a 2-D image x[n, m] instead of a 1-D signal x[m].
In 2-D, there are now three detail images, while in 1-D there is
only one detail signal.
In 1-D, the detail signals are zero except near edges, representing abrupt changes in the signal or in its slope. In 2-D, the
three detail images play the following roles:
(k)
(a) xeLH [n, m] picks up vertical edges,
(2) Stage-2 to stage-K decomposition
(k)
(b) xeHL [n, m] picks up horizontal edges,
(k)
(1)
xeLL [n, m]
(1)
xeLL [n, m]
(1)
xeLL [n, m]
g[n] g[m]
(2,2)
(2)
xeLL [n, m]
(7.100a)
g[n] h[m]
(2,2)
(2)
xeLH [n, m]
(7.100b)
h[n] g[m]
(2,2)
(2)
xeHL [n, m]
(7.100c)
(c) xeHH [n, m] picks up diagonal edges.
7-7.3 Image Synthesis Filter Bank
The image synthesis filter bank combines all of the detail
(K)
images, and the coarsest average image, xeLL [n, m], into the
original image x[n, m], as follows:
(K)
xeLL [n, m]
(K)
(7.100d)
xeLH [n, m]
The decomposition process is continued as above, until output
xeKLL [n, m] is reached, where K is the total number of stages.
xeHH [n, m]
(1)
xeLL [n, m]
h[n] h[m]
(2,2)
(2)
xeHH [n, m]
(3) Final wavelet transform
At the conclusion of the decomposition process, x[n, m] consists
of:
(K)
xeHL [n, m]
(K)
(b) Three detail images at each stage:
(K−1)
(K)
(K−1)
(K−1)
(K−1)
{ xeLH [n, m], xeHL [n, m], xeHH [n, m] }
(K−2)
(K−2)
(K−2)
{ xeLH [n, m], xeHL [n, m], xeHH [n, m] },
g[−n] g[−m]
ALL [n, m]
(2,2)
g[−n] h[−m]
ALH [n, m]
(2,2)
h[−n] g[−m]
AHL [n, m]
(2,2)
h[−n] h[−m]
AHH [n, m]
(K−1)
Average signal xeLL
outputs:
xeLL
(K)
(a) The coarsest average image xeLL [n, m].
(K)
(2,2)
(K)
(K)
(K)
[n, m] is the sum of the above four
(K)
(K)
[n, m] = ALL [n, m] + ALH [n, m]
(K)
(K)
+ AHL [n, m] + AHH [n, m].
(K)
(K)
(K)
(7.101)
Here { ALL [n, m], ALH [n, m], AHL [n, m], AHH [n, m] } are just
four temporary quantities to be added. The analogy to signal
analysis and synthesis filter banks is evident, except that at each
stage there are three detail images instead of one detail signal.
www.dbooks.org
230
CHAPTER 7
WAVELETS AND COMPRESSED SENSING
Repetition of the reconstruction step represented by
Eq. (7.101) through an additional K − 1 stages leads to
50
(0)
x[n, m] = xeLL [n, m].
The condition for perfect reconstruction is the 2-D version of
the Smith-Barnwell condition defined by Eq. (7.60), namely
100
150
|H2 (Ω1 , Ω2 )|2 + |H2(Ω1 − π , Ω2 − π )|2
+|H2 (Ω1 − π , Ω2)|2 + |H2(Ω1 , Ω2 − π )|2 = 4,
(7.102)
where H2 (Ω1 , Ω2 ) is the DSFT of the 2-D wavelet function
h2 [n, m] = h[n] h[m]. Since the 2-D wavelet function h2 [n, m] is
separable, its DSFT is also separable:
H2 (Ω1 , Ω2 ) = H(Ω1 ) H(Ω2 ),
200
250
50
100
150
200
250
(a) 256 × 256 Shepp-Logan phantom image
(7.103)
where H(Ω1 ) is the DTFT of h[n]. The 2-D Smith-Barnwell
condition is satisfied if the 1-D Smith-Barnwell condition given
by Eq. (7.60) is satisfied.
(4)
(4)
(4)
(4)
xLL xLH
(3)
xLH
xHL xHH
(2)
xLH
(3) (3)
xHL xHH
(1)
xLH
(2)
(2)
xHL
xHH
7-7.4 2-D Haar Wavelet Transform of
Shepp-Logan Phantom
The Shepp-Logan phantom is a piecewise-constant image that
has been a test image for tomography algorithms since the
1970s. A 256 × 256 image of the Shepp-Logan phantom is
displayed in Fig. 7-16(a). To illustrate what the coarse and
detail images generated by the application of the 2-D wavelet
transform look like, we applied a 4-stage 2-D Haar wavelet
transform to the image in Fig. 7-16(a). The results are displayed
per the pattern in Fig. 7-16(b), with:
(1)
(1)
xHL
xHH
(b)
Thumbnail image
50
(1) Stage-4 images: 16 × 16
(4)
The coarsest average image, xLL [n, m], is used as a thumbnail
image, and placed at the upper left-hand corner in Fig. 7-16(c).
The three stage-4 detail images are arranged clockwise around
(4)
image xLL [n, m].
100
150
200
(2) Stage-3 images: 32 × 32
The three stage-3 detail images are four times as large, and are
arranged clockwise around the stage-4 images.
(3) Stage-2 images: 64 × 64
250
50
100
150
200
250
(c) 4-stage 2-D Haar wavelet transform images
Figure 7-16 (a) 256 × 256 test image, (b) arrangement of
images generated by a 3-stage Haar wavelet transform, and (c)
the images represented in (b). A logarithmic scale is used to
display the values.
7-7 2-D WAVELET TRANSFORM
231
(4) Stage-1 images: 128 × 128
The largest images in Fig. 7-16(c) are the three stage-1 images. The number of pixels in the 2-D Haar transform can
be computed similarly to the 1-D Haar transform durations in
Eq. (7.46):
20
40
60
80
Stage 4 images: The 4 (16 × 16) images (1 average and 3 detail)
contain a total of 4(16)2 = 1024 pixels.
100
120
Stage 3 images: The 3 (32 × 32) detail images contain a total of
3(32)2 = 3072 pixels. The fourth 32 × 32 image is the average
image, which is decomposed into the stage 4 images.
140
160
180
Stage 2 images: The 3 (64 × 64) detail images contain a total of
3(64)2 = 12288 pixels. The fourth 64 × 64 image is the average
image, which is decomposed into the stage 3 images.
200
Stage 1 images: The 3 (128 × 128) detail images contain a total
of 3(128)2 = 49152 pixels. The fourth 128 × 128 image is the
average image, which is decomposed into the stage 2 images.
The total number of pixels in the wavelet transform of the
Shepp-Logan phantom is then:
20
1024 + 3072 + 12288 + 49152 = 65536.
80
20
40
60
80 100 120 140 160 180 200
(a) 200 × 200 clown image
40
60
100
This equals the number of pixels in the Shepp-Logan phantom,
which is 2562 = 65536.
120
140
Even though the coarsest average image is only 16 × 16, it
contains almost all of the large numbers in the 2-D wavelet
transform of the original image, and captures most of its primary
features. That is why it is used as a thumbnail image. The pixel
values of the stage-4 detail images are almost entirely zeros. The
3-stage composition process preserves the information content
of the original image—composed of 2562 = 65536 pixels—
while compressing it down to 4 images containing only 3619
nonzero pixels. This is a 94.5% reduction in the number of
nonzero pixels.
7-7.5 2-D db3 Daubechies Wavelet Transform of
Clown Image
For a second example, we repeated the steps outlined in
the preceding subsection, but this time we used a 2-D db3
Daubechies wavelet transform on the 200 × 200 clown image
shown in Fig. 7-17(a). The images generated by the 3-stage
decomposition process are displayed in part (b) of the figure,
using the same arrangement as shown earlier in Fig. 7-17(b).
160
180
200
20
40
60
80 100 120 140 160 180 200
(b) 2-D D3 Daubechies wavelet transform
of the clown image
Figure 7-17 (a) Clown image and (b) its 3-stage wavelettransform images. A logarithmic scale is used to display the
values.
7-7.6 Image Compression by Thresholding Its
Wavelet Transform
Image compression is an important feature of the wavelet
transform. Not only is the original image represented by fewer
pixels, but also many of the pixels of the wavelet-transform
images are zero-valued. The compression ratio can be improved
further by thresholding the output images of the final stage of
www.dbooks.org
232
CHAPTER 7
the wavelet-transform decomposition process. Thresholding a
pixel means replacing it with zero if its absolute value is below a
given threshold level λ . As noted earlier in connection with 1-D
signals and 2-D images, most of the wavelet-transform detail
signals and images have very small values, so little information
is lost by setting their values to zero, which means that they no
longer need to be stored, thereby reducing the storage capacity
needed to store the wavelet transform of the image. Furthermore,
since the wavelet transform is composed of orthogonal basis
functions, a small change in the wavelet transform of an image
will produce only a small change in the image reconstructed
from these values (see Section 7-2.3).
To illustrate with an example, we compare in Fig. 7-18 two
images:
20
40
60
80
100
120
140
160
180
200
(a) In part (a), we show the original 200 × 200 clown image,
and
(b) in part (b) we show a reconstructed clown image, generated
from the db3 Daubechies wavelet-transform images after
thresholding the images with λ = 0.11.
The reconstructed image looks almost identical to the original
image, even though only 6% of the pixels in the wavelettransform images are nonzero.
WAVELETS AND COMPRESSED SENSING
20
40
60
80 100 120 140 160 180 200
(a) 200 × 200 clown image
20
40
60
80
Concept Question 7-6: Why is the 2-D wavelet transform useful for generating thumbnail images?
100
120
140
7-8 Denoising by Thresholding and
Shrinking
160
180
200
A noisy image is given by
y[n, m] = x[n, m] + v[n, m],
20
40
60
80 100 120 140 160 180 200
(b) Reconstructed clown image
(7.104)
where x[n, m] is the desired image and v[n, m] is the noise that
had been added to it. The goal of denoising is to recover the
original image x[n, m], or a close approximation thereof, from
the noisy image y[n, m]. We now show that the obvious approach
of simply thresholding the wavelet transform of the image
does not work well. Then we show that the combination of
thresholding and shrinking the wavelet transform of the image
does work well.
7-8.1 Denoising by Thresholding Alone
One approach to denoising is to threshold the wavelet transform
of y[n, m]. For small wavelet transform values, the signal-to-
Figure 7-18 (a) Original clown image, and (b) image reconstructed from thresholded db3 Daubechies wavelet transform
images, requiring only 6% as much storage capacity as the
original image.
noise ratio is low, so little of value is lost by thresholding
these small values to zero. For large wavelet transform values,
the signal-to-noise ratio is large, so these large values should
be kept. This approach works poorly on wavelet transforms of
noisy images, as the following example shows.
Zero-mean 2-D white Gaussian noise with standard deviation
σ = 0.1 was added to the clown image of Fig. 7-19(a). The noisy
7-8 DENOISING BY THRESHOLDING AND SHRINKING
233
20
40
60
80
100
120
140
160
180
200
20
40
60
80 100 120 140 160 180 200
(a) Original noise-free image
20
20
40
40
60
60
80
80
100
100
120
120
140
140
160
160
180
180
200
20
40
60
200
80 100 120 140 160 180 200
(b) Noisy image
20
40
60
80 100 120 140 160 180 200
(c) Image denoised by thresholding its wavelet transform
Figure 7-19 (a) Noise-free clown image, (b) noisy image with SNR = 11.5, and (c) image reconstructed from thresholded wavelet
transform. Thresholding without shrinkage does not reduce noise.
image, shown in Fig. 7-19(b), has a signal-to-noise ratio (SNR)
of 11.5, which means that the noise level is, on average, only
about 8.7% of that of the signal.
The db3 Daubechies wavelet transform was computed for the
noisy image, then thresholded with λ = 0.11, which appeared
to provide the best results. Finally, the image was reconstructed
from the thresholded wavelet transform, and it now appears in
Fig. 7-19(c). Upon comparing the images in parts (b) and (c) of
the figure, we conclude that the thresholding operation failed to
reduce the noise by any appreciable amount.
7-8.2 Denoising by Thresholding and Shrinkage
We now show that a combination of thresholding small wavelettransform values to zero and shrinking other wavelet transform
values by a small number λ performs much better in denoising
images. First we show that shrinkage comes from minimizing
a cost functional, just as Wiener filtering given by Eq. (6.31)
www.dbooks.org
234
CHAPTER 7
came from minimizing the Tikhonov cost functional given by
Eq. (6.29).
In the material that follows, we limit our treatment to 1-D
signals. The results are readily extendable to 2-D images.
Suppose we are given noisy observations y[n] of a signal x[n]
that is known to be sparse (mostly zero),
y[n] = x[n] + v[n].
Λ=
y[n] = 1
λ=2
4
3
Λn
2
1
( y[n] − x[n])2
2
1
0
−1
λ|x[n]|
−0.5
0
0.5
1
1.5
2
x[n]
2
x[n]
(a)
2.5
2
1.5
y[n] = 1
λ = 1/3
Λn
1
( y[n] − x[n])2
2
1
0.5
λ|x[n]|
0
−1
−0.5
0
0.5
1
1.5
(b)
N−1
1
∑ (y[n] − x[n])2 + λ ∑ |x[n]|.
2 n=0
n=0
|
{z
}
| {z }
fidelity to data y[n]
5
(7.105)
All three signals have durations N and the noise v[n] is known
to have zero-mean. The goal is to estimate x[n] from the
noisy observations y[n]; i.e., to denoise y[n] with the additional
information that x[n] is mostly zero-valued.
The prior knowledge that x[n] is sparse can be incorporated
by estimating x[n] from y[n], not by simply using y[n] as an
estimator for x[n], but by minimizing over x[n] the LASSO cost
functional (discussed in more detail in Section 7-9.2).
N−1
WAVELETS AND COMPRESSED SENSING
(7.106)
Figure 7-20 Plots of the first and second terms of Λn , and their
sum for: (a) y[n] = 1 and λ = 2, and (b) y[n] = 1 and λ = 1/3.
sparsity
Readers familiar with basic estimation theory will note that Λ is
the negative log-likelihood function for zero-mean white Gaussian noise v[n] with independent Laplacian a priori distributions
for each x[n]. The coefficient λ is a trade-off parameter between
fidelity to the data y[n] and imposition of sparsity. If λ = 0,
then the estimator x̂[n] of x[n] is just x̂[n] = y[n]. Nonzero λ
emphasizes sparsity, while allowing some difference between
x̂[n] and y[n], which takes into account the noise v[n].
LASSO is an acronym for least absolute shrinkage and
selection operator.
7-8.3 Minimization of LASSO Cost Functional
The minimization of Λ decouples in time n, so each term can
be minimized separately. The goal then is to find an estimate of
x[n], which we denote x̂[n], that minimizes over x[n] the nth term
Λn of Eq. (7.106), namely
1
Λn = (y[n] − x[n])2 + λ |x[n]|.
2
consists of four possible scenarios:
(1) y[n] ≥ 0 and y[n] ≤ λ
(2) y[n] ≥ 0 and y[n] ≥ λ
(3) y[n] ≤ 0 and |y[n]| ≤ λ
(4) y[n] ≤ 0 and |y[n]| ≥ λ
Case 1: y[n] ≥ 0 and y[n] ≤ λ
Let us consider the following example for a particular value of n:
Measurement y[n] = 1
Trade-off parameter λ = 2
Signal x[n]: unknown
The estimated value x̂[n] of x[n] is found by minimizing Λn .
Figure 7-20(a) displays three plots, corresponding to the first
and second terms in Eq. (7.107), and their sum. It is evident
from the plot of Λn that Λn is minimized at x[n] = 0. Hence,
(7.107)
The expression given by Eq. (7.107) is the sum of two terms.
The value of λ is selected to suit the specific application; if
fidelity to the measured observations y[n] is highly prized, then λ
is assigned a small value, but if sparsity is an important attribute,
then λ may be assigned a large value.
Keeping in mind that λ ≥ 0, the minimization problem
x̂[n] = 0
for y[n] ≥ 0 and y[n] ≤ λ .
Case 2: y[n] ≥ 0 and y[n] ≥ λ
(7.108a)
Repetition of the scenario described by case 1, but with λ
changed from 2 to 1/3, leads to the plots shown in Fig. 7-20(b).
7-8 DENOISING BY THRESHOLDING AND SHRINKING
235
In this case, Λn is parabolic-like in shape, and its minimum
occurs at a positive value of x[n] at which the slope of Λn is
zero. That is,
1
d
dΛn
2
=
(y[n] − x[n]) + λ x[n] = 0,
dx[n] dx[n] 2
20
40
60
80
which leads to
100
120
x̂[n] = y[n] − λ ,
for y[n] ≥ 0 and y[n] ≥ λ .
(7.108b)
140
160
Case 3: y[n] ≤ 0 and |y[n]| ≤ λ
180
This case, which is identical to case 1 except that now y[n] is
negative, leads to the same result, namely
x̂[n] = 0,
for y[n] ≤ 0 and |y[n]| ≤ λ .
200
20
40
60
80 100 120 140 160 180 200
(a) Thresholding only
(7.108c)
20
40
Case 4: y[n] ≤ 0 and |y[n]| ≥ λ
60
Repetition of the analysis of case 2, but with y[n] negative, leads
to
80
100
x̂[n] = y[n] + λ ,
for y[n] ≤ 0 and |y[n]| ≥ λ .
The four cases can be combined into


for y[n] > +λ ,
y[n] − λ
x̂[n] = y[n] + λ
for y[n] < −λ ,

0
for |y[n]| < λ .
120
(7.108d)
140
160
180
200
(7.109)
20
40
60
80 100 120 140 160 180 200
(b) Thresholding and shrinkage
Values of y[n] smaller in absolute value than the threshold λ
are thresholded (set) to zero. Values of y[n] larger in absolute
value than the threshold λ are shrunk by λ , making their
absolute values smaller. So x̂[n] is computed by thresholding
and shrinking y[n]. This is usually called (ungrammatically)
“thresholding and shrinkage,” or “soft thresholding.”
The next example shows that denoising images works much
better with thresholding and shrinkage than with thresholding
alone.
When we applied thresholding alone to the noisy image of
Fig. 7-19(b), we obtained the image shown in Fig. 7-19(c),
which we repeat here in Fig. 7-21(a). Application of thresholding and shrinkage in combination, with λ = 0.11, leads to
the image in Fig. 7-21(b), which provides superior rendition
of the clown image by filtering much more of the noise, while
Figure 7-21 Denoising the clown image: (a) denoising by
thresholding alone, (b) denoising by thresholding and shrinkage
in combination.
preserving the real features of the image.
Concept Question 7-7: Why is the combination of
shrinkage and thresholding needed for noise reduction?
Concept Question 7-8: Why does wavelet-based denoising work so much better than lowpass filtering?
www.dbooks.org
236
CHAPTER 7
7-9 Compressed Sensing
In MRI, this reduces acquisition time inside the MRI machine,
and in smartphone cameras, it reduces the exposure time and
energy required to acquire an image.
This section presents the basic concepts behind compressed
sensing and applies these concepts to a few signal and image
inverse problems. Compressed sensing is an active area of research and development, and will experience significant growth
in applications in the future.
The solution of an inverse problem in signal and image processing is the reconstruction of an unknown signal or image from
measurements (known linear combinations) of the values of the
signal or image. Such inverse problems arise in medical imaging, radar imaging, optics, and many other fields. For example,
in tomography and magnetic resonance imaging (MRI), the
inverse problem is to reconstruct an image from measurements
of some (but not all) of its 2-D Fourier transform values.
If the number of measurements equals or exceeds the size
(duration in 1-D, number of pixels in 2-D) of the unknown
signal or image, solution of the inverse problem in the absence
of noise becomes a solution of a linear system of equations. In
practice, there is always noise in the measurements, so some sort
of regularization is required. In Section 6-4.3, the deconvolution
problem required Tikhonov regularization to produce a recognizable solution when noise was added to the data. Furthermore,
often the number of observations is less than the size of the
unknown signal or image. For example, in tomography, the 2-D
Fourier transform values of the image at very high wavenumbers
are usually unknown. In this case, the inverse problem is underdetermined; consequently, even in the absence of noise there is
an infinite number of possible solutions. Hence, regularization is
needed, not only to deal with the underdetermined formulation,
but also to manage the presence of noise in the measurements.
We have seen that many real-world signals and images can
be compressed, using the wavelet transform, into a sparse representation in which most of the values are zero. This suggests
that the number of measurements needed to reconstruct the
signal or image can be less than the size of the signal or image,
because in the wavelet-transform domain, most of the values
to be reconstructed are known to be zero. Had the locations of
the nonzero values been known, the problem would have been
reduced to a solution of a linear system of equations smaller
in size than that of the original linear system of equations. In
practice, however, neither the locations of the nonzero values
nor their values are known.
Compressed sensing refers to a set of signal processing
techniques used for reconstructing wavelet-compressible signals
and images from measurements that are much fewer in number
than the size of the signal or image, but much larger than the
number of nonzero values in the wavelet transform of the signal
or image. The general formulation of the problem is introduced
in the next subsection. There are many advantages to reducing
the number of measurements needed to reconstruct the signal or
image. In tomography, for example, the acquisition of a fewer
number of measurements reduces patient exposure to radiation.
WAVELETS AND COMPRESSED SENSING
7-9.1 Problem Formulation
To cast the compressed sensing problem into an appropriate
form, we define the following quantities:
(a) {x[n], n = 0 . . . N − 1} is an unknown signal of length N
(b) The corresponding (unknown) wavelet transform of x[n] is
{ xe1 [n], xe2 [n], . . . , xeL [n], XeL [n] },
and the wavelet transform of x[n] is sparse: only K values
of all of the {e
xk [n]} are nonzero, with K ≪ N.
(c) { y[n], n = 0, 1, . . . , M − 1 } are M known measurements
y[0] = a0,0 x[0] + a0,1 x[1] + · · · + a0,N−1 x[N − 1],
y[1] = a1,0 x[0] + a1,1 x[1] + · · · + a1,N−1 x[N − 1],
..
.
y[M − 1] = aM−1,0 x[0] + aM−1,1 x[1] + · · ·
+ aM−1,N−1 x[N − 1],
where {an,i , n = 0, 1, . . . , M − 1 and i = 0, 1, . . . , N − 1} are
known.
(d) K is unknown, but we know that K ≪ M < N.
The goal of compressed sensing is to compute signal {x[n],
n = 0, 1, . . . , N − 1} from the M known measurements {y[n],
n = 0, 1, . . . , M − 1}.
The compressed sensing problem can be divided into two
components, a direct problem and an inverse problem. In the
direct problem, the independent variable (input) is x[n] and the
dependent variable (output) is the measurement y[n]. The roles
are reversed in the inverse problem: the measurements become
the independent variables (input) and the unknown signal x[n]
becomes the output. The relationships between x[n] and y[n]
involve vectors and matrices:
7-9 COMPRESSED SENSING
237
A. Signal vector
x = [x[0], x[1], . . . , x[N − 1]]T,
(7.110)
where T denotes the transpose operator, which converts a row
vector into a column vector.
y = [y[0], y[1], . . ., y[M − 1]]T .
(7.111)
B. Wavelet transform vector

  
xe1 [n]
z1
 z2   xe [n] 
   2 
 ..   . 

 
z=
 .  =  ..  ,
 ..   xe [n] 
.  L 
zN
XeL [n]
(7.112)
z is of length N, and xe1 [n] to xeL [n] are the detail signals of the
wavelet transform and XeL [n] is the coarse signal.
C. Wavelet transform matrix
z = W x,
(7.113)
where W is a known N × N wavelet transform matrix that
implements the wavelet transform of x[n] to obtain z.
D. Direct-problem formulation
y = A x,
(7.114)
where A is an M × N matrix. Usually, A is a known matrix based
on a physical model or direct measurement of y for a known x.
Combining Eqs. (7.113) and (7.114) gives
y = A W−1 z = Aw z,
where
−1
Aw = A W .
(7.115a)
(7.115b)
If, somehow, z can be determined from the measurement vector y, then x can be computed by inverting Eq. (7.113):
z.
In seismic signal processing, explosions are set off on the
Earth’s surface, and echoes of the seismic waves created by
the explosion are measured by seismometers. In the 1960s,
sedimentary media (such as the bottom of the Gulf of Mexico)
were modeled as a stack of layers, so the seismometers would
record occasional sharp pulses reflected off of the interfaces
between the layers. The amplitudes and times of the pulses
would allow the layered medium to be reconstructed. However,
the occasional pulses had to be deconvolved from the source
pulse created by the explosions. The deconvolution problem was
modeled as an underdetermined linear system of equations.
A common approach to finding a sparse solution to a system
of equations is to choose the solution that minimizes the sum of
absolute values of the solution. This is known as the minimum
ℓ1 norm solution. The ℓ1 norm is denoted by the symbol ||z||1
and defined as
N
(7.117a)
i=1
E. Inverse-problem formulation
x=W
7-9.2 Inducing Sparsity into Solutions
||z||1 = ∑ |zi |.
Since A and W are both known matrices, Aw also is known.
−1
For the orthogonal wavelet transforms, such as the Haar
and Daubechies transforms covered in Sections 7-4 and 7-6,
W−1 = WT , so the inverse wavelet transform can be computed
as easily as the wavelet transform. In practice, both are computed using analysis and synthesis filter banks, as discussed in
earlier sections.
The crux of the compressed sensing problem reduces to
finding z, given y. An additional factor to keep in mind is that
only K values of the elements of z are nonzero, with K ≪ M.
Algorithms for computing z from y rely on iterative approaches,
as discussed in future sections.
Because z is of length N, y of length M, and M < N
(fewer measurements than unknowns), Eq. (7.115) represents an
underdetermined system of linear equations, whose solution is
commonly called an ill-posed problem.
The goal is to find the solution to the system of equations that
minimizes ||z||1 .
A second approach called the squared ℓ2 norm, which does
not provide a sparse solution, finds the solution that minimizes
the sum of squares of the solution. The squared ℓ2 norm is
denoted by the symbol ||z||22 and is defined as
(7.116)
www.dbooks.org
N
||z||22 = ∑ |zi |2 .
i=1
(7.117b)
238
CHAPTER 7
What is compressed sensing?
Concept Question 7-9:
7-10 Computing Solutions to
Underdetermined Equations
WAVELETS AND COMPRESSED SENSING
which is a straightforward linear programming problem that
can be solved using linprog in MATLAB’s Optimization
Toolbox. The basis pursuit method is limited to noise-free signal
and image problems, so in the general case, more sophisticated
approaches are called for.
7-10.2 LASSO Cost Functional
7-10.1 Basis Pursuit
As noted earlier, the unknown signal vector x can be determined
from Eq. (7.116), provided we have a solution for z (because
W is a known matrix). One possible approach to finding z is to
implement the minimum ℓ1 norm given by Eq. (7.117a), subject
to the constraint given by Eq. (7.115a). That is, the goal is to
find vector z from measurement vector y such that
A serious shortcoming of the basis pursuit solution method
is that it does not account for the presence of noise in the
observations y. As noted earlier in Section 7-8.2, the least
absolute shrinkage and selection operator (LASSOprovides
an effective approach for estimating the true signal from noisy
measurements. In the present context, the LASSO functional is
defined as
Λ = 21 ||y − Aw z||22 + λ ||z||1 ,
|
{z
} | {z }
N
∑ |zi | is minimum,
fidelity
i=1
and
y = Aw z.
The solution method is known as basis pursuit, and it can be
formulated as a linear programming problem by defining the
positive z+ and negative z− parts of z as:
(
+zi
if zi ≥ 0,
+
zi =
(7.118a)
0
if zi < 0,
(
−zi
if zi ≤ 0,
−
zi =
(7.118b)
0
if zi > 0.
(7.122)
sparsity
where λ is a trade-off parameter between sparsity of z and
fidelity to the measurement y. Choosing the solution z that
minimizes Eq. (7.122), for a specified value of λ , is called
basis pursuit denoising. The solution requires the use of an
iterative algorithm. Two such algorithms are presented in later
subsections, preceded by a short review of pseudo inverses.
7-10.3 Review of Pseudo-Inverses
(a) Overdetermined system
Consider the linear system of equations given by
Vector z is then given by
y = Aw z,
z = z+ − z− ,
and its ℓ1 norm is
N
−
||z||1 = ∑ (z+
i + zi ).
(7.120)
i=1
In terms of z+ and z− , the basis pursuit problem becomes:
N
Minimize ∑
−
(z+
i + zi )
i=1
subject to y = Aw z+ − Aw z− ,
(7.123)
(7.119)
−
(z+
i ≥ 0, zi ≥ 0),
(7.121)
with z of length N, y of length M, and Aw of size M × N
with full rank. If M > N, then the system is overdetermined
(more measurements than unknowns) and, in general, it has no
solution. The vector ẑ that minimizes ||y − Aw z||22 is called the
pseudo-inverse solution, and is given by the estimate
ẑ = (ATw Aw )−1 ATw y.
(7.124a)
Note that ATw Aw is an N × N matrix with full rank.
To avoid matrix-inversion problems, ẑ should be computed
not by inverting ATw Aw , but by solving the linear system of
equations
(ATw Aw ) ẑ = ATw y
(7.124b)
7-10 COMPUTING SOLUTIONS TO UNDERDETERMINED EQUATIONS
using the LU decomposition method or similar techniques.
Here, LU stands for lower upper, in reference to the lower triangular submatrix and the upper triangular submatrix multiplying
the unknown vector ẑ.
rewrite Eq. (7.128) in the expanded form
T =

1
2
(b) Underdetermined system
Now consider the underdetermined system characterized by
M < N (fewer measurements than unknowns). In this case, there
is an infinite number of possible solutions. The vector ẑ that
minimizes ||z||22 among this infinite number of solutions also is
called the pseudo-inverse solution, and is given by the estimate
ẑ = ATw (Aw ATw )−1 y.
(7.125)
In the present case, Aw ATw is an M × M matrix with full rank.
Solution ẑ should be computed not by inverting Aw ATw , but by
initially solving the linear system
(Aw ATw ) r̂ = y,
(7.126a)
to compute an intermediate estimate r̂, and then computing ẑ by
applying
ẑ = ATw r̂.
(7.126b)
7-10.4 Iterative Reweighted Least Squares
(IRLS) Algorithm
According to Eq. (7.115a), measurement vector y and wavelet
transform vector z are related by
y = Aw z.
(7.127)
 
a0,0
y[0]
 y[1]   a1,0
− .

..
  .

.
.
y[M − 1]
|
{z
fidelity
}
(7.128)
size
aM−1,0

D11





+λ 





  2
a0,N−1
z1
a1,N−1   z2 
 
  .. 
.

. . . aM−1, N−1 
 .. 
.
 
 .. 
.
zN 2
...
...
a0,1
a1,1
aM−1,1
M×1
M×N
..
N ×1
  2
z1
 z 
 2
 . 
 . 
 . 
 .  .
 . 
 . 
 . 
 . 
 .
zN
DNN
.
..
0
.
..
0
.
..
.
2
N×N
N ×1
(7.129)
where ai, j is the (i, j)th element of Aw , not of A.
Both the unknown vector x and its wavelet vector z are of size
N × 1. This is in contrast with the much shorter measurement
vector y, which is of size M × 1, with M < N.
We now introduce vector y′ and matrix Bw as
#
"
y
}M × 1
′
y =
,
(7.130a)
0
}N × 1
(M+N)×1
"
Aw
Bw = √
2λ D
The iterative reweighted least squares (IRLS) algorithm uses
the Tikhonov regularization functional given by Eq. (6.29),
together with a diagonal weighting matrix D to minimize the
cost function
T = 12 ||y − Aw z||22 + λ ||D z||22 ,
| {z }
239
#
(M+N)×N
}M × N
}N × N
.
(7.130b)
Vector y′ is vector y of length M stacked on top of vector 0,
which is a column vector of zeros of length N. Similarly, matrix
Bw is matrix Aw (of size M × N) stacked on√top of matrix D (of
size N × N), multiplied by the scalar factor 2λ .
where λ is the trade-off parameter between the size of z and the
fidelity to the data y. The goal is to trade off small differences
between y and Aw z so as to keep z small. To compute z, we first
www.dbooks.org
240
CHAPTER 7
Next, we introduce the new cost function T1 as
T1 =
1
2
=
1
2
LASSO functional given by Eq. (7.122) if
"
#
1
D = diag p
.
|zn |
2
y′ − Bw z 2
" # "
# 2
Aw
y
− √
z
0
2λ D
2
1
||y − Aw z||22 + λ ||0 − D z||22
2
1
= ||y − Aw z||22 + λ ||D z||22 = T .
2
N
(7.131)
(7.132)
The vector z minimizing T is the pseudo-inverse given by
=
ATw
#!−1
"
Aw
√
ATw
2 λ DT √
2λ D
= (ATw Aw + 2λ DT D)−1 ATw y.
N
z2n
= ∑ |zn | = ||z||1 .
n=1 |zn |
n=1
||D z||22 = ∑
(7.136)
Given this correspondence between the two cost functionals, the
IRLS algorithm uses the following iterative procedure to find z:
Hence, Eq. (7.128) can be rewritten in the form
ẑ = (BTw Bw )−1 BTw y′
(7.135)
This is because the second terms in the two equations become
identical:
=
T = 21 ||y − Bw z||22 .
WAVELETS AND COMPRESSED SENSING
" #
√
y
2 λ DT
0
(7.133)
As always, instead of performing matrix inversion (which is
susceptible to noise amplification), vector ẑ should be computed
by solving
(ATw Aw + 2λ DT D) ẑ = ATw y.
(7.134)
Once ẑ has been determined, the unknown vector x can be
computed by solving Eq. (7.113).
To solve Eq. (7.134) for ẑ, however, we need to know Aw , λ ,
D, and y. From Eq. (7.115b), Aw = A W−1 , where A is a known
matrix based on a physical model or calibration data, and W is
a known wavelet transform matrix. The parameter λ is specified
by the user to adjust the intended balance between data fidelity
and storage size (as noted in connection with Eq. (7.128)), and
y is the measurement vector. The only remaining quantity is
the diagonal matrix D, whose function is to assign weights to
z1 through zN so as to minimize storage size by having many
elements of z → 0. Initially, D is unknown, but it is possible
to propose an initial function for D and then iterate to obtain a
solution for z that minimizes the number of nonzero elements,
while still satisfying Eq. (7.134).
The Tikhonov function given by Eq. (7.128) reduces to the
(a) Initial solution: Set D = I and then compute z(1) , the initial
iteration of z, by solving Eq. (7.134).
(b) Initial D: Use z(1) to compute D(1) , the initial iteration
of D:


1
.
D(1) = diag  q
(7.137)
(1)
|zn | + ε
(c) Second iteration: Use D(1) to compute z(2) by solving
Eq. (7.134) again.
(d) Recursion: Continue to iterate by computing D(k) from
(k)
z using


1

(7.138)
D(k) = diag  q
(k)
|zn | + ε
for a small deviation ε inserted in the expression to keep D(k)
finite when elements of z(k) → 0.
The iterative process ends when no significant change occurs between successive iterations. The algorithm, also called
focal underdetermined system solver (FOCUSS), is guaranteed
to converge under mild assumptions. However, because the
method requires a solution of a large system of equations at
each iteration, the algorithm is considered unsuitable for most
signal and image processing applications. Superior-performance
algorithms are introduced in succeeding sections.
How can we get a useful solution to an underdetermined system of equations?
Concept Question 7-10:
7-11 LANDWEBER ALGORITHM
241
7-11 Landweber Algorithm
Continuing the pattern leads to
The Landweber algorithm is a recursive algorithm for solving
linear systems of equations y = Ax. The iterative shrinkage
and thresholding algorithm (ISTA) consists of the Landweber
algorithm,with thresholding and shrinkage applied at each recursion. Thresholding and shrinkage were used in Section 7-7
to minimize the LASSO functional.
7-11.1 Underdetermined System
For an underdetermined system y = Ax with M < N, the solution
x̂ that minimizes the sum of squares of the elements of x is, by
analogy with Eq. (7.125), given by
x̂ = AT (A AT )−1 y.
(7.139)
A useful relationship in matrix algebra states that if all of the
eigenvalues λi of (A AT ) lie in the interval 0 < λi < 2, then the
coefficient of y in Eq. (7.139) can be written as
∞
AT (AAT )−1 = ∑ (I − AT A)k AT .
(7.140)
k=0
x(k+1) = x(k) + AT (y − A x(k) ).
(7.144)
The process can be initialized by x(0) = 0, which makes
x(1) = AT y, as it should.
The recursion process is called the Landweber iteration,
which in optics is known as the van Cittert iteration. It is
guaranteed to converge to the solution of y = Ax that minimizes
the sum of squares of the elements of x, provided that the
eigenvalues λi of AAT are within the range 0 < λi < 2. If this
condition is not satisfied, the formulation may be scaled to
A
y
= x
c
c
or, equivalently,
u = Bx,
(7.145)
where u = y/c and B = A/c. The constant c is chosen so that
the eigenvalue condition is satisfied. For example, if c is chosen
to be equal to the sum of the squares of the magnitudes of all of
the elements of A, then the eigenvalues λi′ of BBT will be in the
range 0 < λi′ < 1.
The symbol λi for eigenvalue is unrelated to the trade-off
parameter λ in Eq. (7.128).
Using Eq. (7.140) in Eq. (7.139) leads to
7-11.2 Overdetermined System
∞
T
k
T
x̂ = ∑ (I − A A) A y.
(7.141)
k=0
In analogy with Eq. (7.124a), the solution for an overdetermined
system y = Ax is given by
A recursive implementation of Eq. (7.141) assumes the form
x
(K+1)
K
T
k
T
= ∑ (I − A A) A y,
(7.142)
x̂ = (AT A)−1 AT y.
(7.146)
Using the equality
k=0
∞
(AT A)−1 = ∑ (I − ATA)k ,
where the upper limit in the summation is now K (instead of ∞).
For K = 0 and K = 1, we obtain the expressions
x(1) = AT y,
(7.143a)
x(2) = AT y + (I − ATA)AT y = (I − AT A) x(1) + ATy.
(7.143b)
(7.147)
k=0
we can rewrite Eq. (7.146) in the same form as Eq. (7.141),
namely
∞
x̂ = ∑ (I − AT A)k AT y.
(7.148)
k=0
Extending the process to K = 2 gives
x(3) = AT y + (I − ATA)AT y + (I − ATA)2 AT y
|{z} |
{z
} |
{z
}
k=0
k=1
= x(2) + AT (y − A x(2) ).
Hence, the Landweber algorithm is equally applicable to solving
overdetermined systems of linear equations.
k=2
(7.143c)
www.dbooks.org
242
CHAPTER 7
7-11.3 Iterative Shrinkage and Thresholding
Algorithm (ISTA)
tions on ISTA, with names like SPARSA (sparse reconstruction
by separable approximation), FISTA (fast iterative shrinkage
and thresholding algorithm, and TWISTA (two-step iterative
shrinkage and thresholding algorithm).
For a linear system given by
y = A x,
(7.149)
where x is the unknown signal of length N and y is the (possibly
noisy) observation of length M, the LASSO cost functional is
Λ=
N−1
1 N−1
(y[n] − (Ax)[n])2 + λ ∑ |x[n]|,
∑
2 n=0
n=0
(7.150)
where matrix A is M × N and (A x)[n] is the nth element of A x.
◮ In the system described by Eq. (7.149), x and y are
generic input and output vectors. The Landweber algorithm
provides a good estimate of x, given y. The estimation
algorithm is equally applicable to any other linear system,
including the system y = Aw z, where Aw is the matrix given
by Eq. (7.115b) and z is the wavelet transform vector. ◭
The ISTA algorithm combines the Landweber algorithm with
the thresholding and shrinkage operation outlined earlier in
Section 7-8.3, and summarized by Eq. (7.109). After each
iteration, elements x(k) [n], of vector x(k) , whose absolute values
are smaller than the trade-off parameter λ are thresholded to
zero, and those whose absolute values are larger than λ are
shrunk by λ . Hence, the ISTA algorithm combines Eq. (7.144)
with Eq. (7.109):
x(0) = 0,
x
(k+1)
=x
(k)
(7.151a)
T
(k)
+ A (y − A x ),
(7.151b)
with
 (k+1)

−λ
xi
(k+1)
(k+1)
xi
= xi
+λ


0
(k+1)
(k+1)
if xi
> λ,
(k+1)
if xi
< −λ ,
(k+1)
| < λ,
if |xi
(7.151c)
where xi
is the ith component of x(k+1) .
The ISTA algorithm converges to the value of x that minimizes the LASSO functional given by Eq. (7.150), provided all
of the eigenvalues λi of AAT obey |λi | < 1.
The combination of thresholding and shrinking is often called
soft thresholding, while thresholding small values to zero without shrinking is called hard thresholding. There are many varia-
WAVELETS AND COMPRESSED SENSING
Why do we need an iterative
algorithm to find the LASSO-minimizing solution?
Concept Question 7-11:
7-12 Compressed Sensing Examples
To illustrate the utility of the ISTA described in the preceding
section, we present four examples of compressed sensing:
• Reconstruction of an image from some, but not all, of its
2-D DFT values.
• Image inpainting, which entails filling in holes (missing
pixel values) in an image.
• Valid deconvolution of an image from only part of its
convolution with a known point spread function (PSF).
• Tomography, which involves reconstruction of a 3-D image from slices of its 2-D DFT.
These are only a few of many more possible types of applications of compressed sensing.
The ISTA was used in all four cases, and the maximum
number of possible iterations was set at 1000, or fewer if the
algorithm converges to where no apparent change is observed in
the reconstructed images. The LASSO functional parameter was
set at λ = 0.01, as this value seemed to provide the best results.
7-12.1 Image Reconstruction from Subset of
DFT Values
Suppose that after computing the 2-D DFT of an image x[n, m],
some of the DFT values X[k1 , k2 ] were lost or no longer available. The goal is to reconstruct image x[n, m] from the partial
subset of its DFTs. Since the available DFT values are fewer
than those of the unknown signal, the system is underdetermined
and the application is a good illustration of compressed sensing.
For a 1-D signal { x[n], n = 0, 1, . . . , N − 1 }, its DFT can be
implemented by the matrix-vector product
y = A x,
(7.152)
where the (k, n)th element of A is Ak,n = e− j2π nk/N , the nth
element of vector x is x[n], and the kth element of vector y is
7-12 COMPRESSED SENSING EXAMPLES
243
X[k]. Multiplication of both sides by AH implements an inverse
1-D DFT within a factor 1/N.
The 2-D DFT of an N × N image can be implemented by
multiplication by an N 2 × N 2 block matrix B whose (k2 , n2 )th
block is the N × N matrix A multiplied by the scalar e− j2π n2k2 /N .
So the element Bk1 +Nk2 ,n1 +Nn2 of B is e− j2π n1 k1 /N e− j2π n2k2 /N ,
where 0 ≤ n1 , n2 , k1 , k2 ≤ N − 1.
The absence of some of the DFT values is equivalent to
deleting some of the rows of B and y, thereby establishing
an underdetermined linear system of equations. To illustrate
the reconstruction process for this underdetermined system, we
consider two different scenarios applied to the same set of data.
◮ For convenience, we call the values X[k1 , k2 ] of the 2-D
DFT the pixels of the DFT image. ◭
(a) Least-squares reconstruction
Starting with the (256 × 256) image shown in Fig. 7-22(a), we
compute its 2-D DFT and then we randomly select a subset of
the pixels in the DFT image and label them as unknown. The
complete 2-D DFT image consists of 2562 = 65536 pixel values
of X[k1 , k2 ]. Of those, 35755 are unaltered (and therefore have
known values), and the other 29781 pixels have unknown values.
Figure 7-22(b) displays the locations of pixels with known DFT
values as white dots and those with unknown values as black
dots.
In the least-squares reconstruction method, all of the pixels
with unknown values are set to zero, and then the inverse 2-D
DFT is computed. The resulting image, shown in Fig. 7-22(c),
is a poor rendition of the original image in part (a) of the figure.
stored during the implementation of the reconstruction process.
Multiplication by W−1 is implemented by a 2-D filter bank,
and multiplication by B is implemented by a 2-D FFT. Consequently, ISTA is a very fast algorithm.
7-12.2 Image Inpainting
In an image inpainting problem, some of the pixel values of an
image are unknown, either because those pixel values have been
corrupted, or because they represent some unwanted feature of
the image that we wish to remove. The goal is to restore the
image to its original version, in which the unknown pixel values
are replaced with the, hitherto, unknown pixel values of the
original image. This can be viewed as a kind of interpolation
problem.
It is not at all evident that this can be done at all—how can
we restore unknown pixel values? But under the assumption
that the Daubechies wavelet transform of the image is sparse
(mostly zero-valued), image inpainting can be formulated as a
compressed sensing problem. Let y be the vector of the known
pixel values and x be the vector of the wavelet transform of the
image. Note that all elements of x are unknown, even though
some of the pixel values are actually known. Then the problem
can be formulated as an underdetermined linear system y = Ax.
For example, if x is a column vector of length five, and only
the first, third, and fourth elements of x are known, the problem
can be formulated as
 
    
 x1
y1
x1
1 0 0 0 0 x2 

y2  = x3  = 0 0 1 0 0 
x3  .

y
x
0 0 0 1 0 x 
3
4
4
x5
(b) ISTA reconstruction
The ISTA reconstruction process consists of two steps:
(1) Measurement vector y, representing the 35755 DFT pixels
with known values, is used to estimate the (entire 65536)
wavelet transform vector z by applying the recipe outlined in
Section 7-11.3 with λ = 0.01 and 1000 iterations. The relationship between y and z is given by y = Aw z, with Aw = BW and
some rows of B deleted.
(2) Vector z is then used to reconstruct x by applying the
relation x = W−1 z.
The reconstructed image, displayed in Fig. 7-22(d), is an
excellent rendition of the original image.
It is important to note that while B and W−1 are each
(N 2 × N 2 ), with N = 65536, neither matrix is ever computed or
One application of image inpainting is to restore a painting
in which the paint in some regions of the painting has been
chipped off, scraped off, damaged by water or simply faded,
but most of the painting is unaffected. Another application is
to remove unwanted letters or numbers from an image. Still
another application is “wire removal” in movies, the elimination
of wires used to suspend actors or objects used for an action
stunt in a movie scene.
In all of these cases, damage to the painting, or presence of
unwanted objects in the image, has made some small regions
of the painting or image unknown. The goal is to fill in the
unknown values to restore the (digitized) painting or image to
its original version.
Using a (200 × 200) = 40000-pixel clown image, 19723 pixels were randomly selected and their true values were deleted.
www.dbooks.org
244
CHAPTER 7
50
50
100
100
150
150
200
200
250
WAVELETS AND COMPRESSED SENSING
250
50
100
150
200
250
50
100
150
200
250
(a) Original Shepp-Logan phantom image
(b) Locations of known values of X[k1,k2]
50
50
100
100
150
150
200
200
250
250
50
100
150
200
250
(c) Reconstructed image without ISTA
50
100
150
200
250
(d) Reconstructed image with ISTA
Figure 7-22 (a) Original Shepp-Logan phantom image, (b) 2-D DFT image with locations of pixels of known values displayed in white
and those of unknown values displayed in black, (c) reconstructed image using available DFT pixels and (d) reconstructed image after
filling in missing DFT pixel values with estimates provided by ISTA.
In the image shown in Fig. 7-23(a), the locations of pixels
with unknown values are painted black, while the remaining
half (approximately) have their correct values. The goal is to
reconstruct the clown image from the remaining half.
In terms of the formulation y = AWT z, M = 20277 and
N = 40000, so that just over half of the clown image pixel
values are known. The ISTA is a good algorithm to solve this
compressed sensing problem, since the matrix vector multiplication y = AWT z can be implemented quickly by taking the
inverse wavelet transform of the current iteration (multiplication
by WT ), and then selecting a subset of the pixel values (multiplication by A). The result, after 1000 iterations, is shown in
7-12 COMPRESSED SENSING EXAMPLES
245
(M + L − 1) × (M + L − 1) blurred image y[n, m]. The process is
reversible: the blurred image can be deconvolved to reconstruct
x[n, m] by subjecting y[n, m] to a Wiener filter, as described
earlier in Section 6-4.3. To do so, however, requires that all of
y[n, m] be known.
Often, we encounter deconvolution applications where only
a fraction of y[n, m] is known, specifically, the part of y[n, m]
called the valid convolution. This is the part whose convolution
computation does not require the image x[n, m] to be zero-valued
outside the square 0 ≤ n, m ≤ M − 1.
For L < M (image larger than PSF), the valid 2-D convolution
of h[n, m] and x[n, m] is defined as
20
40
60
80
100
120
140
160
180
200
M−1 M−1
20
40
60
yV [n, m] = ∑ ∑ x[i, j] h[n − i, m − j]
80 100 120 140 160 180 200
i=0 j=0
(a) Locations of known values of image
= h[n, m] ∗ x[n, m], restricted to
{ L − 1 ≤ n, m ≤ M − 1 }.
20
A valid convolution omits all end effects in 1-D convolution
and all edge effects in 2-D convolution. Consequently, the
size of yV [n, m] is (M − L + 1) × (M − L + 1), instead of
(M + L − 1) × (M + L − 1) for the complete convolution y[n, m].
To further illustrate the difference between y[n, m] and
yV [n, m], let us consider the following example:


1 2 3
x[n, m] = 4 5 6
7 8 9
40
60
80
100
120
140
160
180
and
200
(7.153)
20
40
60
80 100 120 140 160 180 200
(b) Reconstructed (inpainted) image
11 12
.
h[n, m] =
13 14
Since M = 3 and L = 2, the 2-D convolution is
Figure 7-23 (a) Locations of known values of clown image in
(M + L − 1) × (M + L − 1) = 4 × 4,
white and those of unknown values in black; (b) restored image.
and y[n, m] is given by


11 34 57 36
 57 143 193 114
y[n, m] = 
.
129 293 343 192
91 202 229 126
Fig. 7-23(b). The image has been reconstructed quite well, but
not perfectly. The db3 Daubechies wavelet transform was used
to sparsify the image.
7-12.3 Valid 2-D Deconvolution
In contrast, the size of the valid 2-D convolution is
(a) Definition of valid convolution
(M − L + 1) × (M − L + 1) = 2 × 2,
Given an M × M image x[n, m] and an L × L point spread
function (PSF) h[n, m], their 2-D convolution generates an
www.dbooks.org
246
CHAPTER 7
and yV [n, m] is given by
transposes of the rows are stacked into a column vector. Finally,
note that multiplication by AT can be implemented as a valid
2-D convolution with the doubly reversed version of h[n, m]. For
example, if


1 2 3 4
5 6 7 8
z[n, m] = 
9 10 11 12
13 14 15 16
yV [n, m] =
143 193
.
293 343
The valid convolution yv [n, m] is the central part of y[n, m],
obtained by deleting the edge rows and columns from y[n, m].
In MATLAB, the valid 2-D convolution of X and H can be
computed using the command
Y=conv2(X,H,’valid’).
(b) Reconstruction from yV [n, m]
The valid 2-D deconvolution problem is to reconstruct an unknown image from its valid 2-D convolution with a known PSF.
The 2-D DFT and Wiener filter cannot be used here, since not all
of the blurred image y[n, m] is known. It may seem that we may
simply ignore, or set to zero, the unknown parts of y[n, m] and
still obtain a decent reconstructed image using a Wiener filter,
but as we will demonstrate with an example, such an approach
does not yield fruitful results.
The valid 2-D deconvolution problem is clearly underdetermined, since the (M − L+ 1)× (M − L+ 1) portion of the blurred
image is smaller than the M × M unknown image. But if x[n, m]
is sparsifiable, then valid 2-D deconvolution can be formulated
as a compressed sensing problem and solved using the ISTA.
The matrix A turns out to be a block Toeplitz with Toeplitz
blocks matrix, but multiplication by A is implemented as a valid
2-D convolution. Multiplication by AT is implemented as a valid
2-D convolution.
The valid 2-D convolution can be implemented as yV = Ax,
where
x = [1 2 3 4 5 6 7 8 9]T ,
yV = [143 193 293 343]T ,
and the matrix A is composed of the elements of h[n, m] as
follows:


14 13 0 12 11 0 0 0 0
 0 14 13 0 12 11 0 0 0 
.
A=
0 0 0 14 13 0 12 11 0 
0 0 0 0 14 13 0 12 11
Note that A is a 2 × 3 block matrix of 2 × 3 blocks. Each
block is constant along its diagonals, and the blocks are constant
along block diagonals. This is the block Toeplitz with Toeplitz
blocks structure. Also note that images x[n, m] and yV [n, m] have
been unwrapped row by row, starting with the top row, and the
WAVELETS AND COMPRESSED SENSING
and
14 13
= h[1 − n, 1 − m],
g[n, m] =
12 11
with n, m = 0, 1,
then the valid 2-D convolution of z[n, m] and g[n, m] is


184 234 284
wv [n, m] = 384 434 484 .
584 634 684
This valid 2-D convolution can also be implemented as w = AT z
where
z = [1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16]T
and
w = [184 234 284 384 434 484 584 634 684]T.
To illustrate the process with an image, we computed the
valid 2-D convolution of the (200 × 200) clown image with a
(20 × 20) PSF. The goal is to reconstruct the clown image from
the (181 × 181) blurred image shown in Fig. 7-24(a). The db3
Daubechies wavelet function was used to sparsify the image.
Here, M = 200 and L = 20, so the valid 2-D convolution has size
(M − L + 1) × (M − L + 1) = 181 × 181. In terms of yV = Ax,
A is (1812 × 2002) = 32761 × 40000.
Parts (b) and (c) of Fig. 7-24 show reconstructed versions
of the clown image, using a Wiener filter and ISTA, respectively. Both images involve deconvolution using the restricted
valid convolution data yV [n, m]. In the Wiener-image approach,
the unknown parts of the blurred image (beyond the edges
of yV [n, m]) were ignored, and the resultant image bears no
real resemblance to the original clown image. In contrast, the
ISTA approach provides excellent reconstruction of the original
image. This is because ISTA is perfectly suited for solving underdetermined systems of linear equations with sparse solutions.
7-12 COMPRESSED SENSING EXAMPLES
247
20
40
60
80
100
120
140
160
180
20
40
60
80 100 120 140 160 180
(a) Valid 2-D convolution yv[m,n] of clown image
20
20
40
40
60
60
80
80
100
100
120
120
140
140
160
160
180
180
200
20
40
60
80 100 120 140 160 180 200
(b) 2-D deconvolution using yv[m,n] and Wiener filter
200
20
40
60
80 100 120 140 160 180 200
(c) 2-D deconvolution using yv[m,n] and ISTA
Figure 7-24 (a) Valid 2-D convolution yV [n, m] of clown image, (b) deconvolution using Wiener filter, and (c) deconvolution using ISTA.
7-12.4 Computed Axial Tomography (CAT)
The basic operation of the CAT scanner was described in
the opening chapter of this book using Fig. 1-24, which we
reproduce here as Fig. 7-25. For a source-to-detection path
through a body at a radius r and at orientation θ , the path
attenuation is given by Eq. (1.18) as
p(r, θ ) =
Z ∞Z ∞
−∞ −∞
α (ξ , η ) δ (r − ξ cos θ − η sin θ ) d ξ d η ,
(7.154)
where α (ξ , η ) is the absorption coefficient of the body under
test at location (ξ , η ) in Cartesian coordinates or (r, θ ) in
polar coordinates. The impulse function δ (r − ξ cos θ − η sin θ )
dictates that only those points in the (ξ , η ) plane that fall along
the path specified by fixed values of (r, θ ) are included in the
integration.
The relation between p(r, θ ) and α (ξ , η ) is known as the
2-D Radon transform of α (ξ , η ). The goal of CAT is to
reconstruct α (ξ , η ) from the measured path attenuations p(r, θ ),
by inverting the Radon transform given by Eq. (7.154). We do
so with the help of the Fourier transform.
www.dbooks.org
248
CHAPTER 7
X-ray
source
WAVELETS AND COMPRESSED SENSING
Recall from entry #1 in Table 2-5 that for variable r,
F {δ (r)} = 1, and from entry #3 in Table 2-4 that the shift
property is
F {x(r − r0 )} = X( f ) e− j2π f r0 .
Fan beam
of X-rays
The combination of the two properties leads to
F {δ (r − ξ cos θ − η sin θ )}
Detector
array
Z ∞
=
=e
δ (r − ξ cos θ − η sin θ ) e− j2π f r dr
0
− j2π f (ξ cos θ +η sin θ )
= e− j2π (µξ +νη ) ,
(7.155)
where we define spatial frequencies µ and ν as
Computer
and monitor
µ = f cos θ ,
ν = f sin θ .
(a) CAT scanner
η
Next, let us define A(µ , ν ) as the 2-D Fourier transform of the
absorption coefficient α (ξ , η ) using the relationship given by
Eq. (3.16a):
X-ray
source
I0
η0
I(ξ0,η0)
X-ray detector
Object
0
Absorption
coefficient
α(ξ,η)
I(ξ0,η0) = ξ
0
I0 exp(− ∫ α(ξ,η0) dξ)
0
ξ0
(b) Horizontal path
0
(7.156a)
(7.156b)
ξ
A(µ , ν ) =
Z ∞Z ∞
−∞ −∞
If we know A(µ , ν ), we can perform an inverse 2-D Fourier
transform to retrieve α (ξ , η ). To do so, we need to relate
A(µ , ν ) to the measured path attenuation profiles p(r, θ ). To that
end, we use Eq. (7.154) to compute P( f , θ ), the 1-D Fourier
transform of p(r, θ ):
P( f , θ ) =
Z ∞
0
η
=
X-ray source
I0
α (ξ , η ) e− j2π µξ e− j2πνη d ξ d η . (7.157)
Z ∞
0
p(r, θ ) e− j2π f r dr
"Z Z
∞
∞
−∞ −∞
α (ξ , η )
#
· δ (r − ξ cos θ − η sin θ ) d ξ d η e− j2π f r dr.
r
θ
(7.158)
α(ξ,η)
I(r,θ)
Detector
By reversing the order of integration, we have
ξ
(c) Path at radius r and orientation θ
Figure 7-25 (a) CAT scanner, (b) X-ray path along ξ , and (c)
X-ray path along arbitrary direction.
P( f , θ ) =
Z ∞Z ∞
−∞ −∞
·
Z ∞
0
α (ξ , η )
δ (r − ξ cos θ − η sin θ ) e
− j2π f r
dr d ξ d η .
(7.159)
We recognize the integral inside the square bracket as the
7-12 COMPRESSED SENSING EXAMPLES
249
Fourier transform of the shifted impulse function, as given by
Eq. (7.155). Hence, Eq. (7.159) simplifies to
P( f , θ ) =
Z ∞Z ∞
−∞ −∞
α (ξ , η ) e− j(2π µξ +2πνη ) d ξ d η ,
which is identical to Eq. (7.157). Hence,
A(µ , ν ) = P( f , θ ),
50
(7.160)
100
(7.161)
where A(µ , ν ) is the 2-D Fourier transform of α (ξ , η ), and P
is the 1-D Fourier transform (with respect to r) of p(r, θ ). The
variables (µ , ν ) and ( f , θ ) and related by Eq. (7.156).
If p(r, θ ) is measured for all r across the body of interest and
for all directions θ , then its 1-D Fourier transform P( f , θ ) can
be computed, and then converted to A(µ , ν ) using Eq. (7.156).
The conversion is called the projection-slice theorem. In practice, however, p(r, θ ) is measured for only a finite number of
angles θ , so A(µ , ν ) is known only along radial slices in the
2-D wavenumber domain (µ , ν ). Reconstruction to find α (ξ , η )
from a subset of its 2-D Fourier transform values is a perfect
example of compressed sensing.
Image reconstruction from partial radial slices
To demonstrate the CAT reconstruction process, we computed
the 2-D DFT X[k1 , k2 ] of a 256 × 256 Shepp-Logan phantom
image, and then retained the data values corresponding to only
12 radial slices, as shown in Fig. 7-26(a). These radial slices
simulate P( f , θ ), corresponding to 12 radial measurements
p(r, θ ). In terms of y = Ax, the number of pixels in the frequency
domain image is N = 65536, and the number of values contained
in the 12 radial slices is M = 11177. The Haar transform was
used to sparsify the image.
150
200
250
50
100
150
200
250
(a) Locations of known values of X[k1,k2]
50
100
150
200
250
50
100
150
200
250
(b) Least-squares reconstruction
(a) Least-squares reconstruction: Unknown values of X[k1 , k2 ]
were set to zero, and then the inverse 2-D DFT was computed.
The resulting image is displayed in Fig. 7-26(b).
50
(b) ISTA reconstruction: Application of ISTA with λ = 0.01
for 1000 iterations led to the image in Fig. 7-26(c), which bears
very good resemblance to the original image.
100
150
200
250
50
100
150
200
250
(c) ISTA reconstruction
Figure 7-26 Shepp-Logan phantom image reconstruction
from partial radial slices of its 2-D DFT: (a) radial slices
of X[k1 , k2 ], (b) least-squares reconstruction, and (c) ISTA
reconstruction.
www.dbooks.org
250
CHAPTER 7
WAVELETS AND COMPRESSED SENSING
Summary
Concepts
• The wavelet transform of an image is an orthonormal
expansion of the image using basis functions that are
localized in wavenumber or space.
• The 1-D wavelet transform of a piecewise-polynomial
signal is sparse (mostly zero-valued). This is why it is
useful.
• The wavelet and inverse wavelet transforms are implemented using filter banks and cyclic convolutions. This
makes their computation very fast.
• The filters in the tree-structured filter banks used to
implement wavelet and inverse wavelet transforms must
satisfy the Smith-Barnwell condition for perfect reconstruction, and also form a quadrature-mirror pair.
• An image can be compressed by thresholding its 2-D
wavelet transform.
• An image can be denoised by thresholding and shrinking
its 2-D wavelet transform. This preserves edges while
reducing noise. Thresholding and shrinkage minimizes
the LASSO cost functional, which favors sparsity.
• Compressed sensing allows an image to be reconstructed
from fewer linear combinations of its pixel values than
the number of pixel values, using the ISTA algorithm or
(rarely) basis pursuit.
• The ISTA algorithm applies thresholding and shrinkage
at each iteration of the Landweber algorithm.
• Applications of compressed sensing include: tomography, image inpainting, and valid deconvolution.
Mathematical Formulae
Zero-stuffing
(
x[n/2]
y[n] =
0
for n even
for n odd
Haar functions
Decimation
y[n] = x[2n]
1
g[n] = √ { 1, 1 }
2
QMF relation
g[n] = −(−1)n h[L − n]
Important Terms
average image
basis pursuit
compressed sensing
cyclic convolution
Daubechies
dbK
Smith-Barnwell condition
(h[n] ∗ h[−n])(−1)n + (h[n] ∗ h[−n]) = 2δ [n]
and
1
h[n] = √ { 1, −1 }
2
Provide definitions or explain the meaning of the following terms:
decimation
detail image
Haar
ISTA algorithm
Landweber algorithm
LASSO functional
orthonormal basis
quadrature mirror filter
pair
Shepp-Logan phantom
shrinkage
Smith-Barnwell condition
sparse
subband decomposition
thresholding
tree-structured filter
banks
zero-stuffing
PROBLEMS
251
PROBLEMS
Section 7-2: Expansions of Signals in Orthogonal
Basis Functions
7.1 The continuous-time Haar functions are defined as
(
1
for 0 < t < 1,
φ (t) =
0
otherwise,


for 0 < t < 21 ,
1
ψ (t) = −1
for 21 < t < 1,

0
otherwise,
ψm,n (t) = 2m/2 ψ (2mt − n).
7.7 Why are time-reversals used in the synthesis filters? Show
that using g[n] and h[n] instead of g[−n] and h[−n] for the
synthesis filters and then h[n] = (−1)n g[n] for the QMF, perfect
reconstruction is possible only if h[n] and g[n] have DTFTs
(
1
for 0 ≤ Ω < π2 ,
G(Ω) =
0
for π2 < Ω < π ,
and
H(Ω) =
(
0
1
for 0 ≤ Ω < π2 ,
for π2 < Ω < π ,
which constitutes an octave-band filter bank. Note that
h[−n] = h[n] and g[−n] = g[n]. Hint: Replace g[−n] with g[n]
and h[−n] with h[n] in Eq. (7.48).
Let B = {φ (t), ψm,n (t), m, n integers} and let F be the set
of piecewise-constant functions with support (nonzero region)
0 ≤ t ≤ 1 whose values change at t = m/2N .
(a) Show that any member of F is a linear combination of
elements of B.
(b) Show that B is an orthonormal basis for F. Hint: Draw
pictures.
7.8 Repeat Problem 7.7, except now change the synthesis
filters from g[−n] and h[−n] to g[n] and −h[n], and then use
h[n] = (−1)n g[n] for the new QMF.
(a) Show that one equation for perfect reconstruction is now
automatically satisified.
(b) Show that the other equation still cannot be satisfied by any
g[n].
7.2 Let B = {e j2π kt , k integer, 0 ≤ t ≤ 1} and F be the set of
continuous functions with support (nonzero region) 0 ≤ t ≤ 1.
Show that B is an orthonormal basis for F.
Hint: Replace g[−n] with g[n] and h[−n] with −h[n] in
Eq. (7.48) and use Eq. (7.34).
Section 7-4: Haar Wavelet Transforms
Section 7-6: Sparsification Using Wavelets of
Piecewise Polynomial Signals
7.3 Let x[n] = {4, 4, 4, 1, 1, 1, 1, 7, 7, 7, 7, 5, 5, 5, 5, 4}.
(a) Compute all of the signals in the Haar analysis filter bank
Fig. 7-8.
(b) Check your answers using Rayleigh’s (Parseval’s) theorem.
You may use MATLAB.
7.9 Use the Smith-Barnwell condition given by Eq. (7.62) to
design the db2 Daubechies wavelet function. Confirm that your
answer matches the coefficients listed in Table 7-3. Do this by
equating coefficients of time n. You should get a large linear
system of equations and a small nonlinear system of equations.
7.4 Let x[n] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16}.
(a) Compute all of the signals in the Haar analysis filter bank
Fig. 7-8.
(b) Check your answers using Rayleigh’s (Parseval’s) theorem.
You may use MATLAB.
7.10 Use the Smith-Barnwell condition given by Eq. (7.62) to
design the db2 Daubechies wavelet function. Confirm that your
answer matches the coefficients listed in Table 7-3. Use q[n] =
{q[0], q[1]} = q[0]{1, b}, where b = q[1]/q[0]. This avoids the
large linear systems of equations and the simultaneous quadratic
equations of the previous problem.
Section 7-5: Discrete-Time Wavelet Transforms
Section 7-7: 2-D Wavelet Transform
h[n] = {a, b, 21 , − 12 }.
Find a, b such that h[n] satisfies
7.5
Smith-Barnwell condition given by Eq. (7.57).
7.6 If h[n] = {a, b, c, d}, find g[n] such that g[n] and h[n] are a
QMF pair given by Eq. (7.52a).
7.11 Use haar.m to compute the 2-D Haar transform of the
image in letters.mat. Set sigma=0 and lambda=0 in the
first line of haar.m. Also depict the image reconstructed from
the wavelet transform.
www.dbooks.org
252
CHAPTER 7
7.12 Use daub.m to compute the 2-D db3 transform of the
SAR image in sar.mat. Change the first line to
(a) Download and run the program daub.m. This adds noise
to the SAR image, computes its 2-D db3 transform,
thresholds and shrinks this wavelet transform, computes
the inverse 2-D db3 wavelet transform of the result, and
displays images. Change the first line to
load sar.mat;sigma=50;lambda=100;
The threshold and shrinkage uses λ = 100, and the signalto-noise ratio is about 6.1.
(b) Why does this work better than the 2-D DFT or convolution
with a lowpass filter?
load sar.mat;sigma=0;lambda=0;
Also depict the image reconstructed from the wavelet transform.
Section 7-8: Wavelet-Based Denoising by
Thresholding and Shrinkage
7.13 This problem investigates denoising the letters image
using the wavelet transform, by thresholding and shrinking the
2-D Haar transform of the noisy image.
(a) Run haar.m. This adds noise to the image in
letters.mat, computes its 2-D Haar transform,
thresholds and shrinks this wavelet transform, computes
the inverse 2-D Haar wavelet transform of the result, and
displays images. The threshold and shrinkage uses λ = 70.
(b) Why does this work better than the 2-D DFT or convolution
with a lowpass filter?
7.14 This problem investigates denoising an MRI head image
using the wavelet transform, by thresholding and shrinking the
2-D Haar transform of the noisy image.
(a) Run the program haar.m. This adds noise to the MRI
image, computes its 2-D Haar transform, thresholds and
shrinks this wavelet transform, computes the inverse 2-D
Haar wavelet transform of the result, and displays images.
Change load letters.mat to load mri.mat;.
The threshold and shrinkage uses λ = 150.
(b) Why does this work better than the 2-D DFT or convolution
with a lowpass filter?
7.15 This problem investigates denoising an MRI head image
using the wavelet transform, by thresholding and shrinking the
2-D db3 transform of the noisy image.
(a) Download and run the program daub.m. This adds noise
to the MRI image, computes its 2-D db3 transform, thresholds and shrinks this wavelet transform, computes the inverse 2-D db3 wavelet transform of the result, and displays
images. Change the first line to
load mri.mat;sigma=50;lambda=100;
The threshold and shrinkage uses λ = 100.
(b) Why does this work better than the 2-D DFT or convolution
with a lowpass filter?
7.16 This problem investigates denoising an SAR image using
wavelet transforms, by thresholding and shrinking the 2-D db3
transform of the noisy image.
WAVELETS AND COMPRESSED SENSING
Section 7-9: Compressed Sensing
7.17 Even if a compressed sensing problem is only slighlty
underdetermined, and it has a mostly sparse solution, there is no
guarantee that the sparse solution is unique. The worst case for
compressed sensing is as follows. Let:
(a) am,n = e− j2π mn/N , n = 0, . . . , N − 1, m = 0, . . . , N − 1; skip
every multiple of N/L.
(b)
(
1
for n is a multiple of L,
xn =
0
for n not a multiple of L.
(c) For N = 12 and L = 4: m = {1, 2, 4, 5, 7, 8, 10, 11} and
{n = 0, 4, 8}, so that in this case A is 8 × 12 and the sparsity
(number of nonzero xn ) is K = 3.
Show that yn = 0! {xn } is called the Dirac comb. Its signficance:
Let xn be any signal of length 12 with nonzero elements only at
{n = 0, 4, 8}. Let ym = ∑11
n=0 am,n xn . Then adding the Dirac comb
to {xn } won’t alter {ym }, so {xn } plus any multiple of the Dirac
comb is a K-sparse solution.
Section 7-10: Computing Solutions to
Underdetermined Systems
7.18 Derive the pseudoinverse given by Eq. (7.124), which is
the solution x̂ to the overdetermined linear system of equations
y = Ax that minimizes E = ||y − Ax||22 . Perform the derivation
by letting x = x̂ + δ for an arbitrary δ and showing that the
coefficient of δ must be zero.
7.19 Derive Eq. (7.125), which is the solution x̂ to the underdetermined linear system of equations y = Ax that minimizes
E = ||x̂||22 . Perform the derivation by minimizing the Tikhonov
criterion given by Eq. (7.128), letting parameter λ → 0, applying
the pseudoinverse given by Eq. (7.124), and using the matrix
identity (AT A + λ 2I)−1 AT = AT (AAT + λ 2I)−1 .
PROBLEMS
253
Section 7-12: Compressed Sensing Examples
7.20 Free the clown from his cage. Run the program P720.m.
This sets horizontal and vertical bands of the clown image to
zero, making it appear that the clown is confined to a cage.
Free the clown: The program then uses inpainting to replace the
bands of zeros with pixels by regarding the bands of zeros as
unknown pixel values of the clown. Change the widths of the
bands of zeros and see how this affects the reconstruction.
7.21 De-square the clown image. Run program P721.m. This
sets 81 small squares of the clown image to zero, desecrating it.
The program then uses inpainting to replace the small squares
with pixels by regarding the 81 small squares as unknown pixel
values of the clown. Change the sizes of the squares and see how
this affects the reconstruction.
7.22 The tomography example at the end of Chapter 7 used 24
rays. Download and run the program P722.m, which generated
this example. Change the number of rays (M) from 24 to 32, and
then 16. Compare the compressed sensing reconstruction results
to the least-squares reconstructions (all unknown DFT values set
to zero).
www.dbooks.org
Chapter
8
Random Variables,
Processes, and Fields
8
n2
n1n2
P[n1n2]
1
11
a2
b
2
12
ab
c
3
13
ac
4
14
ad
Contents
8-1
8-2
8-3
8-4
8-5
8-6
8-7
8-8
8-9
8-10
8-11
Overview, 255
Introduction to Probability, 255
Conditional Probability, 259
Random Variables, 261
Effects of Shifts on Pdfs and Pmfs, 263
Joint Pdfs and Pmfs, 265
Functions of Random Variables, 269
Random Vectors, 272
Gaussian Random Vectors, 275
Random Processes, 278
LTI Filtering of Random Processes, 282
Random Fields, 285
Problems, 288
Objectives
n1 = 1
d
a
n1 = 2
b
a
c
d
■ Compute means, variances, and covariances for
random variables and vectors.
■ Compute autocorrelations and power spectral
densities of wide-sense-stationary random processes
in continuous and discrete time.
■ Use thresholding and shrinkage of its wavelet
transform to denoise an image.
■ Compute autocorrelations and power spectral
densities of wide-sense-stationary random fields.
a
d
n1 = 3
Learn to:
■ Compute conditional probabilities and density
functions.
a
d
a
n1 = 4
d
1
21
ba
b
2
22
b2
c
3
23
bc
4
24
bd
1
31
ca
b
2
32
cb
c
3
33
c2
4
34
cd
1
41
da
b
2
42
db
c
3
43
dc
4
44
d2
This chapter supplies a quick review of probability,
random variables, vectors, processes, and fields for
use in Chapter 9, which reviews estimation theory
and applies it to image estimation. Readers already
familiar with these topics may skip this chapter.
Overview
a coin is flipped, it is equally likely to produce a head or a tail,
then the probabilities of all four events are the same, namely
In this chapter, we offer a brief review of random variables, random processes, and random fields. A random field is a random
process in 2-D. These topics will all be used in Chapter 9, which
covers estimation and Markov random fields. To begin our
review, Section 8-1 provides a primer on the nomenclature used
in the language of probability, illustrated with several examples.
Section 8-2 presents conditional probability, which will play
an important role in Chapter 9. In Section 8-3, we introduce
probability density functions (pdf) and probability mass functions (pmf) for describing the distributions of 1-D random
variables, and then these tools are extended to 2-D in Sections
8-5 and 8-6. Sections 8-7 and 8-8 treat random vectors (vectors
of random variables), which are then extended to 1-D random
processes and 2-D random fields in later sections. A particular
emphasis is placed on wide-sense stationary (WSS) random
processes and fields.
8-1 Introduction to Probability
P[E1 ] = P[E3 ] = P[E3 ] = P[E4 ] =
1
.
4
Event Space A: The set of all subsets of sample space S to which
probabilities can be computed on the basis of the probabilities
assigned to the elements of S. If S has a finite number of
elements M, then A has 2M elements, each of which is a subset
of S. Included in A are both S itself and the null set 0;
/ that is,
S ∈ A and 0/ ∈ A.
8-1.1 Complement, Union, and Intersection
Event E2 = { H1 T2 } represents the outcome that the first coin
flip is a head and the second one is a tail, and its associated
probability of occurrence is P[E2 ]. The complement of event
E2 is E2′ , and it denotes the condition that the outcome is not
H1 T2 . Thus, the complement of an event includes all possible
outcomes of S except for that particular event:
A probability experiment is an experiment in which the outcome
is uncertain (random), but each possible outcome (event) has a
given or computable probability (likelihood) of occurring.
P[E2′ ] = P[S] − P[E2] = 1 − P[E2],
Sample Space S: The set of all distinguishable outcomes. For
a coin flipped once, S = { H, T }, where H denotes a “head”
outcome and T denotes a “tail” outcome. In this case, S consists
of M = 2 elements. For a coin flipped twice in a row, M = 4 and
where we used the probability axiom (Section 8-1.2) that the total probability of all events constituting S is 1. Probability P[E2′ ]
equals P[E1 ]+ P[E3 ]+ P[E4], since only one of { E1 , E3 , E4 } can
occur.
S = { E1 , E2 , E3 , E4 },
(8.1)
where E1 to E4 are outcomes (events) defined as:
(8.2)
◮ The union of two events Ei and E j is an OR statement: Ei occurring, E j occurring, or both, as illustrated by
Fig. 8-1(a). ◭
E1 = { H1 H2 }, E2 = { H1 T2 },
E3 = { T1 H2 }, E4 = { T1 T2 },
where H1 and H2 denote that the results of the first and second
flips are heads, and similarly for tails T1 and T2 .
Event Probability P: Each element of S, such as E1 through
E4 , is called an event. Events may also include combinations
of elements, such as
Ea = { E2 , E4 } = { H1 T2 , T1 T2 },
which in this case represents the outcome that the second coin
flip is a “tail,” regardless of the outcome of the first coin flip.
Associated with each event is a certain probability determined
by the conditions and constraints of the experiment. If each time
Ei
Ej
(a) Union: Ei U Ej
Ei
Ej
(b) Intersection: Ei I Ej
Figure 8-1 (a) The event of the union of two events Ei and E j
encompasses the combined elements of both events, whereas
(b) the event of their intersection includes only the elements
common to both of them.
255
www.dbooks.org
256
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
◮ The intersection of events Ei and E j is an AND statement
(Fig. 8-1(b)), it is denoted Ei ∩ E j , and its associated
probability P[Ei ∩ E j ] represents the condition that both Ei
and E j occur. ◭
For the two-time coin-flip experiment, the occurrence of any
one of the four events, E1 through E4 , negates the possibility
of occurrence of the other three. Hence, if E1 = H1 H2 and
E2 = H1 T2 ,
◮ Probabilities are assigned by the user for all elements of
the sample space S, consistent with the axioms of probability. Using these axioms, the probability of any element of
the event space A can then be computed. ◭
8-1.3 Properties of Unions and Intersections
Commutative property
Ei ∪ E j = E j ∪ Ei ,
Ei ∩ E j = E j ∩ Ei .
(8.8a)
(8.8b)
(Ei ∪ E j ) ∪ Ek = Ei ∪ (E j ∪ Ek ),
(Ei ∩ E j ) ∩ Ek = Ei ∩ (E j ∩ Ek ).
(8.9a)
(8.9b)
P[E1 ∩ E2 ] = P[H1 H2 ∩ H1 T2 ] = 0.
When the intersection Ei ∩ E j = 0,
/ the associated probability is
P[Ei ∩ E j ] = 0
(disjoint),
(8.3)
Associative property
in which case events Ei and E j are said to be disjoint.
8-1.2 Axioms of Probability
The axioms of probability are rules for computing P[E] for
any member E of event space A. The user must first assign
probabilities to each member of the sample space S that satisfies
those axioms.
Axiom 1: Probability of any event Ei is bounded between 0
and 1:
0 ≤ P[Ei ] ≤ 1.
(8.4)
Axiom 2: Total probability for all distinguishable events comprising S is 1:
M
P[S] = ∑ P[Ei ] = 1,
(8.5)
Distributive property
(Ei ∪ E j ) ∩ Ek = (Ei ∩ Ek ) ∪ (E j ∩ Ek ),
(Ei ∩ E j ) ∪ Ek = (Ei ∪ Ek ) ∩ (E j ∪ Ek ).
De Morgan’s law
For two events Ei and E j , De Morgan’s law states:
(E1 ∪ E2 )′ = E1′ ∩ E2′ ,
i=1
where M is the total number of elements in S.
′
Axiom 3: If Ei and E j are disjoint, then P[Ei ∩ E j ] = 0 and
P[Ei ∪ E j ] = P[Ei ] + P[E j ]
(Ei and E j disjoint).
(8.11b)
P[(E1 ∪ E2 )′ ] = P[E1′ ∩ E2′ ],
(8.12a)
(8.6)
(8.7a)
thus stating that the total probability of the presence and absence
of an event is 1. Similarly, the probability of the intersection of
an event Ei and its complement Ei′ is zero:
P[Ei ∩ Ei′ ] = 0.
(8.11a)
(E1 ∩ E2 ) = E1′ ∪ E2′ ,
and hence,
Also, the probability of the union of an event Ei with its
complement Ei′ is 1:
P[Ei ∪ Ei′ ] = 1,
(8.10a)
(8.10b)
(8.7b)
′
P[(E1 ∩ E2 ) ] = P[E1′ ∪ E2′ ],
(8.12b)
where as noted earlier, the prime denotes the complement
(absence) of the specified event. De Morgan’s law also leads to
these useful relationships:
(E1 ∩ E2 ) ∪ (E1′ ∩ E2 ) = (E1 ∪ E1′ ) ∩ E2 = E2 ,
(E1 ∩ E2 ) ∩ (E1′ ∩ E2 ) = (E1 ∩ E1′ ) ∩ E2 = 0.
/
(8.13a)
(8.13b)
8-1 INTRODUCTION TO PROBABILITY
257
Because (E1 ∩ E2 ) and (E1′ ∩ E2 ) are disjoint, the probability
relationship corresponding to Eq. (8.13a) is
P[E1 ∩ E2 ] + P[E1′ ∩ E2 ] = P[E2 ].
(8.14)
8-1.4 Probability Tree for Coin-Flip Experiment
Suppose a coin is flipped N times, and the result of the kth flip—
with 1 ≤ k ≤ N—is either a head and designated Hk , or a tail and
designated Tk .
Independent flips: The result of any coin flip has no effect on
the result of any other flip.
Probability of Union and Intersection Events
For two events, the probability of their union is equal to the sum
of their individual probabilities minus the probability of their
intersection:
P[Ei ∪ E j ] = P[Ei ] + P[E j ] − P[Ei ∩ E j ].
P[Hk ] = a,
P[Tk ] = 1 − a,
1 ≤ k ≤ N.
(8.15)
The relationship given by Eq. (8.15) can be derived by using
Fig. 8-1(a) to write (Ei ∪ E j ) as the union of three intersections:
Ei ∪ E j = (Ei ∩ E ′j ) ∪ (Ei′ ∩ E j ) ∪ (Ei ∩ E j ),
Biased coin: The coin is not necessarily a “fair” coin, meaning
that the probability of heads is not necessarily equal to the
probability of tails. Suppose the probability of heads for any
coin flip is a known value a, where 0 ≤ a ≤ 1. Thus,
(8.16)
For an unbiased coin, a = 0.5. A set designates a specific event,
such as H1 T2 , which can be written as
H1 T2 = H1 ∩ T2 .
and the corresponding probability is given by
P[Ei ∪ E j ] = P[Ei ∩ E ′j ] + P[Ei′ ∩ E j ] + P[Ei ∩ E j ].
(8.17)
Upon setting E1 and E2 in Eq. (8.14) as Ei and E j , respectively,
we obtain
P[Ei ∩ E j ] + P[Ei′ ∩ E j ] = P[E j ].
(8.18a)
Repeating the process but with E1 and E2 set in Eq. (8.14) as E j
and Ei (instead of as Ei and E j ), respectively, leads to
P[E j ∩ Ei ] + P[E ′j ∩ Ei ] = P[Ei ].
(8.18b)
Using Eqs. (8.18a and b) in Eq. (8.17) leads to Eq. (8.15).
To illustrate the meaning of Eq. (8.14), let us consider the
example of a coin toss with N = 2 times. If we use E1 to denote
that the first toss is a head and E2 to denote that the second toss
is a tail, then Eq. (8.14) becomes
P[H1 ∩ T2 ] + P[H1′ ∩ T2 ] = P[T2 ].
(8.20a)
(8.20b)
(8.19)
The first term is the probability that the first toss resulted in a
head and the second toss resulted in a tail, and the second term
is the probability that the first toss did not result in a head, but the
second one did result in a tail. The sum of the two probabilities
is equal to the probability that the second toss is a tail, P[T2 ],
regardless of the outcome of the first toss.
For N = 2, the sample space S has 2N = 22 = 4 elements:
S = { H1 ∩ H2 , H1 ∩ T2 , T1 ∩ H2 , T1 ∩ T2 },
N
and the event space A has 22 = 24 = 16 elements:
{S, H2 , T2 , T1 H2 , T1 T2 , (T1 T2 )′ , (T1 H2 )′ , H1 T2 ∪ T1 H2 }∪
{0,
/ H1 , T1 , H1 H2 , H1 T2 , (H1 H2 )′ , (H1 T2 )′ , H1 H2 ∪ T1 T2 }.
The probability tree of S is shown in Fig. 8-2. Each element
of S and its assigned probability are denoted at the end of each
branch of the tree. Examples of computing probabilities of union
events are given in Example 8-1.
Example 8-1:
Probability Tree for Coin-Flip
Experiment
For the coin-flip experiment with N = 2 and P[Hk ] = a, compute the probabilities of the following events: (a) H1 ∪ H2 , (b)
H1 H2 ∪ T1 T2 , and (c) H1 T2 ∪ T1 H2 .
Solution: (a) According to Eq. (8.15),
P[H1 ∪ H2 ] = P[H1 ] + P[H2] − P[H1 ∩ H2 ].
www.dbooks.org
258
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
a2
a
H2 = H1 I H2 = H1H2
H1
a
a′ = 1 − a
a(1 − a)
T2 =
H1 I T2 = H1T2
Start
a(1 − a)
a′ = 1 − a
a
H2 =
T1 I H2 = T1H2
T1
a′ = 1 − a
(1 − a)2
T2 =
T1 I T2 = T1T2
Figure 8-2 Probability tree for N = 2 for coin-flip experiment.
From the tree in Fig. 8-1,
P[H1 ] = P[H2 ] = a
and
P[H1 ∩ H2 ] = a2 .
Using the tree in Fig. 8-2 and noting that H1 T2 and T1 H2 are
disjoint events, we have
P[H1 T2 ∪ T1 H2 ] = a(1 − a) + (1 − a)a = 2a(1 − a).
8-1.5 Probability Tree for Tetrahedral Die
Experiment
A tetrahedral die is a four-sided object with one of the following
four numbers: 1, 2, 3, 4, printed on each of its four sides
(Fig. 8-3). When the die is rolled, the outcome is the number
printed on the bottom side.
Condition 1: The result of any die roll has no effect on the result
of any other roll.
Condition 2: The die is not necessarily a “fair” die. If we denote
n as the number that appears on the bottom of the die after it has
been rolled, the probabilities that n = 1, 2, 3, or 4 are
P[n = 1] = a,
P[n = 2] = b,
P[n = 3] = c,
P[n = 4] = d,
with the constraints that 0 ≤ a, b, c, d ≤ 1 and
a + b + c + d = 1.
For a fair die, a = b = c = d = 1/4.
Hence,
P[H1 ∪ H2 ] = a + a − a2 = a(2 − a).
(b)
P[H1 H2 ∪ T1 T2 ] = P[H1 H2 ] + P[T1T2 ] − P[H1H2 ∩ T1 T2 ].
From the tree in Fig. 8-2, P[H1 H2 ] = a2 and P[T1 T2 ] = (1 − a)2.
Furthermore, H1 H2 and T1 T2 are disjoint events, since they
cannot both occur. Hence, the probability of their intersection
is zero,
P[H1 H2 ∩ T1 T2 ] = 0,
3
1
and therefore
P[H1 H2 ∪ T1 T2 ] = a2 + (1 − a)2.
(c)
P[H1 T2 ∪ T1 H2 ] = P[H1 T2 ] + P[T1 H2 ] − P[H1T2 ∩ T1 H2 ].
Figure 8-3 Tetrahedral die with 4 sides displaying numerals 1
to 4. When the die is rolled, the outcome is the numeral on the
bottom side.
8-2 CONDITIONAL PROBABILITY
n1 = 1
a
d
a
n1 = 2
b
d
a
c
d
a
n1 = 3
d
a
n1 = 4
d
259
n2
n1n2
P[n1n2]
c = 0.3, and d = 0.4, compute the probabilities of (a) n1 is an
even number and (b) n1 + n2 = 3.
1
11
a2
b
2
12
ab
Solution: (a) The probability of n1 = even number is the sum
of the probabilities for n1 = 2 and n1 = 4:
c
3
13
ac
4
14
ad
1
21
ba
(b) Using the tree in Fig. 8-3,
b
2
22
b2
c
3
23
bc
P[n1 + n2 = 3] = P[n1 = 1 ∩ n2 = 2] + P[n1 = 2 ∩ n2 = 1]
= ab + ba = 2ab = 2 × 0.1 × 0.2 = 0.04.
4
24
bd
1
31
ca
b
2
32
cb
c
3
33
c2
4
34
cd
1
41
da
b
2
42
db
c
3
43
dc
4
44
d2
P[n1 = even] = P[n1 = 2] + P[n1 = 4]
= 0.2 + 0.4 = 0.6.
Concept Question 8-1:
What, exactly, is a probability
tree?
Exercise 8-1: For the coin flip experiment, what is the pmf
for n = # flips of all tails followed by the first head?
p[n] = a(1 − a)(n−1) since we require n − 1
consecutive tails before the first head. This is called a
geometric pmf .
Answer:
8-2 Conditional Probability
Figure 8-4 Probability tree for tetrahedral die experiment with
N = 2 rolls. Symbols a, b, c, and d are the probabilities that the
outcome of the tetrahedral roll is a 1, 2, 3, or 4, respectively.
When rolled N times, the sample space S of the tetrahedral die
has 4N elements, each of which has the form { n1 , n2 , . . . , nN },
where each n is one of the four numbers { 1, 2, 3, 4 }. For N = 2,
S has 42 = 16 elements:
{ n1 n2 } =
{ 11, 12, 13, 14, 21, 22, 23, 24, 31, 32, 33, 34, 41, 42, 43, 44 }.
The probability tree for the tetrahedral die experiment is
shown in Fig. 8-4 for N = 2.
For a given event, say E1 , its probability of occurrence is
denoted P[E1 ]. The value of P[E1 ] is calculated assuming
no foreknowledge that any other event has already occurred.
Sometimes, the prior occurrence of another event, say E3 ,
may impact the probability of occurrence of the current event
under consideration. Such a probability is called a conditional
probability, designated
P[E1 |E3 ],
and is read as “probability of E1 given E3 .” Figure 8-5 displays
the conditional probability tree for the coin flip experiment with
N = 2, and through Examples 8-3 and 8-4 we illustrate how to
compute the conditional probability for specific scenarios.
8-2.1 Conditional Probability Formula
The general form of the conditional probability formula for
events E1 and E2 is
Example 8-2: Tetrahedral Die Probabilities
For the tetrahedral die experiment with N = 2, a = 0.1, b = 0.2,
P[E1 |E2 ] =
www.dbooks.org
P[E1 ∩ E2 ]
P[E1 ∩ E2 ]
=
,
P[E2 ]
P[E1 ∩ E2 ] + P[E1′ ∩ E2 ]
(8.21)
260
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
P[H2|H1] = b
H1
P[H1H2] = ab
T1
P[H1T2] = a(1 − b)
H2
P[T1H2] = (1 − a)b
T2
P[T1T2] = (1 − a)(1 − b)
H1
P[H1] = a
P[T2|H1] = 1 − b
Start
P[H2|T1] = b
P[T1] = 1 − a
T1
P[T2|T1] = 1 − b
Figure 8-5 Conditional probability tree for a coin-flip experiment with N = 2. The top red path represents the outcome H1 H2 and the
bottom blue path represents the outcome T1 T2 .
where we used Eq. (8.14) in the second step.
and
P[Ei ∪ E j ] = P[Ei ] + P[E j ] (mutually exclusive).
8-2.2 Independent and Mutually Exclusive
Events
Two events Ei and E j are said to be independent if the probability of occurrence of either has no effect on the occurrence of the
other:
P[Ei |E j ] = P[Ei ],
P[E j |Ei ] = P[E j ],
(8.22a)
(8.22b)
P[Ei ∩ E j ] = P[Ei ] P[E j ].
(independent events)
(8.22c)
and
In the coin-flip experiment, it was assumed that the outcome of
each flip was independent of the outcomes of other flips. Hence,
P[H1 ] is independent of P[T2 ].
Two events are mutually exclusive if the occurrence of one
of them negates the possibility of the occurrence of the other.
Consequently,
P[Ei ∩ E j ] = 0 (mutually exclusive)
(8.23a)
(8.23b)
Table 8-1 provides a summary of the major terms and symbols
used in this section.
Example 8-3:
Coin-Flip Conditional Probability
Given that the result of the coin-flip experiment for N = 2 was
two heads or two tails, compute the conditional probability that
the result was two heads. Use a = b = 0.4.
Solution:
E1 = H1 H2
E2 = H1 H2 ∪ T1 T2 ,
(both heads),
(both heads or both tails).
The conditional probability that the result is both heads is
P[E1 |E2 ] =
P[E1 ∩ E2 ]
P[H1 H2 ]
=
P[E2 ]
P[H1 H2 ∪ T1 T2 ]
P[H1 H2 ]
,
=
P[H1 H2 ] + P[T1 T2 ]
8-3 RANDOM VARIABLES
261
Table 8-1 Probability symbols and terminology.
Term
Notation
Sample space
Event (examples)
Outcome (examples)
Empty set (impossible event)
Complement of E (“not E”)
Union of Ei and E j (“Ei or E j ”)
S
E1 , E2 , . . .
H, T ; x1 x2
0/
E′
Ei ∪ E j
Intersection of Ei and E j (“Ei and E j ”)
Ei ∩ E j
that
P[n1 = 1 ∩ n2 = 2]
P[n1 + n2 = 3]
P[n1 = 1 ∩ n2 = 2]
=
P[n1 n2 = 12] + P[n1n2 = 21]
ab
= 0.5.
=
ab + ba
P[n1 = 1 | n1 + n2 = 3] =
The last entries were obtained from the probability tree in
Fig. 8-4. Note that because of the a priori knowledge that
n1 + n2 = 3, the probability of n1 = 1 increased from 0.1 to 0.5.
Ei and E j are independent
P[Ei ∩ E j ] =
P[Ei ] P[E j ]
Concept Question 8-2: Why does the conditional probability formula require division by P[B]?
Ei and E j are mutually exclusive
P[Ei ∪ E j ] =
P[Ei ] + P[E j ]
Exercise 8-2: Given that the result of the coin flip experi-
ment for N = 2 was one head and one tail, compute P[H1 ].
where we used the relation (H1 H2 ) ∩ (T1 T2 ) = 0. Using the
probability tree in Fig. 8-5 gives
0.42
a2
=
= 0.31.
P[E1 |E2 ] = 2
2
2
a + (1 − a)
0.4 + (1 − 0.4)2
Example 8-4:
Tetrahedral Die Conditional
Probability
Given that the result of the tetrahedral die experiment of Example 8-2 was n1 + n2 = 3, compute the probability that n1 = 1.
Use a = 0.1, b = 0.2, c = 0.3, and d = 0.4.
Solution: From Fig. 8-4, there are two ways to obtain
n1 + n2 = 3, namely
Answer: P[H1 | H1 T2 ∪ T1 H2 ] = 0.5. (See IP ).
8-3 Random Variables
The number of heads n among N flips is a random variable. A
random variable is a number assigned to each possible outcome
of a random experiment. The range of values that n can assume
is from zero (no heads) to N (all heads). Another possible
random variable is the number of consecutive pairs of heads
among the N flips.
For the tetrahedral die experiment, our random variable might
be the number of times among N tosses that the outcome is the
number 3, or the number of times the number 3 is followed
by the number 2, or many others. In all of these cases, the
random variables have real discrete values, so we refer to them
as discrete random variables. This is in contrast to continuous
random variables in which the random variable may assume
any value over a certain continuous range. We will examine the
properties of both types of variables.
8-3.1 Probability Distributions for Continuous
Random Variables
E1 = n1 n2 = 12,
E2 = n1 n2 = 21.
We need to compute the conditional probability
P[n1 = 1 | n1 + n2 = 3].
Since satisfying both conditions requires that n2 = 2, it follows
Since a continuous random variable can take on a continuum of
values, such as the set of real number in an interval, the probability of a continuous random variable taking on a specific value
is zero. To describe the distribution of a continuous random
variable, we must use a density function, called a probability
www.dbooks.org
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
1
p(x′ ) = lim P[x′ ≤ x < x′ + δ ].
δ →0 δ
(8.24)
Thus, the probability of x lying within a narrow interval
[x′ < x < x′ + δ ) is p(x) δ . The interval probability that x lies
between values a and b is
P[a ≤ x < b] =
Z b
a
p(x′ ) dx′ .
(8.25a)
and the total probability over all possible values of x is
Z ∞
−∞
p(x′ ) dx′ = 1.
(8.25b)
20
x(y)
15
Surface height x (mm)
density function (pdf ). A pdf is not a probability; its units are
the same as those of the random variable (e.g., distance). Instead,
it describes the relative likelihood of the random variable taking
on a specific value. It is analogous to mass density, rather than
pure mass.
Let us consider the example shown in Fig. 8-6. Part (a)
of the figure displays the height profile measured by a laser
ranger as the beam was moved across 1 m of a ground surface.
Relative to the mean surface, the height varies between about
−20 mm and +20 mm, which means that the probability that
the ground surface height may exceed this range is zero. If
we count the number of times that each height occurs, we end
up with the pdf shown in Fig. 8-6(b). The measured pdf is
discrete, but if the discretization is made infinitesimally small,
the pdf becomes continuous. In the present case, the pdf is
approximately Gaussian in shape. The horizontal axis denotes
the surface height x (relative to the mean surface) and the vertical
axis denotes p(x), the probability density function. If a point
on the surface is selected randomly, the probability that it has a
height between 5 mm and 5.001 mm is 0.001 times the value of
the pdf at 5 mm, which is 0.0004.
Formally, p(x) for a continuous random variable is defined as
10
5
0
−5
−10
−15
−20
0 100 200 300 400 500 600 700 800 900 1000
Horizontal location y (mm)
(a) Height profile
0.060
0.055
Probability density p(x) (1/mm)
262
Gaussian
p(x)
0.050
0.045
0.040
0.035
0.030
0.025
0.020
0.015
0.010
0.005
0.000
−20 −15 −10
−5
0
5
10
15
20
Surface height above mean surface x (mm)
(b) pdf
Figure 8-6 (a) Measured height profile x(y) and (b) pdf of
8-3.2 Probability Distributions for Discrete
Random Variables
The term pdf is used to describe the probability function
associated with a continuous random variable. The analogous
function for a discrete random variable n is the probability mass
function (pmf ), p[n].
digitized height profile p(x).
◮ Note that we use curved brackets for a pdf p(x) and
square brackets for a pmf p[n]. ◭
8-4 EFFECTS OF SHIFTS ON PDFS AND PMFS
263
p[n]
R 1/2
Answer: P[x < 1/2] = 0 2x dx = 1/4.
0.4
0.3
Exercise 8-4: Random variable n has the pmf p[n] = ( 12 )n
0.2
for integers n ≥ 1. Compute P[n ≤ 5].
0.1
0
−1
0
1
2
3
4
5
Answer: P[n ≤ 5] = ∑5n=1 ( 12 )n = 31/32.
n
6
Figure 8-7 pmf p[n] for the tetrahedral die experiment.
8-4 Effects of Shifts on Pdfs and Pmfs
8-4.1 Continuous Random Variable
For a discrete random variable n,
p[n′ ] = P[n = n′ ],
(8.26)
where P[n = n′ ] is the probability that n has the value n′ . By
way of an example, Fig. 8-7 displays p[n] for a single toss of
the tetrahedral die, with the random variable n representing the
outcome of the toss, namely the number 1, 2, 3, or 4.
The interval probability that the value of n is between n′ = na
and n′ = nb − 1, inclusive of those limits, is
nb −1
P[na ≤ n < nb ] = ∑ p[n′ ],
(8.27a)
If a random variable x characterized by a pdf p(x) is shifted by
a constant amount τ to form a new random variable y, with
y = x − τ,
(8.28a)
then according to the rules of probability, the pdf of y is the pdf
of x shifted by τ :
(8.28b)
p(y) = p(x − τ ).
When computing mean values and other moments, the dummy
variables used in integrations should be related by
n′ =na
y′ = x′ − τ .
and the total probability over all possible values of n′ is
(8.28c)
∞
p[n′ ] = 1.
∑
′
(8.27b)
n =−∞
8-4.2 Discrete Random Variable
◮ The notation and properties of continuous and discrete
random variables are summarized in Table 8-2. ◭
If x is a continuous random variable, what is P[x = c] for any constant c?
Concept Question 8-3:
For an integer k and two discrete random variables n and m
related by
m = n − k,
(8.29a)
the pmf of m is related to the pmf of n by
p[m = m′ ] = P[m = m′ ]
= P[n − k = m′ ]
Exercise 8-3: Random variable x has the pdf
p(x) =
(
2x
0
for 0 ≤ x ≤ 1,
otherwise.
= P[n = m′ + k] = p[n = n′ ] = p[n′ ],
(8.29b)
where dummy variables n′ and m′ are related by
m′ = n′ − k.
Compute P[x < 1/2].
www.dbooks.org
(8.29c)
264
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
Table 8-2 Notation and properties of continuous and discrete random variables.
A. Continuous Random Variable
pdf of x
x
p(x)
x = E[x] =
mean value of x
mean value of x2
Z ∞
−∞
x2 = E[x2 ] =
x′ p(x′ ) dx′
Z ∞
−∞
(x′ )2 p(x′ ) dx′
p
x2 − x2
standard deviation of x
σx =
variance of x
σx2 = x2 − x2
interval probability over (a, b)
P[a ≤ x < b] =
Z b
a
p(x′ ) dx′
shift property: y = x − τ , with τ = constant
p(y) = p(x), with y = x − τ
covariance of x and y
λx,y = xy − x y
B. Discrete Random Variable
pmf of n
n
p[n]
∞
mean value of n
n=
∑ n′ p[n′]
′
n =−∞
∞
mean value of n2
n2 =
(n′ )2 p[n′ ]
∑
′
n =−∞
p
n2 − n2
standard deviation of n
σn =
variance of n
σn2 = n2 − n2
interval probability over [na ≤ n < nb ]
P[na ≤ n < nb ] =
shift property: m = n − k
p[m] = p[n], with m = n − k
covariance of n and m
λn,m = nm − n m
Example 8-5: Binomial Pmf
In the coin-flip experiment, define random variable n as the
number of heads in N flips. Compute pmf p[n], given that for
any coin flip, the probability for a head outcome is a.
Solution: Since the results of different coin flips are inde-
n′ =nb −1
∑
′
p[n′ ]
n =na
pendent, the probability of any given sequence of results is
the product of the probabilities of the individual results. The
probability for a head is a and the probability for a tail is
(1 − a), and if the number of heads is n, the number of tails
is (N − n). Since multiplication of probabilities is commutative,
the probability of any specific sequence of n heads and (N − n)
8-5 JOINT PDFS AND PMFS
265
8-5 Joint Pdfs and Pmfs
p[n]
0.35
0.30
0.25
0.20
0.15
0.10
0.05
0
0
1
2
3
4
5
6
7
8
9
n
10
This section introduces the joint pdf and joint pmf for two random variables, in continuous and discrete format. Atmospheric
temperature and pressure are each a random variable that varies
with time and the prevailing atmospheric conditions. Hence,
each has its own pdf or pmf. The study of atmospheric phenomena requires knowledge of the statistics of both variables,
including their joint pdf or pmf.
Figure 8-8 Binomial pmf p[n] for N = 10 and a = 0.6.
8-5.1 Continuous Random Variables
tails, in any order, is
n
N−n
P[n] = a (1 − a)
(single sequence).
(8.30a)
The number of such possible sequences, denoted Nn , is
N!
N
=
(number of sequences). (8.30b)
n
n! (N − n)!
The pmf p(n) is the product of the probability for an individual
sequence and the number of sequences:
N
an (1 − a)N−n N!
p(n) = P[n]
=
.
n
n! (N − n)!
(8.31)
This is known as the binomial probability density function. A
plot of p(n) is shown in Fig. 8-8 for N = 10 and a = 0.6.
The expression given by Eq. (8.31) is the pmf for a specific
value of n. If we were to add the probabilities for all possible
values of n (between 0 (no heads) and N (all heads)), among N
tosses, then the sum should be 1. Indeed, a bit of algebra leads
to the conclusion
N
N
∑ p(n) = ∑
′
n′ =0
n =0
′
′
an (1 − a)N−n N!
n′ ! (N − n′)!
N
N−1
= (1 − a) + a(1 − a)
The joint pdf of two continuous random variables x and y is
defined as
1
P[x′ ≤ x < x′ + δ , y′ ≤ y < y′ + δ ].
δ →0 δ 2
(8.32)
The definition is extendable to any number of random variables.
The interval probability over the range (ax ≤ x < bx ) and
(ay ≤ y < by ) is
p(x = x′ , y = y′ ) = lim
P[ax ≤ x < bx , ay ≤ y < by ] =
Z bx Z by
ax
ay
p(x′ , y′ ) dx′ dy′ . (8.33)
Additionally, if we extend the limits to ±∞, we have
Z ∞Z ∞
−∞ −∞
p(x′ , y′ ) dx′ dy′ = 1.
(8.34)
The marginal pdfs p(x) and p(y) are related to the joint pdf
through
p(x) =
p(y) =
Z ∞
−∞
Z ∞
−∞
p(x, y′ ) dy′
(marginal pdf for x),
(8.35a)
p(x′ , y) dx′
(marginal pdf for y).
(8.35b)
The conditional pdf for random variable x, given random
variable y, is given by
N
a2 (1 − a)N−2 N(N − 1)
+
+ · · · + aN
2
= (a + (1 − a))N = 1.
In the last step, we used the binomial expansion of
(a + (1 − a))N .
1
P[x′ ≤ x < x′ + δ , y′ ≤ y < y′ + δ ]
2
δ
p(x = x | y = y ) = lim
1
δ →0
P[y′ ≤ y < y′ + δ ]
δ
p(x, y)
=
(conditional pdf).
(8.36a)
p(y)
′
www.dbooks.org
′
266
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
Similarly,
p(x, y)
p(y|x) =
.
p(x)
(8.36b)
If variables x and y are statistically independent, then
p(x, y) = p(x) p(y),
(8.37a)
in which case
p(x|y) = p(x)
(independent variables).
(8.37b)
The Gaussian (or normal) pdf is an important density function
because it is applicable to the probabilistic behavior of many
physical variables. If random variables x and y are each characterized by a Gaussian pdf, then
1
p(x) = p
e
2πσx2
and
value x is and up to 1. The domain of p(x, y) is the shaded
triangle in Fig. 8-10.
Since the total probability is 1, it follows that
Z 1 Z 1
′ ′
′
p(x , y ) dy dx′
1=
x′
0
Z 1 Z 1
′
C dy dx′
=
=
0
x′
0
Cy′ x′
Z 1
2
2
1
p(y) = q
e−(y−y) /2σy ,
2πσy2
(8.38a)
(8.38b)
where x and y are the mean values of x and y, respectively, and
σx and σy are their associated standard deviations. Moreover, if
x and y are independent, then their joint pdf is
p(x, y) = p(x) p(y)
2
2
2
2
1
e−(x−x) /2σx e−(y−y) /2σy .
=
2πσx σy
Z 1
0
C(1 − x′ ) dx′ =
C
.
2
(8.41)
Z 1
p(x, y′ ) dy′
(
Z 1
2(1 − x)
′
2 dy =
=
0
x
x
0 ≤ x ≤ 1,
otherwise.
(8.42)
(c) Using the results of parts (a) and (b) in Eq. (8.36b), the
conditional pdf p(y|x) is

2
0 ≤ x ≤ y ≤ 1,
p(x, y) 
= 2(1 − x)
p(y|x) =

p(x)
0
otherwise.
(8.39)
(
4
p(y|x = 3/4) =
0
Z 1
3/4
Given that
3/4 ≤ y ≤ 1,
otherwise.
(8.43)
Note that both p(x) and p(y|x = 3/4) integrate to 1, as required
by Eq. (8.5):
Example 8-6: Triangle-Like Joint Pdf
for 0 ≤ x ≤ y ≤ 1,
otherwise,
dx′ =
For x = 3/4,
Part (a) of Fig. 8-9 displays a 1-D plot of the Gaussian pdf with
x = 2 and σx = 0.45, and part (b) displays a 2-D Gaussian with
x = y = 0 and σx = σy = 1.2.
(
C
p(x, y) =
0
Hence, C = 2.
(b) With C = 2, application of Eq. (8.35a) leads to
p(x) =
−(x−x)2 /2σx2
1
Z 1
0
p(x′ ) dx′ =
p(y′ | x = 3/4) dy′ =
Z 1
0
Z 1
2(1 − x′) dx′ = 1,
3/4
4 dy′ = 1.
(8.40)
compute (a) constant C, (b) the marginal pdf p(x), and (c) the
conditional pdf p(y|x) at x = 3/4.
Solution: (a) We note from the definition of p(x, y) that x can
extend between 0 and 1, but y extends only between whatever
8-5.2 Discrete Random Variables
For two discrete random variables n and m, their joint probability mass function is p[n, m]. The probability that n is in the range
between na and nb − 1, inclusive of those limits, and m is in the
8-5 JOINT PDFS AND PMFS
267
p(x)
1.0
x = 2, σx = 0.45
0.6
0.4
0.2
−3
−2
−1
0
1
2
3
4
5
6
7
x
(a) 1-D Gaussian plot with x = 2 and σx = 0.45.
y
x
(b) 2-D Gaussian pdf with x = y = 0 and σx = σy = 1.2.
Figure 8-9 Gaussian pdfs in (a) 1-D and (b) 2-D.
www.dbooks.org
268
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
y
Example 8-7:
1
Discrete Version of Example 8-6
Given that
(
C
p[n, m] =
0
x
1
Figure 8-10 The domain of joint pdf p(x, y) of Example 8-6;
for each value of variable x, variable y extends from that value
to 1.
for 0 ≤ n ≤ m ≤ 2,
otherwise,
compute (a) constant C, (b) the marginal pmf p[n], and (c) the
conditional pmf p[m|n] at n = 1.
Solution: (a) The joint pmf, depicted in Fig. 8-11, is the
discrete equivalent of the joint pdf displayed in Fig. 8-10.
m
range between ma and mb − 1, also inclusive of those limits, is
P
na ≤ n < nb
ma ≤ m < mb
nb −1 mb −1
′
′
∑ p[n , m ].
= ∑
∞
′
′
∑ ∑ p[n , m ] = 1.
1 C
C
0
In analogy with the expressions given by Eq. (8.35) for the
marginal pdfs in the continuous case, the marginal pmfs p[n]
and p[m] in the discrete case are related to the joint pmf by
C
0
1
Since the total probability is 1, it follows that
2
1= ∑
2
(8.45b)
Finally, the conditional pmf for random variable n, given
random variable m, is
p[n, m]
.
p[m]
(8.46)
Of course, if n and m are independent random variables, then
p[n, m] = p[n] p[m], in which case
p[n|m] = p[n]
(n and m independent).
2
m =0
|{z}
m =1
|{z}
|{z}
n′ =1
m =2
n′ =2
= 3C + 2C + C = 6C.
n =−∞
p[n|m] =
2
n′ =0
(8.45a)
∞
∑ p[n′, m].
′
2
p[n′ , m′ ] = ∑ C + ∑ C + ∑ C
∑
′
′
′
′
′
m′ =−∞
p[m] =
n
2
Figure 8-11 Depiction of joint pmf p[n, m] in Example 8-7.
n′ =0 m =n
∞
∑ p[n, m′],
C
(8.44b)
n′ =−∞ m′ =−∞
p[n] =
C
(8.44a)
n′ =na m′ =ma
As required by the axioms of probability, when the limits on the
double sum are extended to ±∞, the total probability adds up
to 1:
∞
2 C
(8.47)
Hence, C = 1/6.
(b) With C = 1/6, application of Eq. (8.45a) leads to

3/6
for n = 0,
2
2
1 
′
p[n] = ∑ p[n, m ] = ∑ = 2/6
for n = 1,
6 
1/6
m′ =n
m′ =n
for n = 2.
(c) Using Eq. (8.46), but with n and m interchanged, the
conditional probability p[m|n] is
p[m|n] =
p[m, n]
1/6
=
.
p[n]
p[n]
8-6 FUNCTIONS OF RANDOM VARIABLES
269
To evaluate p[m|n] at n = 1, we should note that since n ≤ m, the
range of m becomes limited to [1, 2]. Hence,
 1/6 1
for m = 1,

 2/6 = 2
1/6
1/6
1
p[m|1] =
= 2/6 = 2
for m = 2,
p[n = 1] 

0
otherwise.
We note that the total conditional probability adds up to 1:
2
1
1
∑ p[m | n = 1] = 2 + 2 = 1.
m=1
8-6 Functions of Random Variables
8-6.1 Mean Value
The mean value, or simply the mean or expectation, of a
continuous random variable x characterized by a pdf p(x) is
defined as
Z ∞
x′ p(x′ ) dx′ .
(8.48a)
x = E[x] =
−∞
Concept Question 8-4: Since P[y = c] = 0 for any continuous random variable y and constant c, how can p(x|y)
make sense?
Here, we use E[x] to denote the “expected value” of x, synonymous with the abbreviated notation x.
Similarly, for a discrete random variable n characterized by a
pmf p(n), the mean value of n is
∞
n = E[n] =
Exercise 8-5: In Example 8-6, compute the marginal pdf
p(y) and conditional marginal pdf p(x|y).
(
2y
2 dx =
p(y) =
0
0
(
2
p(x, y)
= 2y
p(x|y) =
p(y)
0
For a function f (x, y) of two continuous random variables x and
y, the expectation (mean value) of f (x, y) is computed using the
joint pdf of (x, y), namely p(x, y), as follows:
for 0 ≤ y ≤ 1,
otherwise,
E[ f (x, y)] =
for 0 ≤ x < y ≤ 1,
otherwise.
Z ∞Z ∞
p[m] and conditional marginal pmf p[n|m].
∞
f [n′ , m′ ] p[n′ , m′ ].
∑
∑
′
′
(8.49b)
Expectation is a linear operator: for any two continuous
random variables x and y, and any two constants a and b,
Answer:
for m = 0,
for m = 1,
for m = 2.
which sums to 1, as required for a pmf.
(8.49a)
n =−∞ m =−∞
Exercise 8-6: In Example 8-7, compute the marginal pmf
m+1
for m = 0, 1, 2,
p[m] =
6
p[n, m]
1/6
1
p[n|m] =
=
=
p[m]
(m + 1)/6 m + 1
f (x′ , y′ ) p(x′ , y′ ) dx′ dy′ .
∞
E[ f [n, m]] =
Hence,
−∞ −∞
Similarly, for the discrete case
As expected, this becomes an impulse if y = 0.

1/6
m
1 
p[m] = ∑ = 2/6
6 
3/6
n′ =0
(8.48b)
n =−∞
Answer:
Z y
∑ n′ p[n′].
′
E[ax + by] =
Z ∞
−∞
=a
Z ∞
−∞
+b
for m = 0, 1, 2,
(ax′ + by′ ) p(x′ , y′ ) dx′ dy′
x′ dx′
Z ∞
−∞
Z ∞
y′ dy′
−∞
p(x′ , y′ ) dy′
Z ∞
−∞
p(x′ , y′ ) dx′ .
(8.50)
In view of the relations given by Eq. (8.35) for the marginal pdfs,
Eq. (8.50) can be rewritten as
E[ax + by] = a
Z ∞
−∞
x′ dx′ p(x) + b
Z ∞
−∞
= aE[x] + bE[y] = ax + by.
www.dbooks.org
y′ dy′ p(y′ )
(8.51)
270
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
◮ The mean value of the weighted sum of two random
variables is equal to the sum of their weighted means. ◭
A similar relationship applies to two discrete random variables
n and m:
E[an + bm] = an + bm.
(8.52)
where xy = E[xy].
Unlike the mean, the variance of the linear sum of two random
variables is not a linear operator. Consider the variable x + y:
2
σ(x+y)
= E[((x + y) − E[x + y])2]
= E[(x + y)2 − 2(x + y) E[x + y] + (E[x + y])2]
= E[x2 ] + E[2xy] + E[y2]
− 2E[x] E[x + y] − 2E[y] E[x + y] + (E[x + y])2
8-6.2 Conditional Mean
The conditional expectation, also known as the conditional
mean, of random variable x, given that y = y′ uses the conditional pdf p(x|y = y′ ):
E[x | y = y′ ] =
Z ∞
The expectation of (x + y) is linear; i.e.,
−∞
x′ p(x′ |y′ ) dx′ .
(8.53a)
Note that the conditional mean is defined at a specific value of
the second random variable, namely y = y′ .
For the discrete case,
E[n | m = m′ ] =
2
= x2 + 2xy + y2 − 2x(x + y) − 2y(x + y) + (x + y) .
(8.57)
∞
∑ n′ p[n′ |m′ ].
(8.53b)
n′ =−∞
The variance of a random variable x with mean value x is the
mean value of (x − x)2 , where (x − x) is the deviation of x from
its mean x. Denoted σx2 , the variance is defined as
σx2 = E[(x − x)2 ]
2
= E[x − 2xx + x2 ]
2
σ(x+y)
= x2 + 2xy+ y2 − 2x(x+ y)− 2y(x+ y)+ (x+ y)2 , (8.59)
which simplifies to
(8.60)
In view of the definitions given by Eqs. (8.54) and (8.56),
Eq. (8.60) can be rewritten as
2
σ(x+y)
= σx2 + σy2 + 2λx,y .
(8.61)
8-6.4 Properties of the Covariance
(8.54)
2 = a2 σ 2 .
where x2 = E[x2 ]. Also, σax
x
The square root of the variance is the standard deviation:
q
σx = x2 − x2 .
(8.55)
A pdf with a small standard deviation is relatively narrow in
shape, whereas one with a large value for σx is relatively broad
(Fig. 8-12).
The covariance λx,y of two random variables x and y is
defined as the mean value of the product of the deviation of x
from x and y from y:
λx,y = E[(x − x)(y − y)]
= E[xy] − x E[y] − y E[x] + x y = xy − x y,
(8.58)
Use of Eq. (8.58) in Eq. (8.57) leads to
2
= (x2 − x2 ) + (y2 − y2 ) + 2(xy − x y).
σ(x+y)
8-6.3 Variance and Standard Deviation
= E[x2 ] − 2x E[x] + x2 = x2 − x2 ,
(x + y) = x + y.
(1) The covariance between a random variable and itself is its
variance:
λx,x = σx2 .
(8.62)
(2) The degree of correlation between two random variables is
defined by the correlation coefficient ρxy :
ρx,y =
λx,y
.
σx σy
(8.63)
If x and y are uncorrelated, then
E[xy] = E[x] E[y] = x y,
(8.64)
in which case Eq. (8.56) yields the result
(8.56)
λx,y = 0
(x and y uncorrelated),
(8.65)
8-6 FUNCTIONS OF RANDOM VARIABLES
271
p(x)
1.0
x = 0, σx2 = 0.2
0.8
x = 0, σx2 = 1.0
0.6
x = 0, σx2 = 5.0
0.4
0.2
0
−5
−4
−3
−2
−1
0
1
2
3
4
x
5
Figure 8-12 Three Gaussian distributions, all with x = 0 but different standard deviations.
and, consequently, ρx,y = 0.
(3) Two random variables x and y are uncorrelated if their
covariance is zero, and they are independent if their joint pdf
is separable into the product of their individual pdfs:
p(x, y) = p(x) p(y)
(x and y independent).
Probabilistic Term
Expectation (mean)
Variance σx2
Standard deviation σx
Covariance λx,y
(8.66)
Example 8-8:
◮ Uncorrelated random variables may or may not be
independent, but independent random variables are uncorrelated. ◭
Consider two independent random variables x and y. The mean
of their product is
xy = E[xy] =
=
=
Z ∞Z ∞
−∞ −∞
Z ∞
′ ′
−∞
Z ∞
−∞
=xy
x′ y′ p(x′ , y′ ) dx′ dy′
x y px (x′ ) py (y′ ) dx′ dy′
x′ p(x′ ) dx′
Z ∞
−∞
Triangle-Like Pdf II
Given the joint pdf introduced earlier in Example 8-6, namely
(
C
for 0 ≤ x ≤ y ≤ 1,
p(x, y) =
0
otherwise,
compute (a) x, (b) σx2 , (c) E[y | x = 3/4], and (d) σy2 | x=3/4 .
Solution: (a) From Eq. (8.42),
(
2(1 − x),
p(x) =
0
0 ≤ x ≤ 1,
otherwise.
Hence, the mean value of x is
y′ p(y′ ) dy′
(x and y independent),
Mechanical Term
Center of mass
Moment of inertia
Radius of gyration
Cross moment of inertia
(8.67)
which leads to λx,y = xy − x y = 0, and hence uncorrelation
between x and y.
For readers interested in mechanical systems, we note the
following correspondences:
x = E[x] =
Z 1
0
x′ p(x′ ) dx′ =
Z 1
0
2x′ (1 − x′) dx′ =
1
.
3
(b) To compute σx2 , we start by computing x2 :
x2 = E[x2 ] =
www.dbooks.org
Z 1
0
(x′ )2 p(x′ ) dx′ =
Z 1
0
2(x′ )2 (1 − x′ ) dx′ =
1
.
6
272
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
Hence, by Eq. (8.54),
Concept Question 8-5: When is the variance of the sum
of two random variables the sum of the variances?
2
1
1
1
2
2
2
=
.
σx = x − x = −
6
3
18
(c) With x and y interchanged, the expression for the conditional expectation given by Eq. (8.53a) is
Concept Question 8-6: Does “x and y are uncorrelated”
imply “x and y are independent” or is it the other way
around?
Z ∞
Exercise 8-7: In Example 8-6, compute (a) the mean E(y)
and (b) variance σy2 .
′
E[y | x = x ] =
−∞
y′ p(y′ |x′ ) dy′ .
Using the expression given by Eq. (8.43) and setting x′ = 3/4
leads to
Z 1
7
y′ 4 dy′ = ,
E[y | x = 3/4] =
8
3/4
Answer: (a)
p(y) =
Hence,
Z 1
3/4
(y′ )2 4 dy′ =
a
(b)
Z 1
y2 (2y) dy = 1/2.
Answer: (See IP ).
and
(x )
y(2y) dy = 2/3.
0
8-8, λx,y = 1/36.
The mean and variance of a random variable characterized by a
uniform pdf are
Z b
Z b
1
a+b
x′ p(x′ ) dx′ =
x=
x′ dx′ =
(8.69)
b−a
2
a
a
for 0 ≤ y ≤ 1
otherwise,
Exercise 8-8: Show that for the joint pdf given in Example
A pdf is uniform if it has a constant value over a specified range,
such as a to b, and zero otherwise:

 1
for a < x < b,
(8.68)
p(x) = b − a
0
otherwise.
′ 2
2y
0
1
.
Hence σy2 = 21 − ( 23 )2 = 18
37
.
48
8-6.5 Uniform Pdf
Z b
Z 1
(
0
2
37
1
7
σy | x=3/4 =
=
−
.
48
8
192
σx2 = x2 − x2 =
E[y] =
E[y2 ] =
The first term computes to
E[y2 | x = 3/4] =
2 dx =
0
which is the midpoint between 3/4 and 1.
(d) By Eq. (8.54), the variance σy2 | x=3/4 is given by
σy2 | x=3/4 = E[y2 | x = 3/4] − (E[y | x = 3/4])2 .
Z y
1
b−a
a+b
dx −
2
′
2
=
(b − a)2
.
12
(8.70)
8-7 Random Vectors
Suppose we wish to quantify the probability of an event—
specifically, the time at which students at a certain university
wake up in the morning—in terms of three random variables,
namely the age of an individual student, the temperature of the
room in which the student was sleeping, and the number of
credit hours that the student is enrolled in. Many other factors
impact the time that a student wakes up in the morning, but let
us assume we are focused on only these three random variables.
From the perspective of the prime subject of this book,
namely image processing, our multiple random variables might
be the image intensities of multiple images of a scene acquired
at different wavelengths, different times, or under different
illumination conditions. The objective might be to assign each
image pixel to one of a list of possible classes on the basis of the
intensities of the multiple images for that pixel. If we are dealing
with multi-wavelength satellite images of Earth’s surface, for
8-7 RANDOM VECTORS
273
example, the classes would be water, urban, forest, etc. And if
the multiple images are ultrasound medical images of the same
scene but acquired under different conditions or different times,
the classes would be bone, tissue, etc.
When two or more random variables are associated with an
event of interest, we form a random vector (RV). A random
vector x of length N is a column vector comprising N random
variables { x1 , x2 , . . . , xN }:
where λxi ,x j = xi x j − xi x j . Similarly, the cross-covariance matrix Kx,y between random vector x of length N and random
vector y of length M is given by the (N × M) matrix
x = [x1 , x2 , . . . , xN ]T .
λxN ,y1
(8.71)
We write x in bold to denote that it is a vector, and here the
superscript “T” denotes the transpose operation, which in this
case converts a horizontal vector into a column vector.
Kx,y = E[(x − x)(y − y)T ]

λx1 ,y1
 λx2 ,y1
= E[xyT ] − x yT = 
 ..
.
x′ = [x′1 , x′2 , . . . , x′N ]T ,
···
···
λxN ,y2
···

λx1 ,yM
λx2 ,yM 
.

λxN ,yM
(8.75)
We note that Kx,x = Kx = KTx and Ky,x = KTx,y . Additionally, if
x and y are uncorrelated,
Ky,x = [0]
◮ Throughout this chapter, we assume that all random
variables represent real quantities. ◭
The distribution of a random vector of continuous or discrete
random variables is described respectively by a joint pdf or
joint pmf of the random variables. The dummy variable vector
associated with the random vector x is
λx1 ,y2
λx2 ,y2
(x and y uncorrelated).
(8.76)
8-7.2 Random Vector Ax
Given a random vector x of length N and a constant (M × N)
matrix A, we now examine how to relate the attributes of x to
those of random vector y given by
(8.72)
y = Ax.
(8.77)
and the joint pdf of vector x is p(x).
The mean value of y is simply
◮ The notation and properties of random vectors are summarized in Table 8-3. ◭
y = E[y] = E[Ax] = Ax.
To demonstrate the validity of Eq. (8.78), we rewrite y in terms
of its individual elements:
8-7.1 Mean Vector and Covariance Matrix
N
yi = ∑ Ai, j x j ,
The mean vector x of random vector x is the vector of the mean
values of the components of x:
x = E[x] = [x1 , x2 , . . . , xN ]T .
(8.73)
In analogy with Eq. (8.56), the covariance matrix Kx of random
vector x comprises the covariances between xi and x j :
Kx = E[(x − x)(x − x)T ]

λx1 ,x1
λx2 ,x1

= E[xxT ] − x xT = 
 ..
.
λxN ,x1
j=1
1 ≤ i ≤ M,
···
···
λxN ,x2
···

λx1 ,xN
λx2 ,xN 
 , (8.74)

(8.79)
where Ai, j is the (i, j)th element of A. Taking the expectation of
yi , while recalling from Eq. (8.51) that the expectation is a linear
operator, gives
N
E[yi ] = ∑ Ai, j E[x j ],
j=1
λx1 ,x2
λx2 ,x2
(8.78)
1 ≤ i ≤ M.
(8.80)
Since this result is equally applicable to all random variables yi
of random vector y, it follows that Eq. (8.78) is true.
Using the matrix algebra property
λxN ,xN
www.dbooks.org
(Ax)T = xT AT ,
(8.81)
274
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
Table 8-3 Notation and properties of random vectors.
A. Continuous Random Vector
pdf of x
mean value of x
x = [x1 , x2 , . . . , xN ]T
p(x)
x = E[x] = [x1 , x2 , . . . , xN ]T
covariance matrix of x
cross-covariance matrix
Kx = E[(x − x)(x − x)T ] = xxT − x xT
Kx,y = E[(x − x)(y − y)T ] = x yT − x yT
B. Discrete Random Vector
pmf of n
mean value of n
covariance matrix of n
n = [n1 , n2 , . . . , nN ]T
p[n]
n = E[n] = [n1 , n2 , . . . , nN ]T
Kn = E[(n − n)(n − n)T ] = nnT − n nT
Kn,m = E[(n − n)(m − m)T ] = nmT − n mT
cross-covariance matrix
we derive the following relation for the covariance matrix of y:
Ky = E[(Ax)(Ax)T ] − E[(Ax)] E[Ax]T
= A E[xxT]AT − A E[x] E[x]T AT = AKx AT .
(8.82)
Similarly, the cross-covariance matrix between y and x is obtained by applying the basic definition for Ky,x (as given in the
second step of Eq. (8.75), after interchanging x and y:
Ky,x = E[yxT ] − y xT = E[AxxT] − Ax xT = AKx ,
(8.83)
where we used Eq. (8.74) in the last step.
Noting that (a + b)(c + d) = ac + ad + bc + bd, we can show
that the covariance matrix of the sum of two random vectors x
and y is
K(x+y) = Kx + Kx,y + Ky,x + Ky .
(8.84)
Table 8-4 provides a summary of the covariance relationships
between random vectors x and y.
If x is a random vector and
y = Ax, how is Ky related to Kx ? Is it Ky = AKx AT or
Ky = AT Kx A?
Table 8-4 Properties of two real-valued random vectors.
Random Vectors x and y
Covariance matrix Kx = E[x xT ] − x xT
Cross-covariance matrix Ky,x = E[x yT ] − x yT ,
any 2 random vectors x and y
Cross-covariance matrix Ky,x = KTx,y ,
any 2 random vectors x and y
Cross-covariance matrix Kx,x = Kx = KTx , for x = y
Cross-covariance matrix Kx,y = 0,
if x and y are uncorrelated
Covariance of sum matrix
K(x+y) = Kx + Kx,y + Ky,x + Ky
Concept Question 8-7:
Random Vectors x and y = Ax
Mean value y = Ax
Covariance matrix Ky = AKx AT
Cross-covariance matrix Ky,x = AKx
Cross-covariance matrix Kx,y = Kx AT
8-8 GAUSSIAN RANDOM VECTORS
275
Exercise
8-9:
Random
vector
x has the covariance matrix
Kx =
4 1
4 3
. If y =
x, find Ky and Kx,y .
1 3
2 1
Answer:
4
Ky =
2
4
Kx,y =
2
3 4
1 1
3 4
1 1
where “N ” stands for “Normal Distribution,” another name
for the Gaussian distribution. Next, we examine some of the
important properties of the jointly Gaussian random vectors.
T 1 4 3
115 51
=
.
3 2 1
51 23
19 13
1
.
=
9 5
3
8-8.1 If x Is Gaussian, Then y = Ax Is Gaussian
8-8 Gaussian Random Vectors
The expression given by Eq. (8.39) describes the joint pdf for a
Gaussian random vector consisting of two independent random
variables. For the general case, if x is a random vector of
length N, as defined by Eq. (8.71), and if its N random variables
are jointly Gaussian, then it is considered a Gaussian random
vector and its joint pdf is given by
p(x) =
1
(2π )N/2 (detKx )1/2
1
T
value vector x, and the covariance matrix Kx . Hence, a jointly
Gaussian random vector x often is described by the shorthand
notation:
x ∼ N (x, Kx ),
(8.86)
Per the notation given by Eq. (8.86), if x is a Gaussian random
vector, then random vector y = Ax also is Gaussian and given
by
(8.87)
y ∼ N (Ax, AKx AT ).
To demonstrate the validity of Eq. (8.87), we resort to the
use of the N-dimensional continuous-space Fourier transform
(N-D CSFT). Consider an N-dimensional image f (x), with x a
random vector comprising N random variables (corresponding
to N different pixels). The N-D CSFT of f (x) is F(µ ), where µ
is an N-dimensional spatial frequency vector
−1
e− 2 (x−x) Kx (x−x) ,
µ = [µ1 , µ2 , . . . , µN ]T ,
and
(8.85)
where (det Kx ) is the determinant of the covariance matrix Kx ,
and K−1
x is the inverse of matrix Kx .
The random variables { x1 , x2 , . . . , xN } are said to be jointly
Gaussian random variables. Often, the label gets abbreviated
to Gaussian random variables, but this can be misleading because even though each individual random variable may have
a Gaussian pdf, the combined set of random variables need not
be jointly Gaussian, nor form a Gaussian random vector. Hence,
to avoid ambiguity, we always include the term “jointly,” when
applicable, to random variables
◮ The adjective “jointly” is not attached to a Gaussian
random vector because, by definition, the random variables constituting the Gaussian random vector are jointly
Gaussian. However, two or more Gaussian random vectors
can be jointly Gaussian, in which case we refer to their
combination as jointly Gaussian vectors. ◭
The quantities defining the expression for p(x) given by
Eq. (8.85) are the length N of the random vector, the mean-
F(µ ) =
Z ∞
Z ∞
(8.88)
T
f (x) e− j2πµµ x dx,
···
−∞
−∞
| {z }
(8.89)
N integrals
with
N
µ T x = ∑ µ n xn .
(8.90)
n=1
For N = 2, F(µ ) reduces to the 2-D CSFT given by Eq. (3.16a),
with µ = [µ1 , µ2 ] = [µ , ν ].
The form of the transformation represented by Eq. (8.89) is
equally applicable for computing the N-D CSFT of the joint pdf
p(x), which is called the characteristic function∗ Φ x (µ ):
Φ x (µ ) = F { p(x) } =
Z ∞
Z ∞
T ′
···
p(x′ ) e− j2πµµ x dx′ , (8.91)
−∞
−∞
| {z }
N integrals
For a single random variable x with pdf p(x), the mean value of
∗ The term “characteristic function” is usually associated with the mathematical definition of the Fourier transform, which uses a “+” sign in the exponent
in Eq. (8.91). For consistency with the engineering definition of the Fourier
transform used throughout this book, we use a “−” sign instead.
www.dbooks.org
276
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
8-8.2 Properties of Gaussian Random Vectors
any function of x, such as g(x), is by definition
E[g(x)] =
Z ∞
−∞
g(x′ ) p(x′ ) dx′ .
(8.92)
A. Marginals of Gaussian random vectors are
Gaussian
(8.93)
Let x be a Gaussian random vector partitioned into two vectors
y and z:
y
.
(8.100)
x = [yT , zT ]T =
z
By extension, Eq. (8.91) reduces to
T
Φ x (µ ) = E[e− j2πµµ x ].
Since x is a Gaussian random vector, use of its pdf, as defined
by Eq. (8.85), in Eq. (8.91) can be shown to lead to
Φ x (µ ) = e
µ T x−2π 2 µ T Kx µ
− j2πµ
.
µe = ATµ
(8.95a)
T
µe = µ T A.
(8.95b)
Next, rewriting Eq. (8.93) for y instead of x, we have
T
T
T
In analogy to the correspondence between the two forms of
Φ x (µ ) given by Eqs. (8.93) and (8.94), the form of Eq. (8.94)
applicable to Eq. (8.96) is
T
2 T
T
2 T
(8.97)
T
= e− j2πµµ (Ax)−2π µ (AKx A )µ ,
(8.98)
which is the characteristic function for a Gaussian random vector with mean vector (Ax) and covariance matrix Ky = AKx AT .
Hence,
y ∼ N (Ax, AKx AT ).
(8.99)
This result is consistent with the form of Eq. (8.86); upon
replacing x with y, and using Eqs. (8.78) and (8.82), we have
y ∼ N (y, Ky ) ∼ N (Ax, AKx AT ).
(8.101)
it follows that y is related to x by
y = Ax = I
y
0
.
z
(8.102)
According to the result given by Eq. (8.99), if x is a Gaussian
random vector, so is y, which proves that the marginal of a
Gaussian vector is itself a Gaussian random vector.
B. Uncorrelated jointly Gaussian random vectors
are independent
As demonstrated by the result of Eq. (8.67), independent random
variables are uncorrelated:
Φ y (µ ) = E[e− j2πµµ y ] = E[e− j2πµµ (Ax) ] = E[e− j2π µe x ]. (8.96)
Φ y (µ ) = e− j2π µe x−2π µe Kx µe
A= I 0 ,
(8.94)
We note that the first term is Eq. (8.94) is the N-dimensional
generalization of entry #3 in Table 3-1 and the second term is
the generalization of entry #13 in the same table.
Now, we wish to derive the characteristic function Φ y (µ ) of
vector y = Ax. We start by introducing the spatial frequency
vector µe and its transpose:
and
Next, if we define matrix A as
always
independent
uncorrelated.
The converse, however, is not necessarily true; in general,
uncorrelated random variables may or may not be independent:
sometimes
uncorrelated
independent.
But, for Gaussian random vectors, the relationships are always
bidirectional:
uncorrelated
independent
(Gaussian).
To demonstrate the validity of this assertion, let us define a
Gaussian random vector z, partitioned into two vectors x and y,
x
T T T
,
(8.103)
z = [x y ] =
y
with x of length Nx , y of length Ny , and z of length N = Nx + Ny .
Our task is to demonstrate that if x and y are uncorrelated, then
they are independent. Uncorrelation means that Kx,y = [0] and
independence means that p(x, y) = p(x) p(y).
8-8 GAUSSIAN RANDOM VECTORS
277
The joint pdf of z has the form of Eq. (8.85) with x replaced
with z:
1
T −1
1
e− 2 (z−z) Kz (z−z) .
p(z) =
(2π )N/2 (det Kz )1/2
(8.104)
The mean vector and covariance matrix of z are given by
x
z = E[z] =
(8.105a)
y
and
Kz =
Kx
KTx,y
Kx,y
.
Ky
(8.105b)
Since x and y are uncorrelated, Kx,y = 0, in which case Kz
becomes a block diagonal matrix:
Kx 0
Kz = T
,
(8.106a)
0
Ky
detKz = (det Kx )(det Ky ),
(8.106b)
−1
Kx
0T
(8.106c)
and
K−1
z =
0
.
K−1
y
C. Conditional Gaussian random vectors
If z is a Gaussian random vector partitioned into vectors x and y
as defined by Eq. (8.103), namely
x
,
(8.109)
z=
y
then the conditional pdf p(x | y = y′ ) also is Gaussian:
p(x | y = y′ ) ∼ N (x | y = y′ , Kx|y ),
(8.110)
′
x | y = y′ = E[x | y = y′ ] = x + Kx,yK−1
y (y − y)
(8.111a)
with
and
T
Kx|y = Kx − Kx,y K−1
y Kx,y .
(8.111b)
Interchanging x and y everywhere in Eqs. (8.110) and (8.111)
provides expressions for the conditional pdf p(y | x = x′ ).
Deriving these relationships involves a rather lengthy mathematical process, which we do not include in here (see Problem
8-13).
Moreover, the exponent of Eq. (8.104) becomes
−
1 x − x T K−1
0
x−x
x
y−y
0T K−1
2 y−y
y
1
1
T −1
= − (x − x)T K−1
x (x − x) − (y − y) Ky (y − y).
2
2
(8.107)
In view of the results represented by Eqs. (8.106b) and (8.107),
the pdf of z simplifies to
Example 8-9:
Random vector x =
1
T
−1
1
T
x1
has a joint pdf
x2
p(x) ∼ N (0, Kx ),
2
.
1
(8.113)
y1 = 2x1 + 3x2,
y2 = 4x1 + 5x2.
(8.114a)
(8.114b)
Kx =
−1
(8.108)
5
2
y
Random vector y = 1 is related to x by
y2
where we used the relation N = Nx + Ny .
◮ The result given by Eq. (8.108) confirms that if x and
y are jointly Gaussian and uncorrelated, then they are
independent. ◭
(8.112)
with a covariance matrix
1
p(z) =
N
/2
N
/2
y
x
(2π )
(2π )
(det Kx )1/2 (det Ky )1/2
× e− 2 (x−x) Kx (x−x) e− 2 (y−y) Ky (y−y)
= p(x) p(y),
Gaussian Random Vectors
Also, random variable z is related to x1 and x2 by
z = x1 + 2x2 .
(8.115)
Determine: (a) σx21 and λx1 ,x2 , (b) σz2 , (c) Ky , (d) Ky,x , and
www.dbooks.org
278
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
(e) Ky|x .
(e) Interchanging x and y in Eq. (8.106b) gives
Solution: (a) The covariance matrix given by Eq. (8.113)
represents
5 2
λx1 ,x1 λx1 ,x2
=
.
(8.116)
Kx =
2 1
λx2 ,x1 λx2 ,x2
The variance σx21 of x1 is the same as the covariance λx1 ,x1 .
Hence,
σx21 = 5,
and from Eq. (8.116),
λx1 ,x2 = 2.
(b) Random variable z = x1 + 2x2 is related to x by
x
z = 1 2 1 = Bx,
x2
with
B= 1 2 .
Application of Eq. (8.77) leads to
σz2 = Kz = BKx BT =
5 2 1
2
= 17.
2 1 2
1
Alternatively, using Eq. (8.61),
σz2 = σx21 + 4σx22 + 2λx1x2 = 5 + 4 × 2 + 2 × 2 = 17.
(c) In view of the coefficients in Eq. (8.114), we can relate y
to x by
y = Ax,
with
2
A=
4
3
.
5
5
2
2 2
1 3
2 3
Ky = AKx A =
4 5
4
53 99
=
.
5
99 185
(d) From Eq. (8.83)
Ky,x = AKx =
2
4
3 5
5 2
This result makes perfect sense; if x is known then y = Ax also
is known, and hence the covariance of y given x, is zero.
Concept Question 8-8: If {x1 , x2 , . . . , xN } are all Gaussian random variables, is [x1 , x2 , . . . , xN ]T necessarily a
Gaussian random vector?
8-9 Random Processes
Figure 8-13 displays pdfs for the air temperature x[n] at discrete
times n = 1, 3, and 5. The air temperatures x[1], x[3], and
x[5]—as well as x[n] for all other values of n—are each a
random variable characterized by its own pdf, generated from a
probabilistic model that involves several atmospheric variables.
For example, the pdf of x[3] is p(x[3]) and its mean value is
x[3], and similar attributes apply to x[n] for every n. The set
encompassing all of these random variables for all times n is
called a random process, and since in the present example they
are indexed by discrete time, the process is called a discrete-time
random process. Otherwise, had the indexing been continuous
in time, we would have called it a continuous-time random
process.
The random process x[n] consists of the infinitely long random vector:
x = [. . . , x[−2], x[−1], x[0], x[1], . . . ]T ,
By Eq. (8.77),
T
T
Ky|x = Ky − Ky,x K−1
x Ky,x
53 99
16 7
1 −2 16 30
=
−
99 185
30 13 −2 5
7 13
0 0
.
=
0 0
16 7
2
.
=
30 13
1
(8.117)
and it is characterized by the joint infinite-dimensional pdf p(x).
As we will see shortly, different members of x may or may not be
correlated with each other, or their correlation may be a function
of the time separation between them.
◮ A sample function or realization of a random process
is a deterministic function that results from a specific outcome of the probabilistic experiment generating the random
process. Random processes are also known as stochastic
processes. ◭
8-9 RANDOM PROCESSES
279
p(x[3])
p(x[1])
x[3]
x[3]
p(x[5])
x[1]
x[1]
x[5]
x[5]
n
1
2
3
4
5
6
Figure 8-13 x[n] is the air temperature at discrete time n. At time n = 1, atmospheric conditions lead to a probabilities model for random
variable x[1] given by a pdf p(x[1]) and a mean value x[1]. Similar models characterize x[n] at each n. The sequence of random variables
{ . . . , x[−2], x[−1], x[0], x[1], . . . } constitutes a discrete-time random process.
8-9.1 Examples of Discrete-Time Random
Processes
Gaussian random vector. That is,
x = [x[n1 ], x[n2 ], . . . , x[nN ]]T
A. Independent and identically distributed (IID)
The joint pdf or pmf of two independent random variables
is equal to the product of their individual pdfs or pmfs:
p(x, y) = p(x) p(y). By extension, if all of the elements of vector
x are statistically independent of each other, and if in addition,
they all have the same generic pdf p(x) then x is said to be an
independent and identically distributed (IID) random process
characterized by
∞
p(x) =
p(x[n′ ]).
∏
′
(8.118)
(8.119)
is a Gaussian random vector for any N integer-valued times
{ n1 , n2 , . . . , nN }, and for any N. The joint pdf has the form given
by Eq. (8.85).
8-9.2 Functions of Random Processes
The following definitions pertain to discrete-time random processes. Analogous expressions apply to continuous-time random
processes.
n =−∞
Mean value
B. Gaussian random process
x[n] = E[x[n]].
A random process is Gaussian if each finite subset of the
infinite set of random variables { x[n] } is a jointly Gaussian set
of random variables, and therefore they can be stacked into a
A zero-mean random process is a process with x[n] = 0.
www.dbooks.org
(8.120)
280
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
Autocovariance function
Since the constant mean is known or can be estimated
(Section 9-1), it can be subtracted off from the random process,
thereby producing a zero-mean random process, in which case
Eqs. (8.124) and (8.125b) become applicable.
Kx [i, j] = λx[i],x[ j] = E[(x[i] − x[i])(x[ j] − x[ j])]
= x[i] x[ j] − x[i] x[ j].
(8.121)
Cross-covariance function
◮ For the sake of simplicity, we will henceforth assume that
all WSS random processes have zero mean. ◭
Kxy [i, j] = λx[i],y[ j] = E[(x[i] − x[i])(y[ j] − y[ j])]
= x[i] y[ j] − x[i] y[ j].
(8.122)
Random processes x[n] and y[n] are uncorrelated if
Kxy [n, m] = 0
for all n and m (uncorrelated).
Autocorrelation function
Rx [i, j] = E[x[i] x[ j]] = Kx [i, j] + x[i] x[ j].
(8.123)
For a zero-mean random process with x[i] = x[ j] = 0,
Rx [i, j] = Kx [i, j]
(zero-mean process).
(8.124)
Cross-correlation function
Rxy [i, j] = E[x[i] y[ j]] = Kxy [i, j] + x[i] y[ j].
(8.125a)
For zero-mean random processes
Rxy [i, j] = Kxy [i, j]
(zero-mean process).
(8.125b)
For a zero-mean random process, with x[i] = y[ j] = 0 for all i
and j, the combination of Eqs. (8.121) through (8.127) leads to
and
Rx [i − j] = Kx [i − j] = E[x[i] x[ j]]
(8.128a)
Rxy [i − j] = Kxy [i − j] = E[x[i] y[ j]].
(8.128b)
Changing variables to n = i − j gives, for any j,
Rx [n] = E[x[n + j] x[ j]],
Rxy [n] = E[x[n + j] y[ j]].
(8.129a)
(8.129b)
If the process is IID, x[n + j] and x[ j] are independent random
variables, except for n = 0. Hence, Rx [n] becomes
(
E[x2 [ j]]
for n = 0,
Rx [n] =
(8.130)
E[x[n + j]] E[x[ j]]
for n 6= 0.
Since the process is presumed to be zero-mean, which means
that x[i] = 0 for any i, Rx [n] simplifies to
8-9.3 Wide-Sense Stationary (WSS) Random
Process
Rx [n] = σ 2 δ [n],
(8.131)
σ 2 = x2 [ j].
(8.132)
A. Discrete time
A random process is considered wide-sense stationary (WSS),
also known as weak-sense stationary, if it has the following
three properties:
B. Continuous-time processes
(a) The mean x[n] is constant for all values of n.
(b) The autocovariance function Kx [i, j] is a function of the
difference (i − j), rather than i or j explicitly. That is,
Kx [i, j] = Kx [i − j].
(8.126)
(c) The autocorrelation function also is a function of (i − j):
Rx [i, j] = Rx [i − j].
where the variance is
(8.127)
All of the definitions and relationships introduced earlier
for discrete-time random processes generalize directly to
continuous-time random processes. For example, x(t) is a Gaussian random process if
{ x(t1 ), x(t2 ), . . . , x(tN ) }
(8.133)
are jointly Gaussian random variables for any N real-valued
times {t1 ,t2 , . . . ,tN }, and any integer N. For the discrete-time
8-9 RANDOM PROCESSES
281
WSS random process, we defined the autocorrelation and crosscorrelation functions in terms of the discrete-time difference
n = i − j. By analogy, we define τ = ti − t j for the continuoustime case and then we generalize by replacing t j with simply t,
which leads to
In general, Rx (τ ) is related to the power spectral density of the
signal, Sx ( f ), by the Fourier transform:
Sx ( f ) = F { Rx (τ ) } =
and
Rx (τ ) = Kx (τ ) = E[x(τ + t) x(t)]
(zero-mean WSS)
(8.134a)
and
Rxy (τ ) = Kxy (τ ) = E[x(τ + t) y(t)]
(zero-mean WSS).
(8.134b)
These expressions are for a zero-mean WSS random process.
Furthermore, if the process is also IID,
Rx (τ ) = σ 2 δ (τ ).
(8.135)
Z ∞
′
Rx (τ ′ ) e− j2π f τ d τ ′
−∞
Rx (τ ) = F −1 { Sx ( f ) } =
Z ∞
−∞
′
Sx ( f ′ ) e j2π f τ d f ′ .
E[x2 (t)] = Rx (0) =
Z ∞
−∞
Sx ( f ′ ) d f ′ .
(8.140)
For the special case where Sx ( f ) is zero at all frequencies except
over an infinitesimally narrow band of width B centered at
f ′ = f0 , the expression given by Eq. (8.140) reduces to
A. Continuous-time deterministic signal
For a deterministic (non-random) continuous-time signal x(t),
the signal power at time t is simply |x(t)|2 , and the total energy
of the signal is, from Eq. (2.7),
−∞
Sxy ( f ) = F { Rxy (τ ) } =
and
|x(t)|2 dt.
(8.136)
In image processing, the image intensity is real-valued, so we
will continue to treat 1-D signals and 2-D images as real-valued,
in which case |x(t)|2 = x2 (t).
B. Continuous-time random signal
P(t) = E[x (t)] =
Z ∞
x′ =−∞
Z ∞
′
−∞
Rxy (τ ) = F −1 { Sxy ( f ) } =
Rxy (τ ′ ) e− j2π f τ d τ ′
Z ∞
−∞
(8.142a)
′
Sxy ( f ′ ) e j2π f τ d f ′ . (8.142b)
C. Discrete-time random process
Using the expressions given in Eq. (8.129) for the autocorrelation and cross-correlation functions Rx [n] and Rxy [n] for zeromean WSS random processes, application of the DTFT (instead
of the Fourier transform) leads to
The power at time t of a continuous-time random process x(t)
is defined not only in terms of x2 (t) but also in terms of the
probability of that specific value of x(t). That is,
2
(8.141)
Since x(t) is real-valued, Rx (τ ) = Rx (−τ ) and Sx ( f ) = Sx (− f ),
so the bilateral power spectral density of x(t) at f0 is 2Sx ( f0 ).
Finally, for two zero-mean jointly WSS random processes
x and y, the cross-correlation function Rxy (τ ) and the crossspectral density Sxy ( f ) are related by
8-9.4 Power Spectral Density
Z ∞
(8.139b)
Setting τ = 0 leads to
E[x2 (t)] = Sx ( f0 ) B.
E=
(8.139a)
∞
Sx (Ω) =
∑ Rx [n] e− jΩn
(8.143a)
n=−∞
and
′
2
′
′
(x (t)) p(x (t)) dx (t).
Sxy (Ω) =
∑ Rxy [n] e− jΩn.
(8.143b)
n=−∞
If x(t) is a zero-mean WSS random process, we can express the
power in terms of the autocorrelation function Rx (τ ), as defined
by Eq. (8.134a), for τ = 0:
E[x2 (t)] = Rx (0).
∞
(8.137)
(8.138)
D. White random process
According to Eq. (8.135), the autocorrelation function for a
zero-mean WSS random process is given by Rx (τ ) = σ 2 δ (τ ).
www.dbooks.org
282
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
From Eq. (8.139a), the corresponding power spectral density is
Sx ( f ) =
Z ∞
−∞
Rx (τ ′ ) e
− j2π f τ ′
dτ ′ =
Z ∞
−∞
2
σ 2 δ (τ ′ ) e
=σ .
− j2π f τ ′
dτ ′
(8.144)
Exercise 8-11: A zero-mean WSS random process has the
2
Gaussian autocorrelation function R(τ ) = e−πτ . What is its
power spectral density?
Answer: Sx ( f ) = F {R(τ )}. From entry #6 in Table 2-5,
2
Hence, Sx ( f ) is constant across all frequencies f . By analogy
to white light, which consists of all colors of the spectrum, a
frequency independent power spectral density is called white
and x(t) is called a white random process. One of the properties
of such a process is that x(t1 ) and x(t2 ) are uncorrelated random
variables (and also independent if x(t) is Gaussian), which
means that the value of x(t1 ) at time t1 carries no information
about the value of x(t2 ) at time t2 no matter how small |t2 − t1 |
is, so long as t2 6= t1 .
Often, a white random process is illustrated by a plot like the
one depicted in Fig. 8-14(a). However, such a plot is incorrect
because it displays time continuity of x(t). The correct plot
should look more like the one in part (b) of the figure.
Similarly, for a discrete-time WSS white random processes
with Rx [n] = σ 2 δ [n], use of Eq. (8.131) in Eq. (8.143a) leads to
Sx (Ω) = σ 2
(8.145)
Sx ( f ) = e−π f .
8-10 LTI Filtering of Random Processes
8-10.1 Continuous-Time Random Process
Consider an LTI system with a real-valued impulse response
h(t), a continuous-time random process x(t) at its input, and an
output y(t):
h(t)
x(t)
y(t).
(8.146)
Since x(t) is a random process, it can exhibit many sample
functions (manifestations), and each sample function x(t) is
filtered by h(t) to produce a sample function y(t). Hence, y(t)
also is a random process:
for all Ω. Furthermore, x[n1 ] and x[n2 ] are uncorrelated for any
times n1 6= n2 , and also independent if x[n] is a Gaussian random
process.
y(t) = h(t) ∗ x(t) =
Z ∞
−∞
h(α ) x(t − α ) d α .
(8.147a)
Now we examine the relation between the autocorrelations of
x(t) and y(t), as well as the cross-correlation between x(t) and
y(t). We assume x(t) is a zero-mean WSS random process,
which implies that Rx (t1 ,t2 ) = Rx (t1 − t2 ). Taking the expectation E[·] of Eq. (8.147a) gives
Concept Question 8-9:
Are IID random processes wide-
sense stationary?
E[y(t)] =
Concept Question 8-10: Why can we assume that a
WSS random process has zero mean?
Exercise 8-10: A zero-mean WSS random process has
power spectral density
Sx ( f ) =
4
.
(2π f )2 + 4
What is its autocorrelation function?
Answer: R(τ ) = F −1 {Sx ( f )}. From entry #3 in Table 2-5,
R(τ ) = e−2|τ | .
Z ∞
−∞
h(α ) E[x(t − α )] d α = 0,
(8.147b)
where we assumed that E[x(t − α )] = 0, based on our earlier
assumption that x(t) is a zero-mean WSS random process.
Hence, y(t) is zero-mean.
A. Autocorrelation of output
Let us consider y(t) at times t1 and t2 ; upon replacing t with t1
and dummy variable α with α1 , and then repeating the process
at t2 , we have
y(t1 ) = h(t1 ) ∗ x(t1 ) =
and
Z ∞
−∞
h(α1 ) x(t1 − α1 ) d α1
(8.148a)
8-10 LTI FILTERING OF RANDOM PROCESSES
283
3
2
1
0
−1
−2
−3
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0.7
0.8
0.9
1
(a) Incorrect plot of white process
3
2
1
0
−1
−2
−3
0
0.1
0.2
0.3
0.4
0.5
0.6
(b) Correct plot of white process
Figure 8-14 A white process cannot be continuous in time.
y(t2 ) = h(t2 ) ∗ x(t2 ) =
Z ∞
−∞
h(α2 ) x(t2 − α2 ) d α2 .
(8.148b)
Ry (t1 ,t2 ) =
Multiplication of y(t1 ) by y(t2 ) gives
y(t1 ) y(t2 ) =
Z ∞Z ∞
−∞ −∞
Taking the expectation E[·] of both sides gives
h(α1 ) h(α2 ) x(t1 − α1 ) x(t2 − α2 ) d α1 d α2 .
(8.149)
Z Z ∞
−∞
h(α1 ) h(α2 ) Rx (t1 − t2 − α1 + α2 ) d α1 d α2 .
(8.150)
Sunce we have already shown that E[y(t)] = 0, it follows
that Ry (t1 ,t2 ) = Ry (t1 − t2 ) and y(t) is WSS. Upon defining
www.dbooks.org
284
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
τ = t1 − t2 , we have
Ry (τ ) =
Z Z ∞
−∞
filtering process is a convolution depicted as
h(α1 ) h(α2 ) Rx (τ − α1 + α2 ) d α1 d α2
= h(τ ) ∗ h(−τ ) ∗ Rx(τ ).
i=−∞
The discrete-time counterparts of the autocorrelation and crosscorrelation relations given by Eqs. (8.151) and (8.156) are
derived similarly and given by
−∞
and
h(α1 ) x(t − α1 ) d α1 .
(8.153)
Next, we multiply both sides by x(t − α2 ):
y(t) x(t − α2 ) =
Z ∞
−∞
(8.159)
∞
We begin by rewriting Eq. (8.147a) with the dummy variable α
changed to α1 :
Z ∞
y[n] = h[n] ∗ x[n] = ∑ h[i] x[n − i].
and equivalent to
(8.152)
B. Cross-correlation between input and output
y(t) =
(8.158)
h[n]
(8.151)
Taking the Fourier transform of both sides leads to the following
expression for the power spectral density of y(t):
Sy ( f ) = H( f ) H(− f ) Sx ( f ) = |H( f )|2 Sx ( f ).
y[n],
x[n]
h(α1 ) x(t − α1 ) x(t − α2 ) d α1 . (8.154)
Taking the expectation E[·] of both sides—while keeping in
mind that x(t) is a zero-mean WSS process—leads to
and
Z ∞
Ryx (t, t − α2 ) =
h(α1 ) Rx (α2 − α1 ) d α1 ,
(8.155)
−∞
Ry [n] = h[n] ∗ h[−n] ∗ Rx[n]
(8.160a)
Ryx [n] = h[n] ∗ Rx[n].
(8.160b)
These relationships presume our earlier characterization that
x[n] and y[n] are zero-mean, jointly WSS processes.
The DTFT maps convolutions in the discrete-time domain to
products in the frequency domain:
Sy (Ω) = |H(Ω)|2 Sx (Ω)
(8.161a)
Syx (Ω) = H(Ω) Sx (Ω)
(8.161b)
which states that x(t) and y(t) are jointly WSS. Noting that
Ryx (t, t − α2 ) = Ryx (t − (t − α2 )) = Ryx (α2 ),
Example 8-10:
and the integral in Eq. (8.155) is in the form of a convolution
leads to the result
Ryx (α2 ) = h(α2 ) ∗ Rx (α2 ).
(8.156)
The input x(t) to an LTI system defined by
dy
+ 7y(t) = 5x(t)
dt
The frequency domain equivalent of Eq. (8.156) is
Syx ( f ) = H( f ) Sx ( f ).
Continuous-Time Random
Process
(8.157)
8-10.2 Discrete-Time Random Processes
The relations obtained in the preceding subsection are all for
continuous-time random processes. If, instead, the zero-mean
WSS random process x[n] is passed through an LTI system with
impulse response h[n] to produce a random process y[n], the
is a zero-mean, white random process with Rx (τ ) = 3δ (τ ).
Compute the power spectral density and autocorrelation function of the output y(t).
Solution: Application of the Fourier transform to the system
equation gives
( j2π f + 7) Y( f ) = 5X( f ),
where use was made of property #5 in Table 2-4. The system’s
8-11 RANDOM FIELDS
285
frequency response is
Exercise 8-12: x(t) is a white process with Rx (τ ) = 5δ (τ )
and
5
Y( f )
=
.
H( f ) =
X( f )
j2π f + 7
y(t) =
Z t+1
x(τ ) d τ .
t−1
From Eq. (8.144), Sx ( f ) = σ 2 when x(t) is a white WSS with
Rx (τ ) = σ 2 δ (τ ). Hence, in the present case, Sx ( f ) = 3, and the
power spectral density of y(t) is, from Eq. (8.152),
Sy ( f ) = |H( f )|2 Sx ( f ) =
2
75
5
×3 = 2 2
.
j2π f + 7
4π f + 49
Using entry #3 in Table 2-5, the inverse Fourier transform of
Sy ( f ) is
75 −7|τ |
e
Ry (τ ) =
.
14
Compute the power spectral density of y(t).
Answer: Sy ( f ) = 20 sinc2 (2 f ). (See IP ).
8-11 Random Fields
The preceding sections highlighted the properties and relations
for 1-D random processes, in both continuous and discrete time.
Now, we extend those results to 2-D space, and to distinguish
between 1-D and 2-D processes, we refer to the latter as a
random field, instead of a random process, and we also change
our symbols to match the notation we used in earlier chapters to
represent 2-D images.
Example 8-11: Discrete-Time Random Process
8-11.1 Random Field Notation
(a) Continuous-space random field f (x, y) with 2-D continuous
spatial dimensions x and y.
The input x[n] to a discrete-time LTI system given by
(b) Discrete-space random field f [n, m] with 2-D discrete spatial
dimensions n and m.
y[n] + 2y[n − 1] = 3x[n] + 4x[n − 1]
is a zero-mean white random process with Rx [n] = 3δ [n]. Compute the power spectral density of y[n].
(c) Probability density and mass functions
p( f (x, y) = f ′ (x, y)) = probability density that random field
Solution: Taking the DTFT of the system equation gives
(1 + 2e
− jΩ
) Y(Ω) = (3 + 4e
− jΩ
f (x, y) at (x, y) has value f ′ (x, y)
p[ f [n, m] = f [n, m]] = probability mass that random field
) X(Ω),
where use was made of entry #2 in Table 2-7. The system’s
frequency response is then
f [n, m] at [n, m] has value f ′ [n, m]
E[ f (x, y)] =
Given that x[n] is a zero-mean white process with Rx [n] = 3δ [n],
it follows that Sx [Ω] = 3, and by Eq. (8.161a),
3 + 4e− jΩ
Sy (Ω) = |H(Ω)| Sx (Ω) =
1 + 2e− jΩ
=
(discrete space).
(d) Mean values for each (x, y) and [n, m]
Y(Ω) 3 + 4e− jΩ
.
=
H(Ω) =
X(Ω) 1 + 2e− jΩ
2
(continuous space),
′
−∞
f ′ (x, y) p( f ′ (x, y)) d f ′ ,
(8.162a)
(continuous space)
∞
E[ f [n, m]] =
2
Z ∞
∑ f ′ [n, m] p[ f ′ [n, m]],
′
(8.162b)
f =−∞
×3
(discrete space)
3[(3 + 4 cosΩ)2 + 16 sin2 Ω]
(1 + 2 cosΩ)2 + 4 sin2 Ω
.
where p( f ′ (x, y)) is the pdf of variable f ′ (x, y) and p[ f ′ [n, m]] is
the pmf of f ′ [n, m].
www.dbooks.org
286
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
(e) Autocorrelation functions
and
R f (x1 , y1 ; x2 , y2 ) = E[ f (x1 , y1 ) f (x2 , y2 )],
(continuous space)
R f (n1 , m1 ; n2 , m2 ) = E[ f [n1 , m1 ] f (n2 , m2 )].
(discrete space)
(8.163a)
(8.163b)
Extending the material on 1-D random processes (Section 8-10)
to 2-D, we have
f (x, y)
h(x, y)
R f (x1 , y1 ; x2 , y2 ) = K f (x1 , y1 ; x2 , y2 ) (continuous space),
R f [n1 , m1 ; n2 , m2 ] = K f [n1 , m1 ; n2 , m2 ] (discrete space).
The attributes of 1-D WSS random processes were discussed
earlier in Section 8-9.3. Generalizing to 2-D, a continuous-space
zero-mean random field f (x, y) is WSS if
Sg (µ , ν ) = |H(µ , ν )|2 S f (µ , ν ),
(1) E[ f [n, m]] = constant for all [n, m], and
2
The power spectral density Sf (µ , ν ) in the continuous-space
frequency domain (µ , ν ) is related to R f (x, y) by
R f (x, y) e− j2π (µ x+ν y) dx dy,
(8.164a)
and the 2-D equivalent of Eq. (8.140) is
E[ f 2 (x, y)] = R f (0, 0) =
Z ∞Z ∞
−∞ −∞
S f (µ , ν ) d µ d ν .
(continuous-space zero-mean WSS)
∞
∞
∑ ∑ e− j(Ω1n+Ω2 m) ,
n=−∞ m=−∞
(discrete-space zero-mean WSS)
Sg (Ω1 , Ω2 ) = |H(Ω1 , Ω2 )| S f (Ω1 , Ω2 ),
Sg f (µ , ν ) = H(µ , ν ) S f (µ , ν ),
Sg f (Ω1 , Ω2 ) = H(Ω1 , Ω2 ) S f (Ω1 , Ω2 ).
(8.165a)
(8.166b)
(8.167a)
(8.167b)
(8.167c)
(8.167d)
(8.167e)
(8.167f)
(8.167g)
(8.167h)
Exercise 8-13: A white random field f (x, y) with autocorrelation function R f (x, y) = 2δ (x) δ (y) is filtered by an LSI
system with PSF h(r) = 1r , where r is the radius in (x, y)
space. Compute the power spectral density of the output
random field g(x, y).
Answer:
Sg (µ , ν ) =
(8.164b)
For a discrete-space WSS random field with zero mean,
S f (Ω1 , Ω2 ) =
g[n, m] = h[n, m] ∗ ∗ f [n, m].
Rg (x, y) = h(x, y) ∗ ∗h(−x, −y) ∗ ∗R f (x, y),
Rg [n, m] = h[n, m] ∗ ∗h[−n, −m] ∗ ∗R f [n, m],
Rg f (x, y) = h(x, y) ∗ ∗R f (x, y),
Rg f [n, m] = h[n, m] ∗ ∗R f [n, m],
The corresponding relations for discrete-space random fields are
(2) R f [n, m] = E[ f [n′ + n, m′ + m] f [n′ , m′ ]].
(8.166a)
If f (x, y) and f [n, m] are zero-mean WSS random fields, then
(2) R f (x, y) = E[ f (x′ + x, y′ + y) f (x′ , y′ )].
(continuous-space zero-mean WSS)
h[n, m]
(discrete space)
(1) E[ f (x, y)] = constant for all (x, y), and
−∞ −∞
g(x, y) = h(x, y) ∗ ∗ f (x, y),
(continuous space)
f [n, m]
8-11.2 WSS Random Fields
Z ∞Z ∞
Z
8-11.3 Filtering WSS Random Fields
As in 1-D, if the mean values E[ f (x, y)] and E[ f [n, m]] are
zero, the autocorrelation functions are equal to the autocovariance functions:
S f (µ , ν ) =
Z
π
π
1
S f (Ω1 , Ω2 ) dΩ1 dΩ2 .
2
4 π −π −π
(discrete-space zero-mean WSS)
(8.165b)
E[ f 2 [n, m]] = R f (0, 0) =
(See
IP ).
2
.
µ2 + ν2
8-11 RANDOM FIELDS
287
Summary
Concepts
• A random variable is a number assigned to each random
outcome.
• A probability density function (pdf) is the probability
that a random variable x lies in an interval of length δ x.
• A Gaussian random variable is described by its mean and
variance.
• A random vector is a vector of random variables. Mean
and variance generalize to mean vector and covariance
matrix.
• A random process is a set of random variables indexed
by time.
• A wide-sense stationary (WSS) random process has
constant mean and autocorrelation Rx [i, j] = Rx [i − j].
The constant mean is subtracted off.
• A random field is a 2-D random process in 2-D space.
Mathematical Formulae
Conditional probability
P[E1 ∩ E2 ]
P[E1 |E2 ] =
P[E2 ]
Interval probability
P[a ≤ x < b] =
Z b
a
Covariance matrix
Kx = E[xxT] − xxT
Covariance matrix
KAx = AKx AT
p(x′ ) dx′
Interval probability
P[ax ≤ x < bx , ay ≤ y < by ] =
Z bx Z by
ax
ay
Conditional pdf
p(x, y)
p(x|y) =
p(y)
Expectation
Z ∞
f (x′ ) p(x′ ) dx′
E[ f (x)] = f (x) =
Variance
σx2 = E[x2 ] − x2
−∞
Covariance
λx,y = E[xy] − xy
Gaussian pdf
2
2
1
e−(x−x) /(2σx )
p(x) = p
2πσx2
Mean vector
E[x] = x = [x1 , . . . , xN ]T
Gaussian random vector
x ∼ N (x, Kx )
T
p(x′ , y′ ) dx′ dy′
−1
e−(1/2)(x−x) Kx (x−x)
p(x) =
(2π )N/2 (det Kx )1/2
Conditional Gaussian expectation
E[x|y] = E[x] + Kx,yK−1
y (y − E[y])
Autocovariance function
Kx [i, j] = E[x[i] x[ j]] − E[x[i]] E[x[ j]]
Autocorrelation function
Rx [i, j] = E[x[i] x[ j]]
Rx (t1 ,t2 ) = E[x(t1 ) x(t2 )]
Wide-sense stationary random process
Rx (t1 ,t2 ) = Rx (t1 − t2 ); x(t)] = m
Power spectral density
Sy ( f ) = F {Ry (t)} = |H( f )|2 Sx ( f )
Cross-spectral density
Syx ( f ) = F {Rx,y (t)} = H( f ) Sx ( f )
www.dbooks.org
288
Important Terms
autocovariance function
axioms of probability
conditional probability
covariance matrix
cross-covariance function
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
Provide definitions or explain the meaning of the following terms:
disjoint
iid random process
independent
mutually exclusive
pdf
PROBLEMS
Section 8-2: Conditional Probability
8.1 Three-Card Monte (a game hucksters play with chumps—
don’t be a chump!). There are three cards. Card #1 is red on
both sides. Card #2 is black on both sides. Card #3 is red on one
side and black on the other side. The cards are shuffled and one
chosen at random. The top of the chosen card is red. What is
P[bottom of the chosen card is also red]? (The chump bets even
money.)
8.2 A Tale of Two Tosses. Two coins have P[heads] as follows:
Coin A has P[heads] = 1/3. Coin B has P[heads] = 3/4. The
results of all flips are independent. We choose a coin at random
and flip it. Given that it came up heads, compute P[it was
coin A]. Hint: Draw a probability tree.
8.3 Three Coins in the Fountain (an old movie). Three coins
have P[heads]: Coin A has P[heads] = 2/3. Coin B has
P[heads] = 3/4. Coin C has P[heads] = 4/5. The results of all
flips are independent. Coin A is flipped. Then:
• If coin A lands heads, we flip coin B.
• If coin A lands tails, we flip coin C.
Let H2 denote that the second coin flipped (whatever it is) lands
heads.
(a) Compute P[H2].
(b) Compute P[Coin A landed heads, given H2]. Now the
second coin is flipped n − 1 more times (for a total of n
flips). Let H2n denote that all n flips of the second coin
land heads.
(c) Compute P[H2n].
(d) Compute P[Coin A landed heads, given H2n].
(e) What happens to your answer to (d) as n → ∞? Explain this.
Hint: Draw a probability tree.
pmf
power spectral density
probability and trees
random field
realization
sample and event spaces
sample function
vector random variable
white random process
wide-sense stationary
8.4 Bayes’s Rule. Widgets are made by two factories. Factory
A makes 6000 widgets per year, 2% of which are defective. Factory B makes 4000 widgets per year, 3% of which are defective.
A widget is chosen at random from the 10,000 widgets made in
a year.
(a) Compute P[the chosen widget is defective].
(b) Compute P[the chosen widget came from factory A, given
that it is defective].
8.5 Bayes’s Rule. Widgets are made by two factories. Factory
A makes 7000 widgets per year, 1% of which are defective. Factory B makes 3000 widgets per year, 2% of which are defective.
A widget is chosen at random from the 10,000 widgets made in
a year.
(a) Compute P[the chosen widget is defective].
(b) Compute P[the chosen widget came from factory A, given
that it is defective].
Section 8-3: Random Variables
8.6 Random variables x and y have the joint pdf
(
cx
if 1 < x < y < 2,
p(x, y) =
0
otherwise,
where c is a constant to be determined.
(a) Compute the constant c in the pdf p(x, y).
(b) Are x and y independent? Explain your answer.
(c) Compute the marginal pdf p(y).
(d) Compute the conditional pdf p(x|y) at y = 3/2.
8.7 Random variables x and y have the joint pdf
(
cxy
if 0 < y < x < 1,
p(x, y) =
0
otherwise,
where c is a constant to be determined.
PROBLEMS
289
(a) Compute the constant c in the pdf p(x, y).
(b) Are x and y independent? Explain your answer.
(c) Compute the marginal pdf p(x).
(d) Compute the conditional pdf p(y|x) at x = 1/2.
(a) Write out the formula for the joint pdf p(x, y).
(b) Show that changing variables from (x, y) to (z, w), where
1 1 1
x
z
=√
w
2 1 −1 y
8.8 Random variable x has the exponential pdf
(
for x > 0,
p(x) = λ e−λ x
0
for x < 0.
yields two decorrelated (λz,w = 0) random variables z
and w.
8.12 Random variables {x1 , x2 , x3 } are all zero-mean and
jointly Gaussian, with variances σx2i = 2 and covariances
λxi ,x j = −1 for i 6= j.
R
∞
(a) Confirm −∞
p(x) dx = 1.
(b) Compute the expectation x.
(c) Compute the variance σx2 .
(a) Write out the joint pdf p(x1 , x2 , x3 ). Use Eq. (8.85) notation.
(b) Write out the joint marginal pdf p(x1 , x3 ). Use Eq. (8.85)
notation.
Section 8-5: Joint Pdfs and Pmfs
(c) Let y = x1 + 2x2 + 3x3. Compute σy2 .
8.9 Random variables m and n have the joint pmf
(
c
if 0 ≤ n ≤ m ≤ 4,
p[m, n] =
0
otherwise,
(d) Show that the conditional pdf p(x1 |x2 = 2, x3 = 3) =
δ (x1 + 5). Hint: Compute the eigenvalues and eigenvectors
of the covariance matrix.
8.13 Prove Eq. (8.111a) and Eq. (8.111b), which are:
where c is a constant to be determined and m and n are integers.
(a) Compute the constant c in the pmf p[m, n].
(b) Are m and n independent? Explain your answer.
(c) Compute the marginal pmf p[m].
(d) Compute the conditional pmf p[n|m] at m = 2.
(e) Compute the conditional mean E[n|m] at m = 2.
2 at m = 2.
(f) Compute the conditional variance σn|m
′
E[x|y = y′ ] = E[x] + Kx,yK−1
y (y − E[y]),
T
Kx|y = Kx − Kx,yK−1
y Kx,y .
Hint: Define random vector w = x − Kx,yK−1
y y and show that w
and y are jointly Gaussian and uncorrelated, hence independent,
random vectors.
Section 8-9: LTI Filtering of Random Processes
8.10 Random variables m and n have the joint pmf
(
p[m, n] = cm
if 0 ≤ n ≤ m ≤ 4,
0
otherwise,
where c is a constant to be determined and m and n are integers.
(a) Compute the constant c in the pmf p[m, n].
(b) Compute the marginal pmf p[n].
(c) Compute the conditional pdf p[n|m] at m = 2.
Section 8-7: Random Vectors
8.11 Random variables x and y are zero-mean and jointly
Gaussian, with variances σx2 = σy2 = 1 and covariance λx,y = ρ
for some constant ρ 6= ±1.
8.14 x(t) is a zero-mean WSS random process with power
spectral density
1
.
Sx ( f ) =
9 + (2π f )2
x(t) is passed through the LTI system y(t) = dx
dt + 3x(t).
(a) Compute the power spectral density Sy ( f ).
(b) Compute the cross-spectral density Syx ( f ).
8.15 x(t) is a zero-mean WSS random process with power
spectral density Sx ( f ) = 1. x(t) is passed through the LTI system
dy
dt + 4y(t) = 3x(t).
(a) Compute the power spectral density Sy ( f ).
(b) Compute the cross-spectral density Syx ( f ).
www.dbooks.org
290
CHAPTER 8 RANDOM VARIABLES, PROCESSES, AND FIELDS
8.16 x(t) is a zero-mean WSS random process with power
spectral density Sx ( f ) = 1. x(t) is passed through an LTI system
with impulse response
(
2e−3t
for t > 0,
h(t) =
0
for t < 0.
(a) Compute the power spectral density Sy ( f ).
(b) Compute the cross-spectral density Syx ( f ).
Section 8-11: Random Fields
8.17 f (x, y) is a zero-mean WSS Gaussian random field with
power spectral density S f (µ , ν ) = 1. g(x, y) is f (x, y) filtered by
a brick-wall lowpass filter with cutoff spatial frequency 21 .
(a) Compute the pdf of g(7, 5).
(b) Compute the joint pdf of {g(7, 5), g(2, 4)}.
8.18 f (x, y) is a zero-mean WSS random field with power
spectral density S f (µ , ν ) = 1. g(x, y) is f (x, y) blurred by a
2
2
Gaussian PSF, so g(x, y) = f (x, y) ∗ ∗ e−π (x +y ) .
2
(a) Compute the variance σg(3,5)
.
(b) Compute the covariance λg(3,5),g(2,4).
8.19 A fractal random field f (x, y) has a power spectral
density of form S f (ρ , φ ) = c/ρ d for two constants c and d.
Show that the Laplacian operator g(x, y) = ∇2 f (x, y) defined
in Eq. (5.5) “whitens” f (x, y) (produces a white random field
output g(x, y)) if d = 4.
8.20 f [n, m] is a zero-mean WSS white random field
with power spectral density S f (Ω1 , Ω2 ) = 1. g[n, m] is
f [n, m] filtered with a Laplacian, so g[n, m] = ∇2 f [n, m].
∇2 f [n, m] is the discrete-space Laplacian (Eq. (5.9)). g[n, m] =
f [n, m] ∗ ∗ hLaplace [n, m]. Eq. (5.10):


0 1 0
hLaplace [n, m] = 1 −4 1 .
0 1 0
Compute the power spectral density of g[n, m].
8.21 f [n, m] is a zero-mean WSS Gaussian random field with
autocorrelation function R f [m, n] = 26−|m|−|n| .
(a) Compute the pdf of f [5, 4]
(b) Compute the joint pdf of { f [5, 4], f [3, 1]}.
Chapter
9
Stochastic Denoising
and Deconvolution
9
Contents
9-1
9-2
9-3
9-4
9-5
9-6
9-7
9-8
9-9
9-10
9-11
Overview, 292
Estimation Methods, 292
Coin-Flip Experiment, 298
1-D Estimation Examples, 300
Least-Squares Estimation, 303
Deterministic versus Stochastic Wiener
Filtering, 307
2-D Estimation, 309
Spectral Estimation, 313
1-D Fractals, 314
2-D Fractals, 320
Markov Random Fields, 322
Application of MRF to Image Segmentation, 327
Problems, 331
(a) Fractal tree image
(d) Noisy blurred image
log10(Sf (0,Ω2))
9
8
7
6
5
4
3
2
1
−2
−1.5
−1
−0.5
0
0.5
log(Ω2)
(b) Plot of log10(Sf (0,Ω2)) versus
log10(Ω2) for π/240 ≤ Ω2 ≤ π.
The average slope is −2.
(e) Reconstructed image using
a stochastic Wiener filter
(c) Blurred image
(f ) Reconstructed image using
a deterministic Wiener filter
Objectives
Learn to:
■ Compute MLE, MAP, and LS estimates for small
analytic problems.
■ Estimate parameters of a fractal power spectral
density from a signal or image.
■ Denoise and deconvolve fractal signals and images
using stochastic filters.
■ Use thresholding and shrinkage of its wavelet
transform to denoise an image.
■ Use the ICM algorithm to segment an image.
www.dbooks.org
This chapter provides a quick review of estimation
theory, including MLE, MAP, LS and LLSE
estimators. It then derives stochastic versions of the
deterministic denoising and deconvolution filters
presented in earlier chapters. Incorporating a priori
information, in the form of power spectral density,
is shown to greatly improve the performance of
denoising and deconvolution filters on 1-D and 2-D
problems. A very quick presentation of Markov
random fields and the ICM algorithm for image
segmentation, with examples, are also provided.
Overview
In a deterministic (non-random) inverse problem, the goal is to
compute an unknown 1-D signal x(t) or x[n], or a 2-D image
f (x, y) or f [n, m], from observation of a corresponding signal
y(t) or y[n], or corresponding image g(x, y) or g[n, m], wherein
the unknown quantity and its observed counterpart are linked
by a known model. The inversion process may also take into
consideration side information about the unknown quantity,
such as non-negativity, or a priori information, in the form of
a pdf or pmf for it. The solution of an inverse problem may be
difficult, but a well-posed inverse problem always has a unique
solution.
An estimation problem is a stochastic (random) version of
the inverse problem. The observation is a 1-D random variable
or random process, or a 2-D random field. The random character
of the observation may be associated with the inherent nature
of the observing system itself—an example of which is a laser
imager and certain radar systems—or due to additive noise
introduced in the system’s receiver. An image generated by a
monochromatic laser imager or synthetic-aperture radar usually
exhibits a speckle-like randomness superimposed on the true
image intensity that would have been measured by a widebandwidth sensor.
Based on knowledge of the sources and mechanisms responsible for the randomness associated with the observed signal
or image, we can incorporate randomness (stochasticity) in the
model that relates the unknown quantity to the observation by
modeling the unknown quantity as a random variable with a
characteristic pdf or pmf. This pdf or pmf represents a priori information about the unknown quantities. Because of the
stochastic nature of the inverse problem, we call it an estimation
problem. The formulation of an estimation problem usually
leads to a likelihood function, and the goal of the estimation
problem becomes to maximize the likelihood function; the value
of the unknown signal or image that maximizes the likelihood
function is deemed the solution of the estimation problem.
In Section 9-1, we introduce three common approaches to
estimation:
All three estimation methods are applied in Section 9-2 to the
coin-flip experiment presented earlier in Section 8-1, obtaining
three different estimators of P[heads]. This is then followed in
Sections 9-3 to 9-5 with applications of the estimation methods
to four 1-D estimation problems: (1) estimating the mean of
a wide-sense stationary (WSS) random process, in the context
of polling, (2) denoising a signal containing additive noise, (3)
denoising a signal known to be sparse, and (4) deconvolving
a signal from its noisy convolution with a known impulse
response.
The treatment is extended to 2-D in Section 9-6, and in
Section 9-7 we review the periodogram method for estimating
power spectral densities of random processes and random fields.
The periodogram can be used to obtain parametric forms of
power spectral densities for classes of signals and images, such
as fractals, which can then be used in the formulations developed
in earlier sections. The procedures are outlined in Sections 9-8
for signals and 9-9 for images. The last two sections of the
chapter are focused on an introduction to Markov random fields
(MRF) and their application to image segmentation. An MRF
incorporates a priori information (prior knowledge) about the
relationships between the value of a given pixel and those of its
neighbors.
9-1 Estimation Methods
This section introduces three estimation methods: MLE, MAP,
and LSE, and even though our ultimate goal is to apply these
methods to 2-D images (which we do in later sections), we
will, for the present, limit the presentation to estimating a
scalar unknown x from a scalar observation yobs of a random
variable y. In later sections we generalize the formulations to
random vectors, processes, and fields.
An estimation problem consists of the following ingredients:
(a) x: the unknown quantity to be estimated, which may have a
constant value or it may be a random variable with a known
pdf p(x).
(b) yobs : the observed value of random variable y.
(a) Maximum Likelihood Estimation (MLE),
(b) Maximum A Posteriori Probability (MAP) Estimation,
(c) p(y | x = x′ ): the conditional pdf of y, given that x = x′ ,
provided by a model.
(c) Least-Squares Estimation (LSE).
In a typical situation, y and x are related by a model of the form
In all three methods, the observation is presumed to be random
in nature, and the unknown quantity also is presumed to be
random but only in MAP and LSE; the unknown quantity is
presumed to be non-random in MLE.
y(t) = h(t) ∗ x(t) + υ (t)
(continuous time),
y[n] = h[n] ∗ x[n] + υ [n]
(discrete time),
or
292
9-1 ESTIMATION METHODS
293
where h(t) and h[n] are the continuous-time and discrete-time
impulse responses of the observing system and υ (t) and υ [n]
represent random noise added by the measurement process. To
obtain a good estimate of x(t) (or x[n]), we need to filter out the
noise and to deconvolve y(t) (or y[n]).
We now introduce the basic structure of each of the three
estimation methods.
9-1.1 Maximum Likelihood Estimation (MLE)
Among the three estimation methods, the maximum likelihood
estimation (MLE) method is the one usually used when the
unknown quantity x has a constant value, as opposed to being
a random variable. The basic idea behind MLE is to choose
the value of x that makes what actually happened, namely the
observation y = yobs , the most likely outcome. MLE maximizes
the likelihood of y = yobs (hence the name MLE) by applying
the following recipe:
(1) Set y = yobs in the conditional pdf p(y|x) provided by the
model to obtain p(yobs |x).
(2) Choose the value of x that maximizes the likelihood function p(yobs |x) and denote it x̂MLE.
(3) If side information about x is available, such as x is nonnegative or x is bounded within a specified interval, then
incorporate that information in the maximization process.
In practice it is often easier to maximize the natural logarithm,
ln(p(yobs |x)), rather than to maximize p(yobs |x) itself. Henceforth, ln(p(yobs |x)) will be referred to as the log-likelihood
function.
◮
p(yobs |x) = likelihood function
p(x) = a priori pdf
p(x|yobs ) = a posteriori pdf
ln(p(yobs |x) = log-likelihood function
information about the unknown (input) random variable x in the
form of its pdf p(x). For example, x may be known to have a
Gaussian pdf (Fig. 9-1(a)
2
2
e−[(x−x) /(2σx )]
p
p(x) =
2πσx2
(Gaussian),
(9.1)
where x = E[x] is the mean value of x and σx2 is its variance. Or x
might be known to be a value of a sparse signal with Laplacian
pdf (Fig. 9-1(b))
√
e− 2 |x|/σx
p(x) = √
2 σx
(Laplacian).
(9.2)
Gaussian and Laplacian a priori pdfs will be used later in
Sections 9-3 and 9-4.
The idea behind MAP estimation is to determine the most
probable value of x, given y = yobs , which requires maximizing
the a posteriori pdf p(x|yobs ). This is in contrast to the MLE
method introduced earlier, which sought to maximize the likelihood function p(yobs |x). As we see shortly, in order to maximize
p(x|yobs ), we need to know not only p(yobs |x), but also the
a priori pdf p(x).
As noted earlier, usually we know or have expressions for
p(x) and the likelihood function pobs (y|x), but not for the a
posteriori pdf p(x|y). To obtain an expression for the latter (so
we may maximize it), we use the conditional pdf relation given
by Eq. (8.36b) to relate the joint pdf p(x, y) to each of the two
conditional pdfs:
p(x, y) = p(y|x) p(x)
(9.3a)
p(x, y) = p(x|y) p(y).
(9.3b)
and
Combining the two relations gives Bayes’s rule for pdfs:
p(x|y) = p(y|x)
9-1.2 Maximum A Posteriori (MAP) Estimation
If the unknown quantity x is a random variable, the two methods most commonly used to estimate x, given an observation
y = yobs , are the maximum a posteriori (MAP) estimator and
the least-squares estimator (LSE). This subsection covers MAP
and the next one covers LSE.
The observation y = yobs is called a posteriori information
because it is about the outcome. This is in contrast to a priori
p(x)
.
p(y)
(9.4)
Bayes’s rule relates the a posteriori pdf p(x|yobs ) to the likelihood function p(yobs |x).
The goal of the MAP estimator is to choose the value of x
that maximizes the posteriori pdf p(x|yobs ), given the a priori
pdf p(x) and the likelihood function p(yobs |x). The third pdf in
Eq. (9.4), namely p(y), has no influence on the maximization
process because it is not a function of x. Hence, p(y) can be set
equal to any arbitrary constant value C.
www.dbooks.org
294
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
p(x)
0.4
Gaussian pdf
x=0
σx = 1
0.3
0.2
x=0
σx = 2
0.1
0
−6
x
−4
−2
0
2
4
6
4
6
(a) Gaussian pdf
p(x)
0.8
Laplacian pdf
0.6
σx = 1
0.4
0.2
σx = 2
0
−6
−4
−2
0
2
x
(b) Laplacian pdf
Figure 9-1 Zero-mean Gaussian and Laplacian pdfs.
The MAP estimator recipe consists of the following steps:
given by Eq. (9.4) and set p(y) = C:
p(x|yobs ) = p(yobs |x)
(1) Set y = yobs in the expression for the a posteriori pdf p(x|y)
p(x)
.
C
(9.5)
9-1 ESTIMATION METHODS
295
(2) Choose the value of x that maximizes the a posteriori pdf
p(x|yobs ) and denote it x̂MAP .
(3) If side information about x is available, incorporate that
information in the estimation process.
As noted with the MLE method in the preceding subsection,
it is often easier to maximize the natural logarithm of the
a posteriori pdf:
ln(p(x|yobs )) = ln p(yobs |x) + ln p(x) − lnC,
(9.6)
which is the sum of the logarithms of the likelihood function
p(yobs |x) and the a priori pdf p(x). Subtracting lnC does not
affect the process of maximizing the value of x.
apparent from the integration that both x′ and y′ are treated
as random variables, so we need to keep that in mind in what
follows.
Since expectation is a linear operator,
E[(x − x̂(y))2 ] = E[(x − E[x|y] + ε (y))2]
= E[(x − E[x|y])2] + E[ε 2 (y)]
+ 2E[xε (y)] − 2E[E[x|y]ε (y)].
(9.11)
We now show that the final two terms cancel each other. Using
Eq. (8.36a), namely
p(x′ , y′ ) = p(x′ |y′ ) p(y′ ),
(9.12)
the third term in Eq. (9.11) becomes
9-1.3 Least-Squares Estimator (LSE)
Whereas the MAP estimator sought to estimate x by maximizing
the a posteriori pdf p(x|yobs ) for a given observed value yobs ,
the least-squares estimator (LSE) estimates the most probable
value of x by minimizing the mean square error (MSE) between
the estimated value x̂ and the random variable x. The MSE is
defined as
MSE = E[(x − x̂(y))2 ] =
ZZ
2E[xε (y)] = 2
=2
=2
=2
(x′ − x̂(y′ ))2 p(x′ , y′ ) dx′ dy′ . (9.7)
Of course, x̂ is a function of y, and as we will show shortly,
the value of x estimated by the LSE method, for a given value
y = yobs , is given by
x̂LS = E[x | y = yobs ] =
Z
x′ p(x′ |yobs ) dx′ ,
(9.8)
where p(x|yobs ) is the a posteriori pdf given by Eq. (9.4) with
y = yobs .
We now prove that x̂LS is indeed the value of x that minimizes
the MSE by introducing the perturbation ε (y) as the deviation
of x̂(y) from E[x|y]:
x̂(y) = E[x|y] − ε (y).
(9.9)
For y = yobs , E[x|yobs ] is the LS estimator x̂LS given by Eq. (9.8),
so our task is to demonstrate that the MSE given by Eq. (9.7) is
at its minimum when ε (y) = 0. To that end, let us replace x̂(y)
in the MSE expression given by Eq. (9.7) with the perturbed
definition given by Eq. (9.9):
MSE = E[(x − x̂(y))2 ] = E[(x − E[x|y] + ε (y))2].
ZZ
ZZ
Z
Z
x′ ε (y′ ) p(x′ , y′ ) dx′ dy′
x′ ε (y′ ) p(x′ |y′ ) p(y′ )dx′ dy′
′
′
p(y ) ε (y )
Z
x′ p(x′ |y′ ) dx′ dy′
p(y′ ) ε (y′ ) E[x | y = y′ ] dy′
= 2E[ε (y) E[x|y]].
(9.13)
This result shows that the third and fourth terms in Eq. (9.11)
are identical in magnitude but opposite in sign, thereby canceling one another out. Hence, the MSE is
MSE = E[(x − x̂(y))2 ] = E[(x − E[x|y])2 ] + E[ε 2(y)]. (9.14)
The MSE is minimum when the term E[ε 2 (y)] = 0, which,
in turn, requires that ε (y) = 0. This result proves that the
conditional mean given by Eq. (9.8) is indeed the solution that
minimizes the MSE.
Readers familiar with mechanics will recognize the result
given by Eq. (9.14) as the parallel-axis theorem: The moment
of inertia of a planar object is minimized when the axis of
rotation goes through the center of mass of the object. Here,
MSE represents the moment of inertia and x̂(y) represents the
center of mass.
Now that we have shown that x̂LS is given by Eq. (9.8), let us
examine how to compute it. Using Bayes’s rule from Eq. (9.4),
(9.10)
From the definition on the right-hand side of Eq. (9.7), it is
www.dbooks.org
296
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
9-1.4 Discrete-Value Estimators
Eq. (9.8) becomes
x̂LS =
=
Z
x′ p(yobs |x′ )
1
p(y = yobs )
Z
p(x′ )
dx′
p(y = yobs )
x′ p(yobs |x′ ) p(x′ ) dx′ .
(9.15)
The quantity p(y = yobs ) can be determined from
p(y = yobs ) =
Z
p(yobs |x′ ) p(x′ ) dx′ ,
(9.16)
p(yobs |x′ ) p(x′ ) dx′
Also available from the model relating m to n is:
(9.17)
As noted at the outset of Section 9-1, the pdfs p(x) and p(y|x)
are made available by the model describing the random variable
x and its relationship to random variable y. Hence, computing
x̂LS using Eq. (9.17) should be a straightforward task.
Generalizing to the case where x and y are random vectors
and jointly Gaussian, the LS vector-equivalent of Eq. (9.8) is
x̂LS = E[x | y = yobs ] = x + Kx,y K−1
y (yobs − y),
(9.18)
(jointly Gaussian vectors)
where we used Eq. (8.111a). Here, x and y are the mean values
of vectors x and y, and Ky and Kx,y are the autocorrelation and
cross-correlation matrices defined in Section 8-7.1. We note that
x̂LS is linearly related to the observation vector yobs , which will
prove useful later.
We should note that the MAP and LS methods are sometimes
called Bayesian estimators because they take advantage of
a priori information about x in the form of p(x), whereas the
MLE method does not use p(x). The possible drawback of the
Bayesian methods is that if p(x) is incorrect, their estimates may
prove to be inferior to that of the MLE estimate, but if p(x)
is correct and applicable, the Bayesian methods are likely to
produce more accurate estimates of x than MLE.
Exercise 9-1: How does Eq. (9.18) simplify when random
vectors x and y are replaced with random variables x and y?
Answer:
x̂LS = x +
λx,y
(yobs − y).
σy2
(a) m: the unknown quantity to be estimated, which may have a
constant value or it may be a random variable with a known
pmf p[m].
(b) n: the observed value of a discrete random variable n.
leading to the final expression
R ′
x p(yobs |x′ ) p(x′ ) dx′
.
x̂LS = R
The formulations associated with the three estimation methods
covered in the preceding subsections are specific to continuousvalue quantities, wherein the symbol x denotes the unknown
quantity we wish to estimate and y denotes the observation yobs .
Now we consider discrete-value variables m and n, with:
(c) p[n|m]: the conditional pmf.
Occasionally, we may encounter scenarios in which the observation n is a discrete random variable, but the unknown is a
continuous-value quantity. In such cases, we continue to use n
as the discrete-value observation, but we use x and p(x) instead
of m and p[m] for the continuous-value random variable.
Transforming from (x, y) notation to [m, n] notation and from
integrals to sums leads to the formulations that follow.
A. MLE
m̂MLE = the MLE-estimated value of m, determined by maximizing the likelihood function (or its logarithm):
p[nobs|m]
(both n and m are discrete).
(9.19a)
To maintain notational consistency, if the unknown quantity is
continuous, m should be replaced with x:
p[nobs |x]
(x continuous and n discrete).
(9.19b)
B. MAP
m̂MAP = the MAP-estimated value of m, determined by maximizing the a posteriori pdf
p[nobs |m] p[m]
.
p[nobs ]
(both n and m are discrete)
p[m|nobs ] =
(9.20a)
As noted in connection with Eq. (9.5), p[nobs ] exercises no effect
on the maximization process, so it can be set to any arbitrary
9-1 ESTIMATION METHODS
297
constant value C. Also, if the unknown quantity is continuous,
m and p[m] should be replaced with x and p(x):
p(x|nobs ) =
p[nobs |x] p(x)
p[nobs ]
(x continuous and n discrete).
(9.20b)
C. LSE
∞
m̂LS =
∑ m′ p[nobs|m′ ] p[m′ ]
′
m =−∞
∞
∑
m′ =−∞
.
(9.21a)
p[nobs |m′ ] p[m′ ]
(both m and n are discrete)
This expression, which is the discrete counterpart of the expression given by Eq. (9.17), is applicable only if both m and n are
discrete random variables. If the unknown quantity is a continuous random variable and n is a discrete random variable, then m
and p[m] should be replaced with x and p(x), respectively, and
the summations in Eq. (9.21a) should be converted to integrals,
namely
R
x′ p[nobs |x′ ] p(x′ ) dx′
.
x̂LS = R
p[nobs |x′ ] p(x′ ) dx′
(x continuous and and n discrete)
(9.21b)
Solution: (a) For N = 1, we are given a single observation,
y1 of y. Since p(y|x) = 0 for y > x, we require x̂MLE > y1 . The
height of the pdf for 0 ≤ y ≤ x is 1/x. This height is maximized
by choosing x̂MLE to be as small as possible. The smallest
possible value of x̂MLE for which p(y1 |x) 6= 0 is x̂MLE = y1 .
(b) Repeating this argument for each of N independent
observations {y1 , . . . , yN }, we require the smallest value of x̂MLE
such that x̂MLE > { y1 , . . . , yN }. This is x̂MLE = max{ y1 , . . . , yN }.
The likelihood function is nonzero in the N-dimensional
hypercube 0 ≤ yn ≤ x:
(
1/xN
for 0 ≤ yn ≤ x,
p(y1 , . . . , yN |x) =
0
otherwise,
which is maximized by minimizing xN subject to 0 ≤ yn ≤ x.
Hence x̂MLE = max{ y1 , . . . , yN }.
Concept Question 9-1: What is the essential difference
between MLE and MAP estimation?
Exercise 9-2: Determine x̂LS and x̂MAP , given that only the
a priori pdf p(x) is known.
Answer: x̂LS = x, and x̂MLE = value of x at which p(x) is
the largest.
Example 9-1: MLE
We are given N independent observations of a random variable
y with uniform pdf:
(
1/x
for 0 ≤ y ≤ x,
p(y|x) =
0
otherwise.
p( y|x)
1
x
x
y
Compute x̂MLE for (a) N = 1 and (b) arbitrary N.
Exercise 9-3: We observe random variable y whose pdf
is p(y|λ ) = λ e−λ y for y > 0. Compute the maximum
likelihood estimate of λ .
Answer: The log-likelihood function is
ln p(yobs |λ ) = ln λ − λ yobs.
Setting its derivative to zero gives λ1 − yobs = 0. Solving
gives
1
λ̂MLE (yobs ) =
.
yobs
Exercise 9-4: We observe random variable y whose pdf
is p(y|λ ) = λ e−λ y for y > 0 and λ has the a priori
pdf p(λ ) = 3λ 2 for 0 ≤ λ ≤ 1. Compute the maximum a
posteriori estimate of λ .
www.dbooks.org
298
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
9-2.1 MLE Coin Estimate
Answer: The a posteriori pdf is
p(λ |yobs ) =
p(yobs |λ ) p(λ )
.
p(yobs )
Its logarithm is ln λ − λ yobs + 2 ln λ + ln(3) − ln(p(yobs )).
Setting its derivative to zero gives λ3 − yobs = 0. Solving
gives
3
λ̂MAP (yobs ) =
yobs
if 3/yobs < 1 since λ < 1. Note that λMAP (yobs ) =
3λMLE(yobs ) as λ is likely to be large.
For the coin-flip experiment, the MLE likelihood function
given by Eq. (9.19b) is the conditional binomial pmf given by
Eq. (8.31), with the notation changed from p(n) to p(nobs |x)
and a replaced with x. The pmf given by Eq. (8.31) presumes
that a is a known quantity, whereas in the present case a is the
unknown value of the quantity x. The notation conversion gives
p[nobs |x] =
P[Hk ] = a,
P[Tk ] = 1 − a,
and 0 ≤ a ≤ 1. For a fair coin, a = 0.5, but we wish to consider
the more general case where the coin can be such that a can
assume any desired value between 0 (all tails) and 1 (all heads).
We now use the coin-flip experiment as a vehicle to test the
three estimation methods of the preceding section. In the context of the coin-flip experiment, our unknown and observation
quantities are:
(9.22)
Here, n is a discrete-value random variable, but x is continuous.
The log-likelihood function is
ln(p[nobs |x]) = nobs ln x + (N − nobs) ln(1 − x)
+ ln N! − lnnobs ! − ln(N − nobs)!.
9-2 Coin-Flip Experiment
In Section 8-1.4, we described a coin-flip experiment in which a
coin is flipped N times. The result of the kth flip, with 1 ≤ k ≤ N,
is either a head event and designated by Hk , or a tail event and
designated by Tk . The result of any flip has no effect on the result
of any other flip, but the coin can be biased towards heads or
tails. For any given coin, the probability of heads for any flip is
an unknown number a:
xnobs (1 − x)N−nobs N!
.
nobs ! (N − nobs)!
(9.23)
Only the first two terms are functions of x. The log-likelihood
function is maximized by setting its partial derivative with
respect to x to zero:
nobs N − nobs
∂
∂
(ln x)+ (N − nobs) (ln(1 − x)) =
−
,
∂x
∂x
x
1−x
(9.24)
where we used the relations
0 = nobs
∂
1
(ln x) =
∂x
x
and
1
∂
(ln(1 − x)) = −
.
∂x
1−x
Solving Eq. (9.24) for x gives
x̂MLE = x =
nobs
,
N
(9.25)
(1) x = unknown probability of heads, which we wish to
estimate on the basis of a finite number of observations, N.
If we were to flip the coin an infinite number of times, the
estimated value of x should be a, the true probability of
heads, but in our experiment N is finite. For the sake of the
present exercise, we set N = 10.
and for the coin-flip experiment with N = 10, x̂MLE = nobs /10.
This result not only satisfies the condition 0 ≤ x ≤ 1, but it is
also intuitively obvious. It says that if we flip the coin 10 times
and 6 of those flips turn out to be heads, then the most-likelihood
estimate of the probability that any individual flip is a head is
0.6. As we will see shortly, the MAP and LS estimators provide
slightly different estimates.
(2) nobs = number of heads observed in N = 10 flips.
9-2.2 MAP Coin Estimate
As noted earlier, the MLE method does not use an a priori pdf
for x, but the MAP and LS methods do. Consequently, the three
estimation methods result in three different estimates x̂(nobs ).
Whereas the MLE method does not use a priori probabilistic
information about the unknown quantity x, the MAP method
uses the pmf p[m] if m is a discrete-value random variable or
9-2 COIN-FLIP EXPERIMENT
299
the pdf p(x) if x is a continuous-value random variable. For the
coin-flip experiment, let us assume that x is continuous, with
0 ≤ x ≤ 1, and that we know the coin is biased such that values
of x closer to 1 are more likely. Specifically, we are given the
a priori pdf
(
2x
for 0 ≤ x ≤ 1,
p(x) =
(9.26)
0
otherwise,
which states that the higher the value of x is, the greater is its pdf.
This is in contrast to the uniform pdf p(x) = 1 for 0 ≤ x ≤ 1,
which allocates the same probability density to all values of x
within the specified range. We note that the expression given by
Eq. (9.26) satisfies the condition that
Z 1
x̂
1.0
N = 10
0.8
x̂LS
0.6
x̂MAP
0.4
x̂MLE
0.2
p(x) dx = 1.
0
The MAP estimator obtains x̂MAP by maximizing the a
posteriori pdf given by Eq. (9.20b):
1
p[nobs |x] p(x)
C
2x xnobs (1 − x)N−nobs N!
=
.
C nobs ! (N − nobs)!
p(x|nobs ) =
(9.27)
(9.28)
where used used Eqs. (9.22) and (9.26), and p[n] has been set
to a constant value because it has no effect on the maximization
process. The natural logarithm of the pdf is
2N!
ln(p(x|nobs )) = ln
Cnobs ! (N − nobs)!
+ ln((x)nobs +1 ) + (N − nobs) ln(1 − x)
2N!
= ln
Cnobs ! (N − nobs)!
+ (nobs + 1) ln x + (N − nobs) ln(1 − x). (9.29)
Only the last two terms are functions of x, so when we take the
partial derivative with respect to x and set it equal to zero, we
get
nobs + 1 N − nobs
−
,
0=
x
1−x
whose solution for x yields the MAP estimate
x̂MAP = x =
nobs + 1
.
N +1
(9.30)
For the specific case where N = 10, x̂MAP = (nobs + 1)/11,
0
1
2
3
4
5
6
7
8
9
10
no
Figure 9-2 Variation of x̂MLE , x̂MAP , and x̂LS with nobs , the
number of coin flips that were observed to be heads, for N = 10
coin flips and p(x′ ) = 2x′ for 0 ≤ x′ ≤ 1.
which is different from the MLE estimate x̂MLE = nobs /10.
Figure 9-2 displays a plot of x̂MLE , x̂MAP , and x̂LS as a function
of nobs , all for N = 10. We observe that even when none of
the 10 coin flips is a head (i.e., nobs = 0), the MAP estimator
predicts x̂MAP = 1/11 ≈ 0.09, but as nobs approaches N = 10,
both estimators approach the same limit of 1.
9-2.3 LS Coin Estimate
Since x is a continuous random variable and n a discrete random
variable, the applicable expression for x̂LS is the one given by
Eq. (9.21b):
R ′
x p[nobs|x′ ] p[x′ ] dx′
.
x̂LS = R
p[nobs |x′ ] p[x′ ] dx′
(9.31)
After inserting the expression for p[nobs|x] given by Eq. (9.22)
and the expression for p(x) given by Eq. (9.26) into Eq. (9.31),
and then canceling terms that are common to the numerator and
www.dbooks.org
300
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
denominator but not functions of x, we have
R 1 ′ n +1
(x ) obs (1 − x′)N−nobs dx′
.
x̂LS = 0R 1
′ nobs (1 − x′ )N−nobs dx′
0 (x )
Answer: The a posteriori pdf is
(9.32)
The integration is somewhat cumbersome, but it leads to the
result
nobs + 1
x̂LS =
.
(9.33)
N +2
For N = 10, x̂LS = (nobs + 1)/12. The variation of x̂LS with nobs ,
which is displayed in Fig. 9-2, starts at a value slightly smaller
than x̂MAP at nobs = 0 and concludes at x̂LS = 0.92 at nobs = 10.
The same coin-flip experiment has produced three different
estimators, depending on the estimation criterion and use or not
use of the a priori pdf.
p(x|nobs ) =
p[nobs |x] p(x)
.
p[nobs ]
Its logarithm, excluding constants, is
ln p(x|nobs ) = nobs ln x + (N − nobs + 1) ln(1 − x).
Setting its derivative to zero gives
nobs N − nobs + 1
−
= 0.
x
1−x
Solving gives
x̂MAP (nobs ) =
nobs
.
N +1
Note that x̂MAP (nobs ) < x̂MLE (nobs ) as x is likely to be small.
9-3 1-D Estimation Examples
Concept Question 9-2: How could we get three different
estimates of P[heads] from the same number of heads?
Exercise 9-5: Compute the MAP estimator for the coin-flip
experiment assuming the a priori uniform pdf p(x) = 1, for
0 ≤ x ≤ 1.
Answer: x̂MAP = x̂MLE = nobs /N.
Exercise 9-6: Compute the MAP estimator for the coin-flip
experiment with N = 10 and the a priori pdf p(x) = 3x2 , for
0 ≤ x ≤ 1.
In Section 9-2, we introduced three different approaches to estimation: MLE, MAP, and LS. We used the coin-flip problem as
an example for demonstrating all three approaches, computing
different estimates of P[heads] based on observations of the
number of heads in ten flips of the coin. Now we apply the three
estimation methods to the following discrete-time 1-D estimation scenarios: (1) estimating the mean value of an independent
and identically distributed (IID) random process (as described
in Section 8-9.1 A), in the context of polling, (2) denoising a
random process containing additive white Gaussian noise, given
its power spectral density, (3) deconvolving a random process
containing additive white Gaussian noise, and (4) deconvolving
a sparse random process containing additive white Gaussian
noise. We present these 1-D scenarios as a prelude to similar
presentations for 2-D images in Section 9-4.
Denoising and deconvolution were presented in Chapter 6,
so they should be familiar. The problem of estimating the
expectation of an IID Gaussian random process leads to an
estimator called the sample mean, which we present in the
context of combining poll results.
Answer: x̂MAP = (nobs + 2)/12.
9-3.1 Polling Scenario
Exercise 9-7: Compute the MAP estimator for the coin-flip
experiment with a triangular a priori pdf p(x) = 2(1 − x)
for 0 ≤ x ≤ 1.
Suppose an election is to take place involving two candidates, A
and B. The fraction of voters who will vote for candidate A is
the unknown constant µ , which we wish to estimate. The goal of
9-3 1-D ESTIMATION EXAMPLES
301
a pollster is to estimate µ from a subset of M polled voters, each
of whom tells the pollster for which candidate he or she will
vote. If the subset of voters is representative of the entire voting
population, and if the choice made by each voter is independent
of the choices made by other voters, the number of voters in the
subset of M voters who say they will vote for candidate A is
a discrete random variable m and its pmf is the binomial pmf
defined in Eq. (8.31), namely
M m
p[m] =
µ (1 − µ )M−m
m
µ m (1 − µ )M−m M!
,
=
m! (M − m)!
m = 0, . . . , M.
(9.34)
voters who indicated they plan to vote for candidate A and M is
the total number of polled voters.
For a single poll, the estimation process is straightforward,
but what if we have N different polls, each comprised of M
voters, resulting in N estimates of µ ? Instead of only one random
variable z with only one estimate µ̂ , we now have N random
variables, z[1] through z[N], and N estimates of µ , namely
µ̂ [i] = zobs [i] =
m[i]
,
M
0 ≤ i ≤ N.
(9.41)
How do we combine the information provided by the N polls to
generate a single estimate of µ ? As we see next, the answer to
the question depends on which estimation method we use.
For large values of M (where M is the number of polled
voters), p[m] can be approximated as a Gaussian pdf with a mean
M µ and a variance σm2 = M µ (1 − µ ). In shorthand notation,
m ∼ N (M µ , σm2 ).
(9.35)
where N is shorthand for normal distribution.
Here, random variable m represents the number of voters
who indicated they plan to vote for candidate A, out of a total
of M polled voters. Instead of m, the pollster can formulate
the estimation problem in terms of the normalized random
variable z, where
m
.
(9.36)
z=
M
Furthermore, z can be expressed as
z = µ + ν,
(9.38)
and σz2 is related to σm2 by
σz2 =
M µ (1 − µ ) µ (1 − µ )
σm2
=
=
.
M2
M2
M
(9.39)
If the polling indicates that the vote is likely to be close (between
the two candidates), then the pollster might set µ ≈ 1/2, in
which case σz2 ≈ 1/(4M). Additionally, the value of the unknown constant µ is estimated as
µ̂ = zobs =
m
,
M
The N polling processes can be represented by N random
variables z[i] given by
z[i] = µ + ν [i],
(9.40)
where zobs is the observed value of z, namely the number of
0 ≤ i ≤ N,
(9.42)
where µ is the true mean we wish to estimate and ν [i] is the zeromean Gaussian random variable associated with polling process
z[i]. Our first step involves combining the N random variables
into an N-dimensional random vector z:
(9.37)
with ν as a zero-mean Gaussian random variable:
ν ∼ N (0, σz2 ),
9-3.2 MLE Estimate of Sample Mean
z = [z[1], z[2], . . . , z[N]]T .
(9.43)
Since the various polls are independent processes, we assume
that the { z[i] } are independent random variables. Furthermore,
we assume that M, the number of polled voters, is sufficiently
large to justify describing the random variables { z[i] } by Gaussian pdfs with the same unknown mean µ and variance σz2 . The
joint conditional pdf p(z|µ ) of vector z is the product of the
marginal pdfs of all N members of z:
N
N
2
1
2
p(z|µ ) = ∏ p(z[i]) = ∏ p
e−(z[i]−µ ) /(2σz )
2
2
πσ
i=1
i=1
z
=
N
1
2
2
e−(z[i]−µ ) /(2σz ) . (9.44)
∏
2
N/2
(2πσz )
i=1
Setting z[i] = zobs [i] = the observed result of the ith poll and
µ = µ̂ = the estimate of µ to be computed in Eq. (9.44) gives
the likelihood function. Taking its natural logarithm gives the
www.dbooks.org
302
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
log-likelihood function:
ln(p(zobs |µ̂ )) = −
be expressed as
N
1 N
ln(2πσz2 ) − 2 ∑ (zobs [i] − µ̂ )2 . (9.45)
2
2σz i=1
The log-likelihood function is maximized by setting its partial
derivative with respect to µ̂ to zero:
0=
∂
1 N
ln(p(zobs |µ̂ )) =
∑ (zobs [i] − µ̂ ).
∂ µ̂
2σz2 i=1
1 N m[i]
1 N
1 N
zobs [i] =
=
∑
∑
∑ m[i],
N i=1
N i=1 M
NM i=1
p(µ |zobs ) =
(9.47)
9-3.3 MAP Estimate of Sample Mean
We again have N polls, represented by { z[i] }, but we also
have some additional information generated in earlier polls of
candidate A versus candidate B. The data generated in earlier
polls show that the fraction µ of voters who preferred candidate
A varied from one poll to another, and that collectively µ
behaves like a Gaussian random variable with a mean µ p and
a variance σ p2 :
µ ∼ N (µ p , σ p2 ).
(9.48)
Since the information about the statistics of µ is available ahead
of the N polls, we refer to the pdf of µ as an a priori pdf:
(9.49)
Our plan in the present subsection is to apply the MAP method
outlined in Section 9-1.2, wherein the goal is to maximize
the a posteriori pdf p(unknown mean µ | observation vector zobs ) = p(µ |zobs ). Using the form of Eq. (9.5), p(µ |zobs ) can
(9.50)
1
2
1
2
×q
e−(µ −µ p ) /(2σ p )
C(2πσz2 )N/2
2
2πσ
p
N
which is equal to the total number of voters (among all N polls)
who selected candidate A, divided by the total number of polled
voters, NM. The result given by Eq. (9.47) is called the sample
mean of { zobs [i] }. As with the MLE estimator in Eq. (9.25)
for the coin-flip problem, this is the “obvious” estimator of the
mean µ .
Note that Eq. (9.42) can be interpreted as a set of N observations of a white Gaussian random process with unknown
mean µ . So the sample mean can be used to estimate the
unknown mean of a white Gaussian random process.
2
2
1
e−(µ −µ p) /(2σ p ) .
p(µ ) = q
2πσ p2
p(µ )
p(µ )
= p(zobs |µ )
,
p(zobs )
C
where we set p(zobs ) = C because it is not a function of the
unknown mean µ .
Inserting Eqs. (9.44), with z = zobs , and (9.49) into Eq. (9.50)
leads to
(9.46)
Solving for µ̂ gives the MLE estimate of µ :
µ̂MLE =
p(µ |zobs ) = p(zobs |µ )
2
2
× ∏ e−(zobs [i]−µ ) /(2σz ) .
(9.51)
i=1
The log-likelihood function is
N
1
ln(2πσz2 ) − ln(2πσ p2 )
2
2
1 N
1
− 2 (µ − µ p)2 − 2 ∑ (zobs [i] − µ )2 .
2σ p
2σz i=1
(9.52)
ln(p(µ |zobs )) = − lnC −
The a posteriori estimate µ̂MAP is the value of µ that maximizes
the log-likelihood function, which is obtained by setting its
partial derivative with respect to µ to zero:
#
"
"
#
1
1 N
∂
∂
2
2
− 2 (µ − µ p ) +
− 2 ∑ (zobs [i] − µ ) .
0=
∂µ
2σ p
∂µ
2σz i=1
The solution leads to
µ̂MAP = µ =
σz2
σ p2
!
N
µ p + ∑ zobs [i]
i=1
σ2
N + z2
σp
.
(9.53)
The MAP estimate is the sample mean of the following:
{z[i]}, augmented with (σz2 /σ p2 ) copies of µ p , for a total of
(N + σz2 /σ p2 ) “observations.” For the polling problem, if M is
the number of voters polled in each of the N polls and M ′ is
the number of voters polled in each of the earlier polls, then
(σz2 /σ p2 ) = M ′ /M, so if the previous polls polled more voters,
its estimate µ p of µ is weighed more heavily.
9-4 LEAST-SQUARES ESTIMATION
303
Example 9-2: Polling Example
From Chapter 8, the variance of the sum of
uncorrelated random variables is the sum of their variances, independent random variables are uncorrelated, and
2
σax
= a2 σx2 . Let µ = N1 ∑Ni=1 xi be the sample mean of the
{xi }. Then
σ2
1
σµ2 = 2 N σx2 = x .
N
N
Answer:
Five different polls of 1000 voters each were taken. The fractions of voters supporting candidate A were:
zobs = { 0.51, 0.55, 0.53, 0.54, 0.52 }.
Compute (a) the MLE estimate µ̂MLE , and (b) the MAP estimate
µ̂MAP , given that 2000 voters had been polled in an earlier poll
in which 60% of voters selected candidate A.
Also, E[µ ] = N1 ∑Ni=1 E[xi ] = x, so the sample mean is an
unbiased estimator of x. So the sample mean converges to
the actual mean x of the {xi } as N → ∞. This is why the
sample mean is useful, and why polling works.
Solution: (a) The MLE estimate is simply the sample mean:
µ̂MLE =
9-4 Least-Squares Estimation
1 5
∑ zobs [i]
5 i=1
1
= (0.51 + 0.55 + 0.53 + 0.54 + 0.52) = 0.53.
5
(b) The MAP estimate is given by Eq. (9.53):
µ̂MAP =
σz2
σv2
9-4.1 Gaussian Random Vectors
Generalizing Eq. (9.18) for two jointly Gaussian random vectors
x and y (instead of x and yobs ), the least-squares estimate x̂LS of
vector x, given vector y, is given by the conditional mean
5
x̂LS = E[x|y] = x + Kx,yK−1
y (y − y),
µ p + ∑ zobs [i]
i=1
σz2
N+ 2
σv
.
From the available information, N = 5, µ p = 0.6 (earlier
poll), M = 1000 votes, and M ′ = 2000 voters (earlier a priori
information). Also, σz2 /σv2 = M ′ /M = 2. Using these values
leads to
(9.54)
where x and y are the means of vectors x and y, respectively,
Kx,y is the cross-covariance matrix defined by Eq. (8.75), and
Ky is the covariance matrix given by the form of Eq. (8.74). If x
and y are zero-mean, Eq. (9.54) simplifies to
−1
x̂LS = Kx,y K−1
y y = Rx,y Ry y,
(9.55)
(zero-mean jointly Gaussian)
5
µ̂MAP =
2 × 0.6 + ∑ zobs [i]
i=1
= 0.55.
5+2
The a priori information biased the estimate by raising it from
µ̂MLE = 0.53 to µ̂MAP = 0.55.
Concept Question 9-3: Another term for the sample
mean of a bunch of numbers is their what?
where—because x and y are zero-mean—Kx,y was replaced
with the cross-correlation function Rx,y and Ky was replaced
with the auto-correlation function Rx .
Vector x represents the unknown quantity, vector x̂LS represents the LS estimate of x, and the difference between them,
e = (x − x̂LS ), represents the estimation error. Also, vector y
represents the observation. The expectation of the product of the
estimation error and the observation is
Exercise 9-8: Compute the mean and variance of the
sample mean of N iid random variables {xi }, each of which
has mean x and variance σx2 .
www.dbooks.org
E[(x − x̂LS )yT ] = E[x yT] − E[x̂LS yT ]
T
= Rx,y − Rx,yR−1
y E[y y ]
= Rx,y − Rx,yR−1
y Ry
= Rx,y − Rx,y = 0.
(9.56)
304
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
◮ The result given by Eq. (9.56) is a statement of orthogonality: The estimation error e = (x − x̂LS ) is uncorrelated
with the observation y used to produce the LS estimate
x̂LS . ◭
◮ The result given by Eq. (9.59) is another statement
of orthogonality: When the expectation of the product of
two random variables—in this case, the error e[n] and
the observation y[n − j]—is zero, it means that those two
random variables are uncorrelated. We conclude from this
observation that if x[n] and y[n] are jointly Gaussian WSS
random processes, then x̂LLSE [n] = x̂LS [n]. ◭
9-4.2 Linear Least-Squares Estimation
We now consider a scenario in which the estimator is constrained to be a linear function of the observation. As we shall
see shortly, the derived linear least-squares estimate x̂LLSE is
equal to the LS estimate for Gaussian random vectors of the
preceding subsection.
Given that x[i] and y[i] are zero-mean, jointly wide-sensestationary (WSS, defined in Section 8-9.3) random processes,
our present goal is to compute the linear least-squares estimate
x̂LLSE[n] of x[n] at discrete time n from the infinite set of
observations { y[i], −∞ < i < ∞ }. The estimator is constrained
to be a linear function of the { y[i] }, which means that x̂LLSE [n]
and { y[i] } are related by a linear sum of the form
∞
x̂LLSE [n] = ∑ h[i] y[n − i] = h[n] ∗ y[n].
(9.57)
i=−∞
Here, h[n] is an unknown function (filter) that is yet to be
determined.
Let us consider the error e[n] between x[n] and the estimate
x̂LLSE:
∞
e[n] = x[n] − x̂LLSE[n] = x[n] − ∑ h[i] y[n − i].
(9.58)
9-4.3 1-D Stochastic Wiener Smoothing Filter
We now use the orthogonality relationship to derive 1-D stochastic versions of the deterministic Wiener deconvolution filter
presented in Chapter 6 and the deterministic sparsifying denoising filter presented in Chapter 7. The stochastic versions
of these filters assume that the signals are random processes
instead of just functions. This allows a priori information about
the signals, in the form of their power spectral densities, to be
incorporated into the filters. If all of the random processes are
white, the stochastic filters reduce to the deterministic filters of
Chapters 6 and 7, as we show later.
Another advantage of the stochastic forms of these filters is
that the trade-off parameter λ in the Tikhonov and LASSO
criteria can now be interpreted as an inverse signal-to-noise
ratio, as we also show later. We derive the 1-D forms of
stochastic filters so we may generalize them to 2-D later.
Our task in the present subsection is to determine the filter h[i]
introduced in Eq. (9.57). To that end, we insert Eq. (9.58) into
Eq. (9.59) and apply the distributive property of the expectation:
0 = E[e[n] y[n − j]]
"
=E
Next, let us square the error e[n], take the derivative of its
expectation with respect to h[ j] and then set it equal to zero:
∂
E[e[n]2 ]
∂ h[ j]
∂ e[n]
= 2E e[n]
∂ h[ j]
"
!#
∞
∂
= 2E e[n]
x[n] − ∑ h[ j] y[n − j]
∂ h[ j]
j=−∞
x[n] − ∑ h[i] y[n − i]
i=−∞
#
y[n − j]
∞
= E [x[n] y[n − j]] − ∑ h[i] E[y[n − i] y[n − j]]
i=−∞
0=
= 2E[e[n] (−y[n − j])]
= −2E[e[n] y[n − j]].
!
∞
i=−∞
∞
= Rxy [ j] − ∑ h[i] Ry [ j − i]
i=−∞
= Rxy [ j] − h[ j] ∗ Ry[ j],
(9.60)
where, in the last step, we used the standard definition of
convolution of two discrete-time 1-D signals, Eq. (2.71a).
Taking the DTFT of Eq. (9.60) gives
(9.59)
0 = Sxy (Ω) − H(Ω) Sy (Ω).
9-4 LEAST-SQUARES ESTIMATION
305
The solution for H(Ω) is labeled HSDN (Ω):
HSDN (Ω) =
Taking the DTFTs of Eqs. (9.63) and (9.65) gives
Sxy (Ω)
.
Sy (Ω)
(9.61)
The subscript SDN stands for stochastic denoising Wiener filter.
The LS estimate of x[n] is obtained from observation y[n] by
applying the recipe:
x̂LLSE[n] = x̂LS [n] = hSDN [n] ∗ y[n],
(9.62a)
Sxy (Ω)
.
Sy (Ω)
(9.62b)
with
hSDN [n] = DTFT−1
Application examples follow in the next two subsections.
9-4.4 Stochastic Wiener Denoising
Let us suppose that y[n] are noisy observations of x[n]:
y[n] = x[n] + ν [n],
−∞ < n < ∞,
where ν [n] is a zero-mean IID random noise process. Also, x[n]
and ν [n] are uncorrelated and jointly WSS Gaussian random
processes. Our goal is to compute the linear least-squares estimate x̂LLSE[n] at discrete n from the infinite set of observations
{ y[i], −∞ < i < ∞ }. In order to apply the recipe given by
Eq. (9.62), we first need to obtain expressions for Sxy (Ω) and
Sy (Ω).
Since x[n] and ν [n] are uncorrelated and WSS, it follows that
the cross-correlation
Rxy [i, j] = E[x[i] (x[ j] + ν [ j])]
= E[x[i] x[ j]] + E[x[i] ν [ j]] = Rx [i − j].
Sxy (Ω) = Sx (Ω),
(9.66a)
Sy (Ω) = Sx (Ω) + σv2.
(9.66b)
and
The combination of the two results leads to
HSDN (Ω) =
Sxy (Ω)
Sx (Ω)
=
.
Sy (Ω)
Sx (Ω) + σv2
(9.67)
The expression given by Eq. (9.67) represents a stochastic
denoising filter because at frequency components Ω for which
Sx (Ω) ≫ σv2 , HSDN (Ω) ≈ 1 and x̂LS [n] ≈ y[n], but at frequency
components for which Sx (Ω) ≪ σv2 , HSDN (Ω) ≈ 0, thereby
filtering out the noise at frequencies where the noise drowns out
the signal.
Assuming the power spectral density Sx (Ω) and the noise
variance σv2 are known or can be estimated (as discussed later
in Section 9-7), the denoising recipe consists of the following
steps:
(1) Equation (9.67) is used to compute HSDN (Ω).
(2) The inverse DTFT is applied to obtain
hSDN [n] = DTFT−1 { HSDN (Ω) }.
(3) The LS denoised estimate of random process x[n] at time
n is
x̂LS [n] = hSDN [n] ∗ yobs[n],
(9.68)
where yobs [n] is the observation. Alternatively, yobs [n] can be
transformed to Yobs (Ω), then used to compute
x̂LS (Ω) = HSDN (Ω) Yobs (Ω),
(9.63)
after which an inverse transformation of x̂LS (Ω) yields x̂LS [n].
Similarly,
9-4.5 Stochastic Wiener Deconvolution Example
Ry [i, j] = E[(x[i] + ν [i])(x[ j] + ν [ j])]
= E[x[i] x[ j]] + E[ν [i] x[ j]] + E[x[i] ν [ j]] + E[ν [i] ν [ j]].
(9.64)
Now let y[n] be noisy observations of hblur [n] ∗ x[n]:
Since x[i] and ν [ j] are uncorrelated, the second and third terms
are zero. Hence,
where hblur [n] is a known impulse response. The goal is to
compute the linear least-square estimate x̂LLSE [n] at time n from
the observations { y[i], −∞ < i < ∞ }. The noise v[n] is a zeromean IID random process uncorrelated and jointly WSS with
x[n], and E(v[n]2 ) = σv2 δ [n].
Replacing x[n] with hblur [n] ∗ x[n] in the derivation leading to
Eq. (9.67) for the smoothing filter for noisy observations and
Ry [i, j] = Rx [i − j] + σv2 δ [i − j],
(9.65)
where σv2 is the variance of the noise ν [n]. Equations (9.63) and
(9.65) show that Rxy [i, j] = Rxy [i − j] and Ry [i, j] = Ry [i − j].
y[n] = hblur [n] ∗ x[n] + v[n],
www.dbooks.org
−∞ < n < ∞,
(9.69)
306
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
(3) x[n] at each time n has a Laplacian a priori pdf given by
using Eq. (8.161)(a and b) gives
Sy (Ω) = |Hblur (Ω)|2 Sx (Ω) + σv2 ,
Sxy (Ω) = H∗blur (Ω) Sx (Ω),
(9.70a)
(9.70b)
p(x[n]) = √
√
1
e− 2|x[n]|/σx .
2 σx
(9.75)
which leads to the stochastic deconvolution (SDC) Wiener
filter:
WSDC (Ω) =
H∗blur (Ω) Sx (Ω)
.
|Hblur (Ω)|2 Sx (Ω) + σν2
(9.71a)
For frequencies Ω for which |Hblur (Ω)|2 Sx (Ω) ≫ σν2 ,
H∗blur (Ω) Sx (Ω)
1
WSDC (Ω) ≈
=
,
|Hblur (Ω)|2 Sx (Ω) Hblur (Ω)
(9.71b)
which makes perfect sense for a noiseless deconvolution filter.
The Wiener deconvolution filter given by Eq. (9.71a) can be
rewritten as
WSDC (Ω) =
=
(9.72)
which has the nice interpretation of a Wiener denoising filter
followed by a noiseless deconvolution filter.
Assuming Hblur (Ω), Sx (Ω), and σv2 are known or can be estimated through other means, a reconstructed stochastic Wiener
estimate can be computed
x̂SDC [n] = DTFT−1 [WSDC (Ω) Yobs (Ω)],
(9.73)
where Yobs (Ω) is the DTFT of the noisy, convolved observation
yobs [n].
9-4.6 Stochastic MAP Sparsifying Denoising
Estimator
(9.76a)
T
y = [y[1], y[2], . . . , y[N]] ,
(9.76b)
T
(9.76c)
v = [v[1], v[2], . . . , v[N]] .
Using Eq. (9.75), the a priori pdf of unknown vector x is
(9.77)
For y = yobs , where yobs is the observation vector, the conditional pdf p(yobs |x) is the same as the jointly Gaussian pdf of
the noise vector ν with ν [n] = yobs [n] − x[n]:
N
1
2
2
p(yobs |x) = p(ν ) = ∏ p
e−ν [n] /(2σv )
2
2πσv
n=1
N
2
1
2
=∏p
e−(yobs [n]−x[n]) /(2σv ) . (9.78)
2
2πσv
n=1
The MAP estimate x̂MAP [n] at time n is related to the a posteriori
pdf p(x|yobs), which is related to p(yobs |x) and p(x) by the
vector equivalent of Eq. (9.5):
p(x|yobs) = p(yobs |x)
p(x)
.
C
(9.79)
Inserting Eqs. (9.77) and (9.78) into Eq. (9.79) and then taking
the natural log leads to the MAP log-likelihood function:
We return to the denoising problem:
y[n] = x[n] + v[n],
x = [x[1], x[2], . . . , x[N]]T ,
N
N
√
1
p(x) = ∏ p(x[n]) = ∏ √
e− 2 |x[n]|/σx .
n=1 2 σx
n=1
H∗blur (Ω) Sx (Ω)
|Hblur (Ω)|2 Sx (Ω) + σν2
|Hblur (Ω)|2 Sx (Ω)
1
×
,
|Hblur (Ω)|2 Sx (Ω) + σν2 Hblur (Ω)
We will now show that these conditions lead to a sparse
(mostly zero-valued) estimate of x[n].
To best use the combined information, we form vectors x, y,
and v:
1 ≤ n ≤ N,
(9.74)
and we make the following assumptions:
(1) ν [n] is a zero-mean white Gaussian random process with
Rν [n] = σν2 δ [n].
(2) x[n] and ν [n] are IID and jointly WSS random processes.
ln(p(x|yobs )) = − lnC −
N
ln(2πσv2 )
2
1 N
∑ (yobs[n] − x[n])2
2σv2 n=1
√ N
√
2
− N ln( 2 σx ) −
∑ |x[n]|,
σx n=1
−
(9.80)
9-5 DETERMINISTIC VERSUS STOCHASTIC WIENER FILTERING
The usual procedure for obtaining the MAP estimate x̂MAP [n]
involves taking the partial derivative of the log-likelihood function with respect to x[n], equating the result to zero, and then
solving for x[n]. In the present case, the procedure is not so
straightforward because one of the terms includes the absolute
value of x[n]. Upon ignoring the three terms in Eq. (9.80)
that do not involve x[n] (because they exercise no impact on
minimizing the log-likelihood function), and then multiplying
the two remaining terms by (−σv2 ), we obtain a new cost
functional
Λ=
√ σ2 N
1 N
(yobs [n] − x[n])2 + 2 v ∑ |x[n]|.
∑
2 n=1
σx n=1
(9.81)
The functional form of Λ is identical to that of the LASSO cost
functional given in Eq. (7.106), and so is the form of the solution
given by Eq. (7.109):


for yobs [n] > λ ,
yobs [n] − λ
(9.82)
x̂MAP [n] = yobs [n] + λ
for yobs [n] < −λ ,

0
for |yobs [n]| < λ ,
where λ is the noise-to-signal ratio
λ=
√ σv2
.
2
σx
(9.83)
Concept Question 9-4: How did we use the orthogonality principle of linear prediction in Section 9-4?
How is the Tikhonov parameter
λ interpreted in Section 9-4?
Concept Question 9-5:
Exercise 9-9: What is the MAP sparsifying estimator when
the noise variance σv2 is much smaller than σx ?
Answer: When σv2 ≪ σx , λ → 0, and x̂[n] = yobs [n].
Exercise 9-10: What is the MAP sparsifying estimator
when noise variance σv2 is much greater than σx ?
Answer: When σv2 ≫ σx , λ → ∞, and x̂MAP = 0. This
makes sense: the a priori information that x is sparse
dominates the noisy observation.
307
9-5 Deterministic versus Stochastic
Wiener Filtering
Through a speedometer example, we now compare results of
speed estimates based on two Wiener filtering approaches, one
using a deterministic filter and another using a stochastic filter.
The speedometer accepts as input y(t), a noisy measurement of
the distance traveled at time t, measured by an odometer, and
converts y(t) into an output speed r(t) = dy/dt. [We use symbol
r (for rate) instead of s, to denote speed to avoid notational
confusion in later sections.] Sampling y(t) and r(t) at a sampling
interval ∆ converts them into discrete-time signals y[n] and r[n]:
y[n] = y(t = n∆),
y[n] − y[n − 1]
r[n] =
.
∆
(9.84a)
(9.84b)
The observed distance y[n] contains a noise component ν [n],
y[n] = s[n] + ν [n],
(9.85)
where s[n] is the actual distance that the odometer would have
measured had it been noise-free, and ν [n] is a zero-mean white
noise Gaussian random process with variance σv2 . An example
of a slightly noisy odometer signal y[n] with σv2 = 2 is shown
in Fig. 9-3(a), and the application of the differentiator given
by Eq. (9.84b) with ∆ = 1 s results in the unfiltered speed
ru [n] shown in Fig. 9-3(b). The expression for ru [n] is, from
Eq. (9.84b),
y[n] − y[n − 1]
∆
ν [n] − ν [n − 1]
s[n] − s[n − 1]
+
=
∆
∆
= rtrue [n] + rnoise[n].
(9.86)
ru [n] =
The goal of the Wiener filter is to estimate the true speed rtrue [n],
so for the purpose of present example, we show in Fig. 9-2(c)
a plot of the true speed rtrue [n], against which we will shortly
compare the Wiener-filter results.
9-5.1 Deterministic Wiener Filter
For ∆ = 1 s, the true speed rtrue [n] is related to the true distance
s[n] by
rtrue [n] = s[n] − s[n − 1].
(9.87)
www.dbooks.org
308
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
rtrue[n]
y[n]
800
12
10
600
8
400
6
200
4
2
0
−200
0
0
50
100
150
n −2 0
250
200
50
ru[n]
rˆ D[n]
15
15
10
10
5
5
0
0
0
50
100
150
150
200
250
n
200
250
n
(c) True speed rtrue[n]
(a) Noisy odometer signal y[n]
−5
100
200
250
n −5 0
50
100
150
(d) Speed rˆD[n] estimated by
deterministic Wiener filter
(b) Unfiltered speed ru[n]
rˆs[n]
12
10
8
6
4
2
0
−2
0
50
100
150
200
250
n
(e) Speed rˆS[n] estimated by stochastic Wiener filter
Figure 9-3 Speedometer example: (a) noisy odometer signal y[n], (b) unfiltered speed ru [n] = (y[n] − y[n − 1])/∆, with ∆ = 1 s, (c) true
speed rtrue [n] = (s[n] − s[n − 1])/∆, with ∆ = 1 s, (d) deterministically estimated speed r̂D [n], and (e) stochastically estimated speed r̂S [n].
For a segment of length N, with { n = 1, . . . , N }, the N-point
DFT of Eq. (9.87) gives
R[k] = S[k] − e
− j2π k/N
S[k] = (1 − e
− j2π k/N
) S[k],
(9.88)
where we used property #2 in Table 2-9 to compute the second
term of Eq. (9.88). The frequency response function H[k] of the
noise-free system described by Eq. (9.87) is
H[k] =
1
S[k]
.
=
−
R[k] 1 − e j2π k/N
(9.89)
The deterministic Wiener denoising/deconvolution filter was
presented earlier in Section 6-4.4 for 2-D images. Converting
the notation from 2-D to 1-D, as well as replacing the symbols
in Eqs. (6.32a) and (6.32b) to match our current speedometer
9-6 2-D ESTIMATION
309
problem, we obtain the following expression for the estimated
DFT of the true speed rtrue [n]:
R̂D [k] = Y[k] WDDC [k]
with
WDDC [k] =
(9.90a)
H∗ [k]
,
|H[k]|2 + λ 2
(9.90b)
where Y[k] is the DFT of the noisy observation y[n], WDDC [k]
is the deterministic deconvolution Wiener filter, and λ is
the trade-off parameter in Tikhonov regularization. Combining
Eqs. (9.89), (9.90a), and (9.90b), and multiplying the numerator
and denominator by |1 − e j2π k/N |2 leads to the deterministic
estimate
Y[k] (1 − e− j2π k/N )
.
(9.91)
R̂D [k] =
1 + |1 − e− j2π k/N |2 λ 2
To obtain the “best” estimate of the speed rD [n] using the
deterministic Wiener filter, we need to go through the estimation
process multiple time using different values of λ . The process
entails the following steps:
(1) Compute the DFT Y[k] of the noisy observations y[n].
(2) Compute R̂D [k] using Eq. (9.91) for various values of λ .
(3) Perform an inverse N-point DFT to compute r̂D [n].
For the speedometer example, the outcome with the “seemingly”
best result is the one with λ = 0.1, and its plot is shown in
Fig. 9-3(d). It is an improvement over the unfiltered speed ru [n],
but it still contains a noticeable noisy component.
As noted later in Section 9-7 and 9-8, a practical model for
Ss (Ω) is
C
Ss (Ω) = 2 ,
(9.92c)
Ω
where C is a constant. Using Eqs. (9.89) and (9.92c) leads to the
stochastic estimate
R̂S (Ω) =
Y(Ω)(1 − e− jΩ)
.
σ2
1 + |1 − e− jΩ|2 Ω2 v
C
(9.93)
Implementation of the stochastic Wiener filter entails the following steps:
(1) For the given observation distance signal y[n], compute its
DFT Y[k].
(2) Convert Eq. (9.93) into a numerical format by replacing Ω
with 2π k/N everywhere.
(3) Using C′ = σv2 /C as a parameter, compute R̂S [k] for various
values of C′ .
(4) Perform an inverse DFT to obtain r̂s [n] for each value of C′ .
(5) Select the value of C′ that appears to provide the best result.
The result for the seemingly best value of C′ , which turned out to
be C′ = 0.4, is displayed in Fig. 9-3(e). Comparison of the plots
for r̂D [n] and r̂s [n] reveals that the stochastic approach provides
a much better rendition of the true speed rtrue [n] than does the
deterministic approach.
Concept Question 9-6: Why did the stochastic Wiener
filter produce a much better estimate than the deterministic
Wiener filter in Section 9-5?
9-5.2 Stochastic Wiener Filter
Based on the treatment given in Section 9-4.5, the stochastic
Wiener approach to filtering the noisy signal y[n] uses a stochastic denoising/deconvolution Wiener filter WSDC (Ω) as follows:
R̂S (Ω) = Y(Ω) WSDC (Ω),
with
H∗ (Ω) Ss (Ω)
,
WSDC (Ω) =
|H(Ω)|2 Ss (Ω) + σv2
(9.92a)
Exercise 9-11: What happens to the stochastic Wiener filter
when the observation noise strength σv2 → 0?
Answer:
From Eq. (9.93), letting σv2 → 0 makes
R̂S (Ω) = Y(Ω)(1 − e− jΩ ), whose inverse DTFT is
r̂s [n] = y[n] − y[n − 1], which is Eq. (9.84b).
(9.92b)
where H(Ω) is H[k] of Eq. (9.89) with Ω = 2π k/N, Ss (Ω) is the
power spectral density of s[n] and σv2 is the noise variance. Since
the true distance s[n] is an unknown quantity, its power spectral
density Ss (Ω) is unknown. In practice, Ss (Ω) is assigned a functional form based on experience with similar random processes.
9-6 2-D Estimation
We now extend the various methods that were introduced in earlier sections for estimating 1-D random processes to estimating
2-D random fields. The derivations of these estimators are direct
generalizations of their 1-D counterparts.
www.dbooks.org
310
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
9-6.1 2-D MLE Estimate of Sample Mean
and
Throughout Chapter 8, it was assumed that WSS random fields
were zero-mean, so realized by subtracting the constant mean
from the original random field. To perform the subtraction step,
we need to either know the value of the mean f or we need to
estimate it from the observation of the random field fobs [n, m].
Extending the result of the 1-D MLE estimate of the mean given
by Eq. (9.47) to a 2-D random field of size N × N gives
f MLE =
1 N N
∑ ∑ fobs [n, m].
N 2 n=1
m=1
(9.94)
Sg (Ω1 , Ω2 ) = Sf (Ω1 , Ω2 ) + σv2 ,
(9.97b)
in which case the expression for HSDN (Ω1 , Ω2 ) becomes
HSDN (Ω1 , Ω2 ) =
Sf (Ω1 , Ω2 )
.
Sf (Ω1 , Ω2 ) + σv2
(9.98)
If the power spectral density Sf (Ω1 , Ω2 ) and the noise variance σv2 are known, a 2-D image can be denoised by applying
the following recipe:
(1) Equation (9.98) can be used to compute HSDN (Ω1 , Ω2 ).
◮ Estimating the mean f and subtracting it from fobs [n, m]
to generate a zero-mean observation random field gobs [n, m]
is a required prerequisite step to the application of Wiener
filters for image smoothing and deconvolution. ◭
As in 1-D, the least-squares (LS) estimate for jointly Gaussian
random fields—unknown x[n] and observation y[n]—is equal to
the linear least-squares estimate (LLSE) for any pair of linearly
related random fields, whether jointly Gaussian or not.
◮ Hence, for convenience and with no loss of generality,
we shall assume that all random fields are jointly WSS
Gaussian, for estimation purposes. ◭
Given an unknown random field f [n, m], a zero-mean noise
random field ν [n, m], and an observation random field g[n, m],
with
1 ≤ n, m ≤ N,
(9.95)
the 2-D version of the 1-D stochastic denoising Wiener filter
given by Eq. (9.61) is
HSDN (Ω1 , Ω2 ) =
hSDN [n, m] = DSFT−1 { HSDN (Ω1 , Ω2 ) }.
(9.99)
(3) Then, each element of f [n, m] is denoised by using the
filtering operation
9-6.2 2-D Wiener Filter
g[n, m] = f [n, m] + ν [n, m],
(2) The inverse DSFT is applied to obtain
Sfg (Ω1 , Ω2 )
.
Sg (Ω1 , Ω2 )
(9.96)
9-6.3 Stochastic 2-D Wiener Denoising Filter
Moreover, if ν [n, m] is a zero-mean white Gaussian noise random field with variance σv2 , it follows that
Sfg (Ω1 , Ω2 ) = Sf (Ω1 , Ω2 )
(9.97a)
fˆS [n, m] = hSDN [n, m] ∗ ∗ gobs [n, m],
(9.100)
where gobs [n, m] is the noisy observation of f [n, m]. As
in 1-D, hSDN [n, m] usually is windowed to a finite spatial
extent.
9-6.4 Stochastic 2-D Wiener Deconvolution
Filter
Let us assume we have a noisy 2-D image g[n, m]. In fact, in
addition to the added noise ν [n, m], the true (unknown) image
f [n, m] had been blurred by the measurement process. Thus,
g[n, m] = hblur [n, m] ∗ ∗ f [n, m] + ν [n, m],
(9.101)
where hblur [n, m] is a blur filter with a known PSF, established
through a calibration process.
Our goal is to compute the stochastic reconstruction fˆS [n, m]
of f [n, m] at each location [n, m], from noisy observations
{ gobs [i, j], −∞ ≤ i, j ≤ ∞ }. To that end, we introduce the 2-D
version of Eqs. (9.70a and b):
Sfg (Ω1 , Ω2 ) = H∗blur (Ω1 , Ω2 ) Sf (Ω1 , Ω2 ),
2
Sg (Ω1 , Ω2 ) = |Hblur (Ω1 , Ω2 )|
Sf (Ω1 , Ω2 ) + σv2,
(9.102a)
(9.102b)
and then we use them to obtain the stochastic deconvolution
9-6 2-D ESTIMATION
311
form of a zero-mean white Gaussian random field ν [n, m]:
Wiener filter
Sfg (Ω1 , Ω2 )
Sg (Ω1 , Ω2 )
H∗blur (Ω1 , Ω2 ) Sf (Ω1 , Ω2 )
=
. (9.103)
|Hblur (Ω1 , Ω2 )|2 Sf (Ω1 , Ω2 ) + σv2
WSDC (Ω1 , Ω2 ) =
As before, we assume that we know the power spectral density
Sf (Ω1 , Ω2 ), the noise variance σv2 , and the blur filter hblur [n, m],
in which case we follow the recipe:
(1) Obtain Hblur (Ω1 , Ω2 ) from
Hblur (Ω1 , Ω2 ) = DSFT{ hblur [n, m] }.
(9.104)
(2) Compute WSDC (Ω1 , Ω2 ) using Eq. (9.103).
(3) Compute wSCD [n, m] from
wSDC [n, m] = DSFT−1 { WSDC (Ω1 , Ω2 ) }.
(9.108)
The outcome of the blurring and noise-addition processes is
displayed in Fig. 9-4(b).
A. Deterministic Wiener Deconvolution
By extending the 1-D speedometer recipe of Section 9-5.1 to the
2-D MRI image, we obtain the deterministic Wiener estimate
fˆD [n, m] as follows:
(1) Hblur [k1 , k2 ] is computed from hblur [n, m] by taking the 2-D
DFT of Eq. (9.107).
(2) The deterministic deconvolution Wiener filter
(9.105)
(4) Estimate fˆs [n, m] at each location [n, m] based on observations gobs [n, m] by applying the convolution:
fˆs [n, m] = wSDC [n, m] ∗ ∗ gobs [n, m].
g[n, m] = hblur [n, m] ∗ ∗ f [n, m] + ν [n, m].
(9.106)
Does the stochastic Wiener deconvolution filter provide better
results than the deterministic Wiener deconvolution filter described in Section 6-4.4? The answer is an emphatic yes, as
demonstrated by the following example.
WDDC [k1 , k2 ] =
H∗blur [k1 , k2 ]
|Hblur [k1 , k2 ]|2 + λ 2
(9.109)
is computed for several values of the parameter λ .
(3) F̂D [k1 , k2 ] is estimated using
F̂D [k1 , k2 ] = G[k1 , k2 ] WDDC [k1 , k2 ],
(9.110)
where G[k1 , k2 ] is the 2-D DFT of the blurred image g[n, m].
(4) Application of the inverse DFT gives fˆD [n, m]. The “best”
resultant image, shown in Fig. 9-4(c), used λ = 0.1.
9-6.5 Deterministic versus Stochastic
Deconvolution Example
This is a 2-D “replica” of the 1-D speedometer example of
Section 9-5. The image in Fig. 9-4(a) is a high-resolution lownoise MRI image of a human head. We will treat it as an
“original” image f [n, m]. To illustrate (a) the effects of blurring
caused by the convolution of an original image with the PSF
of the imaging system, and (b) the deblurring realized by
Wiener deconvolution, we use a disk-shaped PSF with a uniform
distribution given by
(
1
for m2 + n2 < 145,
hblur [n, m] =
(9.107)
0
for m2 + n2 ≥ 145,
B. Stochastic Wiener Deconvolution
which provides a good model of out-of-focus imaging. The
circular PSF has a radius of 12 pixels. We also add noise in the
Upon setting Ω1 = 2π k1 /N and Ω2 = 2π k2 /N in Eqs. (9.111)
and (9.103), we obtain the stochastic Wiener deconvolution
The 1-D stochastic Wiener solution to the speedometer problem
is given in Section 9-5.2. The 2-D image deconvolution procedure using the stochastic Wiener approach is identical to the 1-D
procedure outlined earlier in part A of Section 9-6.5, except for
the form of the Wiener filter. For the stochastic case, the 2-D
equivalent of the expression given by Eq. (9.92c) for the power
spectral density is
Sf (Ω1 , Ω2 ) =
www.dbooks.org
C
.
(Ω21 + Ω22 )2
(9.111)
312
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
(a) Original MRI image f [n,m]
(c) Deconvolved image fD[n,m] using
deterministic Wiener filter
(b) Noisy blurred MRI image g[n,m]
(d) Deconvolved image fs[n,m] using
stochastic Wiener filter
ˆ
ˆ
Figure 9-4 MRI images: (a) Original “true” image, (b) image blurred by imaging system, (c) image deconvolved by deterministic Wiener
filter, and (d) image deconvolved by stochastic Wiener filter.
9-7 SPECTRAL ESTIMATION
313
filter:
H∗blur [k1 , k2 ]
.
4 2
σv 2
2π
2
2 2
|Hblur [k1 , k2 ]| +
(k + k2 )
N
C 1
(9.112)
Repetition of the recipe given earlier for the deterministic filter
but replacing WDDC [k1 , k2 ] in Eq. (9.110) with WSDC [k1 , k2 ] led
to the image shown in Fig. 9-4(d). The process was optimized by
repeating it for several different values of the parameter σv2 /C.
The “best” result was for σv2 /C = 1.
Comparison of the images in Fig. 9-4 leads to two conclusions:
(1) The image in part (c), generated by applying the deterministic Wiener filter, is far superior to the blurred image in part (b).
(2) Among the two filtering approaches, the stochastic approach (image in (d)) yields a sharper image than its deterministic counterpart, motivating the stochastic approach.
WSDC [k1 , k2 ] =
(1)
(9.113)
with gobs [n, m] representing the observation, f [n, m] representing the unknown random field, and ν [n, m] representing a white Gaussian random field with known variance
Sv (Ω1 , Ω2 ) = σv2 .
(2) f [n, m] and ν [n, m] are IID, jointly WSS random fields.
(3) Each f [n, m] has a 2-D Laplacian a priori pdf given by
(9.114)
The 2-D LASSO cost-functional equivalent to the 1-D expression given by Eq. (9.81) is
√ σ2 N N
1
Λ = ∑ ∑ (gobs [n, m] − f [n, m])2 + 2 v ∑ ∑ | f [n, m]|,
2 n=1 m=1
σf n=1 m=1
(9.115)
N
where λ is the noise-to-signal ratio
λ=
√ σv2
.
2
σf
(9.117)
Concept Question 9-7: Why did the stochastic Wiener
deconvolution filter produce a much better estimate than the
deterministic Wiener deconvolution filter in Section 9-6?
Exercise 9-12: When σf → ∞, the Laplacian pdf given by
Eq. (9.114) approaches the uniform distribution. What does
the MAP estimate reduce to when σf → ∞?
fMAP [n, m] → gobs [n, m].
The 2-D analogue of the 1-D sparsifying estimator of Section
9-4.6 can be summarized as follows:
√
1
e− 2 | f [n,m]|/σf .
p( f [n, m]) = √
2 σf
for gobs [n] > λ ,
for gobs [n] < −λ ,
for |gobs [n]| < λ ,
(9.116)
Answer: As σf → ∞, the parameter λ → 0, in which case
9-6.6 Stochastic 2-D MAP Sparsifying
Estimator
gobs [n, m] = f [n, m] + ν [n, m],
and the MAP estimate is


gobs [n, m] − λ
ˆfMAP [n, m] = gobs [n, m] + λ

0
9-7 Spectral Estimation
To apply the 1-D and 2-D stochastic denoising and deconvolution operations outlined in Sections 9-4 to 9-6, we need to know
the power spectral densities Sx (Ω) of the 1-D unknown random
process x[n] and Sf (Ω1 , Ω2 ) of the 2-D random field f [n, m].
Had x[n] been available, we could have applied the 1-D
N-order DFT to estimate Sx (Ω) using
2π k
1
Ŝx Ω =
=
N
N
N−1
∑ x[n] e
2
− j2π kn/N
,
(9.118)
n=0
for k = 0, 1, . . . , N − 1. The division by N converts energy
spectral density to power spectral density. Similarly, application
of the 2-D Nth-order DFT to f [n, m] leads
2 π k1
2 π k2
Ŝf Ω1 =
, Ω2 =
=
N
N
1
N2
N
N−1
∑ f [n, m] e
2
− j2π (nk1 +mk2 )/N
,
(9.119)
n=0
for k1 = 0, 1, . . . , N − 1, and k2 = 0, 1, . . . , N − 1.
This estimation method is known as the periodogram spectral
www.dbooks.org
314
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
estimator. Other spectral estimation methods exist as well, but
the central problem is that x[n] and f [n, m] are the unknown
quantities we wish to estimate, so we have no direct way to
determine their power spectral densities. However, we can use
parametric models to describe Sx (Ω) and Sf (Ω1 , Ω2 ), which
can then be used in the stochastic Wiener estimation recipes of
Sections 9-4 to 9-6 to obtain the seemingly best outcome. As we
shall see in the next section, many images and signals exhibit
a fractal-like behavior with corresponding 1-D and 2-D power
spectral densities of the form
C
,
for Ωmin < |Ω| < π ,
(9.120a)
|Ω|a
C
,
for Ωmin < |Ω1 |, |Ω2 | < π ,
Sf (Ω1 , Ω2 ) = 2
(Ω1 + Ω22)b
(9.120b)
definition applies to an image in 2-D. Fractals are quite common
in nature; examples include certain classes of trees, river deltas,
and coastlines (Fig. 9-5). In a perfect fractal, self-similarity
exists over an infinite number of scales, but for real objects the
similarity is exhibited over a finite number of scales.
An example of a fractal signal is shown in Fig. 9-6; note
the statistical resemblance between (a) the pattern of the entire
signal extending over the range between 0 and 1 in part (a) of the
figure and (b) the pattern in part (b) of only a narrow segment of
the original, extending between 0.4 and 0.5 of the original scale.
Sx (Ω) =
where a, b, C, and Ωmin are adjustable constant parameters.
The expressions given by Eq. (9.120) are known as power laws,
an example of which with b = 2 was used in Section 9-6.5 to
deconvolve a blurred MRI image (Fig. 9-4).
Exercise 9-13: Suppose we estimate the autocorrelation
function Rx [n] of data {x[n], n = 0, . . . , N − 1} using the
sample mean over i of {x[i] x[i − n]}, zero-padding {x[n]}
as needed. Show that the DFT of R̂x [n] is the periodogram
(this is one reason why the periodogram works).
Answer:
R̂x [n] =
=
1 N−1
∑ x[i] x[i − n]
N i=0
1 N−1
∑ x[i] x[−(n − i)]
N i=0
1
= (x[n] ∗ x[−n]).
N
From entries #4 and #6 of Table 2-7, the DSFT of
(x[n] ∗ x[−n]) is X(Ω) X(−Ω) = X(Ω) X∗ (Ω) = |X(Ω)|2 ,
which is the periodogram defined in Eq. (9.118).
9-8.1 Continuous-Time Fractals
Perfect self-similarity implies that a signal x(t) is identically
equal to a scaled version of itself, x(at), where a is a positive
scaling constant, but statistical self-similarity implies that if
x(t) is a zero-mean wide-sense stationary (WSS) random process, then its autocorrelation is self-similar. That is,
Rx (τ ) = C Rx (aτ ),
where τ is the time shift between x(t) and x(t − τ ) in
Rx (τ ) = E[x(t) x(t − τ )].
A fractal is a signal, image, or any object that exhibits selfsimilarity across different scales. A signal is self-similar if any
segment of it resembles the overall signal statistically. The same
(9.122)
According to Eq. (9.121), within a multiplicative constant C, the
variation of the autocorrelation Rx (τ ) with the time shift τ is the
same as the variation of the autocorrelation of the time-scaled
version Rx (aτ ) with the scaled time (aτ ).
The self-similarity property extends to the power spectral
density. According to Eq. (8.139a), the power spectral density
Sx ( f ) of x(t) is related to its autocorrelation function Rx (τ ) by
Sx ( f ) =
Z ∞
−∞
Rx (τ ) e− j2π f τ d τ .
(9.123)
Inserting Eq. (9.121) into Eq. (9.123) and then replacing τ with
τ ′ /a leads to
Sx ( f ) = C
=
9-8 1-D Fractals
(9.121)
Z ∞
−∞
Z
C ∞
a
Rx (aτ ) e− j2π f τ d τ
−∞
′
Rx (τ ) e
− j2π f τ ′ /a
C
d τ = Sx
a
′
f
. (9.124)
a
Using functional analysis, it can be shown that strictly speaking,
the only class of signals that are self-similar are power laws
characterized by the form x(t) = Ct a , where C and a are
constants. For a 1-D fractal random process, its power spectral
9-8 1-D FRACTALS
315
Figure 9-5 Fractal patterns in nature.
density has the form
Sx ( f ) =
C
.
| f |a
We should note that a > 0 and S( f ) = S(− f ).
(9.125)
Fractal random processes are often characterized using colors:
White Fractal (a = 0): a random process whose power spectral density, as given by Eq. (9.125), has a frequency exponent
a = 0. Hence, Sx ( f ) = C, which means that all frequencies are
present and weighted equally, just like white light.
www.dbooks.org
316
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
w(t)
w(t)
0.03
0.015
0.02
0.01
0.01
0.005
0
0
−0.01
−0.02
−0.005
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
t (s) −0.010.4
0.41
(a) 1-D signal
0.42
0.43
0.44
0.45
0.46
0.47
0.48
0.49
0.5
t (s)
(b) A small segment of the signal in (a)
Figure 9-6 Fractal signal w(t): Note the self-similarity between: (a) the entire signal and (b) a 10× expanded scale version of the segment
between t = 0.4 s and 0.5 s.
Pink Fractal (a = 1): a random process with Sx ( f ) = C/| f |.
Since higher frequencies (green and blue) are more heavily
attenuated, the combination appears pink in color.
Brown Fractal (a = 2): Brownian motion is characterized by
Sx ( f ) = C/ f 2 , hence the name brown.
A true fractal signal is not realistic because it has infinite total
power:
Z ∞
C
df →∞
P=
a
−∞ | f |
for any a ≥ 0 and C 6= 0. Hence, Sx ( f ) should be limited
to a range of frequencies extending between fmin and fmax .
Continuing with the color analogy, fmin has to be greater than
zero, because if a > 1,
Z ∞
C
df →∞
a
f min f
(a > 1).
unless fmin > 0. This unacceptable condition is known as the
infrared catastrophe because, relative to frequencies in the
viable part of the spectrum, infrared frequencies are considered
to be very low. On the other end of the spectrum, the ultraviolet
catastrophe represents the condition when 0 < a < 1 and
Z fmax
0
C
df →∞
fa
C
,
| f |a
Z fmax
fmin < | f | < fmax .
Sx ( f ) d f = 2
P = Rx (0) = σx2 .
(9.128)
9-8.2 Discrete-Time Fractals
As noted earlier, the power spectral density of a realistic fractal
signal x(t) has to be bandlimited to within a range between
fmin and fmax . Hence, x(t) can be converted to a discrete-time
fractal random process x[n] by sampling it at a rate exceeding the
Nyquist sampling rate of 2 fmax samples/s. The self-similarity
property of x(t) is equally present in x[n], although it is more
difficult to envision it in discrete time.
The functional form of the power spectral density of x[n] is
analogous to that given by Eq. (9.125):
Sx (Ω) =
C
,
|Ω|a
Ωmin < |Ω| < π ,
(9.129)
for some lower frequency bound Ωmin . Since Sx (Ω) = Sx (−Ω)
the average power of x[n] is
(9.126)
Since Sx ( f ) is symmetrical with respect to f = 0,
Sx ( f ) = Sx (− f ), and the total power of the fractal random
Z fmax
C
2C
1−a
df =
( f 1−a − fmin
).
a
1 − a max
f min f
f min
(9.127)
The total power also is related to the zero-shift autocorrelation
function Rx (0) and to the variance σx2 of the zero-mean Gaussian
random process x(t) as in:
P=2
(0 < a < 1),
unless fmax < ∞. Hence, for a realistic fractal random process,
Sx ( f ) =
process is
Pav = 2 ×
=
1
π
1
2π
Z π
Z π
Ωmin
Sx (Ω) dΩ
C
C
dΩ =
(π 1−a − Ω1−a
min ).
a
π (1 − a)
Ωmin Ω
(9.130)
9-8 1-D FRACTALS
317
9-8.3 Wiener Process
A Wiener process w(t) is a zero-mean non-WSS random process generated by integrating a white Gaussian random process
z(t):
Z
t
w(t) =
−∞
z(τ ) d τ .
(9.131)
The Wiener process is a 1-D version of Brownian motion, which
describes the motion of particles or molecules in a solution.
Often w(t) is initialized with w(0) = 0.
Eve though the Wiener process is not WSS—and therefore
it does not have a definable power spectral density, we will
nonetheless proceed heuristically to obtain an expression for
Sw ( f ). To that end, we start by converting Eq. (9.131) into the
differential form
dw
= z(t).
(9.132)
dt
Utilizing entry #5 in Table 2-4, the Fourier transform of the
system described by Eq. (9.132) is
( j2π f ) W( f ) = Z( f ),
W( f )
1
=
.
Z( f )
j2π f
which leads to
Sw ( f ) =
σz2 1
.
4π 2 f 2
Z ∞
−∞
2
w(t) e− j2π f t dt .
(9.137)
Figure 9-7(b) displays a plot of Sw ( f ) as a function of f on a
log-log scale over the range 1 < f < 400 Hz. Superimposed onto
the actual spectrum (in blue) is a straight line in red whose slope
in log-log scale is equivalent to 1/ f 2 , confirming the applicability of the brown fractal model described by Eq. (9.136). From
the intercept in Fig. 9-7(b), it was determined that
σz2 = 0.25.
Hence, Eq. (9.136) becomes
Ŝw ( f ) =
0.25 1
,
4π 2 | f |2
(1 < | f | < 400).
A. Stochastic Wiener Denoising
(9.134)
As noted earlier in the definition for the Wiener process w(t),
the random process z(t) is white Gaussian, so the power spectral
density of z(t) is constant: Sz ( f ) = σz2 . For an LTI system with
transfer function H( f ), the power spectral density Sw ( f ) of the
Wiener random process w(t) is related to the power spectral
density Sz ( f ) of the white random process z(t) by
Sw ( f ) = |H( f )|2 Sz ( f ),
Ŝw ( f ) = |W( f )|2 =
(9.133)
which leads to the system response function H( f ) as
H( f ) =
treat w(t) as a continuous-time signal, even though in reality it
is a discrete-time version of w(t). The power spectral density
Ŝw ( f ) was estimated using the periodogram
To illustrate the utility of the stochastic denoising Wiener filter
described in Section 9-4.4, we added random noise ν (t) to w(t)
to generate
(9.138)
y(t) = w(t) + ν (t).
The sampled noisy signal y[n], shown in Fig. 9-7(c), exhibits large fluctuations because the signal-to-noise ratio is
SNR = −0.21 dB. The negative sign is indicative that the noise
contains more energy than the original signal w(t). The corresponding signal-power to noise-power ratio is
(9.135)
Ps
= 10(−0.21)/10 = 0.95.
Pn
(9.136)
Despite the non-rigorous analysis leading to Eq. (9.136), the
result confirms that the power spectral density of a Wiener
process varies as 1/ f 2 .
The variance of ν (t) is σv2 = 0.01.
Sampling y(t) at a sampling interval of 1 ms (corresponding
to a sampling rate of 1000 samples/s) converts y(t) in Eq. (9.138)
into the discrete-time signal
y[n] = w[n] + ν [n].
9-8.4 Stochastic Wiener Filtering of Wiener
Processes
(9.139)
In the discrete-time frequency domain Ω, the power spectral
density corresponding to Eq. (9.136) is
The plot shown in Fig. 9-7(a) displays a realization of a Wiener
process w(t) extending over a duration of 1 s. For simplicity, we
www.dbooks.org
Sw (Ω) =
σz2 1
1
=
,
4π 2 Ω2 16π 2 Ω2
(9.140)
318
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
w(t)
y[n]
0.04
0.03
0.02
0.02
0.01
0
0
−0.02
−0.01
−0.02
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
t (s) −0.04 0
100
(a) Wiener process w(t)
200
300
400
500
600
700
800
900
1000
n
1000
n
(c) Wiener process w[n] plus noise ν[n]
Sw( f )
10−2
10−3
wˆ LS[n]
10−4
0.02
10−5
0.015
10−6
0.005
10−7
−0.005
0.01
0
−0.01
10−8
3
10
100
300
400
f (Hz)
−0.015
0
100
(b) Log-log plot of Sw( f )
200
300
400
500
600
700
800
900
(d) Denoised Wiener process using
stochastic Wiener filtering
wˆ D[n]
0.04
0.02
0
−0.02
−0.04
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
n
(e) Denoised Wiener process using
deterministicWiener filtering
Figure 9-7 (a) Wiener random process w(t), (b) spectrum Sw ( f ) of w(t) plotted on a log-log scale, (c) noisy signal y[n] = w[n] + ν [n], (d)
denoised estimate ŵLS [n] using stochastic filtering, and (e) deterministic estimate ŵD [n], which resembles the noisy Wiener process in (c).
where we used σz2 = 0.25. With subscript x in Eq. (9.67)
changed to subscript w, and σv2 = 0.01, use of Eq. (9.140) in
Eq. (9.67) gives
HSDN (Ω) =
=
Sw (Ω)
Sw (Ω) + σv2
1/(16π 2Ω2 )
1
,
=
1/(16π 2Ω2 ) + 0.01 1 + 1.58Ω2
(9.141)
9-8 1-D FRACTALS
ŴLS (Ω) = HSDN (Ω) Y(Ω),
319
(9.142)
and
ŵLS [n] = DTFT−1 { ŴLS (Ω) },
(9.143)
where Y(Ω) is the DTFT of y[n].
Application of the recipe outlined in Section 9-4.4 led to
the plot shown in Fig. 9-7(d). The estimated signal ŵLS [n]
bears very close resemblance to the true signal w(t) shown
in Fig. 9-7(a), demonstrating the capability of the stochastic
Wiener filter as a powerful tool for removing noise from noisy
signals.
B. Deterministic Wiener Denoising
In the deterministic approach to noise filtering, signal w(t) is
treated as a white random process with Sw ( f ) = σw2 . Consequently, the expression for the deterministic denoising filter
transfer function becomes
HDDN (Ω) =
σw2
Sw (Ω)
=
.
Sw (Ω) + σv2 σw2 + σv2
tion is equivalent to integration in continuous time or summation
in discrete time. That is, at a sampling rate of 1000 samples/s,
(9.144)
Consequently, since HDDN (Ω) is no longer a function of Ω,
implementation of the two steps in Eqs. (9.142) and (9.143)
leads to the deterministic estimate
σw2
y[n].
(9.145)
ŵD [n] =
σw2 + σv2
Clearly, ŵD [n] is just a scaled version of y[n], and therefore no
filtering is performed. The plot shown in Fig. 9-7(e) is identical
to that in Fig. 9-7(c) for y[n]. For display purposes, we chose to
set the quantity σw2 /(σw2 + σv2 ) = 1.
9-8.5 Stochastic Deconvolution of the Wiener
Process
To illustrate how the stochastic Wiener deconvolution filter of
Section 9-4.5 can by used to deconvolve a random process, we
selected the Wiener process w(t) shown in Fig. 9-8(a) and then
convolved it with a blurring filter characterized by the rectangle
impulse response
(
1000,
0 < n < 100,
hblur [n] =
(9.146)
0,
otherwise.
n
z[n] = 1000 ∑ w[n],
(9.147a)
Zt
(9.147b)
n−100
or equivalently,
z(t) = 1000
w(τ ) d τ .
t−0.1
To perform the summation (or integration), it is necessary to
append w[n] with zeros for −100 ≤ n < 0 (or equivalently,
−0.1 ≤ t < 0 for w(t)). The integrated signal z[n] in Fig. 9-8(b)
is a smoothed version of the original signal w[n].
Next, white Gaussian noise ν [n] was added to the system
output z[n] to produce noisy observation
y[n] = z[n] + ν [n] = hblur [n] ∗ w[n] + ν [n].
(9.148)
The noisy signal, plotted in Fig. 9-8(c), is only slightly different
from z[n] of Fig. 9-8(b), because the signal-to-noise ratio is
36.9 dB, corresponding to an average signal power of about
4900 times that of the noise.
The stochastic Wiener deconvolution method uses the filter
given by Eq. (9.71a), with subscript x changed to w:
WSDC (Ω) =
H∗blur (Ω) Sw (Ω)
.
|Hblur (Ω)|2 Sw (Ω) + σv2
(9.149)
Here, Hblur (Ω) is the DTFT of hblur [n] defined by Eq. (9.146),
Sw (Ω) is given by Eq. (9.140), and from knowledge of the noise
added to z[n], σv2 = 0.01. Once WSDC (Ω) has been computed,
w[n] can be estimated using the form of Eq. (9.73), namely
ŵS [n] = DTFT−1 [WSDC (Ω) Y(Ω)],
(9.150)
where Y(Ω) is the DTFT of the convolved noisy observation
y[n]. Implementation of the stochastic convolution process led
to the plot in Fig. 6-9(d). The mean-square-error between w[n]
and its reconstruction ŵS [n] over the interval 0 < n < 1000 is
2.25 × 10−6.
For the sake of comparison, in part (e) of Fig. 9-8 we
again show ŵD [n], the result of reconstructing w[n] using the
deterministic Wiener filter. Clearly, the deterministic approach
is not effective.
The convolved Wiener process z[n] = w[n] ∗ hblur [n] is displayed
in Fig. 9-8(b). The convolution process with the rectangle func-
www.dbooks.org
320
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
w[n]
y[n]
0.03
2
1.5
0.02
1
0.01
0.5
0
0
−0.5
−0.01
−0.02
−1
0
100
200
300
400
500
600
700
800
900
n
1000
−1.5
0
(a) Wiener process w[n]
100
200
300
400
500
600
700
800
900
1000
n
1000
n
(c) Convolved Wiener process with noise added
z[n]
wˆ SW[n]
2
0.03
1.5
0.02
1
0.01
0.5
0
0
−0.5
−0.01
−1
−1.5 0
100
200
300
400
500
600
700
800
900
1000
n −0.02 0
100
(b) Wiener process convolved with rect
function of duration 100 and amplitude 1000
200
300
400
500
600
700
800
900
(d) Deconvolved Wiener process using
stochastic Wiener deconvolution filter
wˆ DW
0.2
0.15
0.1
0.05
0
−0.05
−0.1
−0.15
0
100
200
300
400
500
600
700
800
900
1000
n
(e) Deconvolved Wiener process using
deterministic Wiener deconvolution filter
Figure 9-8 Wiener deconvolution: (a) original Wiener process w[n], (b) convolved signal z[n] = h[n] ∗ w[n], (c) y[n] = z[n] + ν [n],
where ν [n] is white Gaussian noise with S/N = 36.9 dB, (d) deconvolved reconstruction ŵSW [n] using a stochastic Wiener filter, and
(e) deconvolved reconstruction ŵDW [n] using a deterministic filter.
Why does the reciprocal power
form of the power spectral density of a fractal random
process need only hold over a finite range of frequencies,
as in Eq. (9.126)?
Concept Question 9-8:
9-9 2-D Fractals
The 1-D fractal concepts introduced in the preceding section
generalize readily to 2-D images, which we now demonstrate
through an example:
(1) Figure 9-9(a): 960 × 1280 image f [n, m] of a tree with a
fractal-like branching structure.
9-9 2-D FRACTALS
321
(a) Fractal tree image
(d) Noisy blurred image
log10(Sf (0,Ω2))
9
8
7
6
5
4
3
2
1
−2
−1.5
−1
−0.5
0
0.5
log(Ω2)
(b) Plot of log10(Sf (0,Ω2)) versus
log10(Ω2) for π/240 ≤ Ω2 ≤ π.
The average slope is −2.
(e) Reconstructed image using
a stochastic Wiener filter
(c) Blurred image
(f ) Reconstructed image using
a deterministic Wiener filter
Figure 9-9 2-D deconvolution example.
www.dbooks.org
322
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
(2) Figure 9-9(b): To ascertain the fractal character of the
tree, the 2-D power spectral density Sf (Ω1 , Ω2 ) was estimated
using
959 1279
Ŝf (Ω1 , Ω2 ) = ∑
∑ f [n, m] e− j(Ω1 n+Ω2m) ,
(9.151)
n=0 m=0
and then Ŝf (0, Ω2 ) was plotted in Fig. 9-9(b) on a log-log scale.
The average slope is −2, which means that Ŝf (0, Ω2 ) varies as
1/Ω22. Generalizing to 2-D,
Sf (Ω1 , Ω2 ) =
C
Ω21 + Ω22
.
(9.152)
(3) Figure 9-9(c): To simulate motion blur in the horizontal
direction, the image in Fig. 9-9(a) is convolved with a (1 × 151)
2-D PSF given by
(
1
for 0 ≤ n ≤ 150 and m = 1,
hblur [n, m] =
(9.153)
0
otherwise.
The resultant 960 × 1430 blurred image given by
z[n, m] = hblur [n, m] ∗ ∗ f [n, m]
(9.154)
is shown in Fig. 9-9(c)
(4) Figure 9-9(d): Addition of a slight amount of noise
ν [n, m] to z[n, m] gives
g[n, m] = z[n, m] + ν [n, m] = hblur [n, m] ∗ ∗ f [n, m] + ν [n, m].
(9.155)
The slightly noisy convolved image g[n, m] is shown in
Fig. 9-9(d).
(5) Figure 9-9(e): Generalizing Eq. (9.149) to 2-D gives
WSDC (Ω1 , Ω2 ) =
H∗blur (Ω1 , Ω2 ) Sf (Ω1 , Ω2 )
, (9.156)
|Hblur (Ω1 , Ω2 )|2 Sf (Ω1 , Ω2 ) + σv2
where Hblur (Ω1 , Ω2 ) is the 2-D DSFT of hblur [n, m] and σv2
is the noise variance. Application of the stochastic Wiener
reconstruction recipe in 2-D gives
fˆS [n, m] = DSFT−1 [WSDC (Ω1 , Ω2 ) G(Ω1 , Ω2 )],
ence, namely that instead of using Eq. (9.152) for Sf (Ω1 , Ω2 )
the expression used is Sf (Ω1 , Ω2 ) = σf2 . The absence of the
inverse frequency dependency leads to a fuzzier reconstruction
than realized by the stochastic deconvolution filter.
(9.157)
where G(Ω1 , Ω2 ) is the 2-D DSFT of the observed image
g[n, m]. The reconstructed image is shown in Fig. 9-9(e).
(6) Figure 9-9(f): Deterministic convolution involves the
same steps represented by Eqs. (9.156) and (9.157) for the
stochastic deconvolution process except for one single differ-
9-10 Markov Random Fields
Medical applications—such as ultrasound imaging, MRI, and
others—rely on the use of image processing tools to segment
the images produced by those imaging sensors into regions of
common features. The different regions may belong to different
organs or different types of tissue, and the goal of the segmentation process is to facilitate the interpretation of the information
contained in the images. Another name for segmentation is
classification: assigning each pixel to one of a set of predefined
classes on the basis of its own value as well as those of its
neighbors. Similar tools are used to segment an image captured
by a video camera, an infrared temperature sensor, or a weather
radar system.
An important ingredient of the image segmentation process
is a parameter estimation technique that models an image as a
Markov random field (MRF). The MRF model assigns each
image pixel f [n, m] a conditional probability density based, in
part, on the values of the pixels in its immediate neighborhood.
The purpose of the present section is to introduce the concept
and attributes of MRFs and to demonstrate their applications
through image examples.
9-10.1 1-D Markov Process
Before we delve into the 2-D case, let us first consider the
1-D case of a Markov random process x[n]. The value of
x[n] is continuous, but it is sampled in time in discrete steps,
generating the random vector { . . . , x[0], x[1], . . . , x[N], . . . }. In
Markov language, x[n] at time n is regarded as the present,
x[n + 1] is regarded as the future, and its values x[n − 1],
x[n − 2], . . . , x[0] are regarded as the past. The Markov model
assigns a conditional pdf to “the future value x[n + 1] based on
the present value x[n] but independent of past values,” which is
equivalent to the mathematical statement
p(x[n + 1]|{ x[0], x[1], . . ., x[n] }) = p(x[n + 1]|x[n]).
(9.158)
In 2-D, the “present value of x[n]” becomes the values of the
pixels in the neighborhood of pixel f [n, m], and “past values”
become pixels outside that neighborhood.
Using the Markov condition encapsulated by Eq. (9.158), and
9-10 MARKOV RANDOM FIELDS
323
after much algebra, it can be shown that
the classes pertain to different types of tissue, while in a video
image of terrain, the classes might be roads, cars, trees, etc. The
segmentation procedure involves several elements, the first of
which is the likelihood function of the estimation method used
to implement the segmentation.
Given the general image model
p(x[n]|{ . . . , x[N], x[N − 1], . . ., x[n + 1], x[n − 1], . . ., x[0], . . . })
p(x[n + 1]|x[n]) p(x[n]|x[n − 1])
=Z
, (9.159)
p(x′ [n + 1]|x′[n]) p(x′ [n]|x′ [n − 1]) dx′ [n]
which states that the conditional pdf of x[n], given all other
values { . . . , x[0], . . . , x[N], . . . }, is governed by the product of
two conditional pdfs, one relating x[n] to its immediate past
x[n−1], and another relating x[n] to its immediate future x[n+1].
An MRF generalizes this relationship from 1-D to 2-D using the
concepts of neighborhoods and cliques.
9-10.2 Neighborhoods and Cliques
The neighborhood of a pixel at location [n, m] is denoted∗
∆[n, m] and consists of a set of pixels surrounding pixel [n, m],
but excluding it. Figure 9-10 displays the pixel locations included in 3-, 8-, 15-neighbor systems. Neighborhoods may also
be defined in terms of cliques.
A clique is a set of locations such that any two members of
the clique adjoin each other, either horizontally, vertically, or
diagonally. Figure 9-11 shows 10 cliques, one of which is a
self-adjoining clique. A neighborhood can be represented as a
disjoint union of cliques of various types.
In a typical image, pixel neighbors are more likely to have
similar intensities than distant pixels. Different organs in a
medical image or different objects in an imaged scene tend to
have smooth features and the boundaries between them tend
to have sharp transitions across them. Image texture, defined
as the spatial variability within a given “homogeneous” region,
might exhibit different statistical variations in different types
of regions (classes). Image segmentation (classification) algorithms differ by the type of sensor used to produce the image,
the type of scene, and the intended application. However, most
of these algorithms share a common strategy, which we outline
in this and the next section. Part of the strategy is to define
the “relevant” neighborhood for a particular application, which
entails defining the combination of cliques comprising such a
neighborhood.
9-10.3 Likelihood Functions
The goal of image segmentation is to classify each pixel into
one of K predefined classes. In a medical ultrasound image,
g[n, m] = f [n, m] + ν [n, m],
(9.160)
where g[n, m] is the observed intensity of pixel [n, m], f [n, m]
is the noise-free intensity, and ν [n, m] is the additive noise, the
estimated value fˆ[n, m] is obtained from g[n, m] by maximizing
the likelihood function. The likelihood functions for the maximum likelihood estimation (MLE) method and the maximum a
posteriori probability (MAP) method are given by
MLE:
p(g[n, m]| f [n, m]),
(9.161a)
MAP:
p( f [n, m]|g[n, m]) =
p(g[n, m]| f [n, m]) p( f [n, m])
. (9.161b)
p(g[n, m])
A. MLE Estimate fˆMLE [n, m]
As shown earlier in Sections 9-2 and 9-3 for the 1-D case, the
MLE maximization process entails: (a) introducing an appropriate model for the conditional probability of Eq. (9.161a), (b)
taking the logarithm of the model expression, (c) computing the
derivative of the likelihood function with respect to the unknown
quantity f [n, m] and equating it to zero, and (d) solving for
f [n, m]. The resultant value of f [n, m] is labeled fˆMLE [n, m]. The
computed estimate is then used to assign pixel [n, m] to one of
the K classes, in accordance with a class-assignment algorithm
(introduced shortly).
B. MAP Estimate fˆMAP [n, m]
MAP requires models for two pdfs, namely the same conditional pdf used in MLE, p(g[n, m]| f [n, m]), as well as the
pdf p( f [n, m]). The third pdf, p(g[n, m]) in the denominator
of Eq. (9.161b), is not needed and may be set equal to a
constant. This is because it disappears in the maximization
procedure (taking the log, differentiating with respect to f [n, m],
and equating to zero).
The type of models commonly used to describe the two pdfs
in the numerator of Eq. (9.161b) are introduced shortly.
∗ For purposes of clarity, we use the symbol ∆, even though the Markov field
literature use ∂ .
www.dbooks.org
324
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
n
m
[n, m − 1]
[n − 1, m]
[n, m]
[n +1, m]
[n, m + 1]
[n − 1, m − 1]
[n, m − 1]
[n +1, m − 1]
[n − 1, m]
[n, m]
[n +1, m]
[n − 1, m + 1]
[n, m + 1]
[n +1, m + 1]
(a) 3 neighbors
(b) 8 neighbors
[n − 2, m − 2] [n − 1, m − 2]
[n, m − 2]
[n +1, m − 2] [n + 2, m − 2]
[n − 2, m − 1] [n − 1, m − 1]
[n, m − 1]
[n +1, m − 1] [n + 2, m − 1]
[n − 2, m]
[n − 1, m]
[n, m]
[n +1, m]
[n + 2, m]
[n − 2, m + 1] [n − 1, m + 1]
[n, m + 1]
[n +1, m + 1] [n + 2, m + 1]
[n − 2, m + 2] [n − 1, m + 2]
[n, m + 2]
[n +1, m + 2] [n + 2, m + 2]
(c) 15 neighbors
Figure 9-10 Examples of pixel neighborhoods.
9-10 MARKOV RANDOM FIELDS
0 0
* 0
(a) Self
clique
0 0
* *
(b) Horizontal
clique
0 *
* 0
0 *
* 0
(d) Left-diagonal (e) Right-diagonal
clique
clique
* *
0 *
(g) Inverted
L-clique
0 *
* *
(h) Reversed
L-clique
325
* 0
* 0
(c) Vertical
clique
* 0
* *
(f ) L-clique
The Gibbs distribution is a joint pdf for f [n, m] at all locations
[n, m], and it has the general form:
p({ f [n, m] }) =
* *
* *
( j) Square clique
Figure 9-11 Ten types of pixel cliques. A red star denotes a
location in a clique and a 0 denotes a location not in a clique.
9-10.4 pdf p( f [n, m])
The Hammersley-Clifford theorem is sometimes called “the
fundamental theorem of random fields.” It states that any conditional pdf for a set of random variables in a Markov random
field is always given by the Gibbs distribution, provided the
following conditions hold:
(1) Positivity: p( f [n, m]) > 0, which usually is true for
images. Note that positivity states that the pmf p( f [n, m]) of
f [n, m] is always positive, but f [n, m] itself may assume both
positive and negative values.
(2) Stationarity: p( f [n, m]) does not vary with position. That
is, the same statistics apply across the entire image.
(3) Locality: For a specified neighborhood ∆[n, m], f [n, m], is
a Markov random field that obeys the locality condition
p( f [n, m]|{ f [n′ , m′ ], n′ 6= n, m′ 6= m })
= p( f [n, m]| f∆ [n, m]),
(9.162)
where f∆ [n, m] are the values of pixels in the specified neighborhood ∆[n, m]. Thus, the pdf of f [n, m] at location [n, m] depends
on only the values of the pixels within the neighborhood ∆[n, m].
(9.163)
where { f [n, m]} is f [n, m] at all locations [n, m] and Z is the
partition function that normalizes the pdf so that it integrates to
unity, as every pdf should, and U[n, m] is the sum of the potential
energy over the cliques c:
U[n, m] =
* *
* 0
(i) Invertedreversed
L-clique
1 −U[n,m]
e
,
Z
∑
Uc [n′ , m′ ].
(9.164)
all cliques c
The summation extends over the cliques defining the specified
neighborhood ∆[n, m], and the model for the potential energy of
an individual clique depends on the segmentation algorithm.
The Ising model is a popular image-processing algorithm
used for identifying boundaries between different classes. The
original Ising model, which was developed for characterizing
the statistics of ±1 electron spin in ferromagnetic materials, has
been adapted to image segmentation by restricting the Markov
random field f [n, m] to a few discrete values. Consequently,
f [n, m] is described by a pmf, not a pdf. Each f [n, m] interacts
only with its specified neighbors, such as its 8 immediate
neighbors.
A particular form of the Ising model uses a binary assignment
in which f [n, m] can assume either a value of 0 or a value of 1,
and it interacts with its two horizontal neighbors and two vertical
neighbors (Fig. 9-10(a)). Thus,
∆[n, m] = { [n ± 1, m ± 1] }.
(9.165)
The potential energy of pixel [n, m] is defined as
U[n, m] = β s[n, m],
(9.166)
where β is a model parameter selected through experimentation (as discussed later in Section 9-11.3) and s[n, m] is a
dissimilarity index that accounts for how many of the four
pixels surrounding pixel [n, m] are different from pixel [n, m].
For example, s[n, m] = 0 for the center pixel in Fig. 9-12(a) and
s[n, m] = 4 for the center pixel in Fig. 9-12(b). The full range
of dissimilarity index s[n, m] is between 0 for a pixel surrounded
with like pixels and 4 for a pixel different from all of its four
neighbors.
To compute s[n, m] for each pixel in the image, we first have
to devise a scheme for assigning a value of 0 or 1 to each pixel.
As we shall see in a later section, such an assignment can be
realized using maximum likelihood segmentation. Consider, for
www.dbooks.org
326
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
0
1
0 0 0
1 1 1
0
1
s[n,m] = 0 s[n,m] = 0
(a) All 4 neighbors are the
same as center pixel
1
0
1 0 1
0 1 0
1
0
s[n,m] = 4 s[n,m] = 4
(b) All 4 neighbors are
different from center pixel
1
1 0 0
0
s[n,m] = 2
(c) Two neighbors are different from center pixel
Figure 9-12 Dissimilarity index s[n, m] is equal to the number
of immediate horizontal and vertical pixels that are different
from the pixel under consideration.
information about the neighborhood of that pixel, akin to how
our eye-brain system perceives the boundaries between the
squares in the image of Fig. 9-13(a).
By computing and then plotting the histogram of the observed
image g[n, m], as in Fig. 9-13(b), we can divide the range of
values of g[n, m] into two segments and assign them values of
0 and 1. [If the image is to be classified into more than two
classes, the approach can be extended accordingly, by assigning
discrete values commensurate with the distances between the
peaks in the distribution.] Using the 0/1 assignment, each pixel
in the original image is then assigned a value of 0 or 1. We call
this image a binary image.
Once the dissimilarity index s[n, m] has been determined for
each pixel, the Ising-Gibbs distribution given by Eq. (9.163)
becomes
1
p( f [n, m]) = e−β s[n,m] .
(9.167)
Z
The negative exponential model favors shorter boundaries
(smaller s[n, m]) over longer boundaries. Its role in image segmentation is discussed shortly.
example, the noisy two-class image shown in 9-13(a). Because
of the randomness associated with the pixel values, it is not
easy to assign individual pixels to their correct class. When
our eyes look at the image, however, we discern the boundaries
between the four squares, even though some of the pixels in the
predominantly dark squares are bright, and some of those in the
predominantly bright squares are dark. The goal of segmentation
is to assign each pixel to its correct class by incorporating
Concept Question 9-9: In general terms, what is a
Markov random field model of an image?
Concept Question 9-10:
How can a Markov random
field model be useful?
25
10
20
20
15
30
10
40
5
50
0
−1.5
−1
−0.5
0
0.5
1
1.5
2
2.5
3
60
10
20
30
40
(a) Noisy image
50
60
f1
f2
(b) Histogram of noisy image
Figure 9-13 Noisy image of four squares and associated histogram. The noise-free image had only positive values, but the addition of
random noise expands the range to negative values and to larger positive values.
9-11 APPLICATION OF MRF TO IMAGE SEGMENTATION
9-11 Application of MRF to Image
Segmentation
9-11.1 Image Histogram
Consider the 320 × 320 noisy image shown in Fig. 9-14(a). Our
goal is to segment the image into three classes: bone, other
tissue, and background. To that end, we start by generating
the histogram of the noisy image g[n, m]. The histogram displayed in Fig. 9-14(b) encompasses three clusters of pixels, a
group concentrated around f [n, m] = f1 = 0, another around
f [n, m] = f2 = 150, and a third around f [n, m] = f3 = 300.
The three clusters correspond to dark pixels in the background
part of the image, pixels of non-bone tissue, and pixels of the
bones in the five toes. In the present case, the values of f1 to
f3 were extracted from the image histogram, but in practice
more precise values are available from calibration experiments
performed with the imaging sensor.
9-11.2 MLE Segmentation
The likelihood function for a noisy image g[n, m], modeled as a
Gaussian with variance σ 2 , is given by
N N
1
2
2
e−(1/2σ )(g[n,m]− f [n,m]) .
p(g[n, m]| f [n, m]) = ∏ ∏ √
2
n=1 m=1 2πσ
(9.168)
Here, N = 320. Taking the natural log of both sides gives
ln(p(g[n, m]| f [n, m])) =
−
1 N N
N2
ln(2πσ 2 ) − 2 ∑ ∑ (g[n, m] − f [n, m])2.
2
2σ n=1 m=1
(9.169)
In the present segmentation, each pixel is assigned to one of
three classes:
Class 1: f1 = 0
Class 2: f2 = 150
Class 3: f3 = 300
That is, for each pixel [n, m] with pixel value g[n, m], f [n, m] in
Eq. (9.169) can assume only one of the three listed values. Because of the minus sign ahead of the second term in Eq. (9.169),
the log of the MLE likelihood function is maximized by minimizing the difference (g[n, m] − f [n, m])2 . Hence, for pixel
[n, m] with value g[n, m], the MLE likelihood is maximized
by assigning that pixel the value f1 , f2 , or f3 depending on
which one of them is closest to g[n, m]. Consequently, a pixel
327
with g[n, m] = 50 is closer to f1 = 0 than to f2 = 150, and
therefore it gets classified as class 1 (black). Similarly, a pixel
with g[n, m] = 100 is closest to f2 = 150, and therefore it gets
classified as class 2 (tissue). The MLE segmentation process
leads to the image shown in Fig. 9-14(c).
9-11.3 MAP Segmentation
Inserting Eqs. (9.167) and (9.168) into Eq. (9.161b) provides the
MAP likelihood function
p(g[n, m]| f [n, m]) p( f [n, m])
p( f [n, m]|g[n, m]) =
p(g[n, m])
)
(
N N
1
1 −β s[n,m]
−(1/2σ 2 )(g[n,m]− f [n,m])2
= ∏∏√
e
× e
2
Z
n=1 m=1 2πσ
1
×
,
(9.170)
p(g[n, m])
where β is a selectable (trial-and-error) parameter similar to the
Tikhonov parameter λ , and s[n, m] is the dissimilarity index of
pixel [n, m], computed from the MLE segmented image.
The natural log of the MAP likelihood function is
N2
2
ln(p( f [n, m]|g[n, m])) = − ln(2πσ ) − ln Z − lnC
2
(
)
N N
1
+ − 2 ∑ ∑ (g[n, m] − f [n, m])2 − β s[n, m] ,
2σ n=1 m=1
(9.171)
where we have replaced p(g[n, m]) with constant C because
p(g[n, m]) has no influence on the maximization process. In
Eq. (9.171), we have two groups of terms. The first group—
consisting of three terms—has no role in the maximization
process, whereas the second group does. Of the second group,
let us consider the terms associated with pixel [n, m] and let us
call their combination z[n, m]:
z[n, m] = −
1
((g[n, m] − f [n, m])2 − β s[n, m]).
2σ 2
(9.172)
We wish to assign pixel f [n, m] one of three values, namely
f1 = 0, f2 = 150, or f3 = 300. The assignment also classifies the
pixel as background, tissue, or bone. The assignment seeks to
maximize the likelihood function, which (because of the minus
signs) is accomplished by minimizing the value of z[n, m]. In the
absence of the second term in Eq. (9.172), the process would
collapse to the MLE segmentation of the previous subsection,
www.dbooks.org
328
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
f1 = 0
f2 = 150
f3 = 300
1400
1200
1000
800
600
400
200
0
0
100
200
(a) Noisy x-ray image
300
400
500
g[n,m]
(b) Histogram of noisy image
(c) MLE segmentation
(d) MAP segmentation with ICM algorithm
Figure 9-14 Three-class segmentation example.
but the presence of the term β s[n, m] introduces the degree of
similarity/dissimilarity of pixel [n, m] relative to its neighbors
into the segmentation decision.
For each pixel [n, m], g[n, m] is the observed value of that pixel
in the noisy image, β is a trial-and-error parameter, σ 2 is the
image variance (usually determined through calibration tests),
and s[n, m] is the dissimilarity index obtained from the MLE
segmentation image. By computing z[n, m] three times, once
with f [n, m] in Eq. (9.172) set equal to f1 , another with f [n, m]
set equal to f2 , and a third time with f [n, m] set equal to f3 ,
we obtain three values for z[n, m]. MAP segmentation selects
the smallest of the three absolute values of z[n, m], and assigns
that pixel to the corresponding class. The outcome is shown in
Fig. 9-14(d).
Computationally, a commonly used algorithm for realizing
the MAP segmentation is the Iterated Conditional Modes
(ICM) algorithm. The algorithm repeats the segmentation process iteratively until it reaches a defined threshold.
Example 9-3:
Four-Square Image
Apply the ICM algorithm with β = 300 to segment the 64 × 64
noisy binary image shown in Fig. 9-15(a). The noise level is
characterized by a variance σ 2 = 0.25.
Solution: We start by computing and displaying the image
histogram shown in Fig. 9-15(b). Based on the histogram, we
select f1 = 1.5 for the darker class and f2 = 2.5 for the brighter
class.
Next, we apply MLE segmentation. We assign each pixel
[n, m] the value f1 or f2 depending on which one of them is
PROBLEMS
329
25
10
20
20
15
30
10
40
5
50
0
−1.5
−1
−0.5
0
0.5
1
1.5
2
2.5
3
60
10
20
30
40
50
f1
60
f2
(b) p(g[n,m])
(a) Noisy image
10
10
10
20
20
20
30
30
30
40
40
40
50
50
50
60
60
60
10
20
30
40
50
(c) MLE segmentation
of noisy image
60
10
20
30
40
50
60
(d) ICM segmentation
after 1 iteration
10
20
30
40
50
60
(e) ICM segmentation
after 9 iterations
Figure 9-15 Example 9-2.
closest in value to the pixel value g[n, m]. The result is displayed
in Fig. 9-15(c).
Finally, we apply MAP segmentation using the ICM algorithm to obtain the image in Fig. 9-15(d). The segmentation
can be improved through iterative repetition of the process. The
first iteration generates an MLE image, computes s[n, m] for
each pixel, and then computes a MAP image. Each subsequent
iteration computes a new set of values for s[n, m] and a new
MAP image. The image in Fig. 9-15(e) is the result of 9
iterations. Except for a few misidentified pixels—mostly along
the boundaries of the four squares, the MAP segmentation
procedure provides very good discrimination between the white
and black squares.
What application of Markov
random fields was covered in Section 9-11?
Concept Question 9-11:
www.dbooks.org
330
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
Summary
Concepts
• The goal of estimation is to estimate an unknown x from
observation yobs of a random variable or process y using
a conditional pdf p(y|x) which comes from a model, and
possibly an a priori pdf p(x) for x.
• The maximum likelihood estimate x̂MLE(yobs ) is the
value of x that maximizes the likelihood function
p(yobs |x), or equivalently its logarithm.
• The maximum a posteriori estimate x̂MAP (yobs ) is the
value of x that maximizes the a posteriori pdf p(x|yobs ),
or equivalently its logarithm. p(x|yobs ) is computed from
likelihood function p(yobs |x) and a priori pdf p(x) using
Bayes’s rule (see below).
• The least-squares estimate x̂LS (yobs ) is the value of x that
minimizes the mean square error E[(x − x̂LS(y))2 ].
• The 2-D versions of these estimators are generalizations
of the 1-D ones.
• Using a white power spectral density makes the stochastic Wiener filter reduce to the deterministic Wiener filter.
• A fractal model of power spectral density greatly improves performance.
• A Markov random field (MRF) models each image pixel
as a random variable conditionally dependent on its
neighboring pixels.
• A MRF model and the ICM algorithm can be used to
segment an image.
Mathematical Formulae
Bayes’s rule
1-D Linear least-squares estimator
x̂LLSE [n] = h[n] ∗ yobs[n]
p(x)
p(x|y) = p(y|x)
p(y)
MAP log-likelihood
log p(x|yobs ) = log p(yobs |x) + log p(x) − log(yobs )
Least-squares estimate
R
x′ p(yobs |x′ ) p(x′ ) dx′
x̂LS (yobs ) = E[x | y = yobs ] = R
p(yobs |x′ ) p(x′ ) dx′
Gaussian least-squares estimate
x̂LS (yobs ) = x̄ + Kx,yK−1
y (yobs − ȳ)
Sample mean
x̂LS (yobs ) =
1 N
∑ yobs [n]
N n=1
Least-squares estimation orthogonality principle
E[(x − x̂LS (y))]yT = 0
Important Terms
1-D Deterministic Wiener deconvolution filter
H∗blur (Ω)
x̂(Ω) = yobs (Ω)
|Hblur (Ω)|2 + σv2
1-D Stochastic Wiener deconvolution filter
H∗blur (Ω) Sx (Ω)
x̂(Ω) = yobs (Ω)
|Hblur (Ω)|2 Sx (Ω) + λ 2
Fractal model
c
Sx ( f ) = a , where a = 1 or 2
|f|
Gibbs distribution of a Markov random field
1
p({ f [n, m]}) = e−U[n,m]
Z
Provide definitions or explain the meaning of the following terms:
Bayes’s rule
deterministic Wiener filter
fractal power spectral density
Gibbs distribution
ICM algorithm
least-squares
Markov random field
maximum likelihood
maximum a posteriori
sample mean
stochastic Wiener filter
PROBLEMS
331
PROBLEMS
Section 9-1: Estimation Methods
9.1 A coin with P[heads] = x is flipped N times. The results of
each flip are independent. We observe n0 = number of heads in
N flips.
(a) If x is an unknown constant, compute the MLE estimate
x̂MLE(n0 ).
(b) If x is a random variable with a priori pdf
(
10e−10x
for 0 ≤ x ≤ 1,
p(x) =
0
otherwise,
derive a quadratic equation for the MAP estimate
x̂MAP (n0 ). Neglect P[x > 1] = e−10 = 0.000045.
(c) If N = 92 and n0 = 20, compute x̂MAP (20).
9.2 A coin with P[heads] = x is flipped N times. The results
of flips are independent. We observe n0 = number of heads in N
flips.
(a) If x is a random variable with a priori pdf
1
2
1
1
p(x) = δ x −
+ δ x−
3
4
3
2
(b) If x is a random variable with a priori pdf
(
be−bx
for x > 0,
p(x) =
0
for x < 0,
where b is a known constant, compute the MAP estimate
x̂MAP (y0 ).
(c) Explain the behavior of x̂MAP (y0 ) when b → ∞ and when
b → 0.
9.5 An exponential random variable y has pdf
(
xe−xy
for y > 0,
p(y|x) =
0
for y < 0.
x is a random variable with a priori pdf
(
be−bx
for x > 0,
p(x) =
0
for x < 0,
where b is a known constant. Compute the least-squares estimate
x̂LS (y0 ).
Section 9-4: Least-Squares Estimation
so that P[x = 41 ] = 31 and P[x = 12 ] = 32 , compute an
expression for the least-squares estimator x̂LS (n0 ).
(b) If N = 3 and n0 = 2, compute the least-squares estimate
x̂LS (2).
9.6 y(t) is a zero-mean WSS Gaussian random process with
autocorrelation function Ry (τ ) = e−|τ | .
9.3 An exponential random variable y has pdf
(
xe−xy
for y > 0,
p(y|x) =
0
for y < 0.
(c) Compute the least-squares estimate
(a) Let y = {y(1), y(2), y(3)}. Determine the joint pdf p(y).
(b) Compute the least-squares estimate ŷ(3)LS (y(2) = 6).
ŷ(3)LS (y(2) = 6, y(1) = 4).
We observe five independent values y0 = {y1 , y2 , y3 , y4 , y5 } of
random variable y. x is an unknown constant. Compute the MLE
x̂MLE({y1 , y2 , y3 , y4 , y5 }) of x.
9.4 An exponential random variable y has pdf
(
xe−xy
for y > 0,
p(y|x) =
0
for y < 0.
9.7 x(t) is a zero-mean WSS white Gaussian random process
with autocorrelation Rx (τ ) = 4δ (τ ). x(t) is input into an LTI
system with impulse response
(
3e−2t
for t > 0,
h(t) =
0
for t < 0.
(a) Compute the autocorrelation Ry (τ ) of the output random
process y(t).
(a) If x is an unknown constant, compute the MLE estimate
x̂MLE(y0 ).
(b) Let y = {y(3), y(7), y(9)}. Determine the joint pdf p(y).
(c) Compute the least-squares estimate ŷ(7)LS (y(5) = 6).
www.dbooks.org
332
CHAPTER 9 STOCHASTIC DENOISING AND DECONVOLUTION
Section 9-6: 2-D Estimation Problems
(d) Compute the least-squares estimate
ŷ(7)LS (y(5) = 6, y(3) = 4).
9.8 x[n] is a zero-mean non-WSS random process with autocorrelation Rx [i, j] = min[i, j]. Let i > j > k. Show that
x̂[i]LS (x[ j], x[k]) = x̂[i]LS (x[ j]), so x[k] is irrelevant.
9.9 x[n] is an IID Gaussian random process with
x[n] ∼ N (m, s),
2
where m = E[x[n]] and s = σx[n]
. We are given observations
x0 = {x0 [1], x0 [2], . . . , x0 [N]}
of {x[1], x[2], . . . , x[N]}. The goal is to compute the MLE estimates m̂(x0 ) and ŝ(x0 ) of mean m and variance s.
Section 9-7: Spectral Estimation
9.10 Section 9-7 showed discrete-space fractal images have
power spectral densities
Sf (Ω1 , Ω2 ) =
0 < Ωmin < |Ω1 |,
C
(Ω21 + Ω22 )
,
|Ω2 | < Ωmax < π ,
Deblurring due to an out-of-focus camera can be modeled
crudely as 2-D convolution with a disk-shaped PSF
(
1
for m2 + n2 < R2 ,
h[n, m] =
0
for m2 + n2 > R2 ,
for some radius of R pixels. The program srefocus.m convolves with h[n, m] the image f [n, m] in the file ????.mat,
adds zero-mean white Gaussian noise with variance σv2 to the
blurred image, and deconvolves the blurred image using each of
the following two Wiener filters:
• the deterministic Wiener filter uses power spectral density
Sf (Ω1 , Ω2 ) = C;
• the stochastic Wiener filter uses power spectral density
Sf (Ω1 , Ω2 ) = C/(Ω21 + Ω22).
Both filters depend only on the reciprocal signal-to-noise-type
ratio σv2 /C. In practice, neither C nor σv2 is known, so different
values of σv2 /C would be tried. Use σv2 /C = 1 here. The program
srefocus.m will be used in the next eight problems. [Here
“????” refers to the .mat files used in Problems 9.11 through
9.18.]
9.11 Edit program srefocus.m to deconvolve the image in
clown.mat. Use σv2 /C = 1.
for some Ωmin and Ωmax . This problem suggests many realworld images also have such power spectral densities.
The following program uses a periodogram to estimate the
power spectral density of the image in ????.mat and fits a
line to the log-log plot of Sf (Ω1 , 0) versus Ω1 for Ωmin = 0.1π
and Ωmax = 0.9π . [Here “????” refers to the .mat files listed
below.]
9.12 Edit program srefocus.m to deconvolve the image in
letters.mat. σv2 /C = 1.
clear;load ???.mat;M=size(X,1);
K=round(M/20);
FX=abs(fft2(X))/M;FX=log10(FX.*FX);
Omega1=log10(2*pi*[K:M/2-K]/M);
P=polyfit(Omega1,FX(1,K:M/2-K),1);
Y=polyval(P,Omega1);
subplot(211),plot(Omega1,FX(1,K:M/2-K),
Omega1,Y,’r’),axis tight,grid on
9.15 Edit program srefocus.m so it merely denoises the
noisy clown image. Use σv2 /C = 1 and h[n, m] = δ [n, m] (to
make a denoising, not a deconvolution problem) and
Run this program for the images contained in the following .mat files: (a) clown.mat; (b) letters.mat; (c)
sar.mat; (d) mri.mat. What do these plots suggest about
the power spectral densities of the images?
9.13 Edit program srefocus.m to deconvolve the image in
xray.mat. Use σv2 /C = 1.
9.14 Edit program srefocus.m to deconvolve the image in
mri.mat. Use σv2 /C = 1. Change the power spectral density to
Sf (Ω1 , Ω2 ) = C/((Ω21 + Ω22 ))2 (uncomment a line).
• the deterministic Wiener filter uses the power spectral
density Sf (Ω1 , Ω2 ) = C;
• the stochastic Wiener filter uses the power spectral density
Sf (Ω1 , Ω2 ) = C/(Ω21 + Ω22).
9.16 Edit program srefocus.m so it merely denoises the
noisy letters image. Use σv2 /C = 1 and h[n, m] = δ [n, m] (to
make a denoising, not a deconvolution problem) and
PROBLEMS
333
• the deterministic Wiener filter uses the power spectral
density Sf (Ω1 , Ω2 ) = C;
• the stochastic Wiener filter uses the power spectral density
Sf (Ω1 , Ω2 ) = C/(Ω21 + Ω22).
9.17 Edit program srefocus.m so it merely denoises the
noisy XRAY image. Use σv2 /C = 1 and h[n, m] = δ [n, m] (to
make a denoising, not a deconvolution problem) and
• the deterministic Wiener filter uses power spectral density
Sf (Ω1 , Ω2 ) = C;
• the stochastic Wiener filter uses power spectral density
Sf (Ω1 , Ω2 ) = C/(Ω21 + Ω22).
9.18 Edit program srefocus.m so it merely denoises the
noisy MRI image. Use σv2 /C = 1 and h[n, m] = δ [n, m] (to make
a denoising, not a deconvolution problem) and
• the deterministic Wiener filter uses the power spectral
density Sf (Ω1 , Ω2 ) = C;
• the stochastic Wiener filter uses the power spectral density
Sf (Ω1 , Ω2 ) =
C
.
(Ω21 + Ω22)2
Section 9-10: Markov Random Fields
9.19 Download the image in field.mat and the program
icm.m.
(a) Display the image using
imagesc(X),colormap(gray).
(b) Display its histogram for 200 bins using
hist(X(:),200).
Eight pixel values occur more often than nearby values.
Identify them. Hint: They are in arithmetic progression.
(c) Change the line f=[] in icm.m to contain these values.
(d) Run icm.m. It uses 9 iterations, σ = 30, and β = 1.
(Varying these values seems to make little difference to the
final result.) Display the maximum-likelihood segmented
and ICM-algorithm segmented images.
www.dbooks.org
Chapter
10
10 Color Image Processing
Contents
10-1
10-2
10-3
10-4
Overview, 335
Color Systems, 335
Histogram Equalization and Edge Detection, 340
Color-Image Deblurring, 343
Denoising Color Images, 346
Problems, 351
(a) Blurred Christmas tree
6
4
Objectives
2
0
-2
Learn to:
■ Transform between the RGB and YIQ color systems.
■ Apply image enhancement (histogram equalization
and edge detection) to color images.
-4
-6
-8
-10
-12
-14
0
20
40
60
80
k1
(MATLAB)
100
(b) 1-D spectrum log(|GR[k1,0] + 1)
■ Apply image restoration (deblurring and denoising) to
color images.
(c) Reconstructed Christmas tree
This chapter extends the image processing techniques developed in previous chapters for grayscale
images to color images, which can be represented in
either the RGB color system (suitable for image
deblurring) or the YIQ color system (suitable for
enhancement, so that colors are not distorted).
Examples illustrate all of this.
Overview
So far in this book, we have focused on the processing of
grayscale (black-and-white) images, where the intensity of
the image is described by a single 2-D function f (x, y) of
continuous spatial variables (x, y) or f [n, m] of discrete spatial
variables [n, m]. We considered several types of image processing applications including image denoising, deblurring, and
segmentation. Now in this chapter, we consider color images
and how to process them. It is important to note that by “color
images” we mean true-color images, in which different colors
are actually present in the image, not false-color images, in
which the grayscale intensity of a black-and-white image is
divided into ranges and then assigned to different colors. We use
discrete-space images f [n, m], although our treatment is equally
applicable to continuous-space images f (x, y).
A color image consists of three channels or components,
which taken together contain the information present in the
image. The actual content of the three channels depends on
the color system used for the image. For example the RGB
color system represents a color f [n, m] image as a triplet of red
fR [n, m], green fG [n, m], and blue fB [n, m] images:
f [n, m] = { fR [n, m], fG [n, m], fB [n, m]}.
10-1 Color Systems
A color system is a representation of a color image using three
channels or components. This section presents an overview of
the three color systems most commonly used in displays and
color printers.
(b) Red color fR[n,m]
(c) Green color fG[n,m]
(d) Blue color fB[n,m]
Figure 10-1 The three-color checkerboard image comprises
three single-color images, each comprising one-fourth of the
total pixels of the checkerboard (because one-fourth of the
pixels are black). In (b), fR [n, m] is non-zero in only the nonblack pixels, and a similar arrangement applies to fG [n, m] and
fB [n, m].
(10.1)
It is important to note that it is not true that f [n, m] =
fR [n, m] + fG [n, m] + fB [n, m]—the three components are akin
to elements of a vector. Instead, the color image consists of a
simultaneous display of the 2-D functions fR [n, m], fG [n, m] and
fB [n, m], as shown in Fig. 10-1. The color generation schemes
used in displays and printers are discussed in Section 10-1.
Other color systems, specifically CMYK (cyan, magenta,
yellow, black) and YIQ (luminance, in-phase, quadrature), are
presented in Section 10-1. Image enhancement (histogram
equalization and edge detection) of color images is presented in
Section 10-2, which uses YIQ, because use of RGB can result
in distortion of the colors of f [n, m]. Deconvolution (deblurring)
of color images is presented in Section 10-3, and denoising of
color images is covered in Section 10-4. Both use RGB.
(a) Color checkerboard image
10-1.1 RGB (Additive) Color System
A. Acquiring Color Images
Color images are acquired using three sensors: red, green,
and blue. Film images in color photography were acquired by
splitting the image into three copies, and then filtering them
using red, green, and blue filters to record red, green, and
blue components of the image. TV images when TV color
cameras were used were also acquired by splitting the image
into three copies, and then filtering them using red, green, and
blue filters before inputting them into three separate sensors
(Fig. 10-2). More recently, three sets of charge-coupled device
(CCD) sensors with red, green, and blue filters in front of them
are used to acquire the red, green, and blue components of an
image.
Hence, image processing of color images consists of processing three RGB components. Deconvolution of color images requires deconvolution of each RGB component separately, using
335
www.dbooks.org
336
CHAPTER 10 COLOR IMAGE PROCESSING
Filters
Tubes
Sync
pulses in
Lens
Encoder
Video
signal
combiner
To UHF
transmitter
Chrominance signal
Mirror
Luminance
matrix
Semi-mirror
Luminance signal
Figure 10-2 Color TV camera with three image orthican tubes.
identical procedures (2-D DFT and Wiener filter) for each RGB
component. Image enhancement, in particular edge detection
and histogram equalization, is applied to the YIQ representation,
rather than the RGB representation (the two representations are
related by the linear transformation Eq. (10.4), in order to ensure
that different colors are not enhanced differently. Denoising can
use either RGB or YIQ representation. All of these tools are
illustrated in the remainder of this chapter.
To acquire a color image, most digital cameras and scanners
acquire red, green, and blue images separately. Scanners include
a (M × 3) array of CCD sensors with an (M × 3) array of color
filters in front of them, where M is the width of the scanner in
pixels. One row of filters is red, another row of filters is green,
and another row of filters is blue (see Fig. 1-6). The image is
scanned line by line, and the mth line generates fR [n, m], fG [n, m]
and fB [n, m].
B. Displaying Color Images
As noted above, color photography was developed by filtering
the lens image using red, green, and blue filters, and then developing in a photo lab three separate images in these colors. Color
television with cathode-ray tubes (CRTs) uses three electron
guns to “paint” a phosphor screen with red, green, and blue
images. A mask behind the screen separates the outputs of the
guns, so that the “red” gun illuminates phosphors that glow red
when irradiated with electrons, and similarly for green and blue.
The phosphors are in groups of three, called pixels, one for each
color.
Color televisions and computer monitors with LED displays
use three sets of LEDs to depict the red, green, and blue images.
Each pixel in the display consists of three closely spaced LEDs,
one for each color.
C. RGB
The RGB color system represents color images using
Eq. (10.1). It is called an additive color system because
non-RGB colors are created by taking a linear combination of
{ fR [n, m], fG [n, m], fB [n, m]}. The additive process is illustrated
by the color cube shown in Fig. 10-3. The coordinates are
the weights applied to { fR [n, m], fG [n, m], fB [n, m]} to produce
the color shown in the cube. White, which has coordinates
(1,1,1), is obtained by combining red, green, and blue in equal
strengths. The non-RGB colors are discussed in Subsection
10-1.2 where we discuss the CMYK color scheme.
Since TV screens and monitors are dark when they are not
displaying an image, it is natural to create color images on
them by displaying weighted sums (i.e., linear combinations) of
{ fR [n, m], fG [n, m], fB [n, m]}. The lighting used in a dark theater
for a play is another example of an additive color system.
Web browsers display web pages using an additive
10-1 COLOR SYSTEMS
337
Red
Magenta
Yellow
Blue
Green
Cyan
Figure 10-3 Color cube: various colors are linear combinations of red, green, and blue.
color scheme. The HTML command <body bgcolor=
‘‘#RrGgBb’’> changes the color of the background of a
web page to a linear combination of red, green, and blue, where
the weights are the hexadecimal (base-16) numbers (Rr)16 for
red, (Gg)16 for green, and (Bb)16 for blue, with ( f )16 = 15. So
<body bgcolor=‘‘#ffff00’’> changes the background
color to yellow (red + green), since (Rr)16 = (Gg)16 = ( f f )16
turns the red and green “lights” to their maximum values
( f f )16 = 255. Each hexadecimal weight has 162 = 256 values
so the number of possible colors is (256)3 = 16, 777, 216.
Still, not every conceivable color can be represented as a linear
combination of the colors in { fR [n, m], fG [n, m], fB [n, m]}.
So why is it that systems other than RGB are used?
10-1.2 CMYK (Subtractive) Color System
The C (cyan), M (magenta), Y (yellow), K (black) color system
is used for printing color images on a printer. A CMYK color
system was used to print drafts of this book, for example, on a
color-laser printer. The toner for laser printers and the inks for
color inkjet printers are in CMYK colors.
The colors cyan, magenta, and yellow are generated from red,
green, and blue, as shown in Fig. 10-4. Yellow is the sum of red
and green, or equivalently, yellow is white minus blue.
The CMY colors are also specified as the off-axis corners of the color cube of Fig. 10-3. In fact, on the color
cube, cyan, magenta, and yellow are the complementary colors of red, green, and blue, respectively. For example, cyan
Figure 10-4 Relation of CMY colors to RGB colors.
has the coordinates (0, 1, 1) on the color cube and red has
coordinates (1, 0, 0). The sum of all three additive colors is
(1, 0, 0) + (0, 1, 0) + (0, 0, 1) = (1, 1, 1), which is the coordinate
for white. Similarly, yellow has coordinates (1, 1, 0) and blue
has coordinates (0, 0, 1). This is why the CMYK color system
is called subtractive—it subtracts colors from white, instead of
adding them to black the way RGB does.
10-1.3 Color Printing
A. RGB Printers
Color printers do not use the RGB color system to print text
and images, but let us pretend for the time being that they do.
When the image of a perfectly red tomato is printed on white
paper using an RGB inkjet printer, the printer deposits red ink
onto the paper. A similar process occurs with the cartridge of
a laser printer except that the material deposited into the paper
is a fine red powder and its extent across the paper is defined
by a magnetically charged toner. The printed tomato appears
red because when illuminated by white light, the green and
blue components of the white light are absorbed by the red ink
or red powder, leaving only the red component of the incident
light to be reflected by the printed image of the tomato. Hence,
the perfectly red tomato appears red. However, if the tomato is
yellow in color, which is equivalent to the sum of red and green
in the RGB system (Fig. 10-4), an RGB inkjet printer would
www.dbooks.org
338
CHAPTER 10 COLOR IMAGE PROCESSING
have to deposit both red and green ink onto the paper. When
illuminated by white light, the red ink will absorb the green and
blue components of the light and the green ink will absorb the
red and blue components, leaving no light to be reflected by the
printed page. Hence, the yellow tomato will appear black instead
of yellow. Because the ink used in inkjet printers and the powder
used in laser toners operate by subtracting their colors from
white, they are more compatible with the subtractive CMYK
color system than with the additive RGB system.
B. CMYK Printers
In the subtractive CMYK color system (Fig. 10-5), red can be
created by a mixture of yellow and magenta inks, green by
a mixture of yellow and cyan inks, and so on. Using CMY
ink colors solves the problem we encountered earlier with the
yellow tomato. We can now print many colors, whether pure or
not. Obviously, a yellow tomato can be printed using yellow ink,
but what about a perfectly red tomato? Since yellow consists of
red and green and magenta consists of red and blue, we can use
the combination of yellow and magenta inks to print it since both
colors include red. When illuminated by white light, the yellow
ink absorbs blue and the magenta ink absorbs green, leaving
only the red component of the white light to be reflected.
In CMYK, a color image is depicted as:
f [n, m] = { fC [n, m], fM [n, m], fY [n, m], fK [n, m]}.
(10.2)
Black (K) is included in CMYK because while Fig. 10-4
shows that cyan + magenta + yellow = black, such an addition
of inks requires much ink to produce black, which is a common
color in images. Hence, black is included as a separate color in
printing. Because CMYK consists of three subtractive colors—
yellow, cyan, and magenta—and black, it is often referred to as
a four-color printing process. In CMYK, silver is represented
using the weights {0, 0, 0, 0.25}, and gold by {0, 0, 0.88, 0.15}.
The exact relation between RGB and CMYK color systems is
more complicated than we have presented, since the inks used
in printing are not exactly cyan-, magenta-, and yellow-colored.
We should note that CMYK is not used in image processing,
only for image printing.
10-1.4 YIQ Color System
The Y (luminance), I (In phase), Q (Quadrature) color system
was developed for television by the NTSC (National Television
System Committee) in 1953. It differs from the RGB and
CYMK color systems in that intensity is decoupled from color.
The reason for decoupling intensity from color was so that images transmitted in color could easily be displayed on black-andwhite televisions, which were still in common use throughout
the 1950s. This feature also makes the YIQ color system useful
for histogram equalization and edge detection because image
intensity (luminance) can be varied without distorting colors, as
we show later in Section 10-2.
The YIQ depiction of a color image f [n, m] is given by
f [n, m] = { fY [n, m], fI [n, m], fQ [n, m]}.
Cyan
Green
Blue
Yellow
Magenta
Red
Figure 10-5 Relation of RGB colors to CMY colors.
(10.3)
Even though the symbol fY [n, m] is used in both the YIQ and
CMYK color systems, this is the standard notation used in color
schemes. The meaning of fY [n, m] (yellow or luminance) should
be obvious from the context.
The luminance component fY [n, m] is the intensity of
f [n, m] at location [n, m]. The chrominance (color) components
{ fI [n, m], fQ [n, m]} depict the color of f [n, m] at location [n, m]
using Fig. 10-6.
The horizontal axis (blue to orange) ranges over the colors
to which the eye is most sensitive. The vertical axis (purple
to green) ranges over colors to which the eye is less sensitive.
Accordingly, for analog transmission of television signals, less
bandwidth is needed to transmit the quadrature-modulated signal fQ [n, m] than the in-phase (un-modulated) signal fI [n, m],
with most of the transmission bandwidth reserved for the luminance signal fY [n, m].
The chrominance components { fI [n, m], fQ [n, m]} can be regarded as rectangular color coordinates. In polar coordinates,
10-1 COLOR SYSTEMS
339
image. The formula for this is
fQ[n,m]
fgray [n, m] = 0.299 fR [n, m] + 0.587 fG[n, m] + 0.114 fB[n, m].
(10.5)
The luminance image fY [n, m] is the image displayed on blackand-white televisions and monitors.
B. Significance of the YIQ Color System
Of course, black-and-white monitors, let alone televisions, are
rare today. The significance of the YIQ color system is that
it decouples color from intensity, so we can lighten an image
by applying histogram equalization to fY [n, m], while leaving
fI [n, m] and fQ [n, m] unaltered. Applying histogram equalization
to each of fR [n, m], fG [n, m] and fB [n, m] can alter the colors of
the image (see Section 10-2). There is also the computational
savings of applying image enhancement to a single image
instead of to three images. Similar comments apply to edge
detection in color images.
fI[n,m]
Figure 10-6 YIQ chrominance components.
10-1.5 Other Color Systems
these would be hue (tint) for the angular coordinate and saturation (vividness) for the radial coordinate. These are used in the
HSI (Hue, Saturation, Intensity) color system noted below.
A. Relating the RGB and YIQ Color Systems
The RGB (Eq. (10.1)) and YIQ (Eq. (10.3)) color system
representations of a color image f [n, m] are related by the linear
transformation

 


fY [n, m]
0.299 0.587
0.114
fR [n, m]
 fI [n, m]  = 0.596 −0.275 −0.321  fG [n, m] .
fQ [n, m]
0.212 −0.523 0.311
fB [n, m]
(10.4)
Note that the first row of this matrix sums to one, and each of
the second and third rows sums to zero. This ensures that: (a)
fY [n, m] ≤ 1 if fR [n, m] ≤ 1, fG [n, m] ≤ 1, and fB [n, m] ≤ 1,
and (b) the two color components are zero if all three colors
are present in equal strengths, corresponding to a white image at
location [n, m], in which case fY [n, m] alone suffices to represent
the image at [n, m].
The first row of Eq. (10.4) is a formula for transforming the
color image f [n, m] depicted using Eq. (10.1) to a grayscale
There are many other color systems, such as HSI (Hue, Saturation, Intensity) and HSV (Hue, Saturation, Value). For example,
the HSI color system is related to the RGB system using
1
fI [n, m] = ( fR [n, m] + fG [n, m] + fB [n, m]),
(10.6a)
3
3 min[ fR [n, m], fG [n, m], fB [n, m]]
fS [n, m] = 1 −
,
(10.6b)
fR [n, m] + fG [n, m] + fB [n, m]
"
#
1
2 (( f R − f G ) + ( f R − f B ))
fH [n, m] = cos−1
. (10.6c)
( fR − fG )2 + ( fR − fB )( fG − fB )
We do not discuss or use these color schemes in this book,
nor do we discuss color science, which includes topics such as
photometry (image color spectra) and colorimetry (color matching). These topics cover issues such as nonlinear response of
sensors (including the eye), perception of colors by the eye and
brain (psychophysics of color), imperfect ink colors for printing,
and nonlinear response of inkjet printers (which print dots
of different colors, just as monitors and televisions represent
color images using clusters of dots of different colors). Gamma
correction is an example of a nonlinear correction that arises
in this context. For a good treatment of color science in image
processing, see H. J. Trussel and M. J. Vrhel, Fundamentals of
Digital Imaging, Cambridge University Press, 2008.
www.dbooks.org
340
CHAPTER 10 COLOR IMAGE PROCESSING
◮ In this book we will transform from the RGB color
system to the YIQ color system using Eq. (10.4), then
perform image enhancement using histogram equalization
on fY [n, m], and convert the result back to the RGB color
system using the inverse of Eq. (10.4). ◭
The result can then be converted to another color system, such
as HSI, if desired.
Answer: From Eq. (10.4), fY [n, m] = 1 and
fI [n, m] = fQ [n, m] = 0
for all [n, m], as discussed below Eq. (10.4).
10-2 Histogram Equalization and Edge
Detection
10-2.1 Histogram Equalization
10-1.6 Reading Images to MATLAB
The MATLAB command imread is used to read images into
3-D MATLAB arrays. For example, A=imread(’f.jpg’);
maps the M × N JPEG image f.jpg into the M × N × 3 3-D
MATLAB array A, in which A(:,:,1) is the M × N array
of fR [m, n], A(:,:,2) is the M × N array of fG [m, n], and
A(:,:,3) is the M × N array of fB [m, n]. For the checkerboard
image in Fig. 10-1(a), A(:,:,1) is shown in Fig. 10-1(b),
A(:,:,2) in Fig. 10-1(c), and A(:,:,3) in Fig. 10-1(d).
Many types of images, such as .jpeg, .tiff and .bmp can be
handled using imread. A will be in uint8 format, which
should be converted into double-precision floating-point using
A=double(A);. The values A[i,j] must be scaled to satisfy
0 ≤ A[i, j] ≤ 1, using Eq. (5.1).
imagesc(A) depicts the M × N × 3 3-D MATLAB array A
as a color image, as in Fig. 10-1.
Concept Question 10-1: Why do we use the YIQ color
system in image processing? Why not just use the RGB
color system exclusively?
Exercise 10-1: What should the output of
imagesc(ones(3,3,3)) be?
Answer: A 3 × 3 all-white image (the combination of 3 × 3
red, green, and blue images).
Exercise 10-2: An image has
fR [n, m] = fG [n, m] = fB [n, m] = 1
for all [n, m]. What is its YIQ color system representation?
Histogram equalization (HE) was covered in Chapter 5. It alters
the pixel values of an image f [n, m] so that they are evenly
distributed over the entire range of the output display, thereby
brightening a dim image. In principle, histogram equalization
can be applied to each of the three RGB components of the
color image represented by Eq. (10.1). The drawback of such an
approach is that since the colors of the image are mixtures of the
red, green, and blue components of the image, disproportionate
altering of the pixel values will alter the colors of the image. The
image will appear brighter, but the colors may get distorted.
To avoid the color distortion, the RGB components can
be mapped to YIQ components using Eq. (10.4), and then
histogram equalization can be applied to only the luminance
(brightness) component fY [n, m]. The process brightens the
image but leaves the colors unaltered because the color of each
pixel is determined by the two chrominance components fI [n, m]
and fQ [n, m], whose values are not altered. An inverse form of
Eq. (10.4) is then used to transform the YIQ representation back
to the RGB representation. This approach also has the computational advantage in that histogram equalization is applied
to a single component, fY [n, m], instead of to all three RGB
components.
To compare the result of histogram equalization when applied
(a) directly to an RGB image with (b) the result obtained
by converting the RGB image to a YIQ image, followed by
histogram equalization applied to the Y component and then
converting the YIQ image back to RGB, we consider the dark
checkerboard RGB image shown in Fig. 10-7(a).
Application of the histogram equalization method to the three
RGB channels separately leads to the image in Fig. 10-7(b),
whereas the application of histogram equalization to the Y
channel leads to the image in Fig. 10-7(c). Both approaches
generate brighter versions of the checkerboard image, but only
the Y histogram equalization preserves the true colors of the
original image.
10-2 HISTOGRAM EQUALIZATION AND EDGE DETECTION
341
Example 10-1:
Flag Image in RGB and
YIQ Equalization
Use the image in Fig. 10-8(a) to compare the results of two
histogram equalization experiments, one performed using RGB
and another using YIQ.
Solution: Application of histogram equalization to each of the
three RGB images separately, and then followed by combining
the equalized images into a single color image results in the
brighter, but color-distorted, image shown in Fig. 10-8(b). In
contrast, application of histogram equalization to the Y channel
of the YIQ image (Fig. 10-8(c) preserves the color balance of
the original image. Note the color of the field of stars.
(a) Original, dark checkerboard image
10-2.2 Edge Detection in Color Images
The subject of edge detection as applied to grayscale images
was covered in Section 5-4.2, wherein we demonstrated how the
Sobel edge detector is used to convolve the grayscale image
f [n, m] with two PSFs, hH [n, m] and hV [n, m], to obtain the
horizontal edge image dH [n, m] and the vertical edge image
dV [n, m]:
and
(b) Histogram equalization using RGB
dH [n, m] = f [n, m] ∗ ∗hH [n, m]
(10.7a)
dV [n, m] = f [n, m] ∗ ∗hV [n, m].
(10.7b)
The two edge images are then combined to generate the gradient
image g[n, m]:
q
(10.7c)
g[n, m] = dH [n, m]2 + dV [n, m]2 .
The Sobel edge-detector algorithm identifies a pixel [n, m] as an
edge if g[n, m] exceeds a pre-specified threshold ∆. Accordingly,
the edge image z[n, m] is given by
(
1
for g[n, m] > ∆,
z[n, m] =
(10.8)
0
for g[n, m] < ∆.
(c) Histogram equalization using YIQ
Figure 10-7 Application of histogram equalization to the dark
checkerboard image in (a) results in the image in (b) when the
equalization is applied to all three RGB channels and to the
image in (c) when applied to the Y channel of the YIQ image.
The value of ∆ is application-dependent.
For color images, we can apply edge detection to each of
fR [n, m], fG [n, m] and fB [n, m] in an RGB image, or just to
fY [n, m] in its YIQ equivalent. Aside from the computational
savings of detecting edges in only one image instead of three,
another reason for using just fY [n, m] is illustrated by the
following example.
www.dbooks.org
342
CHAPTER 10 COLOR IMAGE PROCESSING
Edge detection of checkerboard image
The (24 × 24) checkerboard image in Fig. 10-9(a) is in RGB.
Application of the Sobel edge detector algorithm with threshold
∆ = 0.1 to each color channel separately leads to the images
in parts (b) through (d) of the figure. The three colors generate
different edge images. In contrast, Fig. 10-9(e) displays the edge
image using the YIQ image, which correctly identifies all edges
regardless of the pixel colors.
(a) Original dark flag image
(a) Original checkerboard
RGB image
(b) Red color edge
detection
(c) Green color edge
detection
(d) Blue color edge
detection
(b) After histogram equalization of RGB image
(c) After histogram equalization of YIQ image
and conversion back to RGB format
(e) Edge detection using YIQ
Figure 10-8 The application of histogram equalization to
each of the three channels of the RGB image results in color
distortion, but that is not the case when applied to a YIQ image.
Figure 10-9 Sobel edge detection: (a) original color image,
(b)–(d): edge detection of each RGB color channel separately,
and (e) edge detection using Y channel of YIQ image.
10-3 COLOR-IMAGE DEBLURRING
343
Concept Question 10-2: Why does applying histogram
equalization using the RGB color system alter colors?
Exercise 10-3: Should gamma transformation be applied to
images using the RGB or YIQ format?
Answer: YIQ format, because gamma transformation is
nonlinear; applying it to an RGB image would distort its
colors.
Exercise 10-4: Should linear transformation be applied to
images using the RGB or YIQ color system?
RGB, since this is the color system used to
display the transformed image. Applying linear transformation to YIQ images may result in gR [n, m] ≥ gmax ,
gG [n, m] ≥ gmax , or gB [n, m] ≥ gmax in Eq. (5.1).
Answer:
10-3 Color-Image Deblurring
When a color scene/image is blurred as a result of convolution
with the sensor’s PSF h[n, m], each of its RGB components is
blurred. The blurred image consists of the three components
gR [n, m] = fR [n, m] ∗ ∗h[n, m],
gG [n, m] = fG [n, m] ∗ ∗h[n, m],
gB [n, m] = fB [n, m] ∗ ∗h[n, m].
Example 10-2:
Deblurring Checkerboard Image
Consider the 8 × 8 color checkerboard image shown in
Fig. 10-10(a), which we label f [n, m], as it represents an original
unblurred image of the checkerboard.
If the original scene was imaged by a sensor characterized by
a circular PSF given by
(
√
1
for n2 + m2 ≤ 4,
√
(10.11)
h[n, m] =
0
for n2 + m2 > 4.
(a) Generate the convolved image
g[n, m] = f [n, m] ∗ ∗h[n, m],
(10.12)
and (b) apply deconvolution to reconstruct the original image.
Solution: (a) A 2-D display of the PSF, when centered at pixel
(5, 5), is shown in Fig. 10-10(b). The PSF approximates a disk
of radius 4. Included in the PSF are pixels [5, 5], [5, 5 ± 4],
and [5 ± 4, 5]. Hence, to include all its pixels, we approximate
the PSF as a (9 × 9) square. Application of the convolution
operation given by Eq. (10.12) leads to the blurred image shown
in Fig. 10-10(c). Per Section 5-2.5, if the size of image f [n, m]
is (M × M) and if h[n, m] is of size (L × L), then the size of the
convolved image is (N × N) with
N = M + L − 1 = 8 + 9 − 1 = 16.
(10.9)
For the YIQ image format, convolving Eq. (10.4) with h[n, m]
leads to

 


gY [n, m]
0.299 0.587
0.114
gR [n, m]
 gI [n, m]  = 0.596 −0.275 −0.321 gG [n, m] .
gQ [n, m]
0.212 −0.523 0.311
gB [n, m]
Hence, the blurred image in Fig. 10-10(c) is (16 × 16).
(b) To deblur the image in Fig. 10-10(c) and recover the
original image, we follow the recipe outlined in Section 6-4.1:
(1) Transform to the discrete frequency domain:
H[k1 , k2 ] = DFT{[h[n, m]},
GR [k1 , k2 ] = DFT{[gR [n, m]},
GG [k1 , k2 ] = DFT{[gG [n, m]},
GB [k1 , k2 ] = DFT{[gB [n, m]}.
(10.10)
Hence, all three of the YIQ components are also blurred, so
there is no particular advantage to transforming the image from
RGB to YIQ format. In practice deblurring (deconvolution) of a
color image is performed separately on each of gR [n, m], gG [n, m]
and gB [n, m] to obtain the original image components fR [n, m],
fG [n, m] and fB [n, m].
The deblurring process is illustrated through two examples.
(10.13a)
(10.13b)
(10.13c)
(2) Since convolution in the spatial domain is equivalent to
multiplication in the spatial frequency domain using (16 × 16)
2-D DFTs:
GR [k1 , k2 ] = FR [k1 , k2 ] H[k1 , k2 ],
(10.14)
and similar expressions apply to the green and blue images.
www.dbooks.org
344
CHAPTER 10 COLOR IMAGE PROCESSING
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
n
(a) (8 × 8) original checkerboard RGB image
m
(b) Extent of PSF when centered at pixel (5,5)
(c) (16 × 16) blurred checkerboard image
(d) (16 × 16) reconstructed checkerboard image
Figure 10-10 Simulation of blurring and deblurring processes: (a) original image, (b) PSF h[n, m], (c) original image blurred by system
PSF, and (d) reconstructed image after removal of blurring.
Hence, for the red channel we use the computed results to obtain
FR [k1 , k2 ] =
GR [k1 , k2 ]
,
H[k1 , k2 ]
(10.15)
convolved image is shown in Fig. 10-10(d). The blurring caused
by the sensor has been completely removed. The reconstructed
image is a (16 × 16) zero-padded version of the original (8 × 8)
image.
and similar expressions apply to the other two colors.
(3) Compute fR [n, m] for all pixels:
fR [n, m] = DFT−1 {FR [k1 , k2 ]},
(10.16)
and similar expressions apply to fG [n, m] and fB [n, m]. The de-
Concept Question 10-3:
tion in Example 10-2?
Why didn’t we need regulariza-
10-3 COLOR-IMAGE DEBLURRING
345
Exercise 10-5: Can the colors of a deconvolved image
differ from those of the original image?
Answer: Yes. Regularization can alter the values of the
reconstructed { fR [n, m], fG [n, m], fB [n, m]} image, and this
will affect the colors. However, the effect is usually too
slight to be noticeable.
Exercise 10-6: If the PSFs for different colors are different,
can we still deconvolve the image?
Answer: Yes. Use the proper PSF for each color.
(a) Blurred Christmas tree
Example 10-3:
Deblurring of Motion-Blurred
Image
6
4
2
Horizontal motion by a camera relative to a stationary Christmas
tree led to the (691 × 1162) blurry image g[n, m] shown in
Fig. 10-11(a). Apply the motion-deblurring technique of Section 6-6.2 to each of the three RGB images to obtain a deblurred
image f [n, m]. Use λ = 0.01 (varying the value of λ does not
appear to impact the results) and T = 1 s.
Solution:
(1) Image g[n, m] consists of three images, namely gR [n, m],
gG [n, m], and gB [n, m]. Application of the 2-D DSFT, as defined
by Eq. (3.73a), to each of the three images leads to three spectral
images GR (Ω1 , Ω2 ), GG (Ω1 , Ω2 ), and GB (Ω1 , Ω2 ).
(2) For a blur PSF of horizontal extent (N + 1) and an image
of horizontal extent M, the blurred image has a horizontal extent
of M + (N + 1) − 1 = M + N. According to Eq. (6.49), motion
blur in the horizontal direction causing an image to increase in
width by N pixels is mathematically equivalent to a filter with a
spatial frequency response H(Ω1 , Ω2 ) given by
T sin Ω1 N+1
2
e− jΩ1 N/2 .
(10.17)
H(Ω1 , Ω2 ) =
N sin(Ω1 /2)
Here, T is the total recording time and N is the total number
of time shifts that occur during time T . Because the exact value
of T is irrelevant to the deblurring procedure, we set T = 1 s.
In contrast, the value of N is crucial, so we need to extract it
from the spectrum of one of the three color channels. Using the
spectrum GR (Ω1 , Ω2 ) of the red channel, we sample it at
2 π k1
Ω1 =
1162
(10.18a)
0
-2
-4
-6
-8
-10
-12
-14
0
20
40
60
80
k1
(MATLAB)
100
(b) 1-D spectrum log(|GR[k1,0] + 1)
(c) Reconstructed Christmas tree
Figure 10-11
Christmas tree.
www.dbooks.org
Deblurring of motion-blurred image of a
346
CHAPTER 10 COLOR IMAGE PROCESSING
and
(10.18b)
Exercise 10-7: If the length of the blur were N + 1 = 6, at
what Ω1 values would H(Ω1 , Ω2 ) = 0?
to obtain GR [k1 , k2 ]. Next, we plot the horizontal profile of
|GR [k1 , k2 ]| as a function of k1 at k2 = 0. For display purposes,
we offset |GR [k1 , k2 ]| by 1 and plot the logarithm of the sum:
The numerator of Eq. (10.17) is zero when
sin(Ω1 (N + 1)/2) = 0, which occurs when Ω1 (N + 1)/2
is a multiple of π , or equivalently when Ω1 = ±kπ /3 for
integers k. This is independent of the size of the image.
Ω2 =
2 π k2
,
691
log(|GR [k1 , 0]| + 1).
Answer:
(10.19)
The MATLAB-generated plot of Eq. (10.19) is shown in
Fig. 10-11(b). It exhibits a periodic sequence of sharp nulls
corresponding to the zeros of sin[Ω1 (N + 1)/2] in the spatial
frequency response H(Ω1 , Ω2 ) of the PSF that caused the
blurring of the image. The first null is at MATLAB index = 15,
which corresponds to
k1 = 15 − 1 = 14.
Since the first null of the sine function occurs when its argument
is ±π , it follows that
Ω1 (N + 1) 2π k1 N + 1 2π × 14 N + 1
π=
,
=
=
2
1162 2
1162
2
which leads to N = 82.
(3) With H(Ω1 , Ω2 ) fully specified, we now apply the Wiener
filter of Eq. (6.50) to the red-color spectrum to obtain the
deblurred version
GR (Ω1 , Ω2 ) H∗ (Ω1 , Ω2 )
|H(Ω1 , Ω2 )|2 + λ 2
T sin Ω1 N+1
2
e jΩ1 N/2
GR (Ω1 , Ω2 )
N sin(Ω1 /2)
.
=
2
2
sin Ω1 N+1
T
2
+λ2
N
sin(Ω1 /2)
(10.20)
FR (Ω1 , Ω2 ) =
Similar procedures are applied to the two other channels.
(4) Application of the inverse DFT to each of the three color
channels yields fR [n, m], fG [n, m], and fB [n, m], the combination
of which yields the deblurred color image in Fig. 10-11(c). The
motion-caused blurring has essentially been removed. The only
artifact is the black band at the right-hand side of the image,
which is due to the zero-padding used in the DFT (compare with
Fig. 10-10(d)).
10-4 Denoising Color Images
In Chapter 7, we demonstrated how the discrete wavelet transform can be used to denoise an image by applying the combination of thresholding and shrinkage to the noisy image. The
same approach is equally applicable to color images. The recipe
outlined in Section 7-8.2 can be applied to each of the three
RGB channels separately, and then the three modified images
are combined to form the denoised color image. Since noise had
been added to each RGB component separately, the denoising
procedure also is applied to each RGB component separately. If
preserving color is paramount, denoising can be applied to the
Y component of YIQ, instead of to the RGB components.
We illustrate the effectiveness of the denoising technique
through the following two examples.
Example 10-4:
Denoising American Flag Image
To the original image shown in Fig. 10-12(a), a zero-mean white
Gaussian noise random field with σv = 0.5 was added to each
of the RGB components. The resultant noisy image is shown
in Fig. 10-12(b), and the associated signal-to-noise ratios are
3.32 dB for the red channel, 1.71 dB for the green, and 2.36 dB
for the blue. Application of the 2-D Haar transform method
outlined in Section 7-8 with a threshold/shrinkage factor λ = 3
led to the denoised image shown in Fig. 10-12(c). Much of the
noise has been removed.
10-4 DENOISING COLOR IMAGES
347
Example 10-5:
Brightening Toucan Image
The objective of this example is to brighten the toucan image
shown in Fig. 10-13(a) using two different methods, namely the
RGB and YIQ approaches, and then to compare and contrast the
results. The plots in parts (b) to (d) display the histograms of the
R, G, and B channels in the original image.
Method 1: RGB-Equalized Image
Application of the histogram-equalization method of Section
5-3 to the three RGB channels individually led to the image
in Fig. 10-14(a). The underlying histograms are shown in parts
(b)–(d) of the figure. The toucan image is indeed brighter than
that of the original, but because the histogram equalization
was performed separately and independently for the three RGB
channels, the color balance in the RGB-equalized image is not
the same as in the original image. Consequently, some of the
colors got distorted.
(a) Original flag image
Method 2: YIQ-Equalized Image
After (a) converting the original RGB image to YIQ format,
(b) applying histogram-equalization on only the Y channel, and
then (c) converting the YIQ-equalized image to RGB, we end up
with the image and associated histograms shown in Fig. 10-15.
The image in Fig. 10-15(a) is not as bright as the RGB-equalized
image in Fig. 10-14(a), but the color balance is preserved in the
YIQ-equalization method.
(b) Noisy image
Why should wavelet-based denoising be applied to images in YIQ format and not RGB
format?
Concept Question 10-4:
(c) Denoised image
Figure 10-12 Denoising American flag image using 2-D
Haar transform method generates a less noisy image, but at the
expense of high-resolution information.
www.dbooks.org
348
CHAPTER 10 COLOR IMAGE PROCESSING
3500
3000
2500
2000
1500
1000
500
0
50
(a) Original toucan image
100
150
200
250
(b) Histogram of red (R) channel in original image
3500
15000
3000
2500
10000
2000
1500
5000
1000
500
0
50
100
150
200
(c) Histogram of green (G) channel in original image
250
0
50
100
150
200
(d) Histogram of blue (B) channel in original image
Figure 10-13 Original image and associated histograms of its R, G, and B channels.
250
PROBLEMS
349
3500
3000
2500
2000
1500
1000
500
0
0
(a) RGB-equalized image
50
100
150
200
250
(b) Histogram of red (R) channel in RGB-equalized image
12000
2500
10000
2000
8000
1500
6000
1000
4000
500
0
0
2000
50
100
150
200
250
0
0
50
100
150
200
250
(c) Histogram of green (G) channel in RGB-equalized image (d) Histogram of blue (B) channel in RGB-equalized image
Figure 10-14 The RGB-equalized method generates a brighter image, but does not preserve color balance.
Summary
Concepts
• Color images have three components: red, green, and
blue, each of which is a separate 2-D function. This RGB
color system represents how images are acquired and
displayed.
• Other color systems include the CMYK system, used
for color printing, and the YIQ system, used for image
enhancement, which is applied only to the Y (luminance)
component.
• Image restoration, such as denoising and deconvolution,
uses the RGB color system, since each component is
blurred or has noise added to it directly. Image enhancement, such as histogram equalization and edge detection,
uses the YIQ system, since use of the RGB system can
result in distortion of colors.
• Other colors can be represented as a linear combination
of red, green, and blue, as depicted in the color cube.
www.dbooks.org
350
CHAPTER 10 COLOR IMAGE PROCESSING
2500
2000
1500
1000
500
0
(a) YIQ-equalized image
50
100
150
200
250
(b) Histogram of red (R) channel in YIQ-equalized image
3500
3500
3000
3000
2500
2500
2000
2000
1500
1500
1000
1000
500
500
0
0
20
40
60
80
100
120
140
160
180
200
(c) Histogram of green (G) channel in YIQ-equalized image
0
20
40
60
80
100
120
140
160
180
200
(d) Histogram of blue (B) channel in YIQ-equalized image
Figure 10-15 The YIQ-equalized method generates a partially brighter image, but does preserve color balance.
Mathematical Formulae
Components of a color image
f [n, m] = { fR [n, m], fG [n, m], fB [n, m]}
RGB to grayscale
fgray [n, m] = 0.299 fR [n, m] + 0.587 fG[n, m] + 0.114 fB[n, m]
Relation between components of RGB
and YIQ color systems

 


fY [n, m]
0.299 0.587
0.114
fR [n, m]
 fI [n, m]  = 0.596 −0.275 −0.321  fG [n, m]
fQ [n, m]
0.212 −0.523 0.311
fB [n, m]
MATLAB commands for color images
imread, imagesc
Important Terms
additive color scheme
chrominance
Provide definitions or explain the meaning of the following terms:
CMYK color scheme
color cube
luminance
RGB color scheme
subtractive color scheme
YIQ color scheme
PROBLEMS
351
PROBLEMS
10.5 Use sobelc.m to apply Sobel edge detection to the
toucan image in toucan.mat. Display the original image and
the result of edge detection applied to the Y image component.
Use a threshold of 0.5.
Section 10-1: Color Systems
10.1 Hang a checkerboard ornament on a Christmas tree.
The goal of this problem is to hang a color checkerboard
ornament in the middle of the Christmas tree in MATLAB file
xmastree.mat.
(a) Create in MATLAB a (64 × 64) checkerboard image,
consisting of an (8 × 8) array of alternating red and green
(8 × 8) blocks. Zero-pad this to (691 × 1080) (the size of
the image in xmastree.mat).
(b) Add the zero-padded checkerboard image to the Christmas
tree image. Display the resulting image. Why did this
procedure produce a poor result?
(c) Set the pixel values in the Christmas tree image where
the checkerboard image is nonzero to zero. Now add the
zero-padded checkerboard image to this image. Display the
resulting two images. Why did this procedure produce a
much better result?
10.2 Put the clown at the top of a Christmas tree. The goal of
this problem is to put the clown image in clown.mat at the
top of the Christmas tree in MATLAB file xmastree.mat.
(a) Convert the black-and-white clown image in clown.mat
from YIQ to RGB format. ( fI [n, m] and fQ [n, m] are
both zero; fY [n, m] is the black-and-white image.) Zeropad this to (691 × 1080) (the size of the image in
xmastree.mat).
(b) Add the zero-padded checkerboard image to the Christmas
tree image. Display the resulting image. Why didn’t this
work?
(c) Set the pixel values in the Christmas tree image where
the zero-padded clown image is nonzero to zero. Now add
the zero-padded clown image to this image. Display the
resulting two images. Why did this work?
10.6 Use sobelc.m to apply Sobel edge detection to the
plumage image in plumage.mat. Display the original image
and the result of edge detection applied to the Y image component. Use a threshold of 0.5.
Section 10-3: Color Image Deblurring
10.7 Deblur motion-blurred Christmas tree. Download
the (686 × 399) motion-blurred Christmas tree image in
xmasdarkmotion.mat.
(a) Display the motion-blurred Christmas tree image.
(b) Display the log of the magnitude of the red component
of xmasdarkmotion.mat. Zero-pad (686 × 399) to
(686 × 400) to speed up the 2-D FFT. Use the 2-D spectrum
to find the length N + 1 of the PSF of the motion blur.
(c) Use λ = 0.01 and T = 1 in motiondeblurcolor.m to
deblur and display the tree.
10.8 Deblur motion-blurred Christmas tree. Download the
motion-blurred Christmas tree image in xmasmotion.mat.
(a) Display the motion-blurred Christmas tree image.
(b) Use λ = 0.01, T = 1, and N = 15 in
motiondeblurcolor.m to deblur the tree.
10.9 Refocus an out-of-focus image. An out-of-focus image
can be modeled as the image convolved with a disk-shaped PSF.
The program refocuscolor.m convolves an image with a
disk PSF and then deconvolves. Use refocuscolor.m to
blur and deblur the (691 × 1080) image in xmastree.mat.
Uncomment a line in refocuscolor.m and use λ = 0.01
and disk radius R = 20.
10.3 Apply histogram equalization (HE) to the Christmas tree
image in xmastree.mat using the program histcolor.m.
Display results for HE of RGB and YIQ formats.
10.10 Refocus an out-of-focus image. An out-of-focus image
can be modeled as the image convolved with a disk-shaped PSF.
The program refocuscolor.m convolves an image with a
disk PSF and then deconvolves. Use refocuscolor.m to
blur and then deblur the (340 × 453) image in toucan.mat.
Uncomment a line in refocuscolor.m and use λ = 0.01 and
disk radius R = 11.
10.4 Apply histogram equalization (HE) to the Christmas
tree image in xmastreedark.mat using the program
histcolor.m. Display results for HE of RGB and YIQ
formats.
10.11 Refocus an out-of-focus image. An out-of-focus image
can be modeled as the image convolved with a disk-shaped PSF.
The program refocuscolor.m convolves an image with a
disk PSF and then deconvolves. Use refocuscolor.m to
Section 10-2: Histogram Equalization and Edge
Detection
www.dbooks.org
352
blur and then deblur the (512 × 512) image in flag.mat.
Uncomment a line in refocuscolor.m and use λ = 0.01
and disk radius R = 29.
10.12 Refocus an out-of-focus image. An out-of-focus image
can be modeled as the image convolved with a disk-shaped PSF.
The program refocuscolor.m convolves an image with a
disk PSF and then deconvolves. Use refocuscolor.m to
blur and then deblur the plumage image in plumage.mat.
Uncomment a line in refocuscolor.m and use λ = 0.01
and disk radius R = 18.
Section 10-4: Denoising Color Images
10.13 Use Haar transform threshold and shrinkage program
haardenoisecolor.m to denoise each RGB component of
the (8 × 8) checkerboard image in checker.mat. Zero-mean
white Gaussian noise with variance σ 2 is added to each RGB
component. Uncomment a line in haardenoisecolor.m
and use λ = 1 and σ = 0.2.
10.14 Use Haar transform threshold and shrinkage program
haardenoisecolor.m to denoise each RGB component
of the Christmas tree image in xmastree.mat. Zero-mean
white Gaussian noise with variance σ 2 is added to each
RGB component. The image is clipped to (704 × 704) since
haardenoisecolor.m requires square images. Uncomment
a line in haardenoisecolor.m and use λ = 0.2 and
σ = 0.1.
10.15 Use Haar transform threshold and shrinkage program haardenoisecolor.m to denoise each RGB component of the toucan image in toucan.mat. Zero-mean
white Gaussian noise with variance σ 2 is added to each
RGB component. Image is zero-padded to (480 × 480) since
haardenoisecolor.m requires square images. Uncomment
a line in haardenoisecolor.m and use λ = 0.1 and
σ = 0.1.
10.16 Use Haar transform threshold and shrinkage program haardenoisecolor.m to denoise each RGB component of the plumage image in plumage.mat. Zero-mean
white Gaussian noise with variance σ 2 is added to each
RGB component. Image is zero-padded to (512 × 512) since
haardenoisecolor.m requires square images. Uncomment
a line in haardenoisecolor.m and use λ = 0.2 and
σ = 0.1.
10.17 Use db3 transform threshold and shrinkage program
daubdenoisecolor.m to denoise each RGB component
of the Christmas tree image in xmastree.mat. Zero-mean
CHAPTER 10 COLOR IMAGE PROCESSING
white Gaussian noise with variance σ 2 is added to each
RGB component. The image is clipped to (704 × 704) since
daubdenoisecolor.m requires square images. Uncomment
a line in daubdenoisecolor.m and use λ = 0.05 and
σ = 0.05.
10.18 Use db3 transform threshold and shrinkage program daubdenoisecolor.m to denoise each RGB component of the toucan image in toucan.mat. Zero-mean
white Gaussian noise with variance σ 2 is added to each
RGB component. Image is zero-padded to (480 × 480) since
daubdenoisecolor.m requires square images. Uncomment
a line in daubdenoisecolor.m and use λ = 0.05 and
σ = 0.05.
10.19 Use db3 transform threshold and shrinkage program daubdenoisecolor.m to denoise each RGB component of the plumage image in plumage.mat. Zero-mean
white Gaussian noise with variance σ 2 is added to each
RGB component. Image is zero-padded to (512 × 512) since
daubdenoisecolor.m requires square images. Uncomment
a line in daubdenoisecolor.m and use λ = 0.05 and
σ = 0.05.
Chapter
11
11 Image Recognition
Contents
Overview, 354
11-1 Image Classification by Correlation, 354
11-2 Classification by MLE, 357
11-3 Classification by MAP, 358
11-4 Classification of Spatially Shifted Images, 360
11-5 Classification of Spatially Scaled Images, 361
11-6 Classification of Rotated Images, 366
11-7 Color Image Classification, 367
11-8 Unsupervised Learning and Classification, 373
11-9 Unsupervised Learning Examples, 377
11-10 K-Means Clustering Algorithm, 380
Problems, 384
Objectives
Learn to:
■ Use correlation, MLE, and MAP to classify an image.
■ Use cross-correlation to classify a shifted image.
■ Use coordinate transformation (logarithmic or polar)
to classify a spatially scaled or rotated image.
■ Classify color images.
■ Use the SVD to perform unsupervised learning.
www.dbooks.org
(a) Image of text
(b) Thresholded cross-correlation ρ5(n0,m0)
(c) Image of text (green) and thresholded cross-correlation (red)
Image recognition amounts to classification of a
given image as one of several given possible
candidate images. Classification can be regarded as
a discrete estimation problem in which the candidate image that maximizes a likelihood function is
chosen. If there are no candidate images, unsupervised learning can be used to determine image
classes from a set of training images.
Overview
Along the very bottom section of the check shown in Fig. 11-1
are three sequences of numbers. The first sequence, labeled
the “Routing Number,” specifies the identity of the bank that
had issued the check; the second sequence is the “Account
Number” and it identifies the owner of the account; and the
last sequence is the “Check Number” itself. When the check is
inserted inside a check reader, an optical sensor scans across the
rectangular space containing the three sequences of numbers and
then displays the sequences on an electronic screen. How does
the check reader recognize the individual digits comprising the
three sequences? The answers to this and other image recognition questions are at the heart of the present and succeeding
chapter.
The objective of an image recognition algorithm is to correctly classify an observed image gobs [n, m] into one of L
possible classes on the basis of stored, true (reference) images
fk [n, m] of the L classes, where { k = 1, . . . , L }. If the classes are
digits printed in a specific font, then the objective is to identify
the correct digit from among the 10 possible digits between 0
and 9. In text recognition, not only is the number of classes
(letters in the alphabet) much larger, but the letters may be
printed in lower or upper case and may include more than one
type of font. Other image recognition applications may involve
classifying objects on the basis of their shapes and colors.
In some cases, the image recognition algorithm requires
some form of prior training or learning using sample images
of the types of objects under consideration. Otherwise, for L
classes of objects with known true (reference) images fk [n, m]—
with { k = 1, 2, . . . , L }—an algorithm measures the degree of
Figure 11-1 Example of a bank check with three sequences
of numbers.
similarity between gobs [n, m] and each stored, reference image
fk [n, m] and then selects the class index k that maximizes
the similarity. Some algorithms use the correlation between
gobs [n, m] and each fk [n, m] as the parameter with which to
measure the degree of similarity, while other algorithms may use
measurement parameters that incorporate other image properties
as well. Often the observed image gobs [n, m] contains random
noise, and the shape of the imaged object may be distorted due
to some angular rotation or size modification. A robust classification algorithm needs to be adept at dealing with these types
of deviations. These and other aspects of image recognition are
addressed in this chapter.
Image recognition algorithms rely on one of three approaches:
(1) Correlation between the observed image and each of
the stored reference images, in conjunction with a priori
probability information about the frequency of occurrence
of the unknown objects.
(2) Unsupervised training using training images with unknown identities.
(3) Supervised training using training images with known
identities.
The present chapter covers the first two approaches, and the third
approach is covered in Chapter 12.
11-1 Image Classification by
Correlation
Consider the 10 numerals shown in Fig. 11-2(a). The numerals
are printed in white, using a standard bank font, against a black
background. In practice, check numbers are read by a reading
head (optical scanner) whose size in pixels is (9 × 1). As the
head moves horizontally across each (9 × 7) numeral, a 1-D signal of duration 7 samples is generated as the output “signature”
of that numeral. Upon comparing the measured signal with the
10 possible signals stored in the system, the system selects the
numeral (from among 1, 2, . . . , 9, 0) that provides the best match.
An alternative, and more reliable, approach to performing the
numeral recognition problem is to use a 2-D image instead of
a 1-D signal. Let us assume that each numeral in the check
account number is allocated a rectangular space of M × N pixels,
comprising N horizontal pixels and M vertical rows of pixels.
When the optical scanner sweeps across the rectangle containing
the bank account number, it images the rectangle and then
partitions it into individual images each (M × N) pixels in size.
An expanded view of one such image is shown in Fig. 11-2(b).
354
11-1 IMAGE CLASSIFICATION BY CORRELATION
355
(a) Numerals 1,2,…,9,0 in Bank font
(b) Noisy image gobs[n,m] of numeral 3
ρk
x 10
6
14
12
10
8
6
4
2
0
0
1
2
3
4
5
6
7
8
9
k
(c) Correlation ρk
ΛMLE[k]
x 10
6
15
10
5
k
0
−5
0
1
2
3
4
5
6
7
8
9
(d) ΛMLE[k]
Figure 11-2 (a) Numerals 1–9 and 0 in bank font, (b) noisy (30 × 18) image gobs [n, m] of numeral 3 with SRN = −1.832 dB, (c)
correlation ρk , and (d) MLE criterion Λ2 [k].
www.dbooks.org
356
CHAPTER 11 IMAGE RECOGNITION
It contains a single numeral, but it is difficult to recognize the
numeral’s identity because the image is quite noisy (the signalto-noise ratio is −1.832 dB, corresponding to a signal energy
only 66% of that of the noise). The image size is (30 × 18) and
the true identity of the numeral is 3, but we assume at this stage
of the classification example that we do not yet know the true
identity of the numeral in Fig. 11-2(b).
Let us designate the observed image of the unknown numeral
as gobs [n, m], which consists of the sum of two images, namely
images fk [n, m] and υ [n, m]:
gobs [n, m] = fk [n, m] + υ [n, m],
(0 ≤ n ≤ N − 1, 0 ≤ m ≤ M − 1),
(11.1)
where fk [n, m] is the true image of the unknown (but we do
not yet know the value of index k) and υ [n, m] is a randomnoise image. Index k denotes the class of image fk [n, m]: for the
present example, k extends over the range k = 1 for class 1 to
k = 10 for class 10, with class 1 representing numeral 1, class 9
representing numeral 9, and class 10 representing numeral 0.
The goal of the classification algorithm is to determine the true
value of index k, from the observed image gobs [n, m], given the
stored reference images f1 [n, m] through f10 [n, m].
A common approach to establishing the identity of an unknown class is to compute the correlation ρk between the
observed image gobs [n, m] and the reference image fk [n, m] for
all L classes, and the select the value of k associated with the
highest correlation. The correlation ρk is given by
N−1 M−1
ρk = ∑ ∑ gobs [n, m] fk [n, m].
(11.2)
Notation for Training
and Classification
Single-Color Images
fk [n, m] = reference image of class k, { k = 1, . . . , L }
(i)
fk [n, m] = ith training image of class k
L = number of classes
gobs [n, m] = observed image of unknown class
M × N = image size
ρk = correlation between fk [n, m] and gobs [n, m]
p[k] = a priori probability of occurrence of class k
E fk = energy of image fk [n, m]
ΛMLE = MLE maximization parameter
k̂MLE = value of k that maximizes ΛMLE
ΛMAP = MAP maximization parameter
k̂MAP = value of k that maximizes ΛMAP
Color Images
(i)
fk [n, m] = ith training image vector for class k
(i)
(i)
(i)
= [ fk,R [n, m], fk,G [n, m], fk,B [n, m]]
gobs [n, m] = observed image vector
n=0 m=0
Since gobs [n, m] and fk [n, m] are both (M × N), computation of
ρk requires MN multiplications and additions for each value of k.
If gobs [n, m] and fk [n, m] have sparse wavelet transforms, we can
take advantage of Parseval’s theorem for the wavelet transform
(Eq. (7.14b)) to compute ρk , because multiplications by zero do
not count as computations. The
√ Haar transform requires only
subtractions and divisions by 2.
To determine the identity of the numeral in the noisy image
shown in Fig. 11-2(b), ρk was computed using Eq. (11.2) for
all 10 values of k, and the results are displayed in Fig. 11-2(c).
Among the 10 correlations, ρ3 of class 3 (corresponding to
numeral 3) exhibits the largest value, but it exceeds ρ8 of class 8
by only a small amount. The correct numeral in image gobs [n, m]
is indeed 3, so the correlation method is successful in classifying
it correctly, but the margin of error is rather slim. As we shall
see shortly, the MLE classification method outlined in the next
Kk [n, m] = class-specific covariance matrix
Kk0 = joint covariance matrix
section also classifies the observed image correctly, but it does
so with a superior margin of error.
Exercise 11-1: We have two classes of (1 × 1) images
f1 [n, m] = 1 and f2 [n, m] = 4. Use correlation to classify
gobs [n, m] = 2.
ρ1 = (2)(1) = 2 and ρ2 = (2)(4) = 8. Since
ρ2 > ρ1 , classify gobs [n, m] as k = 2. This is counterintuitive
(2 is closer to 1 than to 4), so see Exercise 11-2.
Answer:
11-2 CLASSIFICATION BY MLE
357
11-2 Classification by MLE
all locations [n, m] of the marginal pdfs p(gobs [n, m]) given by
Eq. (11.5):
For L classes, the observed (M × N) image gobs [n, m] is one of:


class #1,
 f1 [n, m] + υ [n, m]


 f2 [n, m] + υ [n, m]
class #2,
(11.3)
gobs [n, m] =
..
.
..


.



fL [n, m] + υ [n, m]
class L,
where υ [n, m] is a (M × N) zero-mean white Gaussian noise
random field with variance σv2 .
11-2.1 Marginal pdfs
n=0 m=0
N−1 M−1
1
2
2
∏ ∏ e−(gobs [n,m]− fk [n,m]) /(2σv )
(2πσv2 )NM/2 n=0 m=0
N−1 M−1
2
2
1
=
e− ∑n=0 ∑m=0 (gobs [n,m]− fk [n,m]) /(2σv ) . (11.6)
NM/2
2
(2πσv )
=
Following Section 9-3, the likelihood
p({gobs[n, m]}) is one of the following joint pdfs:
p({gobs[n, m]}) =
Accordingly, for each location [n, m], exactly one of the following marginal pdfs is the pdf of gobs [n, m]:

2

class #1,
N ( f1 [n, m], σv )


N ( f2 [n, m], σv2 )
class #2,
(11.4)
gobs [n, m] ∼
.
..
..


.


N ( f [n, m], σ 2 )
class L,
L
v
where N ( fk [n, m], σv2 ) is a short-hand notation analogous to
that in Eq. (8.86), and it states that the pdf of gobs [n, m] for class k
has the form of the Gaussian pdf given by Eq. (8.38a):
1
2
2
p(gobs [n, m]) = p
e−(gobs [n,m]− fk [n,m]) /(2σv ) .
2
2πσv
N−1 M−1
p({gobs [n, m]}) = ∏ ∏ p(g[n, m])
(11.5)
The form of the marginal pdf (i.e., the pdf specific to pixel [n, m])
given by Eq. (11.5) is a consequence of the fact that υ [n, m] is a
zero-mean random field, as a result of which the mean value of
gobs [n, m] is simply fk [n, m].
Notationally, p(gobs [n, m]) is the marginal pdf of pixel [n, m]
and p({gobs [n, m]}) is the joint pdf of all pixels in image
gobs [n, m]. Since for class k, image gobs [n, m] is the sum of
fk [n, m] and white random noise υ [n, m], and since the amount of
noise υ [n, m] added to pixel [n, m] is independent of the amount
of noise added to other pixels, image values {gobs[n, m]} can
be regarded as independent if the amount of noise added to
{ fk [n, m]} is significant. This is certainly the case for the image
shown in Fig. 11-2(b). [If the noise is insignificant in comparison with the signal, the classification task becomes rather trivial;
the correlation method of the preceding subsection should be
able to correctly classify the observed image with no error.]
Hence, the joint pdf p({gobs [n, m]}) is equal to the product over
function
1
(2πσv2 )NM/2
 −(1/2σ 2) N−1 M−1 (g [n,m]− f [n,m])2
1
v ∑n=0 ∑m=0 obs
e



M−1
2

e−(1/2σv2) ∑N−1
n=0 ∑m=0 (gobs [n,m]− f 2 [n,m])
× .
 ..



 −(1/2σ 2) ∑N−1 ∑M−1 (gobs [n,m]− fL [n,m])2
v
n=0 m=0
e
class #1,
class #2,
class #L,
(11.7)
and the natural log-likelihood ln[p({gobs[n, m]})] is
1
NM
ln(2πσv2 ) − 2
ln[p({gobs[n, m]})] = −
2
2σv

N−1 M−1
2


n=0 ∑m=0 (gobs [n, m] − f 1 [n, m])
∑N−1

M−1
2
∑
n=0 ∑m=0 (gobs [n, m] − f 2 [n, m])
× .
..




 N−1 M−1 (g [n, m] − f [n, m])2
∑n=0 ∑m=0 obs
L
class #1,
class #2,
class #L.
(11.8)
11-2.2 MLE of Index k
The maximum likelihood estimate k̂MLE is the value of k that
maximizes the log-likelihood function given by Eq. (11.8).
2
2
Since − NM
2 log(2πσv ) is added to all terms, and −1/(2σv )
multiplies all terms, k̂MLE is the value of k that maximizes Λ1 [k]
defined as
N−1 M−1
Λ1 [k] = − ∑ ∑ (gobs [n, m] − fk [n, m])2 .
(11.9)
n=0 m=0
This expression has an evident interpretation: Choose the value
of k such that fk [n, m] is closest to gobs [n, m] in the ℓ2 (sum of
www.dbooks.org
358
CHAPTER 11 IMAGE RECOGNITION
squares) norm defined in Eq. (7.117b).
Computation of k̂MLE can be simplified further by expanding
Eq. (11.9), since
N−1 M−1
2
Λ1 [k] = − ∑ ∑ (gobs [n, m] − fk [n, m])
n=0 m=0
N−1 M−1
N−1 M−1
= − ∑ ∑ gobs [n, m]2 − ∑ ∑ fk [n, m]2
n=0 m=0
N−1 M−1
n=0 m=0
+ 2 ∑ ∑ gobs [n, m] fk [n, m].
(11.10)
n=0 m=0
The first term is the energy of gobs [n, m], which is independent
of k, so we can ignore it. The second term is the energy E fk of
fk [n, m]. The recipe for computing k̂MLE simplifies to choosing
the value of k that maximizes the MLE criterion ΛMLE [k]:
N−1 M−1
ΛMLE[k] = 2 ∑ ∑ gobs [n, m] fk [n, m] − E fk .
(11.11)
n=0 m=0
In view of Eq. (11.2), ΛMLE [k] is related to the correlation ρk
by
ΛMLE [k] = 2ρk − E fk .
(11.12)
Returning to the image of Fig. 11-2(b), computation of the MLE
parameter ΛMLE [k] leads to the stem plot in Fig. 11-2(d), which
correctly selects numeral 3 as the numeral in the noisy image,
with a significant margin between the values of ΛMLE [3] and
ΛMLE[8].
Classified as
True numeral
0 1 2 3 4 5 6 7 8 9
0 78 4 1 5 1 3 2 2 0 4
1 0 77 4 0 6 2 7 3 0 1
2 5 9 69 1 7 1 2 5 1 0
3 2 2 0 85 2 3 4 0 1 1
4 0 3 4 0 70 5 5 3 8 2
5 1 4 2 4 5 76 0 4 1 3
6 0 6 1 0 3 1 83 3 2 1
7 0 3 11 2 5 4 1 69 4 1
8 0 0 1 3 7 1 3 6 75 4
9 0 3 1 0 3 4 7 2 4 76
The first row pertains to numeral 0. Of the 100 noisy images
containing an image of numeral 0, 78 were classified correctly,
4 were misclassified as numeral 1, 1 was misclassified as
numeral 2, etc. Each row sums to 100, and similar interpretations
apply to the other numerals. A perfect classifier would be a
(10 × 10) diagonal matrix of 100s. For the noisy images used
in this trial, the classification accuracy varied between a low of
69% for numeral 7 and a high of 85% for numeral 3.
Why is the maximum likelihood (MLE) classifier different from correlation?
Concept Question 11-1:
Exercise 11-2: We have two classes of (1 × 1) images
f1 [n, m] = 1 and f2 [n, m] = 4. Use the MLE classifier to
classify gobs [n, m] = 2.
Answer:
ρ1 = (2)(1) = 2 and ρ2 = (2)(4) = 8.
ΛMLE [1] = 2(2) − 12 = 3 and ΛMLE [2] = 2(8) − 42 = 0.
Since ΛMLE [1] > ΛMLE [2], classify gobs [n, m] as k = 1.
Including energies of the images resulted in a different
classification.
11-2.3 MLE Bank Numeral Classification
We now extend the numeral classification example by performing 1000 trials comprised of 100 noisy images of each of the 10
numerals 0 to 9. By selecting σv2 = 106 , we decrease the signalto-noise ratio from −1.832 dB, an image example of which is
shown in Fig. 11-1(b), down to an average of −16.7 dB among
the various trials and numerals. At this ratio, the average signal
energy is only 2% of the noise energy!
The results of the trials using the MLE criterion ΛMLE [k] to
classify each of the 1000 trials are summarized in the form of
the following confusion matrix:
11-3 Classification by MAP
Sometimes a priori probabilities p[k] are available for the
various fk [n, m]. This is not the case for zip codes or bank
account numbers (all ten digits are equally likely), but it is
the case for letters of the alphabet. Figure 11-3 depicts the
frequencies of appearances of a letter in typical text. These
frequencies function as a priori probabilities p[k] for each letter
in letter recognition in an image of text.
The a priori probabilities p[k] can be incorporated into the image recognition problem by using an MAP formulation instead
of an MLE formulation, by simply multiplying the likelihood
11-3 CLASSIFICATION BY MAP
359
2 1
and
into one of the two classes defined by the image
5 7
8 6
. Obtain a classification rule in terms of the values of the
4 3
four elements of gobs [n, m] using (a) correlation and (b) MLE.
0.14
0.12
0.10
0.08
0.06
0.04
0.02
0
abcdefghijklmnopqrstuvwxyz
Solution:
(a) Classification by correlation: In terms of the notation
introduced earlier,
2 1
(11.14a)
f1 [n, m] =
5 7
and
f2 [n, m] =
Figure 11-3 Frequencies of appearances of letters, which can
be regarded as a priori probabilities p[k].
8 6
.
4 3
(11.14b)
The energies of f1 [n, m] and f2 [n, m] are
function in Eq. (11.7) by p[k]. The modification adds ln[p[k]] to
the log-likelihood function in Eq. (11.8). Repeating the above
derivation, k̂MAP is computed by choosing the k that maximizes
ΛMAP [k]:
ΛMAP [k] = 2ρk − E fk + 2σv2 ln p[k].
(11.13)
The MAP (maximum a posteriori) classifier is also known
as the minimum error probability (MEP) classifier since it
minimizes the probability of an incorrect choice of k. Note
that if p[k] = 1/L, so that each class is equally likely, the
MAP classifier reduces to the MLE classifier. Also note that
σv2 functions as a trade-off parameter between the a priori
information p[k] and the a posteriori information ρk : the noisier
the observations, the larger is σv2 , and the greater the weight
given to a priori information p[k]. The smaller the noise, the
heavier is the weight given to a posteriori (from the noisy data)
information ρk .
(2 × 2) Image Classification
Example
Example 11-1:
2
2
2
2
E f2 = 8 + 6 + 4 + 3 = 125.
(11.15a)
(11.15b)
The correlations between gobs [n, m] and each of f1 [n, m] and
f2 [n, m] are
ρ1 = 2g0,0 + g1,0 + 5g0,1 + 7g1,1,
ρ2 = 8g0,0 + 6g1,0 + 4g0,1 + 3g1,1.
(11.16a)
(11.16b)
For a given image g[n, m], we classify it as
f1 [n, m],
if ρ1 > ρ2
(11.17a)
f2 [n, m],
if ρ1 < ρ2 .
(11.17b)
or as
(b) Classification by MLE: Using Eq. (11.11), the MLE
parameter Λ2 [k] is given by
Λ2 [1] = 4g0,0 + 2g1,0 + 10g0,1 + 14g1,1 − 79,
Λ2 [2] = 16g0,0 + 12g1,0 + 8g0,1 + 6g1,1 − 125.
(11.18a)
(11.18b)
Upon subtracting the expression for Λ2 [1] from the expression
for Λ2 [2], we obtain the following classification rule:
Image of gobs [n, m] is
Classify noisy image
gobs [n, m] =
E f1 = 22 + 12 + 52 + 72 = 79,
g0,0
g0,1
g1,0
g1,1
f1 [n, m],
www.dbooks.org
if (12g0,0 + 10g1,0 − 2g0,1 − 8g1,1 − 46) < 0
(11.19a)
360
CHAPTER 11 IMAGE RECOGNITION
and
f2 [n, m],
if (12g0,0 + 10g1,0 − 2g0,1 − 8g1,1 − 46) > 0.
(11.19b)
Concept Question 11-2: Why is the maximum a posteriori probability (MAP) classifier different from the maximum likelihood (MLE) classifier?
Exercise 11-3: We have two classes of (1 × 1) images
f1 [n, m] = 1 and f2 [n, m] = 4, with respective a priori
probabilities p[k = 1] = 0.1 and p[k = 2] = 0.9. The additive
noise has σv2 = 3. Use the MAP classifier to classify
gobs [n, m] = 2.
Answer:
ρ1 = (2)(1) = 2 and ρ2 = (2)(4) = 8.
ΛMAP [1] = 2(2) − 12 + 2(3) log(0.1) = −10.8 and
ΛMAP [2] = 2(8) − 42 + 2(3) log(0.9) = −0.63. Since
ΛMAP [2] > ΛMAP [1], classify gobs [n, m] as k = 2. The a
priori probabilities biased the classification towards k = 2.
11-4 Classification of Spatially Shifted
Images
Thus far, we have assumed that the location, size, and orientation
of the observed image gobs [n, m] is the same as those of the
reference images { fk [n, m], k = 1, 2, . . . , L} for all L classes
of numerals, letters, objects, etc. Now, in this and the next
two sections, we examine how to perform image classification
successfully even when the reference images are shifted in
location, of different size, or different orientation relative to the
observed image.
11-4.1 Unknown Relative Shift
gobs [n, m]-Reference Format:


 f1 [n − n0, m − m0 ]


 f2 [n − n0, m − m0 ]
gobs [n, m] =
..


.


 f [n − n , m − m ]
L
0
0
class #1,
class #2,
..
.
class #L,
(11.20)
fk [n, m]-Reference Format:


f1 [n, m]



 f2 [n, m]
gobs [n − n0, m − m0 ] =
..


.


 f [n, m]
L
class #1,
class #2,
..
.
class #L.
(11.21)
or as
11-4.2 Classification by Cross-Correlation
In both cases, the objective is to select the correct class through
some appropriate comparison between gobs and fk for all possible values of k. One such measure of comparison is the crosscorrelation ρk [n0 , m0 ] between gobs and fk when one of them
is shifted by [n0 , m0 ] relative to the other. Of the two formats,
the observation-reference format given by Eq. (11.20) is more
applicable to search algorithms, so we adopt it in here for
computing the cross-correlation:
N−1 M−1
ρk [n0 , m0 ] = ∑ ∑ gobs [n, m] fk [n − n0, m − m0 ].
(11.22)
n=0 m=0
Both images are of size (M × N) pixels.
By rearranging the order of the indices of fk , we can cast the
summations in the form of a convolution:
N−1 M−1
In the absence of a spatial shift between the observation and
reference images, all images are arranged so that their origins
[0, 0] coincide. In some applications, however, the observed
image gobs [n, m] or the reference images fk [n, m] may be shifted
by an unknown amount [n0 , m0 ] relative to each other. One such
application is the search for occurrence of a specific letter in a
block of text, an example of which is presented later in the form
of Example 11-4.
Since it is the relative—rather than the absolute—spatial shift
that matters, the image classification problem can be formulated
as either:
ρk [n0 , m0 ] = ∑ ∑ gobs [n, m] fk [−(n0 − n), −(m0 − m)]
n=0 m=0
= gobs [n0 , m0 ] ∗ ∗ fk [−n0 , −m0 ],
(11.23)
The image representing ρk [n0 , m0 ] is called the crosscorrelation between gobs [n, m] and fk [n, m]. Using properties #4
and #5 in Table 3-3, ρk [n, m] can be computed readily using the
zero-padded 2-D DFT:
Gobs [k1 , k2 ] = (2M × 2N) DFT{gobs[n, m]},
Fk [k1 , k2 ] = (2M × 2N) DFT{ fk [n, m]},
(11.24a)
(11.24b)
11-5 CLASSIFICATION OF SPATIALLY SCALED IMAGES
ρk [n, m] = DFT−1 {Gobs[k1 , k2 ] Fk [2N − k1 , 2M − k2 ]}.
(11.24c)
11-4.3 Classification Recipe for Discrete-Space
Images
The recipe for classification when an unknown relative
shift [n0 , m0 ] may exist between gobs [n, m] and { fk [n, m],
k = 1, 2, . . . , L} consists of the following steps:
1. Compute ρk [n, m] for all values of k: {k = 1, 2, . . . , L} using
Eq. (11.24c).
2. Among the total of (M × N) pixels × L classes = MNL,
identify the combination of pixel location [n, m] and class k
that yields the largest value of ρk [n, m].
3. Label that specific pixel location as [n, m] = [n0 , m0 ] and
label the identified value of k as the unknown class K.
Example 11-2:
361
11-4.4 Classification Recipe for
Continuous-Space Images
In analogy with Eq. (11.20), for continuous-space images the
classification problem is formulated as


f1 (x − x0 , y − y0 )
class #1,



 f2 (x − x0 , y − y0 )
class #2,
(11.25)
gobs (x, y) =
..
.
..


.


 f (x − x , y − y )
class #L,
L
0
0
where the true class is class #K and variables K, x0 and y0 are all
unknown. The cross-correlation for the continuous-space case is
given by
ρk (x0 , y0 ) =
Z ∞Z ∞
−∞ −∞
gobs (ξ , η ) fk (ξ − x0 , η − y0) d ξ d η ,
(11.26)
and the classification recipe proceeds as follows:
1. Given observation image gobs (x, y) and reference images
fk (x, y), compute the cross-correlation ρk (x0 , y0 ) for all
values of k: {k = 1, 2, . . . , L}, and for all spatial shifts
(x0 , y0 ) that offer nonzero overlap between gobs (x, y) and
fk (x − x0 , y − y0).
2. Identify the combination of shift (x0 , y0 ) and class k that
exhibits the largest value of ρk (x0 , y0 ).
Identifying Letter “e”
in Text Image
Figure 11-4(a) displays a (33 × 256) image of two lines of text.
Use cross-correlation to identify all the locations of the letter
“e” within the text image. In the printed text, each lower-case
letter is allocated (9 × 7) pixels and reference images fk [n, m]
are provided for all lower-case letters.
Solution: Application of the recipe outlined in Section 11-5.3
leads to the identification of 7 pixels with cross-correlation
values that were much larger than those of other pixels. After
examining the distribution of values, all values of the crosscorrelation smaller than 800,000 were set to zero, thereby
highlighting the high-value pixels shown in Fig. 11-4(b). With
the text printed in green in Fig. 11-4(c), the identified pixels
in part (b) are now printed in red. Examination of the image
confirms that the cross-correlation algorithm has successfully
identified all occurrences of the letter “e” in the text image.
11-5 Classification of Spatially Scaled
Images
We now consider the problem of image classification in which
the observed image gobs (x, y) is a spatially scaled version of
the reference images fk (x, y), or vice versa. Our presentation
is specific to continuous-space observation and reference images, gobs (x, y) and fk (x, y). In a later part of this section, we
address how to apply the classification algorithm developed for
continuous-space images to discrete-space images. Additionally, to keep the presentation simple and manageable, we assume
that the observed images are noise-free.
The topic of spatial scaling of images was presented earlier
in the book, in Section 3-2.2A. If gobs (x, y) is a spatially scaled
version of fk (x, y), then
gobs (ax x, ay y) = fk (x, y)
or
www.dbooks.org
(11.27a)
362
CHAPTER 11 IMAGE RECOGNITION
(a) Image of text
(b) Thresholded cross-correlation ρ5(n0,m0)
(c) Image of text (green) and thresholded cross-correlation (red)
Figure 11-4 Identifying the letter “e” in Example 11-4.
gobs (x, y) = fk
x y
,
ax ay
.
(11.27b)
The positive-value constants ax and ay are unknown spatialscaling factors. If ax > 1, gobs (x, y) is fk (x, y), but magnified
in the x direction by ax . Conversely, if ax < 1, g(x, y) is fk (x, y)
but shrunk in the x direction by ax . Similar variations apply to
ay along the y direction. An obvious example of image classification in the presence of spatial scaling is when the relative
sizes of letters and numerals in the images to be classified are
different from those in the reference images fk (x, y).
The spatial-scaling classification problem can be formulated
as
 x y


f1
,
class #1,



ax ay 



 f2 x , y
class #2,
ax ay
gobs (x, y) =
(11.28)

..
..


. .




x
y


 fL
class #L,
,
ax ay
where the true class is class #K and all images are nonzero only
for x, y > 0. Class K and variables ax , and ay are all unknown.
11-5.1 Logarithmic Spatial Transformation
Clearly, because gobs and fk have different spatial scales, the
image classification problem cannot be solved using the same
cross-correlation approach presented in earlier sections.
However, we can still solve the problem—and use crosscorrelation—by performing logarithmic warping of spatial coordinates x and y. To that end, we change variables from (x, y)
to (x′ , y′ ):
x′ = ln(x)
′
y = ln(y)
′
x = ex ,
(11.29a)
y′
(11.29b)
y=e .
Next, we define spatially warped images g′obs (x′ , y′ ) and
fk′ (x′ , y′ ):
′
′
g′obs (x′ , y′ ) = gobs (ex , ey ),
(11.30a)
′
′
fk′ (x′ , y′ ) = fk (ex , ey ),
(11.30b)
11-5 CLASSIFICATION OF SPATIALLY SCALED IMAGES
and logarithmically transformed scale factors (a′x , a′y ):
′
a′x = ln(ax )
a x = ea x ,
(11.31a)
a′y = ln(ay )
a′y
(11.31b)
ay = e .
Using these transformations, the spatially warped observation
image g′obs (x′ , y′ ) can be related to the spatially warped reference
image fk′ (x′ , y′ ) as follows:
!
x′ ey′
′
′
e
,
g′obs (x′ , y′ ) = gobs (ex , ey ) = fk
ax ay
!
′
′
ex ey
= fk
′ ,
′
ea x ea y
′
′
′
′
= fk (e(x −ax ) , e(y −ay ) )
= fk′ (x′ − a′x , y′ − a′y ).
(11.32)
Hence, in the logarithmically transformed spatial variables
(x′ , y′ ), spatial scaling by (ax , ay ) becomes a spatial shift by
(a′x , a′y ). An example is illustrated in Fig. 11-5.
The problem statement defined by Eq. (11.28) can now be
reformulated as


f ′ (x′ − a′x , y′ − a′y )
class #1,

 1′ ′

 f (x − a′x , y′ − a′y )
class #2,
2
g′obs (x′ , y′ ) =
(11.33)
.
..
.


.
.


 ′ ′
class #L,
fL (x − a′x , y′ − a′y )
where now the unknown true class is still class k = K, but
the other unknowns are a′x and a′y . The formulation defined by
Eq. (11.33) is the same as that of the spatial-shift classification
problem given by Eq. (11.25) in the previous section. The crosscorrelation between g′obs (x′ , y′ ) and fk′ (x′ , y′ ) when the latter is
spatially shifted by (x0 , y0 ) is given by the convolution
ρk (x0 , y0 ) =
ZZ
g′obs (x′ , y′ ) f ′ (x′ − x0 , y′ − y0) dx′ dy′
= g′obs (x0 , y0 ) ∗ ∗ f ′ (−x0 , −y0 ).
(11.34)
11-5.2 Classification Recipe
The following recipe is for continuous-space images. If the
images are in discrete-space form, interpolation can be applied
to convert them into continuous-space form using the methods
described in Chapter 4.
The recipe for classifying observation image gobs (x, y), given
363
that its coordinates may have an unknown spatial scaling relative
to those of reference images fk (x, y), consists of the following
steps:
1. Use Eqs. (11.29) and (11.30) to transform all images g(x, y)
and fk (x, y) to logarithmic format g′ (x′ , y′ ) and fk′ (x′ , y′ ).
2. Use Eq. (11.34) to compute ρk (x0 , y0 ) for each value of k:
{k = 1, 2, . . . , L}, and all spatial shifts (x0 , y0 ) that offer
nonzero overlap between g′ (x′ , y′ ) and fx′ (x′ − x0 , y′ − y0 ),
using the 2-D CSFT.
3. Identify the combination of shift (x0 , y0 ) and class k that
yields the largest value of ρk (x0 , y0 ). Label that combination as (a′x , a′y ) and k = class K.
4. With (a′x , a′y ) known, the scaling factors (ax , ay ) can then be
determined using Eq. (11.31).
Example 11-3:
Scaled-Image Classification
In Fig. 11-6(a), we are given displays of 10 reference images
of bank-font digits fk [n, m], with k = 1 to 9 representing the
numerals 1 to 9, respectively, and k = 10 representing 0. Each
image is (32 × 20) in size, even though the original size was only
(30 × 18); the original image was zero-padded by inserting two
rows and two columns of zeros at the top and left of the image.
The zero-padding was used to ensure that the logarithmically
transformed image had sufficient space to be shifted.
We also are given an observed image gobs [n, m] shown in
Fig. 11-6(b), which we know to be a downsized scaled version
of one of the 10 digits in Fig. 11-6(a), with a scaling factor of
2 or 4, but we do not know which particular digit it represents,
nor do we know the exact value of the scaling factor. [Actually,
gobs [n, m] is digit “3” at half-size, but that information is yet to
be established by the image classifier.] Classify the unclassified
image gobs [n, m].
Solution: In Section 11-4.1, we used a natural-log transformation to convert a spatially scaled continuous-space image gobs (x, y) into a spatially shifted continuous-space image
g′obs (x′ , y′ ). For a discrete-space image gobs [n, m], we use base
2 logarithms:
www.dbooks.org
n′ = log2 n
′
m = log2 m
′
n = 2n ,
(11.35a)
m′
(11.35b)
m=2 .
364
CHAPTER 11 IMAGE RECOGNITION
5
10
2
4
6
15
8
10
20
12
14
16
25
2
m
4
6
8
10
n
gobs[n,m]
30
m
4
8
12
16
20
n
f3[n,m]
(a) Images in linear coordinates [n,m]
n′
m′
f3′[n′,m′]
n′
m′
n and m
n′ and m′
1 2 4 8 16 32
0 1 2 3 4 5
′ [n′,m′]
gobs
(b) Images in logarithmic coordinates [n′,m′]
(c) Relationship between [n,m] and [n′,m′]
Figure 11-5 (a) (32 × 20) image f3 [n, m] and (16 × 10) image gobs [n, m], (b) the same images in logarithmic format in base 2 (instead of
base e), also note that g′obs [n′ , m′ ] = f3′ [n′ + 1, m′ + 1], and (c) the relation between [n, m] and [n′ , m′ ]. [Note that 32 pixels in [n, m] space
transform into 6 (not 5) pixels in [n′ , m′ ] space.]
11-5 CLASSIFICATION OF SPATIALLY SCALED IMAGES
365
f1′ [n′,m′]
f2′ [n′,m′]
f3′ [n′,m′]
f4′ [n′,m′]
f5′ [n′,m′]
f6′ [n′,m′]
f7′ [n′,m′]
f8′ [n′,m′]
f9′ [n′,m′]
f0′ [n′,m′]
(a) Reference images fk [n,m]
(a) Reference images fk′[n′,m′] in logarithmic format
(b) Observed image gobs[n,m]
(b) Observed image gobs
′ [n′,m′]
in logarithmic format
Figure 11-6 (32 × 20) reference images and (unclassified)
observed image of Example 11-3.
Figure 11-7 (6 × 5) reference images and (5 × 4) (unknown)
observation image in logarithmic format.
The spatially logarithmically transformed image g′obs [n′ , m′ ] in
[n′ , m′ ] space is
Based on the given information,
hn mi
m
n
gobs = fk
,
for 1 ≤ ≤ 20, 1 ≤ ≤ 32, (11.37)
a a
a
a
with the scaling factor a = 1/2 or 1/4. For convenience, we
introduce the inverse scaling factor
(
1
1
for a = 1/2,
i
with i =
(11.38)
b= =2,
a
2
for a = 1/4.
where in the last step we used the definition given by Eq. (11.36)
to convert fk into its logarithmic equivalent fk′ . The result encapsulated by Eq. (11.39) states that in [n′ , m′ ] space, the observed
image g′obs [n′ , m′ ] is a spatially shifted version of fk′ [n′ , m′ ] and
the shift is i pixels along each direction. Hence, application of
the spatial-shift recipe outlined in Section 11-3 should lead to
correctly identifying the values of both k and i. The validity of
this statement is verified by Figs. 11-7 and 11-8. In Fig. 11-7, we
show the reference and observed images in logarithmic format.
Visual inspection reveals the close similarity between f3′ [n′ , m′ ]
and gobs [n′ , m′ ], and that the spatial shift between them is i = 1,
corresponding to a = 1/2.
Computational classification is realized by computing the
MLE criterion ΛMLE [k] as defined by Eq. (11.12):
Upon using Eq. (11.38) in Eq. (11.37) and then incorporating
the scaling relationship into Eq. (11.36), we have
ΛMLE [k] = 2ρk′ − E f ′ ,
′
′
g′obs [n′ , m′ ] = gobs [2n , 2m ].
′
′
g′obs [n′ , m′ ] = fk [2i 2n , 2i 2m ]
′
′
= fk [2n +i , 2m +i ].
= fk′ [n′ + i, m′ + i],
(11.36)
(11.39)
k
(11.40)
where ρk′ is the cross-correlation between g′obs [n′ , m′ ] and
fk′ [n′ , m′ ] and E f ′ is the energy of image fk′ [n′ , m′ ]. Because
k
the energies of the 10 digits in [n′ , m′ ] space vary widely (as
evidenced by the wide range in the number of white pixels
among the 10 digits in Fig. 11-7(a)), use of the MLE criterion
www.dbooks.org
366
CHAPTER 11 IMAGE RECOGNITION
ΛMLE[k]
5
x 10
3
2
1
k
0
−1
−2
0
1
2
3
4
5
6
7
8
9
Figure 11-8 ΛMLE [k] based on correlation between noisy digit g′obs [n, m] and each of the ten digits fk′ [n, m] for {k = 0, 1, . . . , 10}.
ΛMLE[k] provides a better classifier than ρk′ alone. The computed
values of ΛMLE [k], plotted in Fig. 11-8, show that ΛMLE is
largest for digit “3.” Note that because images f2′ [n, m] and
f3′ [n, m] in Fig. 11-7(a) look similar, their computed MLE
criteria (Fig. 11-8) are close in values.
Concept Question 11-3: In classification of spatiallyshifted images, why not just correlate with all possible
shifts of the image?
11-6 Classification of Rotated Images
We now consider the continuous-space and noiseless version
of the image classification problem in which the observation
gobs (x, y) is a rotated (by an unknown angle θo ) version of
fK (x, y), or vice versa. As before, K is the unknown true value
of k.
11-6.1 Polar-Coordinate Transformation
To incorporate the unknown angular rotation into the image
classification problem, we reformulate the problem using polar
coordinates (r, θ ) instead of Cartesian coordinates (x, y). Per
Fig. 3-11(a) and Eq. (3.44), the two pairs of variables are related
by
x = r cos θ
y = r sin θ
p
r = x2 + y2
.
θ = tan−1 (y/x)
(11.41)
The image classification problem with unknown relative rotation
θo can then be reformulated as


fe1 (r, θ − θo )
class #1,



 fe2 (r, θ − θo )
class #2,
geobs (r, θ ) =
(11.42)
..
.
..


.


e
fL (r, θ − θo )
class #L,
where geobs (, θ ) and fek (r, θ ) are gobs (x, y) and fk (x, y) in polar
coordinates, respectively.
The form of Eq. (11.42) is the same as that of Eq. (11.20)
for the spatial-shift problem, except that now the variables
are (r, θ ) instead of (x, y). In fact, the problem is relatively
simpler because we only have one unknown spatial variable,
namely θo , rather than two. Using the cross-correlation concept
that served us well in earlier sections, we define the rotation
cross-correlation ρek (r, θ ) as
ρek (r, θ ) =
Z
geobs (r, θ ′ ) fek (r, θ ′ − θ ) d θ ′ .
(11.43)
11-7 COLOR IMAGE CLASSIFICATION
367
applying Eq. (11.41) to convert the coordinates of the 11 images
from Cartesian (x, y) to polar (r, θ ). The transformed images are
displayed in Fig. 11-10, with the horizontal axis representing θ
and the vertical axis representing r. Close examination of the
images reveals that the observed image is identical with image
“3” of the reference images, except for a 90◦ shift. Hence, it
is not surprising that application of Eq. (11.43) against all 10
reference images and at many values of θ between 0◦ and 360◦
produces the highest value for the cross-correlation when k = 3
and θ = 90◦ .
(a) (17 × 17) clipped reference images
When is interpolation unnecessary for classification of spatially scaled images?
Concept Question 11-4:
(b) Upper half of numeral “3” rotated by 90˚
11-7 Color Image Classification
Figure 11-9 Reference images and rotated observed image of
Example 11-4.
11-6.2 Classification Recipe
1. Transform all images gobs (x, y) and g f (x, y) to polarcoordinates format geobs (r, θ ) and fek (r, θ ).
2. Use Eq. (11.43) to compute ρek (r, θ ) for every possible
combination of k (among L classes) and θ , using the 2-D
CSFT.
3. Identify the combination that yields the largest value of
ρek (r, θ ). Label k = K and θ = θo .
Example 11-4: Rotated Image Classification
The (32 × 20) reference images shown in Fig. 11-6(a) have been
clipped to (17 × 17), so only the upper half of each image is
shown in Fig. 11-9(a). The 90◦ -rotated image of one of the numerals, namely numeral “3,” is shown in Fig. 11-9(b). Classify
the rotated image, presuming that its identity is unknown, as is
its degree of rotation.
Solution: The recipe in Subsection 11-5.2 relies on computing
the cross-correlation ρek (r, θ ) defined by Eq. (11.43). Doing
so requires two prerequisite steps, namely using interpolation
to convert the reference images and the observation image
from discrete-space format to continuous-space format, and then
11-7.1 Notation for Color Images
We now extend the classification methods of the preceding
section to color images consisting of red (R), green (G), and
blue (B) channels. The treatment is equally applicable to nonoptical sensors, such as imaging radars operating at three different wavelengths, and is extendable to multispectral images
comprised of more that three channels.
In Sections 11-1 through 11-3, we considered the scenario
where we were given a set of L possible images { fk [n, m],
k = 1, 2, . . . , L} and the goal was to classify a noisy image
gobs [n, m] into one of the L classes. Now we consider the case
where the { fk [n, m]} images must themselves be estimated from
(i)
a set of I training images { fk [n, m], k = 1, . . . , L; i = 1, . . . , I}.
We will assume that all random variables are jointly Gaussian,
so it will be necessary for us to estimate mean vectors and
covariance matrices for each class k from these training images.
In the numeral classification example of Section 11-1, we had
L = 10 classes of numerals, one observed image gobs [n, m] and
one known reference image fk [n, m] for each value of k, with k
extending between 1 and 10. With color images, the observed
image consists of three channels, and so do all of the training
images. Another consideration that we should now address is
the fact that in many classification problems the identity of a
given class is not unique. If we are dealing with numerals that
are always printed using a single type of font, then we only need
a single reference image per numeral (1 to 9, plus 0), but if
we need to classify numerals that may have been printed using
multiple types of fonts, then we would need a much larger set of
reference images.
Similar within-class variations occur among text letters
www.dbooks.org
368
CHAPTER 11 IMAGE RECOGNITION
(1)
r
r
8
8
6
6
(2)
4
2
0
(3)
2
100
200
300
400
500
600
700
θ
0
r
r
8
8
6
6
(4)
4
2
0
(5)
100
200
300
400
500
600
700
θ
0
r
8
8
6
6
(6)
4
100
200
300
400
500
600
700
θ
0
r
8
8
6
6
(8)
4
500
600
700
θ
100
200
300
400
500
600
700
θ
100
200
300
400
500
600
700
θ
100
200
300
400
500
600
700
θ
400
500
600
700
θ
4
2
100
200
300
400
500
600
700
θ
0
r
r
8
8
6
6
(0)
4
2
0
400
2
2
(9)
300
4
r
0
200
4
r
0
100
2
2
(7)
4
4
2
100
200
300
0
θ
400
500
600
700
100
200
300
(a) Reference images displayed in polar coordinates
r
8
6
4
2
0
100
200
300
400
500
600
700
θ
90°
(b) Observed image
Figure 11-10 In polar coordinates, the observed image in (b) matches the reference image of “3”, shifted by 90◦ .
11-7 COLOR IMAGE CLASSIFICATION
369
printed in different fonts, as well as in many object recognition
applications.
With these two considerations (3-color channels and withinclass variability) in mind, we introduce the following notation:
Number of classes: L, with class index k = 1, 2, . . . , L.
Image size: M × N
Observed-image model for class k:
gobs [n, m] = fk [n, m] + v[n, m],
where v[n, m] is a vector of length 3 (one for each color),
modeled as a zero-mean white Gaussian noise random field with
variance σv2 for each color component.
Gaussian model for gobs [n, m]:
Number of training images per class: I, with training image
index i = 1, 2, . . . , I.
gobs [n, m] = [gk,R [n, m], gk,G [n, m], gk,B [n, m]]T
∼ N ([fk [n, m]], Kk0 ),
(i)
Red channel ith training image for class k: fk,R [n, m]
ith training image vector of 2-D functions of [n, m] for class k:
(i)
(i)
(i)
(i)
fk [n, m] = [ fk,R [n, m], fk,G [n, m], fk,B [n, m]]T
(11.44)
Kk0 = Kk + σv2 I3 ,
(11.45)
11-7.2 Likelihood Functions
Mean training image vector for class k:
fk [n, m] = [ f k,R [n, m], f k,G [n, m], f k,B [n, m]]T
(11.46)
Covariance matrix estimate for class k at location [n, m]:
1 I (i)
(i)
∑ (fk [n, m] − fk [n, m])(fk [n, m] − fk [n, m])T
I i=1
1 I (i)
(i)
∑ [fk [n, m](fk [n, m])T] − fk [n, m](fk [n, m])T.
I i=1
(11.47)
The estimated covariance matrix Kk [n, m] accounts for the
variability at location [n, m] among the I training images, relative
to their mean fk [n, m], and correlation between different colors
in an image.
Location-independent covariance matrix for class k:
Kk =
1 N−1 M−1
∑ ∑ Kk [n, m]
NM n=0
m=0
(11.51)
where I3 is the (3 × 3) identity matrix.
1 I (i)
f k,R [n, m] = ∑ fk,R [n, m]
I i=1
=
(11.50)
where Kk0 is a (3 × 3) joint covariance matrix that accounts
for both Kk , the in-class variability of the noise-free training
images, and the added noise represented by σv2 :
Mean red channel training image for class k:
Kk [n, m] =
(11.49)
(11.48)
The location-independent covariance matrix Kk is the sample mean over [n, m] of Kk [n, m]. It is used to represent the inclass variability when Kk [n, m] is statistically independent of location [n, m], which is a valid assumption in many applications.
Even though for class k, vector fk [n, m] and covariance matrix
Kk are both estimated from the vectors of the I training images
(i)
{fk [n, m]}, we use them as the “true” values, not just estimates,
in the classification algorithms that follow. The goal of the
classification algorithm is to determine the value of the class
index k from among the L possible classes.
Given an observation image vector gobs [n, m] and an average
reference image vector fk [n, m] for each class index k, we define
the difference image vector ∆ k [n, m] as
∆ k [n, m] = gobs [n, m] − fk [n, m]
= [gk,R [n, m], gk,G [n, m], gk,B [n, m]]T
− [ f k,R [n, m], f k,G [n, m], f k,B [n, m]]T .
(11.52)
By treating fk [n, m] as the mean value of gobs [n, m], and in
view of the model given by Eq. (11.50), the location-specific
marginal pdf p(gobs [n, m]) of length-3 Gaussian random vector
gobs [n, m] for class k is given by
p(gobs [n, m]) =
(2π )3/2
1
1
p
det(Kk0 )
T
−1
× e− 2 ((∆∆k [n,m]) Kk0 ∆k [n,m]) .
(11.53)
The {gobs[n, m]} values at different locations [n, m] are independent random vectors because noise vectors v[n, m] are location-
www.dbooks.org
370
CHAPTER 11 IMAGE RECOGNITION
independent. Accordingly, their joint pdf is
The corresponding natural log-likelihood function is given by
N−1 M−1
p({gobs[n, m]}) = ∏ ∏ p(gobs [n, m])
n=0 m=0
=
1
1
3NM/2
(2π )
(det(Kk0 ))NM/2
N−1 M−1
1
−1
× ∏ ∏ e− 2 ((∆∆k [n,m]) Kk0 ∆k [n,m])
T
n=0 m=0
=
1
1
3NM/2
(2π )
(det(Kk0 ))NM/2
N−1 M−1 ∆
1
T −1
× e− 2 (∑n=0 ∑m=0 (∆ k [n,m]) Kk0 ∆k [n,m]) .
(11.54)
The joint pdf given by Eq. (11.54) is the likelihood function of
the observed image vector gobs [n, m]. An expanded version for
the individual classes is given by
3NM
ln(p({gobs[n, m]})) = −
ln(2π )
2

NM


−
ln(det(K10 ))


2




1 N−1 M−1


∆1 [n, m])T K−1
− ∑ ∑ (∆

10 ∆ 1 [n, m]

2 n=0 m=0




NM


−
ln(det(K20 ))


2



1 N−1 M−1
∆2 [n, m])T K−1
− ∑ ∑ (∆
+
20 ∆ 2 [n, m]
2 n=0 m=0




.

.


.





NM


−
ln(det(KL0 ))


2



1 N−1 M−1


 − ∑ ∑ (∆
∆L [n, m])T K−1

L0 ∆ L [n, m]
2 n=0 m=0
class 1,
class 2,
class L.
(11.56)
1
p({gobs[n, m]}) =
3NM/2
(2π )

N−1 M−1
1

T −1
1


e− 2 ((∆∆1 [n,m]) K10 ∆ 1 [n,m]) class 1,

∏
∏
NM/2

(det(K10 ))

n=0 m=0

11-7.3 Classification by MLE


N−1 M−1

1

∆2 [n,m])T K−1
− 12 ((∆
∆ 2 [n,m])

20
class 2,
∏ ∏e
The maximum likelihood estimate k̂MLE of class k is the
× (det(K20 ))NM/2 n=0 m=0

..

value of k that maximizes the log-likelihood function given by


.


Eq. (11.56). Since the first term [−(3NM/2) ln(2π )] is common

N−1 M−1

1

T −1
1

to all terms, it has no impact on the choice of k. Hence, k̂MLE is


∏ ∏ e− 2 ((∆∆L [n,m]) KL0 ∆L [n,m]) class L. the
(det(KL0 ))NM/2 n=0 m=0
value of k that maximizes the MLE criterion Λ1 [k] given by
(11.55)
N−1 M−1
∆k [n, m])T K−1
Λ1 [k] = −NM ln(det(Kk0 ))− ∑ ∑ (∆
k0 ∆ k [n, m] .
n=0 m=0
(11.57)
According to Eq. (11.51), the joint covariance matrix Kk0
incorporates two statistical variations, Kk due to variations
among training images for class k, and σv2 due to the added
noise. If the classification application is such that Kk does
not vary with class k (or the variation from class to class is
relatively minor), we then can treat Kk as a class-independent
covariance matrix K, with a corresponding joint covariance
matrix K0 = K + σv2 I3 . As a consequence, the first term in
Eq. (11.57) becomes class-independent and can be removed
from the maximization process, which leads us to adjust the
definition of the MLE criterion to
N−1 M−1
∆k [n, m])T K−1
Λ1 [k] = − ∑ ∑ (∆
0 ∆ k [n, m] .
n=0 m=0
(11.58)
11-7 COLOR IMAGE CLASSIFICATION
371
Upon replacing ∆ k [n, m] with its defining expression given by
Eq. (11.52), Eq. (11.58) becomes
additional term:
N−1 M−1
ΛMAP [k] = 2 ∑ ∑ (gobs [n, m])T K−1
0 fk [n, m]
N−1 M−1
∆k [n, m])T K−1
Λ1 [k] = − ∑ ∑ (∆
0 ∆ k [n, m]
n=0 m=0
N−1 M−1
n=0 m=0
N−1 M−1
− ∑ ∑ fk [n, m] K−1
0 fk [n, m] + 2 ln(p[k]).
= − ∑ ∑ (gobs [n, m] − fk [n, m])T K−1
0
n=0 m=0
(11.61)
n=0 m=0
× (gobs[n, m] − fk [n, m])
The MAP classifier (estimator of k) selects the value of k that
maximizes ΛMAP [k].
N−1 M−1
= − ∑ ∑ gobs [[n, m]K−1
0 gobs [n, m]]
n=0 m=0
N−1 M−1
− ∑ ∑ fk [n, m]K−1
0 fk [n, m]
n=0 m=0
N−1 M−1
+ 2 ∑ ∑ (gobs [n, m])T K−1
0 fk [n, m] .
(11.59)
Example 11-5:
Color Image Classification
n=0 m=0
The last term comes from noting that a scalar equals its own
transpose, which is useful in matrix algebra.
The first term in Eq. (11.59) is independent of k. Hence,
computation of k̂MLE simplifies to choosing the value of k that
maximizes the modified MLE criterion
N−1 M−1
ΛMLE [k] = 2 ∑ ∑ (gobs [n, m])T K−1
0 fk [n, m]
n=0 m=0
N−1 M−1
− ∑ ∑ fk [n, m] K−1
0 fk [n, m].
(11.60)
n=0 m=0
The expression given by Eq. (11.60) is the vector version of the
expression given earlier in Eq. (11.11) for the scalar case.
11-7.4 Classification by MAP
An noted earlier in connection with Fig. 11-3, the probability
of occurrence p[k] of the letters of the alphabet varies widely
among the 26 letters of the English language. The same may be
true for other classification applications. The maximum a priori
(MAP) classifier takes advantage of this a priori information
by multiplying the likelihood function given by Eq. (11.54) by
p[k]. This modification leads to a MAP classification criterion
ΛMAP [k] given by the same expression for ΛMLE [k], but with an
Develop a classification rule for a two-class, (1 × 1) color image
classifier for a (1 × 1) image [gR , gG , gB ] with no additive noise
and equal class probabilities (p[k = 1] = p[k = 2]), given the
following 4 training images per class:
Class k = 1:
 
 
 
 
0
4
4
4
(1)
(2)
(3)
(4)
f1 = 0 , f1 = 0 , f1 = 4 , f1 = 0 ,
0
0
0
4
Class k = 2:
 
 
 
 
0
0
0
4
(1)
(2)
(3)
(4)
f2 = 0 , f2 = 4 , f2 = 4 , f2 = 4 .
4
0
4
4
(11.62)
(11.63)
Solution: The sample means of the four training sets are
         
1 0 4 4 4 3
0 + 0 + 4 + 0
f1 =
= 1
(11.64)
4
0
0
0
4
1
and
         
1 0 0 0 4 1
0 + 4 + 4 + 4
f2 =
= 3 .
4
4
0
4
4
3
www.dbooks.org
(11.65)
372
CHAPTER 11 IMAGE RECOGNITION
The sample covariance for k = 1 is
 
 

4
1 0
0 [0, 0, 0] + 0 [4, 0, 0]
K1 =
4
0
0
 
 

4
1 4
4 [4, 4, 0] + 0 [4, 0, 4]
+
4
0
4


 
3
3 1
1
− 1 [3, 1, 1] = 1 3 −1 ,
1
1 −1 3
and the sample covariance for k = 2 is
 
 

0
1 0
0 [0, 0, 4] + 4 [0, 4, 0]
K2 =
4
4
0
 
 

4
1 0
4 [0, 4, 4] + 4 [4, 4, 4]
+
4
4
4
 


1
3 1
1
− 3 [1, 3, 3] = 1 3 −1 .
3
1 −1 3
gB
Region f2
(0,0,4)
(4,0,4)
(4,4,4)
(0,4,0)
(0,0,0)
gG
(11.66)
(4,0,0)
Region f1
gR
(4,4,0)
Figure 11-11 Decision regions for Example 11-5.
and for k = 2,
(11.67)
Because K1 = K2 , Eq. (11.60) applies. The classification rule is:
Choose the value of k that maximizes ΛMLE[k] in Eq. (11.60).
Setting K1 = K2 = K, the inverse of K is


1  2 −1 −1
−1
−1 2
1 .
(11.68)
K =
4 −1 1
2
For k = 1, ΛMLE [1] of Eq. (11.60) becomes

 
2
−1
−1
3
1
1  1
ΛMLE [1] = 2[gR, gG , gB ] −1 2
4 −1 1
2
1

 
1 2 −1 −1 3
1
1
− [3, 1, 1] −1 2
4 −1 1
2
1
= 2gR − 3,
(0,4,4)
(11.69)

 
1  2 −1 −1 1
−1 2
1
3
ΛMLE [2] = 2[gR , gG , gB ]
4 −1 1
2
3

 
1 2 −1 −1 1
1
3
− [1, 3, 3] −1 2
4 −1 1
2
3
= 4gG + 4gB − 2gR − 11.
(11.70)
The classification rule (choose the larger of ΛMLE [1] and
ΛMLE [2]) simplifies to the following simple sign test:
• Choose f1 if: −4gR + 4gG + 4gB − 8 < 0,
• Choose f2 if: −4gR + 4gG + 4gB − 8 > 0.
In 3-D space, with axes {gR, gG , gB }, the boundary is a plane
that separates the regions in which {gR , gG , gB } is assigned to
f1 and in which it is assigned to f2 . The regions are shown in
Fig. 11-11.
Example 11-6:
(2 × 2) Image Classifier
We are given two (2 × 2) color image classes:
1 2
4 5
7 8
f1,R =
, f1,G =
, f1,B =
,
3 0
6 0
9 0
11-8 UNSUPERVISED LEARNING AND CLASSIFICATION
3
f2,R =
1
6
2
, f2,G =
4
0
9
5
, f2,B =
7
0
8
.
0
(11.71)
Find the rule for classifying an observed (2 × 2) color image
R
g
gR0,1
gR = R0,0
,
g1,0 gR1,1
G
g0,0 gG
0,1
,
(11.72)
gG = G
g1,0 gG
1,1
B
gB0,1
g
gB = B0,0
.
g1,0 gB1,1
Solution: The second term in Eq. (11.60) represents the energy
Efk of class k. By inspection Ef1 = Ef2 (f1 and f2 have the same
numbers). Hence the second term in Eq. (11.60) is the same
for k = 1 and 2, so we can ignore it. Consequently, Eq. (11.60)
simplifies to
ΛMLE [1] = 1gR0,0 + 2gR0,1 + 3gR1,0
(1) F is any (M × N) matrix.
(2) U is an (M × M) orthogonal matrix, which means
UT U = UUT = IM ,
where IM is an (M × M) identity matrix.
VT V = VVT = IN .
(4) S is an (M × N) diagonal matrix, of the following form:
G
G
+ 6gG
0,0 + 5g0,1 + 4g1,0
(11.74)
Classification Rule: Choose the larger of ΛMLE [1] and ΛMLE [2].
Note that g1,1 is not used for any color, as expected.
How do we estimate image vec(i)
tors fk [n, m] from the training image vectors fk [n, m]?
Concept Question 11-5:
11-8 Unsupervised Learning and
Classification
In unsupervised learning we are given a set of L training
images. We know nothing about what the images are supposed
to represent, or to what classes they belong, or even what the
classes are. The goals of unsupervised learning and classification include:
(2) To classify each of the training images.
Before diving into the mechanics of unsupervised learning and
classification, we provide an overview of an important matrix
tool known as singular value decomposition (SVD).
The SVD is used to factorize an (M × N) matrix F into the
product of two orthogonal matrices U and V, and one diagonal
matrix S:
F = USVT .
(11.75)
(11.73)
ΛMLE [2] = 3gR0,0 + 2gR0,1 + 1gR1,0
(1) To identify classes from the training images.
11-8.1 Singular Value Decomposition
(3) V is an (N × N) orthogonal matrix:
G
G
+ 4gG
0,0 + 5g0,1 + 6g1,0
+ 9gB0,0 + 8gB0,1 + 7gB1,0.
(3) To provide a simple rule for classifying any new image into
one of the classes identified by the training images.
The four matrices have the following attributes:
Assume K1 = K2 = I and equal a priori probabilities.
+ 7gB0,0 + 8gB0,1 + 9gB1,0,
373
(a) If M ≥ N:
diag[σ j ]
S=
,
0M−N, N
where diag[σ j ] is an M × M diagonal matrix of {σ j }, and
in this case F and S are called tall matrices.
(b) If M ≤ N:
S = diag[σ j ] 0M, N−M ,
where diag[σ j ] is an M × M diagonal matrix of {σ j }, and
in this case F and S are called reclining matrices.
Note that 0m,n is an (m × n) matrix of zeros.
The {σ j } are called singular values. The number of singular
values is the smaller of M and N. Note that F and S both have
the same size (M × N). More information on {σ j } is presented
shortly.
11-8.2 SVD Computation by
Eigendecomposition
In this book, we do not prove the existence of the SVD, nor
review algorithms for computing it. However, we offer the
www.dbooks.org
374
CHAPTER 11 IMAGE RECOGNITION
following simple approach to compute the SVD of F by noting
that

12/13 0
1
V= 0
−5/13 0
FFT = (USVT )(VST UT )
= (US)(VT V)(ST U) = U diag[σ 2j ]UT
(11.76a)
and
FT F = (VST UT )(USVT )
= (VST )(UT U)(SVT ) = V diag[σ 2j ]VT .
(11.76b)
These relationships rely on the fact that U and V are orthogonal
matrices: VVT = UT U = I.
As demonstrated shortly in the SVD example, the eigenvectors of FFT constitute the columns of U, the eigenvectors of
FT F constitute the columns of V, and the {σ j } are the nonzero
eigenvalues of FFT and of FT F. Given V and {σ 2j }, or U and
{σ 2j }, we can compute U or V as follows:
U = FVdiag[σ −1
j ],
(11.77a)
T
= diag[σ −1
j ]U F,
(11.77b)
V
T
Postmultiplying FFT in Eq. (11.76a) by U and using UT U = I
gives
(11.78e)
(FFT )U = U diag[σ 2j ].
Next, let u j be the jth column of (M × M) matrix U. Then the
jth column of Eq. (11.78e) is
(FFT )u j = σ 2j u j ,
◮ The SVD of a matrix A can be computed using MATLAB
by the command [U,S,V]=svd(A); ◭
Example 11-7: Compute SVD
Matrix F is given by
96 39 −40
.
F=
−72 52 30
(11.78a)
Compute the SVD of F (a) by MATLAB and (b) by eigendecomposition of FFT .
Solution: (a) The MATLAB command [U,S,V]=svd(F)
yields
4/5 3/5
,
(11.78b)
U=
−3/5 4/5
130 0 0
S=
,
(11.78c)
0 65 0
(11.78d)
The two singular values are σ1 = 130 and σ2 = 65.
The singular values are usually put in decreasing order
σ1 > σ2 > · · · by reordering the columns of U and the rows
of VT .
(b) Using Eq. (11.78a), we compute


96 −72
96
39
−40
 39
52 
FFT =
−72 52 30
−40 30
12337 −6084
.
=
−6084 8788
both of which follow directly from F = USVT .

5/13
0 .
12/13
(11.78f)
This is because postmultiplying U by diag[σ 2j ] multiplies the jth
column of U by σ 2j , and U is an orthogonal matrix characterized
by uTi u j = δ [i − j]. Equation (11.78f) states that u j is an
eigenvector of FFT with associated eigenvalue σ 2j , which means
that (FFT − σ 2j I) is singular and therefore its determinant is
zero:
(11.78g)
det(FFT − σ 2j I) = 0.
The roots of this quadratic polynomial are σ12 and σ22 , which can
be computed by inserting the matrix FFT computed earlier into
Eq. (11.78g):
1 0
12337 −6084
= 0,
− σ 2j
det
0 1
−6084 8788
which leads to
(12337 − σ 2j )(8788 − σ 2j ) − (−6084)2 = 0.
The solution of the quadratic equation gives
σ1 = 130,
σ2 = 65.
11-8 UNSUPERVISED LEARNING AND CLASSIFICATION
To find V, premultiply F = USVT first by UT and then by
T
diag[σ −2
j ]. Using U U = I, the process gives
e T = diag[σ −2 ]UT F,
V
i
(11.78h)
e T is the first M rows of VT . This is as expected:
where V
A = USVT uses only the first M rows of VT . Since VT must
be orthogonal, the remaining rows of VT must be chosen to be
orthogonal to its first M rows. The remaining rows of VT can
be computed from its first M rows using, say, Gram-Schmidt
orthonormalization.
The SVD of F can therefore be computed as follows:
1. Compute eigenvalues and eigenvectors of FF T .
2. The eigenvalues are σi2 and the eigenvectors are ui .
e T of VT using
3. Compute the first M rows V
e T = diag[σ −2 ]UT F.
V
i
4. Compute the remaining rows of VT using Gram-Schmidt
orthonormalization.
Another way to compute the SVD is from
FT F = (VST UT )(USVT )
= (VST )(UT U)(SVT )
0 ]VT .
= Vdiag[σ 2j , |{z}
(11.78i)
N−M
375
e T is computed to be
Using Eq. (11.78h), V
e T = 12/13 0 −5/13 .
V
0
1
0
The third row of VT is computed to be orthogonal to the two
e T using Gram-Schmidt and is
rows of V
vT3 = 5/13 0 12/13 .
11-8.3 Interpretations of SVD
A. Rotations, Reflections and Scalings
An orthogonal matrix can be interpreted as a rotation and/or a
reflection of coordinate axes. A (2 × 2) orthogonal matrix can
often be put into the form of Rθ defined in Eq. (3.38), which we
repeat here as
sin θ
cos θ
.
(11.79)
Rθ =
− sin θ cos θ
For example, by comparing the entries of U in Eq. (11.78b) with
the entries of Rθ , we ascertain that U is a rotation matrix with
θ = 36.87◦. Similarly, V in Eq. (11.78d) represents a rotation
matrix with θ = 22.62◦.
Hence, the SVD of a matrix F can be interpreted as: (1) a
rotation and/or reflection of axes, followed by (2) scaling of the
rotated axes, followed by (3) another rotation and/or reflection
of axes.
B. Expansion in Orthonormal Vectors
Repeating the earlier argument, V is the matrix of eigenvectors,
and {σ 2j } are the nonzero eigenvalues, of (N × N) matrix FT F.
Note that FT F has N − M zero eigenvalues, since the rank of
FT F is M.
Inserting σ1 into Eq. (11.78i) gives
12337 − 16900
−6084
0
u1 =
,
−6084
8788 − 16900
0
which has the solution (after normalization so that uT1 u1 = 1):
u1 = [4/5, −3/5]T.
Similarly, for σ2 ,
0
12337 − 4225
−6084
u =
0
−6084
8788 − 4225 2
Now we introduce another interpretation of the SVD, which will
prove particularly useful in unsupervised learning. In the sequel,
we assume that M < N, so F is a reclining matrix. Let us define:
(1) fi as the ith column of F, for i = 1, 2, . . . , N:

fi,1
 fi,2 

fi = 
 ..  ,
.
fi,M

(11.80)
(2) ui as the ith column of U, for i = 1, 2, . . . , M:
has the solution (after normalization so that uT2 u2 = 1):
u2 = [3/5, 4/5]T.
www.dbooks.org

ui,1
 ui,2 

ui = 
 ..  ,
.

ui,M
(11.81)
376
CHAPTER 11 IMAGE RECOGNITION
C. Comparison with Orthonormal Functions
and
(3) υi, j as the (i, j)th element of V. It follows that υ j,i is the
(i, j)th element of matrix VT .
With these definitions, the nth column of the equation F = USVT
can be rewritten as
M
fi = ∑ u j (σ j υi, j )
j=1
= ci,1 u1 + ci,2 u2 + · · · + ci,M uM ,
for i = 1, . . . , N,
(11.82)
where ci, j = σ j υi, j . The coefficients {ci, j } are called coordinates of vector fi in the basis {u j } of the M-dimensional
space RM .
For example, applying Eq. (11.82) to the SVD example of
Eq. (11.78) gives
−40
f3 =
= u1 (σ1 υ3,1 ) + u2 (σ2 υ3,2 )
30
4/5
3/5
130(−5/13) +
65(0) . (11.83)
=
−3/5 |
4/5
| {z }
{z
}
−50
j = 1, . . . , M;
i = 1, . . . , N,
(11.84)
where (UT fi ) j is the jth element of the column vector UT fi .
Equation (11.84) provides the ( j, i)th element of either UT F or
SVT .
For example, for i = 3 and j = 1,
−40
σ1 υ3,1 = 4/5 −3/5
= −50;
30
and for i = 3 and j = 2,
−40
= 0.
σ2 υ3,2 = 3/5 4/5
30
Equation (11.82) is the same as the expansion given in
Eq. (7.4) for a function x[i], for all i, in orthonormal functions {φi (t)}. Furthermore, Eq. (11.84) is the linear algebraic
equivalent of the formula given by Eq. (7.6) for computing the
coefficients {xi } in the orthonormal expansion in Eq. (7.4). The
orthonormality of the vectors {ui } in U is the linear algebraic
equivalent of orthonormality of the basis functions {φi [n]} in
Eq. (7.5).
For convenience, we repeat Eqs. (7.4)–(7.6), modified to finite
number N of real-valued basis functions φi [n], coefficients xi and
times n, and C = 1:
N
x[n] = ∑ xi φi [n],
(11.86a)
δ [i − j] = ∑ φi [n] φ j [n],
(11.86b)
i=1
N
0
According to Eq. (11.82) each column of F can be written as
a linear combination of the M columns {ui } of the orthogonal
matrix U, with the coefficients in the linear combination being
{σ j υi, j }. Note that the final N–M rows of VT are not used to
compute F, because they multiply the zero part of the reclining
matrix S.
The coefficients σ j υi, j can be computed using
σ j υi, j = (UT fi ) j ,
The column vectors ui of the orthogonal matrix U are themselves orthonormal vectors. This is because UT U = IM is equivalent to stating
(
1
if i = j,
T
ui u j =
(11.85)
0
if i 6= j.
n=1
and
N
xi = ∑ x[n] φi [n].
(11.86c)
n=1
11-8.4 Dimensionality Reduction
Let us arrange the singular values {σi } to be in decreasing order
σ1 > σ2 > · · · > σM ,
which can be accomplished by reordering the columns of U and
the rows of VT .
Often there is a threshold singular value σT with the following properties:
(1) σ1 > σ2 > · · · > σT +1 > σT and
(2) σT ≫ σT −1 > σT −2 > · · · > σM .
Thus, all singular values with index larger than T are much
smaller than σT , and therefore they can be ignored in
11-9 UNSUPERVISED LEARNING EXAMPLES
377
Eq. (11.82). The truncated SVD becomes
Notation for Unsupervised
Training and Classification
T
fi ≈ ∑ u j (σ j υi, j ).
(11.87)
j=1
The summation in Eq. (11.87) provides a reasonable approximation so long as σT is much larger than singular values
with higher indices. In terms of orthogonal functions, this is
analogous to truncating a Fourier series expansion to T terms.
The resulting sum is often a good approximation to the function.
The significance of Eq. (11.87) is that each column fi of
F can now be approximated well by only T of the M terms
in the orthonormal expansion given by Eq. (11.82). Each fi is
represented using T coefficients:
fi ≈ ci,1 u1 + ci,2 u2 + · · · + ci,T uT ,
(11.88)
which contains only the first T terms in Eq. (11.82).
Replacing the expansion in Eq. (11.82) with the truncated
expansion in Eq. (11.87) is called dimensionality reduction.
This not only reduces the amount of computation, but also
allows visualization of the image classes in a T-D subspace
instead of M-D R M . The dimensionality reduction process is
illustrated through forthcoming examples.
Concept Question 11-6:
Why do we set small singular
f (i) [n, m] = (M × N) ith training image, with class unknown, { i = 1, . . . , I }
I = number of training images
L = number of classes
gobs [n, m] = observed image of unknown class
N ′ = MN = number of pixels per image
f(i) = ith training image vector, generated by column
unwrapping f(i) [n, m] into
[ f (i) [0, 0], . . . , f (i) [0, M − 1], . . ., f (i) [1, 0], . . . ,
f (i) [1, M − 1], . . ., f (i) [N − 1, 0], . . .,
f (i) [N − 1, M − 1]]T
F = [f1 f2 . . . fI ]T = matrix composed of the
columns of the I vectors representing images f (i) [n, m],
{ i = 1, 2, . . . , I }. Size of F is N ′ × I, where N ′ = MN
{ σ j } = singular values of SVD representation
U and V = SVD orthogonal matrices
S = SVD diagonal matrix
values to zero?
11-9 Unsupervised Learning Examples
In unsupervised learning, we are given a set of I training images
{ f (i) [n, m], i = 1, . . . , I}. Even though we have no information
about the contents of the I images, the goal is to partition
the images into L classes { fk [n, m], k = 1, . . . L}, as was done
previously in Sections 11-1 to 11-4, but now determined by the
images themselves. If we are then given an additional image
gobs [n, m], we must provide a simple rule for classifying it into
one of the L classes.
We provide two simple examples to illustrate how SVD is
used in unsupervised learning and classification. In the first
example, the images are intentionally chosen to be small in size
(2 × 2) to allow us to carry out the computations manually. A
total of 5 training images and 1 observation image are involved.
The experience gained from the first example then allows us
to consider a second example comprised of 24 training images,
each (3 × 3) in size, and 1 observation image of the same size.
11-9.1 Unsupervised Learning Example with 5
Training Images
In this simple example, we are given I = 5 training images,
{ f (i) [n, m] }, identified by superscript i with i = 1, 2, . . . , 5. All
5 images are (2 × 2) in size. We also are given an observation
image gobs [n, m]. Our goal is to use the training images to
determine the distinct classes contained in the training images,
using unsupervised learning, and then to classify gobs [n, m] into
one of those classes.
The six (2 × 2) images are given by
1.1 0.1
f (1) [n, m] =
,
(11.89a)
0.1 1.0
0.1 1.0
,
(11.89b)
f (2) [n, m] =
1.0 0.1
1.0 0.1
f (3) [n, m] =
,
(11.89c)
0.0 0.9
www.dbooks.org
378
CHAPTER 11 IMAGE RECOGNITION
0.0 0.9
,
f [n, m] =
1.1 0.1
1.1 1.0
,
f (5) [n, m] =
0.9 1.0
1 1
gobs [n, m] =
.
1 1
(4)
(11.89d)
where u1 and u2 are the first and second columns of U. Similarly,
application of Eq. (11.88) for i = 2, 3, 4, and 5 yields
(11.89e)
f(2) ≈ (σ1 υ2,1 )u1 + (σ2 υ2,2 )u2
= 1.06u1 + 0.94u2,
(11.89f)
A. Training Matrix
We unwrap the f (i) [n, m] by columns as in MATLAB’s F(:)
and assemble the unwrapped images into a (4 × 5) training
matrix F. The ith column of the training matrix F is the
unwrapped vector image f(i) [n, m]:
F = f1 f2 f3 f4 f5


1.1 0.1 1.0 0.0 1.1
0.1 1.0 0.0 1.1 0.9
=
.
(11.90)
0.1 1.0 0.1 0.9 1.0
1.0 0.1 0.9 0.1 1.0
Using the recipe given in Section 11-8.1(B) or the MATLAB
command [U,S,V]=svd(F), we obtain the following matrices:


0.54 −0.52 −0.20 −0.63
0.63 −0.26
0.47 0.56
U=
,
(11.91a)
0.49 0.48 −0.69 0.25 
0.50 −0.44 0.29
0.69


2.94
0
0
0
0
1.86
0
0
0
 0
S=
,
(11.91b)
0
0
0.14
0
0
0
0
0
0.02 0


0.40 −0.49 0.45 −0.63 0.01
0.36 0.51 −0.38 −0.44 −0.53


V = 0.35 −0.46 −0.06 0.54 −0.61 . (11.91c)
0.34 0.54
0.70
0.31 −0.01
0.68 −0.01 −0.39 0.17
0.59
It is apparent from matrix S that both σ1 = 2.94 and σ2 = 1.86
are much larger than σ3 = 0.14 and σ4 = 0.02. Hence, we can
truncate the SVD to T = 2 in Eq. (11.88), which gives
f(1) ≈ u1 (σ1 υ1,1 ) + u2 (σ2 υ1,2 )
= (2.94 × 0.40)u1 + (1.86 × (−0.49))u2
= 1.19u1 − 0.90u2,
(11.92a)
f(3) ≈ (σ1 υ3,1 )u1 + (σ2 υ3,2 )u2
= 1.04u1 − 0.86u2,
f(4) ≈ (σ1 υ4,1 )u1 + (σ2 υ4,2 )u2
= 1.01u1 + 1.00u2,
f
(5)
≈ (σ1 υ5,1 )u1 + (σ2 υ5,2 )u2
= 2.00u1 − 0.23u2.
(11.92b)
(11.92c)
(11.92d)
(11.92e)
All five images assume the form of Eq. (11.88):
f(i) ≈ ci,1 u1 + ci,2 u2 .
With u1 and u2 as orthogonal dimensions, ci,1 and ci,2 are the
coordinates of image f(1) in (u1 , u2 ) space.
B. Subspace Representation
If we regard u1 and u2 in Eq. (11.92) as orthogonal axes, then
f(1) , the dimensionally reduced representation of training image 1, has coordinates (1.19, −0.90), as depicted in Fig. 11-12.
Similar assignments are made to the other four training images.
The five symbols appear to be clustered into three image classes
centered approximately at coordinates { (1, 1), (1, −1), (2, 0) }.
We assign:
Class #1:
Class #2:
Class #3:
to cluster centered at (1, 1),
to cluster centered at (1, −1),
to cluster centered at (2, 0).
Given these three clusters, we divide the (u1 , u2 ) domain into
the three regions shown in Fig. 11-12.
C. Classification of Observation Image
For the observation image defined by Eq. (11.89f), we need to
determine its equivalent coordinates (g1 , g2 ) in (u1 , u2 ) space.
We do so by applying the following recipe:
1. Unwrap gobs [n, m] by columns to form vector g, which in
the present case gives
 
1
1
g =  .
(11.93)
1
1
11-9 UNSUPERVISED LEARNING EXAMPLES
379
11-9.2 Unsupervised Learning Example with 24
Training Images
u2
1.5
1
f (4)
f (2)
Class 1
0.5
g
f (5)
0
−0.5
f (3) f (1)
Class 3
Class 2
−1
1
1.5
2
u1
Figure 11-12 Depiction of 2-D subspace spanned by u1
and u2 . The blue
symbols represent each column of the
training matrix (i.e., each training image f (i) [n, m]. They cluster
represents the observation image
into 3 classes. The red
gobs [n, m].
2. Use the form of Eq. (11.84)—with fi replaced with g and
ci, j = σ j υi, j replaced with gi —to compute coordinates g1 to g4 :
 

 
g1
0.54
0.47
0.49
0.50
1
−0.52
0.56
0.48
−0.44
1
g2 



T
g  = U g = −0.20 0.63 −0.69 0.29  1
3
g4
−0.63 −0.26 0.25
0.69
1


2.0
0.08
=
.
(11.94)
0.03
0.05
3. From the result given by Eq. (11.94), we deduce that the
reduced-coordinate representation of gobs [n, m] in (u1 , u2 ) space
is (g1 , g2 ) = (2.0, 0.08).
4. These coordinates place g in Class 3 (Fig. 11-12).
For new observations with (g1 , g2 ) coordinates, the class
regions in Fig. 11-12 correspond to the following classification
scheme:
(a) Class #1 if g2 > 0 and g2 − g1 > −1.5,
(b) Class #2 if g2 < 0 and g2 + g1 < 1.5,
(c) Class #3 if g1 > 1.5 and g1 − 1.5 > g2 > −(g1 − 1.5).
Now that we understand how to use SVD to truncate and classify
simple (2 × 2) images, let us consider a slightly more elaborate
example involving 24 training images, each (3 × 3) in size. The
images are displayed in Fig. 11-13(a) and the goal is to develop
a classification scheme on the basis of unsupervised learning.
The procedure is summarized as follows:
1. Each of the 24 images is unwrapped by columns into a
(9 × 1) vector.
2. The 24 vectors are assembled into a (9 × 24) training
matrix F.
3. MATLAB is used to compute matrices U, S, and V of the
SVD of F, namely F = USVT .
4. From S, its diagonal elements, corresponding to the singular values σ1 through σ9 , are read off and then plotted to
determine how many dimensions the images can be reduced
to within a reasonable degree of approximation. The plotted
values of σi for the images in Fig. 11-13(a) are shown in
Fig. 11-13(b). Because σ1 and σ2 are much larger than the
remaining singular values, we can approximate the images by
reducing the dimensionality from M = 9 down to T = 2.
5. The reduced images can be represented in terms of coefficients similar to the form in Eq. (11.92). These coefficients become coordinates, leading to the 24 “+” symbols in
Fig. 11-13(c).
6. Based on the cluster of points in Fig. 11-13(c), the (u1 , u2 )
space is divided into three classes. For an observation image
with an observation vector g,
(a) Class #1 if g2 > 0 and g2 − g1 > −2,
(b) Class #2 if g2 < 0 and g2 + g1 < 2,
(c) Class #3 if g1 > 2 and g1 − 2 > g2 > −(g1 − 2).
While the two examples presented in this subsection have served
to illustrate the general approach to unsupervised learning and
classification, the image scenarios were limited to small-size
images, the number of reduced dimensions T was only 2 in both
cases, and the clustering of points was fairly obvious, making
the development of the classification schemes rather obvious and
convenient. How should we approach the general case where the
training images are larger than (3 × 3), the number of reduced
dimensions T is greater than 2, and the points are not clustered
into clearly separate groups? The answer is:
(a) Neither the number of available training images nor their
size has any effect on the SVD procedure outlined earlier
in this subsection. We used small-size images simply for the
convenience of writing their matrices on paper.
www.dbooks.org
380
CHAPTER 11 IMAGE RECOGNITION
T ≥ 4, we can no longer rely on our visual perspective to identify
clusters and define classification regions. We therefore need
mathematical tools to help us cluster the training images into any
specified number of classes and in any number of dimensions,
and to do so in accordance with an objective criterion. One such
tool is the K-Means clustering algorithm and its associated
Voronoi sets.
11-10 K-Means Clustering Algorithm
(a) 24 training images, each (3 × 3) in size
11-10.1 Voronoi Sets
σi
Let us start with a review and update of our notation:
15
10
5
0
1
2
3
4
5
6
7
8
9
i
(b) Singular values σi of the training matrix
u2
I:
(M × N):
f (i) [n, m]:
f(i) :
L:
1.5
N ′ = MN:
1
Class 1
0.5
T:
Number of training images
Size of each image, in pixels
ith image
(N ′ × 1) column vector generated by column
unwrapping image f (i) [n, n] into a column of
length N ′ = MN
Specified number of classes into which the
training images are to be clustered
′
Number of dimensions of the RN space in
which SVD clustering has to occur, assuming
no dimensionality reduction is used
Number of dimensions if dimensionality reduction is justified
0
−0.5
Class 2
Class 3
−1
−1.5
1
1.5
2
2.5
3
3.5
u1
(c) Depiction of 2-D subspace spanned by u1 and u2.
The blue symbols represent individual columns of
the training matrix. They cluster into 3 classes.
Figure 11-13 (a) 24 training images, (b) plot of singular
values, and (c) clusters of training images in (u1 , u2 ) space.
(b) For a reduced number of dimensions T = 2, it is easy to
generate 2-D clusters. For T = 3, we can generate and display
3-D clusters onto a 2-D page, but the scales get distorted. For
For many realistic images, M and N may each be on the order of
1000, and therefore N ′ = MN may be on the order of 106 . Even
if the number of dimensions is reduced to a smaller number T , it
remains much larger than the 3-D space we can manage visually.
The material that follows is applicable for clustering images
using SVD in N ′ -dimensional space, as well as in T -dimensional
space if dimensionality reduction had been applied.
Given I images, defined in terms of their unwrapped column
′
vectors {f(i) , i = 1, 2, . . . , I}, we wish to cluster them in RN
space into L classes pre-specified by centroids at locations {µ p ,
p = 1, 2, . . . , L}. With each image represented by a point in
′
′
RN space, the sets of points obtained by partitioning RN into
regions such that each region consists of the set of points closest
to the specified centroids are called Voronoi sets (or regions),
named for Georgy Voronov. Figure 11-14 shows a 2-D space
divided into 8 Voronov regions, each identified by a centroid.
The boundaries of each region are defined such that every point
location within that region is closer to the centroid of that region
than to any of the other centroids. The distance between two
11-10 K-MEANS CLUSTERING ALGORITHM
381
11-10.2 K-Means Algorithm
μ3
μ4
μ2
μ5
μ6
μ1
μ7
μ8
Figure 11-14
Voronoi regions surrounding centroids µ 1
through µ 8 .
points, whether in 2-D or N ′ -D space, is the Euclidean distance
defined in accordance with Eq. (7.117b). In the present context,
for a test image fi and a centroid µ p of region p, defined by


fi,1
 fi,2 

f(i) = 
 .. 
.
fi,N ′
and
(11.95a)
f(2) = 4u1 + 3u2 + 6u3 + (0)u4 ,
f(5) = 2u1 + (0)u2 + 3u3 + 2u4 ,

µ p,1
 µ p,2 

µp =
 ..  ,
.

f(8) = (0)u1 + (0)u2 + 3u3 + 7u4,
(11.95b)
where u1 to u4 are the column vectors of matrix U, as defined in
(1)
Section 11-8.3, then the new centroid µ 3 is given by
µ p,N ′
the Euclidean distance between them is
" ′
#1/2
The boundaries of the Voronoi regions, such as those in
Fig. 11-14, are established by the locations of the pre-specified
centroids {µ p , p = 1, 2, . . . , L}. In unsupervised learning and
classification, no information is available on the locations of
those centroids, so we need an algorithm to create them. The
K-Means algorithm is a procedure for clustering I points—i.e.,
I images defined in terms of their vectors {f(i) , i = 1, 2, . . . , I}—
′
into L clusters of points in RN , where N ′ is the column length
of the image vectors. We still have to specify the number of
classes L, but not the locations of their centroids. The procedure
is iterative and consists of the following steps:
(0)
1. Choose an initial set of L clustering centroids {µ p ,
(0)
p = 1, 2, . . . , L} at random. The superscript of each µ p denotes
that it is the initial choice (zeroth iteration).
2. Compute the L Voronoi sets (regions) using the I available
image vectors, by measuring the distance between each image
(0)
vector f(i) and each centroid µ p and then assigning f(i) to the
nearest centroid.
3. For each Voronoi set p use the locations of the image
vectors assigned to it to compute their combined centroid and
(1)
label it µ p (first iteration). The centroid of a Voronoi set is the
mean (average) location of the image vectors contained within
that set. For purposes of illustration, if Voronoi set 3 contains
(4 × 1) image vectors f(2) , f(5) , and f(8) , and they are given in
SVD format (Eq. (11.82)) by
1
(1)
µ 3 = [(4 + 2 + 0)u1 + (3 + 0 + 0)u2
3
+ (6 + 3 + 3)u3 + (0 + 2 + 7)u4]
= 2u1 + u2 + 4u3 + 3u4 .
N
ℓi,p =
∑ | fi, j − µ p, j |2
.
(11.96)
j=1
After computing ℓi,p for all L regions, image f(i) is assigned
to the region for which ℓi,p is the smallest. The boundaries in
Fig. 11-14 are established by exercising such an assignment to
′
every point in the space RN .
(1)
Here, µ 3 defines the location of the first iteration of centroid 3.
4. Repeat steps 2 and 3 after incrementing each superscript on
µ p by 1.
5. Continue the iterative process until the algorithm has
converged per a defined criterion regarding the average distance
between centroids of successive iterations.
www.dbooks.org
382
CHAPTER 11 IMAGE RECOGNITION
Example 11-8: K-Means Clustering Algorithm
The K-Means algorithm was applied to the N data points shown
in Fig. 11-15(a), which were generated using two sets of random
points. The algorithm was initialized using the K-Means++
algorithm, which works as follows:
• Choose one data point at random. Call this point x0 , the
centroid for the first cluster.
• Choose another data point xn at random. The probability
that the data point xn is chosen is proportional to the square
distance between xn and x0 . Specifically,
P[xn ] =
||xn − x0||22
.
N
∑i=1 ||xi − x0||22
Thus, points far away from the first centroid x0 are more
likely to be chosen. Point xn is assigned as the centroid for
the second cluster.
Results of different iterations of the K-Means algorithm are
shown in parts (b) through (e) of Fig. 11-15, after which the
algorithm seems to have converged.
The K-Means algorithm was run using the MATLAB command idx=kmeans(X,k) available in MATLAB’s Machine
Learning toolbox. Here X is the (N × 2) array of data point
coordinates, k is the number of clusters (which must be set
ahead of time; here k=2), and idx is the (N × 1) column
vector of indices of clusters to which the data points whose
coordinates are in each row of X are assigned. Each element
of idx is an integer between 1 and k. For example, the data
point whose coordinates are [X(3,1),X(3,2)] is assigned
to cluster #idx(3).
kmeans has many options. For information, use help
kmeans.
Concept Question 11-7:
for?
What is the K-Means algorithm
PROBLEMS
383
3
3
2
2
1
1
0
0
-1
-1
-2
-2
-3
-3
-2
-1
0
1
2
3
Cluster 1
Cluster 2
Centroids
-3
-3
4
-2
(a) Random data points to be classified
Cluster 1
Cluster 2
Centroids
1
2
3
4
1
0
0
-1
-1
-2
-2
-2
3
4
Cluster 1
Cluster 2
Centroids
2
1
-3
-3
0
3
3
2
-1
(b) Results of first iteration
-1
0
1
2
3
-3
-3
4
-2
(c) Results of second iteration
-1
0
1
2
(d) Results of third iteration
3
2
Cluster 1
Cluster 2
Centroids
1
0
-1
-2
-3
-3
-2
-1
0
1
2
3
4
(e) Results of fourth iteration
Figure 11-15 (a) Random data points and results of K-means clustering after (b) 1 iteration, (c) 2 iterations, (d) 3 iterations, and (e) 4
iterations.
www.dbooks.org
384
CHAPTER 11 IMAGE RECOGNITION
Summary
Concepts
• Classification is often performed by choosing the image
with the largest correlation with the observed image.
• MLE classification is performed by subtracting each image energy from double the correlation with the observed
image.
• MAP classification is performed by subtracting each
image energy from double the correlation with the observed image, and adding the logarithm of the a priori
probability of the image times double the noise variance.
• Classification of an image with an unknown shift can be
performed using cross-correlation instead of correlation.
Classification of an image with an unknown scaling can
be performed by cross-correlation on logarithmic spatial
scaling.
• Jointly Gaussian color images can be classified using
labelled training images to estimate the mean and covariance matrix of each image class, and using these
estimates in vector versions of the MLE and MAP
classifiers.
• Unsupervised learning can be performed using unlabeled
training images, by unwrapping each image into a column vector, assembling these into a training matrix,
computing the SVD of the training matrix, setting small
singular values to zero, and regarding the orthonormal
vectors associated with the remaining singular values as
subspace bases. Each training image is a point in this
subspace. The points can be clustered using the K-Means
algorithm, if necessary.
Mathematical Formulae
Correlation
Singular value decomposition (SVD)
F = USVT
N−1 M−1
ρk = ∑ ∑ gobs [n, m] fk [n, m]
Expansion in orthonormal vectors
n=0 m=0
T
MAP classifier
ΛMAP [k] = 2ρk − E fk + 2σv2 log p[k]
fi = ∑ u j (σ j vi, j )
j=1
Cross-correlation for unknown shift
ρk [n, m] = gobs [n, m] ∗ ∗ fk [−n, −m]
Important Terms
correlation
cross-correlation
Coefficients of orthonormal vectors
ci, j = σ j vi, j = (UT fi ) j
Provide definitions or explain the meaning of the following terms:
dimensionality reduction
K-Means algorithm
orthonormal vectors
recognition
PROBLEMS
training matrix
unsupervised learning
ρ̃k = ∑ ∑ g̃obs [n, m] f˜k [n, m], where
Section 11-1: Image Classification by Correlation
11.1 A problem with using correlation ρk (defined in Eq.
(11.2)) to compare an observed image gobs [n, m] with each one
of a set of reference images { fk [n, m]} is that the fk [n, m] with
the largest pixel values tends to have the largest ρk . We can
avoid this problem as follows. Define the correlation coefficient
subspace
SVD
and
gobs [n, m]
g̃obs [n, m] = q
∑ ∑ g2obs [n, m]
fk [n, m]
.
f˜k [n, m] = q
∑ ∑ fk2 [n, m]
PROBLEMS
385
2 5
,
f2 [n, m] =
1 7
1 2
,
f3 [n, m] =
5 7
(a) Expand 0 ≤ ∑ ∑(g̃obs [n, m] ± f˜k [n, m])2 and show that
|ρ̃k | ≤ 1.
(b) If gobs [n, m] = a fk [n, m] for any constant a > 0, show that
ρ̃k = 1.
11.2 In Section 11-2, let the reference images { fk [n, m]} all be
energy-normalized to
fk [n, m]
f˜k [n, m] = q
,
∑ ∑ fk2 [n, m]
and the observed image be g[n, m] = f˜k [n, m] + v[n, m], where
v[n, m] is a zero-mean white Gaussian noise random field with
variance σv2 . This is the problem considered in Section 11-2,
except with energy-normalized reference images. Show that the
MLE classifier is now the value of k that maximizes the correlation coefficient ρ̃k = ∑ ∑ g̃obs [n, m] f˜k [n, m] between g[n, m] and
fk [n, m].
Section 11-3: Classification by MAP
into one of the three classes
2
,
4
3
,
1
1
,
4
using
(a) MLE classification;
(b) MAP classification with a priori probabilities p[1] = 0.1,
p[2] = 0.3, p[3] = 0.6. The noise variance is σv2 = 10.
11.4 Determine a rule for classifying the image
g
g1,0
gobs [n, m] = 0,0
g0,1 g1,1
into one of the three classes
f1 [n, m] =
7
5
1
,
2
Section 11-4: Classification of Spatially Shifted
Images
11.5 Download text.jpg and text.m from the book website. These were used in Example 11-2 to find all occurrences
of “e” in text.jpg. Modify text.m to find all occurrences
of “a” in text.jpg. You will need to (i) find a subset of the
image containing “a” instead of “e” and (ii) change the threshold
using trial-and-error. Produce a figure like Fig. 11-11(c).
11.6 Download text.jpg and text.m from the book website. These were used in Example 11-2 to find all occurrences of
“e” in text.jpg. Modify text.m to find all occurrences of
“m” in text.jpg. You will need to (i) find a subset of the image containing “m” instead of “e” and (ii) change the threshold
using trial and error. Produce a figure like Fig. 11-11(c).
11.3 Determine a rule for classifying the image
g0,0 g1,0
gobs [n, m] =
g0,1 g1,1
1
f1 [n, m] =
3
4
f2 [n, m] =
2
3
f3 [n, m] =
2
using
(a) MLE classification;
(b) MAP classification with a priori probabilities p[1] = 0.1,
p[2] = 0.3, p[3] = 0.6. The noise variance is σv2 = 10.
11.7 To machine-read numerals, such as the routing and
account numbers on the check in Fig. 11-1, it is necessary
to segment (chop up) the number into individual numerals,
and then use one of the methods of Sections 11-1 through
11-3 to identify each individual numeral. Segmentation is often
performed by the hardware (camera) used to read the number.
The program bank1.m reads the image of the bank font
numerals in Fig. 11-2(a), segments the image into (30 × 18) images of each numeral, reassembles the images of numerals back
into the sequence of numerals in Fig. 11-2(a), and machinereads the result, using correlation, into a sequence of numbers
that represent the numerals in Fig. 11-2(a).
Modify bank1.m so that it machine reads the account
number in Fig. 11-1. This requires assembling the images of
numerals into the sequence of numerals in the account number
in Fig. 11-1. Plot the computed numerals in a stem plot. Figure
11-2(a) is stored in bank.png.
11.8 To machine-read numerals, such as the routing and
account numbers on the check in Fig. 11-1, it is necessary
to segment (chop up) the number into individual numerals,
and then use one of the methods of Sections 11-1 through
www.dbooks.org
386
11-3 to identify each individual numeral. Segmentation is often
performed by the hardware (camera) used to read the number.
The program bank1.m reads the image of the bank font
numerals in Fig. 11-2(a), segments the image into (30 × 18) images of each numeral, reassembles the images of numerals back
into the sequence of numerals in Fig. 11-2(a), and machinereads the result, using correlation, into a sequence of numbers
that represent the numerals in Fig. 11-2(a).
Modify bank1.m so that it machine-reads the routing number in Fig. 11-1. This requires assembling the images of numerals into the sequence of numerals in the routing number in
Fig. 11-1. Plot the computed numerals in a stem plot. Figure
11-2(a) is stored in bank.png.
Section 11-5: Classification of Spatially Scaled
Images
11.9 To machine-read numerals, such as the routing and
account numbers on the check in Fig. 11-1, it is necessary
to segment (chop up) the number into individual numerals,
and then use one of the methods of Sections 11-1 through
11-3 to identify each individual numeral. Segmentation is often
performed by the hardware (camera) used to read the number.
But the numbers may have a different size from the stored
numerals, in which case the numbers are spatially scaled.
The program scale1.m reads the image of the bank font
numerals in Fig. 11-2(a), segments the image into (30 × 18)
images of each numeral, decimates each numeral by (2 × 2),
reassembles the images of decimated numerals into the sequence
of numerals in Fig. 11-2(a), uses logarithmic spatial scaling
to transform the problem into a shifted images problem, and
machine-reads the result, using correlation, into a sequence of
numbers that represent the numerals in Fig. 11-2(a)
Modify scale1.m so that it machine-reads the account
number in Fig. 11-1. This requires assembling the images of
numerals into the sequence of numerals in the account number
in Fig. 11-1. Plot the computed numerals in a stem plot. Figure
11-2(a) is stored in bank.png.
11.10 To machine-read numerals, such as the routing and
account numbers on the check in Fig. 11-1, it is necessary
to segment (chop up) the number into individual numerals,
and then use one of the methods of Sections 11-1 through
11-3 to identify each individual numeral. Segmentation is often
performed by the hardware (camera) used to read the number.
But the numbers may have a different size from the stored
numerals, in which case the numbers are spatially scaled.
The program scale1.m reads the image of the bank font
numerals in Fig. 11-2(a), segments the image into (30 × 18)
CHAPTER 11 IMAGE RECOGNITION
images of each numeral, decimates each numeral by (2 × 2),
reassembles the images of decimated numerals into the sequence
of numerals in Fig. 11-2(a), uses logarithmic spatial scaling
to transform the problem into a shifted images problem, and
machine-reads the result, using correlation, into a sequence of
numbers that represent the numerals in Fig. 11-2(a)
Modify scale1.m so that it machine-reads the routing
number in Fig. 11-1. This requires assembling the images of
numerals into the sequence of numerals in the routing number
in Fig. 11-1. Plot the computed numerals in a stem plot. Figure
11-2(a) is stored in bank.png.
Section 11-7: Classification of Color Images
11.11 Show that if there is no randomness in image color
components, i.e., Kk = 0, that Eq. (11.61) for MAP classification
of color images reduces to Eq. (11.13) for MAP classification of
grayscale images summed over all three color components.
11.12 Determine a rule for classifying the (2 × 2) color image
g
g0,1,R
gobs,R = 0,0,R
,
g1,0,R g1,1,R
g
g0,1,G
gobs,G = 0,0,G
,
g1,0,G g1,1,G
g
g0,1,B
gobs,B = 0,0,B
,
g1,0,B g1,1,B
into one of the two (2 × 2) color image classes
1 2
5 6
9
f1,R =
, f1,G =
, f1,B =
3 4
7 8
2
3
8 7
4 3
, f2,B =
, f2,G =
f2,R =
1
6 5
2 1
1
,
3
2
,
9
using (assume K1 = K2 = I throughout)
(a) MLE classification;
(b) MAP classification with a priori probabilities p[1] = 0.2,
p[2] = 0.8, and σv2 = 0.
11.13 Determine a rule for classifying the (2 × 2) color image
g
g0,1,R
gobs,R = 0,0,R
,
g1,0,R g1,1,R
g
g0,1,G
gobs,G = 0,0,G
,
g1,0,G g1,1,G
PROBLEMS
387
g
gobs,B = 0,0,B
g1,0,B
g0,1,B
,
g1,1,B
Section 11-8: Unsupervised Learning and
Classification
into one of the two (2 × 2) color image classes
7 1
8 6
9
f1,R =
, f1,G =
, f1,B =
2 4
4 2
5
4 2
2 4
3
f2,R =
, f2,G =
, f2,B =
1 7
6 8
7
7
,
3
5
,
9
11.16 Show that the approximations in the truncated expansions of Eq. (11.92)(a-e) do a good job of approximating the
columns of the training matrix F in Eq. (11.90).
Section 11-9: Unsupervised Learning Examples
using (assume K1 = K2 = I throughout)
(a) MLE classification;
(b) MAP classification with a priori probabilities p[1] = 0.2,
p[2] = 0.8, and σv2 = 0.
11.14 Following Example 11-5, determine an MLE rule for
classifying a (1 × 1) color image [gR , gG , gB ]T into two classes,
determined from the following training images:
For class #1:
 
 
 
 
0
0
8
8
f11 = 0 , f21 = 8 , f31 = 8 , f41 = 8 .
8
8
0
8
For class #2:
 
4
f12 =  4  ,
12
 
4
f22 = 12 ,
12
 
12
f32 = 12 ,
4
 
12
f42 = 12 .
12
Assume no additive noise and equal a priori probabilities
p[1] = p[2] = 0.5.
11.15 Following Example 11-5, determine an MLE rule for
classifying a (1 × 1) color image [gR , gG , gB ]T into two classes,
determined from the following training images:
For class #1:
 
 
 
 
1
5
3
3
f11 = 3 , f21 = 3 , f31 = 5 , f41 = 1 .
5
1
7
5
For class #2:
 
2
f12 = 4 ,
6
 
6
f22 = 4 ,
2
 
4
f32 = 6 ,
8
 
4
f42 = 2 .
6
Assume no additive noise and equal a priori probabilities
p[1] = p[2] = 0.5.
11.17 We are given the eight (2 × 2) training images
1.1 0.1
0.9 0.0
(1)
(2)
f =
, f =
,
0.1 1.1
0.1 0.9
0.9 0.0
1.1 0.1
,
, f(4) =
f(3) =
0.0 0.1
0.0 0.9
0.0 0.9
0.1 1.1
,
, f(6) =
f(5) =
0.9 0.1
1.1 0.1
0.1 1.1
0.0 0.9
f(7) =
, f(8) =
.
0.9 0.0
1.1 0.0
Use unsupervised learning to determine classes for these eight
training images, and a rule for classifying the observed image
0.0 1.1
gobs =
.
1.1 0.0
11.18 We are given the eight (2 × 2) training images
1.1 0.0
0.9 0.1
(2)
(1)
,
, f =
f =
1.1 0.1
0.9 0.1
0.9 0.1
1.1 0.0
f(3) =
, f(4) =
,
1.1 0.0
0.9 0.0
0.0 1.1
0.1 0.9
,
, f(6) =
f(5) =
0.1 1.1
0.1 0.9
0.0 1.1
0.1 0.9
.
, f(8) =
f(7) =
0.0 0.9
0.0 1.1
Use unsupervised learning to determine classes for these eight
training images, and a rule for classifying the observed image
0.0 1.1
.
gobs =
0.0 1.1
www.dbooks.org
388
CHAPTER 11 IMAGE RECOGNITION
Section 11-10: Clustering
(i)
(i)
11.19 We are given a set of N points {( f1 , f2 ), i = 1, . . . , N}
in the plane R 2 . Prove that the point ( f1 , f2 ) minimizing
N
(i)
(i)
∑ ||( f1 , f2 ) − ( f1, f2 )||22
i=1
(i)
(i)
is the centroid of the N points {( f1 , f2 ), i = 1, . . . , N}. This is
how centroids are determined in the K-Means algorithm. Note
that the quantity to be minimized is the sum of squared distances
from the points to the centroid, not the sum of distances.
Chapter
12
Supervised
Learning
and
12
Classification
Contents
12-1
12-2
12-3
12-4
Overview, 390
Overview of Neural Networks, 390
Training Neural Networks, 396
Derivation of Backpropagation, 403
Neural Network Training Examples, 404
Problems, 408
0
1
2
3
4
784 terminals
5
6
Objectives
7
Learn to:
9
8
■ Recognize the difference between unsupervised and
supervised learning.
■ Use a perceptron to implement classification with a
single separating hyperplane.
■ Use a hidden layer of perceptrons to implement more
complicated classification.
■ Train a neural network using backpropagation.
www.dbooks.org
Input terminals
Hidden layer
Output layer
Neural networks and machine learning have become
important tools in signal and image processing and
computer science. No book on image or signal
processing is complete without some coverage of
these, which have revolutionized society. A neural
network can be trained, using sets of labeled
training images, to recognize digits or faces. This
chapter presents a very brief introduction to neural
networks for image classification.
Overview
Having covered image classification algorithms based on unsupervised training in the preceding chapter, we now switch
our attention to algorithms that rely on supervised training
instead. To clearly distinguish between the structures of the two
approaches, we begin this overview with a short summary of the
former and an overview of the latter.
A. Summary of Unsupervised Learning
In unsupervised learning and classification, we are given a set
of I training images, each containing an image of one class (such
as a numeral or a letter) from among L possible classes, but
the identities of the image classes are not known to us. The
goal is to cluster the I images into L domains and to develop
a classification scheme for assigning new observation images to
the correct class.
Each (M × N) training image is unwrapped by columns to
form a column vector of length N ′ = MN. The column vectors,
denoted f(i) with {i = 1, 2, . . . , I}, are stacked alongside each
other to form the (N ′ × I) training matrix F. After computing the
SVD of matrix F, the dimensionality of vectors f(i) is reduced
from length N ′ to a truncated length T by setting small-valued
singular values σi to zero. The reduced-dimensionality image
vectors f(i) are each expressed as the sum of unit vectors {u j ,
i = 1, 2, . . . , T } with coefficients (coordinates) ci, j as expressed
by Eq. (11.92). The coordinates define the location of f(i) in
RT space. Next, Voronoi domains are established by clustering
the I images into L classes using the K-Means algorithm. Finally, boundaries are established between the Voronoi domains,
thereby creating a scheme for classifying new observation images.
B. Overview of Supervised Learning
In supervised learning, we are given a set of I training images,
just as in unsupervised learning, except that now we do know
(i)
the class of each of the training images. Each training image fk
is said to be labeled with its proper image class k, so there is
no need to determine the image classes, which we needed to do
with unsupervised learning. Labeled training images of object
classes, such as faces, are readily available from data collected
from the internet. The goal of supervised training is to use the
labeled training images to develop an algorithm for classifying
new observation images.
The classification algorithm is implemented using a networked (connected) set of simple processors called perceptrons
or neurons. The network of processors is called a neural network. The act of using the labeled training images to compute
the weights (gains) in the neural network is called training the
neural network. The trained neural network accepts as input the
pixel values of a new image to be classified, and outputs the class
to which the new image belongs.
A good example of a supervised learning problem is optical
character recognition of handwritten zip codes. A zip code
is a sequence of five or nine digits used to identify letter
and package destinations for the U.S. Postal System (USPS).
But handwritten zip codes seldom resemble specific images
of digits, because people write digits in different ways and
styles. Supervised learning can be used to recognize handwritten
zip codes by using as training images a labeled database of
handwritten digits maintained by the U.S. National Institute of
Standards and Technology (NIST) and labeled by the actual
digit each handwritten digit represents, as identified by a human
judge. Supervised learning has been applied to optical character
recognition of handwritten Chinese symbols (with a training
image size of 3 GB). We discuss the zip code application further
in Section 12-4.
What is the difference between
supervised and unsupervised learning?
Concept Question 12-1:
12-1 Overview of Neural Networks
A neural network is a network of simple processors called
neurons, as they are perceived to mimic the action of neurons
in a brain. Neural networks are also called multilayered perceptrons.
The idea behind neural networks, when they were originally
developed in the 1950s, was that a neural network mimics the
communication structure in the brain, which consists of about
100 billion neurons, each connected to about one thousand of
its neighbors in an elaborate network. It was thought that such
a network of processors might mimic a brain in its ability to
identify images and objects on the basis of previous exposure
to similar images and objects (learning). The analogy to brain
function is now viewed as “tenuous.”
The major advantage of using trained neural networks for
image classification is that no modeling or understanding of the
image classification problem is necessary. The way in which
the trained neural network works need not be (and seldom
is) understandable to humans. In the remainder of the present
section, we introduce how perceptrons work and how multiple
perceptrons are connected together to form a neural network.
390
12-1 OVERVIEW OF NEURAL NETWORKS
x1
Weights
w1
1
w0
x2
w2
391
1
Out
Σ
0
Weighted
sum
Inputs
Activation
function ϕ(∙)
Perceptron
(a)
x1
Weights
w1
x2
w2
x3
w3
1
w0
x4
w4
x5
w5
x6
w6
Inputs
1
Out
Σ
0
Weighted
sum
Activation
function ϕ(∙)
Perceptron
(b)
Figure 12-1 Perceptrons with (a) 2 inputs and (b) 6 inputs.
Then, in Section 12-2, we discuss how neural networks are
trained, using the labeled training images. An example follows
in Section 12-4.
A. Components of a Perceptron
12-1.1 Perceptrons
Figure 12-1(a) shows a perceptron with N = 2 inputs. The
added constant w0 usually is viewed as another weight multiplying a constant input of 1.
In another example, a perceptron with six inputs is shown in
Fig. 12-1(b).
A perceptron with N inputs {xn , n = 1, . . . , N} is implemented
by the formula
A perceptron is a simple processor that accepts a set of N
numbers {xn , n = 1, . . . , N} as inputs, multiplies each input xn by
a weight wn , adds a constant w0 to the weighted sum of inputs,
feeds the result into an activation function φ (x) that mimics
a step function, and outputs a number y that is close to either 0
or 1. The process mimics a biological neuron, which “fires” only
if the weighted sum of its inputs exceeds a certain threshold.
Synapses in the brain (which connect neurons) are biological
analogues of the weights in a perceptron.
y = φ (w0 + w1 x1 + w2 x2 + · · · + wN xN ).
(12.1)
The activation function φ (x) is commonly chosen to be (for
www.dbooks.org
392
CHAPTER 12 SUPERVISED LEARNING AND CLASSIFICATION
ϕ(x)
g0,0
12
g1,0
10
1
−46
g0,1
−2
g1,1
−8
1
0.8
1
0.6
0.4
0.2
0
−6
−4
−2
0
2
4
6
x
Figure 12-2 Activation function φ (x) for a = 1 (blue) and
gG
4
gB
4
1
−8
0
ϕ(∙)
gR
−4
1
(12.2)
This choice of activation function is plotted in Fig. 12-2 for
a = 1 and a = 4. The similarity to a step function is evident.
The rationale for choosing this particular mathematical representation for the activation function is supported by two
attributes: (1) its resemblance to a step function, and (2) its
derivative has the simple form
dφ
= a φ (x) (1 − φ (x)).
dx
y
(a) Perceptron for Example 11-1
a = 4 (red). The larger the value of a, the more closely φ (x)
resembles a step function.
some activation constant a > 0) the sigmoid function
(
1
1
if x > 0,
≈
φ (x) =
1 + e−ax
0
if x < 0.
Σ
Σ
y
0
ϕ(∙)
(b) Perceptron for Example 11-5
Figure 12-3 Perceptrons for implementing the classification
rules of Examples 11-1 and 11-5.
(12.3)
The derivative of the activation function is used in Section 12-2
for training the neural network.
Note that to implement a simple linear combination of the
perceptron’s inputs, the activation function should be chosen
such that φ (x) = x.
B. Classification Using a Single Perceptron
Another reason for using perceptrons is that single-stage classification algorithms can each be implemented using a single
perceptron.
By way of example, let us consider the classification rules we
derived earlier in Examples 11-1 and 11-5. The classification
rule given by Eq. (11.19) is:
Choose f1 : if 12g0,0 + 10g1,0 − 2g0,1 − 8g1,1 − 46 < 0,
Choose f2 : if 12g0,0 + 10g1,0 − 2g0,1 − 8g1,1 − 46 > 0.
This rule can be implemented using the perceptron in
Fig. 12-3(a) with φ (x) set equal to the step function u(x). If
output y = 1, choose f2 , and if y = 0, choose f1 .
Similarly, consider the classification rule given in Example
11-5, which stated:
Choose f1 : if 4gG + 4gB − 4gR − 8 < 0,
Choose f2 : if 4gG + 4gB − 4gR − 8 > 0.
The rule can be implemented using the perceptron in Fig.
12-3(b). If output y = 1, choose f2 , and if y = 0, choose f1 .
Exercise 12-1: Use a perceptron to implement a digital OR
logic gate (y = x1 + x2 , except 1 + 1 = 1).
Use Fig. 12-1(a) with φ (x) ≈ u(x), weights
w1 = w2 = 1, and any w0 such that −1 < w0 < 0.
Answer:
12-1 OVERVIEW OF NEURAL NETWORKS
393
Exercise 12-2: Use a perceptron to implement a digital
AND logic gate (y = x1 x2 ).
Use Fig. 12-1(a) with φ (x) ≈ u(x), weights
w1 = w2 = 1, and any w0 such that −2 < w0 < −1.
Answer:
12-1.2 Networks of Perceptrons
A. Background
Networks of perceptrons, in which the output of each perceptron
is fanned out and serves as an input into many other perceptrons,
are called neural networks, an example of which is shown in
Fig. 12-4.
The leftmost vertical stack of squares is called the input
layer (of terminals). Each square is a terminal that accepts one
number as an input and fans it out to the perceptrons in the
hidden layer to its immediate right. The number of terminals
in the input layer equals the number of pixels in the image.
Terminals are also called nodes, and they often are depicted
using circles, making them easy to confuse with perceptrons
(which they are not). To distinguish the terminals in the input
layer from the perceptrons in succeeding layers, we depict the
terminals in Fig. 12-4 as squares instead of circles.
The rightmost vertical stack of perceptrons is called the
output layer. Each circle is now a perceptron, which includes
weights and an activation function, which are not shown explicitly in Fig. 12-4. The number of perceptrons in the output
layer is usually the number of image classes. The output of the
neural network is usually close to 1 for one of the perceptrons
in the output layer, and close to zero for all other perceptrons
in the output layer. The output-layer perceptron with an output
of 1 identifies the classification of the image. In terms of the
formulation in Section 11-1, the Kth perceptron in the output
layer outputting 1 is analogous to K maximizing the loglikelihood function. There are also other possibilities; the output
layer may output ones and zeros that constitute bits in a binary
representation of integer K.
The vertical stacks of perceptrons between the input and
output layers are called hidden layers (of perceptrons). The
hidden layers greatly increase the ability of the neural network
to perform classification, partly because there are more weights
and partly because the neural network has a more complicated
structure, including more activation functions. The activation
functions allow classification regions that are more complicated
than those we have seen so far.
An underlying rationale for the validity of neural networks
is the Universal Approximation Theorem, which states that
any “reasonable” nonlinear function f (x1 , x2 , . . . , xN ) can be
approximated as
!
M
N
f (x1 , x2 , . . . , xN ) ≈ ∑ ci φ
∑ wi, j x j ,
i=1
a < xn < b, (12.4)
j=1
for some constants N, M, ci , wi, j , a, b and some nonlinear function φ (·). There are no specific rules for the number of hidden
layers or number of perceptrons in each hidden layer. A very
rough rule of thumb is that the total number of perceptrons in
hidden layers should be one-fourth the sum of the numbers of
inputs and outputs.
B. Deep-Learning Neural Networks
Deep-learning neural networks may have dozens of hidden
layers, with each layer consisting of hundreds of perceptrons,
and requiring teraflops of computation. A trained deep-learning
neural network may be able to perform surprising feats of
classification, such as spam filtering and face recognition. Much
of the research on neural networks is focused on the use of huge
Inputs
x1
x2
x3
x4
Outputs
y1
y2
y3
y4
Layer 2
Layer 1
Layer 3
Layer 0
Layer 4
Hidden layers
Input layer
Output layer
Σ
1
0
Figure 12-4 A neural network with 4 inputs and 4 outputs.
Each blue circle in the hidden and output layers is a perceptron
with associated weights and an activation function (see enlarged
circle).
www.dbooks.org
394
CHAPTER 12 SUPERVISED LEARNING AND CLASSIFICATION
neural networks for deep learning.
Deep-learning neural networks became practical around 2009
when it became possible to construct them using graphical
processing units (GPUs). A GPU is a parallel-processing chip
capable of performing simple computations (like those in a
perceptron) in parallel in thousands of parallel computing cores.
They were developed for creating images for computer games
and similar applications that required computations for rotation
of vertices in 3-D images. An example of a GPU is the NVIDIA
GeForce GTX1080 video card. The applicability of GPUs to
neural networks should be evident.
E. Classification Using Multiple Hyperplanes
Let us consider the XOR (exclusive OR) logic gate defined by
C. Examples of Tiny Neural Networks
For practical applications, even the smallest neural networks
have thousands of perceptrons. For example, a neural network
for reading handwritten zip codes reads each digit using a
camera that generates a (28×28) image of each digit. Hence, the
input layer of the neural network has N ′ = 282 = 784 terminals,
which is too large to display as a figure. The output layer has
L = 10 perceptrons (one for each possible output {0, 1, . . . , 9}).
There is usually one hidden layer with about 15 perceptrons.
Hence, in the forthcoming examples, we limit our discussion
to tiny networks designed to demonstrate how neural networks
operate, even though their sizes are very small.
D. Classification with a Single Separating
Hyperplane
◮ A single perceptron can be used to classify images in
binary (L = 2 classes) classification problems in which the
regions are segregated by a separating hyperplane. ◭
x1
x2
y
(12.5)
) are the coordinates of a point in RN and
where (x1 , x2 , . . . , xN
{a1, a2 , . . . , aN } and b are constants. For N = 2, Eq. (12.5)
becomes
b a1
x2 =
−
x1 ,
a2 a2
which is a line with slope −a1 /a2 . For N = 3, Eq. (12.5)
becomes a plane like the one in Fig. 11-11. A separating
hyperplane divides RN into two classification regions, one for
each image class.
0
0
0
0
1
1
1
0
1
1
1 .
0
The classification regions for the XOR gate are shown in
Fig. 12-5, which include two hyperplanes.
The y = 1 region is not contiguous, so a single perceptron will
not be able to classify the input pair (x1 , x2 ) correctly. Instead,
we use three perceptrons, connected as shown in Fig. 12-5(b).
With the activation functions φ (·) of all three perceptrons set
to act like step functions u(x), the lower perceptron in the hidden
layer of Fig. 12-5(b) computes φ (x2 − x1 − 1), whose value is 1
in the region above the line x2 − x1 = 1 (hyperplane 1 in R2 ).
Similarly, the upper perceptron in the hidden layer computes
φ (x1 − x2 − 1), whose value is 1 in the lower right region below
the line x1 − x2 = 1 (hyperplane 2 in R2 ).
The rightmost perceptron (in the output layer) implements an
OR logic gate operation. The output is y = 1 if either of its inputs
is 1, and 0 if neither of its inputs is 1. The −0.5 could be replaced
by any number b such that −1 < b < 0.
Example 12-1:
A hyperplane in RN is a surface of the form
a1 x1 + a2 x2 + · · · + aN xN = b,
Examples 11-1 and 11-5 are examples of binary classification problems with a separating hyperplane. Classification
was accomplished using the single perceptrons in Figs. 12-3(a)
and (b), respectively. The AND and OR gates in Exercises 12-1
and 12-2 are two other examples of binary classification.
Many classification problems do not have separating hyperplanes separating classification regions. The classification
regions are Voronoi sets separated by multiple hyperplanes, or
more complicated curved boundaries.
Neural Network Classifier
Implement a neural network classifier for the classification rule
developed in Section 11-9.1.
Solution: In Section 11-9.1, we illustrated how to develop an
unsupervised classification rule for an unknown (2 × 2) image
g[0, 0] g[1, 0]
,
(12.6)
g[n, m] =
g[0, 1] g[1, 1]
given five (2 × 2) images { f (i) [n, m], i = 1, 2, . . . , 5} with unknown classes. The procedure led to coordinates g1 and g2 given
12-1 OVERVIEW OF NEURAL NETWORKS
395
x2
Hyperplane 1:
x2 − x1 = 1
y=1
y=0
1
−1
1
x1
Hyperplane 2:
x1 − x2 = 1
and U is a (4 × 4) matrix computed from the five given images
using the SVD method outlined in Section 11-9.1. In Eq. (12.7),
(UT g)i is the ith element of the column vector (UT g). From
Eq. (11.91a), the transpose of U is given by


0.54
0.47
0.49
0.50
0.48 −0.44
−0.52 0.56
UT = 
.
(12.9)
−0.20 0.63 −0.69 0.29 
−0.63 −0.26 0.25
0.69
Using Eqs. (12.8) and (12.9) in Eq. (12.7) gives
y = 0 −1
y=1
g1 = 0.54 g[0, 0] + 0.47 g[0, 1]
+ 0.49 g[1, 0] + 0.50 g[1, 1],
g2 = −0.52 g[0, 0] + 0.56 g[0, 1]
+ 0.48 g[1, 0] − 0.44 g[1, 1].
(a) Classification regions for XOR gate
−1
x1
u(x1 − x2 − 1)
(1) Class #1 if: g2 > 0 and g2 − g1 > −1.5,
(12.11a)
(2) Class #2 if: g2 < 0 and g2 + g1 < 1.5,
(12.11b)
(3) Class #3 if: g1 > 1.5 and g1 − 1.5 > g2 > −(g1 − 1.5).
(12.11c)
−
u(∙)
−0.5
y
To construct a neural network with the appropriate perceptrons,
it is useful to cast the classification rules in the form of AND
or OR statements. To that end, we introduce variables a1 , a2 ,
and a3 , and we use activation functions that mimic step functions
u(·):
−
u(∙)
u(x2 − x1 − 1)
−1
a1 = φ (g2 − g1 + 1.5)
(b) Neural network implementation of XOR gate
≈ u(g2 − g1 + 1.5) =
Figure 12-5 Classification regions and neural network for
XOR gate. Here, φ (·) ≈ u(·) is a step function.
a2 = φ (g2 ) ≈
(
1
0
by
g1 = (UT g)1
(12.7a)
g2 = (UT g)2 ,
(12.7b)
where g is the vector equivalent of unwrapped g[n, m], namely
g = [g[0, 0], g[0, 1], g[1, 0], g[1, 1]]T,
(12.8)
(
1
0
if g2 > 0,
if g2 < 0,
and
and
(12.10b)
The solution in Section 11-9.1 led to the existence of three
classes with the following rule:
u(∙)
x2
(12.10a)
a3 = φ (g2 + g1 − 1.5) ≈
(
1
0
if (g2 − g1 + 1.5) > 0,
if (g2 − g1 + 1.5) < 0,
(12.12a)
(12.12b)
if g2 + g1 − 1.5 > 0,
if g2 + g1 − 1.5 < 0.
(12.12c)
The steps required for realizing a1 to a3 are shown in Fig. 12-6.
The classification rule for Class #1, as given by Eq. (12.11a),
requires that both a1 AND a2 be 1, which can be realized by
www.dbooks.org
396
g[0,0]
CHAPTER 12 SUPERVISED LEARNING AND CLASSIFICATION
0.54
g1
0.47
g[0,1]
0.49
1.5
g2 − g1 + 1.5
−
a1
1
0.50
−1.5
0
a4
−0.52
g[1,0]
g2
0.56
g2
g[1,1]
0
a5
−0.44
g2 + g1 − 1.5
a3
1
y1
0
a2
1
0.48
1
1
y2
0
−0.5
0
−1.5
Figure 12-6 Neural network for Example 12-1.
combining them to obtain
a4 = a1 + a2 − 1.5,
◮ For Class #2, the classification rule is:
(12.13a)
and then subjecting a4 to a final activation-function operation
φ (a4 ):
(12.13b)
y1 = φ (a4 ) = φ (a1 + a2 − 1.5).
The AND table shown in Fig. 12-6 confirms that output y1 = 1
only when both a1 AND a2 are 1.
If y2 = 0, choose Class #2.
For class #3, the classification rule is straightforward:
If y1 = 0 and y2 = 1, then g[n, m] does not belong to either
Class #1 or Class #2, and therefore it has to belong to Class #3.
What is the difference between
a perceptron and a neural network?
Concept Question 12-2:
◮ For Class #1, the classification rule is:
If y1 = 1, choose Class #1.
Next, we consider the rule for Class #2, as defined by
Eq. (12.11b), which also is an AND statement, as it requires that
two simultaneous conditions be satisfied, namely that a2 and
a3 are both zero. Instead of implementing an AND operation
requiring both a2 and a3 to be zero, we can equivalently
implement an OR operation requiring both a2 and a3 to be 1.
This is made possible by the bottom branch in Fig. 12-6:
y2 = φ (a5 ) = φ (a2 + a3 − 0.5).
A perceptron is a nonlinear
function of a linear combination of several inputs. Why
does it have this form?
Concept Question 12-3:
(12.14)
If the input to a neural network
is to be classified into one of two classes separated by a
separating hyperplane, what is the minimum number of
neurons that must be in the neural network?
Concept Question 12-4:
12-2 TRAINING NEURAL NETWORKS
397
12-2 Training Neural Networks
So far, we have presented miniature-size neural networks for
which the weights and structure could be determined rather
easily. Neural networks for real-world problems are far more
complicated. The weights are determined by using labeled training images. The procedure for determining the weights is called
training the neural network. We now present this procedure.
There are actually three different procedures for training a
neural network. At the heart of all three procedures is an algorithm called backpropagation. This term comes from the fact
that the neural network is first run forward (in increasing layer
number, or left to right), using a labeled training image, and the
gradient of the mean-square error with respect to the weights is
then propagated backwards (in decreasing layer number, or right
to left) using the results of the forward run.
The gradient of the mean-square error, computed using backpropagation, is then used in a steepest-descent (or gradient)
algorithm, in one of three ways.
We explain these techniques in the next three subsections.
First, we review steepest descent algorithms. Next, we derive
the backpropagation algorithm, and finally, we present three
procedures for computing the weights from labeled training
images.
x1
x2
x3 x4
x0
Figure 12-7 Contours of equal values of f (x) and the path (in
red) taken by an SD algorithm.
The basic iteration of an SD algorithm is
x(k+1) = x(k) − µ ∇ f (x(k) ),
12-2.1 Gradient (Steepest Descent) (SD)
Minimization Algorithm
A steepest-descent (SD) algorithm, also known as a gradient
or (misleadingly) a gradient-descent algorithm (the gradient
does not descend) is an iterative algorithm for finding the
minimum value of a differentiable function f (x1 , x2 , . . . , xN )
of N spatial variables x = [x1 , x2 , . . . , xN ]T . The function to be
minimized must be real-valued and scalar (not vector-valued),
although SD can also be applied to minimize scalar functions
of vector-valued or complex-valued functions, such as || f (x)||2 .
To maximize f (x), we apply SD to − f (x). The minimum occurs
at x = x∗ , where ∗ denotes the minimizing value, not complex
conjugate (this is standard notation for optimization).
SD relies on the mathematical fact that the gradient ∇ f (x)
specifies the (vector) direction in which f (x) increases fastest.
Recall that, even though f is a scalar, ∇ f is a vector defined as
∇f =
∂f ∂f
∂f
,
,...,
∂ x1 ∂ x2
∂ xN
T
.
(12.15)
(12.16)
where x(k) is the estimate of the minimizing x at the kth iteration,
and µ is the step-size, a small discretization length. Vector x(k) is
perturbed by a distance µ in the direction that decreases f (x(k) )
the fastest.
The iterative process stops when x(k+1) ≈ x(k) , corresponding
to ∇ f (x) ≈ 0. This may be the location x∗ of the global
minimum of f (x), or it may be only a local minimum of f (x).
These are both illustrated in Example 12-2 below.
A useful interpretation of SD is to regard f (x) as the elevation
at a location x in a bowl-shaped surface. The minimizing value
x∗ is at the bottom of the bowl. By taking a short step in the
direction in which f (x) is decreasing, we presumably get closer
to the bottom x∗ of the bowl. However, we may get caught in a
local minimum, at which ∇ f (x) ≥ 0, so taking another step in
any direction would increase f (x).
Figure 12-7 illustrates how taking short steps in the direction
in which f (x) is decreasing may eventually take us to the
minimizing value x∗ . The curves are contours (as in a topographical map) of equal “elevations” of f (x) values. This is merely
illustrative; backpropagation uses a 1-D SD algorithm.
www.dbooks.org
398
CHAPTER 12 SUPERVISED LEARNING AND CLASSIFICATION
different numeral, the output terminals should change values
accordingly.
Example 12-2: 1-D SD Algorithm
The 1-D version of Eq. (12.16) for minimizing a function f (x)
is
d f (k)
(x ),
(12.17)
x(k+1) = x(k) − µ
dx
where k is the iteration index. Apply the algorithm given by
Eq. (12.17) to compute the minimizing value x∗ for
f (x) = cos(3x) − sin(x) + 2,
−2 < x < 2.
(12.18)
Solution: The function f (x) is plotted in blue in Fig. 12-8.
Over the range −2 < x < 2, f (x) has its global minimum x∗
near x = 1, as well as a local minimum near x = −1. Since
f ′ (x) =
df
= −3 sin(3x) − cos(x),
dx
|x| < 2,
the SD algorithm of Eq. (12.17) becomes
x(k+1) = x(k) + µ (3 sin(3x(k) ) + cos(x(k) )).
(12.19)
When initialized using x(0) = 0.1 and µ = 0.05, the SD algorithm converged to the (desired) global minimum at x ≈ 1 after
15 iterations. Ordered pairs {(x(k) , f (x(k) )), k = 1, . . . , 15} are
plotted using red “+” symbols in Fig. 12-8.
But when initialized using x(0) = −0.2, the SD algorithm
converged to the (undesired) local minimum at x ≈ −1. The
ordered pairs {(x(k) , f (x(k) )), k = 1, . . . , 15} are plotted using
green “+” symbols in Fig. 12-8. In both cases the algorithm
had smaller updates at locations x where d f /dx was small, as
expected from Eq. (12.17). At locations where d f /dx = 0, the
algorithm stopped; these were the global and local minima.
12-2.2 Supervised-Training Scenario:
Classifying Numerals 0 to 9
Let us consider how a neural network should perform to correctly classify images of numerals 0 to 9. In Fig. 12-9, the
input layer consists of 784 terminals, corresponding to the
intensity values of (28 × 28) image pixels. The true identity
of the test input image is the numeral 4. The output layer
consists of 10 terminals designed to correspond to the numerals
0 to 9. Once the neural network has been properly trained using
the backpropagation algorithm described later, the fifth output
terminal (corresponding to numeral 4) should report an output
of 1, and all of the other output terminals should report outputs
of zero. If the input image is replaced with an image of a
12-2.3 Backpropagation
A. Overview of Backpropagation
Backpropagation is an algorithm for “training” (computing the
weights) of a neural network. To perform backpropagation, we
are given a set of I labeled training images {f1 , f2 , . . . , fI }, where
each fi is an unwrapped training image. That is, the original 2-D
(M × N) ith image fi [n, m] is unwrapped by column into a vector
fi of length N ′ = MN:
fi = [ f1 [i], f2 [i], . . . , f j [i], . . . , fN ′ [i]]T .
(12.20)
Element f j [i] is the value of the jth pixel of the unwrapped
training image i. The neural network has an input vector and
an output vector. Image fi has N ′ elements, so the input layer of
the neural network should have N ′ terminals. The output of the
last layer consists of L neurons, corresponding to the number of
image classes, and the combination of the L values constitutes
an output vector. The desired variable di is the number denoting
perfect classification of image i with input vector fi , and it is
called the label of fi . For the zip-code example in which each
training image is (28 × 28) in size, the input vector is of length
N ′ = 28 × 28 = 784, but the output vector is only of length
L = 10, corresponding to the digits 0 through 9.
The goal of backpropagation is to determine the weights in
the neural network that minimize the mean-square error E over
all I training images.
B. Notation
The neural-network notation can be both complicated and confusing, so we devote this subsection to nomenclature, supported
by the configuration shown in Fig. 12-10.
With this notation in hand, we now define the mean-square
error E over all I training images as
E=
1 I
∑ E[i],
I i−1
(12.22)
where
E[i] =
1 L
1 L
(eL,p [i])2 = ∑ (d p [i] − yL,p[i])2 .
∑
2 p=1
2 p=1
(12.23)
The quantity inside the summation is the difference between the
12-2 TRAINING NEURAL NETWORKS
399
4
3
2
1
0
−2
−1.5
−1
−0.5
0
0.5
1
1.5
2
Figure 12-8 f (x) (in blue) and {(x(k) , f (x(k) )), k = 1, . . . , 15}, in red when initialized using x(0) = 0.1, and in green when initialized
using x(0) = −0.2.
(28 × 28) = 784 pixels
displaying image
of numeral “4”
NN
Designated numeral Output reading
0
0
1
0
2
0
3
0
4
1
5
0
6
0
7
0
8
0
9
0
Input layer with
784 terminals
Output layer
Figure 12-9 When the image of numeral 4 is introduced at the input of a properly trained neural network, the output terminals should all
be zeros, except for the terminal corresponding to numeral 4.
desired output of neuron p of the output layer and the actual
output of that neuron. The summation is over all L neurons in
the output layer.
12-2.4 Backpropagation Training Procedure
The classification accuracy of the neural network depends on
the relationship between the N ′ inputs associated with the input
image and the L outputs associated with the L classes. That
relationship depends, in turn, on the assignment of weights
wℓ,p,q , representing the gain between the output of the qth
neuron in layer (ℓ − 1) and the input to the pth neuron in layer ℓ
(Fig. 12-11). Backpropagation is the tool used to determine
those weights.
The process involves the use of I training images and
one or more iterations. We will outline three backpropagation
www.dbooks.org
400
f1
CHAPTER 12 SUPERVISED LEARNING AND CLASSIFICATION
f1
w1,1,1
f1w1,1,1
w1,2,1
f1w1,2,1
f2
f3
f2
w1,1,2
w1,2,2
f3
y1,1
1
f2w1,1,2
Σ
w2,1,1
w2,2,1
0
y1,1w2,1,1
f3w1,1,3
w1,1,4
y1,2
1
f3w1,2,3
Σ
0
w2,1,2
w2,2,2
w1,2,0
y2,1
1
0
y1,1w2,2,1
w1,1,3
1
Σ
y1,2w2,1,2
f4w1,1,4
f4
w2,1,0
w1,1,0
f2w1,2,2
w1,2,3
f4
w3,1,0
w3,2,1
w3,3,1
y3,1
Σ
0
w3,2,0
1
y3,2
Σ
y2,2
1
Σ
y1,2w2,2,2
w2,2,0
w3,1,1
0
w3,1,2
0
w3,2,2
w3,3,2
1
y3,3
Σ
0
w3,3,0
f4w1,2,4
w1,2,4
Layer 0
(input)
Layer 1
Layer 2
Layer 3
(output)
Figure 12-10 Neural network with 4 terminals at input layer, corresponding to image vector f = [ f1 f2 f3 f4 ]T , 2 hidden layers of 2
neurons each, and an output layer with three terminals, corresponding to three image classes.
procedures with different approaches. The procedures utilize
relationships extracted from the derivation given later in Section
12-3.
A. Procedure 1: Iterate over k, One Image at a
Time
Image i = 1, Iteration k = 0
1. Initialize: For test image i = 1 (chosen at random and labeled
(0)
image 1), initialize the weights wℓ,p,q [1] with randomly chosen
(0)
values. The superscript (0) denotes that wℓ,p,q [1] are the initial
assignments.
Weight Initialization
The initial values of the weights w in the neural network should
be small enough such that the linear combination of inputs is
between 0 and 1. In addition, even when the activation function
(sigmoid function) has a steep slope, the neuron output will be
roughly proportional to the linear combination (instead of just
being 1 or 0) for a wide range of input values.
12-2 TRAINING NEURAL NETWORKS
401
Symbols and Indices
i:
I:
fi :
f j [i]:
L:
N′:
k:
ℓ:
L:
ℓ, p:
nℓ :
Training-image index label, i = 1, 2, . . . , I
Total number of training images
Vector representing unwrapped image fi [n, m]
Value of jth pixel of vector fi , j = 1, 2, . . . , N ′
Total number of classes, also total number of
outputs
(= MN): Length of training image vectors fi ,
which also is the number of input terminals at
layer 0
Iteration index, as in (·)(k)
Layer index in neural network, ℓ = 0, . . . , L,
with ℓ = 0 representing the input layer and
ℓ = L representing the output layer
Total number of layers, excluding the input
layer (which contains terminals, not perceptrons)
pth neuron in ℓth layer, p = 1, 2, . . . , nℓ
Total number of neurons in ℓth layer
Inputs, Outputs, and Differences
nin = N ′ :
nout = L:
wℓ,p,q [i]:
Number of inputs
Number of outputs
Weight (gain) between output of qth neuron in layer (ℓ − 1) and input to pth
neuron in layer ℓ for image i
υℓ,p[i] = Activity of pth neuron in ℓth layer for image i
nℓ −1
υℓ,p [i] = ∑ wℓ,p,q [i] yℓ−1,q [i],
(12.21a)
q=0
with yℓ−1,0 [i] = 1 for the constant input for each perceptron in layer ℓ.
yℓ,p [i] = output of pth neuron in layer ℓ for image i
1
= φ (υℓ,p [i]) =
,
1 + exp(−aυℓ,p[i])
(12.21b)
yl−1, q
1
Σ
0
wl,p,q
Neuron q in layer l − 1
wl,p,q yl−1, q
1
Σ
0
Neuron p in layer l
Figure 12-11 Neuron q in layer (ℓ − 1) generates output
yℓ−1,q . The contribution of that neuron to neuron p in layer ℓ
is wℓ,p,q yℓ−1,q , where wℓ,p,q is an iteratively adjusted weighting
coefficient.
However, if a weight wℓ,p,q is too small, the output yℓ−1,q of
the qth neuron in layer (ℓ − 1) will have little effect on the input
wℓ,p,q yℓ−1,q to the pth neuron in layer ℓ. Also, if the weight is too
large, it will dominate all of the other inputs to the pth neuron in
layer ℓ. To prevent these extreme situations from occurring, the
initial weights should be chosen appropriately: the more inputs
into a neuron, the smaller the initial weights should be, and in
all cases, the initial weights should be randomly distributed.
A common rule of thumb for initializing the weights is to
choose a random number between 0 and 1 if there are 50
perceptrons in the neural network, between 0 and 0.5 if there
are between 50 and 200 perceptrons in the neural network, and
between 0 and 0.2 if there are more than 200 perceptrons in the
neural network.
2. Feed forward: Run the neural network forward (in increasing
layer index ℓ).
3. Measure outputs of all layers: Measure all of the outputs
(0)
{yℓ,p [1], ℓ = 1, 2, . . . , L} of all perceptrons in all layers of the
neural network, not just the output layer.
(0)
with a = activation constant.
d p [i] = Desired output of pth neuron in output layer
4. Initialize output gradients: Set δL,p [1] for all perceptrons in
the final layer (ℓ = L) using
(0)
(0)
(0)
(0)
δL,p [1] = −a(d p[1] − yL,p[1]) yL,p [1] (1 − yL,p[1]),
ℓ = L for image i
eL,p [i] = Error for pth neuron in output layer ℓ = L
= d p [i] − yL,p [i].
To other neurons
in layer l
(0)
(12.21c)
(12.24)
where yL,p [1] is the output of perceptron p of the last layer,
www.dbooks.org
402
CHAPTER 12 SUPERVISED LEARNING AND CLASSIFICATION
generated in response to test image 1, d p [1] is the desired output
of the pth neuron in the output layer—which is either 1 or 0,
depending on whether image 1 is class p or not, respectively—
and a is a selectable activation constant (see Eq. (12.2)). If image
1 belongs to class 5 from among 8 classes, for example, then
the desired output d p [1] is 1 for p = 5 and zero for the other 7
outputs (p = 1, 2, 3, 4, 6, 7, and 8).
5. Backpropagate: in decreasing layer number, generate values
(0)
for the local gradients δℓ,p [1], starting at ℓ = L and ending at
ℓ = 1, using the recipe
(0)
(0)
nℓ+1
(0)
(0)
(0)
δℓ,p [1] = a yℓ,p [1] × (1 − yℓ,p[1]) × ∑ δℓ+1,q [1] wℓ+1,q,p[1].
q=1
(12.25)
(k)
(k)
(k)
δℓ,p [i] = a yℓ,p [i] (1 − yℓ,p[i])
nℓ+1
(k)
(k)
× ∑ δℓ+1,q [i] wℓ+1,q,p[i],
(12.28b)
q=1
(k)
(k)
(k)
∆ℓ,p,q[i] = δℓ,p [i] yℓ−1,q[i],
(12.28c)
and
(k)
(k+1)
(k)
wℓ,p,q [i] = wℓ,p,q [i] − µ ∆ℓ,p,q[i].
(12.28d)
◮ Procedure 1 computes a local (and hopefully also a
global) minimum of E[i]. The drawback of this procedure
is that it requires many runs of the neural network and
backpropagation to make use of all of the training images.
Hence, the procedure is seldom used in practice. ◭
6. Compute the weight iteration correction:
(0)
(0)
(0)
∆ℓ,p,q [1] = δℓ,p [1] yℓ−1,q [1]
(12.26)
for all layers and neurons in those layers. The relationship given
by Eq. (12.25) is extracted from the complete derivation of
backpropagation given in Section 12-3.
Image 1, Iteration k = 1
B. Procedure 2: Iterate over Images, but
Backpropagate Only Once for Each Image
1. Perform steps 1–7 of Procedure 1.
2. Repeat steps 2–7 of Procedure 1, but with a different training
(1)
image. Use the weights wℓ,p,q computed for the previous training
image.
3. Repeat the preceding step for all remaining training images.
7. Update all weights: from those used in iteration k = 0 to a
new set for iteration k = 1 using
(1)
(0)
(0)
wℓ,p,q [1] = wℓ,p,q [1] − µ ∆ℓ,p,q[1],
(12.27)
where µ is a small step size.
8. Repeat steps 2 to 6.
Image 1, Iterations k = 2 to K
Increment k and repeat steps 2–6 until the process converges to
a predefined set of output thresholds (weights stop changing).
The final iteration is identified as k = K.
◮ Procedure 2 differs from Procedure 1 in that the steepest
descent (SD) algorithm is run for only a single iteration
(k = 0 to k = 1) for each training image. Procedure 2,
called incremental learning, is often described as a gradient descent algorithm, but this is an incorrect description
because the SD algorithm is run for only a single iteration.
(1)
The computed weights {wℓ,p,q [i]} do not minimize E[i]—
they only make it slightly smaller than the randomly chosen
(0)
initial weights {wℓ,p,q [i]} do. However, Procedure 2 does
get all of the training images involved more quickly than
Procedure 1, and is more commonly used in practice. ◭
Images 2 to I
C. Procedure 3: Stochastic Approach
Repeat the process in steps 1–8 for all remaining images i = 2
to I. For each image, initialize the neural network with the last
iteration weights of the first training image. For any iteration k
and image i, the relevant relationships are:
This procedure is similar to Procedure 2 except that it uses a
stochastic gradient descent algorithm. Given I training images,
the images are partitioned into I2 batches of I1 images each,
selected randomly. Hence, I = I1 I2 . Using index i1 for the
images within a batch, with {i1 = 1, 2, . . . , I1 }, and index i2
for the batch of images, with {i2 = 1, 2, . . . , I2 }, the summation
(k)
(k)
(k)
(k)
δL,p [i] = −a(d p[i] − yL,p[i]) yL,p [i] (1 − yL,p[i]),
(12.28a)
12-3 DERIVATION OF BACKPROPAGATION
403
defining the total mean-square error (MSE) in Eq. (12.22) can
be rewritten as a sum of the error over the I2 batches:
E=
1 I2
E[i2 ].
I2 i2∑
=1
(k)
This relationship, which is used to update the weight wℓ,p,q [i] of
(k+1)
iteration k to weight wℓ,p,q [i] in iteration (k + 1), is based on the
steepest descent (SD) algorithm, which uses the iterative form
(12.29a)
(k+1)
Here, E[i2 ] is the MSE of batch i2 , which is given by
E[i2 ] =
1 I1
E[i1 , i2 ],
I1 i2∑
=1
(k)
wℓ,p,q [i] = wℓ,p,q [i] − µ
(12.29b)
where E[i1 , i2 ] is the MSE for image i1 in batch i2 . In view
of Eq. (12.23), for a neural network whose output layer L is
composed of L neurons, E[i1 , i2 ] is given by
1
∑ (d p[i1 , i2 ] − yL,p[i1 , i2 ])2 .
2 p=1
(12.29c)
In Procedures 1 and 2, the weights in the neural network are
updated from one iteration to the next, using the expressions
given by Eq. (11.48) for the local gradients δℓ,p . A similar
process is used in the current procedure, except that now the
weights for each particular neuron are updated using the average
weight for that neuron among I1 training images in one of the I2
batches (in practice, it does not much matter which particular
batch is selected).
Concept Question 12-5:
What is backpropagation and
δℓ,p [i] =
∂ E[i]
(k)
∂ wℓ,p,q [i]
=
ℓ,p,q
∂ υℓ,p [i]
(12.32)
(12.33)
ℓ,p,q
= yℓ−1,q [i],
(12.34)
= δℓ,p yℓ−1,q[i].
(12.35)
(k)
∂ wℓ,p,q [i]
and Eq. (12.33) simplifies to
(k)
∂ wℓ,p,q [i]
A second use of the chain rule gives
δℓ,p [i] =
nℓ+1
∂ E[i]
∂ E[i] ∂ υℓ+1,s [i]
= ∑
∂ υℓ,p [i] s=1 ∂ υℓ+1,s [i] ∂ υℓ,p [i]
nℓ+1
= ∑ δℓ+1,s [i]
12-3 Derivation of Backpropagation
In the preceding section we outlined how to apply the backpropagation algorithm in the form of three slightly different iterative
procedures. A critical ingredient is the relationship given by the
combination of Eqs. (12.28c and d), namely
(k)
∂ E[i]
,
∂ υℓ,p [i]
Use of the definition given by Eq. (12.21a) leads to
What does it mean to train a
Concept Question 12-7: Does a gradient (steepest descent) algorithm always find the global minimum?
(k+1)
(12.31)
∂ υℓ,p [i]
∂ E[i] ∂ υℓ,p [i]
= δℓ,p [i]
.
(k)
∂ υℓ,p [i] ∂ w(k) [i]
∂w
[i]
∂ E[i]
neural network?
.
where υℓ,p [i] is the activity of neuron p in layer ℓ defined in
Eq. (12.21a). Using the chain rule gives
what is it used for?
Concept Question 12-6:
(k)
∂ wℓ,p,q [i]
The second term involves the differential of the mean-square
error E[i]. The derivation that follows will show that the second
term in Eq. (12.31) is indeed equal to the second term in
Eq. (12.30).
We start by defining the local gradient δℓ,p [i] as
L
E[i1 , i2 ] =
∂ E[i]
(k)
(k)
(k)
wℓ,p,q [i] = wℓ,p,q [i] − µ ∆ℓ,p,q [i] = wℓ,p,q − µ δℓ,p [i] yℓ−1,q [i].
(12.30)
s=1
∂ υℓ+1,s [i]
.
∂ υℓ,p [i]
(12.36)
Next, change ℓ to ℓ + 1 in Eq. (12.21a) and also use Eq. (12.21b)
to obtain:
nℓ
(k)
υℓ+1,s [i] = ∑ wℓ+1,s,q [i] φ (υℓ,p [i]).
(12.37)
q=0
When taking a partial derivative with respect to υℓ,p[i], only the
www.dbooks.org
404
CHAPTER 12 SUPERVISED LEARNING AND CLASSIFICATION
δL,p [i]:
q = p term is nonzero, and it is
∂ υℓ+1,s [i]
(k)
= wℓ+1,s,p [i] φ ′ (υℓ,p [i]).
∂ υℓ,p [i]
(12.38)
δL,p [i] = −a(d p[i] − yL,p[i])(1 − yL,p )yL,p .
(12.39)
12-4 Neural Network Training
Examples
Upon factoring out φ ′ (υℓ,p [i]), Eq. (12.36) becomes
mℓ+1
(k)
δℓ,p [i] = φ ′ (υℓ,p [i]) ∑ δℓ+1,s [i] wℓ+1,s,p [i].
s=1
12-4.1 Training Image Selection
Using Eq. (12.3) for the derivative of the sigmoid function φ (x)
defined in Eq. (12.2), and also in Eq. (12.21b), gives
φ ′ (υℓ,p [i]) = a(1 − φ (υℓ,p[i])) φ (υℓ,p [i])
= a(1 − yℓ,p[i]) yℓ,p [i].
(12.40)
Hence, we now can compute δℓ,p [i] recursively in decreasing
layer number ℓ, starting at output layer ℓ = L, using
mℓ+1
(k)
δℓ,p [i] = a(1 − yℓ,p[i]) yℓ,p [i] ∑ δℓ+1,s [i] wℓ+1,s,p [i]. (12.41)
s=1
We still have to initialize this recursion at ℓ = L. To do that,
we apply the chain rule (again!) to the definition given be
Eq. (12.32) at ℓ = L:
δL,p [i] =
(12.46)
∂ E[i]
∂ E[i] ∂ eL,p [i] ∂ yL,p [i]
=
.
∂ υL,p [i] ∂ eL,p [i] ∂ yL,p [i] ∂ υL,p [i]
(12.42)
Each of these partial derivatives can be simplified easily. From
the definition given by Eq. (12.23) for E[i], we have
∂ E[i]
= eL,p [i].
∂ eL,p [i]
(12.43)
From the definition for eL,p [i] given by Eq. (12.21c),
∂ eL,p [i]
= −1.
∂ yL,p [i]
(12.44)
From Eqs. (12.2) and (12.3),
∂ yL,p [i]
= φ ′ (υL,p [i])
∂ υL,p [i]
= a(1 − φ (υL,p [i])) φ (υL,p [i])
= a(1 − yL,p)yL,p .
(12.45)
Substituting these results in Eq. (12.42) gives the initialization
It is customary to use only some members of the training set to
train the neural network, and then use the remaining members
to test the performance of the neural network. If only half of the
training set members are used to train, and the other half to test,
we can repeat training by exchanging these two classes. This is
called cross-validation.
Members of the training set should be selected at random, so
that not all training set images are labeled with the same image
class.
12-4.2 Identifying “+” Symbol
In this simple example, we train a neural network to determine
the presence or absence of a “+” in a noisy (3 × 3) image, from a
set of I labeled training images, all (3 × 3) in size, half of which
contain the “+” symbol—in addition to zero-mean Gaussian
noise—and half contain only noise. Typical examples of the
training images are shown in Fig. 12-12.
This problem can be solved using the procedure MLE described in Section 11-2 of the preceding chapter, but the derivation requires the use of likelihood functions and relies heavily
on the zero-mean white Gaussian nature of the noise. By using a
neural network, we do not need to use the mathematical analyses
associated with the likelihood functions, but we should evaluate
the classification accuracy of the neural network to make sure it
is acceptable
The neural network we use in this example has nine terminals
(nodes) in its input layer (Fig. 12-13), for the nine pixel values
in a (3 × 3) image, nine perceptrons in a single hidden layer, and
a single perceptron in the output layer, which (ideally) outputs
1 if the “+” is present and zero if the ”+” is not present. The
total number of weights is 92 = 81 connecting the input layer
to the hidden layer, and an additional 9 weights connecting
the hidden layer to the output layer. These do not include the
10 constant inputs in the 10 perceptrons. Hence, the total is
81 + 9 + 10 = 100 weights.
The neural network was trained using 100,000 training images, half of which included the “+” and the other half of which
12-4 NEURAL NETWORK TRAINING EXAMPLES
405
f1
f2
f3
f4
f5
(a) Image of noise plus “+” sign
f6
f7
f8
f9
Layer 0
(input)
c1
c2
c3
c4
c5
c6
c10
c7
c8
c9
Layer 1
Layer 2
(output)
(b) Typical image of noise alone, with no “+” sign
Figure 12-12 Typical examples of noisy (3 × 3) images (a)
with and (b) without “+” symbol.
did not. Each training image included a realization of a zeromean Gaussian noise field with σ = 0.1. Typical image examples are shown in Fig. 12-12. The weights were all initialized
with random numbers distributed with a zero-mean Gaussian
distributions with σ = 1. The step size for steepest descent (SD)
was set at µ = 0.1, and for the activation function φ (·), the
activation parameter was set at a = 7. With such a large value
for a, φ (·) resembles a step function. Procedure 2 of Section
12-3 was used.
The neural network was tested using 100 additional training
images. The results are shown in Fig. 12-14. The correct
classification is shown in blue and the neural network output
classification is shown in red. The neural network correctly
classified the “+” symbol in 98% of the 100 test images.
The performance of the neural network depended heavily on
the initial values of the weights wℓ,p,q . For a few initializations, the resulting neural network performed rather poorly, but
Figure 12-13 Basic structure of neural network. The input
image is represented by f = [ f1 , f2 , . . . , fq ]T . Each blue circle
represents a perceptron that weighs the 9 inputs, adds them up,
adds a constant (c0 through c9 , and goes through an activation
function φ (·).
for most initializations the resulting neural networks correctly
classified all test images. A typical neural network correctly
classified 98% of the test images.
12-4.3 Zip-Code Reader
For a zip-code reader, each input f[i] is a vector of length 784.
Each element of f[i] is a pixel value read by a camera that reads
each digit as a (28 × 28) image. The number of pixels is then
282 = 784. Each training image is labeled with a digit (one of
{0, 1, . . . , 9}) selected by a human judge. Training image f[i], the
corresponding output y[i], and the desired output d[i] are given
by
f[i] = [ f1 [i], f2 [i], . . . , f784 [i]]T ,
www.dbooks.org
{i = 1, 2, . . . , I},
(12.47a)
406
CHAPTER 12 SUPERVISED LEARNING AND CLASSIFICATION
1
0.8
0.6
0.4
0.2
0
0
10
20
30
40
50
60
70
80
90
100
0
10
20
30
40
50
60
70
80
90
100
1
0.8
0.6
0.4
0.2
0
Figure 12-14 Performance of neural network for Detection of “+.” Correct in blue, output in red.
y[i] = [y1 [i], y2 [i], . . . , y10 [i]]T ,
and
d[i] = [d1 [i], d2 [i], . . . , d10 [i]]T ,
(12.47b)
(12.47c)
with
d j [i] =
(
1
0
for j = correct digit,
for j = incorrect digit.
(12.47d)
A training set I = 60, 000 images of handwritten digits is available at the U.S. National Institute of Standards and Technology
(NIST). The zip code digits were handwritten by 250 different
people. The neural network, comprised of 784 input terminals,
10 output terminals, and many hidden layers is illustrated in
Fig. 12-15.
PROBLEMS
407
0
1
2
3
4
784 terminals
5
6
7
8
9
Input terminals
Hidden layer
Figure 12-15 The basic structure of a zip-code reader.
www.dbooks.org
Output layer
408
CHAPTER 12 SUPERVISED LEARNING AND CLASSIFICATION
Summary
Concepts
• The output of a perceptron is the application of an
activation function to a weighted sum of its inputs. A
perceptron mimics the biological action of a neuron, and
perceptrons are often called neurons.
• A common choice for activation function is the sigmoid
function (below).
• Classification rules defined by a separating hyperplane
can be implemented using a single perceptron.
• A neural network is a network of perceptrons, connected
in a way that mimics the connections of neurons in the
brain.
• The weights in a neural network are computed using an
algorithm called backpropagation, using a set of labelled
training images. Backpropagation uses one iteration of a
gradient or steepest descent algorithm.
• Computing the weights in a neural network by applying
backpropagation using a set of labelled training images
is called training the neural network. There are three
different ways of performing training.
Mathematical Formulae
Perceptron
y = φ (w0 + w1 x1 + · · · + wN xN )
Important Terms
activation function
backpropagation
gradient
Sigmoid function
1
φ (x) =
1 + e−ax
Provide definitions or explain the meaning of the following terms:
hidden layers
input layer
neural network
neuron
output layer
perceptron
PROBLEMS
Section 12-1: Overview of Neural Networks
12.1 In Exercise 12-1, you had to determine by inspection the
weights of a perceptron so that it implemented an OR gate. Write
out a set of nonlinear equations whose solution is the weights in
Fig. 12-1(a).
12.2 In Exercise 12-2, you had to determine by inspection the
weights of a perceptron so that it implemented an AND gate.
Write out a set of nonlinear equations whose solution is the
weights in Fig. 12-1(a).
g0,0 g1,0
1 0
12.3 An image
is to be classified as either
g1,0 g1,1
0 1
0 1
or
. Write a set of equations whose solution is the
1 0
weights replacing those in Fig. 12-3(a).
sigmoid function
steepest descent
supervised learning
training
g0,0 g1,0
1 0
12.4 An image
is to be classified as either
g1,0 g1,1
0 1
0 1
. Specify the weights in a perceptron like Fig. 12-3(a)
or
1 0
that classifies the image.
12.5 A binary adder implements binary addition (with carry).
It has the truth table
x1
x2
sum
carry
0
0
0
0
0
1
1
0
1
0
1
0
1
1
0
1
where sum = x1 + x2 (mod 2) and carry is the carry (1 + 1 = 10
base 2). Modify the neural network in Fig. 12-5(b) to implement
a binary adder.
12.6 A binary-to-decimal converter accepts as input a 2-bit
binary number (x1 x2 )2 and converts it to a decimal number 0,
PROBLEMS
409
1, 2 or 3. Design a neural network that accepts as inputs {x1 , x2 }
and outputs {y0 , y1 , y2 , y3 } where if (x1 x2 )2 = K, then yK = 1
and {yk , k 6= K} are all zero.
12.7 (This problem may require review of Chapter 2.) A 1-D
signal x(t) of duration 1 s is sampled at 44100 samples/s,
resulting in discrete-time signal x[n] = x(n/44100) of duration
44100 s. Design a neural network for determining the presence
or absence of a trumpet playing note A (fundamental frequency
440 Hz) by examining the first four harmonics of the trumpet
signal. Assume the real parts of their DFT are positive.
12.8 (This problem may require review of Chapter 5.) Design
a neural network for edge detection on a 1-D signal {x[n]} of
duration N. An edge is at n = n0 if |x[n0 ] − x[n0 − 1]| > T for
some threshold T . Let x[−1] = x[0]. The extension to 2-D edge
detection of horizontal or vertical edges is straightforward.
Section 12-2: Training Neural Networks
12.9 The program neuron.m trains a neuron using labeled
training vectors. Run the program to train (determine 2 weights
of) a neuron to implement an OR gate. This is the same problem
as Problem 12.1. The neuron has the form of Fig. 12-1(a). Use
1000 iterations with step size µ = 0.01, a = 7, and initialize
all weights with 0.1. Compare the neuron outputs y[i] with the
desired output d[ j] for j = 1, 2, 3, 4 in a table.
12.10 The program neuron.m trains a neuron using labelled
training vectors. Run the program to train (determine 2 weights
of) a neuron to implement an AND gate. This is the same problem as Problem 12.2. The neuron has the form of Fig. 12-1(a).
Use 1000 iterations with step size µ = 0.01, a = 7, and initialize
all weights with 0.1. Compare the neuron outputs y[i] with the
desired output d[ j] for j = 1, 2, 3, 4 in a table.
12.11 The program neuron.m trains a neuron using labelled
training vectors. Run
to train a neuron to classify a
the program
0 1
1 0
. This is the same problem
or
(2 × 2) image as
1 0
0 1
as Problem 12.3. The neuron has the form of Fig. 12-1(b). Use
1000 iterations with step size µ = 0.01, a = 7, and initialize
all weights with 0.1. Compare the neuron outputs y[i] with the
desired output d[ j] for j = 1, 2 in a table.
Section 12-3: Derivation of Backpropagation
12.12 We clarify the derivation of backpropagation by applying it to a single neuron. Single neurons are discussed in
Section 12-1.1 and illustrated in Fig. 12-1. We are given I
training input M-vectors x[i] = [x1 [i], x2 [i], . . . , xM [i]]T , where
i = 1 . . . I and I labels {d[i], i = 1, . . . , I}, where d[i] is the desired
output for training vector x[i]. The goal is to compute weights
{w j , j = 0, . . . , M} that minimize E[i] = 21 (d[i] − y[i])2, where
y[i] = φ (∑M
j=0 w j x j [i]) and x0 = 1 implements the single neuron.
(a) Derive a steepest descent algorithm to compute {w j ,
j = 0, . . . , M} minimizing E[i].
(b) Show that this is the output layer ℓ = L in the backpropagation derivation.
Section 12-4: Neural Network Training Examples
12.13 Program P1213.m creates 100 random points in the
square −10 < x1 , x2 < 10 and labels them as being inside
or outside a circle of radius 8 centered at the origin (so
the areas inside and outside the circle are rougly equal:
π (8)2 = 201 ≈ 200 = 12 (20)2 ). It then trains a neural network
with 2 inputs (x1 and x2 coordinates of each point), 1 output
neuron (for inside or outside the circle), and a hidden layer of 10
neurons. It uses 1000 iterations, each running over 100 training
points (2 coordinates each), µ = 0.01, and a = 7. Run P1213.m
using different initializations until it successfully assigns each
training point as being in category #1 (inside) or #2 (outside) the
circle. Of course, the neural network doesn’t “know” circles; it
“learns” this from training.
12.14 Program P1214.m creates 100 random points in the
square −10 < x1 , x2 < 10 and labels them as being inside or
outside a parabola x2 = x21 /10. It then trains a neural network
with 2 inputs (x1 and x2 coordinates of each point), 1 output
neuron (for inside or outside the parabola), and a hidden layer
of 10 neurons. It uses 1000 iterations, each running over 100
training points (2 coordinates each), µ = 0.01, and a = 7. Run
P1214.m using different initializations until it successfully
assigns each training point as being in category #1 (inside) or
#2 (outside) the parabola. Of course, the neural network doesn’t
“know” parabolas; it “learns” this from training.
12.15 Program P1215.m creates 100 random points in the
square −10 < x1 , x2 < 10 and labels them as being inside or outside 4 quarter circles centered on the corners.
The areas inside and outside the circle are rougly equal:
π (8)2 = 201 ≈ 200 = 12 (20)2 ). It then trains a neural network
with 2 inputs (x1 and x2 coordinates of each point), 1 output
neuron (for inside or outside the parabola), and a hidden layer
of 10 neurons. It uses 1000 iterations, each running over 100
training points (2 coordinates each), µ = 0.01, and a = 7. Run
P1215.m using different initializations until it successfully
www.dbooks.org
410
CHAPTER 12 SUPERVISED LEARNING AND CLASSIFICATION
assigns each training point as being in category #1 (inside) or
#2 (outside) the circles. Of course, the neural network doesn’t
“know” circles; it “learns” this from training.
A
Appendix
A
Review of Complex Numbers
A complex number z may be written in the rectangular form
z = x + jy,
(z)
(A.1)
x = Re(z),
y = Im(z).
|z| = x2 + y2
(A.2)
θ
(A.3)
(A.4)
we can convert z from polar form, as in Eq. (A.3), into rectangular form, as in Eq. (A.1),
z = |z|e jθ = |z| cos θ + j|z| sin θ ,
(A.5)
which leads to the relations
x = |z| cos θ ,
p
|z| = x2 + y2 ,
y = |z| sin θ ,
θ = tan−1 (y/x).
(z)
x
Figure A-1 Relation between rectangular and polar representations of a complex number z = x + jy = |z|e jθ .
where |z| is the magnitude of z, θ is its phase angle, and the
form θ is a useful shorthand representation commonly used in
numerical calculations. By applying Euler’s identity,
e jθ = cos θ + j sin θ ,
θ = tan−1 (y/x)
|z|
Note that Im(3 + j4) = 4, not j4.
Alternatively, z may be written in polar form as
z = |z|e jθ = |z| θ
z
y
real (Re) and imaginary (Im) parts of z,
where x and y are the√
respectively, and j = −1. That is,
x = |z| cos θ
y = |z| sin θ
(A.6)
(A.7)
The two forms of z are illustrated graphically in Fig. A-1.
Because in the complex plane, a complex number assumes the
form of a vector, it is represented by a bold letter.
When using Eq. (A.7), care should be taken to ensure that θ is
in the proper quadrant by noting the signs of x and y individually,
as illustrated in Fig. A-2. Specifically,


tan−1 (y/x)
if x > 0,


tan−1 (y/x) ± π
if x < 0,
θ=

π /2
if x = 0 and y > 0,



−π /2
if x = 0 and y < 0.
Complex numbers z2 and z4 point in opposite directions and
their phase angles θ2 and θ4 differ by 180◦ , despite the fact that
(y/x) has the same value in both cases.
The complex conjugate of z, denoted with a star superscript
(or asterisk), is obtained by replacing j (wherever it appears)
with − j, so that
z∗ = (x + jy)∗ = x − jy = |z|e− jθ = |z| −θ .
(A.8)
The magnitude |z| is equal to the positive square root of the
product of z and its complex conjugate:
|z| =
411
www.dbooks.org
√
z z∗ .
(A.9)
412
APPENDIX A
Division: For z2 6= 0,
(z)
z2 = −2 + j3
3
θ22
1
θ2 = 180o − θ1
z1
x1 + jy1
=
z2
x2 + jy2
(x1 + jy1 ) (x2 − jy2 )
=
·
(x2 + jy2 ) (x2 − jy2 )
(x1 x2 + y1 y2 ) + j(x2 y1 − x1 y2 )
=
,
x22 + y22
z1 = 2 + j3
3
θ1
θ1 = tan−1 2 = 56.3o
(z)
−3 −2 −1
1
2 3
−1
θ3 = −θ2
θ θ4 θ4 = −θ1
−2 3
z3 = −2 − j3
−3
REVIEW OF COMPLEX NUMBERS
or
z4 = 2 − j3
Figure A-2 √
Complex numbers z1 to z4 have the same
magnitude |z| = 22 + 32 = 3.61, but their polar angles depend
on the polarities of their real and imaginary components.
We now highlight some of the salient properties of complex
algebra.
Equality: If two complex numbers z1 and z2 are given by
z1 = x1 + jy1 = |z1 |e
z2 = x2 + jy2 = |z2 |e
(A.13a)
j θ1
,
(A.10a)
j θ2
,
(A.10b)
then z1 = z2 if and only if (iff ) x1 = x2 and y1 = y2 or,
equivalently, |z1 | = |z2 | and θ1 = θ2 .
z1
|z1 |e jθ1
=
z2
|z2 |e jθ2
|z1 | j(θ1 −θ2 )
e
=
|z2 |
|z1 |
[cos(θ1 − θ2 ) + j sin(θ1 − θ2 )].
=
|z2 |
(A.13b)
Powers: For any positive integer n,
zn = (|z|e jθ )n
= |z|n e jnθ = |z|n (cos nθ + j sin nθ ),
(A.14)
z1/2 = ±|z|1/2 e jθ /2
= ±|z|1/2 [cos(θ /2) + j sin(θ /2)].
(A.15)
Useful relations:
Addition:
z1 + z2 = (x1 + x2 ) + j(y1 + y2 ).
(A.11)
Multiplication:
z1 z2 = (x1 + jy1 )(x2 + jy2 )
= (x1 x2 − y1 y2 ) + j(x1 y2 + x2 y1 ),
j=e
j π /2
= 1 90◦ ,
j π /2
(A.12a)
or
z1 z2 = |z1 |e jθ1 · |z2 |e jθ2
= |z1 ||z2 |e j(θ1 +θ2 )
= |z1 ||z2 |[cos(θ1 + θ2 ) + j sin(θ1 + θ2 )].
−1 = e jπ = e− jπ = 1 180◦ ,
(A.12b)
− j π /2
(A.16a)
(A.16b)
= 1 −90◦ ,
=e
− j = −e
p
±(1 + j)
j = (e jπ /2 )1/2 = ±e jπ /4 = √
,
2
p
±(1 − j)
.
− j = ±e− jπ /4 = √
2
(A.16c)
(A.16d)
(A.16e)
For quick reference, the preceding properties of complex numbers are summarized in Table A-1. Note that if a complex
number is given by (a + jb) and b = 1, it can be written either
as (a + j1) or simply as (a + j). Thus, j is synonymous with j1.
413
Table A-1 Properties of complex numbers.
Euler’s Identity: e jθ = cos θ + j sin θ
sin θ =
e j θ − e− j θ
2j
cos θ =
e j θ + e− j θ
2
z = x + jy = |z|e jθ
z∗ = x − jy = |z|e− jθ
x = Re(z) = |z| cos θ
|z| =
zn = |z|n e jnθ
z1/2 = ±|z|1/2 e jθ /2
z1 = x1 + jy1
z2 = x2 + jy2
z1 = z2 iff x1 = x2 and y1 = y2
z1 + z2 = (x1 + x2) + j(y1 + y2 )
z1 z2 = |z1 ||z2 |e j(θ1 +θ2 )
z1
|z1 | j(θ1 −θ2 )
=
e
z2
|z2 |
p
√
zz∗ = x2 + y2
θ = tan−1 (y/x)
y = Im(z) = |z| sin θ
−1 = e jπ = e− jπ = 1 ±180◦
j = e jπ /2 = 1 90◦
p
(1 + j)
j = ±e jπ /4 = ± √
2
− j = e− jπ /2 = 1 −90◦
p
(1 − j)
− j = ±e− jπ /4 = ± √
2
◦
Example A-1:
V = |V|e jθV = 5e− j53.1 = 5 −53.1◦ ,
Working with Complex
Numbers
|I| =
Since I = (−2 − j3) is in the third quadrant in the complex plane
(Fig. A-3),
θI = −180◦ + tan−1 32 = −123.7◦,
Given two complex numbers
V = 3 − j4,
I = −(2 + j3),
I = 3.61 −123.7◦ .
(a) express V√and I in polar form, and find (b) VI, (c) VI∗ , (d)
V/I, and (e) I .
Solution:
(a)
p
√
22 + 32 = 13 = 3.61.
Alternatively, whenever the real part of a complex number is
negative, we can factor out a (−1) multiplier and then use
Eq. (A.16a) to replace it with a phase angle of either +180◦
or −180◦, as needed. In the case of I, the process is as follows:
√
VV∗
p
√
= (3 − j4)(3 + j4) = 9 + 16 = 5,
I = −2 − j3 = −(2 + j3)
p
◦
−1
= e± j180 · 22 + 32 e j tan (3/2)
|V| =
◦
◦
= 3.61e j57.3 e± j180 .
θV = tan−1 (−4/3) = −53.1◦,
www.dbooks.org
414
APPENDIX A
REVIEW OF COMPLEX NUMBERS
Exercise A-2: Show that
−2
3
θI
|I|
θV
|V|
−3
I
−4
V
Figure A-3 Complex numbers V and I in the complex plane
(Example A-1).
Since our preference is to end up with a phase angle within
the range between −180◦ and +180◦, we will choose −180◦.
Hence,
◦
I = 3.61e− j123.7 .
(b)
VI = (5 −53.1◦)(3.61 −123.7◦)
= (5 × 3.61) (−53.1◦ − 123.7◦) = 18.05 −176.8◦ .
(c)
◦
◦
◦
VI∗ = 5e− j53.1 × 3.61e j123.7 = 18.05e j70.6 .
(d)
◦
5e− j53.1
V
j70.6◦
.
=
◦ = 1.39e
I
3.61e− j123.7
(e)
√
√
◦
I = 3.61e− j123.7
√
◦
◦
= ± 3.61 e− j123.7 /2 = ±1.90e− j61.85 .
Exercise A-1: Express the following complex functions in
polar form:
z1 = (4 − j3)2 ,
z2 = (4 − j3)1/2.
√
Answer: z1 = 25 −73.7◦ , z2 = ± 5 −18.4◦ . (See IP )
√
2 j = ±(1 + j). (See IP )
B
Appendix
B
MATLAB® and MathScript
A Short Introduction for Use in Image
Processing
B-1 Background
“A computer will always do exactly what you tell it to do. But
that may not be what you had in mind.”—a quote from the
1950’s.
This Appendix is a short introduction to MATLAB and MathScript for this book. It is not comprehensive; only commands
directly applicable to signal and image processing are covered.
No commands in any of MATLAB’s Toolboxes are included,
since these commands are not included in basic MATLAB
or MathScript. Programming concepts and techniques are not
included, since they are not used anywhere in this book.
MATLAB
MATLAB is a computer program developed and sold by the
Mathworks, Inc. It is the most commonly used program in signal
processing, but it is used in all fields of engineering.
“MATLAB” (matrix laboratory was originally based on a set
of numerical linear algebra programs, written in FORTRAN,
called LINPACK. So MATLAB tends to formlate problems
in terms of vectors and arrays of numbers, and often solves
problems by formulating them as linear algebra problems.
The student edition of MATLAB is much cheaper than the
professional version of MATLAB. It is licensed for use by all
undergraduate and graduate students. Every program on the
website for this book will run on it.
MATLAB commands used in this book and website also work
in MathScript.
One important exception is colormap(gray). To make
this work in MathScript, G=[0:64]/64;gray=G’*[1 1
1]; must be inserted at the beginning of the program.
Instructions on how to acquire a student version of MathScript
are included on the website accompanying the book, as part of
the student edition of LabVIEW. In the sequel, we use “M/M”
to designate “MATLAB or MathScript.”
Freemat and GNU Octave are freeware programs that are
mostly compatible with MATLAB.
Getting Started
To install the student version of MathScript included on the
website, follow the instructions.
When you run M/M, a prompt >> will appear when it
is ready. Then you can type commands. Your first command
should be >>cd mydirectory, to change directory to your
working directory, which we call “mydirectory” here.
We will use this font to represent typed commands and
generated output. You can get help for any command, such as
plot, by typing at the prompt help plot.
Some basic things to know about M/M:
MathScript
MathScript is a computer program developed and sold by
National Instruments, as a module in LabVIEW. The basic
commands used by MATLAB also work in MathScript, but
higher-level MATLAB commands, and those in Toolboxes,
usually do not work in MathScript. Unless otherwise noted, all
• Inserting a semicolon “;” at the end of a command suppresses output; without it M/M will type the results of the
computation. This is harmless, but it is irritating to have
numbers flying by on your screen.
• Inserting ellipses “...” at the end of a command means
it is continued on the next line. This is useful for long
commands.
• Inserting “%” at the beginning of a line makes the line a
comment; it will not be executed. Comments are used to
explain what the program is doing at that point.
• clear eliminates all present variables. Programs should
always start with a clear.
• whos shows all variables and their sizes.
415
www.dbooks.org
416
APPENDIX
MATLAB AND MATHSCRIPT
• M/M variables are case-sensitive: t and T are different
variables.
√
Both i and‘ j represent −1; answers use i. pi represents
π . e does not represent 2.71828.
• save myfile X,Y saves the variables X and Y in the
file myfile.mat for use in another session of M/M at another
time.
B-2.2
• load myfile loads all variables saved in myfile.mat, so
they can now be used in the present session of M/M.
• quit ends the present session of M/M.
.m Files
An M/M program is a list of commands executed in succession.
Programs are called “m-files” since their extension is “.m,” or
“scripts.”
To write an .m file, at the upper left, click:
File→New→m-file
This opens a window with a text editor.
Type in your commands and then do this:
File→Save as→myname.m
Make sure you save it with an .m extension. Then you can run
the file by typing its name at the prompt: >>myname. Make
sure the file name is not the same as a Matlab command! Using
your own name is a good idea.
You can access previously-typed commands using uparrow
and downarrow on your keyboard.
To download a file from a website, right-click on it, select
save target as, and use the menu to select the proper file type
(specified by its file extension).
B-2 Basic Computation
B-2.1 Basic Arithmetic
• Addition: 3+2 gives ans=5
• Subtraction: 3-2 gives ans=1
• Multiplication: 2*3 gives ans=6
• Division: 6/2 gives ans=3
• Powers: 2b3 gives ans=8
• Others: sin,cos,tan,exp,log,log10
• Square root: sqrt(49) gives ans=7
• Conjugate: conj(3+2j) gives ans=3-2i
Entering Vectors and Arrays
To enter row vector [1 2 3] and store it in A, type at the prompt
A=[1 2 3]; or A=[1,2,3];
To enter the same numbers as a column vector and store
it in A, type at the prompt either A=[1;2;3]; or A=[1 2
3];A=A’; Note A=A’ replaces A with its transpose. “Transpose” means “convert rows to columns, and vice-versa.”
To enter a vector of consecutive or equally-spaced numbers,
follow these examples:
• [2:6] gives ans=2 3 4 5 6
• [3:2:9] gives ans=3 5 7 9
• [4:-1:1] gives ans=4 3 2 1
To enter an array or matrix of numbers, type, for example,
B=[3 1 4;1 5 9;2 6 5]; This gives the array B and its
transpose B’




3 1 4
3 1 2
′
B = 1 5 9
B = 1 5 6
2 6 5
4 9 5
Other basics of arrays:
• ones(M,N) is an M × N array of “1”
• zeros(M,N) is an M × N array of “0”
• length(X) gives the length of vector X
• size(X) gives the size of array X
For B above, size(B) gives ans=3 3
• A(I,J) gives the (I,J)th element of A.
For B above, B(2,3) gives ans=9
B-2.3
Array Operations
Arrays add and subtract point-by-point:
X=[3 1 4];Y=[2 7 3];X+Y gives ans=5 8 7
But X*Y generates an error message.
To compute various types of vector products:
• To multiply element-by-element, use X.*Y This gives
ans=6 7 12. To divide element-by-element, type X./Y
B-3 PLOTTING
417
• To find the inner product of X and Y, which is
(3)(2)+(1)(7)+(4)(3)=25, use X*Y’. This gives ans=25
• To find the outer product of X and Y, which is


(3)(2) (3)(7) (3)(3)
(1)(2) (1)(7) (1)(3) use X’*Y
(4)(2) (4)(7) (4)(3)
• A(A>2)=0 sets to 0 all values of elements of vector A
exceeding 2. For example,
A=[3 1 4 1 5];A(A<2)=0 gives A=3 0 4 0 5
M/M indexing of arrays starts with 1, while signal and image
indexing starts with 0. For example, the DFT is defined using
index n = 0, 1 . . . N − 1, for k = 0, 1 . . . N − 1. fft(X), which
computes the DFT of X, performs
This gives the above matrix.
fft(X)=X*exp(-j*2*pi*[0:N-1]’*[0:N-1]/N);
A common problem is when you think you have a row vector when in fact you have a column vector. Check by using size(X); in the present example, the command gives
ans=1,3 which tells you that X is a 1 × 3 (row) vector.
• The following functions operate on each element of an
array separately, giving another array:
B-2.4
Solving Systems of Equations
To solve the linear system of equations
1 2 x
17
=
3 4 y
39
using
sin,cos,tan,exp,log,log10,sqrt
cos([0:3]*pi) gives ans=1 -1 1 -1
A=[1 2;3 4];Y=[17;39];X=A\Y;X’
• To compute n2 for n = 0, 1 . . . 5, use
[0:5].b2 which gives ans=0 1 4 9 16 25
gives ans=5.000 6.000, which is the solution [x y]′ .
To solve the complex system of equations
16 + 32 j
1+2j 3+4j x
=
48 + 64 j
5+6j 7+8j y
• To compute 2n for n = 0, 1 . . . 5, use
2.b[0:5] which gives ans=1 2 4 8 16 32
Other array operations include:
[1+2j 3+4j;5+6j 7+8j]\[16+32j;48+64j] gives
• A=[1 2 3;4 5 6];(A(:))’
Stacks A by columns into a column vector
and transposes the result to a row vector. In the present
example, the command gives ans=1 4 2 5 3 6
• reshape(A(:),2,3)
Unstacks the column vector to a 2×3 array which, in this
case, is the original array A.
• X=[1 4 1 5 9 2 6 5];C=X(2:8)-X(1:7)
Takes differences of successive values of X. In the present
example, the command gives C=3 -3 4 4 -7 4 -1
• D=[1 2 3]; E=[4 5 6]; F=[D E]
This concatenates the vectors D and E (i.e., it appends
E after D to get vector F). In the present example, the
command gives
F=1 2 3 4 5 6
• I=find(A>2) stores in I locations (indices) of elements
of vector A exceeding 2.
find([3 1 4 1 5]<2) gives ans=2 4
ans=
2 − 2i
,
6 + 2i
which is the solution.
These systems can also be solved using inv(A)*Y, but this
is a bad idea, since computing the matrix inverse of A takes much
more computation than just solving the system of equations.
Computing a matrix inverse can lead to numerical difficulties
for large matrices.
B-3 Plotting
B-3.1
Plotting Basics
To plot a function x(t) for a ≤ t ≤ b:
• Generate, say, 100 values of t in a ≤ t ≤ b using
T=linspace(a,b,100);
• Generate and store 100 values of x(t) in X
www.dbooks.org
418
APPENDIX
• Plot each computed value of X against its corresponding
value of T using plot(T,X)
• If you are making several different plots, put them all on
one page using subplot.
subplot(324),plot(T,X) divides a figure into a 3by-2 array of plots, and puts the X vs. T plot into the 4th
place in the array (the middle of the rightmost column).
One problem with MathScript that does not arise with MATLAB
is that in MathScript subplot(324) opens 6 figures, even if
only one or two of them will actually be used for plots. This is
inelegant but harmless.
Print out the current figure (the one in the foreground; click
on a figure to bring it to the foreground) by typing print
Print the current figure to a encapsulated postscript file
myname.eps by typing print -deps2 myname.eps. Type
help print for a list of printing options for your computer.
For example, use -depsc2 to save a figure in color.
To make separate plots of cos(4t) and sin(4t) for 0 ≤ t ≤ 5 in
a single figure, use the following:
T=linspace(0,5,100);X=cos(4*T);Y=sin(4*T);
subplot(211),plot(T,X)
subplot(212),plot(T,Y)
These commands produce the following figure:
0.5
0
Common problems encountered using plot:
T and X must have the same lengths; and neither T nor X
should be complex; use plot(T,real(X)) if necessary.
The above linspace command generates 100 equally
spaced numbers between a and b, including a and b. This is
not the same as sampling x(t) with a sampling interval of b−a
100 .
To see why:
• linspace(0,1,10) gives 10 numbers between 0 and 1
inclusive, spaced by 0.111;
• [0:.1:1] gives 11 numbers spaced by 0.1.
Try the following yourself on M/M:
• T=[0:10];X=3*cos(T);plot(T,X)
This should be a very jagged-looking plot, since it is only
sampled at 11 integers and the samples are connected by
lines.
• T=[0:0.1:10];X=3*cos(T);plot(T,X)
This should be a much smoother plot since there are now
101 (not 11) samples.
• plot(X). This should be a blue smear! It is about 200
cycles squished together.
−0.5
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
1
0.5
• plot(X(1:100)) This “zooms in” on the first 100
samples of X to see the sinusoid. It is also possible to zoom
in by clicking on the figure.
B-3.3
0
More Advanced Plotting
Plots should be labelled and annotated. Use:
−0.5
−1
Plotting Problems
• T=[1:4000];X=cos(2*pi*440*T/8192);
sound(X,8192) This is musical note “A.”
sound(X,Fs) plays the vector X as sound,
at a sampling rate of Fs samples/second.
1
−1
B-3.2
MATLAB AND MATHSCRIPT
• title(’Myplot’) adds the title “Myplot”
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
The default is that plot(X,Y) plots each of the 100 ordered
pairs (X(I),Y(I)) for I = 1, . . . , 100, and connects the
points with straight lines. If there are only a few data points to
be plotted, they should be plotted as individual ordered pairs, not
connected by lines. This can be done using plot(X,Y,’+’).
• xlabel(’t’) labels the x axis with “t”
• ylabel(’x’) labels the y axis with “x”.
• $\omega$ produces ω in title,xlabel and
ylabel. Similarly for other Greek letters. Note ’, not ‘,
should be used everywhere.
B-5 MISCELLANEOUS COMMANDS
419
• axis tight contracts the plot borders to the limits of
the plot itself.
from 0 to 1 for color images; this must be done separately for
each component:
• axis([a b c d]) changes the horizontal axis limits to
a ≤ x ≤ b and the vertical axis limits to c ≤ y ≤ d.
X(:,:,1)=X(:,:,1)/max(max(X(:,:,1)));
X(:,:,2)=X(:,:,2)/max(max(X(:,:,2)));
X(:,:,3)=X(:,:,3)/max(max(X(:,:,3)));
• grid on adds grid lines to the plot.
• plot(T,X,’g’,T,Y,’r’) plots on the same plot
(T,X,Y must all have the same lengths)
X vs. T in green and Y vs. T in red.
There is much, much more that can be done. Type help plot
to see how to do it.
B-4 Image Commands
B-4.1 Reading Images into M/M
X=imread(’picture.jpg’) reads the (M × N) JPEG image picture.jpg into the (M × N) M/M array X if picture.jpg is a
grayscale (black and white) image.
For color images, X=imread(’picture.jpg’) reads
the (M × N) JPEG image picture.jpg into the (M × N × 3)
3-D M/M array X if picture.jpg is an RGB color image. Then
X(:,:,1) is the red component of the image, X(:,:,2) is
the green component, and X(:,:,3) is the blue component
(see Chapter 10).
The following image formats can be handled using imread:
JPEG, TIFF, BMP, PNG, HDF, PCX. The extension of the
picture filename is used to determine the type of image.
prior to using imagesc(X). This was done for all programs
for Chapter 10.
B-4.3
Image Computations in M/M
To process X using M/M, X must be converted from uint8 format to the double-precision, 64-bit, floating-point format used
by M/M computation. This can be accomplished by inserting
X=double(X) after reading in X.
conv2(X,H) computes the 2-D discrete-space convolution
of X and H. If X is (M1 × M2 ) and H is (L1 × L2 ) then Y is
(N1 ×N2 ), where Ni = Mi +Li −1 for i = 1, 2 (see Section 3-6.2).
conv2(X,H,’valid’) computes the ’valid’ 2-D discretespace convolution, defined in Section 7-12.3.
fft2(X,M,N) computes the (M ×N) 2-D DFT of the image
in array X. If the size of X is smaller than (M × N), it is zeropadded to size (M × N). If M and N are not specified (i.e.,
fft2(X)), then M and N are set to the size of the image. Section
3-9 discusses some issues in the computation and display of
fft2(X,M,N).
real(ifft2(F,M,N)) computes the (M × N) inverse
2-D DFT of F. Even if F is conjugate symmetric, roundoff error
will incorrectly make ifft2(F,M,N) a complex array. So
the real part of ifft2(F,M,N) should always be computed.
Section 3-9 discusses some issues in the computation and
display of real(ifft2(X,M,N)).
B-4.2 Displaying Images in M/M
An image will be displayed using the default axis setting. This
usually alters the aspect ratio of the displayed image. This can
be corrected by axis image, which will display the array as
being (M × N).
figure,imagesc(X),colormap(gray) displays the
(M ×N) array X as a grayscale (black-and-white) image. Adding
axis off suppresses the numbers along the axes, which are
in MATLAB format (see Chapter 3). imagesc(X) scales the
array so that its values range from 0 to 1 for display purposes
(see Chapter 5). image(X) omits this scaling, so in general it
should not be used.
For color images, figure,imagesc(X) displays the
(M × N × 3) 3-D MATLAB array X as a color image,
with colors determined as discussed above. Despite its name,
imagesc(X) does not scale the array so that its values range
B-5 Miscellaneous Commands
B-5.1
Rectangular-to-Polar Complex
Conversion
If an M/M result is a complex number, then it is presented√in its
rectangular form a+bj. M/M recognizes both i and j as −1,
so that complex numbers can be entered as 3+2j or 3+2i.
To convert a complex number X to polar form, use
abs(X),angle(X) to get its magnitude and phase (in
radians), respectively. To get its phase in degrees, use
angle(X)*180/pi
Note atan(imag(X)/real(X)) will not give the correct
phase, since this formula is only valid if the real part is positive.
angle corrects this.
www.dbooks.org
420
The real and imaginary parts of X are found using real(X)
and imag(X), respectively.
B-5.2 Polynomial Zeros
To compute the zeros of a polynomial, enter its coefficients as
a row vector P and use R=roots(P). For example, to find
the zeros of 3x3 − 21x + 18 (the roots of 3x3 − 21x + 18 = 0)
use P=[3 0 -21 18];R=roots(P);R’, giving ans=
-3.0000 2.0000 1.0000, which are the roots.
To find the monic (leading coefficient one) polynomial
having a given set of numbers for zeros, enter the numbers as a column vector R and use P=poly(R). For example, to find the polynomial having {1, 3, 5} as its zeros, use
R=[1;3;5];P=poly(R), giving P=1 -9 23 -15. The
polynomial is therefore x3 − 9x2 + 23x − 15.
Note that polynomial are stored as row vectors, and roots are
stored as column vectors.
B-5.3 Discrete-Time Commands
• stem(N,X) produces a stem plot of X vs. N
• conv(X,Y) convolves X and Y.
• fft(X,N) computes the N-point DFT of X.
• ifft(F) computes the inverse DFT of F. Due to roundoff
error, use real(ifft(F)).
• sinc(X) compute sin(ππx x) for each element.
APPENDIX
MATLAB AND MATHSCRIPT
Index
Index
1-D Fourier transforms, 47–53
1-D continuous-time signals, 41–43
1-D continuous-time systems, 43–47
1-D discrete-time signals and systems, 59–66
1-D estimation examples, 300–303
1-D fractals, 314–320
2-D, 2
2-D continuous-space Fourier transform, 94–107
2-D continuous-space images, 91–93
2-D convolution, 94
2-D discrete Fourier transform, 119–121
2-D discrete space, 113–118
2-D discrete-space Fourier transform, 118–119
2-D estimation, 309–313
2-D fractals, 320–322
2-D impulse, 91
2-D sampling theorem, 107–113
2-D slices, 19
2-D spline interpolation, 149–150
2-D wavelet transform, 228–232
a posteriori, 293
a posteriori pdf, 293
a priori, 293
a priori information, 292
a priori pdf, 293
absorption coefficient, 18
activation constant, 392
activation function, 391
active pixel sensors, 4
additive, 336
additive noise, 181, 184
Airy disc, 7
aliased, 54, 141
aliasing, 73
analysis filter bank, 216
analysis filters, 214
angular frequency, 21
angular resolution, 9
antialias filtering, 141
antialias lowpass filtering, 141–143
application of MRF to image segmentation, 327–329
APSs, 4
associative property, 45
autocovariance functions, 286
average, 213, 217, 221
average image, 229
axial resolution, 25
B-spline, 129
B-splines interpolation, 143–149
backpropagation, 397, 398
bandlimited, 53, 107
basis functions, 203, 206
basis pursuit, 238
basis pursuit denoising, 238
batches, 402
Bayes’s rule, 293
Bayesian estimators, 296
bed of nails, 107
Bessel function, 8, 103
bilateral, 52
bilateral power spectral density, 281
bilinear, 149
binary, 325
binary image, 326
binomial probability density function, 265
black, 335, 337
blackbody, 11
blackbody radiation law, 11
blurring, 181
box image, 91
brickwall lowpass filter, 184
brightness, 5
butterfly diagram, 77
Canny edge detector, 173
CCDs, 4
421
www.dbooks.org
422
CDF, 168
center of mass, 295
central limit theorem, 144
centroids, 380
characteristic function, 275
charge-coupled device, 4, 335
circularly shift, 123
class index, 354
class-assignment algorithm, 323
classification, 322
classification by MAP, 358–360
classification by MLE, 357–358
classification of rotated images, 366–367
classification of spatially scaled images, 361–366
classification of spatially shifted images, 360–361
Classification Rule, 373
clique, 323
CMOS, 4
CMYK, 335
coin-flip experiment, 298–300
color, 335
color cube, 336
color image classification, 367–373
color science, 339
color systems, 335–340
color-image deblurring, 343–346
colorimetry, 339
commutative property, 45
comparison of 2-D interpolation methods, 150
complement, 255, 256
complex conjugate, 411
complex number, 411
compressed, 206
compressed sensing, 203, 236–238
compressed sensing examples, 242–249
compression, 203
computation of continuous-time Fourier transform using the
DFT, 82–84
computation of the 2-D DFT using MATLAB, 121–124
computed tomography, 2, 18
computing solutions to underdetermined equations, 238–240
conditional expectation, 270
conditional pdf, 265
conditional pmf, 268
conditional probability, 259–261
confusion matrix, 358
conjugate symmetry, 50, 67, 95, 120
connecting the dots, 129
INDEX
constant speed, 196
continuous random variables, 261
continuous space, 285, 361
continuous-space Fourier transform, 94
continuous-space systems, 93–94
continuous-time Fourier transform, 47
continuous-time random process, 278
continuous-time signals, 207
convolution integral, 45
convolved image, 166
cooled detectors, 12
coordinates, 376
correlation, 270, 354, 356
correlation coefficient, 270
cost function, 193
covariance, 270
covariance matrix, 273
cross-correlation, 360
cross-covariance matrix, 273
cross-validation, 404
CSFT, 94
CT, 2
CT scan, 18
CTFT, 47
cumulative distribution function, 168
cutoff index, 184
cyan, 335, 337
cyclic, 73
cyclic convolution, 209–213
Daubechies wavelet function, 203, 225
deblurring, 160, 181
decimating, 212
decimation, 141, 203
decomposed, 76
decomposition structure, 218
deconvolution, 160
deconvolution using the DFT, 80–82
deep learning, 393
degree of similarity, 354
denoising, 160, 181, 182
denoising by lowpass filtering, 183–188
denoising by thresholding and shrinking, 232–235
denoising color images, 346–347
density slicing, 9
derivation of backpropagation, 403–404
detail, 214, 217, 221
detail images, 229
detector resolution, 9
INDEX
423
deterministic, 181, 292
deterministic deconvolution, 309
deterministic estimate, 319
deterministic versus stochastic Wiener filtering, 307–309
DFT, 70, 119
difference operator, 171
diffraction pattern, 7
dimensionality reduction, 377
direct, 181
direct and inverse problems, 181–183
direct problem, 236
discrete, 69
discrete Fourier transform, 70–76, 119
discrete random variables, 261
discrete space, 113, 285
discrete time, 59
discrete-space Fourier transform, 118, 183
discrete-space image, 113
discrete-space spatial frequency, 184
discrete-time eternal sinusoid, 61
discrete-time Fourier transform, 66–70
discrete-time frequency, 61
discrete-time (Kronecker) impulse, 61
discrete-time random process, 278, 279
discrete-time rectangle function, 69
discrete-time signal, 59
discrete-time sinc function, 68
discrete-time wavelet transforms, 218–223
disjoint, 256
disk-image, 92
display dynamic range, 160
displaying images, 90–91
dissimilarity index, 325
distribution function, 168
distributive property, 45
downsampled average signal, 214
downsampled detail signal, 214
downsampling, 140–141
DSFT, 118, 183
DSSF, 184
DTFT, 66
DTFT of the Smith-Barnwell condition, 221
duration, 43, 61, 143
dynamic range, 167
edge, 171
edge detection, 160, 171–176
edge image, 341
edge indicator, 173
edge thinning, 173, 175
edge-detection gradient, 173
effects of shifts on pdfs and pmfs, 263–265
electromagnetic, 3
EM, 3
emission, 11
emissivity, 12
energy, 43
energy spectral density, 52, 67
estimate, 234
estimation, 255, 292
estimation accuracy, 193
estimation error, 303
estimation methods, 292–298
estimation problem, 292
Euler’s identity, 411
even, 49, 77, 93
even symmetry, 49
even values of n, 77
event, 255
examples of image interpolation applications, 152–155
excitation, 20
expansion of signals in orthogonal basis functions, 206–209
expectation, 269
false-color, 335
false-color image, 90
fast Fourier transform, 70, 76–80, 119
fast iterative shrinkage and thresholding algorithm, 242
FFT, 70, 119
field gradient, 20
fill in the gaps, 129
filtering, 213
filtering of signals and images, 203
finite-impulse response, 68
FIR, 68
first iteration, 381
FISTA, 242
focal underdetermined system solver, 240
focusing, 23
FOCUSS, 240
four-color printing process, 338
fractal, 290
fractal-like, 314
frequency exponent, 315
frequency filtering, 50
frequency response, 50, 51
functions of random variables, 269–272
fundamental frequency, 186
www.dbooks.org
424
fundamental period, 61
gamma transformation, 162
Gaussian, 266, 279
Gaussian pdf, 293
Gaussian random vector, 275
Gaussian random vectors, 275–278
geometric pmf, 259
Gibbs distribution, 325
global minimum, 397
GPUs, 393
gradient, 397
gradient descent, 397, 402
gradient fields, 21
gradient threshold, 173
graphical processing units, 393
grayscale, 335, 339, 341
grayscale image, 90
gyromagnetic ratio, 21
Haar, 203
Haar transform, 226
Haar wavelet transform, 213–218
Hammersley-Clifford theorem, 325
Hamming window, 68, 186, 188
Hankel transform, 103
hard thresholding, 242
hexagonal sampling, 110
hidden layers, 393
histogram, 168
histogram equalization, 160, 167–170
histogram equalization and edge detection, 340–343
histogram-equalized image, 167
horizontal-direction vertical-edge detector, 172
HSI, 339
HSV, 339
hue, 339
hyperplane, 394
ICM, 328
ideal lowpass filter, 50
iff, 412
IID, 279, 300
ill-posed problem, 237
image, 107
image array size, 131
image classification by correlation, 354–356
image deconvolution, 191–194
image dynamic range, 160
INDEX
image enhancement, 160
image physical size on computer screen, 131
image pixel area, 131
image plane, 5
image recognition, 354
image reconstruction fidelity, 107
image recording, 196
image restoration, 160, 181
image sharpening filter, 166
image shifting, 152
image spatial resolution, 9
image spectra, 92
image texture, 323
images, 92
imaginary, 411
implementation of upsampling using 2-D DFT in MATLAB,
137–140
impulse, 41
impulse response, 8, 44
impulse train, 54
in phase, 338
incremental learning, 402
independent and identically distributed, 279, 300
independent events, 260
independent random variable, 266, 271
infrared, 9
infrared catastrophe, 316
input layer, 393
interpolation, 129, 205
interpolation using sinc functions, 129–130
intersection, 255, 256
interval, 61
interval probability, 262, 263
inverse, 181
inverse problem, 236, 292
IR, 9
IRLS, 239
Ising model, 325
Ising-Gibbs distribution, 326
ISTA, 241
Iterated Conditional Modes, 328
iterative reweighted least squares, 239
iterative shrinkage and thresholding algorithm, 241
jinc function, 106
joint covariance matrix, 369
joint pdf, 265, 357
joint pdfs and pmfs, 265–269
joint probability mass function, 266
INDEX
425
jointly Gaussian random variables, 275
K-Means clustering algorithm, 380–382
K-stage decomposition, 221
knots, 143
L, 43, 44
label, 399
labeled, 390
Lanczos interpolation formula, 130
Landweber algorithm, 241–242
Landweber iteration, 241
Laplacian, 164
Laplacian operator, 164
Laplacian pdf, 293
Laplacian’s spatial frequency response, 165
Larmor frequency, 21
LASSO cost functional, 234, 238
lateral resolutions, 25
LCD, 5, 91
learning, 354, 390
least absolute shrinkage and selection operator, 234, 238
least-squares estimation, 303–307
least-squares estimator, 293, 295
lens law, 4
letter recognition, 358
likelihood, 255, 293
likelihood function, 292, 370
linear, 43, 44
linear convolution, 73
linear least-squares estimate, 304
linear operator, 269
linear programming, 238
linear shift-invariant, 94, 117
linear time-invariant, 41, 44
linear transformation, 160
liquid crystal display, 5, 91
local gradient, 403
local minimum, 397
locality, 325
localization, 21
location-independent covariance matrix, 369
log-likelihood function, 293, 302
logarithmic, 162
logarithmic warping, 362
logarithmically transformed scale factors, 363
long-wave IR, 11
lowpass filter, 99
LSE, 292, 293, 295
LSI, 94, 117
LTI, 41, 43, 44
LTI filtering of random processes, 282–285
LU decomposition, 239
luminance, 338
LWIR, 11
MADs, 78
magenta, 335, 337
magnetic quantum number, 21
magnetic resonance, 19
magnetic resonance imaging, 2, 19–23, 236
magnetized, 20
MAP, 292, 293, 323
marginal pdf, 357, 369
marginal pdfs, 265, 357
marginal pmfs, 268
Markov random fields, 255, 292, 322–326
“mask” image, 164
maximum a posteriori, 293
maximum likelihood estimate, 357
maximum likelihood estimation, 293
mean, 269
mean square error, 295
mean value, 269
mean vector, 273
mean-square error, 399
measurement precision, 193
median filtering, 194
MEP, 359
mesh plot, 90
middle-wave IR, 11
minimum error probability, 359
minimum ℓ1 norm, 237
MLE, 292, 293, 323, 358
MLE criterion, 370
modified functions, 212
modified MLE criterion, 371
modulated signal, 78
moment of inertia, 295
morphing, 152
motion blur, 195
motion-blur deconvolution, 195–197
MR, 19
MRF, 292, 322
MRI, 2, 19, 236
MSE, 295
multilayered perceptrons, 390
multiplications and additions, 76, 78
www.dbooks.org
426
mutually exclusive, 260
MWIR, 11
N-D CSFT, 275
N-dimensional continuous-space Fourier transform, 275
N-dimensional spatial frequency vector, 275
N-point, 71
near IR, 11
nearest-neighbor, 107, 145
neighborhood, 323
neural network training examples, 404–406
neural networks, 390–396
neurons, 390
NIR, 11
NMR, 19
NN, 107, 145
nodes, 393
noise, 181
noise variance, 182
noise-amplification problem, 193
noisy image, 184
non-WSS, 317
normal, 266
normal distribution, 301
normalized intensity, 161
notch filter, 188
notch filtering, 188–191
nuclear magnetic resonance, 19
Nyquist frequency, 54
Nyquist sampling rate, 54, 107
Nyquist-sampled version, 108
object plane, 5
observation, 292, 310
observed value, 292, 296, 301
octaves, 205, 218
odd, 49, 77, 93
odd symmetry, 49
odd values of n, 77
optical imagers, 3–13
original, 133
original sampled image, 133
orthogonal, 206, 373
orthogonal expansion, 206
orthogonality, 304
orthogonality property, 66, 71
orthonormal, 207
orthonormal expansion, 221
output layer, 393
INDEX
overdetermined, 238
parallel-axis theorem, 295
Parseval’s theorem, 48, 67
partition function, 325
path attenuation, 19, 247
pdf, 182, 262
perceptrons, 390
perfect-reconstruction conditions, 219
perfectly red, 337
periodic, 69
periodogram spectral estimator, 314
perturbation, 208
perturbed, 208
phase shifting, 23
phase spectrum, 98
photometry, 339
piecewise-Mth-degree polynomial, 223
piecewise polynomial, 223
pixel, 113
pixel dimensions, 131
pixel value, 168
pixel-value transformation, 160–163, 168
pixels of the DFT image, 243
pmf, 262
point spread function, 8, 94, 173
point spread functions, 92
polar form, 411
potential energy, 325
power laws, 314
power spectral, 281
probabilistic, 181
probability, 255–259
probability density function, 182, 262
probability mass function, 262
probability tree, 257
projection-slice theorem, 249
pseudo inverses, 238
pseudo-inverse solution, 238
pseudocolor, 9
PSF, 8, 92, 94, 173, 181
pulse, 41
pulser, 23
QMF, 203, 219
QMF relation, 219
quadrature, 338
quadrature mirror filter, 203, 219
INDEX
radar imagers, 13–18
radar shadow, 16
radial brickwall lowpass filter, 106
radial discrete-space frequency, 165
radial frequency, 110
radially bandlimited, 110
radio frequency, 20
Radon transform, 19, 247
random, 255
random fields, 255, 285–286
random processes, 255, 278–282
random variables, 255, 261–263
random vectors, 272–275
range, 13
Rayleigh resolution criterion, 9
Rayleigh’s theorem, 48
real, 411
realization, 278
receive beamforming unit, 23
reclining matrix, 373, 375
reconstructed image, 107
reconstruction structure, 218
recorded, 181
rectangle function, 41
rectangular form, 411
reflectance, 11
regularity, 44
regularization parameter, 193
regularized, 193
removing interference, 181
resolution area, 131
RF, 20
RGB, 335
ring impulse, 103
rotated, 195, 366
rotation cross-correlation, 366
rotation matrix, 101
rotationally invariant, 102
RV, 273
salt-and-pepper, 194
sample function, 278
sample mean, 300, 302
sampled image, 107
sampled signal, 54, 129
sampled version, 107
samples, 53
sampling interval, 53, 129
sampling length, 107
427
sampling rate, 53, 107, 129
sampling theorem, 53–59
SAR, 14
SAR PSF, 16
saturation, 339
scaling, 218, 223
scaling constant, 314
scene spatial resolution, 9, 14
SD, 397, 403
SDC, 306
segment, 171, 322
self-similarity, 314
separable, 93, 228
separating hyperplane, 394
sharpened image, 164
Shepp-Logan phantom, 230
shift invariance, 94
shorthand notation, 275
shot noise, 194
shrinkage, 203
shrinking, 232, 233
shrnking, 235
SI, 94
side information, 292, 293
side-looking airborne radar, 14
sifting, 41
sifting property, 91
signal flow graph, 77
signal-to-noise, 184
sinc, 50
sinc function, 69
sinc interpolation formula, 55, 107, 129
singular value decomposition, 373
singular values, 373
sinusoidal interference, 188
SLAR, 14
slice, 18
slices, 2
Smith-Barnwell condition, 203
Smith-Barnwell condition for perfect reconstruction, 220
SNR, 184
Sobel edge detector, 173, 341
soft thresholding, 242
SPARSA, 242
sparse, 215
sparse reconstruction by separable approximation, 242
sparsification using wavelets of piecewise-polynomial signals,
223–228
www.dbooks.org
428
spatial frequency, 94
spatial frequency response, 95
spatial frequency response of the Laplacian operator, 164
spatial interval, 113
spatial resolution, 9
spatial sampling, 113
spatial-scaling factors, 362
spatially scaled, 361, 363
spatially shifted, 363
spatially warped images, 362
spectral estimation, 313–314
spectrum, 51
spin quantum number, 20
square wave, 186
squared ℓ2 norm, 237
standard deviation, 270
static field, 20
statistical self-similarity, 314
steepest descent, 397, 403
steepest-descent, 397
steering, 23
steering the beam, 24
stochastic, 292
stochastic deconvolution, 306, 310
stochastic denoising, 305
stochastic denoising/deconvolution, 309
stochastic denoising filter, 305
stochastic gradient descent, 402
stochastic processes, 278
stochastic Wiener, 306
stochasticity, 292
sub–Nyquist-sampled version, 108
subband coding, 203
subband decomposition, 203
subtractive, 337
summary of image enhancement techniques, 176
superposition, 44
superposition integral, 45
supervised learning, 390
supervised training, 354
support, 43, 61, 143
SVD, 373
synthesis filters, 214
synthetic-aperture radar, 14
tall matrices, 373
terminals, 393
tetrahedral die, 258
thermal imagers, 11
INDEX
thermal infrared imager, 9
three subtractive colors, 338
threshold, 376
threshold level, 232
thresholding, 203, 233, 235
thresholding and shrinking, 235
thumbnail image, 231
TI, 43, 44
Tikhonov regularization, 193
tiling, 110
time delaying, 23
time scaling, 41
time-invariant, 44
time-scaled, 43
Toeplitz blocks, 246
trade-off parameter, 234
training, 354, 390, 396
training images, 367, 373, 377
training matrix, 378
training neural networks, 396–403
transducers, 23
transform, 206
transmit beamforming unit, 23
transmit/receive switch, 23
transpose, 273
tree-structured filter banks, 203–206
true, 181
true resolution area, 131
true value of index k, 356
true-color, 335
truncated SVD, 377
TSFBs, 203
twiddle factors, 79
twiddle multiplications, 79
TWISTA, 242
two-dimensional, 2
two-step iterative shrinkage and thresholding algorithm, 242
ultrasound, 23
ultrasound imager, 23–27
ultraviolet catastrophe, 316
uncooled detectors, 12
uncorrelated, 270, 271, 273, 280
underdetermined, 236
underdetermined system, 237
unfiltered, 307
uniform, 272
uniform pdf, 299
union, 255
INDEX
429
Universal Approximation Theorem, 393
unknown, 292
unsharp masking, 160, 163–167
unsupervised learning, 373, 390
unsupervised learning and classification, 373–377
unsupervised learning examples, 377–380
unsupervised training, 354
upsampled version, 133
upsampling and downsampling modalities, 130–132
upsampling and interpolation, 133–137
upsampling factor, 133
valid convolution, 245
van Cittert iteration, 241
variance, 270
VE, 171
vertical edge, 171
vertical-direction horizontal-edge detector, 172
voltage outputs, 5
Voronoi sets, 380
voxels, 22
warping, 152
wavelength, 25
wavelet, 218, 223
wavelet transform, 203
wavelet transform matrix, 237
wavenumbers, 94
weak-sense stationary, 280
weighting coefficients, 76
white, 282
white random process, 282
wide-sense stationary, 280
Wiener filter, 193, 310
Wiener process, 317
windowed sinc functions, 130
within-class, 367
WSS, 280, 304
X-ray computed tomography, 18–19
yellow, 335, 337
YIQ, 335
zero-padded functions, 80
zero-padding, 74
zero-stuffing, 203, 205, 212
www.dbooks.org
Andrew E. Yagle Fawwaz T. Ulaby
University of Michigan
Andrew E. Yagle is professor of Electrical Engineering and Computer
Science at the University of Michigan. He is the recipient of several
research and teaching awards including the NSF Presidential Young
Investigator, ONR Young Investigator, College of Engineering
Teaching Excellence Award, the Eta Kappa Nu Professor of the Year
Award, and the Class of 1938E Distinguished Service Award.
He is a past member of the IEEE Signal Processing Society Board
of Governors, the Image and Multidimensional Signal Processing
Technical Committee, the Digital Signal Processing Technical
Committee, and the Signal Processing Theory and Methods
Technical Committee. He is a past associate editor of the IEEE
Transactions on Signal Processing, IEEE Signal Processing Letters,
Multidimensional Systems and Signal Processing, and the IEEE
Transactions on Image Processing.
Fawwaz T. Ulaby is the Emmett Leith Distinguished Professor of
Electrical Engineering and Computer Science and former Vice
President for Research at the University of Michigan. He is a member
of the National Academy of Engineering and recipient of the IEEE
James H. Mulligan, Jr. Education Medal. His Applied Electromagnetics
textbook is used at over 100 US universities.