Introduction to Image
Processing
(Signal Processing)
NEU 259
Gina Sosinsky
May 13, 2012
Quantization of images is key….
The Electron Microscope
(Example of a Physical System)
f
H0
H1
Simian SV40
Hn
g
Systems and Signals
• Physical systems can be modeled as input signals that
are transformed by the system, or cause the system
to respond in some way, resulting in other signals,
e.g., all imaging devices.
f0
f1
fn
f0
H0 g0
H1
f
g
g1
H0
H1
Hn gn
H0
f1
g0
g
H1
fn
g1
Hn
gn
f
H
g
What is image processing?
• The analysis, manipulation, storage, and display
of graphical images from sources such as
photographs, drawings, and video.
• Any technique either computational or
photographic which alters the information in
an image.
The analysis, manipulation, storage, and
display of signals in a multi-dimensional
space.
Common Topics in Image Processing
•
•
•
•
•
•
•
•
•
•
•
Acquisition of Images.
Representation and Storage.
Correction of Imaging Defects (Image Restoration).
Image Enhancement (Real and Reciprocal Spaces).
Image Segmentation.
Feature Recognition and Classification.
Boolean Operations.
Morphological Operations.
Image Measuring.
Three-dimensional Imaging.
Image Visualization (2D and 3D).
What is image processing?
• Any technique either computational or photographic which alters
the information in the image.
– Examples
• Reversing the contrast of an image (black becomes white
and vice versa).
• Maximizing the values for color tables (histogram
stretching)
• Pattern recognition & analysis (correlations).
– As Misell points out, image processing will not turn a poor
image into a good one, but extracts the maximum amount of
structural information from the original.
Types of operations
an input image a[m,n] ==> an output image b[m,n] (or another representation)
* Point- the output value at a specific coordinate is dependent only on the input
value at that same coordinate.
* Local - the output value at a specific coordinate is dependent on the input
values in the neighborhood of that same coordinate.
* Global - the output value at a specific coordinate is dependent on all the
values in the input image.
Image Enhancement
Objective: process an image to obtain the
most information from it for a particular
application.
Example: Common operations to enhance
images depend on the convolution of masks
and the Fourier transform.
How do I know that my
new/corrected/enhanced image is
correct?
• Does it resemble the original image?
• Are any unusual features being introduced (e.g.
aliasing)?
• Is it consistent with other results outside of this
image (e.g. biochemistry, NMR, MRI etc.)?
Two simple operations
• Reversing the contrast
new_pix =
max - old_pix +
min
Histogram stretching
(contrast stretching)
Can use histogram to replace out-lying points.
A bit of history
• For electron micrographs, first applications:
Markham et al. in 1963, Klug and Berger in
1964.
– Involved the signal from a periodic specimen
was separated from the non-periodic noise
(electron crystallography).
– Seminal publications:
• Crowther, DeRosier, Klug, Nature 1968
• Unwin and Henderson, JMB 1976
Specific topics
•
•
•
•
•
•
Shannon sampling and Nyquist limits
Fourier analysis
Projection Theorem
Convolution theorem
Resolution & Filtering
Correlation analysis
Shannon Sampling & Nyquist limits
Nyquist Limit is defined as 2  r/M.
M = magnification;
r = step size of scanner or camera
Basically, the Shannon sampling theorem tells you that you
need at least 2 data points to sample a function. But…in
practice, in order to get a given resolution, you need to use
r/(3  M)
or
r/(4  M)
(e.g. if you want 12 Å resolution, you need to use a pixel
size of 3-4 Å)
This is referred to as oversampling your data.
Undersampling can result in an image processing artifact
known as aliasing.
Shannon Sampling & Nyquist Limits
Effect of sampling interval on recovery of information. In this example, a
sampling interval of 32 appears to be just fine enough to recover the
shape of the 1D function without loss of information. At coarser
sampling intervals (4-16), the subtler features in the data are lost. In
practice, one aims to digitize the data at a fine enough interval to be
certain that no information is lost. Thus, using the three-times pixel
resolution criteria, in this example one ought to sample the data 96 (= 3
x 32) or greater to be certain to recover all the information contained in
the data.
Raster size versus Nyquist limits
Jean Baptiste Joseph Fourier
(1768-1830)
“The profound study of nature
is the most fertile source of
mathematical discoveries.”
Fourier Theory using trigonometrical series expansion done in ~1807
http://www-groups.dcs.st-and.ac.uk/~history/Mathematicians/Fourier.html
Fear not the Fourier
Transform, it is your friend!
Continuous FT
Discrete FT
(what we calculate)
Inverse FT
r = T-1 (T(r)) Inverse Theorem
Terms for Fourier analysis
• Real space: Our coordinate system (x,y,z)
• Reciprocal, Fourier, inverse, tranform space:
Coordinate system after Fourier transformation
• Amplitudes and phases or real and imaginary parts due
to complex number analysis
– r(xyz)  Fourier transform (F (hkl)exp i(hkl))
– Where F(hkl) is the amplitude and  is the phase and
– r(xyz) is the density function
Need both phase and amplitude data
Original
correct amplitudes
random phases
random amplitudes
correct phases
Reciprocal Space: The
Final Frontier
• Dimensions in the object (REAL SPACE) are
inversely related to dimensions in the transform
(RECIPROCAL SPACE).
• Small spacings or features in real space are
represented by features spaced far apart in
reciprocal space. Resolution is inversely
proportional to spacings.
• Outer regions of the transform arise from fine
(high resolution) details in the object. Coarse
object features contribute near the central (low
resolution) region of the transform.
Advantages of using Fourier analysis
• The recorded diffraction pattern of an object is the square
of the Fourier transform of that object.
• FT are linear process (like multiplication and division).
Can go backward and forward easily if functions are
known. Advantages for micrographs where the FT is
calculated and we want to do noise reduction, filtering or
averaging.
• Projection Theorem (next slides)
• Convolution Theorem: Deconvolution is more easily
computed in Fourier space rather than in real space
(slides after Projection Theorem).
The Fourier Duck
Behold the duck.
It does not cluck.
A cluck it lacks.
It quacks.
It is specially fond of a puddle or pond.
When it dines or sups, it bottoms ups.
• The Fourier Duck originated in a book of optical
transforms (Taylor, C. A. & Lipson, H., Optical
Transforms 1964). An optical transform is a Fourier
transform performed using a simple optical apparatus.
http://www.ysbl.york.ac.uk/~cowtan/fourier/fourier.html
The Fourier Cat and its Transform
FT
FT-1
Resolution in real versus
reciprocal space
The effect of taking only
low angle diffraction to
form the image of a duck
object. A drawing of a
duck is shown, together
with its diffraction pattern.
Also shown are the images
formed (as the diffraction
pattern of the diffraction
pattern) when stops are
used to progressively more
of the high angle
diffraction pattern. (From
Holmes and Blow)
FT
FT-1
FT
FT-1
This is a test – If we take the amplitudes of the Fourier Duck
and the phases of the Fourier Cat and back transform, what
will we get?
Duck amplitudes, Cat Phases
Yes, we get the Fourier Cat!
Projection Theorem
This is the most fundamental principle for 3D
reconstruction from electron micrographs.
Every micrograph we obtain in TEM is a projection (sum)
of everything in the specimen.
When examining 3D objects, 2D images
may not provide the complete picture
The Projection Theorem
Simply stated it says:
The Fourier Transform of the projected structure
of a 3D object is equivalent to a 2D central
section of the 3D Fourier transform of the object.
The central section intersects the origin of the 3d
transform and is perpendicular to the direction of
the projection. The 3d structure is reconstructed
from several independent 2d views by the inverse
Fourier transform of the complete 3d Fourier
transform.
Radon and X-ray transforms
Illustration of Projection Theorem
Projections
Central Sections
Baumeister et al. (1999) Trends Cell Biol., 9, 81-85.
The Convolution Theorem
The convolution theorem is one of the most
important relationships in Fourier theory
It forms the basis of X-ray, EM and neutron
crystallography.
• Holmes and Blow (1965) give a general statement
of the operation of convolution of two functions:
"Set down the origin of the first function in every
possible position of the second, multiply the value
of the first function in each position by the value
of the second at that point and take the sum of all
such possible operations."
c(u) = f(x) * g(x)
Convolution symbol
Also use 
Properties of Convolution
•Convolution is commutative.
c=a*b=b*a
•Convolution is associative.
c = a * (b * d) = (a * b) * d = a * b * d
•Convolution is distributive.
c = a * (b + d) = (a * b) + (a * d)
where a, b, c, and d are all images, either continuous or
discrete.
A simple example of convolution. One function is a drawing of a
duck, the other is a 2D lattice. The convolution of these functions is
accomplished by putting the duck on every lattice point. (From
Holmes and Blow, p.123)
FT
FT-1
Molecule
convoluted with
lattice points
FT
FT-1
Convolution & Fourier Transforms
• Fourier transform of the convolution of two functions is
the product of their Fourier transforms.
T(ƒ *g) = F x G
• The converse of the above also holds, namely that the
Fourier transform of the product of two functions is
equal to the convolution of the transforms of the
individual functions.
T(ƒ x g) = F * G
Computationally, multiplication and fast-Fourier
transform algorithms are speedier processes than
deconvolution.
Words to image process by
Convolution is easy, Deconvolution
is hard.
(Thursday’s lecture)
Need to know:
T(ƒ x g) = F * G
Simple Filtering Operations
•
•
•
•
Low pass filter
High pass filter
Band pass filter
Crystalline masks, Layer line masks
(All the above can have hard or soft edges)
• Median filter
• Sobel filter
• Rotational harmonics filtering (Fourier-Bessel
analysis)
The Fourier Duck
The Fourier Duck: Low pass filtering
• If we only have the low resolution terms of
the diffraction pattern, we only get a low
resolution duck:
High pass filtering
• If we only have the high resolution terms of
the diffraction pattern, we see only the
edges of the duck but see internal features
(missing box function):
Inverse band pass
(missing shell of data)
• The edges are sharp, but there is smearing
around them from the missing intermediate
resolution terms. The core of the duck is at the
correct level, but the edges are weak.
Hard versus soft edged filters
• Gaussian falloffs at the edges prevent
aliasing artifacts.
• Butterworth filters also have soft edges
Gaussian
Butterfield
Crystalline masking
Median Filter (Real space filter
based on image statistics)
• Ranks the pixels in a neighborhood (kernel)
according to their brightness value (intensity). The
median value in the ordered list is used as a
brightness value for the central pixel.
• Excellent rejector of “shot noise” and for
smoothing operations. Outlying pixels are
replaced by a reasonable value -- the median value
in the neighborhood.
Sobel Filter (Real Space
Filters)
• Uses the derivitives of the values and filters
based on the square root of the sum of the
squares for the values.
• Good for edge detection
• (computationally intensive!)
Edge
Detection
Filters
From Russ’ Book on
Image Processing
Filters within Photoshop
Correlation Analysis & Pattern Recognition
Correlation Analysis & Pattern Recognition
Pearson coefficient as
a measure of
correlation of labeling:
a measure of the
correlation (linear
dependence) between
two variables X and Y,
giving a value between
+1 and −1 inclusive.
Rotational Filtering (Fourier-Bessel)
Friedrich Wilhelm Bessel
1784 - 1846
Good General References
• D.L. Misell, "Image Analysis, Enhancement and
Interpretation" (1978) (Practical Methods in
Electron Microscopy series vol. 7, Audrey
Glauert editor) North-Holland publishers
• J. Frank, "Three-Dimensional Electron
Microscopy of Macromolecular Assemblies"
(2004, 2nd edition) Academic Press publishers
• J. Russ, "The Image Processing Handbook"
(1995, 2nd edition) CRC Press
• Image Processing Fundamentals Web Site
http://www.ph.tn.tudelft.nl/Courses/FIP/nofra
mes/fip.html
• Baker lecture notes