Today
Filtering and the Fourier transform
• Introduction, types of filters
Computational Photography
• Spatial filtering, convolution
• Fourier transform
• Frequency domain filtering
Matthias Zwicker
University of Bern
Fall 2009
Filtering
Examples: smoothing
• Filtering: a process that modifies an image
Input image
Filter
Output image
• In mathematical terminology, filters are
also
l called
ll d operators
– Input: function, output: an other function
– Images: 2D functions
Linear smoothing filter
Input
• Two main filter categories
– Linear
– Nonlinear
Median filter (nonlinear)
http://www.ph.tn.tudelft.nl/Courses/FIP/noframes/fip-Smoothin.html
Examples: smoothing
Examples: sharpening
• Bilateral filter
(nonlinear)
• Unsharp masking
(linear filter)
– Preserves edges
Input
Input
Output
http://users.soe.ucsc.edu/~manduchi/Papers/ICCV98.pdf
Output
http://en.wikipedia.org/wiki/Unsharp_masking
1
Examples: edge detection
Linear filters
• Sobel filter
Advantages
• Can achieve many important effects
(smoothing, sharpening, etc.)
• Rich mathematical theory
• Simple and efficient implementation
Input
Output
http://en.wikipedia.org/wiki/Sobel_operator
Linear filters
Linearity
Brief mathematical derivation
• Interpret images as 2D functions f (x, y)
• Linearity
• Shift invariance
• Impulse function
• Impulse response & convolution
• Important not only for images, but any sort of
signal (soundwave, radiowave, etc.)
• More theory
– Often leave out arguments, i.e., image is f
• Definition
„A filter is linear if its output to a
weighted sum of functions is the same as
the weighted sum of outputs for each
individual function“
• Example of linearity
Filter(Image A+Image B) = Filter(Image A)+Filter(Image B)
http://en.wikipedia.org/wiki/LTI_system_theory
Linearity
Shift invariance
Mathematically
• Definition
„A filter is shift invariant if its output to a
spatially shifted input is the same as the
shifted output to the input without shift“
– Images f, g
– Filter L
– Filtered images L{f }, L{g}
• Linearity
– Scalar weithgs a, b
L{af + bg} = aL{f} + bL{g}
2
LSI filters
Impulse function
• Definition
„Filters that are linear and shift invariant
are called linear shift invariant (LSI)
filters“
• Impulse (Dirac delta) function defined as
• LSI filters can be implemented using
convolution
• Impulse at an arbitrary position
• Let‘s see how...
• Sifting property
– Explanation with 1D functions, but 2D works
exactly the same
Impulse response & convolution
Impulse response & convolution
• Remember: because of sifting property, can
write any function as
• Any LSI filter is specified by its impulse
response
f (x) =
Z
f (ξ)δ(x − ξ)dξ
Z
• The output of an LSI is given by the
convolution of the input with the impulse
response
f (ξ)L{δ(x − ξ)}dξ
• Note: symmetry
• Apply LSI
½Z
¾
L{f
{f ((x)}
)} = L
f (ξ)
(ξ)δ(x
( − ξ)
ξ)dξξ
=
• Define impulse response
w(x − ξ) = L{δ(x − ξ)}
• Define convolution Z
f(x) ? w(x) = f(ξ)w(x − ξ)dξ
Visualization
f (t) ? g(t) =
f (x) ? w(x) = w(x) ? f (x)
• Remember: linearity
(g(x) + f (x)) ? w(x) = f (x) ? w(x) + g(x) ? w(x)
Visualization
Z
f (τ )g(t − τ )dτ
f (t) ? g(t) =
Z
f (τ )g(t − τ )dτ
Inputs
Flip and shift one of the two inputs
http://en.wikipedia.org/wiki/Convolution
Multiply and integrate
for each shift t
http://en.wikipedia.org/wiki/Convolution
3
Discrete convolution
Discrete correlation
• In 2D
• Almost the same as convolution, but filter
mask is not flipped
– Impulse response, or filter mask w(x, y)
– Assume filter is non-zero only within range
[-a,a] x [-b,b]
– Input image f (x, y)
w(x, y) ? f (x, y) =
a
X
b
X
s=−a t=−b
a
X
b
X
w(s, t)f (x + s, y + t)
s=−a t=−b
• Implement convolution using correlation
w(s, t)f (x − s, y − t)
• „Shift flipped filter mask across image. At
each point multiply and sum to get output
value.“
Discrete correlation
a
X
w(x, y) ? f (x, y) =
b
X
– Flip filter mask (rotate by 180 degrees)
– Compute correlation
Correlation vs. convolution
w(s, t)f (x + s, y + t)
s=−a t=−b
Correlation vs. convolution
Some linear filters
• Design the filter mask appropriately to
achieve desired effect
– Smoothing
– Sharpening
– Etc
Etc.
4
Smoothing
Sharpening
• Box filter
• Idea: Use second derivative for sharpening
– „Local moving average“
– E.g., 3x3: w(s, t) = 19 ×
• Derivatives using finite differences
1 1 1
1 1 1
1 1 1
– First derivative
∂f
∂
∂x
≈ f (x + 1) − f (x)
– Second derivative
∂2f
∂x2
Input
≈ f (x + 1) + f (x − 1) − 2f (x)
10x10 box filter
Sharpening
Sharpening with the Laplacian
• Idea: Use second derivative for sharpening
• In 2D, several partial derivatives
• Add negative
multiple of
second derivative
to sharpen
• Laplacian: sum of second partial
derivatives along coordinate axes
• Continuous case
∇2 f =
∂2f
∂x2
2
+ ∂∂yf2
• Finite difference approximation
∇2 f = f (x + 1, y) + f(x − 1, y) + f (x, y + 1) + f (x, y − 1) − 4f (x, y)
• Filter mask for Laplacian
Sharpening with the Laplacian
0 1 0
1 -4 1
0 1 0
Unsharp masking
• Algorithm
Output of Lapacian
with scale and offset
1 1 1
1 -8 1
1 1 1
1. Blur input
2. Subtract blurred image from input
•
Difference is called a mask
3. Add difference (mask) to input
• Can also scale mask to increase sharpening
– Highboost filtering
• Highly popular sharpening methods
– Photoshop etc.
– Flexible: arbitrary smoothing kernel, scaling
parameter
Input
Lapacian
sharpening
Filter extended
with diagonal
• Can be combined into one filter mask
– Very similar to sharpening with Laplacian
5
Unsharp masking
Unsharp masking
• Algorithm
1. Blur input
2. Subtract blurred image from input
•
Difference is called a mask
3. Add difference (mask) to input
• Can also scale mask to increase sharpening
– Called highboost filtering
• Highly popular sharpening methods
– Photoshop etc.
– Flexible: arbitrary smoothing kernel, scaling
parameter
Input
3x3 box,
mask scale 1
• Can be combined into one filter mask
15x15 box
mask scale 1.7
– Very similar to sharpening with Laplacian
Edge detection
Sobel operator
• Basic idea
• Approximation for partial derivative along
x and y including some smoothing
– Edges = quick change in intensity
– Find local maxima of first derivative
x derivative:
y derivative:
• In 2D, use gradient magnitude
q
∇f = fx2 + fy2 ,
fx =
∂f
,f
∂x y
=
∂f
∂y
• Gradient magnitude is noisy
• Smooth image before computing gradient
• Many variations of basic approach
Input
http://en.wikipedia.org/wiki/Edge_detection
Output
http://en.wikipedia.org/wiki/Sobel_operator
Filtering using order statistics
• Idea
– Order pixels in a neighborhood according to their
values
– Replace central pixel by certain value found from
the ordered list
Median filter
• Advantage over smoothing with linear
filter:
– More robust to outliers
– Very effective for „Salt and pepper noise“
• Examples for order statistics filters
– Minimum
– Maximum
– Median: „value in the middle of ordered list“,
50th percentile, 50% of values are smaller, 50%
larger
• Filters are nonlinear!
Input
3x3 box
3x3 median
6
Today
Fourier transform
Filtering and the Fourier transform
• Fourier series
• Spatial filtering, convolution
• Fourier transform
• Fourier transform
• Convolution theorem
• Frequency domain filtering
• Sampling & aliasing
• Discrete Fourier transform
• Extension to 2D
• Nice interactive demonstrations
http://www.fourier-series.com/
Fourier transform
Fourier series
Benefits
• All periodic signals can be represented as a
summation of sinusoidal waves (Fourier, 19th century)
• Analyse & understand linear filters
• Sinusoidal wave defined by
• Understand relationship between
continuous and sampled (discrete) case
• Implement (large) linear filters more
efficiently
– Frequency
– Phase
– Amplitude
• Using complex exponentials
– Complex amplitude c encodes
both amplitude and phase
• How to compute the complex
amplitudes for each wave?
http://axion.physics.ubc.ca/341-02/fourier/fourier.html
Fourier series
Fourier series
• Sinusoidal wave defined by
• Sinusoidal wave defined by
• All periodic signals can be represented as a
summation of sinusoidal waves (Fourier, 19th century)
– Frequency
– Phase
p
– Amplitude
• All periodic signals can be represented as a
summation of sinusoidal waves (Fourier, 19th century)
– Frequency
– Phase
p
– Amplitude
• Using complex exponentials
• Using complex exponentials
– Complex amplitude c encodes
both amplitude and phase
– Complex amplitude c encodes
both amplitude and phase
• How to compute the complex
amplitudes for each wave?
http://axion.physics.ubc.ca/341-02/fourier/fourier.html
• How to compute the complex
amplitude c for each frequency?
http://axion.physics.ubc.ca/341-02/fourier/fourier.html
7
Fourier transform
Fourier analysis
• Generalization of Fourier series to nonperiodic functions
• Signals can be represented in the spatial or
the frequency domain
• Computes amplitude and phase of sinusoidal
wave at each frequency
Fourier transform
– Given input signal/function f, frequency 
Frequency
Complex amplitude
Sinusoidal wave
• Fourier transform is complex valued
Inverse Fourier transform
– Returns complex amplitude for each frequency
Fourier analysis
Fourier analysis
Fourier transform
Fourier transform
Inverse Fourier transform
Inverse Fourier transform
Spatial domain
Frequency domain,
power spectrum
• Power spectrum: squared magnitude of Fourier
transform
– Magnitude: length of complex amplitude
– Squared magnitude given by product of Fourier
transform with its complex conjugate
Fourier transform example
Spatial domain
Spatial domain
Frequency domain,
power spectrum
• Band-limited signal: no frequencies
(amplitudes are 0) above a certain threshold
Fourier transform example
Spatial
domain
Frequency domain
Frequency
domain
cosine (even)
cos(-t) =cos(t)
sine (odd)
sin(-t) =-sin(t)
8
Dirac delta function
Impulse train
Fourier transform
Frequency
d
domain
i
Dirac delta
x)
Impulse train
(shah, comb function)
x
IIIT ( x)    ( x  kT ) III ( ) 
0
k
Period T
Convolution theorem
f(x) ? h(x) ⇔ F (ω)H(ω),
  (  k )
0
k
Period 0  2 / T
Convolution theorem
• “Convolution in the spatial domain
corresponds to multiplication in the
frequency domain, and vice versa”
Spatial domain
Convolution
Fourier tran
nsform
– Convolution in the spatial domain leads to the
same result as Fourier transform,,
multiplication, inverse Fourier transform
– Multiplication in the spatial domain leads to
the same result as Fourier transform,
convolution, inverse Fourier transform
1
2 0
Frequency domain
Multiplication
Inverse Fourier transform
Spatial
d
domain
i
Frequency
domain
Spatial
domain
f (x)h(x) ⇔ F (ω) ? H(ω)
Sampling
Sampling
• Spatial domain: multiply signal with
impulse train
• Spatial domain: multiply signal with
impulse train
• Frequency domain: convolve signal with
Fourier transform of impulse train
• Goal: find Fourier transform of sampled
signal
9
Sampling and reconstruction
Sampling
Sampling and reconstruction
Sampling
Reconstruction
Spatial Domain Frequency Domain
Reconstruction
Spatial Domain Frequency Domain Spatial Domain Frequency Domain
Spatial Domain Frequency Domain
Perfect
reconstruction!
Sampling and reconstruction
Sampling
Sampling and reconstruction
Sampling
Reconstruction
Spatial Domain Frequency Domain
Reconstruction
Spatial Domain Frequency Domain
Spatial Domain Frequency Domain
Spatial Domain Frequency Domain
Aliasing!
Sampling Theorem (Shannon 1949)
• A signal can be reconstructed exactly if it
is sampled, at least, at twice its maximum
frequency
• The minimum sampling frequency is called
the Nyquist frequency
Aliasing!
Aliasing
Sufficiently
sampled
Insufficiently
sampled
[R. Cook ]
10
Aliasing
Antialiasing
• Moiré patterns
• Low pass filter the image first before
resampling
– Band limiting
– Remove frequencies above Nyquist limit of
new
ew sampling
sa pl g g
grid
d
• Smoothing filters have a low pass filtering
effect
Sufficiently sampled
Downsampled,
aliasing appears
http://en.wikipedia.org/wiki/Aliasing
Discrete Fourier transform (DFT)
Discrete Fourier transform (DFT)
• So far, Fourier transform of sampled
function was continuous
• DFT and IDFT with M samples
• Can sample the Fourier transform
• Sampled signal and sampled transform
form discrete Fourier transform pair
Fm =
Sampled transform
fn e−jmn/M
n=0
m = 0...M − 1
DFT
– Transform is invertible as in continuous case
Sampled signal
M
−1
X
−1
1 MX
Fm ejmn/M
2πM n=0
n = 0...M − 1
fn =
IDFT
Fm
fn
Sampled signal
Sampled transform
Note
Note
• Same number of samples M in input and in
DFT
• Samples 0...M-1 of DFT contain two backto-back half periods
• But input usually real, and DFT complex!
– Twice as many numbers in DFT as in input
– How can there be an invertible transformation in
this situation?
• When input is real, then DFT is symmetric
• Need to rearrange coefficients to get
intuitive „centered transform“
fftshift in Matlab
Symmetry
Sampled signal
Sampled transform
11
Properties
Properties
• Note
• Discrete transforms are periodic
F0 =
MX
−1
fn e−j0n/M =
n=0
M
−1
X
fn
n=0
Parseval‘ss theorem
• Parseval
M
−1
X
n=0
|fn |2 =
1
M
M
−1
X
n=0
|Fm |2
– If we evaluate more than M samples in
discrete transforms, the samples repeat
• Convolution theorem holds for circular
convolution
– Wrap convolution kernel around at boundaries
– If circular convolution not desireable, use zero
padding before applying DFT
Extension to 2D
Example 2D transform
• Quite straightforward
• Fourier or frequency spectrum is
magnitude of Fourier transform
– Apply transforms first in one, then other
dimension
• See reading text
• Scaled here for visualization
Input
Fourier spectrum
Today
Frequency domain filtering
Filtering and the Fourier transform
• Remember: convolution theorem
• Spatial filtering, convolution
• Specify desired filter in frequency domain
• Fourier transform
• Can directly control frequencies in output
• Frequency domain filtering
• Computation usually more efficient than
convolution if kernel is large
12
Frequency domain filtering
Low-pass filters
• Procedure
• Low pass filter: remove high frequencies
from signal
1. Compute Fourier transform of input
2. Determine filter function in frequency
domain
3. Multiply
4. Inverse transform
• Shape of low pass filter in spatial and
frequency domain?
• Technical challenges/pitfalls
– Sufficient zero padding, windowing
– Properly centering frequency domain filter
– Filter design
Low-pass filters
Ideal low-pass filter
• The larger the support in the spatial domain, the
smaller the support in the frequency domain
• Sharp cut-off at user specified frequency
• E.g., Gaussian: parameter σ controls filter width
2
2
– Frequency response H(ω) =√Ae−u /2σ
2 2 2
h(x) = 2πσAe−2π σ x
– Impulse response
d
σ decreases
Frequency
domain
(frequency
response)
Spatial domain
(impulse response)
H(ω)
3D plot
As an image
Radial profile
h(x)
Ideal low-pass filter
Butterworth low-pass filter
• Ringing
• Parameter n controls sharpness of
frequency cut-off
– Spatial domain: convolution with a sinc kernel
3D plot
H(u, v) =
Input
Filtered
As an image
Radial profile
1
1 + [D(u, v)/D0 ]2n
13
Butterworth low-pass filter
High-pass filter
• Parameters D0=0.1, n=2
Ideal
Butterworth
Gaussian
Input
Filtered
High-pass filter
Next time
• Ideal high-pass, radius 0.5
• Image restoration
Input
Filtered
14