Colorado School of Mines
Image and Multidimensional Signal
Processing
Professor William Hoff
Dept of Electrical Engineering &Computer Science
Colorado School of Mines
Image and Multidimensional Signal Processing
http://inside.mines.edu/~whoff/
Deconvolution
Colorado School of Mines
Image and Multidimensional Signal Processing
Deconvolution
• Goal is to restore or reconstruct the original image that has been
degraded by convolving with a known degradation function h(x,y)
• The degradation process
g ( x, y)  h( x, y )  f ( x, y)   ( x, y)
G(u, v)  H (u, v) F (u, v)  N (u, v)
F f ( x, y )  F (u, v )
Fh( x, y )  H (u, v )
Fg ( x, y )  G (u, v )
F ( x, y )  N (u, v )
• We will try to “deconvolve”, or undo the effects of the convolution
Colorado School of Mines
Image and Multidimensional Signal Processing
3
Minimum Mean Squared Error (Wiener) Filtering
• We want to find an estimate of the image to minimize the mean
squared error between the original image and the restored image




2
1 m 1 n 1


ˆ
ˆ ( x, y ) 2
e E f  f

f
(
x
,
y
)

f

 mn 
x 0 y 0
2
E[ ]: “expected
value”
• The minimum error solution is
*


H
( u, v )
ˆ
F ( u, v )  
 G ( u, v )
2
 H (u, v )  S (u, v ) / S f (u, v ) 
• where
2
 1

H ( u, v )

 G ( u, v )
2
H
(
u
,
v
)
H (u, v )  S (u, v ) / S f (u, v ) 

avoids
amplifying
noise where
H is small
S (u, v )  N (u, v )  power spectrum of the noise
2
S f (u, v )  F (u, v )  power spectrum of the undegraded image
2
Colorado School of Mines
Image and Multidimensional Signal Processing
4
Minimum Mean Squared Error (Wiener) Filtering
2
 1

H
(
u
,
v
)
ˆ
F ( u, v )  
 G ( u, v )
2
 H (u, v ) H (u, v )  S (u, v ) / S f (u, v ) 
• Where noise is small compared to the signal (i.e., frequencies
where S(u,v) << Sf(u,v) ), the Wiener filter reduces to the
inverse filter
G ( u, v )
Fˆ (u, v ) 
H ( u, v )
• At frequencies (u,v) where noise power is large with respect to
signal power (i.e., S(u,v) >> Sf(u,v) ), result is close to zero
Fˆ (u, v )  0
Colorado School of Mines
Image and Multidimensional Signal Processing
5
Sketch derivation of Wiener filter
• We want to produce an estimate of the original signal that minimizes
the mean square error


2

ˆ
e E f  f 


2
• This is the same as minimizing the error in the frequency domain


2
2
1 


ˆ
ˆ
e E f  f 
E FF 

 mn 

2
(Parseval's Theorem)
• We need to find a filter W(u,v) that that minimizes the error, so that
we can apply to the corrupted signal G(u,v)
Fˆ (u, v )  W (u, v )G (u, v )
This was adapted from http://www.ee.columbia.edu/~xlx/ee4830/notes/lec7.pdf
Colorado School of Mines
Image and Multidimensional Signal Processing
6
Sketch derivation of Wiener filter (continued)
Fˆ  WG  W HF  N   WHF  WN
The reconstructed signal
F  Fˆ  (1  WH ) F  WN The error between the original and reconstructed
1 m n
2
2
Mean squared error
e 
(1  WH ) F  WN

mn u 1 v 1
e2 
1
MN
 (1  WH ) F  WN
2
u
2
Multiplying out, and using the assumption
that F and N are uncorrelated
v
The error is minimized if each term at each (u,v) is a minimum
e2
2
2
 2( (1  W * H * ) H F  W * N  0
W
W* 
HF
2
H F N
2
Colorado School of Mines
2
2
W

( zz * )  2 z *
z
H*
H  N /F
2
2
2
Image and Multidimensional Signal Processing
This is the Wiener filter in
the frequency domain
7
Wiener Filtering
• Problem
– We may not know S(u,v) and Sf(u,v) in advance
– Usually we assume white noise, so Sf(u,v) = |N(u,v)|2 is a
constant
– However, we don’t usually know the spectrum of the undegraded
image |F(u,v)|2
• Instead, we can approximate using a constant K (which is
the same for all u,v) and try different values
2
 1
H ( u, v ) 
ˆ
F ( u, v )  
 G ( u, v )
2
 H (u, v ) H (u, v )  K 
K is the average noise-to-signal ratio
Colorado School of Mines
Image and Multidimensional Signal Processing
8
Simple 1D Example
• Assume this is the Fourier transform of the original undegraded signal, F(u)
F
1.4
1.2
1
0.8
0.6
0.4
0.2
0
-50
Colorado School of Mines
-40
-30
-20
-10
0
10
20
Image and Multidimensional Signal Processing
30
40
50
9
Simple 1D Example
• Assume this is the Fourier transform of the degradation
function H(u)
H
0.04
0.035
0.03
0.025
0.02
0.015
0.01
0.005
0
-50
Colorado School of Mines
-40
-30
-20
-10
0
10
20
Image and Multidimensional Signal Processing
30
40
50
10
Simple 1D Example
• We degrade the input signal and add noise
0.045
F(u) H(u)
0.04
0.035
0.03
0.025
N(u)
0.02
0.015
0.01
0.005
0
-50
Colorado School of Mines
-40
-30
-20
-10
0
10
20
Image and Multidimensional Signal Processing
30
40
50
11
Simple 1D Example
• The degraded signal G(u) = H(u)F(u) + N(u)
G
0.055
0.05
0.045
0.04
0.035
0.03
0.025
0.02
0.015
0.01
0.005
-50
Colorado School of Mines
-40
-30
-20
-10
0
10
20
Image and Multidimensional Signal Processing
30
40
50
12
Simple 1D Example
• The inverse filter, 1/H(u)
1/H
100
90
80
70
60
50
40
30
20
-50
Colorado School of Mines
-40
-30
-20
-10
0
10
20
Image and Multidimensional Signal Processing
30
40
50
13
Simple 1D Example
2
1

H
W 

2
H
 H  K 
• The Weiner filter, W(u)
W
60
50
40
30
20
10
0
-50
Colorado School of Mines
-40
-30
-20
-10
0
10
20
Image and Multidimensional Signal Processing
30
40
50
14
Simple 1D Example
• The transform of the reconstructed signal, F(u) = W(u)G(u)
F reconstructed
1.4
1.2
1
0.8
0.6
0.4
0.2
0
-50
Colorado School of Mines
-40
-30
-20
-10
0
10
20
Image and Multidimensional Signal Processing
30
40
50
15
K is chosen
interactively
Colorado School of Mines
Image and Multidimensional Signal Processing
16
K is chosen
interactively
Colorado School of Mines
Image and Multidimensional Signal Processing
17
Estimating K
• We can estimate K by making some assumptions
• K is the average noise power divided by the average signal power
• We can assume noise is uniformly distributed, or Gaussian, and
estimate its power
• Assume that the average signal power of the degraded image is
about the same as that of the original image
K is the estimated
average noise-to-signal
ratio
 n ( x, y ) 

K
2
  g ( x, y ) 
2
• Rough estimate: if noise is ±1 graylevels, and image is 0..255, then K
is roughly 10^-4. For images with some additional quantization noise
(such as with JPEG compression), this might be closer to 10^-3.
Colorado School of Mines
Image and Multidimensional Signal Processing
18
Matlab functions for deconvolution
• deconvwnr(g, h)
– Does straightforward inverse filter
– Assumes degradation function h(x,y) is known
• deconvwnr(g, h, K)
– Does Wiener inverse filter with constant K (noise to signal ratio)
• deconvwnr(g, h, ncorr, fcorr)
– Uses known autocorrelation functions for noise (ncorr) and original
image (fcorr)
– where
• ncorr = F-1( S(u,v) )
• fcorr = F-1( Sf(u,v) )
Colorado School of Mines
Image and Multidimensional Signal Processing
19
f = im2double(imread('cameraman.tif'));
noise = 0.01*randn(size(f));
h = fspecial('gaussian',7,1);
Blurred = imfilter(f,h,'circular');
g = Blurred + noise;
imshow(g,[]), title('g = Blurred and Noisy');
% noise-to-power ratio
K = sum(noise(:).^2)/sum(f(:).^2);
figure, imshow(deconvwnr(g,h,K),[]);
title('deconvwnr(g,h,K)');
% noise autocorrelation function, centered
NP = abs(fftn(noise)).^2;
NCORR = fftshift(real(ifftn(NP)));
% image autocorrelation function, centered
IP = abs(fftn(f)).^2;
ICORR = fftshift(real(ifftn(IP)));
figure, imshow(deconvwnr(g,h,NCORR,ICORR),[]);
title('deconvwnr(g,h,NCORR,ICORR)');
Colorado School of Mines
Image and Multidimensional Signal Processing
20
Example: Motion Blur
• Motion blur
g ( x, y )   f x  x0 (t ), y  y0 (t )dt
T
0
• For linear motion over time T
– x0(t) = a t/T
– y0(t) = b t/T
• The point spread function h(x,y) is a linear streak
h(x,y)
Colorado School of Mines
Matlab function
h = fspecial(‘motion’, length, angle);
Image and Multidimensional Signal Processing
21
g = Blurred and Noisy
clear all
close all
% Note: "im2double" makes image range 0..1
f = im2double(imread('cameraman.tif'));
% Add Gaussian noise, with zero mean and
% std dev = 1 graylevel
noise = (1/255)*randn(size(f));
h = fspecial('motion',15,45);
Blurred = imfilter(f,h,'circular');
g = Blurred + noise;
imshow(g,[]), title('g = Blurred and Noisy');
deconvwnr(g,h,K)
% True noise-to-signal ratio
K = sum(noise(:).^2)/sum(f(:).^2);
fprintf('K = %f\n', K);
% Estimated noise-to-signal ratio
K = sum(noise(:).^2)/sum(g(:).^2);
fprintf('K = %f\n', K);
figure, imshow(deconvwnr(g,h,K),[]);
title('deconvwnr(g,h,K)');
Colorado School of Mines
Image and Multidimensional Signal Processing
22
Trying different blurring models
• If you didn’t know parameters of the blur (but you knew it
was a motion blur) you could try looping over different
parameters
(loop over values of length)
(loop over values of theta)
h = fspecial('motion', length, theta);
fRestored = deconvwnr(g,h,K);
(display restored image)
Colorado School of Mines
Image and Multidimensional Signal Processing
23
Application: Super-resolution
• Goal: obtain a high resolution image from observed multiple
low-resolution images
• Example applications:
– Get a higher quality digital image from a sequence of images from a LR
camcorder for printing
– Synthetic zooming of region of interest in surveillance, forensic,
scientific, medical, and satellite imaging.
– Improve resolution of medical images such as computed tomography
(CT) and magnetic resonance imaging (MRI)
– Conversion from NTSC video signal to HDTV
Colorado School of Mines
Image and Multidimensional Signal Processing
24
Sampling Limitations
• Recall the Nyquist theorem:
– We can only reconstruct a signal accurately from discrete samples if
the sampling frequency is > 2 times the maximum signal frequency
• With a single image, we have a limited number of samples
• Our approach is to acquire more samples by taking shifted
versions of the original
Colorado School of Mines
Image and Multidimensional Signal Processing
25
From: Park, et al, “Super-Resolution Image Reconstruction: A Technical Overview,”
IEEE ofSIGNAL
PROCESSING MAGAZINE,
May 2003.
Colorado School
Mines
Image and Multidimensional
Signal Processing
26
From: Park, et al, “Super-Resolution Image Reconstruction: A Technical Overview,”
IEEE ofSIGNAL
PROCESSING MAGAZINE,
May 2003.
Colorado School
Mines
Image and Multidimensional
Signal Processing
27
From: Park, et al, “SuperResolution Image
Reconstruction: A
Technical Overview,” IEEE
SIGNAL PROCESSING
MAGAZINE, May 2003.
Colorado School of Mines
Image and Multidimensional Signal Processing
28
Example
• 20 low res images of a
newspaper (256x256)
3
2
1
Colorado School of Mines
Image and Multidimensional Signal Processing
29
Example
Reconstructed hires (1024x1024)
Images created using
super-resolution
toolbox
Licensed from
Professor Peyman
Milanfar from UC
Santa Cruz
Colorado School of Mines
Image and Multidimensional Signal Processing
30
Summary / Questions
• The Wiener filter is a good way to restore images
that have been degraded by a known degradation
function.
– As usual we assume that the degradation process is linear
and position invariant, followed by additive noise.
– The Wiener filter produces a result that has the minimum
mean squared error between the restored image and the
original image.
• Motion blur is an example of a degradation process.
Colorado School of Mines
Image and Multidimensional Signal Processing
31