Computer Vision Restoration
Hanyang University
Jong-Il Park
Restoration vs. Enhancement
To improve an image in some predefined sense
Restoration
Enhancement
Objective process
Subjective process
A priori knowledge on
degradation model
Modeling the degradation and
applying the inverse process
to recover the original
Department of Computer Science and Engineering, Hanyang University
Restoration process
Department of Computer Science and Engineering, Hanyang University
Noise models
Assume noise is independent
of spatial coordinates and it is
uncorrelated w.r.t. the image.
• Gaussian: electronic circuit noise,
sensor noise
• Rayleigh: range images
• Exponential and gamma:
laser images
• impulse(salt-and-pepper):
faulty switching
Department of Computer Science and Engineering, Hanyang University
Eg. Sample noisy images
Department of Computer Science and Engineering, Hanyang University
Eg. Sample noisy images(cont.)
Department of Computer Science and Engineering, Hanyang University
Periodic noise
 Spatially dependent noise
 Periodic noise can be
reduced significantly via
frequency domain filtering
Department of Computer Science and Engineering, Hanyang University
Estimation of noise parameters
 PDF from small patches
Department of Computer Science and Engineering, Hanyang University
When the only degradation is noise
g ( x, y )  f ( x, y )   ( x, y )
and
G(u, v)  F (u, v)  N (u, v)
 Periodic noise  subtraction gives a good result
 Random noise  mean filter, order-statistics filter,…
Department of Computer Science and Engineering, Hanyang University
Mean filters
 Arithmetic mean filters
 For Gaussian or uniform noise
 Geometric mean filters
 For Gaussian or uniform noise
 Harmonic mean filters
 Work well for salt noise but fail for pepper noise
 Contraharmonic mean filters
 Suited for impulse noise but require identification(salt
or pepper)
Department of Computer Science and Engineering, Hanyang University
Arithmetic & Geometric mean filter
Department of Computer Science and Engineering, Hanyang University
Contraharmonic filters
fˆ ( x, y ) 
Q 1
g
(
s
,
t
)

( s ,t )S xy
 g ( s, t )
Q
( s ,t )S xy
 Q<0 : eliminates salt noise
Q=-1 harmonic mean filter
 Q=0 : arithmetic mean filter
 Q>0: eliminates pepper noise
Department of Computer Science and Engineering, Hanyang University
Eg. Contraharmonic filters
Department of Computer Science and Engineering, Hanyang University
Wrong sign in contraharmonic filters
Disaster!
Department of Computer Science and Engineering, Hanyang University
Order-Statistics filters
 Median filter
 Max filter
 Min filter
 Midpoint filter
 Alpha-trimmed mean filter
Department of Computer Science and Engineering, Hanyang University
Median filters
3x3
median
3x3
median
3x3
median
blurred
Department of Computer Science and Engineering, Hanyang University
Max and Min filter
• Max filter
Removes pepper noise
Removes dark pixels
• Min filter
Removes salt noise
Removes light pixels
Makes dark objects larger
Department of Computer Science and Engineering, Hanyang University
Eg. Comparison
(a) Additive uniform
noise
5x5 arithmetic
mean
5x5 median
(b) (a)+additive S&P
5x5 geometric mean
5x5 alpha-trimmed
Mean(d=5)
Department of Computer Science and Engineering, Hanyang University
Adaptive filters
 Behavior changes locally based on statistical
characteristics of local support
 Simple adaptive filter based on mean and variance
1. If global_var is zero, then f(x,y)=g(x,y)
2. If local_var>global_var, then f(x,y)=g(x,y) (high local
var  edge  should be preserved)
3. If local_var==global_var, then arithmetic mean filtering
Department of Computer Science and Engineering, Hanyang University
Eg. Adaptive filter
Department of Computer Science and Engineering, Hanyang University
Adaptive median filter
 Cope with impulse noise with large probability
 Preserve detail while smoothing non-impulse noise
Algorithm
Level A: A1=zmed-zmin
A2=zmed-zmax
If A1>0 AND A2<0, go to level B
Else increase the window size
If window size<=Smax repeat level A
Else output zxy
Level B: B1=zxy-zmin
B2=zxy-zmax
If B1>0 AND B2<0, output zxy
Else output zmed
Department of Computer Science and Engineering, Hanyang University
Eg. Adaptive median filter
median
adaptive median
Department of Computer Science and Engineering, Hanyang University
Periodic noise reduction
 By frequency domain filtering
 Band reject filter
Department of Computer Science and Engineering, Hanyang University
Eg. Periodic noise reduction
Department of Computer Science and Engineering, Hanyang University
Noise extraction
 By bandpass filter
Help understanding noise pattern
Department of Computer Science and Engineering, Hanyang University
Notch filters
Department of Computer Science and Engineering, Hanyang University
Eg. Notch filtering
 Removing sensor scan-
line patterns
Department of Computer Science and Engineering, Hanyang University
Optimum notch filtering
 First isolating the principal contributions of the
interference pattern
 Then subtracting weighted portion of the pattern from
the corrupted image
Department of Computer Science and Engineering, Hanyang University
Eg. Periodic interference(1/3)
 Noisy image
Department of Computer Science and Engineering, Hanyang University
Eg. Periodic interference(2/3)
 Extraction of noise interference pattern
Department of Computer Science and Engineering, Hanyang University
Eg. Periodic interference(3/3)
 Restored image by subtracting weighted portion of
periodic interference (Refer to the derivation of
weights in pp.250-252)
Department of Computer Science and Engineering, Hanyang University
Linear, Position-Invariant Degradation
Spatial Domain:
g ( x, y)  h( x, y)  f ( x, y )   ( x, y)
FrequencyDomain:
G(u, v)  H (u, v) F (u, v)  N (u, v)
Department of Computer Science and Engineering, Hanyang University
Degradation knowledge
 Degradation knowledge about H
1. A priori (known)
2. A posteriori (unknown) 
blind restoration
or blind deconvolution
Restoration:
determine the original image f ,
given the observed image g and
knowledge about the degradation (H).
Department of Computer Science and Engineering, Hanyang University
Fundamental issue
 Restoration problem
T{ f }  g
 restoration is to find T 1 , such that T 1{g}  f
but, 1. T 1 does not exist: singular
2. T 1 may exist, but not be unique: ill-conditioned
1
3. T may exist and unique, but there exists  ,
which can be made arbitrarily small, such that
T 1{g  }  f   ,    , which is not negligible
 Image restoration is ill-conditioned at best and
singular at worst
Department of Computer Science and Engineering, Hanyang University
Estimation of degradation function
 Approaches

Observation

Experimentation

Mathematical modeling
Department of Computer Science and Engineering, Hanyang University
Estimation by observation
 Looking at a small section of the image containing
simple structures and then obtaining degradation
function
Observed sub-image: g s ( x, y)
Estimate of original image: fˆs ( x, y)
Gs (u, v)
H s (u, v) 
Fˆs (u, v)
Department of Computer Science and Engineering, Hanyang University
Estimation by experimentation
 Possible only if equipment similar to the equipment
used to acquire the degraded images is available
 Eg. Use an impulse
Department of Computer Science and Engineering, Hanyang University
Estimation by modeling
 Based on either physical characteristics or basic
principles
Eg.1. Physical characteristics: atmospheric turbulence
H (u , v)  e
k (u 2 v 2 )
5/ 6
Eg.2. Math derivation: motion blur

Starting from
T
g ( x, y)   f [ x  x0 (t ), y  y0 (t )]dt
0

After some manipulation(p.259)
T
H (u, v)   e j 2 [ux0 (t )vy0 (t )]dt
0

Setting the motion model, we obtain the degradation
func.
Department of Computer Science and Engineering, Hanyang University
Eg.1. Physical model
 Atmospheric turbulence
Department of Computer Science and Engineering, Hanyang University
Eg.2. Math modeling
 Motion blur
Department of Computer Science and Engineering, Hanyang University
Restoration methods
 Inverse filtering
 Wiener filtering
 Constrained least square filtering
 Geometric mean filtering
 Etc..
Department of Computer Science and Engineering, Hanyang University
Inverse filtering
G (u, v)
ˆ
F (u, v) 
H (u, v)
 Poor performance!
 Very sensitive to noise
G(u, v)  H (u, v) F (u, v)  N (u, v)
N (u, v)
ˆ
F (u, v)  F (u, v) 
H (u, v)
Noise amplification
when H(u,v) is small
Department of Computer Science and Engineering, Hanyang University
Eg. Inverse filtering
Department of Computer Science and Engineering, Hanyang University
Minimum mean-square error filter
 Necessary to handle noise explicitly
 Statistical characteristics of noise should be
incorporated into the restoration process
 MMSE filter
ˆ
 To find an estimate f of the uncorrupted image f
such that the mean square error between them is
minimized:
2
2
e  E{( f  fˆ ) }


Assume:
 the noise and the image are uncorrelated
 The one or the other has zero mean
 The gray levels in the estimate are a linear function of
the levels in the degraded image
Derivation: Homework
Department of Computer Science and Engineering, Hanyang University
MMSE filter (cont.)
 Frequency domain expression:
Wiener filter


H
*
(
u
,
v
)
S
(
u
,
v
)
f
Fˆ (u, v)  
G (u , v)
2
 S f (u, v) | H (u, v) |  S (u, v) 


H * (u, v)

G (u, v)
2
 | H (u, v) |  S (u , v) / S f (u, v) 
PS of noise
PS of image f
 Approximation of the Wiener filter
2

1
| H (u, v) | 
ˆ
F (u, v)  
G(u, v)
2
 H (u, v) | H (u, v) |  K 
Department of Computer Science and Engineering, Hanyang University
Eg. Wiener filtering
 Using the approximation
 K is chosen interactively
Department of Computer Science and Engineering, Hanyang University
Eg. Restoration by Wiener filter
motion blur
Severe noise
Moderate noise
Negligible noise
Department of Computer Science and Engineering, Hanyang University
Constrained Least Square Filtering
 Difficulty in Wiener filter
 The power spectra of the undegraded image and noise
must be known
 Minimization in a statistical sense
 The constrained LS filtering
 requires knowledge of
 Mean of the noise
 Variance of the noise
 Optimal result for each image
Department of Computer Science and Engineering, Hanyang University
Vector-matrix form of convolution
g  Hf  η
 g: MN-vector (lexicographical order of an image)
 f: MN-vector
 H: MNxMN matrix
Department of Computer Science and Engineering, Hanyang University
Formulation: Constrained LS filter
To find the minimum of a criterion function C defined as
M 1 N 1

C    f ( x, y)
2

2
x 0 y 0
subject to the constraint
|| g  Hfˆ ||2 || η ||2
where || w ||2  wT w is the Euclidean vector norm
Department of Computer Science and Engineering, Hanyang University
Freq. Domain Sol.


H * (u, v)
ˆ
F (u, v)  
G(u, v)
2
2
| H (u, v) |  | P(u, v) | 
 : adjustable parameter
P(u, v) : Fourier transform of the Laplacian operator
Department of Computer Science and Engineering, Hanyang University
Eg. Constrained LS filter
 Significant improvement over Wiener filter
Department of Computer Science and Engineering, Hanyang University
Procedure for computing

 Define a residual vector
r  g  Hfˆ
 Adjust

so that
 ( ) || r || || η || a
2

2
iteration
Calculation
|| η ||  MN[  m ]
2
2
2
In general, automatically determined restoration filter
yields inferior results to manual adjustment of filter
parameters
Department of Computer Science and Engineering, Hanyang University