Wiener Filtering
ECE 7251: Spring 2004
Lecture 19
2/18/04
Prof. Aaron D. Lanterman
School of Electrical & Computer Engineering
Georgia Institute of Technology
AL: 404-385-2548
<lanterma@ece.gatech.edu>
Copyright 2004 Aaron Lanterman
The Setup
• Context: Bayesian linear MMSE estimation
for random sequences
• Parameter sequence { k , k  Z}
• Data sequence {Yk , k  I  Z}
• Goal: Estimate { k } as a linear function of
the observations:
ˆk ( y)   h(k , j ) y j
jI
• Find h to minimize mean square error
Copyright 2004 Aaron Lanterman
O.P. to the Rescue
•
By the orthogonality principle,
*
k
k
i
*
j
k
i
jI
*
E[(ˆ (Y )   )Y ]  0 for i  I
E[( h(k , j )Y   )Y ]  0
 h(k , j ) E[Y Y ]  E[ Y ]
 h( k , j ) r ( j , i )  r ( k , i )
j i
jI
Y
jI
•
*
k i
Y
If processes are stationary, we can write
 h( k , j ) r ( j  i )  r
jI
Y
Y
slight abuse of notation
Copyright 2004 Aaron Lanterman
(k  i )
WienerHopf
Equation
Spectral Representation
• If I  Z it turns out the filter is LTI:
 h( k  j ) r ( j  i )  r (k  i )
WLOG, consider i=0: h(k  j )r ( j )  r
jI
Y
Y
jI
Y
Y
(k )
• Can solve W-H in the Z-transform domain:
H ( z )SY ( z )  SY ( z )
SY ( z )
H ( z) 
SY ( z )
Copyright 2004 Aaron Lanterman
Mean Square Error (1)
2
ˆ
• MSE = E[| (k   (Y )) | ]


ˆ
ˆ
 E[(k  (Y ))(k  (Y ))]
(by O.P.)


ˆ
 E[(k  (Y ))k ]  E[(k ˆ(Y ))ˆ (Y )]

ˆ
 E[k  ]  E[ k (Y )k ]]

k
Copyright 2004 Aaron Lanterman
Mean Square Error (2)

ˆ
MSE  E[k  ]  E[ k (Y )k ]]

 E[k  ]  E[ h(k  j )Y j  k ]]

k

k
jI
 E[ k  ]   h(k  j ) E[Y j  ]

k
jI
 r (0)   h(k  j ) rY  ( j  k )
jI
• Since everything is stationary, can just take k=0
MSE  r (0)  (h  rY  )(0)
Copyright 2004 Aaron Lanterman

k
Mean Square Error (3)
MSE  r (0)  (h  rY  )(0)


 S

( )  H ( ) SY  ( )d


SY ( )
  S ( ) 
SY  ( )d
SY ( )


| SY ( ) |
  S ( ) 
d
SY ( )

2
Copyright 2004 Aaron Lanterman
Deblurring
• Suppose object is observed through a blurring point
spread function f and additive noise W
Yk  ( f )k  Wk
• Suppose  and W are uncorrelated zero-mean
• Recall from ECE6601:
2

Y

W
Y
Y
S  F S  S and S
•
F S
So the Wiener filter is

SY ( z )
F ( z ) S ( z )
H ( z) 

2
SY ( z )
F ( z ) S ( z )  SW ( z )
Copyright 2004 Aaron Lanterman
Interpretation of the Deblurring Filter
• If noise is negligible, i.e. SW ( )  0

F S

F S
1
H ( )  2



F S  SW FF S F
• Even if there is no noise, in
implementation, straight division by
F() is often ill-posed and not a good
idea (round off errors, etc.)
Copyright 2004 Aaron Lanterman
Deblurring Error

| SY ( ) |
MSE   S ( ) 
d
SY ( )

2
2

F ( ) | S ( ) |
  S ( ) 
d
2
F ( ) S ( )  SW ( )









2
S [ F S  SW ]  F | S |
2
2
F S  SW
2
S SW
F S  SW
2
d
Copyright 2004 Aaron Lanterman
2
d
Competing Approaches
• Competing approaches include
iterative methods such as the
“Richardson-Lucy” algorithm (an
EM-style procedure)
– Computationally intensive
– Can naturally incorporate nonnegativity
– Sometimes better match to real
statistics
Copyright 2004 Aaron Lanterman
Discussion
• Advantage of Wiener approach:
– LTI filtering implementation
• Disadvantages of Wiener approach:
– No natural way to incorporate
nonnegativity constraints (in image
processing, for instance)
– Only truly optimal for Gaussian
statistics
Copyright 2004 Aaron Lanterman
Real-Time Wiener Filtering
• What if we don’t have “future”
measurements?
• Must restrict h to be causal
• Solution:
1  SY ( z ) 
H ( z)    

SY ( z )  SY ( z ) 
where the meaning of the plus and minus
superscripts and subscripts will be defined
on later slides
Copyright 2004 Aaron Lanterman
Spectral Factorization
• If Y has a spectrum satisfying the PaleyWiener criterion:

 log S
Y
( )d  

then the spectrum can be factored as
SY ( )  S ( ) S ( )

Y
where
1

Y
1

Y

Y
FF {S } is causal
FF {S } is anticausal
Copyright 2004 Aaron Lanterman
Factoring Rational Spectra
• If the spectrum is a ratio of polynomials, we can
factor as

Y

Y

Y

Y
1
SY ( z )  S ( z ) S ( z )  S ( z ) S ( z )
Poles and zeros
inside unit circle
Poles and zeros
outside unit circle
• Aside: spectral factorization into causal and
anticausal factors is analogous to Cholesky
decomposition of a covariance matrix into lower
and upper triangular factors
Copyright 2004 Aaron Lanterman
Causal Part Extraction
• We can split f into its causal and anticausal parts:
f (k )  { f (k )}  { f (k )}
causal
anticausal
f (k )  f (k )u(k ), { f (k )}  f (k )u(k 1)
• Use similar notation for Z-transform domain
F ( z )  {F ( z )}  {F ( z )}
1
{F }  ZZ {Z
Z {F }u (k )}
1
{F }  Z
Z {ZZ {F}u (k  1)}
Copyright 2004 Aaron Lanterman
How to Extract Causal Parts
• If F is a ratio of polynomials can usually do
a partial fraction expansion:
F ( z )  {F ( z )}  {F ( z )}
Poles and zeros
inside unit circle
Poles and zeros
outside unit circle
• Can also do polynomial long division (see
Ed Kamen’s book)
• Almost always a total pain and really
annoying
Copyright 2004 Aaron Lanterman