1
Homework 4 Fall 2015 EE 573 Due 10/23(F)
SOLUTION
Problem 1. (25pts) Consider a measurement process, xt  st  nt that is the sum of a signal that is statistically
independent of the noise, and where the signal and noise processes are:
Signal: s t is an AR(2) process st  a1st 1  a2 st 2  ut ; Noise: n t is an AR(1) process: nt  b1nt 1  vt .
n
For an ARMA(n,m) process of the form: xt  

m
ak xt k 
k 1
b w
k
; b0  1 where wt is white with power  w2 ,
t k
k 0
n
then X ( z)  G( z)W ( z) , where G( z)  B( z) / A( z) with A( z ) 
a z
k
k
m
& B( z ) 
k 0
b z
k
k

; a0  1 .
k 0
Hence, for z  e i  the Wiener-Kinchine Theorem gives S x () | G(e j ) |2  w2 .
(a) (3pts) Use the above to compute the theoretical closed form, real-valued expression for the psd of the noise, Sn ()
involving ω, b1 &  v2 .
Solution: S n ( ) 
 v2
 S n ( ) 
| 1  b1 z 1 |2
 v2
(1  b12 )  2b1 cos( )
.
(b) (5pts) Use the above to compute the theoretical closed form, real-valued expression for the psd of the signal, S s () .
involving ω, a1 , a2 &  u2 .
Solution: S s ( ) 
Hence:
S s ( ) 
 u2
| 1  a1 z 1  a 2 z  2 |2
(1  a12

 u2
(1  a1 z 1  a 2 z  2 )(1  a1 z  a 2 z 2 )
 u2
2
 a2 )  2a1 (1  a2 ) cos( )  2a2 cos(2 )

 u2
(1  a12  a 22 )  a1 (1  a 2 )( z 1  z )  a 2 ( z  2  z 2 )
.
(c) (3pts) Suppose that the polynomial in the denominator of the transfer function for the signal has complex conjugate
roots z1,2   e  j . Compute the expressions for a1 & a 2 in terms of  &  .
Solution: To obtain the numerical values of the denominator of this psd, write:
z 2  a1z  a2  ( z  z1 )(z  z1 )  ( z  2 Re( z1 ) z  | z1 |2  z  2 cos( ) z   2  a1  2 cos( ) ; a2   2
(d) (5pts) Suppose you have the following numerical information:
Signal: The signal, s t has power Rs (0)  1.0 , and has complex conjugate poles at 0.9 e  j 2 0.2 .
Noise: The noise, nt has power Rn (0)  2.0 , and has a real pole at 0.7.
(i) Show that the noise driving noise has  v2  1.02
Solution: Multiplying nt  b1nt 1  vt by each of {nt k }1k 0 , and taking the expected value gives:
Rn (0)  b1Rn (1)   v2 ; Rn (1)  b1Rn (0) . Hence,  v2  (1  .72 )2  1.02
(ii) Show that the signal driving noise has  u2  0.31 by solving the appropriate autocorrelation equations directly for the


tr
unknowns  u2 Rs (1) Rs (2) directly (i.e. do not integrate the psd).
Solution: To compute the AR(2) driving noise variance,  u2 , use the relations:
E{st st  j  a1st 1st  j  a2 st 2 st  j }   u2   ( j )
for
j  0,1,2 :
Rs (0)  a1Rs (1)  a2 Rs (2)   u2 ; Rs (1)  a1Rs (0)  a2 Rs (1)  0 ; Rs (2)  a1Rs (1)  a2 Rs (0)  0
The unknowns in these 3 equations are Rs (1), Rs (2) &  u2 . Hence, we can write the equations as:
2
 a1
 a 2    u2 
1
1
0  (1  a )
  R (1)   R (0)  a
0
2
s

 s 
 1
0
a 2
 a1
 1   Rs ( 2)


1
  u2  1
 a1
 a2   1

   R (1)   0  (1  a )
0   a1
2
 s  

 
 R ( 2 )  0

 a1
 1  a 2
 s  

 R ( 0)
 s

From (c) we have a2   2  0.92  0.81 & a1  2(.9) cos(0.2 * 2 )  1.786 . Using these, we obtain  u2  0.3114
(e)(4pts) For the numerical values in (d), overlay plots of the psd’s you computed in (a) and (b) for 1000 equally spaced
frequencies over the interval [0, π). They should have units of dBs.
Signal & Noise Theoretical PSDs
15
10
dB
5
0
-5
-10
-15
0
0.05
0.1
0.15
0.2
0.25
0.3
Frequency (Hz)
0.35
0.4
0.45
0.5
Figure 1(e) Theoretical signal (red) and noise (blue) psd’s.
 
(f) (5pts) Recall that the z-transform of a sequence {c k }k 0 is defined as C ( z ) 
c z
k
k
. Use this and the Matlab
k 0
command ‘fft’ to compute and overlay plot the signal and noise psd’s directly on the plots you obtained in (e). They
should have units of dB, and S s () should have the color RED and Sn () should have the color BLUE. Both of these plots
should have the form of dashed lines. If done correctly, you should find that (f) and (g) give exactly the same thing.
NOTE: Your fft must include 2000 frequencies in order to have 1000 frequencies over [0, π).
Solution: From (a) we have S s ( ) 
Similarly, from (b) we have Sn ( ) 
 w2

| 1  a1 z 1  a 2 z  2 |2
 v2
| 1  b1 z 1 |2



 v2
 w2
| Ps (e j ) |2
| Pn (e j ) |2
where Ps (e j )  fft ([1 a1 a 2 ])
where Pn (e j )  fft ([1 b1 ]) .
These expressions, computed using 1000-point fft’s are exactly those given in (a). See Figure 1(e) where the plots are
exactly the same as those computed above.
3
Problem 2. (25pts) Consider again the signal-plus-noise setting where the measurement process is xt  st  nt , and
where the signal and noise are uncorrelated with each other. The Wiener Filter is that filter signal estimator



S ()  H () X () that satisfies orthogonality condition: E{[ S ( )  S ( )]S ( )}  0 (C1).
(a)(5pts) Use the linearity of E(*) to show that H ()  S s () /[S s ()  Sn ()] .



Solution: 0  E{[ S ( )  S ( )]S ( )}  E{[ S ( )  S ( )] H ( ) X ( )}

 0  E{[ S ( )  S ( )] X ( )} . (2a.1)
Because st and n t , so are S () and N () . Hence: E[S () X ()]  E[S (){S ()  N ()}]  E[| S () |2 ]  S s () . (2a.2)

Similarly, we obtain E[S ()]X ()]  H ()E[| X () |2 ]  H ()[Ss ()  Ss ()] (2a.3). (2a.2-3) into (2a.1) gives the result.
(b) (5 pts) For the psd’s as computed in Problem 1(f),
compute and plot the Wiener Filter. [Do not use dB.]
Also, compute it over the entire frequency range
{k  2 (k  1) / 2000}2000
k 1 , but only plot it up to the
Nyquist frequency.
Solution: (See code lines ##-##)
Figure 2(b) Theoretical Wiener filter.
(c) (15pts) To verify that your filter is correct, simulate a 2000-point time series for x t of Problem 1. Then use your filter

in (b) to recover the signal estimate.(i) Overlay plots of s t and s t . (ii) Plot a zoomed region to better see how well the

filter works. (iii) Compute the numerical value of the average squared error mse (1 / n)
n
(s  s )

t
2
t
Note 1: Generate 2500
t 1
points & discard the first 500 points associated with the start-up transient. Note 2: Do not discard this data set. You will
need it in Problem 3. Solution: (See code lines ##-##)

Figure 2(c) An example of the performance of the filter (right is a zoomed plot). (iii) mse  0.278
4
Problem 3 (25pts) You will now proceed to determine whether the numerical value of the average squared error you
obtained in Problem 2(c) is reasonable.
S ( ) S nn ( )
(a)(5pts) Show that the frequency domain theoretical mean squared error is MSE ( )  ss
.
S ss ( )  S nn ( )






Solution: MSE ( )  E[| S ( )  S ( ) |2 ]  E{[ S ( )  S ( )]S s ( )}  E{[ S ( )  S ( )]S s ( )}  E{[ S ( )  S ( )]S s ( )}

Note that the second term was zero due to the fact that the error is uncorrelated with S s ()  H () X () . And so


MSE()  E{[S ()  S ()]S ()}  E{[S ()  S ()]S ()}  S ss ()  H ()S xs ()  S ss ()  H ()S ss () .
Substituting H ()  S s () / S x ()  S s () /[S s ()  Sn ()] into this expression gives
MSE()  S ss ()  H ()S ss ()  S ss ()  [1  H ()]  MSE ( ) 
S ss ( ) S nn ( )
.
S ss ( )  S nn ( )
(b)(5pts) For your psd’s computed in Problem 1(d), compute and plot MSE(ω).
This plot should not be in dB.
Solution: (See code lines ##-##)
Figure 3(b) The MSE(f).
(c)(5pts) Define the relative mean squared frequency domain prediction error:

RMSE ( )  MSE ( ) / S ss ( ) . (i)Obtain the expression for RMSE() . (ii) Plot it.
(iii) Explain why it offers more insight into the performance of the Wiener filter.
Solution: (i) RMSE ( ) 
S nn ( )
MSE ( )

S ss ( )
S ss ( )  S nn ( )
Remark: The RMSE gives a much clearer picture of how the Wiener filter
performs. In particular, it shows here that in the region of the signal peak, expected
squared error is only ~10%. Near ω=0 it is ~100%. Since most of the signal power is
in the peak region, the estimate can be expected to be reasonably good.
Figure 3(c) The RMSE(f).
(d)(5pts) Since all of the processes involved are wss, it should be clear that the time domain mse does not depend on time;

that is mse (t )  E[e(t ) 2 ]  mse . Since MSE(ω) is simply the mean squared error per unit frequency, it follows that mse is
simply the integral of MSE(ω) over [0, 2π). For your 2000 discrete frequencies on this interval, (i) use a Riemann sum
approximation of the integral to get a numerical value for mse. (ii) Discuss how it compares to your answer in Problem
2(c).
n 1
n 1

2 1
Solution: E[e(t ) 2 ]   e2  (1 / 2 ) MSE(k )   (1 / 2 ) MSE(k ) 

MSE(k ) , which gives  e2  0.292


0
0
n
2
n

k
For the signal estimate in Figure 2(c), the average squared error was  e  0.278
0.8
0.7
0.6
(e)(5pts) Recall that mse is nothing more than the variance of the error at any
time. Hence the root mse (denoted rmse), which is the square root of mse, is
simply the standard deviation of the error, e(t) for any time t, is a random
variable. Furthermore, the collection {e(t ) ; t  (, )} is a wss random
process. Use your partial realization of the process obtained in (c) to
investigate the normality of the pdf of e(t) for any time t.
Solution: The N(0,.527) model works well.
0.5
0.4
0.3
0.2
0.1
0
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
Figure 1.6 Scaled histogram of e(t).
5
Problem 4. (25 pts) In Problem 2, you used the theoretical
Wiener filter to obtain the signal estimate. Often, the signal,
noise and/or measurement psd’s are not known, and must
be estimated from data. In this problem you are to assume
that you have access to noise measurements and to signalplus-noise measurements only.
(a)(5pts) Simulate a 10,000 point time series of the noise
process.
Then estimate and plot (in dB) the noise psd via Matlab’s
(or your own) psd code.
Solution: (See code lines ##-##)
Figure 4(a&b) Noise (black) & signal-plus-noise (blue) psd estimates.
(b)(5pts) Repeat (a) for the signal-plus-noise process. Overlay this psd estimate on that obtained in (a).
Solution: (See code lines ##-##)
(c)(5 pts) Using your results in (a) and (b), plot the
estimated Wiener filter. Note: Clearly



H ()  1  Snn () / S xx () . Since the ratio in this expression is
a ratio of psd estimates, it is possible that it might be < 0
for some frequencies. Since, it is impossible for the true
W.F. to be < 0, you should be careful to truncate any
negative number to 0.
Solution: (See code lines ##-##)
Figure 4(c) Wiener filter estimate based on Figure 1.7.
(d)(5pts) Apply your estimated Wiener filter to the data set of Problem 2(c). Then overlay plots of the true signal, the
estimate obtained by the theoretical W.F. and the one obtained using your estimated W.F.
Solution: (See code lines ##-##)
Figure 4(d) Illustration of the performance of the Matlab psd-based estimated Wiener filter in recovering the signal.
(e)(5pts) Compute the average squared error associated with your estimated W.F. and discuss how it compares to that
associated with the theoretical W.F.
Solution: The mse related to this estimate is 0.314. Compared with the value 0.292 achieved by the theoretical filter. This
is not so bad, given that it utilized no knowledge of the signal psd, nor any knowledge of the AR structure of the noise. It
should be noted that it CAN be smaller than .292, as that is the EXPECTED squared error.
6
APPENDIX Matlab Code
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
% PROGRAM NAME: hw4.m
% PROBLEM 1
%Part (e):
dw=2*pi/2000;
w=0:dw:2*pi-dw;
f=(2*pi)^-1 *w(1:1000);
% (a) Compute SIGNAL PSD
rho=0.9; theta=0.2*2*pi;
a1=-2*rho*cos(theta); a2=rho^2;
d0=1+a1^2+a2^2;
d1=a1+a1*a2;
d2=a2;
A=[1 -a1 -a2; 0 1+a2 0;0 a1 1];
v=[1 -a1 -a2]';
b=A^-1*v;
su2=b(1);
ds=(1+a1^2+a2^2)+2*a1*(1+a2)*cos(w)+a2*2*cos(2*w);
Sss = su2 * (ds.^-1);
SssdB = 10*log10(Sss(1:1000));
% ##### FIGURE 1 #####
figure(1)
plot(f,SssdB, 'r')
xlabel('Frequency (Hz)')
ylabel('dB')
% COMPUTE NOISE PSD
b1 = -0.7; varn = 2;
sv2 = varn*(1-b1^2);
nd = (1+b1^2) + 2*b1*cos(w);
Snn = sv2* (nd.^-1);
SnndB = 10*log10(Snn(1:1000));
hold on
plot(f,SnndB,'b')
title('Signal & Noise Theoretical PSDs')
grid
pause
% Part(f): VERIFY PSDS BY COMPUTING VIA FFT
Dz=fft([1 a1 a2],2000);
SAR2=su2*(abs(Dz)).^-2;
SAR2dB = 10*log10(SAR2(1:1000));
plot(f,SAR2dB,'*r')
D1z=fft([1 b1], 2000);
SAR1=sv2*(abs(D1z)).^-2;
SAR1dB = 10*log10(SAR1(1:1000));
plot(f,SAR1dB,'*b')
title('Comparison of Signal & Noise PSDs using fft')
grid
pause
%=======================================
% PROBLEM 2
% Part(b): COMPUTE THE THEORETICAL WIENER FILTER
H=Sss./(Sss + Snn);
H1 = H(1:1000); %Plot only up to the Nyquist frequency
% ##### FIGURE 2 #####
figure(2)
plot(f,H1)
xlabel('Frequency (Hz)')
ylabel('H(f)')
title('Theoretical Wiener Filter')
grid
pause
% Part(c): Test the filter
u=su2^0.5 * randn(1,2500);
v=sv2^0.5 * randn(1,2500);
n = zeros(1,2500);
s = zeros(1,2500);
x = zeros(1,2500);
for k = 3:2500
s(k) = -a1*s(k-1) - a2*s(k-2) + u(k);
n(k) = -b1*n(k-1) + v(k);
end
s = s(501:2500);
n = n(501:2500);
x = s + n;
ss = s; %save this for later
7
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
xx=x; %Save this for later
figure(3)
tvec = 1:2000;
plot(tvec,x)
X = fft(x);
Shat =H.*X;
shat = real(ifft(Shat));
hold on
plot(shat,'r')
plot(s,'k')
legend('x_t','shat_t','s_t')
title('Wiener Filter Signal Estimate')
grid
%Compute Estimate of mse:
msehat = mean((s - shat).^2)
pause
%========================================
% PROBLEM 3
%Part(b) Theoretical MSE(w)
MSE = (Sss .*Snn)./(Sss + Snn);
figure(4)
plot(f,MSE(1:1000));
title('Theoretical Wiener Filter MSE(f)')
grid
xlabel('Frequency (Hz)')
ylabel('MSE(f)')
pause
%Part(c):
% Theoretical relative MSE(w)
RMSE = Snn./(Sss + Snn);
figure(5)
plot(f,RMSE(1:1000));
title('Theoretical Wiener Filter Relative MSE(f)')
grid
xlabel('Frequency (Hz)')
ylabel('RMSE(f)')
pause
%Part(d): Compute Theoretical total mse
mse = mean(MSE)
pause
%Part (e): Error histogram and related fhat
bvec = -1.9:.2:1.9;
err = s - shat;
h = hist(err,bvec);
fhat = (0.2*2000)^-1 * h;
% ##### FIGURE 6 #####
figure(6)
bar(bvec,fhat)
hold on
fnormal = (2*pi*msehat)^-0.5 *exp(-(2*msehat)^-1 *bvec.^2 );
plot(bvec,fnormal,'r')
pause
% PROBLEM 4
%Part(a) & (b):
%Simulate noise alone, and then signal-plus noise with different noise
n = zeros(1,10500);nx = zeros(1,10500);s = zeros(1,10500);x = zeros(1,10500);
v=sv2^.5*randn(1,10500); vx=sv2^.5*randn(1,10500); u=su2^.5*randn(1,10500);
for k = 3:10500
s(k) = -a1*s(k-1) - a2*s(k-2) + u(k);
n(k) = -b1*n(k-1) + v(k);
end
s = s(501:10500);
n = n(501:10500);
x = s + n;
%Compute PSD estimates
nw=200;
navg=10000/nw;
psdn=zeros(1,2000); psdx = zeros(1,2000);
for k = 1:navg
nk=n((k-1)*nw+1:k*nw); xk=x((k-1)*nw+1:k*nw);
psdn = psdn +(1/nw)*abs(fft(nk,2000)).^2;
psdx = psdx +(1/nw)*abs(fft(xk,2000)).^2;
end
psdn = psdn/navg; psdx=psdx/navg;
figure(7)
8
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
psdndB=10*log10(psdn(1:1000));
psdxdB=10*log10(psdx(1:1000));
plot(f,psdxdB,f,psdndB,'k')
title('FFT-based PSD Estimates for n(t) (black) and x(t) (blue')
xlabel('Frequency (Hz)')
ylabel('dB')
grid
pause
%Part(c): Compute Wiener filter
for k = 1:2000
He(k)=1 - psdn(k)/psdx(k);
if He(k) < 0
He(k)=0;
end
end
figure(8)
plot(f,He(1:1000))
title('FFT-based PSD Wiener Filter Estimate')
xlabel('Frequency (Hz)')
grid
pause
%Part(d):
%Apply this W.F. to data xx generated earlier
Shate = He .* fft(xx);
shate = real(ifft(Shate));
figure(9)
plot(tvec,[ss;shat;shate])
legend('s','shat','shate')
xlabel('Time (sampling interval)')
grid
msee = mean((ss - shate).^2)