Homework 3
Prob 2.4.2
p x  X    e X ,   0, X  0 p
p  n* , l*  
n
l* *
n* 
e  l* n* 1
1. find expectation and variance

E ( )   
0
n
l* *
n* 
e  l* n* 1d ,
n
 l *
  * e  l* n* 11d ,
0 n* 

1   l*
l* n* 11 d .
e

n*  0
E ( ) 
1  U
e U n* 11 dU ,

l*n*  0
Let U  l* . We have
1
n*  1 
l*n* 
n
 *.
l*

n

l* *
E ( 2 )    2
n* 
0

n*n* 
,
l*n* 
(ANS)
e  l* n* 1d
1   l *
l* n*  2 1 d .
e

l*n*  0
Again, let U  l* . We have
E ( 2 ) 
Hence, the variance is given by
1


l*2 n*  0
e U U n*  2 1 dU ,

n*  2

(n*  1)( n* )
l*2 n* 
l*2

(n*  1)( n* )n* 
l*2 n* 
n 2  n*
.
 *
l*2
,
 
var    E 2  E 2   ,
n*2  n*


l*2
2. find p x  X 
First, we need to find p x  X 
n*2
n
 *.
l*2 l*2
(ANS)

p x  X    p x   X   p  n* , l* d  ,
0

  e
n
l* *
 X
n* 
0
e  l* n* 1d  ,
n

l* *


0 n*  l*
Let U   l*  X  . Hence, we have
 X
n*
e   l*  X   l*  X n* 11 d ,
n

px X   

l* *

e U U n* 11 dU
0 n*  l*  X
n
n
l* * n*  1
n*l* *
.


n*  l*  X n* 1 l*  X n* 1

n* 1



Using the Bayes’ rule, we obtain
p x  X  
e
p x   X   p  n* , l* 
px X 
 X

n
l* *
n* 
,
e  l* n* 1
,
n
n*l* *
l*  X n* 1

1
e   l*  X   l*  X n* 11 .
(ANS)
n*  1
The PDF of the posterior probability is of the same form as the prior probability.
Hence, the mean square estimate is given by
ˆms  E X  




2
E  ˆms     E ˆ2ms  2ˆms  2 ,


n*  1
l*  X
(ANS)
 

 E X E ˆ2ms  2ˆms  2 X ,


since Ea( x, y)  E x E y a( x, y) X  . It follows that








 
2
E  ˆms     E X ˆ2ms  2ˆms E  X  E 2 X ,


because ̂ms is a function of X only and does not depend on  when X is given. The
above equation can be reduced more and we have
2
E  ˆms     E X ˆ2ms  2ˆms ˆms  E X E 2 X ,


2
E  ˆms     E 2  E X ˆ2ms .


Hence, we obtain
2
 n  1  2 
2  n*  n*



ˆ
  ,
E  ms    
 E X  *


l*2
 l*  X  
2

n*2  n*
 n*  1 


 EX 
2
2

l*
 l*  X  


 
  
 



n*2  n*

n*2  n*
l*2
 n
*


0 l*
 12
 X 2 l*  X n* 1
 n n
* *

n
n*l* *
 12 l* *
n
n 3
0 l*  X  *
n*2  n* n* n*  12
,


n*  2l*2
l*2
n n  1
= * *
,
n*  2l*2
l*2
dX ,
dX ,
(ANS)
3. N observation case
N
p x X     p x   X i  p  n* , l* d ,
i
0 i 1
N

    e
N
  X i
i 1
0

 
0
n*
N l*
 
n* 
n
l* *
n* 
e  l* n* 1d ,
e   l*  l n* 1d  ,
where l is defined in the text page 142. Using the same procedure as the single
observation case, we obtain,
n

px X  
0 l*

 l
l* *
n*  N
n* 
e   l*  l   l*  l n*  N 1 d  l*  l 
,
n
l* * n*  N 
l*  l n*  N n*  .
Using the Bayes’ rule, we obtain
p x  X  
p x  X   p  n* , l* 
p x X 
N
 
N
  X i
e

i 1
n
l* *
n* 
,
e  l* n* 1
,
l* * n*  N 
n
l*  l n*  N n* 
l  l n*  N e   l*  l n*  N 1 .
 *
n*  N 
It follows that
n N
and
̂ms  *
l*  l

(ANS)
 

 
2
E  ˆms     E 2  E X ˆ2ms


 n  N  2 
n*2  n*


 

 E X  *
2
l*
 l*  l  

n*2  n*


l*2

n*2  n*
l*2
n* n*  1n*  N 2
n*  N  1n*  N l*2
,
n n  1n*  N 
 * *
n*  N  1l*2
n* n*  1
n*  N  1l*2
.
Find MAP estimate


ln p x  X  n*  N ln l*  l   ln n*  N    l*  l   n*  N  1ln(  ) ,


 ln p x  X 
n  N  1
 l*  l   *
0


(ANS)
Hence,
ˆMAP 
n*  N  1  ˆ
MS
l*  l 
Prob. 2.4.3
r an
n ~ N 0,  n 
 R  A2 
pr a R A 
exp 

 n 2
2 n2 

1.
  A  m 2 
k0
0
p a  A 
exp 

2
 n 2
 2 n / k 02 
1
Since both a and n have Gaussian distributions and are independent, r is also
Gaussian distributed with
Er   En  Ea  m0
and
var r   var n  var a  
 n2
k 02
k02  1.
Hence, we can write
p r R  
n
 k 2 R  m 2 
0
exp  0
.
2 2
2


2

k

1
k 0  1 2
n 0


k0



The posterior density is given by
p r a R a  p a ( A)
pa r A R  
,
p r R 
  A  m 2 
 R  A2  k 0
0
exp 
exp



2
2
 n 2
 2 n / k 02 
2 n   n 2


,
 k 2 R  m 2 
k0
0
exp  0

2 2
2


2

k

1
 n k 0  1 2
n 0

1





k02  1 exp  R  A2   A  m0 2  k02 R  m0 2  .
 n 2

2 n2
2 n2 / k 02
2 n2 k 02  1 
Consider on the exponent term.
(1)

R  A2   A  m0 2
k 2 R  m0 2
R2
2 RA
A 2 k 02 A 2
 0




2 n2 / k 02
2 n2 k 02  1
2 n2 2 n2 2 n2
2 n2
2 n2


k 02 Am0 k 02 m0 2 k 02 R  m0 2


2 n2
2 n2
2 n2 k 02  1


A2
2 n2

1  k  22A R  m k  2R
2
0
2
n
k 2 m 2 k 2 R  m0 2
- 0 0  0
,
2
2
2 2
2

2

k

1
n
n
n 0
2
2
0 0


2
 
2
2
 1  k 02 

R  m0 k 02 
R  m0 k 02 




A

2 
 2 2 
1

k
2 n2 1  k 02 
n 
0 

2
,

2
2
2
2
2
2
2
 1  k 02 
 A  R  m0 k 0   R  m0 k 0  R - k 0 m0  k 0 R  m0  ,
 
 2 2 
1  k 02 
2 n2 1  k 02
2 n2 2 n2
2 n2 k 02  1
n 




 R 2  k 02 R 2  m02 k 04  k 02 m0 2  k 02 R 2  2 Rm 0 k 02  m02 k 02
,
2 n2 k 02  1
2

2
 


 1  k 02 
R  m0 k 02 
R  m0 k 02




A

 2 2 
1  k 02 
2 n2 1  k 02
n 


 R 2  m02 k 04  2 Rm 0 k 02
,
2 n2 k 02  1

2


2 
2
 
 
2
2
2
 1  k 02 
 A  R  m0 k 0   R  m0 k 0  R  m0 k 0 ,
 
 2 2 
1  k 02 
2 n2 1  k 02
2 n2 1  k 02
n 

2
2
 1  k 02 
 A  R  m0 k 0  .
 
 2 2 
1  k 02 
n 

Hence, (1) becomes
pa r A R  
k02  1 exp   1  k02  A  R  m0 k02  2 
 n 2
  2 2 
n 
 
2. Extend to N independent observations
We know that
1  k 02
 
 
(QED)

pr R    pr a R a  p a ( A)dA

 N

2
.
  Ri  A
2



k0
A

m
i

1
0
dA
 
exp 


N 1 / 2
2
2
2 
N 1

2 n
2 n / k 0
2 
  n




Let us consider on the exponent term. Hence, we have
(2)
N
 Ri  A2

i 1

2 n2
 A  m0 2
2 n2 / k 02

N

1 N 2
R

2
A
Ri  NA2  A 2 k 02  2 Am0 k 02  m02 k 02 ,



i
2 n2 i 1
i 1



 NR
 N
1  2
A N  k 02  2 A N  i  m0 k 02    Ri2  m02 k 02 ,

2 n2 
 i 1 N
 i 1



2
2

 Nl  m0 k 02   
 Nl  m0 k 02 

N  k 02  
1 N 2
2
2



 Ri  m0 k 0  
,

A


2
2
 N  k 2     2 2  i


2 n2 
2

N

k

1

0  
n
n
0



2

 Nl  m0 k 02   

N  k 02  
   f (R ) ,
(3)

A
2 
2  



2 n 
N

k
0  



where

2

Nl  m0 k 02
1 N 2
2 2 

f (R ) 
.
  Ri  m0 k 0  
2
2
2 n2  i 1
 2 n N  k 0

Hence, (2) becomes
pr R  
 n 2
 N 1 2  N 1 / 2
n
k 02
N  k02  exp f R ,
The posterior density is given by
pr a R a  p a ( A)
pa r A R  
,
pr R 

 N

2
  Ri  A
2
 A  m0  
k0
exp  i 1


N 1 / 2
2
N 1

2 n
2 n2 / k 02 
 n 2 




 n 2
 N 1 2  N 1 / 2
n
k 02
,
N  k02  exp f R 
 N

2
  Ri  A

N
 A  m0 2
i 1


exp 

 f R 
2
2
2


 n 2
2 n
2 n / k 0




Using (3), we have
2


2 
 Nl  m0 k 02   
N  k 02
 N  k 0 
   f (R )  f R  ,
pa r A R  
exp 
A


 N  k 2  
 n 2
2 n2 
0  






 k 02


pa r A R  





N  k02  exp  N  k02   A   Nl  m0 k02   2 

 N  k 2     
 n 2
2 n2 
0   




(ANS)
Prob. 2.4.9
1. Does an efficient estimate of the standard deviation  of a zero-mean
Gaussian density exist?
pr  R   
1
 2

R2
2
e 2
Taking ln(.) both sides of the above equation, we obtain


ln pr  R     ln(  )  0.5 ln 2  
R2
.
2 2
Differentiate the above equation with respect to  and let it equal to zero; i.e.,


 ln p r  R  
1 (2) R 2
 
0


2 3
(1)
Hence, we have
2
ˆ ML
 R2 ,
or
̂ ml  R
Replace ̂ ml  R in (1). We reach




2
 ln p r  R  
1 (2)ˆ ml
 


2 3
Obviously, the above equation cannot be written in the form of
 ln p r  R  
 k ( )ˆ ml    .

As a result, we conclude that the efficient estimate of a standard deviation does not
exist.
2.
Does an efficient estimate of the variance  2 of a zero-mean Gaussian
density exist?
We begin the problem by rewriting the conditional probability of R given  2 , and we
obtain
p
 R  2  
r 

2
1
2 2

R2
2
e 2
To avoid the confusion, we denote  2 by v. Thus, the above equation becomes
pr v R v  
1
2v

e
R2
2v
Taking ln(.) of the above equation, we have


1
R2
ln pr v R v    ln( v)  ln( 2 ) 
2
2v
Next, we differentiate the above equation with respect to v, and we have


 ln p r v R v 
1
R2
 
v
2v 2v 2
Let (2) equal to zero; i.e.,

1
R2

 0,
2vml 2vml 2
vˆml  R 2
We replace vˆml  R 2 in (2), and obtain


 ln p r v R v 
1 vˆ
   ml ,
v
2v 2v 2

1
2v 2
vˆml
 v   k v vˆml  v 
Hence, the efficient estimate of variance exists.
(2)
Prob. 2.4.12
s1  x11 A1  x12 A2,
(a)
s 2  x21 A1  x22 A2,
and
r1  s1  n1 ,
(b)
r2  s 2  n2
where ni is i.i.d. N(0,  n )
pr s R S  
 R  s 2  R  s 2 
2
2 
exp   1 1
2
2


 n 2  
2 n

1
Hence,


 R  x A  x A 2  R  x A  x A 2 
1
11 1
12 2
2
21 1
22 2, 
pr a R A  
exp  


2
2
 n 2  
2 n

Taking ln(.) the above equation, we obtain
1




R1  x11 A1  x12 A2 2  R2  x21 A1  x22 A2, 2
2
ln pr a R A   ln  n 2  
2 n2



(1)
Differentiate (1) with respect to A1 we have


R  x A  x A x  R2  x21 A1  x22 A2 x21

ln pr a R A   1 11 1 12 2 11
A1
 n2
Letting the above equation equal to zero, we achieve
R1  x11 A1  x12 A2 x11  R2  x21 A1  x22 A2 x21
 n2


0,
2
2
A1 x11
 x21
 A2 x11 x12  x21 x22   x11 R1  x21 R2
Again, differentiate (1) with respect to A2 we have


R  x A  x A x  R2  x21 A1  x22 A2 x22

ln pr a R A   1 11 1 12 2 12
A2
 n2
(2)
Letting the above equation equal to zero, we achieve


2
2
A1 x11 x12  x21 x22   A2 x12
 x22
 x12 R1  x22 R2
(3)
Using the linear algebra to solve (2) and (3), we obtain that
aˆ1ml 
x22 R1  x12 R2
,
x11 x22  x12 x21
(4)
aˆ 2 m l 
 x21 R1  x11 R2
x11 x22  x12 x21
(5)
and
2
Compute the variances of estimates
 
 x R x R 
E aˆ1ml  E  22 1 12 2  ,
 x11 x22  x12 x21 
x ER1  x12 ER2 
E aˆ1ml  22
x11 x22  x12 x21
 

 
  
var aˆ1m l  E aˆ1m l  E aˆ1m l 2 ,
 x R  E R   x R  E R   2 

1
12 2
2  ,
 E  22 1
 
x11 x22  x12 x21

 




2
2
x22
var( R1 )  x12
var( R2 )
x11 x22  x12 x21 2
2
2
x22
 x12
 2,
2 n
x11 x22  x12 x21 
because R1 and R2 are statistically independent.

,
(6)
By using the same procedure, we obtain

 
  
var aˆ 2 ml  E aˆ 2 ml  E aˆ 2 ml 2

Next, find the cross correlation
2
2
x21
 x11
x11 x22  x12 x21 
2
 n2
(7)

 

E aˆ1m l  E aˆ1m l aˆ 2 m l  E aˆ 2 m l

 x R  E R1   x12 R2  E R2    x21 R1  E R1   x11 R2  E R2   ,


 E  22 1
x
x

x
x
x
x

x
x
11
22
12
21
11
22
12
21









x x E R1  E R1 2  x12 x11 E R2  E R2 2
,
  22 21
x11 x22  x12 x21 2
x x x x
  22 21 12 11  n2
x11 x22  x12 x21 2
Hence, we have

2
2

x22
 x12

x11 x22  x12 x21 2  x22 x21  x12 x11
 n2
 x22 x21  x12 x11 

2
2
x21
 x11

3.
Find fisher’s information by continue differentiation

2

ln pr a R A  
A12
2
2
 x11
 x21




 n2
 x11 x12  x21 x22
2
,
ln pr a R A  
A1A2
 n2
 x11 x12  x21 x22
2
ln pr a R A  
A2 A1
 n2
2
2
 x22
 x12
2
ln pr a R A  
A22
 n2

Hence,
J  

we have
2
2
1  x11
 x21

 n2  x11 x12  x21 x22
x11 x12  x21 x22 
,
2
2
x22
 x12

and
J 1 
2
2

x22
 x12

x11 x22  x12 x21 2  x11 x12  x21 x22
 n2
We observe that
J 1  Σ .
 x11 x12  x21 x22 
,
2
2
x11
 x21

Hence, we can conclude that the estimates are efficient.
Prob. 2.4.17
Eaˆ A  A  B( A)
(1)
ซึง่ หมายความว่า

Eaˆ  A  B( A)  A   aˆ  A  B( A)  pr a R AdR  0

(2)
differentiate สมการข้ างบนเทียบกับ A จะได้


 1  dB A / dA  pr a R AdR   aˆ  A  B( A) 

  

pr a R A
A
dR  0
(3)
1
จาก calculus เรารู้วา่
 ln u 1 u

x
u x
(4)
ดังนันแล้
้ ว เราจะได้ วา่
pr a R A
 ln pr a R A
A
(5)
 ln pr a R A
dR  1  dB A / dA
A
(6)
A
 pr a R A
เอาสมการ (5) .ใส่ไปใน (3) จะได้

 aˆ  A  B( A) pr a R A

เขียนสมการข้ างบนใหม่ได้ เป็ น

   ln pr a R A
 
  
A


pr a R A aˆ  A  B( A)  pr a R A dR  1  dB A / dA

 
ใช้ Schwartz’s inequality ที่เขียนว่า
(7)
 2
  2
 





x
t
dt
y
t
dt
 
 
    xt  y t dt 
 
  
  

จะเท่ากันก็ตอ่ เมื่อ
y (t )  kx(t )
2
(8)
เมื่อ k เป็ นค่าคงที่ ดังนันสมการ
้
(7) เขียนใหม่เป็ น
    ln p R A 2

ra


 pr a R AdR  
  
A

  



 

2
2
  aˆ  A  B( A)  pr a R AdR   1  dB A / dA
 

(9)
จะเท่ากันก็ตอ่ เมื่อ
  ln pr a R A

 pr a R A  k ( A)aˆ  A  B( A)  pr a R A
A


(10)
สมการ (9) สมารถเขียนใหม่เป็ น
  ln p R A 2
ra


E 

A






AE aˆ  A  B( A) 2 A  1  dB A / dA2




หรื อ
  ln p R A 2 
ra


 
var aˆ  A A  1  dB A / dA2 E 
A
 



1
(11)