3.7 Empirical Bayes Analysis
Motivation:
For

X i   i   i ,  i ~ N 0, 
2
, i  1,2,, p.
Then, the MLE estimate is
ˆi  X i (higher dimensional model).
If
1   2     p , the MLE estimate is
p
ˆ  X 
X
i 1
i
(lower dimensional model).
p
Question to ask: which model is better (lower or higher)?
Answer: empirical Bayes analysis is particular desirable in this
situation.
Empirical
“compromise”
Bayes
between
the
method
model
provides
where
i
a
are
completely unrelated (higher dimensional model) and that
where all the
i
are assume to be equal (lower
dimensional model).
(a) Introduction
There are two types of empirical Bayes methods. One is parametric
empirical Bayes (PEB) and the other is nonparametric empirical
Bayes (NPEB).
1
Parametric empirical Bayes: the prior    is in some
parametric class with unknown hyperparameters.
Nonparametric empirical Bayes: one typically assume only that
i
are i.i.d. from some prior
   .
Example 20:
Let




X i ~ N  i ,  2 ,   i  ~ N  ,  2 ,
where

and
 2
are unknown hyperparameters.
◆
Two different ways to carry out empirical Bayes analysis are
1. estimating the prior or posterior by data first, then use
or
ˆ  | x 
ˆ  
to carry out the standard Bayes analysis.
2. finding the Bayes rule in term of unknown prior, and use the
data to estimate the Bayes rule.
(b) Parametric empirical Bayes for normal mean
Example 15 (continue):

X1, X 2 ,, X p , X i ~ N i ,1,    ~ N  , 2
Then,
1.

m  xi  ~ N  ,1   2
The prior using ML-II method is
.
ˆ   ~ N ˆ ,ˆ2   N x , max 0, s 2  1 ,
2

where
̂  x
ˆ 2  max 0, s 2  1.
and
2.
The posterior distribution for X i is
  2 xi  
 2 

f  i | xi  ~ N 
,
2
2 .
1


1



 

Then, the parameter estimate using Bayes rule under square-loss function
is the posterior mean


 2 xi  
1   2 xi   xi  
ˆ
i 

1   2
1   2
x
i

xi
 
1   2
 xi  B xi     1  B xi  B
where B 

1
. Further, the posterior variance is
1   2
2


Vˆi 
1 B .
1   2
Note:
 B  0  ˆi  xi

ˆ
 B  1   i  
◆
Morris (1983, JASA) suggested
p  3 
1
ˆ 

B
 p 1 

 1  ˆ 2





,

and then the empirical Bayes estimate is




ˆ x B
ˆ ˆ  1  B
ˆ x B
ˆx
ˆiEB  1  B
i

i
ˆ  2B
ˆ 2  x  x 2


p  1B
EB
i
ˆ

Vi

1 

p
p3


3
,
where
̂
and
ˆ 2 . are the estimates obtained in step 1.
The estimated posterior distribution is

fˆ i | x ~ N ˆiEB ,Vˆi EB

Note:
 B  0  ˆi  xi

ˆ
 B  1   i  
The 1001   % HPD credible set for  i is
CiEB  ˆiEB  z1 Vˆi EB ,ˆiEB  z1 Vˆi EB 


2
2
and The 1001   % HPD credible set for  is
C
EB

p

 i  ˆiEB

  : 
Vˆi EB

 i 1

2

2
p ,1


.


Example 21:
Y1 , Y2 ,, Yn , Yi ~ N  i ,1,
 i  xi   1 xi1   2 xi 2     p xip   i ,  i ~ N 0, 2 

xi  xi1
Then,
1. Since
xi 2



xip ,   1
2

  i  ~ N xi  , 2 

m  yi  ~ N xi  ,1   2
The marginal distribution is then
4
.
 p t
m  y   m y1 , y2 ,, yn 

 
 2 1   
2
 
 2 1   2

n
2

n
2
n
  y i  x i  2
i 1
e

2 1  2
e

  y  x t  1  y  x 
2
where
 x11 x12
 y1 
x
y 
x22
21
2
y   , X  
 



 
 xn1 xn 2
 yn 
̂
and
ˆ 2
minimizing
 x1 p 
1   2
0


 x2 p 
0
1   2

, 
 
  



 xnp 
0
 0
0 


0 

 

 1   2 

m x  also minimize
 y  X t  1  y  X  .
Thus,

ˆ  X t ˆ 1 X
n

1
ˆ 1 y
X t

 y  x ˆ

i
 i
2
i 1 
ˆ  
n

2
 1

where
1  ˆ 2
0

1  ˆ 2
ˆ   0
 


0
 0
Since
̂
involves
ˆ 2
and
ˆ 2
0 


0 

 .

 1  ˆ 2 

involves
scheme to solve.
2.
The posterior distribution for Yi is
5
̂
, we have to use iterative
  2 yi  xi   2 

f  i | yi  ~ N 
,
2
2 .
1   
 1  
Similar to the previous example, the Empirical Bayes estimates are

ˆiEB  yi  Cˆ i yi  xi ˆ




ˆ Cˆ 
ˆ2

n

l
2
C
EB
i
i
i
ˆ

Vi  1 
yi  xi ˆ

ˆ
n

 n  li  2
where
n  lˆi  2
1
ˆ
Ci 

1  ˆ2
n  lˆi
and


ˆ 1 X
nX X 
ˆ
li 
1  ˆ 2
t

1
,
Xt
(c) Non-parametric empirical Bayes analysis
Example 22:
X 1 , X 2 ,, X p , X i ~ P i ,
i
are i.i.d. from a common prior
0
Under square loss, the estimate is the posterior mean,
6

ii

2
  xi   E f  | x   i 
i
i
  i f  i | xi d i
  i 

f  xi |  i  0  i 
d i
m 0  xi 
1
m 0  xi  
i 
 ix e 
i
  0  i d i
i
xi !

x  1  ix 1e 

  0  i d i
m  xi    xi  1!
x  1m xi  1 f xi  1 |  i 

 m xi  1   0  i d i
m  xi 
x  1m xi  1

 f  i | xi  1d i
m  xi 
x  1m xi  1

m  xi 
i
i
0
0
0
0
0
0
0
0
Further, we can estimate the marginal distribution by the empirical
distribution
p
ˆ  0 x  
m
 I x 
i 1
x
p
i
,
 1 as xi  x


I
x


where x i
. Thus, the empirical Bayes estimate for
0 as xi  x
i
is
7
ˆ EB  x  
 xi
 1   I xi 1 x j 
p
j 1
 I xi x j 
p
j 1
8
.