Finding Bayes Estimators
Notation: For simplicity, write the random variables X = (X1 , X2 , . . . , Xn ) and
˜
let x = (x1 , x2 , . . . , xn ) denote an observed value of X
˜
˜
Bayes Set-up: Think of
(i) θ as a random variable on Θ
(ii) f (x|θ) = f (x1 , x2 , . . . , xn |θ) as the conditional pdf/pmf of X given θ
˜
˜
(iii) π(θ) as the marginal pmf/pdf of θ
(iv) f (x, θ) = f (x|θ)π(θ) as the joint pmf/pdf of (X , θ) together
˜
˜
˜
Definition: The conditional pdf of θ (assumed continuous), given x = (x1 , x2 , . . . , xn ),
˜
Z
f (x|θ)π(θ)
˜
fθ|x (θ) =
, θ ∈ Θ with m(x) =
f (x|θ)π(θ)dθ
m(x)
˜
˜
Θ
˜
˜
is called the posterior pdf of θ on Θ.
Remark: Since a lot of useful parameter spaces Θ are intervals (e.g., Θ = (0, 1) or (0, ∞)), we will think of θ as
being continuous in the Bayes Set-up.
More Notation: (not real important, only for clarity in motivating the next Theorem)
1. For any function h(X ) = h(X1 , X2 , . . . , Xn ) of X , we write
˜
˜
 P
 (x1 ,x2 ,...,xn ) h(x1 , x2 , . . . , xn )f (x1 , x2 , . . . , xn |θ) discrete X
˜
R
Eθ h(X )
≡ EX |θ h(X ) =

˜
˜
h(x
,
x
,
.
.
.
,
x
)f
(x
,
x
,
.
.
.
,
x
|θ)dx
dx
.
.
.
dx
continuous
X
1
2
n
1
2
n
1
2
n
˜
| {z }
˜
earlier notation
Z
2. E(θ) γ(θ) =
γ(θ)π(θ)dθ (i.e, θ continuous r.v.)
Θ
Z
X
3. E(X ) h(X )= h(x)m(x)dx1 . . . dxn (X continuous) or E(X ) h(X ) =
h(x)m(x) (X discrete)
˜
˜
˜
˜
˜
˜
˜ ˜
˜
˜
x
˜
1
Main idea: For an estimator T = h(X ), recall the risk function of T is RT (θ) =
˜
Eθ L(T, θ) = EX |θ L(T, θ). Then, the Bayes risk of T is
˜
BRT = E(θ) RT (θ) = E(θ) [EX |θ L(T, θ)] = E(X ) [Eθ|x L(T, θ)] = E(X ) [Eθ|x L(h(x), θ)]
˜
˜
˜
˜
˜
˜
To find an estimator T = h(X ) to minimize the Bayes risk BRT , it is enough,
˜
at each fixed data x possibility of X , to pick the “h(x)”-value that minimizes
˜
˜
˜
Eθ|x L(h(x), θ).
˜
˜
Theorem: A Bayes estimator is an estimator that minimizes the “posterior
risk” Eθ|x L(h(x), θ), over all estimators T = h(X ), for fixed values x = (x1 , x2 , . . . , xn )
˜
˜
˜
˜
of X = (X1 , X2 , . . . , Xn ).
˜
Corollary: Let T0 denote the Bayes estimator of γ(θ).
¡
¢2
(1). If L(t, θ) = t − γ(θ) , then T0 = Eθ|x γ(θ). posterior mean of γ(θ)
˜ ¡
¯
¯
¢
¯
¯
(2). If L(t, θ) = t − γ(θ) , then T0 = median γ(θ)|x . posterior median of γ(θ)
˜
Example/continued: X ∼ Binomial(θ), θ ∈ (0, 1); uniform(0, 1) prior for θ;
L(t, θ) = (t − θ)2 . We found Bayes estimator T0 =
Corollary 1
2
X+1
n+2
of γ(θ) = θ, but now try