Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
.
.
Biostatistics 602 - Statistical Inference
Lecture 11
Evaluation of Point Estimators
Hyun Min Kang
February 14th, 2013
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
1 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Some News
• Homework 3 is posted.
• Due is Tuesday, February 26th.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
2 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Some News
• Homework 3 is posted.
• Due is Tuesday, February 26th.
• Next Thursday (Feb 21) is the midterm day.
• We will start sharply at 1:10pm.
• It would be better to solve homework 3 yourself to get prepared.
• The exam is closed book, covering all the material from Lecture 1 to
12.
• Last year’s midterm is posted on the web page.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
2 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Last Lecture
1.
What is a maximum likelihood estimator (MLE)?
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
3 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Last Lecture
1.
What is a maximum likelihood estimator (MLE)?
2.
How can you find an MLE?
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
3 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Last Lecture
1.
What is a maximum likelihood estimator (MLE)?
2.
How can you find an MLE?
3.
Does an ML estimate always fall into a valid parameter space?
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
3 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Last Lecture
1.
What is a maximum likelihood estimator (MLE)?
2.
How can you find an MLE?
3.
Does an ML estimate always fall into a valid parameter space?
4.
If you know MLE of θ, can you also know MLE of τ (θ)?
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
3 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Recap - Maximum Likelihood Estimator
.
Definition
.
• For a given sample point x = (x1 , · · · , xn ),
• let θ̂(x) be the value such that
• L(θ|x) attains its maximum.
• More formally, L(θ̂(x)|x) ≥ L(θ|x) ∀θ ∈ Ω where θ̂(x) ∈ Ω.
• θ̂(x) is called the maximum likelihood estimate of θ based on data x,
.
• and θ̂(X) is the maximum likelihood estimator (MLE) of θ.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
4 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Recap - Invariance Property of MLE
.
Question
.
If
. θ̂ is the MLE of θ, what is the MLE of τ (θ)?
.
Example
.
i.i.d.
X1 , · · · , Xn ∼ Bernoulli(p) where 0 < p < 1.
.
1.
What is the MLE of p?
2.
What is the MLE of odds, defined by η = p/(1 − p)?
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
5 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
p
Getting MLE of η = 1−p
from p̂
∑
∗
L (η|x) =
η xi
(1 + η)n
• From MLE of p̂, we know L∗ (η|x) is maximized when
p = η/(1 + η) = p̂.
• Equivalently, L∗ (η|x) is maximized when η = p̂/(1 − p̂) = τ (p̂),
because τ is a one-to-one function.
• Therefore η̂ = τ (p̂).
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
6 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Invariance Property of MLE
.
Fact
.
Denote the MLE of θ by θ̂. If τ (θ) is an one-to-one function of θ, then
MLE
of τ (θ) is τ (θ̂).
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
7 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Invariance Property of MLE
.
Fact
.
Denote the MLE of θ by θ̂. If τ (θ) is an one-to-one function of θ, then
MLE
of τ (θ) is τ (θ̂).
.
.
Proof
.
The likelihood function in terms of τ (θ) = η is
L∗ (τ (θ)|x) =
n
∏
fX (xi |θ)
i=1
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
7 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Invariance Property of MLE
.
Fact
.
Denote the MLE of θ by θ̂. If τ (θ) is an one-to-one function of θ, then
MLE
of τ (θ) is τ (θ̂).
.
.
Proof
.
The likelihood function in terms of τ (θ) = η is
L∗ (τ (θ)|x) =
n
∏
i=1
fX (xi |θ) =
n
∏
f(xi |τ −1 (η))
i=1
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
7 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Invariance Property of MLE
.
Fact
.
Denote the MLE of θ by θ̂. If τ (θ) is an one-to-one function of θ, then
MLE
of τ (θ) is τ (θ̂).
.
.
Proof
.
The likelihood function in terms of τ (θ) = η is
L∗ (τ (θ)|x) =
n
∏
fX (xi |θ) =
i=1
= L(τ
n
∏
f(xi |τ −1 (η))
i=1
−1
(η)|x)
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
7 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Invariance Property of MLE
.
Fact
.
Denote the MLE of θ by θ̂. If τ (θ) is an one-to-one function of θ, then
MLE
of τ (θ) is τ (θ̂).
.
.
Proof
.
The likelihood function in terms of τ (θ) = η is
L∗ (τ (θ)|x) =
n
∏
fX (xi |θ) =
i=1
= L(τ
n
∏
f(xi |τ −1 (η))
i=1
−1
(η)|x)
We know this function is maximized when τ −1 (η) = θ̂, or equivalently,
.when η = τ (θ̂).
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
7 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Invariance Property of MLE
.
Fact
.
Denote the MLE of θ by θ̂. If τ (θ) is an one-to-one function of θ, then
MLE
of τ (θ) is τ (θ̂).
.
.
Proof
.
The likelihood function in terms of τ (θ) = η is
L∗ (τ (θ)|x) =
n
∏
fX (xi |θ) =
i=1
= L(τ
n
∏
f(xi |τ −1 (η))
i=1
−1
(η)|x)
We know this function is maximized when τ −1 (η) = θ̂, or equivalently,
.when η = τ (θ̂). Therefore, MLE of η = τ (θ) is τ (θ̂).
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
7 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Induced Likelihood Function
.
Definition
.
• Let L(θ|x) be the likelihood function for a given data x1 , · · · , xn ,
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
8 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Induced Likelihood Function
.
Definition
.
• Let L(θ|x) be the likelihood function for a given data x1 , · · · , xn ,
• and let η = τ (θ) be a (possibly not a one-to-one) function of θ.
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
8 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Induced Likelihood Function
.
Definition
.
• Let L(θ|x) be the likelihood function for a given data x1 , · · · , xn ,
• and let η = τ (θ) be a (possibly not a one-to-one) function of θ.
We define the induced likelihood function L∗ by
L∗ (η|x) =
sup
θ∈τ −1 (η)
L(θ|x)
where τ −1 (η) = {θ : τ (θ) = η, θ ∈ Ω}.
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
8 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Induced Likelihood Function
.
Definition
.
• Let L(θ|x) be the likelihood function for a given data x1 , · · · , xn ,
• and let η = τ (θ) be a (possibly not a one-to-one) function of θ.
We define the induced likelihood function L∗ by
L∗ (η|x) =
sup
θ∈τ −1 (η)
L(θ|x)
where τ −1 (η) = {θ : τ (θ) = η, θ ∈ Ω}.
.
• The value of η that maximize L∗ (η|x) is called the MLE of η = τ (θ).
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
8 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Invariance Property of MLE
.
Theorem 7.2.10
.
If θ is the MLE of θ̂, then the MLE of η = τ (θ) is τ (θ̂), where τ (θ) is any
.function of θ.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
9 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Invariance Property of MLE
.
Theorem 7.2.10
.
If θ is the MLE of θ̂, then the MLE of η = τ (θ) is τ (θ̂), where τ (θ) is any
.function of θ.
.
Proof - Using Induced Likelihood Function
.
L∗ (η̂|x) = sup L∗ (η|x)
η
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
9 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Invariance Property of MLE
.
Theorem 7.2.10
.
If θ is the MLE of θ̂, then the MLE of η = τ (θ) is τ (θ̂), where τ (θ) is any
.function of θ.
.
Proof - Using Induced Likelihood Function
.
L∗ (η̂|x) = sup L∗ (η|x) = sup
η
η
sup
θ∈τ −1 (η)
L(θ|x)
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
9 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Invariance Property of MLE
.
Theorem 7.2.10
.
If θ is the MLE of θ̂, then the MLE of η = τ (θ) is τ (θ̂), where τ (θ) is any
.function of θ.
.
Proof - Using Induced Likelihood Function
.
L∗ (η̂|x) = sup L∗ (η|x) = sup
η
η
sup
θ∈τ −1 (η)
L(θ|x)
= sup L(θ|x)
θ
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
9 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Invariance Property of MLE
.
Theorem 7.2.10
.
If θ is the MLE of θ̂, then the MLE of η = τ (θ) is τ (θ̂), where τ (θ) is any
.function of θ.
.
Proof - Using Induced Likelihood Function
.
L∗ (η̂|x) = sup L∗ (η|x) = sup
η
η
sup
θ∈τ −1 (η)
L(θ|x)
= sup L(θ|x) = L(θ̂|x)
θ
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
9 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Invariance Property of MLE
.
Theorem 7.2.10
.
If θ is the MLE of θ̂, then the MLE of η = τ (θ) is τ (θ̂), where τ (θ) is any
.function of θ.
.
Proof - Using Induced Likelihood Function
.
L∗ (η̂|x) = sup L∗ (η|x) = sup
η
η
sup
θ∈τ −1 (η)
L(θ|x)
= sup L(θ|x) = L(θ̂|x)
θ
L(θ̂|x) =
sup
L(θ|x)
θ∈τ −1 (τ (θ̂))
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
9 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Invariance Property of MLE
.
Theorem 7.2.10
.
If θ is the MLE of θ̂, then the MLE of η = τ (θ) is τ (θ̂), where τ (θ) is any
.function of θ.
.
Proof - Using Induced Likelihood Function
.
L∗ (η̂|x) = sup L∗ (η|x) = sup
η
η
sup
θ∈τ −1 (η)
L(θ|x)
= sup L(θ|x) = L(θ̂|x)
θ
L(θ̂|x) =
sup
L(θ|x) = L∗ [τ (θ̂)|x]
θ∈τ −1 (τ (θ̂))
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
9 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Invariance Property of MLE
.
Theorem 7.2.10
.
If θ is the MLE of θ̂, then the MLE of η = τ (θ) is τ (θ̂), where τ (θ) is any
.function of θ.
.
Proof - Using Induced Likelihood Function
.
L∗ (η̂|x) = sup L∗ (η|x) = sup
η
η
sup
θ∈τ −1 (η)
L(θ|x)
= sup L(θ|x) = L(θ̂|x)
θ
L(θ̂|x) =
sup
L(θ|x) = L∗ [τ (θ̂)|x]
θ∈τ −1 (τ (θ̂))
Hence,
L∗ (η̂|x) = L∗ [τ (θ̂)|x] and τ (θ̂) is the MLE of τ (θ).
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
9 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Properties of MLE
1.
Optimal in some sense : We will study this later
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
10 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Properties of MLE
1.
Optimal in some sense : We will study this later
2.
By definition, MLE will always fall into the range of the parameter
space.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
10 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Properties of MLE
1.
Optimal in some sense : We will study this later
2.
By definition, MLE will always fall into the range of the parameter
space.
3.
Not always easy to obtain; may be hard to find the global maximum.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
10 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Properties of MLE
1.
Optimal in some sense : We will study this later
2.
By definition, MLE will always fall into the range of the parameter
space.
3.
Not always easy to obtain; may be hard to find the global maximum.
4.
Heavily depends on the underlying distributional assumptions (i.e. not
robust).
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
10 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Method of Evaluating Estimators
.
Definition : Unbiasedness
.
Suppose θ̂ is an estimator for θ, then the bias of θ is defined as
Bias(θ) = E(θ̂) − θ
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
11 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Method of Evaluating Estimators
.
Definition : Unbiasedness
.
Suppose θ̂ is an estimator for θ, then the bias of θ is defined as
Bias(θ) = E(θ̂) − θ
If
. the bias is equal to 0, then θ̂ is an unbiased estimator for θ.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
11 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Method of Evaluating Estimators
.
Definition : Unbiasedness
.
Suppose θ̂ is an estimator for θ, then the bias of θ is defined as
Bias(θ) = E(θ̂) − θ
If
. the bias is equal to 0, then θ̂ is an unbiased estimator for θ.
.
Example
.
X1 , · · · ∑
, Xn are iid samples from a distribution with mean µ. Let
X = n1 ni=1 Xi is an estimator of µ.
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
11 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Method of Evaluating Estimators
.
Definition : Unbiasedness
.
Suppose θ̂ is an estimator for θ, then the bias of θ is defined as
Bias(θ) = E(θ̂) − θ
If
. the bias is equal to 0, then θ̂ is an unbiased estimator for θ.
.
Example
.
X1 , · · · ∑
, Xn are iid samples from a distribution with mean µ. Let
X = n1 ni=1 Xi is an estimator of µ. The bias is
Bias(µ) = E(X) − µ
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
11 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Method of Evaluating Estimators
.
Definition : Unbiasedness
.
Suppose θ̂ is an estimator for θ, then the bias of θ is defined as
Bias(θ) = E(θ̂) − θ
If
. the bias is equal to 0, then θ̂ is an unbiased estimator for θ.
.
Example
.
X1 , · · · ∑
, Xn are iid samples from a distribution with mean µ. Let
X = n1 ni=1 Xi is an estimator of µ. The bias is
Bias(µ) = E(X) − µ
( n
)
1∑
= E
Xi − µ
n
i=1
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
11 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Method of Evaluating Estimators
.
Definition : Unbiasedness
.
Suppose θ̂ is an estimator for θ, then the bias of θ is defined as
Bias(θ) = E(θ̂) − θ
If
. the bias is equal to 0, then θ̂ is an unbiased estimator for θ.
.
Example
.
X1 , · · · ∑
, Xn are iid samples from a distribution with mean µ. Let
X = n1 ni=1 Xi is an estimator of µ. The bias is
Bias(µ) = E(X) − µ
( n
)
1∑
= E
Xi − µ
n
i=1
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
11 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Method of Evaluating Estimators
.
Definition : Unbiasedness
.
Suppose θ̂ is an estimator for θ, then the bias of θ is defined as
Bias(θ) = E(θ̂) − θ
If
. the bias is equal to 0, then θ̂ is an unbiased estimator for θ.
.
Example
.
X1 , · · · ∑
, Xn are iid samples from a distribution with mean µ. Let
X = n1 ni=1 Xi is an estimator of µ. The bias is
Bias(µ) = E(X) − µ
( n
)
n
1∑
1∑
= E
Xi − µ =
E (Xi ) − µ
n
n
i=1
i=1
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
11 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Method of Evaluating Estimators
.
Definition : Unbiasedness
.
Suppose θ̂ is an estimator for θ, then the bias of θ is defined as
Bias(θ) = E(θ̂) − θ
If
. the bias is equal to 0, then θ̂ is an unbiased estimator for θ.
.
Example
.
X1 , · · · ∑
, Xn are iid samples from a distribution with mean µ. Let
X = n1 ni=1 Xi is an estimator of µ. The bias is
Bias(µ) = E(X) − µ
( n
)
n
1∑
1∑
= E
Xi − µ =
E (Xi ) − µ = µ − µ = 0
n
n
i=1
i=1
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
11 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Method of Evaluating Estimators
.
Definition : Unbiasedness
.
Suppose θ̂ is an estimator for θ, then the bias of θ is defined as
Bias(θ) = E(θ̂) − θ
If
. the bias is equal to 0, then θ̂ is an unbiased estimator for θ.
.
Example
.
X1 , · · · ∑
, Xn are iid samples from a distribution with mean µ. Let
X = n1 ni=1 Xi is an estimator of µ. The bias is
Bias(µ) = E(X) − µ
( n
)
n
1∑
1∑
= E
Xi − µ =
E (Xi ) − µ = µ − µ = 0
n
n
i=1
i=1
Therefore
X is an unbiased estimator for µ.
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
11 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
How important is unbiased?
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
12 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
How important is unbiased?
• θ̂1 (blue) is unbiased but has a chance to be very far away from θ = 0.
• θ̂2 (red) is biased but more likely to be closer to the true θ than θ̂1 .
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
12 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Mean Squared Error
.
Definition
.
Mean Squared Error (MSE) of an estimator θ̂ is defined as
MSE(θ̂) = E[(θ̂ − θ)]2
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
13 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Mean Squared Error
.
Definition
.
Mean Squared Error (MSE) of an estimator θ̂ is defined as
MSE(θ̂) = E[(θ̂ − θ)]2
.
.
Property of MSE
.
MSE(θ̂) = E[(θ̂ − Eθ̂ + Eθ̂ − θ)]2
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
13 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Mean Squared Error
.
Definition
.
Mean Squared Error (MSE) of an estimator θ̂ is defined as
MSE(θ̂) = E[(θ̂ − θ)]2
.
.
Property of MSE
.
MSE(θ̂) = E[(θ̂ − Eθ̂ + Eθ̂ − θ)]2
= E[(θ̂ − Eθ̂)2 ] + E[(Eθ̂ − θ)2 ] + 2E[(θ̂ − Eθ̂)]E[(Eθ̂ − θ)]
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
13 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Mean Squared Error
.
Definition
.
Mean Squared Error (MSE) of an estimator θ̂ is defined as
MSE(θ̂) = E[(θ̂ − θ)]2
.
.
Property of MSE
.
MSE(θ̂) = E[(θ̂ − Eθ̂ + Eθ̂ − θ)]2
= E[(θ̂ − Eθ̂)2 ] + E[(Eθ̂ − θ)2 ] + 2E[(θ̂ − Eθ̂)]E[(Eθ̂ − θ)]
= E[(θ̂ − Eθ̂)2 ] + (Eθ̂ − θ)2 + 2(Eθ̂ − Eθ̂)E[(Eθ̂ − θ)]
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
13 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Mean Squared Error
.
Definition
.
Mean Squared Error (MSE) of an estimator θ̂ is defined as
MSE(θ̂) = E[(θ̂ − θ)]2
.
.
Property of MSE
.
MSE(θ̂) = E[(θ̂ − Eθ̂ + Eθ̂ − θ)]2
= E[(θ̂ − Eθ̂)2 ] + E[(Eθ̂ − θ)2 ] + 2E[(θ̂ − Eθ̂)]E[(Eθ̂ − θ)]
= E[(θ̂ − Eθ̂)2 ] + (Eθ̂ − θ)2 + 2(Eθ̂ − Eθ̂)E[(Eθ̂ − θ)]
.
= Var(θ̂) + Bias2 (θ)
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
13 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example
i.i.d.
• X1 , · · · , Xn ∼ N (µ, 1)
• µ1 = 1, µ2 = X.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
14 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example
i.i.d.
• X1 , · · · , Xn ∼ N (µ, 1)
• µ1 = 1, µ2 = X.
MSE(µ̂1 ) = E(µ̂1 − µ)2 = (1 − µ)2
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
14 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example
i.i.d.
• X1 , · · · , Xn ∼ N (µ, 1)
• µ1 = 1, µ2 = X.
MSE(µ̂1 ) = E(µ̂1 − µ)2 = (1 − µ)2
MSE(µ̂2 ) = E(X − µ)2 = Var(X) =
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
1
n
.
.
.
February 14th, 2013
.
14 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example
i.i.d.
• X1 , · · · , Xn ∼ N (µ, 1)
• µ1 = 1, µ2 = X.
MSE(µ̂1 ) = E(µ̂1 − µ)2 = (1 − µ)2
MSE(µ̂2 ) = E(X − µ)2 = Var(X) =
1
n
• Suppose that the true µ = 1, then MSE(µ1 ) = 0 < MSE(µ2 ), and no
estimator can beat µ1 in terms of MSE when true µ = 1.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
14 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example
i.i.d.
• X1 , · · · , Xn ∼ N (µ, 1)
• µ1 = 1, µ2 = X.
MSE(µ̂1 ) = E(µ̂1 − µ)2 = (1 − µ)2
MSE(µ̂2 ) = E(X − µ)2 = Var(X) =
1
n
• Suppose that the true µ = 1, then MSE(µ1 ) = 0 < MSE(µ2 ), and no
estimator can beat µ1 in terms of MSE when true µ = 1.
• Therefore, we cannot find an estimator that is uniformly the best in
terms of MSE across all θ ∈ Ω among all estimators
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
14 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example
i.i.d.
• X1 , · · · , Xn ∼ N (µ, 1)
• µ1 = 1, µ2 = X.
MSE(µ̂1 ) = E(µ̂1 − µ)2 = (1 − µ)2
MSE(µ̂2 ) = E(X − µ)2 = Var(X) =
1
n
• Suppose that the true µ = 1, then MSE(µ1 ) = 0 < MSE(µ2 ), and no
estimator can beat µ1 in terms of MSE when true µ = 1.
• Therefore, we cannot find an estimator that is uniformly the best in
terms of MSE across all θ ∈ Ω among all estimators
• Restrict the class of estimators, and find the ”best” estimator within
the small class.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
14 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Uniformly Minimum Variance Unbiased Estimator
.
Definition
. ∗
W (X) is the best unbiased estimator, or uniformly minimum variance
unbiased estimator (UMVUE) of τ (θ) if,
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
15 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Uniformly Minimum Variance Unbiased Estimator
.
Definition
. ∗
W (X) is the best unbiased estimator, or uniformly minimum variance
unbiased estimator (UMVUE) of τ (θ) if,
1.
E[W∗ (X)|θ] = τ (θ) for all θ (unbiased)
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
15 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Uniformly Minimum Variance Unbiased Estimator
.
Definition
. ∗
W (X) is the best unbiased estimator, or uniformly minimum variance
unbiased estimator (UMVUE) of τ (θ) if,
1.
2.
.
E[W∗ (X)|θ] = τ (θ) for all θ (unbiased)
and Var[W∗ (X)|θ] ≤ Var[W(X)|θ] for all θ, where W is any other
unbiased estimator of τ (θ) (minimum variance).
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
15 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Uniformly Minimum Variance Unbiased Estimator
.
Definition
. ∗
W (X) is the best unbiased estimator, or uniformly minimum variance
unbiased estimator (UMVUE) of τ (θ) if,
1.
2.
.
E[W∗ (X)|θ] = τ (θ) for all θ (unbiased)
and Var[W∗ (X)|θ] ≤ Var[W(X)|θ] for all θ, where W is any other
unbiased estimator of τ (θ) (minimum variance).
.
How to find the Best Unbiased Estimator
.
• Find the lower bound of variances of any unbiased estimator of τ (θ),
say B(θ).
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
15 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Uniformly Minimum Variance Unbiased Estimator
.
Definition
. ∗
W (X) is the best unbiased estimator, or uniformly minimum variance
unbiased estimator (UMVUE) of τ (θ) if,
1.
2.
.
E[W∗ (X)|θ] = τ (θ) for all θ (unbiased)
and Var[W∗ (X)|θ] ≤ Var[W(X)|θ] for all θ, where W is any other
unbiased estimator of τ (θ) (minimum variance).
.
How to find the Best Unbiased Estimator
.
• Find the lower bound of variances of any unbiased estimator of τ (θ),
say B(θ).
• If W∗ is an unbiased estimator of τ (θ) and satisfies
.
Var[W∗ (X)|θ] = B(θ), then W∗ is the best unbiased estimator.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
15 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Cramer-Rao inequality
.
Theorem 7.3.9 : Cramer-Rao Theorem
.
Let X1 , · · · , Xn be a sample with joint pdf/pmf of fX (x|θ). Suppose W(X)
is an estimator satisfying
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
16 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Cramer-Rao inequality
.
Theorem 7.3.9 : Cramer-Rao Theorem
.
Let X1 , · · · , Xn be a sample with joint pdf/pmf of fX (x|θ). Suppose W(X)
is an estimator satisfying
1.
E[W(X)|θ] = τ (θ), ∀θ ∈ Ω.
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
16 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Cramer-Rao inequality
.
Theorem 7.3.9 : Cramer-Rao Theorem
.
Let X1 , · · · , Xn be a sample with joint pdf/pmf of fX (x|θ). Suppose W(X)
is an estimator satisfying
1.
E[W(X)|θ] = τ (θ), ∀θ ∈ Ω.
2.
Var[W(X)|θ] < ∞.
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
16 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Cramer-Rao inequality
.
Theorem 7.3.9 : Cramer-Rao Theorem
.
Let X1 , · · · , Xn be a sample with joint pdf/pmf of fX (x|θ). Suppose W(X)
is an estimator satisfying
1.
E[W(X)|θ] = τ (θ), ∀θ ∈ Ω.
2.
Var[W(X)|θ] < ∞.
For h(x) = 1 and h(x) = W(x), if the differentiation and integrations are
interchangeable, i.e.
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
16 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Cramer-Rao inequality
.
Theorem 7.3.9 : Cramer-Rao Theorem
.
Let X1 , · · · , Xn be a sample with joint pdf/pmf of fX (x|θ). Suppose W(X)
is an estimator satisfying
1.
E[W(X)|θ] = τ (θ), ∀θ ∈ Ω.
2.
Var[W(X)|θ] < ∞.
For h(x) = 1 and h(x) = W(x), if the differentiation and integrations are
interchangeable, i.e.
∫
∫
∂
d
d
h(x) fX (x|θ)dx
h(x)fX (x|θ)dx =
E[h(x)|θ] =
dθ
dθ x∈X
∂θ
x∈X
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
16 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Cramer-Rao inequality
.
Theorem 7.3.9 : Cramer-Rao Theorem
.
Let X1 , · · · , Xn be a sample with joint pdf/pmf of fX (x|θ). Suppose W(X)
is an estimator satisfying
1.
E[W(X)|θ] = τ (θ), ∀θ ∈ Ω.
2.
Var[W(X)|θ] < ∞.
For h(x) = 1 and h(x) = W(x), if the differentiation and integrations are
interchangeable, i.e.
∫
∫
∂
d
d
h(x) fX (x|θ)dx
h(x)fX (x|θ)dx =
E[h(x)|θ] =
dθ
dθ x∈X
∂θ
x∈X
Then, a lower bound of Var[W(X)|θ] is
Var[W(X)] ≥
.
[τ ′ (θ)]2
]
∂
E { ∂θ
log fX (x|θ)}2
[
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
16 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (1/4)
By Cauchy-Schwarz inequality,
[Cov(X, Y)]2 ≤ Var(X)Var(Y)
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
17 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (1/4)
By Cauchy-Schwarz inequality,
[Cov(X, Y)]2 ≤ Var(X)Var(Y)
Replacing X and Y,
[
]2
[
]
∂
∂
Cov{W(X),
log fX (X|θ)}
≤ Var[W(X)]Var
log fX (X|θ)
∂θ
∂θ
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
17 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (1/4)
By Cauchy-Schwarz inequality,
[Cov(X, Y)]2 ≤ Var(X)Var(Y)
Replacing X and Y,
[
]2
[
]
∂
∂
Cov{W(X),
log fX (X|θ)}
≤ Var[W(X)]Var
log fX (X|θ)
∂θ
∂θ
]2
[
∂
Cov{W(X), ∂θ
log fX (X|θ)}
]
[∂
Var[W(X)] ≥
Var ∂θ
log fX (X|θ)
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
17 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (1/4)
By Cauchy-Schwarz inequality,
[Cov(X, Y)]2 ≤ Var(X)Var(Y)
Replacing X and Y,
[
]2
[
]
∂
∂
Cov{W(X),
log fX (X|θ)}
≤ Var[W(X)]Var
log fX (X|θ)
∂θ
∂θ
]2
[
∂
Cov{W(X), ∂θ
log fX (X|θ)}
]
[∂
Var[W(X)] ≥
Var ∂θ
log fX (X|θ)
Using Var(X) = EX2 − (EX)2 ,
[{
[
]
}2 ]
[
]2
∂
∂
∂
Var
log fX (X|θ) = E
log fX (X|θ)
−E
log fX (X|θ)
∂θ
∂θ
∂θ
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
17 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (2/4)
[
]
]
[
∫
∂
∂
E
log fX (X|θ) =
log fX (x|θ) fX (x|θ)dx
∂θ
x∈X ∂θ
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
18 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (2/4)
[
]
]
[
∫
∂
∂
E
log fX (X|θ) =
log fX (x|θ) fX (x|θ)dx
∂θ
x∈X ∂θ
∫
∂
∂θ fX (x|θ)
=
fX (x|θ)dx
x∈X fX (x|θ)
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
18 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (2/4)
[
]
]
[
∫
∂
∂
E
log fX (X|θ) =
log fX (x|θ) fX (x|θ)dx
∂θ
x∈X ∂θ
∫
∂
∂θ fX (x|θ)
=
f (x|θ)dx
f (x|θ) X
∫x∈X X
∂
=
fX (x|θ)dx
x∈X ∂θ
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
18 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (2/4)
[
]
]
[
∫
∂
∂
E
log fX (X|θ) =
log fX (x|θ) fX (x|θ)dx
∂θ
x∈X ∂θ
∫
∂
∂θ fX (x|θ)
=
f (x|θ)dx
f (x|θ) X
∫x∈X X
∂
=
fX (x|θ)dx
x∈X ∂θ
∫
d
=
f (x|θ)dx
(by assumption)
dθ x∈X X
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
18 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (2/4)
[
]
]
[
∫
∂
∂
E
log fX (X|θ) =
log fX (x|θ) fX (x|θ)dx
∂θ
x∈X ∂θ
∫
∂
∂θ fX (x|θ)
=
f (x|θ)dx
f (x|θ) X
∫x∈X X
∂
=
fX (x|θ)dx
x∈X ∂θ
∫
d
=
f (x|θ)dx
(by assumption)
dθ x∈X X
d
=
1=0
dθ
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
18 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (2/4)
[
]
]
[
∫
∂
∂
E
log fX (X|θ) =
log fX (x|θ) fX (x|θ)dx
∂θ
x∈X ∂θ
∫
∂
∂θ fX (x|θ)
=
f (x|θ)dx
f (x|θ) X
∫x∈X X
∂
=
fX (x|θ)dx
x∈X ∂θ
∫
d
=
f (x|θ)dx
(by assumption)
dθ x∈X X
d
=
1=0
dθ[
[
]
{
}2 ]
∂
∂
Var
log fX (X|θ) = E
log fX (X|θ)
∂θ
∂θ
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
18 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (3/4)
[
]
∂
Cov W(X),
log fX (X|θ)
∂θ
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
19 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (3/4)
[
]
∂
Cov W(X),
log fX (X|θ)
∂θ
[
]
[
]
∂
∂
= E W(X) ·
log fX (X|θ) − E [W(X)] E
log fX (X|θ)
∂θ
∂θ
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
19 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (3/4)
[
]
∂
Cov W(X),
log fX (X|θ)
∂θ
[
]
[
]
∂
∂
= E W(X) ·
log fX (X|θ) − E [W(X)] E
log fX (X|θ)
∂θ
∂θ
[
]
∂
= E W(X) ·
log fX (X|θ)
∂θ
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
19 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (3/4)
[
]
∂
Cov W(X),
log fX (X|θ)
∂θ
[
]
[
]
∂
∂
= E W(X) ·
log fX (X|θ) − E [W(X)] E
log fX (X|θ)
∂θ
∂θ
[
] ∫
∂
∂
= E W(X) ·
log fX (X|θ) =
W(x) log fX (x|θ)f(x|θ)dx
∂θ
∂θ
x∈X
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
19 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (3/4)
[
]
∂
Cov W(X),
log fX (X|θ)
∂θ
[
]
[
]
∂
∂
= E W(X) ·
log fX (X|θ) − E [W(X)] E
log fX (X|θ)
∂θ
∂θ
[
] ∫
∂
∂
= E W(X) ·
log fX (X|θ) =
W(x) log fX (x|θ)f(x|θ)dx
∂θ
∂θ
x∈X
∫
∂
f (x|θ)
W(x) ∂θ X
=
f(x|θ)dx
f(x|θ)
x∈X
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
19 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (3/4)
[
]
∂
Cov W(X),
log fX (X|θ)
∂θ
[
]
[
]
∂
∂
= E W(X) ·
log fX (X|θ) − E [W(X)] E
log fX (X|θ)
∂θ
∂θ
[
] ∫
∂
∂
= E W(X) ·
log fX (X|θ) =
W(x) log fX (x|θ)f(x|θ)dx
∂θ
∂θ
x∈X
∫
∫
∂
f (x|θ)
∂
W(x) ∂θ X
W(x) fX (x|θ)
=
f(x|θ)dx =
f(x|θ)
∂θ
x∈X
x∈X
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
19 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (3/4)
[
=
=
=
=
]
∂
Cov W(X),
log fX (X|θ)
∂θ
[
]
[
]
∂
∂
E W(X) ·
log fX (X|θ) − E [W(X)] E
log fX (X|θ)
∂θ
∂θ
[
] ∫
∂
∂
E W(X) ·
log fX (X|θ) =
W(x) log fX (x|θ)f(x|θ)dx
∂θ
∂θ
x∈X
∫
∫
∂
f (x|θ)
∂
W(x) ∂θ X
W(x) fX (x|θ)
f(x|θ)dx =
f(x|θ)
∂θ
x∈X
x∈X
∫
d
W(x)fX (x|θ)
(by assumption)
dθ x∈X
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
19 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (3/4)
[
=
=
=
=
=
]
∂
Cov W(X),
log fX (X|θ)
∂θ
[
]
[
]
∂
∂
E W(X) ·
log fX (X|θ) − E [W(X)] E
log fX (X|θ)
∂θ
∂θ
[
] ∫
∂
∂
E W(X) ·
log fX (X|θ) =
W(x) log fX (x|θ)f(x|θ)dx
∂θ
∂θ
x∈X
∫
∫
∂
f (x|θ)
∂
W(x) ∂θ X
W(x) fX (x|θ)
f(x|θ)dx =
f(x|θ)
∂θ
x∈X
x∈X
∫
d
W(x)fX (x|θ)
(by assumption)
dθ x∈X
d
d
E [W(X)] =
τ (θ) = τ ′ (θ)
dθ
dθ
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
19 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (4/4)
From the previous results
[{
]
}2 ]
∂
∂
Var
log fX (X|θ) = E
log fX (X|θ)
∂θ
∂θ
[
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
20 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (4/4)
From the previous results
[{
]
}2 ]
∂
∂
Var
log fX (X|θ) = E
log fX (X|θ)
∂θ
∂θ
[
]
∂
Cov W(X),
log fX (X|θ) = τ ′ (θ)
∂θ
[
Therefore, Cramer-Rao lower bound is
Var[W(X)] ≥
[
]2
∂
Cov{W(X), ∂θ
log fX (X|θ)}
[∂
]
Var ∂θ
log fX (X|θ)
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
20 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Cramer-Rao Theorem (4/4)
From the previous results
[{
]
}2 ]
∂
∂
Var
log fX (X|θ) = E
log fX (X|θ)
∂θ
∂θ
[
]
∂
Cov W(X),
log fX (X|θ) = τ ′ (θ)
∂θ
[
Therefore, Cramer-Rao lower bound is
Var[W(X)] ≥
=
[
]2
∂
Cov{W(X), ∂θ
log fX (X|θ)}
[∂
]
Var ∂θ
log fX (X|θ)
[τ ′ (θ)]2
]
∂
E { ∂θ
log fX (X|θ)}2
[
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
20 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Cramer-Rao bound in iid case
.
Corollary 7.3.10
.
If X1 , · · · , Xn are iid samples from pdf/pmf fX (x|θ), and the assumptions
in the above Cramer-Rao theorem hold, then the lower-bound of
Var[W(X)|θ] becomes
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
21 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Cramer-Rao bound in iid case
.
Corollary 7.3.10
.
If X1 , · · · , Xn are iid samples from pdf/pmf fX (x|θ), and the assumptions
in the above Cramer-Rao theorem hold, then the lower-bound of
Var[W(X)|θ] becomes
[τ ′ (θ)]2
[ ∂
]
Var[W(X)] ≥
2
nE
{
log
f
(X|θ)}
X
∂θ
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
21 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Cramer-Rao bound in iid case
.
Corollary 7.3.10
.
If X1 , · · · , Xn are iid samples from pdf/pmf fX (x|θ), and the assumptions
in the above Cramer-Rao theorem hold, then the lower-bound of
Var[W(X)|θ] becomes
[τ ′ (θ)]2
[ ∂
]
Var[W(X)] ≥
2
nE
{
log
f
(X|θ)}
X
∂θ
.
.
Proof
.
We need to[show
[{
{ that
}2 ]
}2 ]
∂
∂
E
log fX (X|θ)
= nE
log fX (X|θ)
∂θ
∂θ
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
21 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Corollary 7.3.10
[{
E
∂
log fX (X|θ)
∂θ
}2 ]
{
}2
n
∏
∂
= E
log
fX (Xi |θ)
∂θ
i=1
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
22 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Corollary 7.3.10
[{
E
∂
log fX (X|θ)
∂θ
}2 ]
{
}2
n
∏
∂
= E
log
fX (Xi |θ)
∂θ
i=1
{
}2
n
∑
∂
= E
log fX (Xi |θ)
∂θ
i=1
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
22 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Corollary 7.3.10
[{
E
∂
log fX (X|θ)
∂θ
}2 ]
{
}2
n
∏
∂
= E
log
fX (Xi |θ)
∂θ
i=1
{
}2
n
∑
∂
= E
log fX (Xi |θ)
∂θ
i=1
{
}2
n
∑
∂
= E
log fX (Xi |θ)
∂θ
i=1
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
22 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Corollary 7.3.10
[{
E
∂
log fX (X|θ)
∂θ
}2 ]
=
=
=
=
{
}2
n
∏
∂
E
log
fX (Xi |θ)
∂θ
i=1
{
}2
n
∑
∂
E
log fX (Xi |θ)
∂θ
i=1
{
}2
n
∑
∂
E
log fX (Xi |θ)
∂θ
i=1
[∑ {
}2
n
∂
E
+
i=1 ∂θ log fX (Xi |θ)
]
∑
∂
∂
log
f
(X
|θ)
log
f
(X
|θ)
i
j
X
X
i̸=j ∂θ
∂θ
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
22 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Corollary 7.3.10
Because X1 , · · · , Xn are independent,
∑ ∂
∂
E
log fX (Xi |θ) log fX (Xj |θ)
∂θ
∂θ
i̸=j
]
[
]
[
∑
∂
∂
=
log fX (Xi |θ) E
log fX (Xj |θ) = 0
E
∂θ
∂θ
i̸=j
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
23 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Corollary 7.3.10
Because X1 , · · · , Xn are independent,
∑ ∂
∂
E
log fX (Xi |θ) log fX (Xj |θ)
∂θ
∂θ
i̸=j
]
[
]
[
∑
∂
∂
=
log fX (Xi |θ) E
log fX (Xj |θ) = 0
E
∂θ
∂θ
i̸=j
[{
E
[ n {
}2 ]
}2 ]
∑ ∂
∂
log fX (X|θ)
= E
log fX (Xi |θ)
∂θ
∂θ
i=1
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
23 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Corollary 7.3.10
Because X1 , · · · , Xn are independent,
∑ ∂
∂
E
log fX (Xi |θ) log fX (Xj |θ)
∂θ
∂θ
i̸=j
]
[
]
[
∑
∂
∂
=
log fX (Xi |θ) E
log fX (Xj |θ) = 0
E
∂θ
∂θ
i̸=j
[{
E
[ n {
}2 ]
}2 ]
∑ ∂
∂
log fX (X|θ)
= E
log fX (Xi |θ)
∂θ
∂θ
i=1
[{
}2 ]
n
∑
∂
log fX (Xi |θ)
=
E
∂θ
i=1
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
23 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Proving Corollary 7.3.10
Because X1 , · · · , Xn are independent,
∑ ∂
∂
E
log fX (Xi |θ) log fX (Xj |θ)
∂θ
∂θ
i̸=j
]
[
]
[
∑
∂
∂
=
log fX (Xi |θ) E
log fX (Xj |θ) = 0
E
∂θ
∂θ
i̸=j
[{
E
[ n {
}2 ]
}2 ]
∑ ∂
∂
log fX (X|θ)
= E
log fX (Xi |θ)
∂θ
∂θ
i=1
[{
}2 ]
n
∑
∂
log fX (Xi |θ)
=
E
∂θ
i=1
[{
}2 ]
∂
= nE
log fX (X|θ)
∂θ
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
23 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Remark from Corollary 7.3.10
In iid case, Cramer-Rao lower bound for an unbiased estimator of θ is
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
24 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Remark from Corollary 7.3.10
In iid case, Cramer-Rao lower bound for an unbiased estimator of θ is
Var[W(X)] ≥
1
[ ∂
]
nE { ∂θ log fX (X|θ)}2
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
24 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Remark from Corollary 7.3.10
In iid case, Cramer-Rao lower bound for an unbiased estimator of θ is
Var[W(X)] ≥
1
[ ∂
]
nE { ∂θ log fX (X|θ)}2
Because τ (θ) = θ and τ ′ (θ) = 1.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
24 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Score Function
.
Definition: Score or Score Function for X
.
i.i.d.
X1 , · · · , Xn ∼ fX (x|θ)
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
25 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Score Function
.
Definition: Score or Score Function for X
.
i.i.d.
X1 , · · · , Xn ∼ fX (x|θ)
∂
S(X|θ) =
log fX (X|θ)
∂θ
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
25 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Score Function
.
Definition: Score or Score Function for X
.
i.i.d.
X1 , · · · , Xn ∼ fX (x|θ)
∂
S(X|θ) =
log fX (X|θ)
∂θ
E [S(X|θ)] = 0
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
25 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Score Function
.
Definition: Score or Score Function for X
.
i.i.d.
X1 , · · · , Xn ∼ fX (x|θ)
∂
S(X|θ) =
log fX (X|θ)
∂θ
E [S(X|θ)] = 0
∂
Sn (X|θ) =
log fX (X|θ)
∂θ
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
25 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Fisher Information Number
.
Definition: Fisher Information Number
.
[{
}2 ]
[
]
∂
I(θ) = E
log fX (X|θ)
= E S2 (X|θ)
∂θ
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
26 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Fisher Information Number
.
Definition: Fisher Information Number
.
[{
}2 ]
[
]
∂
I(θ) = E
log fX (X|θ)
= E S2 (X|θ)
∂θ
[{
}2 ]
∂
In (θ) = E
log fX (X|θ)
∂θ
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
26 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Fisher Information Number
.
Definition: Fisher Information Number
.
[{
}2 ]
[
]
∂
I(θ) = E
log fX (X|θ)
= E S2 (X|θ)
∂θ
[{
}2 ]
∂
In (θ) = E
log fX (X|θ)
∂θ
[{
}2 ]
∂
= nE
log fX (X|θ)
= nI(θ)
∂θ
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
26 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Fisher Information Number
.
Definition: Fisher Information Number
.
[{
}2 ]
[
]
∂
I(θ) = E
log fX (X|θ)
= E S2 (X|θ)
∂θ
[{
}2 ]
∂
In (θ) = E
log fX (X|θ)
∂θ
[{
}2 ]
∂
= nE
log fX (X|θ)
= nI(θ)
∂θ
The bigger the information number, the more information we have about
θ,
. the smaller bound on the variance of unbiased estimates.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
26 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Simplified Fisher Information
.
Lemma 7.3.11
.
If fX (x|θ) satisfies the∫ two interchangeability
∫ conditions
∂
d
fX (x|θ)dx =
fX (x|θ)dx
dθ x∈X
x∈X ∂θ
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
27 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Simplified Fisher Information
.
Lemma 7.3.11
.
If fX (x|θ) satisfies the∫ two interchangeability
∫ conditions
∂
d
fX (x|θ)dx =
fX (x|θ)dx
dθ x∈X
x∈X ∂θ
∫
∫
d
∂
∂2
fX (x|θ)dx =
f (x|θ)dx
2 X
dθ x∈X ∂θ
x∈X ∂θ
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
27 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Simplified Fisher Information
.
Lemma 7.3.11
.
If fX (x|θ) satisfies the∫ two interchangeability
∫ conditions
∂
d
fX (x|θ)dx =
fX (x|θ)dx
dθ x∈X
x∈X ∂θ
∫
∫
d
∂
∂2
fX (x|θ)dx =
f (x|θ)dx
2 X
dθ x∈X ∂θ
x∈X ∂θ
which are true for
then
[{exponential family,
]
}2 ]
[ 2
∂
∂
log fX (X|θ)
I(θ) = E
log fX (X|θ)
= −E
∂θ
∂θ2
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
27 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example - Poisson Distribution
i.i.d.
• X1 , · · · , Xn ∼ Poisson(λ)
• λ1 = X
• λ2 = s2X
• E[λ1 ] = E(X) = λ.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
28 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example - Poisson Distribution
i.i.d.
• X1 , · · · , Xn ∼ Poisson(λ)
• λ1 = X
• λ2 = s2X
• E[λ1 ] = E(X) = λ.
−1
Cramer-Rao lower bound is I−1
n (λ) = [nI(λ)] .
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
28 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example - Poisson Distribution
i.i.d.
• X1 , · · · , Xn ∼ Poisson(λ)
• λ1 = X
• λ2 = s2X
• E[λ1 ] = E(X) = λ.
−1
Cramer-Rao lower bound is I−1
n (λ) = [nI(λ)] .
[{
}2 ]
[ 2
]
∂
∂
I(λ) = E
log fX (X|λ)
= −E
log fX (X|λ)
∂λ
∂λ2
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
28 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example - Poisson Distribution
i.i.d.
• X1 , · · · , Xn ∼ Poisson(λ)
• λ1 = X
• λ2 = s2X
• E[λ1 ] = E(X) = λ.
−1
Cramer-Rao lower bound is I−1
n (λ) = [nI(λ)] .
[{
}2 ]
[ 2
]
∂
∂
I(λ) = E
log fX (X|λ)
= −E
log fX (X|λ)
∂λ
∂λ2
]
[ 2
]
[ 2
e−λ λX
∂
∂
log
(−λ
+
X
log
λ
−
log
X!)
= −E
=
−E
∂λ2
X!
∂λ2
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
28 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example - Poisson Distribution
i.i.d.
• X1 , · · · , Xn ∼ Poisson(λ)
• λ1 = X
• λ2 = s2X
• E[λ1 ] = E(X) = λ.
−1
Cramer-Rao lower bound is I−1
n (λ) = [nI(λ)] .
[{
}2 ]
[ 2
]
∂
∂
I(λ) = E
log fX (X|λ)
= −E
log fX (X|λ)
∂λ
∂λ2
]
[ 2
]
[ 2
e−λ λX
∂
∂
log
(−λ
+
X
log
λ
−
log
X!)
= −E
=
−E
∂λ2
X!
∂λ2
[ ]
X
1
1
= E 2 = 2 E(X) =
λ
λ
λ
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
28 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example - Poisson Distribution (cont’d)
Therefore, the Cramer-Rao lower bound is
Var[W(X)] ≥
1
λ
=
nI(λ)
n
where W is any unbiased estimator.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
29 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example - Poisson Distribution (cont’d)
Therefore, the Cramer-Rao lower bound is
Var[W(X)] ≥
1
λ
=
nI(λ)
n
where W is any unbiased estimator.
Var(λ̂1 ) = Var(X) =
λ
Var(X)
=
n
n
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
29 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example - Poisson Distribution (cont’d)
Therefore, the Cramer-Rao lower bound is
Var[W(X)] ≥
1
λ
=
nI(λ)
n
where W is any unbiased estimator.
Var(λ̂1 ) = Var(X) =
λ
Var(X)
=
n
n
Therefore, λ1 = X is the best unbiased estimator of λ.
Var(λ̂2 ) >
λ
n
(details is omitted), so λ̂2 is not the best unbiased estimator.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
29 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
With and without Lemma 7.3.11
.
With Lemma 7.3.11
.
[ 2
]
[ 2
]
∂
1
∂
I(λ) = −E ∂λ
2 log fX (X|λ) = −E ∂λ2 (−λ + X log λ − log X!) = λ
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
30 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
With and without Lemma 7.3.11
.
With Lemma 7.3.11
.
[ 2
]
[ 2
]
∂
1
∂
I(λ) = −E ∂λ
2 log fX (X|λ) = −E ∂λ2 (−λ + X log λ − log X!) = λ
.
.
Without Lemma 7.3.11
.
[{
[{
}2 ]
}2 ]
∂
∂
I(λ) = E
log fX (X|λ)
=E
(−λ + X log λ − log X!)
∂λ
∂λ
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
30 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
With and without Lemma 7.3.11
.
With Lemma 7.3.11
.
[ 2
]
[ 2
]
∂
1
∂
I(λ) = −E ∂λ
2 log fX (X|λ) = −E ∂λ2 (−λ + X log λ − log X!) = λ
.
.
Without Lemma 7.3.11
.
[{
[{
}2 ]
}2 ]
∂
∂
I(λ) = E
log fX (X|λ)
=E
(−λ + X log λ − log X!)
∂λ
∂λ
[{
]
} ]
[
E(X) E(X2 )
X 2
X X2
= E
−1 +
=E 1−2 + 2 =1−2
+
λ
λ
λ
λ
λ2
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
30 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
With and without Lemma 7.3.11
.
With Lemma 7.3.11
.
[ 2
]
[ 2
]
∂
1
∂
I(λ) = −E ∂λ
2 log fX (X|λ) = −E ∂λ2 (−λ + X log λ − log X!) = λ
.
.
Without Lemma 7.3.11
.
[{
[{
}2 ]
}2 ]
∂
∂
I(λ) = E
log fX (X|λ)
=E
(−λ + X log λ − log X!)
∂λ
∂λ
[{
]
} ]
[
E(X) E(X2 )
X 2
X X2
= E
−1 +
=E 1−2 + 2 =1−2
+
λ
λ
λ
λ
λ2
.
= 1−2
λ λ + λ2
1
E(X) Var(X) + [E(X)]2
=
1
−
2
=
+
+
2
2
λ
λ
λ
λ
λ
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
30 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example - Normal Distribution
i.i.d.
• X1 , · · · , Xn ∼ N (µ, σ 2 ), where σ 2 is known.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
31 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example - Normal Distribution
i.i.d.
• X1 , · · · , Xn ∼ N (µ, σ 2 ), where σ 2 is known.
• The Cramer-Rao bound for µ is [nI(µ)]−1 .
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
31 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example - Normal Distribution
i.i.d.
• X1 , · · · , Xn ∼ N (µ, σ 2 ), where σ 2 is known.
• The Cramer-Rao bound for µ is [nI(µ)]−1 .
[
]
∂2
I(µ) = −E
log fX (X|µ)
∂µ2
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
31 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example - Normal Distribution
i.i.d.
• X1 , · · · , Xn ∼ N (µ, σ 2 ), where σ 2 is known.
• The Cramer-Rao bound for µ is [nI(µ)]−1 .
[
]
∂2
I(µ) = −E
log fX (X|µ)
∂µ2
[ 2
{
(
)}]
∂
1
(X − µ)2
= −E
log √
exp −
∂µ2
2σ 2
2πσ 2
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
31 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example - Normal Distribution
i.i.d.
• X1 , · · · , Xn ∼ N (µ, σ 2 ), where σ 2 is known.
• The Cramer-Rao bound for µ is [nI(µ)]−1 .
[
]
∂2
I(µ) = −E
log fX (X|µ)
∂µ2
[ 2
{
(
)}]
∂
1
(X − µ)2
= −E
log √
exp −
∂µ2
2σ 2
2πσ 2
[ 2 {
}]
∂
1
(X − µ)2
2
= −E
−
log(2πσ
)
−
∂µ2
2
2σ 2
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
31 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Example - Normal Distribution
i.i.d.
• X1 , · · · , Xn ∼ N (µ, σ 2 ), where σ 2 is known.
• The Cramer-Rao bound for µ is [nI(µ)]−1 .
[
I(µ) =
=
=
=
]
∂2
−E
log fX (X|µ)
∂µ2
[ 2
{
(
)}]
∂
1
(X − µ)2
−E
log √
exp −
∂µ2
2σ 2
2πσ 2
[ 2 {
}]
∂
1
(X − µ)2
2
−E
−
log(2πσ
)
−
∂µ2
2
2σ 2
{
}]
[
1
∂ 2(X − µ)
= 2
−E
2
∂µ
2σ
σ
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
31 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Applying Lemma 7.3.11
.
Question
.
.When can we interchange the order of differentiation and integration?
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
32 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Applying Lemma 7.3.11
.
Question
.
.When can we interchange the order of differentiation and integration?
.
Answer
.
• For exponential family, always yes.
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
32 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Applying Lemma 7.3.11
.
Question
.
.When can we interchange the order of differentiation and integration?
.
Answer
.
• For exponential family, always yes.
• Not always yes for non-exponential family. Will have to check the
.
individual case.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
32 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Applying Lemma 7.3.11
.
Question
.
.When can we interchange the order of differentiation and integration?
.
Answer
.
• For exponential family, always yes.
• Not always yes for non-exponential family. Will have to check the
.
individual case.
.
Example
.
i.i.d.
X1 , · · · , Xn ∼ Uniform(0, θ)
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
32 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Applying Lemma 7.3.11
.
Question
.
.When can we interchange the order of differentiation and integration?
.
Answer
.
• For exponential family, always yes.
• Not always yes for non-exponential family. Will have to check the
.
individual case.
.
Example
.
i.i.d.
X1 , · · · , Xn ∼ Uniform(0, θ)
∫ θ
∫ θ
d
∂
h(x)fX (x|θ)dx ̸=
h(x) fX (x|θ)dx
dθ 0
∂θ
0
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
32 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Summary
.
Today
.
• Invariance Property
• Mean Squared Error
• Unbiased Estimator
.
• Cramer-Rao inequality
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
33 / 33
Recap
.....
MLE
....
Evaluation
.....
Cramer-Rao
.................
Summary
.
Summary
.
Today
.
• Invariance Property
• Mean Squared Error
• Unbiased Estimator
.
• Cramer-Rao inequality
.
Next Lecture
.
• More on Cramer-Rao inequality
.
.
Hyun Min Kang
Biostatistics 602 - Lecture 11
.
.
.
.
February 14th, 2013
.
33 / 33
0
You can add this document to your study collection(s)
Sign in Available only to authorized usersYou can add this document to your saved list
Sign in Available only to authorized users(For complaints, use another form )