STATISTICAL INFERENCE
PART III
EXPONENTIAL FAMILY, FISHER
INFORMATION, CRAMER-RAO
LOWER BOUND (CRLB)
1
EXPONENTIAL CLASS OF PDFS
• X is a continuous (discrete) rv with pdf f(x;),
. If the pdf can be written in the
following form
k
f ( x; )  h ( x )c( ) exp( w j ( ) t j ( x ))
j1
then, the pdf is a member of exponential
class of pdfs of the continuous (discrete)
type. (Here, k is the number of parameters)
2
REGULAR CASE OF THE EXPONENTIAL
CLASS OF PDFS
•
We have a regular case of the exponential class of
pdfs of the continuous type if
a) Range of X does not depend on .
b) c() ≥ 0, w1(),…, wk() are real valued functions of
 for .
c) h(x) ≥ 0, t1(x),…, tk(x) are real valued functions of x.
If the range of X depends on , then it is called
irregular exponential class or range-dependent
exponential class.
3
EXAMPLES
X~Bin(n,p), where n is known. Is this pdf a
member of exponential class of pdfs? Why?
n x
f ( x; p)   p (1  p) n  x ; x  0,1,...,n; 0  p  1
x
n
p
  (1  p) n exp(x ln(
))
1 p
x
n
h ( x )   
x
t1 ( x )  x
for x  0,...,n; c(p)  (1  p) n
for x  0,...,n;
p
w1 (p)  ln(
)
1 p
for 0  p  1
for 0  p  1
Binomial family is a member of exponential family of distributions.
4
EXAMPLES
X~Cauchy(1,). Is this pdf a member of
exponential class of pdfs? Why?
f ( x; )  ( (1  [ x   ]2 ))1 
h(x) 
1

1

exp{ ln(1  x 2  2x   2 )}
; c( )  1;  ln(1  x 2  2x   2 )  t1 ( x ) w1 ( )
Cauchy is not a member of exponential family.
5
EXPONENTIAL CLASS and CSS
• Random Sample from Regular Exponential Class
Y
n
 t j (X i ) is a css for .
i 1
If Y is an UE of , Y is the UMVUE of .
6
EXAMPLES
Let X1,X2,…~Bin(1,p), i.e., Ber(p).
This family is a member of exponential family of
distributions.
t1 ( x )  x
n
for x  0,...,n
n
 t1( x i )   x i
i 1
is a CSS for p.
i 1
X
X
is UE of p and a function of CSS.
is UMVUE of p.
7
EXAMPLES
X~N(,2) where both  and 2 is unknown.
Find a css for  and 2 .
8
FISHER INFORMATION AND
INFORMATION CRITERIA
• X, f(x;), , xA (not depend on ).
Definitions
and
notations:
  x;   ln f  x; 
 ln f  x; 
  x;  

 2 ln f  x; 
  x;  
 2
f  x; 
f  x;  

 2 f  x; 
f  x;  
 2
9
FISHER INFORMATION AND
INFORMATION CRITERIA
The Fisher Information in a random variable X:
I    E  x; 2  V   x;    E  x;   0
The Fisher Information in the random sample:
I n    nI  
Let’s prove the equalities above.
10
FISHER INFORMATION AND
INFORMATION CRITERIA
d
 f  x; dx  1  d  f  x; dx  0
A
A
  f  x; dx  0
A
  f  x; dx  0
A
 ln f  x;  f  x; 
  x;  


f  x; 
f  x; 
2



  x;  
   x; 
f  x; 
11
FISHER INFORMATION AND
INFORMATION CRITERIA
E  X ;      x;  f  x; dx  0
A
E  X ;      x;  f  x; dx
A
 f  x; 
2

 
   x;   f  x; dx

A  f  x; 
2



  f  x; dx     x;  f  x; dx
A
A
2
 0  E  x; 
12
FISHER INFORMATION AND
INFORMATION CRITERIA
2



 E  x;    E  x;   V   x; 
The Fisher Information in a random variable X:
2

I    E  x;   V   x;    E  x;   0
The Fisher Information in the random sample:
I n    nI  
Proof of the last equality is available on Casella & Berger (1990), pg.
310-311.
13
CRAMER-RAO LOWER BOUND (CRLB)
•
•
•
•
Let X1,X2,…,Xn be sample random variables.
Range of X does not depend on .
Y=U(X1,X2,…,Xn): a statistic; does’nt contain .
Let E(Y)=m().
2


m  
V Y  
 The Cramer - Rao Lower Bound
I n  
• Let prove this!
14
CRAMER-RAO LOWER BOUND (CRLB)
CovY , Z 
1
• -1Corr(Y,Z)1  1 
V Y  V Z 
2




Cov
Y
,
Z
• 0 Corr(Y,Z)21  0 
1
V Y V Z 


CovY , Z 
0
V Z 
2
 V (Y )
• Take Z=′(x1,x2,…,xn;)
• Then, E(Z)=0 and V(Z)=In() (from previous slides).
15
CRAMER-RAO LOWER BOUND (CRLB)
• Cov(Y,Z)=E(YZ)-E(Y)E(Z)=E(YZ)
EY.Z     u x1, x 2 ,, x n   x1, x 2 ,, x n ; f ( x1, x n ; )dx1dx 2  dx n
f  x1,, xn ; 
    u  x1,, xn 
f  x1,, xn ; dx1 dxn
f  x1,, xn ; 
  u x1,, xn  f  x1,, xn ; dx1dxn  m 
16
CRAMER-RAO LOWER BOUND (CRLB)
• E(Y.Z)=mʹ(), Cov(Y,Z)=mʹ(), V(Z)=In()

CovY , Z 
0
V Z 
2
 V (Y )
The Cramer-Rao Inequality
(Information Inequality)

m 2
V Y  
 The Cramer - Rao Lower Bound
I n  
17
CRAMER-RAO LOWER BOUND (CRLB)
• CRLB is the lower bound for the variance of
an unbiased estimator of m().
• When V(Y)=CRLB, Y is the MVUE of m().
• For a r.s., remember that In()=n I(), so,

m 2
V Y  
 The Cramer - Rao Lower Bound
nI  
18
LIMITING DISTRIBUTION OF MLEs
• ˆ : MLE of 
• X1,X2,…,Xn is a random sample.
2

asymptotically
 m    

m ˆ
~
N  m   , RCLB 
m 
nI   






1
ˆ ~ N   ,

large n
 nI   
asympt .

ˆ  
1
nI  


d
ˆ
 nI      
 N  0,1
19
EFFICIENT ESTIMATOR
• T is an efficient estimator (EE) of  if
– T is UE of , and,
– Var(T)=CRLB
• Y is an efficient estimator (EE) of its
expectation, m(), if its variance reaches the
CRLB.
• An EE of m() may not exist.
• The EE of m(), if exists, is unique.
• The EE of m() is the unique MVUE of m().
20
ASYMPTOTIC EFFICIENT ESTIMATOR
• Y is an asymptotic EE of m() if
lim E Y   m 
n 
and
lim V Y   CRLB
n 
21
EXAMPLES
A r.s. of size n from X~Poi(θ).
a) Find CRLB for any UE of θ.
b) Find UMVUE of θ.
c) Find an EE for θ.
d) Find CRLB for any UE of exp{-2θ}. Assume
n=1, and show that (1) x is UMVUE of exp{2θ}. Is this a reasonable estimator?
22
EXAMPLE
A r.s. of size n from X~Exp(). Find UMVUE of ,
if exists.
23
Summary
• We covered 3 methods for finding UMVUE:
– Rao-Blackwell Theorem (Use a ss T, an UE U, and
create a new statistic by E(U|T))
– Lehmann-Scheffe Theorem (Use a css T which is
also UE)
– Cramer-Rao Lower Bound (Find an UE with
variance=CRLB)
24