STATISTICAL INFERENCE
PART III
EXPONENTIAL FAMILY, LOCATION
AND SCALE PARAMETERS
1
EXPONENTIAL CLASS OF PDFS
• X is a continuous (discrete) rv with pdf f(x;),
. If the pdf can be written in the
following form
k
f ( x; )  h ( x )c( ) exp( w j ( ) t j ( x ))
j1
then, the pdf is a member of exponential
class of pdfs of the continuous (discrete)
type. (Here, k is the number of parameters)
2
REGULAR CASE OF THE EXPONENTIAL
CLASS OF PDFS
•
We have a regular case of the exponential class of
pdfs of the continuous type if
a) Range of X does not depend on .
b) c() ≥ 0, w1(),…, wk() are real valued functions of
 for .
c) h(x) ≥ 0, t1(x),…, tk(x) are real valued functions of x.
If the range of X depends on , then it is called
irregular exponential class or range-dependent
exponential class.
3
EXAMPLES
X~Bin(n,p), where n is known. Is this pdf a
member of exponential class of pdfs? Why?
n x
f ( x; p)   p (1  p) n  x ; x  0,1,...,n; 0  p  1
x
n
p
  (1  p) n exp(x ln(
))
1 p
x
n
h ( x )   
x
t1 ( x )  x
for x  0,...,n; c(p)  (1  p) n
for x  0,...,n;
p
w1 (p)  ln(
)
1 p
for 0  p  1
for 0  p  1
Binomial family is a member of exponential family of distributions.
4
EXAMPLES
X~Cauchy(1,). Is this pdf a member of
exponential class of pdfs? Why?
f ( x; )  ( (1  [ x   ]2 ))1 
h(x) 
1

1

exp{ ln(1  x 2  2x   2 )}
; c( )  1;  ln(1  x 2  2x   2 )  t1 ( x ) w1 ( )
Cauchy is not a member of exponential family.
5
REGULAR CASE OF THE EXPONENTIAL
CLASS OF PDFS
• Random Sample from Regular Exponential Class
Y
n
 t j (X i ) is a css for .
i 1
If Y is an UE of , Y is the MVUE of .
6
EXAMPLES
Recall: X~Bin(n,p), where n is known.
This family is a member of exponential family of
distributions.
t1 ( x )  x
n
for x  0,...,n
n
 t1( x i )   x i
i 1
i 1
is a CSS for p.
is UE of p.
is MVUE of p.
7
EXAMPLES
X~N(,2) where both  and 2 is unknown.
Find a css for  and 2 .
8
FISHER INFORMATION AND
INFORMATION CRITERIA
• X, f(x;), , xA (not depend on ).
Definitions
and
notations:
  x;   ln f  x; 
 ln f  x; 
  x;  

 2 ln f  x; 
  x;  
 2
f  x; 
f  x;  

 2 f  x; 
f  x;  
 2
9
FISHER INFORMATION AND
INFORMATION CRITERIA
The Fisher Information in a random variable X:
I    E  x; 2  V   x;    E  x;   0
The Fisher Information in the random sample:
I n    nI  
Let’s prove the equalities above.
10
FISHER INFORMATION AND
INFORMATION CRITERIA
d
 f  x; dx  1  d  f  x; dx  0
A
A
  f  x; dx  0
A
  f  x; dx  0
A
 ln f  x;  f  x; 
  x;  


f  x; 
f  x; 
2



  x;  
   x; 
f  x; 
11
FISHER INFORMATION AND
INFORMATION CRITERIA
E  X ;      x;  f  x; dx  0
A
E  X ;      x;  f  x; dx
A
 f  x; 
2

 
   x;   f  x; dx

A  f  x; 
2



  f  x; dx     x;  f  x; dx
A
A
2
 0  E  x; 
12
FISHER INFORMATION AND
INFORMATION CRITERIA
2



 E  x;    E  x;   V   x; 
The Fisher Information in a random variable X:
2

I    E  x;   V   x;    E  x;   0
The Fisher Information in the random sample:
I n    nI  
Proof of the last equality is available on Casella & Berger (1990), pg.
310-311.
13
CRAMER-RAO LOWER BOUND (CRLB)
•
•
•
•
•
•
Let X1,X2,…,Xn be sample random variables.
Range of X does not depend on .
Y=U(X1,X2,…,Xn): a statistic not containing .
Let E(Y)=m().
Z=′(x1,x2,…,xn;) is a r.v.
E(Z)=0 and V(Z)=In() (from previous slides).

m 2
V Y  
 The Cramer - Rao Lower Bound
I n  
• Let prove this!
14
CRAMER-RAO LOWER BOUND (CRLB)
• Cov(Y,Z)=E(YZ)-E(Y)E(Z)=E(YZ)
EY.Z     u x1, x 2 ,, x n   x1, x 2 ,, x n ; f ( x1, x n ; )dx1dx 2  dx n
f  x1,, xn ; 
    u  x1,, xn 
f  x1,, xn ; dx1 dxn
f  x1,, xn ; 
  u x1,, xn  f  x1,, xn ; dx1dxn  m 
15
CRAMER-RAO LOWER BOUND (CRLB)
• E(Y.Z)=m’()
CovY , Z 
1
• -1Corr(Y,Z)1  1 
V Y  V Z 
2




Cov
Y
,
Z
• 0 Corr(Y,Z)21  0 
1
V Y V Z 

The Cramer-Rao Inequality
(Information Inequality)
2


m  
0
1
V Y I n  

m 2
V Y  
 The Cramer - Rao Lower Bound
I n  
16
CRAMER-RAO LOWER BOUND (CRLB)
• CRLB is the lower bound for the variance of
the unbiased estimator of m().
• When V(Y)=CRLB, Y is the MVUE of m().
• For a r.s., remember that In()=n I(), so,

m 2
V Y  
 The Cramer - Rao Lower Bound
nI  
17
EFFICIENT ESTIMATOR
• T is an efficient estimator (EE) of  if
– T is UE of , and,
– Var(T)=CRLB
• Y is an efficient estimator (EE) of its
expectation, m(), if its variance reaches the
CRLB.
• An EE of m() may not exist.
• The EE of m(), if exists, is unique.
• The EE of m() is the unique MVUE of m().
18
ASYMPTOTIC EFFICIENT ESTIMATOR
• Y is an asymptotic EE of m() if
lim E Y   m 
n 
and
lim V Y   CRLB
n 
19
EXAMPLES
A r.s. of size n from X~Poi(θ).
a) Find CRLB for any UE of θ.
b) Find UMVUE of θ.
c) Find an EE for θ.
d) Find CRLB for any UE of exp{-2θ}. Assume
n=1, and show that (1) x is UMVUE of exp{2θ}. Is this a reasonable estimator?
20
EXAMPLE
A r.s. of size n from X~Exp(). Find UMVUE of ,
if exists.
21
LIMITING DISTRIBUTION OF MLEs
• ˆ : MLE of 
• X1,X2,…,Xn is a random sample.
2

asymptotically
 m    

m ˆ
~
N  m   , RCLB 
m 
nI   






1
ˆ ~ N   ,

large n
 nI   
asympt .

ˆ  
1
nI  


d
ˆ
 nI      
 N  0,1
22
LIMITING DISTRIBUTION OF MLEs
• Let ˆ1 ,ˆ2 ,
,ˆm
be MLEs of 1, 2,…, m.
asympt .

ˆi ~ N i , RCLB
 
i

~ N  m i  , RCLB  , i  1,2,..., m
m  

• EE of m()= m ˆ  fn  ss for  
m ˆi
asympt .

i
•If Y is an EE of , then Z=a+bY is an EE
of a+bm() where a and b are constants.
23
LOCATION PARAMETER
• Let f(x) be any pdf. The family of pdfs f(x)
indexed by parameter  is called the location
family with standard pdf f(x) and  is the
location parameter for the family.
• Equivalently,  is a location parameter for f(x)
iff the distribution of X does not depend on
.
24
Example
• If X~N(θ,1), then X-θ~N(0,1)  distribution is
independent of θ.  θ is a location parameter.
• If X~N(0,θ), then X-θ~N(-θ,θ)  distribution is
NOT independent of θ.  θ is NOT a location
parameter.
25
LOCATION PARAMETER
• Let X1,X2,…,Xn be a r.s. of a distribution with
pdf (or pmf); f(x; ); . An estimator
t(x1,…,xn) is defined to be a location
equivariant iff
t(x1+c,…,xn+c)= t(x1,…,xn) +c
for all values of x1,…,xn and a constant c.
• t(x1,…,xn) is location invariant iff
t(x1+c,…,xn+c)= t(x1,…,xn)
for all values of x1,…,xn and a constant c.
Invariant = does not change
26
Example
• Is X location invariant or equivariant
estimator?
• Let t(x1,…,xn) = X . Then,
t(x1+c,…,xn+c)= (x1+c+…+xn+c)/n =
(x1+…+xn+nc)/n = X +c = t(x1,…,xn) +c
 location equivariant
27
Example
• Is s² location invariant or equivariant
n
n
xi 2
estimator?
 ( x i   ( n ))
i 1
• Let t(x1,…,xn) = s²= i 1
n 1
• Then,
n
n
xi  c 2
 ( x i  c   ( n ))
i 1
i 1
t(x1+c,…,xn+c)=
n 1
=t(x1,…,xn) Location invariant
 s2
(x1,…,xn) and (x1+c,…,xn+c) are located at different points
on real line, but spread among the sample values is same for
both samples.
28
SCALE PARAMETER
• Let f(x) be any pdf. The family of pdfs f(x/)/
for >0, indexed by parameter , is called the
scale family with standard pdf f(x) and  is the
scale parameter for the family.
• Equivalently,  is a scale parameter for f(x) iff
the distribution of X/ does not depend on .
29
Example
•
•
•
•
Let X~Exp(θ). Let Y=X/θ.
You can show that f(y)=exp(-y) for y>0
Distribution is free of θ
θ is scale parameter.
30
SCALE PARAMETER
• Let X1,X2,…,Xn be a r.s. of a distribution with
pdf (or pmf); f(x; ); . An estimator
t(x1,…,xn) is defined to be a scale equivariant
iff
t(cx1,…,cxn)= ct(x1,…,xn)
for all values of x1,…,xn and a constant c>0.
• t(x1,…,xn) is scale invariant iff
t(cx1,…,cxn)= t(x1,…,xn)
for all values of x1,…,xn and a constant c>0.
31
Example
• Is X scale invariant or equivariant estimator?
• Let t(x1,…,xn) = X. Then,
t(cx1,…,cxn)= c(x1+…+xn)/n = c X= ct(x1,…,xn)
 Scale equivariant
32
LOATION-SCALE PARAMETER
• Let f(x) be any pdf. The family of pdfs
f((x) /)/ for >0, indexed by parameter
(,), is called the location-scale family with
standard pdf f(x) and  is a location parameter
and  is the scale parameter for the family.
• Equivalently,  is a location parameter and  is
a scale parameter for f(x) iff the distribution of
(X)/ does not depend on  and.
33
Example
1. X~N(μ,σ²). Then, Y=(X- μ)/σ ~ N(0,1)
Distribution is independent of μ and σ²
μ and σ² are location-scale paramaters
2. X~Cauchy(θ,β). You can show that the p.d.f. of
Y=(X- β)/ θ is f(y) = 1/(π(1+y²))
 β and θ are location-and-scale parameters.
34
LOCATION-SCALE PARAMETER
• Let X1,X2,…,Xn be a r.s. of a distribution with
pdf (or pmf); f(x; ); . An estimator
t(x1,…,xn) is defined to be a location-scale
equivariant iff
t(cx1+d,…,cxn+d)= ct(x1,…,xn)+d
for all values of x1,…,xn and a constant c>0.
• t(x1,…,xn) is location-scale invariant iff
t(cx1+d,…,cxn+d)= t(x1,…,xn)
for all values of x1,…,xn and a constant c>0.
35