Fisher Information
Stat 543 Spring 2005
Definition 1 We will say that the model for X specified by f (x|θ) (either a probability mass function or a
probability density) is FI (Fisher Information) Regular at θ0 ∈ Θ ⊂ Rk provided there is an open neighborhood
of θ0 , say O, such that
i) f (x|θ) > 0 ∀x and ∀θ ∈ O,
ii) ∀x, f (x|θ) has first¯ order partials at θ0 , and¯
R
P
¯
¯
iii) 0 = x ∂θ∂ i f (x|θ)¯
(or 0 = ∂θ∂ i f (x|θ)¯
dx) .
θ=θ0
θ=θ0
Definition 2 If the model for X is FI regular at θ0 and
Eθ0
then the k × k matrix
I(θ0 ) =
Ã
Ã
!2
¯
¯
∂
¯
ln f (X|θ)¯
< ∞ ∀i ,
∂θi
θ=θ0
!
¯
¯
¯
¯
∂
∂
Eθ0
ln f (X|θ)¯¯
ln f (X|θ)¯¯
∂θi
θ=θ0 ∂θ j
θ=θ0
is called the Fisher Information Matrix at θ0 .
Theorem 3 Suppose the model for X is FI Regular at θ0 ∈ Θ ⊂ Rk and that I(θ0 ) exists and is nonsingular.
If ∀x f (x|θ) has continuous second order partials in the neighborhood O, and ∀i, j
0=
or
0=
then
Z
¯
¯
X ∂
X ∂2
¯
¯
f (x|θ)¯
and 0 =
f (x|θ)¯
in the discrete case
∂θ
∂θ
∂θ
θ=θ
θ=θ0
0
i
i
j
x
x
Z
¯
¯
∂
∂2
¯
¯
f (x|θ)¯
dx and 0 =
f (x|θ)¯
dx in the continuous case,
∂θi
∂θi ∂θj
θ=θ0
θ=θ0
I(θ0 ) = −Eθ0
Ã
!
¯
¯
∂2
¯
ln f (X|θ)¯
∂θi ∂θj
θ=θ0
.
Notation 4 In an iid model, i.e. where X = (X1 , ..., Xn ), let I1 (θ0 ) be the Fisher Information Matrix at
θ0 for a single observation. That is, with marginal distribution specified by f (x|θ),
Ã
!
¯
¯
¯
¯
∂
∂
I1 (θ0 ) = Eθ0
ln f (X1 |θ)¯¯
ln f (X1 |θ)¯¯
.
∂θi
θ=θ0 ∂θ j
θ=θ0
Theorem 5 Suppose that X1 , X2 , ..., Xn are iid random vectors and f (x|θ) is either the marginal probability
mass function or the marginal probability density. If the model for X1 is FI regular at θ0 and with X =
(X1 , ..., Xn ), I(θ0 ) is the Fisher Information Matrix for X at θ0 ,
I(θ0 ) = nI1 (θ0 ) .
1