1
4.3
Sampling From A Multivariate Normal Distribution
And Maximum Likelihood Estimation
(a) The Multivariate Normal Likelihood
Let
X 1 , X 2 ,, X n ~ N  , 
be a random sample from a
multivariate normal population. Then, the joint density function of
X 1 , X 2 ,, X n

  x j   t  1 x j    
1

f x1 ,, x n    
exp 

p
1
2
j 1  2  2  2

 


n

1
2 
np
2

n
2
 n

t 1
   x j     x j   
j 1

exp 


2




.
The following results will be used to obtain the maximum likelihood
estimate of  and  .
Result:
Let A be a
p p
Then,
(a)

symmetric matrix and x be a


x t Ax  tr x t Ax  tr Axxt
,
where
p 1
 
A  a ij
p
tr A   aii .
i 1
(b) tr  A 
p
  , where
i 1
i
Based on the result, we have
i
vector.
are the eigenvalues of A.
and
2
 x
n
j 1
  
t
j
1
x


     tr x j     1 x j   
n
j
j 1
t

  tr  1 x j   x j   
n
j 1
t

 1  n

t 
2A.12, p. 98 tr    x j   x j    
  j 1

Then,
 x
n
j 1
  x j      x j  x  x    x j  x  x   
t
j
n
t
t
j 1
  x j  x x j  x     x    x   
n
t
j 1
n
t
j 1
  x j  x x j  x   n x    x   
n
t
t
j 1
n
where x 
x
i 1
i
n
. Further,
 1  n

t
t 
tr    x j  x x j  x   n x    x    

  j 1

 1  n

t 
t
 tr    x j  x x j  x    ntr  1  x    x   
  j 1

 1  n

t 
t


 tr    x j  x x j  x    n x     1  x   

  j 1
Therefore, the likelihood function of
simplified to
X 1 , X 2 ,, X n

can be
3
L ,    f x1 ,  , x n  

1
2 
1
2 
np
2

n
2
np
2

n
2
 n

t 1
   x j     x j   
j 1

exp 


2




  1  n
.

t 
t 1









tr

x

x
x

x

n
x



x


   j

j

   j 1


exp 

2






(b) Maximum Likelihood Estimation of  and 
To obtain the maximum likelihood estimate, the following result will
be used.
Result:
Given a
p p
symmetric positive definite matrix B and a scalar
b  0 , it follows that

  tr  1 B
exp 
b
2


1
 


1
B
b
2b  pb exp  bp 
 
for all positive definite  , with equality holding only for   1 2b B .
Import Result (MLE of
Let
 and  )
X 1 , X 2 ,, X n ~ N  , 
be a random sample from a
multivariate normal population. Then,
 X
n
̂  X
ˆ
and  
j 1
 X X j  X 
t
j
n

n  1S
n
are the maximum likelihood estimators of  and  , respectively,
where
4
 X
n
S
j 1
 X X j  X 
t
j
n 1
is a unbiased estimate of  . Their observed values,
 n x
n
1
j 1
x
and
 x x j  x  , are called the maximum likelihood
t
j
estimates of  and  .
[proof:]
̂
maximizing the function
1
2 
np
2

n
2
  1  n


t 
t
 tr    x j  x x j  x    nx     1 x   
   j 1


exp  

2






also minimizes the function
 1  n

t 
t
tr    x j  x x j  x    nx     1 x    .
  j 1

Since
as
 1
is positive definite, so that
nx     1 x     0 . However,
t
  x , the function
 1  n

t
t
tr    x j  x x j  x    nx     1 x   

  j 1
 1  n

t
 tr    x j  x x j  x  

  j 1
achieves its minimum. It remains to find
̂
maximizing
5
Lˆ ,   
1
2 
np
2

n
2
  1  n

t 
 tr    x j  x x j  x   
   j 1
 
exp  

2






By the previous result with b  n 2 and B 
 x
j 1
j 1
 x x j  x  ,
t
j
 x x j  x 
n
ˆ
the maximum occurs at  
 x
n
t
j
.
n
Note:
Maximum likelihood estimators possess an invariance property. Let
ˆ be the maximum likelihood estimator of  , and consider
estimating the parameter h  , which is a function of  , the

maximum likelihood estimate of h  is given by h ˆ .
For
examples,
1. The maximum likelihood estimator of  t  1  is ˆ t ˆ 1 ˆ .
2. The maximum likelihood estimator of
 X
n
̂ ii 
j 1
 ii is
̂ ii , where
 Xi 
2
ij
n
.
Note:
Let
X 1 , X 2 ,, X n ~ N  , 
be a random sample from a
multivariate normal population. Then,
6
 X
n
̂  X
ˆ
and  
j 1
 X X j  X 
t
j
n
are sufficient statistics.
4.4
The Sampling Distribution of
X
and
S
Definition of the Wishart Distribution:
Let
Z1 , Z 2 ,, Z n ~ N p 0,  
be independently distributed. Then,
n
M   Z j Z tj is distributed as a Wishart
the random matrix
j 1
distribution with n d.f., Wn , . The density of a Wishart
distribution with n d.f. is
( n  p 1)
m
f m |   
pn
2
2

  trm 1 
exp 

2


p
n
1

 2    n  1  i 
2

i 1
2
p ( p 1)
4
Properties of the Wishart Distribution:
1. If
M 1 ~ Wn1 ,  and M 2 ~ Wn2 ,  , then
M 1  M 2 ~ Wn1  n2 ,  .
2. If
M 1 ~ Wn1 ,  , then CM1C t ~ Wn1 , CC t .
Import Result:
 
X
~
N
.
p  ,
1.
n


.
7
2.
n  1S   X j  X X j  X t ~ Wn 1, .
3.
X and n  1S are independent.
n
j 1
4.5
Large-Sample Behavior of
X
and
S
Import Result:
Let
X 1 , X 2 ,, X n
be a random sample from a population with
mean  and finite (nonsingular) covariance  . Then,
n X     N p 0,  
and
n X    S 1  X      p2
t
for n  p large.