8.4
Lecture 11 Friday 02/09/01
Homework and Labs. see the logictics section
Please hand in your labs to Johan by next Monday.
9
Maximum Likelihood Estimation
X1 , X2 , X3 , . . . Xn have joint density denoted
fθ (x1 , x2 , . . . , xn ) = f (x1 , x2 , . . . , xn |θ)
Given observed values X1 = x1 , X2 = x2 , . . . , Xn = xn , the likelihood of θ is the function
lik(θ) = f (x1 , x2 , . . . , xn |θ)
considered as a function of θ.
If the distribution is discrete, f will be the frequency distribution function.
In words: lik(θ)=probability of observing the given data as a function of θ.
Definition:
The maximum likelihood estimate (mle) of θ is that value of θ that maximises lik(θ): it is
the value that makes the observed data the “most probable”.
If the Xi are iid, then the likelihood simplifies to
lik(θ) =
n
Y
f (xi |θ)
i=1
Rather than maximising this product which can be quite tedious, we often use the fact
that the logarithm is an increasing function so it will be equivalent to maximise the log
likelihood:
n
l(θ) =
X
log(f (xi |θ))
i=1
9.0.1
Poisson Example
λx e−λ
x!
For X1 , X2 , . . . , Xn iid Poisson random variables will have a joint frequency function that is
a product of the marginal frequency functions, the log likelihood will thus be:
P (X = x) =
l(λ)
= ni=1 (Xi logλ − λ − logXi !)
P
P
= logλ ni=1 Xi − nλ − ni=1 logXi !
P
We need to find the maximum by finding the derivative:
l0 (λ) =
n
1X
xi − n = 0
λ i=1
1
which implies that the estimate should be
λ̂ = X̄
(as long as we check that the function l is actually concave, which it is).
The mle agrees with the method of moments in this case, so does its sampling distribution.
9.0.2
Normal Example
If X1 , X2 , . . . , Xn are iid N (µ, σ 2 ) random variables their density is written:
f (x1 , . . . , xn |µ, σ) =
n
Y
i
1 x −µ 2
1
√ exp (− [ i
])
2
σ
σ 2π
Regarded as a function of the two parameters, µ and σ this is the likelihood:
n
n
1 X
`(µ, σ) = −nlogσ − log2π − 2
(xi − µ)2
2
2σ i=1
n
∂`
1 X
=
(xi − µ)
∂µ
σ 2 i=1
∂`
P
= − nσ + σ −3 ni=1 (xi − µ)2
∂σ
so setting these to zero gives X̄ as the mle for µ, and σ̂ 2 as the usual.
9.0.3
Gamma Example
f (x|α, λ) =
1 α α−1 −λx
λ x e
Γ(α)
giving the log-likelihood:
l(x|α, λ) =
n
X
[αlogλ + (α − 1)logxi − λxi − logΓ(α)]
i=1
One ends up with a nonlinear equation in α̂ this cannot be solved in closed form, there
are basically two methods and they are called root-finding methods, they are based on the
calculus theorem that says that when a function is continuous, and changes signs on an
interval, it is zero on that interval.
For this particular problem there already coded in matlab a mle method called gamfit,
that also provides a confidence interval.
For general optimization, the function in Matlab is fmin for one variable, and fmins
you could also look at how to use optimize in Splus.
2
9.1
Maximum Likelihood of Multinomial Cell Probabilities
X1 , X2 , . . . , Xm are counts in cells/ boxes 1 up to m, each box has a different probability
(think of the boxes being bigger or smaller) and we fix the number of balls that fall to
be n:x1 + x2 + · · · + xm = n. The probability of each box is pi , with also a constraint:
p1 + p2 + · · · + pm = 1, this is a case in which the Xi0 s are NOT independent, the joint
probability of a vector x1 , x2 , . . . , xm is called the multinomial ,and has the form:
!
n
n! Y xi
f (x1 , x2 , . . . , xm |p1 , . . . , pm ) = Q
pi =
px1 px1 · · · pxmm
xi !
x 1 , x2 , x m 1 1
Each box taken separately against all the other boxes is a binomial, this is an extension
thereof. (look at page 72)
We study the log-likelihood of this :
l(p1 , p2 , ..., pm ) = logn! −
m
X
i=1
logxi ! +
m
X
xi logpi
i=1
However we can’t just go ahead and maximise this we have to take the constraint into account
so we have to use the Lagrange multipliers again.
We use
m
L(p1 , p2 , ..., pm , λ) = l(p1 , p2 , ..., pm ) + λ(1 −
X
pi )
i
By posing all the derivatives to be 0, we get the most natural estimate
p̂i =
xi
n
Maximising log likelihood, with and without constraints, can be an unsolvable problem
in closed form, then we have to use iterative procedures.
I explained about how the parametris bootstrap was often the only way to study the
sampling distribution of the mle.
3