Suppose that X is a discrete random variable with: P(X=0) = (2/3)theta
P(X=1) = (1/3)theta P(X=2) = (2/3)(1-theta) P(X=3) = (1/3)(1-theta)
where theta: [0,1] is a parameter. The following 10 independent
observations were taken from such a distribution: (3,0,2,1,3,2,1,0,2,1) a.
Find the moment of methods estimate of theta b. Find an approximate
standard error for your estimate c. What is the maximum likelihood
estimate of theta? d. What is an approximate standard error of the
maximum likelihood estimate?
𝑃(𝑋 = 0) =
2
𝜃
3
𝑃(𝑋 = 1) =
1
𝜃
3
𝑃(𝑋 = 2) =
2
(1 − 𝜃)
3
1
(1 − 𝜃)
3
𝐸(𝑋) = 0 × 𝑃(𝑋 = 0) + 1 × 𝑃(𝑋 = 1) + 2 × 𝑃(𝑋 = 2) + 3 × 𝑃(𝑋 = 3)
𝑃(𝑋 = 3) =
2
1
2
1
= 0 × 𝜃 + 1 × 𝜃 + 2 × (1 − 𝜃) + 3 × (1 − 𝜃)
3
3
3
3
= 𝜃(1/3 − 4/3 − 3/3) + 4/3 + 3/3 = −2𝜃 + 7/3
𝐸 (𝑋 2 ) = 02 × 𝑃(𝑋 = 0) + 12 × 𝑃(𝑋 = 1) + 22 × 𝑃(𝑋 = 2) + 32 × 𝑃(𝑋 = 3)
2
1
2
1
= 0 × 𝜃 + 1 × 𝜃 + 4 × (1 − 𝜃) + 9 × (1 − 𝜃)
3
3
3
3
16
17
= 𝜃(1/3 − 8/3 − 9/3) + 8/3 + 9/3 = − 𝜃 +
3
3
16
17
2
2
2
𝜎 = 𝑉𝑎𝑟(𝑋) = 𝐸 (𝑋 ) − [𝐸(𝑋)] = − 𝜃 +
− (−2𝜃 + 7/3)2
3
3
16
17
28
49
2
2
=− 𝜃+
− (4𝜃 − 𝜃 + ) = −4𝜃 2 + 4𝜃 +
3
3
3
9
9
3,0,2,1,3,2,1,0,2,1
Sample size = 𝑛 = 10
3+0+2+1+3+2+1+0+2+1
15
3
Sample mean 𝑥̅ =
= =
10
10
2
To find moment estimate of 𝜃, equate population mean and sample
mean.
𝐸(𝑋) = 𝑥̅
−2𝜃 + 7/3 = 3/2
7 3 14 − 9
2𝜃 = − =
= 5/6
3 2
6
5
𝜃=
12
5
The moment estimate of 𝜃 is 𝜃̂ =
12
7
In general −2𝜃 + = 𝑥̅
3
7
2𝜃 = − 𝑥̅
3
7 1
𝜃̂ = − 𝑥̅
6 2
3
When 𝑥̅ = ,
2
𝜃̂ =
The variance of 𝜃̂ is
7 3
5
− =
6 4 12
7 1
1 2
1 𝜎2
̂
(
)
𝑉𝑎𝑟(𝜃 ) = 𝑣𝑎𝑟( − 𝑥̅ ) =
𝑉𝑎𝑟(𝑥̅ ) = ×
6 2
2
4 𝑛
1
2
(−4𝜃 2 + 4𝜃 + )
=
40
9
Substituting 𝜃 by its moment estimate 𝜃̂,
The estimated variance of 𝜃̂ is
1
2
1
25
5 2
(−4𝜃̂ 2 + 4𝜃̂ + ) =
(−4 ×
+4×
+ )
40
9
40
144
12 9
1 −100 + 240 + 32
1 172
43
(
)=
(
)=
=
= 0.029861
40
144
40 144
1440
̂ (𝜃̂) =
𝑉𝑎𝑟
The estimated standard error of 𝜃̂ is √0.029861 = 0.1728
Maximum likelihood estimation
The likelihood function is
𝑓2
𝑓3
2 𝑓0 1 𝑓1 2
1
𝐿(𝜃/𝑥1 , … , 𝑥𝑛 ) = 𝑃 (𝑋 = 𝑥1 ) … 𝑃(𝑋 = 𝑥𝑛 ) = ( 𝜃) ( 𝜃) ( (1 − 𝜃)) ( (1 − 𝜃))
3
3
3
3
𝑓0+𝑓1 (
𝑓2+𝑓3
= 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡 × 𝜃
1 − 𝜃)
(f0, f1, f2, f3 are respectively the frequencies of 0,1,2, and 3.)
𝑙𝑜𝑔 𝐿 = 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡 + (𝑓0 + 𝑓1)𝑙𝑜𝑔 𝜃 + (𝑓2 + 𝑓3)𝑙𝑜𝑔(1 − 𝜃)
𝑑 𝑙𝑜𝑔𝐿 𝑓0 + 𝑓1 𝑓2 + 𝑓3
=
−
=0
𝑑𝜃
𝜃
1−𝜃
(𝑓0 + 𝑓1)(1 − 𝜃) − (𝑓2 + 𝑓3)𝜃 = 0
(𝑓0 + 𝑓1) = (𝑓0 + 𝑓1 + 𝑓2 + 𝑓3)𝜃 = 𝑛𝜃
𝑓0 + 𝑓1
𝜃=
𝑛
The maximum likelihood estimate of 𝜃 is
𝑓0 + 𝑓1
𝜃̃ =
𝑛
𝑛 is the number of observations and f0 and f1 are respectively the number of
0’s and number of 1’s.
3,0,2,1,3,2,1,0,2,1
In this sample, f0 = 2, f1 =3, f2 = 3, f3 = 2, n = 10.
𝜃̃ =
𝑓0 + 𝑓1 2 + 3
5
=
=
= 0.5
𝑛
10
10
Consider the event 𝑋 = 0 𝑜𝑟 1
The frequency of this event is f0+f1.
2
1
3
3
The probability of this event is 𝜃 + 𝜃 = 𝜃
The random variable f0+f1 follows binomial distribution with parameters 𝑛 and 𝜃.
𝐸(𝑓0 + 𝑓1) = 𝑛𝜃
𝑉𝑎𝑟(𝑓0 + 𝑓1) = 𝑛𝜃(1 − 𝜃)
𝑉𝑎𝑟(𝜃̃) =
̃
1
𝜃(1 − 𝜃)
𝑉𝑎𝑟(𝑓0 + 𝑓1) =
2
𝑛
𝑛
̃
𝜃(1−𝜃)
0.5(1−0.5)
0.25
Estimated standard error of 𝜃̃ = √ 𝑛 = √ 10
= √ 10 = √0.025 = 0.1581
SUMMARY
5
Moment estimate of 𝜃 = 𝜃̂ = 12 = 0.4167
Standard error of 𝜃̂ = 0.1728
Maximum likelihood estimate of 𝜃 = 𝜃̃ = 0.5
Standard error of 𝜃̃ = 0.1581