Parameter Estimation &
Confidence Intervals
Stat 430
Fall 2011
Outline
• Parameter Estimation
• Confidence Intervals
• CI for normal distribution and binomial
• CI for differences
Finding Estimators
AMETER ESTIMATION
• i.i.d. observations x , x , ...., x
Maximum Likelihood Estimation
1 2
N from
We have n data values x1 , . . . , xn . The assumption is, that these data values are realiza
distribution Fµ
om variables X1 , . . . , Xn with distribution
Fθ . Unfortunately the value for θ is unknow
X
observed values
x1, x2, x3, ...
f
with
=0
f
with
= -1.8
f
with
=1
ng the value for θ we can “move the density function fθ around” - in the diagram, the thi
ts the data best.idea : choose µ so, that we maximize
since we do not know the true value θ of the distribution, we take that value θ̂ that m
likelihood
to observe x1, x2, ...., xN
the observed values,
i.e.
something like
•
X̄1 − X̄2 ± z ·
n1
+
n2
�
� α
σ
σ
X̄ − z · √ , X̄ + z · √
n
(θ̂n− c, θ̂
Maximum Likelihood + c
Estimation P (|θ̂ − θ| < c)
α
(θ̂ − c, θ̂ + c)
Likelihood function:
P (|θ̂ − θ| < c) > αX
•
P (X1 = x1 ∩ X2 = x2 ∩ . . . ∩ Xn = xn )
i
are indepe
=
Xi are independent!
P (X1 = x1 ∩ X2 = x2 ∩ . . . ∩ Xn = xn )
n
�
=
n
�
=
P (Xi =i=1
xi )
=
P (Xi = xi )
= P (X1
(*)
• for discrete data L(µ; x , ...., x ) = ∏ p (x )
lim
P=(|
lim
P (|
Θ̂) −=θ|
>f�)(x
0Θ̂ − θ| >
continuous
data
L(µ;
x
,
....,
x
∏
)
•
n→∞
i=1
1
1
n→∞
n
n
µ
µ
i
i
V ar[Θ̂1 ] ≤ V ar[Θ̂2 ]
V ar[Θ̂1 ] ≤ V ar
MLE
• Set up likelihood function L
• Find log likelihood l
• Find derivative of l w.r.t parameter
• Set to 0
• solve for parameter
Method of Moments
•
• Example: Gamma Distribution
Estimate kth moment µk by 1/n∑xik
Confidence Intervals
• How close is the estimate to the true value?
• Idea: compute an interval around the
estimated parameter value, in which the true
parameter is “likely” to fall.
�
1
2
n1 �n2
�
�
p̂ − z · p̂(1 − p̂/n, p̂ + z · p̂(1 − p̂/n
�
σ
σ
X̄ − z�· √ , X̄ + z · √
n
s2 n s2 r
�
Confidence
Interval
X̄ − X̄ ± z ·
+
1
2
1
2
n2
α �
�
σ
σ
+−
z ·c,√θ̂ + c)
X̄ − z · √ , X̄(θ̂
n that
n
Find c, such
•
•
n1
P P(|(|θ̂θ̂−
θ|<<c)c)
>α
− θ|
>α
α
α
X
independent!
i are
(
θ̂
−
c,
+
c)
then
is
alpha*100%
Confidence
(
θ̂
−
c,
θ̂
+
c)
x1 ∩ X2 = x2 ∩ . . . ∩ Xn = xn )Xi are independent!
=
= x2 ∩ . . .Interval
∩X
xn<)c) > α =
nθ̂=
P
(|
−
θ|
= P (X1 = x1 ) · P (X2 =
P (|θ̂ − θ| < c) > α
n
�
= P (X1 = x1
=
Pi(X
= xi )
(*)
i independent!
X
are
n n)
= x2 ∩ . . . ∩ X�
=
n = xi=1
Xi are independent!
∩ . . . ∩ X=
=)
n = xn
P) (Xi = x
= P (X(*)
i
1 = x1 ) · P (X2 = x2 ) · . . . ·
n
�
i=1
lim P (|Θ̂ − θ| > =
�) =P0 (X1 = x1 ) · P (X2 = x
n→∞
Confidence Interval for
�
�
�
�
sample
µ
p̂large
− z · p̂(1
− p̂/n, p̂ + zmean
· p̂(1 − p̂/n
•
�
s21
s22 r
X̄1 − X̄2 ± z ·
+
n
n
1
2
Confidence Interval for µ:
�
��
�
σ
σ
X̄ − z · √σ , X̄ + z · √ σ
X̄ − z · √n , X̄ + z n· √
n
n
α
(θ̂ − c, α
θ̂ + c)
P (|θ̂ − θ| < c) > α
(θ̂ − c, θ̂ + c)
Important Normal
Quantiles
alpha
z
0.9
1.65
0.95
1.96
0.98
2.33
0.99
2.58
Confidence Interval for
proportion π
interval for π:
•�Confidence
�
�
�
�
�
��
p̂ −p̂ z−·z · p̂(1
p̂/n,p̂ p̂++
− p̂/n
p̂(1 −
− p̂/n,
z ·z ·p̂(1 p̂(1
− p̂/n
X̄1 − X̄2 ± z ·
�
s21
s22 r
+
n1
n2
Confidence Interval for large
sample difference in
means:µ1 - µ2
�
�
�
�
� z· �
p̂ −
p̂(1 − p̂/n, p̂ + z�· p̂(1 −�p̂/n
p̂ − z · p̂(1 − p̂/n, p̂ + z · p̂(1 − p̂/n
Confidence Interval for µ1 - µ2:
�
�
s21s21 s22 s22 r
X̄22 ±
± zz· ·
++
1 −X̄
X̄1X̄−
n1n n2 n
1
2
�
�
�
�
σ
σ
X̄ − z · √σ , X̄ + z · √ σ
X̄ − z · √n , X̄ + z ·n√
n
n
•
α
i j
ij
�
�
σ
σ
Confidence
Interval
for
large
X̄ − t · √ , X̄ + t · √
n
n
sample difference in
Ho : πijk = πi++ ππ
+j+
proportions:
π2
1π-++k
Ho : πijk = πi++ π+jk
Ho : πijk = πij+ π+jk /π++k
Confidence Interval for π1 - π2:
��
− p̂11)/n
p̂2 (1
p̂1 −p̂p̂1 2−±p̂2z±· z · p̂1p̂(1
)/n1 1++
p̂2−
(1p̂−
p̂22 )/n2
1 (1−
2 )/n
•
�
p̂ − z ·
�
�
�
p̂(1 − p̂/n, p̂ + z · p̂(1 − p̂/n
1
X̄1 − X̄2 ± z ·
�
s21
+
s22 r