Point Estimation
Notes of STAT 6205 by Dr. Fan
Overview
•
•
•
•
•
Section 6.1
Point estimation
Maximum likelihood estimation
Methods of moments
Sufficient statistics
o Definition
o Exponential family
o Mean square error (how to choose an estimator)
6205-Ch6
2
Big Picture
• Goal: To study the unknown distribution of a
population
• Method: Get a representative/random sample and
use the information obtained in the sample to make
statistical inference on the unknown features of the
distribution
• Statistical Inference has two parts:
o Estimation (of parameters)
o Hypothesis testing
• Estimation:
o Point estimation: use a single value to estimate a parameter
o Interval estimation: find an interval covering the unknown parameter
6205-Ch6
3
Point Estimator
• Biased/unbiased: an estimator is called unbiased if
its mean is equal to the parameter of estimate;
otherwise, it is biased
• Example: X_bar is unbiased for estimating mu
6205-Ch6
4
Maximum Likelihood
Estimation
• Given a random sample X1, X2, …, Xn from a
distribution f(x; q) where q is a (unknown) value in the
parameter space W.
• Likelihood function vs. joint pdf
n
L( x;q )   f ( xi ;q )
i 1
• Maximum Likelihood Estimator (m.l.e.) of q, denoted
as qˆ is the value q which maximizes the likelihood
function, given the sample X1, X2, …, Xn.
6205-Ch6
5
Examples/Exercises
• Problem 1: To estimate p, the true probability of heads
up for a given coin.
• Problem 2: Let X1, X2, …, Xn be a random sample from a
Exp(mu) distribution. Find the m.l.e. of mu.
• Problem 3: Let X1, X2, …, Xn be a random sample from a
Weibull(a=3,b) distribution. Find the m.l.e. of b.
• Problem 4: Let X1, X2, …, Xn be a random sample from a
N(m,s^2) distribution. Find the m.l.e. of m and s.
• Problem 5: Let X1, X2, …, Xn be a random sample from a
Weibull(a,b) distribution. Find the m.l.e. of a and b.
6205-Ch6
6
Method of Moments
• Idea: Set population moments = sample moments
and solve for parameters
• Formula: When the parameter q is r-dimensional,
solve the following equations for q:
n
E ( X k )   X ik / n for k  1,2,...r
i 1
6205-Ch6
7
Examples/Exercises
Given a random sample from a population
• Problem 1: Find the m.m.e. of m for a Exp(m)
population.
• Exercise 1: Find the m.m.e. of m and s for a N(m,s^2)
population.
6205-Ch6
8
Sufficient Statistics
• Idea: The “sufficient” statistic contains all
information about the unknown parameter; no
other statistic can provide additional information as
to the unknown parameter.
• If for any event A, P[A|Y=y] does not depend on
the unknown parameter, then the statistic Y is
called “sufficient” (for the unknown parameter).
• Any one-to-one mapping of a sufficient statistic Y is
also sufficient.
• Sufficient statistics do not need to be estimators of
the parameter.
6205-Ch6
9
Sufficient Statistics
6205-Ch6
10
Examples/Exercises
Let X1, X2, …, Xn be a random sample from f(x)
Problem: Let f be Poisson(a). Prove that
1. X-bar is sufficient for the parameter a
2. The m.l.e. of a is a function of the sufficient statistic
Exercise: Let f be Bin(n, p). Prove that X-bar is sufficient
for p (n is known). Hint: find a sufficient statistic Y for p
and then show that X-bar is a function of Y
6205-Ch6
11
Exponential Family
6205-Ch6
12
Examples/Exercises
Example 1: Find a sufficient statistic for p for Bin(n, p)
Example 2: Find a sufficient statistic for a for Poisson(a)
Exercise: Find a sufficient statistic for m for Exp(m)
6205-Ch6
13
Joint Sufficient Statistics
Example: Prove that X-bar and S^2 are joint sufficient
statistics for m and s of N(m, s^2)
6205-Ch6
14
Application of Sufficience
6205-Ch6
15
Example
Consider a Weibull distribution with parameter(a=2, b)
1) Find a sufficient statistic for b
2) Find an unbiased estimator which is a function of
the sufficient statistic found in 1)
6205-Ch6
16
Good Estimator?
• Criterion: mean square error
6205-Ch6
17
Example
• Which of the following two estimator of variance is
better?
6205-Ch6
18