Chapter 2: Statistical properties of the OLS
estimator (part III)
Florian Pelgrin
HEC
September-December, 2010
Florian Pelgrin (HEC)
Ordinary least squares estimator
September-December, 2010
1 / 18
1
What is an estimator?
2
What constitutes a good estimator?
Unbiased estimator
Best unbiased estimator in a parametric model
Best invariant unbiased estimators
3
Summary
Florian Pelgrin (HEC)
Ordinary least squares estimator
September-December, 2010
2 / 18
What is an estimator?
1. What is an estimator?
Definition
A point estimator is any function T (Y1 , Y2 , · · · , Yn ) of a sample. Any
statistic is a point estimator. Examples: Assume that Y1 ,· · · ,Yn are i.i.d. N (m, σ 2 ) random variables.
1
The sample mean
n
Ȳn =
1X
Yi
n
i=1
is a point estimator (or an estimator) of m.
2
The sample variance
n
Sn2 =
2
1 X
Yi − Ȳn
n−1
i=1
2
is a point estimator of σ .
Florian Pelgrin (HEC)
Ordinary least squares estimator
September-December, 2010
3 / 18
What is an estimator?
Remarks:
1
There is no correspondence between the estimator and the
parameter to estimate.
2
In the previous definition, there is no mention regarding the range
of the statistic T (Y1 , · · · , Yn ): the range of the statistic can be
different from the one of the parameter.
3
An estimator is a function of the sample ⇒ it is a random variable
(or vector).
4
An estimate is the realized value of an estimator (i.e. a number)
that is obtained when a sample is actually taken. For instance, ȳn
is an estimate of Ȳn and is given by:
n
ȳn =
1X
yi .
n
i=1
Florian Pelgrin (HEC)
Ordinary least squares estimator
September-December, 2010
4 / 18
What constitutes a good estimator?
2. What constitutes a good estimator?
A good estimator is one that:
1
...is unbiased, i.e.
E(T (Y )) = θ or E(T (Y )) = g(θ)
where g is known;
2
...satisfies some asymptotic properties (when the sample size is large);
3
...is efficient, i.e. has the minimum variance among all estimators of the
quantity of interest;
4
...is the best estimator in a restricted class of estimators that
satisfies some desirable properties (search within a subclass): the class
of unbiased estimators, etc.
5
...is the best estimator, which has some appropriate properties, by
maximizing or minimizing a criterion (or objective function);
6
...
Florian Pelgrin (HEC)
Ordinary least squares estimator
September-December, 2010
5 / 18
What constitutes a good estimator?
Unbiased estimator
2.1. Unbiased estimator...
Definition
An estimator T (Y ) is unbiased for θ if
Eθ [T (Y )] = θ for all θ ∈ Θ.
Florian Pelgrin (HEC)
Ordinary least squares estimator
September-December, 2010
6 / 18
What constitutes a good estimator?
Unbiased estimator
Examples:
1. Let Y1 ,· · · ,Yn be a random sampling from a Bernoulli distribution.
An unbiased estimator of p is
p
1X
T (Y ) =
Yi .
n
i=1
2. Let Y1 ,· · · ,Yn be a random sample from the uniform distribution
U[0,θ] . An unbiased estimator of θ is
n
T (Y ) =
2X
Yi .
n
i=1
Florian Pelgrin (HEC)
Ordinary least squares estimator
September-December, 2010
7 / 18
What constitutes a good estimator?
Unbiased estimator
Examples (cont’d):
3. The multiple linear regression model.
Consider the model
Y = X β0 + u
where Y ∈ Rn , X ∈ Mn×k is nonrandom, E(u) = 0n×1 , and V(u) = σ 2 In .
The OLS estimator
−1 t
T (Y ) = X t X
X Y
is an unbiased estimator of β 0 .
4. The generalized multiple linear regression model.
Consider the model
Y = X β0 + u
where Y ∈ Rn , X ∈ Mn×k is nonrandom, E(u) = 0n×1 , and V(u) = σ 2 Ω.
The Ω matrix is known. The GLS estimator
−1 t −1
T (Y ) = X t Ω−1 X
X Ω Y
is an unbiased estimator of β 0 .
Florian Pelgrin (HEC)
Ordinary least squares estimator
September-December, 2010
8 / 18
What constitutes a good estimator?
Unbiased estimator
Some comments...:
1. The unbiasedness condition must hold for every possible value of
the parameter and not only for some of these values.
2. In general, the property of unbiasedness is not conserved after a
nonlinear transformation of the estimator.
3. Asymptotically unbiased estimators:
Definition
The sequence of estimators θ̂n ≡ Tn (Y ) (with n ∈ N) is
asymptotically unbiased if
lim Eθ (Tn (Y )) = θ for all θ ∈ Θ
n→∞
where Eθ is defined with respect to Pθ,n . Florian Pelgrin (HEC)
Ordinary least squares estimator
September-December, 2010
9 / 18
What constitutes a good estimator?
Unbiased estimator
Unbiasedness is interesting per se but not so much!
(a) The absence of bias is not a sufficient criterion to discriminate
among competitive estimators.
(b) It may exist many unbiased estimators for the same parameter
(vector) of interest.
(c) This is also true when one requires that the estimator is
asymptotically unbiased.
Florian Pelgrin (HEC)
Ordinary least squares estimator
September-December, 2010
10 / 18
What constitutes a good estimator?
Best unbiased estimator in a parametric model
2.2 Best unbiased estimator in a parametric model
When an estimator is unbiased (restrict the class of estimators), its
(matrix) quadratic risk function is given by
Rθ (T (Y ), θ) = Eθ (T (Y ) − θ)(T (Y ) − θ)t
and thus reduced to its variance-covariance matrix Vθ (T (Y )).
Comparing two (or more) unbiased estimates becomes equivalent to
comparing their variance-covariance matrices.
Definition
Suppose that T1 (Y ) and T2 (Y ) are two unbiased estimators. T1 (Y )
dominates T2 (Y ) if and only if
Vθ (T2 (Y )) Vθ (T1 (Y ))
i.e. the matrix Vθ (T2 (Y )) − Vθ (T1 (Y )) is a positive semi-definite matrix for all
θ ∈ Θ.
Florian Pelgrin (HEC)
Ordinary least squares estimator
September-December, 2010
11 / 18
What constitutes a good estimator?
Best unbiased estimator in a parametric model
Definition
Under suitable regularity conditions, a (regular) unbiased estimator of
θ is efficient if its variance-covariance matrix equals the
Frechet-Darmois-Cramer-Rao lower bound:
Vθ (T (Y )) = I(θ)−1
where I(θ) is the Fisher information matrix
∂logL(Y ; θ) ∂logL(Y ; θ)
I(θ) = Eθ
∂θ
∂θt
with L the likelihood function. Florian Pelgrin (HEC)
Ordinary least squares estimator
September-December, 2010
12 / 18
What constitutes a good estimator?
Best invariant unbiased estimators
2.3. Best invariant unbiased estimators
The previous results can only be used to find best unbiased
estimators in parametric models.
These results no longer apply in the case of semi-parametric
models.
The class of estimators must be again restricted: impose
invariance conditions
best linear unbiased estimator
best quadratic unbiased estimator
Florian Pelgrin (HEC)
Ordinary least squares estimator
September-December, 2010
13 / 18
What constitutes a good estimator?
Best invariant unbiased estimators
Best linear unbiased estimators: The Gauss-Markov
theorem
Theorem
Consider the (conditional static) linear regression model:
Y = X β0 + u
with E(ui | X ) = 0 and V(ui | X ) = σ02 In , E(ui3 | X ) = 0,
E(ui4 | X ) = 3σ04 , Y is an n-dimensional vector and X is an n × k
matrix, with P(rk (X ) = k) = 1 or rk(X ) = k . The ordinary least
squares estimator of β 0 defined by
β̂OLS = (X t X )−1 X t Y
is the best estimator in the class of linear (in Y ) unbiased estimators of
β0 . Florian Pelgrin (HEC)
Ordinary least squares estimator
September-December, 2010
14 / 18
What constitutes a good estimator?
Best invariant unbiased estimators
β̂j,OLS is the BLUE of βj , j = 1, · · · , k :
Best = smallest variance
Linear (in Y or yi ) = β̂j =
n
P
wij yi
i=1
Unbiased = E(β̂j ) = βj , ∀β ∈ Θ
Estimator = β̂j = f (data).
Florian Pelgrin (HEC)
Ordinary least squares estimator
September-December, 2010
15 / 18
What constitutes a good estimator?
Best invariant unbiased estimators
Best quadratic unbiased estimators
Theorem
Consider the (conditional static) linear regression model:
Y = X β0 + u
with E(ui | X ) = 0 and V(ui | X ) = σ02 In , E(ui3 | X ) = 0,
E(ui4 | X ) = 3σ04 , Y is an n-dimensional vector and X is an n × k matrix
of rank k (or P(rk(X ) = k) = 1).
The estimator of σ02 defined by
s2 =
1
û 0 û
Y 0 MX Y =
n−k
n−k
is the best quadratic unbiased estimator of σ02 .
Florian Pelgrin (HEC)
Ordinary least squares estimator
September-December, 2010
16 / 18
What constitutes a good estimator?
Best invariant unbiased estimators
To sum up,
parametric
parametric
(Gaussian)
semi-parametric
Florian Pelgrin (HEC)
Table 1: Efficiency of estimators
Fixed regressors
Stochastic regressors
- Best unbiased estimator - Best unbiased estimator
- FDCR bound
- FDCR bound
- Y ∼ N(X β0 , σ02 In )
- Y | X ∼ N(X β0 , σ02 In )
- BUE and FDCR bound
- BUE and FDCR bound
- BLUE
- BLUE
- Gauss-Markov
- Gauss-Markov
Ordinary least squares estimator
September-December, 2010
17 / 18
Summary
Key concepts
What is an estimator?
What is an unbiased estimator? is the unbiasedness property
sufficient to distinguish among competing estimators?
What is an (asymptotically) efficient estimator? In which sense?
How can we characterize efficiency?
Florian Pelgrin (HEC)
Ordinary least squares estimator
September-December, 2010
18 / 18