OLS: Estimation and Standard Errors
Brandon Lee
15.450 Recitation 10
Brandon Lee
OLS: Estimation and Standard Errors
Ordinary Least Squares
The model:
y = Xβ +ε
where y and ε are column vectors of length n (the number of
observations), X is a matrix of dimensions n by k (k is the
number of parameters), and β is a column vector of length k.
For every observation i = 1, 2, . . . , n, we have the equation
yi = xi1 β1 + · · · + xik βk + εi
Roughly speaking, we need the orthogonality condition
E [εi xi ] = 0
for the OLS to be valid (in the sense of consistency).
Brandon Lee
OLS: Estimation and Standard Errors
OLS Estimator
We want to find βˆ that solves
min (y − X β )0 (y − X β )
β
The first order condition (in vector notation) is
0 = X 0 y − X βˆ
and solving this leads to the well-known OLS estimator
−1 0
βˆ = X 0 X
Xy
Brandon Lee
OLS: Estimation and Standard Errors
Geometric Interpretation
The left-hand variable is a vector in the n-dimensional space.
Each column of X (regressor) is a vector in the n-dimensional
space as well, and we have k of them. Then the subspace
spanned by the regressors forms a k-dimensional subspace of
the n-dimensional space. The OLS procedure is nothing more
than finding the orthogonal projection of y on the subspace
spanned by the regressors, because then the vector of residuals
is orthogonal to the subspace and has the minimum length.
This interpretation is very important and intuitive. Moreover,
this is a unique characterization of the OLS estimate.
Let’s see how we can make use of this fact to recognize OLS
estimators in disguise as more general GMM estimators.
Brandon Lee
OLS: Estimation and Standard Errors
Interest Rate Model
Refer to pages 35-37 of Lecture 7.
The model is
rt+1 = a0 + a1 rt + εt+1
where
E [εt+1 ] = 0
2 E εt+1
= b0 + b1 rt
One easy set of moment conditions:
0 = E (1, rt )0 (rt+1 − a0 − a1 rt )
h
i
0
2
0 = E (1, rt ) (rt+1 − a0 − a1 rt ) − b0 − b1 rt
Brandon Lee
OLS: Estimation and Standard Errors
Continued
Solving these sample moment conditions for the unknown
parameters is exactly equivalent to a two-stage OLS
procedure.
Note that the first two moment conditions give us
ET (1, rt )0 (rt+1 − â0 − â1 rt ) = 0
But this says that the estimated residuals are orthogonal to
the regressors and hence â0 and â1 must be OLS estimates of
the equation
rt+1 = a0 + a1 rt + εt+1
Brandon Lee
OLS: Estimation and Standard Errors
Continued
Now define
ε̂t+1 = rt+1 − â0 − â1 rt
then the sample moment conditions
h
i
ET (1, rt )0 (rt+1 − â0 − â1 rt )2 − b̂0 − b̂1 rt = 0
tell us that b̂0 and b̂1 are OLS estimates from the equation
2
ε̂t+1
= b0 + b1 rt + ut+1
by the same logic.
Brandon Lee
OLS: Estimation and Standard Errors
Standard Errors
Let’s suppose that E εi2 |X = σ 2 and E [εi εj |X ] = 0 for i =
6 j.
In other words, we are assuming independent and
homoskedastic errors.
What is the standard error of the OLS estimator under this
assumption?
Var βˆ |X = Var βˆ − β |X
−1 0
= Var X 0 X
X ε|X
−1 0
−1
X Var (ε |X ) X X 0 X
= X 0X
Under the above assumption,
Var (ε|X ) = σ 2 In
and so
−1
Var βˆ |X = σ 2 X 0 X
Brandon Lee
OLS: Estimation and Standard Errors
Continued
We can estimate σ 2 by
n
c2 = 1
σ
∑ ε̂i2
n i=1
and the standard error for the OLS estimator is given by
c2 X 0 X −1
d
V
ar βˆ |X = σ
This is the standard error that most (less sophisticated)
statistical softwares report.
But it is rarely the case that it is safe to assume independent
homoskedastic errors. The Newey-West procedure is a
straightforward and robust method of calculating standard
errors in more general situations.
Brandon Lee
OLS: Estimation and Standard Errors
Newey-West Standard Errors
Again,
Var βˆ |X = Var βˆ − β |X
−1 0
= Var X 0 X
X ε|X
−1
−1
= X 0X
Var X 0 ε|X X 0 X
The Newey-West procedure boils down to an alternative way
of looking at Var (X 0 ε|X ).
If we suspect that the error terms may be heteroskedastic, but
still independent, then
n
d
V
ar X 0 ε|X = ∑ ε̂i2 · xi xi0
i=1
and our standard error for the OLS estimate is
!
n
−1
−1
d βˆ |X = X 0 X
Var
∑ εˆi2 · xi xi0 X 0 X
i=1
Brandon Lee
OLS: Estimation and Standard Errors
Continued
If we suspect correlation between error terms as well as
heteroskedasticity, then
!
n
k
0
k
−
|
j
|
d
V
ar X 0 ε |X = ∑
∑ εˆi εˆi +j · xi xi+j
k
t=1
j=−k
and our standard error for the OLS estimator is
−1
d
V
ar βˆ |X = X 0 X
k
k − |j |
∑ k
j=−k
Brandon Lee
n
!!
0
∑ ε̂i ε̂i+j · xi xi+j
t=1
OLS: Estimation and Standard Errors
X 0X
−1
Continued
We can also write these standard errors to resemble the
general GMM standard errors (see page 23 of Lecture 8).
In the uncorrelated errors case, we have
!
n
−1
−1
d
V
ar βˆ |X = X 0 X
∑ ε̂i2 · xi xi0 X 0 X
i=1
1
=
n
X 0X
−1
n
−1
1
= Ê xi xi0
n
!
X 0 X −1
1 n 2
0
∑ ε̂i · xi xi
n i=1
n
!
−1
1 n 2
ε̂i · xi xi0 Ê xi xi0
∑
n i=1
and for the general Newey-West standard errors, we have
k
−1
d
V
ar βˆ|X = X 0 X
∑
j=−k
−1
1
= Ê xi xi0
n
k − |j|
k
k
n
!!
X 0X
∑ εˆi ε̂i+j · xi xi+j
−1
t=1
1
k − |j|
∑ k
n j=−k
Brandon Lee
0
n
0
!!
∑ ε̂i εˆi+j · xi xi+j
t=1
OLS: Estimation and Standard Errors
ˆ xi x 0
E
i
−1
MIT OpenCourseWare
http://ocw.mit.edu
15.450 Analytics of Finance
Fall 2010
For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms .