Econometrics Lecture Notes (Grad)
Parush Arora
Introduction to Regression
1.1
Conditional Expectations
Let’s assume we are interested in knowing about some variable, say wage. The first thing you can do is plot
the histogram/density.
Having knowledge about some other related variable helps us make inference about the population in a
much better way. The extra information coming from X is used to shrink the distribution of Y (reducing
uncertainty). See Figure 1.
Figure 1:
Rather than considering the whole distribution, we focus on one point (expectation). Unconditional expectation E(Y ) does not take into account relation between Y and X but E(Y |X) does. Conditional expectation
use extra information to make better inference. See Figure 2.
Figure 2:
1.2
Framework
Let’s assume we are interested in finding the effect of education on wage. Let y be a random variable
representing wage and x be a random variable representing education. y is our dependent variable whereas
x is our independent variable. The true model takes the following form
y = f (x) + ϵ ,
E(ϵ|x) = 0
which can be written as
E(y|x)
=
f (x)
y
=
E(y|x) + ϵ
1
We make an assumption about the true model by imposing a linear structure. The benefit of the assumption
is that it allows for easier estimation and clear inference. The cost is that the assumption can be wrong.
y = α + βx + ϵ ,
E(ϵ|x) = 0
where ϵ is the error term which captures all other variables and measurement error. This means
E(y|x)
=
α + βx
y
=
E(y|x) + ϵ
By estimating α and β, we estimate the average relationship between y and x, i.e., when x changes, how
on average y would change. This is called data generating process (DGP) for y. So the eventual aim is to
estimate conditional expectation function (CEF) which is E(y|x).
Let’s assume we have data for wages and education as {(x1 , y1 ), . . . , (xn , yn )}. Each observation is independent and identically distributed (i.i.d). We can write the model as
yi = α + βxi + ϵi ∀ i = 1, . . . , n.
We are interested in estimating α and β.
1.3
Ordinary Least Squares
1.3.1
Population Setting
We minimizes the following objective function to estimate our parameters.
{α̂, β̂} = arg min E(y − α − βx)2
α,β
The objective function is called mean squared errors (MSE) because it can be written as
E(y − α − βx)2 = E(ϵ)2
Taking first order conditions, we can estimate α and β as
∂M SE
∂α
∂M SE
∂β
=
−2E(y − α − βx)
=
−2E(y − α − βx)x
Equating the first order conditions to 0 and solving them
α
0
=
=
E(y) − βE(x)
−2E(y − E(y) + βE(x) − βx)x
0
=
−2E(yx − E(y)x + βE(x)x − βx2 )
β (E(x2 ) − E(x)2 ) =
{z
}
|
V ar(x)
=
=
E(yx) − E(y)E(x)
|
{z
}
Cov(x,y)
E(y − E(y))(x − E(x))
E(x − E(x))2
Cov(x, y)
V ar(x)
Now we consider the vector form which is given as
y
α
x
+ϵ
β
= 1
= x′ β + ϵ
Then we estimate the parameters by minimizing
β = arg min E(y − x′ β)2
β
The first order conditions take the following form
∂M SE
= −2xE(y − x′ β)
∂β
Equating to 0 and solving for β
β
= E(xx′ )−1 E(xy)
2
1.3.2
Sample Setting
We minimizes the following objective function to estimate our parameters.
{α̂, β̂} = arg min
α,β
n
X
(yi − α − βxi )2
i=1
The objective function is called sum of squared errors (SSE) because it can be written as
n
X
(yi − α − βxi )2 =
n
X
i=1
ϵ2
i=1
Taking first order conditions, we can estimate α and β as
∂SSE
∂α
= −2
∂SSE
∂β
= −2
n
X
i=1
n
X
(yi − α − βxi )
(yi − α − βxi )xi
i=1
Equating the first order conditions to 0 and solving them
α̂
β̂
= ȳ − β̂ x̄
Pn
1
(yi − ȳ)(xi − x̄)
Pn
= n 1i=1
2
i=1 (xi − x̄)
n
Cov(xi , yi )
=
V ar(xi )
Thus, the estimated regression looks like
yi = α̂ + β̂xi + ϵ̂i ∀ i = 1, . . . , n.
where ϵ̂i is the residual term or unexplained part of yi .
1.4
OLS is the Best Linear Approximation
What is best? Mean square deviation? mean absolute deviation? Or some other measure of distance? What
Figure 3:
if the true DGP is not linear but takes the following form
Y
= f (X) + ϵ ,
E(ϵ|X) = 0
We are interested in finding best linear approximation of E(Y |X) = f (X). For simplicity, we will consider
the matrix form. We define the best linear estimator such that it minimizes E(f (X) − X ′ β)2 . If we use
3
OLS to estimate the linear approximation then
β
=
arg min E(y − x′ β)2
=
arg min E(f (x) + ϵ − x′ β)2
=
arg min E(f (x) − x′ β)2 + E(ϵ2 ) + 2E(ϵ(f (x) − x′ β))
β
β
β
If E(ϵ|x) = 0 then E(ϵg(x)) = 0 using Law of iterated expectations where g(x) is a function of x
= arg min E(f (x) − x′ β)2 + E(ϵ2 )
β
| {z }
Irrelevant
=
arg min E(f (x) − x′ β)2
β
Showing why E(ϵ(f (x) − x′ β)) = 0. Let ϵ(f (x) − x′ β) = Z where Z is a new random variable.
E(Z)
= E(E(Z|x))
by law of iterated expectations
= E(E(ϵ(f (x) − x′ β)|x))
= E(E(ϵ(f (x)|x) − E(ϵX ′ β)|x))
= E(E(ϵ|x)f (x) − E(ϵ|X)x′ β)
=
0
Therefore, minimizing MSE is equivalent to finding the best linear approximation for f (x). Therefore, OLS
is the best linear approximation.
2
Statistical Review
2.1
Expectations
• If x is an k × 1 random vector.




x1
E(x1 )
 


E( ... ) =  ... 
xk
E(xk )
• Transpose: E(x′ ) = E(x)′
• For any constant a:E(ax) = aE(x).
• For any constant matrix A and B, E(AXB ′ ) = AE(X)B ′ .
• For any two random vectors X and Y, E(X + Y ) = E(X) + E(Y ).
2.2
Variance-Covariance Matrix


µ1
 
• If x is a k × 1 matrix and µ = E(x) =  ... , then V ar(x) = E((x − µ)(x − µ)′ ) which is a variance
µk
covariance matrix. It is symmetric and can written as


V ar(x1 )
Cov(x1 , x2 ) . . .

V ar(x2 )
. . .
V ar(x) = Cov(x2 , x1 )

..
..
..
.
.
.
• If a is a constant vector then V ar(a′ x) = a′ V ar(x)a.
• if A is a constant matrix and b is a constant vector then V ar(Ax + b) = AV ar(x)A′ .
4
2.3
Law of iterated Expectations
EY (Y ) = EX (EY (Y |X))
Proof:
Z
EX (EY (Y |X)) = EX ( yp(y|x)dy)
Z Z
=
( yp(y|x)dy)f (x)dx
Z Z
=
y p(yx)dxdy
Z
=
yp(y)dy
=
EY (Y )
where p() is either pdf or pmf.
2.4
Decomposition of Variance
V arY (Y ) = EX (V arY (Y |X)) + V arX (EY (Y |X))
We will show it on linear regression
V ar(y)
E(V ar(y|x))
V ar(E(y|x))
2.5
=
V ar(x′ β + ϵ)
=
β ′ V ar(x′ )β + V ar(ϵ)
= E(V ar(x′ β + ϵ|x))
=
E(V ar(ϵ|x))
=
V ar(ϵ)
= V ar(E(x′ β + ϵ|x))
=
V ar(E(x′ β|x))
=
V ar(x′ β)
=
β ′ V ar(x′ )β
Differentiation
If X is random matrix and A is a constant matrix then
∂X ′ A
∂X
∂AX
∂X
∂X ′ AX
∂X
2.6
= A
= A′
=
2AX
Independence and No Correlation
Let X and Y are two random variables. X and Y are independent if
p(X, Y ) = p(X)p(Y )
where p() is a pdf/pmf. X and Y are mean independent if
E(X|Y ) = E(X)
X and Y have no correlation if
E(XY ) = E(X)E(Y )
5
Being independent implies mean independence.
Z
E(X|Y )
=
xp(x|y)dx
Z
=
p(xy)
dx
p(y)
Implying independence
Z
p(x)p(y)
dx
x
p(y)
Z
xp(x)dx
=
E(X)
=
=
x
Being mean independent implies no correlation
Z Z
E(XY ) =
xyp(xy)dxdy
Z Z
=
xyp(x|y)p(y)dxdy
Z Z
=
y xp(x|y)dxp(y)dy
Z
=
yE(X|Y )p(y)dy
implying mean independence
Z
=
yE(X)p(y)dy
= E(X)E(Y )
2.7
Statistics and Finite Properties
A statistic is a function of random variable. Statistic is a random variable itself. An estimator is a statistic
directed toward a parameter. A sequence of random variables can have the following relations
1. Independent and identically distributed (i.i.d) - random sample.
2. Independent and non identically distributed (i.n.i.d)
3. Non independent and identically distributed (n.i.i.d)
4. Non independent and non identically distributed (n.i.n.i.d)
2.7.1
Unbiasedness
If θ̂ is an estimator for parameter θ, then θ̂ is unbiased if
E(θ̂) = θ
The bias is defined as
Bias(θ̂) = E(θ̂) − θ
Unbiasedness means the average of the sampling distribution of θ̂ is θ. Sampling distribution is a distribution
of an estimator over multiple samples. This is why, the estimator is treated as a random variable as its value
changes with the sample. Example
Pn1: Let x1 , x2 , . . . , xn be a sequence of independent and identically
distributed random variables. Is n1 i=1 xi an unbiased estimator for µ = E(X)?
n
E(
1X
xi )
n i=1
n
=
1X
E(xi )
n i=1
n
=
=
6
1X
µ
n i=1
µ
Pn
Therefore, n1 i=1 xi is unbiased. Example
2: Let x1 , x2 , . . . , xn be a sequence of independent and identically
Pn
distributed random variables. Is n1 i=1 (xi − x̄)2 an unbiased estimator for σ 2 = V ar(X)?
n
E(
1X
(xi − x̄)2 )
n i=1
= E(
n
n
n
1X 2 1X 2
1X
xi +
x̄ − 2
x̄xi )
n i=1
n i=1
n i=1
n
=
1X
E(x2i ) + E(x̄2 ) − 2E(x̄2 )
n i=1
=
1X
E(x2i ) − E(x̄2 )
n i=1
=
n
n
1X
1X
V ar(xi ) +
(E(xi ))2 − V ar(x̄) − (E(x̄))2
n i=1
n i=1
n
= σ 2 + µ2 −
=
Therefore, n1
2.7.2
σ2
− µ2
n
n−1 2
σ
n
Pn
n−1 2
1 2
2
2
2
i=1 (xi − x̄) is biased estimator of σ . The bias is n σ − σ = − n σ .
Efficiency
Efficiency means lower variance. Among all the set of unbiased estimators, we prefer the one with the lowest
variance. An estimator is efficient if V ar(θ̂) ≤ V ar(θ̃) for all unbiased estimator θ̃, which means E(θ̃) = θ.
Example. Let x1 , x2 , . . . , xn be a sequence of independent
Pn and identically distributed random variables. Find
variance of the following unbiased estimators 1) n1 i=1 xi and 2) x5 which are trying to estimate µ.
n
V ar(
1X
xi )
n i=1
n
=
=
=
1 X
V ar(xi )
n2 i=1
n
V ar(xi )
n2
1
V ar(xi )
n
V ar(x5 ) = V ar(xi )
Pn
1
Estimator n
2.7.3
i=1 xi is more efficient than x5 .
Mean Square Error (MSE)
It is defined as E[(θ̂ − θ)2 ]. How it is different from Variance?
E[(θ̂ − θ)2 ]
= E[(θ̂ − E(θ̂) + E(θ̂) − θ)2 ]
= E[(θ̂ − E(θ̂))2 ] + E[(E(θ̂) − θ)2 ] + 2E[(θ̂ − E(θ̂))(E(θ̂) − θ)]
= E[(θ̂ − E(θ̂))2 ] + E[(E(θ̂) − θ)2 ] + 2 (E(θ̂) − E(θ̂))(E(θ̂) − θ)
|
{z
}
=0
2
2
= E[(θ̂ − E(θ̂)) ] + E[(E(θ̂) − θ) ]
|
{z
} |
{z
}
V ar(θ̂)
Bias(θ̂)2
= V ar(θ̂) + Bias(θ̂)2
2.8
Statistics and Asymptotic Properties
2.8.1
Convergence in mean square
mse
Let x1 , x2 , . . . , xn , . . . be a sequence of random variables. xn −−→ c if
lim E(xn − c)2 = 0
n→∞
7
Or
lim E(xn ) = c
n→∞
lim V ar(xn ) = 0
n→∞
Example: Let x1 , x2 , . . . , xn P
be a sequence of independent and identically distributed random variables.
n
Given V ar(xi ) = σ 2 , does n1 i=1 xi converges in MSE to µ = E(X)?
n
E(
1X
xi )
n i=1
lim E(
1X
xi )
n i=1
V ar(
1X
xi )
n i=1
lim V ar(
1X
xi )
n i=1
= µ
n
n→∞
= µ
n
=
σ2
n
=
0
n
n→∞
mse
i=1 xi −−→ µ.
Thus, n1
Pn
2.8.2
Convergence in Probability
p
Let x1 , x2 , . . . , xn , . . . be a sequence of random variables. xn −
→ c or plimxn = c if
lim P r(|xn − c| < ϵ) = 1
n→∞
Example:
(
0
xn =
n2
with probability 1 − n1
with probability n1
xn converges in probability to 0 as n → ∞, P r(xn = 0) → 1. Does it converge in MSE? Lets check
E(xn )
0(1 −
1
1
) + n2
n
n
=
n
lim E(xn )
=
∞
V ar(xn )
=
02 (1 −
n→∞
lim V ar(xn )
n→∞
mse
=
=
1
1
) + (n2 )2 − n2
n
n
n2 (n − 1)
=
∞
p
mse
Thus, xn −−
̸ → 0. If xn −−→ 0, then it implies that xn −
→ 0.
2.8.3
Convergence in Distribution
d
xn −
→ x, where x is a random variable with cdf F if limn→∞ Fn (xn ) = F (x). As n →
− ∞, the distribution of
xn in the sequence becomes arbitrarily close to fixed distribution x. It does not mean that xn becomes x,
p
d
only it’s distribution becomes identical to that of x. If xn −
→ x then xn −
→ x. where x is a single point with
probability 1.
2.9
Consistency
p
θ̂ is a consistent estimator of θ if θ̂ −
→ θ. We will show different cases of estimator being unbiased and
consistent. Let x1 , x2 , . . . , xn , . . . be a sequence of independent and identically distributed random variables.
Let E(xi ) = µ which is the parameter of interest. We will consider the following estimators.
8
1. Is n1
Pn
i=1 xi an unbiased estimator for µ?
n
E(
1X
xi )
n i=1
n
1X
E(xi )
n i=1
=
n
1X
µ
n i=1
= µ
=
Yes it is. Is n1
Pn
i=1 xi a consistent estimator for µ?
n
E(
1X
xi )
n i=1
lim E(
1X
xi )
n i=1
V ar(
1X
xi )
n i=1
lim V ar(
1X
xi )
n i=1
= µ
n
n→∞
= µ
n
=
σ2
n
=
0
n
n→∞
Yes it is
Pn
2. Is n1 i=1 xi + n1 an unbiased estimator for µ?
n
E(
1
1X
xi + )
n i=1
n
n
=
1
1X
E(xi ) +
n i=1
n
=
1X
µ
n i=1
n
= µ+
No it is not. Is n1
1
n
Pn
1
i=1 xi + n a consistent estimator for µ?
n
E(
1X
1
xi + )
n i=1
n
lim E(
1
1X
xi + )
n i=1
n
V ar(
1X
1
xi + )
n i=1
n
lim V ar(
1X
1
xi + )
n i=1
n
= µ+
1
n
n
n→∞
= µ
n
=
σ2
n
=
0
n
n→∞
Yes it is.
3. Is xi an unbiased estimator for µ?
E(xi )
=
µ
Yes it is . Is xi a consistent estimator for µ?
E(xi )
=
µ
lim E(xi )
=
µ
V ar(xi )
=
σ2
lim V ar(xi )
=
σ2
n→∞
n→∞
Does not converge in mean square since it is converging in distribution to itself. Kind of no convergence
as it is independent of n.
4. Check for xi + n1 .
9
2.10
Law of Large Numbers
2.10.1
Kolmogorov-Khinchin LLN
Let x1 , x2 , . . . , xn be a sequence of independent and identically distributed random variables and E(xi ) =
Pn
p
µ < ∞. If x̄n = n1 i=1 xi , then x̄n −
→ µ or plim(x̄n ) = µ. (It does not require V ar(xi ) < ∞.
2.10.2
Chebychev’s LLN
Let x1 , x2 , . . . , xn be a sequence of non independent and non identically
Pn distributed random variables
with E(xi ) = µi , V ar(xi ) = σi2 and Cov(xi , xj ) = σij . If x̄n = n1 i=1 xi , limn→∞ E(x̄n ) < ∞ and
Pn
p
limn→∞ V ar(x̄n ) = 0, then (x̄n − µ̄n ) −
→ 0 or plim(x̄n − µ̄n ) = 0. (µ̄n = n1 i=1 µi ).
2.11
Continuous Mapping Theorem (CMT)
For any continuous function g that does not depend on n,
p
p
d
d
• If zn −
→ z, then g(zn ) −
→ g(z).
• If zn −
→ z, then g(zn ) −
→ g(z).
Examples:
p
p
p
p
• If x̄n −
→ µ, then x̄2n −
→ µ2 .
→ µ1 .
• If x̄n −
→ µ, then x̄1n −
Unbiasedness does not have this property. Expectation does not go through nonlinear functions. The rule
can be extended to vectors/matrices. If Wn is a random matrix and plimWn = Ω, then plimWn−1 = Ω−1 .
2.12
Slutsky’s Theorem
p
d
If xn −
→ x, yn −
→ c, where x is some random variable and c is a constant, then:
d
xn /yn −
→ x/c
d
xn yn −
→ xc
d
xn + yn −
→x+c
d
xn − yn −
→x−c
p
p
A special case is when xn −
→ d, yn −
→ c, where c and d are constants. The theorem is not valid both random
variable converge . The rule can be extended to vectors/matrices. If Xn and Yn are random matrices and
plimXn = A and plimYn = B, then plimXn Yn = AB.
2.13
Combining Consistent Estimators
We can apply properties of plim to combine consistent estimators in different ways to obtain other consistent
estimators.
Example 1: Let {y1 , . . . , yn } be a random sample of annual earnings from the population of workers with a
high school education with a population mean µy . Let {z1 , . . . , zn } be a random sample of annual earnings
from the population of workers with a college education with a population mean µz . We wish to estimate:
γ = 100 ×
µz − µy
µy
10
We propose 100 × z̄−ȳ
ȳ as a consistent estimator of γ. We know that plimz̄ = µz and plimȳ = µy using LLN.
plim(100 ×
z̄ − ȳ
)
ȳ
z̄ − ȳ
)
ȳ
plim(z̄ − ȳ)
= 100 × (
)
plim ȳ
Using Slutsky Theorem
plim z̄ − plim ȳ)
= 100 × (
)
plim ȳ
µz − µy
= 100 × (
)
µy
100 × plim(
=
Is it an unbiased estimator of γ?
E(100 ×
z̄ − ȳ
)
ȳ
=
̸=
z̄ − ȳ
)
ȳ
E(z̄ − ȳ)
100 × (
)
E(ȳ)
100 × E(
Example 2: Given:
yi = bxi + εi
The OLS estimator is:
P
P
xi εi
xi yi
b̂OLS = P 2 = b + P 2
xi
xi
Then:
plim(b̂OLS )
2.14
P
xi εi
= plim(b + P 2 )
x
P i
plim xi εi
P
= b+
)
plim x2i
Using Slutsky Theorem
P
xi εi
plim n1
P 2
= b+
plim n1
xi
E(xi εi )
= b+
E(x2i )
Using LLN
Asymptotic Property of Estimators - Asymptotic Normality
Given consistency, as n → ∞, the estimator θ̂ has a degenerate distribution that collapses to θ with p = 1.
So we cannot do statistical inference. To obtain a non-degenerate
distribution as n → ∞, we need to magnify
√
or rescale θ̂. Often, the appropriate scale factor is n, so consider:
√
bn = n(θ̂ − θ).
bn generally has a complicated cumulative distribution function (cdf) Fn but Fn might have a well behaved
limit function F . Then, we can say that bn converges in distribution to b where F is the cdf of b and
limn→∞ Fn = F . Generally, the asymptotic distribution of bn is normal. If:
√
d
n(θ̂ − θ) −
→ N (0, σ 2 ),
then θ̂ is asymptotically normal.
2.15
Central Limit Theorem (CLT)
As we increase the number of observations, averages of a large number of random variables behave like
normally distributed variables, regardless of the underlying distributions.
11
2.15.1
Lindeberg-Levy CLT
Assume x1 , x2 , . . . is a sequence of i.i.d. random variables such that E(xi ) = µ and Var(xi ) = σ 2 . Let:
n
1X
xi .
n i=1
x̄n =
Then:
√
d
n(x̄n − µ) −
→ N (0, σ 2 ).
a
2
We say x̄n is asymptotic normal and write as x̄n = N (µ, σn )
2.15.2
Lindeberg-Feller CLT
For i.n.i.d. random variables xi with E(xi ) = µi and Var(xi ) = σi2 , define:
µ̄n =
1X
µi ,
n i
σ̄n2 =
1X 2
σ .
n i i
If limn→∞ max(σi2 )/(nσ̄n2 ) = 0 (no individual variance dominate the sum), then:
√
2.16
d
n(x̄n − µ̄n ) −
→ N (0, σ̄n2 ).
The Delta Method
Univariate version: If
√
d
n(zn − µ) −
→ N (0, σ 2 ), then for any function m(·) such that
• m(·) is differentiable in the neighborhood of µ
• m′ (µ) is finite and m′ (µ) ̸= 0
• m(·) is not a function of n.
√
d
n(m(zn ) − m(µ)) −
→ m′ (µ)N (0, σ 2 )
d
−
→ N (0, (m′ (µ))2 σ 2 )
Example: Given that
√
d
n(zn − α) −
→ N (0, 1), by the Delta method:
√
d
n(zn2 − α2 ) −
→ N (0, 4α2 ).
√
d
Multivariate version:: If n(xn − a) −
→ N (0, Ω), where xn is a P × 1 random vector, a is P × 1 constant
vector and Ω is P × P var-cov matrix, then for any function g(·), which is possibly vector-valued, such that
• g(·) is differentiable in the neighborhood of µ
• g ′ (a) is finite and g ′ (a) ̸= 0
• g(·) is not a function of n.
√
n(g(xn ) − g(a))
∂g(a)
N (0, Ω)
∂a′
∂g(a) ∂g(a)
d
−
→ N (0, (
Ω
)
∂a′
∂a
d
−
→
Example: Let
β1
β̂1
V ar(β̂1 )
Cov(β̂1 , β̂2 )
β=
, β̂ =
, V ar(β̂) =
β2
β̂2
Cov(β̂2 , β̂1 )
V ar(β̂2 )
It is given that
√
d
n(β̂ − β) −
→ N (0, V ar(β̂)). Let g(β) = β1 − β2 . Thus, g(β̂) = β̂1 − β̂2 . Using Delta method
√
n(g(β̂) − g(β))
d
−
→ N (0,
12
∂g(β)
∂g(β)
V ar(β̂)
)
∂β ′
∂β
where
∂g(β)
∂β ′
=
h
=
1
∂g(β)
∂β1
i
−1
" ∂g(β) #
∂g(β)
∂β
∂β1
∂g(β)
∂β2
=
1
−1
=
2.17
∂g(β)
∂β2
Multivariate Normal and Related Distributions
Normal Distribution If z ∼ N (µ, Ω), then for any constant matrix A and vector b:
Az + b ∼ N (Aµ + b, AΩA′ ).
Chi-Square Distribution If zi ∼ N (0, 1), then:
n
X
zi2 ∼ χ2n .
i=1
For a general normal vector z ∼ N (0, In ) and symmetric idempotent matrix A of rank k:
z ′ Az ∼ χ2k .
t Distribution
N (0, 1)
,
tn = p
χ2n /n
where N (0, 1) ⊥
⊥ χ2n . If z ∼ N (0, In ) and A is a matrix, then Az and z ′ Bz are independent if and only if
AB = 0.
F Distribution
Fn1 ,n2 =
χ2n1 /n1
.
χ2n2 /n2
If z ∼ N (0, In ) and A, B are symmetric idempotent matrices with AB = 0, then:
z ′ Az/k1
∼ Fk1 ,k2 .
z ′ Bz/k2
3
Linear Regression Model
Writing the model in matrix notation.
yi
= β0 + β1 x1i + . . . + βk xki + ϵi
 
β0


 β1 
= 1 x1i . . . xki  .  + ϵi
 .. 
βk
=
x′i β + ϵi
∀ i = 1, . . . , n.
Writing in the more extensive form
y1
..
.
= x′1 β + ϵ1
.
..
= ..
.
yn
= x′n β + ϵn
13
 ′
 
x1
ϵ1
 .. 
 .. 
 . β +  . 


y1
 .. 
. =
x′n
ϵn
X β + |{z}
ϵ
|{z}
|{z}
yn
Y
=
|{z}
n×1
n×1
n×k k×1
So if we expand this, the matrix form contains the whole data as

   
 
1 x11 . . . xk1
β0
ϵ1
y1
1 x12 . . . xk2  β1   ϵ2 
 y2 

   
 
 ..  =  ..
..
..   ..  +  .. 
..
.
.
.
.
.  .   . 
1
yn
| {z }
Y
Example:
x1n
|
...
{z
xkn
X
βk
} | {z }
β
Wagei = β0 + β1 Educi + β2 Agei + ui ,
It will be written in the matrix form as



1
W age1
1
 W age2 



 ..  =  ..
.
 . 
1
W agen
3.1
Assumptions
3.1.1
Assumption 1: Linearity
Edu1
Edu2
..
.
Edun
ϵn
| {z }
ϵ
i = 1, . . . , n.

 
Age1  
ϵ1
β0
 ϵ2 
Age2 

 
..  β1  +  .. 
.
.  β2
Agen
ϵn
The first assumption is that the population relationship between the dependent variable and the regressors is
linear. Linear means linear in parameters, non linear functions of regressors are accommodated, for example
X 2 , log X, eX etc.
Example:
yi
yi
3.1.2
=
β0 + β1 e x i + ϵi
- Acceptable
=
β0 +β1 xi
- Not acceptable
e
+ ϵi
Assumption 2: Strict Exogeneity
The conditioning set includes all observations from all explanatory variables. This point may be made more
apparent by writing the assumption as follows:
E[ ϵi | x1 , x2 , . . . , xn ] = 0 ∀ i = 1, . . . , n.
|{z} |{z}
1×1
k×1
The error term ui is mean independent of the explanatory variables. E(ϵi |X) should not be a function of X.
Implications:
1. E(E[ϵi |X]) = E[ϵi ] = 0
2. No correlation between X and ϵi .
Failure of strict exogeneity: Occurs when omitted variables, measurement error, or simultaneity exist. A
violation of strict exogeneity can result in biased and inconsistent estimators, leading to incorrect inference
and predictions. Example:
yt
= yt−1 + ut
where E[ut ] = 0 and E[yt−1 ut ] = 0 for all t.
E[yt ut ]
=
E[(yt−1 + ut )ut ]
=
E[yt−1 ut ] + E[u2t ]
=
V ar[ut ]
̸=
0
14
3.1.3
Assumption 3: No Perfect Collinearity
Explanation: No explanatory variable can be written as an exact linear combination of others (Rank(X) = k).
k is the number of independent variables and n ≥ k. Example: if X1 is income and X2 = X1 /1000, then the
information in X2 is redundant. Implications: (X ′ X) cannot be inverted unless Rank(X) = k.
3.1.4
Assumption 4: Homoskedastic Error and No Serial Correlation
Given strict exogeneity, we assume V ar(ϵi |X) = σ 2 and Cov(ϵi , ϵj |X) = 0 for i ̸= j. In other words, the
variance covariance matrix (Ω ) will take the form
 2

σ
0 ... 0
 0 σ2 . . . 0 


Ω= .
.. . .
.. 
 ..
. .
.
0
0 . . . σ2
3.2
Regression Model for Random Samples
Given a random sample, {yi , xi }ni=1 are i.i.d. (independent and identically distributed), so (yi , xi ) ⊥⊥ (yj , xj )
for i ̸= j. It implies that ϵi ⊥
⊥ (yj , xj ). Therefore, the assumptions are simplified.
Strict exogeneity: From satisfying
E[ϵi |x1 , . . . , xn ] = 0 ∀ i = 1, . . . , n
we now have to satisfy just
E[ϵi |xi ] = 0 ∀ i = 1, . . . , n
Homoskedastic error: From satisfying
V ar[ϵi |x1 , . . . , xn ] = σ 2 ∀ i = 1, . . . , n
we now have to satisfy just
V ar[ϵi |xi ] = σ 2 ∀ i = 1, . . . , n
which is automatically satisfied due to identically distributed.
No serial correlation: Automatically satisfied due to independence.
3.3
The Ordinary Least Squares (OLS) Estimator
The Ordinary Least Squares (OLS) estimates regression coefficients by minimizing the Sum of Squared Errors
(SSE).
• The error of observation i:
ϵi
=
SSE(β)
=
yi − x′i β
n
X
(yi − x′i β)2
i=1
• The error vector:
ϵ =
SSE(β)
Y − Xβ
=
(Y − Xβ)′ (Y − Xβ)
=
Y ′ Y − Y ′ Xβ − (Xβ)′ Y + (Xβ)′ Xβ
=
Y ′ Y − Y ′ Xβ − β ′ X ′ Y + β ′ X ′ Xβ
15
• First Order Condition (FOC) for minimizing SSE:
dSSE(β)
= −X ′ Y −′ X ′ Y + 2X ′ Xβ
dβ
Equating to 0
−X ′ Y − X ′ Y + 2X ′ X β̂
=
0
2X X β̂
=
2X ′ Y
β̂
=
(X ′ X)−1 X ′ Y
′
• The OLS estimator β̂ is given by (X ′ X)−1 X ′ Y and is a function of sample (X and Y ).
Pn
Pn
• Let Sxx = n1 X ′ X, which is n1 i=1 xi x′i , and Sxy = n1 X ′ Y , which is n1 i=1 xi yi . The OLS estimator
−1
β̂ can be written as Sxx
Sxy
• The FOC is a necessary condition for minimization, and we must check the second order condition to
ensure β̂ achieves the minimum, not the maximum.
d2 SSE(β)
= 2X ′ X > 0
dβ 2
X ′ X is a positive definite (a symmetric matrix with positive eigenvalues).
Other ways to estimate OLS: Method of Moments, MLE etc.
3.4
Algebraic Properties of OLS
These properties follow directly from the FOCs.
• ϵ̂ ≡ Y − Ŷ = Y − X β̂, so the FOC can be written compactly as:
−X ′ Y − X ′ Y + 2X ′ X β̂
=
0
′
=
0
X (Y − X β̂)
=
0
′
=
0
=
0
xi x′i β̂
=
0
xi (yi − x′i β̂)
=
0
xi ϵ̂i
=
0
1
xi ϵ̂i
n i=1
=
0
′
2(−X Y + X X β̂)
′
X ϵ̂
1 ′
X ϵ̂
n
• Alternatively, in terms of xi and ϵ̂i , the FOC is:
−2
n
X
xi yi + 2
i=1
n
X
i=1
n
X
i=1
n
X
i=1
n
X
Pn
Pn
The first element of X is 1 which means n1 i=1 ϵ̂i = 0. Rest of them concludes n1 i=1 xi ϵ̂i . These are the
consequences of the assumption E(ϵ|X) = 0.
3.5
Projection Matrix and Residual Maker Matrix
Define the projection matrix P and the residual maker matrix M , both n × n symmetric matrices:
P = X(X ′ X)−1 X ′ ,
|
{z
}
n×n
M = In − X(X ′ X)−1 X ′ = In − P
|
{z
}
n×n
They satisfy the following properties:
16
Both P and M are symmetric and idempotent.
P X = X,
P Y = Ŷ = X β̂,
P ϵ = ϵ − ϵ̂,
P ϵ̂ = 0
= X(X ′ X)−1 X ′ X
PX
= X
= X(X ′ X)−1 X ′ Y
PY
= X β̂
P ϵ = X(X ′ X)−1 X ′ ϵ
= X(X ′ X)−1 X ′ (Y − Xβ)
= X(X ′ X)−1 X ′ Y − X(X ′ X)−1 X ′ Xβ
= X β̂ − Xβ
= Ŷ − Y + ϵ
= ϵ − ϵ̂
= X(X ′ X)−1 X ′ ϵ̂
P ϵ̂
=
M X = 0,
M Y = ϵ̂,
M ϵ = ϵ̂
0
M ϵ̂ = ϵ̂
MX
=
(In − X(X ′ X)−1 X ′ )X
= X −X
MY
=
0
=
(In − X(X ′ X)−1 X ′ )Y
= Y − X β̂
= ϵ̂
Mϵ
=
(In − X(X ′ X)−1 X ′ )ϵ
= ϵ − ϵ + ϵ̂
= ϵ̂
M ϵ̂
=
(In − X(X ′ X)−1 X ′ )ϵ̂
= ϵ̂
It follows that Y = Ŷ + ϵ̂ = P Y + M Y .
3.6
Interpretation of OLS Regression Coefficients: Partial Effects
Frisch-Waugh-Lovell Theorem: Consider splitting the n × k matrix of explanatory variables X into two
parts: X = (X1 , X2 ), where X1 is an n × (k − g) matrix and X2 is an n × g matrix. Write Y in terms of X1 ,
X2 and estimated coefficients β̂1 and β̂2 as:
Y = X1 β̂1 + X2 β̂2 + ϵ̂
One approach to obtain β̂1 and β̂2 is to run a long regression of Y on X1 and X2 . Another approach to
obtain β̂1 is a two step regression called partial effects.
1. Partial out the effects of X2 from Y and X1 :
(a) Regress Y on X2 to obtain M2 Y
(b) Regress X1 on X2 to obtain M2 X1 (Regress each column of X1 on X2 ).
M2 Y and M2 X are residualized Y and residualized X1 obtained by partialling out the effect of X2 .
2. Estimate the net effect of X1 on Y after partialling out the effects of X2 : Regress M2 Y on M2 X1 .
M2 Y
b̂1
= M2 X1 b̂1 + û
=
((M2 X1 )′ M2 X1 )−1 (M2 X1 )′ M2 Y
=
(X1′ M2′ M2 X1 )−1 X1′ M2′ M2 Y
=
(X1′ M2 X1 )−1 X1′ M2 Y
û = M2 Y − M2 X1 b̂1
17
The Frisch-Waugh-Lovell Theorem states: β̂1 = b̂1 and ϵ̂ = û.
Proof:
Y
=
X1 β̂1 + X2 β̂2 + ϵ̂
M2 Y
=
M2 X1 β̂1 + M2 X2 β̂2 + M2 ϵ̂
X1′ M2 Y
=
X1′ M2 X1 β̂1 + X1′ ϵ̂
β̂1
=
(X1′ M2 X1 )−1 X1′ M2 Y
ϵ̂
= Y − X1 β̂1 − X2 β̂2
M2 ϵ̂
= M2 Y − M2 X1 β̂1
Thus, β̂ = b̂.
ϵ̂
= û
Thus, ϵ̂ = û.
Application 1: Including a Constant Term in a Regression
Including a constant term in a regression gives the same results as expressing all regressors in deviations
from their means. Let X1 = l, a column of ones. Then: M0 = In − l(l′ l)−1 l′ . The residual from regressing
Y on X1 is
M0 Y
=
(In − l(l′ l)−1 l′ )Y
= Y − lȲ


Y1 − Ȳ


=  ... 
Yn − Ȳ
Regressing Y on X1 gives an estimated coefficient Ȳ , so the residuals equal Y1 − Ȳ .
Similarly, regressing each column of X2 on X1 gives an estimated coefficient equal to (l′ l)−1 l′ Y and correspondingly residuals equal to (In − l(l′ l)−1 l′ )X2 .
M0 X2
=
(In − l(l′ l)−1 l′ )X2
= X2 − lX̄2


X21 − X̄2


..
= 

.
X2n − X̄2
Therefore, including a constant term in a regression gives the same results as estimating the regression with
one regressor an intercept: y = β0 + β1 x where
Pn
(y − ȳ)(xi − x̄)
Pn i
β̂1 = i=1
2
i=1 (xi − x̄)
Application 2: Visualizing the Partial Relationship
Let X1 consist of a single regressor which is not constant, so X2 contains the remaining K − 1 variables.
Then one can visualize the partial correlation between y and X1 by plotting the residualized Y against
residualized X1 . This is called partial regression scatter plot.
18
3.7
Goodness of Fit: R2
Define the sum of squares residuals (SSR), total sum of squares (SST) and explained sum of squared (SSE).
SST
=
n
X
(yi − ȳ)2
=
(M0 Y )′ (M0 Y )
=
Y ′ M0′ M0 Y
=
=
Y ′ M0 Y
n
X
(ŷi − ȳ)2
=
(M0 Ŷ )′ (M0 Ŷ )
=
Ŷ ′ M0′ M0 Ŷ
=
Ŷ ′ M0 Ŷ
n
X
(yi − ŷi )2
i=1
SSE
i=1
SSR
=
=
i=1
′
ϵ̂ ϵ̂
Prove SST = SSE + SSR
Y ′ M0 Y
(Ŷ + ϵ̂)′ M0 (Ŷ + ϵ̂)
=
= Ŷ ′ M0 Ŷ + Ŷ ′ M0 ϵ̂ + ϵ̂′ M0 Ŷ + ϵ̂′ M0 ϵ̂
= Ŷ ′ M0 Ŷ + Ŷ ′ ϵ̂ + ϵ̂′ Ŷ + ϵ̂′ ϵ̂
= Ŷ ′ M0 Ŷ + ϵ̂′ ϵ̂
R2 measures the fraction of total variation in the dependent variable that can be explained by the linear
regression.
R2 =
SSR
SSE
=1−
SST
SST
When intercept is added, 0 ≤ R2 ≤ 1.
A high R2 does not mean the model is correct.
A low R2 does not mean the model is wrong.
R2 can never get smaller when one adds more regressors to the regression.
Let’s consider two regressions where we add an unnecessary variable Z
Y
= X b̂ + û
Y
= X β̂ + |{z}
Z γ̂ + ê
n×1
For the true model,
MX Y = û
Applying residual maker of X on the second equation
MX Y
=
MX X β̂ + MX Z γ̂ + MX ϵ̂
=
MX Z γ̂ + ϵ̂
The SSR is given as
û′ û =
(MX Z γ̂ + ϵ̂)′ (MX Z γ̂ + ϵ̂)
= γ̂ZMX MX Z γ̂ + γ̂ZMX ϵ̂ + ϵ̂′ MX Z γ̂ + ϵ̂′ ϵ̂
= γ̂ 2 (MX Z)′ MX Z +ϵ̂′ ϵ̂
|
{z
}
≥0
Thus R2 can never get smaller when one adds more regressors to the regression.
19
Goodness of Fit: Adjusted R2
Definition:
R̄2 = 1 −
SSR/(n − k)
SST /(n − 1)
Explanation: There is a penalty associated with each new independent variable introduced as it uses degrees
of freedom. The new independent variable should compensate the penalty in terms of explained variation in
dependent variable for adjusted R2 to increase.
4
Finite Sample Properties of OLS
4.1
Unbiasedness
Under Assumptions 1-3, E[β̂|X] = β, so OLS is unbiased. Also E[β̂] = β, via law of iterative expectations.
= E[(X ′ X)−1 X ′ Y |X]
E[β̂|X]
=
(X ′ X)−1 X ′ E[Y |X]
=
(X ′ X)−1 X ′ E[Xβ + ϵ|X]
=
(X ′ X)−1 X ′ Xβ + (X ′ X)−1 X ′ E[ϵ|X]
= β
Key to proving Unbiasedness is strict exogeneity (E[ϵ|X] = 0). Proving Unbiasedness does not require
Assumption 4 (homoskedasticy and serial correlation).
4.2
Variance
Under Assumptions 1-4,
V ar[β̂|X]
= V ar[(X ′ X)−1 X ′ Y |X]
=
(X ′ X)−1 X ′ V ar[Y |X]X(X ′ X)−1
=
(X ′ X)−1 X ′ V ar[Xβ + ϵ|X]X(X ′ X)−1
=
(X ′ X)−1 X ′ V ar[ϵ|X]X(X ′ X)−1
= (X ′ X)−1 X ′ σ 2 In X(X ′ X)−1
= σ 2 (X ′ X)−1 X ′ X(X ′ X)−1
= σ 2 (X ′ X)−1
The estimator has σ 2 which is unknown. Thus, an estimator for σ 2 is given as
σ̂ 2 =
ϵ̂′ ϵ̂
n−k
Thus, an estimator for V ar[β̂|X] is
V̂ ar[β̂|X] = σ̂ 2 (X ′ X)−1
Show that σ̂ 2 is an unbiased estimator of σ 2
ϵ̂′ ϵ̂
ϵ′ M ϵ
E[
|X] = E[
|X]
n−k
n−k
1
=
E[ϵ′ M ϵ |X]
n − k | {z }
1×1
=
=
=
=
1
E[tr(ϵ′ M ϵ)|X]
n−k
Applicable if the product is scalar. tr(AB) = tr(BA)
1
E[tr(M ϵϵ′ )|X]
n−k
1
tr(E[M ϵϵ′ |X])
n−k
Both trace and expectations are linear operator
1
tr(M σ 2 )
n−k
20
=
=
=
=
=
=
=
4.3
1
tr(M σ 2 )
n−k
σ2
tr(In − X(X ′ X)−1 X ′ ))
n−k
σ2
(n − tr(X(X ′ X)−1 X ′ ))
n−k
σ2
(n − tr(X ′ X(X ′ X)−1 ))
n−k
σ2
(n − tr(Ik ))
n−k
σ2
(n − k)
n−k
σ2
Gauss-Markov Theorem
Under Assumptions 1 − 4, the OLS estimator is the best linear unbiased estimator which means in the class
of linear estimators, it has the lowest variance. Linear estimator:
β̂
= (X ′ X)−1 X ′ Y
= WY
where W = (X ′ X)−1 X. Each element of β̂ is the weighted average of yi . Example, let the model be
yi = a + bxi + ϵi ∀ i = 1, . . . , n.
The estimator will take the form
b̂ =
=
=
=
=
Pn
(x − x̄)(yi − ȳ)
i=1
Pn i
(xi − x̄)2
Pn i=1
Pn
(xi − x̄)yi
(xi − x̄)ȳ
i=1
Pn
− Pi=1
n
2
2
(x
−
x̄)
i=1 i
i=1 (xi − x̄)
P
P
n
n
n
X
ȳ( i=1 xi − i=1 x̄)
Pn
wi yi −
2
i=1 (xi − x̄)
i=1
n
X
ȳ(nx̄ − nx̄)
wi yi − Pn
2
i=1 (xi − x̄)
i=1
n
X
wi yi
i=1
Interpretation of Gauss Markov Theorem: Let β̂ be OLS estimator and β̃ be any other linear unbiased
estimator of β. Then G-M Theorem states that
V ar(β̃|X) − V ar(β̂|X) > 0
4.4
Normality of OLS in Finite Samples
Normality of OLS in finite sample requires the following assumption:
ϵ|X ∼ N (0, σ 2 In )
The OLS estimator follows a normal distribution given the above assumption:
β̂
= β + (X ′ X)−1 X ′ ϵ
ϵ|X
∼ N (0, σ 2 In )
Since,
(X ′ X)−1 X ′ ϵ|X
∼ N (0, (X ′ X)−1 X ′ σ 2 In X(X ′ X)−1 )
∼ N (0, σ 2 (X ′ X)−1 )
β + (X ′ X)−1 X ′ ϵ|X
β̂|X
∼ N (β, σ 2 (X ′ X)−1 )
∼
N (β, σ 2 (X ′ X)−1 )
21


 ′
β1
(X1 M−1 X1 )−1
 ..  2 
 . ,σ 

β̂1
 .. 
 .  |X = N

β̂k
βk

..
.
(Xk′ M−k Xk )−1
!


where M−i is the residual maker for all X except i. Therefore,
β̂k ∼ N (βk , σ 2 (Xk′ M−k Xk )−1 )
Let Vk = (Xk′ M−k Xk )−1 ,
β̂k − βk
√
σ 2 Vk
∼ N (0, 1)
β̂k − βk
√
σ̂ 2 Vk
N (0, 1)
∼ tn−k = q 2
χn−k
n−k
Proof:
β̂k − βk
√
σ̂ 2 Vk
=
β̂k −βk
√
σ 2 Vk
√
2
√σ̂ Vk
σ 2 Vk
where,
β̂k − βk
√
σ 2 Vk
√
σ̂ 2 Vk
√
σ 2 Vk
4.4.1
∼ N (0, 1)
r
σ̂ 2
=
σ2
r
χ2
∼
n−k
Hypothesis Testing
The finite sample normality of β̂ can be useful in performing hypothesis testing in finite samples. Hypothesis
is a mathematical proposition. Null hypothesis (H0 ) is the proposition that we assume to be true at the
start of the analysis. Alternative hypothesis (HA ) are the potential proposition apart from H0 . How to test
hypothesis?
Let’s assume that for the linear regression we considered, we are interesting in testing the following hypothesis.
Null Hypothesis: H0 : βj = βj0 = 0 (the independent variable has no effect)
Alternative Hypothesis: HA : βj ̸= 0 (the independent variable has some significant effect)
The aim is to see whether the sample provides enough evidence against H0 (for HA ) or not. The following
steps are what we follow to perform hypothesis testing.
1. Choose a critical value (tα,n−k /tα/2,n−k /zα /zα/2 ) or significance level (α) which defines what is sufficient evidence against H0 . Here, tα/2,n−k and zα/2 are the critical value from the t-distribution and
standard normal distribution respectively under two sided hypothesis testing and tα,n−k and zα are
the critical value from the t-distribution and standard normal distribution respectively under one sided
hypothesis testing.
2. Use the sample to estimate βj .
3. Calculate some evidence statistic (t/z statistic, CI, p-value) which quantifies the evidence as the distance between estimated value and H0 value.
4. Compare the evidence statistic with critical value and make a decision to either reject or fail to reject
H0
22
We never accept H0 because the evidence is always against H0 and not in favor of.
Test statistic:
t =
β̂j − βj0
se(
ˆ β̂j )
Or
z
=
β̂j − βj0
se(β̂j )
where, β̂j is the estimated coefficient, ŝe(β̂j ) is the estimated value of standard error of β̂j and se(β̂j ) is the
true value of standard error of β̂j . The z/t statistic captures the distance between the estimated coefficient
and H0 value in terms of standard error. Greater the distance, more is the evidence to reject Null.
In two way hypothesis testing, if |t| > tα/2,n−k or |z| > zα/2 , we reject H0 , meaning the predictor is
statistically significant. Else we fail to reject H0 .
In one way hypothesis testing where HA > 0, if t > tα,n−k or z > zα , we reject H0 , meaning the
predictor is statistically significant. Else we fail to reject H0 .
In one way hypothesis testing where HA < 0, if t < tα,n−k or z < zα , we reject H0 , meaning the
predictor is statistically significant. Else we fail to reject H0 .
Confidence Interval:
A (1 − α) × 100% confidence interval for βj for two sided hypothesis testing is:
β̂j ± tα/2,n−k · se(
ˆ β̂j )
Or
β̂j ± zα/2 · se(β̂j )
The confidence interval comments on the probability of interval containing β̂j . If the interval contains H0
which is generally βj = 0, we do not have enough evidence to conclude that βj is significantly different from
zero.
p-Value:
The p-value tells us the probability of obtaining a test statistic at least as extreme as the observed one under
H0 . It can also be interpreted as the probability of obtaining the sample at least as extreme as the one
observed given H0 was true. For two sided, it is computed as:
p = 2 · P (T > |tobs |)
Or
p = 2 · P (Z > |zobs |)
If p ≤ α, reject H0 .
If p > α, fail to reject H0 .
5
Asymptotic Properties of OLS
• Asymptotic properties of OLS describe the behavior of the estimator as n → ∞.
• Asymptotic results can be established under much weaker assumptions than those needed for finite
sample properties.
• In large samples, these asymptotic properties provide a good approximation of the behavior of the
estimator.
23
5.1
Consistency
The following variants of the expression for β̂ will be useful.
β̂
=
(X ′ X)−1 X ′ Y
=
(X ′ X)−1 X ′ (Xβ + ϵ)
= β + (X ′ X)−1 X ′ ϵ
1
1
= β + ( X ′ X)−1 X ′ ϵ
n
n
n
n
1X
1X
= β+(
xi x′i )−1 (
x i ϵi )
n i=1
n i=1
1
Example, when K = 2, xi =
, we have:
zi
n
n 1X 1 1 zi
=
n i=1 zi
n 1 X 1 zi
=
n i=1 zi zi2
n 1 X 1 zi
=
n i=1 zi zi2
1 Pn
Pn
1
n P i=1 1
n P i=1 zi
=
n
n
1
1
2
i=1 zi
i=1 zi
n
n
1
Pnz̄
=
z̄ n1 i=1 zi2
1X
xi x′i
n i=1
n
1X
x i ϵi
n i=1
=
=
=
n 1X 1
ϵi
n i=1 zi
n 1 X ϵi
n i=1 zi ϵi
ϵ̄
P
n
1
i=1 zi ϵi
n
Consistency can be established under general conditions as long as E(xi ϵi ) = 0 which is weaker than
E(ϵi |xi ) = 0.
β̂
plimβ̂
n
n
1X
1X
xi x′i )−1 (
x i ϵi )
n i=1
n i=1
=
β+(
=
n
n
X
1X
′ −1 1
plim β + (
xi xi ) (
x i ϵi )
n i=1
n i=1
=
n
n
1X
1X
plimβ + plim (
xi x′i )−1 (
x i ϵi )
n i=1
n i=1
=
β + (plim
n
n
1X
1X
xi x′i )−1 (plim
x i ϵi )
n i=1
n i=1
By WLLN
=
β + E(xi x′i )−1 E(xi ϵi )
Given E(xi ϵi ) = 0
=
5.2
β
Asymptotic Normality
If the data are well behaved (meaning that the Central Limit Theorem (CLT) applies), then in large samples,
normality of OLS does not depend on normality of the unobservable ϵ; it is a consequence of CLT. We have
24
shown that:
plim β̂ = β,
so as n → ∞, the distribution of β̂ has a mass point at β. Now, rescale the distribution and consider
√
√
1
1
n(β + ( X ′ X)−1 X ′ ϵ − β)
n
n
√ 1 ′ −1 1 ′
=
n( X X)
Xϵ
n
n
√ −1 1 ′
nQ
=
Xϵ
n
√
√
The asymptotic distribution or n(β̂ − β) depends on n n1 X ′ ϵ and Q−1 . Assuming some version of CLT
works, we have:
√ 1 ′
1
d
n( X ϵ − E( X ′ ϵ)) −
→ N (0, V ).
n
n
Assume E(X ′ ϵ) = 0 from the previous section holds.
n(β̂ − β)
=
√ 1 ′ d
→ N (0, V ).
n Xϵ−
n
Assume Var(ϵ|X) = σ 2 In which means homoskedasticty and no serial correlation.
√ 1 ′
√ 1
1√
n Xϵ
n)
V ar
= E( n X ′ ϵϵ′ X
n
n
n
1
=
E(X ′ ϵϵ′ X)
n
= Using LIE
1
E(E(X ′ ϵϵ′ X|X))
=
n
1
=
E(X ′ E(ϵϵ′ |X)X)
n
1
=
E(X ′ V ar(ϵ|X)X)
n
1
=
E(X ′ σ 2 In X)
n
1
= σ 2 E( X ′ X)
n
= σ 2 E(Q)
We can define E(Q) as
E(Q)
=
=
1
E( X ′ X)
n
n
1X
E(
xi x′i )
n i=1
n
=
1X
E(xi x′i )
n i=1
Given i.i.d observations
=
Thus, V ar
√
E(xi x′i )
n n1 X ′ ϵ = σ 2 E(xi x′i ). By CLT:
√
n·
1 ′ d
Xϵ−
→ N (0, σ 2 E(xi x′i ).
n
By the Law of Large Numbers (LLN):
1 ′ p
XX−
→ E(Q) = E(xi x′i ).
n
By the Continuous Mapping Theorem (CMT):
−1
1 ′
p
−
→ E(xi x′i )−1 .
XX
n
25
We assume that E(xi x′i ) is a constant. Applying Slutsky’s Theorem:
√
n(β̂ − β)
=
1 ′
XX
n
−1
·
√
n·
1 ′
Xϵ
n
d
−
→ E(xi x′i )−1 N (0, σ 2 E(xi x′i )).
d
−
→ N (0, σ 2 E(xi x′i )−1 ).
6
Specification Issues
6.1
Omitted Variable Bias
Assume the true model is:
Y
=
Xβ + ϵ
=
X1 β1 + X2 β2 + ϵ
where all the classical linear model assumptions hold.
Regress y on X1 only, the estimator is:
β˜1 = (X1′ X1 )−1 X1′ Y
E[β˜1 |X]
= E[(X1′ X1 )−1 X1′ Y |X]
=
(X1′ X1 )−1 X1′ E[Y |X]
=
(X1′ X1 )−1 X1′ E[X1 β1 + X2 β2 + ϵ|X]
= β1 + (X1′ X1 )−1 X1′ X2 β2 + (X1′ X1 )−1 X1′ E[ϵ|X]
= β1 + (X1′ X1 )−1 X1′ X2 β2
The omitted variable bias is then (X1′ X1 )−1 X1′ X2 β2 which depends on (X1′ X1 )−1 X1′ X2 which means how
omitted variable is associated with the independent variable and β2 which is the effect of omitted variable
on dependent variable.
6.2
Including Irrelevant Covariates
Suppose the true model is correctly given by:
Y = X1 β1 + ϵ.
The estimated value from short regression is
β̂1 = (X1′ X1 )−1 X1′ Y
It is unbiased and has minimum variance is σ 2 (X1′ X1 )−1 because of GM Theorem. Including irrelevant
covariates X2 means β2 = 0 leads to a long regression as
Y = X1 β1 + X2 β2 + ϵ
The estimated value from long regression is
β̃1 = (X1′ M2 X1 )−1 X1′ M2 Y
Check for Unbiasedness
E[β̃1 |X]
=
E[(X1′ M2 X1 )−1 X1′ M2 Y |X]
=
(X1′ M2 X1 )−1 X1′ M2 E[Y |X]
=
(X1′ M2 X1 )−1 X1′ M2 E[X1 β1 + ϵ|X]
=
β1 + (X1′ M2 X1 )−1 X1′ M2 E[ϵ|X]
=
β1
26
Calculate Variance
V ar[β̃1 |X]
=
V ar[(X1′ M2 X1 )−1 X1′ M2 Y |X]
=
V ar[β1 + (X1′ M2 X1 )−1 X1′ M2 ϵ|X]
=
(X1′ M2 X1 )−1 X1′ M2 V ar[ϵ|X]M2 X1 (X1′ M2 X1 )−1
=
(X1′ M2 X1 )−1 X1′ M2 σ 2 In X1 (X1′ M2 X1 )−1
=
σ 2 (X1′ M2 X1 )−1 X1′ M2 X1 (X1′ M2 X1 )−1
=
σ 2 (X1′ M2 X1 )−1
So, when we evaluate it
X1′ M2 X1
=
(X1′ (In − P2 )X1 )
= X1′ X1 − X1 P2 X1
= X1′ X1 − X1′ P2′ P2 X1
= X1′ X1 − (P2 X1 )′ P2 X1
|
{z
}
≥0
Therefore,
X1′ M2 X1
′
(X1 M2 X1 )−1
σ 2 (X1′ M2 X1 )−1
≤ X1′ X1
≥ (X1′ X1 )−1
≥ σ 2 (X1′ X1 )−1
V ar(β̃) ≥ V ar(β̂)
So adding unnecessary variables increase the variance of the estimator as it consumes more degrees of freedom
to estimate β2 and less is left over for β1 .
6.3
Multicollinearity
We concentrate on the single parameter βK in the following model:
Y = X1 β1 + XK βK + ϵ,
where X1 is n × (K − 1) and XK is n × 1 dimensional.
βˆK
′
′
(XK
M1 XK )−1 (XK
M1 Y )
=
E[βˆK |X] = β1
′
V ar[βˆK |X] = σ 2 (XK
M1 XK )−1
When XK is an exact linear function of other explanatory variables, then M1 XK = 0 leading to V ar(βˆK |X) =
∞. This is why with perfect collinearity, OLS does not work. Multicollinearity means M1 XK is very small
(Since it is the residual from XK once X1 is partialled out), thus leading to high estimated values of
V ar(βˆK |X) and low test statistics. It makes rejecting H0 harder.
6.4
Non Homoskedasticity
Consider the model:
y = Xβ + ϵ.
Suppose that the classical linear model Assumptions 1-3 hold, but:
Var[ϵ|X]
=
Σ
E[β̂|X]
=
β
β̂OLS is still unbiased, but the variance for β̂OLS is incorrect. Under Assumption 4 (homoskedasticity and
no serial correlation):
Σ
=
σ 2 In
V ar[β̂|X]
=
σ 2 (X ′ X)−1
27
Under heteroskedasticity (HS) and/or autocorrelation (AC):
V ar[β̂|X]
6.4.1
=
V ar[ˆ(X ′ X)−1 X ′ Y |X]
=
(X ′ X)−1 X ′ V ar[Y |X]X(X ′ X)−1
=
(X ′ X)−1 X ′ V ar[Xβ + ϵ|X]X(X ′ X)−1
=
(X ′ X)−1 X ′ V ar[ϵ|X]X(X ′ X)−1
=
(X ′ X)−1 X ′ ΣX(X ′ X)−1
Heteroskedasticity and No Serial Correlation
With heteroskedasticity, V ar[ϵ|X] = E[ϵϵ′ |X] = Σ where
 2
σ1

..
Σ = 
where σ12 , . . . , σn2 are unknown.
White Standard Errors: σi2 = ϵ̂2i . Which means
 2
ϵ̂1

..
Σ̂w = 
.

.
σn2



ϵ̂2n


Thus,
V̂ ar[β̂|X]
where X ′ Σ̂w X =
6.4.2
=
(X ′ X)−1 X ′ Σ̂w X(X ′ X)−1
Pn
′ 2
i=1 xi xi ϵ̂i . This is how Heteroskedasticity robust standard errors are obtained.
Heteroskedasticity and Serial Correlation
With heteroskedasticity, V ar[ϵ|X] = E[ϵϵ′ |X] = Σ where
 2
σ1 σ12
 σ21 σ22

Σ =  .
..
 ..
.
2
σn1
σn2

. . . σ1n
. . . σ2n 

.. 
..
.
. 
. . . σn2
Generally in the context of time series. Modeling of Σ is required.
7
Generalized Least Square
Consider the model:
Y = Xβ + ϵ,
with V ar(ϵ|X) = Σ ̸= σ 2 In
Assume other classical linear model assumptions still hold. OLS in this case is still unbiased but not the
most efficient estimator. So
Option 1: Apply OLS with white standard errors.
Option 2: Find a more efficient estimator. Basic Idea is to transform the model such that all 4 GM
assumptions hold
Suppose Ω is some n × n symmetric matrix such that in the weighted regression:
ΩY = ΩXβ + Ωϵ, V ar(Ωϵ|X) = σ 2 In
28
For simplicity, we will find Ω such that V ar(Ωϵ|X) = σ 2 In is true.
V ar(Ωϵ|X)
= σ 2 In
′
E(Ωϵϵ Ω |X)
= σ 2 In
ΩE(ϵϵ′ |X)Ω′
= σ 2 In
ΩV ar(ϵ|X)Ω′
= σ 2 In
ΩΣΩ′
= σ 2 In
′
=
Ω
−1
′
ΩΣΩ Ω
′−1
Ω−1 σ 2 In Ω′−1
Σ = σ 2 Ω−1 Ω′−1
Σ = σ 2 (Ω′ Ω)−1
1
σΣ− 2
=
Ω
We can treat σ as the scaling factor. Double check: when Ω = σΣ−1/2 ,
V ar(Ωϵ|X)
= V ar(σΣ−1/2 ϵ|X)
= σ 2 Σ−1/2 V ar(ϵ|X)Σ−1/2
= σ 2 Σ−1/2 ΣΣ−1/2
= σ 2 Σ−1/2 Σ1/2 Σ1/2 Σ−1/2
= σ 2 In
7.1
Estimation
First weight the original regression as
Σ−1/2 Y
=
Σ−1/2 Xβ + Σ−1/2 ϵ
σ is subsumed in Σ−1/2 . Then, estimate the new regression using OLS.
β̂GLS
=
((Σ−1/2 X)′ Σ−1/2 X)−1 (Σ−1/2 X)′ Σ−1/2 Y
β̂GLS
=
(X ′ Σ−1/2 Σ−1/2 X)−1 X ′ Σ−1/2 Σ−1/2 Y
=
(X ′ Σ−1 X)−1 X ′ Σ−1 Y
GLS is just OLS applied to a transformed regression where all classical regression model assumptions hold.
1. GLS is BLUE
2. GLS is consistent and asymptotically normal under general conditions of LLN and CLT.
7.2
Properties of GLS
7.2.1
Unbiasedness
E(β̂GLS |X)
= E[(X ′ Σ−1 X)−1 X ′ Σ−1 Y |X]
= E[(X ′ Σ−1 X)−1 X ′ Σ−1 (Xβ + ϵ)|X]
= β + E[(X ′ Σ−1 X)−1 X ′ Σ−1 ϵ|X]
= β + (X ′ Σ−1 X)−1 X ′ Σ−1 E[ϵ|X]
= β
7.2.2
Minimum Variance
GLS is OLS applied to the transformed model, assuming all classical linear model assumptions hold. Then
GLS is BLUE under Gauss Markov Theorem. It also means V ar[β̂GLS |X] ≤ V ar[β̂OLS |X]. where
V ar[β̂GLS |X]
= V ar[(X ′ Σ−1 X)−1 X ′ Σ−1 Y |X]
=
(X ′ Σ−1 X)−1 X ′ Σ−1 V ar[Y |X]Σ−1 X(X ′ Σ−1 X)−1
=
(X ′ Σ−1 X)−1 X ′ Σ−1 V ar[Xβ + ϵ|X]Σ−1 X(X ′ Σ−1 X)−1
=
(X ′ Σ−1 X)−1 X ′ Σ−1 V ar[ϵ|X]Σ−1 X(X ′ Σ−1 X)−1
=
(X ′ Σ−1 X)−1 X ′ Σ−1 ΣΣ−1 X(X ′ Σ−1 X)−1
=
(X ′ Σ−1 X)−1 X ′ Σ−1 X(X ′ Σ−1 X)−1
=
(X ′ Σ−1 X)−1
29
7.2.3
Consistent
Writing GLS estimator in the following form
β̂GLS
=
(X ′ Σ−1 X)−1 X ′ Σ−1 Y
=
(X ′ Σ−1 X)−1 X ′ Σ−1 (Xβ + ϵ)
= β + (X ′ Σ−1 X)−1 X ′ Σ−1 ϵ
1
1
= β + ( X ′ Σ−1 X)−1 X ′ Σ−1 ϵ
n
n
Checking convergence in probability
1
1
= plim β + ( X ′ Σ−1 X)−1 X ′ Σ−1 ϵ)
n
n
1
1
′ −1
= plimβ + plim ( X Σ X)−1 X ′ Σ−1 ϵ
n
n
1 ′ −1 −1
1
= β + (plim( X Σ X) )(plim X ′ Σ−1 ϵ)
n
n
By WLLN
1
1
= β + E( X ′ Σ−1 X)−1 E( X ′ Σ−1 ϵ)
n
n
Using LIE
1
1
= β + E( X ′ Σ−1 X)−1 E(E[ X ′ Σ−1 ϵ|X])
n
n
1
1
= β + E( X ′ Σ−1 X)−1 E( X ′ Σ−1 E[ϵ|X])
n
n
1
1
= β + E( X ′ Σ−1 X)−1 E( X ′ Σ−1 0)
n
n
= β
plimβ̂
7.2.4
Asymptotic Normality
Given
β̂GLS
=
1
1
β + ( X ′ Σ−1 X)−1 X ′ Σ−1 ϵ
n
n
The asymptotic behavior of β̂GLS −β depends on n1 X ′ Σ−1 ϵ. We know because of strict exogeneity E( n1 X ′ Σ−1 ϵ) =
0 then under CLT, we have:
√
√ 1
1
d
n( X ′ Σ−1 ϵ) −
→ N (0, V ar( n( X ′ Σ−1 ϵ))).
n
n
Assume Var(ϵ|X) = Σ which means homoskedasticity and no serial correlation.
√ 1
V ar( n( X ′ Σ−1 ϵ))
n
√ 1
1√
n)
= E( n X ′ Σ−1 ϵϵ′ Σ−1 X
n
n
1
=
E(X ′ Σ−1 ϵϵ′ Σ−1 X)
n
= Using LIE
1
=
E(E(X ′ Σ−1 ϵϵ′ Σ−1 X|X))
n
1
=
E(X ′ Σ−1 E(ϵϵ′ |X)Σ−1 X)
n
1
=
E(X ′ Σ−1 V ar(ϵ|X)Σ−1 X)
n
1
=
E(X ′ Σ−1 ΣΣ−1 X)
n
1
= E( X ′ Σ−1 X)
n
√
Thus, V ar( n( n1 X ′ Σ−1 ϵ)) = E( n1 X ′ Σ−1 X). By CLT:
√
1
1
d
n( X ′ Σ−1 ϵ) −
→ N (0, E( X ′ Σ−1 X)).
n
n
30
By the Law of Large Numbers (LLN):
1 ′ −1 p
1
XΣ X−
→ E( X ′ Σ−1 X).
n
n
By the Continuous Mapping Theorem (CMT):
1 ′ −1
XΣ X
n
−1
1
p
−
→ E( X ′ Σ−1 X)−1 .
n
We assume that E( n1 X ′ Σ−1 X) is a constant. Applying Slutsky’s Theorem:
√
7.3
1
1
d
n(β̂GLS − β) −
→ E( X ′ Σ−1 X)−1 N (0, E( X ′ Σ−1 X)).
n
n
1
d
−
→ N (0, E( X ′ Σ−1 X)−1 ).
n
Feasible GLS
GLS works only when V ar(u|X) = Σ is known. When Σ is unknown, we need to estimate it first. This leads
to Feasible GLS.
β̂F GLS = (X ′ Σ̂−1 X)−1 X ′ Σ̂−1 Y
parameters to estimate
Note that Σ is an n × n symmetric matrix, so for an unrestricted Σ, there are n(n+1)
2
with only n data points. We usually impose some structure so there are only a few parameters to estimate.
Example: AR(1) error: Suppose ϵt = ρϵt−1 + ut , t = 1, . . . , T and V ar(ϵt ) = V ar(ϵt−1 ) = σ 2 . ut is a
random shock. Then
Cov(ϵt , ϵt − 1)
Cov(ϵt , ϵt−2 )
=
Cov(ρϵt−1 + ut , ϵt−1 )
=
ρCov(ϵt−1 , ϵt−1 )
=
ρV ar(ϵt−1 )
=
ρσ 2
= Cov(ρϵt−1 + ut , ϵt−2 )
= Cov(ρ(ρϵt−2 + ut−1 + ut , ϵt−2 )
= Cov(ρ2 ϵt−2 + ρut−1 + ut , ϵt−2 )
= Cov(ρ2 ϵt−2 , ϵt−2 )
= ρ2 Cov(ϵt−2 , ϵt−2 )
= ρ2 V ar(ϵt−2 )
= ρ2 σ 2
Thus Cov(ϵt , ϵt − p) = ρp σ 2 The var-cov matrix is defined as


1 ρ ρ2 . . .
 ρ 1 ρ . . .


Σ = σ 2 ρ2 ρ 1 . . .


.. .. .. . .
.
. . .
Estimating Σ involves estimating a single parameter ρ which can be obtained through OLS.
The following procedure is used to estimate FGLS if no structure is imposed.
1. Estimate OLS
2. Estimate Σ̂ using residuals given as

ϵ̂21
 ϵ̂2 ϵ̂1

Σ̂ =  .
 ..
ϵ̂1 ϵ̂2
ϵ̂21
..
.
...
...
..
.
ϵ̂n ϵ̂1
ϵ̂n ϵ̂2
...
31

ϵ̂1 ϵ̂n
ϵ̂2 ϵ̂n 

.. 
. 
ϵ̂2n
3. Estimate FGLS using the formula
β̂F GLS = (X ′ Σ̂−1 X)−1 X ′ Σ̂−1 Y
which is applying OLS to the transformed regression
Σ̂−1/2 Y
= Σ̂−1/2 Xβ + Σ̂−1/2 ϵ
4. Repeat Step 2 and 3 untill β̂F GLS is stable.
β̂F GLS is asymptotically equivalent to β̂GLS if
1
1
plim( X ′ Σ̂−1 X − X ′ Σ−1 X) = 0
n
n
and
1
1
plim( X ′ Σ̂−1 ϵ − X ′ Σ−1 ϵ) = 0
n
n
Difficult to comment on the small sample properties.
7.4
Weighted Least Squares
The special case of GLS in the case of heteroskedasticity only is called weighted least squares. All G-M
assumptions hold except for Assumption 4 — in particular, assume:
= σi2 , ∀ i = 1, . . . , n
V ar(ϵi |X)
Cov(ϵi ϵj |X)
=
Example: Assume
yi = α + βxi + γzi + ϵi ,
0,
∀ i ̸= j.
σi2 = σ 2 wi2 ,
i = 1, . . . , n.
wi can be xi or some other regressor not included in the model or n1k where nk is the group size for the k th
group where k = 1, . . . , K.
The constant σ 2 does not matter, so we can first weight the regression as
yi
wi
yi∗
α
βxi
γzi
ϵi
+
+
+
wi
wi
wi
wi
= αi∗ + βx∗i + γzi∗ + ϵ∗i
=
Then, we can estimate the weighted regression by using OLS on the transformed model. This is called
Weighted Least Squares (WLS). It is a special case of GLS when
 1
w1
1

Σ− 2 = 
Example: Consider:
Y = Xβ + ϵ,
Then:

..
.
1
wn


σi2 = exp(x′i γ)σ 2 .

exp(x′1 γ)

Σ = Var(ϵ|X) = σ 2 

..
.
exp(x′n γ)
The constant does not matter so let:
√
1
exp(x′1 γ)

1
Ω = Σ− 2 = 


..
.
√
32
1
exp(x′n γ)

.


.
The weighted regression involves:
√
ΩY
1
exp(x′1 γ)
..

= 


= 

..
.
yn
exp(x′n γ)
√
1
exp(x′1 γ)




..

= 

x1
exp(x′1 γ)
..
.
√ xn1 ′
exp(xn γ)
√
Ωϵ
1
exp(x′1 γ)
√
xn1
√ x1k ′

..
.
...
..
.



...
..
.

x1k
.. 
. 
...
xnk
exp(x1 γ)
√ xnk ′
exp(xn γ)
.
√
√
1
exp(x′n γ)
x11
 .
 .
 .
...
..

= 


= 

.
√
√
1
exp(x′n γ)

y1
 . 
 . 
 .
yn

y1
exp(x′1 γ)
√

= 

.
√
√
ΩX

1
exp(x′n γ)
 
ϵ1
 . 
 . 
 .
ϵn

ϵ1
exp(x′1 γ)
..
.
ϵn
exp(x′n γ)



The weighted regression is then
p
yi
exp(x′i γ)
=
x
x
ϵ
p i1β1 ′ + . . . + p ikβk ′ + p i ′
exp(xi γ)
exp(xi γ)
exp(xi γ)
If γ is known, we can estimate the above by by running OLS on the transformed regression line. Typically
γ is unknown and needs to be estimated first.
σi2 = σ 2 exp(x′i γ) =⇒ log(σi2 ) = log(σ 2 ) + x′i γ.
1. Run OLS on the non transformed regression and obtain ϵ̂.
2. Estimate log(ϵ̂2i ) = α + x′i γ + vi and obtain γ̂.
3. Estimate β̂ using √
1
as weights for ith observation.
exp(x′i γ̂)
4. Estimate ϵ̂ again and repeat step 2 and 3 until β̂ is stable.
7.5
White Test
White test is a general test for heteroskedasticity. Clearer to consider a concrete example. Assume:
yi = β0 + β1 xi + β2 zi + ϵi .
1. Estimate the original regression model and obtain ϵ̂i .
2. Square the residuals and regress them on the original independent variables, their squares, and their
cross-products as follows
ϵ̂2i = γ0 + γ1 xi + γ2 zi + γ3 x2i + γ4 zi2 + γ5 xi zi + ui
3. Test the hypothesis that H0 : γ1 = γ2 = . . . γ5 = 0.
33
The test statistic is nR2 from the auxiliary regression. With p regressors in the auxiliary regression, the
test follows a chi-square distribution with p degrees of freedom. If homoskedasticity is rejected, one can use
robust standard errors or transform the model. Note that rejecting the null implies heteroskedasticity is
present, but failing to reject the null does not confirm homoskedasticity. White test has low power. The
power of a test is the probability of rejecting the null hypothesis when it is false (1 − P (Type II error)).
8
Maximum Likelihood Estimation (MLE)
We will start with two examples which involve probabilistic modeling.
Motivating Example 1: Let y be the outcome of flipping a coin:
(
1, if head
yi =
0, otherwise
Assume that yi comes from a Bernoulli distribution and that we are interested in P (yi = 1) = p. The
probability mass function (PMF) of Bernoulli is given as
f (yi |p) = py (1 − p)1−y
Suppose one has a random sample {y1 , . . . , y100 }, where 100 flips yield 99 heads and 1 tail.
What is an estimate of p? In this example, p is the probability of observing a head and (1 − p) is the
probability of observing a tail. The joint probability of observing 99 heads and 1 tail is:
f (y1 , . . . , y100 |p)
=
100
Y
pyi (1 − p)1−yi
i=1
= p99 (1 − p)1
p is the parameter of the underlying distribution function. The estimation principle: Maximum Likelihood
Estimation (MLE) aims to choose p that maximizes the likelihood function.
What is a likelihood function?
L(p|y1 , . . . , y100 )
=
100
Y
pyi (1 − p)1−yi
i=1
99
= p (1 − p)1
= p99 − p100
The joint probability function is interpreted as the likelihood function if we consider data as given and
maximize with respect to the parameter.
∂L
99
= 99p98
mle − 100pmle
∂p
99p98
mle
99
pmle
∂2L
∂p2
=
0
=
100p99
mle
=
100pmle
99
=
100
=
(99)(98)p97 − (100)(99)p98
=
99p97 (98 − 100p)
=
99(0.99)97 (98 − 99)
=
−99(0.99)97 < 0
Substituting value of pmle
Hence pmle = 0.99 is the maximum.
Motivating Example 2: Observe a random sample {yi }ni=1 of grades of the students. We model the grades
as a normal distribution with unknown mean and variance, i.e.,
yi ∼ N (µ, σ 2 ),
34
i = 1, . . . , n.
How to estimate µ and σ 2 based on these observations? Since y is continuous, pdf of observing yi is:
1
(yi − µ)2
f (yi |µ, σ 2 ) = √
.
exp −
2σ 2
2πσ 2
The joint pdf, or the likelihood function of observing such data, is:
n
Y
1
(yi − µ)2
√
L(µ, σ 2 |y1 , . . . , yn ) =
.
exp −
2σ 2
2πσ 2
i=1
Taking the logarithm of the likelihood function (log-likelihood):
log L(µ, σ 2 |y1 , . . . , yn ) = −
n
n
1 X
log(2πσ 2 ) − 2
(yi − µ)2 .
2
2σ i=1
The MLE estimates for µ and σ 2 maximize this log-likelihood function.
n
∂ log L
1 X
=2 2
(yi − µmle )
∂µ
2σ i=1
=
0
µmle
=
1X
yi
n i=1
=
0
=
1X
(yi − µ)2
n i=1
n
n
X
n
1
∂L
=
−
+
(yi − µ)2
2
2 )2
∂σ 2
2σmle
2(σmle
i=1
n
2
σmle
2
2
is biased but consistent estimator. Check the SOC.
. σmle
Substitute µmle for µ in the estimator of σmle
Motivating Example 3: Observe the number of students who enter late in the class. We treat them as
independent Poisson random variables with EX (Xi ) = λ. The PMF is given as
fX (xi |λ) =
λxi e−λ
,
xi !
x = 1, 2, . . .
The likelihood function L(λ|x1 , . . . , xn ) is as follows.
L(λ|x1 , . . . , xn )
=
n
Y
λxi e−λ
i=1
xi !
We can estimate the maximum likelihood estimator for λ as.
logL =
n
X
λxn e−λ
)
xi !
log(
i=1
=
=
n
X
i=1
n
X
(xi log(λ) − λ − log(xi !))
xi log(λ) − nλ −
i=1
∂logL
∂λ
0
i=1
Differentiate
n
X
1
=
xi − n
λ
i=1
Equate to 0
n
X
1
=
xi
−n
λmle
i=1
n
λmle
∂ 2 logL
∂λ2
=
1X
xi
n i=1
= −
n
X
n
X
i=1
xi
1
<0
λ2
35
log(xi !)
8.1
General Approach
The likelihood of the ith observation is the probability density (for continuous yi ) or probability mass function
(for discrete yi ), given by:
f (yi |θ)
For an i.i.d. sample y1 , . . . , yn , the joint density is:
f (y1 , . . . , yn |θ) =
n
Y
f (yi |θ).
i=1
which is also the likelihood function
L(θ|y1 , . . . , yn ) =
n
Y
f (yi |θ).
i=1
Taking the log-likelihood function:
log L(θ|y1 , . . . , yn ) =
n
X
log f (yi |θ).
i=1
The MLE estimate θ̂M LE is the value that maximizes log L(θ). The first-order condition (FOC) is:
∂ log L(θ)
= 0.
∂θ
The second-order condition (SOC) satisfies:
∂ 2 log L(θ)
≤ 0.
∂θ2
8.2
Linear Regression
Consider the linear regression model:
yi = x′i β + ϵi ,
ϵi |xi ∼ N (0, σ 2 ).
Given a random sample {yi , xi }ni=1 , we estimate the parameters β and σ 2 using MLE. Since yi |xi ∼
N (x′i β, σ 2 ), the likelihood function is:
n
Y
1
(yi − x′i β)2
√
L(β, σ 2 |y1 , . . . , yn ) =
exp −
.
2σ 2
2πσ 2
i=1
Taking the log-likelihood:
log L(β, σ 2 |y1 , . . . , yn ) = −
n
n
1 X
log(2πσ 2 ) − 2
(yi − x′i β)2 .
2
2σ i=1
Solving for MLE estimates involves setting the first-order conditions (FOCs):
n
1 X
∂ log L
=2 2
xi (yi − x′i β̂mle )
∂β
2σ i=1
n
X
i=1
xi yi −
n
X
=
0
xi x′i β̂mle
=
0
β̂mle
=
(
i=1
n
X
xi x′i )−1
i=1
n
X
xi yi
i=1
Same as OLS
n
X
∂L
n
1
=
−
+
(yi − x′i β)2
2
2 )2
∂σ 2
2σmle
2(σmle
i=1
=
0
2
σmle
=
1X
(yi − x′i β)2
n i=1
=
1X
(yi − x′i β)2
n i=1
n
n
36
2
Substitute β with β̂M LE . The estimator σmle
takes the form
n
2
σmle
=
1X 2
ϵ̂
n i=1
It is biased but consistent.
8.3
Consistency
Let the log likelihood be
Ln (θ|y1 , . . . , yn )
=
n
X
log f (yi |θ).
i=1
n
Y
f (yi |θ) .
=
log
=
log f (y|θ).
i=1
Let the true distribution be denoted as f (y|θ0 ) where θ0 is the true parameter. Let the expectations be
defined as
L0 (θ|y1 , . . . , yn )
= E0 (Ln (θ|y1 , . . . , yn ))
Z
=
log f (y|θ)f (y|θ0 )dy
where θ̂M LE denotes the value which maximizes Ln (θ|y1 ,Q
. . . , yn ), θ denotes any possible value of parameter
n
and the joint PDF/PMF is f (y|θ0 ) = f (y1 , . . . , yn |θ0 ) = i=1 f (yi |θ0 ). Proof of consistency
1. θ̂M LE denotes the value which maximizes Ln (θ|y1 , . . . , yn )
2. θ0 denotes the value which maximizes E0 (Ln (θ|y1 , . . . , yn )) (will prove below)
3. Scaled version of log likelihood n1 Ln (θ|y1 , . . . , yn ) converges in probability to E0 (Ln (θ|y1 , . . . , yn )) by
WLLN.
4. Since two functions Ln and L0 are getting closer, the points of maximum should also get closer which
exactly means that plimθ̂M LE = θ0 .
Proof: θ0 is the maximizer of L0 (θ|y1 , . . . , yn ) which can be written as
E0 (log f (y|θ)) ≤ E0 (log f (y|θ0 ))
E0 log f (y|θ) − log f (y|θ0 ) ≤ 0
Proof is as follows
Z
Z
log f (y|θ)f (y|θ0 ) dy − log f (y|θ0 )f (y|θ0 ) dy
Z =
log f (y|θ) − log f (y|θ0 ) f (y|θ0 ) dy
Z
f (y|θ) = log
f (y|θ0 ) dy
f (y|θ0 )
Z f (y|θ)
≤
− 1 f (y|θ0 ) dy
f (y|θ0 )
as log(c) ≤ c − 1 for all c > 0
Z
= f (y|θ) − f (y|θ0 )dy
Z
Z
= f (y|θ)dy − f (y|θ0 )dy
=1−1
=0
37
8.4
Asymptotic Normality
In order to derive asymptotic normality of MLE, need to discuss some preliminary results: the moments of
the first and second derivative of the log likelihood function (corresponding to score function and Hessian
matrix), as well as the information matrix equality (relating the variance of the score to the mean of the
Hessian). Will use L(θ) for the notation of likelihood function.
8.4.1
Score Function
Definition: The first derivative of the log likelihood function.
S(θ) =
k×1
where log L(θ) =
∂ log L(θ)
∂θ
i log f (yi |θ). Thus,
P
si (θ) =
∂ log f (yi |θ)
∂θ
P
Then, S(θ) = i si (θ) since yi , i = 1, . . . , n are i.i.d. random variables, so si (θ) are iid random vectors.
We can prove that E [si (θ0 )] = 0 and further E [S (θ0 )] = 0.
8.4.2
Hessian Matrix
Definition: The second derivative of the log likelihood function
H(θ) = −
k×k
so that H(θ) =
∂ 2 log L(θ)
∂θ∂θ′
Pn
i=1 hi (θ) where
hi (θ) =
We can prove that
∂ 2 log f (yi |θ)
∂θ∂θ′
Var (si (θ0 )) = −E (hi (θ0 )) , and hence with i.i.d. data,
Var (S (θ0 )) = −E (H (θ0 ))
Var (S (θ0 )) = −E (H (θ0 )) is called information matrix equality.
8.4.3
Information matrix/Fisher information
Definition:
I(θ) = −E[H(θ)]
By the above information matrix equality,
I (θ0 ) = Var (S (θ0 ))
1. The larger the Fischer information, (the magnitude of the second derivative), the slope (the first
derivative) changes more quickly at θ.
2. θ0 can be well distinguished from other parameter values (uniqueness).
3. θ0 can be more precisely estimated.
8.4.4
Limiting Distribution
By the FOC of MLE (working with the scaled log likelihood function),
1X si θ̂M LE = 0
n i
Taylor expansion the FOC at θ0 , and then apply to mean value theorem to truncate the Taylor series at the
second term:
1X
∂si (θ0 ) si (θ0 ) +
θ̂M LE − θ0
+ remains
n i
∂θ0′
38
The remains converges to 0 quickly. By WLLN
o p
1 Xn
si (θ0 ) + hi (θ0 ) θ̂M LE − θ0
−
→0
n i
p
1X
1X
si (θ0 ) +
hi (θ0 ) θ̂M LE − θ0 −
→0
n i
n i
#−1
"
p
1X
1X
hi (θ0 )
si (θ0 ) + θ̂M LE − θ0 −
→0
n i
n i
−1 1 P
P
The asymptotic behavior of θ̂M LE − θ0 is the same as that of − n1 i hi (θ0 )
i si (θ0 )
n
−1 √ 1 P
1 P
√ The asymptotic distribution of n θ̂M LE − θ is the asymptotic distribution of − n i hi (θ0 )
n n i si (θ0 )
By CLT.
√ 1X
1X
d
si (θ0 ) −
→ N (0, Var( n
si (θ0 )))
n i
n i
√ 1X
1X
Var( n
si (θ0 )) =
Var (si (θ0 ))
n i
n i
1X
=
(−E (hi (θ0 )))
n i
√
n·
= −E (hi (θ0 ))
Therefore
√ 1X
d
n·
si (θ0 ) −
→ N (0, −E (hi (θ0 )))
n i
Also by WLLN
plim
By CMT, −
−1 p
−1
−
→ −E [hi (θ0 )] . By Slutsky’s, theorem,
1 P
n
1X
hi (θ0 ) = E (hi (θ0 ))
n i
i hi (θ0 )
#−1
√ 1X
1X
d
−1
−
hi (θ0 )
si (θ0 ) −
→ E [hi (θ0 )] N (0, −E [hi (θ0 )])
n·
n i
n i
d
−1
−
→ N 0, −E [hi (θ0 )]
1
d
−1
−
→ N 0, − E [H (θ0 )]
n
"
Thus
√
1
d
−1
n(θ̂M LE − θ0 ) −
→ N (0, − E [H (θ0 )] )
n
a
−1
θM LE ∼ N (θ0 , −E [H (θ0 )] )
a
∼ N (θ0 , I −1 (θ0 ))
8.5
Other Properties
8.5.1
Asymptotic Efficiency
Cramer-Rao Lower Bound: the variance of any unbiased estimator will be at least as large as I −1 (θ0 ). MLE
reaches this lower bound asymptotically, so MLE is asymptotically efficient.
8.5.2
Functional Invariance
If θ̂M LE is the MLE of θ, and γ = g(θ), where g(θ) is bounded and continuous, then γ̂M LE = g θ̂M LE is
the MLE of γ.
39
Estimated Asymptotic Variance of MLE
There are two ways to estimate I (θ0 ), which when inverted is I −1 (θ0 ). These two estimators are
1. Hessian:
hX
i
= −E [H (θ0 )] = −E
hi (θ0 )
X
= −
[E (hi (θ0 )] = −nE [hi (θ0 )]
I (θ0 )
i
Which can be estimated as
X
1X
hi (θ0 ) = −
hi (θ0 ) .
n i
i
−n ·
Thus
ˆ θ̂M LE ) = −
I(
X
hi θ̂M LE
i
2. BHHH or OPG (Outer product of gradient). Recall
I (θ0 )
=
Var(S (θ0 )) = Var(
X
si (θ0 ))
i
X
=
Var(si (θ0 )) = n Var(si (θ0 ))
i
= nE(si (θ0 )(si (θ0 )′ )
Which can be estimated as
n·
1X
(si (θM LE ))(si (θM LE )′ )
n i
Thus
ˆ θ̂M LE ) =
I(
X
(si (θM LE ))(si (θM LE )′ )
i
9
Binary Choice Models
We can model the binary choice model with yi ∈ {0, 1} following a Bernoulli distribution given as
P (y|p) = py (1 − p)1−y
The parameter p which is the probability of choice being made by observation i (yi = 1) can be further
modeled by assuming an underlying latent utility (yi∗ ) that the observation i realizes if yi = 1. Thus, if
yi∗ > 0, then yi = 1 and if yi∗ ≤ 0 then yi = 0. This is called the latent index threshold crossing model given
as:
yi
= I (yi ∗ > 0)
yi∗
= x′i β + νi
where, yi∗ is latent utility which we don’t observe. Thus, we can rewrite the Bernoulli PMF as
yi
=
1 with P (yi∗ > 0)
yi
=
0 with P (yi∗ ≤ 0)
We also know that E(yi = 1) = P (yi = 1) = P (yi∗ > 0). Assuming linear equation for utility
yi
=
1 with P (x′i β + νi > 0)
yi
=
0 with P (x′i β + νi ≤ 0)
yi
=
1 with P (νi > −x′i β)
yi
=
0 with P (νi ≤ −x′i β)
Or
We need to model the probability distribution for νi .
40
9.1
Linear Probability Model
If we estimate a linear regression model using OLS when the dependent variable is binary, then
yi = x′i β + ϵi
E(yi |xi ) = x′i β
and according to Bernoulli distribution, E(yi |xi ) = P (yi = 1|xi ),
P (νi > x′i β) = x′i β
Note however,
Var (ϵi |xi )
=
Var (yi |xi )
2
E yi2 |xi − (E [yi |xi ])
=
E [yi |xi ] − (E [yi |xi ])
=
x′i β − (x′i β)
=
x′i β(1 − x′i β)
=
2
2
The latent error is heteroskedastic. One option is to fix standard errors by using white standard errors or
run GLS for the estimation.
Drawbacks of LPM:
1. The fitted probability x′i β̂ can be greater than 1, or less than 0.
2. The estimated variance of β̂, x′i β̂ 1 − x′i β̂ can be negative.
9.2
Probit or Logit Models
We need to model P (.) such that when yi∗ → ∞, P (yi = 1) → 1 and when yi∗ → −∞, P (yi = 1) → 0. Assume
that νi |xi ∼ F where F is symmetric. Then
P (x′i β + νi ≥ 0|xi ) = P (νi ≥ −x′i β|xi )
= 1 − F (−x′i β)
= 1 − (1 − F (x′i β)) = F (x′i β)
For the identification
• The choice probability stays the same if we replace β with cβ: Some scale normalization is required.
• The choice probability stays the same if we replace the intercept, the first element of β, say β1 by β1 +C
and change the threshold to be C. So need some location normalization. We assume the threshold to
be 0.
Common choices of F include
• F (z) = Φ(z)− which is a normal CDF and we call this Probit model. V ar(νi ) = 1 for scale normalization.
exp(z)
which is the logistic CDF and we call this Logit model. V ar(νi ) = π6 for scale
• F (z) = 1+exp(z)
normalization.
The log likelihood function takes the form
log L(β|yi )
=
yi log(F (x′i )β)) + (1 − yi ) log (1 − F (x′i β))
=
n
X
yi log(F (x′i β)) + (1 − yi ) log (1 − F (x′i β))
The log likelihood of the sample
logL(β|y1 , . . . , yn )
i=1
41
9.3
MLE for Logit and Probit
9.3.1
Score Function
Maximize the log likelihood function with respect to β
β̂M LE = argmax
n X
yi log (F (x′i β)) + (1 − yi ) log (1 − F (x′i β))
i=1
The score function is
1
∂ log L(β) X
′
=
yi
S(β) =
′ β) f (xi β) xi + (1 − yi )
∂β
F
(x
i
i
1
· (−f (x′i β)) · xi
1 − F (x′i β)
X
1 − yi
yi
−
=
f (x′i β) xi
F (x′i β) 1 − F (x′i β)
i
X
yi (1 − F (x′i β) − (1 − yi ) F (x′i β)
=
f (x′i β) xi
F (x′i β) (1 − F (x′i β))
i
X
(yi − F (x′i β))
′
=
f (xi β) xi
F (x′i β) (1 − F (x′i β))
i
F OC : S(β̂) = 0 ⇒
(
X
f (x′i β̂M LE )xi
i
(yi − F (x′i β̂M LE )
f (x′i β̂M LE )(1 − f (x′i β̂M LE )
)
=0
→ non-linear
If we minimize non linear least square given as
n
X
(yi − F (x′i β))
2
i=1
the score function will take the form
2
n
X
f (x′i β)xi (yi − F (x′i β))
i=1
equalizing it to 0 and solving it is equivalent to solving S(β̂) = 0. Thus, estimating Logit or Probit can be
seen as a non linear least square estimation. Verify E[S(β)|X] = 0 evaluated at the true parameter β0
"
#
′
X
(y
−
F
(x
β
))
i
0
i
= E
f (x′i β0 ) xi
′ β ) (1 − F (x′ β )) |X
F
(x
0
i
i 0
i
′
X (yi − F (xi β0 ))
|X
=
E f (x′i β0 ) xi
F (x′i β0 ) (1 − F (x′i β0 ))
i
X
(E [yi |X] − F (x′i β0 ))
=
f (x′i β0 ) xi
F (x′i β0 ) (1 − F (x′i β0 ))
i
X
(F (x′i β0 ) − F (x′i β0 ))
=
f (x′i β0 ) xi
F (x′i β0 ) (1 − F (x′i β0 ))
i
=
0
For the probit model,
Z z
F (z)
√
=
−∞
f (z)
=
√
1
2πσ 2
42
1
2πσ 2
exp(z)dz
exp(z)
For the logit model,
F (z)
=
F (z) · (1 − F (z))
=
exp(z)
1 + exp(z)
exp(z)
exp(z) · 1−
1 + exp(z)
1 + exp(z)
exp(z)
1
·
1 + exp(z) 1 + exp(z)
exp(z)
(1 + exp(z))2
exp(z)(1 + exp(z)) − exp(z)2
(1 + exp(z))2
exp(z)
(1 + exp(z))2
=
=
f (z)
=
=
f (z) = F (z) · (1 − F (z))
9.3.2
Hessian and Information Matrix
si (β)
f (x′i β) xi (yi − F (x′i β))
F (x′i β) (1 − F (x′i β))
= A (x′i β) xi (yi − F (x′i β))
=
f (x′i β )
where A (x′i β) = F x′ β 1−F
. So
( i )(
(x′i β ))
hi (β)
∂si (β)
∂β ′
∂A (x′i β)
xi (yi − F (x′i β)) − A (x′i β) xi x′i f (x′i β)
∂β ′
h ∂A (x′ β)
i
i
′
′
′
′
E
x
(y
−
F
(x
β))
−
A
(x
β)
x
x
f
(x
β)|X
i
i
i
i
i
i
i
∂β ′
′
∂A (xi β)
xi (E[yi |X] − F (x′i β)) − A (x′i β) xi x′i f (x′i β)
∂β ′
f (x′i β)
xi x′i f (x′i β)
−
′
′
F (xi β) (1 − F (xi β))
2
f (x′i β) xi x′i
−
F (x′i β) (1 − F (x′i β))
=
=
E[hi (β)|X]
=
=
=
=
≤0
and
n
X
= E[
hi (β)]
E[H(β)]
i=1
=
n
X
E [hi (β)]
i=1
= nE [hi (β)]
For Logit Model
E[H(β)] = −n · E
h exp(x′ β)x x′ i
i
i i
(1 + exp(x′i β))
For Probit Model,
E[H(β)] = −n · E
h
i
ϕ2 (x′i β)xi x′i
Φ(x′i β)(1 − Φ(x′i β))
The Information matrix can be estimated through the above formulation. Most computer packages have
good routines for computing the MLE. The reason it works so well is that for both Probit and Logit models
the log likelihood function is globally concave.
43
9.4
Properties and Coefficient interpretation
Probit or Logit is estimated by MLE, so assuming the model is correctly specified, their properties follow
from the unusual properties of MLE,
P
• Consistency: β̂M LE −
→β
a
• Asymptotic Normality: β̂M LE ∼ N β, I −1 (β) where I(β) can we estimated using Hessian method or
BHHH or any other numerical algorithm.
• Asymptotic Efficiency: Probit and logit are asymptotically efficient because their variances reach the
Cramer-Rao lower bound asymptotically.
We are typically interested in the effects of the explanatory variables on the choice probability. These are
not directly given by the coefficients in the Probit or Logit model. For a continuous variable xij , we are
interested in the marginal effect of, xij on the choice probability.
∂F (x′i β)
= f (x′i β) βj
∂xij
The marginal effect depends on value of xi , typically
can either be evaluated at the sample mean of xi , i.e;
P
f (x̄′ β) βj or use the average marginal effect n1 i f (x′i β) βj .
10
Endogeneity in a linear model
Setup:
yi = x′i β + ϵi
but E (Xi ϵi ) ̸= 0. We are not considering strict exogeneity E(ϵi |xi ) which ensures unbiasedness. We are
defining endogeniety as defying a weaker condition of no correlation between independent variable and
error, E(ϵi , xi ) ̸= 0, which leads to inconsistency. Our parameter of interest β has a structural (causal)
interpretation in case E(ϵi , xi ) = 0. Linear model with E(ϵi , xi ) = 0 is called linear projection model defined
as
yi = x′i β + ϵi
Model:
E (xi ϵi ) = 0
′ −1
β ∗ = E (xi xi )
E (xi yi )
−1
= β + [E (xi x′i )]
E (xi ϵi )
=β
Under endogeniety, E(ϵi , xi ) ̸= 0 and thus β ∗ ̸= β. And we know that plimβ̂OLS = β ∗ and thus OLS
estimator is inconsistent.
Example: Measurement error in the regressor.
Let (yi , zi ) be joint random variables but zi is not observable. Instead we observe xi which is a proxy for zi .
The model takes the form
yi
=
zi′ β + ei
xi
=
zi + ui
ui is the measurement error. Classical Assumptions: zi ⊥ ui , ei ⊥ ui , and E (ui ) = 0.
yi
=
zi′ β + ei
yi
=
(xi − ui )′ β + ei
=
x′i β − u′i β + ei
=
x′i β + vi where vi = −u′i β + ei
=
E (xi ei ) − E (xi u′i ) β
=
E [(zi + ui ) ei ] − E [(zi + ui ) u′i ] β
=
−E (ui u′i ) β
E (xi vi )
Unless β = 0 or E (ui u′i ) = 0, OLS estimator is inconsistent.
44
Example: Simultaneous equation bias
Demand: qi = −β1 pi + e1i
Supply:
qi = β2 pi + e2i
e1i
1
′
e1 =
, E (ei ) = 0 and E (ei ei ) = I2 =
0
e2i
0
1
Regress qi on pi ?
qi
e1i
1 β1
=
1 −β2
pi
e2i
−1 β2 e1i +β1 e2i qi
e1i
1 β1
β1 +β2
=
=
e1i −e2i
1 −β2
pi
e2i
β1 +β2
OLS β̂ estimate β ∗ in the linear projection model as
qi = β ∗ pi + u∗
E (pi e∗i ) = 0
Thus, OLS will try to estimate
β∗
−1
E (pi qi )
E p2i
−1 (e1i − e2i )2
e1i − e2i β2 e1i + β1 e2i =
E
E
(β1 + β2 )2
β1 + β2
β1 + β2
−1
β2 e21i − β2 e1i e2i − β1 e1i e2i + β1 e22i
2
2
2
= (β1 + β2 ) E(e1i + e2i − 2e1i e2i )
E
(β1 + β2 )2
−1
= E(e21i + e22i − 2e1i e2i )
E(β2 e21i − β2 e1i e2i − β1 e1i e2i + β1 e22i )
=
=
=
β2 E(e21i ) − β2 E(e1i e2i ) − β1 E(e1i e2i ) + β1 E(e22i )
E(e21i ) + E(e22i ) − 2E(e1i e2i )
β2 − β1
2
p
1
Therefore, OLS estimator β̂ −
→ β ∗ = β2 −β
.
2
Example: Omitted Variable Bias
Assume the true model is:
Y
|{z}
Wage
=
X1 β1 + X2 β2 + ϵ
|{z}
|{z}
Edu
Ability
where all the classical linear model assumptions hold. Regress y on X1 only since ability cannot be observed.
The OLS estimator will estimate β1∗ defined as:
β1∗
= E((X1′ X1 )−1 )E(X1′ Y )
= E((X1′ X1 )−1 )E(X1′ (X1 β1 + X2 β2 + ϵ))
= β1 + E((X1′ X1 )−1 )E(X1′ X2 β2 ) + E((X1′ X1 )−1 )E(X1′ ϵ)
= β1 + E((X1′ X1 )−1 )E(X1′ X2 )β2
p
Therefore, OLS estimator β̂ −
→ β ∗ = β1 + E((X1′ X1 )−1 )E(X1′ X2 )β2 .
10.1
Instrumental Variable
We have defined endogeneity as the context where the regressor is correlated with the equation error. In
most applications we only treat a subset of the regressors as endogenous; most of the regressors will be
treated as exogenous, meaning that they are assumed uncorrelated with the equation error. To be specific,
we make the partition
x1i
xi =
, with dimensions k1 , k2
x2i
45
and similarly
β=
β1
β2
so that the structural equation is
yi = x′i β + ei
= x′1i β1 + x′2i β2 + ei .
The regressors are assumed to satisfy:
E(x2i ei ) ̸= 0.
E(x1i ei ) = 0,
We call x1i exogenous and x2i endogenous for the structural parameter β. In matrix notation, we write as:
y = Xβ + e = X1 β1 + X2 β2 + e.
The endogenous regressors x2i are potentially correlated with the equation error ei . The exogenous variables
x1i include the intercept and may be low or high dimensional. To consistently estimate β, we require
additional information. One common approach is to use instruments.
The ℓ × 1 random vector zi is an instrumental variable if:
E(zi ei ) = 0 (Exogeneity Condition)
E(zi zi′ ) > 0
rank (E(zi x′i )) = k (Relevance Condition)
′
We define zi = x1i z2i as x1i is exogenous and satisfy the conditions of an instrument. Exogeneity
condition means the instruments are exogenous, determined outside the model for yi . A necessary condition
for relevancy is that ℓ ≥ k where k = k1 + k2 .
x1i = exogenous variables,
x2i = endogenous variables,
z2i = instrumental variables.
We say the model is just-identified if ℓ = k (and ℓ2 = k2 ) and over-identified if ℓ > k (and ℓ2 > k2 ). These
are interpreted as: instruments are determined outside (yi , x2i ), causally influence x2i , but not yi except via
x2i .
10.1.1
Reduced Form
The reduced form is the relationship between the regressors xi and the instruments zi . A linear reduced
form model for xi is
xi = Γ′ zi + ui
where zi = [xi1 z2i ]′ . The ℓ × k coefficient matrix Γ can be defined by linear projection:
Γ = E(zi zi′ )−1 E(zi x′i )
and E(zi u′i ) = 0. In matrix notation: X = ZΓ + U where U is n × k. Recalling the partitions we can
conformably partition Γ as:
Ik1 Γ12
Γ=
0 Γ22
Then,
x1i = x1i
x2i = Γ′12 x1i + Γ′22 z2i + u2i
Also, substituting into yi equation:
yi = (Γ′ zi + ui )′ β + ei
= zi′ λ + vi
where λ = Γβ, vi = u′i β + ei and E(zi vi ) = E(zi u′i )β + E(zi ei ) = 0. Alternatively, substituting in the
extensive form,
yi = x′1i β1 + (Γ′12 z1i + Γ′22 z2i + u2i )′ β2 + ei
′
= x′1i λ1 + z2i
λ2 + vi
where
λ1 = β1 + Γ12 β2
λ2 = Γ22 β2
46
10.1.2
Reduced Form Estimation
Estimator for the reduced form
Γ̂ =
n
X
zi zi′
n
−1 X
i=1
Reduced form for x2i :
zi x′i = (Z ′ Z)−1 (Z ′ X)
i=1
x2i = Γ̂′12 x1i + Γ̂′22 z2i + û2i
X2 = X1 Γ̂12 + Z2 Γ̂22 + Û2
Estimator:
Γ̂22 = (Z2′ M1 Z2 )−1 Z2′ M1 X2
Reduced form for yi :
yi = zi′ λ̂ + v̂i = x1i λ̂1 + z2i λ̂2 + v̂i
Estimator:
λ̂ =
n
X
zi zi′
n
−1 X
i=1
and
10.1.3
zi yi = (Z ′ Z)−1 (Z ′ Y )
i=1
λ̂2 = (Z2′ M1 Z2 )−1 Z2′ M1 Y
Identification
Parameters Γ and λ are identified:
Γ
=
E(zi zi′ )−1 E(zi x′i )
λ
=
E(zi zi′ )−1 E(zi yi )
Identification condition:
λ = Γβ
Unique solution exists iff:
rank(Γ) = k
Equivalently:
10.1.4
rank(E(zi x′i )) = k
IV Estimator
In the just-identified case (ℓ = k), then:
β
=
Γ−1 λ
=
(E(zi zi′ )−1 E(zi x′i ))−1 E(zi zi′ )−1 E(zi yi )
= E(zi x′i )−1 E(zi yi )
and thus
β2 = Γ−1
22 λ2 ,
β1 = λ1 − Γ12 Γ−1
22 λ2
From method of moments:
The IV estimator is:
E(zi ei )
=
0
E(zi (yi − x′i β))
=
0
E(zi yi )
=
E(zi x′i )β
β
=
(E(zi x′i ))
−1
β̂IV = (Z ′ X)−1 (Z ′ y)
Residual vector:
ê = y − X β̂IV ,
47
Z ′ ê = 0
E(zi yi )
10.2
Two-Stage Least Squares
The IV estimator described in the previous section presumed ℓ = k. Now we allow the general case of ℓ ≥ k.
Examining the reduced-form equation we see
yi
=
zi′ λ + vi
=
zi′ Γβ + vi
E(zi vi ) = 0.
Defining wi = Γ zi we can write this as
′
yi = wi′ β + vi
E(wi vi ) = 0.
Suppose that Γ were known. Then we would estimate β by least-squares of yi on wi = Γ′ zi
β̂
=
(W ′ W )−1 (W ′ y)
=
(Γ′ Z ′ ZΓ)−1 (Γ′ Z ′ y).
While this is infeasible, we can estimate Γ from the reduced form regression. Replacing Γ with its estimate
Γ̂ = (Z ′ Z)−1 (Z ′ X) we obtain
β̂2sls
=
(Γ̂′ Z ′ Z Γ̂)−1 (Γ̂′ Z ′ y)
=
(X ′ Z(Z ′ Z)−1 Z ′ Z(Z ′ Z)−1 Z ′ X)−1 X ′ Z(Z ′ Z)−1 Z ′ y
=
(X ′ Z(Z ′ Z)−1 Z ′ X)−1 X ′ Z(Z ′ Z)−1 Z ′ y.
This is called the two-stage-least squares (2SLS) estimator. If the model is just-identified, so that k = ℓ,
then 2SLS simplifies to the IV estimator. Since the matrices X ′ Z and Z ′ X are square when ℓ = k, we can
factor
−1
−1
X ′ Z(Z ′ Z)−1 Z ′ X
= (Z ′ X)−1 (Z ′ Z)−1
(X ′ Z)−1
=
(Z ′ X)−1 (Z ′ Z)(X ′ Z)−1 .
(Once again, this only works when k = ℓ.) Then
β̂2sls
=
(X ′ Z(Z ′ Z)−1 Z ′ X)−1 X ′ Z(Z ′ Z)−1 Z ′ y
=
(Z ′ X)−1 (Z ′ Z)(X ′ Z)−1 X ′ Z(Z ′ Z)−1 Z ′ y
=
(Z ′ X)−1 (Z ′ Z)(Z ′ Z)−1 Z ′ y
=
(Z ′ X)−1 Z ′ y
= β̂iv
This shows that the 2SLS estimator is a generalization of the IV estimator.
There are several alternative representations of the 2SLS estimator which we now describe. First, defining
the projection matrix
Pz = Z(Z ′ Z)−1 Z ′
we can write the 2SLS estimator more compactly as
β̂2sls = (X ′ Pz X)−1 X ′ Pz y.
Second, define the fitted values for X from the reduced form
X̂ = PZ X = Z Γ̂.
Then the 2SLS estimator can be written as
β̂2sls = (X̂ ′ X)−1 X̂ ′ y.
This is an IV estimator as defined in the previous section using X̂ as the instrument. Third, since PZ is
idempotent, we can also write the 2SLS estimator as
β̂2sls = (X ′ PZ PZ X)−1 X ′ PZ y = (X̂ ′ X̂)−1 X̂ ′ y
which is the least-squares estimator obtained by regressing y on the fitted values X̂. This is the source of
the "two-stage" name is since it can be computed as follows.
48
1. Regress X on Z, vis., Γ̂ = (Z ′ Z)−1 (Z ′ X) and X̂ = Z Γ̂ = PZ X.
2. Regress y on X̂, vis., β̂2sls = (X̂ ′ X̂)−1 X̂ ′ y.
Recall, X = [X1 , X2 ] and Z = [X1 , Z2 ]. Notice X̂1 = PZ X1 = X1 since X1 lies in the span of Z. Then
X̂ = [X̂1 , X̂2 ] = [X1 , X̂2 ].
Thus in the second stage, we regress y on X1 and X̂2 . So only the endogenous variables X2 are replaced by
their fitted values:
X̂2 = X1 Γ̂12 + Z2 Γ̂22 .
This least squares estimator can be written as
y = X1 β̂1 + X̂2 β̂2 + ê.
10.3
Asymptotic Properties
10.3.1
Consistency
Let Qxz = E(xi zi′ ), Qzz = E(zi zi′ ) and Qzx = E(zi x′i ).
β̂2sls
plimβ̂2sls
=
(X ′ Z(Z ′ Z)−1 Z ′ X)−1 X ′ Z(Z ′ Z)−1 Z ′ y
=
(X ′ Z(Z ′ Z)−1 Z ′ X)−1 X ′ Z(Z ′ Z)−1 Z ′ (Xβ + ϵ)
= β + (X ′ Z(Z ′ Z)−1 Z ′ X)−1 X ′ Z(Z ′ Z)−1 Z ′ ϵ
= plimβ + plim (X ′ Z(Z ′ Z)−1 Z ′ X)−1 X ′ Z(Z ′ Z)−1 Z ′ ϵ
1
1
1
1
1
1
= β + plim ( X ′ Z( Z ′ Z)−1 Z ′ X)−1 X ′ Z( Z ′ Z)−1 Z ′ ϵ
n
n
n
n
n
n
1 ′
1 ′
1 ′ −1
1
1
1
−1
= β + plim( X Z)(plim( Z Z)) plim( Z X)
plim( X ′ Z)(plim( Z ′ Z))−1 plim( Z ′ ϵ)
n
n
n
n
n
n
= β + (E(xi zi′ )E(zi zi′ )−1 E(zi x′i ))−1 E(xi zi′ )E(zi zi′ )−1 E(zi ϵi )
−1
= β + (Qxz Q−1
Qxz Q−1
zz Qzx )
zz E(zi ϵi )
= β
Thus, β̂2sls is consistent.
10.3.2
Asymptotic Normality
Also for asymptotic distribution
−1
′
plim(β̂2sls − β) = (Qxz Q−1
Qxz Q−1
zz Qzx )
zz E(zi ϵi ) = 0
Thus, we can derive that
√
d
−1
′ 2
−1
−1
−1
n(β̂2sls − β) −
→ N (0, (Qxz Q−1
Qxz Q−1
)
zz Qzx )
zz E(zi zi ϵi )Qzz Qzx (Qxz Qzz Qzx )
Under homoskedasticity E(zi zi′ ϵ2i ) = Qzz σ 2
√
d
−1
2 −1
−1
−1
n(β̂2sls − β) −
→ N (0, (Qxz Q−1
Qxz Q−1
)
zz Qzx )
zz Qz zσ Qzz Qzx (Qxz Qzz Qzx )
d
−1 2
−
→ N (0, (Qxz Q−1
σ )
zz Qzx )
11
Bayesian Econometrics
Since Statistical Inference is based on probability theory, the major difference between Bayesian and frequentist approaches to inference can be traced to the different views that each have about the interpretation
and scope of probability theory. Frequentists restrict the assignment of probabilities to statements that
describe the outcome of an experiment that can be repeated. Bayesians believe in a subjective interpretation
of probability.
49
11.1
Prior, Likelihood, and Posterior
Using the Bayes Theorem, which is at the center of Bayesian Econometrics
f (y|θ)π(θ)
,
f (y)
π(θ|y) =
R
where f (y) = f (y|θ)π(θ)dθ is the marginal likelihood. The effect of dividing by f (y) is to make π(θ|y) a
normalized probability distribution. π(θ|y) is interpreted as the posterior density function of θ|y and f (y|θ)
is the likelihood function. The prior density π(θ), embodies our beliefs about the distribution of θ before
seeing the data y. These beliefs are based on the researcher’s knowledge of the problem at hand; they may
be based on theoretical considerations or on previous empirical work. The prior distribution usually depends
on parameters, called hyperparameters, which may either be supplied by the researcher or given probability
distributions. f (y), normalizes the posterior distribution. Since it is independent of θ, however, it is often
convenient to write the posterior distribution as
π(θ|y) ∝ f (y|θ)π(θ),
that is, the posterior distribution is proportional to the likelihood function times the prior distribution. In
this form, the right side of the equation does not integrate to one, but as a function of θ, it has the same
shape as π(θ|y).
It is useful to think about the process as a method of updating information, an idea that is reinforced by the
prior-posterior terminology. Before collecting the data y, our information about θ is summarized by the prior
distribution π(θ). After observing y, our information about θ is summarized by the posterior distribution
π(θ|y).
We illustrate these ideas with the coin-tossing example. Let y = 1 if a coin toss results in a head and
0 otherwise, and let P (y = 1) = θ, which is assumed to be constant for each trial. In this model, θ is
a parameter and the value of y is the data. Under these assumptions, y is said to have the Bernoulli
distribution, written as y ∼ Be(θ). We are interested in learning about θ from an experiment in which the
coin is tossed n times yielding the data y = (y1 , y2 , . . . , yn ), where yi indicates whether the ith toss resulted
in a head or tail. The likelihood function for a single toss of a coin can be written as p(yi |θ) = θyi (1 − θ)1−yi ,
which implies P (yi = 1|θ) = θ and P (yi = 0|θ) = 1 − θ. For n independent tosses of a coin, we therefore
have
f (y1 , . . . , yn |θ)
= θy1 (1 − θ)1−y1 · · · θyn (1 − θ)1−yn
n
Y
=
θyi (1 − θ)1−yi
i=1
P
= θ
yi
P
(1 − θ)n−
yi
.
To complete the specification of the model, we need a prior distribution. Since 0 ≤ θ ≤ 1, the prior should
allow θ to take on any value in that interval and not allow it to fall outside that interval. A common choice
is the beta distribution Beta(α, β):
π(θ) =
Γ(α + β) α−1
θ
(1 − θ)β−1 ,
Γ(α)Γ(β)
0 ≤ θ ≤ 1,
α, β > 0.
α and β are hyperparameters. Why choose the beta distribution? First, it is defined in the relevant range.
Second, it is capable of producing a wide variety of shapes. depending on the choice of α and β. The shape
of a beta distribution can be understood by examining its mean and variance:
E(θ) =
α
,
α+β
Var(θ) =
αβ
.
(α + β)2 (α + β + 1)
A third reason for choosing this distribution is that the beta prior in combination with the likelihood function
yields a posterior distribution that has a standard form, which is convenient for analyzing the properties of
the posterior. This is an example of a conjugate prior, where the posterior distribution is in the same family
as the prior distribution.
π(θ|y) ∝ f (y|θ)π(θ)
∝ θ
P
∝ θ
P
(α0 + yi )−1
yi
(1 − θ)n−
P
yi α0 −1
θ
(1 − θ)β0 −1
(1 − θ)(β0 +n−
50
P
yi )−1
.
In this expression, the normalizing constant of the beta distribution has been absorbed into the proportionality constant because the constant does P
not depend on θ. As promised,
π(θ|y) is in the form of a beta
P
distribution with parameters α1 = α0 +
yi and β1 = β0 + n −
yi . We can now find the marginal
distribution for any of the θi , for example, θ1 .


X
π(θ1 |y) ∝ Beta y1 + α1 ,
(yi + αi ) ,
i̸=1
which is a beta distribution. In the die-throwing example, the probability of the 1 spot appearing when a
single die is thrown is given by the beta distribution:
!
6
X
θ1 ∼ Beta y1 + α1 ,
(yi + αi ) .
i=2
Note that this result is equivalent to considering the 1-spot as one outcome and the other die faces as a
second outcome, transforming the multinomial model into a binomial model.
To summarize, when dealing with a model that contains more than one parameter, simply redefine the
parameter as a vector. In addition, the marginal and conditional distributions of individual parameters or
groups of parameters can be found by applying the usual rules of probability.
3.1.3 Bayesian Updating
Let θ represent one parameter or a vector of parameters, and let y1 represent the first set of data obtained
in an experiment. As an example, you may think of y1 as the number of heads found in tossing a coin n1
times, where the probability of heads is θ. As usual,
π(θ|y1 ) ∝ f (y1 |θ)π(θ).
Next, suppose that a new set of data y2 is obtained, and we wish to compute the posterior distribution given
the complete data set π(θ|y1 , y2 ). By the usual rules of probability,
π(θ|y1 , y2 ) ∝
f (y1 , y2 |θ)π(θ)
∝
f (y2 |y1 , θ)f (y1 |θ)π(θ)
∝
f (y2 |y1 , θ)π(θ|y1 ).
If the data sets are independent, f (y2 |y1 , θ) simplifies to f (y2 |θ). This takes the form of a likelihood times
a density for θ, but that the latter density is π(θ|y1 ): the posterior distribution based on the initial set of
data occupies the place where a prior distribution is expected. It is now easy to verify that, if more new
data y3 become available, π(θ|y1 , y2 , y3 ) has π(θ|y1 , y2 ) where you would expect to see π(θ). Thus, as new
information is acquired, the posterior distribution becomes the prior for the next experiment. In this way, the
Bayesian updates the prior distribution to reflect new information. Updating also justifies our interpretation
of the prior distribution as being based on previous data, if such data are available, or on the equivalent of
previous data in the researcher’s view.
As a simple example of updating, consider data generated from the Bernoulli example. Assume
Pa beta prior
with parameters α0 and β0 . Suppose the first experiment
produces
n
trials
and
set
s
=
y1i ; let the
1
1
P
second experiment produce n2 trials and set s2 =
y2i . We can then compute the posterior based on the
first experiment as
f (θ|s1 ) ∝ θα0 −1 (1 − θ)β0 −1 θs1 (1 − θ)n1 −s1 ,
or
θ|s1 ∼ Beta(α0 + s1 , β0 + (n1 − s1 )).
If we take the latter as the prior for the second experiment, we find
f (θ|s1 , s2 ) ∝ θα0 +s1 (1 − θ)β0 +(n1 −s1 ) θs2 (1 − θ)n2 −s2 ,
or
θ|s1 , s2 ∼ Beta(α0 + (s1 + s2 ), β0 + (n1 + n2 ) − (s1 + s2 )).
The latter distribution is implied by a Beta(α0 , β0 ) prior and obtaining s1 + s2 ones on n1 + n2 trials. This
is an intuitively reasonable way of allowing new information to influence beliefs about a parameter, and it
appears as a consequence of standard probability theory.
51
11.2
Consistency
Although the concepts of Bayesian inference hold true for any sample size, it is instructive to examine
how the posterior distribution behaves in large samples. This analysis offers important insights about the
nature of the posterior distribution, particularly about the relative contributions of the prior distribution
and likelihood function in determining the posterior distribution. Consider the case of independent trials,
where the likelihood function L(θ|y) is
Y
Y
L(θ|y) =
f (yi |θ) =
L(θ|yi );
L(θ|yi ) is the likelihood contribution of yi . Also define the log likelihood function l(θ|y) as
X
l(θ|y) = log L(θ|y) =
l(θ|yi ) = n¯l(θ|y),
where l(θ|yi ) is the log likelihood contribution of yi . The posterior distribution can be written as
π(θ|y) ∝ π(θ)L(θ|y) ∝ π(θ) exp[n¯l(θ|y)].
We can now examine the effect of the sample size n on the posterior distribution, which is proportional to
the product of the prior distribution and a term that involves an exponential raised to n times a number.
For large n, the exponential term dominates π(θ), which does not depend on n. Accordingly, we can expect
that the prior distribution will play a relatively smaller role than do the data, as reflected in the likelihood
function, when the sample size is large. Conversely, the prior distribution has relatively greater weight when
n is small. As an example of this phenomenon, recall the coin-tossing example where it was shown that
α
n
α+β
+
ȳ,
E(θ|y) =
α+β+n α+β
α+β+n
which is a weighted average of the mean of the prior α/(α + β) and the sample mean ȳ. For large n, the
weight of the prior mean is small compared to that of the sample mean. This idea can be taken one step
further. If we denote the true value of θ by θ0 , it can be shown that
lim ¯l(θ|y) → ¯l(θ0 |y).
n→∞
Accordingly, for large n, the posterior distribution collapses to a distribution with all its probability at θ0 .
This property is similar to the criterion of consistency in the frequentist literature and extends to the multi
parameter case.
11.3
Inference
• Point Estimate: Choose measures of central tendency from the posterior distribution
• Interval Estimation:
P (θL ≤ θ ≤ θH ) = 95%
which tells us that θL ≤ θ ≤ θH with a probability of 0.95. This is called credible intervals which is
estimated by taking 2.5th and 97.5th percentile from the posterior distribution.
• Prediction:
Z
P (yt+1 |yt )
=
P (yt+1 , θ|yt )dθ
Z
P (yt+1 |θ, yt )π(θ|yt ))dθ
=
Z
P (yt+1 |θ)π(θ|yt ))dθ
=
• Model Comparison: Marginal likelihood
Z
f (y) =
f (y|θ)π(θ)dθ
It is the unconditional probability of data being generated by the model.
52
11.4
Bayesian Linear Regression
Consider the model
yi = x′i β + εi ,
iid
where, for i = 1, . . . , n, we assume that εi ∼ N 0, σ 2 . Stacking the observations, we can write the model
in matrix form as
y = Xβ + ε,
where


y1


y =  ...  ,

x′1


X =  ...  ,

x′n
yn


ε1


ε =  ...  ,
εn
and ε ∼ N 0, σ 2 In , yielding the likelihood function given as
f y|β, σ 2
= fN y|Xβ, σ 2 In
−1
n
′
1
(2π)− 2 σ 2 In 2 e− 2σ2 (y−Xβ) (y−Xβ)
− n
′
1
n
= (2π)− 2 σ 2 2 e− 2σ2 (y−Xβ) (y−Xβ) .
Assume that the priors are given by β ∼ N (β0 , B0 ) and σ 2 ∼ IG ν20 , δ20 .
=
n
− 12
1
′
−1
e− 2σ2 (β−β0 ) B0 (β−β0 )
π(β)
=
(2π)− 2 |B0 |
π(σ 2 )
=
ν0
ν0δ0
δ0
(σ 2 )− 2 −1 exp(− 2 )
Γ(δ0 )
2σ
The posterior distribution takes the form
′
1
n
−1
π(β, σ 2 |y) ∝
(2π)− 2 σ 2 In 2 e− 2σ2 (y−Xβ) (y−Xβ) ×
n
− 12
(2π)− 2 |B0 |
ν δ0
ν0
′ −1
1
δ0 0
(σ 2 )− 2 −1 exp(− 2 )
e− 2σ2 (β−β0 ) B0 (β−β0 ) ×
Γ(δ0 )
2σ
The posterior distribution does not belong to a known family of probability distributions. But the conditional densities of parameters do belong to know families of distribution. Thus, we can use Gibbs Sampler
to draw from the posterior distribution. The full conditional distribution π(β|y, σ 2 ) is proportional to
f (y|β, σ 2 )π(β, σ 2 ) and its kernel, ignoring terms that do not involve β, can be written as
1 (y − Xβ)′ (y − Xβ)
′ −1
π(β|y, σ 2 ) ∝ exp −
+
(β
−
β
)
B
(β
−
β
)
0
0
0
2
σ2
′
1
β ′ X ′ y β ′ X ′ Xβ
y Xβ
′ −1
′ −1
′ −1
∝ exp − − 2 −
+
+
β
B
β
−
β
B
β
−
β
B
β
.
0
0 0
0
0
2
σ
σ2
σ2
Collecting terms and using the definitions of B̂ and β̂, we have that π(β|y, σ 2 ) is proportional to
′
1
X ′X
X ′y
yX
′ −1
exp − β ′ B0−1 + 2
β − β ′ B0−1 β0 + 2 −
+
β
B
β
0 0
2
σ
σ
σ2
i
1h
= exp − β ′ B̂ −1 β − β ′ B̂ −1 β̂ − β̂ ′ B̂ −1 β .
2
Adding and subtracting β̂ ′ B̂ −1 β̂ inside the brackets (note that this is a term that does not involve β), we
can complete the square and write
′
1 π(β|y, z) ∝ exp −
β − β̂ B̂ −1 β − β̂ − β̂ ′ B̂ −1 β̂
2
′
1
∝ exp −
β − β̂ B̂ −1 β − β̂
,
2
53
where the last line follows by recognizing that β̂ ′ B̂ −1 β̂ does not involve β and can therefore be absorbed in
the constant of proportionality. The result is the kernel of the Gaussian density and hence we have shown
that
β|y, σ 2 ∼ N (β̂, B̂)
−1
′
′
where B̂ = B0−1 + Xσ2X
and β̂ = B̂ B0−1 β0 + Xσ2y
To derive the full conditional
distribution π σ 2 |y, β , we drop all terms that do not involve σ 2 from the joint
posterior π β, σ 2 |y to obtain
− ν0 −1 − δ0
− n
′
1
e 2σ2
π σ 2 |y, β ∝ σ 2 2 e− 2σ2 (y−Xβ) (y−Xβ) σ 2 2
0 −1
− n+ν
′
1
2
= σ2
e− 2σ2 [(y−Xβ) (y−Xβ)+δ0 ] ,
which we can recognize as the kernel of the inverse gamma density, whereby
ν0 + n δ0 + (y − Xβ)′ (y − Xβ)
,
π σ 2 |y, β = fIG σ 2
2
2
as required.
Gibbs Sampler
1. Start with an arbitrary value of σ 2 say σ02 and draw β1 from π σ12 |y, β .
2. Draw σ12 from π σ 2 |y, β1 .
3. For i = 2, . . . , B,
2
(a) Draw βi from π σi−1
|y, β .
(b) Draw σi2 from π σ 2 |y, βi .
4. Discard the first M observations as burn in sample and use the rest as parameter values being drawn
from the posterior distribution.
54