Basic Regression – Model, Assumptions, and
Estimation
I. Modeling the Regression from Theory
We now turn to the subject of regression in econometrics. To understand this we
begin with an economic theory that says that X and Y are related. We can write
this as
𝑌 = 𝑓(𝑋)
Our economic theory tells us the qualitative relation between X and Y. Namely,
our theory tells us whether 𝑓 ′ (𝑋) > 0 or 𝑓 ′ (𝑋) < 0. However, we are now
interested in assessing any possible stable quantitative relation between X and Y.
We want to know how much X affects Y and not merely the sign of the effect.
Given the basic functional relation between X and Y , our next step is to convert
this general function relation into a model which is linear in the parameters. This
can be done in a number of ways. Here are some examples.
𝒀 = 𝒇(𝑿)
Linear (in parameters) Model
(1) 𝑌 = 𝛽𝑜 + 𝛽1 𝑋 + 𝜀
(1') 𝑌 = 𝛽𝑜 + 𝛽1 𝑋 + 𝜀
(2) 𝑌 = 𝐴𝑒 𝛼𝑋 𝑒 𝜀
(2') 𝑙𝑛(𝑌) = 𝑙𝑛(𝐴) + 𝛼𝑋 + 𝜀
(3) 𝑌 = 𝐴𝑋 𝛼 𝑒 𝜀
(3') 𝑙𝑛(𝑌) = 𝑙𝑛(𝐴) + 𝛼𝑙𝑛(𝑋) + 𝜀
In the table above note how that each equation on the right hand side is linear in
the parameters. If we let 𝑙𝑛(𝐴) = 𝛽𝑜 and 𝛼 = 𝛽1 , then it is clear that the equations
on the right hand side are linear in the parameters.
The last term in each of the models is ε which we take as a random variable having
a particular PDF. This means that ε has a mean, E[ε] and a variance, var(ε). We
usually assume that E[ε] = 0 and that var(ε) = σ2, particularly when we are
discussing basic regression.
Now suppose that we have a model like (1') above. We also have a random sample
on X and Y. We can write this as X1, X2, ..., XN and Y1, Y2,...,YN. This means
that we have N equations which we can write as
𝑌1 = 𝛽𝑜 + 𝛽1 𝑋1 + 𝜀1
𝑌2 = 𝛽𝑜 + 𝛽1 𝑋2 + 𝜀2
𝑌3 = 𝛽𝑜 + 𝛽1 𝑋3 + 𝜀3
⋮
𝑌𝑁 = 𝛽𝑜 + 𝛽1 𝑋𝑁 + 𝜀𝑁
This system of equations can be written in a concise way as
𝑌𝑡 = 𝛽𝑜 + 𝛽1 𝑋𝑡 + 𝜀𝑡 for t = 1,2,...,N
The goal of our study is to use data on X and Y to estimate or guess the values of
𝛽𝑜 , 𝛽1 , σ2, and 𝜀𝑡 for t = 1,2,...,N. This lecture is concerned with finding a good
way to estimate these parameters. In order to do this we need to make some basic
assumptions about the model.
II. Major Assumptions used in Basic Regression
Our basic regression model uses certain assumptions which we cannot prove to be
true, but which we can later statistically test for their reliability. These
assumptions can be written as follows:
Assumptions: (1) 𝐸[𝜀𝑡 ] = 0 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑡 = 1,2, … , 𝑁
(2) var(𝜀𝑡 ) = 𝜎 2 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑡 = 1,2, … , 𝑁
(3) cov(𝜀𝑡 , 𝜀𝑠 ) = 0 𝑤ℎ𝑒𝑛 𝑡 ≠ 𝑠
(4) the X's and the ε's are independent
The first assumption is easy. It just says that the error is stable and does not have a
mean that is moving over the sample. Since the mean of ε is stable, we can just
assume that it is zero. The second assumption says that the variance is constant
and is equal to a number called σ2 . The third assumption says that any random
disturbance εt will be uncorrelated with any other random disturbance in the
sample εs. These random errors cannot affect each other linearly. The fourth and
last assumption is that the X's and ε's have no relation to each other.
The last assumption is vital if we are to make a good estimation. Unfortunately, the
assumption is violated in many cases that we consider in economics. But, at a
deeper level, our regression model assumes that Y is on the left hand side and X is
on the right hand side. Y is clearly influenced by ε. X is assumed to be
independent of ε, but not of Y. Thus, there is a fundamental asymmetry in basic
regression that X and Y cannot change places across the equality sign. If X is on
the right, then we are assuming the error is independent of X. If we then change
places, putting X on the left and Y on the right, we will be making a clear mistake
in modeling, since it is impossible for ε to be independent of both X and Y.
Regression is therefore more than correlation, which does not distinguish between
such right hand variables and left hand side variables.
We now turn to the issue of estimation of the model.
III. Estimation of the Basic Regression Model
Our model can be written as
𝑌𝑡 = 𝛽𝑜 + 𝛽1 𝑋𝑡 + 𝜀𝑡 for t = 1,2,...,N
Our first step is to use the random sample on X and Y to create estimators for the
coefficients βo and β1. These estimators are called method of moment estimators,
but if the error term ε is distributed as a normal PDF then the method of moment
estimator will be equal to the famous ordinary least squares estimator than we will
discuss later. Here is how we use our assumptions to guide the creation of good
estimators.
To start with we assume that our estimators of βo and β1 are written as 𝛽̂𝑜 and 𝛽̂1 .
In addition, our estimator of 𝜀𝑡 will be written as 𝜀̂𝑡 . The predicted value of 𝒀𝒕
will be written as 𝑌̂𝑡 , and is equal to
𝑌̂𝑡 = 𝛽̂𝑜 + 𝛽̂1 𝑋𝑡
which means that the predicted error, 𝜀̂𝑡 , (called the residual) is equal to
𝜀̂𝑡 = 𝑌𝑡 − 𝑌̂𝑡
which implies that
𝑌𝑡 = 𝛽̂𝑜 + 𝛽̂1 𝑋𝑡 + 𝜀̂𝑡
Step 1: Add the 𝑌𝑡 ′𝑠 together, divide by N, and subtract from 𝑌𝑡 to get the
following:
(𝑌𝑡 − 𝑌̅ ) = 𝛽̂1 (𝑋𝑡 − 𝑋̅) + (𝜀̂𝑡 − 𝜀̂)̅ for t = 1, 2,...,N
Note that we next force 𝜀̂ ̅ = 0 in the above in order to closely follow Assumption 1.
Step 2: Multiply both sides by (𝑋𝑡 − 𝑋̅) to get
(𝑌𝑡 − 𝑌̅)(𝑋𝑡 − 𝑋̅) = 𝛽̂1 (𝑋𝑡 − 𝑋̅)2 + 𝜀̂𝑡 (𝑋𝑡 − 𝑋̅) for t = 1, 2,...,N
Step 3: We next sum these together to get the following
𝑁
𝑁
𝑁
∑(𝑋𝑡 − 𝑋̅)(𝑌𝑡 − 𝑌̅) = 𝛽̂1 ∑(𝑋𝑡 − 𝑋̅)2 + ∑
𝑡=1
𝑡=1
(𝑋𝑡 − 𝑋̅) 𝜀̂𝑡
𝑡=1
̅
After this, we use assumption 4 to force ∑𝑁
𝑡=1(𝑋𝑡 − 𝑋 ) 𝜀̂𝑡 = 0.
̅ 2
Step 4: Finally, we divide both sides by ∑𝑁
𝑡=1(𝑋𝑡 − 𝑋 ) to get the estimator for
𝛽1 as
𝛽̂1 =
̅
̅
∑𝑁
𝑡=1(𝑋𝑡 − 𝑋 )(𝑌𝑡 − 𝑌 )
̅ 2
∑𝑁
𝑡=1(𝑋𝑡 − 𝑋 )
Step 5: Having developed the estimator for 𝛽1 , we now consider the estimator for
𝛽𝑜 in the following way – we know that
𝑌𝑡 = 𝛽̂𝑜 + 𝛽̂1 𝑋𝑡 + 𝜀̂𝑡 and 𝜀̂ ̅ = 0
and therefore taking the average of the Y's gives
𝑌̅ = 𝛽̂𝑜 + 𝛽̂1 𝑋̅
and we can easily solve for our estimator of 𝛽𝑜 as
𝛽̂𝑜 = 𝑌̅ − 𝛽̂1 𝑋̅
We now have successfully calculated two of our estimators -- one for βo and one
for β1. We still must estimate the error terms, as well as the variance of the error.
Here is how we can do this.
Step 6: We can easily create an estimator for the error term, since we have
previously shown how this can be done. We simply let
𝜀̂𝑡 = 𝑌𝑡 − 𝑌̂𝑡 = 𝑌𝑡 − 𝛽̂𝑜 − 𝛽̂1 𝑋𝑡 for t = 1, 2, ..., N
We refer to these 𝜀̂𝑡 's as the regression residuals. We often make a graph of these
residuals much like the following taken from GRETL
A careful study of the residuals can often tell us whether our regression is healthy
and valuable or not. It is a diagnostic tool much like an X-ray or a blood test is a
diagnostic tool for the doctor. The residuals are extremely important and therefore
we will be spending much time trying to understand how to correctly evaluate
them.
Step 7: Finally we can guess σ2 (which we call s2) using the following estimator –
2
∑𝑁
𝑡=1 𝜀̂𝑡
𝜎̂ = 𝑠 =
𝑁−2
2
2
We divide this estimator by N-2 since two degrees of freedom of our data have
been lost calculating the estimators of βo and β1. The estimator s2 shows us the
rough band about zero that the residuals fluctuate. Typically, we can say that 95%
of the residuals will lie in a band which is 2 standard deviations above and below
zero in size. The standard deviation, called the standard error of the regression is
equal to
Standard Error of the Regression = √𝑠 2
Problems:
(1) Explain why we assume 𝜀̂ ̅ = 0 in deriving our estimators.
̅
(2) Explain why we assume ∑𝑁
𝑡=1(𝑋𝑡 − 𝑋 ) 𝜀̂𝑡 = 0 in deriving our estimators.
(3) On page 1 above, why can we say that the right hand side of the table are
"linear in parameters".
(4) Explain why that 𝑌𝑡 = 𝛽̂𝑜 + 𝛽̂1 𝑋𝑡 + 𝜀̂𝑡 , given that 𝑌𝑡 = 𝛽𝑜 + 𝛽1 𝑋𝑡 + 𝜀𝑡 .
(5) Prove that
𝑁
∑
(𝑋𝑡 − 𝑋̅)(𝑌𝑡 − 𝑌̅)
𝑡=1
𝑁
=∑
𝑡=1
𝑋𝑡 (𝑌𝑡 − 𝑌̅) = ∑
𝑁
(𝑋𝑡 − 𝑋̅)𝑌𝑡
𝑡=1
(6) Suppose that our regression model is 𝑌𝑡 = 𝛽1 𝑋𝑡 + 𝜀𝑡 for t = 1, 2,..., N
We can therefore skip Step 1 in the derivation of the estimator for β1. Sow that the
estimator for β1 is equal to 𝛽̂1 =
∑𝑁
𝑡=1 𝑋𝑡 𝑌𝑡
2
∑𝑁
𝑡=1 𝑋𝑡
.
(7) Suppose that our regression model is 𝑌𝑡 = 𝛽𝑜 + 𝜀𝑡 for t = 1, 2,..., N. Derive
an estimator for βo.
(8) Suppose that our regression model is 𝑌𝑡 = 𝛽𝑜 + 𝛽1 𝑋𝑡 + 𝜀𝑡 for t = 1, 2,..., N.
Prove that the average of 𝑌̂𝑡 is equal to the average of 𝑌𝑡 .
(9) Suppose that our regression model is 𝑌𝑡 = 𝛽𝑜 + 𝛽1 𝑋𝑡 + 𝜀𝑡 for t = 1, 2,..., N.
Explain why that the estimator 𝛽̂1 is a random variable. Find the expectation of 𝛽̂1 .
(10) Suppose that our regression model is 𝑌𝑡 = 𝛽𝑜 + 𝛽1 𝑋𝑡 + 𝜀𝑡 for t = 1, 2,..., N.
Find the variance of 𝛽̂1 .