Linear regression with
one regressor
Outline
1. The population linear regression model
2. The ordinary least squares (OLS) estimator and the sample
regression line
3. Measures of fit of the sample regression
4. The least squares assumptions for causal inference
5. The sampling distribution of the OLS estimator
6. The least squares assumptions for prediction
The Population Linear Regression Model
Yi = β0 + β1Xi + ui , i = 1,…, n
where we have n observations, (Xi, Yi), i = 1,.., n,
Xi is the independent variable or regressor,
Yi is the dependent variable,
β0+ β1X is the population regression line or population regression function,
β0 is the intercept of the population regression line,
β1 is the slope of the population regression line,
and ui is the regression error term for the i-th observation.
• The regression error, ui , is the difference between Yi and its predicted value using the
population regression line.
Linear regression
• The population regression line is the expected value of Y given X.
β0 + β1Xi
E(Y|X)=β0 + β1X
• The slope is the difference in the expected values of Y with a
unit difference of in X
• The estimated regression can be used either for:
• causal inference (learning about the causal effect on Y of a change in X)
• prediction (predicting the value of Y given X, for an observation not in
the data set)
The Linear Regression Model (Section 4.1)
The population regression line :
E(Test Score |STR)= β0 + β1STR
β1 = slope of population regression line
Why are β0 and β1 “population” parameters?
• We would like to know the population value of β1.
• We don’t know β1, so must estimate it using data.
Figure 4.1 The population regression model in a picture
ui =Yi – (β0+ β1Xi )
7 (hypothetical) observations on Y and X,
the population regression line,
and the regression error (the “error term”)
Linear regression
• Estimation
• How should we draw a line through the data to estimate the population
slope?
• ordinary least squares (OLS).
• What are advantages and disadvantages of OLS?
• Hypothesis testing
• How to test whether the slope is zero?
• Confidence intervals
• How to construct a confidence interval for the slope?
The Ordinary Least Squares Estimator (SW Section 4.2)
How can we estimate β0 and β1 from data?
Recall that was the least squares estimator of μY : Ȳ solves,
n
min m  (Yi − m) 2
i =1
By analogy, we will focus on the least squares (“ordinary
least squares” or “OLS”) estimator of the unknown
parameters β0 and β1. The OLS estimator solves,
n
min b0 ,b1 [Yi − (b0 + b1 X i )]2
i =1
The population regression line: E(Test Score|STR) = β0 + β1STR
How can we draw the regression line?
Figure 4.2 Scatterplot of Test score vs Student-Teacher Ratio
n
The OLS estimator solves: minb0 ,b1 [Yi −(b0 + b1 X i )]2
i =1
Let b0 and b1 be some estimators of β0 and β1. Then, the
regression line based on these estimators is b0 +b1X.
• The OLS estimator minimizes the average squared
difference between the actual values of Yi and the
prediction (“predicted value”) based on the
estimated line.
• The result is the OLS estimators of β0 and β1.
4.2: The OLS Estimator, Predicted Values,
and Residuals
The OLS estimators of the slope β1 and the intercept β0 are
n
ˆ1 =
 ( X − X )(Y − Y )
i
i =1
i
n
2
(
X
−
X
)
 i
=
s XY
s X2
(4.7)
i =1
늿
0 = Y − 1 X .
(4.8)
The OLS predicted values Yˆi and residuals uˆi are
Yˆ = 늿+  X , i = 1,..., n
i
0
1
i
uˆi = Yi − Yˆi , i = 1,..., n.
(4.9)
(4.10)
ˆ
The estimated intercept (늿
0 ), slope (1 ), and residual (ui ) are computed
from a sample of n observations of X i and Yi , i = 1,..., n. These are estimates
of the unknown true population intercept ( 0 ), slope (1 ), and error term (ui ).
Using R
Application: the California Test Score – Class Size data
• Estimated slope = 𝛽መ1 = −2.28
• Estimated intercept = 𝛽መ0 = 698.9
෣
• Estimated regression line: 𝑇𝑒𝑠𝑡𝑆𝑐𝑜𝑟𝑒
= 698.9 − 2.28 × 𝑆𝑇𝑅
Interpretation of the estimated slope and intercept
෣ = 698.9 − 2.28  STR
𝑇𝑒𝑠𝑡𝑆𝑐𝑜𝑟𝑒
• TestScore
• Districts with one more student per teacher on
average have test scores that are 2.28 points lower.
• The estimated intercept is 698.9.
• For example, for a district with 20 students per teacher,
the predicted test score is 698.9 – 2.28 x 20 = 653.3.
Predicted values & residuals:
One of the districts in the data set is Antelope, CA,
for which STR = 19.33 and Test Score = 657.8
OLS regression: R output
Measures of Fit (Section 4.3)
Two regression statistics provide complementary measures of
how well the regression line “fits” or explains the data:
• The regression R2 measures the fraction of the variance of
Y that is explained by X; it is unitless and ranges between
zero (no fit) and one (perfect fit)
• The standard error of the regression (SER) measures the
magnitude of a typical regression residual in the units of Y.
The R2 is the fraction of the sample variance of Yi “explained”
by the regression.
n
Definition of R 2:
R2 =
2
늿
(
Y
−
Y
)
 i
ESS i =1
= n
TSS
 (Y − Y )
i =1
2
i
• R2 = 0 means ESS = 0
• R2 = 1 means ESS = TSS
• 0 ≤ R2 ≤ 1
• For regression with a single X, R2 = the square of the correlation
coefficient between X and Y
The Standard Error of the Regression (SER)
The SER measures the spread of the distribution of u.
The SER is almost the sample standard deviation of the OLS
residuals:
1 n
2
늿
SER =
(
u
−
u
)
 i
n − 2 i =1
1 n 2
=
uˆi

n − 2 i =1
𝑛
1
The second equality holds because 𝑢lj = ෍ 𝑢ො 𝑖 = 0.
𝑛
𝑖=1
SER =
1 n 2
uˆ i

n − 2 i =1
The SER has the units of u, which are the units of Y
measures the average “size” of the OLS residual (the
average “mistake” made by the OLS regression line)
The root mean squared error (RMSE) is closely
related to the SER:
1 n 2
RMSE =
uˆi

n i =1
The difference is that the RMSE is divided by n
instead of (n – 2).
Why do we divide by (n – 2)?
1 n 2
SER =
uˆi

n − 2 i =1
The reason we divide by (n-2) is that two regression
coefficients(β0 and β1) were estimated. This is called a
“degrees of freedom” correction.
Example of the R2 and the SER
෣ = 698.9 − 2.28  STR, R 2 = .05, SER = 18.6
TestScore
𝑇𝑒𝑠𝑡𝑆𝑐𝑜𝑟𝑒
STR explains only a small fraction of the variation in test scores.
This means that predictions of test scores made using only the student-teacher
ratio for that district will often be wrong by a large amount.
The low R2 implies that other important factors influence test scores.
The Least Squares Assumptions for
Causal Inference (Section 4.4)
• So far we have treated OLS as a way to draw a straight line
through the data on Y and X. Under what conditions does
the slope of this line have a causal interpretation?
• That is, when will the OLS estimator be unbiased for the causal
effect on Y of X?
• What is the variance of the OLS estimator over repeated
samples?
• To answer these questions, we need to make some assumptions
about how Y and X are related to each other, and about how
they are collected (the sampling scheme)
• These assumptions are known as the Least Squares Assumptions
for Causal Inference.
Definition of Causal Effect
• The causal effect on Y of a unit change in X is the expected
difference in Y as measured in a randomized controlled
experiment.
• For a binary treatment, the causal effect is the expected difference
in means between the treatment and control groups, as discussed
in Ch. 3
• With a binary treatment, for the difference in means to
measure a causal effect requires random assignment or
as-if random assignment.
• Random assignment ensures that the treatment (X) is uncorrelated
with all other determinants of Y, so that there are no confounding
variables
• The least squares assumptions for causal inference
generalize the binary treatment case to regression.
The Least Squares Assumptions for Causal Inference
Let β1 be the causal effect on Y of a change in X:
Yi = β0 + β1Xi + ui , i = 1,…, n
1. The conditional mean of u given X is zero,
that is, E(u|X = x) = 0.
− This implies that ˆ1 is unbiased for the causal effect 1
2. (Xi ,Yi), i = 1,…, n, are i.i.d.
• This is true if (X, Y) are collected by simple random sampling
− This delivers the sampling distribution of 늿
0 and 1
3. Large outliers in X and/or Y are rare.
• Technically, X and Y have finite fourth moments
− Outliers can result in meaningless values of ˆ1
Least squares assumption #1: E(u|X = x) = 0.
When β1 is the
causal effect,
for any given
value of X, the
mean of u is
zero:
Test Scorei = β0 + β1STRi + ui , ui = other factors
• What are some of these “other factors”?
• Is E(u|X = x) = 0 plausible for these other factors?
Least squares assumption #1: E(u|X = x) = 0.
• Consider an ideal randomized controlled experiment:
• X is randomly assigned to people (students randomly assigned to different size classes; patients
randomly assigned to medical treatments).
• Randomization is done by computer (using no information about the individual).
• Because X is assigned randomly, all other individual characteristics – the things that make up u –
are distributed independently of X, so u and X are independent.
• Thus, in an ideal randomized controlled experiment, E(u|X = x) = 0 (that is, LSA #1 holds)
• In actual experiments, or with observational data, we will need to think hard about
whether E(u|X = x) = 0 holds.
Least squares assumption #2: (Xi ,Yi), i = 1,…,n are i.i.d.
This arises automatically if the entity (individual, district) is
sampled by simple random sampling:
• The entities are selected from the same population, so (Xi ,Yi)
are identically distributed for all i = 1,…, n.
• The entities are selected at random, so the values of (X,Y) for
different entities are independently distributed.
The main place we will encounter non-i.i.d. sampling is
when data are recorded over time for the same entity
(panel data and time series data) – we will deal with that
complication when we cover panel data.
Least squares assumption #3:Large outliers are rare.
E(X4) < ∞ and E(Y4) < ∞
• A large outlier is an extreme value of X or Y
• On a technical level, if X and Y are bounded, then
they have finite fourth moments. (Standardized test
scores automatically satisfy this; STR, family income,
etc. satisfy this too.)
• The substance of this assumption is that a large
outlier can strongly influence the results – so we
need to rule out large outliers.
• Look at your data! If you have a large outlier, is it a
typo? Does it belong in your data set? Why is it
an outlier?
OLS can be sensitive to an outlier:
• Is the point an outlier in X or Y?
• Eye-econometrics! Plot your data! Sometimes
they are observations that shouldn’t be in your
data set.
The Sampling Distribution of the OLS Estimator
• quantify the sampling uncertainty associated with ˆ1
• use ˆ1 to test hypotheses such as 1 = 0
• construct a confidence interval for β1
• All these require figuring out the sampling distribution
of the OLS estimator. Two steps to get there…
• Probability framework for linear regression
• Distribution of the OLS estimator
Probability Framework for Linear Regression
Population
• The group of interest (ex: all possible school districts)
Random variables: Y, X
• Ex: (Test Score, STR)
Joint distribution of (Y, X).
The three least squares assumptions.
We assume:
• The population regression function is linear
• E(u| X) = 0 (1st Least Squares Assumption)
• X, Y have nonzero finite fourth moments (3rd L.S.A.)
Data Collection by simple random sampling implies:
• {(Xi, Yi)}, i = 1,…, n, are i.i.d. (2nd L.S.A.)
The mean and variance of the sampling
distribution of ˆ (2 of 3)
n
n
 ( X − X )( X − X )  ( X − X )(u − u )
i
ˆ =  i =1
1
1
i
n
(X − X )
i =1
+
i
i =1
2
i
i
n
(X − X )
i =1
2
i
n
SO
 ( X − X )(u − u )
ˆ1 − 1 = i =1 n
i
i
2
(
X
−
X
)
 i
.
i =1
Now
 n

(
X
−
X
)(
u
−
u
)
=
(
X
−
X
)
u
−
(
X
−
X
)


i
i
i
i
 i
u
i =1
i =1
 i =1

n
 n


=  ( X i − X )ui −   X i  − nX  u
i =1
 i =1 

n
n
n
=  ( X i − X )ui
i =1
The mean and variance of the sampling
distribution of ˆ (3 of 3)
n
n
i =1
i =1
Substitute  ( X i − X )(ui − u ) =  ( X i − X )ui into the expression
for ˆ1 − 1 :
n
 ( X − X )(u − u )
ˆ1 − 1 = i =1 n
i
i
2
(
X
−
X
)
 i
i =1
n
SO
 ( X − X )u
ˆ1 − 1 = i =n1
i
i
2
(
X
−
X
)
 i
i =1
෡ 𝟏 ) 𝒂𝒏𝒅 𝒗𝒂𝒓(𝜷
෡ 𝟏 ):
𝑵𝒐𝒘 𝒘𝒆 𝒄𝒂𝒏 𝒄𝒂𝒍𝒄𝒖𝒍𝒂𝒕𝒆 𝑬(𝜷
 n

(
X
−
X
)
u
i 
 i
i
=
1
E ( ˆ1 ) − 1 = E  n

 ( X − X )2 
i
 

i =1
  n


   ( X i − X )u i 



= E  E  i =n1
 X 1 ,..., X n 
   ( X i − X )2 


  i =1

= 0 because E ( ui | X i = x ) = 0 by LSA #1
• Thus LSA #1 implies that E ( ˆ ) = 
1
1
• That is, ˆ1 is an unbiased estimator of  1 .
• For details see App. 4.3
Next calculate var ( ˆ1 ) (1 of 2)
write
n
1 n
( X i − X )u i
vi


n i =1
ˆ1 − 1 = i =n1
=
 n −1  2
2
(Xi − X )


 sX
 n 
i =1
where vi = ( X i − X )ui . If n is large, s X2   X2 and
ˆ1 − 1 
1 n
vi

n i =1

2
X
,
where vi = ( X i − X )ui (see App. 4.3). Thus,
n −1
 1, so
n
Next calculate var ( ˆ1 ) (2 of 2)
ˆ1 − 1 
1 n
vi

n i =1
 X2
n
var(vi )/n


1
2 2
so
var( 늿
−

)
=
var(

)
=
var
v
(

)
=
1
1
1
X
n i
( X2 ) 2
 i =1 
where the final equality holds under the LSA#2. Thus,
Summary
1 var[( X i −  x )ui ]
ˆ
var( 1 ) = 
.
2 2
n
( X )
1. 𝛽መ1 is unbiased: under LSA#1, 𝐸(𝛽መ1 ) = 𝛽1
2. var( 𝛽መ1 ) is inversely proportional to 𝑛
What is the sampling distribution of ˆ1?
The exact sampling distribution depends on the population
distribution of (Y, X). However, when n is large we get some
simple approximations:
p
?
1) Because var( 늿
1 )  1/n and E ( 1 ) = 1 , 1 → 1
2) When n is large, the sampling distribution of ˆ is well
1
approximated by a normal distribution (CLT)
Recall the CLT : suppose {vi }, i = 1,..., n is i.i.d. with E (v) = 0 and
1 n
2
var(v) =  . Then, when n is large,  vi is approximately
n i =1
distributed N (0,  v2 /n).
Large-n approximation to the
distribution of ˆ1:
1 n
1 n
vi
vi


n i =1
n
ˆ1 − 1 =
 i =21 , where vi = ( X i − X )ui
X
 n −1  2

 SX
 n 
• When n is large, vi = ( X i − X )ui  ( X i −  X )ui , which is i.i.d.
1 n
( why ?) and var(vi )   ( why ?). So, by the CLT,  vi is
n i =1
2
approximately distributed N (0,  v /n).
The large-sample normal distribution of
is
2



v
ˆ
1 ~ N  1 ,
, where vi = ( X i −  X )ui
2 2 
 n( X ) 
The larger the variance of X , the smaller
the variance of ˆ1
Where  X2 = var( X i ). The variance of X appears (squared) in the
denominator − so increasing the spread of X decreases the variance
of 1.
With more variation in X, more information is available in the
data that we can use to fit the regression line.
The larger the variance of X , the smaller
the variance of ˆ1
The number of black dots = The number of blue dots
To get a more accurate regression line, which would you like to use?
Summary of the sampling distribution of ˆ1 :
If the three Least Squares Assumptions hold, then
• The exact (finite sample) sampling distribution of ˆ1 has:
− E ( 늿) =  (that is,  is unbiased)
1
1
1
1 var[( X i −  x )ui ]
1
ˆ
− var( 1 ) = 
 .
4
n
X
n
• Other than its mean and variance, the exact distribution of ˆ1 is
complicated and depends on the distribution of (X , u )
p
• 늿
1 → 1 (that is, 1 is consistent)
늿
− E ( 1 )
• When n is large, 1
~ N (0, 1) (CLT)
var( ˆ1 )
• This parallels the sampling distribution of Y .
If the least squares assumptions in Key Concept 4.3 hold, then in
large samples 늿and  have a jointly normal sampling distribution.
0
1
The large-sample normal distribution of ˆ1 is N ( 1 ,  2ˆ ), where the
1
variance of this distribution,  2ˆ , is
1
1 var[( X i −  X )ui ]
 =
.
(4.21)
2
1
n [var( X i )]
The large-sample normal distribution of ˆ is N (  ,  2 ), where
2
ˆ
0
 2ˆ =
0
0
ˆ0
 X 
1 var( H i ui )
,
where
H
=
1
−
Xi.
i

2 2
2 
n [ E ( H i )]
 E( X i ) 
(4.22)
We are now ready to turn to hypothesis tests & confidence intervals…
The Least Squares Assumptions for Prediction (Appendix 4.4)
• Prediction requires using an estimation sample to
estimate a prediction model. Then we use that
model to predict the value of Y for an observation
not in the estimation sample.
• Prediction requires good out-of-sample performance.
• For prediction, β1 is simply the slope of the
population regression line (the conditional
expectation of Y given X).
• The critical LSA for Prediction is that the out-ofsample (“OOS”) observation for which you want to
predict Y comes from the same distribution as the
data used to estimate the model.
• This replaces LSA#1 for Causal Inference
The Least Squares Assumptions for Prediction (Appendix 4.4)
1. The out of sample observation (XOOS,YOOS) is drawn
from the same distribution as the estimation sample
(Xi ,Yi), i = 1,…,n
•
This ensures that the regression line fit using the estimation
sample also applies to the out-of-sample data to be
predicted.
2. (Xi ,Yi), i = 1,…,n are i.i.d.
• This is the same as LSA#2 for causal inference
3. Large outliers in X and/or Y are rare
•
This is the same as LSA#3 for causal inference