Lecture 2:
The Linear Regression Model
Đinh Thi Thanh Binh, PhD
Faculty of International Economics, FTU
1
1. Introduction to regression model
• The term « regression » means «regression to
mediocrity»
• Defined by Galton (1886) when he studied the
relationship between the height of sons and the
height of fathers
2
Distribution of the height of sons respects
to the height of the fathers
3
The study shows that:
• Given the height of fathers, the height of sons will
distribute around a medium value
• On average, when the height of fathers increase, the
height of sons also increase
• If we conect all the medium points, we will have a
linear line
• This line is called regression line, showing the
relationship between the height of sons and the height
of father on average
4
2. Population Regression Function (PRF)
and Sample Regression Function (SRF)
2.1. Definition of PRF
PRF is a regression function that is constructed based
on the survey of the population
For example: For example, Galton studied the
relationship between the height of fathers and the height
of sons in one city. He collected the data of all fathers
having adult sons. So he can build PRF.
5
 So, E(Y|Xi) is a function of independent variable Xi :
E(Y/Xi)= f(Xi) = β0+ β1Xi [1]
• The equation [1] is called Population regression
function (PRF).
– PRF shows how the expected value of Y changes at
different values of X
– If PRF has 1 independent variable  simple
regression function
– If PRF has 2 or more independent variables 
multiple regression function
6
• Suppose that PRF E(Y|Xi) is a linear function,
then:
E(Y|Xi)= β0+ β1Xi [2]
- β0, β1: regression coefficients/ parameters
• β0: constant coefficient
• β1: slope coefficient
• The equation [2] is a simple regression function
7
2.2. Error/ disturbance term
• Because E(Y|Xi) is expected value of Y given Xi,
single values of Yi is not necessary the same with
E(Y|Xi), but they are around E(Y/Xi).
• Note ui is the difference between Yi and E(Y/Xi), we
have:
ui= Yi- E(Y|Xi)
[3]
Or :
Yi= E(Y|Xi)+ ui
[4]
 ui is a random variable/ component or disturbance
8
2.3. Sample regression function (SRF)
• In reality, we can not carry out surveys of population
 we can not build PRF
• Then we only can estimate the expected value of Y, or
in other words, estimate PRF based on sample(s)
taken from population
• Obviously the estimated SRF can not be absolutely
exact
The regression function that is constructed based on a
sample is called Sample Regression Function (SRF).
9
Graph 2.03. Scatter graph and regression line of the 2
samples SRF1 và SRF2
10
• From the population, we can get many samples.
With each sample, we can have a SRF
• To have the “best” SRF, meaning that that SRF is
the closest estimate of PRF, we have to base on
some criteria even when we do not have PRF to
compare.
11
2.3.Sample Regression Function (SRF) – Simple
PRF
SRF
 E (Y / X i )  0  1 X i

Yi  E (Y / X i )  ui  0  1 X i  ui
Yˆi  ˆ0  ˆ1 X i

ˆ
ˆ
ˆ
ˆ
ˆ
Y

Y

u




 i i i
0
1 X i  ui
• 𝑌𝑖 is an estimate of E(Y/Xi) and is a fitted value/
predicted value of Y
• 0 , 1 are estimates of β0, β1
• 𝑢𝑖 is an estimate of ui and is called as residuals
12
2.3.Sample Regression Function (SRF) – Multiple
• PRF:
E (Y / X 1 , X 2 )   0  1 X 1   2 X 2  ...   k X k
Y   0  1 X 1   2 X 2  ...   k X k  u
• SRF:
Yˆ  ˆ0  ˆ1 X 1  ˆ2 X 2  ...  ˆk X k
Y  ˆ0  ˆ1 X 1  ˆ2 X 2  ...  ˆk X k  u
13
3. The Ordinary Least Square (OLS)
• The method OLS is invented by the German
mathematician - Carl Friedrich Gauss.
• It is used to estimate parameters given some
assumptions.
• The estimates have some properties (linearity,
unbiasedness, and efficiency).
• This method is used the most popularly now.
14
3.1. The Ordinary Least Square (OLS)
• Assume that:
PRF has the form: Yi= β0+ β1Xi+ ui
[3.01]
• Because we can not have PRF, we have to estimate it
through SRF
SRF: 𝑌𝑖 = β0 + 1 𝑋𝑖 + 𝑢𝑖 = 𝑌𝑖 + 𝑢𝑖
[3.02]
Where Yˆi is the predicted/ fitted value of Yi
15
3.1. The Ordinary Least Square (OLS)
From [3.02], we have:
uˆi  Yi  Y i [3.03]
[3.03] ûi is the difference between the actual
value and predicted value of Yi.
 If ûi is smaller and smaller, the difference
between Yi and Yˆi is smaller. Then, the estimated
value Yˆi is closer to Yi.
16
3.1. The Ordinary Least Square (OLS)
• Suppose that we have n observations of Y and X,
we try to find SRF so that Yˆi is closest to Y.
• It means that we have to choose SRF so that the
sum of residuals:
n
n
i 1
i 1
 uˆi   (Yi  Yˆi )
has the minimum value
• However, this is not the best choice because of
some following reasons:
17
SRF
Y
Yi
Yˆi  ˆ1  ˆ2 X i
û3
û 4
û1
û 2
X1
X2
X3
X4
X
18
3.1. The Ordinary Least Square (OLS)
We can overcome all these problems by choosing
SRF so that:
n
n
n
i 1
i 1
i 1
2
2
ˆ  ˆ X ) 2
ˆ
ˆ
u

(
Y

Y
)

(
Y


i  i i  i 0 1 i
n
has minimum value. Of which,  uˆi2
i 1
squared residuals (SSR).
is the sum of
19
3.1. The Ordinary Least Square (OLS)
n
2
ˆ
u
From the equation [3.03], we have  i is a function
i 1
ˆ
ˆ
of  and 
1
0
n
n
i 1
i 1
2
ˆ , ˆ )  (Y  ˆ  ˆ X )2
ˆ
u

f
(

i
 i 0 1 i
0
1
20
3.1. The Ordinary Least Square (OLS)
• We have known that, f(X) is minimized when:
↔
 f '(X )  0

 f ''(X )  0
21
3.1. The Ordinary Least Square (OLS)
n
• So, the function  uˆi2 is minimized when:
i 1
 f ' (u )  0

 f ' ' (u )  0
22
• Then, we have ˆ0 and ˆ1
equations:
are the results of the
f ( ˆ0 , ˆ1 ) n
  2(Yi  ˆ0  ˆ1 X i )(1)  0
ˆ
i 1
0
n
n
i 1
i 1
nˆ0  ˆ1  X i   Yi
23
f ( ˆ0 , ˆ1 ) n
  2(Yi  ˆ0  ˆ1 X i )( X i )  0
ˆ
i 1
1
ˆ
0
n

i 1
X  ˆ
i
1
n
n
 X  Y X
i 1
2
i
i 1
i
i
24
• In sum:
n
n
i 1
i 1
nˆ0  ˆ1  X i   Yi
[3.05]
ˆ
0
n

i 1
X  ˆ
i
1
n
n
 X  Y X
i 1
2
i
i 1
i
i
25
•Solve the equations [3.05], we have:
n
ˆ1 
n
n
n X iYi   X i  Yi
i 1
i 1
n
n
i 1
[3.06] n X  ( X i )
i 1
2
i
n
n
 ( X  X )(Y  Y )
 i 1
2
i 1
n
i
n
2
(
X

X
)
 i
i 1
n
n
 X Y   X  X Y
ˆ0  i 1
2
i
i 1
n
i
i 1
n
i
i 1
n X  ( X i )
i 1
2
i
i
i i
 Y  ˆ1 X
2
i 1
26
Example
A random sample as followings:
X: Personal income/ day in thousand vnd
Y: Personal consumption/ day in thousand vnd
Questions:
a. Calculate the main properties of X and Y (Expected value,
variance, median, mod)
b. Estimate the parameters of the SRF
c. Write SRF
27
Example
X: Personal income/ day in thousand vnd
Y: Personal consumption/ day in thousand vnd
X
Y
5
1
4
2
2
3
8
4
8
5
Questions:
a. Calculate the main properties of X and Y (Expected value,
variance, median, mod)
b. Estimate the parameters of the SRF
c. Write SRF
d. Calculate SST, SSE, SSR, R-square
e. Explain the meaning of R-square
28
• Beta0 = 69/68; beta1 = 25/68
• SST = 10, SSE = 125/34, SSR = 215/34, R-square =
0.3676
• R-square = 0.3676
It means that: Income can explain 36.76% of the sample
variation in consumption of people. So 63.24% of the
the sample variation in consumption of people is
explained by other independent variables that are not
included in the model.
29
Example
X: Personal income/ day in thousand vnd
Y: Personal consumption/ day in thousand vnd
X
Y
6
5
5
2
2
2
4
3
4
1
Questions:
a. Calculate the main properties of X and Y (Expected value,
variance, median, mod)
b. Estimate the parameters of the SRF
c. Write SRF
d. Calculate SST, SSE, SSR, R-square
e. Explain the meaning of R-square
30
• E(X) = 21/5
• E(Y) = 13/5
• Var(X) = 1.76
• Var(Y) = 1.84
• Beta0 = 1/44 ; beta1 = 27/44
• SST = 9.2, SSE = 3.3, SSR = 5.9, R-square = 35.86%
It means that: Income can explain 35.86% of the sample
variation in consumption of people. So 64.14% of the the
sample variation in consumption of people is explained by
other independent variables that are not included in the
model.
31
3.2. The statistical properties of OLS estimators
• The OLS estimators are expressed solely in terms of
the observable (i.e., sample) quantities.
• They are point estimators; that is, given the sample,
each estimator will provide only a single (point) value
of the relevant population parameter.
• Once the OLS estimates are obtained from the sample
data, the sample regression line can be easily
obtained.
32
3.2. The statistical properties of OLS estimators
The regression line thus obtained has the following
properties:
1. It passes through the sample means of Y and X ( X , Y )
2. The mean value of the estimated Yˆi is equal to the
mean value of the actual Y
Yˆ  Y
3. The mean value of the residuals is zero.
n
 uˆ  0
i 1
i
33
3.2. The statistical properties of OLS estimators
4. The residuals 𝑢𝑖 are uncorrelated with the predicted
𝑌𝑖 :
n
ˆ
Y
 iuˆi  0
i 1
5. The residuals 𝑢𝑖 are uncorrelated with 𝑋𝑖 :
n
 uˆ X  0
i 1
i
i
34
3.3. The sum of squares
• SST (Total Sum of Squares)
SST   (Yi  Y )2
• SSE: (Explained Sum of Squares)
2
ˆ
SSE   (Yi  Y )
• SSR: (Residual Sum of Squares)
n

SSR   Y i Y i
i 1
 
2

35
u
2
i
• SST is a measure of the total sample variation in the yi; that is,
it measures how spread out the yi are in the sample
• SSE measures the sample variation in the Yˆi
• SSR measures the sample variation in the u i
• The total variation in Y can always be expressed as the sum of
explained variation SSE and the unexplained variation SSR.
Thus,
SSE SSR
1

SST SST
SST = SSE + SSR
36
3.4. Determination Coefficient (R-squared)
R2 is the fraction (percentage) of the sample variation in Y that
is explained by X
SSE
SSR
R 
 1
SST
SST
2
Simple linear regression
n
R2 
n
Multiple linear regression
n
( n  X iYi   X i  Yi ) 2
i 1
n
i 1
n
i 1
n
n
[n  X i  (  X i ) ][n  Yi  (  Yi ) 2 ]
2
i 1
2
i 1
2
i 1
i 1
2
[
()
Y

Y
(

Y
)
]
2 
i
R
i Y
2
2
[
(

Y
)
]
[
(

Y
)
]


Y
Y
i
i
37
Properties of R2
R22≤1
≤1
0≤ R
100* R2 is the percentage of the sample variation
in y that is explained by x
R2 =1: the data points all lie on the same line,OLS
provide a perfect fit to the data
R2 =0: X and Y has no relationship
Weakness: R2 increases when including more X in
the model, even they have no significant effect on Y.
=>Use adjusted-R2 to determine the variables added
in the model
38
3.5. AjustedR2
n 1
R  1  (1  R )
n  k 1
2
2
• When adding new variable into the model,
adjusted R2 increases  that variable should
be included in the model
39
3.6. Assumptions of the OLS
Assumption 1- Linear in parameters: In the PRF, the
dependent variable, y, is related to the independent
variable, x, and the error term, u, as
Y     X u
0
1
Assumption 2 – Random sampling: We have a
random sample of size n
Assumption 3 – Sample variation in the explanatory
variable: The sample outcomes on x, namely {xi , i =
1,…, n}, are not all the same value.
40
3.6. Assumptions of the OLS
• Assumption 4 – No perfect collinearity: In
the sample, there are no exact linear
relationships among the independent variables.
41
• Assumption 5: The error term has an expected value of
zero given any value of the explanatory variable. In other
words, E(u|X)=0.
 This assumption simply says that the factors not explicitly
included in the model, therefore subsumed in 𝑢𝑖 , do not
systematically affect the mean value of Y; the positive 𝑢𝑖
values cancel out the negative 𝑢𝑖 values so that their average
or mean effect on Y is zero.
 Geometrically, this assumption can be pictured as in Figure
3.3, which shows a few values of the variable X and Y
populations associated with each of the them. As shown,
each Y corresponding to a given X is distributed around its
mean value.
42
Figure 3.3. Conditional distribution of the
disturbances ui
43
Assumption 6 - Homoskedasticity: The error term ui
has the same variance given any value of the
independent variable. In other words,
var (ui/Xi)= E[ui- E(ui/Xi)]2= E(ui2/Xi)= σ2
Var(u) reflects the distribution of Y surrounding its
E(Y|X).
 This assumption means that Y corresponding to
various X values have the same variance. The variance
surrouding the regression line is the same across the X
values, it neither increases nor decreases as X varies.
44
Figure 3.4. The simple regression model under
homoskedasticity
45
• Consider figure 3.5, where the conditional variance of
population Y varies with X.
• Let Y represents for weekly consumption expenditure
and X represents for weekly income.
• Figure 3.4 and 3.5 show that as income increases, the
average consumption expenditure also increases.
• In figure 3.4, the variance of consumption expenditure
remains the same at all levels of income.
• In figure 3.5, it increases with the increases in income.
• Richer families on average consume more than poor
families. There is also more variability in the
consumption expenditure of the former.
46
Figure 3.5. The simple regression model under
heteroskedasticity
47
3.7. Properties of the OLS estimators Gauss-Markov Theorem
• The OLS estimators are the best, linear,
unbiased estimators (BLUE).
• The Gauss- Markov Theorem: Under the
OLS assumptions, the estimators are BLUE (
best, linear, unbiased estimators).
48
– Linear: OLS Estimators are linear functions
of a random variable
– Unbiased: E (ˆ j) = βj
– The best: Smallest variance among layers of
unbiased estimators
49
Theorem 1: Unbiasedness of OLS
Given assumptions, we have:
E (  )   , and E (  )  
0
0
1
1
for any values of  0and  1 . In other words,  0 is unbiased
for  0 , and  1 is unbiased for . 1
50
Unbiased
E ( ˆ2 )   2
ˆ2
(a) Phân phối mẫu của β2
51
Theorem 2: Sampling variances of the OLS estimators
Under assumptions 1 through 6,
2
Var( ˆ j )  n
 ( X  X ) (1  R )
2
i 1
ij
j
2
j
j =1,2,….,k
2
2
R is R of regression of X on other independent
j
variable
j
Best - efficient
ˆ2
 2*
2
ˆ2 ,  2*
(c ) Phân phối mẫu của β2 và β2*
53
3.8. Measuring the accuracy of OLS estimates
Variance of estimators
Var( ˆ j )  n
2
2
(

)
 X ij X j (1  R j )
2
i 1
Standard deviation of estimators
sd ( ˆ ) 
j

1/2
(1

)]
[SST R j
2
j
Standard error of the regression ( 𝜎 )
𝑛
𝑢𝑖 2 )/(𝑛 − 𝑘 − 1)
𝜎2 = (
𝑖=1
(n-k-1): degree of freedom; n: observations; k: number
of independent
Standard error of the regression:
𝑠𝑒(𝛽𝑗 ) =
𝜎
[𝑆𝑆𝑇𝑗 (1 − 𝑅𝑗 2 )1/2
3.9. The components of the OLS variances
Var( ˆ j )  n
2
2
(

)
 X ij X j (1  R j )
2
i 1
1. The error variance, 𝟐
• A larger 2 means larger 𝑣𝑎𝑟 𝑗
• More noise, larger 2 , makes it more difficult to
estimate the partial effect of any independent
variables on Y
• To reduce the error variance, there is only one
way is to add more explanatory variables
2. The total sample variance in 𝑿𝒋 ,  ( X  X )
n
i 1
ij
2
j
• The larger the total variation in 𝑥𝑗 is, the smaller
𝑣𝑎𝑟 𝑗 is
• Everything else being equal, for estimating 𝑗 we
prefer to have as much sample variation in 𝑋𝑗 as
possible
• Increase the sample size
3. The linear relationships among the
𝟐
independent variables 𝑹𝒋
• 𝑹𝟐𝒋 is the proportion of the total variation in 𝑋𝑗
that can be explained by other independent
variables
•
𝑹𝟐𝒋 =0, happens if, and only if, 𝑋𝑗 has zero
sample correlation
3.10. Units of measurement
How does changing the units of measurement of
the dependent and/or independent variables affect
OLS estimates?
59
3.10. Units of measurement
• Example: data set “CEO Salary and Return on Equity”
Salary:
Roe:
salary per year in thousands dollar of CEO
average return on equity in percentage
salary  963.191  18.501roe
=> When roe increases by 1%, salary per year of CEO is
expected to increase by 18.501 thousand usd
60
Case 1
• When salary is measured in usd  salarydol =
1000*salary
• The unit of roe is unchanged
salarydol  963191  18501roe
=> If the dependent variable is multiplies or divided by
the constant c, then the OLS intercept and slope
estimates are also multiplies or divided by c.
61
Case 2
• When the unit of salary unchanged
• The unit of roe changed: roedec = roe/100
salary  963.191  1850.1roedec
• Coefficient of roedec is 100 times greater than the
coefficient of roe in [1]
=> If the independent variable is divided or multiplied
by some non zero constant c, then the OLS slope
coefficient is multiplied or divided by c, respectively. The
intercept is unchanged.
62
3.11. “Linear” in regression model
• Linear regression requires “linear” only in
parameters, not variables.
yes
Yi  1   2 X i  ui
1
Y  1   2  ui
X
Y  1  (1  2 )2 X  ui
Ln(𝑌𝑖 ) = 𝛽1 + 𝛽2 /𝑋𝑖 + 𝑢𝑖
yes
no
yes
63
3.12. Functional form
Model
Dependent
variable
Independent
variable
Interpretation
of  1
Lin - lin
y
x
y   x
1
Lin-log
y
log(x)
y  ( /100)%x
1
Log-lin
log(y)
x
%y  (100 )x
1
Log-log
log(y)
log(x)
%y   %x
1
64
Data on “Wage and Education”
Wage: usd/ hour
Educ: years of education
1. Lin-lin:
wage  0,90  0,54educ
•
•
Each year of education is expected to increase the wage
by 54 cent per hour.
Because wage and education has a linear relation  the
effect of education on wage is the same for all the
education levesl (54 cent)  the effect of the 2nd
education level equals the 20th education level, for
example
65
2. Log-lin:
•
•
•
•
logwage  0,584  0,083educ
Explanation:
%y  (100 )x
1
Each year of higher education will increase wage by a
certain %  The change of wage increases when the
year of education increases  increasing return to
education
Each year of higher education will increase wage by
8.3%.
The higher education, the higher value/ effect
66
log(wage)=  + educ+u  wage  exp(   educ  u)
0
1
0
1
wage
u  0,   0
1
educ
67
3. Lin-log:
•
•
demand  0,584  94,3log( price)
Explanation: y  ( 1 /100)%x
When price of product X increases by 1% , demand for
that product decreases by 0,94 thousand of unit.
68
4. Log-log:
log(demand)  0,584  0,253log( price)
•
Explanation: %y  ( )%x
1
•
When price of product X increases by 1%, the demand
for X decreases by 0.25%
69