CHAPTER TWO
Simple Linear Regression
By : Habtamu Legese (M.Sc.)
By: Habtamu Legese Feyisa
1
2.1. Concept of Regression Function
• Regression analysis is one of the most commonly used
tools in econometrics analysis. What is regression?
• Regression is a statistical method that attempts to
determine the strength and character of the relationship
between one dependent variable and a series of other
variables (known as independent variables).
• Regression analysis is concerned with describing and
explaining the relationship between the dependent and
one or more independent variables.
By: Habtamu Legese Feyisa
2
By: Habtamu Legese Feyisa
3
Cont.
• We denote the dependent variable by Y and the explanatory
variables by X 1 , X 2 ,..., X k .
• If k = 1 , that is, there is only one explanatory variable, we
have what is known as simple regression. This is what we
discuss in this unit.
• On the other hand, if K > 1, that is, there are more than one
explanatory variable, we have what is known as multiple
regression.
• In a regression analysis our task is to estimate the population
regression function (PRF) on the basis of sample regression
function (SRF) as accurately as possible.
By: Habtamu Legese Feyisa
4
Regression analysis has following objectives & uses
➢To show the relationship among variables.
➢To estimate average value (mean) of the dependent variable
given the value of independent variable(s);
➢To test hypothesis about sign and magnitude of relationship
➢To forecast future value(s) of the dependent variable
❑It is to explain the variation in the dependent variable based on
the variation in one or more independent variables.
By: Habtamu Legese Feyisa
5
2.1. Stochastic and Non-stochastic Relationships
• A relationship between X and Y, characterized as Y = f(X) is
said to be deterministic or non-stochastic if for each value of
the independent variable (X) there is one and only one
corresponding value of dependent variable (Y).
• On the other hand, a relationship between X and Y is said to
be stochastic if for a particular value of X there is a whole
probabilistic distribution of values of Y.
• In such a case, for any given value of X, the dependent
variable Y assumes some specific value only with some
probability.
By: Habtamu Legese Feyisa
6
Example ………. Supply function
• Assuming that the supply for a certain commodity depends
on its price (other determinants taken to be constant) and the
function being linear, the relationship can be put as:
• The above relationship between P and Q is such that for a
particular value of P, there is only one corresponding value of
Q. This is, therefore, a deterministic (non-stochastic)
relationship since for each price there is always only one
corresponding quantity supplied.
By: Habtamu Legese Feyisa
7
Cont.
• This implies that all the variation in Y is due solely to
changes in X, and that there are no other factors affecting the
dependent variable.
• If this were true all the points of price-quantity pairs, if
plotted on a two dimensional plane, would fall on a straight
line.
• However, if we gather observations on the quantity actually
supplied in the market at various prices and if we plot them
on a diagram we may see that they do not fall on a straight
line.
By: Habtamu Legese Feyisa
8
Scatter Diagram
By: Habtamu Legese Feyisa
9
Cont.
• The deviation of the observation from the line may be
attributed to several factors.
A. Omission of variables from the function
B. Random behaviour of human beings
C. Imperfect specification of the mathematical form of
the model
D. Error of aggregation
E. Error of measurement
By: Habtamu Legese Feyisa
10
Cont.
• In order to take into account the above sources of
errors we introduce a random variable which is
usually denoted by the letter ‘μ’ or ‘ε’ and is called
error term or random disturbance or stochastic term
of the function.
• By introducing this random variable in the function the
model is rendered stochastic of the form:
By: Habtamu Legese Feyisa
11
Cont.
• Thus a stochastic model is a model in which the
dependent variable is not only determined by the
explanatory variable(s) included in the model but also
by others which are not included in the model.
By: Habtamu Legese Feyisa
12
Cont.
• If we have n observations on Y and X, we can write the
simple regression model (2.1.2) by adding subscripts as
• Furthermore, the above regression equation can be rearranged
for the error term as
By: Habtamu Legese Feyisa
13
2.2. Simple Linear Regression model.
• A stochastic relationship with only one explanatory
variable is called simple linear regression model.
➢The term ‘simple’ refers to the fact that we use only two
variables (one dependent and one independent variable).
➢Linear refers to linear in parameters, it may or may not be
linear in the variable. The parameters appear with a power
of one & is not multiplied/divided by other parameters.
By: Habtamu Legese Feyisa
14
Cont.
• The true relationship which connects the variables involved is
split into two parts: a part represented by a line and a part
represented by the random term ‘u’.
By: Habtamu Legese Feyisa
15
Cont.
• The scatter of observations represents the true relationship
between Y and X.
• The line represents the exact part of the relationship and
the deviation of the observation from the line represents the
random component of the relationship.
By: Habtamu Legese Feyisa
16
The Gaussian, standard, or classical linear regression model (CLRM)
• CLRM is the cornerstone of most econometric theory,
makes 10 assumptions.
• It is classical in the sense that it was developed first by
Gauss in 1821 and since then has served as a norm or
a standard against which may be compared the
regression models that do not satisfy the Gaussian
assumptions.
By: Habtamu Legese Feyisa
17
Assumption 1: Linear regression model.
1. The model is linear in parameters.
• The classicals assumed that the model should be linear in the
parameters regardless of whether the explanatory and the
dependent variables are linear or not.
• This is because if the parameters are non-linear it is difficult
for estimation.
• Example :
By: Habtamu Legese Feyisa
18
Assumption 2: X values are fixed in repeated sampling.
• Values taken by the regressor X are considered fixed in
repeated samples.
• More technically, X is assumed to be non-stochastic. In other
words X is assumed to be known with certainty.
• What all this means is that our regression analysis is
conditional regression analysis, that is, conditional on the
given values of the regressor (s) X.
By: Habtamu Legese Feyisa
19
Assumption 3: Zero mean value of disturbance ui.
• That is; given the value of X the mean or expected value of
the disturbance term is zero.
• Technically, the conditional mean value of
is zero.
By: Habtamu Legese Feyisa
20
Assumption 4: Homoscedasticity or equal variance of u .
i
• Given the value of X, the variance of ui is the same for all
observations.
For all values of X, the u’s will show the same dispersion
around their mean.
• This constant variance is called homoscedasticity assumption
and the constant variance itself is called homoscedastic
variance.
By: Habtamu Legese Feyisa
21
Cont.
The disturbance term (U) has a normal distribution
• This means the values of u (for each x) have a bell
shaped symmetrical distribution about their zero mean
and constant variance
, i.e.
By: Habtamu Legese Feyisa
22
Cont.
Randomness of disturbance term 𝒖𝒊 : the error term is
assumed to be a random variable.
• This means that the value which u may assume in any one
period depends on chance; it may be positive, negative or
zero.
• Every value has a certain probability of being assumed by u
in any particular instance.
By: Habtamu Legese Feyisa
23
Assumption 5: No autocorrelation between the disturbances terms.
• This means the value which the random term assumed in one
period does not depend on the value which it assumed in any
other period.
• Algebraically,
By: Habtamu Legese Feyisa
24
Assumption 6: Zero covariance between ui and Xi
This means there is no correlation between the random
variable and the explanatory variable.
• If two variables are unrelated their covariance is zero.
By: Habtamu Legese Feyisa
25
Assumption 7: The number of observations n must be greater than
the number of parameters to be estimated
• Alternatively, the number of observations n must be
greater than the number of explanatory variables.
• From a single observation there is no way to estimate
the two unknowns, α and β.
• We need at least two pairs of observations to estimate
the two unknowns.
By: Habtamu Legese Feyisa
26
Assumption 8: Variability in X values
✓The X values in a given sample must not all be the same.
✓Technically, var (X) must be a finite positive number.
ഥ , it is impossible to estimate the parameters.
✓if 𝑿𝒊 = 𝑿
By: Habtamu Legese Feyisa
27
Assumption 9: The regression model is correctly specified.
• Alternatively, there is no specification bias or error in the
model used in empirical analysis.
• Some important questions that arise in the specification of
the model include the following:
(1) What variables should be included in the model?
(2) What is the functional form of the model? Is it linear in
the parameters, the variables, or both?
(3) What are the probabilistic assumptions made about the
Yi , the Xi, and the ui entering the model?
By: Habtamu Legese Feyisa
28
Assumption 10: There is no perfect multicollinearity.
• That is, there are no perfect linear relationships among the
explanatory variables.
By: Habtamu Legese Feyisa
29
Cont.
• We can now use the above assumptions to derive the following basic
concepts.
A. The dependent variable Y is normally distributed.
i
B. successive values of the dependent variable are
independent, i.e.
By: Habtamu Legese Feyisa
30
A. The dependent variable Y is normally distributed.
By: Habtamu Legese Feyisa
31
B. successive values of the dependent variable are independent
By: Habtamu Legese Feyisa
32
2.2.2 Methods of estimation
•The parameters of the simple linear regression
model can be estimated by various methods.
Three of the most commonly used methods
are:
1. Ordinary least square method (OLS)
2. Method of moments (MM)
By: Habtamu Legese Feyisa
33
Ordinary least square method (OLS)
• The model Y = a + bX + U is called the true relationship
between Y and X because Y and X represent their respective
population value, a and b are called the true parameters
since they are estimated from the population value of Y and
X.
• But it is difficult to obtain the population value of Y and X
because of technical or economic reasons.
• So we are forced to take the sample value of Y and X. The
parameters estimated from the sample value are called the
estimators of the true parameters a and b and are symbolized
as
and
.
i
i
i
By: Habtamu Legese Feyisa
34
Cont.
• Estimation of a and b by least square method (OLS)
involves finding values for the estimates
and
which will minimize the sum of square of the residuals
By: Habtamu Legese Feyisa
35
Method of Ordinary least square method (OLS)
By: Habtamu Legese Feyisa
36
Cont.
Cont.
• Substituting the values of α from (2.10) to (2.13), we get:
Cont.
Cont.
Substituting (2.15) and (2.16) in (2.14), we get
Summary
=
=
Cont.
=
By: Habtamu Legese Feyisa
44
The formula in deviation form
By: Habtamu Legese Feyisa
45
Summary
=
=
By: Habtamu Legese Feyisa
46
2.3 The Method of Least Squares
Numerical
Example:
Explaining sales
= f(advertising)
Sales are in
thousands
of Birr &
advertising
expenses are in
hundreds of Birr.
By: Habtamu Legese Feyisa
Firm (i)
Sales (Yi)
Advertising Expense (Xi)
1
2
3
4
5
6
7
8
9
10
11
10
12
6
10
7
9
10
11
10
10
7
10
5
8
8
6
7
9
10
47
2.3 The Method of Least Squares
.
i
Yi
Xi
X iYi
X i2
1
11
10
110
100
2
10
7
70
49
3
12
10
120
100
4
6
5
30
25
5
10
8
80
64
6
7
8
56
64
7
9
6
54
36
8
10
7
70
49
9
11
9
99
81
10
10
10
100
100
Ʃ
96
80
789
668
By: Habtamu Legese Feyisa
96
Y =
= 9.6
10
80
X=
=8
10
X iYi − nXY

ˆ
b=
2
2
X
−
n
X
 i
789 − 10(8)(9.6)
ˆ
b=
= 0.75
2
668 − 10(8)
aˆ = Y − bˆX
aˆ = 9.6 − 0.75(8) = 3.6
48
2.3 The Method of Least Squares
.
i
Yi
Xi
X iYi
X i2
1
11
10
110
100
2
10
7
70
49
3
12
10
120
100
4
6
5
30
25
5
10
8
80
64
6
7
8
56
64
7
9
6
54
36
8
10
7
70
49
9
11
9
99
81
10
10
10
100
100
Ʃ
96
80
789
668
By: Habtamu Legese Feyisa
49
2.3 The Method of Least Squares
1.4
𝑥y
2.8
𝑥𝑖2
4
-1
0.4
-0.4
1
10
2
2.4
4.8
4
6
5
-3
-3.6
10.8
9
5
10
8
0
0.4
0
0
6
7
8
0
-2.6
0
0
7
9
6
-2
-0.6
1.2
4
8
10
7
-1
0.4
-0.4
1
9
11
9
1
1.4
1.4
1
10
10
10
2
0.4
0.8
4
Ʃ
96
80
0
0
21
28
i
Yi
Xi
𝑋𝑖 − 𝑋ሜ
Y−𝑌ሜ
1
11
10
2
2.
10
7
3
12
4
By: Habtamu Legese Feyisa
50
2.3 The Method of Least Squares
i
Yi
Xi Yˆi = 3.6 + 0.75 X i ei = Yi − Yˆi
1
11
10
11.1
-0.10
0.01
2
10
7
8.85
1.15
1.3225
3
12
10
11.10
0.90
0.81
4
6
5
7.35
-1.35
1.8225
5
10
8
9.60
0.40
0.16
6
7
8
9.60
-2.60
6.76
7
9
6
8.10
0.90
0.81
8
10
7
8.85
1.15
1.3225
9
11
9
10.35
0.65
0.4225
10
10
10
11.10
-1.10
1.21
Ʃ
96
80
96
0
14.65
e
By: Habtamu Legese Feyisa
2
i
Yˆi = 3.6 + 0.75 X i
 e = 14.65
2
i
Yˆ = 9.6
e = 0
i
e X = 0
i
i
51
Example
By: Habtamu Legese Feyisa
52
Cont.
By: Habtamu Legese Feyisa
53
By: Habtamu Legese Feyisa
54
By: Habtamu Legese Feyisa
55
Estimation of a function with zero intercept
By: Habtamu Legese Feyisa
56
Cont.
By: Habtamu Legese Feyisa
57
Statistical Properties of Least Square Estimators
• PROPERTIES OF OLS ESTIMATORS
• The ideal or optimum properties that the OLS estimates
possess may be summarized by well known theorem known as
the Gauss-Markov Theorem.
• Statement of the theorem: “Given the assumptions of the
classical linear regression model, the OLS estimators, in the
class of linear and unbiased estimators, have the minimum
variance, i.e. the OLS estimators are BLUE.
By: Habtamu Legese Feyisa
58
Cont.
• According to the this theorem, under the basic assumptions of
the classical linear regression model, the least squares
estimators are linear, unbiased and have minimum variance
(i.e. are best of all linear unbiased estimators).
• Some times the theorem referred as the BLUE theorem i.e.
Best, Linear, Unbiased Estimator. An estimator is called
BLUE if:
a. Linear: a linear function of the a random variable, such as,
the dependent variable Y.
b. Unbiased: its average or expected value is equal to the
true population parameter.
By: Habtamu Legese Feyisa
59
Cont.
c. Minimum variance: It has a minimum variance in
the class of linear and unbiased estimators. An
unbiased estimator with the least variance is known
as an efficient estimator.
• According to the Gauss-Markov theorem, the OLS
estimators possess all the BLUE properties.
• Lets proof these properties one by one.
By: Habtamu Legese Feyisa
60
By: Habtamu Legese Feyisa
61
By: Habtamu Legese Feyisa
62
Cont.
By: Habtamu Legese Feyisa
63
By: Habtamu Legese Feyisa
64
By: Habtamu Legese Feyisa
65
Cont.
By: Habtamu Legese Feyisa
66
By: Habtamu Legese Feyisa
67
By: Habtamu Legese Feyisa
68
By: Habtamu Legese Feyisa
69
Cont.
By: Habtamu Legese Feyisa
70
Cont.
By: Habtamu Legese Feyisa
71
2.2.2.4. Statistical test of Significance of the OLS Estimators
(First Order tests)
• After the estimation of the parameters and the determination
of the least square regression line, we need to know how
‘good’ is the fit of this line to the sample observation of Y
and X.
• We need to measure the dispersion of observations around
the regression line.
• This knowledge is essential because the closer the
observation to the line, the better the goodness of fit, i.e. the
better is the explanation of the variations of Y by the
changes in the explanatory variables.
By: Habtamu Legese Feyisa
72
Cont.
1. The coefficient of determination (the square of
𝟐
the correlation coefficient i.e. 𝑹 ). This test is
used for judging the explanatory power of the
independent variable(s).
2. The standard error tests of the estimators.
This test is used for judging the statistical
reliability of the estimates of the regression
coefficients.
By: Habtamu Legese Feyisa
73
TESTS OF THE ‘GOODNESS OF FIT’ WITH
By: Habtamu Legese Feyisa
74
By: Habtamu Legese Feyisa
75
By: Habtamu Legese Feyisa
76
By: Habtamu Legese Feyisa
77
By: Habtamu Legese Feyisa
78
By: Habtamu Legese Feyisa
79
By: Habtamu Legese Feyisa
80
By: Habtamu Legese Feyisa
81
By: Habtamu Legese Feyisa
82
2. TESTING THE SIGNIFICANCE OF OLS PARAMETERS
• To test the significance of the OLS parameter estimators we
need the following:
By: Habtamu Legese Feyisa
83
By: Habtamu Legese Feyisa
84
By: Habtamu Legese Feyisa
85
By: Habtamu Legese Feyisa
86
i) Standard error test
By: Habtamu Legese Feyisa
87
Cont.
By: Habtamu Legese Feyisa
88
Cont.
By: Habtamu Legese Feyisa
89
By: Habtamu Legese Feyisa
90
By: Habtamu Legese Feyisa
91
Cont.
By: Habtamu Legese Feyisa
92
ii) Student’s t-test
• Like the standard error test, this test is also important to test
the significance of the parameters. From your statistics, any
variable X can be transformed into t using the general
formula:
By: Habtamu Legese Feyisa
93
Cont.
By: Habtamu Legese Feyisa
94
Cont.
By: Habtamu Legese Feyisa
95
Cont.
By: Habtamu Legese Feyisa
96
Cont.
By: Habtamu Legese Feyisa
97
Cont.
By: Habtamu Legese Feyisa
98
By: Habtamu Legese Feyisa
99
Cont.
By: Habtamu Legese Feyisa
100
Cont.
By: Habtamu Legese Feyisa
101
Cont.
By: Habtamu Legese Feyisa
102
Cont.
By: Habtamu Legese Feyisa
103
By: Habtamu Legese Feyisa
104
Cont.
By: Habtamu Legese Feyisa
105
Cont.
By: Habtamu Legese Feyisa
106
Cont.
By: Habtamu Legese Feyisa
107
Cont.
By: Habtamu Legese Feyisa
108
Cont.
By: Habtamu Legese Feyisa
109
Cont.
By: Habtamu Legese Feyisa
110
2.2.3 Reporting the Results of Regression Analysis
By: Habtamu Legese Feyisa
111
Thank You
By: Habtamu Legese Feyisa
112