Linear Regression
Module - 12
Kiran Sultana
© Islamic Online University
Least Square Method
The regression equation of X on Y is :
𝑋෡𝑖 = 𝑎 + 𝑏𝑌𝑖
The regression equation of Y on X is:
𝑌෡𝑖 = 𝑎 + 𝑏𝑋𝑖
The values of a and b are found with the help of normal equations given below:
(I )
Σ𝑋 = 𝑛𝑎 + 𝑏Σ𝑌
Σ𝑋𝑌 = 𝑎Σ𝑌 + 𝑏Σ𝑌 2
(II )
Σ𝑌 = 𝑛𝑎 + 𝑏Σ𝑋
Σ𝑋𝑌 = 𝑎Σ𝑋 + 𝑏Σ𝑋 2
© Islamic Online University
Example
From the following data obtain the two regression equations
the method of Least Squares.
X
X
Y
3
2
3
Y
6
6
1
7
4
8
8
5
9
 X  24
Y  29
2
1
XY
18
2
56
20
72
7
8
X2 4
5
9
4
Y28
9
36
1
49
16
64
64
25
81
using
 XY 168  X  142 Y  207
2
2
© Islamic Online University
Regression Equation of Y on X
𝑌෡𝑖 = 𝑎 + 𝑏𝑋𝑖
Σ𝑌 = 𝑛𝑎 + 𝑏Σ𝑋
Σ𝑋𝑌 = 𝑎Σ𝑋 + 𝑏Σ𝑋 2
𝑛=5
 X  24
Y  29
 XY  168
 X  142
Y  207
2
2
Plotting the values in given equation
29=5a+24b………………………(i)
168=24a+142b
84=12a+71b……………………..(ii)
Multiplying equation (i ) by 12 and (ii) by 5
348=60a+288b………………………………..(iii)
420=60a+355b………………………………..(iv)
By solving equation(iii)and (iv) we get
a=0.66 and b=1.07
𝑌෠ = 0.66 + 1.07𝑋
© Islamic Online University
Regression Equation of X on Y
𝑋෡𝑖 = 𝑎 + 𝑏𝑌𝑖
Σ𝑋 = 𝑛𝑎 + 𝑏Σ𝑌
Σ𝑋𝑌 = 𝑎Σ𝑌 + 𝑏Σ𝑌 2
𝑛=5
 X  24
Y  29
 XY 168
 X  142
Y  207
2
2
24=5a+29b………………………(i)
Plotting the values in given equation 168=29a+207b…………………..(ii)
Multiplying equation (i)by 29 and in (ii) by 5 we get
696=145a+841b…………………(iii)
840=145a+ 1035b………………..(iv)
By solving equation(iii)and (iv) we get
a=0.49 and b=0.74
𝑋෠ = 0.49 + 0.74𝑌
© Islamic Online University
Simple Linear Regression Equation
The simple linear regression equation provides an estimate of the population regression line
Estimate of the
regression slope
Estimate of regression
intercept
Estimated (or predicted) y
value for observation i
෡𝑖 = 𝑎 + 𝑏𝑋𝑖
𝑌
Value of x for
observation i
The individual random error terms ei have a mean of zero
Σ𝑒 = 0
© Islamic Online University
Assumption Required for a Linear Regression Analysis
1. The distribution of ε at any particular x value has mean value 0.
𝐸(𝜀𝑖 ) = 0
2. The Variance of ε is same for any particular value of x.
𝑣𝑎𝑟 𝜀𝑖 = 𝐸(𝜀𝑖2 ) = 𝜎 2
3. The random errors are independent of one another.
𝐸 𝜀𝑖 , 𝜀𝑗 = 0
4. The distribution of ε at any particular x value is normal.
𝜀𝑖 ~𝑁(0 , 𝜎 2 )
© Islamic Online University
Probability Distribution of 𝜀
For any fixed x, y has normal distribution with mean a + bx and standard
deviation s.
Positive Error
𝐸 𝑌 = 𝛼 + 𝛽𝑋
Negative Error
© Islamic Online University
Error or Ceterus Paribas
Law of Demand:
“The Quantity Demanded increases(decreases) with the
decrease(increase) in price, when other things remaining the same or
Ceterus Paribus”
Mathematically,
Where,
𝑄𝑑 = 𝑓 𝑃 + 𝑐𝑒𝑡𝑒𝑟𝑢𝑠 𝑝𝑒𝑟𝑖𝑏𝑢𝑠
𝑄𝑑 = 𝛼 − 𝛽𝑃 + 𝑐𝑒𝑡𝑒𝑟𝑢𝑠 𝑝𝑒𝑟𝑖𝑏𝑢𝑠
𝛼 = 𝐴𝑢𝑡𝑜𝑛𝑜𝑚𝑜𝑢𝑠 𝐷𝑒𝑚𝑎𝑛𝑑
𝛽 = 𝑆𝑙𝑜𝑝𝑒 𝑜𝑓 𝐷𝑒𝑚𝑎𝑛𝑑 𝐶𝑢𝑟𝑣𝑒 𝑜𝑟 𝑀𝑃𝐶
𝑄𝑑 = 𝛼 − 𝛽𝑃 + 𝜀
It means error term 𝜀 plays the same role in Statistics, as Ceterus paribus in Economics
© Islamic Online University
Regression Equation of Y on X(Alternative Approach)
The calculation by the least squares method are quit cumbersome when the values of X and Y
are large. So the work can be simplified by using this method.
The formula for the calculation of Regression Equations by this method:
𝑌෡𝑖 = 𝑎 + 𝑏𝑋𝑖
ത
ത
Σ(𝑋 − 𝑋)(𝑌
− 𝑌)
𝑏𝑦𝑥 =
Σ 𝑋 − 𝑋ത 2
Or
𝑛Σ𝑋𝑌 − Σ𝑋Σ𝑌
𝑏𝑦𝑥 =
𝑛Σ𝑋 2 − Σ𝑋 2
𝑎𝑦𝑥 = 𝑌ത − 𝑏𝑦𝑥 𝑋ത
© Islamic Online University
Regression Equation of X on Y(Alternative Approach)
When X is regressed on Y, we have the following equation,
𝑋෡𝑖 = 𝑎 + 𝑏𝑌𝑖
Slope
ത
ത
Σ(𝑋 − 𝑋)(𝑌
− 𝑌)
𝑏𝑥𝑦 =
Σ 𝑌 − 𝑌ത 2
Or
𝑛Σ𝑋𝑌 − Σ𝑋Σ𝑌
𝑏𝑥𝑦 =
𝑛Σ𝑌 2 − Σ𝑌 2
Intercept
𝑎𝑥𝑦 = 𝑋ത − 𝑏𝑥𝑦 𝑌ത
© Islamic Online University
Example
From the following data obtain the two regression equations
the method of Least Squares.
X
Y
3
6
2
1
7
8
4
5
8
9
using
For Solution, move to MS Excel
© Islamic Online University
Properties of Least Square
✓ The least square regression line passes through the points of means (𝑋ത , ഥ𝑌), the center of the
observed data.
✓ The sum of residuals is equal to zero Σ𝑒𝑖 = 0
෡𝑖 are equal Σ𝑌𝑖 = Σ𝑌෡𝑖
✓ The Sum of Observed Values 𝑌𝑖 and sum of the fitted values 𝑌
෡𝑖 are equal Σ𝑌ത𝑖 = Σ𝑌ത𝑖 .
✓ The Mean of Observed Values 𝑌𝑖 and Mean of the fitted values 𝑌
✓ Sum of square of residuals is minimum Σ𝑒𝑖2 𝑖𝑠 𝑚𝑖𝑛𝑖𝑚𝑢𝑚
Move to MS Excel
© Islamic Online University
Standard Error of Estimate
“The degree of scatter of observed values about the regression lines is measured
by Standard deviation of regression or Standard error of estimate”
For population data Standard Error of observations about the true regression line 𝐸 𝑌 = 𝛼 + β𝑋
𝑑𝑒𝑛𝑜𝑡𝑒𝑑 𝑏𝑦 𝜎𝑦.𝑥 is given by
𝜎𝑦.𝑥 =
2
Σ 𝑌 − 𝑌෠
𝑁
𝑆𝑦.𝑥 =
2
Σ 𝑌 − 𝑌෠
𝑛−2
For Sample Data:
𝑆𝑦.𝑥 =
or
Σ𝑌𝑖2 − 𝑎Σ𝑌𝑖 − 𝑏Σ𝑋𝑖 𝑌𝑖
𝑛−2
Move to MS Excel
© Islamic Online University
Total Variation, Explained Variation and Unexplained Variation
Total Variation or T.S.S:
“The variability among the values of dependent variable Y coming from the
explained variables and all those which are in Error Term”
𝑇. 𝑆. 𝑆 = Σ 𝑌 − 𝑌ത 2
Explained Variation or R.S.S
Unexplained Variation or E.S.S
The
variation
coming
from
the
explanatory variables e.g., Change in
demand due to change in price
2
෡
ത
𝑅. 𝑆. 𝑆 = Σ 𝑌𝑖 − 𝑌
It is sum of squares of the errors of
′
predicted 𝑌෡𝑖 𝑠 from the sample mean 𝑌ത
The variation coming from the variables
which are unexplained e.g., Change in
demand due to change in Income etc.
2
෠
𝐸. 𝑆. 𝑆 = Σ 𝑌𝑖 − 𝑌
It is sum of squares of the errors of
prediction
© Islamic Online University
Total Variation, Explained Variation and Unexplained Variation
𝑇. 𝑆. 𝑆 = 𝑅. 𝑆. 𝑆 + 𝐸. 𝑆. 𝑆
𝐸. 𝑆. 𝑆
2
Σ 𝑌𝑖 − 𝑌෠
෡𝑖 − 𝑌ത
𝑅. 𝑆. 𝑆 Σ 𝑌
2
© Islamic Online University
Coefficient of Determination
“The Ratio of Explained variation to the
total variation is called the Coefficient of
Determination and Denoted 𝑟 2 ”
2
෡
ത
𝐸𝑥𝑝𝑙𝑎𝑖𝑛𝑒𝑑 𝑉𝑎𝑟𝑖𝑎𝑡𝑖𝑜𝑛 Σ 𝑌𝑖 − 𝑌
𝑟2 =
=
𝑇𝑜𝑡𝑎𝑙 𝑉𝑎𝑟𝑖𝑎𝑡𝑖𝑜𝑛
Σ 𝑌𝑖 − 𝑌ത 2
2
෠
Σ
𝑌
−
𝑌
𝐸.
𝑆.
𝑆
𝑖
𝑟2 = 1 −
=
𝑇. 𝑆. 𝑆 Σ 𝑌𝑖 − 𝑌ത 2
© Islamic Online University
Coefficient of Determination
Simplifying the ratio,
Σ𝑌 2
𝑎Σ𝑌 + 𝑏Σ𝑋𝑌 −
𝑛
2
𝑟 =
2
Σ𝑌
Σ𝑌 2 −
𝑛
•
r2 is the square of correlation coefficient
•
r2 is a number between zero and one and a value close to zero suggests a poor model.
0 ≤ 𝑟2 ≤ 1
•
It gives the proportion of variation in y that can be attributed to an approximate
linear relationship between x and y.
Move to MS Excel
© Islamic Online University
Coefficient of Non-Determination
𝑌෡𝑖 = 𝑎 + 𝑏𝑋𝑖
Logically, it is the total variation in Y that is not explained by the variation in X.
Mathematically,
1 − 𝑟2
For Example:
When 𝑟 2 = 0.75
then
1 − 𝑟 2 = 0.25
Means 25% variation is not accounted by the variation in independent variable
© Islamic Online University
Prediction in Regression Analysis
“Regression line is use to predict the response “Y” for
specific value of the explanatory variable “X””
Accuracy of prediction:
Accuracy of prediction from a regression line depends on how much scatter about line the
data has.
• If the data points are very to the line then we are confident then that our prediction is
accurate and vice versa.
© Islamic Online University
Interpolation and Extrapoltaion
Extrapolation and interpolation are both used to estimate hypothetical values for a variable based upon
other observations
To tell the difference between extrapolation and interpolation, we need to look at the prefixes “extra” and
“inter.”
• The prefix “extra” means “outside” or “in addition to.”
• The prefix “inter” means “in between” or “among.”
Just knowing these meanings (from their originals in Latin) goes a long way to distinguishing between the
two methods.
© Islamic Online University
Interpolation and Extrapolation
. Interpolation is guessing data points that fall within
the range of the data you have, i.e. between your
existing
data
points.
. Extrapolation is guessing data points from beyond
the
range
of
your
data
set.
For Example:
For Example:
•
•
•
Suppose as before that data with x between 0 and
10 is used to produce a regression line y = 2x + 5.
We can use this line of best fit to estimate the y
value corresponding to x = 20, y = 2(20) + 5 =45.
•
Because our x value is not among the range of
values used to make the line of best fit, this is an
example of extrapolation.
Suppose that data with x between 0 and 10 is
used to produce a regression line y = 2x + 5. We
can use this line of best fit to estimate the y value
corresponding to x = 6. y = 2(6) + 5 =17.
Because our x value is among the range of values
used to make the line of best fit, this is an
example of interpolation.
© Islamic Online University
Advantages of Regression Analysis
• Predicting the Future
• Supporting Decision
• Correcting Errors
• New Insights
© Islamic Online University