1.6 Estimation of Regression Function
Method of Least Squares (LS)
When the distribution of errors is not specified, we need to minimize the observed errors Yi −β0 −β1Xi. The least square principle
provides the ”best” fitting line to the data
by minimizing
Q(β0, β1) =
n
∑
(Yi − β0 − β1Xi)2
(1)
i=1
Note: Other criteria may also be proposed, such
as considering the least absolute deviation
∑n
(LAD), i=1 |(Yi − β0 − β1Xi)| but LS offers an enormous theoretical simplification
and has some required good properties of
resulting estimators.
1
Least Square Estimators
The analytical solution for β0 and β1, denoted by b0 and b1, respectively, is obtained
by solving the following simultaneous linear
equations known as the normal equations):
∑
∑
Yi = nb0 + b1
XiYi = b0
∑
∑
Xi
Xi + b1
∑
(2)
Xi2
(3)
These can be explicitly solved to give,
∑
b1 =
(Xi − X̄)(Yi − Ȳ )
∑
(Xi − X̄)2
(4)
b0 = Ȳ − b1X̄
(5)
2
Proof The minimizing equations are
∂Q
= 0
∂β0
∂Q
= 0
∂β1
(6)
(7)
It is easy to obtain
∑
∂Q
= −2 (Yi − β0 − β1Xi)
(8)
∂β0
∑
∂Q
= −2
Xi(Yi − β0 − β1Xi) (9)
∂β1
Equating these to zero and substituting b0
and b1 respectively for β0 and β1, we get
∑
∑
(Yi − β0 − β1Xi) = 0
(10)
Xi(Yi − β0 − β1Xi) = 0
(11)
Expanding the summation over individual
terms we get
∑
∑
Yi − nβ0 − β1
XiYi − β0
∑
Xi − β1
∑
∑
Xi = 0(12)
Xi2 = 0(13)
3
Rearranging the terms gives the normal equations. From the first normal equation, we
get
1∑
1∑
Yi = b0 + b1
Xi
(14)
n
n
or
Ȳ = b0 + b1X̄
(15)
Hence,
b0 = Ȳ − b1X̄
Substituting this in the second normal equation, we get,
∑
XiYi = nX̄(Ȳ − b1X̄) + b1
= nX̄ Ȳ + b1(
∑
∑
Xi2
Xi2 − nX̄ 2)
4
This gives
∑
XiYi − nX̄ Ȳ
b1 = ∑ 2
Xi − nX̄ 2
Using the fact that
∑
XiYi − nX̄ Ȳ =
∑
(Xi − X̄)(Yi − Ȳ )
the above expression becomes
∑
b1 =
(Xi − X̄)(Yi − Ȳ )
SXY
=
∑
(Xi − X̄)2
SXX
where
SXY =
∑
(Xi − X̄)(Yi − Ȳ )
and
SXX =
∑
(Xi − X̄)2
5
Example For the data in Table 1.1, the
following computations are obtained; n =
∑
∑
23;
Xi = 3931.6,
Yi = 1555.3
∑ 2
∑
XiYi = 267951.4;
Xi = 673551.7
Hence
X̄ = 3931.6/23 = 170.94,
Ȳ = 1555.3/23 = 67.6231
For computing b1, the numerator is computed as
∑
XiYi − nX̄ Ȳ =
∑
XiYi −
(
∑
Xi)(
n
∑
Yi)
and denominator as
∑
Xi2 − nX̄ 2 =
∑
Xi2 −
(
∑
X i )2
n
Hence,
b1 =
267951.4 − 3931.6 ∗ 1555.3/23
= 1.40694
2
673551.7 − 3931.6 /23
b0 = 67.6231−1.40694∗170.94 = −172.8792
6
Point Estimation of Mean Response
• Let Xh be a typical value of the independent variable at which the mean response
E(Y ) has to be estimated.
Note that this is equivalent to estimating
the regression function
E(Y ) = β0 + β1X,
(16)
for X = Xh.
• Note that individual value of Y is known as
response and E(Y ) is known as the mean
response.
• The regression function is linear in parameters β0 and β1, hence, its estimate is easily
obtained as
Ŷ
= β̂0 + β̂1X
= b0 + b1X
(17)
7
• For the cases in the study, we call Ŷi :
Ŷi = b0 + b1Xi, i = 1, 2, ..., n;
(18)
the fitted value for the ith case; viewed
as the estimate if the mean response for
X = Xi.
Example 1.2 For the data in Table 1.1, the
estimators of b0 and b1 were obtained as
b0 = −172.88,
b1 = 1.41,
Hence, the estimated regression function is given
by
Ŷ = −172.88 + 1.41X.
This estimated regression function is plotted
in Figure 1.2. The fitted values are reported
in the following table.
8
Table 1.2: Fitted Values and Residuals for
Height-Weight(2001) Data
Student#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Height(X)
Weight(Y)
Fits
183.00
77.0975
84.5908
177.80
90.7029
77.2747
172.72
81.6327
70.1274
167.64
49.8866
62.9802
162.56
45.3515
55.8329
162.56
54.4218
55.8329
172.72
72.5624
70.1274
177.80
74.8299
77.2747
172.72
54.4218
70.7214
185.42
92.9705
87.9956
180.34
81.6327
80.8483
172.72
80.7256
70.1274
180.34
70.2948
80.8483
158.00
55.0000
49.4173
163.00
50.0000
56.4520
158.00
42.0000
49.4173
182.00
72.0000
83.1838
166.00
60.0000
60.6728
168.00
62.0000
63.4867
163.00
55.0000
56.4520
170.00
65.0000
66.3006
172.72
95.2381
70.1274
162.56
72.5624
55.8329
Residuals
-7.4933
13.4282
11.5052
-13.0936
-10.4815
-1.4112
2.4349
-2.4448
-15.7057
4.9749
0.7843
10.5982
-10.5535
5.5827
-6.4520
-7.4173
-11.1838
-0.6728
-1.4867
-1.4520
-1.3006
25.1107
16.7294
9
Regression Plot
Y = -172.879 + 1.40694X
R-Sq = 55.7 %
100
90
Weight(Y)
80
70
60
50
40
160
170
180
Height(X)
Figure 1.2 Scatter Plot and Fitted
Line Plot of Height-Weight Data
of STAT360 2001 Class
10
• The graph shows a good scatter around
the fitted line. Suppose that the mean
weight of a person of typical height X =
171cms is desired; the corresponding point
estimate is given by
Ŷ = −172.88 + 1.41(171) = 68.23Kg.
• Table 1.2 also gives the fitted values for all
the heights in the data; just by substituting
Xi for X in the equation of the fitted line.
• This table also gives the value of the residuals, which are the differences between the
observed values and fitted values.
• In general, the ith residual is given by
ei = Yi − Ŷi
(19)
11
• For the SLR model it can be written as
ei = Yi − b0 − b1Xi = (Yi − Ȳ ) − b1(Xi − X̄)
(20)
• The latter equation is useful for theoretical
derivations in the course.
• The residuals are in some sense estimates
of the errors ϵi. They are used to justify the
validity of the model as well as in finding
departures from the model.
12
Properties of Fitted Regression Line
(i) Sum of all the residuals equal zero, i.e
n
∑
ei = 0
(21)
i=1
Note that this implies that the sample mean
1 ∑n
of the residual values ē = n
i=1 ei = 0.The
sample mean being an estimator of the
population mean, this is inline with the assumption that E(ϵ) = 0.
To prove this use Eq (25) and the fact that
∑n
∑n
i=1 (Xi − X̄) = 0 = i=1 (Yi − Ȳ ) = 0.
∑
(ii) Sum of the squared residuals (Yi − Ŷi)2 is
minimum, for the least squared residuals,
ei = (Yi − Ŷi).
Note that this was the requirement in the
least square estimation.
13
(iii) Sum of the observed values Yi equals the
sum of fitted values Ŷi :
n
∑
Yi =
i=1
n
∑
Ŷi
(22)
i=1
This follows from the first property as
∑
ei =
∑
Yi −
∑
Ŷi = 0.
This implies that the (sample) mean of the
¯
observed values Ȳ and the fitted values Ŷ
are the same.
(iv) Sum of the weighted residuals is zero, when
the residuals are weighted by the corresponding level of the predictor variable:
∑
Xiei = 0
(23)
14
To prove this we see that
∑
Xiei =
=
∑
∑
Xi{(Yi − Ȳ ) − b1(Xi − X̄)}
Xi(Yi − Ȳ )
−b1
∑
Xi(Xi − X̄)
(24)
Furthermore,
Sxy =
=
=
=
∑
∑
∑
∑
(Xi − X̄)(Yi − Ȳ )
Xi(Yi − Ȳ ) −
∑
Xi(Yi − Ȳ ) − X̄
X̄(Yi − Ȳ )
∑
(Yi − Ȳ )
Xi(Yi − Ȳ )
as
∑
(Yi − Ȳ ) = 0
Hence Eq (28) becomes
∑
Xiei = Sxy − b1Sxx
15
Using the formula for b1 :
Sxy
,
b1 =
Sxx
the above equation becomes
∑
Xiei = Sxy −
Sxy
Sxx = Sxy − Sxy = 0
Sxx
(v) Sum of the weighted residuals is zero, when
the residuals are weighted by the corresponding level of the fitted values:
∑
Ŷiei = 0
(25)
This easily follows as
∑
Ŷiei = b0
∑
ei + b1
∑
Xiei
and the facts (proved earlier) that
∑
ei = 0
∑
Xiei = 0
16
(vi) The (fitted) regression line always passes
through the point (X̄, Ȳ ).
Substituting X = X̄, we find that
Ŷ = Ȳ − b1X̄ + b1X̄ = Ȳ
which proves this property.
Notes:
(i) Property (i) follows from the first normal equation as
∑
ei =
=
∑
∑
(Yi − b0 − b1Xi)
Yi − nb0 − b1
∑
Xi
(ii) The property (v) follows from the 2nd
normal equation as:
∑
Xiei =
∑
XiYi − b0
∑
Xi − b1
∑
17
Xi2
(iii) If the data is transformed as Y → y = Y − Ȳ
and X → x = X − X̄, the fitted equation
becomes
ŷ = b1x
(26)
where ŷ = Ŷ − Ȳ . It is clear to see that this
equation passes through the point (0,0);
which is a consequence of shifting the origin to the point (X̄, Ȳ ).
1.7 Estimation of Error Variance
• In general, variation can be estimated by
squared deviations of observations from the
mean or estimate of the mean.
For example for the observations Y1, Y2, ..., Yn
from a normal population N (µ, σ 2), the unbiased estimator of σ 2 is given by
18
1
σ̂ 2 =
n
∑
(Yi − µ)2,
if µ
is known.
n
1 ∑
(Yi − Ȳ )2,
n − 1 i=1
if µ
is unknown.
n i=1
s2 =
• In other words, we say that the estimate
of σ 2 is sum of squared deviations divided
by degrees of freedom; n if µ is known and
n − 1 if µ is estimated by Ȳ .
• In the case of regression model the approximation to deviations of observations
Yi from its mean m(Xi) = β0 +β1Xi is given
by Yi − m(Xi) is given by
ei = Yi − Ŷi = Yi − b0 − b1Xi
and the corresponding sum of squares, denoted by SSE, for Sum of Squares due to
Error. is given by
SSE =
n
∑
(Yi − Ŷi)2 =
i=1
n
∑
e2
i
(27)
i=1
19
The corresponding degrees of freedom is n − 2
(2 degrees of freedom are lost for estimating
two parameters, β0 and β1.) This gives rise to
the following estimate of σ 2,
∑n
e2
SSE
i=1
i
=
M SE =
(28)
n−2
n−2
where M SE stands for Mean Square due to
Error.
• It will be proved later that M SE is unbiased
for σ 2.
Example The estimate of the error variance for the data of Table 1.1 is obtained
as follows:
Note that Sum of Squared Errors (Residuals) is given by
SSE = 2329.5.
based on n = 23 observations.
MSE is given by
Hence,
M SE = 2329.5/21 = 110.93.
20
1.8 Normal Error Regression Models
• The information about the parameters is
given by the distribution of errors Y1, ..., Yn.
• For the normal error regression model, ϵ1, ...ϵn
are independent and normally distributed
with zero mean and variance= σ 2. This implies that Y1, ..., Yn are also normal and independent where
Yi ∼ N (β0 + β1Xi, σ 2).
The probability density function for Yi is
given by
1
1
2}
√ exp{−
(Y
−
m(X
))
i
i
2σ 2
σ 2π
1
1
√
=
exp{− 2 (Yi − β0 − β1Xi)2}
2σ
σ 2π
fi(Yi) =
21
Since , Y1, ..., Yn are independent, the joint
probability density function of Y1, ..., Yn is
given by
f (Y1, ..., Yn) = f1(Y1)f2(Y2)...fn(Yn)
and the likelihood function L(β0, β1, σ 2) is
given by
L(β0 , β1 , σ 2 ) = f (Y1 , ..., Yn)
{
}n ∏
n
1
1
√
=
exp{− 2 (Yi − β0 − β1 Xi))2 }
2σ
σ 2π
i=1
{
}n
n
1
1 ∑
√
=
exp{− 2
(Yi − β0 − β1 Xi )2 } (29)
2σ i=1
σ 2π
• For finding the maximum likelihood estimators of β0, β1, σ 2, the likelihood function
has to be maximized. Equivalently, we consider to maximize the log-likelihood function given by
n
∑
n
n
1
loge L = − loge (2π)− loge σ 2 − 2
(Yi−β0 −β1 Xi)2
2
2
2σ i=1
(30)
22
Maximum Likelihood Estimators of Parameters
The maximum likelihood estimators are obtained by solving the following three equations
∂ loge L
= 0
∂β0
∂ loge L
= 0
∂β1
∂ loge L
= 0
∂σ 2
These partial derivatives are given by
n
∂ loge L
1 ∑
=
(Yi − β0 − β1Xi)
2
∂β0
σ i=1
n
∂ loge L
1 ∑
Xi(Yi − β0 − β1Xi)
=
2
∂β1
σ i=1
n
∂ loge L
n
1 ∑
2
=
−
+
(Y
−
β
−
β
X
)
0
1
i
i
∂σ 2
2σ 2
2σ 4 i=1
23
• Replacing β0, β1, σ 2 by β̂0, β̂1, σ̂ 2, after a little simplification, we obtain
n
∑
(Yi − β̂0 − β̂1Xi) = 0,
i=1
n
∑
Xi(Yi − β̂0 − β̂1Xi) = 0,
i=1
∑n
2
(Y
−
β̂
−
β̂
X
)
0
1
i
i
i=1
= σ̂ 2.
n
(31)
(32)
(33)
• Note that the equations (36) and (37) are
the two normal equations obtained by the
least square method. Hence the Maximum
Likelihood estimators of β0 and β1 are the
same as b0 and b1 respectively. Whereas
the MLE for σ 2 is given by
∑n
σ̂
2
=
=
i=1 (Yi − b0 − b1 Xi )
∑n
2
n
2
i=1 ei
(34)
(35)
n
• Note that the MLE for σ 2 is biased, as
(
E(σ̂ 2 ) = E
n−2
M SE
n
)
=
n−2 2
σ
n
(36)
24