Brief Notes #11
Linear Regression
(a) Simple Linear Regression
•
Model: Y = β0 + β1 g(X) + ε, with ε ~ (0,σ2)
⇒ Y = β0 + β1 X + ε
•
Data: (Xi, Yi), with i = 1, … , n
Yi = β0 + β1 Xi + εi
y
yi
( y i − ŷ i )
ŷ i
( ŷ i − y)
( y i − y)
y
Ŷ = βˆ 0 + βˆ 1 X
x
•
xi
Least Squares Estimation of (β0, β1)
Find ( βˆ 0 , βˆ 1 ) such that
∑ (Y
i
i
− Ŷi ) 2 = min , where Ŷi = βˆ 0 + βˆ 1 X i .
x
•
Solution
S
βˆ 1 = XY ,
S XX
S
βˆ 0 = Y − βˆ 1 X = Y − XY X ,
S XX
1
∑ Xi ,
n i
where X =
Y=
1
∑ Yi
n i
S XX = ∑ (X i − X ) 2 ,
i
•
i
⎡βˆ ⎤
Properties of ⎢ 0 ⎥ for εi ~ iid N(0, σ2)
ˆ
⎣ β1 ⎦
⎛
⎡⎛ 1 X 2 ⎞
⎜
⎟⎟
⎢⎜⎜ +
⎡βˆ 0 ⎤
⎜ ⎡β 0 ⎤
n
S
2 ⎝
XX ⎠
⎢ ˆ ⎥ ~ N⎜ ⎢ ⎥ , σ ⎢⎢
⎛
⎞
β
X
β
⎣ 1⎦
⎜⎣ 1⎦
⎜⎜ −
⎟⎟
⎢
⎜
S
XX
⎝
⎠
⎣
⎝
•
S XY = ∑ (X i − X )(Yi − Y ) .
⎛ X ⎞⎤ ⎞⎟
⎜⎜ −
⎟⎟⎥
S
XX ⎠ ⎥ ⎟
⎝
⎛ 1 ⎞ ⎥ ⎟⎟
⎜⎜
⎟⎟ ⎥
⎝ S XX ⎠ ⎦ ⎟⎠
Properties of Residuals, e i = Yi − Ŷi
-
∑e = ∑e X = ∑e Y
i
i
i
i
- SSe =
∑ (Y
i
i
i
i
=0
i
− Ŷi ) 2 = the residual sum of squares.
i
SS e
~ χ 2n − 2 ,
2
σ
⇒
E[SSe] = σ2(n − 2)
σˆ 2 = SS e /(n − 2) = MS e (mean square error)
σˆ = MS e = "standard error of regression"
•
Significance of Regression
Let S YY = ∑ (Yi − Y ) 2 = total sum of squares.
i
Property: SYY = SSe + SSR,
where SSe =
∑ (Y
i
− Ŷi ) 2 = residual sum of squares,
i
SSR =
∑ (Ŷ
− Y ) 2 = sum of squares explained by the regression.
i
i
Also SSe and SSR are statistically independent.
SS
Notice: if β1 = 0, then 2R ~ χ12
σ
Definition:
R2 =
•
SS
SS R
= 1− e ,
S YY
S YY
coefficient of determination of the regression.
Hypothesis Testing for the Slope β1
1. H0: β1 = β10 against H1: β1 ≠ β10 (t-test)
Property: t (β1 ) =
βˆ 1 − β1
MS e / S XX
~ t n −2
⇒ Accept H0 at confidence level α if:
| t (β10 ) |< t n − 2,α / 2
2. H0: β1 = 0 against H1: β1 ≠ 0 (F-test)
From distributional properties and independence of SSR and SSe, and under H0,
F=
SS R / 1
~ F1,n − 2
SS e /( n − 2)
⇒ Accept H0 if F < F1, n − 2, α
Notice that for H0: β1 = 0, the t-test and the F-test are equivalent.
(b) Multiple Linear Regression
•
k
Model: Y = β 0 + ∑ β j g j (X) + ε , with ε ~ (0,σ2)
j=1
k
⇒ Y = β0 + ∑ β jX j + ε
j=1
•
Data: (Yi, Xi), with i = 1, … , n
k
Y = β 0 + ∑ β j X ij + ε i , with i = 1, … , n
j=1
⎡β 0 ⎤
⎢β ⎥
β = ⎢ 1⎥,
⎢#⎥
⎢ ⎥
⎣β k ⎦
⎡ Y1 ⎤
Let Y = ⎢⎢ # ⎥⎥ ,
⎢⎣Yn ⎥⎦
⎡ ε1 ⎤
ε = ⎢⎢ # ⎥⎥ ,
⎢⎣ε n ⎥⎦
⎡1 X 11
#
H = ⎢⎢#
⎢⎣1 X n1
X 12
#
X n2
" X 1k ⎤
#
# ⎥⎥
" X nk ⎥⎦
⇒Y=Hβ+ε
•
Least Squares Estimation
k
e i = Yi − Ŷi = Yi − βˆ 0 − ∑ βˆ j X ij
j=1
⎡ e1 ⎤
e = ⎢⎢ # ⎥⎥ = Y − Hβˆ
⎢⎣e n ⎥⎦
T
T
T
T
T
SS e = ∑ e i2 = ∑ e e = (Y − Hβˆ ) T (Y − Hβˆ ) = Y Y − 2Y Hβˆ + βˆ H Hβˆ
i
dSS e (β)
dβ
i
=0
⇒
T
T
βˆ = (H H) −1 H Y
•
Properties of β̂ (if εi ~ iid N(0, σ2))
T
βˆ ~ N(β, σ 2 (H H) −1 )
•
Properties of Residuals
SS e
~ χ 2n − k −1
2
σ
⇒
σˆ 2 =
SS e
= MS e
n − k −1
SYY = SSe + SSR
R2 = 1−
•
SS e
S YY
Hypothesis Testing
⎡β ⎤
Let β = ⎢ 1 ⎥ , where β1 has τ1 components and β2 has τ2 = k − τ1 components. We
⎣β 2 ⎦
want to test H0: β2 = 0 against H1: β2 ≠ 0 (at least one component of β2 is non-zero).
The procedure is as follows:
- Fit the complete regression model and calculate SSR and SSe;
- Fit the reduced model with β2 = 0, and calculate SS R1 ;
- Let SS2|1 = SSR − SS R1 = extra sum of squares due to β2 when β1 is in the
regression.
- Distributional property of SS2|1. Under H0,
SS 2|1
σ2
~ χ 2τ 2
Also, SS2|1 and SSe are independent. Therefore,
F=
SS 2|1 / τ 2
SS e /( n − k − 1)
~ Fτ2 ,n − k −1
⇒ Accept H0 if F < Fτ 2 ,n − k −1,α