Example
Animal Studies of Side Effects
y = pressure in the pancreas,
Simple Linear Regression
Basic Ideas
x = dose,
In simple linear regression, there is an
approximately linear relation between two
variables, say
x
0
5
10
15
20
25
y
14.6
24.5
21.8
34.5
35.1
43.0
35
40
y = α + βx + ,
y
30
where
20
25
• x and y are observed;
15
• α and β are unknown;
• is a random error with mean 0.
0
5
10
15
20
25
x
Figure 1: A Scatterplot
Note: Here x is a design variable, set by the
experimenter.
Slide 1
Slide 2
Example
From the Coleman Report
Drawing the Line
Least Squares Estimators
The Problem: Given (x1, y1 ), · · · , (xn , yn ), find
a and b to minimize
y = Ave 6th Grade Verbal Score,
x = Mother0 s Education (yrs),
SS(a, b) =
x
12.38
10.34
14.08
14.20
12.3
11.46
y
37.01
26.51
36.51
40.70
37.1
33.40
n
X
i=1
(yi − a − bxi)2 .
The Solution:
sxy
,
sxx
a = ȳ − bx̄,
36
38
40
b=
28
30
32
y
34
where x̄ and ȳ are the sample means of x1 , · · · , xn
and y1 , · · · , yn ,
26
sxy =
11
12
13
14
x
Figure 2: A Scatterplot
n
X
i=1
sxx
(xi − x̄)(yi − ȳ),
n
X
i=1
(xi − x̄)2
Note: Here x is a covariate, measured with y.
Slide 3
Slide 4
Here
The Details
∂
SS(a, b) = −2n(ȳ − a − bx̄).
∂a
So,
a = ȳ − bx̄.
Recall:
SS(a, b) =
n
X
i=1
(yi − a − bxi)2 .
Differentiate:
Then
n
X
∂
SS(a, b) = −2
[yi − ȳ − b(xi − x̄)]xi ,
∂b
i=1
= −2
n
X
∂
SS(a, b) = −2
(yi − a − bxi )
∂a
i=1
n
X
∂
(yi − a − bxi)xi .
SS(a, b) = −2
∂b
i=1
Now
n
X
i=1
Solve:
∂
SS(a, b) = 0,
∂a
∂
SS(a, b) = 0.
∂b
i=1
(yi − ȳ)xi − b
(xi − x̄) = 0 =
n
X
i=1
(xi − x̄)xi
n
X
(yi − ȳ).
i=1
So,
n
X
i=1
(yi − ȳ)xi =
n
X
i=1
(yi − ȳ)(xi − x̄)
= sxy ,
Slide 5
Slide 6
Example
Coleman
a = .1312 and b = 2.8149
say, and
n
X
n
X
i=1
i=1
(xi − x̄)xi =
n
X
(xi − x̄)2
38
40
= sxx ,
34
32
30
26
and
28
∂
SS(a, b) = −2 sxy − bsxx ,
∂b
sxy
,
b=
sxx
Verbal Score
36
say. So,
a = ȳ − bx̄.
The Least Squares Line: y = a + bx.
Slide 7
11
12
13
14
Mother’s Ed
Figure 3: A Scatterplot and Least Squares Line
Slide 8
Calculating a and b
By Machine: Using Excel-for example.
Example
Dose and Response
By Hand: Recall
sxx =
n
X
i=1
(xi − x̄)2 =
n
X
i=1
x2i − nx̄2 .
Similarly,
sxy =
n
X
=
i=1
n
X
(xi − x̄)(yi − ȳ)
i=1
xiyi − nx̄ȳ.
Sums
xy
x2
14.6
0
0
5
24.5
122.5
25
10
21.8
218
100
15
34.5
517.5
225
20
35.1
702
400
x
y
0
25
43.0
1075
625
75
173.5
2635
1375
So, a, and b can be calculated from the sums of
xi, yi , xiyi , and x2i .
Slide 9
Slide 10
The Calculations
A Least Squares Line
Pressure vs. Dose
a = 15.595 and b = 1.066
40
75
x̄ =
= 12.5,
6
173.5
ȳ =
= 28.92,
6
35
sxy = 2635 − 6 × 12.5 × 28.92
25
sxx = 1375 − 6 × (12.5)2
y
30
= 466.25,
15
20
= 437.5,
b=
466.25
= 1.066,
437.5
and
0
5
10
15
20
25
x
Figure 4: A Scatterplot and Least Squares Line
a = 28.92 − 1.066 × 12.5 = 15.595.
Slide 11
Slide 12
Some Terminology
Least Squares Estimators:
sxy
b=
,
sxx
a = ȳ − bx̄.
Error Sum of Squares AKA Residual Sum of
Squares:
n
X
e2i .
SSE =
Fitted Values AKA Predicted Values:
Let
i=1
ŷi = a + bxi = ȳ + b(xi − x̄).
Residuals:
syy =
n
X
i=1
(yi − ȳ)2 .
Then
syy = SSR + SSE.
ei = yi − yˆi
Let
= yi − ȳ − b(xi − x̄)
R2 =
Then
Regression Sum of Squares:
SSR =
n
X
i=1
SSR
.
syy
100R2 = % ExplainedVariation
(ŷi − ȳ)2
Note: SSE = syy − SSR = syy − b2sxx .
= b2 sxx
Slide 13
Slide 14
Derivation of syy = SSR + SSE
syy =
n
X
=
i=1
n
X
Inference
(yi − ŷi + ŷi − ȳ)2
e2i + 2
i=1
n
X
i=1
+
n
X
i=1
ei (ŷi − ȳ)
(ŷi − ȳ)
2
Model: Now suppose
yi = α + βxi + i ,
where
1 , · · · , n ∼ind Normal[0, σ 2].
Notes -a) Here −∞ < α, β < ∞ and σ 2 > 0 are
unknown.
Here
1st = SSE,
3rd = SSR,
and
2nd = 2
n
X
i=1
b) If x1, · · · , xn are covariates, then the
conditions must hold conditionally given
x1 , · · · , x n .
c) Then
[(yi − ȳ) − b(xi − x̄)b(xi − x̄)
= 2[bsxy − b2 sxx ] = 0
Slide 15
yi ∼ Normal[α + βxi, σ 2 ],
are (conditionally) independent.
Slide 16
The Likelihood Function
That is,
sxy
,
sxx
α̂ = a = ȳ − bx̄.
β̂ = b =
The Likelihood Function is
n
Y
1
1
√
exp[− 2 (yi − α − βxi)2 ]
2
2σ
2πσ
i=1
n
1 n
1 X
exp[− 2
= √
(yi − α − βxi )2 ]
2
2σ i=1
2πσ
The Profile Likelihood Function: Then
`(α̂, β̂, σ 2 |x, y) = −
1
− n[log(σ 2) + log(2π)].
2
So, the log-likelihood function is
`(α, β, σ 2 |x, y) = −
n
1 X
(yi − α − βxi)2
2σ2
1
SSE
2σ2
The MLE of σ 2 :
i=1
∂
1
1
`=
SSE − 2 n.
∂σ2
2σ4
2σ
1
− n[log(σ 2 ) + log(2π)].
2
So,
σ̂2 =
Maximum Likelihood Estimators: α̂ and β̂ must
minimize the sum of squares. So,
SSE
.
n
Let
M SE =
MLE = LSE.
Slide 17
SSE
.
n−2
Slide 18
Means and Variances
Of The Estimators
Unbiasedness: α̂ and β̂ are unbiased; that is
E(β̂) = β,
E(α̂) = α.
Variances:
σβ̂2 =
σ2
sxx
1
x̄2 2
σα̂2 =
+
σ
n sxx
Derivation For β̂: First,
sxy =
n
X
=
n
X
i=1
i=1
since
Pn
i=1 (xi
(xi − x̄)(yi − ȳ)
So,
1
E(SxY )
sxx
n
1 X
=
(xi − x̄)E(Yi )
sxx i=1
E(β̂) =
=
=
n
1 X
sxx
1
sxx
i=1
n
X
i=1
(xi − x̄)(α + βxi)
(xi − x̄)β(xi − x̄)
1
sxx β
sxx
= β.
=
(xi − x̄)yi ,
− x̄) = 0.
Slide 19
Slide 20
Sampling Distributions
Similarly,
σβ̂2 =
=
=
1
s2xx
n
X
Var
(xi − x̄)Yi
n
1 X
s2xx
1
s2xx
α̂ ∼ Normal[α, σα̂2 ],
β̂ ∼ Normal[β, σβ̂2 ],
i=1
i=1
(xi − x̄)2σ2
and
βsxx
SSE
∼ χ2n−2 ;
σ2
and (α̂, β̂) is independent if SSE
σ2
.
=
sxx
Corollary: MSE is unbiased; that is
E(MSE ) = σ 2 .
Notes: • Unbiasedness requires (only) E(i ) = 0.
• Variance requires also, E(2i ) = σ 2 .
Note: The proof is similar to the independence of
X̄ and S 2 in the one-sample normal problem.
Note: Unbiasedness of MSE requires only
E(i ) = 0 and E(2i ) = σ 2 .
• Similarly for α̂.
Slide 21
Slide 22
Confidence Intervals
Studentization: Let
σ̂β̂2 =
MSE
sxx
T =
β̂ − β
σ̂β̂
Here
and
Then
T ∼ tn−2
So, if c is the 97.5th percentile of tn−2 , for
example, then
P [−c ≤ T ≤ c] = .95.
−c ≤ T =
β̂ − β
≤c
σ̂β̂
iff
β̂ − cσ̂β̂ ≤ β ≤ β̂ + cσ̂β̂ .
Confidence Interval For β: So,
β̂ ± cσ̂β̂
is a 95% confidence interval for β.
Confidence Interval for α: Similarly,
α̂ ± cσ̂α̂
is a 95% confidence interval for α.
0.1
i
O
Slide 23
Slide 24
Here
Example
United Data Services
Here
n = 14,
x = Units Serviced,
c = 2.180,
y = Time.
β̂ = 15.509,
sxx = 114,
150
MSE = 29.074
100
σ̂β̂2 =
29.074
= (.505)2
114
and
50
Time
So,
2
4
6
8
β̂ ± cσ̂β̂ = 15.509 ± 2.18 × .505
10
Units
= 15.51 ± 1.10
Figure 5: A Scatterplot
Slide 25
Slide 26
Testing H0 : β = 0
Review
From the Confidence Interval: Accept if
β̂ − cσ̂β̂ ≤ 0 ≤ β̂ + cσ̂β̂ .
• Least squares estimators, a and b.
• Properties of the estimators
Equivalently, reject if
|T0 | =
• Simple Linear Regression: Y = α + βx + |β̂|
> c.
σ̂β̂
Example: Dose Response.
β̂ ± cσ̂β̂ = 1.066 ± .449 and, therefore, H0 is
rejected.
Note: This is the GLRT (as in the one sample
problem).
Slide 27
• Sampling distributions
• Confidence intervals
• Testing
Today
• Estimating expected response
• Predicting a future value
Slide 28
Estimating Expected Response
Let
Let
µ̂0 = α̂ + β̂x0.
µ(x) = α + βx = E(Y |x).
Then
E(µ̂0 ) = µ0
Fix an x0 and let
σµ̂2 0 =
µ0 = µ(x0) = α + βx0
Let
Then
σ̂µ̂2 0 =
µ(x) = µ0 + β(x − x0 ).
So,
1
n
+
(x0 − x̄)2 2
σ .
sxx
1
(x0 − x̄)2 MSE .
+
n
sxx
Then
µ0 = α 0 ,
when
x0 = x − x 0 .
µ̂0 ± cσ̂µ̂0
is a level 95% confidence interval for µ0.
Slide 29
Slide 30
Predicting a Future Value
Now let
Y0 ∼ Normal[µ0 , σ̂ 2 ]
Then
∆
∼ tn−2
σ̂∆
and
Ŷ0 = µ̂0 .
and
P [−c ≤
Then
∆ := Y0 − Ŷ0 = Y0 − µ0 − (µ̂0 − µ0 ).
Here
−c ≤
So,
E(∆) = 0
2
σ∆
= σ 2 + σµ̂2 0
1
(x0 − x̄)2 2
= 1+ +
σ
n
sxx
∆
≤ c] = .95.
σ̂∆
∆
≤c
σ̂∆
iff
Ŷ0 − cσ̂∆ ≤ Y0 ≤ Ŷ0 + cσ̂∆ .
So,
P [Ŷ0 − cσ̂∆ ≤ Y0 ≤ Ŷ0 + cσ̂∆ ] = .95.
The interval
Y0 − Ŷ0 ∼
Let
2
Normal[0, σ∆
].
Ŷ0 ± cσ̂∆
is called a 95% prediction interval for Y0 .
(x0 − x̄)2 1
2
MSE .
σ̂∆
= 1+ +
n
sxx
Slide 31
Slide 32
Example
United Data Service
The Prediction Interval
Take
x0 = 4,
For this
Ŷ0 = µ̂0 = 66.198
µ0 = α + 4β.
Then
µ̂0 = 4.162 + 4 × 15.509 = 66.198.
1
(4 − 6)2 2
σ̂∆
= 1+
+
29.074
14
114
2
= (5.672)
Next
σ̂µ̂2 0
x̄ = 6
1
(4 − 6)2 +
29.074 = (1.76)2.
=
14
114
So
µ̂0 ± cσ̂µ̂0 = 66.198 ± 2.18 × 1.76
= 66.20 ± 3.84.
Slide 33
Ŷ0 ± cσ̂∆ = 66.198 ± 2.18 × 5.672
= 66.20 ± 12.36
Note: Average response versus individual
response.
Slide 34