Chapter 1 – Linear Regression with 1 Predictor
Statistical Model
Yi  0  1 X i  i
i  1, , n
where:




Yi is the (random) response for the ith case
0 , 1 are parameters
X i is a known constant, the value of the predictor variable for the ith case
 i is a random error term, such that:
E{i }  0  2 {i }   2  {i ,  j }  0  i, j  i  j
The last point states that the random errors are independent (uncorrelated), with mean 0,
and variance  2 . This also implies that:
E{Yi }  0  1 X i
 2 {Yi }   2  {Yi , Yj }  0
Thus, 0 represents the mean response when X  0 (assuming that is reasonable level of
X ), and is referred to as the Y-intercept. Also, 1 represent the change in the mean
response as X increases by 1 unit, and is called the slope.
Least Squares Estimation of Model Parameters
In practice, the parameters 0 and 1 are unknown and must be estimated. One widely
used criterion is to minimize the error sum of squares:
Yi  0  1 X i  i
n
Q
 i2 
i 1

n
 (Y  (
i 1
i
0
i  Yi  ( 0  1 X i )
 1 X i )) 2
This is done by calculus, by taking the partial derivatives of Q with respect to 0 and 1
and setting each equation to 0. The values of 0 and 1 that set these equations to 0 are
the least squares estimates and are labelled b0 and b1 .
First, take the partial derivates of Q with respect to 0 and 1 :
n
Q
 2 (Yi  ( 0  1 X i ))(  1)
0
i 1
(1)
n
Q
 2 (Yi  ( 0  1 X i ))(  X i )
0
i 1
( 2)
Next, set these these 2 equations to 0, replacing 0 and 1 with b0 and b1 since these are
the values that minimize the error sum of squares:
n
n
 2 (Yi  b0  b1 X i )  0 
n
Y
i 1
i
i 1
n
 nb0  b1  X i
n
 2 (Yi  b0  b1 X i ) X i  0 
 XY
i 1
(1a )
i 1
i i
i 1
n
n
i 1
i 1
 b0  X i  b1  X i2
( 2a )
These two equations are referred to as the normal equations (although, note that we
have said nothing YET, about normally distributed data).
Solving these two equations yields:
n
b1 
 (X
i 1
i
 X )(Yi  Y )
n
 (X
i 1
i
 X)
2
( Xi  X )
n


i 1
n
 (X
i 1
1


 n  X k i Yi 
i 1
n
b0  Y  b1 X 
n
Yi 
i
 X)
2
kY
i 1
i i
n
 lY
i 1
i i
where k i and li are constants, and Yi is a random variable with mean and variance given
above:
ki 
Xi  X
n
 (X
i 1
li 
i
 X )2
X ( Xi  X )
1
1
 Xk i   n
n
n
 ( Xi  X )2
i 1
The fitted regression line, also known as the prediction equation is:
^
Y  b0  b1 X
The fitted values for the individual observations aye obtained by plugging in the
corresponding level of the predictor variable ( X i ) into the fitted equation. The residuals
^
are the vertical distances between the observed values ( Yi ) and their fitted values ( Y i ),
and are denoted as ei .
^
^
Y i  b0  b1 X i
ei  Yi  Y i
Properties of the fitted regression line
n

e
i 1
n


Xe
Y
i
The residuals sum to 0
0
The sum of the weighted (by X ) residuals is 0
ei  0
The sum of the weighted (by Y ) residuals is 0
i i
i 1
n ^
i 1

0
i
^
The regression line goes through the point ( X , Y )
These can be derived via their definitions and the normal equations.
Estimation of the Error Variance
Note that for a random variable, its variance is the expected value of the squared
deviation from the mean. That is, for a random variable W , with mean W its variance is:
 2 {W}  E{(W  W ) 2 }
For the simple linear regression model, the errors have mean 0, and variance  2 . This
means that for the actual observed values Yi , their mean and variance are as follows:
E{Yi }  0  1 X i
 2 {Yi }  E{(Yi  (0  1 X i )) 2 }   2
^
First, we replace the unknown mean 0  1 X i with its fitted value Y i  b0  b1 X i , then
we take the “average” squared distance from the observed values to their fitted values.
We divide the sum of squared errors by n-2 to obtain an unbiased estimate of  2 (recall
how you computed a sample variance when sampling from a single population).
n
s2
 (Y  Y
i
i 1

n
^
i
)
e
2
i 1

n 2
2
i
n 2
Common notation is to label the numerator as the error sum of squares (SSE).
n
SSE


i 1
n
^
(Yi  Y i ) 2
e

i 1
2
i
Also, the estimated variance is referred to as the error (or residual) mean square
(MSE).

MSE

s2
SSE
n 2
To obtain an estimate of the standard deviation (which is in the units of the data), we take
the square root of the erro mean square. s  MSE .
A shortcut formula for the error sum of squares, which can cause problems due to roundoff errors is:
n
SSE
 (Y  Y )

i
i 1
n
2
 b1  ( X i  X )(Yi  Y )
i 1
Some notation makes life easier when writing out elements of the regression model:
n
SS XX 
 (X
i 1
n
 X) 
2
i

i 1
 n

  Xi 
 i 1 
X i2 
n
n
SS XY 
SSYY

i 1
n
( X i  X )(Yi  Y ) 

i 1
2
 n
 n 
  X i    Yi 
 i 1   i 1 
X i Yi 
n
 n 
  Yi 
n
n
 i 1 
2
2
  (Yi  Y )   Yi 
n
i 1
i 1
2
Note that we will be able to obtain most all of the simple linear regression analysis from
these quantities, the sample means, and the sample size.
SS XY
b1 
SS XX
SSE  SSYY
( SS XY ) 2

SS XX
Normal Error Regression Model
If we add further that the random errors follow a normal distribution, then the response
variable also has a normal distribution, with mean and variance given above. The
notation, we will use for the errors, and the data is:
i ~ N (0,  2 )
Yi ~ N (0  1 X i ,  2 )
The density function for the ith observation is:
fi 
 1  Yi  0  1 X i  2 
1
exp   
 
 

2
 2
The likelihood function, is the product of the individual density functions (due to the
independence assumption on the random errors).
L( 0 , 1 ,  ) 
2

1
(2 )
2 n/2
n
1
 (2
i 1

1
exp  
2
 2
2 1/ 2
)
1

2
exp  
2 (Yi  0  1 X i ) 
 2

n
 (Y  
i 1
i
0

 1 X i ) 2 

The values of 0 , 1 ,  2 that maximize the likelihood function are referred to as
^
^
^
maximum likelihood estimators. The MLE’s are denoted as: 0 ,  1 ,  2 . Note that the
natural logarithm of the likelihood is maximized by the same values of 0 , 1 ,  2 that
maximize the likelihood function, and it’s easier to work with the log likelihood function.
log e L  
n
n
1
log(2 )  log( 2 ) 
2
2
2 2
n
 (Y  
i 1
i
0
 1 X i ) 2
Taking partial derivatives with respect to 0 , 1 ,  2 yields:
 log L
1
 2 2
0
2
n
 (Y  
i
i 1
 log L
1
 2 2
1
2
 1 X i )(  1)
0
n
 (Y  
i
i 1
 1 X i )(  X i )
0
 log L
n
1
 
2
2 

2
2( 2 ) 2
n
 (Y  
i
i 1
0
 1 X i ) 2
( 4)
(5)
(6)
Setting these three equations to 0, and placing “hats” on parameters denoting the
maximum likelihood estimators, we get the following three equations:
n
n
 Yi  n  0   1  X i
^
^
i 1
n
 XY
i i
i 1
1
^ 4

( 4a )
i 1
n
n
  0  X i   1  X i2
^
i 1
n
 (Y  
^
i
i 1
^
(5a )
i 1
^
 1 Xi )2 
0
n
(6a )
^ 2

From equations 4a and 5a, we see that the maximum likelihood estimators are the same
as the least squares estimators (these are the normal equations). However, from equation
6a, we obtain the maximum likelihood estimator for the error variance as:
n
^ 2
 
 (Y  
i 1
^
i
n
^
 1 Xi )2
0
n

 (Y  Y
i 1
^
i
i
)2
n
This estimator is biased downward. We will use the unbiased estimator s 2  MSE
throughout this course to estimate the error variance.
Example – LSD Concentration and Math Scores
A pharmacodynamic study was conducted at Yale in the 1960’s to determine the
relationship between LSD concentration and math scores in a group of volunteers. The
independent (predictor) variable was the mean tissue concentration of LSD in a group of
5 volunteers, and the dependent (response) variable was the mean math score among the
volunteers. There were n=7 observations, collected at different time points throughout the
experiment.
Source: Wagner, J.G., Agahajanian, G.K., and Bing, O.H. (1968), “Correlation of
Performance Test Scores with Tissue Concentration of Lysergic Acid Diethylamide in
Human Subjects,” Clinical Pharmacology and Therapeutics, 9:635-638.
The following EXCEL spreadsheet gives the data and pertinent calculations.
Time (i)
Sum
Mean
Score (Y)
Conc (X)
Y-Ybar
X-Xbar
(Y-Ybar)**2
(X-Xbar)**2
(X-Xbar)(Y-Ybar) Yhat
e
e**2
1
78.93
1.17
28.84286
-3.162857
831.9104082 10.0036653
-91.22583673
78.5828
0.3472
0.1205
2
58.20
2.97
8.112857
-1.362857
65.81845102 1.85737959
-11.05666531 62.36576
-4.1658
17.354
3
67.47
3.26
17.38286
-1.072857
302.1637224 1.15102245
-18.64932245 59.75301
7.717
59.552
4
37.47
4.69
-12.61714
0.357143
159.1922939 0.12755102
-4.506122449 46.86948
-9.3995
88.35
5
45.65
5.83
-4.437143
1.497143
19.68823673 2.24143673
-6.643036735 36.59868
9.0513
81.926
6
32.92
6.00
-17.16714
1.667143
294.7107939 2.77936531
-28.62007959 35.06708
-2.1471
4.6099
7
29.97
6.41
-20.11714
2.077143
404.6994367 4.31452245
-41.78617959 31.37319
-1.4032
1.969
350.61
30.33
0
0
2078.183343 22.4749429
-202.4872429
1E-14
253.88
50.08714286
4.3328571
b1
-9.009466
b0
89.123874
MSE
50.776266
^
The fitted equation is: Y  89.12  9.01X and the estimated error variance is
s 2  MSE  50.78 , with corresponding standard deviation s  7.13 .
350.61
A plot of the data and the fitted equation are given below, obtained from EXCEL.
Math Score vs LSD Concentration
90
80
Math Score (Y)
70
60
50
40
30
20
10
0
0
1
2
3
4
5
6
7
LSD Concentration (X)
Output from various software packages is given below. Rules for standard errors and tests
are given in the next chapter. We will mainly use SAS, EXCEL, and SPSS throughout the
semester.
1) EXCEL (Using Built-in Data Analysis Package)
Data Cells
Time (i)
1
2
3
4
5
6
7
Score (Y) Conc (X)
78.93
1.17
58.2
2.97
67.47
3.26
37.47
4.69
45.65
5.83
32.92
6
29.97
6.41
Regression Coefficients
Intercept
Conc (X)
Coefficient Standard
t Stat
P-value
Lower
Upper
s
Error
95%
95%
89.12387 7.047547 12.64608 5.49E-05 71.00761 107.2401
-9.00947 1.503076 -5.99402 0.001854 -12.8732 -5.14569
Fitted Values and Residuals
Observation
1
2
3
4
5
6
7
Predicted
Score (Y)
78.5828
62.36576
59.75301
46.86948
36.59868
35.06708
31.37319
Residuals
0.347202
-4.16576
7.716987
-9.39948
9.051315
-2.14708
-1.40319
2) SAS (Using PROC REG)
Program (Bottom portion generates graphics quality plot for WORD)
options nodate nonumber ps=55 ls=76;
title ‘Pharmacodynamic Study’;
title2 ‘Y=Math Score
X=Tissue LSD Concentration’;
data lsd;
input score conc;
cards;
78.93 1.17
58.20 2.97
67.47 3.26
37.47 4.69
45.65 5.83
32.92 6.00
29.97 6.41
;
run;
proc reg;
model score=conc / p r;
run;
symbol1 c=black i=rl v=dot;
proc gplot;
plot score*conc=1 / frame;
run;
quit;
Program Output (Some output suppressed)
Pharmacodynamic Study
Y=Math Score
X=Tissue LSD Concentration
The REG Procedure
Model: MODEL1
Dependent Variable: score
Parameter Estimates
Variable
Intercept
conc
DF
1
1
Parameter
Estimate
89.12387
-9.00947
Standard
Error
7.04755
1.50308
t Value
12.65
-5.99
Pr > |t|
<.0001
0.0019
Output Statistics
Obs
1
2
3
4
5
6
7
Dep Var
score
78.9300
58.2000
67.4700
37.4700
45.6500
32.9200
29.9700
Predicted
Value
78.5828
62.3658
59.7530
46.8695
36.5987
35.0671
31.3732
Std Error
Mean Predict
5.4639
3.3838
3.1391
2.7463
3.5097
3.6787
4.1233
Residual
0.3472
-4.1658
7.7170
-9.3995
9.0513
-2.1471
-1.4032
Std Error
Residual
4.574
6.271
6.397
6.575
6.201
6.103
5.812
Student
Residual
0.0759
-0.664
1.206
-1.430
1.460
-0.352
-0.241
Plot (Including Regression Line)
3) SPSS (Spreadsheet/Menu Driven Package)
Output (Regression Coefficients Portion)
Coeffi cientsa
Model
1
(Const ant)
Conc (X)
Unstandardized
Coeffic ients
B
St d. Error
89.124
7.048
-9. 009
1.503
a. Dependent Variable: Sc ore (Y)
St andardiz ed
Coeffic ients
Beta
-.937
t
12.646
-5. 994
Sig.
.000
.002
Plot of Data and Regression Line
Math Score vs LSD Concentration (SPSS)
80.00

Linear Regression
70.00
Score (Y)

60.00

50.00

40.00



30.00
1.00
2.00
Score (Y) = 89.12 + -9.01 * conc
R-Square
3.00
4.00= 0.88 5.00
6.00
Conc (X)
4) STATVIEW (Spreadsheet/Menu Driven Package from SAS)
Output (Regression Coefficients Portion)
Regression Coefficients
Score (Y) vs. Conc (X)
Coefficient
Std. Error
Std. Coeff.
t-Value
P-Value
Intercept
89.124
7.048
89.124
12.646
<.0001
Conc (X)
-9.009
1.503
-.937
-5.994
.0019
Graphic output
Regression Plot
80
70
Score (Y)
60
50
40
30
20
1
2
3
4
Conc (X)
5
6
7
Y = 89.124 - 9.009 * X; R^2 = .878
5) S-Plus (Also available in R)
Program Commands






x <- c(1.17, 2.97, 3.26, 4.69, 5.83, 6.00, 6.41)
y <- c(78.93, 58.20, 67.47, 37.47, 45.65, 32.92,
29.97)
plot (x,y)
fit <- lm(y ~ x)
abline (fit)
summary (fit)
Program Output
Residuals:
1
2
3
4
5
6
7
0.3472 –4.166 7.717 –9.399 9.051 –2.147 –1.403
Coefficients:
(Intercept)
x
Value Std. Error
89.1239
7.0475
-9.0095
1.5031
t value Pr(>|t|)
12.6461
0.0001
-5.9940
0.0019
Residual standard error: 7.126 on 5 degrees of freedom
30 40 50 y 60 70 80
Graphics Output
1
2
3
4
5
6
x
6) STATA
Output (Regression Coefficients Portion)
score Coef.
Std. Err.
conc -9.009467
_cons 89.12388
1.503077
7.047547
t
P>t
[95% Conf.
Interval]
-5.99 0.002 -12.87325
12.65 0.000 71.00758
-5.145686
107.2402
Graphics Output
Math Scores vs LSD Concentration
60
50
40
30
score/Fitted values
70
80
STATA Output
1
2
3
4
conc
score
Fitted values
5
6
Chapter 2 – Inferences in Regression Analysis
Rules Concerning Linear Functions of Random Variables (P. 1318)
n
Let Y1 ,, Yn be n random variables. Consider the function
a Y
i 1
i i
where the coefficients
a1 ,, a n are constants. Then, we have:
n

E  ai Yi  
 i 1

n

  ai Yi  
 i 1

2
n
 a E{Y }
i 1
n
i
i
n
 a a  {Y , Y }
i
i 1 j 1
j
i
j
When Y1 ,, Yn are independent (as in the model in Chapter 1), the variance of the linear
combination simplifies to:

n

 2  ai Yi  
 i 1

n
a 
i 1
2
i
2
{Yi }
n
When Y1 ,, Yn are independent, the covariance of two linear functions
a Y
i 1
i i
and
n
c Y
i 1

i i
n
can be written as:
n

  ai Yi , ci Yi , 
 i 1
i 1

n
a c 
i 1
i i
2
{Yi }
^
We will use these rules to obtain the distribution of the estimators b0 , b1 , Y  b0  b1 X
Inferences Concerning 1
Recall that the least squares estimate of the slope parameter, b1 , is a linear function of
the observed responses Y1 ,, Yn :
n
b1 
 (X
i 1
i
 X )(Yi  Y )
n
 (X
i 1
ki 
i
( Xi  X )
n


i 1
 X )2
n
 (X
i 1
i
 X )2
n
Yi 
kY
i 1
i i
( Xi  X )
n
 (X
 X )2
i
i 1
Note that E {Yi }  0  1 X i , so that the expected value of b1 is:
n
E {b1 } 
( Xi  X )
n

i 1
k i E {Yi } 

i 1
 (X
i 1

( 0  1 X i )
n
i
 X)
2
n
n



(
X

X
)


 0
i
1 ( Xi  X ) Xi 
i 1

( X i  X ) 2  i 1
1
n

i 1
n
Note that
 (X
i 1
i
 X)  0
(why?), so that the first term in the brackets is 0, and that
n
we can add 1 X  ( X i  X )  0 to the last term to get:
i 1
E {b1 } 
n
 n


(
X

X
)
X


 1
i
i
1 ( Xi  X ) X  
i 1

( X i  X ) 2  i 1
1
n

i 1
Thus, b1 is an unbiased estimator of the parameter 1 .
To obtain the variance of b1 , recall that  2 {Yi }   2 . Thus:
1
n
 (X
i 1
i
 X )2
n
1  ( X i  X ) 2  1
i 1
2
n


( Xi  X )2


n
n 
(
X

X
)
2
  2  i 1
 2 {b1 }   k i2 2 {Yi }    n i

2 
n


i 1
i 1 
2 
2
  ( Xi  X ) 
 ( Xi  X ) 
 i 1

 i 1

2
n
 (X
i 1
i
 X )2
Note that the variance of b1 decreases when we have larger sample sizes (as long as the
added X levels are not placed at the sample mean X ). Since  2 is unknown in practice,
and must be estimated from the data, we obtain the estimated variance of the estimator b1
by replacing the unknown  2 with its unbiased estimate s 2  MSE :
s {b1 } 
s2
2

n
 (X
i 1
 X )2
i
MSE
n
 (X
i 1
i
 X )2
with estimated standard error:
s{b1} 
s

n
 (X
i 1
i
 X )2
MSE
n
 (X
i 1
i
 X )2
Further, the sampling distribution of b1 is normal, that is:




2



b1 ~ N 1 , n


( Xi  X )2 




i 1
since under the current model, b1 is a linear function of independent, normal random
variables Y1 ,, Yn .
Making use of theory from mathematical statistics, we obtain the following result that
allows us to make inferences concerning 1 :
b1  1
~ t (n  2) where t(n-2) represents Student’s t-distribution with n-2 degrees of
s{b1}
freedom.
Confidence Interval for 1
As a result of the fact that
b1  1
~ t (n  2) , we obtain the following probability
s{b1}
statement:
b1  1
 t (1   / 2; n  2)}  1   where t ( / 2; n  2) is the
s{b1}
(/2)100th percentile of the t-distribution with n-2 degrees of freedom. Note that since the
t-distribution is symmetric around 0, we have that t ( / 2; n  2)  t (1   / 2; n  2) .
Traditionally, we obtain the table values corresponding to t (1   / 2; n  2) , which is the
value of that leaves an upper tail area of /2. The following algebra results in obtaining a
(1-)100% confidence interval for 1:
P{t ( / 2; n  2) 
P{t ( / 2; n  2) 
b1  1
 t (1   / 2; n  2)}
s{b1 }
 P{t (1   / 2; n  2) 
b1  1
 t (1   / 2; n  2)}
s{b1 }
 P{t (1   / 2; n  2) s{b1 }  b1   1  t (1   / 2; n  2) s{b1 }}
 P{b1  t (1   / 2; n  2) s{b1 }   1  b1  t (1   / 2; n  2) s{b1 }}
 P{b1  t (1   / 2; n  2) s{b1 }   1  b1  t (1   / 2; n  2) s{b1 }}
This leads to the following rule for a (1-)100% confidence interval for 1:
b1  t (1   / 2; n  2)s{b1}
Some statistical software packages print this out automatically (e.g. EXCEL and SPSS).
Other packages simply print out estimates and standard errors only (e.g. SAS).
Tests Concerning 1
b1  1
~ t n  2 to test hypotheses concerning
s{b1 }
the slope parameter. As with means and proportions (and differences of means and
proportions), we can conduct one-sided and two-sided tests, depending on whether a
priori a specific directional belief is held regarding the slope. More often than not (but not
necessarily), the null value for 1 is 0 (the mean of Y is independent of X) and the
alternative is that 1 is positive (1-sided), negative (1-sided), or different from 0 (2sided). The alternative hypothesis must be selected before observing the data.
We can also make use of the of the fact that
2-sided tests


Null Hypothesis: H 0 : 1  10
Alternative (Research Hypothesis): H A : 1  10

Test Statistic: t* 


b1  1
s{b1}
Decision Rule: Conclude HA if | t* |  t (1   / 2; n  2) , otherwise conclude H0
P-value: 2 P(t (n  2) | t* |)
All statistical software packages (to my knowledge) will print out the test statistic and Pvalue corresponding to a 2-sided test with 10=0.
1-sided tests (Upper Tail)


Null Hypothesis: H 0 : 1  10
Alternative (Research Hypothesis): H A : 1  10

Test Statistic: t* 


b1  1
s{b1}
Decision Rule: Conclude HA if t *  t (1   ; n  2) , otherwise conclude H0
P-value: P(t (n  2)  t*)
A test for positive association between Y and X (HA:1>0) can be obtained from standard
statisical software by first checking that b1 (and thus t*) is positive, and cutting the
printed P-value in half.
1-sided tests (Lower Tail)
H 0 : 1  10

Null Hypothesis:

Alternative (Research Hypothesis): H A : 1  10

Test Statistic: t* 


b1  1
s{b1}
Decision Rule: Conclude HA if t *   t (1   ; n  2) , otherwise conclude H0
P-value: P(t (n  2)  t*)
A test for negative association between Y and X (HA:1<0) can be obtained from standard
statisical software by first checking that b1 (and thus t*) is negative, and cutting the
printed P-value in half.
Inferences Concerning 0
Recall that the least squares estimate of the intercept parameter, b0 , is a linear function
of the observed responses Y1 ,, Yn :


1
n
(X  X )X 
Yi   l i Yi
b0  Y  b1 X     n i
i 1  n
i 1
( X i  X ) 2 


i 1

n
Recalling that E{Yi }   0  1 X i :






1


n 
n 
(X  X )X
(X  X )X
(X  X )X 
1
1
 (  0  1 X i )   0    n i
 1    n i
Xi
E{b0 }     n i
2 
2 
2 
i 1  n
i 1  n
i 1  n
(X i  X ) 
(Xi  X ) 
(Xi  X ) 






i 1
i 1
i 1



n


2 
1 n
n
(X  X )
   0  1 ( X  X (1))   0
  0 (1  0)  1   X i  X  n i
 n i 1
2 
i 1
(Xi  X ) 


i 1

Thus, b0 is an unbiased estimator or the parameter 0. Below, we obtain the variance of
the estimator of b0.
2






2
n 
n
(X i  X )X  2
X (X i  X )2
2X (X i  X ) 
1
1
2
2

    2 
 {b0 }     n
 n
2


n
2 
2
i 1  n
i 1 n


2
(
X

X
)
n
(
X

X
)




i
i
(
X

X
)




 i
i 1
i 1
 i 1





2
X
2 n


2
n2
 n
2

(X i  X ) 

 i 1



n
n
2X
2
(Xi  X )  n
( X i  X )



i 1
n ( X i  X ) 2 i 1

i 1



2
1

X
2



 n
n
2 
(X i  X ) 


i 1

Note that the variance will decrease as the sample size increases, as long as X values are
not all placed at the mean. Further, the sampling distribution is normal under the
assumptions of the model. The estimated standard error of b0 replaces 2 with its
unbiased estimate s2=MSE and taking the square root of the variance.


2


1
X
1
X

s{b0 }  s
 n
 MSE   n
n
n

2
2 
(X i  X )
(X i  X ) 



i 1
i 1

2
b0   0
~ t (n  2) , allowing for inferences concerning the intercept parameter
s{b0 }
0 when it is meaningful, namely when X=0 is within the range of observed data.
Note that
Confidence Interval for 0
b0  t (1   / 2; n  2) s{b0 }
It is also useful to obtain the covariance of b0 and b1, as they are only independent under
very rare circumstances:

n
n

n
i 1

i 1
 {b0 , b1 }    l i Yi , k i Yi    l i k i  2 {Yi }
 i 1


1
X (X i  X )  (X i  X )
 n
  n
2

i 1  n
(X i  X )2   (X i  X )2


i 1

 i 1
n

2
n
n ( X i  X ) 2
 (X i  X ) 
i 1
i 1
 0
2X
n
 (X i  X )2
i 1
2X
n
 
 n

 (X i  X )2 
 i 1

n
2
 (X
i 1
i
 X )2
2X
n
 (X
i 1
i
 X )2
In practice, X is usually positive, so that the intercept and slope estimators are usually
negatively correlated. We will use the result shortly.
Considerations on Making Inferences Concerning 0 and 1
Normality of Error Terms
If the data are approximately normal, simulation results have shown that using the tdistribution will provide approximately correct significance levels and confidence
coefficients for tests and confidence intervals, respectively. Even if the distribution of the
errors (and thus Y) is far from normal, in large samples the sampling distributions of b0
and b1 have sampling distributions that are approximately normal as results of central
limit theorems. This is sometimes referred to as asymptotic normality.
Interpretations of Confidence Coefficients and Error Probabilities
Since X levels are treated as fixed constants, these refer to the case where we repeated the
experiment many times at the current set of X levels in this data set. In this sense, it’s
easier to interpret these terms in controlled experiments where the experimenter has set
the levels of X (such as time and temperature in a laboratory type setting) as opposed to
observational studies, where nature determines the X levels, and we may not be able to
reproduce the same conditions repeatedly. This will be covered later.
Spacing of X Levels
The variances of b0 and b1 (for given n and 2) decrease as the X levels are more spread
n
out, since their variances are inversely related to
(X
i 1
i
 X ) 2 . However, there are
reasons to choose a diverse range of X levels for assessing model fit. This is covered in
Chapter 4.
Power of Tests
The power of a statistical test refers to the probability that we reject the null hypothesis.
Note that when the null hypothesis is true, the power is simply the probability of a Type I
error (). When the null hypothesis is false, the power is the probability that we correctly
reject the null hypothesis, which is 1 minus the probability of a Type II error (=1-),
where denotes the power of the test and  is the probability of a Type II error (failing to
reject the null hypothesis when the alternative hypothesis is true). The following
procedure can be used to obtain the power of the test concerning the slope parameter with
a 2-sided alternative.
1) Write out null and alternative hypotheses: H 0 : 1  10
H A : 1  10
2) Obtain the noncentrality measure, the standardized distance between the true value of
   10
1 and the value under the null hypothesis (10):   1
 {b1 }
3) Choose the probability of a Type I error (=0.05 or =0.01)
4) Determine the degrees of freedom for error: df = n-2
5) Refer to Table B.5 (pages 1346-7), identifying (page),  (row) and error degrees of
freedom (column). The table provides the power of the test under these parameter
values.
Note that the power increases within each tables as the noncentrality measure increases
for a given degrees of freedom, and as the degrees of freedom increases for a given
noncentrality measure.
Confidence Interval for E{Yh}=0+1Xh
When we wish to estimate the mean at a hypothetical X value (within the range of
observed X values), we can use the fitted equation at that value of X=Xh as a point
estimate, but we haveto include the uncertainty in the regression estimators to construct a
confidence interval for the mean.
Parameter: E{Yh }   0  1 X h
^
Estimator: Y h  b0  b1 X h
We can obtain the variance of the estimator (as a function of X=Xh) as follows:
 2 Y h    2 b0  b1 X h    2 {b0 }  X h2 2 {b1 }  2 X h {b0 , b1 }
^




2
1

X
  X h2
 2  n
n

( X i  X ) 2 


i 1



2



 X



2
X

h
n
n

2
2 
(X i  X )
(X i  X ) 

 
i 1
i 1

2


2 
1
(X  X )

 2  n h
n
2 
(X i  X ) 


i 1



2 
1
(X  X )

Estimated standard error of estimator: s{Y h }  MSE   n h
n
2 
(Xi  X ) 


i 1

^
^
Y h  E{Yh }
~ t (n  2) which can be used to construct confidence intervals for the mean
^
s{Y h }
response at specific X levels, and tests concerning the mean (tests are rarely conducted).
(1-)100% Confidence Interval for E{Yh}:
^
^
Y h  t (1   / 2; n  2) s{Y h }
Predicting a Future Observation When X is Known
If  0 , 1 ,  were known, we’d know that the distribution of responses when X=Xh is
normal with mean  0  1 X h and standard deviation  . Thus, making use of the normal
distribution (and equivalently, the empirical rule) we know that if we took a sample item
from this distribution, it is very likely that the value fall within 2 standard deviations of
the mean. That is, we would know that the probability that the sampled item lies within
the range ( 0  1 X h   ,  0  1 X h   ) is approximately 0.95.
In practice, we don’t know the mean  0  1 X h or the standard deviation  . However,
we just constructed a (1-)100% Confidence Interval for E{Yh}, and we have an estimate
of  (s). Intuitively, we can approximately use the logic of the previous paragraph (with
the estimate of  ) across the range of believable values for the mean. Then our
prediction interval spans the lower tail of the normal curve centered at the lower bound
for the mean to the upper tail of the normal curve centered at the upper bound for the
mean. See Figure 2.5 on page 64 of the text book.
The prediction error is for the new observation is the difference between the observed
^
value and its predicted value: Yh  Y h . Since the data are assumed to be independent, the
new (future) value is independent of its predicted value, since it wasn’t used in the
regression analysis. The variance of the prediction error can be obtained as follows:


2

^
^
(X  X ) 
1

 2 { pred }   2 {Yh  Y h }   2 {Yh }   2 {Y h }   2   2   n h
n
2 
(Xi  X ) 


i 1



2 
 1
(X  X )

  2 1   n h
 n
2 
(Xi  X ) 


i 1

and an unbiased estimator is:


2 
 1
(X  X )

s 2 { pred }  MSE 1   n h
 n
2 
(X i  X ) 


i 1

(1-)100% Prediction Interval for New Observation When X=Xh


2 
 1
(X  X )

Y h  t ( / 2; n  2) MSE 1   n h
 n
2 
(X i  X ) 


i 1

^
It is a simple extension to obtain a prediction for the mean of m new observations when
X=Xh. The sample mean of m observations is
2
and we get the following variance for
m
for the error in the prediction mean:


2 
1 1
(X  X )

s 2 { predmean}  MSE    n h
m n
2 
(X i  X ) 


i 1

and the obvious adjustment to the prediction interval for a single observation.
(1-)100% Prediction Interval for the Mean of m New Observations When X=Xh


2 
1 1
(X  X )

Y h  t ( / 2; n  2) MSE    n h
m n
2 
(X i  X ) 


i 1

^
Confidence Band for the Entire Regression Line (Working-Hotelling Method)
^
^
Y h  Ws{Y h }
W  2 F (1   ;2, n  2)
Analysis of Variance Approach to Regression
Consider the total deviations of the observed responses from the mean: Yi  Y . When
these terms are all squared and summed up, this is referred to as the total sum of squares
(SSTO).
n
SSTO   (Yi  Y ) 2
i 1
The more spread out the observed data are, the larger SSTO will be.
Now consider the deviation of the observed responses from their fitted values based on
^
the regression model: Yi  Y i  Yi  (b0  b1 X i )  ei . When these terms are squared and
summed up, this is referred to as the error sum of squares (SSE). We’ve already
encounterd this quantity and used it to estimate the error variance.
n
^
SSE   (Yi  Y i ) 2
i 1
When the observed responses fall close to the regression line, SSE will be small. When
the data are not near the line, SSE will be large.
Finally, there is a third quantity, representing the deviations of the predicted values from
the mean. Then these deviations are squared and summed up, this is referred to as the
regression sum of squares (SSR).
n
^
SSR   (Y i  Y ) 2
i 1
The error and regression sums of squares sum to the total sum of squares:
SSTO  SSR  SSE which can be seen as follows:
^
^
^
^
Yi  Y  Yi  Y  Y i  Y i  (Yi  Y i )  (Y i  Y )
^
^
^

^
^
^
(Yi  Y ) 2  [(Yi  Y i )  (Y i  Y )] 2  (Yi  Y i ) 2  (Y i  Y ) 2  2(Yi  Y i )(Y i  Y )



SSTO   (Yi  Y ) 2   (Yi  Y i ) 2  (Y i  Y ) 2  2(Yi  Y i )(Y i  Y ) 

i 1
i 1 
n
n
^
n
n
^
^
n
^
^
^
^
^
 (Yi  Y i ) 2   (Y i  Y ) 2  2 (Yi  Y i )(Y i  Y ) 
i 1
i 1
n
 (Y
i 1
i
n
 (Y
i 1
i
n
 (Y
i 1
i
^
n
i 1
n
^
 Y i ) 2   (Y i  Y ) 2  2 ei (b0  b1 X i  Y ) 
i 1
i 1
n
n
 n

 Y i ) 2   (Y i  Y ) 2  2 b0  ei  b1  ei X i  Y  ei  
i 1
i 1
i 1
 i 1

^
n
^
^
n
^
n
^
n
^
 Y i ) 2   (Y i  Y ) 2  2(0)   (Yi  Y i ) 2   (Y i  Y ) 2  SSE  SSR
i 1
i 1
The last term was 0 since
i 1
e  e X
i
i
i
 0,
Each sum of squares has associated with degrees of freedom. The total degrees of
freedom is dfT = n-1. The error degrees of freedom is dfE = n-2. The regression degrees of
freedom is dfR = 1. Note that the error and regression degrees of freedom sum to the total
degrees of freedom: n  1  1  (n  2) .
Mean squares are the sums of squares divided by their degrees of freedom:
MSR 
SSR
1
MSE 
SSE
n2
Note that MSE was our estimate of the error variance, and that we don’t compute a total
mean square. It can be shown that the expected values of the mean squares are:
E{MSE}   2
n
E{MSR}   2  12  ( X i  X ) 2
i 1
Note that these expected mean squares are the same if and only if 1=0.
The Analysis of Variance is reported in tabular form:
Source
Regression
Error
C Total
df
1
n-2
n-1
SS
SSR
SSE
SSTO
MS
MSR=SSR/1
MSE=SSE/(n-2)
F
F=MSR/MSE
F Test of 1 = 0 versus 1  0
As a result of Cochran’s Theorem (stated on page 76 of text book), we have a test of
whether the dependent variable Y is linearly related to the predictor variable X. This is a
very specific case of the t-test described previously. Its full utility will be seen when we
consider multiple predictors. The test proceeds as follows:

Null hypothesis: H 0 : 1  0

Alternative (Research) Hypothesis: H A : 1  0

Test Statistic: TS : F * 

Rejection Region: RR : F *  F (1   ;1, n  2)

P-value: P{F (1, n  2)  F *}
MSR
MSE
Critical values of the F-distribution (indexed by numerator and denominator degrees’ of
freedom) are given in Table B.4, pages 1340-1345.
Note that this is a very specific version of the t-test regarding the slope parameter,
specifically a 2-sided test of whether the slope is 0. Mathematically, the tests are
identical:
 ( X  X )(Y  Y )  ( X  X )(Y  Y )
0
 (X  X )   (X  X )

i
b
t*  1
s{b1 }
Note that:
i
i
i
2
2
i
i
MSE
 (X
i
 X )2
MSE
^
MSR  SSR   (Y i  Y ) 2   (b0  b1 X i  Y ) 2 
nb02  b12  X i2  nY  2b0 b1  X i  2nb0 Y  2b1 Y  X i
2

n(Y  b1 X ) 2  b12  X i2  nY  2(Y  b1 X )b1 n X  2n(Y  b1 X )Y  2b1 Y n X
2

nY  nb12 X  2nb1 X Y  b12  X i2  nY  2nb1 X Y  2nb12 X  2nY  2nb1 X Y  2nb1 X Y
2
2
2
2
2
 (nY  nY  2nY )  (2nb1 X Y  2nb1 X Y  2nb1 X Y  2nb1 X Y )  (b12  X i2  nb12 X  2nb12 X )
2
2
2
2
  ( X i  X )(Yi  Y ) 
 0  0  b  X  nb X  b  ( X i  X )  

2
  ( X i  X )

2
1
 ( X

i
2
i
 X )(Yi  Y )
(X
i
2
1
2
2
1
2
2
(X
i
 X )2

2
 X )2
Thus:
  ( X i  X )(Yi  Y ) 


2


(
X

X
)

i
(t*) 2  
 
MSE






2
 ( X
i
 X )(Yi  Y )
(X
i
 X )2
MSE

2

MSR
F*
MSE
Further, the critical values are equivalent: (t (1   / 2; n  2)) 2  F (1   ;1, n  2) ,
check this from the two tables. Thus, the tests are equivalent.
2
General Linear Test Approach
This is a very general method of testing hypotheses concerning regression models. We
first consider the the simple linear regression model, and testing whether Y is linearly
associated with X. We wish to test H 0 : 1  0 vs H A : 1  0 .
Full Model
This is the model specified under the alternative hypothesis, also referrred to as the
unrestricted model. Under simple linear regression with normal errors, we have:
Yi   0  1 X i   i
Using least squares (and maximum likelihood) to estimate the model parameters
^
( Y i  b0  b1 X i ), we obtain the error sum of squares for the full model:
^
SSE ( F )   (Yi  (b0  b1 X i )) 2   (Yi  Y i ) 2  SSE
Reduced Model
This the model specified by the null hypothesis, also referred to as the restricted model.
Under simple linear regression with normal errors, we have:
Yi   0  0 X i   i   0   i
Under least squares (and maximum likelihood) to estimate the model parameter, we
obtain Y as the estimate of 0, and have b0  Y as the fitted value for each observation.
We when get the following error sum of squares under the reduced model:
SSE ( R)   (Yi  b0 ) 2   (Yi  Y ) 2  SSTO
Test Statistic
The error sum of squares for the full model will always be less that or equal to the error
sum of squares for reduced model, by definition of least squares. The test statistic will be:
SSE ( R)  SSE ( F )
df R  df F
where df R , df F are the error degrees of freedom for the
F* 
SSE ( F )
df F
full and reduced models. We will use this method throughout course.
For the simple linear regression model, we obtain the following quantities:
SSE( F )  SSE
df F  n  2
SSE( R)  SSTO
df R  n  1
thus the F-Statistic for the General Linear Test can be written:
SSE ( R)  SSE ( F )
SSTO  SSE
SSR
df R  df F
MSR
(n  1)  (n  2)
F* 

 1 
SSE ( F )
SSE
SSE
MSE
df F
n2
n2
Thus, for this particular null hypothesis, the general linear test “generalizes” to the F-test.
Descriptive Measures of Association
Along with the slope, Y-intercept, and error variance; several other measures are often
reported.
Coefficient of Determination (r2)
The coefficient of determination measures the proportion of the variation in Y that is
“explained” by the regression on X. It is computed as the regression sum of squares
divided by the total (corrected) sum of squares. Values near 0 imply that the regression
model has done little to “explain” variation in Y, while values near 1 imply that the model
has “explained” a large portion of the variation in Y. If all the data fall exactly on the
fitted line, r2=1. The coefficient of determination will lie beween 0 and 1.
r2 
SSR
SSE
 1
SSTO
SSTO
0  r2 1
Coefficient of Correlation (r)
The coefficient of correlation is a measure of the strength of the linear association
between Y and X. It will always be the same sign as the slope estimate (b1), but it has
several advantages:

In some applications, we cannot identify a clear dependent and independent
variable, we just wish to determine how two variables vary together in a population
(peoples heights and weights, closing stock prices of two firms, etc). Unlike the slope
estimate, the coefficient of correlation does not depend on which variable is labeled
as Y, and which is labeled as X.
 The slope estimate depends on the units of X and Y, while the correlation
coefficient does not.

The slope estimate has no bound on its range of potential values. The correlation
coefficient is bounded by –1 and +1, with higher values (in absolute value) implying
stronger linear association (it is not useful in measuring nonlinear association which
may exist, however).
r  sgn( b1 ) r 2 
 ( X  X )(Y  Y )  s
 ( X  X )(Y  Y ) s
i
i
i
i
x
b1
1  r  1
y
where sgn(b1) is the sign (positive or negative) of b1, and s x , s y are the sample standard
deviations of X and Y, respectively.
Issues in Applying Regression Analysis

When using regression to predict the future, the assumption is that the conditions
are the same in future as they are now. Clearly any future predictions of economic
variables such as tourism made prior to September 11, 2001 would not be valid.
 Often when we predict in the future, we must also predict X, as well as Y, especially
when we aren’t controlling the levels of X. Prediction intervals using methods
described previously will be too narrow (that is, they will overstate confidence
levels).
 Inferences should be made only within the range of X values used in the regression
analysis. We have no means of knowing whether a linear association continues
outside the range observed. That is, we should not extrapolate outside the range of X
levels observed in experiment.
 Even if we determine that X and Y are associated based on the t-test and/or F-test,
we cannot conclude that changes in X cause changes in Y. Finding an association is
only one step in demonstrating a causal relationship.
 When multiple tests and/or confidence intervals are being made, we must adjust our
confidence levels. This is covered in Chapter 4.
 When Xi is a random variable, and not being controlled, all methods described thus
far hold, as long as the Xi are independent, and their probability distribution does not
depend on  0 ,  1 ,  2 .