Statistics for Managers
using Microsoft Excel
3rd Edition
Chapter 11
Simple Linear Regression
© 2002 Prentice-Hall, Inc.
Chap 11-1
Chapter Topics







Types of regression models
Determining the simple linear regression
equation
Measures of variation
Assumptions of regression and correlation
Residual analysis
Measuring autocorrelation
Inferences about the slope
© 2002 Prentice-Hall, Inc.
Chap 11-2
Chapter Topics



(continued)
Correlation - measuring the strength of the
association
Estimation of mean values and prediction of
individual values
Pitfalls in regression and ethical issues
© 2002 Prentice-Hall, Inc.
Chap 11-3
Purpose of Regression Analysis

Regression analysis is used primarily to
model causality and provide prediction


Predicts the value of a dependent (response)
variable based on the value of at least one
independent (explanatory) variable
Explains the effect of the independent variables
on the dependent variable
© 2002 Prentice-Hall, Inc.
Chap 11-4
Types of Regression Models
Positive Linear Relationship
Negative Linear Relationship
© 2002 Prentice-Hall, Inc.
Relationship NOT Linear
No Relationship
Chap 11-5
Simple Linear Regression Model



Relationship between variables
is described by a linear function
The change of one variable
causes the change in the
other variable
A dependency of one variable
on the other
© 2002 Prentice-Hall, Inc.
Chap 11-6
Population Linear Regression
Population regression line is a straight line that
describes the dependence of the average value
(conditional mean) of one variable on the other
Population
Slope
Coefficient
Population
Y intercept
Dependent
(Response)
Variable
© 2002 Prentice-Hall, Inc.
Random
Error
Yi     X i  i
Population
Regression
YX
Line
(conditional mean)

Independent
(Explanatory)
Variable
Chap 11-7
Population Linear Regression
(continued)
Y
(Observed Value of Y) = Yi
    X i  i
 i = Random Error

YX     X i

(Conditional Mean)
X
Observed Value of Y
© 2002 Prentice-Hall, Inc.
Chap 11-8
Sample Linear Regression
Sample regression line provides an estimate
of the population regression line as well as a
predicted value of Y
Sample
Y Intercept
Yi  b0  b1 X i  ei
Sample
Slope
Coefficient
Residual
Sample Regression Line
ˆ
Y  b0  b1 X  (Fitted Regression Line, Predicted Value)
© 2002 Prentice-Hall, Inc.
Chap 11-9
Sample Linear Regression
(continued)

b0 and b1 are obtained by finding the values
of b0 and b that minimizes the sum of the
1
squared residuals

n
i 1


Yi  Yˆi
  e
2
n
i 1
2
i
b0 provides an estimate of  
b1 provides and estimate of 
© 2002 Prentice-Hall, Inc.
Chap 11-10
Sample Linear Regression
(continued)
Yi  b0  b1 X i  ei
Y
ei
Yi     X i  i
b1
i

YX     X i

b0
Y i  b0  b1 X i
X
Observed Value
© 2002 Prentice-Hall, Inc.
Chap 11-11
Interpretation of the
Slope and the Intercept

  E Y | X  0 is the average value of Y
when the value of X is zero.

E Y | X 
1 
measures the change in the
X
average value of Y as a result of a one-unit
change in X.
© 2002 Prentice-Hall, Inc.
Chap 11-12
Interpretation of the
Slope and the Intercept
(continued)

b  Eˆ Y | X  0  is the estimated average
value of Y when the value of X is zero.

Eˆ Y | X 
b1 
X
is the estimated change in
the average value of Y as a result of a oneunit change in X.
© 2002 Prentice-Hall, Inc.
Chap 11-13
Simple Linear Regression:
Example
You want to examine
the linear dependency
of the annual sales of
produce stores on their
size in square footage.
Sample data for seven
stores were obtained.
Find the equation of
the straight line that
fits the data best.
© 2002 Prentice-Hall, Inc.
Store
Square
Feet
Annual
Sales
($1000)
1
2
3
4
5
6
7
1,726
1,542
2,816
5,555
1,292
2,208
1,313
3,681
3,395
6,653
9,543
3,318
5,563
3,760
Chap 11-14
Scatter Diagram: Example
Annua l Sa le s ($000)
12000
10000
8000
6000
4000
2000
0
0
Excel Output
© 2002 Prentice-Hall, Inc.
1000
2000
3000
4000
5000
6000
S q u a re F e e t
Chap 11-15
Equation for the Sample
Regression Line: Example
Yˆi  b0  b1 X i
 1636.415  1.487 X i
From Excel Printout:
C o e ffi c i e n ts
I n te r c e p t
1 6 3 6 .4 1 4 7 2 6
X V a ria b le 1 1 .4 8 6 6 3 3 6 5 7
© 2002 Prentice-Hall, Inc.
Chap 11-16
Graph of the Sample
Regression Line: Example
Annua l Sa le s ($000)
12000
10000
8000
6000
4000
2000
0
0
1000
2000
3000
4000
5000
6000
S q u a re F e e t
© 2002 Prentice-Hall, Inc.
Chap 11-17
Interpretation of Results:
Example
Yˆi  1636.415 1.487 Xi
The slope of 1.487 means that for each increase of
one unit in X, we predict the average of Y to
increase by an estimated 1.487 units.
The model estimates that for each increase of one
square foot in the size of the store, the expected
annual sales are predicted to increase by $1487.
© 2002 Prentice-Hall, Inc.
Chap 11-18
Simple Linear Regression in
PHStat


In excel, use PHStat | regression | simple
linear regression …
EXCEL spreadsheet of regression sales on
footage
© 2002 Prentice-Hall, Inc.
Chap 11-19
Measure of Variation:
The Sum of Squares
SST
=
Total
=
Sample
Variability
© 2002 Prentice-Hall, Inc.
SSR
Explained
Variability
+
SSE
+
Unexplained
Variability
Chap 11-20
Measure of Variation:
The Sum of Squares

SST = total sum of squares


Measures the variation of the Yi values around
their mean Y
SSR = regression sum of squares


(continued)
Explained variation attributable to the relationship
between X and Y
SSE = error sum of squares

Variation attributable to factors other than the
relationship between X and Y
© 2002 Prentice-Hall, Inc.
Chap 11-21
Measure of Variation:
The Sum of Squares
(continued)

SSE =(Yi - Yi )2
Y
_
SST = (Yi - Y)2
 _
SSR = (Yi - Y)2
Xi
© 2002 Prentice-Hall, Inc.
_
Y
X
Chap 11-22
Venn Diagrams and
Explanatory Power of Regression
Variations in
store sizes not
used in
explaining
variation in
sales
Sizes
Sales
Variations in
sales explained
by the error
term  SSE 
Variations in sales
explained by sizes or
variations in sizes
used in explaining
variation in sales
 SSR
© 2002 Prentice-Hall, Inc.
Chap 11-23
The ANOVA Table in Excel
ANOVA
df
Regression
p
SS
MS
SSR
MSR
=SSR/p
Residuals
n-p-1 SSE
Total
n-1
© 2002 Prentice-Hall, Inc.
F
Significance
F
MSR/MSE
P-value of
the F Test
MSE
=SSE/(n-p-1)
SST
Chap 11-24
Measures of Variation
The Sum of Squares: Example
Excel Output for Produce Stores
Degrees of freedom
ANOVA
df
SS
MS
Regression
1
30380456.12
30380456
Residual
5
1871199.595 374239.92
Total
6
32251655.71
F
81.17909
Regression (explained) df
Error (residual) df
Total df
© 2002 Prentice-Hall, Inc.
SSE
SSR
Significance F
0.000281201
SST
Chap 11-25
The Coefficient of Determination


SSR Regression Sum of Squares
r 

SST
Total Sum of Squares
2
Measures the proportion of variation in Y
that is explained by the independent
variable X in the regression model
© 2002 Prentice-Hall, Inc.
Chap 11-26
Venn Diagrams and
Explanatory Power of Regression
r 
2
Sales
Sizes
© 2002 Prentice-Hall, Inc.
SSR

SSR  SSE
Chap 11-27
Coefficients of Determination (r 2)
and Correlation (r)
Y r2 = 1, r = +1
Y r2 = 1, r = -1
^=b +b X
Y
i
^=b +b X
Y
i
0
1 i
0
X
Yr2 = .8, r = +0.9
X
© 2002 Prentice-Hall, Inc.
X
Y
^=b +b X
Y
i
0
1 i
1 i
r2 = 0, r = 0
^=b +b X
Y
i
0
1 i
X
Chap 11-28
Standard Error of Estimate

n


SYX
SSE


n2
i 1
Y  Yˆi

2
n2
The standard deviation of the variation of
observations around the regression line
© 2002 Prentice-Hall, Inc.
Chap 11-29
Measures of Variation:
Produce Store Example
Excel Output for Produce Stores
R e g r e ssi o n S ta ti sti c s
M u lt ip le R
0 .9 7 0 5 5 7 2
R S q u a re
0 .9 4 1 9 8 1 2 9
A d ju s t e d R S q u a re
0 .9 3 0 3 7 7 5 4
S t a n d a rd E rro r
6 1 1 .7 5 1 5 1 7
O b s e r va t i o n s
7
Syx
r2 = .94
94% of the variation in annual sales can be
explained by the variability in the size of the
store as measured by square footage
© 2002 Prentice-Hall, Inc.
Chap 11-30
Linear Regression Assumptions

Normality




Y values are normally distributed for each X
Probability distribution of error is normal
2. Homoscedasticity (Constant Variance)
3. Independence of Errors
© 2002 Prentice-Hall, Inc.
Chap 11-31
Variation of Errors around
the Regression Line
f(e)
• Y values are normally distributed
around the regression line.
• For each X value, the “spread” or
variance around the regression line is
the same.
Y
X2
X1
X
© 2002 Prentice-Hall, Inc.
Sample Regression Line
Chap 11-32
Residual Analysis

Purposes



Examine linearity
Evaluate violations of assumptions
Graphical Analysis of Residuals

Plot residuals vs. Xi , Yi and time
© 2002 Prentice-Hall, Inc.
Chap 11-33
Residual Analysis for Linearity
Y
Y
X
e
X
X
e
X
Not Linear
© 2002 Prentice-Hall, Inc.

Linear
Chap 11-34
Studentized Residual
X  X 
 X  X 
2

SRi 
SYX
ei
1  hi
where
1
hi  
n
i
n
i 1



2
i
Residual divided by its standard error
Standardized residual adjusted for the distance
from the average X value
Allow us to normalize the magnitude of the
residuals in units reflecting the variation around
the regression line
© 2002 Prentice-Hall, Inc.
Chap 11-35
Residual Analysis for
Homoscedasticity
Y
Y
X
SR
X
SR
X
Heteroscedasticity
© 2002 Prentice-Hall, Inc.
X
Homoscedasticity
Chap 11-36
Residual Analysis:Excel Output
for Produce Stores Example
Observation
1
2
3
4
5
6
7
Excel Output
Predicted Y
4202.344417
3928.803824
5822.775103
9894.664688
3557.14541
4918.90184
3588.364717
Residuals
-521.3444173
-533.8038245
830.2248971
-351.6646882
-239.1454103
644.0981603
171.6352829
Residual Plot
0
1000
© 2002 Prentice-Hall, Inc.
2000
3000
4000
Square Feet
5000
6000
Chap 11-37
Residual Analysis
for Independence

The Durbin-Watson Statistic


Used when data is collected over time to detect
autocorrelation (residuals in one time period are
related to residuals in another period)
Measures violation of independence assumption
n
D
2
(
e

e
)
 i i 1
i2
n
e
i 1
© 2002 Prentice-Hall, Inc.
2
i
Should be close to 2.
If not, examine the model
for autocorrelation.
Chap 11-38
Durbin-Watson Statistic
in PHStat

PHStat | regression | simple linear regression
…

Check the box for Durbin-Watson Statistic
© 2002 Prentice-Hall, Inc.
Chap 11-39
Obtaining the Critical Values of
Durbin-Watson Statistic
Table 13.4 Finding critical values of Durbin-Watson Statistic
 5
p=1
p=2
n
dL
dU
dL
dU
15
1.08
1.36
.95
1.54
16
1.10
1.37
.98
1.54
© 2002 Prentice-Hall, Inc.
Chap 11-40
Using the
Durbin-Watson Statistic
H0 :
H1
No autocorrelation (error terms are independent)
: There is autocorrelation (error terms are not
independent)
Reject H0
(positive
autocorrelation)
0
© 2002 Prentice-Hall, Inc.
dL
Inconclusive
Accept H0
(no autocorrelatin)
dU
2
4-dU
Reject H0
(negative
autocorrelation)
4-dL
4
Chap 11-41
Residual Analysis
for Independence
Graphical Approach

Not Independent
e
Independent
e
Time
Cyclical Pattern
Time
No Particular Pattern
Residual is plotted against time to detect any autocorrelation
© 2002 Prentice-Hall, Inc.
Chap 11-42
Inference about the Slope:
t Test

t test for a population slope


Null and alternative hypotheses



Is there a linear dependency of Y on X ?
H0: 1 = 0
H1: 1  0
(no linear dependency)
(linear dependency)
Test statistic


b1  1
t
where Sb1 
Sb1
d. f .  n  2
© 2002 Prentice-Hall, Inc.
SYX
n
(X
i 1
i
 X)
2
Chap 11-43
Example: Produce Store
Data for Seven Stores:
Store
Square
Feet
Annual
Sales
($000)
1,726
1,542
2,816
5,555
1,292
2,208
1,313
3,681
3,395
6,653
9,543
3,318
5,563
3,760
1
2
3
4
5
6
7
© 2002 Prentice-Hall, Inc.
Estimated
Regression
Equation:

Yi = 1636.415 +1.487Xi
The slope of this
model is 1.487.
Is square footage of
the store affecting its
annual sales?
Chap 11-44
Inferences about the Slope:
t Test Example
Test Statistic:
H0: 1 = 0
From Excel Printout b
Sb1 t
H1: 1  0
1
Coefficients Standard Error t Stat P-value
  .05
Intercept
1636.4147
451.4953 3.6244 0.01515
df  7 - 2 = 5
Footage
1.4866
0.1650 9.0099 0.00028
Critical Value(s):
Reject
.025
Reject
.025
-2.5706 0 2.5706
© 2002 Prentice-Hall, Inc.
Decision:
Reject H0
t
Conclusion:
There is evidence that
square footage affects
annual sales.
Chap 11-45
Inferences about the Slope:
Confidence Interval Example
Confidence Interval Estimate of the Slope:
b1  tn2 Sb1
Excel Printout for Produce Stores
L o w er 95%
I n te r c e p t
U p p er 95%
475.810926
2797.01853
X V a r i a b l e 11 . 0 6 2 4 9 0 3 7
1.91077694
At 95% level of confidence, the confidence interval
for the slope is (1.062, 1.911). Does not include 0.
Conclusion: There is a significant linear dependency
of annual sales on the size of the store.
© 2002 Prentice-Hall, Inc.
Chap 11-46
Inferences about the Slope:
F Test

F Test for a population slope


Null and alternative hypotheses



Is there a linear dependency of Y on X ?
H0: 1 = 0
H1: 1  0
(No Linear Dependency)
(Linear Dependency)
Test statistic


SSR
1
F 
SSE
 n  2
Numerator d.f.=1, denominator d.f.=n-2
© 2002 Prentice-Hall, Inc.
Chap 11-47
Relationship between
a t Test and an F Test

Null and alternative hypotheses


H0: 1 = 0
H1: 1  0
t 
n2
© 2002 Prentice-Hall, Inc.
2
(No linear dependency)
(Linear dependency)
 F1,n 2
Chap 11-48
Inferences about the Slope:
F Test Example
H0: 1 = 0
H1: 1  0
  .05
numerator
df = 1
denominator
df  7 - 2 = 5
Test Statistic:
From Excel Printout
ANOVA
df
Regression
Residual
Total
1
5
6
Reject
 = .05
0
© 2002 Prentice-Hall, Inc.
6.61
F1,n2
SS
MS
F Significance F
30380456.12 30380456.12 81.179
0.000281
1871199.595 374239.919
32251655.71
Decision: Reject H0
Conclusion:
There is evidence that
square footage affects
annual sales.
Chap 11-49
Purpose of Correlation Analysis

Correlation analysis is used to measure
strength of association (linear relationship)
between two numerical variables


Only concerned with strength of the relationship
No causal effect is implied
© 2002 Prentice-Hall, Inc.
Chap 11-50
Purpose of Correlation Analysis
(continued)


Population correlation coefficient  (Rho) is
used to measure the strength between the
variables
Sample correlation coefficient r is an estimate
of  and is used to measure the strength of
the linear relationship in the sample
observations
© 2002 Prentice-Hall, Inc.
Chap 11-51
Sample of Observations from
Various r Values
Y
Y
Y
X
r = -1
X
r = -.6
Y
© 2002 Prentice-Hall, Inc.
X
r=0
Y
r = .6
X
r=1
X
Chap 11-52
Features of  and r





Unit free
Range between -1 and 1
The closer to -1, the stronger the negative
linear relationship
The closer to 1, the stronger the positive
linear relationship
The closer to 0, the weaker the linear
relationship
© 2002 Prentice-Hall, Inc.
Chap 11-53
Test for a Linear Relationship

Hypotheses



H0:  = 0 (no correlation)
H1:   0 (correlation)
Test statistic
t

r
where
 r
n2
2
n
r  r2 
© 2002 Prentice-Hall, Inc.
 X
i 1
n
 X
i 1
i
i
 X Yi  Y 
X
2
n
 Y  Y 
i 1
2
i
Chap 11-54
Example: Produce Stores
From Excel Printout
Is there any
evidence of a linear
relationship between
the annual sales of a
store and its square
footage at .05 level
of significance?
© 2002 Prentice-Hall, Inc.
r
R e g r e ssi o n S ta ti sti c s
M u lt ip le R
R S q u a re
0 .9 7 0 5 5 7 2
0 .9 4 1 9 8 1 2 9
A d ju s t e d R S q u a re 0 . 9 3 0 3 7 7 5 4
S t a n d a rd E rro r
O b s e rva t io n s
6 1 1 .7 5 1 5 1 7
7
H0:  = 0 (No association)
H1:   0 (Association)
  .05
df  7 - 2 = 5
Chap 11-55
Example:
Produce Stores Solution
r
.9706
t

 9.0099
2
1  .9420
 r
5
n2
Critical Value(s):
Reject
.025
Reject
.025
-2.5706 0 2.5706
© 2002 Prentice-Hall, Inc.
Decision:
Reject H0
Conclusion:
There is evidence of a
linear relationship at 5%
level of significance
The value of the t statistic is
exactly the same as the t
statistic value for test on the
slope coefficient
Chap 11-56
Estimation of Mean Values
Confidence interval estimate for
Y | X  X :
i
The mean of Y given a particular Xi
Standard error
of the estimate
Size of interval varies according
to distance away from mean, X
(Xi  X )
1
ˆ
Yi  tn2 SYX
 n
n
2
(Xi  X )
t value from table
with df=n-2
© 2002 Prentice-Hall, Inc.
2
i 1
Chap 11-57
Prediction of Individual Values
Prediction interval for individual response
Yi at a particular Xi
Addition of one increases width of interval
from that for the mean of Y
1 ( Xi  X )
ˆ
Yi  tn2 SYX 1   n
n
2
( Xi  X )
2
i 1
© 2002 Prentice-Hall, Inc.
Chap 11-58
Interval Estimates
for Different Values of X
Y
Confidence
Interval for the
mean of Y
Prediction Interval
for a individual Yi
X
© 2002 Prentice-Hall, Inc.
X
A given X
Chap 11-59
Example: Produce Stores
Data for seven stores:
Store
Square
Feet
Annual
Sales
($000)
1
2
3
4
5
6
7
1,726
1,542
2,816
5,555
1,292
2,208
1,313
3,681
3,395
6,653
9,543
3,318
5,563
3,760
© 2002 Prentice-Hall, Inc.
Predict the annual
sales for a store
with 2000 square
feet.
Regression Model Obtained:

Yi = 1636.415 +1.487Xi
Chap 11-60
Estimation of Mean Values:
Example
Confidence Interval Estimate for
Y | X  X
i
Find the 95% confidence interval for the average annual
sales for a 2,000 square-foot store.

Predicted Sales Yi = 1636.415 +1.487Xi = 4610.45 ($000)
X = 2350.29
SYX = 611.75
1
ˆ
Yi  tn2 SYX

n
tn-2 = t5 = 2.5706
( X i  X )2
n
2
(
X

X
)
 i
 4610.45  612.66
i 1
© 2002 Prentice-Hall, Inc.
Chap 11-61
Prediction Interval for Y :
Example
Prediction Interval for Individual Y
Find the 95% prediction interval
for the annual sales of a 2,000 square-foot store

Predicted Sales Yi = 1636.415 +1.487Xi = 4610.45 ($000)
X = 2350.29
Yˆi  tn2 SYX
SYX = 611.75
tn-2 = t5 = 2.5706
1 ( X i  X )2
1  n
 4610.45  1687.68
n
2
( Xi  X )
i 1
© 2002 Prentice-Hall, Inc.
Chap 11-62
Estimation of Mean Values and
Prediction of Individual Values in PHStat

In excel, use PHStat | regression | simple
linear regression …


Check the “confidence and prediction interval for
X=” box
EXCEL spreadsheet of regression sales on
footage
© 2002 Prentice-Hall, Inc.
Chap 11-63
Pitfalls of Regression Analysis




Lacking an awareness of the assumptions
underlying least-squares regression
Not knowing how to evaluate the assumptions
Not knowing the alternatives to least-squares
regression if a particular assumption is violated
Using a regression model without knowledge of
the subject matter
© 2002 Prentice-Hall, Inc.
Chap 11-64
Strategies for Avoiding
the Pitfalls of Regression



Start with a scatter plot of X on Y to
observe possible relationship
Perform residual analysis to check the
assumptions
Use a histogram, stem-and-leaf display,
box-and-whisker plot, or normal probability
plot of the residuals to uncover possible
non-normality
© 2002 Prentice-Hall, Inc.
Chap 11-65
Strategies for Avoiding
the Pitfalls of Regression
(continued)


If there is violation of any assumption, use
alternative methods (e.g.: least absolute
deviation regression or least median of squares
regression) to least-squares regression or
alternative least-squares models (e.g.:
Curvilinear or multiple regression)
If there is no evidence of assumption violation,
then test for the significance of the regression
coefficients and construct confidence intervals
and prediction intervals
© 2002 Prentice-Hall, Inc.
Chap 11-66
Chapter Summary






Introduced types of regression models
Discussed determining the simple linear
regression equation
Described measures of variation
Addressed assumptions of regression and
correlation
Discussed residual analysis
Addressed measuring autocorrelation
© 2002 Prentice-Hall, Inc.
Chap 11-67
Chapter Summary




(continued)
Described inference about the slope
Discussed correlation -- measuring the
strength of the association
Addressed estimation of mean values and
prediction of individual values
Discussed possible pitfalls in regression and
recommended a strategy to avoid them
© 2002 Prentice-Hall, Inc.
Chap 11-68