Regression Basics (§11.1 – 11.3)
Regression Unit Outline
• What is Regression?
• How is a Simple Linear Regression Analysis done?
• Outline the analysis protocol.
• Work an example.
• Examine the details (a little theory).
• Related items.
• When is simple linear regression appropriate?
STA6166-RegBasics
1
Relationships
What is Regression?
In science, we frequently measure two or more variables on the same
individual (case, object, etc). We do this to explore the nature of the
relationship among these variables. There are two basic types of
relationships.
• Cause-and-effect relationships.
• Functional relationships.
Function: a mathematical relationship enabling us to predict what
values of one variable (Y) correspond to given values of another
variable (X).
• Y: is referred to as the dependent variable, the response
variable or the predicted variable.
• X: is referred to as the independent variable, the explanatory
variable or the predictor variable.
STA6166-RegBasics
2
Examples
•
•
•
•
•
•
The time needed to fill a soft
drink vending machine
The tensile strength of wrapping
paper
Percent germination of begonia
seeds
The mean litter weight of test
rats
Maintenance cost of tractors
The repair time for a computer
•
•
•
•
•
•
The number of cases needed to
fill the machine
The percent of hardwood in the
pulp batch
The intensity of light in an
incubator
The litter size
The age of the tractor
The number of components
which have to be changed
In each case, the statement can be read as; Y is a function of X.
Two kinds of explanatory variables:
Those we can control
Those over which we have little or no control.
STA6166-RegBasics
3
An operations supervisor measured how long it takes one of her drivers to put 1,
2, 3 and 4 cases of soft drink into a soft drink machine. In this case the levels of
the explanatory variable, X are {1,2,3,4}, and she controls them. She might repeat
the measurement a couple of times at each level of X. A scatter plot of the
resulting data might look like:
STA6166-RegBasics
4
A forestry graduate student makes wrapping paper out of different
percentages of hardwood then measure its tensile strength. He has
the freedom to choose at the beginning of the study to have only five
percentages to work with, say {5%, 10%, 15%, 20%, and 25%}. A
scatter plot of the resulting data might look like:
STA6166-RegBasics
5
A farm manager is interested in the relationship between litter size and
average litter weight (average newborn piglet weight). She examines
the farm records over the last couple of years and records the litter
size and average weight for all births. A plot of the data pairs looks
like the following:
STA6166-RegBasics
6
A farm operations student is interested in the relationship between
maintenance cost and age of farm tractors. He performs a telephone
interview survey of the 52 commercial potato growers in Putnam County,
FL. One part of the questionnaire provides information on tractor age
and 1995 maintenance cost (fuel, lubricants, repairs, etc). A plot of these
data might look like:
STA6166-RegBasics
7
Questions needing answers.
• What is the association between Y and X?
• How can changes in Y be explained by changes in X?
• What are the functional relationships between Y and X?
A functional relationship is symbolically written as:
Eq: 1
Y  f(X)
Example: A proportional
relationship (e.g. fish weight to
length).
Y  b1 X
b1 is the slope of the line.
STA6166-RegBasics
8
Example: Linear relationship (e.g. Y=cholesterol
versus X=age)
Y  b0  b1X
b0 is the intercept,
b1 is the slope.
STA6166-RegBasics
9
Example: Polynomial relationship
(e.g. Y=crop yield vs. X=pH)
Y  b0  b1X  b2 X
2
b0: intercept,
b1: linear coefficient,
b2: quadratic coefficient.
STA6166-RegBasics
10
Nonlinear relationship:
2
Y = b0 sin(b1X + b2X )
STA6166-RegBasics
11
Concerns:
• The proposed functional relationship will not fit
exactly, i.e. something is either wrong with the
data (errors in measurement), or the model is
inadequate (errors in specification).
• The relationship is not truly known until we
assign values to the parameters of the model.
The possibility of errors into the proposed relationship is
acknowledged in the functional symbolism as follows:
Eq: 2
Y  f (X )  
 is a random variable representing the result of both errors in
model specification and measurement. As in AOV, the variance
of  is the background variability with respect to which we will
assess the significance of the factors (explanatory variables).
STA6166-RegBasics
12
The error term:
Eq: 3
Another way to emphasize
 Y  f(X)
or, emphasizing that f(X) depends on unknown parameters.
Eq: 4
Y  f ( X | 0 , 1)  
What if we don’t know the functional form of the relationship?
• Look at a scatter plot of the data for suggestions.
• Hypothesize about the nature of the underlying
process. Often the hypothesized processes will
suggest a functional form.
STA6166-RegBasics
13
The straight line -- a conservative
starting point.
Regression Analysis: the process of fitting a line to data.
Sir Francis Galton (1822-1911) -- a British
anthropologist and meteorologist coined the
term “regression”.
Regression towards mediocrity in hereditary stature - the tendency
of offspring to be smaller than large parents and larger than small
parents. Referred to as “regression towards the mean”.
Expected
offspring
height
2
ˆ
Y  Y  (X  X)
3
Average sized offspring
Adjustment for how
far parent is from
mean of parents
STA6166-RegBasics
14
Regression to the Mean: Galton’s Height Data
mean parent height
mean parent height
45 degree line
regression line
mean child height
Data: 952 parent-child
pairs of heights. Parent
height is average of the
two parents. Women’s
heights have been
adjusted to make them
comparable to men’s.
STA6166-RegBasics
15
Regression to the Mean is a Powerful Effect!
Same data, but suppose
response is now blood
pressure (bp) before &
after (day 1, day 2).
If we track only those
with elevated bp before
(above 3rd quartile) , we
see an amazing
improvement, even
though no treatment took
place!
This is the regression
effect at work. If it is not
recognized and taken into
account, misleading
results and biases can
occur.
STA6166-RegBasics
16
How is a Simple Linear Regression
Analysis done? A Protocol
no
Assumptions
OK?
yes
STA6166-RegBasics
17
Steps in a Regression Analysis
1. Examine the scatterplot of the data.
• Does the relationship look linear?
• Are there points in locations they shouldn’t be?
• Do we need a transformation?
2. Assuming a linear function looks appropriate, estimate the regression
parameters.
• How do we do this? (Method of Least Squares)
3. Test whether there really is a statistically significant linear
relationship. Just because we assumed a linear function it does
not follow that the data support this assumption.
• How do we test this? (F-test for Variances)
4. If there is a significant linear relationship, estimate the response, Y,
for the given values of X, and compute the residuals.
5. Examine the residuals for systematic inadequacies in the linear model
as fit to the data.
• Is there evidence that a more complicated relationship (say a
polynomial) should be considered; are there problems with the
regression assumptions? (Residual analysis).
• Are there specific data points which do not seem to follow the
proposed relationship? (Examined using influence measures).
STA6166-RegBasics
18
Simple Linear Regression - Example and Theory
SITUATION: A company that repairs
small computers needs to develop a
better way of providing customers typical
repair cost estimates. To begin this
process, they compiled data on repair
times (in minutes) and the number of
components needing repair or
replacement from the previous week.
The data, sorted by number of
components are as follows:
Paired Observations (xi, yi)
Number
Repair
of components
time
i
xi
yi
1
1
23
2
2
29
3
4
64
4
4
72
5
4
80
6
5
87
7
6
96
8
6
105
9
8
127
10
8
119
11
9
145
12
9
149
13
10
165
14
10
154
STA6166-RegBasics
19
Assumed Linear
Regression Model
 i  yi  yˆ i
for i  1, 2,..., n
Computer repair times
Estimating the
regression parameters
180
160
140
120
Y
Objective: Minimize the
difference between the
observation and its
prediction according to
the line.
yi   0  1 xi   i
100
80
60
40
20
0
2
 yi  ( ˆ0  ˆ1 xi )
yˆi  predictedy valuewhen x  xi
4
6
8
10
X
STA6166-RegBasics
20
We want the line which is best for all points. This is done by
finding the values of 0 and 1 which minimizes some sum of
errors. There are a number of ways of doing this. Consider these
two:
n
min   i
 0 , 1
i 1
n
min   i
 0 , 1
i 1
2
Sum of squared
residuals
The method of least squares produces estimates with statistical
properties (e.g. sampling distributions) which are easier to
determine.
Regression => least squares estimation
ˆ0 ˆ1
Referred to as least squares estimates.
STA6166-RegBasics
21
Normal Equations
Calculus is used to find the least squares estimates.
n
n
E (  0 , 1 )    i   ( yi   0  1 xi ) 2
i 1
2
i 1
E
0
 0
E
0
1
Note:
Solve this system of two equations in two unknowns.
The parameter estimates will be functions of the data,
hence they will be statistics.
STA6166-RegBasics
22
Let:

S xx
Sums of Squares
n
 (x  x)
2
i
i 1
 ( x1  x )  ( x2  x )    ( xn  x )
2

Syy

2
1

2
(
x
)

x



i
i
n  i 1 
i 1
n
n
2
n
2
(
y

y
)
 i
i 1
 ( y1  y )  ( y 2  y )    ( y n  y )
2
2
1 n 
2
  ( y i )    y i 
n  i 1 
i 1
n
Sxy
Sums of
squares of
x.
2

n
(x
i 1
i
Sums of
squares of
y.
2
2
 x )(y i  y )
 ( x1  x )(y 1  y )    ( xn  x )(y i  y )
n
1  n  n 
  ( x i y i )    x i   y i 
n  i 1  i 1 
i 1
Sums of
cross
products
of x and y.
STA6166-RegBasics
23
Parameter
estimates:
S XY
ˆ
1 
S XX
ˆ0  y  ˆ1 x
Easy to compute with a spreadsheet program.
Easier to do with a statistical analysis package.
Example:
ˆ1  7.71
ˆ0  15.20
ˆi  15.20  7.71xi
y
Prediction
STA6166-RegBasics
24
Testing for a Statistically Significant
Regression
Ho: There is no relationship between Y and X.
HA: There is a relationship between Y and X.
Which of two competing models is more appropriate?
LinearModel: Y   0  1 X  
Mean Model:
Y   
We look at the sums of squares of the prediction
errors for the two models and decide if that for the
linear model is significantly smaller than that for
the mean model.
STA6166-RegBasics
25
Sums of Squares About the Mean (TSS)
Sum of squares about the mean: sum of the
prediction errors for the null (mean model)
hypothesis.
n
TSS  S yy   ( yi  y ) 2
i 1
TSS is actually a measure of the variance of the responses.
STA6166-RegBasics
26
Residual Sums of Squares
Sum of squares for error: sum of the prediction errors
for the alternative (linear regression model) hypothesis.
n
n
2
ˆ
ˆ
SSE   ( yi  yˆ i )   ( yi   0  1 xi )
2
i 1
i 1
SSE measures the variance of the residuals, the part of
the response variation that is not explained by the model.
STA6166-RegBasics
27
Regression Sums of Squares
Sum of squares due to the regression: difference
between TSS and SSE, i.e. SSR = TSS – SSE.
n
n
SSR   ( yi  yi )   ( yi  yˆ i )
2
i 1
2
i 1
n
2
ˆ
  ( y  yi )
i 1
SSR measures how much variability in the response is
explained by the regression.
STA6166-RegBasics
28
Graphical View
Linear Model
Mean Model
yˆi  ˆ0  ˆ1xi
TSS = SSR + SSE
Total
variability
in y-values
=
Variability
accounted
for by the
regression
+
Unexplained
variability
STA6166-RegBasics
29
TSS = SSR + SSE
Total
variability
in y-values
=
Variability
accounted
for by the
regression
+
Unexplained
variability
regression model fits well
Then SSR approaches TSS and SSE gets small.
regression model adds little
Then SSR approaches 0 and SSE approaches TSS.
STA6166-RegBasics
30
Mean Square Terms
Mean Square Total
ˆ
Sample variance of
the response, y:
ˆ
2
T
1 n
( yi  y ) 2


n  1 i 1
T SS

n 1
 MST
n
2
R
  ( yˆ i  y )2
i 1
SSR
1
 MSR
Regression Mean Square:

ˆ 
2
Residual Mean Square
1 n
 ˆ 
( yi  yˆ i ) 2

n  2 i 1
SSE

n2
 MSE
2
STA6166-RegBasics
31
F Test for Significant Regression
Both MSE and MSR measure the same underlying variance
quantity under the assumption that the null (mean) model holds.
 
2
2
R

Under the alternative hypothesis, the MSR should be much
greater than the MSE.
2
2
R

 
Placing this in the context of a test of variance.
 R2 MSR
F 2 
  MSE
Test Statistic
F should be near 1 if the regression is not significant, i.e. H0:
mean model holds.
STA6166-RegBasics
32
Formal test of the significance of the
regression.
H0:
HA:
No significant regression fit.
The regression explains a significant amount of
the variability in the response.
or
The slope of the regression line is significant.
or
X is a significant predictor of Y.
Test Statistic:
MSR
F 
MSE
Reject H0 if:
F  F1,n2,a
Where a is the probability of a type I error.
STA6166-RegBasics
33
Assumptions
1.
2.
1, 2, … n are independent of each other.
The i are normally distributed with mean
zero and have common variance  2.
How do we check these assumptions?
I.
II.
III.
Appropriate graphs.
Correlations (more later).
Formal goodness of fit tests.
STA6166-RegBasics
34
Analysis of Variance Table
We summarize the computations of this test in a table.
TSS
STA6166-RegBasics
35
Number
of components
i
xi
1
1
2
2
3
4
4
4
5
4
6
5
7
6
8
6
9
8
10
8
11
9
12
9
13
10
14
10
Repair
time
yi
23
29
64
72
80
87
96
105
127
119
145
149
165
154
STA6166-RegBasics
36
SAS output
ˆ   MSE
MSE
STA6166-RegBasics
37
Parameter Standard Error Estimates
Under the assumptions for regression inference, the least
squares estimates themselves are random variables.
1.
2.
1, 2, … n are independent of each other.
The i are normally distributed with mean zero and
have common variance 2.
Using some more calculus and mathematical statistics we can
determine the distributions for these parameters.
2


x

i
2

ˆ0  N   0 , 


nS
XX


2


ˆ1  N  1 ,
 S XX



STA6166-RegBasics
38
Testing regression parameters
important
The estimate of 2 is the mean square error: MSE
Test H0: 1=0:
Reject H0 if:
t 1 
ˆ1  0
MSE
2
ˆ
  MSE
S XX
t 1  t n  2 ,a / 2
(1-a)100% CI for 1:
ˆ1  t n  2,a / 2
MSE
S XX
STA6166-RegBasics
39
ˆ0
P-values
ˆ1
MSE
S XX
t 1 
ˆ1  0
MSE
S XX
STA6166-RegBasics
40
Regression
in Minitab
STA6166-RegBasics
41
Specifying
Model and
Output
Options
STA6166-RegBasics
42
STA6166-RegBasics
43
> y_c(23,29,64,72,80,87,96,105,127,119,145,149,165,154)
> x_c(1,2,4,4,4,5,6,6,8,8,9,9,10,10)
> myfit <- lm(y ~ x)
> summary(myfit)
Residuals:
Min
1Q
-10.2967 -4.1029
Median
0.2980
3Q
Max
4.2529 11.4962
Regression in R
Coefficients:
Estimate
Std. Error t value
Pr(>|t|)
(Intercept) 7.7110
4.1149
1.874
0.0855 .
x
15.1982
0.6086
24.972
1.03e-11 ***
--Signif. codes: 0 `***' 0.001 `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1
Residual standard error: 6.433 on 12 degrees of freedom
Multiple R-Squared: 0.9811, Adjusted R-squared: 0.9795
F-statistic: 623.6 on 1 and 12 DF, p-value: 1.030e-11
> anova(myfit)
Analysis of Variance Table
Response: y
Df Sum Sq Mean Sq F value Pr(>F)
x
1 25804.4 25804.4 623.62 1.030e-11 ***
Residuals 12
496.5
41.4
--Signif. codes: 0 `***' 0.001 `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1
STA6166-RegBasics
44
Residuals vs. Fitted Values
> par(mfrow=c(2,1))
> plot(myfit$fitted,myfit$resid)
> abline(0,0)
> qqnorm(myfit$resid)
STA6166-RegBasics
45