1/27/2018
12
Multiple Linear Regression
CHAPTER OUTLINE
12-1 Multiple Linear Regression Model
12-3 Confidence Intervals in Multiple Linear
12-1.1 Introduction
Regression
12-1.2 Least squares estimation of the
12-4.1 Use of t-tests
parameters
12-3.2 Confidence interval on the mean response
12-1.3 Matrix approach to multiple linear
12-4 Prediction of New Observations
regression
12-5 Model Adequacy Checking
12-1.4 Properties of the least squares estimators 12-5.1 Residual analysis
12-2 Hypothesis Tests in Multiple Linear
12-5.2 Influential observations
Regression
12-2.1 Test for significance of regression
12-2.2 Tests on individual regression coefficients
& subsets of coefficients
12-6 Aspects of Multiple Regression
Modeling
12-6.1 Polynomial regression models
12-6.2 Categorical regressors & indicator
variables
12-6.3 Selection of variables & model building
12-6.4 Multicollinearity
1
Learning Objectives for Chapter 12
After careful study of this chapter, you should be able to do the
following:
1.
2.
3.
4.
5.
6.
7.
8.
Use multiple regression techniques to build empirical models to
engineering and scientific data.
Understand how the method of least squares extends to fitting multiple
regression models.
Assess regression model adequacy.
Test hypotheses and construct confidence intervals on the regression
coefficients.
Use the regression model to estimate the mean response, and to make
predictions and to construct confidence intervals and prediction
intervals.
Build regression models with polynomial terms.
Use indicator variables to model categorical regressors.
Use stepwise regression and other model building techniques to select
the appropriate set of variables for a regression model.
2
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
1
1/27/2018
12-1: Multiple Linear Regression Models
12-1.1 Introduction
• Many
applications of regression analysis involve
situations in which there are more than one
regressor variable.
• A regression model that contains more than one
regressor variable is called a multiple regression
model.
3
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-1: Multiple Linear Regression Models
12-1.1 Introduction
• For example, suppose that the effective life of a cutting
tool depends on the cutting speed and the tool angle. A
possible multiple regression model could be
where
Y – tool life
x1 – cutting speed
x2 – tool angle
4
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
2
1/27/2018
12-1: Multiple Linear Regression Models
12-1.1 Introduction
Figure 12-1 (a) The regression plane for the model E(Y) = 50 + 10x1 + 7x2. (b) The
contour plot
5
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-1: Multiple Linear Regression Models
12-1.1 Introduction
6
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
3
1/27/2018
12-1: Multiple Linear Regression Models
12-1.1 Introduction
Figure 12-2 (a) Threedimensional plot of the
regression model E(Y) = 50 +
10x1 + 7x2 + 5x1x2. (b) The
contour plot
7
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-1: Multiple Linear Regression Models
12-1.1 Introduction
Figure 12-3 (a) Threedimensional plot of the
regression model E(Y) = 800 +
10x1 + 7x2 – 8.5x12 – 5x22 +
4x1x2. (b) The contour plot
8
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
4
1/27/2018
12-1: Multiple Linear Regression Models
12-1.2 Least Squares Estimation of the Parameters
9
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-1: Multiple Linear Regression Models
12-1.2 Least Squares Estimation of the Parameters
• The least squares function is given by
• The least squares estimates must satisfy
10
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
5
1/27/2018
12-1: Multiple Linear Regression Models
12-1.2 Least Squares Estimation of the Parameters
• The least squares normal Equations are
• The solution to the normal Equations are the least
squares estimators of the regression coefficients.
11
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-1: Multiple Linear Regression Models
Example 12-1
12
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
6
1/27/2018
12-1: Multiple Linear Regression Models
Example 12-1
13
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-1: Multiple Linear Regression Models
Figure 12-4 Matrix of scatter plots (from Minitab) for the wire bond pull strength
data in Table 12-2.
14
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
7
1/27/2018
12-1: Multiple Linear Regression Models
Example 12-1
15
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-1: Multiple Linear Regression Models
Example 12-1
16
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
8
1/27/2018
12-1: Multiple Linear Regression Models
Example 12-1
17
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-1: Multiple Linear Regression Models
12-1.3 Matrix Approach to Multiple Linear Regression
Suppose the model relating the regressors to the
response is
In matrix notation this model can be written as
18
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
9
1/27/2018
12-1: Multiple Linear Regression Models
12-1.3 Matrix Approach to Multiple Linear Regression
where
19
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-1: Multiple Linear Regression Models
12-1.3 Matrix Approach to Multiple Linear Regression
We wish to find the vector of least squares
estimators that minimizes:
The resulting least squares estimate is
20
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
10
1/27/2018
12-1: Multiple Linear Regression Models
12-1.3 Matrix Approach to Multiple Linear Regression
21
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-1: Multiple Linear Regression Models
Example 12-2
22
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
11
1/27/2018
Example 12-2
23
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-1: Multiple Linear Regression Models
Example 12-2
24
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12
1/27/2018
12-1: Multiple Linear Regression Models
Example 12-2
25
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-1: Multiple Linear Regression Models
Example 12-2
26
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
13
1/27/2018
12-1: Multiple Linear Regression Models
Example 12-2
27
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
28
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
14
1/27/2018
12-1: Multiple Linear Regression Models
Estimating 2
An unbiased estimator of 2 is
29
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-1: Multiple Linear Regression Models
12-1.4 Properties of the Least Squares Estimators
Unbiased estimators:
Covariance Matrix:
30
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
15
1/27/2018
12-1: Multiple Linear Regression Models
12-1.4 Properties of the Least Squares Estimators
Individual variances and covariances:
In general,
31
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-2: Hypothesis Tests in Multiple Linear Regression
12-2.1 Test for Significance of Regression
The appropriate hypotheses are
The test statistic is
32
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
16
1/27/2018
12-2: Hypothesis Tests in Multiple Linear Regression
12-2.1 Test for Significance of Regression
33
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-2: Hypothesis Tests in Multiple Linear Regression
Example 12-3
34
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
17
1/27/2018
12-2: Hypothesis Tests in Multiple Linear Regression
Example 12-3
35
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-2: Hypothesis Tests in Multiple Linear Regression
Example 12-3
36
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
18
1/27/2018
12-2: Hypothesis Tests in Multiple Linear Regression
Example 12-3
37
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-2: Hypothesis Tests in Multiple Linear Regression
R2 and Adjusted R2
The coefficient of multiple determination
• For the wire bond pull strength data, we find that R2 =
SSR/SST = 5990.7712/6105.9447 = 0.9811.
• Thus, the model accounts for about 98% of the
variability in the pull strength response.
38
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
19
1/27/2018
12-2: Hypothesis Tests in Multiple Linear Regression
R2 and Adjusted R2
The adjusted R2 is
• The adjusted R2 statistic penalizes the analyst for
adding terms to the model.
• It can help guard against overfitting (including
regressors that are not really useful)
39
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-2: Hypothesis Tests in Multiple Linear Regression
12-2.2 Tests on Individual Regression Coefficients and
Subsets of Coefficients
The hypotheses for testing the significance of any
individual regression coefficient:
40
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
20
1/27/2018
12-2: Hypothesis Tests in Multiple Linear Regression
12-2.2 Tests on Individual Regression Coefficients and
Subsets of Coefficients
The test statistic is
• Reject H0 if |t0| > t/2,n-p.
• This is called a partial or marginal test
41
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-2: Hypothesis Tests in Multiple Linear Regression
Example 12-4
42
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
21
1/27/2018
12-2: Hypothesis Tests in Multiple Linear Regression
Example 12-4
43
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-2: Hypothesis Tests in Multiple Linear Regression
The general regression significance test or the extra
sum of squares method:
We wish to test the hypotheses:
44
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
22
1/27/2018
12-2: Hypothesis Tests in Multiple Linear Regression
A general form of the model can be written:
where X1 represents the columns of X associated with
1 and X2 represents the columns of X associated with
2
45
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-2: Hypothesis Tests in Multiple Linear Regression
For the full model:
If H0 is true, the reduced model is
46
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
23
1/27/2018
12-2: Hypothesis Tests in Multiple Linear Regression
The test statistic is:
Reject H0 if f0 > f,r,n-p
The test in Equation (12-32) is often referred to as a
partial F-test
47
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-2: Hypothesis Tests in Multiple Linear Regression
Example 12-6
48
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
24
1/27/2018
12-2: Hypothesis Tests in Multiple Linear Regression
Example 12-6
49
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-2: Hypothesis Tests in Multiple Linear Regression
Example 12-6
50
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
25
1/27/2018
12-3: Confidence Intervals in Multiple Linear Regression
12-3.1 Confidence Intervals on Individual
Regression Coefficients
Definition
51
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-3: Confidence Intervals in Multiple Linear Regression
Example 12-7
52
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
26
1/27/2018
12-3: Confidence Intervals in Multiple Linear Regression
12-3.2 Confidence Interval on the Mean Response
The mean response at a point x0 is estimated by
The variance of the estimated mean response is
53
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-3: Confidence Intervals in Multiple Linear Regression
12-3.2 Confidence Interval on the Mean Response
Definition
54
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
27
1/27/2018
12-3: Confidence Intervals in Multiple Linear Regression
Example 12-8
55
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-3: Confidence Intervals in Multiple Linear Regression
Example 12-8
56
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
28
1/27/2018
12-4: Prediction of New Observations
A point estimate of the future observation Y0 is
A 100(1-)% prediction interval for this future
observation is
57
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-4: Prediction of New Observations
Figure 12-5 An example of
extrapolation in multiple regression
58
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
29
1/27/2018
12-4: Prediction of New Observations
Example 12-9
59
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-5: Model Adequacy Checking
12-5.1 Residual Analysis
Example 12-10
Figure 12-6 Normal probability plot of
residuals
60
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
30
1/27/2018
12-5: Model Adequacy Checking
12-5.1 Residual Analysis
Example 12-10
61
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-5: Model Adequacy Checking
12-5.1 Residual Analysis
Example 12-10
Figure 12-7 Plot of residuals
62
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
31
1/27/2018
12-5: Model Adequacy Checking
12-5.1 Residual Analysis
Example 12-10
Figure 12-8 Plot of residuals against x1.
63
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-5: Model Adequacy Checking
12-5.1 Residual Analysis
Example 12-10
Figure 12-9 Plot of residuals against x2.
64
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
32
1/27/2018
12-5: Model Adequacy Checking
12-5.1 Residual Analysis
65
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-5: Model Adequacy Checking
12-5.1 Residual Analysis
The variance of the ith residual is
66
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
33
1/27/2018
12-5: Model Adequacy Checking
12-5.1 Residual Analysis
67
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-5: Model Adequacy Checking
12-5.2 Influential Observations
Figure 12-10 A point that is
remote in x-space.
68
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
34
1/27/2018
12-5: Model Adequacy Checking
12-5.2 Influential Observations
Cook’s distance measure
69
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-5: Model Adequacy Checking
Example 12-11
70
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
35
1/27/2018
12-5: Model Adequacy Checking
Example 12-11
71
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-6: Aspects of Multiple Regression Modeling
12-6.1 Polynomial Regression Models
72
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
36
1/27/2018
12-6: Aspects of Multiple Regression Modeling
Example 12-12
73
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-6: Aspects of Multiple Regression Modeling
Example 12-11
Figure 12-11 Data for Example
12-11.
74
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
37
1/27/2018
Example 12-12
75
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-6: Aspects of Multiple Regression Modeling
Example 12-12
76
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
38
1/27/2018
12-6: Aspects of Multiple Regression Modeling
12-6.2 Categorical Regressors and Indicator Variables
Many problems may involve qualitative or
categorical variables.
• The usual method for the different levels of a
qualitative variable is to use indicator variables.
• For example, to introduce the effect of two different
operators into a regression model, we could define an
indicator variable as follows:
•
77
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-6: Aspects of Multiple Regression Modeling
Example 12-13
78
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
39
1/27/2018
12-6: Aspects of Multiple Regression Modeling
Example 12-13
79
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-6: Aspects of Multiple Regression Modeling
Example 12-13
80
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
40
1/27/2018
Example 12-12
81
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-6: Aspects of Multiple Regression Modeling
Example 12-13
82
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
41
1/27/2018
12-6: Aspects of Multiple Regression Modeling
Example 12-13
83
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-6: Aspects of Multiple Regression Modeling
12-6.3 Selection of Variables and Model Building
84
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
42
1/27/2018
12-6: Aspects of Multiple Regression Modeling
12-6.3 Selection of Variables and Model Building
All Possible Regressions – Example 12-14
85
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-6: Aspects of Multiple Regression Modeling
12-6.3 Selection of Variables and Model Building
All Possible Regressions – Example 12-14
86
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
43
1/27/2018
12-6: Aspects of Multiple Regression Modeling
12-6.3 Selection of Variables and Model Building
All Possible Regressions – Example 12-14
Figure 12-12 A matrix of
Scatter plots from Minitab
for the Wine Quality Data.
87
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
88
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
44
1/27/2018
12-6.3: Selection of Variables and Model Building - Stepwise Regression
Example 12-14
89
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-6.3: Selection of Variables and Model Building - Backward Regression
Example 12-14
90
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
45
1/27/2018
12-6: Aspects of Multiple Regression Modeling
12-6.4 Multicollinearity
Variance Inflation Factor (VIF)
91
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-6: Aspects of Multiple Regression Modeling
12-6.4 Multicollinearity
The presence of multicollinearity can be detected in several
ways. Two of the more easily understood of these are:
92
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
46
1/27/2018
Important Terms & Concepts of Chapter 12
All possible regressions
Model parameters & their
interpretation in multiple regression
Analysis of variance test in multiple
regression
Multicollinearity
Categorical variables
Multiple regression
Confidence intervals on the mean
Outliers
response
Polynomial regression model
Cp statistic
Prediction interval on a future
Extra sum of squares method
observation
Hidden extrapolation
PRESS statistic
Indicator variables
Residual analysis & model adequacy
checking
Inference (test & intervals) on individual
model parameters
Significance of regression
Influential observations
Stepwise regression & related methods
Variance Inflation Factor (VIF)
93
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
47