The Multiple Regression Model
Prepared by Vera Tabakova, East Carolina University






5.1 Model Specification and Data
5.2 Estimating the Parameters of the Multiple
Regression Model
5.3 Sampling Properties of the Least Squares
Estimator
5.4 Interval Estimation
5.5 Hypothesis Testing for a Single Coefficient
5.6 Measuring Goodness-of-Fit
Principles of Econometrics, 3rd Edition
Slide 5-2
S  1  2 P  3 A

(5.1)
β2 = the change in monthly sales S ($1000) when the price index P is increased by
one unit ($1), and advertising expenditure A is held constant
=

S
S

P ( A held constant) P
β3 = the change in monthly sales S ($1000) when advertising expenditure A is
increased by one unit ($1000), and the price index P is held constant
=
S
S

A ( P held constant) A
Principles of Econometrics, 3rd Edition
Slide 5-3
Figure 5.1 The multiple regression plane
Principles of Econometrics, 3rd Edition
Slide 5-4
Principles of Econometrics, 3rd Edition
Slide 5-5
Si  E ( Si )  ei  1  2 Pi  3 Ai  ei

(5.2)
The introduction of the error term, and assumptions about its probability
distribution, turn the economic model into the econometric model in (5.2).
Principles of Econometrics, 3rd Edition
Slide 5-6
yi  1  2 xi 2  3 xi 3 
  K xiK  ei
(5.3)
E  y 
E  y 
k 

xk other x's held constant
xk
yi  1  2 xi 2  3 xi 3  e
Principles of Econometrics, 3rd Edition
(5.4)
Slide 5-7
1.
E (ei )  0

Each random error has a probability distribution with zero mean. Some errors will
be positive, some will be negative; over a large number of observations they will
average out to zero.
Principles of Econometrics, 3rd Edition
Slide 5-8
2.

var(ei )  2
Each random error has a probability distribution with variance σ2. The variance σ2
is an unknown parameter and it measures the uncertainty in the statistical model. It
is the same for each observation, so that for no observations will the model
uncertainty be more, or less, nor is it directly related to any economic variable.
Errors with this property are said to be homoskedastic.
Principles of Econometrics, 3rd Edition
Slide 5-9
3.
cov(ei , e j )  0

The covariance between the two random errors corresponding to any two different
observations is zero. The size of an error for one observation has no bearing on the
likely size of an error for another observation. Thus, any pair of errors is
uncorrelated.
Principles of Econometrics, 3rd Edition
Slide 5-10
4.

ei ~ N  0, 2 
We will sometimes further assume that the random errors have normal probability
distributions.
Principles of Econometrics, 3rd Edition
Slide 5-11
The statistical properties of yi follow from the properties of ei.
1.
E ( yi )  1  2 xi 2  3 xi 3

The expected (average) value of yi depends on the values of the explanatory
variables and the unknown parameters. It is equivalent to E (ei )  0. This assumption
says that the average value of yi changes for each observation and is given by the
regression function E ( yi )  1  2 xi 2  3 xi 3 .
Principles of Econometrics, 3rd Edition
Slide 5-12
2.
var( yi )  var(ei )  2

The variance of the probability distribution of yi does not change with each
observation. Some observations on yi are not more likely to be further from the
regression function than others.
Principles of Econometrics, 3rd Edition
Slide 5-13
3.
cov( yi , y j )  cov(ei , e j )  0

Any two observations on the dependent variable are uncorrelated. For example, if
one observation is above E(yi), a subsequent observation is not more or less likely
to be above E(yi).
Principles of Econometrics, 3rd Edition
Slide 5-14
4.
yi ~ N (1  2 xi 2  3 xi 3 ), 2 

We sometimes will assume that the values of yi are normally distributed about their
mean. This is equivalent to assuming that ei ~ N  0, 2  .
Principles of Econometrics, 3rd Edition
Slide 5-15
Assumptions of the Multiple Regression Model
MR1.
yi  1  2 xi 2 
  K xiK  ei , i  1,
MR2.
E ( yi )  1  2 xi 2 
MR3.
var( yi )  var(ei )  2
MR4.
cov( yi , y j )  cov(ei , e j )  0
MR5.
The values of each xtk are not random and are not exact linear
,N
  K xiK  E (ei )  0
functions of the other explanatory variables
MR6.
yi ~ N (1  2 xi 2 
Principles of Econometrics, 3rd Edition
 K xiK ), 2   ei ~ N (0, 2 )
Slide 5-16
yi  1  2 xi 2  3 xi 3  e
N
S  1 , 2 , 3     yi  E ( yi ) 
(5.4)
2
i 1
(5.5)
N
   yi  1  2 xi 2  3 xi 3 
2
i 1
Principles of Econometrics, 3rd Edition
Slide 5-17
Principles of Econometrics, 3rd Edition
Slide 5-18
E ( yi )  1  2 xi 2  3 xi 3
yˆi  b1  b2 xi 2  b3 xi 3
 118.91  7.908 xi 2  1.863xi 3
Sˆi  118.91  7.908Pi  1.863 Ai
(5.6)
SALES  118.91  7.908 PRICE  1.863 ADVERT
Principles of Econometrics, 3rd Edition
Slide 5-19
Suppose we are interested in predicting sales revenue for a price of
$5.50 and an advertising expenditure of $1,200.
This prediction is given by
Sˆ  118.91  7.908 PRICE  1.863 ADVERT
 118.914  7.9079  5.5  1.8626  1.2
 77.656
Principles of Econometrics, 3rd Edition
Slide 5-20
Remark: Estimated regression models describe the relationship between
the economic variables for values similar to those found in the sample
data. Extrapolating the results to extreme values is generally not a good
idea. Predicting the value of the dependent variable for values of the
explanatory variables far from the sample values invites disaster.
Principles of Econometrics, 3rd Edition
Slide 5-21
2  var(ei )  E(ei2 )
eˆi  yi  yˆi  yi   b1  b2 xi 2  b3 xi 3 
N
ˆ2 
Principles of Econometrics, 3rd Edition
2
ˆ
e
i
i 1
(5.7)
N K
Slide 5-22
75
2
ˆ
e
i
1718.943
ˆ 

 23.874
N K
75  3
2
i 1
N
SSE   eˆi2  1718.943
i 1
ˆ  23.874  4.8861

Principles of Econometrics, 3rd Edition
Slide 5-23
The Gauss-Markov Theorem: For the multiple
regression model, if assumptions MR1-MR5
listed at the beginning of the Chapter hold, then
the least squares estimators are the Best Linear
Unbiased Estimators (BLUE) of the parameters.
Principles of Econometrics, 3rd Edition
Slide 5-24
var(b2 ) 
2
r23 
N
(1  r ) ( xi 2  x2 )
2
23
2
i 1
 ( xi 2  x2 )( xi 3  x3 )
2
2
(
x

x
)
(
x

x
)
 i 2 2  i3 3
Principles of Econometrics, 3rd Edition
(5.8)
(5.9)
Slide 5-25
1. Larger error variances 2 lead to larger variances of the least
squares estimators.
2. Larger sample sizes N imply smaller variances of the least squares
estimators.
3. More variation in an explanatory variable around its mean, leads
to a smaller variance of the least squares estimator.
4. A larger correlation between x2 and x3 leads to a larger variance
of b2.
Principles of Econometrics, 3rd Edition
Slide 5-26
 The covariance matrix for K=3 is
 var  b1 
cov  b1 , b2  cov  b1 , b3  


cov  b1 , b2 , b3   cov  b1 , b2 
var  b2 
cov  b2 , b3  
 cov  b1 , b3  cov  b2 , b3 
var  b3  
 The estimated variances and covariances in the example are
 40.343 6.795 .7484 
cov  b1 , b2 , b3    6.795
1.201 .0197 
 .7484 .0197
.4668
Principles of Econometrics, 3rd Edition
(5.10)
Slide 5-27
 Therefore, we have
var  b1   40.343
cov  b1 , b2   6.795
var  b2   1.201
cov  b1 , b3   .7484
var  b3   .4668
cov  b2 , b3   .0197
Principles of Econometrics, 3rd Edition
Slide 5-28
Principles of Econometrics, 3rd Edition
Slide 5-29
 The standard errors are
se(b1 )  var(b1 )  40.343  6.352
se(b2 )  var(b2 )  1.201  1.096
se(b3 )  var(b3 )  .4668  .6832
Principles of Econometrics, 3rd Edition
Slide 5-30
yi  1  2 xi 2  3 xi 3 
yi ~ N (1  2 xi 2 
  K xiK  ei
 K xiK ), 2   ei ~ N (0, 2 )
bk ~ N 
 k , var  bk 
Principles of Econometrics, 3rd Edition
Slide 5-31
z
bk  k
var  bk 
t
~ N  0,1 , for k  1, 2,, K
bk  k
var  bk 
Principles of Econometrics, 3rd Edition
bk  k

~ t N K 
se(bk )
(5.11)
(5.12)
Slide 5-32
P(tc  t(72)  tc )  .95
(5.13)


b2  2
P  1.993 
 1.993   .95
se(b2 )


(5.14)
P b2  1.993  se(b2 )  2  b2  1.993  se(b2 )  .95
b2  1.993  se(b2 ),
Principles of Econometrics, 3rd Edition
b2  1.993  se(b2 )
(5.15)
Slide 5-33
 A 95% interval estimate for β2 based on our sample is given by
(10.092,  5.724)
 A 95% interval estimate for β3 based on our sample is given by
(1.8626  1.993  .6832, 1.8626  1.993  .6832)  (.501, 3.224)
 The general expression for a 100(1  )% confidence interval is
[bk  t(1/2, N K )  se(bk ), bk  t(1/2, N K )  se(bk )
Principles of Econometrics, 3rd Edition
Slide 5-34
STEP-BY-STEP PROCEDURE FOR TESTING HYPOTHESES
1.
Determine the null and alternative hypotheses.
2.
Specify the test statistic and its distribution if the null hypothesis is true.
3.
Select α and determine the rejection region.
4.
Calculate the sample value of the test statistic and, if desired, the pvalue.
5.
State your conclusion.
Principles of Econometrics, 3rd Edition
Slide 5-35
H 0 : k  0
H1 :  k  0
bk
t
~ t( N  K )
se  bk 
 For a test with level of significance α
tc  t(1/2, N  K ) and  tc  t( /2, N  K )
Principles of Econometrics, 3rd Edition
Slide 5-36
 Big Andy’s Burger Barn example
1.
The null and alternative hypotheses are: H 0 : 2  0 and H1 : 2  0
2.
The test statistic, if the null hypothesis is true, is t  b2 se  b2  ~ t( N  K )
3.
Using a 5% significance level (α=.05), and 72 degrees of freedom, the critical
values that lead to a probability of 0.025 in each tail of the distribution are
t(.975, 72)  1.993 and t(.025, 72)  1.993
Principles of Econometrics, 3rd Edition
Slide 5-37
4.
The computed value of the t-statistic is t 
7.908
 7.215
1.096
the p-value in this case can be found as

 

P t 72  7.215  P t 72  7.215  2  (2.2 1010 )  .000
5.
Since 7.215  1.993 , we reject H 0 : 2  0 and conclude that there is evidence
from the data to suggest sales revenue depends on price. Using the p-value to
perform the test, we reject H 0 because .000  .05.
Principles of Econometrics, 3rd Edition
Slide 5-38
 Testing whether sales revenue is related to advertising expenditure
1.
H 0 : 3  0 and H1 : 3  0
2.
The test statistic, if the null hypothesis is true, is t  b3 se  b3  ~ t( N  K )
3.
Using a 5% significance level, we reject the null hypothesis if
t  1.993 or t  1.993 . In terms of the p-value, we reject H0 if p  .05 .
Principles of Econometrics, 3rd Edition
Slide 5-39
 Testing whether sales revenue is related to advertising expenditure
4.
The value of the test statistic is t  1.8626  2.726 ;

the p-value in given by P t72
5.
.6832
 2.726  P t72  2.726  2  .004  .008
 

Because 2.726  1.993 , we reject the null hypothesis; the data support the
conjecture that revenue is related to advertising expenditure. Using the p-value
we reject H 0 because .008  .05 .
Principles of Econometrics, 3rd Edition
Slide 5-40
 5.5.2a Testing for elastic demand
We wish to know if

2  0 : a decrease in price leads to a decrease in sales revenue (demand is price
inelastic), or

2  0 : a decrease in price leads to an increase in sales revenue (demand is price
elastic)
Principles of Econometrics, 3rd Edition
Slide 5-41
1.
H 0 : 2  0 (demand is unit elastic or inelastic)
H1 : 2  0 (demand is elastic)
2.
To create a test statistic we assume that H 0 : 2  0 is true and use
t  b2 se(b2 ) ~ t N  K 
3.
At a 5% significance level, we reject H 0 if t  1.666 or if the p  value  .05
Principles of Econometrics, 3rd Edition
Slide 5-42
4.
The value of the test statistic is t  b2
se  b2 

7.908
 7.215
1.096
The corresponding p-value is P(t(72)  7.215)  .000
5.
Since  7.215  1.666 we reject H 0 : 2  0 . Since .000  .05 , the same conclusion
is reached using the p-value.
Principles of Econometrics, 3rd Edition
Slide 5-43
5.5.2b Testing Advertising Effectiveness
1.
H 0 : 3  1 and H1 : 3  1
2.
To create a test statistic we assume that H 0 : 3  1 is true and use
t
3.
b3  1
~t
se(b3 )  N K 
At a 5% significance level, we reject H 0 if t  1.666 or if the p  value  .05
Principles of Econometrics, 3rd Edition
Slide 5-44
5.5.2b Testing Advertising Effectiveness
1.
The value of the test statistic is t  b3  3  1.8626  1  1.263
se  b3 
.6832
The corresponding p-value is P(t(72)  1.263)  .105
5.
Since 1.263<1.666 we do not reject H 0 . Since .105>.05, the same conclusion is
reached using the p-value.
Principles of Econometrics, 3rd Edition
Slide 5-45
N
SSR
R 

SST
2
  yˆi  y 
2
  yi  y 
2
i 1
N
i 1
(5.16)
N
SSE
 1
 1
SST
2
ˆ
e
i
i 1
N
  yi  y 
2
i 1
Principles of Econometrics, 3rd Edition
Slide 5-46
yˆi  b1  b2 xi 2  b3 xi 3 
ˆ y 
 bk xiK
1 N
2
 yi  y  

N  1 i 1
SST
N 1
SST  ( N  1)ˆ 2y
Principles of Econometrics, 3rd Edition
Slide 5-47
 For Big Andy’s Burger Barn we find that
SST  74  6.488542  3115.485
SSE  1718.943
N
R  1
2
ˆ
e
i
i 1
2
N
  yi  y 
2
1718.943
 1
 .448
3115.485
i 1
Principles of Econometrics, 3rd Edition
Slide 5-48

An alternative measure of goodness-of-fit called the adjusted-R2, is
usually reported by regression programs and it is computed as
SSE / ( N  K )
R  1
SST / ( N  1)
2
Principles of Econometrics, 3rd Edition
Slide 5-49

If the model does not contain an intercept parameter, then the measure
R2 given in (5.16) is no longer appropriate. The reason it is no longer
appropriate is that, without an intercept term in the model,
N
  yi  y 
i 1
2
N
N
2
ˆ
ˆ
   yi  y    ei
2
i 1
i 1
SST  SSR  SSE
Principles of Econometrics, 3rd Edition
Slide 5-50
SALES  118.9  7.908 PRICE  1.8626 ADVERT
(se)
(6.35) (1.096)
R2  .448
(5.17)
(.6832)
 From this summary we can read off the estimated effects of changes in the
explanatory variables on the dependent variable and we can predict values of the
dependent variable for given values of the explanatory variables. For the
construction of an interval estimate we need the least squares estimate, its
standard error, and a critical value from the t-distribution.
Principles of Econometrics, 3rd Edition
Slide 5-51









BLU estimator
covariance matrix of
least squares
estimator
critical value
error variance
estimate
error variance
estimator
goodness of fit
interval estimate
least squares
estimates
least squares
Principles of Econometrics, 3rd Edition









estimation
least squares
estimators
multiple regression
model
one-tailed test
p-value
regression
coefficients
standard errors
sum of squared errors
sum of squares of
regression
testing significance


total sum of squares
two-tailed test
Slide 5-52
Principles of Econometrics, 3rd Edition
Slide 5-53
N
S (1 , 2 , 3 )   ( yi  1  2 xi 2  3 xi 3 ) 2
(2A.1)
i 1
S
 2 N 1  22  xi 2  23  xi 3  2 yi
1
S
 21  xi 2  22  xi22  23  xi 2 xi 3  2 xi 2 yi
2
S
 21  xi 3  22  xi 2 xi 3  23  xi23  2 xi 3 yi
3
Principles of Econometrics, 3rd Edition
Slide 5-54
Nb1   xi 2b2   xi 3b3   yi
 xi 2b1   xi22b2   xi 2 xi 3b3   xi 2 yi
(5A.1)
 xi 3b1   xi 2 xi 3b2   xi23b3   xi 3 yi
let yi  yi  y ,
Principles of Econometrics, 3rd Edition
xi2  xi 2  x2 , xi3  xi 3  x3
Slide 5-55
b1  y  b2 x2  b3 x3
b2
y x   x     y x   x



  x   x     x x 
b3
y x   x     y x   x



  x   x     x x 
 
i i2
2
i3
2
i2
Principles of Econometrics, 3rd Edition
 
i i3
x

 
i3 i 2

  2
i 2 i3
2
i3
2
i2
2
i2
 
i 2 i3
 
i i3
 
i i2
2
i3
x
  2
i 2 i3
Slide 5-56