Random Regressors and Moment Based
Estimation
Prepared by Vera Tabakova, East Carolina University

10.1 Linear Regression with Random x’s

10.2 Cases in Which x and e are Correlated

10.3 Estimators Based on the Method of Moments

10.4 Specification Tests
Principles of Econometrics, 3rd Edition
Slide 10-2
The assumptions of the simple linear regression are:

SR1. yi  1  2 xi  ei i  1, , N

SR2. E (ei )  0

SR3. var(ei )  2

SR4. cov(ei , e j )  0

SR5. The variable xi is not random, and it must take at least two
different values.

SR6. (optional) ei ~ N (0, 2 )
Principles of Econometrics, 3rd Edition
Slide 10-3
The purpose of this chapter is to discuss regression models in which
xi is random and correlated with the error term ei. We will:

Discuss the conditions under which having a random x is not a
problem, and how to test whether our data satisfies these conditions.

Present cases in which the randomness of x causes the least squares
estimator to fail.

Provide estimators that have good properties even when xi is random
and correlated with the error ei.
Principles of Econometrics, 3rd Edition
Slide 10-4

A10.1
yi  1  2 xi  ei correctly describes the relationship
between yi and xi in the population, where β1 and β2 are
unknown (fixed) parameters and ei is an unobservable
random error term.

A10.2
The data pairs  xi , yi  i  1, , N , are obtained by random
sampling. That is, the data pairs are collected from the
same population, by a process in which each pair is
independent of every other pair. Such data are said to be
independent and identically distributed.
Principles of Econometrics, 3rd Edition
Slide 10-5

A10.3
E  ei | xi   0. The expected value of the error term ei,
conditional on the value of xi, is zero.
This assumption implies that we have (i) omitted no important variables,
(ii) used the correct functional form, and (iii) there exist no factors that
cause the error term ei to be correlated with xi.
 If E  ei | xi   0 , then we can show that it is also true that xi and ei are
uncorrelated, and that cov  xi , ei   0 .
 Conversely, if xi and ei are correlated, then
show that E  ei | xi   0 .
Principles of Econometrics, 3rd Edition
cov  xi , ei   0 and we can
Slide 10-6

A10.4
In the sample, xi must take at least two different values.

A10.5
var  ei | xi   2 . The variance of the error term, conditional
on xi is a constant σ2.

A10.6
ei | xi ~ N  0, 2  . The distribution of the error term,
conditional on xi, is normal.
Principles of Econometrics, 3rd Edition
Slide 10-7

Under assumptions A10.1-A10.4 the least squares estimator is
unbiased.

Under assumptions A10.1-A10.5 the least squares estimator is the
best linear unbiased estimator of the regression parameters,
conditional on the x’s, and the usual estimator of σ2 is unbiased.
Principles of Econometrics, 3rd Edition
Slide 10-8

Under assumptions A10.1-A10.6 the distributions of the least squares
estimators, conditional upon the x’s, are normal, and their variances
are estimated in the usual way. Consequently the usual interval
estimation and hypothesis testing procedures are valid.
Principles of Econometrics, 3rd Edition
Slide 10-9
Figure 10.1 An illustration of consistency
Principles of Econometrics, 3rd Edition
Slide 10-10
Remark: Consistency is a “large sample” or “asymptotic” property. We have
stated another large sample property of the least squares estimators in Chapter
2.6. We found that even when the random errors in a regression model are not
normally distributed, the least squares estimators still have approximate
normal distributions if the sample size N is large enough. How large must the
sample size be for these large sample properties to be valid approximations of
reality? In a simple regression 50 observations might be enough. In multiple
regression models the number might be much higher, depending on the quality
of the data.
Principles of Econometrics, 3rd Edition
Slide 10-11

A10.3* E  ei   0 and cov  xi , ei   0
E  ei | xi   0  cov  xi , ei   0
E  ei | xi   0  E  ei   0
Principles of Econometrics, 3rd Edition
Slide 10-12

Under assumption A10.3* the least squares estimators are consistent.
That is, they converge to the true parameter values as N.

Under assumptions A10.1, A10.2, A10.3*, A10.4 and A10.5, the least
squares estimators have approximate normal distributions in large
samples, whether the errors are normally distributed or not.
Furthermore our usual interval estimators and test statistics are valid,
if the sample is large.
Principles of Econometrics, 3rd Edition
Slide 10-13
 If assumption A10.3* is not true, and in particular if cov  xi , ei   0
so that xi and ei are correlated, then the least squares estimators are
inconsistent. They do not converge to the true parameter values
even in very large samples. Furthermore, none of our usual
hypothesis testing or interval estimation procedures are valid.
Principles of Econometrics, 3rd Edition
Slide 10-14
Figure 10.2 Plot of correlated x and e
Principles of Econometrics, 3rd Edition
Slide 10-15
y  E  y   e  1  2 x  e  1  1 x  e
yˆ  b1  b2 x  .9789  1.7034 x
Principles of Econometrics, 3rd Edition
Slide 10-16
Figure 10.3 Plot of data, true and fitted regressions
Principles of Econometrics, 3rd Edition
Slide 10-17
When an explanatory variable and the error term are correlated the
explanatory variable is said to be endogenous and means
“determined within the system.” When an explanatory variable is
correlated with the regression error one is said to have an
“endogeneity problem.”
Principles of Econometrics, 3rd Edition
Slide 10-18
yi  1  2 xi*  vi
xi  xi*  ui
Principles of Econometrics, 3rd Edition
(10.1)
(10.2)
Slide 10-19
yi  1  2 xi*  vi
 1  2  xi  ui   vi
 1  2 xi   vi  2ui 
(10.3)
 1  2 xi  ei
Principles of Econometrics, 3rd Edition
Slide 10-20
cov  xi , ei   E  xi ei   E  xi*  ui   vi  2ui  
 E  2ui2   2u2  0
Principles of Econometrics, 3rd Edition
(10.4)
Slide 10-21
WAGEi  1  2 EDUCi  ei
(10.5)
Omitted factors: experience, ability and motivation.
Therefore, we expect that cov  EDUCi , ei   0.
Principles of Econometrics, 3rd Edition
Slide 10-22
Qi  1  2 Pi  ei
(10.6)
There is a feedback relationship between Pi and Qi. Because of this
feedback, which results because price and quantity are jointly, or
simultaneously, determined, we can show that cov( Pi , ei )  0.
The resulting bias (and inconsistency) is called the simultaneous
equations bias.
Principles of Econometrics, 3rd Edition
Slide 10-23
yt  1  2 yt 1  3 xt  et
AR(1) process: et  et 1  vt
If   0 there will be correlation between yt 1 and et .
In this case the least squares estimator applied to the lagged
dependent variable model will be biased and inconsistent.
Principles of Econometrics, 3rd Edition
Slide 10-24
When all the usual assumptions of the linear model hold, the method
of moments leads us to the least squares estimator. If x is random and
correlated with the error term, the method of moments leads us to an
alternative, called instrumental variables estimation, or two-stage
least squares estimation, that will work in large samples.
Principles of Econometrics, 3rd Edition
Slide 10-25
E Y k   k  k th moment of Y
(10.7)
E Y k   ˆ k  k th sample moment of Y   yik N
(10.8)
var Y     E Y     E Y 2    2
(10.9)
2
Principles of Econometrics, 3rd Edition
2
Slide 10-26
Population Moments Sample Moments
E Y   1  
ˆ   yi N
E Y 2    2
(10.10)
ˆ 2   yi2 N
ˆ   yi N  y
(10.11)
yi  y 
y
y  Ny




2
2
2
  ˆ 2  ˆ 
y 

N
N
N
2
i
Principles of Econometrics, 3rd Edition
2
i
2
2
(10.12)
Slide 10-27
E  ei   0  E  yi  1  2 xi   0
(10.13)
E  xi ei   0  E  xi  yi 1 2 xi   0
(10.14)
1
N
1
N
  yi  b1  b2 xi   0
 xi  yi  b1  b2 xi   0
Principles of Econometrics, 3rd Edition
(10.15)
Slide 10-28
xi  x  yi  y 


b2 
2
  xi  x 
b1  y  b2 x
Under "nice" assumptions, the method of moments principle of
estimation leads us to the same estimators for the simple linear
regression model as the least squares principle.
Principles of Econometrics, 3rd Edition
Slide 10-29
Suppose that there is another variable, z, such that

z does not have a direct effect on y, and thus it does not belong on the
right-hand side of the model as an explanatory variable.

zi is not correlated with the regression error term ei. Variables with
this property are said to be exogenous.

z is strongly [or at least not weakly] correlated with x, the endogenous
explanatory variable.
A variable z with these properties is called an instrumental variable.
Principles of Econometrics, 3rd Edition
Slide 10-30
E  zi ei   0  E  zi  yi 1 2 xi   0

(10.16)

1
yi  ˆ 1  ˆ 2 xi  0

N


(10.17)
1
zi yi  ˆ 1  ˆ 2 xi  0

N
Principles of Econometrics, 3rd Edition
Slide 10-31
ˆ  N  zi yi   zi  yi 
2
N  zi xi   zi  xi
  zi  z  yi  y 
  zi  z  xi  x 
(10.18)
ˆ 1  y  ˆ 2 x
Principles of Econometrics, 3rd Edition
Slide 10-32
These new estimators have the following properties:

They are consistent, if E  zi ei   0.

In large samples the instrumental variable estimators have
approximate normal distributions. In the simple regression model
2



ˆ 2 ~ N  2 , 2
2 
 r  x  x  
zx
i


Principles of Econometrics, 3rd Edition
(10.19)
Slide 10-33

The error variance is estimated using the estimator
ˆ 2IV 
Principles of Econometrics, 3rd Edition
  yi  ˆ 1  ˆ 2 xi 
2
N 2
Slide 10-34
 
var ˆ 2 
2
2
zx
r
  xi  x 
2
var  b2 

rzx2
Using the instrumental variables estimation procedure when it is not
required leads to wider confidence intervals, and less precise
inference, than if least squares estimation is used.
The bottom line is that when instruments are weak instrumental
variables estimation is not reliable.
Principles of Econometrics, 3rd Edition
Slide 10-35
yˆOLS  .9789  1.7034 x
yˆ IV _ z1  1.1011  1.1924 x
(se) (.088) (.090)
(se)
yˆ IV _ z2  1.3451  .1724 x
yˆ IV _ z3  .9640  1.7657 x
(se)
(se)
(.256) (.797)
Principles of Econometrics, 3rd Edition
(.109) (.195)
(.095) (.172)
Slide 10-36
ln WAGE   1  2 EDUC  3 EXPER  4 EXPER 2  e
ln WAGE  =  .5220  .1075  EDUC  .0416  EXPER  .0008  EXPER 2
(se)
(.1986) (.0141)
Principles of Econometrics, 3rd Edition
(.0132)
(.0004)
Slide 10-37
EDUC  9.7751  .0489  EXPER  .0013  EXPER 2  .2677  MOTHEREDUC
(se)
(.4249) (.0417)
(.0012)
(.0311)
ln WAGE   .1982  .0493  EDUC  .0449  EXPER  .0009  EXPER 2
(se)
(.4729) (.0374)
Principles of Econometrics, 3rd Edition
(.0136)
(.0004)
Slide 10-38
E  wi ei   E  wi  yi 1 2 xi   0


1
yi  ˆ 1  ˆ 2 xi  mˆ 1  0

N




1
ˆ
ˆ
ˆ
z
y





i
i
1
2 xi  m2  0
N
(10.20)
1
wi yi  ˆ 1  ˆ 2 xi  mˆ 3  0

N
Principles of Econometrics, 3rd Edition
Slide 10-39
A 2-step process.

Regress x on a constant term, z and w, and obtain the predicted
values xˆ.

Use x̂ as an instrumental variable for x.
Principles of Econometrics, 3rd Edition
Slide 10-40


1
yi  ˆ 1  ˆ 2 xi  0

N


(10.21)
1
xˆi yi  ˆ 1  ˆ 2 xi  0

N
Principles of Econometrics, 3rd Edition
Slide 10-41
xˆi  xˆ   yi  y    xˆi  x  yi  y 


ˆ 2 

  xˆi  xˆ   xi  x    xˆi  x  xi  x 
(10.22)
ˆ 1  y  ˆ 2 x
Principles of Econometrics, 3rd Edition
Slide 10-42
Two-stage least squares (2SLS) estimator:

Stage 1 is the regression of x on a constant term, z and w, to obtain
the predicted values x̂ . This first stage is called the reduced form
model estimation.

Stage 2 is ordinary least squares estimation of the simple linear
regression
yi  1  2 xˆi  errori
Principles of Econometrics, 3rd Edition
(10.23)
Slide 10-43
2

var ˆ 2 
2
ˆ
  xi  x 
 
ˆ 
2
IV

yi  ˆ 1  ˆ 2 xi
Principles of Econometrics, 3rd Edition

2
N 2
2
ˆ

IV
var ˆ 2 
2
  xˆi  x 
 
(10.24)
(10.25)
Slide 10-44
xˆ  .1947  .5700 z1  .2068 z2
(se) (.079) (.089)
(.077)
yˆ IV _ z1 , z2  1.1376  1.0399 x
(se)
Principles of Econometrics, 3rd Edition
(.116)
(.194)
(10.26)
(10.27)
Slide 10-45
Principles of Econometrics, 3rd Edition
Slide 10-46
ln WAGE = .0481  .0614 EDUC  .0442 EXPER  .0009 EXPER 2
(se)
(.4003) (.0314)
Principles of Econometrics, 3rd Edition
(.0134)
(.0004)
Slide 10-47
G exogenous variables
B endogenous variables
y  1  2 x2 
G xG  G1xG1 
xG  j  1 j   2 j x2 
  Gj xG  1 j z1 
 K xK  e
  Lj zL  v j ,
j  1,
Principles of Econometrics, 3rd Edition
,B
(10.28)
(10.29)
Slide 10-48
xˆG  j  ˆ 1 j  ˆ 2 j x2 
 ˆ Gj xG  ˆ 1 j z1 
 ˆ Lj z L ,
j  1,
y  1  2 x2 
Principles of Econometrics, 3rd Edition
G xG  G 1 xˆG 1 
,B
  K xˆK  error (10.30)
Slide 10-49
When testing the null hypothesis H 0 : k  c use of the test statistic
t  ˆ  c se ˆ is valid in large samples. It is common, but not

k
  
k
universal, practice to use critical values, and p-values, based on the
distribution rather than the more strictly appropriate N(0,1)
distribution. The reason is that tests based on the t-distribution tend to
work better in samples of data that are not large.
Principles of Econometrics, 3rd Edition
Slide 10-50
When testing a joint hypothesis, such as H 0 : 2  c2 , 3  c3 , the test
may be based on the chi-square distribution with the number of
degrees of freedom equal to the number of hypotheses (J) being
tested. The test itself may be called a “Wald” test, or a likelihood ratio
(LR) test, or a Lagrange multiplier (LM) test. These testing procedures
are all asymptotically equivalent .
Principles of Econometrics, 3rd Edition
Slide 10-51
y  1  2 x  e
eˆ  y  ˆ  ˆ x
1
2
R  1   eˆ
2
2
i
  yi  y 
2
Unfortunately R2 can be negative when based on IV estimates.
Therefore the use of measures like R2 outside the context of the least
squares estimation should be avoided.
Principles of Econometrics, 3rd Edition
Slide 10-52

Can we test for whether x is correlated with the error term? This
might give us a guide of when to use least squares and when to use IV
estimators.

Can we test whether our instrument is sufficiently strong to avoid the
problems associated with “weak” instruments?

Can we test if our instrument is valid, and uncorrelated with the
regression error, as required?
Principles of Econometrics, 3rd Edition
Slide 10-53
H 0 : cov  xi , ei   0

H1 : cov  xi , ei   0
If the null hypothesis is true, both the least squares estimator and the
instrumental variables estimator are consistent. Naturally if the null
hypothesis is true, use the more efficient estimator, which is the least
squares estimator.

If the null hypothesis is false, the least squares estimator is not consistent,
and the instrumental variables estimator is consistent. If the null hypothesis
is not true, use the instrumental variables estimator, which is consistent.
Principles of Econometrics, 3rd Edition
Slide 10-54
yi  1  2 xi  ei
Let z1 and z2 be instrumental variables for x.
1.
Estimate the model xi  1  1 zi1  2 zi 2  vi by least squares, and
obtain the residuals vˆ  x  ˆ  ˆ z  ˆ z . If there are more than
i
i
1
1 i1
2 i2
one explanatory variables that are being tested for endogeneity,
repeat this estimation for each one, using all available instrumental
variables in each regression.
Principles of Econometrics, 3rd Edition
Slide 10-55
2.
Include the residuals computed in step 1 as an explanatory variable
in the original regression,yi  1  2 xi  vˆi  ei . Estimate this
"artificial regression" by least squares, and employ the usual t-test
for the hypothesis of significance
H 0 :   0  no correlation between xi and ei 
H1 :   0  correlation between xi and ei 
Principles of Econometrics, 3rd Edition
Slide 10-56
3.
If more than one variable is being tested for endogeneity, the test
will be an F-test of joint significance of the coefficients on the
included residuals.
Principles of Econometrics, 3rd Edition
Slide 10-57
y  1  2 x2 
 G xG  G 1 xG 1  e
xG 1  1   2 x2 
  G xG  1 z1  v
If we have L > 1 instruments available then the reduced form equation is
xG 1  1   2 x2 
Principles of Econometrics, 3rd Edition
  G xG  1 z1 
L zL  v
Slide 10-58
1.
Compute the IV estimates ˆ k using all available instruments,
including the G variables x1=1, x2, …, xG that are presumed to be
exogenous, and the L instruments z1, …, zL.
2.
Obtain the residuals eˆ  y  ˆ 1  ˆ 2 x2 
Principles of Econometrics, 3rd Edition
 ˆ K xK .
Slide 10-59
3.
Regress ê on all the available instruments described in step 1.
4.
Compute NR2 from this regression, where N is the sample size and
R2 is the usual goodness-of-fit measure.
5.
If all of the surplus moment conditions are valid, then NR 2 ~ (2L B ) .
If the value of the test statistic exceeds the 100(1−α)-percentile from
2

the ( L  B ) distribution, then we conclude that at least one of the
surplus moment conditions restrictions is not valid.
Principles of Econometrics, 3rd Edition
Slide 10-60
10.4.4a The Hausman Test
vˆ  x  xˆ  x  .1947  .5700 z1  .2068z2
(10.31)
yˆ  1.1376  1.0399 x  .9957vˆ
(se) (.080) (.133)
(.163)
Principles of Econometrics, 3rd Edition
Slide 10-61

10.4.4b Test for Weak Instruments
xˆ  .2196  .5711z1
(t )
(6.23)
xˆ  .2140  .2090 z2
(t )
Principles of Econometrics, 3rd Edition
(2.28)
Slide 10-62

10.4.4c Testing Surplus Moment Conditions

If we use z1 and z2 as instruments there is one surplus moment
condition.
eˆ  .0189  .0881z1  .1818 z2
The R2 from this regression is .03628, and NR2 = 3.628. The .05
critical value for the chi-square distribution with one degree of
freedom is 3.84, thus we fail to reject the validity of the surplus
moment condition.
Principles of Econometrics, 3rd Edition
Slide 10-63

10.4.4c Testing Surplus Moment Conditions

If we use z1, z2 and z3 as instruments there are two surplus moment
conditions.
eˆ = 0.0207  .1033z1  .2355z2 + .1798z3
The R2 from this regression is .1311, and NR2 = 13.11. The .05 critical
value for the chi-square distribution with two degrees of freedom is
5.99, thus we reject the validity of the two surplus moment
conditions.
Principles of Econometrics, 3rd Edition
Slide 10-64
Principles of Econometrics, 3rd Edition
Slide 10-65















asymptotic properties
conditional expectation
endogenous variables
errors-in-variables
exogenous variables
finite sample properties
Hausman test
instrumental variable
instrumental variable estimator
just identified equations
large sample properties
over identified equations
population moments
random sampling
reduced form equation
Principles of Econometrics, 3rd Edition





sample moments
simultaneous equations bias
test of surplus moment conditions
two-stage least squares estimation
weak instruments
Slide 10-66
Principles of Econometrics, 3rd Edition
Slide 10-67

10A.1 Conditional Expectations
E Y | X  x    yP Y  y | X  x   yf  y | x 
y
(10A.1)
y
var Y | X  x     y  E Y | X  x   f  y | x 
2
y
Principles of Econometrics, 3rd Edition
Slide 10-68

10A.2 Iterated Expectations
E Y   EX  E Y | X 
(10A.2)
f  x, y   f  y | x  f  x 
Principles of Econometrics, 3rd Edition
Slide 10-69

10A.2 Iterated Expectations


E Y    yf  y    y   f  x, y  
y
y
x



  y  f  y | x  f  x 
y
x



    yf  y | x   f  x 
x  y

[by changing order of summation]
  E Y | X  x  f  x 
x
 E X  E Y | X  
Principles of Econometrics, 3rd Edition
Slide 10-70

10A.2 Iterated Expectations
E  XY   EX  XE Y | X 
(10A.3)
cov  X , Y   EX  X   X  E Y | X 
(10A.4)
Principles of Econometrics, 3rd Edition
Slide 10-71

10A.3 Regression Model Applications
E  ei   Ex  E  ei | xi   Ex 0  0
(10A.5)
E  xi ei   Ex  xi E  ei | xi   Ex  xi 0  0
(10A.6)
cov  xi , ei   Ex  xi   x  E  ei | xi   Ex  xi   x  0  0 (10A.7)
Principles of Econometrics, 3rd Edition
Slide 10-72
yi  E  yi   2  xi  E  xi   ei
 xi  E  xi    yi  E  yi   2  xi  E  xi    xi  E  xi  ei
2

E  xi  E  xi   yi  E  yi   2 E  xi  E  xi   E  xi  E  xi  ei
2

cov  x, y   2 var  x   cov  x, e 
Principles of Econometrics, 3rd Edition
Slide 10-73
cov  x, y  cov  x, e 
2 

var  x 
var  x 
cov  x, y 
2 
var  x 
xi  x  yi  y    xi  x  yi  y  /  N  1 cov( x, y )


b2 


2
2
var( x)
  xi  x 
  xi  x  /  N  1
Principles of Econometrics, 3rd Edition
(10B.1)
(10B.2)
(10B.3)
Slide 10-74
cov( x, y )
cov( x, y )
b2 

 2
var( x)
var( x)
cov  x, y  cov  x, e 
2 

var  x 
var  x 
cov  x, y 
cov  x, e 
b2 
 2 
 2
var  x 
var  x 
Principles of Econometrics, 3rd Edition
(10B.4)
Slide 10-75
ˆ    zi  z  yi  y   N  1  cov  z , y 
2
  zi  z  xi  x   N  1 cov  z, x 
ˆ  cov  z , y 
2
cov  z , x 
Principles of Econometrics, 3rd Edition
(10C.1)
(10C.2)
Slide 10-76
cov  z , y  cov  z , e 
2 

cov  z , x  cov  z , x 
(10C.3)
ˆ  cov  z , y   
2
2
cov  z , x 
(10C.4)
Principles of Econometrics, 3rd Edition
Slide 10-77
y  1  2 x  e
(10D.1)
x  0  1 z  v
(10D.2)
x  E  x  v
(10D.3)
y  1  2 x  e  1  2  E  x   v   e
 1  2 E  x   2v  e
(10D.4)
Principles of Econometrics, 3rd Edition
Slide 10-78
x  xˆ  vˆ
y  1  2 x  e  1  2  xˆ  vˆ  e
 1  2 xˆ  2vˆ  e
(10D.5)
(10D.6)
y  1  2 xˆ  vˆ  e
(10D.7)
y  1  2 xˆ  e
(10D.8)
Principles of Econometrics, 3rd Edition
Slide 10-79
y  1  2 xˆ  vˆ  e  2vˆ   2vˆ
 1  2  xˆ  vˆ       2  vˆ  e
(10D.9)
 1  2 x  vˆ  e
Principles of Econometrics, 3rd Edition
Slide 10-80