Expectations, Variances & Covariances
The Rules of Summation
n
å xi ¼ x1 þ x2 þ þ xn
covðX; YÞ ¼ E½ðXE½XÞðYE½YÞ
i¼1
n
¼ å å ½x EðXÞ½ y EðYÞ f ðx; yÞ
å a ¼ na
x y
i¼1
n
covðX;YÞ
r ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
varðXÞvarðYÞ
n
å axi ¼ a å xi
i¼1
n
i¼1
n
n
i¼1
i¼1
E(c1X þ c2Y ) ¼ c1E(X ) þ c2E(Y )
E(X þ Y ) ¼ E(X ) þ E(Y )
å ðxi þ yi Þ ¼ å xi þ å yi
i¼1
n
n
n
i¼1
i¼1
å ðaxi þ byi Þ ¼ a å xi þ b å yi
i¼1
n
var(aX þ bY þ cZ ) ¼ a2var(X) þ b2var(Y ) þ c2var(Z )
þ 2abcov(X,Y ) þ 2accov(X,Z ) þ 2bccov(Y,Z )
n
å ða þ bxi Þ ¼ na þ b å xi
i¼1
If X, Y, and Z are independent, or uncorrelated, random
variables, then the covariance terms are zero and:
i¼1
n
å xi
x1 þ x2 þ þ xn
x ¼ i¼1n ¼
n
varðaX þ bY þ cZÞ ¼ a2 varðXÞ
n
å ðxi xÞ ¼ 0
þ b2 varðYÞ þ c2 varðZÞ
i¼1
2
3
2
å å f ðxi ; yj Þ ¼ å ½ f ðxi ; y1 Þ þ f ðxi ; y2 Þ þ f ðxi ; y3 Þ
i¼1 j¼1
i¼1
¼ f ðx1 ; y1 Þ þ f ðx1 ; y2 Þ þ f ðx1 ; y3 Þ
þ f ðx2 ; y1 Þ þ f ðx2 ; y2 Þ þ f ðx2 ; y3 Þ
Expected Values & Variances
EðXÞ ¼ x1 f ðx1 Þ þ x2 f ðx2 Þ þ þ xn f ðxn Þ
n
¼ å xi f ðxi Þ ¼ å x f ðxÞ
x
i¼1
E½gðXÞ ¼ å gðxÞ f ðxÞ
x
E½g1 ðXÞ þ g2 ðXÞ ¼ å ½g1ðxÞ þ g2 ðxÞ f ðxÞ
x
¼ å g1ðxÞ f ðxÞ þ å g2 ðxÞ f ðxÞ
x
Normal Probabilities
Xm
Nð0; 1Þ
s
2
If X N(m, s ) and a is a constant, then
a m
PðX aÞ ¼ P Z s
If X Nðm; s2 Þ and a and b are constants; then
am
bm
Z
Pða X bÞ ¼ P
s
s
If X N(m, s2), then Z ¼
Assumptions of the Simple Linear Regression
Model
SR1
x
¼ E½g1 ðXÞ þ E½g2 ðXÞ
E(c) ¼ c
E(cX ) ¼ cE(X )
E(a þ cX ) ¼ a þ cE(X )
var(X ) ¼ s2 ¼ E[X E(X )]2 ¼ E(X2) [E(X )]2
var(a þ cX ) ¼ E[(a þ cX) E(a þ cX)]2 ¼ c2var(X )
Marginal and Conditional Distributions
f ðxÞ ¼ å f ðx; yÞ
for each value X can take
f ðyÞ ¼ å f ðx; yÞ
for each value Y can take
SR2
SR3
SR4
SR5
SR6
The value of y, for each value of x, is y ¼ b1 þ
b2x þ e
The average value of the random error e is
E(e) ¼ 0 since we assume that E(y) ¼ b1 þ b2x
The variance of the random error e is var(e) ¼
s2 ¼ var(y)
The covariance between any pair of random
errors, ei and ej is cov(ei, ej) ¼ cov(yi, yj) ¼ 0
The variable x is not random and must take at
least two different values.
(optional) The values of e are normally distributed about their mean e N(0, s2)
y
x
f ðxjyÞ ¼ P½X ¼ xjY ¼ y ¼
f ðx; yÞ
f ðyÞ
If X and Y are independent random variables, then
f (x,y) ¼ f (x)f ( y) for each and every pair of values
x and y. The converse is also true.
If X and Y are independent random variables, then the
conditional probability density function of X given that
Y ¼ y is f ðxjyÞ ¼
f ðx; yÞ f ðxÞ f ðyÞ
¼
¼ f ðxÞ
f ðyÞ
f ðyÞ
for each and every pair of values x and y. The converse is
also true.
Least Squares Estimation
If b1 and b2 are the least squares estimates, then
^yi ¼ b1 þ b2 xi
^ei ¼ yi ^yi ¼ yi b1 b2 xi
The Normal Equations
Nb1 þ Sxi b2 ¼ Syi
Sxi b1 þ Sx2i b2 ¼ Sxi yi
Least Squares Estimators
b2 ¼
Sðxi xÞðyi yÞ
S ðxi xÞ2
b1 ¼ y b2 x
Elasticity
percentage change in y Dy=y Dy x
¼
¼
h¼
percentage change in x Dx=x Dx y
h¼
DEðyÞ=EðyÞ DEðyÞ x
x
¼
¼ b2 Dx=x
Dx EðyÞ
EðyÞ
Least Squares Expressions Useful for Theory
b2 ¼ b2 þ Swi ei
wi ¼
Sðxi xÞ2
Swi xi ¼ 1;
Sw2i ¼ 1=Sðxi xÞ2
Properties of the Least Squares Estimators
"
#
Sx2i
s2
varðb1 Þ ¼ s2
varðb2 Þ ¼
2
NSðxi xÞ
Sðxi xÞ2
"
#
x
covðb1 ; b2 Þ ¼ s2
Sðxi xÞ2
Gauss-Markov Theorem: Under the assumptions
SR1–SR5 of the linear regression model the estimators
b1 and b2 have the smallest variance of all linear and
unbiased estimators of b1 and b2. They are the Best
Linear Unbiased Estimators (BLUE) of b1 and b2.
If we make the normality assumption, assumption
SR6, about the error term, then the least squares estimators are normally distributed.
!
!
s2 å x2i
s2
; b2 N b2 ;
b1 N b1 ;
NSðxi xÞ2
Sðxi xÞ2
Estimated Error Variance
s
^2 ¼
Type II error: The null hypothesis is false and we decide
not to reject it.
p-value rejection rule: When the p-value of a hypothesis test is smaller than the chosen value of a, then the
test procedure leads to rejection of the null hypothesis.
xi x
Swi ¼ 0;
Rejection rule for a two-tail test: If the value of the
test statistic falls in the rejection region, either tail of
the t-distribution, then we reject the null hypothesis
and accept the alternative.
Type I error: The null hypothesis is true and we decide
to reject it.
S^e2i
N 2
Prediction
y0 ¼ b1 þ b2 x0 þ e0 ; ^y0 ¼ b1 þ b2 x0 ; f ¼ ^y0 y0
"
#
qffiffiffiffiffiffiffiffiffiffiffiffiffi
1
ðx0 xÞ2
2
b
bf Þ
varð f Þ ¼ s
^ 1þ þ
varð
;
seð
f
Þ
¼
N Sðxi xÞ2
A (1 a) 100% confidence interval, or prediction
interval, for y0
^y0 tc seð f Þ
Goodness of Fit
Sðyi yÞ2 ¼ Sð^yi yÞ2 þ S^e2i
SST ¼ SSR þ SSE
SSR
SSE
¼1
¼ ðcorrðy; ^yÞÞ2
R2 ¼
SST
SST
Log-Linear Model
lnðyÞ ¼ b1 þ b2 x þ e; b
lnð yÞ ¼ b1 þ b2 x
100 b2
% change in y given a one-unit change in x:
^yn ¼ expðb1 þ b2 xÞ
^yc ¼ expðb1 þ b2 xÞexpð^
s2 =2Þ
Prediction interval:
h
i
h
i
b tc seð f Þ ; exp lnð
byÞ þ tc seð f Þ
exp lnðyÞ
Estimator Standard Errors
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
varðb1 Þ; seðb2 Þ ¼ b
varðb2 Þ
seðb1 Þ ¼ b
Generalized goodness-of-fit measure R2g ¼ ðcorrðy;^yn ÞÞ2
t-distribution
MR1 yi ¼ b1 þ b2xi2 þ þ bKxiK þ ei
If assumptions SR1–SR6 of the simple linear regression
model hold, then
MR3 var(yi) ¼ var(ei) ¼ s2
t¼
bk bk
tðN2Þ ; k ¼ 1; 2
seðbk Þ
Interval Estimates
P[b2 tcse(b2) b2 b2 þ tcse(b2)] ¼ 1 a
Hypothesis Testing
Components of Hypothesis Tests
1. A null hypothesis, H0
2. An alternative hypothesis, H1
3. A test statistic
4. A rejection region
5. A conclusion
If the null hypothesis H0 : b2 ¼ c is true, then
t¼
b2 c
tðN2Þ
seðb2 Þ
Assumptions of the Multiple Regression Model
MR2 E(yi) ¼ b1 þ b2xi2 þ þ bKxiK , E(ei) ¼ 0.
MR4 cov(yi, yj) ¼ cov(ei, ej) ¼ 0
MR5 The values of xik are not random and are not
exact linear functions of the other explanatory
variables.
MR6 yi N½ðb1 þ b2 xi2 þ þ bK xiK Þ; s2 , ei Nð0; s2 Þ
Least Squares Estimates in MR Model
Least squares estimates b1, b2, . . . , bK minimize
Sðb1, b2, . . . , bKÞ ¼ åðyi b1 b2xi2 bKxiKÞ2
Estimated Error Variance and Estimator
Standard Errors
s
^2 ¼
å ^e2i
NK
seðbk Þ ¼
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
b
varðb
kÞ
Hypothesis Tests and Interval Estimates for Single Parameters
bk bk
tðNKÞ
Use t-distribution t ¼
seðbk Þ
t-test for More than One Parameter
H0 : b2 þ cb3 ¼ a
b2 þ cb3 a
tðNKÞ
seðb2 þ cb3 Þ
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
2b
b
b
seðb2 þ cb3 Þ ¼ varðb
2 Þ þ c varðb3 Þ þ 2c covðb2 ; b3 Þ
t¼
When H0 is true
Joint F-tests
ðSSER SSEU Þ=J
SSEU =ðN KÞ
To test the overall significance of the model the null and alternative
hypotheses and F statistic are
F¼
H0 : b2 ¼ 0; b3 ¼ 0; : : : ; bK ¼ 0
H1 : at least one of the bk is nonzero
F¼
ðSST SSEÞ=ðK 1Þ
SSE=ðN KÞ
RESET: A Specification Test
yi ¼ b1 þ b2 xi2 þ b3 xi3 þ ei
yi ¼ b1 þ b2 xi2 þ b3 xi3 þ g1 ^y2i þ ei ;
^yi ¼ b1 þ b2 xi2 þ b3 xi3
H0 : g1 ¼ 0
yi ¼ b1 þ b2 xi2 þ b3 xi3 þ g1 ^y2i þ g2 ^y3i þ ei ;
H0 : g1 ¼ g2 ¼ 0
Model Selection
AIC ¼ ln(SSE=N) þ 2K=N
SC ¼ ln(SSE=N) þ K ln(N)=N
Collinearity and Omitted Variables
yi ¼ b1 þ b2 xi2 þ b3 xi3 þ ei
s2
varðb2 Þ ¼
2
ð1 r23 Þ å ðxi2 x2 Þ2
b
covðx2 ; x3 Þ
When x3 is omitted; biasðb2 Þ ¼ Eðb2 Þ b2 ¼ b3
b
varðx2 Þ
Heteroskedasticity
var(yi) ¼ var(ei) ¼ si2
General variance function
s2i ¼ expða1 þ a2 zi2 þ þ aS ziS Þ
Breusch-Pagan and White Tests for H0: a2 ¼ a3 ¼ ¼ aS ¼ 0
When H0 is true x2 ¼ N R2 x2ðS1Þ
Goldfeld-Quandt test for H0 : s2M ¼ s2R versus H1 : s2M 6¼ s2R
When H0 is true F ¼ s
^ 2M =^
s2R FðNM KM ;NR KR Þ
Transformed model for varðei Þ ¼ s2i ¼ s2 xi
pffiffiffiffi
pffiffiffiffi
pffiffiffiffi
pffiffiffiffi
yi = xi ¼ b1 ð1= xi Þ þ b2 ðxi = xi Þ þ ei = xi
Estimating the variance function
¼
Finite distributed lag model
yt ¼ a þ b0 xt þ b1 xt1 þ b2 xt2 þ þ bq xtq þ vt
Correlogram
rk ¼ å ðyt yÞðytk yÞ= å ðyt yÞ2
pffiffiffiffi
For H0 : rk ¼ 0;
z ¼ T rk Nð0; 1Þ
LM test
yt ¼ b1 þ b2 xt þ r^et1 þ ^vt
Test H 0 : r ¼ 0 with t-test
^et ¼ g1 þ g2 xt þ r^et1 þ ^vt
Test using LM ¼ T R2
yt ¼ b1 þ b2 xt þ et
AR(1) error
To test J joint hypotheses,
lnð^e2i Þ
Regression with Stationary Time Series Variables
lnðs2i Þ
Grouped data
varðei Þ ¼ s2i ¼
þ vi ¼ a1 þ a2 zi2 þ þ aS ziS þ vi
(
s2M i ¼ 1; 2; . . . ; NM
s2R i ¼ 1; 2; . . . ; NR
Transformed model for feasible generalized least squares
.pffiffiffiffiffi
.pffiffiffiffiffi
.pffiffiffiffiffi
.pffiffiffiffiffi
s
^ i ¼ b1 1
s
^ i þ ei
s
^i
s
^ i þ b2 xi
yi
et ¼ ret1 þ vt
Nonlinear least squares estimation
yt ¼ b1 ð1 rÞ þ b2 xt þ ryt1 b2 rxt1 þ vt
ARDL(p, q) model
yt ¼ d þ d0 xt þ dl xt1 þ þ dq xtq þ ul yt1
þ þ up ytp þ vt
AR(p) forecasting model
yt ¼ d þ ul yt1 þ u2 yt2 þ þ up ytp þ vt
Exponential smoothing ^yt ¼ ayt1 þ ð1 aÞ^yt1
Multiplier analysis
d0 þ d1 L þ d2 L2 þ þ dq Lq ¼ ð1 u1 L u2 L2 up Lp Þ
ðb0 þ b1 L þ b2 L2 þ Þ
Unit Roots and Cointegration
Unit Root Test for Stationarity: Null hypothesis:
H0 : g ¼ 0
Dickey-Fuller Test 1 (no constant and no trend):
Dyt ¼ gyt1 þ vt
Dickey-Fuller Test 2 (with constant but no trend):
Dyt ¼ a þ gyt1 þ vt
Dickey-Fuller Test 3 (with constant and with trend):
Dyt ¼ a þ gyt1 þ lt þ vt
Augmented Dickey-Fuller Tests:
m
Dyt ¼ a þ gyt1 þ å as Dyts þ vt
s¼1
Test for cointegration
D^et ¼ g^et1 þ vt
Random walk:
yt ¼ yt1 þ vt
Random walk with drift:
yt ¼ a þ yt1 þ vt
Random walk model with drift and time trend:
yt ¼ a þ dt þ yt1 þ vt
Panel Data
Pooled least squares regression
yit ¼ b1 þ b2 x2it þ b3 x3it þ eit
Cluster robust standard errors cov(eit, eis) ¼ cts
Fixed effects model
b1i not random
yit ¼ b1i þ b2 x2it þ b3 x3it þ eit
yit yi ¼ b2 ðx2it x2i Þ þ b3 ðx3it x3i Þ þ ðeit ei Þ
Random effects model
yit ¼ b1i þ b2 x2it þ b3 x3it þ eit
bit ¼ b1 þ ui random
yit ayi ¼ b1 ð1 aÞ þ b2 ðx2it ax2i Þ þ b3 ðx3it ax3i Þ þ vit
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
a ¼ 1 se
Ts2u þ s2e
Hausman test
h
i1=2
b
b
t ¼ ðbFE;k bRE;k Þ varðb
FE;k Þ varðbRE;k Þ