Econ 427 lecture 24 slides
Forecast Evaluation
Byron Gangnes
Review: Linear Projection
•
•
Consider the h-step ahead forecast for the
following general covariance stationary model:
Wold (MA) representation for the series of
interest is:
yt     t  b1 t 1  b2 t 2  ...
 t WN (0, 2 )
•
The h-step-ahead linear least squares forecast is
obtained by writing out the series for period
T+h,
yT h    T h  b1T h1  ...  bhT  bh1T 1  ...
Byron Gangnes
Linear Projection
•
and then projecting on the info set available
today (time T). (See Ch. 9) This gives:
yT h ,T    bh T  bh 1 T 1  ...
•
The h-step-ahead forecast error is then:
eT  h ,T  yT  h  yT  h ,T   T  h  b1 T  h 1  ...  bh 1 T 1
•
This is an MA(h-1)
And the forecast error variance is:
h 1

2
2
2
 h   1   bi 
 i 1 
Byron Gangnes
Optimal Forecasts
•
Four key properties follow:
–
–
Optimal forecasts are unbiased.
Optimal forecasts have 1-step-ahead forecasts errors
that are white noise
Optimal forecasts have h-step-ahead forecasts that
are at most MA(h-1)
–
•
–
We saw above that optimal forecasts will have MA(h-1)
errors.
Optimal forecasts have h-step-ahead errors with
variances that are non-decreasing in h and that
converge to the unconditional variance of the process
(series).
Byron Gangnes
Test of forecast bias
•
To test for forecast bias, regress the forecast errors on a
constant:
et  h,t  0  ut
IsΚ0  0?
•
But we need to be sure that residuals from this
regression are white noise. So we select an ARMA
model to do so and then estimate:
 
et  h,t  0  ARMA p,q
IsΚ0  0?
•
See the discussion in the book of how we can test the
other characteristics of optimal forecasts.
Byron Gangnes
Mincer-Zarnowitz regression
•
A key overarching property: forecast errors
should be unforecastable on the basis of info
avail at the time the forecast is made.
Testing this with a Mincer-Zarnowitz
regression:
yt  h   0  1 yt  h ,t  ut
•
•
Then use an F-test to test whether β0=0 and
β1=1.
–
Note that in this case , yt+h = yt+h,t+ ut
they only differ by unforecastable error.
Byron Gangnes
Statistical tests of forecast accuracy
•
•
At the beginning of the term we looked at
various loss functions such as MSE, RMSE, etc.
We may want to test whether the “expected loss
differential” between two forecasts a and b is
significantly diff from zero:
E (dt )  E  L  eta h,t   E  L(etbh,t )   0
•
In the book (pp. 262-263 ) they show a
nonparametric test you can do to test whether
one forecast is statistically smaller loss than the
other.
Byron Gangnes
Statistical tests of forecast accuracy
•
How can we test whether the difference d is
significant using and OLS regression?
Regress d on a constant and look at the t-stat on
the constant term, β
•
dt    ARMA( p, q)
–
Note that here you have to model the error (which
won’t nec. be white noise—why not?—in order to
get an unbiased est. of c term.
Byron Gangnes
•
•
Forecasting encompassing and
forecast combination
It may be the case that one forecast
encompasses all of the relevant info in
another candidate forecast. In that case
your forecast cannot be improved by using
the second forecast.
But it is frequently the case that
alternative forecasts do not encompass
each other and so combining them can
yield improved forecast performance.
Byron Gangnes
Forecast Combination
•
•
How can we do this?
Regression-based method of forecast
combination. Run the regression:
yt h  0  1 ytah,t  2 ytbh,t   t h,t
–
–
The regression equation basically lets the data tell us
what the optimal weights are to put on the two
forecasts.
Normally you will want to allow for ARMA(p,q)
error terms since we know that even optimal
forecasts can have MA(h-1) errors and non-optimal
forecasts could have other dynamics.
Byron Gangnes
Forecast Combination
•
These methods can be applied to any
forecasts. They don’t have to be formal
model-based forecasts.
–
•
See the application at the end of the chapter.
Go through it carefully to understand the
procedure they follow to compare and
combine the two forecasts and the resulting
gains. How big are they?
We’ll look at an example next time.
Byron Gangnes