Evaluating starting point output gap estimate errors
James Twaddle
Editor’s note
The previous paper shows that bias in our estimate of the starting point for the output gap
may be a significant driver of the bias in our CPI inflation forecasts. We therefore decided to
investigate more closely what was driving this error – our estimates of total growth, or of
trend growth, or Statistics New Zealand revisions to the GDP series?
Executive summary
Previous work has showed that we have tended to underestimate the starting point output
gap and that this has likely been a significant contributor to our CPI inflation forecast
errors. Our estimates of GDP growth are, if anything, biased towards over-prediction. In
this paper, we look at whether data revisions are to blame, or whether the problem is with
the ways we use the data to calculate an output gap.
In this analysis we are able to examine only errors post-1997, since this is when we began
to formally use the output gap concept in our macroeconomic model. Note that we have
previously concluded (eg our 1999 business cycle review) that we over-estimated potential
output growth in the mid-1990s.
We find that both data revisions and how we use that data have played a significant part in
our underestimating the output gap starting point bias, with the latter being a larger cause.
The bias coming from how we use the data has both technical elements (the end-point
problem) and judgemental aspects.
In terms of policymaking, we probably cannot do much about data revisions and there will
always be endpoint problems with a two-sided filter. This underscores the importance of
putting considerable thought into our estimates of starting point potential output, and
looking at a range of indicators of resource strain when formulating policy.
1
Introduction
Recent work has shown that our forecasts of the starting point output gap are biased towards
underestimation by around half a per cent of GDP (“Analysis of bias and RMSE in forecasts
of key variables”). Further work has indicated that output gap forecast errors are likely to be
a material source of our inflation forecast bias (“Attributing inflation forecast bias to other
forecast variables using FPS”). This paper looks at our starting point output gap estimates to
try to uncover the sources of our output gap forecast bias.
This analysis centres on the period from 1997, when we began using the FPS framework to
bring together our projections. This time period is important for two reasons. First, the
output gap took on increased prominence in our projections because of the large role FPS
accords it. Second, this period coincides with our use of the Multivariate (MV) filter to
calculate our estimates of starting-point potential output.
2
Background
The MV filter as we use it has three main components:
•
At its heart it has a Hodrick-Prescott (HP) filter. This is a two-sided filter that creates
a smooth path for potential output based on the idea that persistent movements in
output are attributable to supply shocks, while more temporary/transitory movements
are caused by variation in aggregate demand. As with all filters that rely on past and
future observations, it suffers from an endpoint problem at the end of the sample
because it lacks future data to guide it as to the persistence characteristics of output
movements.
•
It also uses what we call ‘conditioning relationships’ to help inform our estimates of
potential output. These conditioning relationships use other indicators of resource
strain to guide our output gap estimates (eg capacity utilisation). The conditioning
relationships currently have a lower weighting than actual GDP outturns.
•
The final component is what is referred to as a stiffener. The stiffener allows a
constant assumed growth rate of potential output to influence our potential output
estimates over the last few years of estimation. This serves two purposes:
(i) it mitigates (but does not eliminate) the endpoint problem by stopping the filter
following the data too closely at the end of the sample.
(ii) it allows some judgemental adjustment of the output gap so that information
outside the MV filter can be used.
For a majority of the sample period the stiffener growth rate has been set to 3 per cent.
Occasionally it has been adjusted in response to, for example, net migration data.
In using the MV filter, we treat the monitoring quarter output estimates as actual data, and
then estimate the output gap over the period inclusive of the monitoring quarters. As such,
FPS considers the starting point output gap to be that at the end of the monitoring quarters, ie
two quarters after the last official GDP release. In terms of the output gap forecast bias
calculated in “Analysis of bias and RMSE in forecasts of key variables”, this is the two
quarter ahead forecast of the output gap (reproduced in table 1 for the period 1994 to 2002),
and is the one that this analysis will concentrate on.
Bias in output gap forecasts beyond the starting point output gap is not directly considered
here. Output gap forecast bias at longer horizons could potentially be a function of a number
of factors, one of which is the persistence in the model carrying bias in starting point output
gap estimates through to forecasts of the output gap at longer horizons. Given that, if
anything, we over-estimate quarterly GDP growth, the large bias towards under-estimation of
the output gap starting point is likely largely responsible for the negative bias in subsequent
quarters.1
Table One. Output Gap Forecast Bias, 1994:4-2002:1.
1
2
3
4
5
6
7
8
Output gap
Bias
-0.42*
-0.54** -0.58** -0.56** -0.47
-0.34
-0.18
0.00
RMSE
0.97
1.06
1.19
1.28
1.25
1
1.13
1.11
1.16
In theory, other potential causes of output gap forecast bias beyond the starting point error could
include asymmetric shocks, bias in judgements, and model mis-specification.
Notes to table:
The statistical significance of the bias is indicated with asterisks:
*
= significant at the 10 per cent level
**
= significant at 5 per cent
*** = significant at 1 per cent
See “Analysis of bias and RMSE in forecasts of key variables” for full details on how these statistics were
calculated.
3
Analysis
We can construct a series of real-time starting point output gap estimates. At each point in
time this series equals what we thought the starting point output gap was at the time. Figure 1
shows this series from our projections. Figure 1 also shows an approximation of this realtime output gap, calculated by using a slightly adapted version of the MV filter from the
August 2002 projection on real-time GDP data. The advantage of this approximation is that it
allows us to do decompositions of our output gap forecast bias.2 From here on we will treat
our approximation as the real-time output gap for ease of analysis, dropping any reference to
it being an approximation.
Figure 1
Real-time output gaps
% of GDP
% of GDP
2
2
Published Real-time
Real-time Approximation
1
1
0
0
-1
-1
-2
-2
-3
-3
-4
-4
Apr-97
Oct-97
Apr-98
Oct-98
Apr-99
Oct-99
Apr-00
Oct-00
Apr-01
Oct-01
In figure 2 we compare the real-time output gap over the past five years to our current best
estimate assessment of the output gap from the August 2002 MPS (‘final’). For most of the
sample period the real-time output gap has been indicating less strain on resources than
hindsight indicates there was. The difference between the series is smaller for more recent
observations; recent output gap estimates have had less chance to be proven wrong.
2
The reasons the lines are different are that we impose the Phillips Curve conditioning relationship
coefficients from the latest projection instead of estimating them, and only approximate the weight on
the stiffener and the stiffener growth rate. Another reason for the difference is that we assume we
correctly forecast monitoring quarter capacity utilisation. In fact we have tended to underestimate it,
but earlier work has indicated this is unlikely to be material.
Figure 2
Output gaps
% of GDP
3
% of GDP
3
Final
2
Real-time
2
1
1
0
0
-1
-1
-2
-2
-3
-3
-4
-4
Apr-97
Oct-97
Apr-98
Oct-98
Apr-99
Oct-99
Apr-00
Oct-00
Apr-01
Oct-01
On average the difference between the real-time and final output gap estimates is over half a
percentage point. The difference is largest over 1999, where cumulatively over the calendar
year the difference in output gaps is over four per cent.
We can decompose the difference in the real-time vs. final output gaps into the two
categories. The first relates to data revisions and not having the final vintage of data (that is,
not having the latest, revised output data). We call this ‘data issues’. The second group
consists of issues relating to the endpoint problem and the stiffener, which we will call ‘other
issues’.
The impact of data revisions on the level of GDP is illustrated in figure 3 (taken from “GDP
forecast errors”). There have been large revisions to GDP, particularly with the move from
calculating GDP by fixed weighting to using chain-linked data.3
3
It is important to remember that both GDP and the output gap are levels concepts. Large revisions to
the growth rate of GDP may not necessarily lead to large revisions to the output gap (especially for
flexible filters like the HP and MV filters). This follows from the fact that the filter follows the data,
forcing some of the GDP revision into potential output. See Nelson and Nikolov (2001) pp10-11 for a
good explanation of this point.
Figure 3
Revisions to production GDP (from “GDP forecast errors”)
Historical GDPP figures
1350
1300
GDPP index (Base: 1990=1000)
1250
1200
1150
1100
1050
1000
950
Jan-01
Jul-00
May-01
Jan-00
Jul-99
Jul-98
Jan-99
Jul-97
May-00
May-99
Jan-98
May-98
Jan-97
Jul-96
Jun-97
Jan-96
Jul-95
Jun-96
Jan-95
Jun-95
Jul-94
Jul-93
Jun-94
Jan-94
Jan-93
Jan-92
Jul-91
Jan-91
Jul-90
Jul-89
Jan-90
Jan-89
Jul-92
May-93
900
Date
We can separate the ‘data’ and ‘other’ issues by feeding the latest, revised, GDP data into the
MV filter in place of the unrevised and monitoring quarter data. We do not feed in more data
than was available at the time – that is, we do not feed in information relating to quarters in
the future. This gives us an estimate of the impact of not having the ‘best’ data at the time of
our initial assessment of the output gap, with the residual source of errors being grouped as
other issues (figure 4). It is worth reiterating that the bias reduces for the more recent
forecasts because our estimates have had less time to be proven wrong (so RMSEs in table 2
are likely to be biased downwards due to the period analysed being so recent). These recent
forecast errors will be revised with future revisions to GDP and as further data allows the
filter to make a better distinction between demand and supply shocks.
Figure 4
Contributors to forecast bias
%
%
3
3
Data Issues
Other Issues
2
2
1
1
0
0
-1
-1
-2
-2
-3
-3
Apr-97
Oct-97
Apr-98
Oct-98
Apr-99
Oct-99
Apr-00
Oct-00
Apr-01
Oct-01
Figure 4 suggests that what we are calling ‘other issues’ caused us to consistently
underestimate the strain on resources. Data issues have also caused underestimation on
average, but with a considerable amount of volatility. Data issues seem to have been
particularly important over 1999, when the starting point output gap error was largest. At this
time both ‘data’ and ‘other’ issues were contributing to an underestimation of the output gap
starting point. Table 2 decomposes the forecast bias numerically.
Table Two: Output Gap Bias
Mean
Standard Deviation
Maximum
Minimum
RMSE
Data Issues
-0.14
0.97
2.01
-1.81
0.95
Other Issues
-0.59
0.64
0.25
-1.83
0.86
Overall
-0.73
0.94
0.71
-2.62
1.17
Notes to table:
Sample period is June 1997 to March 2002
Data revisions and monitoring quarter errors are a material cause of output gap forecast
errors. Practically all of the ‘data issues’ related output gap bias relates to not having the
latest vintage of data rather than from poor monitoring quarter forecasts. If we replace the
monitoring quarter GDP numbers with the first GDP outturns the bias is almost unchanged,
though the standard deviation falls. This matches up with earlier findings that our monitoring
quarter GDP forecasts are unbiased (see Ranchhod 2002). In other words, data revisions are
driving the bias coming from data issues.
Other issues - those relating to the endpoint problem and the stiffener - are a slightly larger
cause of starting point output gap estimate errors. That finding is consistent with work done
by van Norden and Orphanides (2002) using a range of filters for US data. They find that
“although important, the ex-post revisions of published data is not the primary source of
revisions in output gap estimates…the bulk of the problem is due to the pervasive
unreliability of end-of-sample estimates”. Though generally smaller in absolute magnitude
than errors due to data issues, the contribution of other issues to the bias has been consistently
one-sided, making them a key contributor to the bias.
There are two potential causes for the bias coming from the ‘other issues’ category. The first
is the technical endpoint problem. We lack future data when compiling real-time output gap
estimates, so when that future information becomes available our output gap estimates will
change because the filter can get a better steer of the temporary and persistent movements in
output. We use the stiffener to help us mitigate this problem, though it will never completely
eliminate it. The earlier part of the sample period, when we had less weight on the stiffener,
is probably more affected by the endpoint problem than later, when the weight was increased.
However, we have not used the stiffener solely as a technical device to reduce the extent of
the endpoint problem over the past five years. The stiffener has also been used as a means of
introducing judgement into our estimates of the output gap, which is the second cause of bias
in the ‘other issues’ category. For example, if we felt our estimates of the output gap were
higher than we felt comfortable with (or other evidence had suggested at the time), but our
monitoring quarter GDP estimates had seemed about right, we could have increased the
potential output growth rate used in the stiffener. Increasing this growth rate allocates more
of the end-of-sample output growth to potential output and less to the output gap (the reverse
is also true, whereby the stiffener can be used to increase our output gap estimates and reduce
our potential output estimates at the end of the sample).
On occasion over the period 1997 to 2002 we have set the stiffener to a value other than 3 per
cent. On these occasions we can think of the stiffener as largely a judgement. When the
stiffener has been left at 3 per cent (as it was for most of the sample) we cannot ex-post be
sure how much of this choice was a judgement, and how much was due to just using the
stiffener as a technical device.
Figure 5 shows that over most of the FPS sample period (June 1997 to September 2002) the
MV filter estimate of potential output (taken from the latest projection) is below a growth rate
of 3 per cent (which was used in the stiffener for around two thirds of the sample). Therefore
3 per cent appears to have been a slightly over-optimistic prediction for the average rate of
potential output growth. Setting the stiffener to a lower growth rate over the sample period
could potentially have reduced the extent of our output gap forecast bias. However, using
hindsight to suggest that setting the stiffener to 2.75 per cent may have reduced output gap
forecast bias is not the same thing as making that determination in real-time.
Figure 5
Annual growth rate of potential output
MV filter estimate
%
%
5
5
4
4
3
3
2
2
1
1
0
1991
0
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
Even if we set the stiffener to the unknown ex-post average potential output growth rate in the
future, we would still have revisions to our output gap estimates (aside from GDP data
revisions) because of the endpoint problem and because potential output does not grow at a
constant rate. Rather, it tends to cycle around a longer-run trend.
The short period that has been examined here (June 1997 to September 2002) covers a period
where our latest estimates indicate that potential output grew at a slower rate than the mid1990s. Over a longer sample the bias in our estimates of potential output coming in from the
stiffener and the endpoint problem may be different, so it would be too simplistic to take from
these results that we should immediately and permanently lower the potential output growth
rate built into the stiffener.
4
Summary
Our starting point output gap estimate errors are due both to data revisions and issues relating
to the endpoint problem and stiffener. The endpoint problem, and the stiffener that we use to
mitigate that problem, are a somewhat larger cause of starting point output gap estimate bias.
We cannot be definite about classifying this later category as simply a technical filter issue
because the stiffener has also been used as a device to introduce judgement into our output
gap estimates. What we can say is that data revisions are only part of the story about our
output gap forecast bias.
Previous forecast errors work has suggested that the bias in our output gap starting point
estimates does matter for our inflation forecast bias. Unfortunately there are no quick
solutions. There is no ‘fix’ for the issue of ongoing GDP revisions by Statistics New Zealand
and there will always be endpoint problems with a two-sided filter. This underscores the
importance of putting considerable thought into our estimates of starting point potential
output and not taking any single estimate of the output gap as the ‘true’ output gap to
formulate policy.
References
McCaw, S (2002), “Bias in forecasts of key variables,” Reserve Bank of New Zealand
Memorandum.
Nelson, E and K Nikolov (2002), “UK inflation in the 1970s and 1980s: the role of output gap
mismeasurement,” Bank of England Working Paper no 148.
Orphanides, A and S Van Norden (2002), “The reliability of output gap estimates in real
time,”. Forthcoming in the Review of Economics and Statistics. An earlier version of this
paper is available as a Federal Reserve Board of Governors’ working paper.
Ranchhod, S (2002), “GDP forecast errors,” Reserve Bank of New Zealand Memorandum.
Twaddle, J (2002), “Attributing inflation forecast bias using FPS,” Reserve Bank of New
Zealand Memorandum.