advertisement

Evaluating starting point output gap estimate errors James Twaddle Editor’s note The previous paper shows that bias in our estimate of the starting point for the output gap may be a significant driver of the bias in our CPI inflation forecasts. We therefore decided to investigate more closely what was driving this error – our estimates of total growth, or of trend growth, or Statistics New Zealand revisions to the GDP series? Executive summary Previous work has showed that we have tended to underestimate the starting point output gap and that this has likely been a significant contributor to our CPI inflation forecast errors. Our estimates of GDP growth are, if anything, biased towards over-prediction. In this paper, we look at whether data revisions are to blame, or whether the problem is with the ways we use the data to calculate an output gap. In this analysis we are able to examine only errors post-1997, since this is when we began to formally use the output gap concept in our macroeconomic model. Note that we have previously concluded (eg our 1999 business cycle review) that we over-estimated potential output growth in the mid-1990s. We find that both data revisions and how we use that data have played a significant part in our underestimating the output gap starting point bias, with the latter being a larger cause. The bias coming from how we use the data has both technical elements (the end-point problem) and judgemental aspects. In terms of policymaking, we probably cannot do much about data revisions and there will always be endpoint problems with a two-sided filter. This underscores the importance of putting considerable thought into our estimates of starting point potential output, and looking at a range of indicators of resource strain when formulating policy. 1 Introduction Recent work has shown that our forecasts of the starting point output gap are biased towards underestimation by around half a per cent of GDP (“Analysis of bias and RMSE in forecasts of key variables”). Further work has indicated that output gap forecast errors are likely to be a material source of our inflation forecast bias (“Attributing inflation forecast bias to other forecast variables using FPS”). This paper looks at our starting point output gap estimates to try to uncover the sources of our output gap forecast bias. This analysis centres on the period from 1997, when we began using the FPS framework to bring together our projections. This time period is important for two reasons. First, the output gap took on increased prominence in our projections because of the large role FPS accords it. Second, this period coincides with our use of the Multivariate (MV) filter to calculate our estimates of starting-point potential output. 2 Background The MV filter as we use it has three main components: • At its heart it has a Hodrick-Prescott (HP) filter. This is a two-sided filter that creates a smooth path for potential output based on the idea that persistent movements in output are attributable to supply shocks, while more temporary/transitory movements are caused by variation in aggregate demand. As with all filters that rely on past and future observations, it suffers from an endpoint problem at the end of the sample because it lacks future data to guide it as to the persistence characteristics of output movements. • It also uses what we call ‘conditioning relationships’ to help inform our estimates of potential output. These conditioning relationships use other indicators of resource strain to guide our output gap estimates (eg capacity utilisation). The conditioning relationships currently have a lower weighting than actual GDP outturns. • The final component is what is referred to as a stiffener. The stiffener allows a constant assumed growth rate of potential output to influence our potential output estimates over the last few years of estimation. This serves two purposes: (i) it mitigates (but does not eliminate) the endpoint problem by stopping the filter following the data too closely at the end of the sample. (ii) it allows some judgemental adjustment of the output gap so that information outside the MV filter can be used. For a majority of the sample period the stiffener growth rate has been set to 3 per cent. Occasionally it has been adjusted in response to, for example, net migration data. In using the MV filter, we treat the monitoring quarter output estimates as actual data, and then estimate the output gap over the period inclusive of the monitoring quarters. As such, FPS considers the starting point output gap to be that at the end of the monitoring quarters, ie two quarters after the last official GDP release. In terms of the output gap forecast bias calculated in “Analysis of bias and RMSE in forecasts of key variables”, this is the two quarter ahead forecast of the output gap (reproduced in table 1 for the period 1994 to 2002), and is the one that this analysis will concentrate on. Bias in output gap forecasts beyond the starting point output gap is not directly considered here. Output gap forecast bias at longer horizons could potentially be a function of a number of factors, one of which is the persistence in the model carrying bias in starting point output gap estimates through to forecasts of the output gap at longer horizons. Given that, if anything, we over-estimate quarterly GDP growth, the large bias towards under-estimation of the output gap starting point is likely largely responsible for the negative bias in subsequent quarters.1 Table One. Output Gap Forecast Bias, 1994:4-2002:1. 1 2 3 4 5 6 7 8 Output gap Bias -0.42* -0.54** -0.58** -0.56** -0.47 -0.34 -0.18 0.00 RMSE 0.97 1.06 1.19 1.28 1.25 1 1.13 1.11 1.16 In theory, other potential causes of output gap forecast bias beyond the starting point error could include asymmetric shocks, bias in judgements, and model mis-specification. Notes to table: The statistical significance of the bias is indicated with asterisks: * = significant at the 10 per cent level ** = significant at 5 per cent *** = significant at 1 per cent See “Analysis of bias and RMSE in forecasts of key variables” for full details on how these statistics were calculated. 3 Analysis We can construct a series of real-time starting point output gap estimates. At each point in time this series equals what we thought the starting point output gap was at the time. Figure 1 shows this series from our projections. Figure 1 also shows an approximation of this realtime output gap, calculated by using a slightly adapted version of the MV filter from the August 2002 projection on real-time GDP data. The advantage of this approximation is that it allows us to do decompositions of our output gap forecast bias.2 From here on we will treat our approximation as the real-time output gap for ease of analysis, dropping any reference to it being an approximation. Figure 1 Real-time output gaps % of GDP % of GDP 2 2 Published Real-time Real-time Approximation 1 1 0 0 -1 -1 -2 -2 -3 -3 -4 -4 Apr-97 Oct-97 Apr-98 Oct-98 Apr-99 Oct-99 Apr-00 Oct-00 Apr-01 Oct-01 In figure 2 we compare the real-time output gap over the past five years to our current best estimate assessment of the output gap from the August 2002 MPS (‘final’). For most of the sample period the real-time output gap has been indicating less strain on resources than hindsight indicates there was. The difference between the series is smaller for more recent observations; recent output gap estimates have had less chance to be proven wrong. 2 The reasons the lines are different are that we impose the Phillips Curve conditioning relationship coefficients from the latest projection instead of estimating them, and only approximate the weight on the stiffener and the stiffener growth rate. Another reason for the difference is that we assume we correctly forecast monitoring quarter capacity utilisation. In fact we have tended to underestimate it, but earlier work has indicated this is unlikely to be material. Figure 2 Output gaps % of GDP 3 % of GDP 3 Final 2 Real-time 2 1 1 0 0 -1 -1 -2 -2 -3 -3 -4 -4 Apr-97 Oct-97 Apr-98 Oct-98 Apr-99 Oct-99 Apr-00 Oct-00 Apr-01 Oct-01 On average the difference between the real-time and final output gap estimates is over half a percentage point. The difference is largest over 1999, where cumulatively over the calendar year the difference in output gaps is over four per cent. We can decompose the difference in the real-time vs. final output gaps into the two categories. The first relates to data revisions and not having the final vintage of data (that is, not having the latest, revised output data). We call this ‘data issues’. The second group consists of issues relating to the endpoint problem and the stiffener, which we will call ‘other issues’. The impact of data revisions on the level of GDP is illustrated in figure 3 (taken from “GDP forecast errors”). There have been large revisions to GDP, particularly with the move from calculating GDP by fixed weighting to using chain-linked data.3 3 It is important to remember that both GDP and the output gap are levels concepts. Large revisions to the growth rate of GDP may not necessarily lead to large revisions to the output gap (especially for flexible filters like the HP and MV filters). This follows from the fact that the filter follows the data, forcing some of the GDP revision into potential output. See Nelson and Nikolov (2001) pp10-11 for a good explanation of this point. Figure 3 Revisions to production GDP (from “GDP forecast errors”) Historical GDPP figures 1350 1300 GDPP index (Base: 1990=1000) 1250 1200 1150 1100 1050 1000 950 Jan-01 Jul-00 May-01 Jan-00 Jul-99 Jul-98 Jan-99 Jul-97 May-00 May-99 Jan-98 May-98 Jan-97 Jul-96 Jun-97 Jan-96 Jul-95 Jun-96 Jan-95 Jun-95 Jul-94 Jul-93 Jun-94 Jan-94 Jan-93 Jan-92 Jul-91 Jan-91 Jul-90 Jul-89 Jan-90 Jan-89 Jul-92 May-93 900 Date We can separate the ‘data’ and ‘other’ issues by feeding the latest, revised, GDP data into the MV filter in place of the unrevised and monitoring quarter data. We do not feed in more data than was available at the time – that is, we do not feed in information relating to quarters in the future. This gives us an estimate of the impact of not having the ‘best’ data at the time of our initial assessment of the output gap, with the residual source of errors being grouped as other issues (figure 4). It is worth reiterating that the bias reduces for the more recent forecasts because our estimates have had less time to be proven wrong (so RMSEs in table 2 are likely to be biased downwards due to the period analysed being so recent). These recent forecast errors will be revised with future revisions to GDP and as further data allows the filter to make a better distinction between demand and supply shocks. Figure 4 Contributors to forecast bias % % 3 3 Data Issues Other Issues 2 2 1 1 0 0 -1 -1 -2 -2 -3 -3 Apr-97 Oct-97 Apr-98 Oct-98 Apr-99 Oct-99 Apr-00 Oct-00 Apr-01 Oct-01 Figure 4 suggests that what we are calling ‘other issues’ caused us to consistently underestimate the strain on resources. Data issues have also caused underestimation on average, but with a considerable amount of volatility. Data issues seem to have been particularly important over 1999, when the starting point output gap error was largest. At this time both ‘data’ and ‘other’ issues were contributing to an underestimation of the output gap starting point. Table 2 decomposes the forecast bias numerically. Table Two: Output Gap Bias Mean Standard Deviation Maximum Minimum RMSE Data Issues -0.14 0.97 2.01 -1.81 0.95 Other Issues -0.59 0.64 0.25 -1.83 0.86 Overall -0.73 0.94 0.71 -2.62 1.17 Notes to table: Sample period is June 1997 to March 2002 Data revisions and monitoring quarter errors are a material cause of output gap forecast errors. Practically all of the ‘data issues’ related output gap bias relates to not having the latest vintage of data rather than from poor monitoring quarter forecasts. If we replace the monitoring quarter GDP numbers with the first GDP outturns the bias is almost unchanged, though the standard deviation falls. This matches up with earlier findings that our monitoring quarter GDP forecasts are unbiased (see Ranchhod 2002). In other words, data revisions are driving the bias coming from data issues. Other issues - those relating to the endpoint problem and the stiffener - are a slightly larger cause of starting point output gap estimate errors. That finding is consistent with work done by van Norden and Orphanides (2002) using a range of filters for US data. They find that “although important, the ex-post revisions of published data is not the primary source of revisions in output gap estimates…the bulk of the problem is due to the pervasive unreliability of end-of-sample estimates”. Though generally smaller in absolute magnitude than errors due to data issues, the contribution of other issues to the bias has been consistently one-sided, making them a key contributor to the bias. There are two potential causes for the bias coming from the ‘other issues’ category. The first is the technical endpoint problem. We lack future data when compiling real-time output gap estimates, so when that future information becomes available our output gap estimates will change because the filter can get a better steer of the temporary and persistent movements in output. We use the stiffener to help us mitigate this problem, though it will never completely eliminate it. The earlier part of the sample period, when we had less weight on the stiffener, is probably more affected by the endpoint problem than later, when the weight was increased. However, we have not used the stiffener solely as a technical device to reduce the extent of the endpoint problem over the past five years. The stiffener has also been used as a means of introducing judgement into our estimates of the output gap, which is the second cause of bias in the ‘other issues’ category. For example, if we felt our estimates of the output gap were higher than we felt comfortable with (or other evidence had suggested at the time), but our monitoring quarter GDP estimates had seemed about right, we could have increased the potential output growth rate used in the stiffener. Increasing this growth rate allocates more of the end-of-sample output growth to potential output and less to the output gap (the reverse is also true, whereby the stiffener can be used to increase our output gap estimates and reduce our potential output estimates at the end of the sample). On occasion over the period 1997 to 2002 we have set the stiffener to a value other than 3 per cent. On these occasions we can think of the stiffener as largely a judgement. When the stiffener has been left at 3 per cent (as it was for most of the sample) we cannot ex-post be sure how much of this choice was a judgement, and how much was due to just using the stiffener as a technical device. Figure 5 shows that over most of the FPS sample period (June 1997 to September 2002) the MV filter estimate of potential output (taken from the latest projection) is below a growth rate of 3 per cent (which was used in the stiffener for around two thirds of the sample). Therefore 3 per cent appears to have been a slightly over-optimistic prediction for the average rate of potential output growth. Setting the stiffener to a lower growth rate over the sample period could potentially have reduced the extent of our output gap forecast bias. However, using hindsight to suggest that setting the stiffener to 2.75 per cent may have reduced output gap forecast bias is not the same thing as making that determination in real-time. Figure 5 Annual growth rate of potential output MV filter estimate % % 5 5 4 4 3 3 2 2 1 1 0 1991 0 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 Even if we set the stiffener to the unknown ex-post average potential output growth rate in the future, we would still have revisions to our output gap estimates (aside from GDP data revisions) because of the endpoint problem and because potential output does not grow at a constant rate. Rather, it tends to cycle around a longer-run trend. The short period that has been examined here (June 1997 to September 2002) covers a period where our latest estimates indicate that potential output grew at a slower rate than the mid1990s. Over a longer sample the bias in our estimates of potential output coming in from the stiffener and the endpoint problem may be different, so it would be too simplistic to take from these results that we should immediately and permanently lower the potential output growth rate built into the stiffener. 4 Summary Our starting point output gap estimate errors are due both to data revisions and issues relating to the endpoint problem and stiffener. The endpoint problem, and the stiffener that we use to mitigate that problem, are a somewhat larger cause of starting point output gap estimate bias. We cannot be definite about classifying this later category as simply a technical filter issue because the stiffener has also been used as a device to introduce judgement into our output gap estimates. What we can say is that data revisions are only part of the story about our output gap forecast bias. Previous forecast errors work has suggested that the bias in our output gap starting point estimates does matter for our inflation forecast bias. Unfortunately there are no quick solutions. There is no ‘fix’ for the issue of ongoing GDP revisions by Statistics New Zealand and there will always be endpoint problems with a two-sided filter. This underscores the importance of putting considerable thought into our estimates of starting point potential output and not taking any single estimate of the output gap as the ‘true’ output gap to formulate policy. References McCaw, S (2002), “Bias in forecasts of key variables,” Reserve Bank of New Zealand Memorandum. Nelson, E and K Nikolov (2002), “UK inflation in the 1970s and 1980s: the role of output gap mismeasurement,” Bank of England Working Paper no 148. Orphanides, A and S Van Norden (2002), “The reliability of output gap estimates in real time,”. Forthcoming in the Review of Economics and Statistics. An earlier version of this paper is available as a Federal Reserve Board of Governors’ working paper. Ranchhod, S (2002), “GDP forecast errors,” Reserve Bank of New Zealand Memorandum. Twaddle, J (2002), “Attributing inflation forecast bias using FPS,” Reserve Bank of New Zealand Memorandum.