Some comments about forecasting
Based on a paper provisionally entitled
“Forecasting GDP and its expenditure components by
the Economist Intelligence Unit: Are Country Reports
worth paying for?”
Corné van Walbeek
Forecasting techniques
• Non-quantitative techniques
– “I think that…”
– Consensus seeking (e.g. Delphi method)
– Scenario planning
• Quantitative techniques
– Time series methods (e.g. ARIMA)
– Predicting with simple (single equation) behavioural models
– Multiple equation models
• Others
– Technical analysis, especially for shares and currencies
Some background about macroeconomic
forecasting
• Economists are not particularly good at forecasting
– Especially not in turbulent times (Granger, 1996)
– Very poor at predicting recessions (Loungani, 2001)
– Forecasts tend to cluster together, often quite far from the
actual value (Granger, 1996)
• Most studies consider the accuracy of GDP growth and
inflation forecasts (Ash et al, 1998, Oller & Barot, 2000, Vogel, 2007)
• Strong focus on industrialised countries (US agencies,
IMF, OECD)
• Strong focus on institutional forecasts; not much on
private sector forecasts
Criteria for forecast accuracy
• Bias
– Mean error
 (F  A ) / n
i
i
i
• Size of forecast error
– Root mean square error
 (F
i
 Ai ) 2 / n
i
• Ability to beat naïve alternative
– RMSEEIU/RMSEnaive < 1
• Directional accuracy
– Forecasting accelerations and decelerations correctly
A typical forecasting process
• Use econometric models
– Details are often published if organisation is “public”
– If it is a private company, details typically not provided
• Model consists of
– Behavioural equations
– Standard macroeconomic identities (e.g. GDP = C + I + G + X - M
– Global identities (e.g. ΣX = ΣM) if relevant
• Distinguish between exogenous and endogenous variables
• Manual adjustments are made to forecasts if deemed necessary
• Rigorous and iterative process of quality control and checking of
forecasts
An example of the data: Austria, January 2007
Next-year (t+1) forecast
Current year (t) forecast
“Actual” value of
last year (t-1)
Against this value
the forecasts for
2006 are
measured
Magnitudes of the forecast errors
Table 5: Root mean square error for current-year (t) forecasts
Private consumption
Public consumption
Gross fixed investment
Total domestic demand
Exports of goods and services
Imports of goods and services
Gross domestic product
High
income
(n=32)
1.5
Upper
middle
income
(n=11)
2.9
Lower
middle
income
(n=6)
2.3
Developing
countries
(n=17)
2.7
All
countries
(n=49)
1.8
(320)
(87)
(46)
(133)
(453)
1.6
3.4
4.1
3.6
2.2
(313)
(87)
(46)
(133)
(446)
4.2
7.3
7.9
7.5
5.1
(310)
(86)
(46)
(132)
(442)
2.0
3.5
2.5
3.2
2.3
(312)
(78)
(31)
(109)
(421)
4.3
6.1
8.8
7.0
5.1
(320)
(87)
(46)
(133)
(453)
4.6
7.4
9.8
8.3
5.6
(320)
(87)
(46)
(133)
(453)
1.4
2.0
2.3
2.1
1.6
(320)
(87)
(49)
(136)
(456)
The number of observations for which the RMSE is calculated is shown in parentheses below each value
Some comments about the RMSEs
1.
They are large
–
–
For current year forecasts: between 1.4 and 9.8 percentage
points; median = 3.5 percentage points
For next-year forecasts: between 15 and 30 per cent larger
than current-year forecasts
2. Large differences in RMSEs between magnitudes
–
–
RMSEs around 2 percentage points: C, G, TDD and GDP
RMSEs around 5 percentage points: I, X and M
3. Lower RMSEs for developed countries; higher
RMSEs for developing countries
Comparing the EIU’s forecasts against
naïve predictions
• Assumption used for this paper:
– The naively predicted growth rate for this year and for next
year is the “estimated” growth rate for the previous year
• Calculate RMSE ratio = RMSEEIU/RMSEnaive
• If RMSE ratio < 1, then EIU forecasts are better
(have smaller errors) than naïve alternative
Table 9: RMSE of EIU current-period (t) forecasts as a ratio of RMSE of naïve
forecasts
Private consumption
Public consumption
Gross fixed investment
Total domestic demand
Exports of goods and services
Imports of goods and services
Gross domestic product
High
income
(n=32)
0.88
0.87
0.73
0.78
0.73
0.72
0.73
Upper
middle
income
(n=11)
0.81
0.79
0.71
0.74
0.73
0.78
0.68
Lower
middle
income
(n=6)
0.80
0.87
0.79
0.77
0.71
0.68
0.72
Developing
countries
(n=17)
0.81
0.82
0.74
0.75
0.72
0.75
0.70
All
countries
(n=49)
0.86
0.85
0.73
0.77
0.73
0.73
0.72
Average of 0.77
Table 10: RMSE of EIU next-period (t+1) forecasts as a ratio of RMSE of naïve
forecasts
Private consumption
Public consumption
Gross fixed investment
Total domestic demand
Exports of goods and services
Imports of goods and services
Gross domestic product
High
income
(n=32)
0.92
0.84
0.80
0.85
0.80
0.76
0.89
Upper
middle
income
(n=11)
0.87
0.73
0.71
0.76
0.67
0.79
0.76
Lower
middle
income
(n=6)
0.81
0.85
0.88
0.77
0.80
0.78
0.93
Developing
countries
(n=17)
0.85
0.77
0.77
0.76
0.71
0.79
0.82
All
countries
(n=49)
0.90
0.82
0.79
0.83
0.78
0.77
0.87
Average of 0.82
Two recommendations
1.
–
2.
More modesty please!
Words like “prescient”, “decisive verdicts”, “precision”, etc. do not
belong in a forecaster’s vocabulary
Publish confidence intervals
–
–
–
–
E.g. 67% confidence intervals (= point estimate ± RMSE)
67% (or 50%) confidence intervals
1.
2.
Are not affected by outlying forecast errors
Are not as large as 95% confidence intervals (see Granger, 1996)
•
Be honest (“This magnitude is very difficult to forecast”)
1.
2.
Emphasises the stochastic nature of forecasting to clients
Increases the credibility of the EIU (“Now they are always wrong. At
least they will be right two thirds of the time”)
Allows users to do scenario planning with realistic “optimistic” and
“pessimistic” scenarios
The existing RMSEs would be a good first approximation for such
intervals
What if the intervals are embarrassingly large?
Advantages of publishing confidence intervals:
3.