The ECB Survey of
Professional Forecasters
Luca Onorante
European Central Bank*
(updated from A. Meyler and I.Rubene)
October 2009
*The views and opinions expressed are those of the presenter and not necessarily those of the ECB
Outline of the presentation
(1) main features of the ECB SPF
(2) evaluation of short-term forecasts and performance
to date
(3) forecast uncertainty as viewed by SPF participants
(4) longer-term expectations
2
Main features of the SPF panel
• quarterly survey
• conducted since 1999Q1
• euro area macroeconomic expectations for HICP
inflation, real GDP growth and the unemployment
rate
• short- and more medium- and longer-term horizons
surveyed (including rolling and calendar year
horizons)
• probability distributions
• qualitative answers also possible
• survey panel of financial and non-financial institutions
• 75 active panellists (with average response of around
60) located throughout the EU
3
(2) evaluation of short-term forecasts and
performance to date
4
Summary: inflation forecasting performance
Chart: Inflation - 12 month ahead rolling horizons
Actual HICP outcome
HICP exc. unp fd & nrg
Error
SPF forecast
4.5
4.0
3.5
3.0
2.5
2.0
1.5
1.0
0.5
0.0
10
09
08
07
06
05
04
03
02
01
00
99
-0.5
• inflation persistently under-forecast
Sample statistics (1999Q1-2008Q4)*
Inflation
Actual value
Mean
2.2 (2.2)
Std dev. (in pp)
0.6 (0.6)
1 year
2 years
Forecast value
ahead
ahead
Mean
1.8
1.8
Std dev. (in pp)
0.2
0.1
Forecast error
1 year
2 years
statistics
ahead
ahead
ME (in pp)
0.6 (0.5) 0.6 (0.5)
MAE (in pp)
0.6 (0.6) 0.6 (0.6)
RMSE (in pp)
0.8 (0.7) 0.8 (0.7)
Theil’s U
1.0 (0.9) 1.0 (0.9)
* data in brackets are current vintage
data; otherwise data refer to real-time
• results broadly similar for one- and two-year ahead forecasts
• can largely be explained by specific shocks to food and energy
5
Summary: GDP growth forecasting performance
Sample statistics (1999Q1-2008Q4)*
GDP
Actual value
Mean
1.8 (2.1)
Std dev. (in pp)
1.0 (1.1)
1 year
2 years
Forecast value
ahead
ahead
Mean
1.9
2.3
Std dev. (in pp)
0.9
0.4
Forecast error
1 year
2 years
Chart: GDP - 12 month ahead rolling horizons
Error (based on first vintage)
Outcome (first vintage)
Outcome (current vintage)
SPF forecast
5
4
3
2
1
0
-1
-2
10
09
08
07
06
05
04
03
02
01
00
99
-3
• persistence in growth forecast errors
statistics
ahead
ahead
ME (in pp)
-0.3 (0.0) -0.7 (-0.5)
MAE (in pp)
0.8 (0.8) 1.1 (1.0)
RMSE (in pp)
0.9 (1.0) 1.3 (1.2)
Theil’s U
0.7 (0.7) 0.8 (0.7)
* data in brackets are current vintage
data; otherwise data refer to real-time
• larger for two-year ahead errors (which are smoother)
• mean error sensitive to data vintage
6
Summary: unemployment forecasting
performance
Chart: Unemp. - 12 month ahead rolling horizons
Error (based on first vintage), rhs
Outcome (first vintage), lhs
Outcome (current vintage), lhs
SPF forecast, lhs
11
4
4
-3
10
-2
09
5
08
-1
07
6
06
0
05
7
04
1
03
8
02
2
01
9
00
3
99
10
• little bias on average but persistent
Sample statistics (1999Q1-2008Q4)*
Unemployment
Actual value
Mean
8.5 (8.3)
Std dev. (in pp)
0.8 (0.6)
1 year
2 years
Forecast value
ahead
ahead
Mean
8.4
8.1
Std dev. (in pp)
0.9
0.8
Forecast error
1 year
2 years
statistics
ahead
ahead
ME (in pp)
-0.1 (-0.2) -0.1 (-0.1)
MAE (in pp)
0.4 (0.5) 0.6 (0.7)
RMSE (in pp)
0.5 (0.7) 0.8 (0.9)
Theil’s U
0.8 (1.2) 0.8 (0.9)
* data in brackets are current vintage
data; otherwise data refer to real-time
• errors on real-time data slightly lower on average
7
Short-term forecasting and performance:
Main conclusions
• Large and persistent errors with evidence of bias
• However, sample period (1999-2008) characterised
by substantial shocks
• Considerable heterogeneity across forecasters, but
difficult to identify consistently good or bad ones
• Little evidence of systematic differences across types
of forecaster or nationality of forecaster
8
(3) Macroeconomic uncertainty according to
the SPF
9
Different measures of uncertainty
The SPF provides several dimensions to measure
uncertainty e.g.
• Using information about point estimates e.g. spread
or standard deviation of point estimates
• Using information about probability distribution e.g.
average std. dev. of individual or aggregation
distributions, skew or kurtosis of distributions, event
probability, etc.
10
Short-term GDP growth uncertainty
• Level of disagreement 0.35 p.p. on
average, but fluctuated in range 0.2
p.p. to 0.6 p.p.
Disagreement
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
99
00
01
02
03
04
05
06
07
08
11
Short-term GDP growth uncertainty
• Level of disagreement 0.35 p.p. on
average, but fluctuated in range 0.2
p.p. to 0.6 p.p.
• Individual uncertainty has been
slightly higher, around 0.5 p.p. on
average, and more stable
Disagreement
Individual uncertainty
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
99
00
01
02
03
04
05
06
07
08
12
Short-term GDP growth uncertainty
• Level of disagreement 0.35p.p. on
average, but fluctuated in range
0.2p.p. to 0.6p.p.
• Individual uncertainty has been
slightly higher, around 0.5p.p. on
average, and more stable
• Aggregate uncertainty has
averaged around 0.6p.p., with
cyclical movements mainly driven
by disagreement
Disagreement
Individual uncertainty
Aggregate uncertainty
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
99
00
01
02
03
04
05
06
07
08
• Respondents appear not to have captured fully nature and extent of actual
uncertainty, particularly when one takes into consideration upward bias (see
Stuart & Ord 1994) in uncertainty measures above – this is a general result
across variables and horizons22
13
Uncertainty correlates strongly with other
business cycle indicators
Implied volatility of the euro area stock market (% per
annum; left-hand scale)
Average overall uncertainty (right-hand scale)
0.70
70
Economic Sentiment Indicator (inverted; left-hand scale)
Average overall uncertainty (right-hand scale)
0.70
0.65
80
0.65
40
0.60
90
0.60
30
0.55
100
0.55
0.50
110
0.50
0.45
120
0.45
70
60
50
2009
2008
2007
2006
2005
2004
2003
2002
2001
2009
2008
2007
2006
2005
2004
2003
2002
2001
2000
1999
0
2000
10
1999
20
• GDP growth and unemployment uncertainty indicators from SPF
correlate strongly with other proxies such as:
- implied stock market volatility
- business/consumer tendency surveys
14
(4) Longer-term inflation expectations of SPF
participants
15
Longer-term inflation expectations
Euro area SPF
US SPF
US SPF: CPI over the next 10 years
euro area SPF - HICP inflation 5 yrs ahead
Median
Mean
Low er quartile
Upper quartile
2.4
2.2
2.2
2.2
2.0
2.0
2.0
1.8
1.8
1.8
1.6
1.6
1.4
1.4
Median
Low er quartile
Upper quartile
1.6
2008Q4
2008Q1
2007Q2
2006Q3
2005Q4
2005Q1
2004Q2
2003Q3
2002Q4
2002Q1
2001Q2
2000Q3
1999Q4
1.4
1999Q1
2009Q1
2.4
2008Q1
2.4
2007Q1
2.6
2006Q1
2.6
2005Q1
2.6
2004Q1
2.8
2003Q1
2.8
2002Q1
2.8
2001Q1
3.0
2000Q1
3.0
1999Q1
3.0
Mean
• Average longer-term inflation expectations in line with ECB
definition of price stability; also less dispersed over time
16
Longer-term growth and unemployment
expectations
GDP growth
inter-quartile range
mean
Unemployment rate
median
inter-quartile range
3.0
mean
median
11
2.9
2.8
10
2.7
2.6
9
2.5
2.4
8
2.3
2.2
7
2.1
2.0
6
1.9
1.8
1.7
09
08
07
06
05
04
03
02
01
00
99
09
08
07
06
05
04
03
02
01
00
99
5
• Balance of risks to longer-term growth expectations has generally been
judged to the downside, while those for the unemployment rate have been
to the upside
• For longer-term unemployment expectations there has been
considerable correlation between revisions to short-term expectations
and longer-term expectations, indicating SPF respondents perceive
significant hysteresis
17
Concluding comments
• Evidence of persistent errors and possible bias in inflation, growth and
unemployment rate expectations from professional forecasters (small
sample caveat). Errors are comparable with those in other surveys (e.g.
Consensus Economics).
• Notwithstanding significant heterogeneity in forecast accuracy (MSE)
across survey participants, errors are largely common across
forecasters. Dispersion of point forecasts provides a poor indication of
the true level of uncertainty.
• Beyond point forecasts, SPF also provides insight into other aspects
such as uncertainty and longer-term expectations. Some evidence that
SPF respondents appear not to have captured fully nature and extent of
actual macroeconomic risks
• Overall empirical expectations in line with the recent emphasis on
learning and sticky information rather than fully rational expectations.
18
The ECB Survey of
Professional Forecasters
1 Frequency of the updates and data
Chart 1a When do you update your forecasts?
100%
Chart 1b If it is calendar driven, how often do you
update your forecasts?
80%
80%
60%
60%
40%
40%
20%
20%
0%
Q1a
calendar driven
data dependent
both
84%
30%
34%
0%
Q1b
Note: may sum to more than 100% as some respondents report both categories. In addition,
some of those who responded calendar-driven also stated they might occasionally update
quarterly
monthly
continously
other
59%
38%
6%
18%
Note: may sum to more than 100% as some respondents report both categories.
Chart 1c When responding to the SPF do you provide…
Chart 2 What is the highest frequency of data at which
you model / forecast?
80%
80%
60%
60%
40%
40%
20%
20%
0%
Q1c
latest available
new forecast
it depends
66%
7%
27%
Note: many of those providing qualitative information indicated that they may do a partial
update when responding to the SPF, if changes are significant to do it.
0%
Q2
monthly
quarterly
annual
it depends
59%
25%
0%
30%
Note: Many respondents reported that they follow the frequency of the underlying variable
being forecast (i.e. monthly for inflation, quarterly for GDP)
20
2a Role of judgment
Number of respondents reporting: judgment = 100%
HICP inflation GDP growth
Short term
(one year or less)
Medium term
(up to two years)
Long term
(five years ahead)
Unemployment rate
5
5
8
6
6
7
10
8
9
Chart 3a Judgment applied to the forecast, %
Short term
53
60
37
39
Medium term
43
42
45
Long term
47
45
48
40
20
0
HICP
GDP
Unemployment rate
Note: calculations include responses with 100% judgement.
21
2b Model versus judgement
HICP Inflation
Short term
GDP growth
Medium term
Long term
Short term
60
60
50
50
40
40
30
30
20
20
10
10
0
0
Macro
traditional
Macro DSGE
Time series
Judgment
Unemployment rate
Short term
Macro
traditional
Other, please
specify
Medium term
Macro DSGE
Time
series
Long term
Judgment
Other,
please
specify
What model do you use?
Medium term
Long term
80%
70%
65%
60
59%
60%
50
54%
40
30
40%
20
10
16%
20%
0
5%
Macro
traditional
Macro DSGE
Time
series
Judgment
Other,
please
specify
5%
0%
ARIMA
Note: calculations exclude responses with 100% judgement.
single
equation
VAR/VEC
Other (e.g.
factor
models)
Macro
DSGE
Marco
traditional,
other
Other,
please
specify
22
3 Probability distributions
Do you report the mean, mode or
median of the probability distribution?
80%
How do you generate your
reported probability distribution?
70%
70%
60%
60%
50%
50%
40%
40%
30%
30%
18%
20%
7%
10%
79%
80%
75%
20%
17%
5%
10%
0%
0%
mean
mode
median
model
functional form
judgmentally
23
4 External assumptions – how derived?
Oil prices
Exchange rate
100%
100%
80%
80%
60%
60%
40%
40%
20%
20%
0%
Q5a)
in house
markets
consensus
other
78%
32%
7%
27%
Note: may sum to more than 100% as some respondents report
both categories.
0%
Q5b)
consensus
other
5%
10%
10%
Wage growth
100%
100%
80%
80%
60%
60%
40%
40%
20%
20%
Q5c)
markets
88%
Note: may sum to more than 100% as some respondents report
both categories.
ECB’s interest rate
0%
in house
in house
markets
consensus
other
93%
5%
5%
2%
Note: may sum to more than 100% as some respondents report
both categories.
0%
Q5d)
in house
consensus
other
95%
8%
5%
Note: may sum to more than 100% as some respondents report
both categories.
24