Sources of predictability and error in
ECMWF long range forecasts
Tim Stockdale
European Centre for Medium-Range Weather Forecasts
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
Outline
 Overview of System 4
 Some recent research results
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
Seasonal prediction at ECMWF
 Started in the 1990’s
 Strategy: fully coupled global GCMs
 Real-time forecasts since early 1997
 Forecasts issued publicly from December 1997
 Now using “System 4”
 Lifetime of systems has been about 5 years each
© ECMWF
S1
S2
Dec
1997
Mar
2002
LRF Training, Belgrade
S3
S4
Mar
2007
Nov
2011
13th - 16th November 2013
System 4 seasonal forecast model
 IFS (atmosphere)
 TL255L91 Cy36r4, 0.7 deg grid for physics (operational in Dec 2010)
 Full stratosphere, enhanced stratospheric physics
 Singular vectors from EPS system to perturb atmosphere initial conditions
 Ocean currents coupled to atmosphere boundary layer calculations
 NEMO (ocean)
 Global ocean model, 1x1 resolution, 0.3 meridional near equator
 NEMOVAR (3D-Var) analyses, newly developed.
 Coupling
 Fully coupled, no flux adjustments
 Sea-ice based on sampling previous five years
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
Reduced mean state errors
T850
U50
850hPa temperature S4(15)-ERA Int 1991-2008 JJA
135°W
Global
rms error:
0.663 NH:0.669
TR:0.662
SH:0.66
90°W
45°W
0°
45°E
90°E
50hPa zonal wind S4(15)-ERA Int 1991-2008 DJF
135°E
135°W
Global rms45°W
error: 1 NH:1.43
TR:0.853
SH:0.72
0°
45°E
90°E
90°W
135°E
[m/s]
[K]
60°N
60°N
8
60°N
60°N
4
30°N
30°N
3
8
30°N
30°N
2
0°
0°
1
30°S
-2
0°
2
60°S
30°S
30°S
60°S
60°S
-8
-8
90°W
45°W
0°
45°E
90°E
-10
135°E
135°W
135°E
135°W
850hPa temperature S3(11)-ERA Int 1991-2008 JJA
135°W
Global
rms error:
1.07 NH:1.06
TR:0.798
SH:1.48
90°W
45°W
0°
45°E
90°E
90°W
45°W
0°
45°E
90°E
135°E
50hPa zonal wind S3(11)-ERA Int 1991-2008 DJF
Global
rms 45°W
error: 3.26 NH:5.53
TR:2.02
SH:2.03
90°W
0°
45°E
90°E
135°E
[m/s]
[K]
60°N
60°N
8
60°N
60°N
4
30°N
30°N
3
0°
0°
1
30°N
30°N
30°S
-2
0°
2
-2
30°S
30°S
60°S
60°S
-3
60°S
60°S
-8
-8
135°W
90°W
45°W
0°
45°E
-10
90°E
135°E
135°W
90°W
90°E
135°E
135°W
90°W
T850 S4(15)-S3(11) 1991-2008 JJA
© ECMWF
60°N
mean 0.275 0°rms diff: 0.786
45°E
45°W
-4
-6
-4
90°W
6
4
0°
-1
30°S
10
8
2
135°W
-4
-6
-4
135°W
45°W
0°
45°E
90°E
135°E
90°E
135°E
U50 S4(15)-S3(11) 1991-2008 DJF
LRF Training, Belgrade
S4
-2
-3
60°S
6
4
0°
-1
30°S
10
[K]th
13th - 16
November 2013
4
60°N
60°N
2
mean -1.080° rms diff: 2.7
45°E
45°W
[m/s]
60°N
5
4
S3
Tropospheric scores
Spatially averaged grid-point temporal ACC
ACC S3 and S4 (m2-4; 30y)
0.7
0.7
S3
0.6
0.5
0.5
0.4
0.4
0.3
0.2
0.1
0.1
0
0
TR
2mT
NH
Z500
NH
T850
NH
2mT
S4
0.3
0.2
TR
T850
S3
0.6
S4
ACC
ACC
ACC S3 and S4 (m5-7; 30y)
TR
T850
One month lead
© ECMWF
LRF Training, Belgrade
TR
2mT
NH
Z500
NH
T850
Four month lead
13th - 16th November 2013
NH
2mT
Capturing trends is important.
Time-varying CO2 and other
factors are important in this.
NATL SST forecast anomalies
ECMWF forecasts, mean for months 1- 3, plotted at centre of verification period
Ensemble size is 11 SST obs: HadISST1/OIv2
Obs. anom.
Fcast S3
2
1
1
0
0
-1
-1
There is a strong link between
seasonal prediction and decadal/
multi-decadal climate prediction.
NATL SST forecast anomalies
ECMWF forecasts, mean for months 4- 6, plotted at centre of verification period
Ensemble size is 11 SST obs: HadISST1/OIv2
Obs. anom.
-2
1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004
6.11 cressida - net Sat Sep 1 00:30:54 2007
2
1
1
0
0
-1
-1
-2
© ECMWF
LRF Training, Belgrade
Fcast S3
2
-2
Anomaly (deg C)
Anomaly (deg C)
2
1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004
13th - 16th November 2013
MAGICS 6.11 cressida - net Sat Sep 1 00:29:36 2007
-2
More recent ENSO forecasts are better ....
NINO3.4 SST rms errors
NINO3.4 SST rms errors
180 start dates from 19810101 to 19951201, amplitude scaled
Ensemble size is 15
95% confidence interval for 0001, for given set of start dates
Fcast S4
Persistence
180 start dates from 19960101 to 20101201, amplitude scaled
Ensemble size is 15
95% confidence interval for 0001, for given set of start dates
Fcast S4
Ensemble sd
0.8
0.8
0.6
Ensemble sd
0.6
0.4
0.4
0.2
0
Persistence
Rms error (deg C)
1
Rms error (deg C)
1
0.2
0
1
2
3
4
5
6
0
7
0
1
2
Forecast time (months)
3
4
5
6
1981-1995
1996-2010
NINO3.4 SST anomaly correlation
NINO3.4 SST anomaly correlation
wrt NCEP adjusted OIv2 1971-2000 climatology
wrt NCEP adjusted OIv2 1971-2000 climatology
0.9
0.9
Anomaly correlation
1
Anomaly correlation
1
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.4
0.5
0
1
2
3
4
5
6
7
0.4
0
1
Forecast time (months)
© ECMWF
7
Forecast time (months)
LRF Training, Belgrade
MAGICS 6.12 nautilus - net Wed May 16 12:21:31 2012
2
3
4
Forecast time (months)
13th - 16th November 2013
MAGICS 6.12 nautilus - net Wed May 16 12:19:01 2012
5
6
7
System 4 configuration
 Real time forecasts:
 51 member ensemble forecast to 7 months
 SST and atmos. perturbations added to each member
 15 member ensemble forecast to 13 months
 Designed to give an ‘outlook’ for ENSO
 Only once per quarter (Feb, May, Aug and Nov starts)
 Back integrations from 1981-2010 (30 years)
 15 member ensemble every month
 15 members extended to 13 months once per quarter
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
How many back integrations?
 Back integrations dominate total cost of system
 System 4:
5400 back integrations
(must be in first year)

612 real-time integrations (per year)
 Back integrations define model climate
 Need both climate mean and the pdf, latter needs large sample
 May prefer to use a “recent” period (30 years? Or less??)
 System 2 had a 75 member “climate”, S3 had 275, S4 has 450.
 Sampling is basically OK
 Back integrations provide information on skill
 A forecast cannot be used unless we know (or assume) its level of skill
 Observations have only 1 member, so large ensembles are less helpful
than large numbers of cases.
 Care needed e.g. to estimate skill of 51 member ensemble based on past
performance of 15 member ensemble
 For regions of high signal/noise, System 4 gives adequate skill estimates
© ECMWF
LRF Training, Belgrade 13th - 16th November 2013
 For regions of low signal/noise (eg <= 0.5), need hundreds of years
QBO
5N-5S U30 forecast anomalies
System 4
ECMWF forecasts at month 7
Ensemble size is 11 U30 obs: ec_erai
Anomaly (m/s)
Obs. anom.
Fcast S4
24
24
12
12
0
0
-12
-12
-24
-24
5N-5S U50 forecast anomalies
1991 1992 1993 1994 1995 5N-5S
1996 1997
1998
1999 2000
2001 2002 2003 2004 2005
U50
forecast
anomalies
System 3
ECMWF forecasts at month 7
Ensemble size is 11 U50 obs: ec_erai
ECMWF forecasts at month 7
Ensemble size is 11 U50 obs: ec_erai
Fcast S3
Obs. anom.
MAGICS 6.12 nautilus - net Wed Sep 5 16:45:10 2012
Fcast S4
24
24
24
24
12
12
12
12
0
0
0
0
Anomaly (m/s)
Anomaly (m/s)
Obs. anom.
50hPa
-12
-12
-12
-12
-24
1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
-24
-24
1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
-24
© ECMWF
MAGICS 6.12 nautilus - net Wed Sep 5 16:44:39 2012
LRF Training, Belgrade
30hPa
13th - 16th November 2013
MAGICS 6.12 nautilus - net Wed Sep 5 16:43:59 2012
Problematic ozone analyses
GLOBAL O30 forecast anomalies
ECMWF forecasts at month 7
Ensemble size is 5 O30 obs: ec_erai
Obs. anom.
Fcast S4
2
1
1
0
0
-1
-1
-2
-2
Anomaly (ppm)
2
1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
MAGICS 6.12 nautilus - net Thu May 31 10:57:12 2012
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
Stratospheric trends
GLOBAL T30 forecast anomalies
ECMWF forecasts at month 5
Ensemble size is 15 T30 obs: ec_erai
Obs. anom.
Fcast S4
4
2
2
0
0
-2
-2
Anomaly (K)
4
1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011
MAGICS 6.12 nautilus - net Wed May 16 12:43:33 2012
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
Stratospheric temperature trend
problem. This is due to an
erroneous trend in initial
conditions of stratospheric water
vapour.
Land surface
Snow depth limits, 1st April
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
Sea ice
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
Tropical storm forecasts
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
Model errors are still serious …
 Models have errors other than mean bias
 Eg weak wind and SST variability in System 2
 Past models underestimated MJO activity (S4 better)
 Suspected too-weak teleconnections to mid-latitudes
 Mean state errors interact with model variability
 Nino 4 region is very sensitive (cold tongue/warm pool boundary)
 Atlantic variability suppressed if mean state is badly wrong
 Forecast errors are often larger than they should be
 With respect to internal variability estimates and (occasionally) other
prediction systems
 Reliability of probabilistic forecasts is often not particularly high (S4 better)
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
Recent Research
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
S4 extended hindcast set
15 members
51 members
DJF Europe T2m>upper tercile
Re-forecasts from 1 Nov, 1981-2010
Reliability score: 0.902
ROC skill score: 0.06
DJF Europe T2m>upper tercile
Re-forecasts from 1 Nov, 1981-2010
Reliability score: 0.981
ROC skill score: 0.22
(Figures from Susanna Corti)
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
S4 extended hindcast set
15 members
51 members
JJA Europe T2m>upper tercile
Re-forecasts from 1 May, 1981-2010
Reliability score: 0.987
ROC skill score: 0.38
JJA Europe T2m>upper tercile
Re-forecasts from 1 May, 1981-2010
Reliability score: 0.996
ROC skill score: 0.43
(Figures from Susanna Corti)
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
S4 ACC
DJF Z500
© ECMWF
LRF Training, Belgrade
S4 ACC perfect model limit
13th - 16th November 2013
Local p-value for perfect model
Indistinguishable from perfect
Worse than perfect
Better than perfect
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
ACC=0.61
Fig. 1. The Arctic Oscillation Index for DJF, as analysed from ERAI (blue) and as predicted by the S4
ensemble mean from the 1st November (red). The S4 ensemble mean is scaled by a factor of 6 to be
of comparable amplitude to the observed index.
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
QBO
A big reduction in vertical diffusion, and a further tuning of non-orographic GWD,
has given a big additional improvement in the QBO compared to S4.
Period and downward penetration match observations
Semi-annual oscillation still poorly represented
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
QBO forecasts
S3
S4
New
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
NH winter forecasts: vertical diffusion
0.371
0.319
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
NH winter forecasts
Even with 101 members,
ensemble mean signal
not always well defined
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013
Conclusions
 Models are improving
 Gradual but continuous improvement in scores
 Reliability can be high in many situations
 Forecast systems still have deficiencies
 Need calibration, and often cannot be trusted at face value
 Some issues may affect real-time forecasts more than re-forecasts
 Further improvements lie ahead
 Research results suggesting that previous estimates of predictability limits
might be wrong.
 Hard work needed to improve models and capture new sources of
predictability.
© ECMWF
LRF Training, Belgrade
13th - 16th November 2013