Final report for Project 3.1.2
Intercomparison of seasonal forecast models for
South-Eastern Australian Climate
Eun-Pa Lim, Harry Hendon, Andrew Charles, and Oscar Alves
Centre for Australian Weather and Climate Research
The Bureau of Meteorology
GPO Box 1289
Melbourne 3001
hhh@bom.gov.au
Tel: (03) 9669 4120, Fax: (03) 9669 4660,
15 Jan 2009
Abstract
This study compared currently available seasonal forecast schemes and evaluated
their skill over the south-eastern Australian region (33.5–38.5°S, 137.5–152.5°E). We
compared one dynamical model (POAMA), two statistical–dynamical models based
on POAMA products, and one statistical model (the current operational seasonal
forecast scheme run by the National Climate Centre, NCC). We began by noting that
the direct forecasts from POAMA are, in a sense, “statistically corrected” in that some
basic bias is removed (by forming anomalies relative to the hindcast model
climatology) and standardization is applied (by normalising the anomalies to the
standard deviation of predicted anomalies). In the following we will refer to the biascorrected and standardised forecasts of rainfall and temperatures from POAMA as the
“direct” POAMA forecasts, and the statistical–dynamical model forecasts as
“statistically post-processed”.
Our results suggest that the current NCC operational scheme for predicting above and
below median rainfall and temperature provides conservative but reliable forecasts in
all seasons (that is, all forecasts are close to a climatological forecast). Compared to
the NCC operational scheme, the direct forecasts from POAMA demonstrate up to
30–40% Brier Skill Score improvement in predicting above-median rainfall in autumn
and spring; however they show significant decrease of skill in summer and winter
over the SEACI region at lead time 0. Likewise, up to 40–50% skill improvement
over the NCC operational scheme is found for the direct POAMA temperature
forecasts (especially in spring), but there is a 30–40% decrease in skill for summer.
Statistically calibrated and bridging forecasts add skill to POAMA only over some
parts of the SEACI region and only in some seasons.
Based on findings in the literature that the overall performance of a multi-model
ensemble system can be better than the individual single models included in it, we
have designed a “homogeneous” multi-model ensemble system (HMME) using a
combination of the statistical–dynamical models and the direct predictions from
POAMA. According to our analysis, the HMME provides, for all seasons over southeastern Australia, about 10–30% skill improvement over the direct predictions from
POAMA for above-median rainfall. On the other hand, the effect for maximum
temperature is less dramatic when the direct forecasts from POAMA and the forecasts
from the statistical bridging and calibration models are combined; this is because
direct POAMA predictions for south-eastern temperature is already skilful and
superior to any of the statistical or statistical–dynamical models in the winter and
spring seasons.
Significant research highlights, breakthroughs, and snapshots

POAMA demonstrates good skill in predicting South-Eastern Australian
rainfall and temperature. It shows, for the hindcast period 1980–2006, a 10–
30% skill improvement compared to the current operational system run by the
NCC (a purely statistical model) in all seasons except for summer.

A preliminary version of our homogeneous multi-model ensemble prediction
system has been constructed to statistically predict regional rainfall and
temperature. It is based on raw POAMA forecasts and forecasts from
statistically post-processed models, and uses predictions from POAMA of
hemispheric mean sea-level pressure, rainfall, and temperature. The multimodel forecast demonstrates some distinctive improvements over the direct
predictions from POAMA, having forecast skill increases for rainfall of up to
20% compared to direct predictions from POAMA.

Real-time forecasts of South-Eastern Australian rainfall and temperature are
available at the POAMA official website:
http://poama.bom.gov.au/experimental/poama15/sp_seaci.html

Implementation of the HMME for real-time forecasts is under way.
Statement of results, their interpretation, and practical significance against each
objective

Evaluate the skill of predictions from statistical–dynamical models
directly from POAMA and from purely statistical models for South-East
Australian rainfall and temperature
In this study, we have compared the skill of four different models in predicting
seasonal rainfall and temperature for South-Eastern Australia: the direct predictions
from a dynamical model (POAMA), two statistical–dynamical models (a statistical
calibration model and a statistical bridging model), and a purely statistical model run
operationally by the National Climate Centre.
POAMA is a seasonal forecasting system based on comprehensive atmosphere and
ocean dynamical models (Alves et al. 2003). One of the major improvements of the
current version of the POAMA system (v1.5b) is that it has a new atmosphere–land
initialisation scheme (ALI) which initialises POAMA’s atmosphere–land surface
models with conditions close to those observed. Consequently, compared to previous
versions, POAMA v1.5b demonstrates improved skill at intra-seasonal and seasonal
time-scales (Hudson et al. 2009). For this study, a 10-member ensemble of hindcasts
for the period 1980–2006 were generated and analysed.
The statistical–dynamical schemes have been developed through Project 3.2.3, and
are based on the statistical relationship between observed Australian rainfall and
temperature and the direct predictions of rainfall and mean sea-level pressure from
POAMA. A statistical bridging model uses POAMA-predicted mean sea-level
pressure over the Southern Hemisphere at mid and high latitudes to predict regional
Australian rainfall and temperature. A statistical calibration model adjusts the direct
predictions of Australian rainfall and temperature from POAMA towards observed
rainfall and temperature..
The Bureau of Meteorology’s seasonal forecast system operated in the NCC is based
on a statistical model that was designed to utilise the historical relationship between
Australian climate and the leading modes of the tropical Pacific–Indian Ocean
variability (Drosdowsky and Chambers 2001).
The current operational predictions from NCC for above- and below-median rainfall
and temperature provide conservative and reliable forecasts in all seasons (that is, all
forecasts are close to a climatological forecast). In comparison, POAMA
demonstrates up to 30–40% Brier Skill Score improvement over the operational
scheme in predicting above-median rainfall in autumn and spring, but POAMA shows
a significant decrease of skill in summer and winter over the SEACI region at lead
time 0. Likewise, compared to the operational scheme, POAMA demonstrates up to
40–50% skill improvement (except for summer) for forecasts of maximum
temperature over the SEACI region. In summer, a 30–40% skill decrease occurs.
Compared to direct predictions of rainfall and temperature from POAMA, the
statistically calibrated forecasts and the statistically bridged forecasts have better skill
for particular locations, but over wider areas they show reduced skill for both rainfall
and temperature. The reduced skill of the statistical–dynamical forecasts can be partly
attributed to the thorough cross-validation processes within the short hindcast period.
Statistical–dynamical models can, however, provide more skilful predictions than
POAMA over regions where raw POAMA skill is low. An implication of these results
is that as the POAMA system is further improved, it will become harder for
statistical–dynamical schemes to beat raw POAMA forecasts.
Although statistically post-processed forecasts do not necessarily outperform direct
predictions from POAMA, they can contribute to skill improvement in POAMA’s
seasonal forecasts; a multi-model ensemble prediction can improve matters provided
some skill is available and that forecast errors are independent of POAMA (Zebiak
2003, Hagedorn et al. 2005). We have designed a “homogeneous” multi-model
ensemble system (HMME) using statistical–dynamical models and direct predictions
from POAMA; when its probabilistic forecast skill for above-median rainfall is tested
over the South-Eastern Australia region, the HMME results, for all seasons, in about
10–20% skill improvement over the direct predictions from POAMA. The multimodel forecasts are much less emphatic than the raw forecasts from POAMA, while
at the same time being more skilful. For temperature forecasts, the effect of
combining POAMA and statistical–dynamical models is positive in summer and
autumn, but rather negative in winter and spring (because POAMA’s prediction for
South-Eastern Australian temperature is already very skilful). In the case of
temperature prediction, excluding the bridging model might help the performance of
the HMME.
Summary of methods and modifications (with reasons)

Utilise 10-member, 9-month hindcasts from POAMA for the period 1980–
2006.

Apply statistical post-processing to POAMA hindcasts and evaluate the
performance of the statistical–dynamical models for South-Eastern Australian
rainfall and temperature predictions. In this report we focus on the skill
measured with the probabilistic method to make a comparison with the
Bureau’s operational seasonal outlook for probabilistic forecasts of rainfall
and temperature. The deterministic forecast skill assessment is discussed in
Project 3.2.3.

Compare the skill from the dynamical and statistical–dynamical models to that
from the operational statistical seasonal forecast model.

Develop a homogeneous multi-model ensemble system consisting of POAMA
and statistical calibration and bridging models which capitalises on POAMApredicted Australian rainfall and temperature (calibration) and mean sea-level
pressure (bridging).
(a) Rainfall
OPR
POAMA
CALIBRATION
BRIDGING
HMME
DJF
(summer)
MAM
(autumn)
JJA
(winter)
SON
(spring)
(b) Tmax
OPR
POAMA
CALIBRATION
BRIDGING
HMME
DJF
MAM
JJA
SON
Figure 1: The Brier Skill Score (refer to Lim et al., 2009, for the formula) of five
different seasonal prediction systems in predicting above-median rainfall (a) and
temperature maximum (b) over South-Eastern Australia at their shortest lead times.
The contour indicates percent improvement of forecast skill compared to a reference
forecast system. OPR: the current operational system to predict greater than median
rainfall, taking the climatological forecast as a reference. POAMA: POAMA, taking
the current operational scheme as a reference. CALIBRATION, BRIDGING, and
HMME: the statistical calibration, statistical bridging, and the homogeneous multimodel ensemble models, respectively, taking POAMA as a reference.
Summary of links to other projects

This project is the follow-on of Project 3.2.2 that recommended statistical
techniques for making use of predictable climate components from POAMA.

It builds on Project 3.2.3 that developed and tested several statistical–
dynamical methods to improve South-Eastern Australian rainfall and
temperature by bridging POAMA’s sea-surface temperature (SST) and mean
sea-level pressure (MSLP) predictions and calibrating POAMA’s rainfall and
temperature predictions.

The data used for this project have been extended and improved by Project
3.1.4.
Publications arising from this project
Lim, E.-P., Hendon, H.H., and Alves, O. (2009). Dynamical, statistical–dynamical,
and homogeneous multi-model ensemble forecasts of Australian cool
season rainfall (manuscript in preparation for submission to Mon. Wea.
Rev.)
Acknowledgement
This research was in part supported by the South Eastern Australian Climate
Initiative.
Recommendations for changes to work-plan from your original table
None
References
Alves, O., Wang, G., Zhong, A., Smith, M., Tzeitkin, F., Warren, G., Schiller, A., Godfrey,
S., and Meyers, G. (2003). POAMA: Bureau of Meteorology operational coupled
model seasonal forecast system. Science for drought – Proceedings of the National
Drought Forum. Queensland Government, Brisbane.
Drosdowsky, W. and Chambers, L. (2001). Near-global sea surface temperature anomalies as
predictors of Australian seasonal rainfall. J. Climate 14: 1677–1687.
Hagedorn, R., Doblas-Reyes, F.J.. and Palmer, T.N. (2005). The rationale behind the success
of multi-model ensembles in seasonal forecasting – I. Basic concept. Tellus 57A: 219–
233.
Hudson, D., Alves, O. Wang, G., and Hendon, H. (2009). The impact of atmospheric
initialisation on seasonal prediction in the Indo-Pacific region (draft to be submitted)
Zebiak, S.E. (2003). Research potential for improvements in climate prediction. Bull. Amer.
Met. Soc. 84: 1692–1696.
Project Milestone Reporting Table
To be completed prior to commencing the project
Completed at each Milestone date
Milestone
description1
Performance
indicators2
Completion
date3
Budget4 for
Milestone
($) (SEACI
contribution)
Progress5
Recommended
changes to
workplan6
1. Apply statistical
calibration and
bridging to
extended hindcast
set from POAMA.
Develop purely
statistical scheme
(as an extension of
the bridging
technique) for SE
Aust rainfall and
temperature based
on major modes of
SST variability
Short report
prepared
(4 pages).
Jul 2008
25K
* Forecast skill of
two statistical
bridging schemes
and one statistical
calibration scheme
has been assessed
and compared to
the skill of
POAMA and the
current operational
statistical model
for the prediction
of SE Aust rainfall.
* Multi-model
ensemble prediction system has
been designed and
tested.
None
2. Evaluate and
document forecast
skill of SE Aust
rainfall and
temperature from
POAMA,
statistical bridging
and calibration
schemes and a
statistical model
for the period
1980–2006
Report
prepared
(10 pages)
Dec 2008
25K
A manuscript for
publication has
been prepared (Lim
et al. 2009)
None
Attachment
Probabilistic forecasts of Australian cool
season rainfall
Eunpa Lim, Harry Hendon, David Anderson, Oscar Alves
e.lim@bom.gov.au
1. Introduction
Over the last two decades, significant amount of research and efforts has been poured
into utilizing general circulation models for seasonal climate prediction and
improving their skill. The strength of dynamical models is that the models can
incorporate linear and non-linear processes in the atmosphere and ocean to predict
climate, and such ability is especially important for seasonal climate prediction over
subtropical and extratropical regions where their climate is governed by internal
dynamics as much as by the lower boundary forcing (i.e. sea surface temperature).
In order to improve the quality of seasonal climate forecasts over Australia,
the Australian Bureau of Meteorology together with CSIRO has been developing an
atmosphere and ocean coupled model, POAMA (Predictive Ocean Atmosphere Model
for Australia). POAMA became operational in 2002 and currently it issues the latest
30 member ensemble forecasts for El Niño or La Niña states in the next 9 months. As
ENSO (El Niño and the Southern Oscillation) is one of the dominant drivers of the
Australian seasonal climate, it is a major focus of POAMA to predict the tropical SST
anomalies associated with ENSO with good skill. In fact, the current operational
version of POAMA (v1.5b) demonstrates internationally competitive skill in ENSO
predictions (see http://iri.columbia.edu/climate/ENSO/currentinfo/SST_table.html).
Consequently, some useful skill in POAMA’s prediction for the Australian seasonal
climate is expected. The first aim of this study is, therefore, to analyse characteristics
of POAMA’s probabilistic forecasts for the Australian rainfall and to assess the skill,
using its hindcast set for 1980-2006.
Apart from purely dynamical prediction, previous studies have suggested that
reasonable forecast skill can be obtained by using dynamically predicted climate
components in a statistical model. This method is referred as statistical postprocessing or statistical-dynamical prediction. Statistical calibration and bridging
(downscaling) are common techniques for statistical-dynamical prediction: Statistical
calibration attempts to adjust spatial patterns of variability of a dynamically predicted
variable against its observed counterpart (Ward and Navarra 1997, Feddersen et al.
1999, Mo and Straus 2002, Kang et al. 2004). Statistical bridging is to predict a
variable through its statistical relationship with a model predicted large scale climate
component (Voldoire et al. 2002, Tippett et al. 2005, Lin and Derome 2005, Kang et
al. 2007). Therefore, the second aim of this study is to explore the feasibility of
increasing forecast skill over POAMA through statistical post-processing by
correcting POAMA’s rainfall prediction against observation or by bridging POAMA
predicted large scale climate component to rainfall over Australia.
This paper is outlined as follows: details of the POAMA system and
verification methods will be described in the next section. Then we will discuss
probabilistic forecasts of POAMA and our statistical-dynamical models in sections 3
and 4, respectively. Finally, the concluding remarks will be given in section 5.
2. Description of the POAMA system and verification methods
The POAMA version 1.5b system uses the latest versions of the Bureau’s
Atmospheric Model (BAM3; Colman et al. 2005) and the Australian Community
Ocean Model (ACOM2; Schiller et al. 2002, Oke et al. 2005). The horizontal
structure of BAM3 is represented by spherical harmonics with a truncation at
triangular 47 (T47, approximately 250 km), and the vertical variation is represented
by 17 sigma levels. ACOM2 has a zonal resolution of 2° latitude (lat) and a
telescoping meridional resolution of 0.5° lat within 8° lat of the equator, gradually
changing to 1.5° lat near the poles. The atmosphere and ocean models are coupled by
the Ocean Atmosphere Sea Ice Soil (OASIS) coupling software.
The atmospheric initial conditions are obtained from the Bureau’s operational
numerical weather prediction system (GASP) for the real-time forecasts and from the
atmosphere and land initialization scheme (ALI) for the hindcasts. According to
Hudson and Alves (2007) and Wang et al. (2008), the use of ALI enables the initial
conditions of the hindcasts to be more consistent with those of the real-time forecasts,
and therefore, the skill assessed in the hindcasts can be a good indicator of the skill of
the real-time forecasts. The ocean data assimilation system provides the best estimate
of the present state of the tropical upper ocean, based on the optimum interpolation
(OI) technique (Smith et al. 1991). Further details of the POAMA system can be
found in Lim et al. (2008) and http://poama.bom.gov.au/documentation/index.html.
In this study, we analyse ten-member ensemble hindcasts at lead time 0 (LT 0)
generated from 1980 to 2006. We will limit our interest to the rainfall in the SH
winter (June-July-August) and spring (September-October-November) when skilful
rainfall prediction would be greatly valued for decision making for water management
and agriculture. The ensemble hindcasts of Australian rainfall prediction were verified
against the Australian National Climate Centre gridded data (0.25° lat x 0.25° lon;
Jones and Weymouth 1997). The standardized rainfall anomaly (hereafter, refer to
rainfall) was obtained from seasonally averaged data.
3. Probabilistic forecasts of POAMA
As a compact way of displaying probabilistic forecast features, reliability diagram is
widely used (Wilks 2006, Palmer et al. 2008) (Figure 1). Taking all the grid points
over Australia in the 27 year hindcasts at LT 0 into account, Figure 1 suggests that
POAMA demonstrates moderate reliability for predicting lower probabilities (2050%, relatively close to the perfect reliability line) but poor reliability for predicting
higher probabilities (60-80%) of exceeding median rainfall. Also, the forecasts with
greater than 90% chance of being above median are found to be reliable forecasts
especially in spring. The graphs also show that POAMA has a good forecast
resolution, having the spread of the observed relative frequency of above median
rainfall, ranging from 0.2 to 0.8.
In order to assess the skill of POAMA probabilistic forecasts, we have
computed the Brier Score (BS), which is the mean squared error of the probability
forecasts against the occurrence of the event (Wilks 2006, their eq. 7.34):
BS 
1 n
( Yk  Ok ) 2

n k 1
where n is the number of the hindcast years, Yk is POAMA forecast probability
for above median rainfall and Ok is the observed rainfall in the kth year. Ok is 1 if it is
above median or 0 if it is not. Figure 2 displays the Brier Skill Score (BSS) which
indicates a % improvement of the BS by POAMA forecasts compared to the BS of the
Bureau’s current operational seasonal forecast system in winter and spring. The
present operational seasonal forecast system is purely based on statistics (Drosdowsky
and Chambers 2001). Here, POAMA’s LT 0 forecasts are compared to the current
operational forecasts at lead time 1 month which is the shortest lead time for this
system.
According to Figure 2, 10-20% of the forecast skill improvement from
POAMA is found over the eastern and the south western parts of Australia in both
winter and spring. In particular, south eastern Australia obtains more than 20%
improvement of forecast skill in spring. By contrast, POAMA does not have much
skill for predicting winter rainfall over the southern end of the country even at LT 0,
which could be related to some deficiencies in the model physics or the model
resolution for resolving synoptic scale weather systems over the Great Australian
Bight. These issues should be properly addressed in the subsequent versions of
POAMA.
4. Statistical-dynamical forecasts and a preliminary homogeneous multi-model
ensemble approach
As mentioned earlier in the introduction, it has been suggested in the literature that
statistical-dynamical prediction can be a good way of obtaining reasonable forecast
skill. Therefore, we developed a statistical calibration scheme using POAMA
predicted rainfall and a statistical bridging scheme using POAMA predicted pressure
in order to predict Australian rainfall. As the first step to develop our statisticaldynamical models, we found the most covarying spatial patterns between the
observed Australian rainfall and POAMA rainfall (or POAMA pressure) and their
temporal coefficients, using the Singular Value Decomposition Analysis (SVDA)
technique. Then, the temporal coefficients of the first 5 dominant SVD modes of
POAMA rainfall or pressure were multiple-linearly regressed onto the observed
rainfall at each grid point over Australia. Finally, rainfall forecast at a new time was
made by 1) projecting POAMA predicted rainfall (or pressure) for the new time onto
the 5 SVD spatial patterns obtained earlier, 2) calculating the resultant temporal
coefficients, and 3) plugging them to the regression coefficients computed in the
training period. In this study, all the processes of computing SVD modes and
regression coefficients were cross-validated. For the details of the statisticaldynamical techniques, refer to Hendon et al. (2007).
Figure 3 displays the skill from POAMA and our statistical calibration and
bridging models for predicting above median rainfall averaged over Australia. It is
seen that the forecast skill of the statistical-dynamical models is competitive with that
of POAMA but not better than the POAMA skill in both seasons at LT 0. Our detailed
analysis suggests that whether these statistical-dynamical prediction schemes can add
extra skill to POAMA depends on the locations, seasons and lead times of interest
(not shown). This implies that a multi-model ensemble approach using all the
available information from the dynamical and statistical-dynamical models could
bring some positive results in forecast skill by compensating the weaknesses of each
model. Therefore, we combined the three sets of the ten member ensemble forecasts
from POAMA and the statistical calibration and bridging models, and measured the
skill with the BSS. This multi-model ensemble scheme is called “homogenous multimodel ensemble (HMME)” scheme as the individual models utilize the outputs of one
model - POAMA.
The results of the HMME prediction are shown in Figures 4 and 5. The
reliability diagram in Figure 4 exhibits that the forecast reliability is greatly improved
by our HMME prediction compared to the reliability of POAMA (shown in Figure 1),
having the forecasts in each probability bin closely aligned with the perfect reliability
line. The forecast resolution increases, ranging from 0 to 0.9. The frequency of
emphatic forecasts is reduced, and the frequency of forecast probabilities is more
normally distributed. Furthermore, the plots in Figure 5 demonstrate that there are up
to 20% improvement of the BS of the HMME compared to that of POAMA in both
winter and spring at LT 0. Much benefit of using the HMME is found in winter
especially over the southern part of South Australia and over Victoria where POAMA
does not have good forecast skill, whereas the use of the HMME decreases the
forecast skill over the border of the New South Wales and Queensland where
POAMA exhibits about 30% higher skill than the current operational system in
spring. Despite the sensitivity of forecast skill improvement to different regions and
seasons, the HMME prediction seems able to add extra skill to POAMA over most
areas of Australia.
5. Concluding remarks
In this study, we have analysed the characteristics and skill of the probabilistic
forecasts of POAMA and the statistical-dynamical prediction schemes for Australian
winter and spring season rainfall. The results have suggested that POAMA can
provide more skilful forecasts than the current operational forecast system for
below/above median rainfall in winter and spring at LT 0. Especially, the
improvement is significant over south eastern Australia in spring time.
In order to examine if additional skill improvement is obtainable by
capitalizing on POAMA predicted outputs to predict Australian rainfall in statistical
schemes, we have developed a statistical calibration scheme and a statistical bridging
scheme which take POAMA predicted rainfall and pressure as a predictor,
respectively, in a multiple linear regression model. The forecast skill from these
statistical-dynamical prediction schemes is seen to be comparable with POAMA skill
but does not seem to offer additional skill to POAMA at least at the shortest lead time.
As an inventive experimental attempt, we have concatenated our dynamical
and statistical-dynamical models to make a multi-model ensemble prediction for
Australian rainfall. The results show that in both winter and spring at LT 0 our
homogeneous multi-model ensemble scheme can improve the forecast reliability
significantly and demonstrate improved forecast skill over the regions where POAMA
forecasts are not very skilful.
The POAMA system is continually evolving and improving – the subsequent
versions of POAMA will address issues such as correcting model bias and drift,
increasing model horizontal resolution, improving model physics and initializing
model with more realistic initial conditions so as to provide skilful prediction of
regional climate variability. In the meantime, the homogeneous multi-model ensemble
prediction seems worth further exploring and being extended to longer lead time
forecasts and to different climate variables such as temperature over Australia.
SON
JJA
1
0.8
POAMA
0.6
Perfect reliability
No resolution
0.4
no skill
0.2
0
Observed relative frequency
Observed relative frequency
1
0.8
POAMA
0.6
Perfect reliability
No resolution
0.4
no skill
0.2
0
0
0.2
0.4
0.6
Forecast probability
0.8
1
0
0.2
0.4
0.6
0.8
1
Forecast probability
Figure 1: Reliability diagrams for POAMA prediction for above median rainfall over
Australia at lead time 0 (LT 0). The size of the solid circles is proportional to the bin
population. The circles below (above) the ‘no skill’ line in the forecast probabilities
smaller (larger) than the climatological forecast for above median (0.5) contribute to
positive Brier Skill Score (BSS). The ‘no resolution’ line indicates no difference in the
observed frequencies of above median rainfall between the different forecast
categories.
Figure 2: BSS which indicates a % improvement of POAMA forecasts over the
current operational forecasts for exceeding median rainfall for the period of 19802006. Left and right plots show winter and spring skill, respectively. The contour
interval is 0.1 which means 10% change.
Figure 3: Percent consistent score (hit rates) of below/above median rainfall averaged
over Australia and BSS of above median of Australian mean rainfall, taking the
climatological forecast as a reference at LT 0.
JJA
SON
1
0.8
Multimodel
Perfect reliability
0.6
No resolution
no skill
0.4
0.2
0
Observed relative frequency
Observed relative frequency
1
0.8
Multimodel
0.6
Perfect reliability
No resolution
0.4
no skill
0.2
0
0
0.2
0.4
0.6
Forecast probability
0.8
1
0
0.2
0.4
0.6
0.8
1
Forecast probability
Figure 4: Reliability diagrams for the homogeneous multi-model ensemble (HMME)
prediction for above median rainfall over Australia at LT 0.
Figure 5: BSS which indicates a % improvement of the HMME forecasts over
POAMA forecasts at LT 0 for exceeding median rainfall for the period of 1980-2006.
Left and right plots display winter and spring skill, respectively. The contour interval
is 0.1 which means 10% change.
References
Colman, R., L. Deschamps, M. Naughton, L. Rikus, A. Sulaiman, K. Puri, G. Roff,
Z. Sun, and G. Embery, 2005: BMRC Atmospheric Model (BAM) version
3.0: Comparison with mean climatology. BMRC Research Report No. 108,
Bureau of Meteorology Research Centre, 32 pp. (available from
http://www.bom.gov.au/bmrc/pubs/researchreports/researchreports.htm).
Drosdowsky, W. and L. Chambers, 2001: Near-global sea surface temperature
anomalies as predictors of Australian seasonal rainfall. J. Climate, 14, 16771687.
Feddersen, H., A. Navarra, and M. N. Ward, 1999: Reduction of model systematic
error by statistical correction for dynamical seasonal predictions. J. Climate,
12, 1974-1989.
Hendon, H. H., E. Lim, O. Alves, and G. Wang, 2007: Review of techniques to
bridge/calibrate dynamical seasonal predictions with focus on south eastern
Australia, SEACI milestone report.
Hudson, D. and O. Alves, 2007: The impact of land-atmosphere initialisation on
dynamical seasonal prediction. BMRC Research Report No. 133, Bureau of
Meteorology Research Centre, p 19-22.
Jones, D. A. and G. Weymouth, 1997: An Australian monthly rainfall dataset.
Technical Report 70, Bureau of Meteorology.
Kang, I.-S., J.-Y. Lee, and C.-K. Park, 2004: Potential predictability of summer
mean precipitation in a dynamical seasonal prediction system with
systematic error correction. J. Climate, 17, 834-844.
Kang, H., K.-H. An, C.-Kyu Park, A. L. S. Solis and K. Stitthichivapak, 2007:
Multimodel output statistical downscaling prediction of precipitation in the
Philippines and Thailand. Geophys. Res. Lett., 34, L15710,
doi:10.1029/2007GL030730
Lim, E.-P., H. H. Hendon, O. Alves, D. Hudson, and G. Wang, 2008: Dynamical
forecasts of Australian seasonal rainfall. manuscript in preparation.
Lin, H. and J. Derome, 2005: Correction of atmospheric dynamical seasonal forecasts using the leading ocean-forced spatial patterns. Geophys. Res. Lett., 32,
L14804.
Mo, R. and D. M. Straus, 2002: Statistical-dynamical seasonal prediction based on
principal component regression of GCM ensemble integrations. Mon. Wea.
Rev., 130, 2167-2187.
Oke, P. R., A. Schiller, D. A. Griffin, and G. B. Brassington, 2005: Ensemble data
assimilation for an eddy-resolving ocean model of the Australian region. Q.
J. Roy. Met. Soc., 131, 3301-3311.
Palmer, T. N. and F. J. Doblas-Reyes, A. Weisheimer, and M. J. Rodwell, 2008:
Toward Seamless Prediction: Calibration of Climate Change Projections Using
Seasonal Forecasts, Bull. Amer. Met. Soc., 89, 459-470.
Schiller, A., J. S. Godfrey, P. C. McIntosh, G. Meyers, N. R. Smith, O.Alves, G.
Wang, and R. Fiedler, 2002: A New Version of the Australian Community
Ocean Model for Seasonal Climate Prediction. CSIRO Marine Research
Report No. 240.
Smith, N. R., J. E. Blomley, and G. Meyers, 1991: A univariate statistical interpolation scheme for subsurface thermal analyses in the tropical oceans.
Prog. Oceanog., 28, 219-256.
Tippett, M. K., L. Goddard, and A. G. Barnston, 2005: Statistical-dynamical
seasonal forecasts of central-southwest Asian winter precipitation. J.
Climate, 18, 1831-1843.
Voldoire, A., B. Timbal, and S. Power, 2002: Statistical-dynamical seasonal
forecasting. BMRC Research Report No. 87, Bureau of Meteorology
Research Centre, 62 pp.
Wang, G., O. Alves, and H. Hendon, 2008: SST Skill Assessment from the new
POAMA-1.5 system. SEACI milestone report.
Ward, M. N. and A. Navarra, 1997: Pattern analysis of SST-forced variability in
ensmeble GCM simulations: Examples over Europe and the tropical Pacific. J.
Climate, 10, 2210-2220.
Wilks, D. 2006: Statistical Methods in the Atmospheric Sciences, Academic Press,
592 pp.
Attachment 2.
Dynamical, statistical-dynamical, and homogeneous multi-model
ensemble forecasts of Australian spring season rainfall
Eun-Pa Lim, Harry H. Hendon, David L. T. Anderson and Oscar Alves
ABSTRACT
We assess skill of the Australian Bureau of Meteorology dynamical seasonal
forecast model, POAMA, for probabilistic forecasts of spring season rainfall in
Australia and examine the feasibility of increasing forecast skill through statistical
post-processing. Two statistical post-processing techniques are explored: calibrating
POAMA’s rainfall prediction against observations and using predicted large-scale
climate components to infer regional rainfall over Australia (bridging). We also
introduce a “homogeneous” multi-model ensemble prediction method that consists of
the direct rainfall prediction from POAMA together with the two statistically postprocessed predictions.
Using hindcasts for the period 1980-2006, the homogeneous multi-model
ensemble significantly improves skill over both the direct predictions from the model
and from these statistically post-processed predictions in regard to forecasting
below/above median rainfall. The forecast reliability is also improved.
1. Introduction
Over the last two decades, a significant amount of research and effort has been poured
into utilizing general circulation models for seasonal climate prediction (e.g.
Stockdale et al. 1998) and improving their skill. The strength of dynamical models is
that they incorporate linear and non-linear processes in the atmosphere and ocean, so
that climate evolution, which is chaotic in nature, can be captured in the probabilistic
domain through multi-member or multi-model ensemble forecasts. This merit of
dynamical models is especially important for climate prediction over subtropical and
extratropical regions where climate is governed by internal dynamics as much as by
the lower boundary forcing (i.e. sea surface temperature).
In order to improve the quality of seasonal climate forecasts over Australia,
the Australian Bureau of Meteorology (BOM) together with Commonwealth
Scientific and Industrial Research Organization (CSIRO) has been developing a
coupled atmosphere-ocean climate prediction model, POAMA (Predictive Ocean
Atmosphere Model for Australia). The current operational version of POAMA (v1.5b)
demonstrates internationally competitive skill in ENSO predictions (Wang et al. 2008;
also see http://iri.columbia.edu/climate/ENSO/currentinfo/SST_table.html). This
implies some useful skill in POAMA prediction for the Australian seasonal climate,
based on the model’s ability to predict tropical SST variations especially those
associated with ENSO and the strong association of Australian climate variability
with ENSO (e.g. McBride and Nicholls 1987). The first aim of this study is, therefore,
to assess the skill of POAMA seasonal forecasts for Australian rainfall, using its
retrospective forecast set for the period 1980-2006.
Apart from purely dynamical prediction, statistical-dynamical prediction is
another way of making use of products of a dynamical model for regional climate
prediction. Previous studies have suggested that reasonable forecast skill can be
obtained by using dynamically predicted climate components in a statistical model
(this method is also referred to as statistical post-processing; e.g., Feddersen et al.
1999). Statistical calibration and bridging (downscaling) are common techniques for
statistical-dynamical prediction. Statistical calibration attempts to adjust the spatial
pattern of the variability of a dynamically predicted variable, comparing it to its
observed counterpart (Ward and Navarra 1997, Feddersen et al. 1999, Mo and Straus
2002, Kang et al. 2004). Whereas, statistical bridging is the prediction of a variable
through its statistical relationship with any model-predicted large-scale climate
component such as temperature, pressure and wind fields (Voldoire et al. 2002,
Tippett et al. 2005, Lin and Derome 2005, Kang et al. 2007). Therefore, the second
aim of this study is to test the feasibility of increasing regional forecast skill of rainfall
through statistical post-processing. We explore correcting rainfall prediction against
observation (calibration) and bridging some predicted large-scale climate component,
for which POAMA shows skill, to rainfall over Australia. The component chosen for
this study is mean sea level pressure (MSLP) as MSLP is a climate variable directly
related to rainfall and also serves as an atmospheric bridge between the lower
boundary forcing and regional climate.
Finally, we have designed a “homogeneous” multi-model ensemble (HMME)
prediction method1 that consists of the direct forecasts from POAMA and the
statistical-dynamical forecasts in order to seek improved rainfall forecast skill over
Australia. It has been shown in previous studies that the overall performance of a
multi-model ensemble system can be better than the individual single models included
in the multi-model ensemble system as a result of offsetting errors (Zebiak 2003;
Hagedorn et al. 2005, Weigel et al. 2008). Provided that there is some independent
information available in the predictions that go into our HMME, it is worth exploring
if this HMME approach can improve the prediction skill in a fashion similar to that
achieved by a conventional multi-model ensemble based on different dynamical
models, but in a more cost-effective way.
Details of the POAMA system and verification methods will be described in
section 2. Then we will analyse the skill of probabilistic forecasts for below/above
median rainfall from POAMA in section 3. In section 4 the development methods of
statistical-dynamical models are discussed and the skill from these models will be
compared to that of POAMA. This will be followed by the skill assessment of the
HMME in section 5. Finally, concluding remarks will be given in section 6.
We call this system “homogeneous” multi-model ensemble as the individual models participating in
this system utilize the outputs of one model - POAMA.
1
2. Description of the POAMA system and verification methods
The POAMA version 1.5b system consists of the BOM Atmospheric Model version 3
(BAM3; Colman et al. 2005) and the Australian Community Ocean Model version 2
(ACOM2; Schiller et al. 2002, Oke et al. 2005). The horizontal structure of BAM3 is
represented by spherical harmonics with a triangular truncation at wave number 47
(denoted T47, which has approximately 250 km resolution), and the vertical variation
is represented by 17 sigma levels. ACOM2 has a zonal resolution of 2° longitude and
a telescoping meridional resolution of 0.5° latitude within 8° latitude of the equator,
gradually changing to 1.5° latitude near the poles. Vertically, ACOM2 has 25 levels,
with 12 levels in the top 185 m. The atmosphere and ocean models are coupled every
3 hours by the Ocean Atmosphere Sea Ice Soil (OASIS) coupling software (Valke
2000).
POAMA forecasts are initialized with observed atmospheric and oceanic
conditions. The atmospheric initial conditions are provided by the atmosphere and
land initialization scheme called ALI (Hudson and Alves 2007). ALI nudges zonal
and meridional winds, temperature, and humidity from BAM3 to those of the ERA-40
reanalysis for the hindcasts for 1980-2001 and those of the BOM NWP system
(GASP) for the latter period of the hindcasts. Likewise, ALI nudges the BAM3
variables to the analysis from the BOM NWP system for the real-time forecasts. The
initial conditions produced from ALI are similar to the analyses of ERA-40/GASP but
cause less initial shock than if the ERA-40/GASP analyses were directly used as
initial conditions. Land surface is initialized to be consistent with the atmospheric
conditions. Furthermore, ALI enables the initial conditions of the hindcasts to be more
consistent with those of the real-time forecasts, and therefore, the skill assessed in the
hindcasts can be a good indicator of the skill of the real-time forecasts (Hudson and
Alves 2007).
The ocean data assimilation system provides an estimate of the present state of
the tropical upper ocean, based on the optimum interpolation (OI) technique of using
available sub-surface temperature observations (Smith et al. 1991), together with a
strong relaxation of the SST to observed analyses. Further details of the POAMA
system
can
be
found
in
Lim
et
al.
(2009)
and
http://poama.bom.gov.au/documentation/index.html.
For this study, ten-member ensemble hindcasts for the period of 1980-2006
from POAMA v1.5b were used. We limit our interest to the probabilistic forecasts
based on the ten members for above median rainfall in the austral spring (SeptemberOctober-November; SON) when skilful rainfall prediction would be greatly valued for
decision making for water management and agriculture in Australia. The ensemble
hindcasts of Australian rainfall prediction were verified against the Australian
National Climate Centre gridded rainfall data (0.25° lat x 0.25° lon; Jones and
Weymouth 1997).
Rainfall anomalies from the seasonal cycle of both model and verification data
sets were seasonally averaged and then standardized by their respective standard
deviations. The seasonal cycle of the model is a function of forecast start month and
lead time. By forming anomalies relative to the model’s climatology and
standardizing the anomalies relative to the model’s variability, the mean model bias is
removed and the hindcast rainfall is “calibrated” to have the same standard deviation
as observed.
Rainfall anomalies of both model and observation were obtained with
thorough cross-validation: 1) rainfall data in each verification year were left out; 2)
the climatology, standard deviation, and median were obtained with the remaining
data for both model and observation; 3) the rainfall anomalies of POAMA and
observation in the year left out were calculated based on the climatology and standard
deviation obtained in the training period and were compared to the medians of the
training data sets of POAMA and observation; 4) the probabilistic forecast of above
median rainfall with ten ensemble members was verified. This process was repeated
throughout the entire period. Hereafter, rainfall refers to the standardized rainfall
anomaly.
3. POAMA forecasts
As a compact way of displaying many of the features of probabilistic forecasts, the
reliability diagram is widely used (Wilks 2006; Palmer et al. 2008). Figure 1 displays
the reliability of seasonal mean rainfall forecasts for all grid points over Australia for
SON at lead time zero month 2 (LT 0) based on the entire 27 year record of hindcasts.
POAMA demonstrates moderate reliability in predicting above median rainfall,
competing with the climatological forecast (i.e. 50% for below/above median) (which
is indicated by its closeness to the “no skill” line (Wilks 2006)). In the categories of
50-80% probabilities of exceeding median rainfall, POAMA forecasts are more
erroneous than the climatological forecast. On the other hand, forecasts for greater
than 90% chance of being above median rainfall are reliable, and such overconfident
forecasts are not rare in POAMA (9% of the total forecast population in SON). Figure
1 indicates that POAMA has reasonably good forecast resolution having a broad
range of the frequency of observed rainfall occurrence, given forecast probabilities.
In order to further assess the skill of the probabilistic rainfall forecasts, we
compute the Brier Score (BS), which is the mean squared error of probability forecasts
against the occurrence of an event (Wilks 2006, their equation 7.34):
1 n
BS   ( Fk  Ok ) 2
n k 1
(1)
where n is the total number of the hindcast years in this study, Fk is the forecast
probability for above the median rainfall and Ok is the observed rainfall occurrence in
the kth year. Ok is 1 if it is above the median, or 0 if it is not. The Brier Skill Score
(BSS) of POAMA, which is expressed as a % change of the BS of the POAMA
forecasts compared to the BS of the climatological forecast, is displayed in Figure 2.
The BS of the climatological forecast was computed with a sampling error correction
term as suggested by Weigel et al (2007) and Mason and Stephenson (2008). In
predicting above median rainfall, forecasts of spring rainfall at LT 0 from POAMA
exhibit up to 30-40% skill improvement over the climatological forecast in eastern
Australia whereas the performance of POAMA is comparable to the climatological
forecast in the west.
2
The period of time between the issue time of the forecast and the beginning of the forecast validity
period (WMO users guide, http://www.bom.gov.au/wmo/lrfvs/users.shtml)
4. Utilization of POAMA forecasts in statistical models: statistical-dynamical
prediction
As a method to improve the direct predictions of rainfall from POAMA, we develop
two statistical models that use as inputs dynamically predicted variables from
POAMA: a statistical calibration scheme using POAMA’s prediction of Australian
rainfall (using all model grid points over Australia) and a statistical bridging scheme
using POAMA predicted mean sea level pressure (MSLP) over the Southern
Hemisphere (SH) extratropical region (20°-75°S). The rationale for the calibration
scheme is that direct forecasts of rainfall from POAMA may suffer from systematic
bias as a result of, for instance, a systematic bias in the rainfall teleconnections to
Australia associated with ENSO, which can be corrected a posteriori. The rationale
for the bridging scheme is that POAMA may have some skill in predicting the largescale variations of circulation (in this case, MSLP) that exert a strong control over
local rainfall but that the conversion into skilful rainfall predictions in the POAMA
model is not faithfully realized.
The basic algorithm is the same in both the calibration and bridging models.
First, we extract the temporally covarying components of the predictors with observed
Australian rainfall by using the Singular Value Decomposition Analysis (SVDA)
technique (Bretherton et al. 1992, Ward and Navarra 1997). This technique expands a
predictor field X(i,t) and a predictand field Y(j,t) in terms of spatial patterns G(i,m)
(right singular vector) and H(j,m) (left singular vector), respectively, which maximize
the temporal covariance between the two fields. The expansion coefficients, u(m,t)
and v(m,t,) of those patterns are given as
M
X ( i ,t )   u( m ,t )G( i , m )
(2)
m 1
M
Y ( j ,t )   v( m ,t )H ( j , m )
(3)
m 1
where i and j indicate the number of grid points of X and Y, respectively, t indicates
the number of time, and m and M represent SVD modes.
Second, we linearly regress the predictand field Y(j,t) – in this case observed
Australian rainfall - on the time series of a selected number of SVD modes of the
predictor (u(m,t)), using multiple linear regression technique, i.e.
M
Ŷ ( j ,t )   A( j , m )u( m ,t )
(4)
m 1
where Ŷ ( j ,t ) is the predicted Y(j,t) in the regression model, and A(j,m) is a set of
regression coefficients which minimize the expected root-mean-square difference
between Ŷ ( j ,t ) and Y(j,t). In this step, we need to decide how many SVD modes to
include as predictors in this regression model. According to Mo and Straus (2002), the
number of predictors in a multiple linear regression model should satisfy two criteria
to avoid overfitting: There should be a minimum of 10 degrees of freedom (t-M-1
≥10), and there should be at least five data points per predictor (M ≤ t/5).
Consequently, the leading five SVD modes, which explain 96% of the
covariance between POAMA predictions of Australian rainfall and observed
Australian rainfall, were retained in our statistical calibration model. The first five
SVD time series of POAMA predicted rainfall explain up to 50% of observed rainfall
variance in Western Australia but much less in the east (Fig. 3).
For the statistical bridging model based on POAMA predictions of MSLP over
the SH, our predictors are the projections of POAMA predictions of MSLP onto the
leading 10 EOF modes of observed MSLP (the first four EOF modes are displayed in
Fig. 4). The resultant principal component time series X(i,t), where i runs from 1-10,
were plugged into the SVDA algorithm, together with observed rainfall. Such
prefiltering of MSLP data with EOF analysis reduces the number of degrees of
freedom in the MSLP field, and therefore, can result in more stable SVD modes
(Barnett and Preisendorfer 1987, Bretherton et al. 1992).
The 10 leading EOFs of observed MSLP explain 90% of the total MSLP
variability in the SH extratropics (20°-75°S) according to our analysis on the NCEPDOE reanalysis II data (Kanamitsu et al. 2002). The first EOF mode (Fig. 4)
represents the Southern Annular Mode (SAM; Thompson and Wallace 2000, Marshall
2003). The second EOF mode shows the wavetrain pattern known as the PacificSouth American pattern, and it has been suggested to be associated with ENSO,
especially in response to western Pacific SST variability (Karoly 1989, Mo and
Higgins 1998). The correlation of its principal component time series (PCs) and the
Niño 3 SST index is -0.5. The third EOF mode shows some association with the El
Niño Modoki (Ashok et al. 2006), exhibiting a correlation of -0.6 with the El Niño
Modoki index which captures the variability of the sea surface temperature (SST)
between the central and the western and eastern basins of the tropical Pacific Ocean.
The fourth EOF mode also has a moderate correlation of -0.4 with the Niño 3 index.
These four EOF modes explain about 70% of the total variance of the SH
extratropical MSLP. The higher modes EOFs (5th-10th) used in this study explain
about 15% of the total variance of the MSLP.
The first 10 leading EOF modes of observed MSLP account for more than
40% of the rainfall variance over most of the country and up to 70% in centralwestern Australia (Fig 5). The rainfall in the northern and western parts of the country
is associated with the first four leading modes of MSLP EOFS shown in Figure 4
whereas the rainfall in the south and central parts is largely explained by the higher
mode EOFs. In order to understand the upper limit of predictability of rainfall by
perfectly predicted SH extratropical MSLP, we re-calculated observed rainfall
anomalies and MSLP PCs in a cross-validated manner (i.e. each year PC values and
rainfall anomalies were obtained from the climatology and EOF patterns of the
independent period) and then the explained variance of rainfall was computed by
squaring its correlation with the cross-validated MSLP PCs (Fig. 6). The rainfall
variance explained by MSLP PCs with the assumption of perfect forecasts of MSLP
PCs is small in the central-eastern part of Australia due to the large year-to-year
variations of the higher EOF modes of MSLP variability (Fig. 6 (a), (c)). In contrast,
the potential predictability of the rainfall in Western Australia is high (40-70%) as the
rainfall is associated with the first few leading modes of MSLP variability which are
more stable through the entire hindcast years.
The skill of POAMA for predicting the leading 10 EOFs of MSLP is assessed
by the correlation of the observed PCs with the predicted PCs that are obtained by
projection of the predicted MSLP onto the observed EOFs (Table 1). It is encouraging
to see that the first few EOF modes of MSLP variability, which represent SAM
(EOF1) and the tropical SST related modes (EOFs 2 and 3), are skilfully predictable
by POAMA at least at LT 0.
Figure 7 (a) shows the right singular vector of the leading five SVD modes of
predicted MSLP with observed Australian rainfall. The first five SVD modes explain
98% of the total covariance between observed rainfall and predicted (with
prefiltering) MSLP, and the temporal coefficients of the right hand vectors for the
five SVD modes account for 20-40% of observed rainfall variance in northern and
eastern Australia in spring (Fig. 7 (f)). The temporal coefficients of the first 5 right
hand vectors were used in the regression model (eq. (4)), and a set of regression
coefficients was obtained for rainfall prediction.
These statistical-dynamical models were developed and verified with careful
cross-validation by leaving out the verification year while the models were set up with
the data in the remaining years. This process was repeated throughout the entire
hindcast period.
Although some local improvements are found in different parts of Australia in
the statistical calibration and bridging models, forecast skill of both statisticaldynamical models is not as good as that of raw POAMA in predicting above median
rainfall over most of Australia (Fig. 8). The reason for the overall poorer performance
of the statistical-dynamical models compared to raw POAMA prediction is likely to
be related to the thorough cross-validation process within the short period of the
hindcasts coupled together with only modest signal strength. In the case of
constructing a statistical bridging model, it is difficult to find a dominant predictor to
explain the variability of Australian rainfall over the entire country and in all seasons
(e.g. Murphy and Timbal 2008). Tropical SST might be a better candidate as a
predictor field in that sense as there is a reasonably strong empirical relationship
between Australian rainfall and ENSO, but POAMA ensemble forecasts for tropical
SST tend to have very narrow spread, which would increase the confidence of the
already over-confident rainfall forecasts from POAMA (cf. Fig. 1).
5. Homogeneous Multi-Model Ensemble prediction
Although the statistical-dynamical models by themselves do not outperform the direct
predictions from POAMA, they possibly could contribute to skill improvement in the
context of a multi-model ensemble, provided that there is some independent
information in each component model (Doblas-Reyes et al. 2000, Zebiak 2003,
Coelho et al. 2004, Hagedorn et al. 2005). Some indication of the independence of the
3 sets of forecasts is shown in Figure 9, which displays the correlation of the
ensemble mean rainfall anomaly from the two statistical-dynamical models with the
direct prediction from POAMA. While the bridging forecasts are clearly largely
independent (low correlation) from the direct predictions, the calibrated forecasts are
dependent on the direct predictions in the south east where POAMA demonstrates its
best skill in predicting spring rainfall. Nonetheless, about 20% of the variance of the
calibrated forecasts is unexplained by the direct predictions over the south east and
the unexplained variance increases away from the south east. We constructed our
HMME by simply combining the rainfall predictions of these two statisticaldynamical models together with the direct rainfall predictions from POAMA.
The reliability of the HMME predictions is shown in Figure 10. Forecast
reliability from the HMME is much improved compared to the reliability of the direct
prediction of rainfall from POAMA (Fig. 1) as the forecasts in each probability bin
are more closely aligned with the perfect reliability line. Furthermore, the forecast
frequency as a function of forecast probability is more normally distributed, and the
occurrence of over-confident forecasts is markedly reduced. According to Figure 11
(a) there is up to 20-30% improvement of the BS for the HMME compared to that of
the direct predictions from POAMA over the northern and southern parts of the
country in SON at LT 0. However, near the border of New South Wales and
Queensland where direct prediction from POAMA exhibits about 40% higher skill
than the climatological forecast, skill is reduced by the HMME. Despite such negative
impacts locally, the HMME provides much improved forecasts compared to the
climatological forecast over most of the country except for the central to northern part
of Western Australia (Fig. 11 (b)). The improvement by the HMME over POAMA
and over the climatological forecast seems to stem from keeping the probability of
rainfall signal in the component models and at the same time cancelling the forecast
errors. As pointed out above, the HMME forecasts are much less emphatic than the
raw forecasts from POAMA, and Weigel et al. (2008) reported that the multi-model
ensemble approach can result in skill improvement over a single model whose
ensemble forecasts are overconfident.
6. Concluding remarks
We have analysed the characteristics and skill of probabilistic forecasts of Australian
spring rainfall from the POAMA dynamic coupled model forecasting system and have
examined the feasibility of forecast skill improvement through statistical-dynamical
prediction methods. A 10 member ensemble of hindcasts for 1980-2006 generated
from the current operational version of POAMA was employed for this study.
Direct prediction from POAMA of below/above median rainfall in the austral
spring exhibits poor to moderate reliability at LT 0. However, POAMA outperforms a
climatological forecast over the eastern part of the country. In order to examine if
additional skill improvement is obtainable through statistical post-processing, we
have developed a statistical calibration scheme and a statistical bridging scheme that
use POAMA’s prediction of Australian rainfall and POAMA’s predictions of
extratropical MSLP in the SH, respectively, as predictors for Australian rainfall.
Forecast skill, as measured by the BSS, from these statistical-dynamical prediction
schemes is less than the direct rainfall prediction from POAMA in most of Australia
basically due to the modest strength of the relationship between Australian spring
rainfall and the dominant modes of large-scale climate components (e.g. SAM,
ENSO) and also due to POAMA’s inability to adequately transfer the climate signals
to Australian rainfall in the case of statistical calibration.
As an innovative experiment, we combined the direct rainfall predictions from
POAMA together with the statistically calibrated and bridged predictions to make a
multi-model ensemble prediction. In SON at LT 0 our homogeneous multi-model
ensemble scheme improves forecast reliability significantly and its BS is higher than
POAMA or the climatological forecast over broad areas of Australia.
The POAMA seasonal forecast system is continually evolving and improving
– subsequent versions of POAMA will address issues such as correcting model bias
and drift, increasing model horizontal resolution, improving model physics, and
improved initialization so as to provide improved prediction of regional climate
variability. The potential for further skill improvement using statistical postprocessing seems to be somewhat limited due to the short length of hindcasts (post
1982) and the large level of climate variability and climate change in Australia.
However, our study suggests that even if statistical or statistical-dynamical forecasts
are not very skilful, they can still contribute to improving seasonal forecast skill as
participants in a multi-model ensemble system.
Acknowledgements
This research was supported in part by the South Eastern Australian Climate Initiative
(SEACI; http://www.mdbc.gov.au/subs/seaci/ ).
.
REFERENCES
Ashok, K., S. K. Behera, S. A. Rao, and H. Weng, 2007: El Niño Modoki and its
possible
teleconnection.
J.
Geophys.
Res.,
112,
C1107,
doi:10.1029/2006JC003 798.
Barnett, T. P. and R. W. Preisendorfer, 1987: Origins and levels of monthly and
seasonal forecast skill for United States surface air temperatures determined
by canonical correlation analysis. Mon. Wea. Rev., 115, 1825–1850.
Bretherton, C. S., C. Smith., and J. M. Wallace, 1992: An intercomparison of
methods for finding coupled patterns in climate data. J. Climate, 5, 541–560.
Coelho, C. A. S., S. Pezzulli, M. Balmaseda, F. J. Doblas-Reyes, and D. B.
Stephenson, 2004: Forecast calibration and combination: A simple Bayesian
approach for ENSO. J. Climate, 17, 1504-1516.
Colman, R., L. Deschamps, M. Naughton, L. Rikus, A. Sulaiman, K. Puri, G. Roff,
Z. Sun, and G. Embery, 2005: BMRC Atmospheric Model (BAM) version
3.0: Comparison with mean climatology. BMRC Research Report No. 108,
Bureau of Meteorology Research Centre, 32 pp. (available from
http://www.bom.gov.au/bmrc/pubs/researchreports/researchreports.htm).
Doblas-Reyes, F. J., M. Deque, and J.-Ph. Piedelievre, 2000: Multi-model spread
and probabilistic seasonal forecasts in PROVOST. Quart. J. Roy. Meteor.
Soc., 126, 2069-2088.
Drosdowsky, W. and L. Chambers, 2001: Near-global sea surface temperature
anomalies as predictors of Australian seasonal rainfall. J. Climate, 14, 16771687.
Feddersen, H., A. Navarra, and M. N. Ward, 1999: Reduction of model systematic
error by statistical correction for dynamical seasonal predictions. J. Climate,
12, 1974-1989.
Hagedorn, R., F. J. Doblas-Reyes, and T. N. Palmer, 2005: The rationale behind the
success of multi-model ensembles in seasonal forecasting – I. Basic concept.
Tellus, 57A, 219-233.
Hudson, D. and O. Alves, 2007: The impact of land-atmosphere initialisation on
dynamical seasonal prediction. BMRC Research Report No. 133, Bureau of
Meteorology Research Centre, p 19-22.
Jones, D. A. and G. Weymouth, 1997: An Australian monthly rainfall dataset.
Technical Report 70, Bureau of Meteorology.
Kanamitsu, M., R. E. Kistler, and R. W. Reynolds, S.-K. Yang, J. J. Hnilo, M.
Fiorino, and G. L. Potter, 2002: NCEP/DOE AMIP-II Reanalysis (R-2).
Bull. Amer. Meteor. Soc., 83, 1641–1643.
Kang, I.-S., J.-Y. Lee, and C.-K. Park, 2004: Potential predictability of summer
mean precipitation in a dynamical seasonal prediction system with
systematic error correction. J. Climate, 17, 834-844.
Kang, H., K.-H. An, C.-Kyu Park, A. L. S. Solis and K. Stitthichivapak, 2007:
Multimodel output statistical downscaling prediction of precipitation in the
Philippines and Thailand. Geophys. Res. Lett., 34, L15710,
doi:10.1029/2007GL030730
Lim, E.-P., H. H. Hendon, D. Hudson, G. Wang, and O. Alves, 2009: Dynamical
forecast of inter El-Niño variations of tropical SST and Australian spring
rainfall, submitted to Mon.Wea. Rev.
Lin, H. and J. Derome, 2005: Correction of atmospheric dynamical seasonal forecasts using the leading ocean-forced spatial patterns. Geophys. Res. Lett., 32,
L14804.
Marshall, G. J., 2003: Trends in the Southern Annular Mode from observations and
reanalyses. J. Clim., 16, 4134-4143.
Mason, S. J. and D. B. Stephenson, 2008: How do we know whether seasonal
climate forecasts are any good?, Seasonal Climate: Forecasting and
Managing
Risk,
edited
by
Alberto Troccoli,
Mike Harrison,
David L. T. Anderson and Simon J. Mason, Springer, pp 259-289
McBride, J. L., and N. Nicholls, 1983: Seasonal relationships between Australian
rainfall and the Southern Oscillation. Mon. Wea. Rev., 111, 1998–2004.
Mo, K. C., and R. W. Higgins, 1998: The Pacific–South American Modes and
Tropical Convection during the Southern Hemisphere Winter. Mon. Wea.
Rev., 126, 1581–1596.
Mo, R. and D. M. Straus, 2002: Statistical-dynamical seasonal prediction based on
principal component regression of GCM ensemble integrations. Mon. Wea.
Rev., 130, 2167-2187.
Murphy, B. F. and B. Timbal, 2008: A review of recent climate variability and
climate change in southeastern Australia. Int. J. Climatol. 28, 859–879.
Oke, P. R., A. Schiller, D. A. Griffin, and G. B. Brassington, 2005: Ensemble data
assimilation for an eddy-resolving ocean model of the Australian region. Q.
J. Roy. Met. Soc., 131, 3301-3311.
Palmer, T. N. and F. J. Doblas-Reyes, A. Weisheimer, and M. J. Rodwell, 2008:
Toward Seamless Prediction: Calibration of Climate Change Projections Using
Seasonal Forecasts, Bull. Amer. Met. Soc., 89, 459-470.
Schiller, A., J. S. Godfrey, P. C. McIntosh, G. Meyers, N. R. Smith, O. Alves, G.
Wang, and R. Fiedler, 2002: A New Version of the Australian Community
Ocean Model for Seasonal Climate Prediction. CSIRO Marine Research
Report No. 240.
Smith, N. R., J. E. Blomley, and G. Meyers, 1991: A univariate statistical interpolation scheme for subsurface thermal analyses in the tropical oceans.
Prog. Oceanog., 28, 219-256.
Stockdale, T. N., D. L. T. Anderson, J. O. S. Alves, M. A. Balmaseda, 1998: Global
seasonal rainfall forecasts using a coupled ocean-atmosphere model. Nature
392, 370-373.
Thompson, D.W. J. and J. M.Wallace, 2000: Annular modes in the extratropical
circulation. Part I: Month-to-month variability. J. Climate, 13, 1000–1016.
Tippett, M. K., L. Goddard, and A. G. Barnston, 2005: Statistical-dynamical
seasonal forecasts of central-southwest Asian winter precipitation. J.
Climate, 18, 1831-1843.
Valke, S, Terray, L, Piacentini, A, 2000: The OASIS coupled user guide version
2.4, Technical Report TR/ CMGC/00-10, CERFACS
Voldoire, A., B. Timbal, and S. Power, 2002: Statistical-dynamical seasonal
forecasting. BMRC Research Report No. 87, Bureau of Meteorology
Research Centre, 62 pp.
Wang, G., O. Alves, and H. Hendon, 2008: SST Skill Assessment from the new
POAMA-1.5 system. SEACI milestone report.
Ward, M. N. and A. Navarra, 1997: Pattern analysis of SST-forced variability in
ensemble GCM simulations: Examples over Europe and the tropical Pacific. J.
Climate, 10, 2210-2220.
Weigel, A. P., M. A. Liniger and C. Appenzeller, 2008: Can multi-model
combination really enhance the prediction skill of probabilistic ensemble
forecasts?, Q. J. R. Meteorol. Soc., 134, 241-260.
Wilks, D. 2006: Statistical Methods in the Atmospheric Sciences, Academic Press,
592 pp.
Zebiak, S. E., 2003: Research potential for improvements in climate prediction. Bull.
Amer. Met. Soc., 84, 1692-1696.
List of Figures
Figure 1: Reliability diagram of POAMA prediction for above median rainfall, taking
into account the 27 year hindcasts over all grid points over Australia at lead time 0 (LT
0). The size of solid dots is proportional to the population in each probability bin. The
dots below (above) the ‘no skill’ line in the forecast probabilities smaller (larger) than the
climatological forecast (0.5) contribute to positive Brier Skill Score (BSS).
Figure 2: BSS which indicates a % improvement of POAMA forecasts over the
climatological forecast for exceeding median rainfall for the period of 1980-2006. The
contour interval indicates10% change.
Figure 3: (a)-(e) Correlation of observed rainfall with the expansion coefficients of
the first 5 leading SVD right vectors of POAMA predicted rainfall at LT 0 in 19812006, and (f) explained variance of the observed rainfall by the 5 leading SVD modes
of POAMA predicted rainfall.
Figure 4: First four standardized principal components of observed MSLP (PCs; left
panel) and the corresponding EOF patterns obtained by regression of observed MSLP
field onto the standardized PCs (right panel) in the period of 1981-2006 from the
NCEP-DOE reanalysis II data set. The domain is 20°-75°S. Contour interval is 0.5
hPa. The number in the parentheses indicates the MSLP variance explained by each
mode.
Figure 5: Explained observed rainfall variance by the first 10 observed EOF modes of
MSLP variability in SON in the period of 1981-2006.
Figure 6: Explained rainfall variance by the first 10 PCs of MSLP obtained with a
perfect forecast assumption (i.e. each year PC values and rainfall anomaly were
obtained from the climatology and EOF patterns of the independent period with the
observed data set.)
Figure 7: (a) The first five right vectors of the SVD modes of POAMA predicted
MSLP at LT0 (represented by POAMA prediction of observed 10 EOFs of MSLP),
(b)-(e) Correlation of observed rainfall with the temporal coefficients of the five SVD
right vectors shown in (a), and (f) explained variance of the observed rainfall by the
five SVD modes in the period of 1981-2006.
Figure 8: BSS of statistical calibration and bridging schemes at LT 0, taking POAMA
forecasts as reference. The contour interval is 10% change.
Figure 9: Correlation of ensemble means between direct rainfall predictions from
POAMA and the statistical-dynamical predictions at LT 0. Correlation coefficients
greater than 0.4 are statistically significant at the 95% confidence level.
Figure 10: The same as Figure 1 except for the reliability diagram of the HMME at
LT 0.
Figure 11: BSS of the HMME at LT 0, taking POAMA forecasts (left) and the
climatological forecast (right) as reference.
SON
Observed relative frequency
1
0.8
0.6
POAMA
Perfect reliability
no skill
0.4
0.2
0
0
0.2
0.4
0.6
0.8
1
Forecast probability
Figure 1: Reliability diagrams of POAMA prediction for above median rainfall,
taking into account the 27 year hindcasts over all grid points over Australia at lead
time 0 (LT 0). The size of solid dots is proportional to the population in each
probability bin. The dots below (above) the ‘no skill’ line in the forecast probabilities
smaller (larger) than the climatological forecast (0.5) contribute to positive Brier Skill
Score (BSS).
Figure 2: BSS expressed as a % improvement of POAMA forecasts over the
climatological forecast for exceeding median rainfall for the period of 1980-2006. The
contour interval is 10%.
(a) SVD 1
(c) SVD 3
(e) SVD 5
(b) SVD 2
(d) SVD 4
(f) Explained rainfall variance
Figure 3: (a)-(e) Correlation of observed rainfall with the expansion coefficients of
the first 5 leading SVD right vectors of POAMA predicted rainfall at LT 0 in 19812006, and (f) explained variance of the observed rainfall by the 5 leading SVD modes
of POAMA predicted rainfall.
PC1
Standardized amplitude
3
2
1
0
-1
-2
2005
2003
2001
1999
1997
1995
1993
1991
1989
1987
1985
1983
1981
-3
Year
(a) EOF 1 (33.7%)
PC2
Standardized amplitude
3
2
1
0
-1
-2
2005
2003
2001
1999
1997
1995
1993
1991
1989
1987
1985
1983
1981
-3
Year
(b) EOF 2 (15.9%)
PC3
Standardized amplitude
3
2
1
0
-1
-2
2005
2003
2001
1999
1997
1995
1993
1991
1989
1987
1985
1983
1981
-3
Year
(c) EOF3 (11.2%)
PC4
Standardized amplitude
3
2
1
0
-1
-2
2005
2003
2001
1999
1997
1995
1993
1991
1989
1987
1985
1983
1981
-3
Year
(d) EOF 4 (8.6%)
Figure 4: First four standardized principal components of observed MSLP (PCs; left
panel) and the corresponding EOF patterns obtained by regression of observed MSLP
field onto the standardized PCs (right panel) in the period of 1981-2006 from the
NCEP-DOE reanalysis II data set. The domain is 20°-75°S. Contour interval is 0.5
hPa. The number in the parentheses indicates the MSLP variance explained by each
mode.
(a) PC1-10
(b) PC1-4
(c) PC5-10
Figure 5: Explained observed rainfall variance by the first 10 observed EOF modes of
MSLP variability in SON in the period of 1981-2006.
(a) PC1-10
(b) PC1-4
(c) PC5-10
Figure 6: Explained rainfall variance by the first 10 PCs of MSLP obtained with a
perfect forecast assumption (i.e. each year PC values and rainfall anomaly were
obtained from the climatology and EOF patterns of the independent period with the
observed data set.)
(a) Five SVD right vectors
1
0.8
0.6
PC1
PC2
0.4
PC9 PC1
0.2
PC9 PC1
PC9
PC9
0
PC9
-0.2
-0.6
-0.8
PC2
PC1
PC2
-0.4
PC2
PC2
PC1
-1
SVD1
SVD2
SVD3
SVD4
SVD5
(b) SVD1
(c) SVD2
(d) SVD3
(e) SVD4
(f) SVD5
(g) Explained rainfall variance
Figure 7: (a) The first five right vectors of the SVD modes of POAMA predicted
MSLP at LT0 (represented by POAMA prediction of observed 10 EOFs of MSLP),
(b)-(f) Correlation of observed rainfall with the temporal coefficients of the five SVD
right vectors shown in (a), and (g) the explained variance of the observed rainfall by
the five SVD modes in the period of 1981-2006.
(a) Statistical Calibration
(b) Statistical Bridging
Figure 8: BSS of statistical calibration and bridging schemes at LT 0, taking POAMA
forecasts as reference. The contour interval is 10% change.
(a) Statistical Calibration
(b) Statistical Bridging
Figure 9: Correlation of ensemble means between direct rainfall predictions from
POAMA and the statistical-dynamical predictions at LT 0. Correlation coefficients
greater than 0.4 are statistically significant at the 95% confidence level.
SON
Observed relative frequency
1
0.8
0.6
HMME
Perfect reliability
no skill
0.4
0.2
0
0
0.2
0.4
0.6
0.8
1
Forecast probability
Figure 10: The same as Figure 1 except for the reliability diagram of the HMME at
LT 0.
(a) HMME over POAMA
(b) HMME over the climatological forecast
Figure 11: BSS of the HMME at LT 0, taking POAMA forecasts (left) and the
climatological forecast (right) as reference.
r
PC1
0.71
PC2
0.58
PC3
0.73
PC4
0.17
PC5
0.15
PC6
0.55
PC7
0.17
PC8
0.30
PC9
0.27
PC10
0.19
Table 1: Correlation of the observed and the predicted PCs of observed EOFs of
MSLP in 1981-2006. r greater than 0.4 is statistically significant at the 95%
confidence level (bold faced)