Multi-model Ensembles, Poor Man's Ensembles and Super-Ensembles - a brief review
Prepared for the WMO Workshop on Applications of Ensemble Forecasting,
16-20 October 2000, Beijing, China
Ken Mylne
The Met Office, Bracknell, UK
Introduction
This is intended as a brief overview of some of the different types of ensemble forecasting
systems currently being developed or tested in the World. It is not intended to be a
comprehensive literature review, but to give some references for further study if required.
Current Operational Ensemble Prediction Systems (EPS) for medium-range forecasting
mostly use a single model with a set of perturbed initial conditions to take account of the
analysis uncertainty eg ECMWF - Molteni et al (1996), Buizza et al (1998); NCEP - Toth and
Kalnay (1993). This approach essentially assumes that the model is perfect, and that forecast
uncertainty is due only to initial condition errors. Some attempt to allow for model errors is
now made in the ECMWF ensemble, by the use of stochastic perturbations of the tendency
due to model physics ("stochastic physics"), Buizza et al (1999). A more comprehensive
approach is followed by Houtekamer et al (1996) who use a number of different versions of
their model with different physical parametrization schemes as well as initial condition
perturbations. All these systems find that the spread of the ensemble is too small, indicating
that the simulation of the error sources is incomplete. Some of the new approaches to
ensemble forecasting, such as multi-model ensembles, are attempting to address some of this
additional uncertainty.
Multi-model and Multi-analysis Ensembles
One approach to taking account of model uncertainties is to include other model(s) in the
ensemble system. Ideally the models used should be as independent of each other as possible
so that their errors and weaknesses are as independent as possible. Evans et al (2000), Mylne
et al (1999) and Mylne et al (2000) describe experiments combining the ECMWF ensemble
with an ensemble using the Met Office's Unified Model (UM). The UM is completely
independent of the ECMWF model, having different dynamics and physics, but has
competitive average forecast skill. The UM ensemble used the Met Office's analysis to which
ECMWF initial perturbations were added, so the combined ensemble sampled both different
initial condition uncertainties (through use of an independent analysis system) and different
model uncertainties. Results showed considerable improvements in skill for the multi-model
multi-analysis (or multi-system) ensembles compared to the ECMWF ensemble, with no
increase in ensemble size. This is believed to be due to the fact that in synoptic situations or
geographical locations where one model performs poorly, another independent skilful model
may often perform relatively better.
A full multi-model multi-analysis ensemble system as used by the above authors requires
extra effort in maintenance for operational use compared to a single model system, and
Richardson (2000) has experimented with multi-analysis ensembles where only the ECMWF
model is used, but analyses from a number of NWP centres are used to improve the estimate
of initial condition uncertainty from that given by the ECMWF singular vector perturbations.
He also experimented with using just the consensus analysis obtained by averaging the
analyses of several NWP centres. Results showed that 50-80% of the benefit of the full multimodel multi-analysis ensemble could be achieved by the multi-analysis ensemble, rather less
using just the consensus analysis. This is consistent with the results of Evans et al (2000) who
found that it was necessary to include both multi-model and multi-analyses to get the full
benefit.
The use of multi-model multi-analysis ensembles is now becoming standard practice in
seasonal-range forecasting where the benefits were demonstrated in the PROVOST project
(eg Graham et al 2000) and will be used extensively in the forthcoming European DEMETER
project. Multi-model multi-analysis techniques are also being used increasingly for shortrange ensemble forecasting where it is often important to sample local details of precipitation,
for example. Stensrud et al (1999) found that the inclusion of two different models in an
ensemble for short-range forecasting “assists in increasing the ensemble spread significantly”.
Mullen et al (1999) found that the combined effect of uncertainties in model physics and in
the initial state provided a suitable means for creating a short-range ensemble for QPF
(quantitative precipitation forecasting), with the inclusion of different forms of cumulus cloud
parametrization schemes within a mesoscale model ensemble.
“Poor Man's” Ensembles
For many years operational forecasters, particularly medium-range forecasters in major
centres such as the Met Office in the UK, have had access to some forecast products from
global NWP centres other than their own. They routinely compare forecasts from different
centres to assess confidence in forecasts from their own models, and to determine possible
alternative solutions. This set of available forecasts is often termed the “Poor Man's”
ensemble because it is relatively cheap compared to the cost of developing and running a full
EPS such as the ECMWF and NCEP ones. Forecasters in the Met Office often observe
subjectively that model forecasts from other NWP centres are synoptically more different
from the ECMWF model than are the members of the ECMWF EPS. They find that the EPS
members follow too closely to the solution of the ECMWF high-resolution deterministic
model (or of the EPS control) - in other words that the EPS does not spread sufficiently to
incorporate the full range of uncertainty. Thus even with the advent of the EPS, the
forecasters still attach a lot of weight to the subjective assessment of the Poor Man's
ensemble.
Some recent research (eg Ziehmann, 2000) has shown more objectively that the Poor Man's
approach can, as well as being cheap, also be a very efficient method of sampling the forecast
uncertainty, particularly in the short-range (up to 5 days). Given the success of multi-model
multi-analysis ensembles described above, particularly for short-range forecasting, this is
perhaps not surprising since the Poor Man's ensemble inherently uses multiple models and
multiple analysis systems. In the longer medium-range (beyond 5 days) it is not surprising if
the Poor Man's system becomes less competitive, since the perturbations generated by
ECMWF and NCEP are purposely designed to maximise ensemble growth in the mediumrange.
One of the major objectives of ensemble forecasting, particularly in the short- and earlymedium-range, is the probabilistic prediction of severe weather events. Severe weather is
often extremely difficult to predict deterministically as its development can be very sensitive
to small errors even at very short range, but successful prediction is of crucial importance to
the success and credibility of any forecasting service. A good example of the difficulty of
predicting severe weather was the storms which struck France and Germany in late December
1999, when most NWP models, including high-resolution regional and meso-scale models,
failed to capture the severity of the storms even at 24 hours or less. In the discussions
following these storms it was generally agreed that reliable deterministic prediction of such
severe developments was unlikely to be possible in the foreseeable future, and that the use of
the Poor Man's ensemble provided the best approach to capturing such events.
One limitation of Ziehmann's (2000) work was that she used only four models in the Poor
Man's ensemble, compared to 51 members in the EPS with which she was comparing it. This
made comparison difficult and she had to make allowance for the numbers of ensemble
members to draw conclusions. A four-member ensemble is unlikely to be competitive in
operational use, although on occasions it may still provide useful synoptic information not
included in even a much larger perturbed EPS. In order to address this short-coming in more
detail, the Met Office is currently leading a new project to assess the benefits of a Poor Man's
ensemble using a larger set of models, in collaboration with NCEP and with other global
NWP centres who are being asked to supply forecast data for the project. The project aims to
measure the skill of the larger Poor Man's ensemble compared to other ensemble methods,
particularly for short-range forecasting. Particular attention will be paid to the ability of the
system to capture the probability of severe weather events. (Further information about this
project can be obtained from the author: Mr Ken Mylne, Rm 329, The Met Office, London
Road, Bracknell, Berks RG12 7BU, UK; tel +44 1344 856070; email krmylne@meto.gov.uk).
Multi-Model Super-Ensembles
Multi-model super-ensembles have been developed by Krishnamurti et al (1999a and 1999b)
at Florida State University. This is essentially a development of the Poor Man’s ensemble, in
that it takes output from multiple operational deterministic forecasts. However instead of
treating the combination of models as an ensemble, they perform a statistical combination of
the forecasts, based on past performance, to obtain the best deterministic forecast. Unlike
other ensemble techniques, the super-ensemble provides only a deterministic forecast, not a
probability forecast. Since it is a statistical technique, the super-ensemble is first trained using
a dataset of past forecasts and observations (analyses). The training consists of a linear
multiple regression of the different models against the observations to determine the statistical
weights for each model. This process is carried out separately for each variable at each gridpoint. In the forecast phase these regressions and weights are used to determine the best
estimate of the forecast at each gridpoint from the output of all the models. For a single model
this process is equivalent to making a simple bias correction at each grid-point, but for
multiple models it applies bias corrections to all models and applies different weights to each
model according to its past skill. Results show that the super-ensemble provides the best
deterministic forecast, better than all individual models or the simple ensemble mean of the
models. It is also better than the ensemble mean of individually bias-corrected models,
because of the statistical weights applied to the models, which put greatest weight on the most
skilful model in any location.
References
Buizza,R., Petroliagis,T., Palmer,T.N, Barkmeijer,J. Hamrud,M., Hollingsworth,A.,
Simmons,A., and Wedi,N. , 1998 Impact of model resolution and ensemble size on the
performance of an ensemble prediction system Quart. J. Royal Meteorol. Soc. 124, 19351960.
Buizza,R., Miller,M. and Palmer,T.N. 1999 Stochastic representation of model uncertainties
in the ECMWF EPS Quart. J. Roy. Meteorol. Soc, 125, No. 560, 2887-2908.
Evans,R.E., Harrison,M.S.J. and Graham,R.J., 2000: Joint Medium Range Ensembles from
the UKMO and ECMWF Systems. Mon. Wea. Rev.,128, No 9, .3104-3127.
Graham,R.J., Evans,A.D.L., Mylne,K.R., Harrison,M.S.J. and Robertson,K.B. 2000:
An assessment of seasonal predictability using Atmospheric GCMs Quart. J. Royal
Meteorol. Soc 126, pp2211-2240.
Houtekamer,P.L., Lefaivre,L., Derome,J., Ritchie,H., and Mitchell,H.L., 1996: A system
simulation approach to ensemble prediction. Mon. Wea. Rev. 124, 1225-1242.
Krishnamurti,T.N., Kishtawal,C.M., Zhang,Z, LaRow,T., Bachiochi,D., Williford,E.,
Gadgil,S. and Surendran,S.. (1999a): Multi-model superensemble forecasts for
weather and seasonal climate FSU (Florida State Univ.) Report 99-8, July 1999.
Krishnamurti,T.N., Kishtawal,C.M., LaRow,T., Bachiochi,D., Zhang,Z, Williford,E.,
Gadgil,S. and Surendran,S.. (1999b): Improved weather and seasonal climate
forecasts from multimodel superensemble Science 285 No 5433, pp 1548-1550, 3 Sep
1999.
Molteni,F., Buizza,R., Palmer,T.N. and Petroliagis,T., 1996: The ECMWF Ensemble
Prediction System: Methodology and Validation. Quart.J.Roy.Meteor.Soc. 122, 73-119.
Mullen,S.L., Du,J. and Sanders,F., 1999: The dependence of ensemble dispersion on analysisforecast systems: implications to short-range ensemble forecasting of precipitation. Mon.
Wea. Rev 127 No 7, 1674-1686.
Mylne,K.R., Clark,R.T. and Evans,R.E. 1999: Quasi-operational multi-model multi-analysis
ensembles on medium-range timescales AMS 13th Conf on Numerical Weather Prediction,
13-17 September 1999, Denver, Colorado, pp204-209. (Also provided as lecture notes to the
WMO Workshop on the use of ensemble prediction, 16-20 October 2000, Beijing, China.)
Mylne,K.R., Evans,R.E. and Clark,R.T. 2000: Multi-model multi-analysis ensembles in
quasi-operational medium-range forecasting, To be submitted to Quart.J.Roy.Meteor.Soc.
Richardson,D, 2000: Ensembles using multiple model and analyses, submitted to Quart. J.
Royal Meteorol. Soc, January 2000. (Also in: Proceedings of a seminar held at ECMWF on
diagnosis of models and data assimilation systems, 6-10 September, 1999, Shinfield Park,
Reading.)
Stensrud,D.J., Brooks,H.E., Du,J., Tracton,M.S. and Rogers,E. 1999: Using
Ensembles for Short-Range Forecasting Mon. Wea. Rev, 127, 433- 446.
Toth,Z., and Kalnay,E., 1993: Ensemble Forecasting at the NMC: The generation of
perturbations. Bull.Amer.Meteor.Soc., 74, 2317-2330.
Ziehmann, C. (2000): Comparison of a single model EPS with a multi-model
ensemble consisting of a few operational models Tellus 52A, 280-299.