Strategies for the verification of ensemble forecasts
(Laurie Wilson, Meteorological Service of Canada)
Abstract:
The subject of ensemble forecast verification naturally divides itself into two parts:
Verification of the ensemble distribution of deterministic forecasts and verification of
probability forecasts derived from the ensemble. The verification of the ensemble distribution
presents unique problems because it inevitably involves comparing the full ensemble of
forecast values against a single observation at the verifying time. A strategy to accomplish
this kind of evaluation of the ensemble distribution, proposed by Wilson et al, 1999, will be
described, along with other methods that have been used to evaluate aspects of the
ensemble distribution.
By comparison, evaluation of probability forecasts extracted from the ensemble is simpler;
any of the existing measures applied to probability forecasts can be used. Following the
work of Murphy, the attributes of probability forecasts will be described, along with a set of
verification measures, which are used to evaluate these attributes. Examples of application
of each technique to ensemble forecasts will be shown, and the interpretation of the
verification output will be discussed. The verification methods to be described include the
Brier and the Rank probability score, skill scores based on these, reliability tables, and the
relative operating characteristic (ROC), which has been widely used in the past few years to
evaluate ensemble forecasts.