WORLD METEOROLOGICAL ORGANIZATION
CAS/WWRP/SSC7/DOCXX.X
COMMISSION FOR ATMOSPHERIC SCIENCES
(CAS)
(9 November 2015)
_______
8th Scientific Steering Committee of the World
Weather Research Programme (WWRP)
Agenda item : X.X
GENEVA, SWITZERLAND
24-27 NOVEMBER 2015
ENGLISH ONLY
REPORT TO WWRP/SSC ON THE ACTIVITIES OF JWGFVR
Submitted by JWGFVR
1. Objectives
Activities (in brief) for each of the WG’s objectives are listed below.
1.1 To plan and implement the verification component of WWRP
WG members have contributed to the verification plans in HiW, PPP and S2S. Input has also been
provided to various RDPs, FDPs, e.g. AvRDP, TLFDP.
1.2 To serve as a focal point for the development and dissemination of new verification methods
for WWRP and EPAC, as required
The WG has endorsed a follow-on spatial methods inter-comparison. See highlights section below.
1.3 To facilitate and encourage training and dissemination of information on verification
methodologies
WG members have provided a 3-day verification tutorial. See highlights section below.
1.4 To take into account the needs of users so as to ensure the relevance of the practice of
forecast verification
A new challenge has been launched to find the best new user-relevant verification metric. See
highlights below.
1.5 To facilitate the development and application of improved diagnostic verification methods to
assess and enable improvement of the quality of weather forecasts, including forecasts from
numerical weather and climate models
See 1.2. Links to the WCRP Grand challenge on extremes. See below.
1.6 To encourage the sharing of observational data for verification purposes
The co-chair is actively engaging with CBS as well as through colleagues at the Met Office
involved in WIGOS to improve the quality control of synoptic observations on the GTS, reduce the
number of silent stations, and improve the metadata for observing sites, especially latitudelongitude precision. See more below.
1.7 To encourage greater awareness in the research community of the importance of verification
as a vital part of numerical and field experiments rather than an "afterthought"
Sessions at conferences such as EMS/ECAM have raised the profile of verification to new levels to
promote best practices and highlight that verification is an integral part of any modelling or
forecasting study. The biannual verification session at ECAM should now be considered as one of
our flagship WG activities. In 2014 the first verification session was also held in the Applied Climate
(EMS/ECAC) conference to promote cross over to the seasonal and near-term climate
communities.
1.8 To promote collaboration among scientists conducting research on various aspects of forecast
verification as well as with model developers and forecast providers
The MesoVICT project serves as a good example of promoting collaboration. See below.
1.9 To collaborate on forecast verification with the Working Group on Numerical Experimentation
(WGNE) and WCRP and in coordination with CBS.
See 1.6 above.
2. Key highlights







The European Meteorological Society (EMS)/ European Conference of Applied Meteorology
(ECAM) conference in Sofia during September 2015 was a great success. The high standard
of contributions was gratifying. With 19 presentations and 12 posters this has become one of
the largest sessions at the conference and a shop window for the WG whose two members (P.
Nurmi and M. Goeber) have been conveners of these sessions since 2003. The session was
very well attended with standing room only most of the time. A special section was devoted to
the results of FROST 2014 FDP/RDP, with 6 excellent contributions on the evaluation of the
Sochi winter Olympics.
The WG held its annual meeting alongside the verification sessions at the EMS/ECAM in Sofia.
JWGFVR launched the “Challenge to Develop and Demonstrate the Best New User-Oriented
Forecast Verification Metric” at EMS/ECAM in Sofia during September 2015. User-oriented
verification has been identified as important to national meteorological and hydrological
services, and is a key component of the WMO’s THORPEX legacy projects - Polar Prediction
(PPP), Sub-seasonal to Seasonal Prediction (S2S), and High Impact Weather (HIWeather), as
well as the Aviation Research Demonstration Project (AvRDP). The challenge aims at
encouraging development of user-oriented verification approaches, and will consider all
applications of meteorological and hydrological forecasts; the metrics can be quantitative
scores or diagnostics (e.g., diagrams). Criteria for judging the best new metric will include
originality, user relevance, intuitiveness, and simplicity, ease of computing, robustness, and
resistance to hedging. Metrics with a clear statistical foundation that can be applied to a
broader set of problems are especially encouraged. The winning entry will be awarded with an
invited keynote talk at the 7th International Verification Methods Workshop in 2017. The closing
date for entries is October 2016. This project is undertaken in collaboration with the HIW, S2S,
and PPP programs.
Four members of the WG presented a 3-day “roving” tutorial at BMKG in Jakarta Indonesia 1719 November 2015, with around 20 students in person, and the lectures also broadcast via
webinars. The tutorial consisted of lectures and labs using precipitation forecasts for the region.
A special issue of the Indian meteorology journal Mausam containing papers from the 6th
International Verification Methods Workshop (New Delhi, 2014) has been published, and is
accessible online. Members of the WG acted as guest editors, contributors, and reviewers.
A chapter on forecast verification (“Chapter 21. Numerical Prediction of the earth system:
Crosscutting research on verification techniques”) was prepared by working group members
and included in the WWRP book, Seamless Prediction of the Earth System from Minutes to
Months.
The “Mesoscale Verification Inter-Comparison over Complex Terrain” (MesoVICT) project
continued with a separate session at the EMS/ECAM conference in Sofia during September
2015. Introductory presentations were also made at EGU 2015 and at the International
Conference on Alpine Meteorology (ICAM) to raise awareness. The session had 8 speakers



and a number of posters, and was well attended. A side meeting was also held to discuss
specific project issues. A key development is the instigation of another project within the
COSMO consortium known as INSPECT with clear deliverables for the consortium. Several
centres have or are in the process of contributing reruns of the core case, with the prospect of
at least two reruns at around 250 -- 300 m resolution for the domain covering the Alps. This will
be a very exciting additional dataset. At present the project has a strong capacity development
component but the project committee is working hard to ensure that the science is also being
advanced. This project will continue to be one of the group’s flagship activities for the next few
years, and will provide opportunities to work with the Mesoscale and Nowcasting working group,
especially through the availability of the sub-km scale reruns.
Marion Mittermaier was invited to present at the “Understanding, modelling and predicting
weather and climate extremes” workshop, convened to discuss and make progress on one of
the WCRP Grand Challenges. The presentation covered the recent application of featurebased methods to global model features such as jets, lows and highs (Mittermaier et al. 2015,
MWR, in press) and how this has recently been expanded by Barb Brown and colleagues at
NCAR to evaluate drought in current climate projections. Interaction with the WCRP grand
challenges was a particular request of CAS.
Laurie Wilson (co-chair) continues to be involved in SWFDPs, for verification and postprocessing. The South Africa project is now being managed by local experts, though some
training continues. The aim is for the local offices to do the verification themselves. For E Africa
the training methodology has changed to case studies as there is insufficient data for larger
evaluations. The new SE Asia project is starting; a meeting in Bangkok in September was
attended by Laurie Wilson. The training workshops have been mostly attended by forecasters
with the typical training being one-half or one-third of a day.
The working group's web page on "Forecast Verification: Issues, Methods and FAQ",
http://www.cawcr.gov.au/projects/verification/, continues to be a valuable resource for basic
verification methodology and promotion of new techniques, and provides an extensive
reference list.] The verification discussion mailing list also continues to be a community
resource used for discussion of specific verification questions.
3. Links to working groups, projects and/or international initiatives





Pertti Nurmi has been the WG representative for the PPP steering group during 2012-15. He
together with Barbara Casati were involved in the writing of the verification section for the
Implementation Plan for YOPPP (Year of Polar Prediction). Barbara Casati will be the
representative on the steering group from 2016 on.
Caio Coelho has been the main representative for the group in S2S. There was a workshop in
Jeju in June with around 18 talks and many posters. The S2S verification science plan has
been updated. There are suggestions that the WG should forge stronger links with the WGNE
metrics panel.
Former co-chair Beth Ebert is now leading the evaluation task team for HiWeather. A number
of WG members are members of the TT: Marion Mittermaier, Thomas Haiden, Manfred
Dorninger. Pertti Nurmi and Martin Goeber will remain on the TT.
Barb Brown attended the IWTC workshop organised by the Tropical WG. A paper on
verification methods for tropical cyclones was prepared for this conference and has been
submitted to the journal Tropical Cyclone Research and Review. A recommendation was
made that WMO identify a standard set of verification methods for TCs. Development of this
set of metrics will involve collaboration with the TC committee and others.
Barb Brown visited the Korean Institute of Advanced Prediction Systems and the Shanghai
Typhoon Institute to provide lectures on verification methods and to collaborate with scientists
on verification activities. At STI, plans were also discussed for future activities of the Tropical
Cyclone Landfall Forecast Demonstration Project (TLFDP).Caio Coelho provided lectures on
understanding and communicating uncertainties in seasonal forecasts and on seasonal
forecast verification methods during the Seventh International Training Workshop Climate
Variability and Predictions (7ITWCVP) Jakarta, Indonesia, 14 – 20 May, 2015. This workshop
was sponsored by BMKG, USAID, WMO and NOAA, with attendance of participants from
Southeast Asia Met Services.





Marion Mittermaier presented an overview of verification to the AvRDP steering group at its
meeting on 24 June 2015 in Shanghai.
Whilst the mandatory exchange of surface weather verification scores for global model
forecasts is continuing under the banner of the CBS, the WG continues to explore the scientific
issues, and it has recently been found that the lack of lat-lon precision in positional information
of synoptic weather stations exchanged on the GTS is now hindering our ability to uniquely
extract the nearest global model grid point to an observing site. Many global models are now
sub- 20 km, with metadata having no more than 2 decimal places. This has a tangible impact
on verification scores especially for spatially discontinuous variables such as precipitation but
potentially also temperature in regions of complex terrain. This improvement of observations
metadata continues to be an important activity and should also be promoted by WWRP, not
just CBS and WIGOS.
Building on the successes of Sochi FROST 2014 FDP/RDP with Pertti Nurmi as the WG
representative in verification issues, there was the initial meeting to discuss the next
PyeongChang 2018 Olympics RDP/FDP. Pertti Nurmi has been invited as foreign chair of
verification working group of this project. One of the focus areas is improving the verification of
nowcasts and very short-range forecasts.
Yuejian Zhu is actively involving RDP/SCMREX project as ISSC member, to provide a
guidance of heavy precipitation verification, scientific design of the experiments. A project is
very success, and preparing BAMS review that is accepted. TMRWG has recommended to
extend project to 2019.
Links with Lake Victoria, UPDRAFT, La Plata at this stage are not clear or not active.
4. The year ahead
Key activities include:
 involvement in THORPEX legacy projects, and associated meetings,
 involvement in various RDPs and FDPs, and SWFDP training workshops
 scientific organising committee for WSN16 in Hong Kong in July 2016,
 promoting the verification metric challenge,
 EMS/European Conference on Applied Climatology verification session to further encourage
cross over between the weather and climate communities,
 the MesoVICT workshop in Bologna in September 2016 and
 planning for the 7th international verification methods workshop and tutorial in Berlin in 2017.
5. Other topics
5.1 Membership
 Martin Goeber has stepped down from the group. Dr Chiara Marsigli from ARPA-SIMC has
been approached as a new member and she has accepted. Her membership is currently being
processed and subject to WMO approval.
 Simon Mason’s term has ended and a new member will join the working group. Someone has
been approached but at the time of writing has not yet accepted.
 Pertti Nurmi is stepping down. The WG still needs to agree on whom to approach as a new
member.
 The co-chairs want to express their thanks to the outgoing members for their contributions to
the activities of the WG.
5.2 Fund raising
Members of the WG are involved in a number of Horizon 2020 proposals submitted under the 2015
DRS-1 theme call. The outcome will be announced in January 2016 with successful projects
starting in autumn 2016. These projects could contribute significantly to the development of tools
for the processing of “citizen observations” and their use in the evaluation of forecasts of hazards
and/or impacts.
5.3 Meetings
Date
3-5 Apr 2016
27-22 Apr 2016
Meeting title
Sea ice verification workshop
EGU Session on Innovative Evaluation and
Prediction for Large Earth Science Datasets
25-29 Apr 2016 HiWeather workshop
23-25 May 2016 PPP Steering Group Meeting #7
6-8 Jul 2016
Towards a strategy for the public
communication of the risk of high-impact
weather using probabilistic forecasts – miniworkshop as part of the RMetS/NCAS
conference
6-8 Jul 2016
User-focused evaluation methods for weather
and water related impacts – mini-workshop as
part of the RMetS/NCAS conference
25-29 Jul 2016
WMO WWRP 4th International Symposium on
Nowcasting and Very-short-range Forecast
2016 (WSN16) + AvRDP steering committee
meeting
11-16 Sep 2016 EMS/ECAC
20-21 Sep 2016 JWGFVR meeting
21-23 Sep 2016 2nd MesoVICT workshop
2017
7th verification methods workshop and tutorial
Where
Frascati, Italy
Vienna, Austria
Exeter, UK
Beijing, China
Manchester, UK
Manchester, UK
Hong Kong
Trieste, Italy
Bologna, Italy
Bologna, Italy
Berlin, Germany