WAFS Workshop on the Use and Visualization of Gridded
SIGWX Forecasts
14-15 September 2009
Overview
Introduction
 Compatibility of forecasts of the two WAFCs
 Quality issues: seasonal aspects
 Quality issues: CB diurnal variation
 Quality issues: CB associated with TC
 Verification issues
 Calibration issues
 User survey
 Conclusions

Introduction
WAFSOPSG/4: WAFC Provider States, in
coordination with WAFSOPSG members from
IATA, IFALPA and WMO, to undertake
systematic comparisons of trial gridded
forecasts of icing, turbulence and CB clouds of
the two models, highlighting characteristics of
areas with different values.
 The results should lead to the alignment of
algorithms used by the two WAFCs.

Introduction
To facilitate comparison of the trial gridded
forecasts, HK, China set up website (http://wafsgrid-fc.weather.gov.hk) to display gridded values
(without smoothing/interpolation).
 The observations were reported in CNS/MET
SG/13 meeting in July 2009 and were agreed to
be referred to the WAFSOPSG/5 meeting.

(Earlier observations reported in CNS/MET SG/12 meeting in 2008)
Compatibility of forecasts of the two WAFCs
UK
US
T+24 h Max CAT Potential at FL340 based on 12 UTC 2 July 2009 )

Difference map shows absolute difference in the two
forecasts (little agreement between the two forecasts if
difference map appears to be the union of them).
Diff
Compatibility of forecasts of the two WAFCs

CB horizontal extent
- CB coverage of UK more extensive
- Forecast values of US reaching 50-75% (OCNL
following Annex 3 spec) over tropics, larger than those
given in the UK forecast

ICAO height at CB top
- UK still gives more extensive CB areas
- UK over-forecast CB for vast areas in the Pacific
Ocean

Mean in-cloud turbulence potential (FL180)
- UK and US forecast values with similar range
- UK forecast shows small values almost everywhere
Compatibility of forecasts of the two WAFCs

Max icing potential (FL180)
- Significant differences in the spatial coverage over the
tropical and higher latitude regions
- WAFS Science Coordination Meeting 2009: “percentage
overlap of all (icing) forecasts between UK and US ~ 20%”

Max CAT potential (FL340)
- UK gives much less extensive CAT areas (except over
certain spots)
Compatibility of forecasts of the two WAFCs

Difference maps in most cases appear to be the
union of the forecasts from the two centres,
indicating little agreement
Latest CB forecasts of UK
CB top height valid
12 UTC 14 Sep 2009
CB horiz extent valid
12 UTC 14 Sep 2009
MTSAT-1R infrared
satellite imagery, with
deep convection
indicated in red,
at 08 UTC on
14 Sep 2009
Quality issues: seasonal aspects
mean in-cloud turbulence at FL 180
T+24 h Gridded Forecasts
UK (based on 00 UTC 6 Jan 2009)
T+24 h Gridded Forecasts
(based on 00 UTC 2 Jul 2009)
US
For both
winter &
summer: large
values are
forecast by
UK for high
latitudes in
both
hemispheres.
They are not
seen in the
US forecasts.
Quality issues: seasonal aspects
max icing potential at FL 180
UK
T+24 h Gridded Forecasts
(based on 00 UTC 6 Jan 2009)
T+24 h Gridded Forecasts
(based on 00 UTC 2 Jul 2009)
US
At high latitudes
in winter: large
values
(> 85%) forecast
by UK, but
much smaller
values (mostly
< 30%) forecast
by US
Scientific paper
on icing
climatology:
non-occurrence
of icing at
FL180 in the
wintertime north
of 25N
Quality issues: seasonal aspects
max CAT potential at FL340 – 3 days in Jan 2009
- large values are
forecast by UK for a
number of isolated
locations
- stationary “hot
spots” associated
with mountainous
regions
Quality issues: seasonal aspects
Some common issues can be identified
when data for different seasons are
monitored (e.g. in-cloud turb, icing)
 Some issues (e.g. terrain-related) may
show up at different locations in different
seasons
 The above observations are not exhaustive
 It is imperative for the WAFCs to:

 conduct verification for all seasons
 examine model climatological biases
Quality issues: seasonal aspects
Climatology of differences between two WAFCs (May – Aug 2009)
CB
horiz extent
Max icing
CB
top height
Max CAT
?
Mean
in-cloud turb
?
Quality issues: CB diurnal variation
ICAO height at CB top (UK)
T+24h gridded forecasts
T+24h gridded forecasts
Based on 00 UTC 17 August 2008 Based on 06 UTC 17 August 2008
Solar
heating
Quality issues: CB diurnal variation
ICAO height at CB top (US)
T+24h gridded forecasts
Based on 00 UTC 17 August 2008
T+24h gridded forecasts
Based on 06 UTC 17 August 2008
Solar
heating
Again the UK forecasts are giving much more extensive CB
areas compared to the US forecasts
Quality issues: CB diurnal variation
• Actual weather as shown by
the satellite images
• Weather was generally fine
over southern China
throughout the period
MTSAT-1R infrared
satellite imagery, with
deep convection
indicated in red,
at 23:30 UTC on
17 August 2008
MTSAT-1R infrared
satellite imagery, with
deep convection
indicated in red,
at 05:30 UTC on
18 August 2008
• Both UK and US overforecast CB occurrence over
this region at 06 UTC (14 H
local time), probably due to
difficulties of the models in
handling CB development
induced by daytime solar
heating
Quality issues: CB diurnal variation

2009 WAFS Science Coordination Meeting
acknowledged that further work would
need to be undertaken by the WAFCs for
improving the CB gridded forecasts,
including to “look at diurnal variations of
forecasts
(manual
and
automatic),
particularly in tropics”
Quality issues: CB associated with TC
UK
US
Based on 06 UTC 19 June 2009
Satellite imagery clearly
shows deep convection
Need TC (and other SIGWX) information associated with tropical
cyclone Linfa
in future visualization models
Rather difficult to identify the area associated with
TC without referring to the satellite imagery
Verification issues
APANPIRG called for systematic
SIGWX verification in 2006
 WAFSOPSG/4: Evaluation of the
SIGWX forecasts proposed by
APANPIRG would be sufficiently
addressed in the systematic comparison
of WAFS SIGWX forecasts with the
gridded forecasts

2009 Science Coordination Meeting
Forecast
verified
Icing
CAT
Period
Nov 2008 –
Jan 2009
Nov 2008 –
Jan 2009
Parameters
verified
Maximum &
mean icing
potential at
FL100 &
FL140
CAT
potential@
In-cloud
Turbulence
CB
Data used as “ground
truth” for verification
CloudSat Icing
Potential (CLIP)
product (derived from
CloudSat cloud
classification data and
numerical model
temperature forecast)
Global Aircraft Data Set
(GADS)
based on British
Airways fleet of Boeing
747-400 aircraft
Verification
metrics
considered
Criteria for counting
hit and false alarm
POD+ & FAR^
Icing potential
thresholds –
0.1, 0.3 & 0.5
CLIP –
trace icing
POD & FAR
CAT potential thresholds
– numerous
GADS –
DEVG# >= 2 m/s
POD & FAR
Rainfall rate
thresholds – 16 levels
0 to 0.0046 kg/m2/s
Sferics data - ??
Not verified
Nov 2008 &
Jan 2009
UK convective
rainfall rate*
US - ??
UK Sferics data
+ POD = probability of detection, or hit rate = hit / (hit + miss).
^ FAR = false alarm rate = false alarm /(false alarm + correct rejection) . For rare events, correct rejection is very large!
@ No indication whether max or mean turb potential was considered, and whether or not forecasts for all flight levels were verified.
# DEVG = Derived equivalent vertical gust
* Not a product of the CB gridded forecasts
Verification issues


In-cloud turbulence – systematic verification yet to be
done.
Icing – verification based on another icing product
(CLIP) as ground truth, with “trace icing” criterion
Kay et al (2009) on CLIP: “It is important to note that our algorithm
only states where icing conditions have the potential to exist rather
than stating exactly where within the clouds exactly the icing does
or does not exist. ... Therefore our algorithm should be considered
as a biased representation of icing conditions in the atmosphere.
In the absence of a true measure of icing conditions the amount of
bias cannot be known”.
The choice of the ground truth for the verification, as
well as the verification method itself, must be agreed by
WAFSOPSG.
“Trace icing”  calibration issue.
Verification issues


CAT – verification based on DEVG >= 2 m/s from
aircraft data, with numerous thresholds, rather
than ICAO adopted turbulence metrics, i.e.
MOD/SEV in PIREP or EDR thresholds.
Numerous thresholds  calibration issue.
CB – verification based on sferics data, with
numerous thresholds, rather than observations of
CB (e.g. from satellite). Also lack of information on
how CB forecast parameters are derived, and use
of smoothing algorithm by UK.
Calibration issues
Need to be mindful about users’
perception and expectation based on
their experience with the existing
SIGWX forecasts
 Thresholds for verifying and visualizing
gridded forecasts need to be determined
in consultation with users

User survey

HKO conducted a user survey on the WAFS trial
gridded forecasts in Jul to Aug 2009, 15 returns
from IFALPA, pilots, airline management and
dispatchers:
 80% consider gridded products are NOT suitable for
operational use in flight planning and flight
documentation;
 93% considered gridded products CANNOT replace
the SWH and SWM charts.

Specific user comments are summarized below.
User comments
Comments
User
Why is there so much difference from the London and
Washington charts if they are from the same time period?
IFALPA
Rep
Have lots of deficiencies: too general; discontinuities;
disagreement between centres; loss of useful information;
too long forecast time periods etc.
“Potential” appears to have no scientific definition or
validity and of little or no assistance to pilot, whereas IFALPA
PROB gives the indication of the likelihood of the Pilot
turbulence/icing/CAT being encountered.
They must reflect the situation and not be so far forecast
that the time period does not apply or that the forecast is
so general as to be meaningless.
User comments
Comments
User
Not suitable for operational use without considerable
improvement. The improvement suggested by the HKO
would help a lot, particularly if the horizontal "real time"
strips can be developed. These would be very useful and
very practical for the pilots.
IFALPA
Pilot
The gridded products could not replace the high- (cont’d)
level/medium level at this stage as the information is too
general and some of the previous information to be omitted.
User comments
Comments
User
The scale is too large and the areas of hazard are too
wide. If pilot was presented with so much weather, he/she
would be unable to fly around it.
Pilots
Different aircraft types have different rides through
turbulence depending on wing loading.
Icing is only a problem at cruise levels in large tropical
convective clouds.
The technological advance and gridded forecasts could Member of
not take the place of the forecaster’s expertise of the MET WAFS
situation, particularly for SIGWX like CB.
OPSG
User comments
User
Comments
Large difference
Washington.
between
WAFC
London
and
Two WAFCs should align to produce an integrated
WAFS.
Very large scale, too big an area, insufficient details for Airlines
short-haul flights.
It is a rough forecast; user cannot obtain detailed
information just by a glance; not precise and detailed in
comparison with JMA data.
We do prefer using High-level SIGWX and Medium-level
SIGWX charts (the current one). It's simple, easy to
understand and contains lots of information.
Conclusions

Suitability of WAFS gridded forecasts of icing,
turbulence and CB for operational use :
• Issues with compatibility, quality, verification, and
calibration need to be satisfactorily addressed before
they could be considered suitable for operational use.
• Systematic verification: WAFS output performance
indicators (including methodology, thresholds, choice of
ground truth) for verifying the gridded forecasts should
be systematically developed and agreed.
• Calibration: the thresholds for verifying and visualizing
the gridded forecast should be calibrated for compatibility
with users’ expectation/perception. Pilots have problems
with the use of “potential” for icing and turbulence.
Conclusions

Suitability of WAFS gridded forecasts of
icing, turbulence and CB to replace the
current SIGWX forecasts (SWH and SWM):
 Not at this stage (not for operational use)
 Visualization issues (including thresholds to be
adopted, forecast irregularities e.g. CB top/base
heights) need to be resolved
 High “at-a-glance” visualization models should
be agreed by users
 SIGWX features other than icing, turbulence and
CB required by users should be included,
especially if they are safety-related (e.g. TC, VA)
Thank you
Supplementary information
Observations reported in CNS/MET SG/12, 2008
CB horizontal extent
UK
valid 00 UTC 23 Jun (based on 00 UTC 22 June 2008)
US
CB coverage is more extensive with generally larger forecast values in the UK forecast
compared with that of US. Furthermore, small values of CB horizontal extent are given
almost everywhere in the UK forecast
Back
Observations reported in CNS/MET SG/12, 2008
ICAO height at CB top
UK
US
valid 00 UTC 23 Jun (based on 00 UTC 22 June 2008)
UK forecast generally gives more extensive but lower heights of CB top
(generally below 37kft) compared with that of US (above 43 kft)
Back
Observations reported in CNS/MET SG/12, 2008
Mean in-cloud turbulence potential
UK
US
valid 00 UTC 23 Jun at FL180 (based on 00 UTC 22 June 2008)
UK forecast generally gives wider range and higher turbulence potential values
compared with that of US. On the other hand, small values of mean in-cloud
turbulence potential are given almost everywhere in the US forecast
Back
Observations reported in CNS/MET SG/12, 2008
Maximum icing potential
UK
US
valid 00 UTC 23 Jun at FL180 (based on 00 UTC 22 June 2008)
At FL180: UK forecast giving generally higher icing potential over the higher
latitude regions; US forecast giving generally higher icing potential over the tropical
region and the UK forecast giving generally higher icing potential over the higher
latitude regions
Back
Observations reported in CNS/MET SG/12, 2008
Maximum CAT potential
UK
US
valid 00 UTC 23 Jun at FL340 (based on 00 UTC 22 June 2008)
At FL340: UK forecast generally gives much less extensive CAT
areas compared with that of US
Back
Back
Bernstein, B.C. and Christine LeBot, 2009: An Inferred Climatology of Icing Conditions Aloft,
Including Supercooled Large Drops Part II: Europe, Asia and the Globe, Journal of Applied
Meteorology and Climatology, American Meteorological Society
Back
CB horizontal extent at FL340
T+24 h Gridded Forecasts (based on 12 UTC 2 July 2009)
UK
US
Diff
Back
Back
ICAO height at CB top
T+24 h Gridded Forecasts (based on 12 UTC 2 July 2009)
UK
US
Diff
Back
ICAO height at CB top
T+24 h Gridded Forecasts (based on 12 UTC 2 July 2009)
UK
US
MTSAT-1R IR sat pic
With deep convection indicated in red
At 11:30 UTC 3 July 2009
Back
Back
Mean in-cloud turbulence potential at FL 180
T+24 h Gridded Forecasts (based on 12 UTC 2 July 2009)
UK
US
Diff
Back
Back
Maximum icing potential at FL 180
T+24 h Gridded Forecasts (based on 12 UTC 2 July 2009)
UK
US
Diff
Back
Back
Maximum CAT potential at FL 340
T+24 h Gridded Forecasts (based on 12 UTC 2 July 2009)
UK
US
Diff
Back
Back
List of Respondents to WAFS Survey
IFALPA Rep (Captain Miguel Marin, Chairman of ATS
IFALPA Pilot (Captain Brian Greeves, HKALPA)
Committee)
Pilots:
The Guild of Air Pilots and Air Navigators (GAPAN) (Captain Brent
Hawkins, Vice-Chairman, Hong Kong Region)
Metrojet (Captain Peter Garraty)
Airlines:
Cathay Pacific Airways (Neil Phillips, Manager Line Operations)
China Airlines (Lo Wai Lun Kevin, Dispatcher)
Dragonair (Andy Jepps, FTU A320; Jim Ashby, Deputy Manager Line
Operations)
El Al Israel Airlines Limited (c/o Hammond Law, Team Manager)
Fedex (Andy Yeung, Manager - Operations)
Hong Kong Airport Services Limited (Gary Hor, Flight Dispatch Manager;
Jackson Lam, Quality Assurance Manager)
Japan Airlines (Stephen Chan, Manager of Flight Operation)
Jardine Airport Services Limited (Hammond Law, Team Manager)
South African Airways (Ringo Chui, Airport Manager & Vice-chairman of
Back
Airline Operators Committee)