Translating verification experience
from meteorology to space weather
Suzy Bingham
ESWW splinter session, Thurs. 20th Nov.
Current space weather
validation, verification & metrics
Met Office
forecaster
Webpages:
Enlil & REFM.
• Validation: plans to work with CCMC for TEC validation.
• Verification: plan to use experience in terrestrial methods; to verify models &
forecaster warnings.
•
Application metrics/KPIs: have extended terrestrial infrastructure to monitor
space weather systems. Plan to verify for stakeholders.
Verification of forecasts
• Why verify: to understand quality/accuracy, evidence for stakeholders,
forecaster feedback, to further improve, to compare models/methods.
• Space weather forecasters produce guidance: twice daily. These forecasts
include probability forecasts for geomagnetic storms, X-ray flares, high energy
protons & high energy electrons.
Example probability
forecasts
Verification of forecasts
Metrics/performance indicators used in terrestrial weather forecasting verification
for stakeholders (with examples):
1. Severe weather warning accuracy (score: impact level, area covered & validity time)
2. Forecast accuracy (e.g. daily minimum temp accuracy should be +/-2°C)
3. Public value (“How useful are forecasts these days”?)
4. Public reach (“Have you seen any weather warnings in the last few days?”)
5. Service quality (Timeliness scores for model delivery.)
6. Emergency responder value (How satisfied are you with the service?”)
7. Responder reach (Availability of Hazard Manager application to emergency responder
community)
8. National capability (95% of lightning location data messages should be available within
a certain time.)
9. Milestone achievement (Develop a national electronic weather data archive)
Verification of warnings
Introducing the warnings verification system
MISS
Event
threshold
Low Event
threshold
EARLY
HIT
EARLY
LOW
HIT
NONEVENT
Issue
time
© Crown copyright Met Office
HIT
LATE
HIT
LOW HIT
LATE
LOW
HIT
FALSE
ALARM
Warning period
MISS
NONEVENT
End time +
late hit period
time
Gale warning verification
St Judes Storm 27-28 October 2013...
... a hit!
Performance: ROC plot
Performance of Forecaster issued Heavy Rainfall Alerts in 2012:
1
0.9
0.8
0.6
ROC area
HIT RATE
0.7
0.5
0.4
0.3
1
0.9
0.8
0.7
0.6
0.5
NO FLEX
TEMPORAL FLEX
TEMPORAL + INTENSITY FLEX
no skill
0.2
0.1
0
0
© Crown copyright Met Office
0.2
0.4
0.6
FALSE ALARM RATE
0.8
1
Application metrics:
timely, reliable, robust
Timeliness plot for Global
Model.
Server monitoring to
verify system is
robust/reliable.
Data latency
• ENLIL: run every 2 hours
• Run completed / graphics produced around 4
hrs after model analysis time (ie T=0).
• SOHO: EIT 4 channels, LASCO – C2, C3
•
LASCO C2 & C3 – 12 min cad / ~1-2 hrs latency (but data gaps – sometimes up to 5 hrs)
• SDO: AIA 12 channels, HMI-magnetogram
• STEREO A&B – COR1, COR2, EUVI
•
cadence: COR1 – 1hr, COR2 – 15 min, EUVI – 10 min
•
latency – 1-3 hrs (but currently very big data gaps/no data).
WMO Coronograph image requirements, OSCAR webpage
Met Office Business
Performance Measures (BPMs)
The Met Office BPMs are: Forecast accuracy, Growth, Reach, Customer &
Service Delivery, Efficiency & Sustainability Excellence.
Verification underpins the Forecast Accuracy BPM which is set by government.
This BPM is to improve:
1.
Global NWP Index
2.
UK NWP Index
3.
Public Forecasts
4.
Customer Forecasts
Global NWP Index is compiled from:
• Mean sea-level pressure,
• 500 hPa height,
• 850 hPa wind,
• 250 hPa wind.
Plot showing increase in Global NWP Index
which shows an improvement in Global model accuracy.
Summary
• Met Office is currently applying some validation, verification & metrics
to space weather models/service (e.g. verifying reliability of service).
• Met Office is planning to take part in CCMC TEC challenge with initial
longitudinal section case studies over Europe.
• Met Office is planning to adapt terrestrial verification methods to
space weather to allow forecasters to understand warning accuracy
(e.g. flexible verification system).
• Met Office would like to provide simple metrics to stakeholders.
Useful links
• WMO space weather observation requirements, Observing Systems
Capability Analysis & Review Tool: http://www.wmosat.info/oscar/applicationareas/view/25
• Forecast verification, Centre for Australian Weather & Climate Research:
www.cawcr.gov.au/projects/verification/
• http://www.eumetcal.org/resources/ukmeteocal/temp/msgcal/www/english/co
urses/msgcrs/crsindex.htm
• NOAA verification metrics glossary:
http://www.nws.noaa.gov/oh/rfcdev/docs/Glossary_Verification_Metrics.pdf
Questions and answers
Verification of forecasts
Metrics/performance indicators used in terrestrial weather forecasting verification
for stakeholders:
1. Severe weather warning accuracy
2. Forecast accuracy
3. Public value
4. Public reach
5. Service quality
6. Emergency responder value
7. Responder reach
8. National capability
9. Milestone achievement
Scoring for quality of sever weather warning.
0-2 Very Poor
3-4 Poor
Guidance
5-7 Good
Guidance
8-9 Excellent
Guidance
Warning was missed or gave very poor
guidance to customer, perhaps being classed
as a “False Alarm”
Although a warning was issued it gave poor
guidance to the customer
A warning was issued which gave generally
good guidance to the customer
The warning issued gave excellent guidance
to the customer
Performance: reliability
Performance of Forecaster issued Heavy Rainfall Alerts in 2012:
thousands of warnings
1.0
Observed Relative
Relative Frequency
Frequency
Observed
0.9
0.8
0.7
0.6
7
6
5
4
3
2
1
0
no resolution
no skill
NO FLEX
TEMPORAL FLEX
TEMPORAL+INTENSITY FLEX
0.3
0.5
0.5
perfect reliability
0.8
probability
0.4
0.3
0.2
0.1
0.0
0.0
0.1
0.2
0.3
0.4
0.5
Probability
© Crown copyright Met Office
0.6
0.7
0.8
0.9
1.0
Met Office Business
Performance Measures (BPMs)
The Met Office BPMs are: Forecast accuracy, Growth, Reach, Customer &
Service Delivery, Efficiency & Sustainability Excellence.
Verification underpins the Forecast Accuracy BPM which is set by government.
This BPM for FY13/14 was to improve:
1.
Global NWP Index - 101.88 with a stretch of 102.43.
2.
UK NWP Index - 123.05 with a stretch of 123.9.
3.
Public Forecasts- 12/17 forecast targets met with a stretch of 17/17.
4.
Customer Forecasts- 2/3 forecast targets met with a stretch of 3/3.
Global NWP Index is compiled from:
• Mean sea-level pressure,
• 500 hPa height,
• 850 hPa wind,
• 250 hPa wind.
Plot showing increase in Global NWP Index
which shows an improvement in Global model accuracy.