Evaluation of a Challenging
Warm Season QPF month at HPC:
June 2009
Brendon Rubin-Oster
Richard Otto
(with contributions from Mike Bodner, Keith Brill,
David Novak and HPC management)
June 3, 2010
1
Motivation For This Review
HPC 1” Threat Score was 0.131 for June 2009
• Lowest since 1999 (0.119)
• FY 2009 goal is 0.290
• NAM threat score of 0.111 is lowest since
at least 1998
• GFS threat score of 0.092 tied its lowest in
10 years (1999)
• HPC % Improvement over models:
18% NAM, 43% GFS, 16% ECMWF
2
(2009)
3
Seek To Answer the Following
• What (if anything) was so different about June 2009?
• Was there a common theme associated with the busted
forecasts? (MCS activity, scattered convection, synoptic
scale forcing, mesoscale boundaries, etc.)
• What kinds of errors were observed?
(timing/duration, placement, magnitude, etc.)
• What were the causes of these errors?
• How did the models perform?
4
Data & Methodology
Selected model data archived at HPC for later review
deterministic (NAM, GFS, ECMWF, CMC, UKMET)
ensembles (SREF, CMCE, ECENS)
•
•
•
2 km base reflectivity radar composite archive (NMAPRAD)
VIS/IR satellite & analysis data from internet
(SPC archives, UCAR case studies)
NCEP/NCAR reanalysis for monthly means
Limitations on available data
•
•
•
•
•
Surface: winds, moisture convergence, T, Td, θe, moisture transport,
1000 mb frontogenesis, mixing ratio
Thickness: 1000-850 mb & 850-700 mb
Instability: CAPE, lifted indices
Miscellaneous: Q-vectors, Corfidi vectors
And others…
5
6
What (if anything) was so different
about June 2009?
7
500mb Geopotential Height means (m)
Source: http://www.esrl.noaa.gov/psd/data/composites/day/
8
250 mb mean zonal winds (m/s): June 1-30
Source: http://www.esrl.noaa.gov/psd/data/composites/day/
9
Observed Precipitation (in) (June 1-30)
Source: http://water.weather.gov/
10
Percent of Normal Precipitation (June 1-30)
Source: http://water.weather.gov/
11
Was there a common theme associated
with the busted forecasts?
Defining Modes of Convection
● Mesoscale Convective System (MCS):
 organized cluster of thunderstorms
 resides significant distance from synoptic boundaries
 persists for at least 3 to 6 hours
Defining Modes of Convection
● Convection along a synoptic scale boundary:
 thunderstorms developing within 150 km of a synoptic
boundary
Defining Modes of Convection
● Convection along a mesoscale boundary:
 thunderstorms developing within 150 km of a mesoscale
surface boundary
 consists of: sea breezes, surface trofs, outflow boundaries
Defining Modes of Convection
● Scattered convection:
 thunderstorms not appearing to be associated with
synoptic/mesoscale boundaries
 does not meet MCS guidelines
Was there a common theme associated
with the busted forecasts?
Busted forecasts (14 cases)
7%
Convection along synoptic
boundaries (7)
MCS activity (3)
7%
14%
22%
50%
Convection along
mesoscale boundaries (2)
Scattered convection (1)
Stratiform precip from
synoptic system (1)
What kinds of errors were observed?
Error types (14 cases)
7%
Magnitude (6)
21%
43%
Placement & Magnitude
(4)
Placement (3)
29%
Timing/Duration &
Magnitude (1)
What were the causes of these errors?
Reasons for busts (14 cases)
14%
22%
14%
22%
14%
14%
Unforecast mesoscale
boundary (3)
Mishandling of MCV(s)
(3)
Erroneous convective
feedback H5 vort (2)
Incorrect placement of
synoptic boundary (2)
Uncertain MCS
development (2)
Unknown (2)
How did the models perform
during these 14 cases?
Rank
Performer
Threat Score
Rank
Performer
Bias
1
NAM
0.042
1
NAM
1.353
2
HPC
0.033
2
ECMWF
1.462
3
GFS
0.027
3
GFS
1.932
4
ECMWF
0.026
4
HPC
2.810
Source: http://www2.hpc.ncep.noaa.gov/npvu/
20
June 26, 2009 case study
45mm / 1.78” PW
OBSERVED
NAM f036
AUTO f036
HPC f036
GFS f036
EC f036
24-hr radar loop {June 25 (1200Z)- June 26 (1200Z)}
Satellite loop June 25 (1415Z-2315Z)
Final Notes / Future Work?
• Explore how atypical the June 2009 observed
phenomena and forecast performance were
relative to other June cases
• Compare performance of high resolution
mesoscale models
• Use object oriented verification to objectively
quantify errors
• HPC is currently applying MODE (method for
object-based diagnostic evaluation) in real-time