Are Numerical
Weather
Prediction Models
Getting Better?
Cliff Mass, David Ovens, and Jeff
Baars
University of Washington
Getting Better?
• The overall performance of numerical weather
prediction has gotten better over the past
several decades.
• But such improvements could have come from
a number of sources:
•
•
•
•
Increased resolution
Better data assimilation
Improved global data assets (e.g., satellite data)
Better models and associated parameterizations.
The Question of This Talk
To what degree are our
modeling systems
IMPROVING in their
ability to deliver better
simulations and
forecasts?
Additional Question
To What Degree Do We Actually
KNOW how much our models
are improving?
Most modeling systems have
shown a flattening of the
improvement curve the last 5
years or so—even with improving
data for initialization and
increasing resolution
ECMWF Flat
GFS and NAM Heavy Precipitation
Threat Scores: Flat-lined
500 hPa
Day 2
Most Global Models Show Minimal Improvement the Last 5 Years
What about mesoscale models
like WRF?
Developmental Testbed Center (DTC)
Reference Configuration Tests and
Comparisons (3.6 versus 3.4)
Purple: 3.6
Green: 3.4
Upper Air
Temperature
No real overall improvement
between WRF 3.4 and 3.6
Let’s Check Out the Long-Term
Verification of the Northwest
Regional Prediction System
• Frozen MM5 driven by operational National
Weather Service NAM
• Continuous upgraded WRF driven by
operational NWS GFS and recently by RAP
• So we are driving the local models with
supposedly improving national guidance.
July 1, 2006
June 1, 2015
Bottom Line: No evidence of
significant improvement in UW
WRF over 9 years, even with
synoptic models getting slightly
better
But let’s do this more rigorously
• Run a history of WRF models from
substantially different “eras” using the SAME
large scale forcing.
• Jan 1-Feb 28, 2010, Pacific Northwest Domain
• Attempted to keep physics parameterizations
consistent (but they are being updated as the
modeling system changes)
WRF Versions Tested
•
•
•
•
2.2.1 - October 31, 2007
3.1.1 - July 31, 2009
3.5 - April 18, 2013
3.7 - April 20, 2015
A stroll through WRF history
• WRF version 2.2.1 with Kain-Fritsch cumulus
parameterization, simple ice physics, YSU PBL, Noah LSM, and
RRTM longwave and Dudhia shortwave radiation
• WRF version 3.1.1 with Kain-Fritsch cumulus
parameterization, Thompson microphysics, YSU PBL, Noah
LSM, and RRTM longwave and Dudhia shortwave radiation
• WRF version 3.5 with Kain-Fritsch cumulus parameterization,
Thompson microphysics, YSU PBL, Noah LSM, and RRTMG
longwave and shortwave radiation
• WRF version 3.7 with Kain-Fritsch cumulus parameterization,
Thompson microphysics, YSU PBL, Noah LSM, and RRTMG
longwave and shortwave radiation
Some Results
Sea Level Pressure (24h Forecast)
2.2.1 WINS
Forecasts GET WORSE Over
Time
Wind Speed
3.5 and 3.7 Better Than Early Versions
3.5 lower MAE than 3.7
2-m Temperature
3.1.1 has lowest MAE,
2.2.1 Worst MAE and ME
Wind Direction
3.5 best, followed by 3.7
2.21 better than 3.11
Conclusions (based on many more
stats)
• The WRF modeling system does not get steadily
better, but staggers forward in a halting manner
• A new version of the modeling system is not
necessarily better.
• Some statistics have generally improved, others
have not.
• Precipitation (not shown above) has been one of
the winners (improvement over the last
decade). Positive Definite advection helped.
Other modeling systems are no
better than WRF
Why Slow Progress? Some
Possibilities
• NWP is now a mature technology and we
should not expect significant advances
• There is no organized, coherent system in
place to improve physics parameterizations
and model numerics/structure.
• There is a lack of organized
verification/evaluation of model simulations
to find problems.
Without detail evaluations of the
strengths of weaknesses of our
modeling system are we driving
blind?
Our Challenge
• We need a completely new paradigm on how
we develop and fund NWP model
development, and how we apply the
enormous resources of our community more
effectively.
• Need to bring together the huge,
uncoordinated NWP research effort in the U.S
• Reduce the number of models, determine the
key weaknesses in NWP systems, and direct
resources to rectify them.
The END