APPENDIX III
AFWA MM5 Model Verification Statistics
The following provides a series of graphics that depict the bias and RMSE plots
for many of the AFWA MM5 products currently available via AFWIN. If you discover
trends or other characteristics on these products, please feed that information back to us.
AFWA verifies all MM5 coarse grid windows and some of nests (If sufficient obs are
available). Standard parameters are verified (Geopotential Height, Temperature (at
mandatory levels), Relative Humidity, Winds, and Sea Level Pressure. AFWA uses the
AGROMET database to verify precipitation , and we hope to use the National Weather
Service (NWS) rain gauge network for verification in the near future. The AVN, ETA,
and NOGAPS also verified, and compared to MM5 for the same parameters (we only
compare domain to domain). We verify every cycle, whether it is a 00Z/12Z cycle, or a
06Z/18Z cycle. There is an obvious dilemma with 12/00Z versus 06/18Z fields: We have
limited upper air data at 06Z/18Z, so we have to verify with that limited data that is
available. Clearly, this could be a source of potential inaccuracy in our verification
statistics.
The method used to conduct the verification is as follows: model data
interpolated to observation site. Bi-quadratic interpolation is used to fit the model data
to the observation site. This method is highly useful, since it not only allows for
comparison of the ACTUAL values (at a point in time), but also allows for comparison of
rates of change of these parameters. Nearest neighbor used in precipitation interpolation.
The differences between the actual values and model values are determined (the residuals
are calculated) and the differences are summed over the entire window domain.
A wide variety of products are available from the final raw numbers:
- Monthly bias charts
- Three month trends
- Skill score comparisons with other models
As of 01 September 1998, this data is available on the World Wide Web at:
http://www.scott.af.mil/afwa/dn/dnt/offutt/mm5/mm5stats.htm
Other products can be requested from Mr. Robert Craig (DSN 271-3186). Data is stored
in Microsoft Access format, for those interested in conducting independent studies.
Bias statistics for the MM5 at AFWA have been compiled for seven of the 36 km
windows. We have put together the statistics for Geopotential Height, Relative Humidity
(Rh), Temperature, and Wind speed at each mandatory level from 1000 mb through 100
mb. These graphs (below) are vertical displays. To see a specific graphic, refer to the
table below by region and parameter.
Additional note: on all the graphs below, the following applies:
Both "T1A" and "t1a" mean "Alaska Region"
Both "T2A" and "t2a" mean "CONUS Region"
Both "T3A" and "t3a" mean "Europe Region"
Both "T4A" and "t4a" mean "Southwest Asia Region"
Both "T5A" and "t5a" mean "South Central America Region"
Both "T6A" and "t6a" mean "Southeast Asia / Western Pacific Region"
RMSE statistics for the MM5 at AFWA have been compiled for some of the 36
km windows. We have put together the statistics for Sea Level Pressure, 700 mb Relative
Humidity (Rh), 850 mb and Surface Temperature, and Wind speed at 250 mb. These
graphs (below) are displays of average Root Mean Square Error over the entire domain of
the window in question.
We are only providing a selection of bias and RMSE graphics from the CONUS (T2A)
36 km window. If you wish to see the graphics for other regions, please see our MM5
Information and Training Page, which is accessed via AFWIN. . To access this page, go
to AFWIN, then select any theater, such as “CONUS”. Then you need to select “Model
Charts” in the blue field/frame on the left. After this, select “MM5 Fine Scale Model
Output” in the main hypertext field on the left. Finally, select the next hyperlink, which
is entitled “About the MM5 (strengths, weaknesses, resolution, products, etc.)”. This is
the location for the MM5 information and training page. Hit the “Enter Here” Button.
After this, you will be directed to the CONTENTS page. At the Bottom of this page, you
will see a button for MM5 STATISTICS. This is the location for all MM5 statistics from
AFWA. Be sure to check this site often, since AFWA will be moving this site to a new
location (in the future) to support increased user bandwidth needs.
Model comparisons for MM5 and the AVN, ETA, and NOGAPS have been also
compiled for the CONUS region and others. Model skill scores are used for comparison
since they tend to be less biased to different model resolution than RMSEs. Skill scores
compare how well mode forecast gradients of a quantity compare to observation
gradients of the same quantity. The better the comparison, the lower the skill score. Only
AVN, NOGAPS, and ETA data over a particular MM5 are used in the calculations.
For additional questions on the MM5 statistics, contact :
HQ Air Force Weather Agency (AFWA)
Office Code : DNTT \ DSN: 271-2110
Commerical: (402) 294-2110
Email: mm5info@afwa.af.mil
-9
Ap 8
r
7- -98
Ap
10 r-9
-A 8
p
13 r-9
-A 8
p
16 r-9
-A 8
p
19 r-9
-A 8
p
22 r-9
-A 8
p
25 r-9
-A 8
p
28 r-9
-A 8
p
1- r-9
M 8
a
4- y-9
M 8
a
7- y-9
M 8
10 ay-M 98
13 ay-M 98
16 ay-M 98
19 ay-M 98
22 ay-M 98
25 ay-M 98
28 ay-M 98
31 ay-M 98
ay
3- -98
Ju
n
6- -98
Ju
n
9- -98
Ju
12 n-9
-J 8
u
15 n-9
-J 8
u
18 n-9
-J 8
u
21 n-9
-J 8
u
24 n-9
-J 8
u
27 n-9
-J 8
u
30 n-9
-J 8
un
-9
8
Ap
r
Ap
r
-9
Ap 8
r-9
7Ap 8
10 r-9
-A 8
p
13 r-9
-A 8
p
16 r-9
-A 8
p
19 r-9
-A 8
p
22 r-9
-A 8
p
25 r-9
-A 8
p
28 r-9
-A 8
p
1- r-9
M 8
ay
4- -9
M 8
a
7- y-9
M 8
10 ay-M 98
13 ay-M 98
16 ay-M 98
19 ay-M 98
22 ay-M 98
25 ay-M 98
28 ay-M 98
31 ay-M 98
ay
3- -98
Ju
n
6- -98
Ju
n
9- -98
Ju
12 n-9
-J 8
u
15 n-9
-J 8
u
18 n-9
-J 8
u
21 n-9
-J 8
u
24 n-9
-J 8
u
27 n-9
-J 8
u
30 n-9
-J 8
un
-9
8
4-
1-
m
4-
1-
T2A 18hr 500mb Height, RMSE 3 month trend
60
50
40
30
20
10
0
T2A 18hr Surface Temperature, RMSE 3 month trend
5.5
5.0
4.5
4.0
3.5
C
3.0
2.5
2.0
1.5
1.0
0.5
0.0
4-
Ap
r
-9
Ap 8
r
7- -98
Ap
10 r-9
-A 8
p
13 r-9
-A 8
p
16 r-9
-A 8
19 pr-9
-A 8
p
22 r-9
-A 8
p
25 r-9
-A 8
p
28 r-9
-A 8
p
1- r-9
M 8
a
4- y-9
M 8
a
7- y-9
M 8
10 ay-M 98
13 ay-M 98
16 ay-M 98
19 ay-M 98
22 ay-M 98
25 ay-M 98
28 ay-M 98
31 ay-M 98
ay
3- -98
Ju
n
6- -98
Ju
n
9- -98
Ju
12 n-9
-J 8
u
15 n-9
-J 8
u
18 n-9
-J 8
u
21 n-9
-J 8
u
24 n-9
-J 8
u
27 n-9
-J 8
u
30 n-9
-J 8
un
-9
8
1-
T2A 18hr Sea Level Pressure, RMSE 3 month trend
4.5
4.0
3.5
3.0
2.5
mb
2.0
1.5
1.0
0.5
0.0