An Evaluation of Red Flag Warnings
and Numerical Trend Forecasts
in the Northwest Geographic Area
during the 2004 Fire Season
The Pot Peak Fire encounters heavy fuels and pushes a smoke column to 30,000 ft.
Wenatchee NF, Washington
Prepared by:
The Fire Weather Working Team
Pacific Northwest Wildfire Coordinating Group
This page intentionally left blank.
-2-
Executive Summary
The Memorandum of Understanding (MOU) between the Pacific Northwest Wildfire
Coordinating Group (PNWCG) and the National Weather Service (NWS) establishes the
framework for cooperation and commitment by these parties to improve fire weather in the
Northwest and to document standards for forecasts and services. The National Weather Service
has agreed to maintain staffing and services for the fire weather program until it has the
advanced operational technology in place and demonstrates its ability to provide fire weather
services equal to those prior to the implementation of Modernization and Restructuring (MAR).
Forecast accuracy standards set in place within the MOU were determined from those achieved
by the NWS in the 1980s and 90s, prior to MAR.
Evaluation and verification of fire weather products must:
 be based upon a quantifiable process
 be consistent from year to year
 be viewed with respect to trends over a number of years, and
 identify solutions to improve accuracy.
Evaluation’s ultimate goal is to improve the quality and accuracy of fire weather products and
services used in the Pacific Northwest that affect fire management decisions with respect to
resource allocation, firefighter and public safety.
The following document is an evaluation of Red Flag Warnings and Numerical Trend Forecasts
issued by the six NWS offices (Seattle, Spokane, Portland, Pendleton, Medford and Boise)
within the Northwest Geographic Area during the 2004 fire season with comparisons to previous
years. The evaluation was performed in accordance with Section V, paragraph five of the MOU
between PNWCG as defined in EXHIBIT C, Forecast and Service Standards. The initial results
of the evaluation were presented to the NWS in November, 2004 with modifications for the final
report based upon input provided by individual NWS offices.
Red Flag Warnings were verified using the published criteria in the NWS Annual Operating Plan
(AOP) for 2004, lightning data, Remote Automatic Weather Station (RAWS) data, National Fire
Danger Rating (NFDRS) indices, fuel moisture values, and fire information. Red Flag Warnings
are issued by the NWS to notify wildland fire agencies of weather conditions that, in conjunction
with critically dry fuels, could lead to a dramatic increase in fire danger or wildfire activity.
Timely and accurate warnings enable wildland fire fighting agencies to manage critical resources
and prepare appropriate suppression responses for protecting life and property. The combination
of critical weather events (i.e. strong wind, low relative humidity and/or dry lightning) and low
fuel moisture is defined as a “Red Flag” event. Fire Weather Watches are used to alert land
management agencies in advance of possible Red Flag events.
Numerical Trend Forecasts were evaluated by comparing observed and forecast weather data
archived in the Weather Information Management System (WIMS). NWS forecasts are
compared against the persistence error for particular weather stations. The persistence error
refers to the numeric difference (or the change) from one day to the next for a particular weather
element at approximately 1400 LDT. For example, if the observed temperature is 75 degrees F
one day and 78 degrees F the next, the persistence error is 3 degrees F. The elements verified are
temperature, relative humidity, and wind speed. The “improvement over persistence”
technique is superior to other verification methods because it captures the ability of
forecasts to predict changes in the weather. Changes in wind speed and relative humidity
have a great impact on fire behavior and fire danger.
-3-
In summary, the evaluation produced the following results:













There were far fewer Red Flag Warnings issued in the Pacific Northwest this year
compared to the last five years (1999-2003). This is a reflection of the mild 2004 Oregon
fire season.
Portland was the only NWS office whose Red Flag Warnings were more accurate than
the 1999 NWS Western Region average. The 1999 results are used as benchmark values
because this was the last year of dedicated fire weather forecasters and prior to full
compliance of NWS’ Modernization and Restructuring (MAR).
In general, Red Flag Warnings issued by the five other NWS offices failed to match the
accuracy results published by the NWS Western Region in 1999.
Seattle, Medford and Pendleton had False Alarm Rates (FAR) greater than .50, which
means the majority of warnings issued by those offices did not verify.
The majority of actual Red Flag events were forecast.
There were very few procedural errors in issuing and canceling Red Flag Warnings this
year.
No problems were noted in the coordination of Red Flag Warnings among NWS offices
this year. This was due to the creation of “seamless” red flag criteria among adjacent
NWS offices and daily conference calls with NWS and NWCC meteorologists.
Seattle, Spokane, Portland and Medford exceeded the 60% goal of warnings preceded by
a watch established in the PNWCG/NWS MOU. This was much better than 2003 when
only Portland reached this goal. Most of Pendleton’s warnings were issued without the
advanced notification of a watch.
In general, Numerical Trend Forecasts do not achieve the accuracy standards established
in the MOU, those attained by the NWS in the 1980s and 1990s. The one exception is
Boise’s temperature and relative humidity forecasts which exceed the standard.
Temperature is the weather element most accurately forecast by the NWS, followed by
relative humidity and wind speed.
Seattle, Pendleton, and Boise are the only NWS offices to produce wind speed forecasts
better than persistence.
Although most NWS offices continue to show slight improvement in forecast accuracy,
MOU accuracy standards are not likely to be met by most NWS offices within the next 5
years (based upon 2000-2004 trends). Questions have been raised that the MOU accuracy
standards are too strict. However, NWS verification statistics, from Seattle and Boise
during the 1980s and 1990s, show these standards were met about 50% of the time.
The fact that the MOU accuracy standards have not been met, and that Red Flag Warning
accuracy has not matched pre-MAR, indicates there has been little if any improvement in
forecast accuracy since MAR was instituted in 2000, and in some cases a degradation in
accuracy.
A concerted effort should be made by the NWS to ensure the issuance of timely and accurate
Red Flag Warnings that enhance fire management decisions and safety concerns for both the
public and firefighters. Weather conditions must be monitored (i.e. radar, RAWS observations)
and updated forecasts and warnings issued when necessary. A quantifiable measure of fuel
dryness needs to be determined by the fire agencies to assist the NWS when conditions are
favorable for large fires.
Numerical Weather Trend Forecasts, used in narrative fire weather and fire danger forecasts,
could be improved by using innovative forecast techniques, if the NWS chose to do so. The Fire
Weather Working Team remains willing to assist the NWS with any endeavor to improve Red
Flag Warning and Numerical Trend forecast accuracy.
-4-
The Evaluation of 2004 Red Flag Warnings
and Numerical Weather Trend Forecasts
Red Flag Warning Evaluation:
Red Flag Warnings are issued by the National Weather Service (NWS) to notify wildland fire
agencies of weather conditions that, in conjunction with critically dry fuels, could lead to a
dramatic increase in fire danger or wildfire activity. Timely and accurate warnings enable
wildland fire fighting agencies to manage critical resources and prepare appropriate suppression
responses for protecting life and property. The combination of critical weather events (i.e. strong
wind, low relative humidity and/or dry lightning) and low fuel moisture is defined as a “Red
Flag” event. Fire Weather Watches are used to alert land management agencies in advance of
possible Red Flag events.
The following report details the evaluation of Red Flag Warnings issued by the six NWS offices
(Seattle, Spokane, Portland, Pendleton, Medford and Boise) within the Northwest Geographic
Area during the 2004 fire season. The evaluation was performed in accordance with the
Memorandum of Understanding (EXHIBIT C Forecast and Service Standards, Section E)
between the Pacific Northwest Wildfire Coordinating Group (PNWCG) and the NWS. Red Flag
Warnings were verified using the published criteria in the NWS Annual Operating Plan (AOP)
for 2004, lightning data, Remote Automatic Weather Station (RAWS) data, National Fire Danger
Rating (NFDRS) indices, fuel moisture values, and fire information. An examination of these
Red Flag Warnings follows. (See Appendix A for specifics)
Findings
1. False Alarm Rate (FAR)
The False Alarm Rate (FAR) is the percentage of Red Flag warnings that do not verify.
The FAR can vary from 1.00 (no warnings correct) to .00 (all warnings correct). The
more often events are forecast and do not occur, the higher (worse) the score. Thus, the
FAR is a measure of Red Flag Warning accuracy. The closer the FAR is to .00, the more
accurate the warnings. The False Alarm Rate for all warnings varied between .33 and
1.00, depending upon the office. Seattle, Pendleton, and Medford all had False Alarm
Rates greater than .50, which means the majority of warnings issued by those offices did
not verify. The Portland FAR was significantly better this year and it was the only office
that exceeded the .36 FAR published as the NWS Western Region average in 1999.
Listed below are the dry lightning, wind/low RH and all warning False Alarm Rates for
the six NWS offices.
Office
Spokane
Seattle
Portland
Pendleton
Medford
Boise
FAR
FAR
FAR
Dry lightning Wind/Low RH All Warnings
.57
.67
.33
.67
N/A
.25
.00
N/A
N/A
.50
1.00
.60
.50
.67
.33
.59
1.00
.44
-5-
2. Probability of Detection (POD)
The Probability of Detection (POD) is the percentage of actual Red Flag events that are
correctly forecast. The more often events are accurately forecast, the better the score.
The POD can vary from 1.00 (all Red Flag events are correctly forecast) to .00 (all Red
Flag events are not forecast). The Probability of Detection ranged from 1.00 at Seattle
and Portland to .00 at Medford. Except for Medford, the majority of Red Flag events
were correctly forecast.
Listed below are the dry lightning, wind/low RH and all warning Probability of Detection
values for the six NWS offices.
Office
Spokane
Seattle
Portland
Pendleton
Medford
Boise
POD
POD
POD
Dry lightning Wind/Low RH All Warnings
.86
1.00
1.00
1.00
.00
.60
1.00
N/A
N/A
.83
N/A
1.00
.89
1.00
1.00
.90
.00
.71
3. Critical Success Index (CSI)
The Critical Success Index (CSI) is a combination of FAR and POD. It is the ratio of
correct forecasts to the number of actual Red Flag events plus the number of incorrect
forecasts. The best score is 1.00, the worst is 0. Critical Success Index values ranged
from .67 at Portland to .00 at Medford. The Portland CSI was significantly better this
year and it was the only office that exceeded the 1999 NWS Western Region CSI average
of .58.
Listed below are the dry lightning, wind/low RH and all warning Critical Success Index
values for the six NWS offices.
Office
Spokane
Seattle
Portland
Pendleton
Medford
Boise
CSI
CSI
CSI
Dry lightning Wind/Low RH All Warnings
.40
.33
.67
.33
.00
.50
1.00
N/A
N/A
.45
.00
.40
.47
.33
.67
.39
.00
.45
-6-
4. Number of Red Flag Warnings
There were 64 Red Flag Warnings issued in the Northwest Geographic Area in 2004.
This compares to 153 in 2003, 214 in 2002, 100 in 2001 and 183 in 2000. This is far
fewer than last year due to the mild fire season in Oregon this summer. The number of
warnings issued by office are as follows: Medford 5, Spokane 16, Pendleton 22, Portland
9, Boise 9 and Seattle 3.
5. Coordination of Red Flag warnings/criteria among National Weather Service offices
No problems were noted with the coordination of Red Flag Warnings this year. This was
due in part to the daily telephone conference call between the Geographic Area
Coordination Center and NWS meteorologists, and the creation of “seamless” red flag
criteria among adjacent Weather Service offices with similar fire weather patterns, fuels
and topography.
6. Missed Red Flag events
An evaluation of “missed” Red Flag events compared each office’s published Red Flag
criteria to hourly RAWS data. The Predictive Services Branch of the Northwest Area
Coordination Center daily archives hourly observations from approximately 200 RAWS.
A data base query for each weather station produced a list of the hours during the day in
which the criteria was either met or exceeded. Isolated occurrences were discarded. Only
those instances with multiple hours and stations were counted as “missed” warnings. The
majority of missed warnings were due to dry lightning and not strong wind and low
relative humidity. There were only 5 missed warnings this year.
7. Procedural errors in the issuance of Red Flag Warnings
Procedural errors are those in conflict with National Weather Service (National Weather
Service Instruction 10-401) or NFDRS directives. There were very few errors noted this
year compared to past fire seasons. Most of the errors involved format, improper
cancellation, or inconsistent statements between the general forecast (FWF) and the
warning statement (RFW).
8. Percentage of Red Flag Warnings preceded by a Fire Weather Watch
The PNWCG and NWS Memorandum of Understanding (MOU) in Exhibit C states that,
“at least 60% of all Red Flag Warnings will be preceded by a Fire Weather Watch.” This
is to ensure that there is sufficient advance notification of Red Flag events to properly
prepare firefighting resources. Four of the six NWS offices achieved the 60% goal and
all did significantly better than 2003. The following is the percentage of time watches
preceded warnings by office: Portland 78%, Pendleton 41%, Medford 100%, Boise 56%,
Seattle 100%, and Spokane 63%.
-7-
Fire Weather Numerical Trend Forecast Evaluation
An evaluation of Numerical Trend Forecasts produced by six National Weather Service (NWS)
offices as input into the National Fire Danger Rating System (NFDRS) were evaluated at the
conclusion of the 2004 fire season by the Predictive Services Section at the Northwest
Interagency Coordination Center. This was the fifth year in which a detailed assessment was
performed in accordance with Section V, paragraph five of the Memorandum of Understanding
between the Pacific Northwest Wildfire Coordinating Group (PNWCG) and the NWS. The
purpose of monitoring and evaluating in the Pacific Northwest is to improve the quality and
accuracy of all fire weather products and services used in fire management decisions that affect
resource allocation and firefighter and public safety.
Trend forecasts are evaluated by comparing observed and forecast weather data archived in the
Weather Information Management System (WIMS). Daily NWS forecasts are compared against
the persistence error for a particular weather station. The persistence error refers to the numeric
difference (or the change) from one day to the next for a particular weather element at
approximately 1400 LDT. For example, if the observed temperature is 75 degrees F one day and
78 degrees F the next, the persistence error is 3 degrees F. The elements verified are
temperature, relative humidity, and wind speed. Changes in wind speed and relative humidity
have a much greater impact on fire behavior and fire danger than temperature.
Improvement over persistence reflects the ability of forecasts to correctly predict weather
changes. Since persistence forecasts can be determined without any meteorological input (just
forecast the same weather as yesterday’s observed), NWS skill should result in forecast accuracy
that consistently exceeds the persistence error. A total of 100 RAWS stations were evaluated
between June 1 and September 30: 22 in the Medford fire weather district, 15 in Portland, 27 in
Pendleton, 17 in Spokane, 13 in Seattle and 6 in the Boise district.
Spreadsheets were constructed for each RAWS to calculate daily persistence and forecast errors
for temperature, relative humidity and wind speed. Once persistence and forecaster errors were
calculated for each day, the average error for each was calculated over the entire fire season.
These values were then input into the following formula to calculate forecaster improvement
over persistence.
Forecaster improvement = (persistence error – forecaster error)/persistence error * 100
A negative value indicates a forecast worse than persistence. The higher a positive value (perfect
forecasts = 100%), the better the forecast in relation to persistence.
These figures were then entered into a summary spreadsheet that divided the 100 RAWS into
NWS fire weather districts and fire weather zones. Additional information, such as the
beginning and ending dates of NFDRS forecasts and the number of forecasts generated during
the four-month (122 day) period were also included.
The table on the next page summarizes the improvement over persistence of NWS forecasts
during the 2000–2004 fire seasons. In general, NWS forecasts are still below the accuracy
standards established in the Memorandum of Understanding between the NWS and the Pacific
Northwest firefighting agencies. This is especially evident in the relative humidity and wind
speed forecasts. The wind speed forecasts are less accurate than persistence, or what was
observed the previous day at Medford, Portland and Spokane. (Note: the negative red figures in
the wind column indicate forecast skill less than that of persistence.) Although most NWS offices
continue to show slight improvement in forecast accuracy, MOU accuracy standards are not
-8-
likely to be met by most NWS offices within the next 5 years (based upon 2000-2004 trends).
Questions have been raised that the MOU accuracy standards are too strict. However, NWS
verification statistics, from Seattle and Boise during the 1980s and 1990s, show these standards
were met about 50% of the time.
Appendix B contains the verification results by NWS office and graphics showing trends since
2000 compared to the MOU Accuracy standards. (Note: Seattle provided 2004 verification
numbers were generally within 2% of the FWWT figures.)
Office
Seattle
2004
2003
2002
2001
2000
Improvement over Persistence
Temperature
Relative Hum.
Wind
18.63
23.39
25.89
23.24
22.42
17.96
18.35
20.59
21.17
19.10
3.71
-4.15
5.23
-11.18
-18.12
25.92
26.25
25.56
20.69
19.43
13.96
8.67
14.52
17.89
8.64
-4.86
-16.20
-6.86
-13.94
-17.59
29.69
30.40
27.36
26.71
29.54
23.13
23.09
22.45
18.53
18.72
-2.02
-12.39
-7.94
-11.09
-11.69
28.51
36.81
35.12
31.26
28.76
19.57
22.13
22.45
20.62
19.06
0.26
-1.82
-1.49
-6.94
-15.88
26.41
29.77
31.75
28.80
25.75
15.85
17.58
18.15
14.92
14.54
-3.64
-5.51
-11.83
-2.53
-6.29
2004
2003
2002
2001
2000
37.71
28.19
29.28
32.25
N/A
25.40
18.01
20.51
23.54
N/A
1.57
5.37
7.31
0.37
N/A
MOU Fcst
Standards
35.00
25.00
10.00
Spokane
2004
2003
2002
2001
2000
Portland
2004
2003
2002
2001
2000
Pendleton
2004
2003
2002
2001
2000
Medford
2004
2003
2002
2001
2000
Boise
-9-
This page intentionally left blank.
- 10 -
Appendix A
- 11 -
This page intentionally left blank.
- 12 -
Seattle 2004 Red Flag Warnings
Date
Zones
Reason
Large Fire
Potential
Aug 13-14
662
Dry Lightning
High
Aug 14
658, 659
Dry Lightning
High
Total Warnings: 3
Dry Lightning: 3
Correct warnings: 1
Incorrect warnings: 2
Warnings Preceded with a Watch: 3 of 3 or 100%
Verification
No – 662 (no lightning observed)
Preceded with a watch. Canx 0325 PDT 8-15
Yes – 658
No – 659 (no lightning observed)
Preceded with a watch.
Wind/low RH: 0
Missed warnings: 0
False Alarm Rate:
Dry Lightning . 67
Wind/low RH N/A All . 67
Probability of Detection:
Dry Lightning 1. 00 Wind/low RH N/A All 1.00
Critical Success Index:
Dry Lightning . 33
Wind/low RH N/A All .33
Note:
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (1/1 + 2) = 1 – (1/3) = 1 - .33 = . 67
Dry Ltng
1 – (a/a + b) = 1 – (1/1 + 2) = 1 – (1/3) = 1 - .33 = . 67
Wind/Low RH 1 – (a/a + b) = 1 – (0/0 + 0) = 1 – (0/0) = 1 – N/A = N/A
Probability of Detection: All
a/(a + c) = 1/(1 + 0) = 1/1 = 1.00
Dry Ltng
a/(a + c) = 1/(1 + 0) = 1/1 = 1.00
Wind/Low RH a/(a + c) = 0/(0 + 0) = 0/0 = N/A
Critical Success Index:
All
a/(a + b + c) = 1/(1 + 2 + 0) = 1/3 = .33
Dry Ltng
a/(a + b + c) = 1/(1 + 2 + 0) = 1/3 = . 33
Wind/Low RH a/(a + b + c) = 0/(0 + 0 + 0) = 0/0 = N/A
- 13 -
Spokane 2004 Red Flag Warnings
Large Fire
Potential
C1, E3 High
E1, E2 Mod
Date
Zones
Reason
July 13
673, 676, 677
Dry lightning
July 26
684
Wind/Low RH
High
Aug 2-3
673, 686
Dry lightning
High
Aug 14-15
673, 676, 677,
680, 682, 684,
685
Dry lightning
E3 High
E1, C1 Mod
Aug 14-15
686, 687
Dry lightning
E2 Mod
Sept 30
684
Wind/ Low RH
Low
July 13
684
Dry lightning
E1, E2 Mod
Total Warnings: 16
Dry Lightning: 14
Correct Warnings: 8
Incorrect Warnings: 8
Warnings Preceded with a Watch: 10 of 16 or 63%
Verification
Yes – 673, 676, 677
Not preceded with a watch
Yes – 684
Not preceded with a watch
Yes – 673, 687
Not preceded with a watch
Yes – 685
No – 673, 676, 677, 680, 682, 684 (no
lightning strikes observed)
Preceded with a watch
No – 686, 687 (no lightning strikes observed)
Preceded with a watch
Yes – 684
Preceded with a watch
Missed 684 (lightning strikes, little if any
rain, and NFDRS fire danger VH)
Wind/low RH/Haines: 2
Missed Warnings: 1
False Alarm Rate:
Dry Lightning . 57
Wind/low RH/Haines . 00
All . 50
Probability of Detection:
Dry Lightning . 86
Wind/low RH/Haines 1. 00 All . 89
Critical Success Index:
Dry Lightning . 40
Wind/low RH/Haines 1. 00 All . 47
Note:
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (8/8 + 8) = 1 – (8/16) = 1 - .50 = . 50
Dry Ltng
1 – (a/a + b) = 1 – (6/6 + 8) = 1 – (6/14) = 1 - .43 = . 57
Wind/Low RH 1 – (a/a + b) = 1 – (2/2 +0) = 1 – (2/2) = 1 - 1 = . 00
Probability of Detection: All
a/(a + c) = 8/(8 + 1) = 8/9 = . 89
Dry Ltng
a/(a + c) = 6/(6 + 1) = 6/7 = . 86
Wind/Low RH a/(a + c) = 2/(2 + 0) = 2/2 = 1. 00
Critical Success Index:
All
a/(a + b + c) = 8/(8 + 8 + 1) = 8/17 = . 47
Dry Ltng
a/(a + b + c) = 6/(6 + 8 + 1) = 6/15 = . 40
Wind/Low RH a/(a + b + c) = 2/(2 + 0 + 0) = 2/2 = 1. 00
- 14 -
Portland 2004 Red Flag Warnings
Date
Zones
Reason
Large
Fire
Potential
Aug 1
607, 608
Episode Lightning
W3 – Mod
W4 - High
Aug 2-3
608
Episode Lightning
W3 – Mod
W4 - High
Aug 13
608
Episode Lightning
W3 – Mod
W4 - High
Aug 14
605, 606, 607,
608, 660
Episode Lightning
W3 – Mod
W4 - High
Total Warnings: 9
Dry Lightning: 9
Correct Warnings: 6
Incorrect Warnings: 3
Warnings Preceded with a Watch: 7 of 9 or 78%
Verification
Yes – 608
No – 607 (no lightning strikes)
Preceded with a watch in 608 but not 607
Expired at 2056 PDT
Yes – 608
Not preceded with a watch
Ended at 0615 PDT 8-3
Yes - 608
Preceded with a watch
Yes – 605, 606, 608
No – 607, 660 (no lightning strikes)
Preceded with a watch
Wind/low RH/Haines: 0
Missed Warnings: 0
False Alarm Rate:
Dry Lightning . 33
Wind/low RH N/A All . 33
Probability of Detection:
Dry Lightning 1. 00 Wind/low RH N/A All 1. 00
Critical Success Index:
Dry Lightning . 67
Wind/low RH N/A All . 67
Note:
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (6/6 + 3) = 1 – (6/9) = 1 - .67 = . 33
Dry Ltng
1 – (a/a + b) = 1 – (6/6 + 3) = 1 – (6/9) = 1 - .67 = . 33
Wind/Low RH 1 – (a/a + b) = 1 – (0/0 +0) = 1 – (0/0) = 1 – N/A = N/A
Probability of Detection: All
a/(a + c) = 6/(6 + 0) = 6/6 = 1. 00
Dry Ltng
a/(a + c) = 6/(6 + 0) = 6/6 = 1. 00
Wind/Low RH a/(a + c) = 0/(0 + 0) = 0/0 = N/A
Critical Success Index:
All
a/(a + b + c) = 6/(6 + 3 + 0) = 6/9 = . 67
Dry Ltng
a/(a + b + c) = 6/(6 + 3 + 0) = 6/9 = . 67
Wind/Low RH a/(a + b + c) = 0/(0 + 0 + 0) = 0/0 = N/A
- 15 -
Pendleton 2004 Red Flag Warnings
Date
Zones
Reason
Large
Fire
Potential
July 23-24
609, 610, 611,
681
Low RH/Poor RH
Recovery
C1 – High
C2 – High
July 24
630, 631, 638
Low RH/Poor RH
Recovery
C2 – High
E3 – N/A
E4 - High
July 25
610
Wind/Low RH
C2 - High
July 25-26
631
Wind/Low RH
E3 - High
July 26
610
Wind/Low RH
C2 - High
Aug 14
All 12 Zones
Lightning after a
dry period
C1 – High
C2 – High
E3 – High
E4 - High
July 26
638
Wind/Low RH
E4 - High
Total Warnings: 22
Dry Lightning: 12
Correct warnings: 9
Incorrect warnings: 13
Warnings Preceded with a Watch: 9 of 22 or 41%
Verification
Yes – 611
No – 609, 610, 681 (daytime criteria not met)
Not preceded with a watch.
Yes – 630, 638
No – 631 (daytime criteria not met)
Not preceded with a watch. Expired 2304 PDT
Yes - 610
Not preceded with a watch.
Yes - 631
Not preceded with a watch. Expired 2147 PDT
No - 610
Not preceded with a watch. Expired 2204 PDT
Yes – 610, 611, 630 634
No – 609, 631, 632, 633, 635, 638, 675, 681
(no lightning occurred in these zones)
Preceded with a watch in 9 of 12 zones.
Missed zone 638 (criteria met at 2 stations for
at least 2 hours)
Wind/low RH/Poor Recovery: 10
Missed warnings: 1
False Alarm Rate:
Dry Lightning . 67
Wind/low RH . 50
All . 59
Probability of Detection:
Dry Lightning 1. 00 Wind/low RH . 83
All . 90
Critical Success Index:
Dry Lightning . 33
All . 39
Wind/low RH . 45
Note:
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (9/9 + 13) = 1 – (9/22) = 1 - .41 = . 59
Dry Ltng
1 – (a/a + b) = 1 – (4/4 + 8) = 1 – (4/12) = 1 - .33 = . 67
Wind/Low RH 1 – (a/a + b) = 1 – (5/5 +5) = 1 – (5/10) = 1 - .50 = . 50
Probability of Detection: All
a/(a + c) = 9/(9 + 1) = 9/10 = . 90
Dry Ltng
a/(a + c) = 4/(4 + 0) = 4/4 = 1. 00
Wind/Low RH a/(a + c) = 5/(5 + 1) = 5/6 = . 83
Critical Success Index:
All
a/(a + b + c) = 9/(9 + 13 + 1) = 9/23 = . 39
Dry Ltng
a/(a + b + c) = 4/(4 + 8 + 0) = 4/12 = . 33
Wind/Low RH a/(a + b + c) = 5/(5 + 5 + 1) = 5/11 = . 45
- 16 -
Medford 2004 Red Flag Warnings
Date
Zones
Reason
Large
Fire
Potential
Sept 3-4
618, 619, 620
Wind/Low RH
W4 - Low
Oct 8
624, 625
Wind/Low RH
C2 - Mod
Aug 13
624
Dry Lightning
C3 - High
Total Warnings: 5
Dry Lightning: 0
Correct Warnings: 0
Incorrect Warnings: 3
Warnings Preceded with a Watch: 5 of 5 or 100%
Verification
No – 618, 619, 620 (no stations met criteria)
Preceded with a watch
No- 624, 625 (criteria not met)
Preceded with a watch
Missed dry lightning event (numerous fire starts
Fremont NF with one 4,000+ acre fire)(SPC
had this zone in a Critical Fire Weather area
for dry lightning)
Wind/low RH/HI: 3
Missed Warnings: 1
False Alarm Rate:
Dry Lightning N/A Wind/low RH 1. 00 All 1. 00
Probability of Detection:
Dry Lightning . 00
Wind/low RH N/A
All . 00
Critical Success Index:
Dry Lightning . 00
Wind/low RH . 00
All . 00
Note:
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (0/0 + 5) = 1 – (0/5) = 1 - .00 = 1. 00
Dry Ltng
1 – (a/a + b) = 1 – (0/0 + 0) = 1 – (0/0) = N/A
Wind/Low RH 1 – (a/a + b) = 1 – (0/0 +5) = 1 – (0/5) = 1 - .00 = 1. 00
Probability of Detection: All
a/(a + c) = 0/(0 + 1) = 0/1 = . 00
Dry Ltng
a/(a + c) = 0/(0 + 1) = 0/1 = . 00
Wind/Low RH a/(a + c) = 0/(0 + 0) = 0/0 = N/A
Critical Success Index:
All
a/(a + b + c) = 0/(0 + 5 + 1) = 0/6 = . 00
Dry Ltng
a/(a + b + c) = 0/(0 + 0 + 1) = 0/1 = . 00
Wind/Low RH a/(a + b + c) = 0/(0 + 5 + 1) = 0/6 = . 00
- 17 -
Boise 2004 Red Flag Warnings
Date
Zones
Reason
Large
Fire
Potential
June 24
636, 637
Dry lightning
Mod
Aug 5-6
637
Wind/Low RH
High
Aug 13-14
636, 637
Dry lightning
High
Sept 1
636, 637
Wind/Low RH
Mod
Oct 8
636, 637
Wind/Low RH
Mod
July 24-25
636, 637
Dry lightning
High
Total Warnings: 9
Dry Lightning: 4
Correct warnings: 5
Incorrect warnings: 4
Warnings Preceded with a Watch: 5 of 9 or 56%
Verification
Yes – 636, 637
Not preceded with a watch.
No- no stations met criteria
Preceded with a watch.
Yes – 636
No - 637
Preceded with a watch.
No – no stations met criteria
Not preceded with a watch.
Yes – 636, 637
Preceded with a watch
Missed Warning - Watch not upgraded to a
warning based upon scattered lightning strikes
and meager rainfall amounts
Wind/low RH/Haines: 5
Missed warnings: 2
False Alarm Rate:
Dry Lightning . 25
Wind/low RH . 60
All . 44
Probability of Detection:
Dry Lightning . 60
Wind/low RH 1. 00 All . 71
Critical Success Index:
Dry Lightning . 50
Wind/low RH . 40
All . 45
Note:
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (5/5 + 4) = 1 – (5/9) = 1 - .56 = . 44
Dry Ltng
1 – (a/a + b) = 1 – (3/3 + 1) = 1 – (3/4) = 1 - .75 = . 25
Wind/Low RH 1 – (a/a + b) = 1 – (2/2 +3) = 1 – (2/5) = 1 - .40 = . 60
Probability of Detection: All
a/(a + c) = 5/(5 + 2) = 5/7 = . 71
Dry Ltng
a/(a + c) = 3/(3 + 2) = 3/5 = . 60
Wind/Low RH a/(a + c) = 2/(2 + 0) = 2/2 = 1. 00
Critical Success Index:
All
a/(a + b + c) = 5/(5 + 4 + 2) = 5/11 = . 45
Dry Ltng
a/(a + b + c) = 3/(3 + 1 + 2) = 3/6 = . 50
Wind/Low RH a/(a + b + c) = 2/(2 + 3 + 0) = 2/5 = . 40
- 18 -
2000 Red Flag Warning Verification
NWS Office
POD
FAR
CSI
# Wrngs
Seattle
Spokane
Portland
Pendleton
Medford
Boise
N/A
.86
N/A
.84
.78
.90
N/A
.62
1.00
.76
.63
.18
N/A
.36
N/A
.23
.33
.75
0
65
1
88
19
10
1999 NWS WR
Average
.83
.36
.58
2001 Red Flag Warning Verification
NWS Office
POD
FAR
CSI
# Wrngs
Seattle
Spokane
Portland
Pendleton
Medford
Boise
N/A
N/A
N/A
N/A
N/A
N/A
1.00
.52
.21
.30
.56
.29
N/A
N/A
N/A
N/A
N/A
N/A
3
33
14
27
16
7
1999 NWS WR
Average
.83
.36
.58
2002 Red Flag Warning Verification
NWS Office
POD
FAR
CSI
# Wrngs
Seattle
Spokane
Portland
Pendleton
Medford
Boise
1.00
.75
.50
.67
.54
.85
.80
.62
.55
.36
.42
.15
.20
.33
.31
.48
.39
.73
5
56
22
50
54
13
1999 NWS WR
Average
.83
.36
.58
- 19 -
2003 Red Flag Warning Verification
NWS Office
POD
FAR
CSI
# Wrngs
Seattle
Spokane
Portland
Pendleton
Medford
Boise
1.00
.78
.44
.88
.45
.67
.65
.56
.87
.46
.57
.45
.35
.39
.11
.50
.29
.43
20
41
30
28
23
11
1999 NWS WR
Average
.83
.36
.58
2004 Red Flag Warning Verification
NWS Office
POD
FAR
CSI
# Wrngs
Seattle
Spokane
Portland
Pendleton
Medford
Boise
1.00
.89
1.00
.90
.00
.71
.67
.50
.33
.59
1.00
.44
.33
.47
.67
.39
.00
.45
5
16
9
22
5
9
1999 NWS WR
Average
.83
.36
.58
Note: Figures in red indicate less accuracy than 1999 NWS Western Region average
POD = Probability of Detection which is the percentage of Red Flag events correctly forecast.
The best possible score is 1.00 and the worst .00
FAR = False Alarm Rate which is the percentage of forecasts which were incorrect. The more
often a Red Flag event is forecast and does not occur, the worse the score. The best possible
score is .00 and the worst 1.00
CSI = Critical Success Index which is the percentage of correct Red Flag Warnings to the
number of events plus the number of incorrect forecasts. The best possible score is 1.00 and the
worst .00
- 20 -
- 21 -
- 22 -
Appendix B
- 23 -
This page intentionally left blank.
- 24 -
2004 Forecaster Improvement over Persistence (NFDRS weather trends)
June 1 through Sept 30, 2004
(Forecaster Improvement over
Persistence)
Relative
Temperature
Hum.
Wind
Portland NWS
Zone Sta. #
Fcst
Dates
Number
Days
Fcst
%
Days
Fcst
601
350215
Cedar Creek
30.26
22.17
10.18
6/02-9/30
98
80
602
350216
South Fork
36.62
28.21
14.95
6/01-9/30
104
85
451209
Abernathy Mtn
17.36
10.39
-9.00
6/09-9/30
87
71
603
352547
Village Creek
34.00
27.34
17.44
6/02-9/30
114
93
351710
Rockhouse
34.87
21.61
8.82
6/02-9/30
56
46
606
352024
Yellowstone
38.19
25.63
-2.67
6/03-9/30
92
75
352552
Trout Creek
25.59
20.94
-17.46
6/02-9/30
111
91
607
350604
Log Creek
26.39
20.31
0.00
6/01-9/30
111
91
350718
Red Box
28.33
32.29
-10.53
6/01-9/30
111
91
350726
Wanderer's Pk
30.06
30.06
10.09
6/01-9/30
112
92
608
352554
Pebble
28.57
27.99
4.35
6/02-9/30
114
93
352558
Emigrant Creek
29.40
22.20
-10.55
6/02-9/30
113
93
612
352545
Goodwin Peak
26.90
21.10
-40.24
6/02-9/30
77
63
660
451921
Canyon Creek
31.06
23.68
-3.71
6/01-9/30
103
84
451208
Elk Rock
27.75
13.10
Ques? Obs
6/09-9/30
87
71
AVG
29.69
23.13
-2.02
99
81
MEDIAN
29.40
22.20
-1.34
MAX
38.19
32.29
17.44
MIN
17.36
10.39
-40.24
TOTAL FCSTS
Pendleton NWS
Zone Sta. #
609
610
611
630
631
632
633
1490
(Forecaster Improvement over
Persistence)
Relative
Temperature
Hum.
Wind
Fcst
Dates
Number
Days
Fcst
%
Days
Fcst
350912
Polliwog
24.37
15.50
-20.42
6/01-9/30
106
87
350913
Wamic Mill
29.75
21.02
-14.29
6/01-9/30
106
87
350917
Mutton Mountain
35.01
22.55
4.82
6/01-9/30
101
83
350920
HeHe #1
30.46
22.09
2.06
6/01-9/30
103
84
352620
Colgate
32.48
4.77
0.00
6/01-9/30
110
90
353402
Cabin Lake
26.27
17.54
-9.29
6/01-9/30
118
97
352618
Lava Butte
30.78
21.89
10.71
6/01-9/30
118
97
352107
Haystack
33.39
18.75
17.40
6/01-9/30
116
95
352712
Salt Creek
32.08
21.64
-1.01
6/01-9/30
115
94
352207
Slide Mountain
28.98
19.58
-4.19
6/01-9/30
116
95
352701
Cold Spring
26.45
20.26
-0.86
6/01-9/30
116
95
453201
Juniper Dunes
26.80
17.71
4.36
6/02-9/30
104
85
351001
Patjens
28.87
11.21
8.82
6/01-9/30
118
97
350915
North Pole Ridge
27.93
13.93
20.45
6/01-9/30
117
96
352329
Case
33.77
24.32
1.06
6/01-9/30
113
93
352327
Fall Mountain
18.32
18.51
-8.78
6/02-9/30
114
93
353424
Antelope
27.71
19.08
7.80
6/01-9/30
114
93
351518
Eden
28.59
20.23
-18.27
6/01-9/30
112
92
- 25 -
453803
Alder
27.51
19.58
11.59
6/02-9/30
112
92
351317
Black Mountain 2
27.45
15.99
-10.89
6/02-9/30
111
91
351417
LaGrande
22.29
23.15
-9.67
6/01-9/30
106
87
634
351416
Minam Lodge
30.07
24.43
-10.70
6/01-9/30
106
87
635
351502
Harle Butte
29.67
18.85
0.56
6/01-9/30
100
82
351520
Roberts Butte
30.80
18.60
0.64
6/01-9/30
100
82
638
352416
Blue Canyon
30.34
24.58
10.19
6/01-9/19
94
77
352418
Sparta Butte
30.36
29.92
-4.36
6/01-9/30
104
85
681
452404
Greyback
30.75
25.29
20.64
6/19-9/29
85
70
AVG
28.94
19.67
0.31
109
89
MEDIAN
29.33
19.58
0.28
MAX
35.01
29.92
20.64
MIN
18.32
4.77
-20.42
TOTAL FCSTS
(Forecaster Improvement over
Persistence)
Relative
Temperature
Hum.
Wind
Boise NWS
Zone Sta. #
636
637
2935
Fcst
Dates
Number
Days
Fcst
%
Days
Fcst
353520
Basque Hills
36.05
28.57
24.48
6/02-9/29
96
79
353526
Moon Hill
40.61
30.92
-35.92
6/02-9/29
99
81
353512
Wagontire
34.36
19.55
30.04
6/02-9/29
99
81
353612
Grassy Mountain
42.36
27.37
-5.34
6/02-9/29
101
83
353613
Kelsey Butte
37.27
21.48
-5.17
6/02-9/29
101
83
353614
Owyhee Ridge
6/02-9/29
101
83
100
82
35.62
24.52
1.34
AVG
37.71
25.40
1.57
MEDIAN
36.66
25.95
-1.92
MAX
42.36
30.92
30.04
MIN
34.36
19.55
-35.92
TOTAL FCSTS
597
(Forecaster Improvement over
Persistence)
Relative
Temperature
Hum.
Wind
Medford NWS
Zone Sta. #
Fcst
Dates
Number
Days
Fcst
%
Days
Fcst
615
352814
Powers 2
20.11
14.37
4.40
6/03-9/30
97
80
616
352816
Signal Tree
6.34
15.25
8.90
6/02-9/29
64
52
617
352546
Sugarloaf
39.44
30.15
-1.85
6/04-9/30
93
76
353031
Cinnamon Butte
32.09
19.63
-5.26
6/04-9/30
87
71
353040
Buckeye
33.56
25.43
-16.88
6/04-9/30
97
80
618
352920
Red Mound
26.04
22.12
-4.76
6/02-9/29
63
52
619
352915
Quail Prairie
27.76
19.84
-2.03
6/03-9/30
93
76
620
352919
Calvert
1.90
6.18
-4.42
6/03-9/30
92
75
353115
Illinois Valley
23.85
-7.59
3.51
6/03-9/30
93
76
353114
Onion Mtn
32.38
21.55
2.56
6/03-9/30
93
76
353120
Provolt
22.82
-1.82
-6.28
6/03-9/30
92
75
621
353214
Star R.S.
32.07
-2.97
-26.50
6/04-9/28
87
71
622
353228
Evans Creek
27.80
23.99
-6.06
6/04-9/30
93
76
353230
Buckhorn
27.02
6.62
-12.14
6/04-9/30
93
76
623
353345
Rover
insufficient observations
353227
Zim
44.13
-2.40
6/02-9/30
71
58
27.71
- 26 -
624
625
353339
Seldom
27.54
18.99
-24.67
6/02-9/30
353307
Calimus Butte
30.40
35.83
-15.05
353328
Gerber
24.11
18.07
-3.15
353337
Timothy
20.25
9.50
8.45
353422
Coffee Pot
23.18
16.88
-3.23
353406
Fort Rock
25.88
8.71
353424
Rock Creek
32.37
20.18
AVG
26.41
15.85
-3.64
MEDIAN
27.28
18.53
-3.19
MAX
44.13
35.83
15.64
MIN
1.90
-7.59
-26.50
101
83
6/02-9/29
99
81
6/03-9/30
102
84
6/02-9/30
92
75
6/02-9/30
106
87
11.25
6/02-9/30
104
85
15.64
6/02-9/30
104
85
92
75
TOTAL FCSTS
2016
(Forecaster Improvement over
Persistence)
Relative
Temperature
Hum.
Wind
Seattle NWS
Zone Sta. #
Fcst
%
Days
Dates
Number
Days
Fcst
3.58
6/11-9/30
74
61
Fcst
650
450130
Ellis Mtn
6.37
19.31
651
450306
Minot
18.26
15.46
8.72
6/11-9/30
84
69
652
450211
Owl Mtn
22.76
26.59
17.09
6/11-9/30
73
60
658
451415
Sumas
21.45
12.31
2.76
7/05-9/30
85
70
451613
Gold Mountain
23.78
23.39
Bad Data
6/06-9/30
97
80
659
661
662
451611
Johnson
19.84
22.50
Bad Data
6/10-9/30
96
79
451509
Finney
21.18
23.38
Bad Data
6/06-9/30
97
80
451919
Orr Creek
19.62
16.42
-8.60
6/13-9/30
96
79
451105
Kosmos
25.20
18.52
0.79
8/08-9/30
74
61
451718
Greenwater
21.33
23.94
Bad Data
6/06-9/30
100
82
450117
Cougar
11.85
13.54
-9.47
6/06-9/30
90
74
450911
Jefferson
11.53
8.64
4.88
6/06-9/30
92
75
452121
Stehekin
6/12-9/30
84
69
88
72
18.99
9.43
13.64
AVG
18.63
17.96
3.71
MEDIAN
19.84
18.52
3.58
MAX
25.20
26.59
17.09
MIN
6.37
8.64
-9.47
TOTAL FCSTS
(Forecaster Improvement over
Persistence)
Relative
Temperature
Hum.
Wind
Spokane NWS
Zone Sta. #
673
677
680
682
684
685
1142
Fcst
%
Days
Dates
Number
Days
Fcst
8.90
6/19-9/29
75
61
72
59
Fcst
452601
Douglas
453601
Escure
37.59
15.55
11.47
6/19-9/29
452136
Entiat
21.67
5.53
-1.61
6/19-9/29
87
71
452132
Camp Four
22.76
15.14
-8.18
6/19-9/29
87
71
86
70
34.68
15.57
452134
Dry Creek
16.81
17.48
6.02
6/19-9/29
452206
Peoh Point
23.88
17.61
-9.34
6/19-9/29
86
70
87
71
452601
Sedge Ridge
35.28
-2.94
1.97
6/19-9/29
452219
Swuak
31.00
23.13
-6.31
6/19-9/29
79
65
2.39
6/19-9/30
85
70
6/19-9/30
87
71
452030
NCSB
452040
Kramer
452006
First Butte
36.78
10.43
insufficient forecasts
20.20
- 27 -
17.19
-14.86
452036
686
687
83 Monument
17.58
16.20
-20.27
6/19-9/30
88
72
82
67
453412
Deer Mountain
29.54
18.63
-15.22
6/19-9/30
452916
Kettle Falls
11.43
4.67
-4.86
6/19-9/29
79
65
452510
Gold Mountain
452514
Brown Mountain
-13.58
6/19-9/30
86
70
6/19-9/30
89
73
84
69
452029
Lost Lake
insufficient forecasts
21.05
15.41
28.60
19.86
-9.49
AVG
25.92
13.96
-4.86
MEDIAN
23.88
15.57
-6.31
MAX
37.59
23.13
11.47
MIN
11.43
-2.94
-20.27
TOTAL FCSTS
1255
- 28 -
- 29 -
- 30 -
- 31 -