The Evaluation of Red Flag Warnings
in the Northwest Geographic Area
during the 2003 Fire Season
D:\626080504.doc
Executive Summary
Red Flag Warnings are issued by the National Weather Service (NWS) to notify wildland fire
agencies of weather conditions that, in conjunction with critically dry fuels, could lead to a
dramatic increase in fire danger or wildfire activity. Timely and accurate warnings enable
wildland fire fighting agencies to manage critical resources and prepare appropriate suppression
responses for protecting life and property. The combination of critical weather events (i.e. strong
wind, low relative humidity and/or dry lightning) and low fuel moisture is defined as a “Red
Flag” event. Fire Weather Watches are used to alert land management agencies in advance of
possible Red Flag events.
The following summarizes the evaluation of Red Flag Warnings issued by the six NWS offices
(Seattle, Spokane, Portland, Pendleton, Medford and Boise) within the Northwest Geographic
Area during the 2003 fire season. The evaluation was performed in accordance with the
Memorandum of Understanding (EXHIBIT C Forecast and Service Standards, Section E)
between the Pacific Northwest Wildfire Coordinating Group (PNWCG) and the NWS. Red Flag
Warnings were verified using the published criteria in the NWS Annual Operating Plan (AOP)
for 2003, lightning data, Remote Automatic Weather Station (RAWS) data, National Fire Danger
Rating (NFDRS) indices, fuel moisture values, and fire information. In summary:








Spokane, Seattle, Medford and Portland had False Alarm Rates (FAR) greater than .50,
which means the majority of warnings issued by those offices did not verify. These
offices had a tendency to “over-warn” for the situation.
The majority of actual Red Flag events were forecast, however the probability that these
events are correctly forecast has worsened since 2000. There were a number of days in
from late June through early September in which Red Flag Warnings should have been
issued and weren’t for strong wind and low humidity in eastern Washington and eastern
Oregon.
There were 153 Red Flag Warnings issued by the NWS in the Northwest Geographic
Area in 2003. This compares to 214 in 2002, 100 in 2001 and 183 in 2000. This is fewer
than last year due to minimal large fire activity in southern Oregon this summer.
Procedural errors in issuing and canceling Red Flag Warnings were less frequent this
year, but still continue at a few offices.
No problems were noted in the coordination of Red Flag Warnings among NWS offices
this year. This was due to the creation of “seamless” red flag criteria among adjacent
NWS offices and daily conference calls with NWCC meteorologists.
Except for Portland, the percentage of Warnings preceded by a Watch DID NOT achieve
the 60% goal established in the PNWCG/NWS MOU. The other five NWS offices were
all under 40% as most Warnings were issued with little or no advance notification.
The 2003 Critical Success Index and False Alarm Rates of Red Flag Warnings in the
Northwest failed to meet the NWS Western Region averages published in 1999. The
Probability of Detection was met or exceeded at two of the six NWS Offices. The 1999
NWS Western Region averages are used as “benchmark” values because this was the last
year of “dedicated” fire weather forecasters and prior to full compliance of NWS’s
Modernization and Restructuring (MAR).
A concerted effort must be adopted by NWS and NWCC meteorologists to monitor
current weather (i.e. radar, RAWS observations) and fuel conditions to ensure that Red
Flag Warnings are timely and accurate enhancing firefighter safety.
D:\626080504.doc
Introduction:
Red Flag Warnings are issued by the National Weather Service (NWS) to notify wildland fire
agencies of weather conditions that, in conjunction with critically dry fuels, could lead to a
dramatic increase in fire danger or wildfire activity. Timely and accurate warnings enable
wildland fire fighting agencies to manage critical resources and prepare appropriate suppression
responses for protecting life and property. The combination of critical weather events (i.e. strong
wind, low relative humidity and/or dry lightning) and low fuel moisture is defined as a “Red
Flag” event. Fire Weather Watches are used to alert land management agencies in advance of
possible Red Flag events.
The following summarizes the evaluation of Red Flag Warnings issued by the six NWS offices
(Seattle, Spokane, Portland, Pendleton, Medford and Boise) within the Northwest Geographic
Area during the 2003 fire season. The evaluation was performed in accordance with the
Memorandum of Understanding (EXHIBIT C Forecast and Service Standards, Section E)
between the Pacific Northwest Wildfire Coordinating Group (PNWCG) and the NWS. Red Flag
Warnings were verified using the published criteria in the NWS Annual Operating Plan (AOP)
for 2003, lightning data, Remote Automatic Weather Station (RAWS) data, National Fire Danger
Rating (NFDRS) indices, fuel moisture values, and fire information. An examination of these
Red Flag Warnings follows.
Findings:
1. False Alarm Rate (FAR)
The False Alarm Rate (FAR) is the percentage of Red Flag warnings that did not verify.
The FAR can vary from 1.00 (no warnings correct) to .00 (all warnings correct). The
more often an event is forecast and does not occur, the higher (worse) the score. Thus, it
is a measure of Red Flag Warning accuracy. The closer the FAR is to .00, the more
accurate the warnings. The False Alarm Rate for all warnings varied from .45 to .87,
depending upon the office. Spokane, Seattle, Medford and Portland had False Alarm
Rates greater than .50, which means that the majority of warnings issued by those offices
did not verify. None of the NWS offices matched the FAR of .36 published as the NWS
Western Region average in 1999.
Listed below are the dry lightning, wind/low RH and all warning False Alarm Rates for
the six NWS offices.
Office
Spokane
Seattle
Portland
Pendleton
Medford
Boise
D:\626080504.doc
FAR
FAR
FAR
Dry lightning Wind/Low RH All Warnings
.44
.68
.95
.36
1.00
.50
.65
.50
.72
.57
.29
.33
.56
.65
.87
.46
.57
.45
2. Probability of Detection (POD)
The Probability of Detection (POD) is the percentage of actual Red Flag events that were
correctly forecast. The more often an event is accurately forecast, the better the score. It
can vary from 1.00 (all Red Flag events are correctly forecast) to .00 (all Red Flag events
were not forecast). The Probability of Detection ranged from 1.00 at Seattle to .44 at
Portland and .45 at Medford. Except for Portland and Medford, the majority of Red Flag
events were correctly forecast.
Listed below are the dry lightning, wind/low RH and all warning Probability of Detection
values for the six NWS offices.
Office
Spokane
Seattle
Portland
Pendleton
Medford
Boise
POD
POD
POD
Dry lightning Wind/Low RH All Warnings
.91
1.00
1.00
1.00
N/A
1.00
.67
1.00
.38
.75
.45
.40
.56
1.00
.44
.88
.45
.67
3. Critical Success Index (CSI)
The Critical Success Index (CSI) is a combination of FAR and POD. It is the ratio of
correct forecasts to the number of actual Red Flag events plus the number of incorrect
forecasts. The best score is 1.00, the worst is 0. Critical Success Index values ranged
from .50 at Pendleton to .11 at Portland. None of the offices matched or exceeded the
1999 NWS Western Region CSI average of .58.
Listed below are the dry lightning, wind/low RH and all warning Critical Success Index
values for the six NWS offices.
Office
Spokane
Seattle
Portland
Pendleton
Medford
Boise
D:\626080504.doc
CSI
CSI
CSI
Dry lightning Wind/Low RH All Warnings
.53
.33
.05
.64
.00
.50
.30
.50
.19
.38
.38
.33
.39
.35
.11
.50
.29
.43
4. Number of Red Flag Warnings
There were 153 Red Flag Warnings issued in the Northwest Geographic Area in 2003.
This compares to 214 in 2002, 100 in 2001 and 183 in 2000. This is fewer than last year
due to minimal large fire activity in southern Oregon this summer. The number of
warnings by office were as follows: Medford 23, Spokane 41, Pendleton 28, Portland 30,
Boise 11 and Seattle 20.
5. Coordination of Red Flag warnings/criteria among National Weather Service offices
No problems were noted with the coordination of Red Flag Warnings this year. This was
due to the daily telephone conference call between the Geographic Area Coordination
Center and NWS meteorologists, and the creation of “seamless” red flag criteria among
adjacent Weather Service offices with similar fire weather patterns, fuels and topography.
6. Missed Red Flag events
An evaluation of “missed” Red Flag events compared each office’s published Red Flag
criteria to hourly RAWS data. The Predictive Services Branch of the Northwest Area
Coordination Center daily archives 24-hour observations from approximately 200
RAWS. A data base query of each weather station produced a list of the hours during
each day in which the criteria was either met or exceeded. Isolated occurrences were
discarded; only those instances with multiple hours and stations were counted as
“missed” warnings. The vast majority of missed warnings were due to strong wind and
low relative humidity, and not dry lightning. There were a total of 27 missed warnings,
most of which occurred in eastern Oregon north-central Washington.
7. Procedural errors in the issuance of Red Flag Warnings
Procedural errors are those in conflict with National Weather Service (WSOM Chapter
D-06, Western Region ROML W-10-96) or NFDRS directives. There were relatively
few errors noted this year compared to past fire seasons. Most of the errors detected
involved format, improper cancellation, or inconsistent statements between the general
forecast (FWF) and the warning statement (RFW).
8. Percentage of Red Flag Warnings preceded by a Fire Weather Watch
The PNWCG and NWS Memorandum of Understanding (MOU) in Exhibit C states that,
“at least 60% of all Red Flag Warnings will be preceded by a Fire Weather Watch.” This
is to ensure that there is sufficient advance notification of Red Flag events to properly
prepare firefighting resources. Of the six NWS offices, only Portland achieved the 60%
goal. The following is the percentage of time warnings were preceded by watches by
office: Portland 70%, Pendleton 36%, Medford 39%, Boise 27%, Seattle 5%, and
Spokane 34%.
D:\626080504.doc
Appendix A
D:\626080504.doc
Seattle 2003 Red Flag Warnings
Reason
Large
Fire
Potential
Date
Zones
June 29
649, 650, 651,
652, 661, 653,
654, 655, 656,
657, 658, 659
Dry Lightning
High
June 29
662
Wind/Low RH
High
July 30
662
Low RH/Haines 6
High
Sept 3
651, 654, 658,
655, 657, 659
Dry Lightning
High
Total Warnings: 20
Dry Lightning: 18
Correct warnings: 7
Incorrect warnings: 13
Warnings Preceded with a Watch: 1 of 20 or 5%
Verification
Yes – 652, 653, 654, 656, 658, 659
No – 649,650,651,661,655,657 (no lightning
observed)
Not preceded with a watch.
No (criteria not met)
Not preceded with a watch.
Yes
Not preceded with a watch.
No – 651, 655, 657, 659, 654, 658 (no observed
lightning strikes) (canceled 0730 am Sept 4)
Not preceded with a watch except for zone 658.
Wind/low RH: 2
Missed warnings: 0
False Alarm Rate:
Dry Lightning . 67
Wind/low RH . 50
Probability of Detection:
Dry Lightning 1. 00 Wind/low RH 1.00 All 1.00
Critical Success Index:
Dry Lightning . 33
Wind/low RH .50
All . 65
All .35
Note:
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (7/7 + 13) = 1 – (7/20) = 1 - .35 = . 65
Dry Ltng
1 – (a/a + b) = 1 – ( 6/6 + 12) = 1 – (6/18) = 1 - .33 = . 67
Wind/Low RH 1 – (a/a + b) = 1 – (1/1 +1) = 1 – (1/2) = 1 - .50 = . 50
Probability of Detection: All
a/a + c = 7/7 + 1 = 7/7= 1.00
Dry Ltng
a/a + c = 6/6 + 0 = 1/1 = 1.00
Wind/Low RH a/a + c = 1/1 + 1 = 1/1 = 1.00
Critical Success Index:
D:\626080504.doc
All
a/a + b + c = 7/7 + 13 = 7/20 = .35
Dry Ltng
a/a + b + c = 6/6 + 12 + 0 = 6/18 = . 33
Wind/Low RH a/a + b + c = 1/1 + 1 = 1/2 = .50
Spokane 2003 Red Flag Warnings
Date
Zones
Reason
Large
Fire
Potential
June 29
673, 676, 677,
684
Wind/Low RH
High
July 16
684, 685
Wind/Low RH
High
July 23
673, 676, 677,
680, 682, 684,
685, 686, 687
Wind/Low RH
High
Low RH/Haines
High
Dry lightning
High
July 30
Aug 5
676, 677, 680,
682, 684, 685,
687
676, 677, 680,
682, 684, 685,
687
Verification
No – 673, 676, 677, 684 (criteria not met)
Preceded with a watch
No – 684, 685 (criteria not met at 2 stations)
Not preceded with a watch
No – 673, 680, 684 (criteria not met at 2
stations)
No – 676, 677, 682, 685, 686, 687 (criteria not
met)
Preceded with a watch
Yes – 676, 677, 680, 682, 684, 685, 687
Not preceded with a watch
Yes – 676, 677, 682, 684, 685
No – 687 (no lightning) 680 (wet lightning)
Not preceded with a watch
Yes – 686
Preceded with a watch
Yes – 673, 684, 685, 686, 687
No – 676, 677, 680, 682 (no lightning)
Not preceded with a watch
No – 686, 687 (no lightning)
Not preceded with a watch
Missed 684
Aug 15
686
Low RH/Haines
High
Aug 15
673,676, 677,
680, 682, 684,
685, 686, 687
Dry lightning/Low
RH/ Haines
High
Sept 6
686, 687
Dry lightning/Wind
High
July 12
684
Wind/Low RH
High
Missed 684 (3 stations multiple hours)
Aug 26
684
Wind/Low RH
High
Missed 684 (2 stations multiple hours)
Sept 1
684
Wind/Low RH
High
Missed 684 (2 stations multiple hours)
Sept 26
684
Wind/Low RH
High
Missed 684 (2 stations multiple hours)
Total Warnings: 41
Dry Lightning: 18
Correct Warnings: 18
Incorrect Warnings: 23
Warnings Preceded with a Watch: 14 or 34%
Wind/low RH/Haines: 23
Missed Warnings: 5
False Alarm Rate:
Dry Lightning . 44
Wind/low RH/Haines . 65
Probability of Detection:
Dry Lightning . 91
Wind/low RH/Haines . 67 All . 78
Critical Success Index:
Dry Lightning . 53
Wind/low RH/Haines . 30
Note:
All . 56
All . 39
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
D:\626080504.doc
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (18/18 + 23) = 1 – (18/41) = 1 - .44 = . 56
Dry Ltng
1 – (a/a + b) = 1 – ( 10/10 + 8) = 1 – (10/18) = 1 - .56 = . 44
Wind/Low RH 1 – (a/a + b) = 1 – (8/8 +15) = 1 – (8/23) = 1 - .35 = . 65
Probability of Detection: All
a/a + c = 18/18 + 5 = 18/23 = . 78
Dry Ltng
a/a + c = 10/10 + 1 = 10/11 = . 91
Wind/Low RH a/a + c = 8/8 + 4 = 8/12 = . 67
Critical Success Index:
D:\626080504.doc
All
a/a + b + c = 18/18 + 23 + 5 = 18/46 = . 39
Dry Ltng
a/a + b + c = 10/10 + 8 + 1 = 10/19 = . 53
Wind/Low RH a/a + b + c = 8/8 + 15 + 4 = 8/27 = . 30
Portland 2003 Red Flag Warnings
Date
Zones
Reason
Large
Fire
Potential
July 21
607, 608, 609,
610, 611
Dry Lightning
High
July 22
607, 608, 609,
610, 611
Dry Lightning/Low
RH/Gusty
Thunderstorm wind
High
Dry Lightning/
Low RH/Haines
High
Wind/Low RH
Mod - W2
High - W3
Sept 3-4
Sept 26-27
603, 604, 605,
606, 607, 608,
609, 610, 611,
660
601, 602, 603,
604, 605, 606,
607, 608 612,
660
Verification
Yes – 611
No – 607, 608, 609, 610 (no lightning
observed) Preceded with a watch in 608, 611
Yes – 610 for wind and low RH
No – 607, 608 609, 611 (no lightning observed)
Not preceded with a watch
No – (no lightning observed)
Canceled 0930 am Sept 4
Preceded with a watch in all zones but 660
Yes - 602, 660
No – 601, 603, 604, 605, 606, 607, 608, 612
Preceded with a watch all zones
June 27-29
610
Wind/Low RH
High
Missed (2 stations met criteria)
June 29
611
Wind/Low RH
High
Missed (3 stations met criteria)
July 22-24
610
Wind/Low RH
High
Missed (3 stations met criteria)
July 30
611
Wind/Low RH
High
Missed (2 stations met criteria)
Sept 1
610
Wind/Low RH
High
Missed (2 stations met criteria)
Total Warnings: 30
Dry Lightning: 20
Correct Warnings: 4
Incorrect Warnings: 26
Warnings Preceded with a Watch: 21 or 70%
Wind/low RH/Haines: 10
Missed Warnings: 5
False Alarm Rate:
Dry Lightning . 95
Wind/low RH . 72
All . 87
Probability of Detection:
Dry Lightning 1. 00 Wind/low RH . 38
All . 44
Critical Success Index:
Dry Lightning . 05
All . 11
Note:
Wind/low RH . 19
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
D:\626080504.doc
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (4/4 + 26) = 1 – (4/30) = 1 - .13 = . 87
Dry Ltng
1 – (a/a + b) = 1 – ( 1/1 + 18) = 1 – (1/19) = 1 - .05 = . 95
Wind/Low RH 1 – (a/a + b) = 1 – (3/3 +8) = 1 – (3/11) = 1 - .27 = . 72
Probability of Detection: All
a/a + c = 4/4 + 5 = 4/9 = . 44
Dry Ltng
a/a + c = 1/1 + 0 = 1/1 = 1. 00
Wind/Low RH a/a + c = 3/3 + 5 = 3/8 = . 38
Critical Success Index:
D:\626080504.doc
All
a/a + b + c = 4/4 + 26 + 5 = 4/35 = . 11
Dry Ltng
a/a + b + c = 1/1 + 18 + 0 = 1/19 = . 05
Wind/Low RH a/a + b + c = 3/3 + 8 + 5 = 3/16 = . 19
Pendleton 2003 Red Flag Warnings
Date
Zones
Reason
Large
Fire
Potential
June 29
631, 675
Wind/Low RH
High
July 7
630,632, 633,
634, 635, 638
Lightning after an
extended dry
period
High
July 22-23
630, 632, 633,
634, 635, 638
Lightning after a
dry period
High
July 30
630, 631, 632,
681
Low RH/Haines 6
High
July 31
630, 631, 632,
638
Wind/Low RH
High
Aug 10
638
Wind/Low
RH/Haines
High
Aug 15-16
630, 631,675
Wind/Low RH
High
Sept 3-4
630, 632
Dry Lightning
High
July 12
631
Wind/Low RH
High
July 23
631
Wind/Low RH
High
Total Warnings: 28
Dry Lightning: 14
Correct warnings: 15
Incorrect warnings: 13
Warnings Preceded with a Watch: 10 or 36%
Verification
No – 675, 631 (only 1 station met the criteria)
Preceded with a watch.
Yes – 630, 632, 633, 634, 635, 638 (canceled
1008 am July 8)
Not preceded with a watch.
Yes- 630, 632, 638
No – 633, 634, 635, (no lightning observed)
(canceled 1057 pm July 23)
Preceded with a watch in 633, 634 and 635 but
not in 630, 632 and 638
Yes – 630, 631, 632, 681 (wind criteria also
met in 630 and 631)
Not preceded with a watch.
Yes – 631
No – 630, 632, 638 (criteria not met)
Preceded with a watch.
Yes – 638
Not preceded with a watch.
No – 630, 675, 631 (only 1 station met the
criteria) Expired 1100 am Aug 16
Not preceded with a watch.
No – 630, 632 (no lightning observed)
(canceled 0940 am Sept 4)
Preceded with a watch in 630 but not in 632.
Missed (criteria met by 1 station for 3 hours
and a second station for 6 hours)
Missed (criteria met by 1 station for 4 hours
and a second station for 7 hours)
Wind/low RH/Haines: 14
Missed warnings: 2
False Alarm Rate:
Dry Lightning . 36
Wind/low RH . 57
All . 46
Probability of Detection:
Dry Lightning 1. 00 Wind/low RH .75
All . 88
Critical Success Index:
Dry Lightning . 64
All . 50
Note:
Wind/low RH . 38
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
D:\626080504.doc
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (15/15 + 13) = 1 – (15/28) = 1 - .54 = . 46
Dry Ltng
1 – (a/a + b) = 1 – (9/9 + 5) = 1 – (9/14) = 1 - .64 = . 36
Wind/Low RH 1 – (a/a + b) = 1 – (6/6 +8) = 1 – (6/14) = 1 - .43 = . 57
Probability of Detection: All
a/a + c = 15/15 + 2 = 15/17 = . 88
Dry Ltng
a/a + c = 9/9 + 0 = 6/6 = 1. 00
Wind/Low RH a/a + c = 6/6 + 2 = 6/8 = . 75
Critical Success Index:
D:\626080504.doc
All
a/a + b + c = 15/15 + 13 + 2 = 15/30 = . 50
Dry Ltng
a/a + b + c = 9/9 + 5 + 0 = 9/14 = . 64
Wind/Low RH a/a + b + c = 6/6 + 8 + 2 = 6/16 = . 38
Medford 2003 Red Flag Warnings
Date
Zones
Reason
Large
Fire
Potential
July 30
617
HI 6
High
Sept 3
615, 616, 617,
618, 619, 620,
621, 622, 623
Dry Lightning
High
Sept 12-14
618, 619
Wind/Low RH
Low
Sept 21-23
618, 619
Wind/Low RH
High
Sept 25
618
Wind/Low RH
High
Sept 26-27
620, 621, 623
Wind/Low RH
High
Oct 24-25
618, 619, 620,
621, 622
Wind/Low RH
N/A
June 29
617, 622, 623,
624, 625
Wind/Low RH
High
Missed (criteria met at multiple stations)
Aug 9
625
Wind/Low RH
High
Missed (criteria met)
Sept 2
620, 622
Wind/Low RH
High
Missed (criteria met and 3 stations)
Sept 6
624, 625
Wind/Low RH
High
Missed (criteria met)
Total Warnings: 23
Dry Lightning: 9
Correct Warnings: 10
Incorrect Warnings: 13
Warnings Preceded with a Watch: 9 or 39%
Verification
Yes – 617
Not preceded with a watch
No - (no lightning observed until after the
warning was canceled at 0845 am Sept 4)
All but 618, 620, 621, 622, 623 preceded with a
watch
No – Fuel moisture high due to recent rains of
up to 2 inches
Preceded with a watch
Yes – 618, 619
Not preceded with a watch
Missed 623
Yes – 618
Not preceded with a watch
Yes – 620, 623 (no lead time)
No – 621
Not preceded with a watch
Yes – 618, 619, 620, 622
No – 621
Zones 618, 619, 620 preceded with a watch
Missed 623
Wind/low RH/HI: 14
Missed Warnings: 12
False Alarm Rate:
Dry Lightning 1.00 Wind/low RH . 29
All . 57
Probability of Detection:
Dry Lightning N/A Wind/low RH .45
All . 45
Critical Success Index:
Dry Lightning . 00
All . 29
Note:
Wind/low RH . 38
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
D:\626080504.doc
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (10/10 + 13) = 1 – (10/23) = 1 - .43 = . 57
Dry Ltng
1 – (a/a + b) = 1 – (0/0 + 9) = 1 – (0/9) = 1 - 0 = 1. 00
Wind/Low RH 1 – (a/a + b) = 1 – (10/10 +4) = 1 – (10/14) = 1 - .71 = . 29
Probability of Detection: All
a/a + c = 10/10 + 12 = 10/22 = . 45
Dry Ltng
a/a + c = 0/0 + 0 = 0/0 = N/A
Wind/Low RH a/a + c = 10/10 + 12 = 10/22 = . 45
Critical Success Index:
D:\626080504.doc
All
a/a + b + c = 10/10 + 13 + 12 = 10/35 = . 29
Dry Ltng
a/a + b + c = 0/0 + 9 + 0 = 0/9 = . 00
Wind/Low RH a/a + b + c = 10/10 + 4 + 12 = 10/26 = . 38
Boise 2003 Red Flag Warnings
Date
Zones
Reason
Large
Fire
Potential
June 18
637
Dry lightning/Wind
High
June 30 –
July 1
637
Wind/Low RH
High
July 7
636, 637
Dry lightning/Wind
High
July 22
636
Lightning after a dry
period
High
July 23
636, 637
Dry lightning
High
Aug 10
636, 637
Wind/Low
RH/Haines
High
Aug 26
636, 637
Dry lightning/Wind
High
June 29
636, 637
Wind/Low RH
High
Aug 9
636
Wind/Low RH
High
Missed 636 (criteria met)
Aug 15
636
Wind/Low RH
High
Missed 636 (criteria met)
Sept 6
636
Wind/Low RH
High
Missed 636 (criteria met)
Total Warnings: 11
Dry Lightning: 8
Correct warnings: 6
Incorrect warnings: 5
Warnings Preceded with a Watch: 3 of 11 or 27%
Verification
Yes
Not preceded with a watch.
No – 637 (criteria not met)
Not preceded with a watch.
Yes – 637
No – 636 (only isolated strikes)
Preceded with a watch.
No
Not preceded with a watch.
Yes
Zone 636 preceded with a watch.
Yes
Not preceded with a watch.
No (no lightning observed)
Not preceded with a watch.
Missed 636, 637 (multiple stations, multiple
hours) (local users did not want a warning
issued)
Wind/low RH/Haines: 3
Missed warnings: 3
False Alarm Rate:
Dry Lightning . 50
Wind/low RH . 33
All . 45
Probability of Detection:
Dry Lightning 1. 00 Wind/low RH . 40
All . 67
Critical Success Index:
Dry Lightning . 50
All . 43
Note:
Wind/low RH . 33
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
D:\626080504.doc
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (6/6 + 5) = 1 – (6/11) = 1 - .55 = . 45
Dry Ltng
1 – (a/a + b) = 1 – ( 4/4 + 4) = 1 – (4/8) = 1 - .50 = . 50
Wind/Low RH 1 – (a/a + b) = 1 – (2/2 +1) = 1 – (2/3) = 1 - .67 = . 33
Probability of Detection: All
a/a + c = 6/6 + 3 = 6/9 = . 67
Dry Ltng
a/a + c = 4/4 + 0 = 1/1 = 1. 00
Wind/Low RH a/a + c = 2/2 + 3 = 2/5 = . 40
Critical Success Index:
D:\626080504.doc
All
a/a + b + c = 6/6 + 5 + 3 = 6/14 = . 43
Dry Ltng
a/a + b + c = 4/4 + 4 + 0 = 4/8 = . 50
Wind/Low RH a/a + b + c = 2/2 + 1 + 3 = 2/6 = . 33
2000 Red Flag Warning Verification
NWS Office
POD
FAR
CSI
# Wrngs
Seattle
Spokane
Portland
Pendleton
Medford
Boise
N/A
.86
N/A
.84
.78
.90
N/A
.62
1.00
.76
.63
.18
N/A
.36
N/A
.23
.33
.75
0
65
1
88
19
10
1999 NWS WR
Average
.83
.36
.58
2001 Red Flag Warning Verification
NWS Office
POD
FAR
CSI
# Wrngs
Seattle
Spokane
Portland
Pendleton
Medford
Boise
N/A
N/A
N/A
N/A
N/A
N/A
1.00
.52
.21
.30
.56
.29
N/A
N/A
N/A
N/A
N/A
N/A
3
33
14
27
16
7
1999 NWS WR
Average
.83
.36
.58
2002 Red Flag Warning Verification
NWS Office
POD
FAR
CSI
# Wrngs
Seattle
Spokane
Portland
Pendleton
Medford
Boise
1.00
.75
.50
.67
.54
.85
.80
.62
.55
.36
.42
.15
.20
.33
.31
.48
.39
.73
5
56
22
50
54
13
1999 NWS WR
Average
.83
.36
.58
D:\626080504.doc
2003 Red Flag Warning Verification
NWS Office
POD
FAR
CSI
# Wrngs
Seattle
Spokane
Portland
Pendleton
Medford
Boise
1.00
.78
.44
.88
.45
.67
.65
.56
.87
.46
.57
.45
.35
.39
.11
.50
.29
.43
20
41
30
28
23
11
Note: Figures in red indicate less accuracy than 1999 NWS Western Region average
1999 NWS WR
Average
.83
.36
.58
POD = Probability of Detection which is the percentage of Red Flag events correctly forecast.
The best possible score is 1.00 and the worst .00
FAR = False Alarm Rate which is the percentage of forecasts which were incorrect. The more
often a Red Flag event is forecast and does not occur, the worse the score. The best possible
score is .00 and the worst 1.00
CSI = Critical Success Index which is the percentage of correct Red Flag Warnings to the
number of events plus the number of incorrect forecasts. The best possible score is 1.00 and the
worst .00
D:\626080504.doc
2003 False Alarm Rate (FAR)
1
0.9
0.8
Percentage
0.7
0.6
0.5
0.4
1999 NWS Western Region Average
0.3
0.2
0.1
0
Highest Accuracy
SEA
PDX
MFR
GEG
PDT
BOI
NWS Office
FAR by Office
2003 Probability of Detection (POD)
1.00
Highest Accuracy
0.90
1999 NWS Western Region Average
0.80
Percentage
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
SEA
PDX
MFR
GEG
NWS Office
POD by Office
D:\626080504.doc
PDT
BOI
2003 Critical Success Index (CSI)
1
Highest Accuracy
0.9
0.8
Percentage
0.7
0.6
1999 NWS Western Region Average
0.5
0.4
0.3
0.2
0.1
0
SEA
PDX
MFR
GEG
NWS Office
CSI by Office
D:\626080504.doc
PDT
BOI