An Evaluation of Red Flag Warnings
and Numerical Trend Forecasts
in the Northwest Geographic Area
during the 2003 Fire Season
B and B Complex, August 21, 2003
Prepared by:
The Fire Weather Working Team
Pacific Northwest Wildfire Coordinating Group
This page intentionally left blank.
-2-
Executive Summary
The Memorandum of Understanding (MOU) between the Pacific Northwest Wildfire
Coordinating Group (PNECG) and the National Weather Service (NWS) establishes the
framework for cooperation and commitment by these parties to improve fire weather in the
Northwest and to document standards for forecasts and services. The National Weather Service
has agreed to maintain staffing and services for the fire weather program until it has the
advanced operational technology in place and demonstrates its ability to provide fire weather
services equal to those prior to the implementation of Modernization and Restructuring (MAR).
Forecast accuracy standards set in place within the MOU were determined from those achieved
by the NWS in the 1980s and 90s, prior to MAR.
Evaluation and verification of fire weather products must:
 be based upon a quantifiable process
 be consistent from year to year
 be viewed with respect to trends over a number of years, and
 identify solutions to improve accuracy.
Evaluation’s ultimate goal is to improve the quality and accuracy of fire weather products and
services used in the Pacific Northwest that affect fire management decisions with respect to
resource allocation, firefighter and public safety.
The following document is an evaluation of Red Flag Warnings and Numerical Trend Forecasts
issued by the six NWS offices (Seattle, Spokane, Portland, Pendleton, Medford and Boise)
within the Northwest Geographic Area during the 2003 fire season with comparisons to previous
years. The evaluation was performed in accordance with Section V, paragraph five of the MOU
between PNWCG as defined in EXHIBIT C, Forecast and Service Standards. The initial results
of the evaluation were presented to the NWS in November, 2003 with modifications for the final
report based upon input provided by individual NWS offices.
Red Flag Warnings were verified using the published criteria in the NWS Annual Operating Plan
(AOP) for 2003, lightning data, Remote Automatic Weather Station (RAWS) data, National Fire
Danger Rating (NFDRS) indices, fuel moisture values, and fire information. Red Flag Warnings
are issued by the NWS to notify wildland fire agencies of weather conditions that, in conjunction
with critically dry fuels, could lead to a dramatic increase in fire danger or wildfire activity.
Timely and accurate warnings enable wildland fire fighting agencies to manage critical resources
and prepare appropriate suppression responses for protecting life and property. The combination
of critical weather events (i.e. strong wind, low relative humidity and/or dry lightning) and low
fuel moisture is defined as a “Red Flag” event. Fire Weather Watches are used to alert land
management agencies in advance of possible Red Flag events.
Numerical Trend Forecasts were evaluated by comparing observed and forecast weather data
archived in the Weather Information Management System (WIMS). NWS forecasts are
compared against the persistence error for particular weather stations. The persistence error
refers to the numeric difference (or the change) from one day to the next for a particular weather
element at approximately 1400 LDT. For example, if the observed temperature is 75 degrees F
one day and 78 degrees F the next, the persistence error is 3 degrees F. The elements verified are
temperature, relative humidity, and wind speed. Changes in wind speed and relative humidity
have a much greater impact on fire behavior and fire danger than temperature.
-3-
In summary, the evaluation produced the following results:










In general, Red Flag Warnings issued in the Pacific Northwest failed to match the
accuracy results published by the NWS Western Region in 1999. The 1999 results are
used as benchmark values because this was the last year of dedicated fire weather
forecasters and prior to full compliance of NWS’s Modernization and Restructuring
(MAR).
Spokane, Seattle, Medford and Portland had False Alarm Rates (FAR) greater than .50,
which means the majority of warnings issued by those offices did not verify. These
offices had a tendency to over-warn for the situation.
The majority of actual Red Flag events were forecast, however the probability that these
events are correctly forecast has worsened since 2000.
Procedural errors in issuing and canceling Red Flag Warnings were less frequent this
year, but still continue at a few offices.
No problems were noted in the coordination of Red Flag Warnings among NWS offices
this year. This was due to the creation of “seamless” red flag criteria among adjacent
NWS offices and daily conference calls with NWS and NWCC meteorologists.
Portland was the only office to achieve the 60% goal of warnings preceded by a watch
established in the PNWCG/NWS MOU. The other five NWS offices were all under 40%
as most warnings were issued with little or no advance notification.
In general, Numerical Trend Forecasts do not achieve the accuracy standards established
in the MOU, those attained by the NWS in the 1980s and 1990s. The one exception is
Pendleton’s temperature forecasts which exceed the standard.
Temperature is the weather element most accurately forecast by the NWS, followed by
relative humidity and wind speed.
Boise is the only NWS office to produce wind speed forecasts better than persistence.
Pendleton is the only NWS office to have shown consistent improvement in their
temperature, relative humidity and wind speed forecasts over the last four years.
A concerted effort should be made by the NWS to ensure the issuance of timely and accurate
Red Flag Warnings that enhance fire management decisions and safety concerns for both the
public and firefighters. Weather conditions must be monitored (i.e. radar, RAWS observations)
and updated forecasts and warnings issued when necessary. A quantifiable measure of fuel
dryness needs to be determined by the fire agencies to assist the NWS when conditions are
favorable for large fires. Numerical Trend Forecasts, used in narrative fire weather and fire
danger forecasts, could be improved by using forecast guidance that has been developed by
Predictive Services using RAWS observations. The Fire Weather Working Team is willing to
assist the NWS with any endeavor to improve Red Flag Warning and Numerical Trend forecast
accuracy.
-4-
The Evaluation of 2003 Red Flag Warnings
and Numerical Trend Forecasts
Red Flag Warning Evaluation:
Red Flag Warnings are issued by the National Weather Service (NWS) to notify wildland fire
agencies of weather conditions that, in conjunction with critically dry fuels, could lead to a
dramatic increase in fire danger or wildfire activity. Timely and accurate warnings enable
wildland fire fighting agencies to manage critical resources and prepare appropriate suppression
responses for protecting life and property. The combination of critical weather events (i.e. strong
wind, low relative humidity and/or dry lightning) and low fuel moisture is defined as a “Red
Flag” event. Fire Weather Watches are used to alert land management agencies in advance of
possible Red Flag events.
The following details the evaluation of Red Flag Warnings issued by the six NWS offices
(Seattle, Spokane, Portland, Pendleton, Medford and Boise) within the Northwest Geographic
Area during the 2003 fire season. The evaluation was performed in accordance with the
Memorandum of Understanding (EXHIBIT C Forecast and Service Standards, Section E)
between the Pacific Northwest Wildfire Coordinating Group (PNWCG) and the NWS. Red Flag
Warnings were verified using the published criteria in the NWS Annual Operating Plan (AOP)
for 2003, lightning data, Remote Automatic Weather Station (RAWS) data, National Fire Danger
Rating (NFDRS) indices, fuel moisture values, and fire information. An examination of these
Red Flag Warnings follows. (See Appendix A for specifics)
Findings
1. False Alarm Rate (FAR)
The False Alarm Rate (FAR) is the percentage of Red Flag warnings that do not verify.
The FAR can vary from 1.00 (no warnings correct) to .00 (all warnings correct). The
more often events are forecast and do not occur, the higher (worse) the score. Thus, the
FAR is a measure of Red Flag Warning accuracy. The closer the FAR is to .00, the more
accurate the warnings. The False Alarm Rate for all warnings varied between .45 to .87,
depending upon the office. Spokane, Seattle, Medford and Portland had False Alarm
Rates greater than .50, which means the majority of warnings issued by those offices did
not verify. None of the NWS offices matched the FAR of .36 published as the NWS
Western Region average in 1999.
Listed below are the dry lightning, wind/low RH and all warning False Alarm Rates for
the six NWS offices.
Office
Spokane
Seattle
Portland
Pendleton
Medford
Boise
FAR
FAR
FAR
Dry lightning Wind/Low RH All Warnings
.44
.68
.95
.36
1.00
.50
.65
.50
.72
.57
.29
.33
.56
.65
.87
.46
.57
.45
-5-
2. Probability of Detection (POD)
The Probability of Detection (POD) is the percentage of actual Red Flag events that are
correctly forecast. The more often events are accurately forecast, the better the score.
The POD can vary from 1.00 (all Red Flag events are correctly forecast) to .00 (all Red
Flag events are not forecast). The Probability of Detection ranged from 1.00 at Seattle to
.44 at Portland and .45 at Medford. Except for Portland and Medford, the majority of
Red Flag events were correctly forecast.
Listed below are the dry lightning, wind/low RH and all warning Probability of Detection
values for the six NWS offices.
Office
Spokane
Seattle
Portland
Pendleton
Medford
Boise
POD
POD
POD
Dry lightning Wind/Low RH All Warnings
.91
1.00
1.00
1.00
N/A
1.00
.67
1.00
.38
.75
.45
.40
.56
1.00
.44
.88
.45
.67
3. Critical Success Index (CSI)
The Critical Success Index (CSI) is a combination of FAR and POD. It is the ratio of
correct forecasts to the number of actual Red Flag events plus the number of incorrect
forecasts. The best score is 1.00, the worst is 0. Critical Success Index values ranged
from .50 at Pendleton to .11 at Portland. None of the offices matched or exceeded the
1999 NWS Western Region CSI average of .58.
Listed below are the dry lightning, wind/low RH and all warning Critical Success Index
values for the six NWS offices.
Office
Spokane
Seattle
Portland
Pendleton
Medford
Boise
CSI
CSI
CSI
Dry lightning Wind/Low RH All Warnings
.53
.33
.05
.64
.00
.50
.30
.50
.19
.38
.38
.33
.39
.35
.11
.50
.29
.43
-6-
4. Number of Red Flag Warnings
There were 153 Red Flag Warnings issued in the Northwest Geographic Area in 2003.
This compares to 214 in 2002, 100 in 2001 and 183 in 2000. This is fewer than last year
due to much less fire activity in southern Oregon this summer. The number of warnings
issued by office are as follows: Medford 23, Spokane 41, Pendleton 28, Portland 30,
Boise 11 and Seattle 20.
5. Coordination of Red Flag warnings/criteria among National Weather Service offices
No problems were noted with the coordination of Red Flag Warnings this year. This was
due to the daily telephone conference call between the Geographic Area Coordination
Center and NWS meteorologists, and the creation of “seamless” red flag criteria among
adjacent Weather Service offices with similar fire weather patterns, fuels and topography.
6. Missed Red Flag events
An evaluation of “missed” Red Flag events compared each office’s published Red Flag
criteria to hourly RAWS data. The Predictive Services Branch of the Northwest Area
Coordination Center daily archives hourly observations from approximately 200 RAWS.
A data base query for each weather station produced a list of the hours during the day in
which the criteria was either met or exceeded. Isolated occurrences were discarded. Only
those instances with multiple hours and stations were counted as “missed” warnings. The
vast majority of missed warnings were due to strong wind and low relative humidity, and
not dry lightning. There were a total of 27 missed warnings, most of which occurred in
eastern Oregon and north-central Washington.
7. Procedural errors in the issuance of Red Flag Warnings
Procedural errors are those in conflict with National Weather Service (National Weather
Service Instruction 10-401) or NFDRS directives. There were relatively few errors noted
this year compared to past fire seasons. Most of the errors involved format, improper
cancellation, or inconsistent statements between the general forecast (FWF) and the
warning statement (RFW).
8. Percentage of Red Flag Warnings preceded by a Fire Weather Watch
The PNWCG and NWS Memorandum of Understanding (MOU) in Exhibit C states that,
“at least 60% of all Red Flag Warnings will be preceded by a Fire Weather Watch.” This
is to ensure that there is sufficient advance notification of Red Flag events to properly
prepare firefighting resources. Of the six NWS offices, only Portland achieved the 60%
goal. The following is the percentage of time watches preceded warnings by office:
Portland 70%, Pendleton 36%, Medford 39%, Boise 27%, Seattle 5%, and Spokane 34%.
-7-
Fire Weather Numerical Trend Forecast Evaluation
An evaluation of Numerical Trend Forecasts produced by six National Weather Service (NWS)
offices as input into the National Fire Danger Rating System (NFDRS) were evaluated at the
conclusion of the 2003 fire season by the Predictive Services Section at the Northwest
Interagency Coordination Center. This was the fourth year in which a detailed assessment was
performed in accordance with Section V, paragraph five of the Memorandum of Understanding
between the Pacific Northwest Wildfire Coordinating Group (PNWCG) and the NWS. The
purpose of monitoring and evaluating in the Pacific Northwest is to improve the quality and
accuracy of all fire weather products and services used in fire management decisions that affect
resource allocation and firefighter and public safety.
Trend forecasts are evaluated by comparing observed and forecast weather data archived in the
Weather Information Management System (WIMS). NWS forecasts are compared against the
persistence error for a particular weather station. The persistence error refers to the numeric
difference (or the change) from one day to the next for a particular weather element at
approximately 1400 LDT. For example, if the observed temperature is 75 degrees F one day and
78 degrees F the next, the persistence error is 3 degrees F. The elements verified are
temperature, relative humidity, and wind speed. Changes in wind speed and relative humidity
have a much greater impact on fire behavior and fire danger than temperature.
Since persistence forecasts can be determined without any meteorological input, NWS skill
should result in forecast accuracy that consistently exceeds the persistence error. A total of 101
RAWS stations were evaluated between June 1 and September 30: 23 in the Medford fire
weather district, 22 in Portland, 20 in Pendleton, 17 in Spokane, 13 in Seattle and 6 in the Boise
district.
A spreadsheet was constructed for each RAWS to calculate daily persistence and forecast errors
for temperature, relative humidity and wind speed. Once persistence and forecaster errors were
calculated for each day, the average error for each was calculated over the entire fire season.
These values were then input into the following formula to calculate forecaster improvement
over persistence.
Forecaster improvement = (persistence error – forecaster error)/persistence error * 100
A negative value indicates a forecast worse than persistence. The higher a positive value, the
better the forecast in relation to persistence. Perfect forecasts would result in a 100 percent
forecaster improvement over persistence.
These figures were then entered into a summary spreadsheet that divided the 101 RAWS into
NWS fire weather districts and fire weather zones. Additional information, such as the
beginning and ending dates of NFDRS forecasts and the number of forecasts generated during
the four-month period were also included.
The table below summarizes the improvement over persistence of NWS forecasts during the
2003, 2002, 2001 and 2000 fire seasons. In general, NWS forecasts are still below the accuracy
standards established in the Memorandum of Understanding between the NWS and the Pacific
Northwest firefighting agencies. This is especially evident in the relative humidity and wind
speed forecasts. Except for Boise, the wind speed forecasts are less accurate than persistence, or
what was observed the previous day. (Note: the negative red figures in the wind column indicate
forecast skill less than that of persistence.)
-8-
Appendix B contains the verification results by NWS office and graphics showing trends since
2000 compared to the MOU Accuracy standards.
Office
Seattle
2003
2002
2001
2000
Improvement over Persistence
Temperature
Relative Hum.
Wind
23.39
25.89
23.24
22.42
18.35
20.59
21.17
19.10
-4.15
5.23
-11.18
-18.12
26.25
25.56
20.69
19.43
8.67
14.52
17.89
8.64
-16.20
-6.86
-13.94
-17.59
30.40
27.36
26.71
29.54
23.09
22.45
18.53
18.72
-12.39
-7.94
-11.09
-11.69
36.81
35.12
31.26
28.76
22.13
22.45
20.62
19.06
-1.82
-1.49
-6.94
-15.88
29.77
31.75
28.80
25.75
17.58
18.15
14.92
14.54
-5.51
-11.83
-2.53
-6.29
2003
2002
2001
2000
28.19
29.28
32.25
N/A
18.01
20.51
23.54
N/A
5.37
7.31
0.37
N/A
MOU Fcst
Standards
35.00
25.00
10.00
Spokane
2003
2002
2001
2000
Portland
2003
2002
2001
2000
Pendleton
2003
2002
2001
2000
Medford
2003
2002
2001
2000
Boise
-9-
This page intentionally left blank.
- 10 -
Appendix A
- 11 -
This page intentionally left blank.
- 12 -
Seattle 2003 Red Flag Warnings
Reason
Large
Fire
Potential
Date
Zones
June 29
649, 650, 651,
652, 661, 653,
654, 655, 656,
657, 658, 659
Dry Lightning
High
June 29
662
Wind/Low RH
High
July 30
662
Low RH/Haines 6
High
Sept 3
651, 654, 658,
655, 657, 659
Dry Lightning
High
Total Warnings: 20
Dry Lightning: 18
Correct warnings: 7
Incorrect warnings: 13
Warnings Preceded with a Watch: 1 of 20 or 5%
Verification
Yes – 652, 653, 654, 656, 658, 659
No – 649,650,651,661,655,657 (no lightning
observed)
Not preceded with a watch.
No (criteria not met)
Not preceded with a watch.
Yes
Not preceded with a watch.
No – 651, 655, 657, 659, 654, 658 (no observed
lightning strikes) (canceled 0730 am Sept 4)
Not preceded with a watch except for zone 658.
Wind/low RH: 2
Missed warnings: 0
False Alarm Rate:
Dry Lightning . 67
Wind/low RH . 50
Probability of Detection:
Dry Lightning 1. 00 Wind/low RH 1.00 All 1.00
Critical Success Index:
Dry Lightning . 33
Wind/low RH .50
All . 65
All .35
Note:
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (7/7 + 13) = 1 – (7/20) = 1 - .35 = . 65
Dry Ltng
1 – (a/a + b) = 1 – ( 6/6 + 12) = 1 – (6/18) = 1 - .33 = . 67
Wind/Low RH 1 – (a/a + b) = 1 – (1/1 +1) = 1 – (1/2) = 1 - .50 = . 50
Probability of Detection: All
a/a + c = 7/7 + 1 = 7/7= 1.00
Dry Ltng
a/a + c = 6/6 + 0 = 1/1 = 1.00
Wind/Low RH a/a + c = 1/1 + 1 = 1/1 = 1.00
Critical Success Index:
All
a/a + b + c = 7/7 + 13 = 7/20 = .35
Dry Ltng
a/a + b + c = 6/6 + 12 + 0 = 6/18 = . 33
Wind/Low RH a/a + b + c = 1/1 + 1 = 1/2 = .50
- 13 -
Spokane 2003 Red Flag Warnings
Date
Zones
Reason
Large
Fire
Potential
June 29
673, 676, 677,
684
Wind/Low RH
High
July 16
684, 685
Wind/Low RH
High
July 23
673, 676, 677,
680, 682, 684,
685, 686, 687
Wind/Low RH
High
Low RH/Haines
High
Dry lightning
High
July 30
Aug 5
676, 677, 680,
682, 684, 685,
687
676, 677, 680,
682, 684, 685,
687
Verification
No – 673, 676, 677, 684 (criteria not met)
Preceded with a watch
No – 684, 685 (criteria not met at 2 stations)
Not preceded with a watch
No – 673, 680, 684 (criteria not met at 2
stations)
No – 676, 677, 682, 685, 686, 687 (criteria not
met)
Preceded with a watch
Yes – 676, 677, 680, 682, 684, 685, 687
Not preceded with a watch
Yes – 676, 677, 682, 684, 685
No – 687 (no lightning) 680 (wet lightning)
Not preceded with a watch
Yes – 686
Preceded with a watch
Yes – 673, 684, 685, 686, 687
No – 676, 677, 680, 682 (no lightning)
Not preceded with a watch
No – 686, 687 (no lightning)
Not preceded with a watch
Missed 684
Aug 15
686
Low RH/Haines
High
Aug 15
673,676, 677,
680, 682, 684,
685, 686, 687
Dry lightning/Low
RH/ Haines
High
Sept 6
686, 687
Dry lightning/Wind
High
July 12
684
Wind/Low RH
High
Missed 684 (3 stations multiple hours)
Aug 26
684
Wind/Low RH
High
Missed 684 (2 stations multiple hours)
Sept 1
684
Wind/Low RH
High
Missed 684 (2 stations multiple hours)
Sept 26
684
Wind/Low RH
High
Missed 684 (2 stations multiple hours)
Total Warnings: 41
Dry Lightning: 18
Correct Warnings: 18
Incorrect Warnings: 23
Warnings Preceded with a Watch: 14 or 34%
Wind/low RH/Haines: 23
Missed Warnings: 5
False Alarm Rate:
Dry Lightning . 44
Wind/low RH/Haines . 65
Probability of Detection:
Dry Lightning . 91
Wind/low RH/Haines . 67 All . 78
Critical Success Index:
Dry Lightning . 53
Wind/low RH/Haines . 30
Note:
All . 56
All . 39
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
- 14 -
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (18/18 + 23) = 1 – (18/41) = 1 - .44 = . 56
Dry Ltng
1 – (a/a + b) = 1 – ( 10/10 + 8) = 1 – (10/18) = 1 - .56 = . 44
Wind/Low RH 1 – (a/a + b) = 1 – (8/8 +15) = 1 – (8/23) = 1 - .35 = . 65
Probability of Detection: All
a/a + c = 18/18 + 5 = 18/23 = . 78
Dry Ltng
a/a + c = 10/10 + 1 = 10/11 = . 91
Wind/Low RH a/a + c = 8/8 + 4 = 8/12 = . 67
Critical Success Index:
All
a/a + b + c = 18/18 + 23 + 5 = 18/46 = . 39
Dry Ltng
a/a + b + c = 10/10 + 8 + 1 = 10/19 = . 53
Wind/Low RH a/a + b + c = 8/8 + 15 + 4 = 8/27 = . 30
- 15 -
Portland 2003 Red Flag Warnings
Date
Zones
Reason
Large
Fire
Potential
July 21
607, 608, 609,
610, 611
Dry Lightning
High
July 22
607, 608, 609,
610, 611
Dry Lightning/Low
RH/Gusty
Thunderstorm wind
High
Dry Lightning/
Low RH/Haines
High
Wind/Low RH
Mod - W2
High - W3
Sept 3-4
Sept 26-27
603, 604, 605,
606, 607, 608,
609, 610, 611,
660
601, 602, 603,
604, 605, 606,
607, 608 612,
660
Verification
Yes – 611
No – 607, 608, 609, 610 (no lightning
observed) Preceded with a watch in 608, 611
Yes – 610 for wind and low RH
No – 607, 608 609, 611 (no lightning observed)
Not preceded with a watch
No – (no lightning observed)
Canceled 0930 am Sept 4
Preceded with a watch in all zones but 660
Yes - 602, 660
No – 601, 603, 604, 605, 606, 607, 608, 612
Preceded with a watch all zones
June 27-29
610
Wind/Low RH
High
Missed (2 stations met criteria)
June 29
611
Wind/Low RH
High
Missed (3 stations met criteria)
July 22-24
610
Wind/Low RH
High
Missed (3 stations met criteria)
July 30
611
Wind/Low RH
High
Missed (2 stations met criteria)
Sept 1
610
Wind/Low RH
High
Missed (2 stations met criteria)
Total Warnings: 30
Dry Lightning: 20
Correct Warnings: 4
Incorrect Warnings: 26
Warnings Preceded with a Watch: 21 or 70%
Wind/low RH/Haines: 10
Missed Warnings: 5
False Alarm Rate:
Dry Lightning . 95
Wind/low RH . 72
All . 87
Probability of Detection:
Dry Lightning 1. 00 Wind/low RH . 38
All . 44
Critical Success Index:
Dry Lightning . 05
All . 11
Note:
Wind/low RH . 19
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
- 16 -
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (4/4 + 26) = 1 – (4/30) = 1 - .13 = . 87
Dry Ltng
1 – (a/a + b) = 1 – ( 1/1 + 18) = 1 – (1/19) = 1 - .05 = . 95
Wind/Low RH 1 – (a/a + b) = 1 – (3/3 +8) = 1 – (3/11) = 1 - .27 = . 72
Probability of Detection: All
a/a + c = 4/4 + 5 = 4/9 = . 44
Dry Ltng
a/a + c = 1/1 + 0 = 1/1 = 1. 00
Wind/Low RH a/a + c = 3/3 + 5 = 3/8 = . 38
Critical Success Index:
All
a/a + b + c = 4/4 + 26 + 5 = 4/35 = . 11
Dry Ltng
a/a + b + c = 1/1 + 18 + 0 = 1/19 = . 05
Wind/Low RH a/a + b + c = 3/3 + 8 + 5 = 3/16 = . 19
- 17 -
Pendleton 2003 Red Flag Warnings
Date
Zones
Reason
Large
Fire
Potential
June 29
631, 675
Wind/Low RH
High
July 7
630,632, 633,
634, 635, 638
Lightning after an
extended dry
period
High
July 22-23
630, 632, 633,
634, 635, 638
Lightning after a
dry period
High
July 30
630, 631, 632,
681
Low RH/Haines 6
High
July 31
630, 631, 632,
638
Wind/Low RH
High
Aug 10
638
Wind/Low
RH/Haines
High
Aug 15-16
630, 631,675
Wind/Low RH
High
Sept 3-4
630, 632
Dry Lightning
High
July 12
631
Wind/Low RH
High
July 23
631
Wind/Low RH
High
Total Warnings: 28
Dry Lightning: 14
Correct warnings: 15
Incorrect warnings: 13
Warnings Preceded with a Watch: 10 or 36%
Verification
No – 675, 631 (only 1 station met the criteria)
Preceded with a watch.
Yes – 630, 632, 633, 634, 635, 638 (canceled
1008 am July 8)
Not preceded with a watch.
Yes- 630, 632, 638
No – 633, 634, 635, (no lightning observed)
(canceled 1057 pm July 23)
Preceded with a watch in 633, 634 and 635 but
not in 630, 632 and 638
Yes – 630, 631, 632, 681 (wind criteria also
met in 630 and 631)
Not preceded with a watch.
Yes – 631
No – 630, 632, 638 (criteria not met)
Preceded with a watch.
Yes – 638
Not preceded with a watch.
No – 630, 675, 631 (only 1 station met the
criteria) Expired 1100 am Aug 16
Not preceded with a watch.
No – 630, 632 (no lightning observed)
(canceled 0940 am Sept 4)
Preceded with a watch in 630 but not in 632.
Missed (criteria met by 1 station for 3 hours
and a second station for 6 hours)
Missed (criteria met by 1 station for 4 hours
and a second station for 7 hours)
Wind/low RH/Haines: 14
Missed warnings: 2
False Alarm Rate:
Dry Lightning . 36
Wind/low RH . 57
All . 46
Probability of Detection:
Dry Lightning 1. 00 Wind/low RH .75
All . 88
Critical Success Index:
Dry Lightning . 64
All . 50
Note:
Wind/low RH . 38
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
- 18 -
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (15/15 + 13) = 1 – (15/28) = 1 - .54 = . 46
Dry Ltng
1 – (a/a + b) = 1 – (9/9 + 5) = 1 – (9/14) = 1 - .64 = . 36
Wind/Low RH 1 – (a/a + b) = 1 – (6/6 +8) = 1 – (6/14) = 1 - .43 = . 57
Probability of Detection: All
a/a + c = 15/15 + 2 = 15/17 = . 88
Dry Ltng
a/a + c = 9/9 + 0 = 6/6 = 1. 00
Wind/Low RH a/a + c = 6/6 + 2 = 6/8 = . 75
Critical Success Index:
All
a/a + b + c = 15/15 + 13 + 2 = 15/30 = . 50
Dry Ltng
a/a + b + c = 9/9 + 5 + 0 = 9/14 = . 64
Wind/Low RH a/a + b + c = 6/6 + 8 + 2 = 6/16 = . 38
- 19 -
Medford 2003 Red Flag Warnings
Date
Zones
Reason
Large
Fire
Potential
July 30
617
HI 6
High
Sept 3
615, 616, 617,
618, 619, 620,
621, 622, 623
Dry Lightning
High
Sept 12-14
618, 619
Wind/Low RH
Low
Sept 21-23
618, 619
Wind/Low RH
High
Sept 25
618
Wind/Low RH
High
Sept 26-27
620, 621, 623
Wind/Low RH
High
Oct 24-25
618, 619, 620,
621, 622
Wind/Low RH
N/A
June 29
617, 622, 623,
624, 625
Wind/Low RH
High
Missed (criteria met at multiple stations)
Aug 9
625
Wind/Low RH
High
Missed (criteria met)
Sept 2
620, 622
Wind/Low RH
High
Missed (criteria met and 3 stations)
Sept 6
624, 625
Wind/Low RH
High
Missed (criteria met)
Total Warnings: 23
Dry Lightning: 9
Correct Warnings: 10
Incorrect Warnings: 13
Warnings Preceded with a Watch: 9 or 39%
Verification
Yes – 617
Not preceded with a watch
No - (no lightning observed until after the
warning was canceled at 0845 am Sept 4)
All but 618, 620, 621, 622, 623 preceded with a
watch
No – Fuel moisture high due to recent rains of
up to 2 inches
Preceded with a watch
Yes – 618, 619
Not preceded with a watch
Missed 623
Yes – 618
Not preceded with a watch
Yes – 620, 623 (no lead time)
No – 621
Not preceded with a watch
Yes – 618, 619, 620, 622
No – 621
Zones 618, 619, 620 preceded with a watch
Missed 623
Wind/low RH/HI: 14
Missed Warnings: 12
False Alarm Rate:
Dry Lightning 1.00 Wind/low RH . 29
All . 57
Probability of Detection:
Dry Lightning N/A Wind/low RH .45
All . 45
Critical Success Index:
Dry Lightning . 00
All . 29
Note:
Wind/low RH . 38
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
- 20 -
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (10/10 + 13) = 1 – (10/23) = 1 - .43 = . 57
Dry Ltng
1 – (a/a + b) = 1 – (0/0 + 9) = 1 – (0/9) = 1 - 0 = 1. 00
Wind/Low RH 1 – (a/a + b) = 1 – (10/10 +4) = 1 – (10/14) = 1 - .71 = . 29
Probability of Detection: All
a/a + c = 10/10 + 12 = 10/22 = . 45
Dry Ltng
a/a + c = 0/0 + 0 = 0/0 = N/A
Wind/Low RH a/a + c = 10/10 + 12 = 10/22 = . 45
Critical Success Index:
All
a/a + b + c = 10/10 + 13 + 12 = 10/35 = . 29
Dry Ltng
a/a + b + c = 0/0 + 9 + 0 = 0/9 = . 00
Wind/Low RH a/a + b + c = 10/10 + 4 + 12 = 10/26 = . 38
- 21 -
Boise 2003 Red Flag Warnings
Date
Zones
Reason
Large
Fire
Potential
June 18
637
Dry lightning/Wind
High
June 30 –
July 1
637
Wind/Low RH
High
July 7
636, 637
Dry lightning/Wind
High
July 22
636
Lightning after a dry
period
High
July 23
636, 637
Dry lightning
High
Aug 10
636, 637
Wind/Low
RH/Haines
High
Aug 26
636, 637
Dry lightning/Wind
High
June 29
636, 637
Wind/Low RH
High
Aug 9
636
Wind/Low RH
High
Missed 636 (criteria met)
Aug 15
636
Wind/Low RH
High
Missed 636 (criteria met)
Sept 6
636
Wind/Low RH
High
Missed 636 (criteria met)
Total Warnings: 11
Dry Lightning: 8
Correct warnings: 6
Incorrect warnings: 5
Warnings Preceded with a Watch: 3 of 11 or 27%
Verification
Yes
Not preceded with a watch.
No – 637 (criteria not met)
Not preceded with a watch.
Yes – 637
No – 636 (only isolated strikes)
Preceded with a watch.
No
Not preceded with a watch.
Yes
Zone 636 preceded with a watch.
Yes
Not preceded with a watch.
No (no lightning observed)
Not preceded with a watch.
Missed 636, 637 (multiple stations, multiple
hours) (local users did not want a warning
issued)
Wind/low RH/Haines: 3
Missed warnings: 3
False Alarm Rate:
Dry Lightning . 50
Wind/low RH . 33
All . 45
Probability of Detection:
Dry Lightning 1. 00 Wind/low RH . 40
All . 67
Critical Success Index:
Dry Lightning . 50
All . 43
Note:
Wind/low RH . 33
For highest accuracy, False Alarm Rate should approach .00 and Critical Success
Index and Probability of Detection 1.00
- 22 -
Calculations:
a = correct warnings
b = incorrect warnings
c = missed warnings
False Alarm Rate:
All
1 – (a/a + b) = 1 – (6/6 + 5) = 1 – (6/11) = 1 - .55 = . 45
Dry Ltng
1 – (a/a + b) = 1 – ( 4/4 + 4) = 1 – (4/8) = 1 - .50 = . 50
Wind/Low RH 1 – (a/a + b) = 1 – (2/2 +1) = 1 – (2/3) = 1 - .67 = . 33
Probability of Detection: All
a/a + c = 6/6 + 3 = 6/9 = . 67
Dry Ltng
a/a + c = 4/4 + 0 = 1/1 = 1. 00
Wind/Low RH a/a + c = 2/2 + 3 = 2/5 = . 40
Critical Success Index:
All
a/a + b + c = 6/6 + 5 + 3 = 6/14 = . 43
Dry Ltng
a/a + b + c = 4/4 + 4 + 0 = 4/8 = . 50
Wind/Low RH a/a + b + c = 2/2 + 1 + 3 = 2/6 = . 33
- 23 -
2000 Red Flag Warning Verification
NWS Office
POD
FAR
CSI
# Wrngs
Seattle
Spokane
Portland
Pendleton
Medford
Boise
N/A
.86
N/A
.84
.78
.90
N/A
.62
1.00
.76
.63
.18
N/A
.36
N/A
.23
.33
.75
0
65
1
88
19
10
1999 NWS WR
Average
.83
.36
.58
2001 Red Flag Warning Verification
NWS Office
POD
FAR
CSI
# Wrngs
Seattle
Spokane
Portland
Pendleton
Medford
Boise
N/A
N/A
N/A
N/A
N/A
N/A
1.00
.52
.21
.30
.56
.29
N/A
N/A
N/A
N/A
N/A
N/A
3
33
14
27
16
7
1999 NWS WR
Average
.83
.36
.58
2002 Red Flag Warning Verification
NWS Office
POD
FAR
CSI
# Wrngs
Seattle
Spokane
Portland
Pendleton
Medford
Boise
1.00
.75
.50
.67
.54
.85
.80
.62
.55
.36
.42
.15
.20
.33
.31
.48
.39
.73
5
56
22
50
54
13
1999 NWS WR
Average
.83
.36
.58
- 24 -
2003 Red Flag Warning Verification
NWS Office
POD
FAR
CSI
# Wrngs
Seattle
Spokane
Portland
Pendleton
Medford
Boise
1.00
.78
.44
.88
.45
.67
.65
.56
.87
.46
.57
.45
.35
.39
.11
.50
.29
.43
20
41
30
28
23
11
Note: Figures in red indicate less accuracy than 1999 NWS Western Region average
1999 NWS WR
Average
.83
.36
.58
POD = Probability of Detection which is the percentage of Red Flag events correctly forecast.
The best possible score is 1.00 and the worst .00
FAR = False Alarm Rate which is the percentage of forecasts which were incorrect. The more
often a Red Flag event is forecast and does not occur, the worse the score. The best possible
score is .00 and the worst 1.00
CSI = Critical Success Index which is the percentage of correct Red Flag Warnings to the
number of events plus the number of incorrect forecasts. The best possible score is 1.00 and the
worst .00
- 25 -
2003 False Alarm Rate (FAR)
1
0.9
0.8
Percentage
0.7
0.6
0.5
0.4
1999 NWS Western Region Average
0.3
0.2
0.1
0
Highest Accuracy
SEA
PDX
MFR
GEG
PDT
BOI
NWS Office
FAR by Office
2003 Probability of Detection (POD)
1.00
Highest Accuracy
0.90
1999 NWS Western Region Average
0.80
Percentage
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
SEA
PDX
MFR
GEG
NWS Office
POD by Office
- 26 -
PDT
BOI
2003 Critical Success Index (CSI)
1
Highest Accuracy
0.9
0.8
Percentage
0.7
0.6
1999 NWS Western Region Average
0.5
0.4
0.3
0.2
0.1
0
SEA
PDX
MFR
GEG
NWS Office
CSI by Office
- 27 -
PDT
BOI
This page intentionally left blank.
- 28 -
Appendix B
- 29 -
This page intentionally left blank.
- 30 -
2003 Forecaster Improvement over Persistence (NFDRS weather trends)
June 1 through Sept 30, 2003
(Forecaster Improvement over
Persistence)
Relative
Temperature
Hum.
Wind
Portland NWS
Zone Sta. #
Fcst
Dates
Number
Days
Fcst
%
Days
Fcst
601
350215
Cedar Creek
27.57
21.43
-1.90
6/06-9/30
89
72
602
350216
South Fork
28.68
12.48
7.14
8/07-9/30
46
37
451209
Abernathy Mtn
29.18
22.00
-35.45
6/05-9/30
98
80
603
352547
Village Creek
25.14
21.66
-1.63
6/05-9/30
107
87
351710
Rockhouse
21.80
21.88
3.63
6/06-9/30
59
48
606
352024
Yellowstone
39.37
34.82
-1.67
6/05-9/30
64
52
352552
Trout Creek
37.89
30.61
-17.47
6/05-9/30
104
85
607
350604
Log Creek
29.07
25.15
-36.17
6/07-9/30
95
77
350718
Red Box
42.22
29.63
-37.17
6/07-9/30
101
82
350726
Wanderer's Pk
26.76
34.90
-2.78
8/06-9/30
45
37
608
352554
Pebble
40.40
35.70
-11.76
6/05-9/30
102
83
352558
Emigrant Creek
31.30
20.33
-2.75
6/05-9/30
102
83
609
350912
Polliwog
26.77
14.96
-19.28
6/07-9/30
100
81
350913
Wamic Mill
28.76
5.31
-10.42
6/05-9/30
97
79
610
350917
Mutton Mountain
34.57
31.93
6.91
6/07-9/30
88
72
350920
HeHe #1
31.81
23.89
6.64
6/05-9/30
90
73
352620
Colgate
33.22
21.21
6.67
6/05-9/30
103
84
611
353402
Cabin Lake
24.15
15.04
-24.49
6/06-9/30
108
88
352618
Lava Butte
21.92
20.51
1.22
6/05-9/30
109
89
612
352545
Goodwin Peak
18.14
13.96
-64.62
6/05-9/30
73
59
660
451921
Canyon Creek
32.48
26.31
-11.54
6/06-9/30
89
72
451208
Elk Rock
37.65
24.18
-25.71
6/06-9/30
96
78
AVG
30.40
23.09
-12.39
89
73
MEDIAN
29.13
21.94
-6.60
MAX
42.22
35.70
7.14
MIN
18.14
5.31
-64.62
TOTAL FCSTS
Pendleton NWS
Zone Sta. #
630
631
632
633
1965
(Forecaster Improvement over
Persistence)
Relative
Temperature
Hum.
Wind
Fcst
Dates
Number
Days
Fcst
%
Days
Fcst
352107
Haystack
43.08
19.93
-2.22
6/10-9/30
97
79
352712
Salt Creek
39.48
28.99
3.48
6/11-9/30
97
79
352207
Slide Mountain
42.10
29.70
-29.58
6/10-9/30
102
83
352701
Cold Spring
38.04
26.52
-5.69
6/10-9/30
102
83
453201
Juniper Dunes
32.30
18.62
11.52
6/10-9/30
87
71
351001
Patjens
41.57
24.02
5.94
6/11-9/30
99
81
350915
North Pole Ridge
42.33
23.12
15.18
6/11-9/30
100
81
352329
Case
36.33
13.17
6.29
6/10-9/30
100
81
352327
Fall Mountain
34.46
17.00
-18.26
6/10-9/30
100
81
353424
Antelope
36.75
20.32
6.27
6/11-9/30
105
85
351518
Eden
35.98
22.92
-6.25
6/10-9/30
99
81
- 31 -
453803
Alder
38.16
23.95
4.83
6/10-9/30
98
80
351317
Black Mountain 2
45.26
23.78
-19.74
6/11-9/30
96
78
351417
LaGrande
36.94
19.16
-4.39
6/10-9/30
89
72
634
351416
Minam Lodge
35.74
20.00
-11.47
6/10-9/30
95
77
635
351502
Harle Butte
29.48
20.71
-2.68
6/11-9/30
89
72
351520
Roberts Butte
31.47
24.90
-7.09
6/11-9/30
88
72
638
352416
Blue Canyon
37.48
18.69
4.23
6/10-9/30
95
77
352418
Sparta Butte
34.70
26.50
-6.64
6/11-9/30
95
77
681
452404
Greyback
39.05
23.72
19.63
6/13-9/30
60
49
AVG
37.54
22.29
-1.83
95
77
MEDIAN
36.94
22.92
-2.68
MAX
45.26
29.70
19.63
MIN
29.48
13.17
-29.58
TOTAL FCSTS
(Forecaster Improvement over
Persistence)
Relative
Temperature
Hum.
Wind
Boise NWS
Zone Sta. #
636
637
1893
Fcst
%
Days
Dates
Number
Days
Fcst
12.47
6/11-9/30
91
74
Fcst
353520
Basque Hills
32.51
15.03
353526
Moon Hill
27.98
25.71
-2.47
6/11-9/30
92
75
353512
Wagontire
30.29
13.56
13.86
6/11-9/30
92
75
353612
Grassy Mountain
30.00
22.26
5.48
6/11-9/30
97
79
353613
Kelsey Butte
29.98
23.60
0.67
6/11-9/30
100
81
353614
Owyhee Ridge
6/11-9/30
96
78
95
77
18.36
7.87
2.20
AVG
28.19
18.01
5.37
MEDIAN
29.99
18.65
3.84
MAX
32.51
25.71
13.86
MIN
18.36
7.87
-2.47
TOTAL FCSTS
568
(Forecaster Improvement over
Persistence)
Relative
Temperature
Hum.
Wind
Medford NWS
Zone Sta. #
Fcst
Dates
Number
Days
Fcst
%
Days
Fcst
615
352814
Powers 2
17.85
22.52
-9.09
6/11-9/29
95
77
616
352816
Signal Tree
23.65
18.21
-9.09
6/11-9/29
55
45
617
352546
Sugarloaf
32.06
26.03
-38.00
8/07-9/29
50
41
353031
Cinnamon Butte
32.94
22.3
-0.61
6/11-9/29
100
81
353040
Buckeye
38.27
19.72
-14.58
6/11-9/29
100
81
618
352920
Red Mound
14.44
15.37
-6.92
6/11-9/29
65
53
619
352915
Quail Prairie
25.62
22.51
-7.66
6/11-9/29
93
76
620
352919
Calvert
28.21
-28.22
4.79
6/12-9/29
93
76
353115
Illinois Valley
32.60
18.06
3.79
6/12-9/29
93
76
353114
Onion Mtn
33.75
19.79
4.76
6/11-9/29
94
76
353120
Provolt
16.19
-5.13
-10.23
6/12-9/29
93
76
621
353214
Star R.S.
32.48
12.14
-30.58
6/11-9/29
106
86
622
353228
Evans Creek
39.83
31.89
3.85
6/11-9/29
102
83
353230
Buckhorn
32.11
10.73
-13.97
6/11-9/29
100
81
623
353345
Rover
insufficient observations
353227
Zim
36.18
-15.53
6/11-9/29
106
86
26.89
- 32 -
624
625
353339
Seldom
33.40
23.08
353307
Calimus Butte
31.48
353328
Gerber
33.27
353337
Timothy
35.94
353422
Coffee Pot
32.79
353406
Fort Rock
353424
Rock Creek
-0.67
6/11-9/29
101
82
22.10
9.28
6/11-9/29
103
84
23.34
-3.80
6/11-9/29
106
86
26.72
7.20
6/11-9/29
108
88
22.9
-2.67
6/11-9/29
107
87
38.41
19.86
5.28
6/11-9/29
105
85
36.75
20.32
6.27
6/11-9/29
105
85
AVG
30.83
17.78
-5.37
95
77
MEDIAN
32.70
21.21
-3.24
MAX
39.83
31.89
9.28
MIN
14.44
-28.22
-38.00
TOTAL FCSTS
2080
(Forecaster Improvement over
Persistence)
Relative
Temperature
Hum.
Wind
Seattle NWS
Zone Sta. #
Fcst
Dates
Number
Days
Fcst
%
Days
Fcst
650
450130
Ellis Mtn
12.44
16.14
1.86
6/11-9/30
69
56
651
450306
Minot
18.85
16.21
-7.19
6/11-9/30
76
62
652
450211
Owl Mtn
22.57
25.47
-6.22
6/11-9/30
85
69
658
451415
Sumas
35.82
32.20
-19.01
7/05-9/30
67
55
451613
Gold Mountain
22.61
28.05
Bad Data
6/06-9/30
99
81
659
661
662
451611
Johnson
26.48
12.51
Bad Data
6/10-9/30
92
75
451509
Finney
23.63
26.57
Bad Data
6/06-9/30
99
81
451919
Orr Creek
29.37
11.53
-8.60
6/13-9/30
91
74
451105
Kosmos
21.71
12.89
-2.48
8/08-9/30
71
58
451718
Greenwater
26.54
20.95
-10.83
6/06-9/30
99
81
450117
Cougar
20.56
21.98
-5.59
6/06-9/30
99
81
450911
Jefferson
23.42
19.68
0.41
6/06-9/30
98
80
452121
Stehekin
6/12-9/30
96
78
88
72
20.08
-5.65
16.19
AVG
23.39
18.35
-4.15
MEDIAN
22.61
19.68
-5.91
MAX
35.82
32.20
16.19
MIN
12.44
-5.65
-19.01
TOTAL FCSTS
(Forecaster Improvement over
Persistence)
Relative
Temperature
Hum.
Wind
Spokane NWS
Zone Sta. #
673
677
680
682
684
685
1141
Fcst
%
Days
Dates
Number
Days
Fcst
-3.47
6/12-9/30
59
48
57
46
Fcst
452601
Douglas
453601
Escure
38.70
10.31
10.75
6/12-9/30
452136
Entiat
30.98
3.07
-0.36
6/12-9/30
90
73
452132
Camp Four
27.01
11.05
-19.35
6/12-9/30
91
74
86
70
37.54
19.41
452134
Dry Creek
19.70
8.46
14.78
6/12-9/30
452206
Peoh Point
26.69
17.27
-17.05
6/12-9/30
90
73
59
48
452601
Sedge Ridge
37.54
19.41
-3.47
6/12-9/30
452219
Swuak
30.96
11.25
-64.00
6/12-9/30
91
74
-20.89
6/12-9/30
89
73
6/12-9/30
89
72
452030
NCSB
452040
Kramer
452006
First Butte
25.06
-7.95
insufficient observations and forecasts
18.32
-6.67
- 33 -
-37.75
452036
686
687
83 Monument
16.58
11.46
-42.42
6/12-9/30
90
73
81
66
453412
Deer Mountain
23.79
8.31
-9.66
6/13-9/30
452916
Kettle Falls
17.93
3.67
-9.04
6/13-9/30
77
63
452510
Gold Mountain
452514
Brown Mountain
Bad Data
6/13-9/30
81
66
6/12-9/30
89
72
81
66
452029
insufficient observations and forecasts
Lost Lake
22.88
10.40
20.03
10.66
-24.84
AVG
26.25
8.67
-16.20
MEDIAN
25.06
10.40
-13.36
MAX
38.70
19.41
14.78
MIN
16.58
-7.95
-64.00
TOTAL FCSTS
1219
Forecaster Improvement over Persistence
Temperature
40.00
Percentage of Improvement
MOU Accuracy Standard
35.00
30.00
25.00
20.00
15.00
10.00
2000
Seattle
2001
Spokane
Year
Portland
- 34 -
2002
Pendleton
2003
Medford
Boise
Forecaster Improvement over Persistence
Relative Humidity
Percentage of Improvement
35.00
30.00
MOU Accuracy Standard
25.00
20.00
15.00
10.00
5.00
0.00
2000
2001
Seattle
Spokane
Year
Portland
2002
Pendleton
2003
Medford
Boise
Forecaster Improvement over Persistence
Wind Speed
15.00
Percentage of Improvement
MOU Accuracy Standard
10.00
5.00
0.00
-5.00
-10.00
-15.00
-20.00
2000
Seattle
2001
Spokane
Year
Portland
- 35 -
2002
Pendleton
2003
Medford
Boise