Fleet Weather Center Norfolk
Assessing the value of ensemble guidance for FWC-Norfolk forecasters
Presented to the NUOPC Users Workshop
This briefing is UNCLASSIFIED / FOUO
June 2011
LT Rich Rainer
N5 Department Head
Forecast Team
AGC Brown
AG1 Cummings
AG2 Nelson
AG2 Didier
AG3 Dietrich
AGAN Gagle
Fleet Weather Center - Norfolk
AGENDA
• Objectives
• Challenge
• Description of experiment
• General results
• Data collection
• Objective results
• Subjective findings
• Possible next steps
Fleet Weather Center - Norfolk
2
Objectives
• In an environment where resources are scarce and
billets are being reduced to “optimal manning”
levels
– How do we maintain forecast quality?
– How to we increase forecaster efficiency (reduce the
amount of time necessary to make quality forecasts)?
Fleet Weather Center - Norfolk
3
Challenge
• Fleet Weather Center forecasters have a heavy
workload
– 2 forecasters
– 2 techs
– Forecasts for 18 OPAREAS
– Forecasts for 20-50 ships
– Wind and seas warnings for the
• North Atlantic
• Mediterranean
• Black Sea
• South Atlantic
Fleet Weather Center - Norfolk
4
Description of experiment
• SGOT shadow forecast team
– Two forecasters, two assistant forecasters
– March 15, 2011 through May 15, 2011
– FOXX1 plus North Atlantic high winds and seas
– In addition to traditional guidance products, use NOGAPS
ensemble guidance in MetCast/JMV
• Mean/spread plots
• Probability of exceedance for winds (35kts) and seas (12’ and 18’)
– Compete with official forecasts from Watch Floor
– Develop best practices for use of ensemble guidance
• Chose to use ensemble as primary source of guidance
– Validate using satellite, ship, and buoy observations
Fleet Weather Center - Norfolk
5
OPAREA Forecasts
• FOXX01
– VACAPES
• WEAX Parameters
- Winds and
- Direction
- Speed
- Seas
- Height
Fleet Weather Center - Norfolk
6
Example OPAREA forecast
UNCLASSIFIED
FOXX01 KNGU 041200
MSGID/GENADMIN/FLEWEACEN NORFOLK VA//
SUBJ/VACAPES/CHERRY PT/CHARLESTON/JACKSONVILLE/
PORT CANAVERAL/TONGUE OF THE OCEAN OPAREAS FORECAST//
POC/COMMAND DUTY OFFICER/-/FLEWEACEN NORFOLK/LOC:NORFOLK VA//
TEL:757-444-7750/SECTEL:-0963/EMAIL: CDO.FWC.NRFK.FCT(AT)NAVY.MIL//
RMKS/1.HIGH PRESSURE RIDGES IN FROM THE MIDWEST PRODUCING PARTLY CLOUDY SKIES, MODERATE WINDS AND SEAS THROUGHOUT THE AREA.
EXPECT WINDS AND SEAS TO DECREASE AS THIS HIGH PRESSURE TRANSITS SOUTHEAST TONIGHT THROUGH FRIDAY AFTERNOON.
2. VACAPES OPAREA 24 HOUR FORECAST COMMENCING 050000Z.
SKY, WEATHER: PARTLY CLOUDY.
B. VSBY (NM): 7.
C. SURFACE WIND (KTS):NORTHWEST 20 TO 25 GUST 30, DECREASING 13 TO 18 GUST 23 BY 05/18Z.
D. COMBINED SEAS (FT):COASTAL:NORTHERLY 3 TO 5.
SEAWARD:SOUTHERLY NORTHERLY 5 TO 7.
E. MAX/MIN TEMPS (F): 68/52.
F. SST(F/C): COASTAL: 55/13. VICINITY OF GULF STREAM: 74/22.
G. OUTLOOK TO 48 HOURS:
WINDS: NORTHWEST 13 TO 18 GUST 23, DECREASING 10 TO 15 GUST 20 BY 06/06Z, BACKING WEST-NORTHWEST BY 06/12Z, BECOMING WEST
5 TO 10 BY 06/18Z.
SEAS:COASTAL:NORTHERLY 3 TO 5, ABATING 2 TO 4 BY 06/06Z.
SEAWARD: NORTHERLY 5 TO 7, ABATING 4 TO 6 BY 06/18Z.
3.CHERRY PT OPAREA 24 HOUR FORECAST COMMENCING 050000Z.
A. SKY, WEATHER: PARTLY CLOUDY.
B. VSBY (NM): 7.
C. SURFACE WIND (KTS): NORTHWEST 18 TO 23 GUST 28, DECREASING 15 TO 20 BY 05/06Z, DECREASING 13 TO 18 BY 05/18Z.
COMBINED SEAS (FT):COASTAL: NORTH 2 TO 4.
SEAWARD:NORTH 5 TO 7, ABATING 4 TO 6 BY 05/18Z.
E. MAX/MIN TEMPS (F): 72/54.
F. SST(F/C): COASTAL: 56/13. VICINITY OF GULF STREAM: 72/22.
G. OUTLOOK TO 48 HOURS:
WINDS:NORTHWEST 13 TO 18, DECREASING 8 TO 13 BY 06/06Z, BECOMING WEST 5 TO 10 BY 06/12Z, BACKING SOUTHWEST BY 06/18Z.
SEAS:COASTAL: NORTH 2 TO 4, BECOMING CONFUSED BY 06/12Z.
SEAWARD: NORTH 4 TO 6, BECOMING NORTHWEST 3 TO 5, BY 06/12Z.
4. CHARLESTON OPAREA 24 HOUR FORECAST COMMENCING 050000Z.
A. SKY, WEATHER: PARTLY CLOUDY.
B. VSBY (NM): 7.
C. SURFACE WIND (KTS):NORTHWEST 15 TO 20, BECOMING NORTH 10 TO 15 BY 05/18Z.
Fleet Weather Center - Norfolk
7
OPAREA validation
C. SURFACE WIND (KTS):NORTHWEST 20 TO 25 GUST 30,
DECREASING 13 TO 18 GUST 23 BY 05/18Z.
D. COMBINED SEAS (FT):COASTAL:NORTHERLY 3 TO 5.
• Collect all available observations in OPAREA
• Compute average observation value
• Calculate distance outside of forecast range
• Example: Average observation was 7ft seas,
forecast error was +2ft.
Fleet Weather Center - Norfolk
8
Example wind and seas warning
HIGHLIGHTED CONTOURS:
WINDS >35KTS & >50KTS;
Fleet Weather
- Norfolk
SEASCenter
>12FT
& >18FT
SGOT Shadow in Black
Watch Floor in Yellow
9
General results
• VACAPES forecasts made with the help of
ensemble guidance outperformed official
VACAPES forecasts
• No quantitative assessment of high winds and
seas forecasts (yet), but sense is that they are
comparable and quicker to generate
• Uncertainty information is welcome, but slows
down the forecast process when large uncertainty
is present
– Need for products that clarify the “form” of the
uncertainty and the reason for the uncertainty
Fleet Weather Center - Norfolk
10
Data collection
Fleet Weather Center - Norfolk
11
Objective results (24hr-48hr)
VACAPES wind direction
Bias
RMSE
% outside threshold
Prob. guidance
48.7deg
16.3 (39)
Det. guidance
57.1deg
23.8 (57)
14.8%
31.6%
Percent improvement
VACAPES wind speed
Bias
RMSE
% outside threshold
Prob. guidance
2.3kts
4.6kts
6.9 (16)
Det. guidance
3.1kts
5.4kts
10.8 (25)
25%
13.8%
36%
Percent improvement
VACAPES wave height
Bias
RMSE
% outside threshold
Prob. guidance
-0.02ft
0.57ft
0.42 (1)
Det. guidance
0.25ft
1.06ft
7.59 (18)
Percent improvement
93.3%
46.7%
94.4%
Fleet Weather Center - Norfolk
12
Objective results (48hr-72hr)
VACAPES wind direction
Bias
RMSE
% outside threshold
Prob. guidance
54.4deg
24.1 (54)
Det. guidance
61.2deg
26.3 (59)
11.2%
8.5%
Percent improvement
VACAPES wind speed
Bias
RMSE
% outside threshold
Prob. guidance
2.0kts
4.9kts
8.9 (20)
Det. guidance
2.6kts
4.9kts
9.4 (21)
21%
1%
4.8%
Percent improvement
VACAPES wave height
Bias
RMSE
% outside threshold
-0.004ft
0.64ft
1.5 (3)
Det. guidance
0.33ft
1.5ft
4.6 (14)
Percent improvement
98.7%
48.4%
68.2%
Prob. guidance
Fleet Weather Center - Norfolk
13
24hr wind direction errors
Wind direction, 24hr forecast error (deg)
200
180
160
Direction error (deg)
140
120
100
Shadow
Watch Floor
80
60
40
20
0
Fleet Weather Center - Norfolk
14
Wind direction errors
Wind direction threshold
exceedance (%)
80
70
60
50
40
30
20
10
0
50
Shadow
Shadow
Watch Floor
Exceedance (%)
RMSE (deg)
Wind direction RMSE (deg)
Watch Floor
40
30
20
10
0
24
30
36
42
48
54
Fleet Weather Center - Norfolk
60
66
24
30
36
42
48
54
60
66
15
24hr wind speed errors
Wind speed, 24hr forecast error (kts)
16
14
12
Speed error (kts)
10
8
6
Shadow
Watch Floor
4
2
0
-2
-4
Fleet Weather Center - Norfolk
16
Wind speed errors
Bias (kts)
Wind speed bias (kts)
4
3.5
3
2.5
2
1.5
1
0.5
0
Shadow
Watch Floor
24
30
36
42
48
Watch Floor
30 36 42 48 54 60
Fleet Weather Center - Norfolk
66
Exceedance (%)
RMSE (kts)
Shadow
24
60
66
Wind speed threshold
exceedance (%)
Wind speed RMSE (kts)
7
6
5
4
3
2
1
0
54
16
14
12
10
8
6
4
2
0
Shadow
Watch Floor
24
30
36
42
48
54
60
66
17
24hr wave height forecast errors
Wave height, 24hr forecast error (ft)
3
2
Height error (ft)
1
0
Shadow
-1
Watch floor
-2
-3
-4
Fleet Weather Center - Norfolk
18
Wave height forecast errors
Wave height bias (ft)
0.6
Shadow
0.5
Watch Floor
Bias (ft)
0.4
0.3
0.2
0.1
0
-0.1
-0.2
24
30
36
42
48
54
1.6
Shadow
9
Shadow
Watch Floor
8
Watch Floor
Percent exceeded
RMSE (ft)
1.4
1.2
1
0.8
0.6
0.4
7
6
5
4
3
2
0.2
1
0
0
24
30
36
42
48
54
66
Wave height threshold
exceedance (%)
Wave height RMSE (ft)
1.8
60
60
Fleet Weather Center - Norfolk
66
24
30
36
42
48
54
60
66
19
Subjective wind/sea warnings findings
• Shadow wind and seas warnings seem comparable
to Watch Floor products.
– Data exists for objective validation
• Shadow wind and seas production was faster.
Fleet Weather Center - Norfolk
20
Utility of ensemble guidance
• Used ensemble as primary source of guidance.
Used deterministic to validate.
– Not obviously best way forward
• Ensemble is great when uncertainty is low
(typically associated with high pressure).
• Ensemble slows the forecast process when
uncertainty is high (especially for fast moving and
deep low pressure systems).
• Found it useful to apply color bar lower thresholds
to cut down on overwhelming data.
Fleet Weather Center - Norfolk
21
Needs
• Ensembles raise questions. We need new
products to tell us why the uncertainty looks the
way it does
– Sensitivity products
• Need quick, detailed information about form of
uncertainty.
– E.g. plumes, skewness.
• Guidance on how to define relevant uncertainty
thresholds.
• Uncertainty parameters that support all relevant
mission areas.
Fleet Weather Center - Norfolk
22
Personal takeaways…
• How I would use this on the watch floor tomorrow:
– Probability products for winds and seas
– Use deterministic models as primary form of guidance
and ensemble as means of identifying where additional
attention is necessary
• But…
– Really want information about the form of the uncertainty
and information about why the uncertainty is there
Fleet Weather Center - Norfolk
23
Possible next steps
• NPS thesis topics
– Validate winds/seas
– Conditional verification
– Compare with objective guidance
• Automated winds and seas warnings
• Possible additional experiments
– Alter CONOPS so that deterministic model is primary
source of guidance and use ensemble secondarily
– Inclusion of GFS ensemble
– Experiment with products that communicate form of
uncertainty and sensitivity information
– Write 120hr forecast products
Fleet Weather Center - Norfolk
24