Forecast Performance
Assessment of a
Kinematic and a
Magnetohydrodynamic
Solar Wind Model
[Simulation meets Reality]
09 October 2012
Integrity  Service  Excellence
Donald C. Norquist
AFRL/RVBXS
Kirtland AFB, NM
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
1
Presentation Outline
•Motivation for this study
•Models considered in this study
•Data for model initialization and forecast verification
•Method of forecast verification
•Results of forecast verification
•Conclusions reached from this study
•Acknowledgements
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
2
Motivation
•Comprehensive forecast verification informs forecasters
who use the products as guidance
•Provides feedback to model authors on performance
strengths and weaknesses
•Gives quantitative skill measures to modelers
considering use of forecast products as forcing data –
what to expect
•Guides decision-makers in operational centers regarding
incorporation or continuation of the model
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
3
Models
•Wang-Sheeley-Arge (WSA) solar wind model
– WSA corona model: potential field source surface model
inputs photosphere Br maps, computes 2.5 Rs Br grid
– Schatten Current Sheet model extends Br to 5 Rs
– Vr specified from Br field at 5.0 Rs empirically
– 1-D kinematic code propagates Vr, Br polarity (±1) to 1 AU
•Enlil solar wind model
–
–
–
–
3-D magnetohydrodynamic model from NOAA/SWPC
Computational domain: ± 60° latitude, 21.5 Rs to 1.1 AU
Computational grid: 2° lat.-lon., 0.67 Rs radial
21.5 Rs inner boundary specification: radial components
of mag. field Br from WSA corona model, also empirical Vr
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
4
Data
•Model initialization
– Global Oscillation Network Group (GONG) magnetogram
Br maps from National Solar Observatory
– 1° lat.-lon., daily at 0 UTC, 2007-2011 as available
– Uncorrected (Unc) and zero-point corrected (Cor) maps
– WSA corona model used to process maps to 5 Rs, 21.5 Rs
•Forecast verification
– Advanced Composition Explorer (ACE) observations at L1
from ACE Science Center at Cal Tech
– Bmag, Bx from MAG – used Level 2 (quality checked) for
entire period
– Proton speed from SWEPAM – used Level 2 through 10
Jun 11, Level 0 (unchecked) 11 Jun 11 – end of period
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
5
Method
•Model / Init. Cond. Pairs: Enl-Unc, Enl-Cor, WSA-Unc, WSA-Cor
•Forecast variables verified:
– Vr (models) vs. proton speed (obs): “solar wind (SW) speed”
– Interplanetary magnetic field (IMF) polarity (models) vs. sign
of Bx (obs): “IMF polarity”
•WSA forecast data used at 4.55 h output intervals (Δ tW)
– Hourly ACE averaged to Δ tW centered at WSA output time
•Nominal Enlil interval of 3.64 h rounded to 4 h interval (Δ tE)
– Enlil 2.73 min interval output averaged to Δ tE
– Hourly ACE averaged to Δ tE centered at Δ tE / 2
•ACE ave. polarity: if |Bx|/Bmag < .1, set to -999; ± 1 otherwise
•24 h obs set to -999 starting interval prior to shock arrival at L1
•Seven-day forecasts verified separately by forecast days 1 - 7
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
6
Number of Forecast Times Verified
Daily forecasts by year: 2007 (363), 2008 (355), 2009 (357), 2010 (358), 2011
(355); 1788 X 2 (Unc, Cor) = 3576 in all
Number of Forecast Times Verified by Forecast Day
WSA/Enl
Day 1
Day 2
Day 3
Day 4
Day 5
Day 6
Day 7
2007-SW
1871/2138 1888/2138 1869/2138 1879/2138 1878/2138 1877/1693 1858/2138
2007-IMF
1698/1915 1713/1915 1697/1917 1691/1917 1687/1917 1693/1917 1688/1919
2008-SW
1852/2107 1840/2107 1867/2107 1848/2107 1851/2107 1846/2107 1813/2107
2008-IMF
1688/1937 1675/1941 1709/1941 1663/1938 1684/1941 1677/1942 1648/1931
2009-SW
1853/2104 1860/2104 1838/2104 1855/2106 1845/2104 1856/2104 1825/2104
2009-IMF
1631/1819 1634/1819 1622/1815 1612/1818 1610/1817 1634/1821 1599/1820
2010-SW
1806/2081 1833/2081 1834/2084 1827/2082 1827/2081 1829/2081 1822/2081
2010-IMF
1647/1924 1697/1921 1694/1922 1706/1924 1698/1928 1708/1938 1684/1929
2011-SW
1740/2000 1758/2004 1755/2001 1750/1995 1740/1995 1767/1995 1746/2005
2011-IMF
1599/1835 1623/1837 1632/1832 1615/1828 1618/1827 1628/1826 1600/1836
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
7
Forecast Verification Metrics
•Fcst – Obs Difference (i=fcst time index):
•Mean:
•Mean Square:
•Standard Deviation:
•Absolute Mean:
X i = xFi − xOi
1 N
X = ∑ Xi
N i =1
1 N 2
2
X = ∑ Xi
N i =1
σX =
1
X =
N
1
N
N
2
(
X
−
X
)
∑ i
i =1
N
∑X
i =1
i
•Skill Score (Recurrence): SS R = 1 − X 2 R 2 ; Ri = xO
i
•Skill Score (Persistence): SS P = 1 − X
2
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
0 −27 D
− xOi
P 2 ; Pi = xOi − xOi
0
8
Results
Annual Statistics from ACE Observations
2007
2008
2009
2010
2011
Mean SW Speed (km/s)
441
450
365
406
422
Std Dev SW Speed (km/s, 4-h ave)
111
113
65
88
89
Mean IMF magnitude (nT)
4.50
4.25
3.94
4.75
5.29
No. of Shocks (24h pds removed)
6
3
4
5
21
No. of High Speed Events (HSEs)*
26
26
9
13
18
No. of IMF Polarity Changes (IPCs)*
58
48
69
71
47
*as determined from 1 h ACE observations using MacNeice (2009)
algorithm
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
9
SW Speed Fcst – Obs Difference Mean
(% of Obs Mean)
SW Speed Fcst - Obs Difference Mean (% of Obs Mean)
30
Enl-Unc
Enl-Cor
WSA-Unc
WSA-Cor
25
20
15
10
5
0
-5
07: 1 2 3 4 5 6 7 08: 1 2 3 4 5 6 7 09: 1 2 3 4 5 6 7 10: 1 2 3 4 5 6 7 11: 1 2 3 4 5 6 7
Year: Forecast Day
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
10
SW Speed Fcst – Obs Difference Absolute Mean
(% of Obs Mean)
32
SW Speed Fcst - Obs Difference Absolute Mean (% of Obs
Mean)
Enl-Unc
Enl-Cor
WSA-Unc
WSA-Cor
28
24
20
16
12
07: 1 2 3 4 5 6 7 08: 1 2 3 4 5 6 7 09: 1 2 3 4 5 6 7 10: 1 2 3 4 5 6 7 11: 1 2 3 4 5 6 7
Year: Forecast Day
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
11
SW Speed Skill Score Based on Recurrence
1
SW Speed Skill Score Based on Recurrence
0.5
0
-0.5
-1
-1.5
-2
-2.5
Enl-Unc
Enl-Cor
WSA-Unc
WSA-Cor
-3
-3.5
07: 1 2 3 4 5 6 7 08: 1 2 3 4 5 6 7 09: 1 2 3 4 5 6 7 10: 1 2 3 4 5 6 7 11: 1 2 3 4 5 6 7
Year: Forecast Day
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
12
SW Speed Fcst – Obs Difference
Standard Deviation (km/s)
120
SW Speed Fcst - Obs Difference Standard Deviation
(km/s)
Enl-Unc
Enl-Cor
WSA-Unc
WSA-Cor
110
100
90
80
70
07: 1 2 3 4 5 6 7 08: 1 2 3 4 5 6 7 09: 1 2 3 4 5 6 7 10: 1 2 3 4 5 6 7 11: 1 2 3 4 5 6 7
Year: Forecast Day
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
13
Forecast and Observation Means,
Standard Deviations, and Correlation
•Means:
1 N
1 N
xF = ∑ xF i , xO = ∑ xO i
N i =1
N i =1
•Standard Deviations: σ x =
F
•Correlation:
rxF xO =
1
N
1
N
N
∑ (x
i =1
Fi
N
∑ ( xF i − xF ) , σ xO =
2
i =1
1
N
N
2
(
x
−
x
)
∑ Oi O
i =1
− xF )( xOi − xO )
σx σx
F
O
•Alternatively, we compute correlation of day-average (DA)
forecasts and observations
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
14
Forecast and Observation SW Speed
Standard Deviation (km/s)
Forecast and Observation SW Speed Standard Deviation
(km/s)
120
Enl-Unc
Enl-Cor
WSA-Unc
WSA-Cor
Obs
110
100
90
80
70
60
50
07: 1 2 3 4 5 6 7 08: 1 2 3 4 5 6 7 09: 1 2 3 4 5 6 7 10: 1 2 3 4 5 6 7 11: 1 2 3 4 5 6 7
Year: Forecast Day
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
15
Day-Average SW Speed Forecast-Observation
Correlation
Day-Average SW Speed Forecast-Observation Correlation
0.8
Enl-Unc
Enl-Cor
WSA-Unc
WSA-Cor
0.7
0.6
0.5
0.4
0.3
0.2
07: 1 2 3 4 5 6 7 08: 1 2 3 4 5 6 7 09: 1 2 3 4 5 6 7 10: 1 2 3 4 5 6 7 11: 1 2 3 4 5 6 7
Year: Forecast Day
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
16
Percentage of Correct IMF Polarity
Forecast Intervals
Percentage of Correct IMF Polarity Forecast Intervals
84
81
Enl-Unc
Enl-Cor
WSA-Unc
WSA-Cor
78
75
72
69
66
07: 1 2 3 4 5 6 7 08: 1 2 3 4 5 6 7 09: 1 2 3 4 5 6 7 10: 1 2 3 4 5 6 7 11: 1 2 3 4 5 6 7
Year: Forecast Day
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
17
Analysis of High Speed Events (HSEs) and
IMF Polarity Changes (IPCs)
•Owens et al. (2005) HSE criteria
– A net solar wind increase of ≥ 100 km/s in ≤ 48 hours
– HSEs within two days of each other considered one event
•MacNeice (2009) HSE criteria
– Identify pds of fcst intervals w/ 1-day increase of ≥ 50 km/s
– A contiguous set is an HSE pd, pds < 18 h apart combined
– Accept HSE pds w/ min Vsw ≤ 500 km/s, max Vsw ≥ 500 km/s
– Accept HSE pds that maintain 50 km/s day-1 for ≥ 12 h
– Accept HSE pds w/ max – min difference ≥ 200 km/s
•MacNeice (2009) IPC criteria
– Convert Bpol =1 to ϕ=-45°, Bpol =-1 to ϕ=135°
– Apply 60-h running mean to each ϕ in 7-day fcst period
– Set filtered ϕ to 135° if > 45°, to -45° otherwise
– ID 1st, last fcst time, polarity of constant polarity pds (CPP)
– Iteratively remove CPP of ≤τ=.1,.2,.3,.4,.5 days & separated
from nearest CPP of like polarity by ≥ 3τ days
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
18
600
200
0
-1.1
2007051502
2007051506
2007051510
2007051514
2007051518
2007051522
2007051602
2007051606
2007051610
2007051614
2007051618
2007051622
2007051702
2007051706
2007051710
2007051714
2007051718
2007051722
2007051802
2007051806
2007051810
2007051814
2007051818
2007051822
2007051902
2007051906
2007051910
2007051914
2007051918
2007051922
2007052002
2007052006
2007052010
2007052014
2007052018
2007052022
2007052102
2007052106
2007052110
2007052114
2007052118
2007052122
SW Speed (km/s)
700
2007051502
2007051506
2007051510
2007051514
2007051518
2007051522
2007051602
2007051606
2007051610
2007051614
2007051618
2007051622
2007051702
2007051706
2007051710
2007051714
2007051718
2007051722
2007051802
2007051806
2007051810
2007051814
2007051818
2007051822
2007051902
2007051906
2007051910
2007051914
2007051918
2007051922
2007052002
2007052006
2007052010
2007052014
2007052018
2007052022
2007052102
2007052106
2007052110
2007052114
2007052118
2007052122
IPCs:
2007051510
2007051810
2007052018 1.1
IMF Polarity (+/-)
Seven-Day Enlil Forecast
Initialized 00 UTC 20070515 vs. ACE Obs
Enlil-Unc SW Spd
Enlil-Cor SW Spd
ACE SW Spd
500
400
300
HSE
Date/Time (YYYYMMDDHH)
Enlil IMF Polarity
ACE IMF Polarity
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
Date/Time (YYYYMMDDHH)
19
2007-2011 HSE Contingency Tables
HSEs (Owens et al., 2005)
HSEs (MacNeice, 2009)
Predicted
Predicted
Values shown are % of all forecast periods
Yes
No
Yes
No
O
Yes
24
36
3
18
b
No
10
30
3
76
s
Yes
22
39
3
17
e
No
8
31
2
78
r
Yes
47
12
4
15
v
No
25
16
9
72
e
Yes
44
15
5
14
d
No
22
19
6
75
Enl-Unc
Enl-Cor
WSA-Unc
WSA-Cor
Contingency Table Attributes
HSEs (Owens, 2005)
HSEs (MacNeice, 2009)
CSI=a/(a+b+c) POD=a/(a+c) Bias=(a+b)/(a+c) CSI=a/(a+b+c) POD=a/(a+c) Bias=(a+b)/(a+c)
Enl-Unc
0.34
0.40 (2/5)
0.57
0.13
0.14 (1/7)
0.29
Enl-Cor
0.32
0.36
0.49
0.13
0.15
0.25
WSA-Unc
0.56
0.80 (4/5)
1.22
0.14
0.21 (1/5)
0.68
WSA-Cor
0.54
0.75
1.12
0.20
0.26
0.58
a c
b d
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
20
HSE Arrival Time and Peak Speed
for 2007-2011 Y/Y Forecast Periods
Owens, 2005
Model # Y/Y FPs
ICs
HSE Arrival Time Obs-Fcst (h)
HSE Peak Speed Fcst-Obs (km/s)
Mean
Abs Mean
RMS
Mean
Abs Mean
RMS
Enl-Unc
437
4
28
40
9
76
96
Enl-Cor
385
-2
30
41
-27
79
96
WSA-Unc
862
4
28
39
-7
81
101
WSA-Cor
794
1
28
38
-33
75
93
MacNeice, 2009
HSE Arrival Time Obs – Fcst (h) HSE Peak Speed Fcst – Obs (km/s)
Model # Y/Y FPs
ICs
Mean
Abs Mean RMS
Mean
Abs Mean
RMS
Enl-Unc
57
14
22
29
-28
48
62
Enl-Cor
59
6
19
24
-39
53
65
WSA-Unc
66
3
27
36
-13
56
66
WSA-Cor
84
1
21
28
-30
59
75
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
21
2007-2011 IPC Contingency Tables
IPCs (MacNeice, 2009)
Predicted
Values shown are % of all forecast periods
Yes
No
O
Yes
36/3
28
b
No
9
24
s
Yes
34/3
29
e
No
7
27
r
Yes
36/2
26
v
No
8
28
e
Yes
34/3
27
d
No
7
29
Enl-Unc
Enl-Cor
WSA-Unc
WSA-Cor
Contingency Table Attributes
a c
b d
IPCs (MacNeice, 2009)
CSI=a/(a+b+c) POD=a/(a+c) Bias=(a+b)/(a+c)
Enl-Unc
0.49
0.56 (5/9)
0.70
Enl-Cor
0.49
0.54
0.65
WSA-Unc
0.51
0.58
0.71
WSA-Cor
0.50
0.56
0.67
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
22
Count and Average Duration of All CPPs
Predicted and Observed in 2007-2011
No. Positive CPPs No. Negative CPPs Ave. Duration (h)
Enl-Unc
446
453
69
Enl-Cor
411
406
67
Obs (4h)
811
838
56
WSA-Unc
441
444
66
WSA-Cor
409
435
67
Obs (4.55 h)
806
827
56
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
23
Conclusions
•COR magnetograms reduced small (except 2009) pos. F-O diff.
•WSA Vsw fcsts had 2-3% smaller abs mean F-O diff. than Enlil
•COR magnetograms reduced stnd dev of Vsw fcsts in years
with greatest observed stnd dev
•They increased fcst stnd dev when obs stnd dev was least
•Smaller F-O diff. for smoothed fcsts, larger for amplified fcsts
•F-O diff. were least at 3-5 days and increased greatly later
•Vsw skill < recurrence in more variant years, > in less variance
•Models/ICs: no significant difference in IMF polarity fcst skill
•# FPs with predicted HSE / # FPs with obs HSE: in Owens, 2/5
(Enlil), 4/5 (WSA); in MacNeice, 1/7 (Enlil), 1/5 (WSA)
•IPCs: ~ 5/9 (Enlil, WSA); 45-50% fewer CPPs predicted than
obs; predicted CPPs average 10-12 h longer than obs
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
24
Acknowledgements
•Nick Arge, David MacKenzie and Christina Lee of AFRL
solar section for their assistance with the models
•Sean McManus of National Solar Observatory for the
corrected magnetograms
•ACE Science Center at Cal Tech for the ACE Level 3
observations
•DoD High Performance Computing Modernization
Program for computer processing time on Cray, IBM
supercomputers
•AFRL applied research program and the Space Weather
Forecasting Laboratory for funding
Place
Proper A:
DISTRIBUTION
Here
Distribution
Approved forSTATEMENT
Public Release
25