Towards a Homogeneous Solar
Irradiance Dataset
Selection, Merging and Quality
Assessment
Micha Schöll, Mattiheu Kretzschmar & Thierry Dudok de Wit
LPC2E/CNRS
Micha Schöll
SORCE
Science
Meeting
28-31/01/2014
●
Introduction
–
SOLID
–
The Big Question
●
Data
●
Handling
●
Overview
–
Missing data interpolate
–
Outliers detection & removal
–
Holtz-Winters interpolation
–
Noise estimation
SORCE
Science
Meeting
Outlook
Micha Schöll
28-31/01/2014
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
The goal of SOLID is to bring added value to the European
spectral irradiance observations by combining all existing
observations, and at the same time, filling the temporal
and spectral gaps through modelling and reconstruction of
spectral irradiance variations.
(projects.pmodwrc.ch/solid)
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
1/13
The Big Question
What is the actual SSI for each day
of the year for the last few solar
cycles (~50 years)?
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
The Big Question
What is the actual SSI for each day
of the year for the last few solar
cycles (~50 years)?
And how certain are
we about that?
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
Measuring Spectral Solar Irradiance
●
●
●
●
Micha Schöll
Space based observation are
required for direct observation of
SSI
First observations from sounding
rocket flights in the 1970s
First long-term SSI from the
Atmospheric Experiment E,
1977-1981
Data provided by instrument
teams (thank you!)
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
6
Measuring Spectral Solar Irradiance
Instrument
Wavelength
(nm)
Time
(Year CE)
AEE-EUV FLUX
17
122
1978 1981
● Space based observation are
CDS
31
63
1998 2010
of 1987
SBUV-N07 required for direct
170 observation
399
1979
SBUV2-N09 SSI
170
399
1985 1997
SBUV2-N11
170
399
1989 1995
● First observations from sounding
SCIAMACHY
212
2385
2002 2012
SNOE_SXP rocket flights in the
4 1970s
4
1998 2001
SORCE SOLSTICE
FUV
115 from
179the 2003 2013
● First long-term
SSI
SORCE SOLSTICE MUV
180
309
2003 2013
Atmospheric
Experiment
E,
SORCE SIM
310
2412
2004 2011
1977-1981
Timed SEE EGS
27
190
2002 2013
Timed SEE● SSI
190
2002 2013
Data provided by27instrument
UARS Solstice
120
420
1991 2005
teams (thank you!)
UARS SUSIM
116
410
1992 2006
DeLand SSI Composite
120
400
1979 2006
SORCE Composite
0.5
2412
2003 2013
Total of 14 Instruments + 2 Composites
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
Measuring Spectral Solar Irradiance
Reference Spectra
Instrument
AEE-EUV FLUX
CDS
SBUV-N07
Arvesen
SBUV2-N09
ATLAS Composite 1
SBUV2-N11
ATLAS Composite 3
SCIAMACHY
Solspec ATLAS Composite
SNOE_SXP
Thuillier
SORCEATLAS1
SOLSTICE FUV
Thuillier
SORCEATLAS3
SOLSTICE MUV
Colina
'96 SIM
SORCE
Timed
SEE EGS
Hall
Anderson
'78
Timed'94
SEE SSI
Kurucz
UARS'95
Solstice
Kurucz
UARS SUSIM
Thekaekara
'73
DeLand SSI Composite
Total of 11 Reference Spectra
SORCE Composite
Reference Spectrum
Wavelength
(nm)
17
Time
(Year CE)
122
1978
1981
Wavelength
Time
31
63
1998
(nm)
(Year
CE) 2010
170
399
205
2495
170
399
0.5
2398
170
399
0.5
2398
212
2385
199
2385
4
4
0.5
2398
115
179
116
420
180
309
120
2500
310
2412
190
20027
310
190
20027 200 μm
120
420
299
1001
116 400 μm
410
115
120
400
0.5
2412
1979
1969 1987
1985 1997
2005
1989 1995
2005
2002 2012
2005
1998 2001
2005 2013
2003
2005 2013
2003
1996 2011
2004
2002
1978 2013
2002
1994 2013
1991
1995 2005
1992
1973 2006
1979 2006
2003 2013
Total of 14 Instruments + 2 Composites
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
Measuring Spectral Solar Irradiance
Reference Spectra
Reference Spectrum
Wavelength
(nm)
Arvesen
205
2495
ATLAS Composite 1
0.5
2398
ATLAS Composite 3
0.5
2398
Solspec ATLAS Composite
199
2385
Thuillier ATLAS1
0.5
2398
Thuillier ATLAS3
116
420
Colina '96
120
2500
Hall Anderson '78
200
310
Kurucz '04
200
200 μm
Kurucz '05
299
1001
1994
DSA
TSI1995
Thekaekara '73
115
400 μm
PSI1973
Proxies
Total of 11 Reference Spectra
Micha Schöll
Homogenous Solar Irradiance Dataset
Time
(Year CE)
ISN
1969
F 10.7
2005
Mg2005
II
MPSI
2005
Ly 2005
a
2005
MWSI
1996
Ca 1978
K
B Field
SORCE, Jan-2014
SBUV2-N09
SBUV2-N11
Colina '96
Kurucz 05
Hall Anderson 78
SBUV-N07
SORCE
SSI composite
SSI SIM
SOLSTICE MUV
SUSIM
FUV
DeLand SSI Composite
AEE
CDS
SNOE SXP
Timed
SEE EGS
Thuillier ATLAS
ATLAS Composite
Models
Thekaeara 73
Wavelength (nm)
Arvesen
Kurucz 04
Proxies
Available Data
Time (Year CE)
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
Internal Data Handling
●
Encapsulate data!
●
Pipeline approach
1. Homogenize & Re-grid data
2. Interpolate missing data
3. Normalize in time
4. Removal of outliers
–
●
Estimate uncertainties at
every step
Keep track of changes!
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
2. Interpolate Missing Data
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
2. Interpolate Missing Data
●
Iterative SVD interpolation
up to 10th order
–
Remove “ludicrous”
outliers ( > 21 σ)
–
Some data interpolated
with adjacent wavelength
included, e.g., SUSIM
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
Interpolating SBUVN 07
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
Interpolating SUSIM
?
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
3. Outliers
●
Outliers: Data that are most
likely not of physical origin
–
Instrumental artefact
–
Processing errors
–
Data transmission
errors
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
3. Outliers
SORCE MUV
309 nm
308 nm
307 nm
W/m2/nm
306 nm
305 nm
304 nm
May 2003
Micha Schöll
Oct 2005
Apr 2008
Homogenous Solar Irradiance Dataset
Sept 2010
Feb 2013
SORCE, Jan-2014
3. Outliers
●
●
Outliers: Data that are most
likely not of physical origin
–
Instrumental artifact
–
Processing errors
–
Data transmission
Flag and replace
–
Flag via auto-regressive (AR)
model
–
Replace via linear interpolation
–
Include previously interpolated
values as outlier candidates
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
AR Fit Outlier Removal
Probability Density (nm/m2/W
Irradiance (W/m2/nm)
350 nm @ SBUV2 N09
Micha Schöll
.
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
Irradiance
(Normalized)
Normalized
Radiance
Detection of Real ./. Instrumental Anomalies
(137 − 141 nm)
SORCE SSI SOLSTICE FUV
Wavelength (nm)
Quality Map
137
“Real” Anomaly
AR Fitted
Refitted AR
141
140
139
Proxy
138
ISN
100
0
Micha Schöll
9/11/2003
2/10
17/10
Time (Year CE)
Homogenous Solar Irradiance Dataset
1/11
16/11
1/12004
SORCE, Jan-2014
Physical Outliers
Time Series
Replace Outliers by
Linear Interpolation
Construct AR Model
Difference of Data
to Model > nσ?
Yes
Physical Outliers
Candidate?
No
Return Value
Difference of Data
to Model > (n+1)σ?
Flowchart of Outliers Detection Routine
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
●
●
●
Adapted Holtz Winters
extrapolation by forward &
backwards estimator with forced
endpoints
Uncertainty estimates via
bootstrapping or error-propagation
Irradiance (W/m2/nm)
4. Replacing Outliers & Missing Data
Year (CE)
Bootstrapping and error
propagation estimator display
similar behaviour
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
4. Replacing Outliers & Missing Data
Probability Density (nm/m2/W
Irradiance (W/m2/nm)
350 nm @ SBUV2 N09
Micha Schöll
Time (CE)
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
4. Replacing Outliers & Missing Data
SBUV2−N09 (259 nm)
0.114
Probability Density (nm/m /W
Irradiance (W/m2/nm)
2
Irradiance (W/m2/nm)
350 nm @ SBUV2 N09
0.113
0.112
0.111
0.11
1990.4
Micha Schöll
1990.6
1990.8
1991
1991.2
Homogenous
Time
(CE) Solar Irradiance Dataset
1991.4
SORCE, Jan-2014
5. Empirical Noise Estimate
●
Overview
●
Wavelet estimator
●
Coloured noise
●
Comparison & conclusion
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
5. Empirical Noise Estimate
●
●
Noise: Irrelevant or
meaningless data or output
occurring along with desired
information (Merriam
Webster).
Only consider high
frequency noise, i.e.
blue(-ish) noise / the HF
component of white noise.
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
5. Empirical Noise Estimate
●
●
Noise: Irrelevant or
meaningless data or output
occurring along with desired
information (Merriam
Webster).
Only consider high
frequency noise, i.e.
blue(-ish) noise / the HF
component of white noise.
Micha Schöll
●
●
●
Standard noise estimator:
∥∇ 2 d∥
With edge detection:
∥∇ 2 d :| ∇ d |<n σ d ∥
Wavelet noise estimator
(Donoho et al, 1994):
median ( W (d ) ) / 0.67
●
CC estimator
●
Fourier noise estimation
●
SVD noise estimation
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
Wavelet Noise Estimator
●
SBUVN09
Normalized median of wavelet
coefficients (Donoho et al, 1994):
Log(Noise) (W/m2/nm)
●
●
For time dependent
uncertainty, calculate noise
over a running window
Wavelength (nm)
median ( W (d ) ) / 0.67
Width = 90 days, every 30 days.
Time (Year CE)
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
Wavelet Noise Estimator
●
Normalized median of wavelet
coefficients (Donoho et al, 1994):
For time dependent
uncertainty, calculate noise
over a running window
Log(Normalised Noise) (#)
●
TIMED SEE EGS
Width = 90 days, every 30 days.
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
Log(Noise) (W/m2/nm)
median ( W (d ) ) / 0.67
●
SBUVN09
Wavelet Noise Estimator
Violet
White
Underestimates coloured
noise
Blue
Donoho Noise Estimator
Brown
✗
Wavelet noise estimator is
colour sensitive
Pink
●
Power of f (Colour Parameter)
Normal random coloured noise of length 1000, 300 trials
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
Coloured Noise Estimate
median ( WHF (d ) ) / 0.67
➔
This overestimates blue/violet
noise
Violet
Only use HF-wavelet
White
Underestimates coloured
noise
Blue
➔
Donoho Noise Estimator
HF Component Est.
Pink
✗
Wavelet noise estimator is
colour sensitive
Brown
●
Power of f (Color Parameter)
random colored noise of length 1000, 300 trials
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
Coloured Noise Estimate
median ( WHF (d ) ) / 0.67
➔
●
This overestimates blue/violet
noise
Use weighted average
Micha Schöll
Homogenous Solar Irradiance Dataset
Violet
Only use HF-wavelet
White
Underestimates coloured
noise
Blue
➔
Donoho Noise Estimator
HF Component Est.
New Noise Estimator
Pink
✗
Wavelet noise estimator is
colour sensitive
Brown
●
Power of f (Color Parameter)
random colored noise of length 1000, 300 trials
SORCE, Jan-2014
Physical Signal or Noise?
●
●
Construct MISO AR model with
data and proxy
(Radio Flux/ISN/Mg II/...) to
extract possible physical outliers
d' ← ARMISO ( d, p )
n
← WaveletNoise ( d− d' )
Estimate noise via wavelet
estimator of the difference of data
to model
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
Physical Signal or Noise?
●
●
Construct MISO AR model with
data and proxy
(Radio Flux/ISN/Mg II/...) to
extract some physical signal.
Estimate noise via wavelet
estimator of the difference of data
to model
Noise (% of σ)
●
Wavelet Estimator
MISO AR
Insignificant difference (for our
purpose)
Wavelength
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
External Noise Estimate
●
●
Other sources of
uncertainty:
–
Degradation
–
Absolute calibration
–
Orbital effect
Use estimates provided by
instrument teams
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
Conclusion
●
Available
–
time series
–
reference spectra
–
proxy data
●
Next step:
–
Create composite
are in one homogeneous data format
●
●
●
Missing data interpolated via SVD
approximation
Outliers flagged and removed
Possibility of “real” outliers accounted
for
●
Noise estimated
●
All changes tracked in quality map
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
Outlook
Use reference spectra as basis
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
Outlook
Instrument dependent variability?
Ly a
Micha Schöll
Homogenous Solar Irradiance Dataset
SORCE, Jan-2014
Thanks!
Questions,
Comments
Micha Schöll
Micha Schöll
LPC2E/CNRS
Homogenous
Solar Irradiance Dataset
SORCE, Jan-2014