CheckUVSpec
A quality assurance program for measured surface UV spectra
Contact: Ola Engelsen (Norwegian Polar Institute)
Last update: March 31, 2003
The CheckUVSpec QA program performs quality checks of measured downward hemispherical UV
spectra in flexstor file format (Heikkilä et al. 1998). It joins the two quality indicator programs
AtmosphericSignature and SHICrivm outlined below, and provides flagging output information from
either of the programs or both. The CheckUVSpec QA program is meant to supplement other existing
on site quality check procedures, not replace them. It is run operationally on all surface UV spectra
submitted to the EDUCE European UV database. The CheckUVSpec program is freely available at
http://zardoz.nilu.no/~olaeng/CheckUVSpec/CheckUVSpec.html
AtmosphericSignature by Ola Engelsen (Norwegian Polar Institute)
Introduction
The AtmosphericSignature QA program provides checks on the levels of irradiances in measured UV
spectra to flag obviously erroneous spectra or indicate approximately what kind of atmospheric
scenarios each measured spectrum resembles. This may provide important information both to the user
of the UV spectra as well as the data provider. Most users would want to reject any spectra with
obvious instrument errors from their analysis, or may want to focus on particular atmospheric
scenarios. As most UV spectra are measured automatically, little ancillary metadata on atmospheric
and surface conditions will usually be available. Furthermore, automatic unattended measurements of
UV spectra may be influenced by circumstances like ice and snow, birds, dirt, calibration errors, etc.
Most UV instruments are delicate and its calibration can easily be degraded.
Method
The Fastrt simulation tool (http://zardoz.nilu.no/~olaeng/fastrt/fastrt.html) is a central
component in the diagnosis of UV spectra. The Fastrt program is called by the AtmosphericSignature
tool to perform UV simulations in the upper and lower end of UVA of relevant atmospheric scenarios,
when needed. The relevant atmospheric scenarios are denoted MIN, MAX, AERO, CLEA and CLOU
and are described below.
MIN: We simulate the lowest naturally observable radiation levels. These, we assume to exist under
an extremely thick homogenous water cloud. This cloud is here assumed to be four km thick and have
a liquid water column of 4000g m-2. This is equivalent to a cloud optical depth of 650 at a wavelength
of 360nm. The cloud optical properties match that of alto-stratus clouds (Shettle, 1989) and the cloud
water droplets are distributed evenly between 2 and 7 km height above the terrestrial surface. There is
no surface reflection.
MAX: We have a broken clouds scenario where all downwelling radiation is initially transmitted
through the atmosphere as if only a clear atmosphere and a black surface were present. All radiation
reaching the ground is however trapped between a snow-covered ground and an extremely thick
homogenous alto-stratus cloud. This cloud is identical to the one in the MIN scenario above. We
assume that the MAX scenario yields the highest radiation levels obtainable naturally.
MAX_0: This is similar to the MAX scenario, except that now all diffuse downward radiation
transmitted through the clear atmosphere also is absorbed and scattered by the thick homogeneous
cloud.
AERO: This pertains to a cloudless but very turbid aerosol-loaded atmosphere with a visibility of 5km.
There is no surface reflection.
CLEA: This is a clear atmosphere with no aerosols or clouds present, and there is no surface
reflection.
CLOU: Cloudy atmosphere similar to MAX but with variable cloud density. The resulting irradiances
are computed iteratively in the search for a cloud liquid water column which yields the best match
between measurements and simulations.
The AtmosphericSignature QA tool relies on comparisons of measured UV spectra with modelled UV
spectra under identical observation conditions for the various atmospheric scenarios above. The
quality indicator program will print the spectrum ID followed by one of the rows in table 1 for each
measured spectrum.
Output
Spectrum
Flag
Diagnosis
name
FILE
BLACK
Flexstor file read error
CHECK
FILE
BLACK/GREY
Wavelengths
CHECK
inadequately
specified
FILE
BLACK
Irradiances
CHECK
inadequately
specified
Metric,
f
m1
M2
Sim. 1 Sim. 2
Metric,
f
m1
M2
and then followed by one of the lines below:
Spectrum
name
ATMSIG
ATMSIG
Flag
GREY
RED if:
f < 1.15
Diagnosis
Solar zenith angles off
limits [0-84] degrees
Radiation level too low
ATMSIG
BLACK if: Radiation level too high
f > 1.15
RED if:
0.85 < f <= 1.15
ATMSIG YELLOW if:
Radiation
m2 > MAX2_0
enhancement
ATMSIG
GREEN
otherwise
GREEN
Cloudless sky
ATMSIG
GREEN
CL-NCU
Sim. 1 Sim. 2
m2

MIN2
M2
MIN2
m2

MAX2
M2
MAX2
(m2-CLEA2)

(MAX2-CLEA2)
(0 > f >1)
Atmospheric visibility
(5-340 km)
M1/M2-CLEA1/CLEA2
MAX2_0 M2 CLEA2 MAX2
M2 CLEA2 AERO2
m1
M2 CLEA1 CLEA2
m1
M2 MIN1
(f < -0.15)
ATMSIG
GREEN
NCL-CU
ATMSIG
GREEN
Cloudy
M1/M2 - MIN1/MIN2
(f > 0.15)
Cloud liquid water
column (g m-2)
m2
Table 1. Output from NPI’s QA diagnosis tool AtmosphericSignature. m1 and m2 are measurements in
lower and upper part of the UVA spectral region, respectively. Subscripts 1 and 2 associated with
simulated scenarios MIN, MAX, AERO and CLEA indicates the lower and upper part of UVA,
respectively. Empty cells are filled with NaN which indicates that no legitimate number exists for this
MIN2
cell under this indication. The NaN quantity is set by the C compiler, e.g. to -99999.99. Cloud optical
depth at 360nm is the cloud liquid water column multiplied by 650/4000. N is the spectrum number
within the flexstor file. We conservatively estimate an overall joint uncertainty of 15% for
measurements and simulations.
More detailed information on some of the spectrum indicators:
Flexstor file read error: The input flexstor file is missing, has a format not fully recognized by the
program, or essential information cannot be read correctly from the file, e.g. unrecognisable units has
been encountered in the file.
Wavelengths inadequately specified: The spectra may be flagged GREY if there are no data at
wavelengths of interest to the AtmosphericSignature program, BLACK if e.g. the wavelength
units are unrecognisable.
Radiation enhancement: there is most likely a high surface albedo present, and/or a cloud constellation
causing the radiation level to be higher than for a clear atmosphere.
Cloudless sky: there are most likely no clouds present. Aerosols and haze may still be present, but not
necessarily.
CL-NCU (Cloud at Lower band- No Cloud at Upper band): there are most likely clouds in the lower
end of the scanned spectrum and cloudless sky in the upper end.
NCL-CU (No Cloud at Lower band- Cloud at Upper band): there are clouds in the upper end of the
spectrum and possibly no clouds in the lower end. There are at least clouds with very different
thickness in the lower end of the spectrum than in the upper part.
The other indicators should be self-explanatory.
Flagging from SHICrivm package to be included in the EDUCE database
Harry Slaper 29-11-2002
Introduction
Quality indicators will be included in the EDUCE-database. The quality indicators will be
related to several aspects of the spectral quality, including wavelength scale errors, spectral
shape errors and irradiance scale errors. The quality indicators will be referred to as quality
flags. In general GREEN flags meet the highest quality criteria, YELLOW flags do not fully
meet the highest criteria, but do meet the secondary criteria, RED flags identify spectra with a
doubtful quality, not meeting the secondary criteria, but not exceeding the lowest/rejection
criteria. BLACK flags indicate spectra that do not meet the lowest criteria and therefore
normally are not retrieved from the database, unless specifically requested by the user. In
addition a GREY flag is used if the spectra are doubtful but the algorithm is not allowing a
final conclusion.
Several flagging indicators are used at the database to allow a check of the data for different
aspects.
This document describes the flagging contribution from RIVM by means of the SHICrivm
package. In addition NPI has provided a quality checking routine for the irradiance scale,
which is described in a separate document.
Quality reporting by SHICrivm
Each spectral run provides quality-flagging output as shown below in the example. It is
proposed to include the following flags/indicators in the EDUCE data-base:
Shift_1
quality-flag
Shift_2
quality-flag
Start_irradiance
quality-flag
Spike+local_shape
Transmission_2
Scan_variability_2
quality-flag
quality-flag also indicator for possible atmospheric conditions
indicator providing information on the variability of conditions during
the scan
Example standard output for Quality flags obtained using SHIC:
1350801G 2002 sza 48.164 UVA_transmission: GREEN LOW_OR_NO_CLOUDS
scan_variability: STABLE_SCAN
local: GREEN STABLE indicative eff. ozone: 301.17 DU
shift1_flagging
GREEN
-0.009 (nm)
shift2_flagging
GREEN
-0.000 (nm)
start_irradiance_flag
GREEN
0.0000066 maximum_below_first: 0.0000024
Spike+local_shape
GREEN
Transmission_2
GREEN LOW_OR_NO_CLOUDS
0.86454561 2.40 all 0.81811091 12.58
scan_variability_2
STABLE_SCAN
2.40 =sd trans2
Transmission_1
NOT_IMPLEMENTED
0.68294455 18.33
local_shape_flag
GREEN STABLE
2.11 % criterium and nr_spikes 0.250 0 0.500 0 5.000 0
spike_flag_median_based GREEN nr_spikes_above_irr: 0.00050000 0 nr_spikes_above_irr0: 0.00010000 0 detected:
start_wavelength_flag GREEN
295.25 (nm) efuv_below: 0.01 %
last_wavelength_flag
GREEN
400.00 (nm) efuv_above: 0.00 %
Median_irradiance_flag NOT_EXTREME
Median_Irradiance: 0.03711040 at 310.00 nm
0
1
Shift1_flagging
Wavelength range 300-325 nm wavelength shifts of measured spectrum; numeric value
provides shift in nm.
criterium for flagging based on absolute number of shift, which is given in nm: ABS(f)
0 < GREEN < 0.1 < YELLOW < 0.2 < RED < 0.4 < BLACK
values of wavelength shifts: -0.9 up to +0.9 nm in standard analysis
The flag is returned as GREY if the algorithm does not find at least five reliable shift
determinations, or if the flag is BLACK but the median irradiance around 310 nm is
lower than 5e-4 W/(m2 nm)
2
Shift2_flagging
Wavelength range 325-400 nm wavelength shifts of measured spectrum; numeric value
provides shift in nm.
criterium for flagging based on absolute number of shift, which is given in nm: ABS(f)
0 < GREEN < 0.1 < YELLOW < 0.2 < RED < 0.4 < BLACK
values of wavelength shifts: -0.9 up to +0.9 nm in standard analysis; if outside the wavelength
range the values 9.999 is given (together with GREY flag identifier)
The flag is returned as GREY if the algorithm does not find at least five reliable shift
determinations, or if the flag is BLACK but the median irradiance around 310 nm is
lower than 5e-4 W/(m2 nm)
3
start_irradiance_flag
lowest reliable irradiance reading (criterium five subsequent ratios of irradiance readings
require to be within 25% of modelled ratios); two numeric values provide: the irradiance at
the first reliable reading, and the highest irrradiance reading below the first reliable reading;
criterium based on highest of the two numeric values:
GREEN < 5e-4 (W/(m2 nm)) < YELLOW < 1.5e-3 < RED < 5e-3 < BLACK
Values can be in principle any real number
Flag is GREY if the median irradiance level around 310 nm is lower than 5e-4 W/(m2 nm)
4
Spike and local shape flag
This flag combines results from the spike flag and the local shape flag. The worst of those two
flags is taken: the spike_flag_median_based and the local_shape_flag. There is not one
number corresponding to this flag, so it should be identified by the colour only. In addition to
the flag-colour the variability for the local_shape flag could be used as a number that is
included in the database, but it is not possible to link this one number to the flag-colour.
A spike is identified if the ratio of an irradiance measurement with the median of 10 readings
around the measured wavelength deviates more than a factor of two from a similar ratio
obtained from the modelled spectrum. A spike should at the database al ways lead to a
BLACK flag. A RED flag for spikes is identified if the spectral ratio of two subsequent
readings deviates more than 50% from the modelled ratio for all spectral irradiance levels
above the start_irradiance. Similarly a YELLOW flag is identified if a measured ratio deviates
25%.
The local shape flag indicates the variability of the ratio of two subsequent readings versus a
similar modelled ratio and therefore identifies local deviations/variations in the spectral shape.
The flagging is as follows:
GREEN < 10 % < YELLOW < 15 % <RED < 20% < BLACK
If the local shape flag is GREEN a further specification occurs:
STABLE < 3% < VARIABLE < 5% < HIGHLY_VARIABLE < 10%
Reported as GREEN STABLE etc. I propose to not include this differentiation of the GREEN
flag at the database.
5
Transmission_2
Transmission above 325 nm (up to highest wavelength, maximally 400 nm)
The transmission is calculated comparing with cloudless modelled values taking into account
earth sun distances.
Transmission is given in the first numeric value, usually between 0 and indefinite, but an
undetermined value leads to –999.999 (this occurs if the wavelength range does not include
wavelengths above 325 nm).
I now consider the implementation of a transmission for the 320-325 nm range, because many
single Brewers are found at the database for which we now do not have an irradiance check.
This will change the Transmission_2 line from:
Transmission_2
GREEN LOW_OR_NO_CLOUDS
0.86454561 2.40 all
0.81811091 12.58
To:
Transmission_2 GREEN LOW_OR_NO_CLOUDS 0.8645 2.40 region_320_325 0.8543
1.15 all 0.81811091 12.58
At the database just the numbers 0.8645 (transmission from 325-400 nm) and 0.8543
(transmission from 320-325 nm) are required, and the flagging criteria are compared with the
first number unless the value of –999.999 is given. In the latter case the value following the
region_320_325 should be used with similar border settings. The 2.40 and 1.15 in the
example are numbers to identify if large variations occur (percental variation), and are not
required for this flag.
Here the flagging is GREEN, YELLOW, RED and BLACK and in addition an identification
on the transmission is given
>2.0 'BLACK
EXTREMELY_HIGH
';
>1.5;
>1.25;
>0.75;
>0.25;
>0.10;
>0.05;
>0.01;
<0.01;
'RED
'YELLOW
'GREEN
’GREEN
'GREEN
'YELLOW
‘RED
‘BLACK
VERY_VERY_HIGH
';
VERY_HIGH
';
LOW_OR_NO_CLOUDS
';
CLOUDS
';
THICK_CLOUDS
';
VERY_THICK_CLOUDS
';
VERY_VERY_THICK_CLOUDS ';
EXTREMELY_LOW_TRANSMISS';
GREY is used instead of BLACK flags if the irradiance levels are below 5e-4 W/(m2 nm);
NOT_DETERMINED is provided if the spectral range does not cover wavelengths above 325
nm. GREY NOT_DETERMINED if irradiance level below 5e-4 W/(m2 nm) and the spectral
range does not cover wavelengths above 325 nm.
At the EDUCE meeting comments were received, implying that only BLACK and GREEN
flagging should be used as quality indicator at the database for the irradiance scale. I propose
to take over only the margins for BLACK and GREEN for the quality indication and in
addition the other transmission remarks can be relevant for data-selections, not so much as a
quality flag but rather an identifier of the conditions.
I think we still need to look at the exact borders to be used. Gunther Seckmeyer argued that
the lower limit at 0.01 is still too high. This is under consideration. The too high flag was not
disputed and makes sense when looking at the data from the database. I note that Ola
Engelsens high scenario corresponds to a factor of nearly five. It is my impression that this is
quite an extreme scenario which I have not seen in data tested so far, except for some obvious
errors.
Suggestion just take the two numbers, both transmissions as indicated above. In the testing
phase you could use a similar approach but taking the transmission following the string “all “.
For the rest this will remain identical.
6
Scan_variability_2
(NOT primarily a quality flag, but important to identify variability of conditions during the
scan)
This identifies the variability in the transmission over the scan for the wavelength range above
325 nm (up to 400 nm). It is not so much a direct quality indicator, but it is a diagnostic
identifier for large variations occurring during a scan.
Indicator is numerical real number.
> 30 %
> 20 %
>15 %
>10 %
> 5%
> 3%
>2%
>0
corresponds to
‘EXTREME_VARIATION_SCAN
';
'VERY_LARGE_VARIATION_SCAN ';
'LARGE_VARIATION_SCAN
';
'CONSIDERABLE_VARIATION_SCAN ';
'SOME_VARIATION_SCAN
';
'LOW_VARIATION_SCAN
';
'STABLE_SCAN
';
'VERY_STABLE_SCAN
';
NOT_DETERMINED is provided if the wavelength range above 325 nm is not covered in the
scan. Number of 100.00 is then given.
Acknowledgment
The development of CheckUVSpec, SHICrivm and AtmosphericSignature tools has been funded by
the European Commission (EC) through the CEC project "European Database for UV Climatology
and Evaluation (EDUCE)".
References
Shettle E P, "Models of aerosols, clouds and precipitation for atmospheric propagation
studies, In Atmospheric propagation in the UV, visible, IR and MM-region and related system
aspects, AGARD Conf. Proc. pp. 15-1-15-13, 1989
A. Heikkilä, A. Albold, G. Bernhard, B. Gardiner, K. Hurtta, P. Kirsch, A. Kylling, B. Mayer,
H. Slaper, P. Taalas, B. Walravens, A. Webb, G. Seckmeyer, SUVDAMA Data Storage,
Edition 4, 1998, Documentation for the European UV Database, http://www.muk.unihannover.de/EDUCE