Calibration Concepts for ATST Data
January 20, 2014
K. Reardon
version 0.5a
Table of Contents
From Bits to Science
Data Processing Purposes
Data Calibration
Calibration Types
Photometric
Spatial
Temporal
Spectral
Polarimetric
Calibration Recipes
Data Pipelines
Data Product Generation
Example Broadband Imager Calibration Recipe
Example Imaging Spectrograph Calibration Recipe
Example Slit Spectrograph Calibration Recipe
Calibration Implementations
Detector Calibration
Photometric Calibrations
Spatial Calibrations
Spectral Calibrations
Polarimetric Calibrations
Calibration Model
Revision History
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
Calibration Concepts for ATST Data
Date
Version
Released By
21/05/2013
20/08/2013
0.2
0.3
K. Reardon
K. Reardon
08/10/2013
0.5
K. Reardon
20/01/2014
0.5a
K. Reardon
Comments
for internal review
expanded to include calibration recipes for
different types of instruments
for review by instrument partners and
others
minor updates based on AT comments
From Bits to Science
Instrumentation at ATST will record series of two-dimensional data frames with digital
detectors. These frames will encode information about the emergent intensities from the
solar atmosphere in various ways. These same frames and their representation of the
flux will also contain signatures of the terrestrial atmosphere, the telescope, the
detector, and the instrument functional behavior. It is necessary, for a variety of
purposes, to disentangle these different contributions from the detected signal. This
requires a good knowledge of the various components in the image formation chain.
Only through the careful understanding and separation of the different components
contributing to the measured signal can we reliably analyze the information clearly
pertaining to the solar atmosphere.
The purpose of this document is to describe some of the processes that are
necessary to enable the conversion of the detected counts in the data frames into
science products. The document reviews most of the known or common data calibration
steps that are needed to produce well-understood data, presenting a general
classification scheme for these different processes. Calibration recipes are then
presented for several different classes of solar instrumentation relevant to ATST.
The “processing” of the data, calculations applied to specific data contents,
encompasses several broader concepts that can and should be separated when
thinking about the implementation of the overall data handling system. These concepts
are the actual data calibration methods, the optimized algorithms that implement those
methods, the data pipeline architecture, and the generation of public data-products. This
document primarily focuses on the first of these issues, the types and methods of data
calibration.
Data Processing Purposes
The processing of data must address several different needs.
First, a primary goal is to generate data products that are easily amenable to further
scientific evaluation. This purpose serves the need of many scientific users of the facility
in providing them with datasets with well-defined characteristics and that are stable in
time, allowing them to develop their own analysis workflows.
A second goal of data processing may also be to reduce the overall volume of data,
either by collapsing redundant, unnecessary content, or by extracting a subset of
parameters of interest.
Third, the processing of the data can also address several operational needs for the
facility. Rapid processing (minutes to days timescales) is needed for quality control of
Page 2
Calibration Concepts for ATST Data
the acquired data and can provide feedback on the successful execution of various
observing programs. This can be done both through real-time visual inspection or
through automated processes that evaluate a (semi-) fixed set of parameters related to
the scientific and calibration data. Such a process might evaluate seeing quality,
instrument setup, or completeness of (multi-instrument) datasets or observing
programs.
Fourth, the processing of the data can also allow knowledge on the instrument
performance to be extracted. These data can be aggregated over time to generate an
understanding of the long-term temporal changes in the instrument characteristics. This
can be incorporated into overall performance monitoring plans. This has an operational
impact by allowing deviations or degradations from nominal performance to be spotted
as early as possible. It also provides a baseline of instrumental characteristics and their
variability that further allows data reduction pipelines to be optimally tuned to provide
valid results for the expected range of data content, while being able to reliably flag
datasets with anomalous content.
Data Calibration
As stated above, the data frames recorded at the telescope contain signatures from a
set of components in series – the solar atmosphere, the terrestrial atmosphere, the
telescope, the instrument, and the detector. The goal of the calibration process is to
characterize the magnitude of the non-solar contributions to the acquired data.
The data can be said to be well calibrated as long as the relative contributions to the
measured data values are properly quantified. The correction of the data for the
determined calibration parameters is a possible subsequent step. In some cases, only
the calibration parameters are needed in successive analysis procedures, without any
actual removal of the signature necessary. In some cases, only a characterization of the
contribution is possible, not the actual removal, such as for some noise sources in the
measured signal.
As part of the calibration process it may be necessary to remove some contributions
in order to (accurately) determine the magnitude of successive contributions. The
effects on the data frames for some signatures are largely independent, while others
can be intertwined through the measurement process.
More specifically, the calibration process entails precisely defining the information
content of each pixel in the recorded array. The data pixels in the 2-D recorded array,
actually represent a more complex volume in the complete measurement space
(sometimes termed a “voxel”). The calibration process seeks simply to describe the
boundaries and coverage of the pixel/voxel in a complete coordinate space which
encompasses spatial, spectral, polarimetric, photometric and temporal axes. The
intensity of the recorded image can be thought of as lying along a photometric axis
(which can be relative or absolute), with some calibrations seeking to normalize the
intensity information across the array onto a common, linear axis.
Many calibration techniques fall into common classes, since they are addressing the
same general issues of data content definition. However, because different types of
instruments have dissimilar methods of sampling the incoming flux, specific calibration
technique may vary.
Page 3
Calibration Concepts for ATST Data
What does this mean for the software processes and
methods for calibrating the data from the ATST? The
essential implication is that all the instruments will
have to address, in more or less detail, the same
conceptual calibration steps, though order and
methods may vary. Many of the calibration steps will
produce outputs with similar content, which can all be
handled in a similar way. This will allow common
terminology and software components to be used
across the full instrument suite. Instrument-specific
calibration techniques will still need to be defined, but
it may be possible to identify significant overlap that
permits savings in development and maintenance of
some components.Calibration Types
We classify here a variety of standard calibration processes according to the primary
component of the data content that they seek to quantify.
Photometric
One of the fundamental calibrations, and one often given the most attention, is
correction of the signal level for varying system responses. For most scientific goals,
only a relative calibration among pixels or among a series of observations is necessary.
Some scientific studies might benefit from an absolute calibration.
Photometric calibrations typically start with corrections for the detector response,
including linearity and removal (or better, substitution) of pixels with non-predictable
response (bad pixels, dead columns, etc.). The removal of the detector bias and dark
signal are also part of the photometric correction, to ensure the signal level has a known
zero point. For instruments in the IR, the calibration of the thermal background and its
temporal variability is particularly important. If there are any pixel transfer effects
induced by the detector, such as smearing in a frame-transfer device or register
saturation, these are also part of the detector calibration.
Further photometric calibrations measure the relative variations in the system
throughput or response, also known as flat-fielding or gain-correction. Various
techniques are used for measuring the response variations, either by summing images
at different pointings to average-out any background solar structures, or through
dithering techniques that allow the system response to be solved through a set of linear
equations. One vexing issue in this process is fringe removal – this is particularly acute
in spectral images and can be introduced by interference in optical elements, filter, or
the detector substrate. Because the fringes are temperature dependent, and hence
temporally variable, they often require special treatment in the calibration process.
Page 4
Calibration Concepts for ATST Data
The knowledge of any background flux summed with the image is a further calibration
that allows the relative variations within a given pixel or among pixels to be properly
quantified. In the solar case, this background is often due to scattered light, arising both
from the telescope and instrumentation and from the sky. The former is typically
relatively stable in time, for a given instrument configuration, while the latter varies
strongly on a variety of timescales.
Straylight is a special case of scattered light due to the smearing by near wings of the
system’s point spread function (PSF). The overall form of the PSF is due to a
contribution of turbulent changes in the terrestrial atmosphere, the optical properties of
the telescope and feed optics, and the instrument itself. The instrumental PSF may be
relatively stable, but it can produce significant smearing or mixing of contributions (e.g.
crosstalk among slits or fibers). The accuracy of the focusing of the instrument will also
contribute to the overall PSF of the data.
A full photometric calibration would require knowledge of the spatially varying PSF
across the field of view, and in each image. In some cases this information may be
available, or a significant correction can be made through image processing techniques
(speckle reconstruction, deconvolution). For other data or certain types of instruments,
this information may not be available and can only be estimated or derived from other
sources (e.g. the AO statistics).
The transmission of prefilters or blocking filters, or even the overall coating
responses, may produce spectral variations in the system response that require a
photometric calibration. The distribution of such changes on the final image is often
corrected in the flat-fielding process, while spectrally dependent transmission changes
often require a separate calibration.
The total transmission of the Earth’s atmosphere also varies with time, either through
increased scattering or absorption. Absorption changes are produced by variations in
the path length through the atmosphere (i.e. elevation dependent). This will affect the
photometric stability between successive images or series of measurements, and the
transmissions variations can be modeled or measured. Telluric lines are another source
of wavelength-dependent atmospheric absorption that may also have a significant
temporal variation, especially at longer wavelengths.
Absolute flux calibrations may be needed by some scientific programs. Standard or
absolute flux calibration also provides a direct measurement of overall system
throughput performance. This calibration can be performed by comparing measured
intensities in particular wavelengths and locations (disk center) with flux atlases.
Alternatively, measurements of standard stars provide an absolute calibration.
Spatial
The calibration of the spatial coordinates for each pixel involves determination of both
the relative position of each pixel in the data array, as well as absolute offsets from a
known coordinate system. The calibration of spatial pixels in standard reference
coordinate systems is important because it allows reliable alignment of the data with
data from other instruments, both internally to ATST and externally. The spatial
calibration aims to provide the coordinates at the center or corner of a given pixel,
ignoring the question of spatial smearing that is described in the photometric
Page 5
Calibration Concepts for ATST Data
calibrations. The spatial calibration process also defines any areas masked in the image
by instrumental stops or obstructions, and hence not validly described by the sky
coordinates.
A basic spatial calibration is to determine the relative pixel positions compared to an
internal reference grid. This can be done using a images of a target with a known
geometry. Static deviations might be due to distortions induced by the optical train, or
rotation of the data axes with respect to the detector axes (e.g. tilted spectral lines on a
spectrograph image). There may be distortions or spatial variations that are a result of
the detector pixel grid non-uniformities or pixel dimensions (e.g. non-square pixels).
A second aspect of the spatial calibration is to define the pointing on a common or
absolute spatial grid. High-resolution solar images are naturally described in helioprojective coordinates. The telescope pointing is available in absolute coordinates,
including helioprojective. It is necessary to calibrate the offset of each detector’s image
coordinate grid from the telescope pointing (for each wavelength). The use of reference
external targets, such as the solar limb images or comparison to full disk images, allows
the calculation of both spatial translation and rotation with respect to the helioprojective
axes.
There may also be time varying offsets between the telescope and instrument image
coordinate grid. The most prominent cause of such variations would be atmospheric
dispersion. These drifts either need to be either measured from the data themselves or
modeled based the terrestrial atmospheric parameters. Note that there will also be timevarying image rotation of the image plane with respect to the helioprojective coordinate
scale, either due to the telescope motion or produced by the differential atmospheric
refraction.
One of the most complicated aspects of the spatial calibration is the correction for the
subfield distortions caused by the anisoplanatism of the atmospheric seeing. Even with
adaptive optic corrected images there are differential motion and other variations
between different portions of the field. In essence, this effect causes a continuous
distortion of the pointing coordinates at any given pixel. These relative offsets can be
calibrated by measuring a time series of observations of given area on the Sun, or
reduced through post-facto image correction techniques that combine multiple images.
Temporal
The temporal definition of the data contents is often very straightforward. For a
simple exposure it is simply the start and end time of the exposure. If an analog shutter
is used the voltage pulse well defines the integration time. For a direct trigger to a
camera head, the exact time of the integration is sometimes harder to determine, since
the internal delays are not well constrained. In any case, the exposure time and duration
uncertainty is typically on the order of a millisecond or less, which is of secondary
importance in most solar observations. As observations push to cadences of less than
one second, the timing jitter may need a more precise calibration. Absolute timing with
respect to global time systems are important for observations of flares or combining
data from multiple facilities.
In some cases, however, the definition of the time of observations becomes more
complicated. When summing multiple images, for example in polarimetric or image
Page 6
Calibration Concepts for ATST Data
reconstruction datasets, the time coverage may not be evenly sampled (depending on
modulation scheme or image selection process). The definition of the observing time
may be less straightforward, but again may not be a crucial factor in the scientific
utilization of the data. The achieved cadence of the observations might be more
important for the scientific evaluation.
Spectral
The calibration of the spectral response of an instrument is needed to provide a
known wavelength scale and/or reference for the interpretation of the measured spectral
profiles at each pixel. There are multiple sources of instrumental effects than can induce
spatially dependent variations in the wavelength scale.
For a grating spectrograph, it should be possible to fit a polynomial function to
measured line positions or the known atlas in order to define the spectral format. For a
fiber-fed spectrograph or an imaging spectrometer, there will be similar, but more
complex, variations offsets in the spectral scale that need to be measured and fit (to
reduce the noise inherent to the measurement process). These processes allow the
relative wavelength shifts or dispersion variations in the wavelength grid to be
determined.
The absolute wavelength calibration of the data can either be derived from the data
themselves, if high accuracy is not required, or determined from additional, targeted
calibrations. There are often temporal variations in the absolute wavelength calibration
for a given instrument, and as such the required frequency or method of the calibration
may depend on specific scientific goals.
The calibration of the instrumental (spectral) profile can also be included with the
spectral calibration (it is in some ways analogous to the spatial PSF dealt with under the
photometric calibration). The instrumental profile is important for the interpretation of the
measured spectra (especially through inversions or comparison with simulated spectra),
but is not typically used (at present) to actually correct or deconvolve the observed
spectra. Note that the instrumental profile may both vary spatially across the field and
across large wavelength ranges.
Even a simple imager may be subject to a basic spectral calibration. The
transmission profile of any filter used, narrow or broad band, needs to be determined
(and followed in time). In addition, the effective transmission wavelength of the filter,
which can vary due to changes in temperature or tilt, needs to be evaluated and tracked
in time in order to allow the long-term interpretation of the data.
Polarimetric
The ability to make detailed measurements of the magnetic field on the Sun relies on
an accurate calibration of the polarimetric properties of the telescope (and subsystems)
and instruments. The polarimetric calibration is assumed to be both spectrally and
temporally varying. The calibration methodology may rely on measurements and
modeling. There are various techniques currently employed in solar physics to measure
the polarization characteristics of a telescope or instrument. These calibration
processes may seek to treat different components of the optical system separately, in
order to simplify the measurement or inversion process. It is also the case that the
Page 7
Calibration Concepts for ATST Data
polarization characteristics of different optical components or subsystems may change
on different timescales and hence may be handled differently in the calibration process.
The polarimetric calibration may vary as a function of wavelength or position in the
field of view. The detector itself may introduce pixel-by-pixel changes in the polarimetric
response. Such variations require polarization calibration information to be available
with various levels of granularity, which may require significant volumes of calibration
data or processing to apply the corrections.
Different scientific programs may require different
levels of polarimetric calibration accuracy. Also, the
polarimetric calibration is applied at different
locations within the data reduction process,
depending on scientific goals.Calibration Recipes
While the previous section described the general aspects of data reduction, the
implementations of those steps will depend on the specific instruments and scientific
programs. Instrument teams should be able to produce calibration strategies, or recipes,
that are capable of producing data with sufficient accuracy that they are able to satisfy
broad or multiple scientific goals. A principal component of these strategies should be
the documentation of possible methods and quantitative analysis justifying the selection
and expected accuracy of particular calibration schemes. The strategy should also be
illustrated with working code that provides a functional implementation of the calibration
scheme for some trial datasets.
Since the detectors for ATST are being provided as a facility subsystem, it would
make sense for the detector developers to provide similar calibration recipes for the
removal of detector signatures from the raw data frames (to the extent possible). This
could be a natural outcome of the detailed detector calibration studies. This will allow
the implementation in the data center of efficient routines for correcting or mitigating any
detector signatures in a consistent manner.
The calibration steps described in the recipes will rely on a variety of sources for
calibration information. There will be daily or at least regular calibration measurements
taken to determine some of the time-variable instrumental contributions. In addition,
some calibration information will be derived from the data themselves (or from
associated datasets from other instruments), in particular those details about the effects
of atmospheric turbulence or transmission on the recorded data. Other types of
calibration may be taken on longer time scales and not be closely associated with any
given dataset, but rather provide general information on the instrument performance,
which may lead to preventative maintenance actions. Finally, there will be some semistatic information about the instrument characteristics that are encoded in an instrument
model that provides a reasonable description of the instrument outputs, both average
performance and expected limits. All these different forms of calibration information
need to be managed and made available to scientists and to the data processing
software.
Page 8
Calibration Concepts for ATST Data
Data Pipelines
The calibration recipes, both documentation and code, provided by the instrument
and detector teams will be a significant source of input for producing robust, monitored
data pipelines capable of autonomous processing in a workflow environment of the
majority of the ATST datasets. These pipelines will be tested with sample data provided
by the instrument teams to ensure that they produce the same results as outlined in the
calibration recipes. It is envisioned that some of the pipeline development will done in
close collaboration with the instrument and detector teams.
The pipelines will be built, to the extent possible, on a common framework of data
processing code. This will exploit commonalities in processing steps among different
instruments. This will allow extensive code reuse, which will simplify software
maintenance and facilitate further developments of the pipeline capabilities. This
approach is similar to that taken for many other astronomical facilities.
These pipelines will employ both the calibration data taken in coordination with a
specific observing program, as well as the other sources of calibration information to
process the data. The Data Center will provide an internet-accessible database of
instrument calibrations, at least for those calibrations that are more stable in time. In this
way, the data pipeline can always have access to the latest calibration information,
regardless of where it is being run. The Calibration Database will be populated with
calibration information that is generated by the calibration recipes or other information
derived from the observations and calibrations.
The data pipelines will have the ability to track which calibration processes were
applied to the data and provide an initial assessment of the validity and accuracy of the
calibration results. The calibration process may imply a correction to the values stored
in a data array, or may simply involve revising the definition of a metadata parameter(s)
to more accurately describe the data contents.
The data pipeline software will allow flexibility in the choice of which calibration
processes are required or suitable for a given dataset. Currently, this is often
determined during the data processing, as problems are identified in the dataset. With a
more autonomous data processing pipeline, this ad-hoc approach should be reduced.
Indeed, it would be advantageous if the scientific and data reduction goals were
captured during the design of the observing program. This design process seeks to
match scientific requirements to instrumental parameters, but should also be used to
extract information on the types and accuracies of the parameters to be extracted
during the data calibration and processing steps. This information, called processing
directives, would help provide a meaningful guide to the initial or minimal data
calibration needed to meet the proposers’ scientific goals. It would also help ensure that
there was a coherent, end-to-end plan for the full data lifecycle, and allow for flexible
data quality assurance applications.
Page 9
Calibration Concepts for ATST Data
Fig. 1 – a diagram taken from Peron (2012) showing how the raw data and
calibrations are combined through pipeline recipes to generate data products.
Data Product Generation
The data pipelines will produce a variety of outputs, including calibration data
products and scientific data products.
The former are the result of reduced calibration datasets and the generation of the
associated calibration values or arrays. These calibration data products will be used
operationally for quality control on the dataset acceptability as well as long-term
monitoring of instrument performance. These calibration data products will also be
made available, through central repository as described above (the Calibration
Database), for the calibration and processing of the scientific datasets.
The scientific data products will be the result of the application of the calibration
procedures to the scientific datasets. The exact calibration steps that need to be applied
may be dependent on the desired scientific goal for the observing program or for the
archival investigator. Some of the initial processing requirements may be captured in
the proposal or experiment design stage, others may be driven by specific atmospheric
conditions. The data processing algorithms will have incorporate information on the
dataset contents that are required to allow meaningful application of each processing
method. The data and metadata contents will be evaluated autonomously (guided by
the processing directives) to determine which calibrations and processing steps to apply
to each dataset. Additional data processing may be run on specific datasets in order to
ensure their compliance to the scientific goals, or if new scientific goals are identified for
that data set.
Below we outline, for different classes of instruments, the basic sets of calibrations
that should be performed in order to generate data products whose contents are well
defined and easily processed into higher level data products. The Calibration classes
are described in the section above for the various calibration types.
Page 10
Calibration Concepts for ATST Data
In some cases, the calibration process will renormalize the data onto a regular array
with grid axes corresponding to physical coordinate axes. This re-gridding might lead to
a significant increase in data volume or lead to one or more interpolations that might
degrade overall data quality. In such cases, the calibration information might be applied
on demand in order to optimize resource usage. The choice of how to apply calibration
information may be application dependent, but the key is to have the information
available and in a common framework easily utilized by software components.
A calibration recipe developed for each instrument will outline the methods to be
used to perform each calibration step and the accuracy required (or achievable). The
calibration plan will also define the maximum allowed temporal intervals between the
acquisition of a scientific dataset and the associated calibration datasets such that the
application of those calibrations allows the processed data to meet the defined accuracy
goals.
Detector:
Calibration
Type
Calibration Class
Frequency
Supplemental
Photometric
Bias Correction
Linearity Correction
Pixel Masking
Detector electronic gain
Charge Transfer Correction
(optional)
Detector Response Variations
daily
maintenance
maintenance
baseline
monthly
Spatial
Spatial Grid Non-uniformities
baseline
detector ID
Spectral
Quantum Efficiency Calibration
baseline
detector ID
Information
detector ID;
detector
temperature;
maintenance
Page 11
Calibration Concepts for ATST Data
Fig. 2 – An activity diagram showing the data processing sequence for the detector calibration.
Page 12
Calibration Concepts for ATST Data
Broadband Imaging:
Calibration
Type
Calibration Class
Supplemental
Information
Photometric
Dark signal correction
Flat-fielding (system response calibration)
Fringe removal
Scattered light - static
Atmospheric transmission
Secondary image correction
Focus Calibration
(optional)
Atmospheric PSF correction
scattered light magnitude
solar light level
Spatial
Mask definition
Optical distortions
Telescope pointing offset / rotation
Helioprojective coordinate conversion
Atmospheric dispersion offset
Seeing-induced spatial distortions
image distortion map
Spectral
Filter Wavelength Calibration
Temporal
Exposure interval
Page 13
Calibration Concepts for ATST Data
Narrowband Imaging:
Calibration
Type
Calibration Class
Supplemental
Information
Photometric
Dark signal correction
Flat-fielding (system response calibration)
Fringe removal
Atmospheric transmission
Prefilter transmission
Instrumental PSF calibration
Focus Calibration
Secondary image correction
(optional)
Scattered light - static
Atmospheric PSF correction
scattered light magnitude
atmospheric transmission
Spatial
Mask definition
Optical geometric distortions
Telescope pointing offset / rotation
Helioprojective coordinate conversion
Atmospheric dispersion offset
(optional)
Seeing-induced spatial distortions
image distortion map
Spectral
Wavelength scale calibration
Absolute wavelength calibration (x,y)
Spectral parasitic light calibration
Instrumental spectral profiles
Temporal
Exposure interval
Polarimetric
Instrumental Polarization matrix
Telescope Polarization matrix
Polarization modulation scheme
Grating Spectrograph:
Page 14
Calibration Concepts for ATST Data
Calibration
Type
Calibration Class
Supplemental
Information
Photometric
Dark signal correction
Flat-fielding (system response calibration)
Fringe removal
Scattered light - static
Atmospheric transmission
Prefilter transmission
Secondary image correction
(optional)
Spectrum separation
Spatial
Mask definition
Optical distortions
Telescope pointing offset / rotation
Helioprojective coordinate conversion
Atmospheric dispersion offset
(optional)
Seeing-induced spatial distortions
Spectral
Wavelength scale calibration
Absolute wavelength calibration (x,y)
Temporal
Exposure interval
Polarimetric
Instrumental Polarization matrix
Telescope Polarization matrix
Polarization modulation scheme
scattered light magnitude
atmospheric transmission
Instrumental spectral profiles
Example Broadband Imager Calibration Recipe
Detector
Read Data Array
Correct Linearity
Correct Bad Pixels
Correct Detector Response Variations
Provide Detector Gain, Readout Speed, Transfer Speed
Dark
Load Dark/Bias Data
Make Dark/Bias Calibration
Flat
Load Flat Field Data
Remove Dark/Bias
Define Field-of-View Mask
Derive System Response
Page 15
Calibration Concepts for ATST Data
Derive Fringes
Broadband
Load Broadband Data
Collate by wavelength, pointing, modulation state
Remove Dark/Bias / Remove System Response / Remove Fringes
Define Field-of-View Mask
Derive Intensity Variations
Image Reconstruction
Identify image bursts (by common pointing, wavelength)
Measure bulk shifts from images for bursts
Remove bulk-shifts
Perform speckle reconstruction/MFBD on broadband images
Grid
Load Grid Data
Remove Dark/Bias / Remove System Response / Remove Fringes
Determine Optical Distortions
Determine Spatial Scale, Rotation Angle
Coalignment
Extract pointing from metadata
Extract solar image rotation from metadata
Measure atmospheric dispersion from data
– or –
Derive atmospheric Dispersion from model
Remove atmospheric dispersion
Remove time-varying image rotation
Define spatial coordinates
Destretch
Measure sub-field image motions
Detrend destretch vectors
Remap destretch vector maps to original data coordinates
Destretch reconstructed images from detrended vectors
Apply corrections for image rotation and optical distortions
Target
Load Target Data
Remove Dark/Bias / Remove System Response / Remove Fringes
Determine Focus Quality
Pinhole
Load Pinhole Data
Remove Dark/Bias / Remove System Response / Remove Fringes
Identify secondary images
Light Level Monitor
Page 16
Calibration Concepts for ATST Data
Determine Atmospheric Transmission
Wavelength
Determine temporal variation in wavelength reference frame
Determine wavelength calibration
Semi-static Calibrations
Provide Instrumental PSF Calibration
Provide Static Scattered Light Calibration
Provide Prefilter Profile Calibration
Provide Spectral Parasitic Light CalibrationExample
Imaging Spectrograph Calibration Recipe
Detector
Read Data Array
Correct Linearity
Correct Bad Pixels
Correct Detector Response Variations
Store Detector Gain, Readout Speed, Transfer Speed
Dark
Load Dark/Bias Data
Make Dark/Bias Calibration
Flat
Load Flat Field Data
Remove Dark/Bias
Define Field-of-View Mask
Derive System Response
Derive Fringes
Target
Load Target Data
Remove Dark/Bias / Remove System Response / Remove Fringes
Determine Focus Quality
Grid
Load Grid Data
Remove Dark/Bias / Remove System Response / Remove Fringes
Determine Optical Distortions
Determine Spatial Scale, Angle
Pinhole
Load Pinhole Data
Remove Dark/Bias / Remove System Response / Remove Fringes
Identify secondary images
Light Level Monitor
Determine Atmospheric Transmission
Page 17
Calibration Concepts for ATST Data
Narrowband
Load Narrowband Data Array
Collate by wavelength, filter, polarization state
Remove Dark/Bias / Remove System Response / Remove Fringes
Define Field-of-View Mask
Broadband
Load Broadband Data
Remove Dark/Bias / Remove System Response / Remove Fringes
Define Field-of-View Mask
Derive Intensity Variations
Determine Optical Distortions
Determine Spatial Scale, Angle
Coalignment
Extract pointing from metadata
Extract solar image rotation from metadata
Determine broadband/narrowband image coalignment parameters
Measure atmospheric dispersion from data
– or –
Derive atmospheric Dispersion from model
Destretch
Measure bulk shifts from broadband images
Perform speckle reconstruction on broadband images
Measure sub-field image motions
Detrend destretch vectors
Destretch speckle reconstructed images from detrended vectors
Scale broadband and narrowband images to common spatial scale
– or –
Derive conversion parameters for destretch vector maps
Destretch each broadband image to corresponding speckle reconstructed image
Apply broadband destretch vectors to narrowband data
Remove bulk shifts
Correct Image distortions
Remove atmospheric dispersion
Define spatial coordinates
Wavelength
Extract Wavelength Scale from metadata and calibration data
Define spatially varying wavelength scale offsets (from flat fields)
Determine temporal variation in wavelength calibration or reference frame
Determine absolute wavelength calibration
Prefilter
Extract Prefilter Transmission / wavelength-dependent fluctuations
Determine Prefilter Wavelength Shift
Page 18
Calibration Concepts for ATST Data
Apply Prefilter Transmission Correction
Semi-static Calibrations
Provide Instrumental PSF Calibration
Provide Static Scattered Light Calibration
Provide Spectral Instrumental Profile Calibration
Provide Spectral Parasitic Light CalibrationExample Slit
Spectrograph Calibration Recipe
Detector
Read Data Array
Correct Linearity
Correct Bad Pixels
Correct Detector Response Variations
Store Detector Gain, Readout Speed, Transfer Speed
Dark
Load Dark/Bias Data
Make Dark/Bias Calibration
Flat
Load Flat Field Data
Remove Dark/Bias
Define Field-of-View Mask
Derive System Response
Derive Fringes
Target
Load Target Data
Remove Dark/Bias / Remove System Response / Remove Fringes
Determine Focus Quality
Grid
Load Grid Data
Remove Dark/Bias / Remove System Response / Remove Fringes
Determine Optical Distortions
Determine Spatial Scale, Angle
Light Level Monitor
Determine Atmospheric Transmission
Spectral Data
Load Spectral Array
Collate by wavelength, filter, modulation state
Remove Dark/Bias / Remove System Response / Remove Fringes
Define Field-of-View mask
Measure spectral drifts and image motion
Page 19
Calibration Concepts for ATST Data
Field Monitor Data
Load Field Monitor Data
Remove Dark/Bias / Remove System Response / Remove Fringes
Define Field-of-View Mask
Determine Optical Distortions
Determine Spatial Scale, Rotation Angle
Coalignment
Extract pointing from metadata
Extract solar image rotation from metadata
Construct pseudo-scans from field monitor data
Construct continuum scans from spectral data
Determine spectral/field monitor image coalignment parameters
Measure slit position for individual spectra
Measure atmospheric dispersion from data
– or –
Derive atmospheric dispersion from model
Destretch
Apply bulk shift corrections to individual spectral data
Correct Image distortions
Correct time-varying image rotation
Remove atmospheric dispersion
Define spatial coordinates
Wavelength
Determine dispersion direction
Determine slit tilt
Determine spectral curvature and spatially varying wavelength scale offsets
Extract wavelength scale from metadata and calibration data
Determine temporal variation in wavelength calibration or reference frame
Determine absolute wavelength calibration
Prefilter
Extract Prefilter Transmission Profile
Determine Prefilter Wavelength Shift
Apply Prefilter Transmission Correction
Semi-static Calibrations
Provide Instrumental PSF Calibration
Provide Static Scattered Light Calibration
Provide Spectral Instrumental Profile Calibration
Page 20
Calibration Concepts for ATST Data
Provide Spectral Parasitic Light CalibrationCalibration
Implementations
We now examine some practical implementations of the calibration classes described
earlier. This provides practical details related to calibration execution as well as an
overview of the requirements of the information needed to convert the calibration
datasets into correction or conversion parameters that can be applied to the science
data.
Calibrations are described in terms of pixel position (x, y), detector channel (c),
wavelength (λ), temperature (T), and Stokes state (s). In addition, where temporal
changes in the calibration values are known to be significant on daily timescales (or
less), the temporal coordinate (t) is also included. However, it should be assumed that
temporal changes are possible on timescales relevant for the instrument for all types of
calibrations. Further, some calibrations are better described in terms of relative spatial
position (px, py) rather than pixel positions.
Detector Calibration
Bias Level
Measurements of the signal level for a zero (or minimal)
duration exposure. Provides a measurement of the level of the
electronic pedestal in the readout electronics.
Parameters : O(x, y, c, T, t) – electronic bias as function of
pixel position, detector channel, and temperature.
Note: multiple individual bias images are co-added to reduce
contribution of noise to corrected data.
Read Noise
The sequence of bias level exposures can also be used to
measure the readout noise of the detector.
Parameters : N(x, y, c) – readout noise as function of pixel
position, and detector channel.
Dark Current
A sequence of images at a given exposure time that can be
used to measure the non-signal noise contribution, including
thermal noise and spurious illumination sources.
Parameters : D(x, y, c, T, t) – dark signal as function of pixel
position, and detector channel, as well as temperature and
exposure duration
Note: multiple individual bias images are co-added to reduce
contribution of noise to corrected data.
Page 21
Calibration Concepts for ATST Data
Linearity
Measurements of an intensity of a reference source at multiple
exposure times or signal levels. Allows non-linear detector
response to be determined. A correction function can be
determined, either by fitting an analytical function or using a
lookup table.
Parameters : �(x, y, c, DN, t) – response linearity as a function
of pixel locations, detector channel, and input data number
Changes in detector readout rate, gain, binning, or other
configurations may also result in different linearity calibrations.
Detector Response
Measurements of a reference source with even illumination,
possibly using image offsets, to determine the relative
response of each pixel. Response variations may vary with
wavelength, in particular in the presence of fringing (within the
detector) or non-uniform substrate of gate structures.
Parameters : R(x, y, c, λ, t) – detector responses as a function
of pixel position, detector channel, and wavelength.
Detector Gain
Measurement of signal level and total signal noise are used to
determine the conversion between detected photons and data
number units output by A/D converter.
Parameters : G(x, y, c) – detector gain as a function of pixel
position and channel. For some detectors gain may be the
same for all pixels or for all pixels read by a common A/D
converter.
Pixel Charge
Transfer
Known issue for frame-transfer devices. Unknown impact on
other chip types (e.g. CMOS).
Compare (quasi-) point source observations to expected
charge distribution. Transfer characteristics might depend on
camera readout settings.
Parameters : T(x, y, c) – amount of additional charge in a
given pixel and detector channel
Spatial Grid Nonuniformities
Measure a target with known geometrical properties to
compare measured and expected grid coordinates
Parameters : Sd(x, y) – vector offsets from nominal grid
positions at each pixel position
Alternatively, for non-square pixels, the non-uniformities could
be represented by a single scaling factor along one axis of the
detector array.
Quantum Efficiency
Relative response of detector to different wavelengths of light.
May come from manufacturer specifications
Parameters : Q(λ) – relative or absolute efficiency at different
wavelengths
Page 22
Calibration Concepts for ATST Data
Photometric Calibrations
Dark Current
A sequence of images at a given exposure time that can be used to
measure the non-target signal contribution, including thermal noise
and spurious illumination sources.
Parameters : D(x, y, c, T, t) – dark signal as function of pixel position,
and detector channel, as well as temperature and exposure duration
Note: multiple individual bias images are co-added to reduce
contribution of calibration noise to corrected data.
Flat-fielding (system
response calibration)
Measurements of a reference source with uniform illumination, to
determine the relative response of each pixel. Uniform illumination
can be achieved through summing of multiple fields or use of internal
lamp illumination. Alternatively, samples of the same field at different
detector offsets can be inverted to determine relative responses.
Many instruments will encode spectral information in the observed
images. If using the Sun as a flat-field reference, this spectral
signature needs to be removed to determine the proper system
response.
Parameters : F(x, y, c, λ) – system response as function of pixel
position, and detector channel, as well as wavelength
Note: multiple individual bias images are co-added to reduce
contribution of calibration noise to corrected data.
Fringe removal
Any fringes due to interference in lenses, windows or other
optical elements need to be removed. These can be highly
variable due to the sensitivity of element thickness to temperature
variations. Methods such as Fourier analysis or substrate modeling
can be used to determine fringe patterns. Fringe removal can also
be handled through the system response calibration, but is often
better corrected separately because of the temporal variability.
Parameters : f(x, y, λ) – fringe-induced response variations as a
function of pixel position and wavelength.
Page 23
Calibration Concepts for ATST Data
Atmospheric
transmission
A determination of the (relative) atmospheric extinction can be
determined by one or more of: light-level monitors; normalized
mean data counts (assuming a fixed pointing); atmospheric
models.
Parameters : A(λ, t) – the relative atmospheric transmission or
absorption.
Prefilter transmission
A curve of the wavelength dependence of filter or prefilter
transmission. The transmission may be provided by the
manufacturer (generally averaged over an unknown area), but can
also be sampled or measured in the lab or in the instrument itself.
Parameters : P(x,y, λ, θ) – the relative prefilter transmission as a
function of wavelength, tilt angle, and optionally spatial position.
Instrumental PSF/MTF
The instrumental PSF or MTF can be measured by the use of
knife-edges, sinusoidal grids or other reference targets. This
provides an estimation of the image degradation added by the
instrumental optics. It may be wavelength dependent, with
different typical scales.
Parameters : Mi(px, py, λ) – the two-dimensional PSF or MTF in
relative spatial coordinates and as a function of wavelength.
Telescope PSF
As above for the instrumental PSF, but for the PSF of the
telescope. Additional information generated by optical control
and adaptive optics systems may provide information on the
telescope figure errors or other optical misalignments that would
allow a more accurate determination of the telescope PSF.
Parameters : Mt(px, py, λ) – the two-dimensional PSF or MTF in
relative spatial coordinates and as a function of wavelength.
Page 24
Calibration Concepts for ATST Data
Focus Accuracy
Target images are also evaluated to determine the accuracy of the
focus values utilized in the observations. May be an output of an
automated focus routine.
Parameters : Z(λ) – the offset (in mm or mechanical units) of the
image plane from the optimal focus position. Alternatively, a
quantitative evaluation of the image degradation (e.g. Strehl ratio)
produced by focus errors.
Secondary images
Limb, target, or pinhole images can be used to determine the
presence and magnitude of any ghost images or internal
reflections.
Parameters : I(λ) – a list of offsets, relative magnitudes,
orientations, and blurring of any secondary images.
Scattered light - static
A measurement of the magnitude of the spatially scattered light
in the image produced by internal, and largely static
contributions. This is related to the PSF/MTF calibration listed
above, but is produced by the broad wings of the PSF that
typically need to be measured and corrected using different
techniques.
Parameters : Ms(px, py, λ) – the two-dimensional PSF or MTF in
relative spatial coordinates and as a function of wavelength.
Atmospheric PSF
Estimation or determination of the terrestrial atmospheric
contribution to the PSF. This PSF is variable temporally, spatially,
and spectrally. The PSF can be calculated from AO information or
derived from a multiple images of the (constant) object through
multiple realizations of the atmosphere. Several of the algorithms
currently in use both estimate and correct the PSF as part of the
same process.
Parameters : Ma(x, y ,px, py, λ) – the two-dimensional PSF or MTF
across the image field of view and as a function of wavelength.
Page 25
Calibration Concepts for ATST Data
Spatial Calibrations
Mask definition
Define location of sampled pixels which do and do not sample
regions of the target. Typically, these would be masked off
areas (e.g. a circular or rectangular mask).
Parameters : t(x,y) – the binary or percentage map of pixels’
correspondence to imaging of target area versus an internal
obstruction or mask.
Optical distortions
A map of the semi-static offsets between the nominal and
actual location sampled by each pixel.
Parameters : So(x, y) – vector offsets from nominal grid positions
at each pixel position for internally induced distortions or offsets
Telescope pointing
Offset and rotation of pixel grid array with respect to pointing
offset / rotation
of optical center of telescope.
Parameters : [x, y, θ] – offset and rotation parameters for
instrument pointing
Helioprojective
Definition of factors needed to derive absolute coordinates
coordinate
(typically in helioprojective coordinates) of each pixel in the
conversion
field of view.
Parameters : C(x,y) – definition of nominal plate scale,reference
pixel, rotation, that given the telescope pointing information, and
following the telescope pointing and optical distortion corrections
defined above, allow the exact coordinate positions for each pixel
to be defined.
Atmospheric
The offset (in elevation only) of the image observed at one
dispersion offset
wavelength with an image taken at the same time at a different
wavelength, or the expected position at a different wavelength.
Typically the telescope will report its pointing position in
helioprojective coordinates, but it should specify the applicable
wavelength range for that pointing.
Parameters : P(x, y, t) – time-dependent shifts of images due to
atmospheric dispersion. If detector axes are not aligned with
geocentric coordinates, then shift may be applied in both
directions.
Page 26
Calibration Concepts for ATST Data
Seeing-induced
Measurement of the time-varying turbulent changes in
spatial distortions
terrestrial atmosphere produces independent shifts within
subfields of the image. May also be considered as part of the
PSF variations described above, but often for convenience are
considered separately.
Parameters : Sa(x, y, t) – vector offsets from nominal grid
positions at each pixel position due to shifts due to rapid changes
in atmospheric transmission and reaction.
Spectral Calibrations
Wavelength scale
Define (relative) spectral sampling of data, valid for each
pixel. For a spectrograph, much of the wavelength scale is
described by the dispersion, but the wavelength scale also
includes any pixel-level distortions in the spectral scale (e.g.
line curvature). For an imaging spectrograph will include
both the nominal spectral position for each tuning, but also
the offsets from the nominal position for each pixel (e.g.
blueshift, plate errors). The wavelength position of each
pixel is typically defined as the weighted mean (or COG)
wavelength.
Parameters : W(x, y, s) – the spectral sampling at each pixel
and, optionally for each instrument setting.
Absolute wavelength
Offset factor that can be applied to relative wavelength
scale defined above to convert values to stable absolute
wavelength scale. The reference frame of the absolute
wavelength scale should be stated.
Parameters : Wa – an offset factor, separate for each
wavelength scale defined above
Page 27
Calibration Concepts for ATST Data
Spectral parasitic light
The relative contribution of the detected signal that comes
from spectral regions outside the nominal spectral bins
defined above. This could be due to scattered light,
additional orders, or secondary peaks in an interference
device transmission profile.
Parameters : p(x,y) – the magnitude of the scattered light
relative to the measured signal at some spectral position
(i.e. continuum), possibly as a function of pixel position.
Filter wavelength
An estimate of the central wavelength of the transmission
profile of a broad interference filter. May be expressed as a
filter tilt, assuming the conversion to wavelength offset is
known.
Parameters : Wi – the parameters needed to define the
central wavelength (as the weighted average of the
transmission profile) of an interference filter.
Instrumental profile
The shape of the instrumental spectral transmission profile,
as a function of wavelength around the nominal central
wavelength.
Parameters : I(x, y, λ) – the relative (or absolute) spectral
profile as a function of wavelength, and optionally as a
function of pixel position.
Polarimetric Calibrations
Instrumental and
transfer optics
polarization matrix
Determination of the Mueller matrix defining the conversion
between different polarization states due to interactions of
the incident photons with the instrumental optics.
Parameters : X(x, y, s) – the 4 x 4 matrix defining the relative
contribution of each Stokes state to the output signal,
optionally for each instrumental setting, and possibly as a
function of spatial position.
Page 28
Calibration Concepts for ATST Data
Telescope polarization
matrix
Determination of the Mueller matrix defining the conversion
between different polarization states due to interactions of
the incident photons with the telescope optics.
Parameters : T(s, t) – the 4 x 4 matrix defining the relative
contribution of each Stokes state to the output signal,
optionally for different telescope configurations and as a
function of time or pointing. The telescope polarization
characteristics typically do not produce variations across the
field in the polarization behavior.
Polarization
modulation scheme
The definition of the modulation scheme by which different
polarization states are measured. This allows the measured
signal to be processed, when combined with the
polarization matrices, so as to separate each of the Stokes
parameters.
Parameters : m(s) – a definition of the nominal modulation
output for each instrument state or recorded datum.
Calibration Model
The calibration data and the derived parameters will be tracked by the Data Center and
stored in a central repository for easy access and to allow better management of a
broad range of calibration types. The following model shows how the calibration inputs,
processing, and output parameters would be organized within this Calibration Database.
Page 29
Calibration Concepts for ATST Data
Figure 3: The Calibration data model, showing how calibration datasets (recorded at the
telescope, in a lab, or elsewhere) can be processed, or models be run, in order to
generate calibration parameters suitable for application to the data.
Page 30