Calibration Concepts for ATST Data January 20, 2014 K. Reardon version 0.5a Table of Contents From Bits to Science Data Processing Purposes Data Calibration Calibration Types Photometric Spatial Temporal Spectral Polarimetric Calibration Recipes Data Pipelines Data Product Generation Example Broadband Imager Calibration Recipe Example Imaging Spectrograph Calibration Recipe Example Slit Spectrograph Calibration Recipe Calibration Implementations Detector Calibration Photometric Calibrations Spatial Calibrations Spectral Calibrations Polarimetric Calibrations Calibration Model Revision History 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Calibration Concepts for ATST Data Date Version Released By 21/05/2013 20/08/2013 0.2 0.3 K. Reardon K. Reardon 08/10/2013 0.5 K. Reardon 20/01/2014 0.5a K. Reardon Comments for internal review expanded to include calibration recipes for different types of instruments for review by instrument partners and others minor updates based on AT comments From Bits to Science Instrumentation at ATST will record series of two-dimensional data frames with digital detectors. These frames will encode information about the emergent intensities from the solar atmosphere in various ways. These same frames and their representation of the flux will also contain signatures of the terrestrial atmosphere, the telescope, the detector, and the instrument functional behavior. It is necessary, for a variety of purposes, to disentangle these different contributions from the detected signal. This requires a good knowledge of the various components in the image formation chain. Only through the careful understanding and separation of the different components contributing to the measured signal can we reliably analyze the information clearly pertaining to the solar atmosphere. The purpose of this document is to describe some of the processes that are necessary to enable the conversion of the detected counts in the data frames into science products. The document reviews most of the known or common data calibration steps that are needed to produce well-understood data, presenting a general classification scheme for these different processes. Calibration recipes are then presented for several different classes of solar instrumentation relevant to ATST. The “processing” of the data, calculations applied to specific data contents, encompasses several broader concepts that can and should be separated when thinking about the implementation of the overall data handling system. These concepts are the actual data calibration methods, the optimized algorithms that implement those methods, the data pipeline architecture, and the generation of public data-products. This document primarily focuses on the first of these issues, the types and methods of data calibration. Data Processing Purposes The processing of data must address several different needs. First, a primary goal is to generate data products that are easily amenable to further scientific evaluation. This purpose serves the need of many scientific users of the facility in providing them with datasets with well-defined characteristics and that are stable in time, allowing them to develop their own analysis workflows. A second goal of data processing may also be to reduce the overall volume of data, either by collapsing redundant, unnecessary content, or by extracting a subset of parameters of interest. Third, the processing of the data can also address several operational needs for the facility. Rapid processing (minutes to days timescales) is needed for quality control of Page 2 Calibration Concepts for ATST Data the acquired data and can provide feedback on the successful execution of various observing programs. This can be done both through real-time visual inspection or through automated processes that evaluate a (semi-) fixed set of parameters related to the scientific and calibration data. Such a process might evaluate seeing quality, instrument setup, or completeness of (multi-instrument) datasets or observing programs. Fourth, the processing of the data can also allow knowledge on the instrument performance to be extracted. These data can be aggregated over time to generate an understanding of the long-term temporal changes in the instrument characteristics. This can be incorporated into overall performance monitoring plans. This has an operational impact by allowing deviations or degradations from nominal performance to be spotted as early as possible. It also provides a baseline of instrumental characteristics and their variability that further allows data reduction pipelines to be optimally tuned to provide valid results for the expected range of data content, while being able to reliably flag datasets with anomalous content. Data Calibration As stated above, the data frames recorded at the telescope contain signatures from a set of components in series – the solar atmosphere, the terrestrial atmosphere, the telescope, the instrument, and the detector. The goal of the calibration process is to characterize the magnitude of the non-solar contributions to the acquired data. The data can be said to be well calibrated as long as the relative contributions to the measured data values are properly quantified. The correction of the data for the determined calibration parameters is a possible subsequent step. In some cases, only the calibration parameters are needed in successive analysis procedures, without any actual removal of the signature necessary. In some cases, only a characterization of the contribution is possible, not the actual removal, such as for some noise sources in the measured signal. As part of the calibration process it may be necessary to remove some contributions in order to (accurately) determine the magnitude of successive contributions. The effects on the data frames for some signatures are largely independent, while others can be intertwined through the measurement process. More specifically, the calibration process entails precisely defining the information content of each pixel in the recorded array. The data pixels in the 2-D recorded array, actually represent a more complex volume in the complete measurement space (sometimes termed a “voxel”). The calibration process seeks simply to describe the boundaries and coverage of the pixel/voxel in a complete coordinate space which encompasses spatial, spectral, polarimetric, photometric and temporal axes. The intensity of the recorded image can be thought of as lying along a photometric axis (which can be relative or absolute), with some calibrations seeking to normalize the intensity information across the array onto a common, linear axis. Many calibration techniques fall into common classes, since they are addressing the same general issues of data content definition. However, because different types of instruments have dissimilar methods of sampling the incoming flux, specific calibration technique may vary. Page 3 Calibration Concepts for ATST Data What does this mean for the software processes and methods for calibrating the data from the ATST? The essential implication is that all the instruments will have to address, in more or less detail, the same conceptual calibration steps, though order and methods may vary. Many of the calibration steps will produce outputs with similar content, which can all be handled in a similar way. This will allow common terminology and software components to be used across the full instrument suite. Instrument-specific calibration techniques will still need to be defined, but it may be possible to identify significant overlap that permits savings in development and maintenance of some components.Calibration Types We classify here a variety of standard calibration processes according to the primary component of the data content that they seek to quantify. Photometric One of the fundamental calibrations, and one often given the most attention, is correction of the signal level for varying system responses. For most scientific goals, only a relative calibration among pixels or among a series of observations is necessary. Some scientific studies might benefit from an absolute calibration. Photometric calibrations typically start with corrections for the detector response, including linearity and removal (or better, substitution) of pixels with non-predictable response (bad pixels, dead columns, etc.). The removal of the detector bias and dark signal are also part of the photometric correction, to ensure the signal level has a known zero point. For instruments in the IR, the calibration of the thermal background and its temporal variability is particularly important. If there are any pixel transfer effects induced by the detector, such as smearing in a frame-transfer device or register saturation, these are also part of the detector calibration. Further photometric calibrations measure the relative variations in the system throughput or response, also known as flat-fielding or gain-correction. Various techniques are used for measuring the response variations, either by summing images at different pointings to average-out any background solar structures, or through dithering techniques that allow the system response to be solved through a set of linear equations. One vexing issue in this process is fringe removal – this is particularly acute in spectral images and can be introduced by interference in optical elements, filter, or the detector substrate. Because the fringes are temperature dependent, and hence temporally variable, they often require special treatment in the calibration process. Page 4 Calibration Concepts for ATST Data The knowledge of any background flux summed with the image is a further calibration that allows the relative variations within a given pixel or among pixels to be properly quantified. In the solar case, this background is often due to scattered light, arising both from the telescope and instrumentation and from the sky. The former is typically relatively stable in time, for a given instrument configuration, while the latter varies strongly on a variety of timescales. Straylight is a special case of scattered light due to the smearing by near wings of the system’s point spread function (PSF). The overall form of the PSF is due to a contribution of turbulent changes in the terrestrial atmosphere, the optical properties of the telescope and feed optics, and the instrument itself. The instrumental PSF may be relatively stable, but it can produce significant smearing or mixing of contributions (e.g. crosstalk among slits or fibers). The accuracy of the focusing of the instrument will also contribute to the overall PSF of the data. A full photometric calibration would require knowledge of the spatially varying PSF across the field of view, and in each image. In some cases this information may be available, or a significant correction can be made through image processing techniques (speckle reconstruction, deconvolution). For other data or certain types of instruments, this information may not be available and can only be estimated or derived from other sources (e.g. the AO statistics). The transmission of prefilters or blocking filters, or even the overall coating responses, may produce spectral variations in the system response that require a photometric calibration. The distribution of such changes on the final image is often corrected in the flat-fielding process, while spectrally dependent transmission changes often require a separate calibration. The total transmission of the Earth’s atmosphere also varies with time, either through increased scattering or absorption. Absorption changes are produced by variations in the path length through the atmosphere (i.e. elevation dependent). This will affect the photometric stability between successive images or series of measurements, and the transmissions variations can be modeled or measured. Telluric lines are another source of wavelength-dependent atmospheric absorption that may also have a significant temporal variation, especially at longer wavelengths. Absolute flux calibrations may be needed by some scientific programs. Standard or absolute flux calibration also provides a direct measurement of overall system throughput performance. This calibration can be performed by comparing measured intensities in particular wavelengths and locations (disk center) with flux atlases. Alternatively, measurements of standard stars provide an absolute calibration. Spatial The calibration of the spatial coordinates for each pixel involves determination of both the relative position of each pixel in the data array, as well as absolute offsets from a known coordinate system. The calibration of spatial pixels in standard reference coordinate systems is important because it allows reliable alignment of the data with data from other instruments, both internally to ATST and externally. The spatial calibration aims to provide the coordinates at the center or corner of a given pixel, ignoring the question of spatial smearing that is described in the photometric Page 5 Calibration Concepts for ATST Data calibrations. The spatial calibration process also defines any areas masked in the image by instrumental stops or obstructions, and hence not validly described by the sky coordinates. A basic spatial calibration is to determine the relative pixel positions compared to an internal reference grid. This can be done using a images of a target with a known geometry. Static deviations might be due to distortions induced by the optical train, or rotation of the data axes with respect to the detector axes (e.g. tilted spectral lines on a spectrograph image). There may be distortions or spatial variations that are a result of the detector pixel grid non-uniformities or pixel dimensions (e.g. non-square pixels). A second aspect of the spatial calibration is to define the pointing on a common or absolute spatial grid. High-resolution solar images are naturally described in helioprojective coordinates. The telescope pointing is available in absolute coordinates, including helioprojective. It is necessary to calibrate the offset of each detector’s image coordinate grid from the telescope pointing (for each wavelength). The use of reference external targets, such as the solar limb images or comparison to full disk images, allows the calculation of both spatial translation and rotation with respect to the helioprojective axes. There may also be time varying offsets between the telescope and instrument image coordinate grid. The most prominent cause of such variations would be atmospheric dispersion. These drifts either need to be either measured from the data themselves or modeled based the terrestrial atmospheric parameters. Note that there will also be timevarying image rotation of the image plane with respect to the helioprojective coordinate scale, either due to the telescope motion or produced by the differential atmospheric refraction. One of the most complicated aspects of the spatial calibration is the correction for the subfield distortions caused by the anisoplanatism of the atmospheric seeing. Even with adaptive optic corrected images there are differential motion and other variations between different portions of the field. In essence, this effect causes a continuous distortion of the pointing coordinates at any given pixel. These relative offsets can be calibrated by measuring a time series of observations of given area on the Sun, or reduced through post-facto image correction techniques that combine multiple images. Temporal The temporal definition of the data contents is often very straightforward. For a simple exposure it is simply the start and end time of the exposure. If an analog shutter is used the voltage pulse well defines the integration time. For a direct trigger to a camera head, the exact time of the integration is sometimes harder to determine, since the internal delays are not well constrained. In any case, the exposure time and duration uncertainty is typically on the order of a millisecond or less, which is of secondary importance in most solar observations. As observations push to cadences of less than one second, the timing jitter may need a more precise calibration. Absolute timing with respect to global time systems are important for observations of flares or combining data from multiple facilities. In some cases, however, the definition of the time of observations becomes more complicated. When summing multiple images, for example in polarimetric or image Page 6 Calibration Concepts for ATST Data reconstruction datasets, the time coverage may not be evenly sampled (depending on modulation scheme or image selection process). The definition of the observing time may be less straightforward, but again may not be a crucial factor in the scientific utilization of the data. The achieved cadence of the observations might be more important for the scientific evaluation. Spectral The calibration of the spectral response of an instrument is needed to provide a known wavelength scale and/or reference for the interpretation of the measured spectral profiles at each pixel. There are multiple sources of instrumental effects than can induce spatially dependent variations in the wavelength scale. For a grating spectrograph, it should be possible to fit a polynomial function to measured line positions or the known atlas in order to define the spectral format. For a fiber-fed spectrograph or an imaging spectrometer, there will be similar, but more complex, variations offsets in the spectral scale that need to be measured and fit (to reduce the noise inherent to the measurement process). These processes allow the relative wavelength shifts or dispersion variations in the wavelength grid to be determined. The absolute wavelength calibration of the data can either be derived from the data themselves, if high accuracy is not required, or determined from additional, targeted calibrations. There are often temporal variations in the absolute wavelength calibration for a given instrument, and as such the required frequency or method of the calibration may depend on specific scientific goals. The calibration of the instrumental (spectral) profile can also be included with the spectral calibration (it is in some ways analogous to the spatial PSF dealt with under the photometric calibration). The instrumental profile is important for the interpretation of the measured spectra (especially through inversions or comparison with simulated spectra), but is not typically used (at present) to actually correct or deconvolve the observed spectra. Note that the instrumental profile may both vary spatially across the field and across large wavelength ranges. Even a simple imager may be subject to a basic spectral calibration. The transmission profile of any filter used, narrow or broad band, needs to be determined (and followed in time). In addition, the effective transmission wavelength of the filter, which can vary due to changes in temperature or tilt, needs to be evaluated and tracked in time in order to allow the long-term interpretation of the data. Polarimetric The ability to make detailed measurements of the magnetic field on the Sun relies on an accurate calibration of the polarimetric properties of the telescope (and subsystems) and instruments. The polarimetric calibration is assumed to be both spectrally and temporally varying. The calibration methodology may rely on measurements and modeling. There are various techniques currently employed in solar physics to measure the polarization characteristics of a telescope or instrument. These calibration processes may seek to treat different components of the optical system separately, in order to simplify the measurement or inversion process. It is also the case that the Page 7 Calibration Concepts for ATST Data polarization characteristics of different optical components or subsystems may change on different timescales and hence may be handled differently in the calibration process. The polarimetric calibration may vary as a function of wavelength or position in the field of view. The detector itself may introduce pixel-by-pixel changes in the polarimetric response. Such variations require polarization calibration information to be available with various levels of granularity, which may require significant volumes of calibration data or processing to apply the corrections. Different scientific programs may require different levels of polarimetric calibration accuracy. Also, the polarimetric calibration is applied at different locations within the data reduction process, depending on scientific goals.Calibration Recipes While the previous section described the general aspects of data reduction, the implementations of those steps will depend on the specific instruments and scientific programs. Instrument teams should be able to produce calibration strategies, or recipes, that are capable of producing data with sufficient accuracy that they are able to satisfy broad or multiple scientific goals. A principal component of these strategies should be the documentation of possible methods and quantitative analysis justifying the selection and expected accuracy of particular calibration schemes. The strategy should also be illustrated with working code that provides a functional implementation of the calibration scheme for some trial datasets. Since the detectors for ATST are being provided as a facility subsystem, it would make sense for the detector developers to provide similar calibration recipes for the removal of detector signatures from the raw data frames (to the extent possible). This could be a natural outcome of the detailed detector calibration studies. This will allow the implementation in the data center of efficient routines for correcting or mitigating any detector signatures in a consistent manner. The calibration steps described in the recipes will rely on a variety of sources for calibration information. There will be daily or at least regular calibration measurements taken to determine some of the time-variable instrumental contributions. In addition, some calibration information will be derived from the data themselves (or from associated datasets from other instruments), in particular those details about the effects of atmospheric turbulence or transmission on the recorded data. Other types of calibration may be taken on longer time scales and not be closely associated with any given dataset, but rather provide general information on the instrument performance, which may lead to preventative maintenance actions. Finally, there will be some semistatic information about the instrument characteristics that are encoded in an instrument model that provides a reasonable description of the instrument outputs, both average performance and expected limits. All these different forms of calibration information need to be managed and made available to scientists and to the data processing software. Page 8 Calibration Concepts for ATST Data Data Pipelines The calibration recipes, both documentation and code, provided by the instrument and detector teams will be a significant source of input for producing robust, monitored data pipelines capable of autonomous processing in a workflow environment of the majority of the ATST datasets. These pipelines will be tested with sample data provided by the instrument teams to ensure that they produce the same results as outlined in the calibration recipes. It is envisioned that some of the pipeline development will done in close collaboration with the instrument and detector teams. The pipelines will be built, to the extent possible, on a common framework of data processing code. This will exploit commonalities in processing steps among different instruments. This will allow extensive code reuse, which will simplify software maintenance and facilitate further developments of the pipeline capabilities. This approach is similar to that taken for many other astronomical facilities. These pipelines will employ both the calibration data taken in coordination with a specific observing program, as well as the other sources of calibration information to process the data. The Data Center will provide an internet-accessible database of instrument calibrations, at least for those calibrations that are more stable in time. In this way, the data pipeline can always have access to the latest calibration information, regardless of where it is being run. The Calibration Database will be populated with calibration information that is generated by the calibration recipes or other information derived from the observations and calibrations. The data pipelines will have the ability to track which calibration processes were applied to the data and provide an initial assessment of the validity and accuracy of the calibration results. The calibration process may imply a correction to the values stored in a data array, or may simply involve revising the definition of a metadata parameter(s) to more accurately describe the data contents. The data pipeline software will allow flexibility in the choice of which calibration processes are required or suitable for a given dataset. Currently, this is often determined during the data processing, as problems are identified in the dataset. With a more autonomous data processing pipeline, this ad-hoc approach should be reduced. Indeed, it would be advantageous if the scientific and data reduction goals were captured during the design of the observing program. This design process seeks to match scientific requirements to instrumental parameters, but should also be used to extract information on the types and accuracies of the parameters to be extracted during the data calibration and processing steps. This information, called processing directives, would help provide a meaningful guide to the initial or minimal data calibration needed to meet the proposers’ scientific goals. It would also help ensure that there was a coherent, end-to-end plan for the full data lifecycle, and allow for flexible data quality assurance applications. Page 9 Calibration Concepts for ATST Data Fig. 1 – a diagram taken from Peron (2012) showing how the raw data and calibrations are combined through pipeline recipes to generate data products. Data Product Generation The data pipelines will produce a variety of outputs, including calibration data products and scientific data products. The former are the result of reduced calibration datasets and the generation of the associated calibration values or arrays. These calibration data products will be used operationally for quality control on the dataset acceptability as well as long-term monitoring of instrument performance. These calibration data products will also be made available, through central repository as described above (the Calibration Database), for the calibration and processing of the scientific datasets. The scientific data products will be the result of the application of the calibration procedures to the scientific datasets. The exact calibration steps that need to be applied may be dependent on the desired scientific goal for the observing program or for the archival investigator. Some of the initial processing requirements may be captured in the proposal or experiment design stage, others may be driven by specific atmospheric conditions. The data processing algorithms will have incorporate information on the dataset contents that are required to allow meaningful application of each processing method. The data and metadata contents will be evaluated autonomously (guided by the processing directives) to determine which calibrations and processing steps to apply to each dataset. Additional data processing may be run on specific datasets in order to ensure their compliance to the scientific goals, or if new scientific goals are identified for that data set. Below we outline, for different classes of instruments, the basic sets of calibrations that should be performed in order to generate data products whose contents are well defined and easily processed into higher level data products. The Calibration classes are described in the section above for the various calibration types. Page 10 Calibration Concepts for ATST Data In some cases, the calibration process will renormalize the data onto a regular array with grid axes corresponding to physical coordinate axes. This re-gridding might lead to a significant increase in data volume or lead to one or more interpolations that might degrade overall data quality. In such cases, the calibration information might be applied on demand in order to optimize resource usage. The choice of how to apply calibration information may be application dependent, but the key is to have the information available and in a common framework easily utilized by software components. A calibration recipe developed for each instrument will outline the methods to be used to perform each calibration step and the accuracy required (or achievable). The calibration plan will also define the maximum allowed temporal intervals between the acquisition of a scientific dataset and the associated calibration datasets such that the application of those calibrations allows the processed data to meet the defined accuracy goals. Detector: Calibration Type Calibration Class Frequency Supplemental Photometric Bias Correction Linearity Correction Pixel Masking Detector electronic gain Charge Transfer Correction (optional) Detector Response Variations daily maintenance maintenance baseline monthly Spatial Spatial Grid Non-uniformities baseline detector ID Spectral Quantum Efficiency Calibration baseline detector ID Information detector ID; detector temperature; maintenance Page 11 Calibration Concepts for ATST Data Fig. 2 – An activity diagram showing the data processing sequence for the detector calibration. Page 12 Calibration Concepts for ATST Data Broadband Imaging: Calibration Type Calibration Class Supplemental Information Photometric Dark signal correction Flat-fielding (system response calibration) Fringe removal Scattered light - static Atmospheric transmission Secondary image correction Focus Calibration (optional) Atmospheric PSF correction scattered light magnitude solar light level Spatial Mask definition Optical distortions Telescope pointing offset / rotation Helioprojective coordinate conversion Atmospheric dispersion offset Seeing-induced spatial distortions image distortion map Spectral Filter Wavelength Calibration Temporal Exposure interval Page 13 Calibration Concepts for ATST Data Narrowband Imaging: Calibration Type Calibration Class Supplemental Information Photometric Dark signal correction Flat-fielding (system response calibration) Fringe removal Atmospheric transmission Prefilter transmission Instrumental PSF calibration Focus Calibration Secondary image correction (optional) Scattered light - static Atmospheric PSF correction scattered light magnitude atmospheric transmission Spatial Mask definition Optical geometric distortions Telescope pointing offset / rotation Helioprojective coordinate conversion Atmospheric dispersion offset (optional) Seeing-induced spatial distortions image distortion map Spectral Wavelength scale calibration Absolute wavelength calibration (x,y) Spectral parasitic light calibration Instrumental spectral profiles Temporal Exposure interval Polarimetric Instrumental Polarization matrix Telescope Polarization matrix Polarization modulation scheme Grating Spectrograph: Page 14 Calibration Concepts for ATST Data Calibration Type Calibration Class Supplemental Information Photometric Dark signal correction Flat-fielding (system response calibration) Fringe removal Scattered light - static Atmospheric transmission Prefilter transmission Secondary image correction (optional) Spectrum separation Spatial Mask definition Optical distortions Telescope pointing offset / rotation Helioprojective coordinate conversion Atmospheric dispersion offset (optional) Seeing-induced spatial distortions Spectral Wavelength scale calibration Absolute wavelength calibration (x,y) Temporal Exposure interval Polarimetric Instrumental Polarization matrix Telescope Polarization matrix Polarization modulation scheme scattered light magnitude atmospheric transmission Instrumental spectral profiles Example Broadband Imager Calibration Recipe Detector Read Data Array Correct Linearity Correct Bad Pixels Correct Detector Response Variations Provide Detector Gain, Readout Speed, Transfer Speed Dark Load Dark/Bias Data Make Dark/Bias Calibration Flat Load Flat Field Data Remove Dark/Bias Define Field-of-View Mask Derive System Response Page 15 Calibration Concepts for ATST Data Derive Fringes Broadband Load Broadband Data Collate by wavelength, pointing, modulation state Remove Dark/Bias / Remove System Response / Remove Fringes Define Field-of-View Mask Derive Intensity Variations Image Reconstruction Identify image bursts (by common pointing, wavelength) Measure bulk shifts from images for bursts Remove bulk-shifts Perform speckle reconstruction/MFBD on broadband images Grid Load Grid Data Remove Dark/Bias / Remove System Response / Remove Fringes Determine Optical Distortions Determine Spatial Scale, Rotation Angle Coalignment Extract pointing from metadata Extract solar image rotation from metadata Measure atmospheric dispersion from data – or – Derive atmospheric Dispersion from model Remove atmospheric dispersion Remove time-varying image rotation Define spatial coordinates Destretch Measure sub-field image motions Detrend destretch vectors Remap destretch vector maps to original data coordinates Destretch reconstructed images from detrended vectors Apply corrections for image rotation and optical distortions Target Load Target Data Remove Dark/Bias / Remove System Response / Remove Fringes Determine Focus Quality Pinhole Load Pinhole Data Remove Dark/Bias / Remove System Response / Remove Fringes Identify secondary images Light Level Monitor Page 16 Calibration Concepts for ATST Data Determine Atmospheric Transmission Wavelength Determine temporal variation in wavelength reference frame Determine wavelength calibration Semi-static Calibrations Provide Instrumental PSF Calibration Provide Static Scattered Light Calibration Provide Prefilter Profile Calibration Provide Spectral Parasitic Light CalibrationExample Imaging Spectrograph Calibration Recipe Detector Read Data Array Correct Linearity Correct Bad Pixels Correct Detector Response Variations Store Detector Gain, Readout Speed, Transfer Speed Dark Load Dark/Bias Data Make Dark/Bias Calibration Flat Load Flat Field Data Remove Dark/Bias Define Field-of-View Mask Derive System Response Derive Fringes Target Load Target Data Remove Dark/Bias / Remove System Response / Remove Fringes Determine Focus Quality Grid Load Grid Data Remove Dark/Bias / Remove System Response / Remove Fringes Determine Optical Distortions Determine Spatial Scale, Angle Pinhole Load Pinhole Data Remove Dark/Bias / Remove System Response / Remove Fringes Identify secondary images Light Level Monitor Determine Atmospheric Transmission Page 17 Calibration Concepts for ATST Data Narrowband Load Narrowband Data Array Collate by wavelength, filter, polarization state Remove Dark/Bias / Remove System Response / Remove Fringes Define Field-of-View Mask Broadband Load Broadband Data Remove Dark/Bias / Remove System Response / Remove Fringes Define Field-of-View Mask Derive Intensity Variations Determine Optical Distortions Determine Spatial Scale, Angle Coalignment Extract pointing from metadata Extract solar image rotation from metadata Determine broadband/narrowband image coalignment parameters Measure atmospheric dispersion from data – or – Derive atmospheric Dispersion from model Destretch Measure bulk shifts from broadband images Perform speckle reconstruction on broadband images Measure sub-field image motions Detrend destretch vectors Destretch speckle reconstructed images from detrended vectors Scale broadband and narrowband images to common spatial scale – or – Derive conversion parameters for destretch vector maps Destretch each broadband image to corresponding speckle reconstructed image Apply broadband destretch vectors to narrowband data Remove bulk shifts Correct Image distortions Remove atmospheric dispersion Define spatial coordinates Wavelength Extract Wavelength Scale from metadata and calibration data Define spatially varying wavelength scale offsets (from flat fields) Determine temporal variation in wavelength calibration or reference frame Determine absolute wavelength calibration Prefilter Extract Prefilter Transmission / wavelength-dependent fluctuations Determine Prefilter Wavelength Shift Page 18 Calibration Concepts for ATST Data Apply Prefilter Transmission Correction Semi-static Calibrations Provide Instrumental PSF Calibration Provide Static Scattered Light Calibration Provide Spectral Instrumental Profile Calibration Provide Spectral Parasitic Light CalibrationExample Slit Spectrograph Calibration Recipe Detector Read Data Array Correct Linearity Correct Bad Pixels Correct Detector Response Variations Store Detector Gain, Readout Speed, Transfer Speed Dark Load Dark/Bias Data Make Dark/Bias Calibration Flat Load Flat Field Data Remove Dark/Bias Define Field-of-View Mask Derive System Response Derive Fringes Target Load Target Data Remove Dark/Bias / Remove System Response / Remove Fringes Determine Focus Quality Grid Load Grid Data Remove Dark/Bias / Remove System Response / Remove Fringes Determine Optical Distortions Determine Spatial Scale, Angle Light Level Monitor Determine Atmospheric Transmission Spectral Data Load Spectral Array Collate by wavelength, filter, modulation state Remove Dark/Bias / Remove System Response / Remove Fringes Define Field-of-View mask Measure spectral drifts and image motion Page 19 Calibration Concepts for ATST Data Field Monitor Data Load Field Monitor Data Remove Dark/Bias / Remove System Response / Remove Fringes Define Field-of-View Mask Determine Optical Distortions Determine Spatial Scale, Rotation Angle Coalignment Extract pointing from metadata Extract solar image rotation from metadata Construct pseudo-scans from field monitor data Construct continuum scans from spectral data Determine spectral/field monitor image coalignment parameters Measure slit position for individual spectra Measure atmospheric dispersion from data – or – Derive atmospheric dispersion from model Destretch Apply bulk shift corrections to individual spectral data Correct Image distortions Correct time-varying image rotation Remove atmospheric dispersion Define spatial coordinates Wavelength Determine dispersion direction Determine slit tilt Determine spectral curvature and spatially varying wavelength scale offsets Extract wavelength scale from metadata and calibration data Determine temporal variation in wavelength calibration or reference frame Determine absolute wavelength calibration Prefilter Extract Prefilter Transmission Profile Determine Prefilter Wavelength Shift Apply Prefilter Transmission Correction Semi-static Calibrations Provide Instrumental PSF Calibration Provide Static Scattered Light Calibration Provide Spectral Instrumental Profile Calibration Page 20 Calibration Concepts for ATST Data Provide Spectral Parasitic Light CalibrationCalibration Implementations We now examine some practical implementations of the calibration classes described earlier. This provides practical details related to calibration execution as well as an overview of the requirements of the information needed to convert the calibration datasets into correction or conversion parameters that can be applied to the science data. Calibrations are described in terms of pixel position (x, y), detector channel (c), wavelength (λ), temperature (T), and Stokes state (s). In addition, where temporal changes in the calibration values are known to be significant on daily timescales (or less), the temporal coordinate (t) is also included. However, it should be assumed that temporal changes are possible on timescales relevant for the instrument for all types of calibrations. Further, some calibrations are better described in terms of relative spatial position (px, py) rather than pixel positions. Detector Calibration Bias Level Measurements of the signal level for a zero (or minimal) duration exposure. Provides a measurement of the level of the electronic pedestal in the readout electronics. Parameters : O(x, y, c, T, t) – electronic bias as function of pixel position, detector channel, and temperature. Note: multiple individual bias images are co-added to reduce contribution of noise to corrected data. Read Noise The sequence of bias level exposures can also be used to measure the readout noise of the detector. Parameters : N(x, y, c) – readout noise as function of pixel position, and detector channel. Dark Current A sequence of images at a given exposure time that can be used to measure the non-signal noise contribution, including thermal noise and spurious illumination sources. Parameters : D(x, y, c, T, t) – dark signal as function of pixel position, and detector channel, as well as temperature and exposure duration Note: multiple individual bias images are co-added to reduce contribution of noise to corrected data. Page 21 Calibration Concepts for ATST Data Linearity Measurements of an intensity of a reference source at multiple exposure times or signal levels. Allows non-linear detector response to be determined. A correction function can be determined, either by fitting an analytical function or using a lookup table. Parameters : �(x, y, c, DN, t) – response linearity as a function of pixel locations, detector channel, and input data number Changes in detector readout rate, gain, binning, or other configurations may also result in different linearity calibrations. Detector Response Measurements of a reference source with even illumination, possibly using image offsets, to determine the relative response of each pixel. Response variations may vary with wavelength, in particular in the presence of fringing (within the detector) or non-uniform substrate of gate structures. Parameters : R(x, y, c, λ, t) – detector responses as a function of pixel position, detector channel, and wavelength. Detector Gain Measurement of signal level and total signal noise are used to determine the conversion between detected photons and data number units output by A/D converter. Parameters : G(x, y, c) – detector gain as a function of pixel position and channel. For some detectors gain may be the same for all pixels or for all pixels read by a common A/D converter. Pixel Charge Transfer Known issue for frame-transfer devices. Unknown impact on other chip types (e.g. CMOS). Compare (quasi-) point source observations to expected charge distribution. Transfer characteristics might depend on camera readout settings. Parameters : T(x, y, c) – amount of additional charge in a given pixel and detector channel Spatial Grid Nonuniformities Measure a target with known geometrical properties to compare measured and expected grid coordinates Parameters : Sd(x, y) – vector offsets from nominal grid positions at each pixel position Alternatively, for non-square pixels, the non-uniformities could be represented by a single scaling factor along one axis of the detector array. Quantum Efficiency Relative response of detector to different wavelengths of light. May come from manufacturer specifications Parameters : Q(λ) – relative or absolute efficiency at different wavelengths Page 22 Calibration Concepts for ATST Data Photometric Calibrations Dark Current A sequence of images at a given exposure time that can be used to measure the non-target signal contribution, including thermal noise and spurious illumination sources. Parameters : D(x, y, c, T, t) – dark signal as function of pixel position, and detector channel, as well as temperature and exposure duration Note: multiple individual bias images are co-added to reduce contribution of calibration noise to corrected data. Flat-fielding (system response calibration) Measurements of a reference source with uniform illumination, to determine the relative response of each pixel. Uniform illumination can be achieved through summing of multiple fields or use of internal lamp illumination. Alternatively, samples of the same field at different detector offsets can be inverted to determine relative responses. Many instruments will encode spectral information in the observed images. If using the Sun as a flat-field reference, this spectral signature needs to be removed to determine the proper system response. Parameters : F(x, y, c, λ) – system response as function of pixel position, and detector channel, as well as wavelength Note: multiple individual bias images are co-added to reduce contribution of calibration noise to corrected data. Fringe removal Any fringes due to interference in lenses, windows or other optical elements need to be removed. These can be highly variable due to the sensitivity of element thickness to temperature variations. Methods such as Fourier analysis or substrate modeling can be used to determine fringe patterns. Fringe removal can also be handled through the system response calibration, but is often better corrected separately because of the temporal variability. Parameters : f(x, y, λ) – fringe-induced response variations as a function of pixel position and wavelength. Page 23 Calibration Concepts for ATST Data Atmospheric transmission A determination of the (relative) atmospheric extinction can be determined by one or more of: light-level monitors; normalized mean data counts (assuming a fixed pointing); atmospheric models. Parameters : A(λ, t) – the relative atmospheric transmission or absorption. Prefilter transmission A curve of the wavelength dependence of filter or prefilter transmission. The transmission may be provided by the manufacturer (generally averaged over an unknown area), but can also be sampled or measured in the lab or in the instrument itself. Parameters : P(x,y, λ, θ) – the relative prefilter transmission as a function of wavelength, tilt angle, and optionally spatial position. Instrumental PSF/MTF The instrumental PSF or MTF can be measured by the use of knife-edges, sinusoidal grids or other reference targets. This provides an estimation of the image degradation added by the instrumental optics. It may be wavelength dependent, with different typical scales. Parameters : Mi(px, py, λ) – the two-dimensional PSF or MTF in relative spatial coordinates and as a function of wavelength. Telescope PSF As above for the instrumental PSF, but for the PSF of the telescope. Additional information generated by optical control and adaptive optics systems may provide information on the telescope figure errors or other optical misalignments that would allow a more accurate determination of the telescope PSF. Parameters : Mt(px, py, λ) – the two-dimensional PSF or MTF in relative spatial coordinates and as a function of wavelength. Page 24 Calibration Concepts for ATST Data Focus Accuracy Target images are also evaluated to determine the accuracy of the focus values utilized in the observations. May be an output of an automated focus routine. Parameters : Z(λ) – the offset (in mm or mechanical units) of the image plane from the optimal focus position. Alternatively, a quantitative evaluation of the image degradation (e.g. Strehl ratio) produced by focus errors. Secondary images Limb, target, or pinhole images can be used to determine the presence and magnitude of any ghost images or internal reflections. Parameters : I(λ) – a list of offsets, relative magnitudes, orientations, and blurring of any secondary images. Scattered light - static A measurement of the magnitude of the spatially scattered light in the image produced by internal, and largely static contributions. This is related to the PSF/MTF calibration listed above, but is produced by the broad wings of the PSF that typically need to be measured and corrected using different techniques. Parameters : Ms(px, py, λ) – the two-dimensional PSF or MTF in relative spatial coordinates and as a function of wavelength. Atmospheric PSF Estimation or determination of the terrestrial atmospheric contribution to the PSF. This PSF is variable temporally, spatially, and spectrally. The PSF can be calculated from AO information or derived from a multiple images of the (constant) object through multiple realizations of the atmosphere. Several of the algorithms currently in use both estimate and correct the PSF as part of the same process. Parameters : Ma(x, y ,px, py, λ) – the two-dimensional PSF or MTF across the image field of view and as a function of wavelength. Page 25 Calibration Concepts for ATST Data Spatial Calibrations Mask definition Define location of sampled pixels which do and do not sample regions of the target. Typically, these would be masked off areas (e.g. a circular or rectangular mask). Parameters : t(x,y) – the binary or percentage map of pixels’ correspondence to imaging of target area versus an internal obstruction or mask. Optical distortions A map of the semi-static offsets between the nominal and actual location sampled by each pixel. Parameters : So(x, y) – vector offsets from nominal grid positions at each pixel position for internally induced distortions or offsets Telescope pointing Offset and rotation of pixel grid array with respect to pointing offset / rotation of optical center of telescope. Parameters : [x, y, θ] – offset and rotation parameters for instrument pointing Helioprojective Definition of factors needed to derive absolute coordinates coordinate (typically in helioprojective coordinates) of each pixel in the conversion field of view. Parameters : C(x,y) – definition of nominal plate scale,reference pixel, rotation, that given the telescope pointing information, and following the telescope pointing and optical distortion corrections defined above, allow the exact coordinate positions for each pixel to be defined. Atmospheric The offset (in elevation only) of the image observed at one dispersion offset wavelength with an image taken at the same time at a different wavelength, or the expected position at a different wavelength. Typically the telescope will report its pointing position in helioprojective coordinates, but it should specify the applicable wavelength range for that pointing. Parameters : P(x, y, t) – time-dependent shifts of images due to atmospheric dispersion. If detector axes are not aligned with geocentric coordinates, then shift may be applied in both directions. Page 26 Calibration Concepts for ATST Data Seeing-induced Measurement of the time-varying turbulent changes in spatial distortions terrestrial atmosphere produces independent shifts within subfields of the image. May also be considered as part of the PSF variations described above, but often for convenience are considered separately. Parameters : Sa(x, y, t) – vector offsets from nominal grid positions at each pixel position due to shifts due to rapid changes in atmospheric transmission and reaction. Spectral Calibrations Wavelength scale Define (relative) spectral sampling of data, valid for each pixel. For a spectrograph, much of the wavelength scale is described by the dispersion, but the wavelength scale also includes any pixel-level distortions in the spectral scale (e.g. line curvature). For an imaging spectrograph will include both the nominal spectral position for each tuning, but also the offsets from the nominal position for each pixel (e.g. blueshift, plate errors). The wavelength position of each pixel is typically defined as the weighted mean (or COG) wavelength. Parameters : W(x, y, s) – the spectral sampling at each pixel and, optionally for each instrument setting. Absolute wavelength Offset factor that can be applied to relative wavelength scale defined above to convert values to stable absolute wavelength scale. The reference frame of the absolute wavelength scale should be stated. Parameters : Wa – an offset factor, separate for each wavelength scale defined above Page 27 Calibration Concepts for ATST Data Spectral parasitic light The relative contribution of the detected signal that comes from spectral regions outside the nominal spectral bins defined above. This could be due to scattered light, additional orders, or secondary peaks in an interference device transmission profile. Parameters : p(x,y) – the magnitude of the scattered light relative to the measured signal at some spectral position (i.e. continuum), possibly as a function of pixel position. Filter wavelength An estimate of the central wavelength of the transmission profile of a broad interference filter. May be expressed as a filter tilt, assuming the conversion to wavelength offset is known. Parameters : Wi – the parameters needed to define the central wavelength (as the weighted average of the transmission profile) of an interference filter. Instrumental profile The shape of the instrumental spectral transmission profile, as a function of wavelength around the nominal central wavelength. Parameters : I(x, y, λ) – the relative (or absolute) spectral profile as a function of wavelength, and optionally as a function of pixel position. Polarimetric Calibrations Instrumental and transfer optics polarization matrix Determination of the Mueller matrix defining the conversion between different polarization states due to interactions of the incident photons with the instrumental optics. Parameters : X(x, y, s) – the 4 x 4 matrix defining the relative contribution of each Stokes state to the output signal, optionally for each instrumental setting, and possibly as a function of spatial position. Page 28 Calibration Concepts for ATST Data Telescope polarization matrix Determination of the Mueller matrix defining the conversion between different polarization states due to interactions of the incident photons with the telescope optics. Parameters : T(s, t) – the 4 x 4 matrix defining the relative contribution of each Stokes state to the output signal, optionally for different telescope configurations and as a function of time or pointing. The telescope polarization characteristics typically do not produce variations across the field in the polarization behavior. Polarization modulation scheme The definition of the modulation scheme by which different polarization states are measured. This allows the measured signal to be processed, when combined with the polarization matrices, so as to separate each of the Stokes parameters. Parameters : m(s) – a definition of the nominal modulation output for each instrument state or recorded datum. Calibration Model The calibration data and the derived parameters will be tracked by the Data Center and stored in a central repository for easy access and to allow better management of a broad range of calibration types. The following model shows how the calibration inputs, processing, and output parameters would be organized within this Calibration Database. Page 29 Calibration Concepts for ATST Data Figure 3: The Calibration data model, showing how calibration datasets (recorded at the telescope, in a lab, or elsewhere) can be processed, or models be run, in order to generate calibration parameters suitable for application to the data. Page 30