Instrument Science Report NICMOS-97-029
The STScI NICMOS Pipeline: CALNICB,
Reduction of Image Associations
John MacKenty, Chris Skinner, Howard Bushouse
October 28, 1997
ABSTRACT
This ISR defines CALNICB, the second and final stage of the STScI NICMOS calibration
pipeline. CALNICB contains the elements of the pipeline which operate on the members of
an association of NICMOS exposures. It removes cosmic rays by anti-coincidence detection, constructs mosaics from pre-defined patterns of observations, and measures and subtracts the background.
1. Introduction
Motivations
The calibration of NICMOS observations can be divided into two logically distinct parts.
The first part depends upon calibration information which is stable over relatively long
periods or on engineering data which is returned with each observation. The second part
depends upon calibration obtained contemporaneously with the observation to be calibrated. A simple example of the first kind is the correction for sensitivity non-uniformities
(i.e. flat fielding) while one of the second kind is the removal of cosmic ray events by the
comparison of two or more exposures of the same field. The existing STScI data processing pipelines only perform calibrations of the first type. Early in the requirements
definition phase for the second generation science instruments (i.e. NICMOS and STIS) it
was recognized that support for contemporaneous calibrations was both necessary to a
greater degree than required for the first generation of HST instruments and also more
practical with the improvements in computer technology over the past 5 to 10 years.
The primary goals of the calibration pipeline are to: (1) process data to the point that it can
be used for initial scientific analysis, (2) permit uniform and routine quality checking of
observations at STScI, and (3) to populate the Hubble Data Archive (HDA) with consistent products suitable for archival analysis without the need for significant reprocessing.
1
CALNICB incorporates several calibration steps which would otherwise have to be performed by users of NICMOS data prior to reaching the above goals.
Patterns and Associations
To make the contemporaneous calibration of multiple exposures robust and economical,
one essential limitation has been imposed on this process. This is that the exposures to be
calibrated as a set must be members of an association defined on the Phase 2 proposal
logsheet (i.e. at the start of the observing process). Mechanisms to define such associations have been defined and implemented within the pre-observation process (i.e. RPS2
and TRANS). For NICMOS observations, two mechanisms exist to create associations of
exposures: (1) the optional parameter “NEXP” (number of identical exposures to be
obtained) may be set greater than 1 (the default), and (2) the optional parameter “PATTERN”. The PATTERN parameter results in a sequence of NUM-POS pointings (with
NEXP exposures being obtained at each pointing). Each pointing position is controlled by
the combination of the optional parameters DITH-SIZE, CHOP-SIZE, PATTERN, and
PATTERN-ORIENT (see ISR NICMOS-013). This information is passed to the image
headers for use by CALNICB.
2. Development Strategy
Multiple Builds
CALNICB presents a number of development challenges. First, the complexity of the
project drives us to implement it in stages. Not all observing options will be supported in
the first “Build”. Second, there exist issues related to NICMOS performance and observing strategies which will not be resolved until after NICMOS is installed in HST and
tested extensively. Of particular importance are the stability of the thermal background
with time, the possible spatial variation of the illumination pattern (as compared to the
absolute level) of the background, and the severity of the effects of the charged particle
environment (i.e. cosmic rays).
In this ISR we outline the elements to be included in Builds 1 and 2 of CALNICB. The
division of capabilities is designed to support many observations with Build 1, and nearly
all with Build 2, under our current assumptions. Build 1 by itself is sufficient for testing all
aspects of the integrated TRANS, OPUS, and HDA ground systems. Build 3 would occur
after SMOV and will reflect our increased understanding of the limiting factors in the
NICMOS calibration, and may implement more sophisticated algorithms for certain subsets of the observational parameter space (e.g. observations with very large numbers of
exposures).
2
Content and Assumptions for Build 1
Build 1 will correct for cosmic rays in registered (NEXP>1) and dithered frames. It will
assume that the headers contain sufficiently accurate pointing information to accomplish
the registration and mosaicing of images. That is, it will not properly process long exposures obtained with gyro hold pointing control (where “long” means ~10s - 15s for
Camera 1 and ~60s - 100s for Camera 3.
Background removal will be limited to averages of the sky frames (or target frames when
no sky frames exist) and a sanity check with the expected levels. This will require only
single image source location and iterative sigma clipping (not comparison of source reality
between exposures). Highly discrepant outcomes will result in the background level
defaulting to the CALNICA model prediction.
Contents and Assumptions for Build 2
Build 2 will add two significant capabilities. First, frame registration by correlation will be
supported. This will enable mosaicing images obtained with gyro hold pointing control.
Second, source identification and iterative removal of sources for the purpose of robust
background level estimate will be included.
3. High Level Design
Inputs
CALNICB receives two forms of input: (1) The association table (ASN) contains the
names of each member of the association. This includes the names assigned by TRANS
and OPUS for the output file(s). (2) The individual calibrated exposures (CAL files) and
their supporting engineering data (SPT files) as FITS files with image extensions. Only the
name of the ASN table is required as an input to the task.
It also requires a calibration reference file containing the background illumination pattern.
Outputs
CALNICB generates two or three types of output. It will always create an output association table (ASC) containing updated pointing information and other association level data.
It will also always create an output image of the “target” field (MOS file). Some patterns
also result in observations of one or more (currently up to eight) offset pointings for measurements of the background. As these are distinct HST observations, each one requires a
mosaiced background image (MOS) to be generated. Appropriate logging information is
sent to the standard output for inclusion in the trailer file in OPUS and for display on the
user’s terminal within STSDAS.
3
Major Elements
CALNICB should contain three major processing modules incorporating each of its major
functions. These are: (1) image combination, including cosmic ray removal, (2) background subtraction, and (3) mosaic generation.
A high level outline of the task might look like this:
•
Get input ASN table, get images, perform initial checks
•
combine all NEXP exposures into single images
•
determine and subtract background estimate from target and background fields
•
combine (mosaic) each pointing into a single image
•
output ASC table, target and (if present) background images
Software Practices
CALNICB should follow the design principles used for CALNICA. Statistical errors, pixel
flagging, and pixel specific information (TIME and SAMP) must be handled and propagated. All image i/o should be accomplished with the “hstio” layer and all data should be
loaded into memory before processing starts. The existing NICMOS data structures must
be used and updated if necessary. Each algorithmic element should be a separate and distinct module to permit its replacement without modification to the other parts of the task.
The task is to be written in the “C” language and appropriate documentation for both software developers and end users must be created.
4. Algorithm Definitions
NEXP Combination
This module requires a conservative cosmic ray rejection (e.g. 5 pixel rejection) and simple combination of the images.
In Build 2 a capability to check and refine the alignment of each image will be added.
Mosaic Construction
For Build 1, this module will use the pointing information from the headers (i.e. the
planned motions of the telescope). For each non-overlapping pointing (by definition the
“chop” regions) an output image will be created. This image will have dimensions which
include all of the observed sky (i.e. it will typically be larger than the 256 x 256 pixel NICMOS array). Each image will be shifted using bi-linear interpolation onto a common
frame (established by the first image in the sequence). Then the image combine/cosmic
ray flag routine will be used to create the output image.
4
For Build 2 the image alignment will be more sophisticated. The module must determine
which regions overlap and use all available data in those regions (it will still create an output image for each chop region). The relative position of each exposure (of the NEXP
combined exposures) will be checked and updated by correlation. This will permit proper
combination of gyro hold pointing control observations.
Background Estimation and Removal
The background estimates come from two sources. First, the single frame estimates created by CALNICA (based on image averages and also a model of the expected
backgrounds). [These will be implemented in Build 3 of CALNICA.] Second, estimates
from the target and, if present, background images. For Builds 1 and 2 only a scalar background level is determined.
A simple priority scheme will be used for both Builds 1 and 2. If background images are
present, they are used to compute the background level. An average of these samples
(excluding outliers if more than 2 samples are present) will be used. If background images
are not present, then the estimate derived from the target image is used if it agrees (TBD
how closely) with the CALNICA model estimates. If it does not, then the model is used
with appropriate warnings.
For Build 1 the background estimates for each image are based on iterative sigma clipping
of the pixels in the image.
For Build 2 sources will be located (iteratively) and removed from the iterative sigma clipping average.
Cosmic ray flagging
For Build 1 cosmic ray rejection should be pixel based. The module should assume that it
is provided with frames aligned to the subpixel level (better than 0.5 pixels). The algorithm should flag pixels which exceed the ERR array estimate by TBD sigma. The module
should permit the flagging of the individual pixel, its four adjacent neighbors (“+” pattern), or all eight neighbors.
Image Combination
Build 1 image combination should assume that the images are aligned at the subpixel level
(<0.5 pixels) and should handle three cases:
•
2 images (average with exclusion of flagged pixels)
•
3 or 4 images (add exclusion of extreme outliers -- e.g. 5 sigma)
•
5 or more images (add iterative sigma clipping)
5
5. Possibilities for Further Development
Build 3 and beyond may include support for subsets of the available observing possibilities. One example is that the optimal methodologies for averaging frames varies
considerably with the number of frames available. Extension of CALNICB to handle cases
with large numbers (e.g. >10 or 20) of overlapping images may be desirable.
6