Instrument Science Report STIS 96-018
The STScI STIS Pipeline V: Cosmic Ray
Rejection
Stefi Baum, J. C. Hsu, Phil Hodge, and Harry Ferguson
July 22, 1996
ABSTRACT
In this ISR we describe calstis-2, the calstis calibration module which combines CRSPLIT
exposures to produce a single cosmic ray rejected image. Cosmic ray rejection in the STIS
pipeline will follow the same basic philosophy as does the STSDAS task crrej - a series of
separate CRSPLIT exposures are combined to produce a single summed image, where discrepant (different by some number of sigma from the guess value) are discarded in forming
the output image. The calstis pipeline is able to perform this cosmic ray rejection because
the individually commanded exposures are associated together into a single dataset by
TRANS and generic conversion. The crrej will also exist as a task in STSDAS to allow
users to reperform the cosmic ray rejection, altering the input parameters.
1. Introduction
In this ISR, we describe calstis-2, the portion of the STIS calibration pipeline which
processes multiple CRSPLIT exposures to produce a single cosmic-ray (CR) rejected output image. This ISR is organized as follows:
•
in “Motivation” on page 2 we describe the motivation for performing cosmic ray rejection in the pipeline,
•
in “Associations and CRSPLIT Data” on page 3 we briefly describe the association
and packaging of individually commanded CRSPLIT exposures by the ground system,
which allows the processing of these exposures as a unit through calstis,
•
in “The Place of calstis-2 in the STIS Pipeline” on page 3 we describe calstis-2 within
the context of the overall STIS calibration pipeline,
•
in “Calstis-2, Design of the CR-rejection Algorithm” on page 6, we describe the algorithmic details of the cosmic ray rejection process used in the calstis pipeline
1
•
in “Output of Calstis-2; the CR-rejected image” on page 9 we describe the output
product from calstis-2
•
in “The crrejtab.fits Calibration Reference File” on page 10 we describe the reference
file which controls the performance of calstis-2 in the pipeline, and
•
in “Crrej, the STSDAS Task” on page 10 we describe the requirements for calstis-2 to
run as a task in STSDAS.
2. Motivation
One of STIS’s three detectors is a CCD operating from the near-UV to the near-IR
(2500Å-11000Å). Long slit CCD spectroscopy is expected to comprise ~1/2 of the science
conducted with STIS, and the STIS CCD will likely be frequently used in parallel observations, either in imaging mode or with a first order grating to produce slitless
spectroscopic images of the sky. As with WF/PC and WFPC2, CCD images will be subject to cosmic rays which compromise the utility of single-exposure CCD observations.
The STIS Proposal Instructions support the taking of CRSPLIT CCD exposures; where a
single observing duration is broken into n (as specified by the CR-SPLIT=n optional
parameter) individual exposures. For STIS, these CRSPLIT exposures will all have
roughly the same exposure time (to within ~20%).
In the WFPC and WFPC2 calibration pipelines the individual CRSPLIT exposures are
passed through the pipeline individually and sent individually to the archive. GO’s and
archival users must extract the exposures and combine them to reject cosmic rays before a
scientifically (or cosmetically) useful image is produced. Ideally, the cosmic ray rejection
would be performed in the pipeline itself, rendering the products delivered to the user and
the archive more immediately useful. For STIS, the motivation to reject cosmic rays in the
pipeline and deliver cosmic-ray rejected calibration end products to the user and archive is
doubly strong; STIS spectroscopic data will not only be processed through traditional
CCD two-dimensional image reduction (calstis-1 which takes the data through flatfielding, see STIS ISR 95-07) but will also be spectroscopically calibrated, with 2-D and 1-D
spectral extractions performed on the data (see e.g., STIS ISR 95-06).
Goals of Pipeline CR-Rejection
The goal of cosmic-ray rejection in the pipeline is not to produce the optimum cosmic
ray rejected product for all observational and scientific scenarios. This is in fact not realistic within the confines of a pipeline. Rather the goal of cosmic-ray rejection in the pipeline
is to produce an output product which is significantly less compromised by cosmic rays
than the input product, which can be immediately used for scientific investigation in some
sizeable fraction (~>50%) of cases, and which significantly improves the value of the
quick-look products in all cases. We fully anticipate that a substantial fraction (~>20%) of
users will wish to re-perform the cosmic-ray rejection step on their data or will perform
2
additional CR-rejection (e.g., flagging residual cosmic rays based on shape) on the pipeline-delivered CR-rejected image. For this reason, the calstis pipeline also processes each
of the CRSPLIT exposures through calstis-1, outputting the rootname_flt.fits file, which
contains the original CRSPLIT exposures calibrated through flatfielding. In addition, the
calstis-2 task will be available as a IRAF/STSDAS task which users can run outside of the
pipeline environment, changing the input parameters to better optimize the rejection process for their own specific data and science aims.
3. Associations and CRSPLIT Data
Data taken using the CRSPLIT=n Phase II proposal logsheet optional parameter will be
associated together by TRANS (the software system which TRANSLATES the Phase II
proposal instructions into commandable units) for OPUS (see ISR STIS 96-107). During
generic conversion, OPUS will collect the individually commanded but associated
CRPSLIT exposures and package them together into a single dataset, with the science
data in a single fits image extension file. The individual exposures populate successive
image extensions in the file. Each extension will have its own image extension header with
exposure specific header keywords. The primary header file contains keywords describing
the total observation set contained in the file and the extension headers contains exposure
specific information. For example, the TEXPTIME primary header keyword gives the total
exposure time in all the CRSPLIT exposures, while the EXPTIME image extension header
keyword gives the exposure time in the specific single exposure in its extension. The
CRSPLIT primary header keyword indicates the number of CRSPLIT exposures in the file.
The CRCORR calibration switch header keyword (indicating that a cosmic-ray rejected
image should be produced) will be set to PERFORM whenever CRSPLIT>1.
4. The Place of calstis-2 in the STIS Pipeline
As described in STIS ISR 95-006, calstis consists of a large number of individual executable elements, dubbed calstis-1 through calstis-10. The CR-rejection element of calstis
is dubbed calstis-2. Calstis-2 will be executed if the CRCORR header keyword is equal to
PERFORM.
Figure 1 shows the flow of CRSPLIT data through the calstis pipeline. We have elected
to perform the combination of the individual CRSPLIT exposures into a single cosmic-ray
rejected frame early in the flow. The cosmic ray rejection is performed after each exposure
has had its data quality file initialized and the overscan bias level subtraction (BLEVCORR)
performed upon it, but prior to subtraction of a bias frame (BIASCORR), dark (DARKCORR) and flatfielding of the data (FLATCORR). The CR-rejected, bias level subtracted
image is then passed through the remainder of the two-dimensional image reduction (calstis-1) to produce a flat-fielded CR-rejected image (rootname_crj.fits). This CR-rejected
flat-fielded image is then passed through the subsequent processing steps in calstis: geo-
3
metric correction for imaging data and 2-D and 1-D spectroscopic extraction and
calibration for spectroscopic data.
We have elected to place the CR-rejection step early in the flow for two reasons. First,
this is the optimal place to perform the CR-rejection, as the data values (DN) have not yet
been scaled by the flat field response. If the flat fields exhibit the effects of vignetting or
other strong large scale spatial variations, then performing CR-rejection following flat
fielding exhibits poorer results; the flat fielding increases the dispersion of the DN values
in regions of low flat field response reducing the effectiveness of the CR-rejection algorithm. Second, by performing the CR-rejection prior to the bulk of two-dimensional image
reduction, we assure that the errors produced in the output file are correct. If we did the
CR-rejection following flat fielding, the errors calculated as the individual exposure errors
added in quadrature would be incorrect, as they would incorrectly scale the errors contributed by the calibration reference files. Calibration contributes to the errors in a non-linear
fashion and those errors cannot be separated from the Poisson errors due to counting statistics following calibration.
If the EXPSCORR calibration switch in the header is set to perform, the pipeline will
also process each of the rootname_raw.fits file with the CRSPLIT individual exposures
through calstis-1, outputting an intermediate t product, rootname_flt.fits, containing the
flatfielded CRSPLIT exposures in successive extensions of a single file. This file will not be
passed through the subsequent calibration steps (e.g., spectroscopic reduction), but will be
retained as an intermediate data product, to allow users to examine the affects of the pipeline cosmic-ray rejection.
4
Figure 1: Flow of CRSPLIT data through calstis
rootname_raw.fits
DQCORR
BLEVCORR
(initialize data quality,
subtract overscan derived
bias level)
calstis-1a
EXPCORR
(process individual exposures as well)
CRCORR
(reject cosmic rays)
calstis-2
rootname_raw.fits
calstis-1
(data quality mask flagged
for cosmic rays)
BIASCORR
DARKCORR
FLATCORR
(PHOTCORR)
rootname_flt.fits
calstis-1
(flat fielded CRSPLIT
exposures, in image extensions)
(perform 2-dimensional
image reduction)
rootname_crj.fits
(flat fielded, cosmic ray
rejected image, err and dq)
spectroscopic
reduction
rootname_x2d.fits
rootname_x1d.fits
two and one dimensional
spectroscopically extracted
and calibrated data
rootname_geo.fits
(geometrically corrected
imaging data)
or
geometric
correction for
imaging data
5
5. Calstis-2, Design of the CR-rejection Algorithm
To reject cosmic rays, calstis-2 will utilize the basic method employed in the earlier
STSDAS task crrej. As described in the on-line STSDAS help to this task, the basic philosophy is to take multiple exposures of the same field and combine the images by
rejecting very high counts in each pixel stack. STIS-crrej will differ from the current version of crrej in that (1) it will be written in C and utilize the HSTIO package, (2) it will
utilize the input data quality flags to discard pixels from the input images when forming
the output image, (3) it will output an error array for the CR-rejected image, utilizing the
input error arrays.
Calstis-2, as envisioned for the calstis pipeline will do the following specific things:
•
form a stack of images to be combined,
•
form an initial guess image,
•
form a summed ‘CR-rejected’ image, using the guess image to reject high and low values in the stack,
•
iterate, using different (usually decreasing) statistical criterion, producing a new guess
image, and new summed image,
•
output final CR-rejected image, including science, data quality and error extensions
•
flag the data quality arrays of the input files, to indicate where a cosmic ray has been
found.
The algorithmic details of each step are summarized in the subsections below.
Input parameters are read from both a calibration reference file (crrejtab.fits) and from
the primary header of the input science file (rootname_flt.fits).
The parameters and keywords to be read are summarized in Table Table 1 and Table 2.
6
Table 1. Header Keywords to be Read by Calstis-x
Keyword
meaning
populated by
location
READNSE
read noise in DN
calstis-1
read from reference file
primary header
ATODGAIN
calibrated gain
DN=counts/ATODGAIN
calstis-1
read from reference file
primary header
TEXPTIME
total exposure time in
observation set (seconds)
generic conversion
primary header
EXPTIME
exposure time in each
exposure (seconds)
generic conversion
extension
header
Table 2.Parameters controlling Calstis-2, read from crrejtab.fits,
parameter
purpose
valuetype
example
scalenoise
controls multiplicative contribution to noise
string
“2.0”
initial
controls how initial guess image is produced
string
“min” or
“med’’
sky
controls whether a sky value is subtracted
string
“none” or
“mode”
sigmas
controls statistical rejection criteria of cosmic rays in
each iteration
string
“5,4”
radius
rejection propagation radius (in pixels) for rejecting
neighboring pixels
real
1.5
pfactor
controls threshold for rejecting neighboring pixels
in each iteration
real
0.5
badinputdq
data quality flag bits to reject from input stack; if set
to zero, omit this step
16 bit int
4
mask
controls whether or not data quality flags in input files
are set for cosmic rays found by calstis-2
string
perform or
omit
Form a stack of images to be combined
This step performs a simple input of the science data and data quality flags for each
image into a stack. The science values are scaled to the total exposure time (read from the
TEXPTIME primary header keyword) of the CRSPLIT series on input. The EXPTIME
extension header keyword gives the exposure time of each exposure in the stack.
7
If sky=mode, a sky value is calculated as the mode in all pixels for each input image
and subtracted from each pixel value prior to placement in the stack. If sky = “none”, no
background is subtracted.
Form an initial guess image.
The initial guess image can either be constructed as the median value in the stack at
each pixel. Only input pixels whose baddqinput data quality flags are not set (determined by the performance of a bitwise logical AND between baddqinput and the input
pixel DQ flag) are used to form the guess. This capability is needed to reject pixels compromised from bad data in a single input image from contaminating the output product.
The algorithm to be used in forming the guess image is controlled by the parameter
initial, which can have values “med’’ (median), or “min” (minimum).
Compute an average value for each pixel from the stack, flagging discrepant values
A CR-rejected ‘summed’ output science image is created, by setting the value at each
pixel to the average values of all good pixels in the stack whose values are within plus or
minus sigmas*noise of the initial guess image. Deviant (out of range) stack pixels are
flagged as cosmic ray impacted by setting their stack data quality flags to 8192.
The value of noise (in DN) is computed as:
noise= SQRT[(READNSE/ATODGAIN)2 + (DN/ATODGAIN) + (scalenoise*0.01*DN)2]
where,
•
DN = the data number of the stack pixel value,
•
READNSE is the read noise in electrons, read from the primary header,
•
ATODGAIN is the calibrated conversion from electrons to DN, read from the primary
header, and
•
scalenoise is an input parameter, read from the crrejtab.fits calibration reference
file.
The sigmas parameter is read from the crrejtab.fits calibration reference file. Sigma
is a string, e.g., sigmas = “4,3”. The number of entries in the string dictates the number
of iterations to be performed (in this example two) and the values in the string indicate the
value of sigmas for each iteration. In this example in the first iteration good stack values
greater than the guess image value plus 4*noise or less than the guess image value minus
4*noise are considered to be outliers and excluded from the average on the first iteration.
A second iteration is then performed in which sigmas is set to 3 and good stack values
disparate by more than +/- 3*noise from the guess image are excluded when determining
the average. If radius is >=1, then pixels neighboring rejected pixels are also excluded
8
in forming the average as described in “Rejection Propagation to Neighboring Pixels” on
page 9 below.
Scalenoise is a string. If scalenoise = “2.0”, then the term 0.02*value is added
to the noise. Experience has shown that a multiplicative factor such as scalenoise is
important to include in forming the noise in order to avoid flagging the centers of stars, as
jitters from the telescope may slightly change the pointing from image to image.
Rejection Propagation to Neighboring Pixels
Pixels next to outliers can also be flagged as cosmic ray impacted and excluded from
use in determining the output CR-free summed image. Rejection of neighboring pixels is
controlled by the radius and pfactor parameters. Radius is a real number in the
unit of pixels; if radius=1.0 a total of 5 pixels will be considered for rejection in a “+”
shaped pattern where the central pixel is the pixel originally flagged as an outlier. If radius
= 1.5, a total of 9 pixels will be considered for rejection in a square centered on the originally flagged pixel. The neighboring pixels are discarded if their values are discrepant
from the guess image by more than pfactor*sigmas*noise, where pfactor is a
real. If pfactor = 0., then all n pixels are immediately rejected.
6. Output of Calstis-2; the CR-rejected image
The output of calstis-2 is the rootname_crj.fits file. This file has a primary header, and
3 image extensions; the science, error and data quality images.
Science Extension
The output science image is constructed as the sum of all the good pixels in the stack,
where the value at each output image pixel is scaled by the ratio of input to output exposure times. For example, if CRSPLIT=3 and TEXPTIME = 180, and each extension had
EXPTIME=60, then if for a given output pixel, one of the input pixels was flagged as bad,
then the summed value would be scaled by 3/2. The EXPTIME extension header keyword
is set to the value of TEXPTIME in the rootname_crj.fits file. A block of header keywords
(Cosmic Ray Rejection Parameters) is created in the output file, indicating the parameters
used to control the cosmic ray rejection process.
Error Extension
Errors for the output file are calculated as the errors from all stack pixels which contribute to the SCI extension, added in quadrature.
9
Data Quality Extension
The output data quality file is formed as the bitwise OR of all stack data quality flags
for stack pixels which contributed to the SCI extension. Note that no stack pixel flagged as
a cosmic ray will be in the output SCI extension, thus there should be no pixels in the output data quality extension flagged as cosmic ray impacted.
Flagging of Data Quality Flags in the Input images
If mask is set to yes, calstis-2 will also flag the data quality values in the input
(rootname_raw.fits) file, to indicate pixels determined by crrej to be cosmic-ray impacted.
If mask=yes the CR-impacted bit will be set either on or off in the input file i.e., if the
pixels was not flagged as cosmic ray impacted, then the corresponding flag (8192) will be
set off, if it was flagged as a cosmic ray the bit will be set on. This resetting of the bit is
important, as it allows people to run calstis-2 (or stis-crrej) multiple times on the same
input file.
We will set mask to yes in the pipeline. Thus the data quality extensions in the
rootname_flt.fits file (produced from the rootname_raw.fits file following the performance
of calstis-2, see the data flow in Figure 1), will be flagged to indicate pixels identified as
being cosmic ray impacted.
7. The crrejtab.fits Calibration Reference File
The crrejtab.fits calibration reference file contains the parameters which control the
flow of calstis-2. The values of those parameters may be set differently depending on the
number of CRSPLIT input images and depending on the exposure time of each image. The
columns of the crrejtab.fits reference file are summarized in Table 3 below. Calstis-2
should select the appropriate row of this table based first on the value of CRSPLIT, and secondly on the value of TEXPTIME/CRSPLIT. Specifically calstis-2 should find all rows
with an exact match on crsplit, and then select the row in which the value of
mean_exptime is closest to but still greater than TEXPTIME/CRSPLIT.
Table 3. Columns of the crrjectab.fits Reference File
crsplit
mean_exptime
scalenoise
initial
sky
sigmas
radius
pfactor
badinputdq
mask
8. Crrej, the STSDAS Task
Calstis-2 must be available also as an STSDAS task to allow users to reperform the
cosmic ray rejection, optimizing the controlling parameters for their particular science
10
goals. When run as an STSDAS task, the parameters controlled by the crrejctab.fits calibration reference file, will be input parameters. In addition to these, to maintain the
general applicability of the crrej task, values read from header keywords (e.g.,
READNSE,ATODGAIN, keywords specifying exposure time) should be specifiable as
input parameters, by the user. Ideally, the current crrej task as the calstis-2/crrej task
described here (which access data in different formats) would be a single task.
11