Instrument Science Report ACS 2002-10
ACS software tool development
W.B. Sparks, W.J. Hack, R.N. Hook, A. Koekemoer
December 16, 2002
ABSTRACT
We describe the anticipated software tool development requirements for ACS for the
period September 2002 to mid-2003. The most substantial area for development is in
issues related to image registration and combination with appropriate handling of data
quality information (especially cosmic rays and hot pixels).
Software Tools Overview
Software tools are required in support of an HST instrument in order to (i) simulate observations with the instrument (ii) access, visualize and manipulate data arrays (iii) provide
calibration reference files in support of the calibration pipeline for the instrument (iv) provide the calibration of the instrument, i.e. removal of the instrument specific signatures
from the data, through the calibration pipeline itself and on-the-fly reprocessing (OTFR)
and in stand-alone mode for more complex operations (v) provide analysis tools that
enhance the scientific usefulness of the data and allow scientific measurements to be
made. Here we outline software development plans for ACS, building on the work
described in ACS ISRs 01-10, 01-04 and 00-3 (Sparks et al). The ACS pipeline operations
are described in ACS ISRs 99-03 (Hack), 99-04 (Mutchler et al) and 99-08 (Hack).
The previously described software development, see above ISRs, has reached sufficient
maturity that we are now in a position to consider implementation of the upgrade path
described therein. A large fraction of the elements of that upgrade path for image combination and data quality handling are captured in the task “MultiDrizzle” written by
Koekemoer (Koekemoer et al 2002, Sparks 2002). This task uses PyDrizzle at its core and
allows iteration to handle cosmic rays more effectively than a single pass. The script is
being used by the GOODS team and others to process their ACS data.
Copyright© 1999 The Association of Universities for Research in Astronomy, Inc. All Rights Reserved.
Instrument Science Report ACS 2002-10
Given our need for robustness in user software and software support, as well as resource
practicalities, we envisage a staged, initially limited, implementation of the MultiDrizzle
functionality for the ACS pipeline.
In this document, we also describe a variety of utilities allowing better use of context
images and tools to transform between sky, raw image and geometrically corrected image
coordinates. Additional tasks are outlined that will allow flux calibration of ramp filter
observations and reference file generation support utilities.
The degree to which these are implemented will depend on resource allocations.
PyDrizzle; calacs; MultiDrizzle
All ACS data on retrieval from the archive or when first acquired are processed through
the calibration pipeline calacs and the geometric distortion correction/image combination
task PyDrizzle. It is likely that retrievals and OTFR will dominate over “first run” processing in terms of the demands on computing resources. Users should in general use OTFR
since time dependent calibrations typically are delayed by at least two weeks compared to
the observations.
The baseline version of calacs is 4.1c, and for PyDrizzle is currently version 3.3d installed
in OPUS 14.1. This version correctly handles subarrays and fourth order geometric distortion coefficients. For single images, the current calacs+PyDrizzle processing makes no
attempt to identify cosmic rays, but the data are debiassed, dark subtracted and flat-fielded
by calacs and corrected for geometric distortion using PyDrizzle. For “associated” data
(CR-SPLITs, PATTERNs) cosmic rays are removed using acsrej if the data have been CRSPLIT, otherwise no attempt is made to identify cosmic rays. Acsrej is run prior to dark
subtraction and flat-fielding. PyDrizzle then combines the reduced images if they are associated, simultaneously correcting for geometric distortion.
Clearly a major deficiency to this approach is the limited degree to which cosmic rays may
be removed. In stand-alone mode it is usual to iterate on the image combination, firstly
making a median (or similar) image combination to eliminate cosmic rays, to transform
back onto the raw data frame, flag cosmic rays in the original images, and then re-combine
the data this time rejecting pixels affected by cosmic rays hits. At the same time, adjustments are made for pointing mismatches and background sky offsets. A script,
MultiDrizzle, has been written by A. Koekemoer to carry out this process, based around
the use of PyDrizzle together with other tasks from the stsdas dither package. The resulting data are of significantly superior quality to a single run of calacs+PyDrizzle.
The current status of drizzle related software support for ACS is summarized at http://
www.stsci.edu/hst/acs/analysis/drizzle, which includes a link to the MultiDrizzle documentation and software.
Our proposal is to use MultiDrizzle as a proof of concept and template for review and
study by SSG and the ACS group members, with a view to incorporating some or all of its
functionality in the ACS pipeline. Irrespective of pipeline implementation, we would like
2
Instrument Science Report ACS 2002-10
to have some or all of the MultiDrizzle functionality available and supported for standalone use by ACS observers. To facilitate stand-alone reprocessing at the “PyDrizzle”
stage, we require the calacs flag EXPSCORR to be set to PERFORM. This then carries
out calacs processing on all individual images, optimizes the effectiveness of OTFR, and
avoids the need for users to download all reference files should they need to reprocess
their data.
The MultiDrizzle task can carry out the following steps, either in a single pass or alternatively by selecting various steps individually:
1. StaticMask
- Identifies negative bad pixels, based on examining all the images,
and includes them in the dq file
2. SkySub
- Sky-subtracts each frame
3. Single-Image CR - Carries out CR rejection on each input image separately,
using Ferguson’s "crsingle.py" script.
4. Driz_Separate
- Drizzle the input images onto separate, registered
outputs (using shifts computed from the headers)
4a. Tweakshift1
- Optionally refine the shifts using SExtractor catalogs,
working on the separate drizzled output images
5. Median
6. Blot
7. Driz_cr
- Create a median image from the separate drizzled images
- Blots the median image back to the original input frames
- Uses the blotted image, creates a derivative image,
and computes CR masks
7a. Tweakshift2a,b - Optionally refine the shifts using SExtractor catalogs,
comparing each input image with its blotted median
8. Driz_Combine
- Does the final drizzle combination
Generically, that is, the tasks required include:
image registration
object identification
catalogs
background level determination
3
Instrument Science Report ACS 2002-10
single image PyDrizzle, even for associated data
median combine or minimum combine
blot to raw datasets
single image CR identification and DQ file update
re-process with PyDrizzle.
To enable such processing it will be necessary to set EXPSCORR to PERFORM so that
each raw image, even in an association, is processed individually to the flat-fielding stage.
It is not currently possible to generate association tables on-the-fly as part of the archive
interface. It is possible to do so, however, in stand-alone mode. Hence the MultiDrizzle
approach becomes vastly more general since all relevant data on a target may be assembled and processed in a straightforward fashion. The task BuildAsn already exists as a
support utility to do this in pyraf.
Subject to the review described below, we anticipate a conservative, robust implementation of the MultiDrizzle functionality in stages. The highest priority is to provide a cosmic
ray rejection capability, hence a strawman initial implementation is to defer pointing
issues (for which we feel additional experience on the best approach is needed, and which
have administrative concerns such as software licensing) until a later stage. We propose to
proceed as quickly as possible with a basic but robust cosmic ray identification loop
assuming the header information is accurate (for the pipeline), and noting that in standalone mode, PyDrizzle can recognize pointing corrections supplied as part of an association table.
Drizzle Kernel Enhancements
The MultiDrizzle script invokes PyDrizzle which in turn uses the stsdas drizzle task
(Fruchter & Hook 2002) to actually combine images. In addition to the developments of
the framework the core drizzle task is also undergoing some enhancements which are
briefly noted here.
Drizzle was developed for the combination of sets of dithered, geometrically distorted
images. For this purpose the "shrunken pixel" approach was close to ideal. However, drizzle is also applied to other applications, such as the geometric correction of single images.
In this case the standard drizzle approach (with a "pixfrac" of 1.0) is similar to linear interpolation and not optimum. The drizzle "kernel" controls how the weight of an input pixel
is distributed on the output image and is analogous to an interpolation kernel.
We are therefore experimenting with kernels such as the Lanczos (a damped sinc) which
are recommended in the image processing literature for the interpolation of marginally
sampled images. Experiments show that this kernel results in images with a sharper PSF
and lower level of apparent noise correlation, although the negative excursions of this ker-
4
Instrument Science Report ACS 2002-10
nel inevitably cause artifacts around features such as cosmic rays. Despite some
limitations these new kernels may offer ways of better retaining the exquisite image quality obtainable with ACS.
Schedule
October 2002: Delivery of prototype MultiDrizzle code to Hack/Hook/ACS group to
begin study/review
November/December 2002: Code review; design strategy for potential pipeline compatibility (SSG)
Requirements for pipeline operation (ACS instrument scientists)
Decision whether to proceed with pipeline/OTFR implementation
December 2002: “stop gap” pilot script to be made available for community use in standalone mode
Approx May 2003: for coding, testing and delivery if “yes”; stand-alone implementation
of required new software in either case (essentially, to move away from reliance on
scripts).
Ongoing 2002/2003: develop experience and improved understanding of approaches to
refining STScI supported software, such as pointing information and image registration.
Documentation is an ongoing and crucial requirement throughout and requires resource
allocation.
Support utilities
The following support utilities have been requested and are, for the most part, selfexplanatory.
pixel area map generator to provide geometric area of raw data pixels.
skytoxy (RA/dec to pixel coordinate transformation on either the geometrically corrected
image or the raw image)
xytoxy (pixel coordinate transformation between the geometrically corrected image and
the raw image)
exposure time image generator, constructed from the context image
variance image generator for the final drizzled data, according to selected algorithm
acslook to provide quick look at multiextension data
drizzle <---> IDCTAB conversion utility to allow drizzle convention to be put into IDTAB
format or vice versa.
Additional calacs/PyDrizzle Upgrades and Studies
Velocity aberration correction. It has been found that differential velocity aberration can
cause small image shifts which in turn cause CR-SPLITs to be mis-registered. The acsrej
processing can fail in consequence. The problem is entirely deterministic, involves essen-
5
Instrument Science Report ACS 2002-10
tially only a small scale change on the input image, and so may by “allowed for” in
PyDrizzle.
Polarizer IDCTAB recognition. The polarizing elements in ACS have optical power and
introduce small image shifts. Separate IDCTABs are now available for the polarizers.
CDBS needs minor modification to allow their use in the pipeline. This capability may
prove useful for correcting other individual filter differences.
Offset flat fields. Keywords are being introduced to provide the filter wheel position used
during an observation (as opposed to commanded which can be off slightly). Calacs could
be modified to take advantage of this keyword if offset flats are available.
Potential feedback from King-Anderson analysis. It is possible that the improved geometric analysis of King & Anderson may lead to modifications to PyDrizzle processing for
ACS data.
Overhaul of DQ handling. See TIR in draft by Sparks, Hack and Van Orsow.
The longer term upgrade path in this area will depend heavily on hands-on users experience. In addition to the elements of MultiDrizzle, we anticipate input in at least the
following areas: object cataloging; alternates to core routine of drizzle; optimized parameter settings for drizzle and single image cosmic ray identification.
Existing software maintenance and contingency
With the advent of actual ACS data, tasks that have already been completed will potentially be tested much more rigorously by external users. Debugging and maintenance of
such tasks, especially calacs, may be anticipated as part of the contingency allowance.
Contingency time must be allowed for, since ACS is a relatively new instrument. Unforeseen developments lead to new, unanticipated requirements in the area of software
support. Hands-on experience with real data may change priorities and introduce new
needs, potentially at short notice and potentially as a matter of urgency.
Reference File Generation Tools and CTE Analysis
ACS requires time-dependent reference file generation. In particular, bad pixels come and
go, as for other CCDs on HST. Development of tools to generate appropriate reference
files as automatically as possible is desirable, following an initiative by the STIS group to
do this.
6
Instrument Science Report ACS 2002-10
Degradation of CTE may eventually lead to time dependent reference files. The observed
CTE degradation of WFPC2 and STIS together with the large format of the ACS CCDs
has given rise to great concern over the rate of CTE decline in ACS, particularly the WFC.
Hence, CTE monitoring will form an important part of the ACS calibration plan.
Analysis tools to facilitate that monitoring will be helpful once the program is under way.
There are currently two options: (i) Mike Jones has designed an engineering program first
pixel response (FPR) test to estimate CTE (ii) Also, Riess WFPC2 ISR 99-04 describes a
technique to measure CTE from cosmic rays in dark frames. The ACS group will utilize
these techniques to monitor CTE and, if possible, automate the measurement process
given experience with on-orbit data. Such a process may become straightforward with an
application tool.
Ramp filter calibration
Conceptually, the process of calibrating ramp filter observations may be envisaged as twostages in which firstly, the data are flat-fielded in such a way as to enable continuum subtraction to be carried out, and secondly, the resulting monochromatic data are fluxcalibrated to correct for the changing transmission, or sensitivity, around the field of view,
knowing (or assuming) the wavelength of the monochromatic data. These tools have not
been implemented for WFPC2 for which a point-source treatment was adequate, however
the ramp filter field of view of ACS is much wider, about 80x40 arcsec, and hence tools to
correct and manipulate the full area of an ACS ramp filter observation are desirable for the
data analysis stage
Ramp filter flat-fielding
Ramp filter P-flats (pixel-to-pixel changes) were constructed using nearest-filter full-field
flat fields. Given L-flat sensitivity changes across the field of view, a utility to scale the
continuum emission according to the known sensitivity curves may be needed. Investigation is required to determine whether automatic scaling is sufficiently accurate or
necessary, and whether software tools are needed in this area.
Flux calibration of ramp filter data
An extended monochromatic source image is the expected result of a typical ramp filter
observation following continuum subtraction. Continuum subtraction is a specialized
operation, so we do not envisage including it in the pipeline. Hence, we cannot include the
subsequent step of flux calibration in the pipeline over an extended area. We do expect to
provide flux calibration keywords that will be valid for the specified wavelength of the
observation at the center of the field. Additionally, to allow access to the full two-dimensional character of these observations, a stand-alone utility interfacing with the synphot
library of transmission curves will enable a complete areal flux calibration to be carried
out.
7
Instrument Science Report ACS 2002-10
Item: generate inverse sensitivity maps for ramp filter observations for (i) fixed wavelength (ii) default central wavelength (iii) an input wavelength map.
Other ACS software utilities
Polarization Analysis
Integrate Sparks IDL code based on Sparks & Axon 1999 PASP 111, 1298 procedure,
with ST-ECF iraf software (Walsh).
Tools for coronagraphic analysis
ACS has a powerful coronagraphic capability that will allow faint source to be seen very
close to bright stars and quasars. In order to process data of this form, we will require techniques to determine azimuthal profiles using a variety of methods (mean, median, mode
for example), plus construction of model profiles utilizing the azimuthal averaging
information.
New techniques involving the wavelength dependence of the coronagraphic PSF may
come into play as described in Sparks & Ford 2002 ApJ, 578, 543. Supporting software to
process resulting coronagraphic datacubes may eventually be required however it is not
anticipated for the current cycle.
Synphot
Ensure accurate and complete functionality of ACS synphot software which is needed by
both calacs and the exposure time calculator. The ETC will transition to the APT version
during this year.
PSF modelling
Validate and maintain the Tiny Tim implementation for ACS (Krist).
Extracted spectra for grisms: aXe
ST-ECF have developed grism spectra analysis tools, and have assumed formal responsibility for support of the software they provide. Some discussion of incorporating this in
the ACS pipeline has taken place. SSG support will be needed to assist ST-ECF in installing and integrating the software they are to provide within STScI systems, and eventually
possibly the ACS pipeline.
ST-ECF are working the following items:
* implement weighted extraction
* allow summing of multiple extraction tables from different exposures
* script optimal combination of dithered image-grism pairs, perhaps
using MultiDrizzle
8
Instrument Science Report ACS 2002-10
References
Fruchter, A.S., Hook, R.N., 2002, PASP, 114, 144.
Hack, W.; ACS ISR 99-03 CALACS Operation and Implementation
Hack, W.; ACS ISR 99-08 CALACS Reference Files
Koekemoer, A., Fruchter, A., Hook, R., 2002; HST Calibration Workshop, “MultiDrizzle:
An Integrated Pyraf Script for Registering, Cleaning and Combining Images”
Mutchler, M., Jedrzejewski, R., Cox, C.; ACS ISR 99-04 ACS calibration pipeline testing:
basic image reduction
Sparks et al ACS ISR 00-03 Software tools for ACS: Geometrical Issues and Overall Software Tool Development
Sparks, W.B., Hack, W., Hook, R.N.; ACS ISR 01-04 Inital Implementation Strategy for
Drizzle with ACS
Sparks, W.B., Hack, W.J., Clampin, M.; ACS ISR 01-10 ACS Software Tool Developmen
Sparks, W.B., 2002; HST Calibration Workshop, “ACS Calibration Software”
9
Instrument Science Report ACS 2002-10
Summary
Item
Timescale
Priority
Pipeline
Simplified MultiDrizzle functionality
(without tweakshift)
individual image PyDrizzle
blot
driz_cr
Dec 2002
1
no
Simplified MultiDrizzle functionality
for potential pipeline use
May 2003
1
yes
xytosky, skytoxy, xytoxy convert x,y to RA,Dec;
pixel area map
Cycle 11
2
no
drizzle support utilities:
exposure time map
pixel area map
variance image
Cycle 11
calacs and other software maintenance post-launch
Cycle 11
1
yes
Velocity aberration correction
Cycle 11
1
yes
Individual IDCTAB/filter recognition
Cycle 11
2
yes
Cycle 11/12
3
yes
Nov/Dec 2002
3
yes
Offset flat-fields
data quality procedures for calacs
no
1
1
2
ACS quick display
2
reference file generation tools
Cycle 11
3
yes
CTE analysis tools
Cycle 12
3
no
Ramp filter calibration
L-flat procedures
Flux calibration
Cycle 11
4
2
no
coronagraphic analysis
Cycle 12
4
no
Contingency time for instrument anomaly
1
ACS ETC / APT / synphot
1
PSF modelling
1
no (Krist)
Extracted spectra for grisms
3
(ST-ECF)
10