Instrument Science Report WFPC2 2001-001
Creating WFPC2 Dark
Reference Files
J. Mack, M.S. Wiggs
January 4, 2001
ABSTRACT
In this ISR, we describe the procedures for making WFPC2 dark reference files for the
HST data reduction pipeline. These calculations incorporate several IRAF scripts which
are linked by the newly developed task “wkdark”. These scripts are currently available for
FTP from the WFPC2 website to users who wish to create their own dark reference files.
1. Introduction
We present IRAF scripts written by the WFPC2 group for creating dark reference files to
be used in the HST calibration pipeline. In Section 2, we describe how to obtain the necessary files and how to set up the scripts in your IRAF account. In Section 3, we present
the criteria for retrieving darks from the HST Data Archive, and in Section 4 we describe
how to prepare the images for processing. The procedure for running the task wkdark and
for setting the necessary parameters is described in Section 5. The last section gives a
detailed description of the individual scripts linked together by wkdark and of the calculations performed in each script.
2. Setting Up Your IRAF Account
All the necessary files for making dark reference files are available for FTP from the
WFPC2 Dark Reference File Website:
http://www.stsci.edu/instruments/wfpc2/Wfpc2_memos/wfpc2_darks_tab.html
Copyright© 1999 The Association of Universities for Research in Astronomy, Inc. All Rights Reserved.
Instrument Science Report WFPC2 2001-001
The FTP file contains several cl scripts, text files, and images and requires about 40 MB of
disk space. These files should be placed together in a single directory before starting.
Among these files is the script “mksuper.cl” which sets up the package mksuper in IRAF.
The task wkdark links together the individual scripts used in processing which are listed
below:
prompt> more mksuper.cl
package mksuper
set mksuper = /mydir/darks/
task wkdark
= mksuper$wkdark.cl
task superimg = mksuper$superimg.cl
task superdqf = mksuper$superdqf.cl
task dowfstat = mksuper$dowfstat.cl
task dohedit
= mksuper$dohedit.cl
task doheader = mksuper$doheader.cl
task dodqfqs = mksuper$dodqfqs.cl
task calcheck = mksuper$calcheck.cl
task wfarith
= mksuper$wfarith.cl
clbye()
You will need to edit the second line: set mksuper = /.../... to reflect the directory into which
you have FTP’d the above files. Then, add a line in your loginuser.cl file telling IRAF
where this package is located.
task $mksuper= "/mydir/darks/mksuper.cl"
When you start up IRAF, you will need to load the mksuper package before you can
access the tasks defined above. The only task you will actually run is wkdark, since all the
others are called from this script.
cl> mksuper
calcheck
dodqfqs
doheader dowfstat
dohedit
superdqf
superimg
wfarith
wkdark
3. Retrieving Files from the Archive
You now want to get a set of weekly dark images from the archive. To do this, you must
use the StarView software and enter the relevant search parameters. These steps are outlined below.
2
Instrument Science Report WFPC2 2001-001
Under the Searches menu in StarView, select
New Format Screens
Instruments
WFPC2
WFPC2 Instrument
Then set the following search criteria:
IMAGE TYPE = "DARK"
SERIALS
= "OFF"
EXPOSURE = "1800"
START TIME = "mm/dd/yy..mm/dd/yy"
Then hit Begin Search and mark the images you want to retrieve..
To set the field START TIME, choose the range of dates you wish to examine using the format in the example above. If you are instead interested in looking at all the darks taken
after a certain date, you would use the format “>mm/dd/yy”. When submitting your
request to the archive, you need only retrieve the calibrated datasets. To do this, select
either “OTFC” or “OTFR” for the data type requested.
There are two types of weekly dark monitoring programs, each with their own prososal ID:
one is a regular monitoring program and the other takes darks specifically just before and
just after a decontamination. There are usually five dark images for each weekly monitoring program. These individual images will be combined to make the weekly dark
reference file. Keep this naming convention in mind, “dark image” vs “dark file”, throughout this ISR.
A separate monitoring program takes darks on a daily basis. If you decide to work with
dark images from the daily dark monitoring program instead, these are taken in groups of
three and have shorter exposure times (1000 sec) than the weekly dark images (1800 sec).
To achieve the same exposure time as the weekly dark reference file, you would thus need
9*1000 sec exposures to equal 5*1800 sec exposures. (The readnoise will be slightly
higher when using 9 images, though is not a significant increase compared with the dark
count rate.). If you want to attain the same signal-to-noise level per pixel as the weekly
dark files, you would need to use 15 daily dark images. When selecting these images, try
to keep them as close as possible to the date of your observations and be sure the images
you select are all taken within the same decontamination cycle.(see WFPC2 Decontamination Dates on the Web).
http://www.stsci.edu/instruments/wfpc2/Wfpc2_memos/wfpc2_decon_dates.html
3
Instrument Science Report WFPC2 2001-001
In addition to the individual dark images, you will need to retrieve the dark reference file
created for the week just prior to the date of your observations. This information is posted
on the WFPC2 Reference Files List website from page 1.
To retrieve this dataset, you must enter the file rootname in StarView:
Under the View menu, select
Retrieval
Add Datasets
By Hand
Then enter the rootname of the reference file corresponding to Gain=7, Serials=Off. This
will be the third of the four files listed on the web for each weekly dark program. When
submitting your request to the archive for this file, you should retrieve the calibrated
datasets only.
Note: You must create a separate directory for each set of weekly darks you are working
on and you must ONLY use dark images which were taken within the same decontamination cycle. You should place the previous week’s dark reference file in this same directory.
This directory must be different from the one containing the IRAF scripts and other files.
4. Preparing The Dark Images
The images will arrive from the archive in “fits” waiver format. These must first be converted to “geis” format with the task tables.fitsio.strfits. The only datasets you will need
to use are the rootname_c0f.fits images which are then converted to rootname.c0h and
rootname.c0d using the following command:
cl> strfits fits_files="*_c0f.fits" file_list="" iraf_files=""
For the previous weekly dark file, the archive will deliver the files rootname_b3f.fits and
rootname_r3f.fits. To convert these files to the appropriate format, type:
cl> strfits fits_files="*_?3f.fits" file_list="" iraf_files=""
You should delete all the files in this directory with the “fits” ending before you run
wkdark.
Next, display each group of all five weekly dark images and mask out "bad" areas using
the task stlocal.testwfpc.uimedit. This task is identical to images.tv.imedit, except it
works on the multi-group format of WFPC2 images. For more information, refer to the
help documentation for imedit.
4
Instrument Science Report WFPC2 2001-001
Figure 1: Dark image with CTE effects (top) which are then masked (bottom)
5
Instrument Science Report WFPC2 2001-001
Bad areas are usually Charge Transfer Efficiency (CTE) effects. These primarily take the
form of multi-column vertical bars or charge trails from bright cosmic rays that are a few
DN above the normal background. These are illustrated in Figure 1. Also keep an eye out
for jumps in the bias level which will appear as horizontal bands across the chip. These
bands are typically about 0.5 DN higher or lower than the rest of the image. Images with
bias jumps will need to be either corrected, following the methods outlined in ISR 95-06,
or simply rejected.
To mask out "bad" areas, set the pixel values in the “bad” region to be some very large
number so that when the individual files are combined, these pixels will be rejected. This
is done interactively using IRAF:
cl> uimedit *c0h[1] value=20000
To mask the bad region with a rectangular box, type "d" at the lower left and then again at
the upper right corner of the box you wish to mask. The uimedit task will replace this
region on your screen with the value specified and will overwrite the original file. Be sure
that you examine and mask all four chips (groups 1,2,3,4) in each dark image.
5. Making the Darks
The wkdark task is all you will need to run to make the dark reference files. This program
links together the following 4 tasks: superimg, superdqf, doheader, calcheck, each of
which are described in detail in Section 6. Your input dark files and previous dark reference image should be in a separate directory from files you FTP’d to create the darks.
Before running wkdark, the following 2 parameters must be set. You may enter these
parameters prior to running, by typing "epar wkdark", or you may type them on the command line when IRAF prompts you for input. When entering these parameters, use the
formats given in the following example.
mk> wkdark
Rootname of Previous Dark (kau1324lu):
Useafter Date (Jan 25 2000 05:05:05):
The useafter date format is (Mon dd yyyy hh:mm:ss). If the darks were taken just after a
decontamination, use the date/time of the decontamination from the website listed on page
3. Otherwise, use the date/time of the first dark observation, determined by typing:
cl> hedit *c0h fields=“date-obs,time-obs” value=“.”
6
Instrument Science Report WFPC2 2001-001
Once processing commences, no further steps need to be taken. Numerous processing
comments will be written to the screen while the code is running and you may monitor this
output if desired. These comments will follow the outline described in the next section.
6. Detailed Description of Processing using “wkdark”
The IRAF task wkdark is a compilation of 4 individual scripts, each dealing with a different phase in the process of producing darks. For reference, we describe each of these
scripts in detail below.
SUPERIMG.CL
The task superimg is used to construct the weekly dark reference file from the 5 individual darks files. These calculations are summarized as follows:
1. Runs the task stsdas.hst_calib.wfpc.mkdark using all the “c0h” images in the
directory. This task generates a dark image from multiple exposures of the same field
which is free of cosmic rays. The sigmas used in iterative cosmic ray rejections were set
to “4,4,3,2”. The WFPC2 noisepar parameters were set with read noise=1.73 DN and
gain=7.5 electrons per DN. For more information, type help mkdark in the
stsdas.hst_calib.wfpc package.
2. Normalizes all 4 groups of the "dark.r3h" file, dividing the "dark.r3h" image
by 1843.60. This is the value of the effective exposure time of the 1800 sec dark images.
The output file "weekly.r3h" contains the dark current per second in each pixel.
3. Calculates the mean and standard deviation of each chip in the weekly dark file
using images.imutil.imstat. This calculation is iterated 4 times, each time further
constraining the upper and lower limits, where upper=(mean+3*sigma) and
lower=(mean-3*sigma). The final sigma for each chip is written to the file ”sigma”.
4. Compares the statistics of the weekly dark to the current superdark file
"superdark.r3h", which was created using an average of 120 input darks. This file
should exist in the directory with your dark scripts. If the weekly and the super dark
images differ by more than 5 sigma, where sigma is the standard deviation calculated in
step 4, then the pixel value from the weekly dark image is used. This implies that the
pixel’s dark current has changed relative to the superdark. Otherwise, the value is
taken from the superdark image.
5. Uses the header information from the superdark combined with the pixel data
from the weekly dark “weekly.r3h” and writes the file "ndark7.r3h".
6. Uses the task dohedit to updates numerous header keywords for the "ndark7.r3h"
image, which corresponds to "Gain=7,Serials=Off".
7. Calculates the “Gain=15” image from the “Gain=7” image using the task wfarith
7
Instrument Science Report WFPC2 2001-001
which works on the WFPC2 multi-group file format, where ndark7*0.5=ndark15.
8. Copies the “Serials=Off” images to “Serials=On” images: ndark7on, ndark15on.
SUPERDQF.CL
The task superdqf is used to update the data quality files (DQF) which are named
ndark*.b3h. To do this, it compares the current weekly dark DQF with the superdark
DQF and with the DQF from the previous weekly dark .
1. Uses the following criteria to determine the DQF pixel values
a) data quality value=’2’
if the pixel value came from the weekly dark and if crrej used 3 or less
of the 5 input frames, indicating that the pixel value was questionable
b) data quality value=’same as superdark’
if the pixel value came from the superdark
c) data quality value=’1024’
if the pixel value came from the weekly dark and if the value changed less
than 0.003 DN/s since the previous weekly dark
d) data quality value=’512’
if the pixel value came from the weekly dark and if the value changed more
than 0.003 DN/s since the previous weekly dark
2. Uses the header information from the superdark combined with the data quality values
determined above and writes the file "ndark7.b3h".
3. Updates numerous header keywords for the DQF "ndark7.b3h",where
"Gain=7, Serials=Off".
4. Copies “Gain=7” DQF to “Gain=15” DQF and the “Serials=Off” DQF’s to
“Serials=On” DQF’s
5. Removes all temporary files and moves the original images and any remaining
intermediate products to the directory INFILES. The only things left are the ndark*
files and the sigma file.
DOHEADER.CL
The task doheader is dedicated to updating the image header keywords and to giving
detailed history comments describing the dark creation procedure.
1. Uses the task dowfstat to calculate the number of good pixels, the data min and
max, and the good pixel min and max, and sets these keywords.
2. Edits the header keywords PEDIGREE, USEAFTER, DATE-OBS, DESCRIP using
information from the image headers and from the wkdark parameters.
8
Instrument Science Report WFPC2 2001-001
3. Adds history comments describing how this weekly dark file was created and how the
superdark file was created.
4. Rechecks the header information by printing the min and max keywords for groups
1 and 3 and then by calculating the min and max values using groups 1 and 3 of the dark
files. These are printed to the file "out.chk". Examine this file at the end of processing
to ensure that the first four lines of statistics match the last four lines.
CALCHECK.CL
The task calcheck is used to verify that the stsdas.hst_calib.wfpc.calwp2 will be able to
handle these new dark reference files when you recalibrate your data.
1. Copies a dummy uncalibrated image *.d0h (with value 1000 everywhere) and the
engineering data file *.x0h from your dark scripts directory and runs the task
dodqfqs to create the q0h,q1h quality files.
2. Runs calwp2 on each of the 4 dark reference files, saving the output log to calchk.log
and creating images out*.c0h. Statistics are then computed for each output image.
3. For comparison with these statistics, the code also performs a manual check: It
multiplies the dark reference files ndark7.r3h and ndark15.r3h by 500 seconds and
subtracts each from the d0h file, writing to the output files "min*.d0h"
4. Compares the statistics derived using calwp2 (Step 2) with the statistics derived
using a manual check (Step 3). If these numbers are not identical, the differences will
be printed to the screen and logged to the file "out.mm". If all has run correctly, this
file will be empty.
5. All files except the ndark* images are moved to a directory "INFILES". The final
remaining product is the four dark reference files (*.r3h) and data quality files (*.b3h)
corresponding to both gains 7 and 15 and to both serials ON and OFF.
6. Acknowledgements
Many have contributed to the dark procedures and scripts over the years. We would like to
thank the WFPC2 IDT, the WFPC2 group at STScI, including Sylvia Baggett, Stefano
Casertano, and John Biretta. Others who have made considerable contributions include
Calvin Tullos, Jin-Chung Hsu, Max Mutchler, and Christine Ritchie.
9