Instrument Science Report NICMOS 2006-003
Correcting the NICMOS count-rate
dependent non-linearity
Roelof S. de Jong
March 29, 2006
ABSTRACT
We describe a routine to correct NICMOS imaging data for the NICMOS count-rate
dependent non-linearity recently discovered by Bohlin et al. (2005) and quantified by de
Jong et al. (2006) and Bohlin et al. (2006). The routine has been implemented in the
python scripting language and is callable from the shell command line and from iraf. The
routine corrects NICMOS count-rate images assuming the non-linearity follows a powerlaw behavior. The wavelength dependence of the non-linearity is interpolated between
the measured points of de Jong et al. (2006) and Bohlin et al. (2006) if necessary. The
count rates in the output images are modified and hence the standard NICMOS
calibration zero-points are no longer valid. New calibration zero-points have been
derived from standard star images corrected with the routine. The routine was tested on
the lamp-on/off data used in de Jong et al. (2006) to measure the non-linearity effect. We
apply the correction to the NGC1850 stellar cluster field and the Hubble Ultra Deep
Field (HUDF) to show the magnitude offsets expected due to the non-linearity on objects
with a range in luminosity and surface brightness.
Introduction
In a recent analysis of NICMOS, STIS and ACS spectral data Bohlin et al. (2005) have
shown that NICMOS shows a systematic count rate dependent non-linearity, primarily at
the shorter wavelengths. This non-linearity was further investigated and quantified by de
Jong et al. (2006) using the NICMOS internal flat field lamp to artificially increase the
count rate on NIC1 and NIC2 images of a star cluster. Bohlin et al. (2006) further
Operated by the Association of Universities for Research1in Astronomy, Inc., for the National Aeronautics and
Space Administration
quantified the non-linearity in NIC3 using grism spectra in three tests: the flat field lampon/off test, comparing the NICMOS and STIS spectra in the spectral overlap region, and
by comparing NICMOS White Dwarf measurements to models. There are a few more
indications that NICMOS suffers from a non-linearity dependent on the incoming flux: 1)
narrowband filters at the shorter wavelengths required larger corrections from their
ground-based determined throughputs than the broadband filters, 2) high redshift
supernova fluxes are slightly fainter in F110W than expected based on their ACS fluxes
and well tested SN models (Adam Riess, private communication), and 3) galaxies in the
HUDF are slightly fainter than expected based on ACS and ground-based J&K
magnitudes combined with SED modeling (Mobasher & Riess 2005; Coe et al. 2005).
Here we describe the software implementation of a routine that corrects NICMOS
count-rate images for the non-linearity. We derive new photometric zero-points for the
corrected images. We show the implications of the non-linearity for galaxies in the
HUDF as an example.
Non-linearity correction software implementation
We correct the NICMOS images under the assumption that a power law model can
describe the non-linearity:
cr(x,y) ∝ (ftot(x,y))α,
(1)
with cr(x,y) the measured count rate in ADU/s and ftot(x,y) the total flux falling on a
detector pixel. For a non-linearity of ~5% per dex in flux change this corresponds to
α~1.02. In magnitudes we have an offset of Δm=2.5(α-1) per dex change in incident
flux. The amount of non-linearity depends on wavelength and was quantified for NIC1
and NIC2 in three filters each by de Jong et al. (2006) and with grism spectra for NIC3
by Bohlin et al. (2006). These measurements are shown in Fig. 1 as solid points. To
determine the amount of non-linearity in filters not measured we interpolate between or
extrapolate beyond the measured wavelength as indicated by the solid lines. For
simplicity we use the pivot wavelength of a filter as stored in the NICMOS image header
keyword PHOTPLAM to determine the interpolation or extrapolation point. The nonlinearity quantification in Bohlin et al. (2006) was actually in percent per dex flux change
and not by a power law, but it is trivial to show that for the four orders of magnitude
change in incident flux for which the non-linearity was measured the two descriptions are
identical within the measurement uncertainties, i.e. for small ∆ we have
log(x+∆)≈log(x)+∆/2.5.
2
Figure 1: Non-linearity index in magnitude offset (∆m) per dex change in incident flux as
function of wavelength. The solid points indicate the measurements by de Jong et al. (2006 and
Bohlin et al. (2006). The lines going through the points indicate the interpolation and
extrapolation used to calculate the non-linearity for other filters than those used for the measured
points.
To correct the images for the non-linearity we have to raise each pixel in the
image to the power 1/α. However, we cannot raise a negative number to a fractional
power, and many pixels in a typical NICMOS image have negative count rates in areas
free of objects due to low sky levels and relatively large read noise. We could opt not to
correct negative pixels at all, but to preserve a more symmetric noise distribution near the
sky level we use the absolute values of the pixels in the correction:
crcorrected(x,y) ∝ |crmeasured(x,y)|1/α.
(2)
The non-linearity is thought to depend on all the infalling flux on the detector, and
therefore it is important that the routine described here must only be used on images that
have their original sky background level. The NICMOS routine pedsky for instance
removes the estimated sky background from the image, and before running the current
correcting routine this sky value (stored in header keyword SKYVAL) should be added
back into the image.
Equations (1) and (2) do not provide an absolute calibration, only a
proportionality. This is because the measurements in Bohlin et al. (2005) show that the
non-linearity is the same over four orders of magnitude in incident flux range, with no
turnover at the bright or faint end. We can therefore not say whether we are missing flux
at the low count-rate end or having extra flux at the high count-rate end. The
3
normalization of the correction is therefore arbitrary, and for simplicity we chose to
correct all count rates relative to 1ADU/s; count rates higher than 1 ADU/s are brought
down, count rates lower than 1 ADU/s are increased to bring all count rate levels on a
linear scale.
The non-linearity correction routine is implemented in python. It assumes that the
numarray and pyfits python modules are installed. They are part of the pyraf package
needed to run multidrizzle on NICMOS images. The routine can be run from any
command line that runs python programs and takes one or two input parameters. In the
case that only an input file name (including extension) is provided, an output file will be
created named after the input file name, but with everything after the first underscore
replace with _ncl, or if no underscore is present, with _ncl added at the end. For example:
$ nonlincor.py n9a01beq_cal.fits
will result in an output file n9a01beq_ncl.fits, while
$ nonlincor.py hudfF110W.fits
will result in an hudfF110W_ncl.fits output file. When providing two arguments, the
second argument is taken to be the output file, which will overwrite any existing file of
that name. From the python command line the arguments have to be put in quotes as the
routine is called as a function (assuming nonlincor.py is in your PYTHONPATH):
>>> import nonlincor
>>> nonlincor.nonlincor(‘hudfF110W.fits’)
An iraf script has been implemented for confenience, which takes the usual lists of files
in iraf form as input (comma separated filenames, wildcards, or @filelist). Output file
names can optinally provided through the output parameter like:
cl> task nonlcor = nonlcor.cl
cl> nonlcor nic1*.fits output=%nic1%cnic1%*.fits,
which will take all files starting with nic1 as input, and will give output files starting with
cnic1.
The routine runs on both simple FITS files as well as multiextension FITS files
(as delivered by the MAST archive). In the case of a multiextension file, the first image is
assumed to be the science image in ADU/s and the pixels of the second image in the
image stack are multiplied with the same amount as the first image, assuming it is a
sigma error image. It should be noted however, that NICMOS images have incorrect
4
values in the sigma error images in the current pipeline, because of an incorrect treatment
of the estimating the photon noise, and pipeline error images should not be used in
analysis at the current time.
The routine expects the CAMERA and FILTER keywords to be present in the
image header; the first is needed to determine which non-linearity table to use, the second
is used to calculate the pivot wavelength of the filter and which is then used to interpolate
between the table values if needed. If the data was processed with software that does not
preserve these header keywords they will have to be added in by hand.
Zero-point derivation
The routine changes the count rates of sources and hence the normal NICMOS
calibrations are no longer valid. We performed a recalibration of all standard star data
after applying the non-linearity correction. Calibration was performed in the same way as
the regular NICMOS calibration, which will be described in more detail in forthcoming
ISRs (Sosey et al. 2006; de Jong & Sosey 2006).
In short, pipeline *_cal.fits standard star images were corrected for pedestal with
pedsky, shading was removed with a 1D fit in the proper direction, and the correction for
count-rate dependent non-linearity was applied as described above. Aperture photometry
was obtained for each dither position and filter dependent aperture corrections to go from
fixed to “infinite” apertures were applied. The mean count rate of all measurements for a
given camera, filter, and star was determined after removing bad measurements due to
bad pixels, loss of guidelock, etc. This provides the count rates and their uncertainties for
the NICMOS standard stars after correction for non-linearity.
These count rates have to be compared to absolute (spectro)-photometry to
provide absolute calibrations. No absolute photometry exists for the NICMOS filters, and
hence calibration is done using spectro-photometric standard stars in combination with
the synphot package to calculate the effective stimulus of each filter. NICMOS
calibration is performed using two principle standard stars: P330E, a solar analog star,
and G191B2B, a white dwarf. For P330E we used the calibrated spectrum of Bolin,
Dickinson & Calzetti (2001), available in the synphot database under the name
p330e_stis_001.fits. For G191B2B we used NICMOS grism spectra as described in
Bohlin et al. (2006), which were absolute flux calibrated using a white dwarf model
described in Bohlin et al. (2001).
5
Figure 2: The PHOTFNU values derived for the different standard stars relative to those of star
G191B2B. The PHOTFNU parameter indicates the sensitivity of a filter/detector by relating the
effective stimulus of a flat spectrum source in fν to the observed count rate. For each star we list
the mean deviation from G191B2B and the rms averaged over all filters. The horizontal line
indicates the effective width of the filter, which is typically half the FWHM width of the filter.
On the left we show the PHOTFNU ratios derived before non-linearity correction (a), on the right
after correction (b).
While P330E and G191B2B form the principle standards for NICMOS, other
stars were observed in a subset of filters to provide crosschecks of the calibrations.
NICMOS grism observations of solar analog stars P177D and SNAP-2, white dwarfs
GD71 and WD1657+343, and principle SDSS standard BD+17o4708 were absolutely
calibrated after correcting for non-linearity as described in Bohlin et al. (2006). These
stars, with a wide range in brightness, required different multiplication factors to match
them to the STIS observations (and their models for the white dwarfs), which is how the
non-linearity was discovered in the first place. The NICMOS grism spectra were
extended at the shorter wavelengths with STIS spectra and at longer wavelength with
6
stellar models (when available), because some filters cover a broader wavelength range
than covered by the grisms (no grism observations were taken of the fainter stars beyond
1.9 micron).
Based on the observed count rates and the absolutely calibrated spectra we can
now use synphot to calculate a PHOTFNU for each star: the flux of an object with a flat
spectrum in fν that produces 1 ADU/s in the detector. In Fig. 2 we show the ratios of
PHOTFNU of the different stars relative to those of G191B2B before and after correcting
for non-linearity. For a linear system deviations from unity indicate that the absolute
calibrations of the spectra are in error, or that, in case the spectra deviate substantially
from each other, the filter throughput shape used to calculate the PHOTFNU values are
incorrect. We just test whether the ratio in the observed aperture photometry count rate
for two stars is the same as expected from the two calibrated spectra in the filter
wavelength region. A non-linear detector will show deviations from unity depending on
the source brightness.
The PHOTFNU ratios in Fig. 2a before correcting for non-linearity show indeed
such non-linearity; faint stars SNAP-2 and WD1657+343 have a PHOTFNU that is too
high relative to G191B2B, while bright star BD+17 o4708 has a too low PHOTFNU,
especially for the filters at the shorter wavelengths. After correcting for non-linearity
almost all filters of all stars have PHOTFNU values that agree with those of G191B2B
within a few percent. There is a hint that the NIC3 correction may be too small, because
there are still some deviations for our faintest reference star WD1657+343.
7
Figure 3: The observed count rate versus the expected count rate based on the spectrophotometric standard stars and the HST sensitivity as calculated with synphot for all three
cameras. On the left the ratios before correcting for non-linearity (a), on the right after corrections
(b). The horizontal line indicates the effective width of the filter, which is typically half the
FWHM width of the filter.
These PHOTFNU ratio plots will however not show whether the absolute
calibration of the system is correct, all stars could be equally wrong in absolute sense.
Fig. 3 shows the observed count rate before and after non-linearity correction relative to
the expected count rate given the absolute spectra and the synphot throughput sensitivities
that are currently in the standard database. The synphot throughputs were calibrated in
Cycle 12 to the weighted average of the P330E and G191B2B observations, and hence it
is not surprising that these stars have count rates close to those predicted by synphot
before correction. However, the faint and bright stars deviate as expected from the
predicted count rates before non-linearity correction. After non-linearity correction all
stars do agree with each other in almost all filters (see also Fig. 2b), but the absolute
calibrations are no longer correct. This is why we have to recalibrate the zero-points after
non-linearity correction in the first place! Broadband filters at short wavelengths deviate
8
more from the expected count rates than narrow band filters, because they have higher
count rates and are farther away from the normalization at 1 ADU/s we used for the nonlinearity correction and receive the largest correction. This difference between broad and
narrowband filters mimics the corrections needed after inflight calibrations, when inflight
sensitivities were compared to the throughputs measured for the filters on the ground,
providing further evidence for the non-linearity.
The newly determined values for PHOTFNU and its equivalent PHOTFLAM for
flat spectrum sources in fλ are written in header keywords by the non-linearity correction
routine. They are currently hard coded in the routine, but will in the future be provided as
separate FITS tables. PHOTFNU can be used to convert observed count rates in the
corrected images to AB magnitudes using
mAB = –2.5 log (crcorrected × PHOTFNU × 10-23 ) – 48.6,
or in ST magnitudes by
mST = –2.5 log (crcorrected× PHOTFLAM) – 21.1,
where PHOTFLAM is in units of erg cm-2 Å-1 ADU-1, and PHOTFNU in Jy sec ADU-1.
The recalibration of the zero-points for non-linearity corrected images has so far
only been performed on standard star data obtained after installation of the NICMOS
Cryo-cooler System (NCS; Cycle 11 and beyond), and hence the derived calibration
keywords are only valid for post-NCS data. While the non-linearity probably has not
changed much between pre and post-NCS data and the routine makes the right
correction, the zero-points are completely different for the two eras due to the different
operating temperatures, and the zero-points provided by the routine should not be used
with Cycle 7/7n data.
Testing
It is not trivial to test the non-linearity correction routine completely independently. The
photometric calibrations investigated above show that the photometric consistency
between bright and faint stars dramatically improves after applying the correction.
However, there is some circularity applied here, as the NICMOS grism spectra used in
the calibrations were corrected for non-linearity as well. However, this was necessary to
make them agree with the STIS spectra, and the circularity would only apply to NIC3
(the grism camera). The NIC1 and NIC2 corrections were derived independent of the
spectra, and their improved calibration consistency was obtained with completely
different non-linearity correction parameters.
The correction routine was also tested using the lamp-on/off data that was used to
characterize the non-linearity by de Jong et al. (2006). Both the cluster observations
without and with the flatfield lamp switched on were corrected using the correction
routine. The resulting images were analyzed in identical fashion as by de Jong et al.,
9
comparing the flux levels pixel-by-pixel with lamp on and off. Figure 4 shows an
example for the F110W filter in NIC2. Before the correction, the increase in count rates
in pixels is higher than expected based on the lamp count rate alone, and the amount of
extra increase depends on the original flux in the pixel with the lamp off. After applying
the correction, the count rate increase in each pixel switching the lamp on shows no
longer a dependence on the count rate originally in the pixel. While not completely
unexpected, as this is how the α non-linearity parameters were determined in the first
place, it shows the routine at least works as expected, and the α values become equal to 1
for all filters as it should for a linear system.
Figure 4: Testing the non-linearity correction on F110W images of star cluster NGC1850. The
top panels show the data before correction, the bottom panels after correction. On the horizontal
axis we show the count rate in each pixel before the flat field lamp is switched on. On the vertical
axis we show on the left the difference in count rate in the lamp-on minus the lamp-off image and
the lamp background, on the right the same difference, divided by the lamp-off count rate (i.e. the
relative difference in count rate). While before correction there is significant counts in the
difference image left at the star positions, after correction the count rates in the objects are the
same whether the lamp is on or off.
10
Science impact
To estimate the science impact of the non-linearity we will study two examples that are
fairly typical for NICMOS science observations, a fairly crowded stellar field and faint
galaxies in the Hubble Ultra Deep Field. In each case we calculate the photometric offset
between uncorrected and corrected images.
Crowded field photometry of NGC1850
We calculate the effect of non-linearity for a fairly crowded field using images of star
cluster NGC1850 from program 10726, the data that was acquired in the lamp-on/off test
to measure the non-linearity. The data reduction was described in de Jong et al. (2006),
and we use the average of all lamp-off images in the different filters. The final images
were then corrected for non-linearity with the routine described above. Aperture
photometry was obtained with the same settings as used in the calibration program, using
apertures with radius 11.5 pixels for NIC1 and 6.5 pixels for NIC2.
In Fig. 5 we show the difference in magnitudes before and after correction as
function of star magnitude. The correction is nearly zero for bright stars that have
luminosities comparable to our standard stars, and increases toward fainter stars. Stars
brighter than our standard stars would require a negative correction. The relation is not
very tight and has an increasing scatter at the faint end. The field is crowded enough that
faint stars can sit on top of the faint PSF tails of brighter objects and hence require less of
a correction because the local count rate is higher. This shows that it is not trivial to
correct existing photometry for the non-linearity if one has no good estimate of the local
background for each object, but Fig. 5 provides an estimate of the size of the effect.
Figure 5: The difference in magnitude before and after correction for non-linearity of objects in
the NGC1850 as function of uncorrected magnitude.
11
Faint galaxy photometry in the Hubble Ultra Deep Field
We use the NICMOS treasury data of program 9803 (PI Thompson) to estimate the
science impact on sparse field galaxy photometry. We use an independent data reduction
from the one provided by Thompson et al. (2005) available in the MAST archive,
because Thompson et al. used an independent data reduction for which the STScI
provided photometric zero-points are not valid. However, the offsets calculated here are
valid for the Thompson et al. reduction as well. Bergeron et al. (in prep.) describe the
details of the HUDF NICMOS data reduction used here.
The sky backgrounds were subtracted from the original frames before they were
mosaiced together. However, because the non-linearity depends on the total incoming
flux, we added the average sky values back in the final mosaic. The average sky values
over all frames were 0.12 ADU/s at F110W and 0.14 ADU/s at F160W. The variation
from frame to frame was very small at about 0.01 ADU/s. However, the accuracy to
which we can determine the exact sky value will limit the accuracy to which we can
determine the magnitude offsets.
Galaxy photometry was obtained with the SExtractor package, v2.2.2. The
extraction parameters were set identical to those used by Thompson et al. (2005), but
without the use of a weight image and with fitting of the sky background. We ran
SExtractor in multi-image mode, using the uncorrected images for detection of galaxies,
and using both the uncorrected and corrected images to do photometry.
The difference between the corrected and the uncorrected MAG_AUTO magnitudes as
function of object magnitude is shown in Fig. 6. Almost all objects are galaxies, and
fainter than 21 AB-mag all galaxies are sufficient below sky level that they have the same
magnitude offset after correction. The corrections are 0.158 mag for F110W and 0.042
mag for F160W. Brighter than 21 AB-mag we see the offset decrease, as these objects
(some of which are stars) significant flux above the sky level and need less correction.
Figure 6 makes clear that the exact offset of the HUDF galaxy magnitudes
depends critically on the value of the non-linearity correction parameter and the value of
the sky level. It is therefore important to obtain NIC3 imaging measurements of the nonlinearity in the F110W and the F160W filters using the lamp-on/off test, as the grism
determined offset has not been folded trough the filter curves to determine the exact nonlinearity parameter. The random variation in the sky value was about 0.01 ADU/s, which
includes contributions from real sky variations and from pedestal variations. However,
there might be a systematic error in the sky values due to errors in the dark current
subtraction, flat field errors propagating in pedsky corrections and unknown other
additive effects at these low count rates. Raising the sky value by 0.03 ADU/s would
decrease the offsets to 0.153 mag for F110W and 0.040 mag for F160W, while
decreasing the sky by 0.03 ADU/s would increase the offsets to 0.164 mag for F110W
and 0.044 mag for F160W. Luckily, the magnitude offset due to the non-linearity
depends on the logarithm of the flux change, and hence even quite large errors in the sky
12
level result only in a limited uncertainty in the correction. The uncertainties in the
correction are small compared to the overall uncertainty in the absolute NICMOS
calibration, which is in the order of 3% for these NIC3 filters.
Figure 6: The magnitude difference before and after correction for non-linearity of objects in the
Hubble Ultra Deep Field as function of corrected magnitude.
Downloading and installing
Information how to download and install the non-linearity correction scripts can be found
at http://www.stsci.edu/hst/nicmos/performance/anomalies/nonlinearity.html .
Conclusions
We have developed a software routine implemented in the python scripting language to
correct NICMOS images for the non-linearity as quantified in de Jong et al. (2006) and
Bohlin et al. (2006). New zero-point calibrations have been calculated for the corrected
images using the two principle standard stars P330E and G191B2B. Comparing the
imaging photometry to spectro-photometric observations for stars with a range in
13
brightnesses shows an increased consistency in calibrations when the non-linearity
correction is applied. The count rates behave as expected for a linear system after
applying the correction to star cluster images obtained with flatfield lamp switched on
and off. Typical corrections to the NICMOS treasury HUDF galaxies amount to 0.158
mag for F110W and 0.042 mag for F160W.
Recommendations
We recommend that all imaging NICMOS observations that require photometric
accuracy better than 20% are corrected for the count rate dependent non-linearity by this
software routine. Further development of the routine is recommended to make it more
robust, more flexible to further calibrations of the non-linearity and zero-points, and
suitable for the calnica and archive pipeline. Finally, we recommend that NIC3 imaging
lamp-on/off measurements be taken as soon as possible to measure the non-linearity
parameter in imaging mode, as the corrections for deep photometry (like the HUDF)
depend critically on it.
Acknowledgements
The NICMOS team, the non-linearity working group members, and Rodger Thompson
are gratefully acknowledged for their help in this investigation into the NICMOS linearity
and calibration.
References
Bohlin, R.C., Dickinson, M.E. & Calzetti, D. 2001, AJ 122, 2118
Bohlin, R., Lindler, D. & Riess, A. 2005, NICMOS ISR 2005-002
Bohlin, R., Riess, A. & de Jong, R.S. 2006, NICMOS ISR 2006-002
De Jong, R.S., Bergeron, E., Riess, A., Bohlin, R., 2006, NICMOS ISR 2006-001
Thompson, R.I., et al. 2005, AJ 130, 1
14