Instrument Science Report NICMOS 2006-001
NICMOS count-rate dependent nonlinearity tests using flatfield lamps
Roelof S. de Jong, Eddie Bergeron, Adam Riess, Ralph Bohlin
February 15, 2006
ABSTRACT
We investigate the recently discovered NICMOS count-rate dependent non-linearity
(Bohlin et al. 2005) using the flatfield lamps to artificially increase the count rate. A star
cluster field was imaged in a lamp off-on-off sequence in all cameras in a selected set of
filters, followed by a series of darks to investigate persistence and to clean the images
from any remaining charge for the next orbit. Subtracting the lamp-off images from the
lamp-on images clearly shows residual ADUs at the star positions, indicating that a
higher background (and thus total) count rate increases the number of ADUs registered
from an object. We model the non-linearity with a power law (count-rate ∝ fluxα) and fit
this model to the data. Both NIC1 and NIC2 (NIC3 was not tested in this program) show
non-linearity, becoming stronger at shorter wavelengths, but with larger amplitude than
predicted by the Bohlin et al. NIC3 measurements. The non-linearity in NIC1 and NIC2
amounts to 0.06-0.10 mag offset per factor ten change in incident flux for the shortest
wavelength (F090M and F110W), about 0.03 mag/dex at F160W, and less at longer
wavelengths. Archival data from Cycle 7 are also analyzed, showing that the nonlinearity has not changed in NIC2 F110W, and suggesting that this effect is independent
of detector temperature.
Introduction
In a recent analysis of NICMOS, STIS and ACS spectral data Bohlin et al. (2005) have
shown that NICMOS shows a systematic count rate dependent non-linearity, primarily at
the shorter wavelengths. The same spectra show a similar non-linearity when compared
1
Operated by the Association of Universities for Research in Astronomy, Inc., for the National Aeronautics and
Space Administration
to ACS photometry. However, the NICMOS spectra in general show good agreement
with the NICMOS photometry of the same objects, indicating that the NICMOS system is
internally consistent and that it is not the spectral data reduction that is at fault. A few
more indications have been found that NICMOS suffers from a non-linearity dependent
on the incoming flux: 1) narrowband filters at the shorter wavelengths required larger
corrections from their ground-based determined throughputs than the broadband filters,
2) high redshift supernova fluxes are slightly fainter in F110W than expected based on
their ACS fluxes and well tested SN models (Adam Riess, private communication), and
3) galaxies in the HUDF are slightly fainter than expected based on ACS and groundbased J&K magnitudes combined with SED modeling (Mobasher & Riess 2005; Coe et
al. 2005).
However, all these lines of evidence rely on modeling of filter throughputs and/or
spectral modeling of sources. Here we describe a test that depends on the change in
incoming flux on the detector alone. NICMOS is a shutterless instrument and observes
the sky while obtaining calibration flatfields using its internal lamps. The same object can
be observed with an artificially increased flux and the count rate for objects can be
compared with and without extra lamplight. For a fully linear system adding a
background flux should not enhance the flux in the object, but any flux dependent nonlinearity is revealed immediately when subtracting lamp-off images from lamp-on
images.
Data
Normal lamp flatfield observations are obtained in empty fields and are not suitable for
the current analysis. Because the non-linearity effect is rather small (~5% extra ADU per
dex in incoming flux), observations have to be carefully planned to match the count rates
from the objects to the count rate from the lamps. Furthermore, to measure such a 5%
offset from linearity the lamp background will be 10x brighter than the object flux per
pixel, and many pixels and observations must be averaged to obtain the signal-to-noise
required to accurately measure the ~5% effect on such a high background.
During Cycle 7 a few observations were obtained in focus monitoring program
7901 that happened to be useful for this analysis. These observations were obtained at the
end of a focus sweep observation in NIC2 to determine the position of the coronagraphic
hole (this program was discontinued because software was developed to determine the
hole position during coronographic acquisition). These observations were only obtained
in F110W and have very short exposure times (<3 seconds), because both object and the
lamp (FLAT1) are bright. The images are severely out of focus, but this does not hamper
pixel-by-pixel analysis of relative count rates in the lamp-off and on images.
To cover more wavelengths and to have count rates somewhat closer to real
observations, special observations were obtained in program 10726. The FLAT2 lamp
2
was used in combination with the NGC1850 star cluster. The FLAT2 lamp was used
instead of the lowest count rate FLAT3 in order to have enough flux to build high enough
signal-to-noise to measure the non-linearity effect within one orbit. Furthermore, the
FLAT3 lamp has never been used before in orbit. In the current test only NIC1 and NIC2
were measured, but the FLAT3 lamp may be more appropriate to test NIC3 for nonlinearity.
It was decided not to put the image out of focus (even though this provides more
pixels at intermediate count rates and mitigates image misalignments) in order to be able
to do real aperture photometry in the different images. However the lamp on/off analysis
cannot be performed with aperture photometry because the non-linearity is different for
all pixels in an aperture and not equal to the non-linearity of the average flux in an
aperture.
All observations (lamp off and on) were obtained without dithering, in order to
keep everything as stable as possible and to allow pixel-by-pixel comparison of the lampon and off images. At the time of the implementation of the program charge trapping, as
revealed by persistence, was considered the main candidate for the cause of the nonlinearity. The charge trapping hypothesis was later found to be a less likely solution (de
Jong, in prep., Bohlin et al. 2006) but the program was set up in the following sequences
to address charge trapping issues: slew to NGC1850 and obtain darks to make sure there
is no persistence, obtain a few exposures with lamp off to set the baseline count rate
measurement, obtain a large number of exposures with the lamp on (supposedly filling
the traps), take another set of exposure with the lamp off (now with the traps still filled
but discharging), followed by dark exposures during the occulted part of the orbit to
make sure no flux is falling on the detectors before the next orbit. The total exposure time
in each off and on sequence was set such that a signal-to-noise of 100 could be obtained
for pixels with flux 1/10 the lamp flux. Identical MULTIACCUM exposure sequences
(SAMP-SEQ) were used in the lamp-off, lamp-on and dark frames to make observations
as similar as possible, and total exposure times per exposure was set such that saturation
in the lamp on exposures was avoided. NIC1 and NIC2 were run in parallel for 2 out of 3
orbits, but only one of the cameras was pointed at the center of the cluster (NIC2 F160W
and F187W observations were obtained off cluster center and have less stars). NIC3 was
not used in parallel as it would require very short exposure times and the buffer dump
overhead from the high data rate would have severely reduced the efficiency of the
program.
All observations were reduced with the standard pipeline of calnica using the
regular reference files. Flatfield observations are normally not processed in the standard
data reduction. Therefore the header items NLINCORR, DARKCORR, FLATCORR and
UNITCORR of the lamp-on *_raw.fits files were set to PERFORM and the files were
processed again by calnica. All separate sets of off, on, off, and darks were combined
per filter with IRAF task mscombine.
3
After these basic data reduction steps it was found that the current pipeline
flatfields are a poor match to the lamp-on data (RdJ has recently shown that they have
been getting a progressively worse match to the flat monitoring data since they were
created during the post-NCS SMOV-3B). The lamp-on images show large-scale flat field
residuals of several percent between minimum and maximum, and also the noise in the
images is much larger than expected theoretically. Stacking the different lamp-on images
did not reduce the pixel-to-pixel noise as expected, also suggesting that there is a
systematic data reduction problem. New flat fields were created from the latest lamp flat
monitoring data and a flat field correction was applied. This significantly reduced the
large-scale flat field residuals (by a factor 1.5-3) and reduced the pixel-to-pixel noise by a
factor of ~2. The remaining large-scale flat field residuals were removed by dividing the
on and off images by a 15x15 box median filtered image of the lamp-on minus lamp-off
images (which should in theory have no star residuals in them, but do, and hence the
median filter to remove them). All these extra steps do not change the basic final
conclusion of this investigation that there is count rate dependent non-linearity, but help
to make the signal more significant and allow more accurate measurement of the strength
of the effect. These corrections were not necessary for the Cycle 7/7n data that were also
analyzed, as the temperature dependent Cycle 7/7n flatfields were accurate within the
errors.
Analysis
The data were analyzed under the assumption that a power law can model the nonlinearity:
cr(x,y) ∝ (ftot(x,y))α,
with cr(x,y) the measured count rate in ADU/s and ftot(x,y) the total flux falling on a
detector pixel. For a non-linearity of ~5% per dex this corresponds to α~1.02. In
magnitudes we have an offset of Δm=2.5(α-1) per dex change in incident flux. When we
have a lamp-on and lamp-off observations and subtract the off from the on observation
we expect to see positive residuals at positions where there are objects if α>1:
cron-croff ∝ (fobj+fsky+flamp)α-(fobj+fsky)α ~ (fobj+flamp)α-(fobj)α,
where it is assumed that the sky flux is small compared to the other fluxes. Such image
residuals are shown in Figure 1. The absolute boost in measured count rate is largest for
bright objects, but the relative increase in measured count rate is larger for lower object
fluxes, because the relative increase in flux by switching on the lamp is much larger.
However, at low count rates the noise dramatically increases and we have to average
many points to see the effect. This is shown in Figure 2, where we plot both the absolute
count rate increase
4
(cron-croff-crlamp) vs croff
Figure 1: (Left) NIC1 F110W lamp-off image of NGC1850. (Right) NIC1 F110W lamp-on
minus lamp-off image. Bright stars are clearly not well subtracted and leave residual flux as
expected for count rate dependent non-linearity. The worm-like structures are due to residual flat
field problems.
as well as the relative count rate increase
(cron-croff-crlamp)/croff vs croff .
To determine α for a particular camera and filter we have to fit the equation that links the
flux measurements in all pixels in the lamp-on image to the lamp-off image:
F(x,y)=cron= (fobj+fsky+flamp)α=(croff1/α+ flamp)α.
We use a Levenberg-Marquard fitting routine to determine α and the lamp flux, which
requires the derivatives of the fitting function:
∂F(x,y)/∂α = (croff1/α+ flamp)α[log(croff1/α+ flamp)- croff1/αlog(croff)/(α( croff1/α+ flamp))]
∂F(x,y)/∂flamp = α(croff1/α+ flamp)α-1
An error weighted fit was performed, excluding from the fits the points in the (poor
quality) bottom 20 rows and in the NIC2 choronographic hole region. The σ values in the
ERR images of NICMOS pipeline imsets are incorrect, and proper ERR images were
calculated by taking the ERR image of the pipeline after the FLATCORR step and add
the appropriate photon noise derived from the SCI image. We used the σ values of the
lamp-on image for weights, as we are minimizing the differences between the model and
the lamp-on image. However, the independent variable, the count rates from the lamp-off
image, has also significant noise (mainly read noise at low count rates), and therefore a
5
minimum signal-to-noise cutoff was imposed on the lamp-off input values. Even the
improved σ values are actually not completely realistic due to the poor flat fields. We are
therefore also determining a σ estimate directly from the data, calculating the r.m.s. in the
lamp-on minus lamp-off image. We have been using both σ values to select pixels in the
lamp-off image to use in the fitting process. The resulting fits show some dependence on
the imposed cutoff using cutoffs between 2 to 3σ, but one would expect that this
dependence decreases when the noise is reduced with better flat fields. The results vary
by a few percent depending on whether the initial estimate of α fed into the fit routine
was lower or higher than the final fit value. Therefore, to find the best α and to determine
the uncertainty in the fit, we first made a pass to approximately determine α, and then
made fits with a range of initial values of α around this first estimate and with low signal
cutoffs in the lamp-off images of 2, 2.5, and 3σ. The α values listed in Table 1 are the
median values of all α values thus determined; the errors are the 1-sigma deviations in
the distribution of α. As these systematic uncertainties are probably larger than the
sampling uncertainties, we did not perform a full bootstrap resampling error analysis of
the data.
The fitted non-linearity functions are overplotted in Figure 2 and the measured α
values are tabulated in Table 1. A number of points can immediately be taken from the
Table. The effect becomes more severe at shorter wavelengths, similar as predicted by
the analysis of Bohlin et al. (2005). However, NIC1 and NIC2 still have significant nonlinearity at F160W, unlike the Bohlin et al. prediction for NIC3. NIC1 and NIC2 may not
have the same non-linearity at all wavelengths, with NIC1 and NIC2 differing by 2σ at
F110W and NIC1 being more affected. We do not see any statistical significant
difference between the non-linearity in NIC2 F110W as measured in Cycle 7/7n and in
the current test taken during Cycle 14.
In some of the observations there was some drift in pointing of the telescope. This
will add somewhat to the uncertainty of the measurement. While at first sight one would
think that binning the data on the array would mitigate this problem. Unfortunately this
does not work, because the expected average increase in count rate in the individual
pixels combined in a bin is not equal to the expected increase in count rate for the
average flux in the binned pixels. This is also the reason why aperture photometry cannot
be used to measure α, as within an aperture there will be a range in flux increases, which
will not be equal to the increase expected for the average flux.
We also investigated the lamp-off images taken before and after the lamp-on
images for signs of charge trapping and non-linearity. The lamp-off images taken after
the lamp-on images show a decrease in overall background count rate level consistent
with persistence. The set of lamp-off images taken of the star cluster after the initial darks
generally but before the lamp-on images show a small gradual increase in the background
count rate level. Furthermore, the very first image taken after the initial darks seems to
6
have lower count rates in the objects compared to the rest of the lamp-off images,
fractionally somewhat larger in the fainter objects than the brighter objects. These early
lower count rates are suggestive of a charge trapping effect. The count rates stabilize
within about one minute, even at the low count rates, and we have to recommend that no
low count rate observations should be taken with exposure times less than one minute.
However, this small amount of count rate buildup, that has some of the characteristics of
charge trapping, does not provide the explanation for the non-linearity measured in this
ISR. The amount of non-linearity measured between the lamp-on and lamp-off images is
independent from whether all lamp-off images are used, or only the lamp-of images
before the lamp-on, or the images after the lamp-on (Table 2).
A similar test as presented here using the flatfield lamps to artificially increase the
count rates on grism exposures is described in Bohlin et al. (2006). They find a similar
wavelength dependent non-linearity in NIC3 as described here, but with a lower
amplitude.
7
Conclusions
NICMOS has a significant count rate dependent non-linearity that also depends on
wavelength. This is a different non-linearity from the well-known total count dependent
non-linearity. The non-linearity in NIC1 and NIC2 amounts to 0.06-0.10 mag offset per
dex change in incident flux for the shortest wavelength (F090M and F110W), about 0.03
mag/dex at F160W and less than that at longer wavelengths. These corrections are larger
than predicted from the Bohlin et al. (2005) NIC3 grism results, which may point to
intrinsic detector differences or might be the result of a different analysis method. The
non-linearity seems to have changed very little from Cycle 7/7n to Cycle 14 (in F110W
NIC2), and hence is unlikely to depend on detector temperature. The fact that there is a
wavelength dependence to the effect in the lamp off/on/off test and that this trend
quantitatively agrees with the grism observations strongly argues against this being the
result of a data reduction error and that the cause is intrinsic to the measurement.
To what extend NICMOS photometry is affected by the non-linearity depends on
the wavelength of the observations (i.e. the α parameter), whether the object is a point
source or extended, and on the count rate of the sky background (as the count rate will
never go below the sky level and hence the non-linearity will level off, even if the
sources have lower count rates). Given that the NICMOS standard stars are of about the
12th magnitude, the maximum expected offset for the UDF for example is about 0.15-0.2
mag at 22 F110W AB-mag, where the objects are comparable to or below the sky count
level.
Recommendations
Further tests are necessary to measure the effect more accurately, but the NICMOS
community should be made aware of these results and their significance for the different
kind of observations as soon as possible. Some off the necessary steps to improve the
calibrations of NICMOS data are:
- Improve the current flat fields in the NICMOS pipeline, probably by taking
temperature dependence into account,
- Perform lamp-on/off tests for all regularly used passbands to determine the α
parameters, at least at the shorter wavelengths. The images should be taken outof-focus to mitigate the effect of drift during the observations,
- Include NIC3 in the imaging lamp-on/off non-linearity testing,
- Similar tests should be performed on the ground for detectors in future space
programs (e.g. WFC3 and JWST),
- Develop software to correct fluxes of observations based on these α parameters. It
will be impossible to implement on a pixel-to-pixel basis without modifying the
properties of the read-noise somewhat (negative pixels cannot be corrected with a
8
fractional power law) and without having to derive the calibration zeropoints from
scratch.
In a forthcoming ISR we will describe how one can calculate corrections for the countrate non-linearity for aperture photometry of point sources. Accurate corrections for
extended sources are more complicated to calculate and will have to await modifications
of the NICMOS data reduction pipeline.
Acknowledgements
The NICMOS team, the non-linearity working group members, and Rodger Thompson
are gratefully acknowledged for their help in this investigation into the NICMOS linearity
and calibration.
References
Bohlin, R., Lindler, D. & Riess, A. 2005, NICMOS ISR 2005-002
Bohlin, R., Riess, A. & de Jong, R.S. 2006, NICMOS ISR 2006-002
9
Table 1. Measured α and Δm values in Cycle 7/7n and 14 with their 1 sigma errors. The
NIC2, F110W observations have been shaded gray.
Date
Camera
Filter
1998/02/18
1998/04/17
1998/06/04
1998/08/06
1998/09/24
2
2
2
2
2
F110W
F110W
F110W
F110W
F110W
2005/11/17
2005/11/17
2005/11/17
2005/11/17
2005/11/17
2005/11/17
1
1
1
2
2
2
F090M
F110W
F160W
F110W
F160W
F187W
α
Cycle 7/7n
1.022 +/- 0.001
1.025 +/- 0.004
1.023 +/- 0.001
1.024 +/- 0.001
1.022 +/- 0.001
Cycle 14
1.040 +/- 0.003
1.030 +/- 0.003
1.012 +/- 0.002
1.025 +/- 0.002
1.012 +/- 0.006
1.005 +/- 0.004
Δ m/dex
lamp
0.055 +/- 0.003
0.063 +/- 0.010
0.059 +/- 0.002
0.061 +/- 0.002
0.054 +/- 0.002
2152.9
2102.8
2129.5
2119.9
2177.1
0.101 +/- 0.008 79.8
0.074 +/- 0.009 316.8
0.031 +/- 0.006 148.3
0.063 +/- 0.006 1206.5
0.029 +/- 0.015 535.8
0.013 +/- 0.009 209.5
Table 2. Measured α and Δm values for Cycle 14 data, using for the lamp-off reference
data either only the lamp-off data taken before the lamp-on data, only the lamp-off data
taken after the lamp-on data, or both.
Date
2005/11/17 both
2005/11/17 before
2005/11/17 after
2005/11/17 both
2005/11/17 before
2005/11/17 after
2005/11/17 both
2005/11/17 before
2005/11/17 after
Camera
1
1
1
1
1
1
2
2
2
Filter
F110W
F110W
F110W
F160W
F160W
F160W
F110W
F110W
F110W
α
1.030 +/ 0.003
1.029 +/ 0.003
1.029 +/ 0.003
1.012 +/ 0.002
1.014 +/ 0.002
1.012 +/ 0.002
1.025 +/ 0.002
1.025 +/ 0.003
1.024 +/ 0.003
10
Δ m/dex
0.074 +/ 0.009
0.074 +/ 0.009
0.072 +/ 0.009
0.031 +/ 0.006
0.035 +/ 0.005
0.030 +/ 0.005
0.063 +/ 0.006
0.063 +/ 0.006
0.059 +/ 0.006
lamp
316.8
317.3
317.9
148.3
147.4
148.7
1206.5
1205.3
1218.5
Figure 2: The absolute and relative difference in lamp-on minus lamp-off count rates as function
of the lamp-off count rates on a pixel-by-pixel basis. The yellow +’s are for all data, the blue +’s
for the data used in the fit (bad and low S/N points filtered). The green circles are the averages in
bins of 50 pixels in ascending lamp-off count rate for all pixels. The red circles are binned
averages in 30 equal logarithmic steps in lamp-off count rate for the selected pixels. The green
lines are the fitted non-linearity functions, with the fitted α parameter labeled at the top. Left is
the absolute difference (cron-croff-crlamp), right is the relative difference (cron-croff-crlamp)/ croff. While
the brightest points have the largest flux change in absolute sense and are easy to measure above
the noise (and are not due to background subtraction errors), the fainter points change relatively
the most and have larger calibration errors relative to bright standard stars.
11
12
13
14