IPAC Photometric Pipeline
Russ Laher
PTF Summer School
August 26, 2013
26 August 2014
iPTF Summer School
1
Outline
•
•
•
•
•
Raw Data Overview
Significant Project Events
Pipeline Software
Major Processing Steps
Products Overview
Reference: Laher et al., PASP 126:674, 2014 July
26 August 2014
iPTF Summer School
2
Raw Data Overview
•
•
•
•
•
A PTF exposure is a FITS file with multiple
header-data units (13 HDUs)
The primary HDU has information about
the observation, and no image data
12 HDUs more are for the CCD images
Science exposure times are nominally 60 s
(95.9%; 3.85% are 30 s)
Some CCDs are better than others in terms
of image quality
26 August 2014
iPTF Summer School
3
Raw Data
Overview (cont.)
PTF200903011447_2_o_12442.fits,
CCDID=7 (lower-right portion)
• Raw CCD Image Data
– 2048x4096 imaged pixels
(2078x4128 including the bias
overscan region)
– Dynamic range from 0 to
65,535 DN (unsigned integers)
– Data are stored as signed
integers, and FITS-header
keyword BZERO=32,768 DN
instigates the required data
shift
26 August 2014
iPTF Summer School
4
Significant Project Events
• 2008-12-13: First light
• 2009-03-01: Start of data epoch for routine IPACPTF pipeline processing
• 2009-06-30: End of commissioning phase
• 2009-10-09: Cameraelectronics upgrade
(which improved the
dynamic range)
26 August 2014
Saturation values vs. epoch [DN].
| CCDID | 20080101 | 20091009 |
0
34000 53000
1
36000 54000
2
25000 55000
4
31000 49000
5
33000 50000
6
26000 55000
7
17000 55000
8
42000 53000
9
19000 52000
10
25000 52000
11
36000 53000
iPTF Summer School
5
Significant Project Events (cont.)
• 2010-07-15: The R and g’
filters were swapped in
the filter wheel (which
altered ghost positions)
• 2010-09-02: “Fogging”
problem solved (which
caused an oily build-up
on the camera’s cold CCD
window)
• 2013-01-01: Start of iPTF
26 August 2014
iPTF Summer School
6
Pipeline Software
• Pipelines are wrapped with Perl scripts
• Binary executables (developed for PTF, imported
from Spitzer, SExtractor, SCAMP, SWarp,
astrometry.net, Montage, WCS tools, etc.)
• Libraries (cfitsio, fftw, NaN, Atlas, etc.)
• All software and CDFs under CVS version control
• Changes tracked under web-based GNATS
• Builds on 64-bit Linux (RHE 6) and Mac platforms
• Metadata tracked in a PostgreSQL database
26 August 2014
iPTF Summer School
7
Major Daily Processing Steps
Data Ingest
Camera-imagesplitting pipeline
(ppid=7)
Receive “end of
night” signal
Super-bias
calibration
pipeline (ppid=1)
Imagepreprocessing
pipeline (ppid=3)
Super-flat
calibration
pipeline (ppid=4)
Image-flattener
pipeline (ppid=10)
Frame-processing
pipeline (ppid=5
or ppid=13)
Product archiver
IRSA product
metadata
deliveries
26 August 2014
iPTF Summer School
8
Super-Bias Calibr. Pipeline (ppid=1)
• Inputs are all bias images for a given night and CCD
• Since 2009-01-16, ~40 bias images per night are commonly taken
• Bias images (exptime=0 s, shutter closed) are taken at start and end
of night
• Input pixel mask identifies CCD dead/bad pixels, which are set to
zero
• Inputs are corrected individually for the floating bias
• At each pixel location, outliers in the image stack are rejected if
more than 2.5 times data dispersion around median (double-sided)
• Data dispersion = 0.5 * (P84.1%-P15.9%)
• Surviving data are averaged for each pixel
• Outputs:
– Super-bias calibration FITS file (2078x4128 pixels)
– FITS keyword: NBIASES = 30 / Number of biases frames averaged
– Ancillary FITS file of data cube with uncertainties, number of samples,
and number of outliers rejected
26 August 2014
iPTF Summer School
9
Super-Bias Example
PTF_201208280000_i_s_bias_t120000_u000088205_f00_p000000_c04.fits
Computing image statistics...
number of NaNs, Infs = 0, 0
image min, max = -757.096313, 34390.914063
image n, ave, stdev = 8577984, -5.298940e-01, 1.443749e+01
image median, scale = -4.796706e-01, 1.981689e+00
image 16 and 84 percentiles = -2.458933e+00, 1.504445e+00
26 August 2014
iPTF Summer School
10
Floating-Bias Correction
• Done separately for each science image using the floatingbias strip contained in the science image
• Floating-bias image region:
x1 = 2066,
x2 = 2077,
y1 = 30,
y2 = 4084,
• Outliers in the region are rejected if more than 2.5 times data
dispersion around median (double-sided)
• Data dispersion = 0.5 * (P84.1%-P15.9%)
• Non-rejected data are averaged to obtain the floating bias
• The floating bias is subtracted from the image data at each
pixel position (pipelines ppid= 1 and 3)
26 August 2014
iPTF Summer School
11
Image-Preprocessing Pipeline (ppid=3)
• Executed on a per night, per CCD, per filter basis
• Prepares science images to be fed into downstream pipelines (e.g.,
superflat pipeline, flattener pipeline)
• Subtracts the floating bias and super-bias from each pixel
• Crop science images to remove overscan regions
• Recompute an improved value for the seeing (after running Sextractor); at
least 25 “good” sources are required, otherwise infobit 6 is set for the
corresponding science image
– Perform source selection based stellarity and instrumental magnitude
– Compute mode of the distribution of FWHM samples
• FITS keyword for improved seeing: FWHMSEX
• Compute initial data masks for saturated (bit 8), “dirty” (bit 11), and
source-detected (bit 1) pixels
• Incorporate existing pixel mask (dead/bad pixels) into initial data mask
• Infobit 7 is set for science images containing no sources
26 August 2014
iPTF Summer School
12
Super-Flat Calibration Pipeline
(ppid=4)
• Run for a given night, CCD and filter
• Computes a flat-field or nonuniformity correction
• Generally uses all science images
• Inputs are preprocessed images and data masks
• Inputs are normalized by each image global median (which excludes
source-detected, saturated and dead/bad pixels)
• Normalized values < 0.01 are reset to unity
• Images are stacked and outliers are rejected at +/-5 sigma (with
10% data-distribution tails clipped) about the median
• The median is recomputed for the remaining data at each pixel
location
• The resulting image is normalized by its global mean
• Normalized values < 0.01 are reset to unity
26 August 2014
iPTF Summer School
13
Super-Flat Example
PTF_201208310000_i_s_flat_t120000_u000088315_f02_p000000_c04.fits
Computing image statistics...
number of NaNs, Infs = 52253, 0
image min, max = 0.426518, 1.504622
image n, ave, stdev = 8336355, 1.000000e+00, 1.433028e-02
image median, scale = 1.000021e+00, 1.302055e-02
image 16 and 84 percentiles = 9.869365e-01, 1.012978e+00
26 August 2014
iPTF Summer School
14
Image-Flattener Pipeline (ppid=10)
• Run for a given night, CCD and filter
• Applies super-flat corrections to science
images
• Computes weight images for final sourcecatalog generation (later in frame-processing
pipeline); uses the super flat and data mask
• Sextractor’s Eye software detects CCD bleeds
(bit 6) and radiation hits (bit 7); data mask
updated for these bits
26 August 2014
iPTF Summer School
15
Frame-Processing Pipeline (ppid=5)
26 August 2014
iPTF Summer School
16
Absolute Photometric Calibration
•
•
•
•
•
•
•
•
Ofek et al., PASP, vol. 124, pp 62-73, 2012 January
Absolute photometric calibration using SDSS catalog stars
Separate calibration per night, CCD, and filter
Based on Sextractor’s MAG_AUTO instrumental magnitudes
R and g’ bands only
Coefficients stored in FITS header
Color terms are ignored when applying the calibration
Terms that depend on image position comprise the “Zero Point Variability Map”, an
illumination correction for spatial variations caused by the “fogging” problem
ZPVM
26 August 2014
iPTF Summer School
17
Photometric Calibration (cont.)
•
Applying the calibration (ignore color terms):
# i = image index
# j = source index (within image i)
deltime = obsjds[i] - APMEDJD; # image (obs. JD + 0.5*exptime) minus median JD.
absphotzpfixed = APPN01
+ APPN03 * airmasses[i]
+ APPN05 * deltime
+ APPN06 * deltime * deltime
+ 2.5 * log10(exptimes[i]);
# bilinear interpolation
src_zpvm[j] = getZpvmValueForPixel(x_image[j], y_image[j], ZPVM, 256);
if (isNaN(src_zpvm[j])) {
src_zpvm[j] = 0.0;
}
absphotzp[j] = absphotzpfixed + src_zpvm[j];
26 August 2014
iPTF Summer School
18
•
•
•
•
•
•
•
Products Overview
Raw Images
Processed Images
– Single-image FITS files (32-bit floating-point data)
– Infobits (set of flags about processing anomolies for each image)
Data Masks
– Single-Image FITS files (16-bit integer data)
– Per-pixel flags to give useful information encountered for each image pixel
– Processed-image FITS header included as an HDU extension
SExtractor Catalogs (and DOAPHOT Catalogs for data after January 2013)
– FITS binary-table files
– Includes per-source ZEROPOINT column (Sextractor catalog only)
– Processed-image FITS header included as an HDU extension
Calibration Files (not explicit in IRSA interface)
– Super bias
– Super flat
– Zero-point variation map (ZPVM)
– Calibration ancillary files
Pipeline Logs
Source Matches & Relative Photometry
26 August 2014
iPTF Summer School
19
Any questions? Please feel free to
e-mail laher@ipac.caltech.edu
26 August 2014
iPTF Summer School
20
Backup Slides
26 August 2014
iPTF Summer School
21
Raw Data Overview (cont.)
• An example of a camera-image filename is
PTF201108182046_2_o_8242.fits
–
–
–
–
Date and fractional day (GMT)
Filter number (1=g’, 2=R, 11=HA656, 12=HA663)
Image type (o=object, b=bias, k=dark current)
Counter (caution: not a unique number, since it resets
to zero upon camera reboot)
26 August 2014
iPTF Summer School
22
Generate-Catalog Pipeline (ppid=13)
•
•
•
•
•
•
•
•
•
•
•
•
Stopgap designed to generate a catalog from an existing processed image in the
IRSA archive, and fix known “features” in the products
Fixes a weight-map issue where weight-map pixels were too aggressively set to zero
when associated data-mask bits were set to 1
Calibration files associated with the processed image are migrated to the output disk
location
Copies the entire processed image FITS header to the mask FITS header
New ZEROPOINT column in Sextractor binary FITS catalogs
Some FITS header improvements (e.g., new and updated keyword values, minor
correction to APDOF value, and corrections to bit definitions in data mask)
Changed Sextractor constant gain=1.5 to CCD-dependent settings
New infobit 17 for flagging high-distortion cases
Removed the 1.0e3 scaling of instrumental magnitudes in sources.sql (CR #282), and
replaced with calibrated magnitudes (CR #326)
Added new columns in sources.sql: obsmjd, hmjd, ptffield, exptime, fwhmsex,
airmass, colorterm, airmasscolorterm, bsrms, limitmag, medianskymag, moonphas,
moonesb
New SDQA rating medianSkyMagAbsPhotCal and new FITS keyword MSMAPCZP
Products from this pipeline will be deprecated after next round of PTF reprocessing
26 August 2014
iPTF Summer School
23
PTF Standardized Filenaming Scheme
•
Example filenames:
PTF_200903011372_i_p_scie_t031734_u008648839_f02_p000642_c10.fits
PTF_200908184228_i_p_scie_t100852_u011892777_f02_p100043_c01.fits
PTF_200908184228_i_p_mask_t100852_u011892777_f02_p100043_c01.fits
PTF_200908184228_c_p_scie_t100852_u011892777_f02_p100043_c01.ctlg
•
•
•
•
•
•
•
•
•
Concatenation of year (4 digits), month (2 digits), day (2 digits), and fractional day (4 digits)
One-character product format: Choice of “i” for image or “c” for catalog
One-character product category: Choice of “p” for processed, “s” for super, or “e” for external
Four-character product type: Choice of “scie” for science, “mask” for mask, “bias” for superbias,
“banc” for superbias-ancillary file, “flat” for superflat, “twfl” for twilight flat, “fmsk” for flat
mask, “weig” for weight, “zpvm” for zero-point variability map, “zpve” for zero-point-variabilitymap error, “sdss” for SDSS, “uca3” for UCAC3, “2mas” for 2MASS (Two-Micron All-Sky Survey),
or “usb1” for USNO-B1
Prefix “t” for time followed by hours (2 digits), minutes (2 digits), and seconds (2 digits)
Prefix “u” for unique index followed by relevant database-table primary key (e.g., processedimage ID or pid)
Prefix “f” for filter followed by 2-digit filter number (FILTERID)
Prefix “p” for PTF field and is followed by PTF field number (PTFFIELD)
Prefix “c” for CCD followed by two-digit CCD index (CCDID)
26 August 2014
iPTF Summer School
24
PTF Operations Database
•
•
•
•
PostgreSQL (open source)
Relates raw data to pipeline products
Tracks the latest (best) versions of products
Tracks pipeline software versions and
configurations (CDFs)
• Generates unique IDs for product versions
• Stores SDQA ratings of products
• Enhances pipeline reliability and data integrity
26 August 2014
iPTF Summer School
25
PTF Ops. Database Schema
Nights
nid
nightDate
filename
receivedAt
filesize
checksum
status
Exposures
1
1..*
Fields
fieldid
ptfField
object
prid (F.K.)
objRa
objDec
objRaD
objDecD
descrip
1
1..*
sdqa_statusid
statusName
definition
26 August 2014
ppid
priority
script
descrip
Filters
fid
filter
Projects
1
1..*
prid
ptfPid
piid (F.K.)
1
1..*
PIs
1
1..*
sdqa_ratingid
pid (F.K.)
sdqa_metricid (F.K.)
(F.K.)
1sdqa_thresholdid
1..
metricValue
metricErr
flagValue
1
1
SDQA_Metrics
sdqa_metricid
metricName
physicalUnits
dataType
definition
1
1
1..*
SDQA_Thresholds
sdqa_thresholdid
sdqa_metricid (F.K.)
version
vBest
upperThreshold
lowerThreshold
createdDate
ProcImages
1
1..*
RawImages
rid
expid (F.K.)
ccdid (F.K.)
filename
checksum
createdDate
status
infobits
nid (F.K.)
itid (F.K.)
piid (F.K.)
CalFileUsage
1
SDQA_Statuses
itid
imgType
SDQA_Ratings
1
1
1
1..*
Pipelines
piid
ptfPrPI
ProcImages
pid
rid
ppid
version
vBest
sdqa_statusId (F.K.)
image metadata…
1
1..*
expid
obsdate
filename
receivedAt
1
1..
filesize
checksum
status
infobits
nid (F.K.)
fid (F.K.)
itid (F.K.)
fieldid (F.K.)
piid (F.K.)
prid (F.K.)
skipFilterCheck
filterCheck
FITS-header metadata…
ImgTypes
1
1..*
1..*
1..*
1..*
pid
ppid (F.K.)
rid (F.K.)
version
filename
checksum
createdDate
status
SDQA_statusId
infobits
nid (F.K.)
expid (F.K.)
ccdid (F.K.)
fid (F.K.)
piid (F.K.)
vBest
pBest
svid (F.K.)
cvid (F.K.)
avid (F.K.)
archiveStatus
pid (F.K.)
cid (F.K.)
1..*
1
1..*
CalFiles
1 1..*
1..*
1
Tracks
1..*
1
1..*
Catalogs
1..*
catid
pid (F.K.)
ppid (F.K.)
version
filename
checksum
createdDate
status
infobits
nid (F.K.)
expid (F.K.)
ccdid (F.K.)
fid (F.K.)
piid (F.K.)
vBest
pBest
svid (F.K.)
cvid (F.K.)
avid (F.K.)
archiveStatus
1
AncilFiles
caid
cid (F.K.)
ancilType
filename
checksum
avid (F.K.)
archiveStatus
aid
pid (F.K.)
ancilType
filename
checksum
avid (F.K.)
archiveStatus
1..*
1
1
1
ArchiveVersions
1..*
iPTF Summer School
1
avid
archived
cvid
cvsTag
installed
1
CalAncilFiles
1..*
svid
cvsTag
installed
CdfVersions
1..*
1..*
cid
calType
ccdid (F.K.)
fid (F.K.)
startDate
endDate
filename
checksum
createdDate
status
avid (F.K.)
archiveStatus
SwVersions
1
tid
pid (F.K.)
num
metadata…
26
Data-Analysis “Gotchas”
• Ghost (bit 5) and Halo (bit 12) masking needs to be improved
• Currently, data-mask bits 6 (CCD bleeds) and 7 (radiation hits) are always set
together
• For raw data, DS9 shifts image positions according if not set up to ignore the
FITS-header keyword TRIMSEC
TRIMSEC = '[16:2063,1:4096]' / trim section
• PTF images have moderate geometrical distortion (4th-order polynomials)
– Variable observing weather and seeing requires that each PTF exposure
have a separate distortion characterization
– DS9 does not compute distortion corrections involving PVij FITS-header
keywords, but does for SIP distortion
– The CD matrix is recomputed for sharing between SIP and PV distortion
(Sextractor uses only PV distortion)
• Some ZPVM and ZPVE FITS files erroneously contain the BZERO FITS-header
keyword and have non-standard file size (not padded to 2880 bytes)
• Products from ppid=13 pipeline did not have astrometry.net run in the case
of SCAMP failure
26 August 2014
iPTF Summer School
27
Processed-Image InfoBits
(located in FITS header and database)
Bit Description
-------------------------------0 Scamp failed
1 WCS solution determined to be bad
2 mShrink execution failed
3 mJPEG execution failed
4 No output from ptfQA module (because SExtractor found no sources...)
5 Seeing was found to be zero; reset it to 2.5 arcseconds
6 ptfSeeing had insufficient number of sources for seeing determination (<1)
7 No sources found by Sextractor
8 Insufficient number of 2Mass sources in image for WCS verification (20)
9 Insufficient number of 2Mass matches for WCS verification (20)
10 2Mass astrometric RMSE(s) exceeded threshold (1.5 arcsec)
11 SExtractor before scamp failed
12 pv2sip failed
13 scamp terminated with exitcode=0 but had insufficient number of catalog stars (20)
14 scamp terminated with exitcode=0 but had insufficient number of matches (20)
15 Anomalous low-order WCS terms (CDELT1, CDELT2, CROTA2, CRPIX1, and CRPIX2)
16 Track-finder module failed
17 High distortion in WCS solution
18 Astrometry.net was run
19 Error from sub runAstrometryDotNet
20 Time limit reached in sub runAstrometryDotNet
perl -e '$a = 2**1 + 2**9 + 2**17 + 2**18 + 2**19 + 2**20 ; print ”infobits value = $a\n";'
infobits value = 1966594
26 August 2014
iPTF Summer School
28
Science Data Quality Analysis
• Frame-processing pipeline computes 109
different image statistics and stores them in
the database (SDQA ratings)
• Superflat pipeline computes SDQA ratings for
11 different SDQA metrics.
• These SDQA ratings can be queried over the
entire data set and studied
ProcImages
pid
rid
ppid
version
vBest
sdqa_statusId (F.K.)
image metadata…
SDQA_Ratings
1
1..*
sdqa_ratingid
pid (F.K.)
sdqa_metricid (F.K.)
(F.K.)
1sdqa_thresholdid
1..
metricValue
metricErr
flagValue
1
1
1
1
1
SDQA_Statuses
sdqa_statusid
statusName
definition
SDQA_Metrics
sdqa_metricid
metricName
physicalUnits
dataType
definition
1
1
1..*
SDQA_Thresholds
sdqa_thresholdid
sdqa_metricid (F.K.)
version
vBest
upperThreshold
lowerThreshold
createdDate
Column
median - mode (DN)
Number of good data points 2740614
Number of bad data points 0
Average
15.331820
Median
6.877991
Standard deviation
50.667200
Scale
9.169171
RMS value
52.936083
Minimum
-1819.593800
Maximum
11965.768000
Percentile1
0.328857
Percentile16
2.829102
Percentile84
21.167443
Percentile99
135.954096
26 August 2014
iPTF Summer School
29
Data-Mask Bit Definitions
• Located in the FITS header of the data-mask file:
/ PTF DMASK BIT DEFINITIONS
BIT00
BIT01
BIT02
BIT03
BIT04
BIT05
BIT06
BIT07
BIT08
BIT09
BIT10
BIT11
BIT12
BIT13
BIT14
BIT15
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
0 / AIRCRAFT/SATELLITE TRACK
1 / OBJECT (detected by SExtractor)
2 / HIGH DARK-CURRENT
3 / RESERVED FOR FUTURE USE
4 / NOISY
5 / GHOST
6 / CCD BLEED
7 / RAD HIT
8 / SATURATED
9 / DEAD/BAD
10 / NAN (not a number)
11 / DIRTY (10-sigma below coarse local median)
12 / HALO
13 / RESERVED FOR FUTURE USE
14 / RESERVED FOR FUTURE USE
15 / RESERVED FOR FUTURE USE
perl -e '$a = 2**0 + 2**1 + 2**8; print ”Data-mask value = $a\n";'
Data-mask value = 259
26 August 2014
iPTF Summer School
30