Telescope and Instrument Performance
Summary (TIPS)
18 July 2002
AGENDA
1.
2.
3.
CADC/ECF WFPC2 Associations
Status of Pydrizzle Software
GSC2 Status and Implementation Plans
Next TIPS Meeting: 15 August 2002
Dave Schade
Warren Hack
Brian McLean
STScI TIPS Meeting - 18 July 2002
David Schade
Canadian Astronomy Data Centre
WFPC2 Science Products
Pipeline
Developed as a partnership of the Canadian Astronomy Data Centre and the
Space Telescope-European Coordination Facility
Built on the foundation of the WFPC2 calibration pipeline
1
Canadian Astronomy Data Centre
STScI TIPS Meeting - 18 July 2002
Goals of the present pipeline/VO prototype delivery system
1.
To provide a science-quality image products in a ready-to-use
form
2.
To provide science-quality source and object catalogues
3.
To provide a basic archive interface to these products
4.
To provide a VO prototype interface to these products (in
combination with other products)
We view production of VO content as part of our VO-targeted work
2
Canadian Astronomy Data Centre
STScI TIPS Meeting - 18 July 2002
“A” associations: 1st generation created by ST-ECF and CADC
•
3
Jitter information complete for 57% of the assocations
Canadian Astronomy Data Centre
STScI TIPS Meeting - 18 July 2002
“B” associations: 2nd generation created by CADC and ST-ECF
•
Cross-correlation used to get dither offsets
•
Total of 28167 “B”-assocations
•
Top 6 filters comprise 14977 associations with T(sum)=35.15 million
seconds
•
4
Exposure times: Average 2347. Seconds, median ~ 900 seconds
Canadian Astronomy Data Centre
STScI TIPS Meeting - 18 July 2002
“B” associations: 2nd generation created with ST-ECF
Number of associations
Median exposure time
Average exposure time
Association exposure time
5
Canadian Astronomy Data Centre
STScI TIPS Meeting - 18 July 2002
Number of associations
“B” associations: 2nd generation created with ST-ECF
Half of the associations
Filter name
6
Canadian Astronomy Data Centre
STScI TIPS Meeting - 18 July 2002
Producing stacked images and catalogues for WFPC2
Associations
Alberto has
reviewed the history
7
Dither Patterns
Stacking
Jitter
Cross-Correlation
WCS
Shift method
Drizzle?
Combine method
(Artificial Skepticism)
X-Corr
X-Corr took 2.5
years to complete:
b-associations
Source Catalogues
Astrometry
Photometry
Detection
Properties
These two steps can be executed in
5 days for the full WFCP2 collection
Object Catalogues
colors
morphology
(redshift)
redshift)
(X-ray,FIR,radio)
These will be
released fall 2002
Canadian Astronomy Data Centre
STScI TIPS Meeting - 18 July 2002
Producing stacked images and catalogues for WFPC2
CNOC1
Images
2d-spectra
1d-spectra
Derived parameters
Associations
Dither Patterns
Stacking
Jitter
Cross-Correlation
WCS
Shift method
Drizzle?
Combine method
(Artificial Skepticism)
Source Catalogues
Astrometry
Photometry
Detection
Properties
Object Catalogues
colors
morphology
(redshift)
redshift)
(X-ray,FIR,radio)
Other catalogue at
different
wavelength
8
Canadian Astronomy Data Centre
STScI TIPS Meeting - 18 July 2002
Astrometry Corrections
Subset of 1680 stacks:
89% have 1 or more USNO2 stars on the mosaic
Offset in RA (seconds of arc)
Error in the RA offset
This is a completely random set of filters: red filters will be better
9
Canadian Astronomy Data Centre
STScI TIPS Meeting - 18 July 2002
Things that we need to do
1.
Quality Verification: Design and execute a set of data quality tests
2.
Documentation: Produce online documentation and a refereed
publication describing the project
3.
Develop automated maintenance for the system
4.
Develop, enhance, extend the VO environment for WFPC2, ACS,
other collections
10
Canadian Astronomy Data Centre
July TIPS
SPACE TELESCOPE SCIENCE INSTITUTE
18 July 2002
Warren Hack
PyDrizzle and ACS Pipeline Calibration
1
July TIPS
SPACE TELESCOPE SCIENCE INSTITUTE
18 July 2002
Warren Hack
Standard ACS Calibrations
The standard ACS pipeline calibration software was designed from the STIS pipeline
software. This software performs a wide-range of calibration steps, including bias
subtraction, dark subtraction, flat-fielding, and even cosmic-ray rejection.
ACS observations have some properties which standard pipeline processing performed for STIS or NICMOS could not address; specifically, significant optical distortion.
In addition, a 50 pixel wide gap between the chips in the Wide-Field Camera (WFC)
would require the use of dithering in order to get data from behind the gap.
The tasks in the STSDAS Dither package provide the tools to address both of these
issues.
2
July TIPS
SPACE TELESCOPE SCIENCE INSTITUTE
18 July 2002
Warren Hack
Dither and the Pipeline
Drizzle was developed to combine the dithered observations of the Hubble Deep Field
into a mosaic for analysis. It can also be used to remove distortion as well.
However, there are many problems with getting this software to run in the pipeline.
• a large number of parameters need to be determined, as documented in the
164-page HST Dither Handbook
• IRAF CL scripts do not support error handling necessary for a pipeline
• IRAF scripts tend to be fragile and can be hard to maintain
• drizzle only understands how to work on a single chip at a time
• drizzle can not properly apply distortion to subarray data at this time
A new task was written to control the operation of ‘drizzle’ for use in the calibration
pipeline for ACS: PyDrizzle.
3
July TIPS
SPACE TELESCOPE SCIENCE INSTITUTE
18 July 2002
Warren Hack
What is PyDrizzle?
PyDrizzle \Py” driz” zle\, n. 1. A program written in Python to be run under the
PyRAF environment that automates the use of the Dither task ‘drizzle’.
PyDrizzle automates the use of ‘drizzle’ through the following features:
• interprets association tables to determine input images to be combined into a
final product.
• understands how all the chips in an observation need to be combined into a
single image for ACS (and WFPC2)
• computes the shifts between exposures from the WCS header keywords
• converts distortion coefficients for ACS into a format understood by ‘drizzle’
• only requires one command to run drizzle on all input images
• allows single exposures to be drizzled to separate outputs if desired
• also controls the use of blot, for use in cosmic-ray rejection
4
5
July TIPS
SPACE TELESCOPE SCIENCE INSTITUTE
18 July 2002
Warren Hack
PyDrizzle and Association tables
ACS observations can be associated in the pipeline for a number of reasons:
• CR-SPLIT or REPEAT-OBS data
• Dither pattern specified]
A simple CR-SPLIT, 2-point dither association table, JNXXAA10T_ASN.FITS:
MEMNAME
1st
2nd
CR product
CR product
Dither product
MEMTYPE
MEMPRSNT
JNXXAAB1T
EXP-CR1
yes
JNXXAAB2T
EXP-CR1
yes
JNXXAAC1T
EXP-CR2
yes
JNXXAAC2T
EXP-CR2
yes
JNXXAA11T
PROD-CR1
yes
JNXXAA12T
PROD-CR2
yes
JNXXAA10T
PROD-DTH
no
July TIPS
SPACE TELESCOPE SCIENCE INSTITUTE
18 July 2002
Warren Hack
PyDrizzle Processing Steps
PyDrizzle performs a number of operations in preparation for running drizzle.
1. Determine input image(s) and name of output product
2. Interrogate each input image to determine the detector
3. Read each extension of each input image to extract WCS information
4. Read in distortion coefficients for each observation
5. Build a virtual ‘meta-chip’ which combines all chips from an exposure into a
single image after applying distortion coefficients
6. Compare WCS information for each exposure to determine any shifts, scaling, and/or rotation needed to create final combined product
7. Sets up drizzle parameters for each chip based on info from previous step
8. Run drizzle for each chip
9. Combine all three products from drizzle to create a single distortion-corrected, multi-extension FITS image.
6
July TIPS
SPACE TELESCOPE SCIENCE INSTITUTE
18 July 2002
Warren Hack
PyDrizzle and Exposures
Drizzle was only designed to operate on a single group or extension of an image at a
time. Thus, it falls on the user to figure out (if at all possible) the shifts, output image
size and plate scale for each chip or extension to combine a multi-chip observation
into a single combined image.
A single distortion-corrected image was desired as the final product for the ACS calibration pipeline, therefore, PyDrizzle needed to know how to combine the chips into a
single image. The IDCTAB, distortion coefficients reference table, not only contains
the distortion coefficients, but also the absolute position of each chips reference point.
This allows PyDrizzle to compute the offsets between the chips. This offset gets
recorded in the coefficients file passed to drizzle for each chip.
7
July TIPS
SPACE TELESCOPE SCIENCE INSTITUTE
18 July 2002
Warren Hack
PyDrizzle and Shifts
Image registration can pose serious problems when drizzling images, as slight offsets
in pointing can result in multiple images. PyDrizzle addresses this in two ways:
• Determination of the shifts by PyDrizzle start with the WCS information read
in from the image headers.
– HST observations taken as part of a dither-pattern rely on the same guide
stars, thereby minimizing pointing errors.
– The same assumptions can not be made, however, for mosaics or images
taken with different instruments at different epochs.
• PyDrizzle can read in delta’s from optional columns in the association table.
– These columns provide the offset from the header WCS values necessary to
accurately combine the images.
– These offsets can be computed using any method desired by the user.
8
July TIPS
SPACE TELESCOPE SCIENCE INSTITUTE
18 July 2002
Warren Hack
Running PyDrizzle
The actual operation of PyDrizzle has been designed to be as simple as possible. The
default case only requires the name of the input image or association table as input,
information the pipeline has when processing the data.
Taking the association table presented earlier as an example, we can run PyDrizzle to
create a dither-combined product in just one command.
xena> pyraf
>>> stsdas
>>> analysis
>>> dither
>>> pydrizzle jnxxaa10t_asn.fits
(PyDrizzle messages printed here...)
>>> display jnxxaa10t_drz.fits[sci,1] 1
9
July TIPS
SPACE TELESCOPE SCIENCE INSTITUTE
18 July 2002
Warren Hack
Where it is being used...
The ACS calibration pipeline currently uses PyDrizzle for:
♦ geometric distortion correction for single images
♦ dither-combining, and distortion correction for associated data
In addition, the Dither Working Group has been providing input into further development of PyDrizzle. Development has begun on:
• shift refinement with PyDrizzle
• cosmic-ray detection and removal using the dither package algorithms
A simple, proof-of-concept Python script has already been written by Anton
Koekemoer using PyDrizzle to perform cosmic-ray rejection and dither-combination
of ACS and WFPC2 data.
10
DSS-II/GSC-II Project Status
Brian McLean
CASB/ACDSD
CASB Mission Statement
CASB is committed to producing and distributing all-sky digital images, deep object catalogues, and software
tools to support operations of current and future ground and space based astronomical observatories, and to
provide a research and educational resource to the community
Catalogs and Surveys Branch
Operational Goals
DSS : Scan, archive, compress and put on-line, all available Schmidt
sky survey plates
GSC-II : Build database to create a deep all-sky catalog with positions,
proper motions, magnitudes, colors and classifications from all available
observational material and published catalogs
Mission Support : Provide access and tools to use the DSS and GSC
for operational support of telescopes and scientific research
July 22, 02
Images/Catalogs for Observation Planning
STIS/(COS) Bright Object Protection (integrated with APT)
Update HST GSC with better positions
NGST near-IR Guide Stars (K<18)
GEMINI & VLT GS (tracking & adaptive optics)
TIPS Meeting
2
Catalogs and Surveys Branch
DSS-II Overview
Scan new survey plates from the Palomar and UK
Schmidt telescopes
STScI/CASB scan archive (8TB raw data)
All-sky, minimum 3 bandpasses (J, F, N)
15 µ (1”) sampling, 23040x23040 pixels of 1.2 GB each
Older 25 µ scans to be replaced as resources permit
Raw FITS images on MOD with 8mm tape backup
Compressed images on-line (CD jukebox+NAS RAIDarray)
Distributed to Data Centers worldwide
DSS retrievals are “most requested’ service by community
Photometric and astrometric calibration of all images
calibrations provided in image headers
July 22, 02
TIPS Meeting
Survey
Number
Plates
(7952)
POSS-I E
880
POSS-I O
880
Pal-QV
614
POSS-II J
897
POSS-II F
897
POSS-II N
897
SERC-J/EJ
894
SERC-QV
90
PPARC-ER
288
AAO-SES
606
AAO-GR
109
UKSTU-IR
894
Special XX
3
6
Catalogs and Surveys Branch
Summary of Plate Processing
(June 2002)
Survey
Epoch
Plate/Filter
Band
Depth
Scanned
CDROM
Process
Pal- QV
1983 -85
IIaD+W12
V
19.5
100%
100%
42%
0%
SERC -J/EJ
1975 -87
IIIaJ+GG395
BJ
23.0
100%
100%
100%
100%
POSS -I E
1950 -58
103aE+red
R
20.0
100%
100%
90%
3%
POSS -I O
1950 -58
103aO
B
21.0
55%
15%
1%
0%
POSS- II J
1987 -00
IIIaJ+GG385
22.5
100%
100%
100%
100%
POSS -II F
1987 -99
IIIaF+RG610
R
20.8
100%
100%
100%
100%
POSS -II N
1987 -01
IV-N +RG9
I
19.5
99%
99%
99%
87%
AAO - SES/SERC - ER
1990 -00
IIIaF+OG590
R
22.0
100%
100%
100%
100 %
SERC -I
199 0-02
IV -N +RG715
I
19.5
96%
96%
96%
86%
July 22, 02
B
J
TIPS Meeting
DB
4
Catalogs and Surveys Branch
GSC-II Overview
All-sky catalog of positions, proper motions, magnitudes and
colors
Use all available images : DSS-I/DSS-II images + unpublished (QV,XO)
complete to a minimum of V=18 (Goal is plate limited)
GSC-II will have multiple observations for ~2 billion objects
Used commercial OODb
All observed object parameters stored in db, ~3TB final size
“Master Index” links all observations and catalogs to individual objects on the sky.
Recalibrations can be applied within db
Export catalog as binary FITS tables
GSC-II project is partially funded by external partners
July 22, 02
OATo, GEMINI, ESO, ESA/ST-ECF and ESA/SA
Receive prepublication data access for telescope operations and science projects
TIPS Meeting
5
Catalogs and Surveys Branch
GSC-II Goals
Positions
Proper Motions
Errors due to limited time baseline between observation epochs
typically 5-30 mas/year in North and 20-40 mas/year in South
Photometry
< 0.50”absolute; average across a single plate; < 0.20” relative,
over 1/2o field
< 0.15”adherence to ICRS reference frame, average over all plates
0.2mag error between 12-18 magnitude
Classification:
July 22, 02
95% accuracy within two magnitudes of plate limit
TIPS Meeting
6
Catalogs and Surveys Branch
Overview of Precision
Photometry
Astrometry
Zero pt. error less than <0.04mag
Rms errors vary from 0.15mag near
sequence to 0.28mag at edges
Zero pt. offsets are negligible
<0.01”
Rms errors vary from 0.15” in plate
center to 0.5” at edge
Magnitude equation < 0.1” not
corrected yet
Classification
July 22, 02
90-95% to 18th mag
Known errors with bright stars –
corrected by final release date
TIPS Meeting
7
Catalogs and Surveys Branch
Catalog Releases
PREVIOUS RELEASES
GSC 2.0
PLANNED RELEASES
delivered 7/99
Sky coverage prioritized for GEMINI SV, HST BOP
Two-passbands (J,F), without proper motions
Deliverable to GEMINI and ESO/VLT only
GSC 2.3
GSC 2.1.0
delivered 8/00
Same as GSC 2.1.0, increased sky coverage
Improved Photometric calibration with GSPC 2
Improved Astrometric calibration with Tycho 2
GSC 2.2.0
Same as GSC 2.0, but with available sky coverage
Preliminary astrometry, photometry, classification
GSC 2.1.1
delivered 3/00
GSC 2.n+
delivered 06/01
Complete sky coverage in 2 passbands
Magnitude limited (F<18.5, J<19.5)
Initial community release (www, data-centers)
estimated March 2003
Adds available XE, XO, QV plates + IR (XI,
XN)
Recalibration with final astrometric masks,
magnitude equation and vignetting
function
Reclassification with final decision trees
Proper motions computed
“Maintenance Mode”
Inclusion of 2MASS data (HST science
planning/NGST)
Galaxy photometry
Improved global error analysis and
removal of systematic errors
GSC 2.2.1
July 22, 02
color corrections
proper motion corrections
delivered 06/01
Complete sky coverage in 2 passbands
Plate limited
Internal consortium release for Telescope
Operations and Science Teams
TIPS Meeting
8
Catalogs and Surveys Branch
HST Pointing
Improved positional accuracy of Guide Stars
July 22, 02
More recent epoch of observation
More accurate astrometry (reduced systematic errors)
Proper motions
ICRS reference frame
NGSS will select GSC-I or GSC-II based on RPS2 keyword
Requires addition of dynamic WCS updates to DSS retrieval
with GSC-II astrometry
Plan to test GSC-II using selected cycle 12 calibration
observations
General GO usage for cycle 13
TIPS Meeting
9
Catalogs and Surveys Branch
HST Astrometry
Future observations using GSC-II will have improved
astrometry for aligning with observations at other
wavelengths = Better Science
Archived observations using GSC-I can be ‘updated’ with
GSC-II positions.
Can be accomplished either
July 22, 02
‘On-the-fly’
Updated headers in archive
Schedule TBD
TIPS Meeting
10
Catalogs and Surveys Branch
Observation Planning
DSS and GSC-II already integrated into APT
http services
In process of switching DSS from CDROM jukebox to RAIDarray
for improved reliability and response.
GSC-II Export catalog also on RAIDarray.
Bright Object Protection
Plan to merge 2MASS data into database
July 22, 02
Provide IR data for HST science planning
Create GS catalog for NGST pointing
TIPS Meeting
11
Catalogs and Surveys Branch
Summary
DSS approaching completion
GSC-II completed next year
Worked with APT to integrate DSS and GSC-II
Plans already in place for transition of HST pointing
from GSC-I to GSC-II
Discussions started for implementing improved
astrometry of HST images from the archive
July 22, 02
TIPS Meeting
12
MEMORANDUM
TO:
Distribution
DATE:
July 18, 2002
SUBJECT:
Questions and Answers from the 18 July 2002 TIPS Meeting
CADC/ECF WFPC2 Associations
Presenter – Dave Schade
Q: Do the associations span across different visits? Do the association span across different proposals?
A: The associations do not span across different proposals, but they do span across visits with the same roll
angle.
Q: Will the number of associations increase significantly if they span different proposals?
A: We do not know, but we expect the number to be small.
Q: Are the data currently available publicly? What is the current means to access the data?
A: The data do exist but not in a public form. CADC can arrange for immediately access to available data
on a case-by-case basis, and we are working with the STScI archive to make the data available publicly.
Q: What is the assessment of the quality of the current data? What fraction of the data is being considered
as problematic? How do you measure the quality of the data?
A: The assessment of data quality spans across the entire reduction process, which includes source
detection problems. Data towards the blue side seem to have more problems. Problematic data constitute
only a few percent of the entire dataset. An empirical parameter set, which includes filters used, exposure
time, etc., was used to provide quantitative measures to assess the quality of the data.
Status of PyDrizzle Software
Presenter – Warren Hack
Q: Does PyDrizzle refine shifts with delta-associations and rotation as well?
A: Yes, rotation is part of it.
Q: Does PyDrizzle give shifts in pixels?
A: In PyDrizzle, shifts are given in RA and Dec.
Q: Does OTFR result in PyDrizzle products?
A: Yes, but only the default parameters will be used for the process.
Q: When should we run PyDrizzle externally beyond the standard pipeline calibration?
A: Some examples of situation where one would want to run PyDrizzle externally include deep
observations and observations with different dither patterns.
1
Q: Are there plans to implement other techniques, such as cross-correlations, to improve on the shifts?
A: We are not clear at this point as to what specific techniques will be incorporated into the current
pipeline. We might need to implement a set of different techniques for different situations, or to develop a
specific toolsets. The group will continue to investigate the options.
Q: Is there a plan to develop a completely automated PyDrizzle pipeline?
A: We do have a plan but without a schedule.
GSC2 Status and Implementation Plans
Presenter – Brian McLean
Q: Is there a need to improve on the astrometric calibration for GSC II when other more current surveys,
such as UCAC, can provide much better astrometry?
A: There are two aspects to be considered. One is the astrometric accuracy for post-observational analysis.
The other is the accuracy needed for operational use. GSC II provides better coverage and better positions
for the faint end. For example, UCAC will be complete only down to 16th magnitude, whereas GSC II will
cover down to 21st magnitude.
Q: Will GSC II satisfy NGST requirements?
A: Studies have indicated that NGST will require 1 arc-sec accuracy. GSC II will certainly satisfy this
requirement.
2