CUC Oct 2005
Science Data Systems
Jonathan McDowell
SDS – Oct 2005
●
CIAO Evolution Overview
●
PSFs
●
Chandra Source Catalog
Staffing
●
Shrinking science team:
–
Martin Elvis back to full time research (2004)
–
Mike Wise leaving for Amsterdam (Dec 2005)
–
Eric Schlegel to U.Tex/San Antonio (Aug 2005)
–
Dan Harris to “retirement” (aka science) (Sep 05)
–
No replacements
CIAO Releases
●
CIAO 3.2.1. - Feb 05 bugfix
–
●
acis_build_badpix header problems, support for
parameter block input
CIAO 3.2.2 – Jun 05 – Repro3 support
–
Asphist fixes for multiple observations
–
Mkacisrmf bugfixes and support for new CALDB
–
New algorithm for HRC degap: will improve HETG
wavelengths
–
Change to bad pixel handling in VFAINT mode
CIAO Schedule
●
●
CIAO 3.3 – November (Tool release)
–
New science tools and bug fixes
–
reproject_image, reproject_aspect
CIAO 4.0 – 2006 (Infrastructure release)
–
Sherpa, Chips: major functionality improvements
–
Data Model improvements
●
CIAO script releases – TBD
●
SAOSAC release - TBD
CIAO 3.3
●
●
reproject_image, reproject_aspect
DM ability to handle data cubes (filter on regions,
make slices, etc.)
●
Proposal tool support for Cycle 8
●
Fixes to region area bugs
●
Fix problem with FTOOLS/CIAO clash (renamed
cfitsio library)
CIAO Future
●
Areas needing improvement for users:
–
PSFs
–
Merging observations
–
Background modelling
–
Easy fluxes and upper limits
–
Publication quality plots
–
Robust, fast and programmable spectral and spatial
fitting (improved Sherpa)
PSFs: User needs and our plans
●
User need:
–
●
●
Plan: Distribute portable SAOSAC/MARX
combo
User need:
–
●
Ability to make custom PSFs
Estimate PSF fraction easily
Plan: encircled-energy fraction tables in CALDB
and software to use them; some numerical issues
at present
PSFs: ECF table status
●
●
Normalization issue in wings: tracking down
some discrepancies
HRC-only (flat plane) values: characterizing error
introduced wrt ACIS tilted-chip, appears
significant
PSFs: SAOSAC status
●
●
●
●
SAOSAC (CXC Optics Group) consists of a
heterogeneous group of 40 programs and
associated scripts: C,C++, Fortran, Perl, Lua.
Designed as engineering tool
Available to users via CHART web interface
Moving to portable version isolated from
references to local disks, new build system
Modifying to handle numerical differences
between Sparc and x86 architecture
PSFs: SAOSAC and users
●
●
●
SAOSAC outputs ray files. Need to project to
focal plane and intersect with instrument
Will model instrument effects on PSF (but should
be optional, since PSU has its own detector
model)
Will use MARX for both projection and default
instrument model (may replace/enhance
instrument model later)
PSFs: SAOSAC/MARX driver script
●
We will write a driver script with a simple
interface. For a first release:
–
Single point source only, no grating
–
Specify spectrum (via PHA file or Sherpa model syntax?)
–
Specify off-axis angle and azimuth
–
Specify SIM position and fiducial correction
–
Specify date? (for time-dependent instrument cal?)
–
Specify instrument model (MARX or none)
Implementation Plan
●
CXC Optics to complete SAOSAC portable
infrastructure by December
●
Perform SDS testing and resolve Linux issues
●
Identify resources for driver script work
●
Documentation effort to guide users
●
Identify release schedule by end January
●
Release asynchronously with respect to CIAO?
Chandra Source Catalog
●
●
●
●
●
Chandra Level 3 Pipeline
Make a definitive catalog of sources useful for
individual and statistical studies
“Blind”, automatic processing capable of
handling different instrument configurations and
different kinds of field (crowded, extended, etc.)
Runs over full field (handle large off-axis PSFs)
Good for stars, galaxies, AGN, clusters, etc.. not
just optimized for one kind of source
Chandra Source Catalog
●
●
●
●
Not just a static catalog:
Access to extracted data products for each source
via the catalog
Dynamic interaction to produce user-configured
views with on-the-fly processing
Continuous updating as mission continues and
data becomes public
Chandra Source Catalog
●
●
●
In early years of mission, steadily improved L1
and L2 calibrated products (event lists)
We ran detect and made source lists, but these
products were provided as a guide only.
Our understanding of the instruments is now at
the point where automatic processing can go
further along the analysis chain – this was always
part of our plan.
Chandra Source Catalog
●
●
●
New standard data products: extracted spectra, ARF, RMF
and 4-band postage-stamp images for each source.
New standard data product: high quality source list
New standard data product: source properties table for each
source, with accurate positions and fluxes from 2D PSF
fitting as well as extent and variability flags
Chandra Source Catalog
●
●
●
Energy bands for ACIS:
–
Broad 0.2-7.5 keV
–
Soft 0.2-0.5 keV
–
Medium 0.5-2.0 keV
–
Hard 2.0 – 7.5 keV
Only one broad band for HRC
Bands selected after comparison with those in use in the
community; compatibility with XMM
Chandra Source Pipeline - 1
●
●
●
Reprocess event lists with latest cal, and perform high
background (flare) cleaning
Make exposure maps in each band for false edge source
suppression and construction of sensitivity map
Run multi-scale WAVDETECT in each band
–
●
●
Scales 1,2,4,8,16
Identify sources across scales and bands
Define source and (annular) background regions for each
source
Overlapping sources
Chandra Source Pipeline - 2
●
Make postage stamp fluxed image in each band
●
Use SAOSAC to make PSF for source
●
●
2D gaussian (PSF-convolved) fits to image, attempt to
identify source as single, double or complex
Derive source fluxes and position(s). Use power law spectral
fit with alpha=1 and galactic NH. (Free alpha if enough
counts, currently n=300).
●
Correct with ECF (PSF fraction) tables
●
Variability test (currently KS, soon Gregory-Loredo)
Near term enhancements
●
●
●
●
Merge pipeline: combine source data from multiple
observations and instruments, get best fluxes and long-term
variability estimate
Remove readout streaks (“out of time events”)
Revise wavdetect parameters based on pilot studies and
characterize completeness, sensitivity, false source rate
Complication: 2D fitting can discover extra sources, so
threshold is not simple to define.
Proposed enhancements
●
●
PSF fraction correction (as fn. of energy) for sources. Plan to
use fraction within given contour, rather than PSU method of
selecting contour based on given fraction. (Will be useful
user tool). Method more appropriate from blind pipeline with
avoidance of nearby sources (interactive region tweaking not
possible in pipeline)
Background maps
Longer term enhancements
●
Handle very extended sources (SNRs, clusters) using Voronoi
tesselation detect
●
Mosaic regions (M31, Orion..)
●
Gratings data (some day...)
Reviews of other processing pipes
●
●
●
●
●
ACIS_EXTRACT reviewed by M. McCullogh, F. Primini for
functionality; further tests planned. We are adopting the
important approach of energy-dependent PSF fraction
corrections to the ARF.
AE Currently uses mkpsf for PSF generation, which we don't
have confidence in.
XASSIST (A. Ptak) also under review; has no PSF fitting yet
XMM Catalog: Plan is to do simple celldetect with no
position refinement, and bright threshold (fewer sources)
CHAMP: Used in initial evaluation; no HRC support.
Operations plan
●
●
●
●
Initial evaluation based on 6 test sequences
Run larger number of sequences for science testing over a
period of months, while merge pipeline in development
Start production processing circa Fall '06
Release initial public product when about 1/5 of mission has
been processed, catalog will evolve as more is processed.
●
1 year to process first 5 years
●
Access to “live” evolving catalog and to fixed snapshots
User interface
●
●
●
●
Web-based GUI interface
Links to data, source and background events, images, spectra
and responses; searches on all columns
VO-aware interface with ability to define on-the-fly catalogs
in spreadsheet-like way; iterative searches
Later enhance to allow workflows (run queries which involve
processing on the actual data)