TIPS/JIM
July 19, 2012
Agenda:
INS News (Danny Lennon)!
New MAST Discovery Portal Interface (Tom Donaldson)!
Demo of the Hubble Legacy Archive (HLA) Interface (Lee Quick)
Hubble Source Catalog (Steve Lubow)
Next TIPS/JIM: August 16, 2012
1
MAST Data Discovery Portal A new website for accessing MAST data Tom Donaldson June 19, 2012 Portal Goals… Build on Success of HLA Interface Access all MAST data via single web page Collaborate With VAO Access all Virtual Observatory data Flexible and Extensible Infrastructure •  Keep client and server code independent –  Single server can support mulQple clients Scripts APT Mashup Server Mobile Apps Aladin MAST Portal Other Web Pages Flexible and Extensible Infrastructure •  Server can gather and homogenize data from a wide variety of sources !
!
!
!
!
!
!
12.3!45'!
#67!
.%+*8/%'!!
,-!.%#+/0!
?>:%+6#)!
.%+*8/%'!
"#$%!
&%'()*%+!
4(96)(#7!
4#:#!
;+(7</:'!
1#'0<@!.%+*%+!
!
=68:8#)!.%+*8/%!
27#@:%+'!
A<:<+%!.%+*8/%!
27#@:%+'!
4#:#!/())#:%7!#67!#*#8)#B)%!:(!
/)8%6:'!86!$<):8@)%!C(+$#:'D!
• E.-"!
• ,-!3#B)%!
• F.,!
• G31H!
!
,-!=67%>!
.%#+/0!
Core Portal Features/Demo… All-­‐Sky VisualizaQon Displays for Images, Catalogs, Spectra Filters for Narrowing Results Flexible Layouts Moving Forward •  We want to work with archive users to make this tool as useful as possible. •  If you’d be interested in sharing ideas on features or trying out the interface as it evolves, please let us know. –  Tom Donaldson (tdonaldson@stsci.edu) –  Tony Rogers (rogers@stsci.edu) –  Karen Levay (klevay@stsci.edu) TIPS/JIM
July 19, 2012
Agenda:
INS News (Danny Lennon)!
New MAST Discovery Portal Interface (Tom Donaldson)!
Demo of the Hubble Legacy Archive (HLA) Interface (Lee Quick)
Hubble Source Catalog (Steve Lubow)
Next TIPS/JIM: August 16, 2012
1
Hubble Legacy Archive Lee Quick -­‐ TIPS mee9ng July 19, 2012 •
•
•
•
•
Goals Data History Current Work Demo Goals of the HLA Project –  Provide intui9ve searching, browsing and download of Hubble data –  Create online, enhanced data products for all Hubble data –  Develop enhanced user interface capabili9es –  hla.stsci.edu Hierarchy of HLA Image Products •
Level 1 data are the individual exposures, projected ("drizzled") onto a common frame for each visit. •
Level 2 data are the combined images (exposures with the same filter, same camera, and within the same visit). •
Level 3 images are deep combined images or mosaics crea9ng by combining data taken in different visits to the target. •
Level 4 data are false color images (a combina9on of two or three colors, depending on what is available). •
Level 5 data are the community-­‐contributed high-­‐level science products (HLSP). •
Level 0 (calibrated, unprocessed) are not available for direct searches but may oUen by found using the More... link. HLA -­‐ Data Holdings HLA (a brief) History DR2 9-­‐2008 EDR 8-­‐2007 DR1 2-­‐2008 DR4 3-­‐2010 DR3 5-­‐2009 DR6 1-­‐2012 DR5 3-­‐2011 HLA Early Data Release -­‐ August 2007 DR2 9-­‐2008 EDR 8-­‐2007 DR1 2-­‐2008 •  Data online for immediate access •  Footprint service to make it easier to browse and download •  Interac9ve image display •  HLA combined images for ~25% of ACS data •  Prototype ACS source lists •  NICMOS grism extrac9ons for ~10% of G141 data (produced at ST-­‐ECF) DR4 3-­‐2010 DR3 5-­‐2009 DR6 1-­‐2012 DR5 3-­‐2011 HLA Data Release 1 – February 2008 DR2 9-­‐2008 EDR 8-­‐2007 DR1 2-­‐2008 •
•
•
•
•
•
•
•
•
•
•
•
•
•
DR4 3-­‐2010 DR3 5-­‐2009 NEW DATA PRODUCTS HLA combined images for ~90% of ACS data NICMOS grism extrac9ons for ~80% of available data (produced at ST-­‐ECF) DAOphot source lists for ~50% of ACS images Prototype SExtractor source lists for ~10% of ACS images A few prototype combined WFPC2 images (produced at CADC USER INTERFACE ENHANCEMENTS Footprints for ACS, WFPC2, STIS, NICMOS, FOS, and GHRS STIS images accessible in HLA interac9ve display Non-­‐HLA data products included in inventory and images, accessible through DADS Ability to filter inventory table to select observa9ons of interest Readout of mouse RA/Dec posi9on in interac9ve display Plot cuts of rows or columns through images (or 2-­‐D spectra) in interac9ve display Introductory training movies on how to use the HLA DR6 1-­‐2012 DR5 3-­‐2011 HLA Data Release 2-­‐ September 2008 DR2 9-­‐2008 EDR 8-­‐2007 DR1 2-­‐2008 •
•
•
•
•
•
•
•
•
•
•
DR4 3-­‐2010 DR3 5-­‐2009 NEW DATA PRODUCTS Combined HLA images for ~95% of WFPC2 data (produced at CADC) Combined HLA images for ~95% of ACS observa9ons Both DAOphot and SExtractor source lists for 70% of ACS observa9ons USER INTERFACE ENHANCEMENTS (Shopping) Cart to allow users to download mul9ple files at once Enhanced plogng capabili9es Addi9onal catalog overlays in interac9ve display (SDSS, 2MASS, GSC2, FIRST) User defined search lists Much faster footprint interface Addi9onal advanced search op9ons (Proposal ID, Moving Targets, Filter) DR6 1-­‐2012 DR5 3-­‐2011 HLA Data Release 3 – May 2009 DR2 9-­‐2008 EDR 8-­‐2007 DR1 2-­‐2008 •
•
•
•
•
•
•
•
•
•
•
•
•
DR4 3-­‐2010 DR3 5-­‐2009 NEW DATA PRODUCTS HLA drizzled combined images for ~30% of NICMOS data Sample of 1235 ACS extracted grism spectra (provided by ST-­‐
ECF) Preliminary DAOphot and SExtractor source lists for ~80% of WFPC2 images Sample prototypes of ACS mul9-­‐visit mosaics USER INTERFACE ENHANCEMENTS User search list posi9ons are accepted in a variety of input formats Selected datasets (e.g., from clicking on footprints) can be added to cart Selec9on of datasets can be used for sor9ng and filtering (e.g., to show selected data at the top of the table) Cart can be used to retrieve both DADS and HLA data products Cart includes es9mated download data volume GALEX catalog is available for overlay in interac9ve display Advanced contrast controls in interac9ve display can be used to change color balance DR5 3-­‐2011 DR6 1-­‐2012 HLA Data Release 4 – March 2010 DR2 9-­‐2008 EDR 8-­‐2007 DR1 2-­‐2008 •
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
DR4 3-­‐2010 DR3 5-­‐2009 NEW DATA PRODUCTS Fully reprocessed NICMOS data Fully reprocessed WFPC2 data (produced at CADC) with new source lists Access to WFC3 Early Science release data New high-­‐level science products Addi9onal ACS mul9-­‐visit mosaics Updated ACS combined images USER INTERFACE ENHANCEMENTS Plogng tool allows for more dynamic user interac9on, including zooming and panning Spectra from FOS and GHRS are displayed in the plogng tool using the "Interac9ve Display" links in the inventory and images views. COS and WFC3 observa9ons are included in the search results and footprints Proprietary data are included in the search results plus footprints and can be retrieved from DADS (by privileged users) via the HLA cart Search by IPPPSOOT exposures and HST visit names (propid_visit) Inventory table updates Consolidated Help Center holds HLA resources such as FAQ, Help, Movies etc., and includes a new HLA documenta9on search tool. Source lists are dynamically generated from our databases, which improves consistency between the various catalog sources and makes it possible to download them from the inventory Search results get automa9cally gzip-­‐compressed when the browser permits it (which is usually the case). The interac9ve display shows the original image pixel value in the coordinate box when the mouse cursor stops moving. DR5 3-­‐2011 DR6 1-­‐2012 HLA Data Release 5 – March 2011 DR2 9-­‐2008 EDR 8-­‐2007 DR1 2-­‐2008 •
•
DR4 3-­‐2010 DR3 5-­‐2009 USER INTERFACE ENHANCEMENTS A completely new footprint interface, featuring: Zooming in/out capability Panning Fully integrated table, so that selec9ons in one screen remain valid in the others –  Selec9on across instruments possible –  Op9on to search through proprietary and/or non-­‐
proprietary data –
–
–
•
Improved download flow from shopping cart –
–
•
•
•
•
•
•
•
Single or sequen9al download Par9al downloads can be restarted Sources clickable in interac9ve display When filtering RA or Dec columns, the values may be entered either in decimal degrees or in "sexagesimal" format (hh:mm:ss, dd:mm:ss) Mul9band source list (ACS and WFPC2) NEW DATA PRODUCTS Completed WFPC2 processing (over 300 new visits with source lists) New High Level Science Products ECF-­‐generated ACS and NICMOS extracted GRISM data hosted at STScI DR5 3-­‐2011 DR6 1-­‐2012 HLA Data Release 6 – January 2012 DR2 9-­‐2008 EDR 8-­‐2007 DR1 2-­‐2008 •
•
•
•
•
•
•
•
•
DR4 3-­‐2010 DR3 5-­‐2009 NEW DATA PRODUCTS WFC3 image data: includes ~62% of data from observa9ons that have a release date up to 11-­‐30-­‐2011 New High Level Science Products Addi9onal ACS (updated to release date 2-­‐16-­‐2011), WPFC2 and NICMOS data USER INTERFACE ENHANCEMENTS Interac9ve Display enhancements. The inventory and images interfaces now include a "Select All" buoon that can be used to select all the datasets in the current list. The HLA Simple Image Access Protocol server is now fully compliant with the Virtual Observatory standards. The HLA SIAP server will be added to the VO registry for easy access to HLA data from scripts and other VO tools. Searches using lists of posi9ons now show the number of lines in the list. DR6 1-­‐2012 DR5 3-­‐2011 Current work – WFC3 Source lists •
•
Based on exis9ng HLA pipelines (ACS, NICMOS) with structural changes Some algorithmic changes –  New alignment module performs very well on crowded fields –  Sky equaliza9on included –  More rigorous handling of header informa9on •
Source lists are undergoing quality review, will be released later Current work – Spectral Data •
•
•
•
•
Strong desire to beoer integrate spectral data in the HLA General purpose spectral pipeline not yet available -­‐ focus on High Level Science Products First release will include collec9ons of processed spectra (e.g., STARCAT) •  Interface modifica9ons to handle spectral products •  Preview code replaces image previews and cutouts •  New Spectral Plogng tool with beoer func9onality to view and interact with spectra Difficul9es: •  Spectral data products inhomogeneous in proper9es, combina9on process •  Naming conven9ons in development •  Large number of data points in some spectra can challenge some browsers •  User preferences not yet established -­‐ may need to iterate on specifics Expect ~200 products at first release TIPS/JIM
July 19, 2012
Agenda:
INS News (Danny Lennon)!
New MAST Discovery Portal Interface (Tom Donaldson)!
Demo of the Hubble Legacy Archive (HLA) Interface (Lee Quick)
Hubble Source Catalog (Steve Lubow)
Next TIPS/JIM: August 16, 2012
1
HUBBLE SOURCE CATALOG
Steve Lubow (STScI)
Collaborator: Tamas Budavari (JHU)
July 19, 2012
DESCRIPTION
•
Hubble Legacy Archive provides Source
Extractor source lists for ACS/WFC and
WFPC2 images (combined within a visit for
single filter & detector).
•
Crossmatch the source lists by position from
different overlapping images. Matched sources
correspond to single astronomical object.
•
Matches contain spectral and temporal
properties of astronomical objects
Source Detections
count
1000
100
•
Catalog has matches of detected and
nondetected sources. Improved astrometry.
Eta Car
10
5
10
50 100
cosmos
HDF UDF
500 1000
# overlapping images
RESULTS
Budavari & Lubow 2012, arXiv:1206.0644v1, ApJ, submitted
astrometric and matching
algorithms. All in SQL Server database.
• 45
million detections and 15 million
color nondetections
• Corrected
relative astrometry for about
50% of ACS/WFC and WFPC2 images
• Matches
time and spectral coverage
• Conservative
matching
" matches per mas
• Novel
200 000
Astrometry
150 000
After correction
100 000
Before correction
50 000
0
0
20
40
60
Σ !mas"
Coverage Map
80
100
CHECK ON RESULTS
• Matches
based on position
" pairs per bin
8 ! 106
Matched pairings
6
6 ! 10
ACS!WFC
4 ! 106
2 ! 106
Random pairings
0
• Check
• Should
be nearly zero
0.2
0.4
0.6
0.8
fractional flux difference
1.0
1.0
normalized ! pairs
on flux differences in same
match for same filter, detector, and
instrument (different visits)
0.0
0.8
WFPC2
0.6
0.4
0.2
0.0
0.00
ACS
0.02
0.04
0.06
0.08
0.10
fractional flux difference
0.12
0.14
STATUS
•
3 year NASA AISRP grant for Budavari & Lubow ended 6/30/12
•
Beta Version online http://archive.stsci.edu/hst/hla_cat/
•
Future activities supported by MAST and Hubble Division may include
•
Testing and tuning matching parameters
•
Improve Source Lists
•
Astrometric corrections in HLA images
•
Add ACS/HRC, WFC3, NICMOS
•
Add DAOPhot Source Lists
•
User interfaces: Graphical and CAS Jobs (SQL)
EXAMPLE
http://archive.stsci.edu/hst/hla_cat/