Gijs Verdoes Kleijn
TarGet
NOVA-OmegaCEN
Kapteyn Astronomical Institute
University of Groningen
The Netherlands
Astro-WISE’s origin: OmegaCAM,
Kilo Degree Survey (KiDS) + GT surveys
VISTA: 4m
VST: 2.6m
VIRCAM: 0.6 sq.deg.
NIR camera
OmegaCAM: 1 sq.deg.
optical camera
16 x 2kx2k detectors
32 x 2kx4k ccds
0.35” pixels
0.21” pixels
VIKING: 250 nights
KiDS:440 nights
1500 sq.deg ZYJHKs
1500 sq.deg ugri
Euclid-SGS
538
Tbyte data
322 Tbyte data
Paranal Monthly Data Rates
2007 statistics
All Current Paranal
Instruments 4% (433.2 GB)
OmegaCAM 24%
(~2500 GB)
VIRCAM 72%
(~7500 GB)
Mark Neeser
VIKING
Compared to VIKING
+
KiDS
Compared to KiDS
UKIDSS-LAS=
KIDSS-DXS=
SDSS=
CFHTLS-WIDE=
3.3 x area
1/43th area
6 x area
1/10th area
1.4 mag
shallower
0.5 mag
deeper
2 mag shallower
1.3 mag deeper
2x worse seeing
~same
AstroWISE =
share data+resources+changes
• Flexible pooling distributed survey teams
– Calibrations, Quality Control, Analysis
– Code – methods – pipelines
• Federate varying hardware resources
– Scalable hardware cornerstones: Databases,
dataservers, compute clusters
• Make change & growing insight “king”
– Physical changes, improved code/methods,
improved insight (even following the “final”
release)
Procedurized observations ->
Data Model
–Sanity
checks / QC0
–Quality
control 1
–Image
pipeline
–Calibration
–
–procedure
s
–Source
pipeline
Astro-WISE
fully object-data centric
• Data model -> Object model
• All data beyond pixel data is metadata
all pixel data <–>DataServers
all metadata <–> DataBase
(cluster of 2)
• Persistent Object Hierarchy for DB+DS
Persistent Object Hierarchy
class DBObjectMeta: # python<->db
def __new__
#makes any derived Class persistent
def __call__
# instantiate persistent object-attributes
class DBObject:
__metaclass__= DBObjectMeta
object_id = persistent(„object identifier', oidtype)
#object_id = unique: primary key in DB table
class ClassA(DBObject):
attribute = persistent(„Description‟, attribute_type, default_value)
# Make it
example = ClassA()
example.commit()
# Retrieve it, query for it
oid = xample.object_id
result = ClassA(object_id = oid)
query = ClassA.attribute==value
Persistency DataObject
#Base class for objects with files on Dataservers
from astro.database.DBMain import DBObject, persistent
class DataObject(DBObject):
filename = persistent('File part of this object', str, '')
#Store on DataServer and commit to DataBase
example =DataObject(pathname='example1.txt')
example.store()
example.commit()
#Query and retrieve from DataServer
query = DataObject.filename.like('example*')
g=query[0].retrieve()
Tracing Lineage in DB
#A simple persistent class
class ClassA(DBObject):
attribute = persistent(„Description‟, attribute_type, default_value)
#A simple persistent class with links
Class ClassB(Object)
a = persistent(„a link‟, ClassA, None)
as = persisnt(„list of links‟, ClassA, [])
b= persistent(„Link to object of ClassB‟)
#links via user-defined types in DB
Astro-Wise Pipelines
Bias pipeline
Flatfield pipeline
Photometric pipeline
Image pipeline
Source pipeline
TARGET diagram:
backward chaining
Webservice Target Processor
“system-generated
workflow”
Target processing:
the unix make metaphor
#Astro-WISE command-line request for target
awe> targethot=HotPixelMap.get(date='2003-02-14', chip=„002')
The make chain is
ReadNoise <-- Bias <-- HotPixels
# HotPixelMap inherits from ProcessTarget
class HotPixelMap(ProcessTarget):
def self.make()
# ProcessTarget: base class of objects made via processing
class ProcessTarget():
def get(date, chip) # if not exist/up-to-date then make()
def exist()
# does the target exist?
def uptodate()
# is each dependency up to date?
Make is fully recursive
Webservice QualityWISE
Webservice QualityWISE
“Bundling & tracing Quality Control”
CalTS: calibration scientist
monitoring tool
Astro-WISE publishes to VO
Quality Controlled Publishing
Quality: flags & timestamps
Target.verify() #automated inspection
Target.quality_flags #set by system
Target.inspect() # User inspection
Target.is_valid=value #set by user
0,1,2 = bad,OK,Qualified - ready for delivery
Target.timestamp_start,end # validity time range
Context: privileges
1 = Mydb
2 = Project, eg KiDS
Project favorite flag
3,4,5 = AstroWise,World,VO
user_CalFile
project CalFile
awe Calfile
5 Line Scripts
#Find ScienceFrames for a ccd named ccd53 and filter
Awe> q = (ReducedScienceFrame.chip.name == 'ccd53„) and
(ReducedScienceFrame.filter == „#841‟)
# From the query result, get the rms of the sky in image
Awe> x = [k.imstat.stdev for k in q]
# get the rms of the used Masterflat
Awe> y = [k.flat.imstat.stdev for k in q]
# Make a plot
Awe> pylab.scatter(x,y)
Customized Quality Control at
source level
Customized analysis in wavelength
space: Phot-z’s
Astro-WISE federation:
nodes & satellites
Brasil
VIKING Astrometry- global




work in progress
Challenging due to
poor spatial
sampling in dithers
Only close jitters
can be solved
globally
RMS~0.21”,
RMS_OVERLAP~0.04”
using 2MASS-PSC
HST/ACS Coma LS
Profile fitting
• AWE enables COMA LS team
• 70K objects via galphot/ galfit /
Sextractor in parallel mode
• Key results
0.2
d(F475W-F814W)/dlogR
1. Photometric catalog full survey
2. Structural parameters catalog full survey
3. Hint of extensive Intracluster GCs
population
4. Extreme faint-end luminosity function
Colour gradients
0.2
0.1
0.1
0.0
0.0
-0.1
-0.1
-0.2
-0.2
-23 -22 -21 -20 -19 -18 -17 -16 -15
F814W(AB) Absolute magnitude
-23 -22 -21 -20 -19 -18 -17 -16 -15
F814W(AB) Absolute magnitude
Bright galaxy subtraction
Carter et al. 2008; Hammer et al, 2010, ApJS; Hoyos, et al., 2010, MNRAS
+ VIRCAM, MegaCAM, LBC, ISAAC, LOFAR.
Euclid-SGS
Pan-STARRS
Astro-WISE – LOFAR
LTA
IBM- Blue Gene/P
5 Petabyte/yr
- EoR
Euclid-SGS
- LOFAR Surveys
Multi-disciplinary EScience
TASK24
Euclid-SGS
Euclid-SGS
LifeLines
165.000 patients 30 yr
Molecular
MOLGENIS
METABASE
Care
Healtcare Transaction Base
(HTB/HL7)
Euclid-SGS
Integration
Physiology
Astro-WISE paradigm
“Classical” paradigm Target processing - Awe
Forward chaining
Backward chaining
waterfall model
TIER architecture
User hunts upstream
driven by input raw data
Driven by query of user
Process in pipeline
workflow
Process in bits and pieces on the fly
Backward chaining
Operators push data
User pulls data
Results in releases
Provide information system
Static archives – publish
Dynamic archives –publish Internet
Raw data - obsolete
Raw data is sacred