ASTRO-WISE
An Astronomical Wide-field
Imaging System for Europe
Konrad Kuijken, Edwin Valentijn
Kapteyn Institute, Groningen
10 June 2002
Towards an International VO Garching bei Munchen
1
Astro-Wise
Wide field imaging
system for Europe
VST + OmegaCAM
6x WFI
1o x 1o
16k x 16k
pixels
10 June 2002
Towards an International VO Garching bei Munchen
2
Astro-Wise
Wide field imaging
system for Europe
The Problem
• Many wide-field imaging instruments, vast amounts of data
– E.g.: VST = Southern sky (30 min exp, 300 nights/y) in 3 years.
Vast amount of data! 100 Tbyte
• Science on these data should be archive-based
• Handling of the data is non-trivial
– Reduction
– Calibration and re-calibration
– Image comparisons and combinations
– Working with source lists
– Visualization
• ASTRO-WISE aim:
– provide a system for maximal data handling flexibility
10 June 2002
Towards an International VO Garching bei Munchen
3
Astro-Wise
Wide field imaging
system for Europe
Concept solution
• Environment which
–
–
–
–
Has access to all raw and calibration data
Can run and modify reduction/calibration pipelines
Can run source extraction algorithms
Archives reduced data and source lists, or regenerates
these dynamically
– Can be federated to link different data centers
• This dynamical archive continuously grows as more raw
data enter the system, and can be used for
– `small’ or large science projects
– generating and checking calibration data
– exchanging methods and scripts
• A key functionality is the link back from source data to
the original raw pixel data and calibration files
10 June 2002
Towards an International VO Garching bei Munchen
4
Astro-Wise
Wide field imaging
system for Europe
Example uses
• Deep multi-colour fields
– No need to take all data in one campaign
– Combine data of particular quality, assess results
– Select sources, visualize interesting ones, …
• 1-in-1,000,000 events spurious or not?
• Large homogeneous surveys
– Eg weak lensing maps, cluster searches, star counts
• Variability (source list- or pixel based)
– Proper motions (asteroids, nearby stars)
– Flux variations
10 June 2002
Towards an International VO Garching bei Munchen
5
Astro-Wise
Wide field imaging
system for Europe
Example uses
• Monitor instrument (calibration files)
• Planning observations
– View quality of existing data
– Build on what already exists, add more filters, more
exposure time, better seeing, …
• All this can be done on the continuously
growing archive
10 June 2002
Towards an International VO Garching bei Munchen
6
Astro-Wise
Wide field imaging
system for Europe
Philosophy
• System is not geared to a single data product,
but to being a flexible tool.
• Not focused on making a single,
homogeneous survey with static data
release(s) - but could be used for this
• Optimise interaction between users and their
data
 Dynamic archive
• Geared to optical (IR) wide-field image data
10 June 2002
Towards an International VO Garching bei Munchen
7
Astro-Wise
Wide field imaging
system for Europe
Partners
• Distribute the load/environment over National
centers, each serving their community
• Partners:
–
–
–
–
–
–
NL - Groningen (+ Leiden)
ESO – EIS
I – Napoli +
F - Terapix
D – USM +
UK - VISTA Edinburgh/Cambridge
• 1.5 MEuro grant from EU RTD programme
10 June 2002
Towards an International VO Garching bei Munchen
8
Astro-Wise
Wide field imaging
system for Europe
•
•
•
•
•
How?
Procedurizing data taking at telescope for
both science and calibration data
Full integration with data reduction
OmegaCAM FDR
Data model (classes) defined for data
reduction and calibration
Design
View pipeline as an administrative problem
10 June 2002
Towards an International VO Garching bei Munchen
9
Astro-Wise
Wide field imaging
system for Europe
Paranal
QC1
Calibration
Pipeline
re q5 6 2 - doit
re q5 6 2 - doit
re q5 4 1 - doit
Ca l5 4 2 L
s e q6 3 1
MON ITOR IN G
MON ITOR IN G
B IA S
Do m e L a m p
Ingest & statistics
USER Specific
re q5 2 2
re q5 4 7 - doit
re q5 4 7 - doit
QU IC K C H E C K
QU IC K C H E C K
H OT P IX E LS
Ca l5 2 2
Se q6 3 1
Ho t p i x e l s
S tatistics
Ca l5 4 1
re q5 2 1 - doit
re q5 2 1 - doit
R E A D N OIS E
R E A D N OIS E
re q5 3 1 - doit
re q5 3 1 - doit
DARK CURRE NT
DARK CURRE NT
re q5 3 5
Ca l5 3 5
C OLD P IX E LS
Co l d p i x e l s
Bi a s
re q5 4 2
Ca l5 4 2
D OME FLA T
Do m e f l a t
re q5 4 3
re q5 3 2
P A R TIC LE R A TE
re q5 7 1
FOC U S
re q5 6 5
U S E R -> K E Y
re q5 5 2
Ca l5 4 3
TW ILIGH T FLA T
T wi l i g h t f l a t
s e q6 3 2
F l a t fi e l d
de-B ias & Flat
re q5 4 6
Ca l5 4 5
MA S TE R FLA T
MA S TE R W E IGH T
F ri n g e f l a t
Ca l5 4 4
Ca l5 4 6 W
N IGH T S K Y FLA T
Ni g h t s k y f l a t
W e ig h t
m ake lists
sE xtr actor
m ake lists
sE xtr actor
s e q6 3 3
Individual weight
Ca l5 4 5
Ca l5 2 3
F ri n g e f l a t
FR IN GE FLA T
Se q6 3 2
S tatistics
re q5 4 4
re q5 4 5
P OIN TIN G
Ca l5 4 6
A DU c o n v .
Se q6 3 3
re q5 5 3
re q5 3 4
TR A C K IN G
C C D C TE
re q5 6 6
re q5 5 6
D E P . ON A N GLE
A S T. GU ID E C C D
re q5 2 4
re q5 6 1
E .M. C OMP A T.
S H U TTE R TIMIN G
re q5 5 1
re q5 6 7
C A ME R A P OS .
LIN E A R ITY
# A r tefacts
re q5 2 3
GA IN
re q5 3 3
LIN E A R ITY
Ca l in1
s e q6 3 4 / re q5 5 5
A s t ro m e t ri c
re f e re n c e
A pply astr om etr y
Se q6 3 4
re q5 6 9
Ca l5 6 9
re q5 6 3 - doit
Ca l5 6 3
2N D A R Y S TA N D .
S e c o n d a ry
s t a n d a rd s
ZE R OP N T K E Y
Z e ro p o i n t +
e x ti n c ti o n
re q5 6 8
P S F A N IS OTR OP .
D E TE C T. LIMIT
A str om etr ic
par am eter s
s e q6 3 5
re q5 4 8
A pply photom etr y
ILLU MIN . C OR R .
re q5 5 4
Ca l5 6 2 /PAF
E x t i n c t i o n re p o rt
re q5 6 4
Ca l5 6 4
ZE R OP N T U S E R
Z e ro p o i n t +
e x ti n c ti o n
Se q6 3 4
+ P hotom etr ic
par am eter s
Ca l5 6 5
re q5 7 2
re q5 2 5
GH OS TS - A D C
C R OS S TA LK
com bine lists
sA ssociate
Tell m e
ever ything
Us e r-> K e y
s e q6 3 6
C oadd im ages
re q5 3 6
Ca l i b ra ti o n fi l e , PAF o r l o g fi l e
H Y S TE R E S IS
Pe rfo rm e d fre q u e n tl y d u ri n g ro u ti n e p h a s e
Pe rfo rm e d o n l y d u ri n g c o m m i s s i o n i n g o r l e s s th a n twi c e a y e a r i n ro u ti n e p h a s e
A
B
10 June 2002
Se q6 3 6
C alibr ated Im g
S tatistics
B u s e s s a m e d a ta a s A
Towards an International VO Garching bei Munchen
10
Astro-Wise
Wide field imaging
system for Europe
Paranal
QC1
Calibration
Pipeline
re q5 6 2 - doit
re q5 6 2 - doit
re q5 4 1 - doit
Ca l5 4 2 L
s e q6 3 1
MON ITOR IN G
MON ITOR IN G
B IA S
Do m e L a m p
Ingest & statistics
USER Specific
re q5 2 2
re q5 4 7 - doit
re q5 4 7 - doit
QU IC K C H E C K
QU IC K C H E C K
H OT P IX E LS
Ca l5 2 2
Se q6 3 1
Ho t p i x e l s
S tatistics
Ca l5 4 1
re q5 2 1 - doit
re q5 2 1 - doit
R E A D N OIS E
R E A D N OIS E
re q5 3 1 - doit
re q5 3 1 - doit
DARK CURRE NT
DARK CURRE NT
re q5 3 5
Ca l5 3 5
C OLD P IX E LS
Co l d p i x e l s
Bi a s
re q5 4 2
Ca l5 4 2
D OME FLA T
Do m e f l a t
re q5 4 3
re q5 3 2
P A R TIC LE R A TE
re q5 7 1
FOC U S
re q5 6 5
U S E R -> K E Y
re q5 5 2
Ca l5 4 3
TW ILIGH T FLA T
T wi l i g h t f l a t
s e q6 3 2
F l a t fi e l d
de-B ias & Flat
re q5 4 6
Ca l5 4 5
MA S TE R FLA T
MA S TE R W E IGH T
F ri n g e f l a t
Ca l5 4 4
Ca l5 4 6 W
N IGH T S K Y FLA T
Ni g h t s k y f l a t
W e ig h t
m ake lists
sE xtr actor
m ake lists
sE xtr actor
s e q6 3 3
Individual weight
Ca l5 4 5
Ca l5 2 3
F ri n g e f l a t
FR IN GE FLA T
Se q6 3 2
S tatistics
re q5 4 4
re q5 4 5
P OIN TIN G
Ca l5 4 6
A DU c o n v .
Se q6 3 3
re q5 5 3
re q5 3 4
TR A C K IN G
C C D C TE
re q5 6 6
re q5 5 6
D E P . ON A N GLE
A S T. GU ID E C C D
re q5 2 4
re q5 6 1
E .M. C OMP A T.
S H U TTE R TIMIN G
re q5 5 1
re q5 6 7
C A ME R A P OS .
LIN E A R ITY
# A r tefacts
re q5 2 3
GA IN
re q5 3 3
LIN E A R ITY
Ca l in1
s e q6 3 4 / re q5 5 5
A s t ro m e t ri c
re f e re n c e
A pply astr om etr y
Se q6 3 4
re q5 6 9
Ca l5 6 9
re q5 6 3 - doit
Ca l5 6 3
2N D A R Y S TA N D .
S e c o n d a ry
s t a n d a rd s
ZE R OP N T K E Y
Z e ro p o i n t +
e x ti n c ti o n
re q5 6 8
P S F A N IS OTR OP .
D E TE C T. LIMIT
A str om etr ic
par am eter s
s e q6 3 5
re q5 4 8
A pply photom etr y
ILLU MIN . C OR R .
re q5 5 4
Ca l5 6 2 /PAF
E x t i n c t i o n re p o rt
re q5 6 4
Ca l5 6 4
ZE R OP N T U S E R
Z e ro p o i n t +
e x ti n c ti o n
Se q6 3 4
+ P hotom etr ic
par am eter s
Ca l5 6 5
re q5 7 2
re q5 2 5
GH OS TS - A D C
C R OS S TA LK
com bine lists
sA ssociate
Tell m e
ever ything
Us e r-> K e y
s e q6 3 6
C oadd im ages
re q5 3 6
H Y S TE R E S IS
Bi a s p i p e l i n e
F l a t fi e l d p i p e l i n e
Se q6 3 6
Im a g e p i p e l i n e
C alibr ated Im g
S tatistics
Ph o to m e try p i p e l i n e
Us e r s p e c i fi c
10 June 2002
Towards an International VO Garching bei Munchen
11
Astro-Wise
Wide field imaging
system for Europe
Tbyte source lists- brains make the
associations
Link -lists
Source
S
S
Source
List
V filter
A
C
S
S
E
S
A
T
Red quasars
Source Source
List
List
B filter B filter
RA 0, Dec0
RA 1, Dec1
S
RA 0, Dec0
Proper motion
S
A
E
S
t1
10 June 2002
t2
T
RA 0?, Dec0?
RA 0?, Dec0?
A
t0
Source
List
V filter
Source
List
V filter
I
RA 0, Dec0
C
S
O
Source
List
V filter
Supernovae
E
RA 1, Dec1
O
A
Source
List
B filter
Source
List
B filter
Source
List
B-V
RA 0, Dec0
I
Source
List
B filter
T
RA 0, Dec0
RA 0, Dec0
C
A
RA 0, Dec0
O
I
List
V filter
Towards an International VO Garching bei Munchen
Source Source Source Source
List
List
List
List
V filter V filter V filter V filter
RA 0, Dec0
t0
RA 0?, Dec0?
t1
RA 0?, Dec0?
RA 0?, Dec0?
t2
as fast as possible 12
tn
Astro-Wise
Wide field imaging
system for Europe
Share the load
–
–
–
–
–
–
Hardware: Beowulf processors – 32 (most cases)
Hardware: multi Terabyte disks (10 – 100)
Data reduction: derive calibration
Data reduction: run image pipeline (1 Mpx/s)
Archive: storage of images (100’s Tbyte)
Archive: storage of source parameters (1-10 Tbyte)
Network speed free parameter:
- from 5 Mb/s (24 hours/day provides full replication of all
OmegaCAM data)
- to 200 Mb/s no replication
10 June 2002
Towards an International VO Garching bei Munchen
13
Astro-Wise
Wide field imaging
system for Europe
Federate
• Calibration results
• Methods (pipelines) for processing
calibration
• Raw data
• Reduced images
• Source lists - catalogues
10 June 2002
Towards an International VO Garching bei Munchen
14
Astro-Wise
Wide field imaging
system for Europe
Concepts
• Federation maintained by a single db
• Full history tracking of all input that went into result providing
on-the fly reprocessing – tell me everything tools
• Dynamical archive - Context areas in object attributes
– Project: eg Cal, Science, Survey, Personal
– Owner: eg pipeline, developer, user
Partitions
– Strategy: Standard, Deep, Freq (monitoring)
Link to AVO
– Mode: Stare, Jitter, Dither
VO table
– Time: time stamping
• Standards
- classes/data model/ procedures
- 00 – inheritance/ persistency
- Python scripts/ libraries
10 June 2002
Towards an International VO Garching bei Munchen
15
Astro-Wise
Wide field imaging
system for Europe
DB – engine
Oracle contract:
– reference
– licenses
- consultancy
Oracle 9i:
- federation through “Advanced replication” to evolve into cross
site links (=references= pointers)
- Python I/F SQL – OCI- Oracle db
- Full oo support + Python to db persistency
- Terabyte scalability through “partitioned tables”
- Administration tools
– Interoperability: STREAMS connects to others: SyBase, MySQL
10 June 2002
Towards an International VO Garching bei Munchen
16
Astro-Wise
Wide field imaging
system for Europe
Intra-operability
peer to peer
• CVS
• “Advanced Replication” evolving to pointers
WRITE
WRITE
RE
PL
IC
ON
TI
CA
AT
I
I
PL
RE
Y
D-O
NL
N
REA
N
LICA
TIO
Y
NLY
TIO
Towards an International VO Garching bei Munchen
NL
D-O
REA
LICA
REP
REP
-O
-
AD
RE
ON
10 June 2002
RE
AD
Y
L
ON
17
Astro-Wise
Wide field imaging
system for Europe
WP s
• WP 1 – pipelines / same as ESO-DMD
» Implement and operate
• WP 2 – visualization tools
» Panorapix – fast Gbyte image display + true colour
» Graphic cards – OpenGL -
• WP3 – federated db’s
» CVS
» Oracle
• WP4 – processing power 4 Beowulf centers
• WP5 – direct access 100 Tbyte storage
• WP6 – coordinate – AVO – CERN-HEP- Datagrid
ASTRO-GRID
10 June 2002
Towards an International VO Garching bei Munchen
18
Astro-Wise
Wide field imaging
system for Europe
Schedule
• ADD – review Q2 2002- OK
• Basic operations – Q4 2003
• Evaluate and prepare for mass
production 2004
• Qualify and populate 2005
• Deliver survey system – satellites
• Links to LOFAR, JIVE,WHIRCAM, GAIA
Personal Observatory
10 June 2002
Towards an International VO Garching bei Munchen
19