Sky Survey Data Management
Bob Mann
Wide-Field Astronomy Unit
University of Edinburgh
1
Logistics
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
Introductions
Wireless
Toilets
Dinner
Tea, coffee & lunch breaks
Presentations
2/11
e-Science Institute
ƒ Mission
ƒ “To facilitate the e-Science community”
ƒ First phase: community building
ƒ Training events: lectures and hands-on
ƒ Now: supporting the community
ƒ Focus on longer-term issues – esp. research
ƒ Themes – series of connected workshops
3/11
Theme Programme
Kick-Off Meeting
The Transient Sky
Sky Survey Data Management
Future Directions of N-Body Simulations in
Cosmology
ƒ Sky surveys and data mining
ƒ Weak lensing shape measurement
ƒ
ƒ
ƒ
ƒ
4/11
Outcomes from Theme
ƒ Presentations on the web
ƒ New understanding & collaborations
ƒ Special issue of New Astronomy Reviews
ƒ ~6-8 mini-review articles of ~10-15 pages each
ƒ Targeting major topics addressed during Theme
ƒ Who’d like to co-author one on data management?
5/11
Motivations for this workshop
Next generation of sky surveys will be different
qualitatively, as well as quantitatively
ƒ “SDSS model” is being stretched & must break
ƒ SDSS model (also GALEX, UKIDSS, VISTA, PS1?,…)
ƒ Static catalogues in relational database on single server
ƒ Data accessed via webforms:
VO access available, but less well used, as less useful
ƒ Most users download data to desktop for analysis
ƒ Large statistical analyses done with downloaded copies
ƒ But, data volumes will soon be too great for this…
(when? – VISTA and PS1 mark the boundary) 6/11
What will be different?
ƒ Network speed not keeping up with data volumes
ƒ At some point people will stop being able to download the
size of dataset they want to work with
ƒ Analysis code must be run at the data centre
ƒ Catalogues too large for a single-server RDBMS
ƒ Partitioning databases over a cluster poses new problems:
not all RDBMS do this well
ƒ Science drivers changing
ƒ Time domain – enabled by great increased in étendue
ƒ Weak lensing – does this fit within standard pipeline?
7/11
Data management issues
ƒ Do we stick with standard RDBMSs?
ƒ Are there better technologies?
ƒ How do we support new science drivers?
ƒ What are their requirements in detail?
ƒ What role does the VO play in all this?
ƒ What about forthcoming radio surveys?
8/11
Workshop Programme
ƒ Three components
ƒ The state of the art
ƒ What we do now – and where it’s starting to fail
ƒ Planning for future sky surveys
ƒ What we’ll have to do & where the problems lie
ƒ Enabling technologies
ƒ What new technologies and techniques might help
(Practical constraints have messed up order somewhat)
9/11
Data management issues
ƒ Do we stick with standard RDBMSs?
ƒ Are there betterVDFS
technologies?
– Nigel Hambly
Astro-Wise
– Gijs
Verdoes
Kleijn
MonetDB
–
Martin
Kersten
ƒ How do we support
science
drivers?
SDSSnew
–
Ani
Thakar
SciDB – Jacek Becla
ƒ What are their requirements in detail?
Hadoop – Miles Obsorne
PS1 Transients – Ken Smith
ƒ What role does the VO play in all this?
Euclid – Tom Kitching
Virtual Observatory
– Keith Noddle
LSST – Timstatus
Axelrod
ƒ What about forthcoming radio surveys?
ASKAP and SKA – Kevin Vinsen
LOFAR – Michael Wise
10/11
Workshop programme
ƒ Plenty of time for discussion
ƒ 45 min slots, but speakers aiming for ~30-35
ƒ All tomorrow afternoon for discussion
ƒ Discuss as we go along and identify topics
to discuss at greater length tomorrow
ƒ With an eye to New Astronomy Reviews paper
11/11