Virtual Observatory Status
Keith Noddle
WFAU Tech Lead
Keith Noddle, WFAU Tech Lead
Topics
●
●
●
●
●
●
●
●
●
Goal of VO
IVOA
Under the Hood
Scalability and Extensibility
New Opportunities
Toolkits and Applications
Deployment scenarios
CADC
Why would you?
Keith Noddle, WFAU Tech Lead
Goal: Turning this...
Keith Noddle, WFAU Tech Lead
...into this
Keith Noddle, WFAU Tech Lead
Assets
Desktop
Data centre
Algorithm
Algorithm
Data set
GUI
Application
CPU
Data set
Community
CPU
Resource list
User details
Keith Noddle, WFAU Tech Lead
Assets turned into client & services
Desktop
Data centre
GUI
GUI
with
Astro
Application
Runtime
GUI
App server
Application
Data
set
App
App
GUI
Data server
Application
Data
set
Community
VO support
GUI
App server
Application
GUI
VOSpace
Application
User
details
App
Resource
list
GUI
Community
Application
Keith Noddle, WFAU Tech Lead
GUI
Registry
Application
International Virtual Observatory
Alliance
IVOA: Standards
Keith Noddle, WFAU Tech Lead
Working Groups & Standards
●
Work Groups & Standards for:
–
–
Registry
Grid & Web Services
●
●
–
Data Access Layer
●
–
–
–
●
VOSpace
Universal Worker Service (UWS)
ADQL/TAP, SIAP, SSAP, SLAP etc
Data Models
Applications
Theory
At last count there are ~25 standards agreed or being
ratified
Keith Noddle, WFAU Tech Lead
AstroGrid: Under the Hood
Keith Noddle, WFAU Tech Lead
VODesktop...
Keith Noddle, WFAU Tech Lead
...Tip of the Iceberg
Keith Noddle, WFAU Tech Lead
...Tip of the Iceberg
Keith Noddle, WFAU Tech Lead
...Tip of the Iceberg
Astro Runtime [AR]
Astro Runtime [AR]
Community
Community
Registry
VOSpace
CEA [UWS]
DSA
App App App
Data Data Data
Keith Noddle, WFAU Tech Lead
Registry
VOSpace
CEA [UWS]
DSA
App App App
Data Data Data
AstroGrid
VODesktop
Keith Noddle, WFAU Tech Lead
AstroGrid
VODesktop
Astro Runtime [AR]
Keith Noddle, WFAU Tech Lead
AstroGrid
VODesktop
Astro Runtime [AR]
Community
Keith Noddle, WFAU Tech Lead
AstroGrid
VODesktop
Astro Runtime [AR]
Community
Registry
Keith Noddle, WFAU Tech Lead
AstroGrid
VODesktop
Astro Runtime [AR]
Community
Registry
VOSpace
Keith Noddle, WFAU Tech Lead
AstroGrid
VODesktop
Astro Runtime [AR]
Community
Registry
VOSpace
CEA [UWS]
App
App
App
Keith Noddle, WFAU Tech Lead
AstroGrid
VODesktop
Astro Runtime [AR]
Community
App
Registry
VOSpace
CEA [UWS]
DSA
App
App
Data
Keith Noddle, WFAU Tech Lead
Data
Data
AstroGrid
VODesktop
Astro Runtime [AR]
Community
App
Registry
VOSpace
CEA [UWS]
DSA
App
App
Data
Keith Noddle, WFAU Tech Lead
Data
Data
Registry
The Virtual Observatory's Yellow Pages
Keith Noddle, WFAU Tech Lead
AstroGrid
VODesktop
Astro Runtime [AR]
Community
App
Registry
VOSpace
CEA [UWS]
DSA
App
App
Data
Keith Noddle, WFAU Tech Lead
Data
Data
DAL : Accessing Data
Catalogue [TAP]
Keith Noddle, WFAU Tech Lead
DAL : Accessing Data
Catalogue [TAP]
Images [SIAP]
Keith Noddle, WFAU Tech Lead
DAL : Accessing Data
Catalogue [TAP]
Images [SIAP]
Time Series [STAP]
Keith Noddle, WFAU Tech Lead
DAL : Accessing Data
Catalogue [TAP]
Images [SIAP]
Time Series [STAP]
etc
Keith Noddle, WFAU Tech Lead
DAL + DM : Accessing Data
Data Model
Catalogue
Spectra
Keith Noddle, WFAU Tech Lead
Data Model
DAL : DSA architecture
ADQL QUERY
OBJECT BUILDER
CONE
INTERNAL
QUERY
OBJECTS
ASYNC JOBS
(with queue)
DATA
CACHE
(etc)
SYNC JOBS
(no queue)
RESULTS
FORMATTER
CEA
ADQL TO SQL
TRANSLATOR
TAP
NATIVE
SQL
SQL
RDBMS
WITH
JDBC
JDBC
RESULTSET
VOTABLE, CSV, HTML etc
INTERFACE
LAYER
JOB MANAGEMENT
LAYER
Keith Noddle, WFAU Tech Lead
RDBMS ACCESS
LAYER
Accessing Data
Note: The DSA service simply publishes
existing data to the VO, nothing about the
existing database set-up changes as a result
of deploying DSA.
Keith Noddle, WFAU Tech Lead
AstroGrid
VODesktop
Astro Runtime [AR]
Community
App
Registry
VOSpace
CEA [UWS]
DSA
App
App
Data
Keith Noddle, WFAU Tech Lead
Data
Data
Processing in the VO
Application
(via AR?)
Registry
Common
Execution
Connector
Application as
.exe file (local)
Common
Execution
Connector
Application as
.exe file (grid)
Common
Execution
Connector
Application as
.HTTP service
Common
Execution
Connector
Application as
Java class
Keith Noddle, WFAU Tech Lead
AstroGrid
VODesktop
Astro Runtime [AR]
Community
App
Registry
VOSpace
CEA [UWS]
DSA
App
App
Data
Keith Noddle, WFAU Tech Lead
Data
Data
Distributed Storage
DSA
Process
Process
VOSpace
Keith Noddle, WFAU Tech Lead
Desktop
AstroGrid
VODesktop
Astro Runtime [AR]
Community
App
Registry
VOSpace
CEA [UWS]
DSA
App
App
Data
Keith Noddle, WFAU Tech Lead
Data
Data
Access control
Desktop application, e.g. Taverna
{User-name, password, community}
Astro Runtime
{User-name,
password}
X.509/RFC3820
proxy
X.509/RFC3820
proxy
Community/
MyProxy
Worker services
X.509/RFC3820
proxy
User details
CPU, storage,archives etc.
Keith Noddle, WFAU Tech Lead
Access policy
Scalability and Extensibility
●
●
Web services
Load balancing / Fail-over
–
–
●
Queues
–
–
●
Multiple deployments
Off the shelf sysAdmin tasks
Execution
Pending
Tunable
–
–
Service
Container
Keith Noddle, WFAU Tech Lead
Scalability and Extensibility
SQL
INTERNAL
QUERY
OBJECTS
ASYNC
JOBS (with
queue)
DATA
CACHE
(etc)
SYNC JOBS
(no queue)
RESULTS
FORMATTER
CEA
ADQL TO XXX
TRANSLATOR
CONE
ADQL QUERY
OBJECT BUILDER
TAP
VOTABLE, CSV, HTML etc
Keith Noddle, WFAU Tech Lead
XML
Hadoop
New Opportunities
OGSA-DAI
Keith Noddle, WFAU Tech Lead
New Opportunities
OGSA-DAI: Joins
Keith Noddle, WFAU Tech Lead
New Opportunities
OGSA-DAI: Distributed joins
Keith Noddle, WFAU Tech Lead
New Opportunities
OGSA-DAI: Activities for astronomy
Keith Noddle, WFAU Tech Lead
New Opportunities
OGSA-DAI: Astronomy cross-match
Keith Noddle, WFAU Tech Lead
New Opportunities
OGSA-DAI: Distributed Query Processing (DQP)
Keith Noddle, WFAU Tech Lead
New Opportunities
OGSA-DAI: DQP over TAP
Keith Noddle, WFAU Tech Lead
New Opportunities
OGSA-DAI: TAP over DQP over TAP
Keith Noddle, WFAU Tech Lead
Toolkits
Keith Noddle, WFAU Tech Lead
AstroGrid
VODesktop
Astro Runtime [AR]
Community
App
Registry
VOSpace
CEA [UWS]
DSA
App
App
Data
Keith Noddle, WFAU Tech Lead
Data
Data
Astro Runtime
●
An API for VO services
Keith Noddle, WFAU Tech Lead
Astro Runtime
●
●
An API for VO services
A client side layer
Keith Noddle, WFAU Tech Lead
Astro Runtime
●
●
●
An API for VO services
A client side layer
Accessible from any language supporting XMLRPC
Keith Noddle, WFAU Tech Lead
Astro Runtime
●
●
●
●
An API for VO services
A client side layer
Accessible from any language supporting XMLRPC
As well as natively from Java
Keith Noddle, WFAU Tech Lead
Astro Runtime
●
●
●
●
●
An API for VO services
A client side layer
Accessible from any language supporting XMLRPC
As well as natively from Java
Python abstraction layer
–
–
More “Pythonic”
Aggregates some lower level functions.
Keith Noddle, WFAU Tech Lead
SAMP
●
Simple Application Messaging Protocol
Keith Noddle, WFAU Tech Lead
SAMP
●
●
Simple Application Messaging Protocol
Client-side applications message bus
Keith Noddle, WFAU Tech Lead
SAMP
●
●
●
Simple Application Messaging Protocol
Client-side applications message bus
Client applications interoperate...
Keith Noddle, WFAU Tech Lead
SAMP
●
●
●
●
Simple Application Messaging Protocol
Client-side applications message bus
Client applications interoperate...
...without having to know the details of how the
other application works
Keith Noddle, WFAU Tech Lead
SAMP
●
●
●
●
●
Simple Application Messaging Protocol
Client-side applications message bus
Client applications interoperate...
...without having to know the details of how the
other application works
Publish-Subscribe mechanism
Keith Noddle, WFAU Tech Lead
SAMP
●
●
●
●
●
●
Simple Application Messaging Protocol
Client-side applications message bus
Client applications interoperate...
...without having to know the details of how the
other application works
Publish-Subscribe mechanism
Simple to implement / add to existing applications
Keith Noddle, WFAU Tech Lead
SAMP
●
●
●
●
●
●
●
Simple Application Messaging Protocol
Client-side applications message bus
Client applications interoperate...
...without having to know the details of how the
other application works
Publish-Subscribe mechanism
Simple to implement / add to existing applications
SAMP defines message types
Keith Noddle, WFAU Tech Lead
SAMP
●
●
●
●
●
●
●
●
Simple Application Messaging Protocol
Client-side applications message bus
Client applications interoperate...
...without having to know the details of how the
other application works
Publish-Subscribe mechanism
Simple to implement / add to existing applications
SAMP defines message types
New types can be added without restriction...
Keith Noddle, WFAU Tech Lead
SAMP
●
●
●
●
●
●
●
●
●
Simple Application Messaging Protocol
Client-side applications message bus
Client applications interoperate...
...without having to know the details of how the
other application works
Publish-Subscribe mechanism
Simple to implement / add to existing applications
SAMP defines message types
New types can be added without restriction...
...but need formal IVOA approval to be added to
standard.
Keith Noddle, WFAU Tech Lead
SAMP
VODesktop
TopCat
Keith Noddle, WFAU Tech Lead
Aladin
A Partial list of VO-Capable
Applications & Libraries
●
●
●
●
●
●
●
●
●
VODesktop
Topcat
SPLAT-VO
Aladin
VOSpec
VOPlot
Montage
Specview
Web Portals:
–
–
●
●
●
●
●
●
AstroRuntime
PLASTIC / SAMP
Python AR Library
STILTS
SAVOT
JAVOT
NVO
JVO
Keith Noddle, WFAU Tech Lead
New generation of Apps emerging
●
●
●
VOWarehouse
SAMP over XMPP
Distributed cross-matching
Keith Noddle, WFAU Tech Lead
What to deploy?
So many services, so little time...
Keith Noddle, WFAU Tech Lead
It depends...
Keith Noddle, WFAU Tech Lead
It depends...
Small data holding:
DAL(DSAs)
Keith Noddle, WFAU Tech Lead
It depends...
Small data holding:
DAL(DSAs)
Multiple data holdings:
DAL
Publishing Registry
Keith Noddle, WFAU Tech Lead
It depends...
Small data holding:
DAL(DSAs)
Multiple data holdings:
DAL
Publishing Registry
Multiple data holdings:
DAL
Publishing Registry
Community
VOSpace
Keith Noddle, WFAU Tech Lead
It depends...
Small data holding:
DAL(DSAs)
Multiple data holdings:
Large Facility
DAL
Publishing Registry
CEA/UWS
Community
VOSpace
DAL
Publishing Registry
Multiple data holdings:
DAL
Publishing Registry
Community
VOSpace
Keith Noddle, WFAU Tech Lead
It depends...
Small data holding:
DAL(DSAs)
Multiple data holdings:
DAL
Publishing Registry
CEA/UWS
Community
VOSpace
DAL
Publishing Registry
Multiple data holdings:
DAL
Publishing Registry
Community
VOSpace
Large Facility
Large Facility
DAL
Full Registry
CEA/UWS
Community
VOSpace
Keith Noddle, WFAU Tech Lead
It depends...
Small data holding:
DAL(DSAs)
Multiple data holdings:
Large Facility
DAL
Publishing Registry
CEA/UWS
Community
VOSpace
DAL
Publishing Registry
Multiple data holdings:
DAL
Publishing Registry
Community
VOSpace
Large Facility
DAL
Full Registry
CEA/UWS
Community
VOSpace
Keith Noddle, WFAU Tech Lead
Small Facility:
Community
VOSpace
Pub Registry
For example at ROE
●
Services
–
–
–
–
–
–
DSA/TAP
SIA/SSA
Community
VOSpace
Registry
VO enabled services
Keith Noddle, WFAU Tech Lead
For example at ROE
●
Services
–
–
–
–
–
–
●
DSA/TAP
SIA/SSA
Community
VOSpace
Registry
VO enabled services
Developments
–
–
Applications
Research made possible using VO
Keith Noddle, WFAU Tech Lead
Another example: CADC
VO at the heart of a Data Centre
Keith Noddle, WFAU Tech Lead
CADC
●
New perspective on VO standards
–
–
●
VO has moved from periphery to core
Re-using the intellectual investment of the IVOA
Applying to new CADC use cases:
–
–
–
–
–
Annotation service: TAP, GMS, SSO
PI distribution: VOSpace, GMS, SSO
Collaboration repositories: VOSpace, GMS, SSO
Virtual machine repositories: VOSpace, GMS, SSO
Sharing virtual machines: VOSpace, GMS, SSO
Keith Noddle, WFAU Tech Lead
CADC
Keith Noddle, WFAU Tech Lead
CADC
Keith Noddle, WFAU Tech Lead
So, why would you adopt VO?
Keith Noddle, WFAU Tech Lead
So, why whould you adopt VO?
Actually, why wouldn't you??
Keith Noddle, WFAU Tech Lead