V-GISC – SIMDAT
Gil Ross (Met Office UK)
NESC Workshop 6th to 8th September 2005
© Crown copyright 2005
NESC Workshop 6th-8th September 2005
Page 1
Contents
 Part 1
 World Meteorological Organisation
 Data - Communications now
 WMO Operational Meteorology
 Real-Time Daily Data Flows
 WMO Information System
 GISC - Global Information System Centre
 data Portal data Mirror
 Part 2
 vGISC and SIMDAT
 SIMDAT EU Project
 SIMDAT Meteorology
© Crown copyright 2005
NESC Workshop 6th-8th September 2005
Page 2
WMO is….
 UN Agency (1950)
derived from IMO (1873)
 Co-ordinate meteorology
weather
 observations
 forecasts
climate change
hydrology
disaster mitigation
 187 Members
© Crown copyright 2005
NESC Workshop 6th-8th September 2005
Page 3
WMO - Data Exchange
 Now
 GTS Global Telecommunications System
Observations
Forecasts
Warnings
 Private Network
Node to Node network
NOT an Internet
 Future
 WIS - WMO Information System
Replacement for GTS
Portal for Operational Meteorology
Catalogue driven based on  ISO 19115 Metadata WMO Core Profile
© Crown copyright 2005
NESC Workshop 6th-8th September 2005
Page 4
GTS Real-Time Data in Met Office UK
Main Message Switch
 300,000 bulletins received per day
 5.2 million bulletins sent/day.
Main File Server
 17,000 files received/day
 27,000 files sent/day
Other routes – e.g. Satellite data ~ 27GB/day
Ephemeral and Persistent databases
 Ephemeral completely repopulated every day
 Persistent < 1 TB new data per day
© Crown copyright 2005
NESC Workshop 6th-8th September 2005
Page 5
Surface Observations 0000Z 3rd September
http://www.metoffice.gov.uk/research/nwp/observations/data_coverage/index.html
© Crown copyright 2005
NESC Workshop 6th-8th September 2005
Page 6
Aircraft observations 000Z 3rd September
© Crown copyright 2005
NESC Workshop 6th-8th September 2005
Page 7
WIS - Information collection data flow
Various communication
networks
DCPC
NC
NC
NC
GISC
NC
DCPC
GISC
NC
GISC
NC
DCPC
NC
GISC
NC
NC
NC
GISC
Global Information
System Centre
© Crown copyright 2005
NC
NC
DCPC
NC
Data Collection or Product Centre
National Centre
NESC Workshop 6th-8th September 2005
Page 8
WIS - Information distribution data flow
Various communication
networks
DCPC
NC
NC
GISC
NC
NC
GISC
GISC
DCPC
DCPC
NC
GISC
National users
National users
NC
DCPC
NC
NC
Routine dissemination
(Internet, private network,
satellite, etc.)
© Crown copyright 2005
National dissemination
Request/reply (Internet)
NESC Workshop 6th-8th September 2005
Page 9
Part 2
© Crown copyright 2005
NESC Workshop 6th-8th September 2005
Page 10
V-GISC
V-GISC – SIMDAT
Gil Ross
Met Office
SIMDAT / NESC Workshop Sept 05
11
SIMDAT
vGISC

V-GISC
Instead of each National Met Service having a GISC
(Global Information System Centre)

a “distributed” or “virtual” GISC

3 NMSs –
 Meteo France
 Deutcher Wetterdienst DWD
 Met Office UK – MetO

and two DCPCs Data Collection and Production Centre
 ECMWF
 EUMETSAT
SIMDAT / NESC Workshop Sept 05
12
SIMDAT
SIMDAT
V-GISC
 4 years project funded by the EU
 Contract with EU was signed on 1 September 2004
 Data Grids for Process and Product Development using Numerical
Simulation and Knowledge Discovery
 SIMDAT focuses on 4 application area:
 product design in automotive and aerospace,
 process design in pharmacology
 service provision in meteorology
 Budget of 11 M € of which 10.5% for meteorological activity
 project managed by ECMWF
SIMDAT / NESC Workshop Sept 05
13
SIMDAT
V-GISC
SIMDAT - Consortium
SIMDAT / NESC Workshop Sept 05
14
SIMDAT
SIMDAT
V-GISC
 Seven Grid-technology areas have been identified to achieving
SIMDAT objectives:
 Integrated Grid infrastructure offering basic services to applications
 Access to data distributed on Grid sites
 Management of Virtual Organisation
 Workflows
 Ontologies
 Integration of analysis services
 Knowledge Services
 Not all will be addressed in Meteo - SIMDAT
SIMDAT / NESC Workshop Sept 05
15
SIMDAT
SIMDAT - Strategy
V-GISC
 The heart of the issue is data
Phase 1: Connectivity
. Deployment of Grid
infrastructure with particular
attention to data transport
and management.
. Distributed DB access
SIMDAT / NESC Workshop Sept 05
Phase 2: Interoperability
. Virtual Data
Repository
. Introduction of grid
technologies research
Phase 3: Knowledge
Workflows for nextgeneration aggregated
knowledge capture,
discovery and mining.
16
SIMDAT
SIMDAT - vGISC
V-GISC
 vGISC working groups adopted by SIMDAT (meteorology)
 SIMDAT is a technical solution for vGISC type operations
 will NOT deliver a vGISC
 has specified requirements similar to vGISC
 is investigating a vGISC Virtual Organisation
 is developing catalogue operations
 has developed message infrastructure
 SIMDAT 18 month test system due - NOW!
SIMDAT / NESC Workshop Sept 05
17
SIMDAT
SIMDAT requirements
SIMDAT / NESC Workshop Sept 05
V-GISC
18
SIMDAT
DCI Distributed Architecture
SIMDAT / NESC Workshop Sept 05
V-GISC
19
SIMDAT
VGISC Demonstrator: UKMO local db (1)
V-GISC
Internet
JEDDS
Internet
SIMDAT / NESC Workshop Sept 05
20
SIMDAT
VGISC Demonstrator: UKMO local db (2)
V-GISC
 Local database is a derivative of JEDDS – the ‘Joint
Environmental Dynamic Data Server’
 Originally developed by Met Office to provide data on
demand to UK military
 Features:
 Live data feed; continuously refreshed
 Serving global TAF, METAR & GRIB bulletins
 Request-oriented SOAP web-service interface
SIMDAT / NESC Workshop Sept 05
21
SIMDAT
VGISC Demonstrator: UKMO local db (3)
V-GISC
Application DMZ
Synchr
o
Pull
(h
Portal &
Catalogue
(local)
nisatio
n
ttp)
Internet
Internal
Pu
s
EAL4
Firewall
h
(ft
p)
Database DMZ
Pull
FROST
Portal &
Catalogue
(Remote)
Router
User
SIMDAT
Demonstrator
Deployment
July 2005
Jeremy Tandy
JEDDS ++
SIMDAT / NESC Workshop Sept 05
22
SIMDAT
V-GISC
Catalogue Status
in Test system
SIMDAT / NESC Workshop Sept 05
23
SIMDAT
Working decisions (1)
V-GISC
 Teleconference, wiki & email used for remote collaboration
 focus on HOW the catalogue operates, not the minutiae of
metadata standards
 use a ‘random’ UID scheme and not try to ‘overload’ identifiers
with semantics
 use a simple ‘Google-like’ and/or directory-based search
interface
 3 classes of search – by Keyword, time and/or location
 ignore internationalisation for now
 Issue: WMO Core (& ISO19115) is TOO flexible – there are too
many options for spatial & temporal descriptions … we need
best practice ‘application schemas’
SIMDAT / NESC Workshop Sept 05
24
SIMDAT
Working decisions (2)
V-GISC
 Metadata
 create linking schemas using the WMO Core Profile by using
additional namespaces (i.e. <v:id>)
 metadata documents should be split into a ‘hierarchy’ of XML
‘fragments’ are used within the catalogue
 model for merging metadata fragments will uses Java
templating rather than an XSLT-based approach … key issue to
demonstrate the concept is WHAT the fragments are not the
mechanics of HOW they are merged. This position may be
revised post-demonstrator
 Agreement: nomenclature …
•
a specific SYNOP (12:00UTC, 16/06/05 at Heathrow) is an instance
•
the SYNOP ‘product’ is a generalised
identify a specific synop instance
SIMDAT / NESC Workshop Sept 05
instance that requires instantiation parameters to
25
SIMDAT
What Now?
V-GISC
 Metadata demo
 http://www.ecmwf.int/products/realtime/d/vgisc/
 Re-consider parked issues
 extend beyond real-time data
 Project tasks
SIMDAT / NESC Workshop Sept 05
26
SIMDAT