JERICO WP5 Data management and distribution Alessandro Crise Istituto Nazionale di Oceanografia e di Geofisica Sperimentale JERICO Project Kick-Off meeting Maison de la Recherche Paris, 24-25 May 2011 JERICO WP5 Data management and distribution: main goal The main goal is to design the JERICO Data Flow and Management infrastructure based on the implementation of an end-to-end ‘open and free’ conduit from data to users and upon the guidelines proposed in the GOOS Report 148 (2005). The implementation will be consistent and complementary with the major European initiatives for the establishment and coordination of infrastructures for the management and distribution of data and products in Europe which can use JERICO data (SeaDataNet, EuroGOOS Regional Centres, MyOcean). YET ANOTHER DATA MANAGEMENT SYSTEM?: NO! WP5 Structure JERICO will need to ensure that the flow of real-time and delayed mode will be reliable, accessible and easy to distribute. The WP5 structure: • Creating value for measured data: from raw data issued from the sensors to validated data stored and marked. • Facilitating data and meta-data flow to existing data repository infrastructures (SeaDataNet); • Helping in delivering real-time JERICO data to ROOSs and My Ocean Task 5.1: Quality assurance: create value for measured data Task responsible: OGS Partners: HCMR Duration M1-M42 • This task is aimed at adding value to observations by associating them with characteristic uncertainties. • By definition, the “expected” accuracy associated with an observation is an attribute of the measurement chain that produces it. Task 5.1: Quality assurance objectives The objectives are: Collecting pre-existing knowledge regarding the key identified sensors and critically revise state-of-art literature on in lab and in situ performances of the most popular sensors (in conjunction with WP4) defining procedures for ascribing a total uncertainty to measurements that will account for the actual circumstances and conditions under which the measurements were taken. Testing and verifying uncertainty computational procedures Task 5.1: Quality assurance methodologies . Methodology A novel model of a-priori uncertainty will be proposed as a sum of fixed and time-dependent source of error and uncertainty (precision and accuracy, drifts, experimental uncertainties, the representativity error, etc.) The variables temperature, salinity and fluorescence/chlorophyll will be considered at first. Controlled experiments will be performed in order to document and validate the different uncertainty components composing the final uncertainty estimates. All the relevant assessments carried out in WP4 will be included as basic information for the combined uncertainty estimation. WP4.1 will provide an updated summary of these achievements to WP5.1, no later than the dates set down in the WP4 D4.2 delivery schedule (M14) Task 5.2: Harmonization of Delayed Mode data management procedures with SEADATANET Task responsible: IFREMER, Partners: HCMR, MUMM, OGS Duration: M1-M42 • This task will define, establish and oversee the data management infrastructure for dealing with delayed-mode data in JERICO. • The goal is to provide JERICO users with relevant informations for seamless data integration with the EU SEADATANET initiative. Task 5.2 Methodology - - - - - - - Survey the existing delayed-mode data handling practices of JERICO partners. Formulate a viable proposal for a common JERICO delayed-mode data management platform that can serve to reinforce the SEADATANET effort; the guidelines established by DG-Mare/EMODNET and WISE-Marine will be also taken in account when doing this. Create common vocabularies for JERICO delayed-mode data formats and meta-databases, building upon the work that has already been done in SEADATANET. Implement delayed-mode data and metadata formats that are compatible with those of SEADATANET (in compliance with the EU INSPIRE Directive); the Sensor Web Enablement family of the OGC consortium family of standards (SensorML, Sensor Registry, O&M) will be considered in the implementation. Ensure easy sharing and secure archival of delayed-mode data within JERICO by employing common data transport formats and storage criteria making use of the experience gained by SEADATANET. Task 5.2 Metadata implementation Discovery metadata To let all know what have been deployed during the project (where, what, how, who…) To help to retrieve data related to observatories To be compliant with European directives (Inspire) Enabling technology Use of SeaDataNet EDIOS directory to describe observatories Typing / Generation of metadata using SeaDataNet procedures and tools (Mikado) complemented with automatic tools developed in Emodnet-Physics to better use the RT Netcdf files provided in RT by the ROOSs Task 5.2 Data implementation Delayed mode data Delayed mode quality check Uniform file format (based on OceanSites NetCDF, compatibility with CF convention) + distribution in ASCII flat file format (Ocean Data View) Full metadata (Common Data Index) generated by Mikado for long term reuse Data access via web portal (all users) Task 5.3 Harmonization of Real Time data anagement procedures with MyOcean and EuroGOOS & the institution of data access services for JERICO TOP activities Task responsible: IFREMER, Partners: CNR, NIVA, IMR, HCMR,PUERTO, SMHI Duration: M1-M42 • This task will define, establish and oversee the data management infrastructure for dealing with real-time data in JERICO. • The goal is to provide for seamless integration with the EuroGOOS and EU Marine Core Service’s MyOcean component and in order to render JERICO real-time data accessible for use in generating downstream services and products. Task 5.3 Methodology • Survey the existing real-time data handling practices of JERICO partners. • Formulate a viable proposal for a common JERICO real-time data management platform that can serve to reinforce EuroGOOS and MyOcean • Create common vocabularies for JERICO real-time data formats and metadatabases, based on any existing • recommendations by MyOcean/EuroGOOS for doing this. • Implement real-time data and metadata formats based on any existing recommendations by EuroGOOS and MyOcean for doing this. • Reconcile, wherever possible, the data quality assurance procedures/protocols for real-time data amongst the JERICO partners (link to WP4, WP5.1). • Ensure easy exchange of real-time data within JERICO by employing common data transport formats and storage criteria, based on any existing recommendations by EuroGOOS/MyOcean for doing this. WP5.3 Implementation Real-Time data Real-Time Quality check based on EuroGOOS/MyOcean procedure that would need to be extended to new parameters whenever possible Uniform file format (based on OceanSites NetCDF, compatibility with CF convention) ftp file server (with an index file) per region Oceanotron (OpenDap protocol) for direct access by remote software WP5 in JERICO Data flow WP5 act as mediator in JERICO Data Flow Model A priori quality assurance (WP4) Data management procedures (WP5) • Time-dependent data uncertainty estimation • Real-time data management • Delayed mode data management Value added products and data dissemination (WP6) EuroGOOS ROOSs engagement (WP2) Demonstation of data provision (WP7) SeaDataNet MyOcean Emodnet Physical Parameters EuroGOOS MyOcean Insitu TAC implementation of EuroGOOS regional approach Arctic: IMR / Norway All parameters North West Shelves: BSH/Germany •Temp & Salinity : BSH/Germany •Current: SMHI/Sweden •Sea Level: DMI/Denmark Global Ocean : Coriolis/France •Link with international network s: Coriolis/France •Argo: Coriolis,BODC,BSH •European Vessels: South West Shelves: Puertos Del NIVA/Norway Estado/Spain •Mooring : PdE •Underway data: Coriolis/France Baltic Sea: SMHI/Sweden •Temp & Salinity : BSH/Germany •Current: SMHI/Sweden •Sea Level: DMI/Denmark •Bio : FMI-Syke/Finland Black Sea IOBAS/Bulgaria All parameters Mediterranean Sea : HCMR/Greece •Mooring : HCMR/Greece •XBT/CTD: ENEA/Italy •Drifter & Argo: OGS/Italy •Glider and Argo: MyOcean In-situ TAC role Limited number of parameters: Physical: T&S, current, sea level Biogeochemical: Chlorophyll/Fluorescence, Oxygen, Nutrients Integrate in-situ data in product accessible through global and regional portals Common format Common NRT QC Common Quality flags Common distribution tools Single access point Ensure a minimum level of quality on the data delivered In Near real time ( 24h to a week) In delayed mode Assessment of the products at basin scale EMODNET PHYSICAL PARAMETERS The European Commission, represented by DG MARE, concluded a service contracts aiming to: Create a prototype components of the European Marine Observation and Data Network (EMODnet). Provide access to archived and real-time data on physical variables in Europe's seas and oceans determine how well the data meet the users needs. THE EMODNET PP PILLARS SeaDataNet JERICO WP5 EuroGOOS MyOcean EMODnet PP EMODnet PP aims to bridge the communities behind MyOcean, EuroGOOS and SeaDataNet in order to strengthen their position inside the Commission agenda on IS and expand in future the participation to all partners EuroGOOS SEPRISE network NEW FEATURES Add new stations and models Add profiles Make use of the results from ECOOP Add data from FerryBox and RV Common quality control Link to EDIOS SEPRISE data in SDN ODV format Put data on GTS E-HYPE Cooperation between MyOcean in-situ TAC and SeaDataNet MoU signed Beginning 2010 Operational users Marine governmental agencies and research institutes (including NODCs) MyOcean in-situ TAC Direct users real-time Re-analysis users Observing systems Data integration (QC, « best copy », archival, re-analysis datasets Vocabularies Metadata (EDIOS, CDI, CAMIOON, SensorML) Technical WG Metadata (e.g. EDIOS) Other observations (e.g. scientific cruises…) Yearly regional products (climatologies, datasets) SeaDataNet International organisations Direct users (IOC/IODE, ICES, ICSU World Data System…) Users JERICO role in the European Data Management landscape Operational users JERICO Coastal MyOcean in-situ TAC Observatory Network (WP7) Re-analysis users Observing Network ‘EMECOlike’ Web – enabled Datatools Information (QC, « best copy », archival, re-analysis datasets Vocabularies Metadata (EDIOS, CDI, CAMIOON, SensorML) Technical WG ‘EMECO’ like & evidence pipeline (WP6) Assessment Web database Data integration Yearly regional products (climatologies, datasets) Portal delayed-mode SeaDataNet International organisations Direct users Data XML, CSV (IOC/IODE, ICES, ICSU World Data System…) Users WP5 Milestones/Deliverables ACTIONS To propose a formal agreement with SDN, EuroGOOS/MyO for JERICO data provision, to ensure more visibility to JERICO project To find an agreement with DG MARE/EMODNET for the availability to JERICO of the developments achieved/to be achieved in EMODNET Physical Parameter lot. Thanks!