Webinar on SDMX
implementation in
Research & Development
Eurostat
SDMX-RD Webinar – Agenda
•
•
•
•
SDMX Overview
SDMX-RD Project overview
SDMX – how do I get there?
Demonstrations
•
•
•
•
SDMX Converter
SDMX Reference Infrastructure (SDMX-RI)
SDMX-RI Mapping Assistant
SDMX-RI clients (PUSH and PULL Mode)
• Questions & Answers
Eurostat
The SDMX
Components
SDMX
Information
Model
Content
Oriented
Guidelines
IT Infrastructure
for exchange and
sharing
3
Eurostat
The SDMX
Components
Describe statistics in a standard way
Objects and their relationships


Data Structure Definition (DSD), Concepts, Code List

Central management and standard access

SDMX Registry, SDMX Web Services





Cross Domain Concepts
Cross Domain Code Lists
Statistical Domains
Metadata Common Vocabulary

Push

Provider generates and sends file to receiver
Pull



Provider opens web service to data
Receiver downloads regularly
Hub


Special case of pull: receiver downloads on end user request
Eurostat
4
SDMX and Validation
 After Transmission
( “Second Level”)
 Before/During Transmission
(“First Level”)
- Covered by SDMX standard
- Format Check (SDMX-ML)
- Code Check (SDMX DSD)
Eurostat
- Not yet covered
(planned end-2014)
- Detailed value check
-…
Data flows in R&D
Eurostat
SDMX-RD project overview
• Participants:
• Eurostat, OECD, & UNESCO
• Scope and objectives
• Create joint DSDs, harmonised codelists, and
simple validation rules for the R&D data
• Operationalise the use of these SDMX artefacts
into our processes
Eurostat
SDMX-RD "Pilot Package"
• DSD Matrix
• R&D Coding sheet
• SDMX Data Structure Definitions
• Related tools and support
Eurostat
Technical Integration
Source
Source
National
databases
(Mapping)
Source
National data
sources
National
Organisations
SDMX Data
Structure
Defintions
& Data
Flows
International
Metadata driven process based on SDMX standards and tools
Full automation of the data exchange possible
Eurostat
Data provider view:
One goal – different possibilities
Webforms
EDAMIS
Single Entry
Point
SDMX-ML file
Excel
sheets
SDMX Converter
SDMX-ML file
Database
export
SDMX Converter
SDMX-ML file
Database
SDMX Reference
Infrastructure
Eurostat
SDMX-ML file
SDMX Web Service
Same
DSD

Same
Format
SDMX-RD Webinar – Agenda
•
•
•
•
SDMX Overview
SDMX-RD Project overview
SDMX – how do I get there?
Demonstrations
•
•
•
•
SDMX Converter
SDMX Reference Infrastructure (SDMX-RI)
SDMX-RI Mapping Assistant
SDMX-RI clients (PUSH and PULL Mode)
• Questions & Answers
Eurostat
SDMX-RD Webinar – Agenda
•
•
•
•
SDMX Overview
SDMX-RD Project overview
SDMX – how do I get there?
Demonstrations
•
•
•
•
SDMX Converter
SDMX Reference Infrastructure (SDMX-RI)
SDMX-RI Mapping Assistant
SDMX-RI clients (PUSH and PULL Mode)
• Questions & Answers
Eurostat
SDMX Reference Infrastructure
https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Reference_Infrastr
ucture_SDMX-RI
Data Provider
Data Collector
SDMX Registry
Mapping
Assistant
DSD
Web
Svc
Non-SDMX
local database
Test
Client
SDMX-RI
Eurostat
Web
Client
SDMX data set
SDMX Reference Infrastructure
https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Reference_Infrastr
ucture_SDMX-RI
Data Provider
Data Collector
SDMX Registry
Mapping
Assistant
DSD
Web
Svc
Non-SDMX
local database
Test
Client
SDMX-RI
Eurostat
Web
Client
SDMX data set
SDMX-RD Webinar – Agenda
•
•
•
•
SDMX Overview
SDMX-RD Project overview
SDMX – how do I get there?
Demonstrations
•
•
•
•
SDMX Converter
SDMX Reference Infrastructure (SDMX-RI)
SDMX-RI Mapping Assistant
SDMX-RI clients (PUSH and PULL Mode)
• Questions & Answers
Eurostat
SDMX Reference Infrastructure
https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Reference_Infrastr
ucture_SDMX-RI
Data Provider
Data Collector
SDMX Registry
Mapping
Assistant
DSD
Web
Svc
Non-SDMX
local database
Test
Client
SDMX-RI
Eurostat
Web
Client
SDMX data set
SDMX-RD Webinar – Agenda
•
•
•
•
SDMX Overview
SDMX-RD Project overview
SDMX – how do I get there?
Demonstrations
•
•
•
•
SDMX Converter
SDMX Reference Infrastructure (SDMX-RI)
SDMX-RI Mapping Assistant
SDMX-RI clients (PUSH and PULL Mode)
• Questions & Answers
Eurostat
SDMX Tools – Summary
• Eurostat tools  Eurostat SDMX Info Space
https://webgate.ec.europa.eu/fpfis/mwikis/sdmx
• SDMX Converter
• SDMX Reference Infrastructure & Mapping Assistant
• SDMX Registry & Data Structure Wizard
• Other tools availble in the community
http://www.sdmxtools.org
•
•
•
•
•
Fusion Weaver (validation, transformation, creation)
Fusion Transformer (command line conversion)
Fusion Registry
ECB Checker/Loader (conversion and validation)
FAO OpenSDMX project
Eurostat
Aim of the pilot / open questions
• Feedback on data structures and coding
• For pilot: DSDs in Eurostat test registry
• For production: Global Registry
• Note: maintenance agencies not yet final
• Setup / configuration of national systems
• Support is given by international organisations
Eurostat
Questions?
• SDMX Support Team
• ESTAT-Support-SDMX@ec.europa.eu
• SDMX Website
• http://www.sdmx.org
• Eurostat SDMX Info Space
• https://webgate.ec.europa.eu/fpfis/mwikis/sdmx
20
Eurostat