Statistical Data and Metadata Exchange Initiative
BACKGROUND
BRIEFING
AND
NEXT
STEPS
METIS
Geneva, February 2004
Overview
 Challenges and Opportunities
 What is SDMX?
 Progress
 Next steps
1
Challenges
 Exchange of statistical data is still complex, resource
intensive, and expensive
 Various international organisations have individual
approaches for their constituencies
 Uncertainties about how to proceed with new technologies
(XML, web services …)
2
180 + Countries
International Organisations accounts
Regional Organisations statistics
National Statistical
Organisations
accounts
statistics
Banks, Corporates
Individual Households
transactions
accounts
www.z.org
www.hub.org
www.y.org
www.x.org
Internet, Search, Navigation
Statistical information building blocks
3
Opportunities
 Constituencies of international organisations have asked
for common solutions
 International organisations are in a position to expand
their cooperation
 IT technology seems to offer some ways forward that we
should not ignore
4
What is SDMX? (1)
 Initiative of BIS, ECB, EUROSTAT, IMF, OECD, World
Bank and UN
 “ ... to focus on business practices in the field of statistical
information that would allow more efficient processes for
exchange and sharing of data and metadata within the
current scope of our collective activities”
 Global workshop at IMF, reports to UN Statistical
Commission, liaison with international standards
community and private sector initiatives
 www.sdmx.org
5
6
7
8
What is SDMX? (2)
 Projects (shared funding and resources) launched in June 2002
 batch data exchange
 metadata common vocabulary
 metadata repositories
 a practical case study on emerging e-standards
9
Case Study
 Overall goal: reality check on emerging standards by
building prototype.
 Closely coordinated work with other projects –
exploring ways to validate practical impact of project
deliverables.
 Builds on experimentation taking place in different
SDMX institutions.
 Support from other institutions and central banks.
10
Case Study: general approach
 Selected the BIS-IMF-OECD-World Bank hub on external
debt for the demo example
 Reviewed available and emerging standards
 Established experimental XML standards based on
existing statistical exchange standards (i.e. GESMES/TS
metadata “key family” concepts)
 Built a metadata registry and populated it with links to
where the data can be found
 Built web services (registration, search, transformation,
comparison)
11
Case Study: key findings
 Technology could be made to work better for statistical data
exchange
 Common information model underlies all statistical data
 Common standards cut across different patterns of data exchange:
bilateral, gateway, data sharing
 Potential of (new) web technology is significant
 XML is widely supported by software vendors, making it easier to use
off-the-shelf tools to develop applications
 registry-based architecture for metadata can facilitate navigation
(i.e. data google)
 use of web services enables automated data exchange
 Possible to build on existing infrastructures, technologies and
standards
12
Example of information model: metadata “key family”
Country
Stock/Flow
Unit Multiplier
Unit
Time/Frequency
Topic
13
Simple tool for creating a key family (Demo)
14
Spreadsheet input to XML output (Demo)
15
Patterns of Data Exchange
 Bilateral
 individual provider-to-consumer links
 could be batch or web dissemination
 variety of format standards
 Gateway
 many bilateral-to-central authority then
on to others (bilaterally)
 still many bilateral formats
 typical in data bank exercises
 Data sharing
 statistical hubs through common
standards and formats
 data/metadata useable without prior
agreement
 dissemination could replace reporting
16
GESMES/TS Reporting & Registry-Based Dissemination (1)
GESMES/TS
Data
Database
EDI
Gateway
EDI
Gateway
Database
(1) Today, the sender renders the data out of the database as
GESMES/TS EDIFACT, and sends it to a counter-party according
to prior agreement. When received by the counter-party, it goes through
a gateway and is passed into their internal systems.
17
GESMES/TS Reporting & Registry-Based Dissemination (2)
Transmission of GESMES/TS
& SDMX-ML Data
EDI & SDMX-ML
Gateway
EDI & SDMX-ML
Gateway
Notification
Database
Database
Registry
SDMX-ML
Gateway Registration
SDMX-ML
Gateway
Retrieval of SDMX-ML Data
Website
18
Possible registry-based infrastructure
Registry
Registry Services - Registration
Key family
Central Bank
Website
Classification
Scheme
BIS
Website
IMF
Website
Publisher
Search, Retrieve, Transform
Dataset
OECD
Website
World Bank
Website
Dataset
Key family
metadata
Statistics
User
Consumer
19
Web Services for statistical information exchange and sharing
Service/Data Provider
Service/Data Locator
Service/Data Consumer
Web Service
Registry
Web Service
20
21
22
23
24
25
Selecting Data Category to Search (Demo)
26
Results of Registry Search – Comparison View (Demo)
27
Anticipated benefits
 Reduced reporting burden – via common formats adopted
by international organisations for data and metadata
exchange
 More user-friendly access when publishing national data
and metadata on the web - via global standards for data
formats, catalogs/registries and associated services
 Enhanced opportunity for navigation, discovery and
comparative data analysis – via global guidelines for
metadata vocabularies and repositories linked to statistical
information in common formats
 Can replicate these models and tools for statistical
information systems at national levels
28
Envisaged next steps
 Transition from projects to SDMX deliverables, on a learnby-doing basis
 metadata standards
 technical standards
 Pilot for Joint External Debt Hub (involving cost sharing by
participating international organisations)
 to implement initial version 1.0 standards
 to implement re-usable and freely available tools and
registries to provide the basis for capacity building,
fostering the adoption of SDMX standards
 to implement a pilot for creditor and debtor statistics
based on these standards and tools
29
SDMX
International
Sponsoring
Institutions
Aligned Subject-Matter
Activities
(institutions and other fora
adopting SDMX standards)
SDMX
Technical Standards
www.sdmx.org
Standards
Tools
Registries
Repositories
Conformance Tests
Monitor Projects / Activities
Training / Capacity Building
Copyright and Intellectual
Property
Contacts/Liaison
use
examples
examples
OECD.Stat
deliverables
Joint External Debt Hub
NAWWE
Comtrade
BIS Data Bank
Core Statistical Concepts
Metadata Common Vocabulary
deliverables
Core Subject-Matter Domains
Monetary and Financial
National Accounts
Balance of Payments
External Debt
SDMX
Metadata Standards
Information Model
Formats (EDI, XML)
Architecture
(institutions and other fora
adopting SDMX standards)
examples
Aligned
Implementation
Activities
Aligned Subject-Matter Activity
SDMX Standards-setting (Version 1.0) Joint External Debt Hub Project: Overview
UN Statistical Commission:
Inter-Agency Task Force on Finance Statistics
(institutions and other fora adopting SDMX standards)
SDMX
Metadata Standards
SDMX
Technical Standards
deliverables
www.sdmx.org
Standards
Tools
Registries
Repositories
Conformance Tests
Monitor Projects / Activities
Training / Capacity Building
Copyright and Intellectual
Property
Contacts/Liaison
Information Model
Formats (EDI,XML)
Architecture
Core Statistical Concepts
Metadata Common Vocabulary
Core Subject-Matter Domains
External Debt
deliverables
Aligned Implementation Activity
BIS-IMF-OECD-World Bank
(institutions and other fora adopting SDMX
standards)
More about SDMX to follow in Working Paper 7 :
Inter-agency cooperation for the dissemiantion and
exchange of standard metadata
www.sdmx.org
32