Global Merchandise Trade SDMX
Data Structure Definition
Overview
Markie Muryawan
United Nations Statistics Division
WCO IT Conference and Exhibition © 2015
General SDMX Information
2
What is it?
• What‘s SDMX?
–
–
–
–
SDMX = Statistical Data and Metadata Exchange
In layman’s term = standard data format for statistics
Standard in both statistical and technical aspect
Sponsored by 7 organizations: BIS - ECB - EUROSTAT IMF - OECD - UN - World Bank
• SDMX consists of
– Technical standards (including Information Model),
– Statistical guidelines, and
– An IT architecture and tools.
3
Why SDMX
• Why uses SDMX?
–
–
–
–
To reduce international reporting burden
To facilitate data processing
To increase efficiency of international data exchange
ISO Standard: 17369:2013 (SDMX v.2.1)
• In 2008, UN Statistical Commission (UNSC)
recognized and supported the SDMX standards
and guidelines
– And SDMX sponsors report regularly to UNSC since
2002 (see the latest E/CN.3/2013/30 and
E/CN.3/2015/33).
4
Data Structure Definition (DSD)
• A DSD is a central component of the SDMX. It
specifies:
– i) a set of concepts which describe and identify
the data;
– ii) the attachment level of each of these concepts
as well as their status (mandatory or conditional);
and
– iii) code lists which provide acceptable values for
the concepts and their standard coding and
formatting.
5
Latest SDMX Development - 2015
• Three global data
structure definitions:
– System of National
Accounts (SNA)
– Balance of Payments
(BOP)
– Foreign Direct
Investment (FDI)
• SDMX Global Registry
https://registry.sdmx.org ,
consists of internationally
agreed:
– Code list
– Concept Scheme
– Data structure definition
6
Latest SDMX Development (cont.)
• Ongoing development of SDMX data structure
definition for statistics on
– research and development
– Education
– short-term business
– government finance and
– international trade
7
Some example of SDMX
implementations
• Eurostat collects census and national account data via SDMX
• The European System of Central Banks exchanges statistics on BOP
in SDMX
• OECD receives short-term economic statistics through SDMX web
services
• IMF Special Data Dissemination Standard Plus required submission
of national data in SDMX
• A joint initiative between the AfDB and IMF provides a countryspecific cloud-based data dissemination platform that converts data
into SDMX
• BIS and some 60 central banks operate of its statistical data
exchange based on SDMX
• UNSD established SDMX exchange of data linked to MDG, and it’s
expected the similar approach for post-2015 indicators
8
SDMX technical development
• Draft specification for
SDMX-JSON
• Design new validation and
transformation language
• Working on common IT
tools,
http://www.sdmxtools.org/
e.g., Eurostat Reference
Infrastructure
9
Now, Trade SDMX
10
SDMX-IMTS Working Group
• Established in 2013 [UN, OECD, Eurostat, ITC,
UNCTAD]
• UNSD acts as secretariat for the Working Group
• The purpose is to specify uniform structures,
concept definitions and code lists for the
transmission of IMTS data and metadata
• Comply with latest technical specification (SDMX
v.2.1) and follow the latest recommendations on
IMTS (IMTS 2010)
11
UN Comtrade: Analysis of data flow
Original
commodity
data user
UN Comtrade
3
Disseminate
original data
Data Brokers
1
Register
4
5
Preprocess
Normalize
Tariffline data
Eurostat
Original
commodity data
8
Convert
National
Data Provider
7
6
Aggregate
Disseminate
tariffline data
Supplementary
data
IMF
Tariffline data
user
Reported totals
Processed data
9
10
Publish
Disseminate
published data
2
OPEC
Download
Published
data user
Published data
WTO
12
Latest update – May 2015
• Participation of OECD, Eurostat, UN, ITC, UNCTAD
• Data flows analysed (previous slide)
• Concept scheme created for the whole domain. 31 concepts
total!!!
• Most concepts coded, still work-in-progress, will be discussed
during global consultation
• Concepts mapped to Data flows resulting in 3 groups of
dimensionality.
– DSD 1: All concepts, for data collection
– DSD 2: Most concepts, for public dissemination
– DSD 3: For aggregate data, and indices
• Concepts classified into 18 dimensions , 12 attributes and 1
measurement
13
Dimensions (total of 18)
FREQ
TIME_PERIOD
REF_AREA
TRADE_FLOW
COMMODITY_1
COMMODITY_1_CONF
COMMODITY_2
COMMODITY_2_CONF
COMMODITY_CUSTOM_BREAKDOWN
COUNTERPART_AREA_1
COUNTERPART_AREA_1_CONF
COUNTERPART_AREA_2
COUNTERPART_AREA_2_CONF
TRANSPORT_MODE_BORDER
CUSTOMS_PROC
ECONOMIC_ACTIVITY
TRANSFORMATION
OBS_TYPE
Annual/Quarterly/Monthly/etc.
Reporting Period
Reporting Country, mostly ISO country codes
Exports/Imports/etc.
HS-all editions, SITC all versions, BEC
Confidentialized commodity code
HS-all editions, SITC all versions, BEC
Confidentialized commodity code
Extension of commodity (tariff line level)
Trading partner 1, mostly ISO country codes
Confidentialized primary partner area
Trading partner 2, mostly ISO country codes
Confidentialized additional partner area
Mode of transport
Customs procedures codes, Revised Kyoto
Economic Activity/Industry based on ISIC
Transformation performed on OBS_VALUE
CIF-type value, FOB-type value, net weight,
gross weight, supplementary quantity, price
index, unit value index, volume index
14
Attributes (total of 12)
TIME_PERIOD_START_DATE
UNIT_MULT
UNIT_MEASURE
COMMENT_OBS
TRADE_SYSTEM
COMMODITY_CUSTOM_BREAKDOWN_CODE
COMMODITY_CUSTOM_BREAKDOWN_DESC
COUNTERPART_AREA_1_TYPE
COUNTERPART_AREA_2_TYPE
COUNTERPART_AREA_1_ANNOTATION
COUNTERPART_AREA_2_ANNOTATION
OBS_STATUS
Reference Period Start Date
Unit multiplier
Unit of measure
Comments to the observation value
Trade System
Custom commodity breakdown code
Custom commodity breakdown
description
Type of primary partner area
Type of additional partner area
Annotation of primary partner area
Annotation of additional partner area
Observation Status
15
Some examples
DSD with basic data
16
17
18
Some examples
DSD with dual commodity
classifications
19
20
Some examples
DSD with full commodity breakdown
– tariff line level
21
22
Some examples
DSD with confidential data when
country does NOT provide details for
internal purposes
23
24
25
Some examples
DSD with confidential data when
country does provide details for
internal purposes
26
27
28
Simplified example of
one single trade record
Implementation using SDMX-XML
29
Codes used in this example:
HS12_010121: "Pure-bred breeding horses"
HS12_0101XX: "Live horses, asses, mules and hinnies // Confidential Item"
SITC4_0015: "Horses, asses, mules & hinnies, live"
ISIC4_014: "Animal production"
_Z = "Not applicable"
_X = "Not allocated / unspecified"
30
Roadmap
Completed
Work in
Progress
•
•
•
•
•
Visualization/Analysis of data flows
Definition of list of concepts to be covered for the reporting framework
Review of existing Code lists in other domains
Draft concept scheme for IMTS
Draft code lists for IMTS
• Worldwide consultation with the countries (May-June 2015)
• Finalize a first global reporting framework for IMTS
• Gradual implementation of version 1 of global reporting framework for
IMTS
• Agreement on maintenance and governance
• Further refinement of DSD, especially for DSD3
31
Global Consultation on IMTS DSD
• From 8 May 2015 – 30 June 2015
• National agency responsible for IMTS
• Review of the following aspects:
– Concepts (dimensions and attributes): How do the
concepts defined apply to your practices? Is there any
gaps?
– Code lists: Are the code lists of all concepts
comprehensive and appropriate? Is there a need to
omit/include/modify any category?
– Implementation of SDMX: What are the challenges
and requirements?
32
Reflection
• Full implementation of SDMX would benefit
everyone (e.g., harmonized coding for all data
items)
• However, it’s required additional investment
(people, time) and probably upgrade of IT
existing system
• How to better align the work on Trade SDMX
and WCO Data Model?
33
Thank you
Please send your comments to
muryawan@un.org