7c. SDMX practical use case:
National Accounts
Daniel Suranyi and Cristina Blanaru
Eurostat
SDMX Basics course, 27-29 October 2015
Eurostat
Content
• Projects around SDMX and National Accounts
• Governance, maintenance, sharing
• Implementation in NA
• Business process re-engineering
• Structural and content validation
• Interational data sharing
• National perspective
• Working with the DSD Matric
• Working with the SDMX-MES DSDs
• Architecture & tools, validation demonstration
Eurostat
National process
Why? : The business case
 I want to…
 … structure my statistical datasets
 … exchange data with other organisations
 … align my definitions internationally using
standards
 … validate that data is correct
Eurostat
The big picture: ongoing activities
ESS.VIP
Validation
Eurostat
NAPS-S
ESS.VIP
SERV
Service Oriented
Architecture for
National
Accounts
SDMX-MES
SDMX-VTL
Validation &
Transformation
Language
TFIDC
Task Force
International Data
Sharing
Eurostat
Ownership Group
on SDMX for
Macroeconomic
Statistics
The big picture: ongoing activities
ESS.VIP
Validation
Eurostat
NAPS-S
ESS.VIP
SERV
Service Oriented
Architecture for
National
Accounts
SDMX-MES
SDMX-VTL
Validation &
Transformation
Language
TFIDC
Task Force
International Data
Sharing
Eurostat
Ownership Group
on SDMX for
Macroeconomic
Statistics
SDMX-MES Ownership Group (OG)
• SDMX for Macro-Economic Statistics
• Mandated by SDMX Sponsors in January 2014
• As a single OG for NA & GFS, BOP and FDI
• Maintenance:
NA/GFS (https://sdmx.org/?page_id=1498) and
BOP/FDI (https://sdmx.org/?page_id=1747)
• New domains can be added (e.g. Prices)
• Representation of management
from content and SDMX side
• Oversees maintenance of SDMX artefacts
• Mandated technical group(s)
National Accounts
Shared
Concepts and
Code Lists
•SNA 2008
•ESA 2010
•Other NA related
domains
Balance of
Payments
FDI
statistics
Statistics
Other domains
Cross Domain
Concepts and
Code Lists
Concepts in SDMX-MES DSDs
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
19
20
21
22
23
24
FREQ
ADJUSTMENT
REF_AREA
COUNTERPART_AREA
REF_SECTOR
COUNTERPART_SECTOR
CONSOLIDATION
ACCOUNTING_ENTRY
STO
INSTR_ASSET
MATURITY
ACTIVITY
ACTIVITY_TO
PRODUCT
PRODUCT_TO
EXPENDITURE
PENSION_FUNDTYPE
UNIT_MEASURE
CURRENCY_DENOM
VALUATION
PRICES
TRANSFORMATION
CUST_BREAKDOWN
Concepts
Cross Domain
Shared
NA specific
Obs
Obs
Obs
Ser
Ser
Ser
Ser
Ser
Dset
…
OBS_STATUS
CONF_STATUS
EMBARGO_DATE
DECIMALS
TABLE_IDENTIFIER
UNIT_MULT
COMPILING_ORG
REF_YEAR_PRICE
COMMENT_DSET
…
Coding in SDMX-MES DSDs
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
19
20
21
22
23
24
FREQ
ADJUSTMENT
REF_AREA
COUNTERPART_AREA
REF_SECTOR
COUNTERPART_SECTOR
CONSOLIDATION
ACCOUNTING_ENTRY
STO
INSTR_ASSET
MATURITY
ACTIVITY
ACTIVITY_TO
PRODUCT
PRODUCT_TO
EXPENDITURE
PENSION_FUNDTYPE
UNIT_MEASURE
CURRENCY_DENOM
VALUATION
PRICES
TRANSFORMATION
CUST_BREAKDOWN
Concept
CL Maintenance
Cross-Domain
SDMX
IMF/ESTAT
Cross-Domain
no CD code list
IMF/ESTAT
Cross-Domain
code list extended
Shared/Specific
IMF/ESTAT
Not coded
-
Obs
Obs
Obs
Ser
Ser
Ser
Ser
Ser
Dset
…
OBS_STATUS
CONF_STATUS
EMBARGO_DATE
DECIMALS
TABLE_IDENTIFIER
UNIT_MULT
COMPILING_ORG
REF_YEAR_PRICE
COMMENT_DSET
…
Data Structure Definitions (DSDs)
DSD ID
NA_...
MAIN
SEC
DSD Title
Dataflow defintions
Main aggregates
Sector accounts
and government
finance
ESA / OECD: 1, 3, 5, 20, 22
ESA / OECD: 2, 6, 7, 8, 9, 11, 26, 27, 28
ECB: MUFA, GFS
EDP: 1, 2, 3, 4
OECD: 119, 7HH, 7II, PSD
G20: Recommendation 15
SU
Supply / use and
input / output
ESA : 15, 16, 17, 18, 19
REG
PENS
Regional statistics
Pension schemes
ESA: 10, 12, 13
ESA: 29
The big picture: ongoing activities
ESS.VIP
Validation
Eurostat
NAPS-S
ESS.VIP
SERV
Service Oriented
Architecture for
National
Accounts
SDMX-MES
SDMX-VTL
Validation &
Transformation
Language
TFIDC
Task Force
International Data
Sharing
Eurostat
Ownership Group
on SDMX for
Macroeconomic
Statistics
SDMX Validation Possibilities
 “Technical”
 “Statistical Domain”
- Covered by SDMX today
- Not yet covered by SDMX
(VTL project)
- Format Check (SDMX-ML)
- Codes exist (SDMX DSD)
- Codes used correctly
(SDMX Dataflow & Constraint)
- Value check
- Time series
- Revisions
- Validation expressions
What is VTL
A standard language
Considers the validation as a kind of Transformation (calculation)
For defining validation and transformation rules
Validation (now)
Transformation (partially now, to be enriched at a later stage)
Main goals:
Define and preserve validation and transformation rules
Exchange and share rules
Apply rules in industrialized processes
Apply to several standards (e.g. SDMX, DDI, GSIM) thanks to a
generic information model
13
VTL features (4)
VTL 1.0 allows:
• Persistent and temporary results
• Operations on mono and multi measure data
• Dealing with missing data
• Dealing with Attributes and their propagation rules
https://sdmx.org/?page_id=5096
VTL 1.1 will introduce:
• Other operators, mainly for validation purposes
• Reusable rules
• Bug fixing, fine tuning
14
Summary: Current ESA business process
Data provider
National
Production
Generate
SDMX-ML
Eurostat
PUSH
Transmission
Structural
Validation
Content
Validation
Processing &
Calculation
Data Sharing
&
Dissemination
Structural
Validation
15
Eurostat
Possible future architecture (NAPS-S)
Shared / Replicated
Services
For NA
and others
Structural
Validation
Content
Validation
Statistical
Service A
Statistical
Service B
SDMX
Registry
VTL
Repository
Common
Repository
Common
Repository 16
Eurostat
SDMX Dataflow defintion
• 1 DSD covers multiple Dataflows
• NA_MAIN DSD:
•
•
•
•
Table 1: T0101_A, T0102_A, T0103_A …
Table 3: …
Table 5: …
…
• Each of those Dataflows gives a partial picture
• "cutting the DSD": SDMX cube region
• "listing each series": SDMX key set
Eurostat
SDMX Information Model - Summary
Reference
Reference
DSD
Role
Attachement
level
Dimension
Dimension
Dimension
Data Flow
Dimension
Dimension
Dimension
Dimension
Measure
Attribute
series
Concept
Scheme
Reference
Concept ID
Reference
Concept Name
FREQ
REF_AREA
ACTIVITY
ADJUSTMENT
PRICES
STO
TIME_PERIOD
OBS_VALUE
TABLE_IDENTIFIER
Frequency
Reference area
Activity classification
Constraint
Adjustment
indicator
Prices
Stocks, Transactions, Other Flows
Reference period
Observation value
Table identifier
Eurostat
Code lists
Code list ID
CL_FREQ
CL_AREA
CL_ACTIVITY
CL_ADJUSTMENT
CL_NA_PRICES
CL_NA_STO
CL_NA_TABLEID
SDMX Cube Region Constraint
• The DSD has an Activity code list with >170 items
• In table 101 we only report A*10
• SDMX cube region constraint (linked to dataflow):
• List of allowed codes by concept
?
Eurostat
SDMX Key Set Constraint
• Which series are allowed / forbidden?
Eurostat
The big picture: ongoing activities
ESS.VIP
Validation
Eurostat
NAPS-S
ESS.VIP
SERV
Service Oriented
Architecture for
National
Accounts
SDMX-MES
SDMX-VTL
Validation &
Transformation
Language
TFIDC
Task Force
International Data
Sharing
Eurostat
Ownership Group
on SDMX for
Macroeconomic
Statistics
The necessary pre-conditions
• SDMX maintenance agreements
Ownership and maintaining agency to be defined
(normally between the international organisations)
• Internationally agreed data validation
Data validation rules and SDMX data validation standards
(beyond file structure validation)
• Streamlined data exchange processes
Frequency and timeliness for data dissemination, data
quality management etc.
A possible data sharing model
IMF, UN, WB, BIS,
other IOs
U
S
E
R
S
SDMX Web Services
OECD
Eurostat ECB
EU
OECD
other than EU
World other
than OECD
Technical Integration
Source
Source
National
databases
(Mapping)
SDMX Data
Structure
Defintions &
Data Flows
Source
National
data sources
National
statistical organisations
International
organisations
Metadata driven process based on SDMX standards and tools
Full automation of the data exchange possible
SDMX Basic Course
National perspective:
National Accounts
29th October 2015
Daniel SURANYI
Cristina BLANARU
Eurostat
National process
Why? : The business case
 I want to…
 … structure my statistical datasets
 … exchange data with other organisations
 … align my definitions internationally using
standards
 … validate that data is correct
Eurostat
Why? : The business case
 How we succeeded to…
 … structure my statistical datasets
using SDMX Data Structure Definitions
 … exchange data with other organisations
using Push, Pull or Hub infrastructure
 … align my definitions internationally using
standards
with SDMX cross-domain concepts, code lists
 … validate that data is correct
using SDMX schemas and VTL
Eurostat
Overall process
Exchange
Needs
Concept
Scheme
Coding
Publishing
Artefacts
Optimising
DSD
Matrix
Eurostat
SDMX-NA DSD Matrix explainationOverview Sheet(1)
Eurostat
SDMX-NA DSD Matrix explainationOverview Sheet(2)
 summarises all concepts and code lists
 Define the concepts (Cross domain
concepts, Domain specific concepts)
 Adding code lists to the concepts
Eurostat
SDMX-NA DSD Matrix explaination
Matrix Sheet(3)
Eurostat
SDMX-NA DSD Matrix explaination
Matrix Sheet(4)
 define the relationship between the datasets
included in the scope of the transmission and the
concepts
 each concept has a hyperlink pointing to the
corresponding code list sheet.
Eurostat
SDMX-NA DSD Matrix explaination
Matrix Sheet(5)
DSD Matrix: data flows X concepts
FRE
Q
REF_ARE
A
ST
O
ACTIVIT
Y
Main Aggregates
%
#
%
#
GDP and main
indicators
%
#
%
Sector Accounts
%
#
%
#
General Government
%
#
%
N
Supply / Use
A
#
%
Data Flow
CON
S
PRODUC
T
%
SDMX-NA DSD Matrix explaination
Matrix Sheet(6)
•
•
•
•
#
%
(code)
(blank)
concept fully used
concept partially used
one code of concept used
concept not used
SDMX-NA DSD Matrix explaination
Code list Sheets(7)
Showing the contents of each of the code lists
used:
 CL_FREQ sheet
 CL_AREA sheet
 CL_ACCOUNT_ENTRY sheet
 CL_MATURITY sheet………….
Data Structure Definitions (DSDs)
• 5 DSDs covers multiple Dataflows
•
•
•
•
•
NA_MAIN DSD
NA_SEC
NA_REG
NA_SU
NA_PENS
• All structures (DSDs) stored on the SDMX Global
Registry+ on SDMX website: www.sdmx.org
Eurostat
Data Structure Definitions on the
SDMX Global Registry
Data Structure Definitions on the
on SDMX website
DSD ID and Name
Version
SDMX Download
NA_MAIN
Main Aggregates
1.5
2.0 / 2.1
NA_SEC
Sector Accounts & Government Finance
1.5
Excel templates*
last update
V1.4a
12/12/2014
V1.4a
12/12/2014
V1.4a
12/12/2014
V1.4c **
01/06/2015
2.0 / 2.1
NA tables
ESA: 1, 3, 5, 20, 22
ECB: MUFA, GFS, FAM
ESA-FINA: 6, 7
ESA-SA: 8, 26
V1.4a 12/12/2014
ESA-GFS: 2, 9, 11, (25), 27, 28
V1.4a 12/12/2014
OECD: 119, 7HH, 7II, PSD
ESA: 10, 12, 13
NA_REG
Regional Accounts
2.0 ***
2.0 / 2.1
V1.4a *
12/12/2014
NA_SU
Supply/Use & Input/Output Statistics
1.5
2.0 / 2.1
V1.4b **
19/01/2015
ESA: 15, 16, 17, 18, 19
NA_PENS
Pensions
1.5
2.0 / 2.1
V1.4b **
09/05/2015
ESA: 29
NA_MAIN93
Main Aggregates, SNA93
1.5
2.0 / 2.1
N/A
N/A
NA_SEC93
Sector Accounts, SNA93
1.5
2.0 / 2.1
N/A
N/A
IMF:GFS
1.0
IMF Government Finance Statistics Manual 2014
2.0 / 2.1
N/A
N/A
Eurostat
Data provider view:
One goal – different possibilities
Webforms
Online data
collection tools
SDMX-ML file
Excel
sheets
SDMX Converter
SDMX-ML file
Database
export
SDMX Converter
SDMX-ML file
Database
SDMX Reference
Infrastructure
SDMX-ML file
SDMX Web Service
Same
DSD

Same
Format
Data provider view:One goal –
different possibilities
Webforms
Excel sheets
Online data
collection tools
SDMX-ML file
SDMX Converter
SDMX-ML file
Database
export
SDMX Converter
SDMX-ML file
Database
SDMX Reference
Infrastructure
PHASE OUT FOR ESA IN 09/2016
SDMX-ML file
SDMX Web Service
Same
DSD

Same
Format
SDMX Reference Infrastructure
https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Reference_Infrastr
ucture_SDMX-RI
Data Provider
Data Collector
SDMX Registry
Mapping
Assistant
DSD
Web
Svc
Non-SDMX
local database
Test
Client
SDMX-RI
Web
Client
SDMX data set
NSIs: Local Database (1)
 Create a DB for National Accounts (if
you don't have one in your NSI)
 One table per DSD (just 5 tables for
the whole transmission program)
 Avoid the proliferation of tables
 A simply DB structure
 Load data into table
 Use Mapping Assistant tool of SDMXRI package
NSIs: Local Database (2)
 Create a link from NSI source DB to
the DSD
 Use Test Client of the SDMX –RI
package (to export data from the
table to an SDMX file)
 Validate the table
 Send the table to EUROSTAT
Create a link from NSI source DB to the DSD
Mapping Assistant: Database  SDMX (3)
NSI Web Client
Web Client pivot table
Fusion Cloud SDMX Validation Service
https://data.sdmxcloud.org/
 Validation against DSD
 Correct DSD used
 Correct concepts and identifiers
 Dimensions and mandatory attributes present
 Correct codes used
 Validation against Data Flow
 Correct codes for specific ESA2010 table
 Simple data viewer
Eurostat
Demo fusion cloud validation (1)
Eurostat
Demo fusion cloud validation (2)
Eurostat
Demo fusion cloud validation (3)
Eurostat
Summary
• Projects around SDMX and National Accounts
• Governance, maintenance, sharing
• Implementation in NA
• Business process re-engineering
• Structural and content validation
• Interational data sharing
• National perspective
• Working with the DSD Matric
• Working with the SDMX-MES DSDs
• Architecture & tools, validation demonstration
Eurostat