UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
UN Comtrade Version
Ref : DTM_DST001CEn.doc
1 Feb 2006
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
Page ii
UN Comtrade Version
UPDATE FORM
Name and Version
number
DST001C
DISTRIBUTION
Andreas Lindner
Trevor Fletcher
Grégory Legoff
Lynda Hawe
Ronald Jansen
Markie Muryawan
Henri Kayali
VERSION
A
B
C
INFO³OECD
Sent
By
1 Feb 06
OECD/STD
Signed Approved
By
Signed
OECD
OECD
OECD
OECD
UN
UN
Info³
DATE
OPERATION SPECIFICS
19 Oct Creation of the document
05
12 Dec Add-ons: Comtrade Distribution details (operating mode)
05
1 Feb 06 Cover page and Introduction edited
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
Page iii
UN Comtrade Version
TABLE OF CONTENTS
1.
1.1.
1.2.
1.3.
INTRODUCTION .................................................................................................................................................... 1
W HAT IS MEANT BY “JOINT SYSTEM”? ............................................................................................................... 1
ABOUT THIS DOCUMENT ...................................................................................................................................... 2
ASSOCIATED DOCUMENTS ................................................................................................................................... 2
2.
OVERVIEW ............................................................................................................................................................. 3
3.
TECHNICAL SPECIFICATIONS.......................................................................................................................... 4
3.1.
REQUIREMENTS ................................................................................................................................................... 4
3.2.
DATA STORAGE FORMATS ................................................................................................................................... 4
3.2.1.
Data Capture ............................................................................................................................................. 4
3.2.2.
Data Processing and Dissemination ...................................................................................................... 4
3.3.
DATA FLOW FORMATS ......................................................................................................................................... 5
3.4.
DATA TRANSFER PROTOCOLS ............................................................................................................................. 6
3.5.
SECURITY ............................................................................................................................................................. 7
3.6.
SYNCHRONIZATION .............................................................................................................................................. 7
4.
OPERATING MODE .............................................................................................................................................. 8
4.1.
DOWNLOAD COMTRADETOOLS SETUP FILES ..................................................................................................... 8
4.2.
INSTALL COMTRADETOOLS.................................................................................................................................. 9
4.3.
TEST W EB SERVICES ........................................................................................................................................ 11
4.4.
DOWNLOAD DATA TO A CSV FILE ..................................................................................................................... 12
4.5.
USING THE CONFIGURATION FILE ..................................................................................................................... 14
4.5.1.
Change Column Mapping...................................................................................................................... 15
4.5.2.
Use ProjectID .......................................................................................................................................... 15
4.6.
IMPORT DATA INTO A SQL SERVER DATABASE ................................................................................................ 16
4.6.1.
Check SQL Server Updates ................................................................................................................. 16
4.6.2.
Create the Destination Table ................................................................................................................ 16
4.6.3.
Define the Connection String ................................................................................................................ 16
4.6.4.
Define the Column Mapping Between CSV File and Database Table ............................................ 17
4.6.5.
Import a Dataset into the Table – Debug Mode ................................................................................. 17
4.6.6.
Import Datasets ...................................................................................................................................... 17
4.6.7.
Use the New ‘Resume’ Parameter ....................................................................................................... 17
FIGURES ........................................................................................................................................................................ 18
INFO³OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
1.
Page 1
UN Comtrade Version
Introduction
1.1.
What Is Meant by “Joint System”?
Since 2000, UNSD 1 and OECD 2 have joined forces to reduce the response burden for national data
providers in the field of merchandise trade statistics and to agree on one unique and transparent set of data
instead of having parallel (and often not harmonized) data collections. This considerable amount of very
detailed investigation has led to the signing of an UNSD/OECD Memorandum of Understanding in January
2001 and further extensions, where UNSD and OECD jointly developed
(i) an agreement on data collection and data sharing,
(ii) common trade data processing standards, and
(iii) computer applications for use in processing of trade data.
This set of agreements, standards and applications is referenced by both organizations as:
UN─OECD Joint Trade Data Collection and Processing System (Joint System)
Both organizations agreed to use the above name in any documents describing their cooperative activities.
It is also to be used when referring to the respective databases.
The Joint System is the foundation of the trade databases maintained by both organizations. Both
organizations continue to keep separate (institutionally and legally) trade databases to ensure rights and
ability of each organization to pursue their own policies with regard to
(i) storing various data in addition to those agreed in Joint System [e,g., historical data, memorandum
items, derived data etc],
(ii) developing additional analytical and presentational functionalities and controlling user access to
them, and
(iii) dissemination practices including pricing of user access to the database services, entering into
contracts with users, etc.
Each organization maintains its own brand name for use in data dissemination activities. The names of
those databases are:

For UNSD: UN Comtrade3

For OECD: OECD ITCS4 Database
To ensure user awareness of the joint efforts of both organizations in the area of international merchandise
trade statistics, both organizations agreed to make data responsibility very clear.
Consequently, the UN Comtrade disclaimer contains the following statement:
“Beginning 2005, trade data of OECD member countries are compiled by OECD in accordance with
UN-OECD Common Trade Data Processing Standards and are available in UN Comtrade in
accordance with UN/OECD Data Sharing Agreement. Any deviation from this practice is reflected in
the country/period metadata.”
Similarly, the OECD trade database disclaimer contains the following statement:
“The trade data for the selected non-OECD member countries are compiled by UNSD in
accordance with UN/OECD Common Trade Data Processing Standards and are available in OECD
ITCS Database in accordance with UN-OECD Data Sharing Agreement. Any deviation from this
practice is reflected in the country/period
1
United Nations Statistics Division
Organization for Economic Co-operation and Development
3 Commodity Trade Statistics Database
4 International Trade by Commodity Statistics
2
INFO³/OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
1.2.
Page 2
UN Comtrade Version
About this Document
This document makes up the technical specifications of the Joint System data transfer mechanism (DTM)
between UNSD and OECD databases. DTM is introduced in Section 2. below.
Data transfers between UNSD and OECD are the responsibility of specialized OECD statistics database
Administrators.
Word links are available throughout this document –like for instance, at the beginning of the section.
Pictures include detailed comments in callouts or in their captions. These comments contribute to trimming
down the text and making it clearer. This layout is generally appreciated by non-expert readers.
1.3.
Associated Documents
The UNSD – OECD Joint System package also includes the following documents:




Data Capture / Data Processing Manual
Web Browser Client User Manual
Database Administrator Client Manual
Database Dissemination Model Guide
INFO³/OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
2.
Page 3
UN Comtrade Version
Overview
As mentioned in the previous section, UNSD and OECD will use a unique logical storage to make their
statistical data available through the Web.
In fact, the Joint System will still use two physical SQL Server databases, one for each organization, to
store the collected and processed commodity trade statistical data.
These two databases will be continuously updated and synchronized through a data exchange process
explained later in this document. Thus, the databases will stay identical regarding their logical contents. A
specific “flavor” will also be added to each database contents to reflect OECD and UNSD standards,
particular additions and habits.
This process is illustrated below:
OECD
Disseminate
Working
Working
Database
Database
Data Exchange &
Synchronisation
Dissemination
Dissemination
Database
Database
UNSD
Disseminate
Working
Working
Database
Database
Dissemination
Dissemination
Database
Database
Process
Process
Common
Common
Processing
Processing
Module
Module
Common
Common
Processing
Processing
Module
Module
Data Access
Data capture
Data capture
5
ITS Meeting Sept 2005
Figure 1: statistical data flow for UNSD and OECD in Joint System
Data transfers and conversions are physically carried out by an UNSD software package, named
ComtradeTools, which makes use of Web Services. ComtradeTools functions, Configuration file and
commands are discussed in the Operating Mode below.
The purpose of this document is to describe, step by step, the technical specifications of data transfer as
represented above:






Transfer protocols
Data flow formats
Data storage formats
Security
OECD – UNSD synchronization
Operating mode
Some parts of this document are closely related to the Database Administrator Client Manual.
INFO³/OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
3.
Page 4
UN Comtrade Version
Technical Specifications
3.1.
Requirements
The OECD Client who wishes to import updated data from the UNSD database should be equipped with the
following software components:




Microsoft Windows 2000 or a higher version;
Microsoft .NET Framework version 1.1;
Microsoft SQL Server 2000;
WinZip program (note: some versions of Windows have a built-in unzip capability)
In addition, the OECD Client should have a basic knowledge of XML files and SQL Server Query Analyzer.
3.2.
Data Storage Formats
3.2.1.
Data Capture
Contributing countries generally send their annual statistical data as plain text, Excel or .DBF (FoxPro) files.
In fact, any format is acceptable.
MS Access is used as an intermediate storage for all these files before loading them up to the SQL Server
database. Importation into Access and SQL Server database are carried out through the Get External Data
(Access) and Import Data (SQL Server) functions.
Note: data in Access format is kept for archival purposes.
3.2.2.
Data Processing and Dissemination
As mentioned before, SQL Server is used to store the statistical data to be processed (Working Databases)
and distributed (Dissemination Databases)
Data processing (importation, checking, conversion, aggregation…) is described in the Data Capture /
Processing Manual.
INFO³/OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
3.3.
Page 5
UN Comtrade Version
Data Flow Formats
The OECD and UNSD databases are kept synchronized through bi-directional data transfers. The
synchronization process is explained in this document and in the Database Administrator Client Manual.
During these transfers, the XML5 format is used, with SDMX6 cross-sectional schema. The incoming data
is finally converted into CSV7 format and loaded into the destination database.
The following insertion is an example of SDMX cross-sectional schema:
-
-
-
-
<?xml version="1.0" encoding="utf-8" ?>
<CrossSectionalData xmlns="http://www.SDMX.org/resources/SDMXML/schemas/v1_0/message"
xmlns:uncs="http://unstats.un.org/structure/key_families/cross/UN_COMTRADE_BE"
xmlns:cross="http://www.SDMX.org/resources/SDMXML/schemas/v1_0/cross"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:schemaLocation="http://www.SDMX.org/resources/SDMXML/schemas/v1_0/message
SDMXMessage.xsd
http://unstats.un.org/structure/key_families/cross/UN_COMTRADE_BE
UN_COMTRADE_BE_CrossSectional.xsd
http://www.SDMX.org/resources/SDMXML/schemas/v1_0/cross
SDMXCrossSectionalData.xsd">
<Header>
<ID>UN626573874</ID>
<Test>true</Test>
<Truncated>false</Truncated>
<Prepared>2005-11-10T04:34:38</Prepared>
<Sender id="UN">
<Name xml:lang="en">United Nations</Name>
</Sender>
<KeyFamilyRef>UN_COMTRADE_BE</KeyFamilyRef>
<DataSetID>01</DataSetID>
<DataSetAction>Update</DataSetAction>
<Extracted>2005-11-10T04:34:38</Extracted>
</Header>
<uncs:DataSet>
<uncs:Group RPT="266" time="2004" CL="BE" UNIT_MULT="1" DECIMALS="1" CURRENCY="USD"
FREQ="A" TIME_FORMAT="P1Y" REPORTED_CLASSIFICATION="H2" FLOWS_IN_DATASET="MX">
<uncs:Section
TF="1"
REPORTED_CURRENCY="XOF"
CONVERSION_FACTOR="0.001891"
VALUATION="Special" TRADE_SYSTEM="CIF" PARTNER="Origin">
<uncs:Obs CC-BE="122" PRT="12" netweight="0" qty="0" QU="1" value="12990" />
</uncs:Section>
</uncs:Group>
</uncs:DataSet>
</CrossSectionalData>
Figure 2: example of SDMX cross-sectional schema
Link to
http://www.sdmx.org
for more detail about SDMX.
The same data format is used for transfers between working and dissemination databases (see fig. 1).
The SDMX data is compressed by 70% to 80% before the data transfer.
5
Extensible Markup Language
Statistical Data and Metadata Exchange
7 Comma Separated Values; e.g., the contents of a plain text file. Such a file can be used by Notepad or Excel
6
INFO³/OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
3.4.
Page 6
UN Comtrade Version
Data Transfer Protocols
The common HTTP8 is used all over the Internet via Web Services. Web Services is a remote library (or
Class, from a language point of view), published on the Web via XML standards, with properties and
methods, some of which devoted to Web data transfers. In our case, due to data size, REST Web service
is selected because of its ability to stream and use on-the-fly compression.
The picture below illustrates the Joint System data transfer mechanism between UNSD and OECD:
Figure 3: transfer mechanism between UNSD and OECD
Transfers are “Pulled” by OECD: data is sent on OECD Users’ requests to ComtradeTools –see Operating
Mode below.
8
Hypertext Transfer Protocol
INFO³/OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
3.5.
Page 7
UN Comtrade Version
Security
It is based on IP Addresses of Users who wish to connect to the Web Server and on passwords to the
dissemination database. The IP address seen by ComtradeTools is the Web access provider’s, not the
local network address of the connecting station.
An example of denied access is given in section 4. Web Services is also used via the Access Control List
function. In addition, the User ID / password combination used when logging on to Windows is used to
access the CoprA application (see Data Capture / Data Processing Manual).
The secured (encrypted) HTTP, HTTPS, could possibly be used.
3.6.
Synchronization
The OECD and UNSD databases are synchronized through a time stamp mechanism inside
ComtradeTools: each time a data set is imported from reporting countries to the UNSD database, its time
stamp is updated to the current date, and a comparison with the previous time stamp tells whether this data
item must be downloaded to the OECD database to update it.
The format of a time stamp is YYYY-MM-DDTHH:MM:SS (labels in italics are placeholders)
From a general point of view, the updating data can be filtered by country, year or commodity, in addition to
being selected by the time stamp. Examples of such selection parameters are given in the next section.
Selection criteria can be defined directly in the command line or by Keys in the ComtradeTools
Configuration file. The example below is the definition of such a key:
<add key="2000Data" value="LastSynch=2004-01-06T18:08:46;Synchoption=TIMESTAMP; parameters=/r:360” />
The key is identified by its name, ‘2000Data’. It selects a reporting country, Indonesia (code 360) and its
time stamp is transmitted by the keyword LastSynch (date of the last synchronization process).
More precisions about the Configuration file and its usage are given in the next section.
CoprA
Reporting countries
The transfer/synchronization process is illustrated below:
Timestamp
(Comtrade
Tools)
UNSD
OECD
Figure 4: the import - synchronization process
The synchronization process isn’t automated. Its periodicity is left to the Administrator’s assessment – e.g.,
one week. In the near future, the process will be automated and will take place, for instance, every night.
CoprA, the UNSD data gathering and processing tool, is described in the Data Capture / Data Processing
Manual
INFO³/OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
4.
Page 8
UN Comtrade Version
Operating Mode
This section is an illustration of the operating procedure used to import UN data into the OECD database.
The conversion and synchronization phases are also reproduced. An example of column mapping and SQL
Server data creation is provided. In fact, all the useful operations are discussed. Depending on the status of
the operating workstation and the data to be transferred, some of the operations described here won’t be
necessary to carry out an import.
As mentioned above, a software package, ComtradeTools, is used to initiate the various transfer /
conversion functions. ComtradeTools is a command line program: it is launched from the Run window, like,
for instance, the Registry Editor.
The first thing to do is to download the ComtradeTools package, if not already present on the User’s
workstation.
4.1.
Download ComtradeTools Setup Files
Installation package (compressed file)
Get ComtradeTools installation files by linking to
http://unstats.un.org/unsd/comtradedist/comtradeToolsSetup.zip
The usual Open/Save interface is displayed. The easiest way to proceed is to choose Open; WinZip will be
activated and will display the three expanded installation files (.exe, .ini, .msi):
Figure 5: the three ComtradeTools setup files – unpacked by WinZip
Transfer the installation files to a proper local directory
(suggested: <System Drive>:\Program Files\UNSD\ComtradeTools; System Drive is usually C).
INFO³/OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
4.2.
Page 9
UN Comtrade Version
Install ComtradeTools
Double-click9 the .exe file. A wizard will help you install the Application; in fact, you just have to choose the
Application folder (the installation folder defined during the previous step is the default). The installation
process takes a few seconds. The following files are created:
Installation / Application folder
ComtradeTools task
Configuration file
Figure 6: the ComtradeTools files
(See next page)
9
Depending on the System options, a single click may be enough.
INFO³/OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
Page 10
UN Comtrade Version
The presence of the Application can also be checked through the usual Add/Remove Programs interface:
Figure 7: Add/Remove Programs interface
This utility should be used to uninstall ComtradeTools, as it is for any other application.
INFO³/OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
4.3.
Page 11
UN Comtrade Version
Test Web Services
To make sure that everything is in order –especially regarding access rights–, a request to ComtradeTools
can be sent through the Web Browser. For instance:
http://unstats.un.org/unsd/comtradedist/getSdmxV1.aspx?px=H1&r=381&y=2003,2002&cc=TOTAL&p=0&comp=false
This request asks for the transfer of a selected data set from the UN database to the OECD Client
workstation. The selection parameters are identified by keywords like ‘px’ or ‘r’. The meanings of these
keywords are explained below, in section 4.4. If the transfer operation works fine, the destination is a XML
file with contents like:
Figure 8: contents of the destination XML file after the transfer (partial view, may need enlargement)
The access control concerns the sender IP address as seen by the UNSD Server, as mentioned in the
Security section. If access is denied, the following message is sent:
Figure 9: access to ComtradeTools function denied
INFO³/OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
4.4.
Page 12
UN Comtrade Version
Download Data to a CSV File
The next requests will be sent from the Command Line (aka DOS) window. This window pops up when
selecting Start > Run. The following command is an example of download request:
ComtradeTools /r:360 /y:2003 /px:H1 /action:DownloadAndConvertToCSV
The program name, ComtradeTools, is followed by a list of slash-separated parameters. These parameters
are:
r: reporting country code. Here, 360 stands for Indonesia
y: reporting year
px: classification used. H1 means HS1996
action: here, a CSV file will be created with the converted imported data.
The Command Line window with the command (leftmost part) in it is reproduced below:
Figure 10: command line window
Clicking OK submits the transfer request to ComtradeTools. The resulting process can be followed on a
DOS pane that pops up:
Figure 11: command line execution – error(s) occurred
We see that the transfer failed. Submitting the URL that appears in this DOS pane would return the access
denied message in fig. 9.
(See next page)
INFO³/OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
Page 13
UN Comtrade Version
Had the access been granted, the transfer would have been successful, with the following messages on the
DOS pane:
Another
command line
Figure 12: command line execution – successful transfer
The code of the reporting country is now 8 (Albania): Indonesia would have returned too large a data set.
We notice an extra parameter in the command line: outputDirectory with C:\Temp as value. This is the local
directory the CSV file is created in. The name of this file is ‘Temph120018_CSV.txt’, with the selected
dataset values converted to CSV in it. The destination file name is built from the selection criteria:
classification, year and country, prefixed by the destination folder.
The picture below shows the first records of the destination CSV file:
Column
headers
Figure 13: dataset downloaded into a CSV file – first records
The first line is made of column headers, with, for instance, ‘8’ as reporting country. The next records are
the extracted data values.
INFO³/OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
4.5.
Page 14
UN Comtrade Version
Using the Configuration File
This file has the XML format and contains the definition of Keys that are used by ComtradeTools as
parameters. It is part of the ComtradeTools package (see fig. 6)
Its original contents (for the example we use) are displayed below:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<appSettings>
<!-- Web Service URL, DO NOT CHANGE -->
<add key="SdmxTariffLine" value="http://unstats.un.org/unsd/comtradedist/getSdmxTariffLineV1.aspx" />
<add key="XmlTariffLine" value="http://unstats.un.org/unsd/comtradedist/getTariffLine.aspx" />
<add key="SdmxComtrade" value="http://unstats.un.org/unsd/comtradedist/getSdmxV1.aspx" />
<add key="XmlComtrade" value="http://unstats.un.org/unsd/comtradedist/get.aspx" />
<add key="DaComtrade" value="http://unstats.un.org/unsd/comtradedist/refs/getDataAvailability.aspx" />
<add key="DaTariffLine" value="http://unstats.un.org/unsd/comtradedist/refs/getDaTariffLine.aspx" />
<!-- Mapping from SDMX to CSV Header, used only for action DownloadAndConvertToCSV, DownloadAndImportToSQLDB -->
<add key="AttributeMapping" value="RPT=reporter; time=year; CL=classification; TF=flow; CC-XX=commodity; PRT=partner;
netweight=netweight; qty=quantity; QU=QuantityUnit;value=TradeValue" />
<!-- SQL Mapping from CSV Headers to Column Names plus the Constant Value, used only for action
DownloadAndImportToSQLDB -->
<add key="SQLDestConnectionString"
value="UserName=sa;Password=password;Server=(local);Database=EmptyComtradeDev;Table=tempLoad" />
<!-- SQL Mapping from CSV Headers to Column Names plus the Constant Value, used only for action
DownloadAndImportToSQLDB -->
<add key="SQLColumnMapping"
value="year=yr;flow=rgCode;commodity=cmdCode;QuantityUnit=qtCode;classification=pfCode;reporter=rtCode;partner=ptCode;Trad
eValue=TradeValue;netweight=NetWeight;quantity=TradeQuantity" />
<add key="SQLConstValueMapping" value="source=UN" />
<!-- Time based Synch, updated automatically when SynchOption:TimeBased is selected-->
<!-- DateTime format is ProjectID=YYYY-MM-DDTHH:MM:SS -->
<add key="2000Data" value="LastSynch=2004-01-06T18:08:46;SynchOption=TIMESTAMP;Parameters=/y:2000" />
<add key="MexicoData" value="LastSynch=2004-11-06T18:08:46;SynchOption=TIMESTAMP;Parameters=/r:484" />
<add key="TFData" value="LastSynch=;SynchOption=TIMESTAMP;Parameters=" />
<add key="2003IndonesiaData" value="LastSynch=;SynchOption=TIMESTAMP;Parameters=/r:360 /y:2003 /cc:TOTAL" />
<add key="2003Data" value="LastSynch=2004-04-27T15:37:01;SynchOption=TIMESTAMP;Parameters=/y:2003 /cc:TOTAL" />
</appSettings>
</configuration>
Figure 14: the original ComtradeTools Configuration file – notice the comments (HTML format)
The keys in this file are defined by their names and values within an ‘add key’ tag. The ‘2000Data’ key has
been introduced in the Synchronization section. In the rest of this section, we’ll give examples of how the
Configuration file can be used.
INFO³/OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
4.5.1.
Page 15
UN Comtrade Version
Change Column Mapping
When the original UNSD data (SDMX format) is transferred to a CSV file, and later when the CSV data is
imported into the OECD SQL Server database, a column mapping must take place between the source and
the destination. The correspondence is defined by the two following keys in the Configuration file:
AttributeMapping:
SQLColumnMapping:
SDMX to CSV
CSV to SQL Server table
In the Configuration file, we’ll first change the AttributeMapping entry:
<add key="AttributeMapping" value="RPT=reporter; time=year; CL=classification; TF=flow; CC-XX=commodity;
PRT=partner; netweight=netweight; qty=quantity; QU=QuantityUnit;value=TradeValue" />
to:
<add key="AttributeMapping"
value="REPORTED_CURRENCY=cur;CONVERSION_FACTOR=cf;CL=classification; TF=flow; CCXX=commodity; PRT=partner; netweight=netweight; qty=quantity; QU=QuantityUnit;value=TradeValue" />
We notice that:
1. two correspondences have been removed: RPT=reporter; time=year.
2. two other correspondences have been added:
REPORTED_CURRENCY=cur; CONVERSION_FACTOR=cf.
We can then run the transfer request again:
ComtradeTools /r:8 /y:2001 /px:H1 /action:DownloadAndConvertToCSV OutputDirectory:C:\temp
And compare the new CSV file we get with the previous one (fig. 13).
Finally, we restore the two removed correspondences (reporter and year) in the AttributeMapping key.
SQLColumnMapping will be used later.
4.5.2.
Use ProjectID
Keys can be added to the Configuration file to replace a parameter string in the ComtradeTools command.
These keys (and therefore the parameter string) can be submitted via the ProjectID label in the command
line. For instance, we can insert the following definition in the Configuration file:
<add key="IndonesiaH1"value="LastSynch=;SynchOption=TIMESTAMP;Parameters=/r:360/px:H1" />
The name of the key is ‘IndonesiaH1’ and its contents make a selection by reporting country
(360=Indonesia) and classification (H1=HS1996). LastSynch is reset and TIMESTAMP is used as
synchronization option. The command line can now be:
ComtradeTools /y:2003 /action:DownloadAndConvertToCSV /outputDirectory:c:\temp\sdmx\
/ProjectID:IndonesiaH1
The transfer and conversion to CSV will be carried out, according to the parameters in IndonesianH1 and in
the command line. When the process is completed, the timestamp will be updated to keep the
synchronization process in order.
INFO³/OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
4.6.
Page 16
UN Comtrade Version
Import Data into a SQL Server Database
4.6.1.
Check SQL Server Updates
The User should first make sure that the sql2ksp3.exe Client component is installed on their workstation by
linking to:
http://www.microsoft.com/sql/downloads/2000/sp3.asp file
If this component is missing, the following error will occur:
"Unhandled Exception: System.InvalidCastException: QueryInterface for interface DTS.CustomTask failed."
Of course, this operation should be carried out only once for a given system.
4.6.2.
Create the Destination Table
The following commands and source data should be submitted to SQL Server Query Analyzer, either from
the keyboard or from a script file, to create the destination table in a database (for this example, any
existing database managed by SQL Server will do):
if exists (select * from dbo.sysobjects where id =
object_id(N'[dbo].[tempLoad]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)
drop table [dbo].[tempLoad]
GO
CREATE TABLE [dbo].[tempLoad] (
[pfCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[yr] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[rgCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[rtCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[ptCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[cmdCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[cmdID] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[qtCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[TradeQuantity] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[NetWeight] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[TradeValue] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[source] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[currency] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[conversion_factor] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL
) ON [PRIMARY]
GO
Figure 15: SQL Query Analyzer input to create the destination table in a database
This SQL sequence first checks for table ‘tempLoad’ existence; if the table exists, it is removed from the
database. ‘tempLoad’ is then (re-)created and its columns are defined with their names, type and length.
Here, type=Character and length=50 or 255.
4.6.3.
Define the Connection String
The following key should be added to the Configuration file:
<add key="SQLDestConnectionString"
value="UserName=sdmx;Password=sdmx;Server=(local);Database=Northwind;Table=tempLoad"/>
This is the path to the SQL Server database and the target table.
In this example, Northwind, the standard Windows demonstration database, is used. Optional value
parameters make it possible to customize the destination table according to Year, Reporting Country or
Classification.
INFO³/OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
4.6.4.
Page 17
UN Comtrade Version
Define the Column Mapping Between CSV File and Database Table
Key SQLColumnMapping in the Configuration file is used. Its value should be:
<add key="SQLColumnMapping"
value="reporter=rtCode;classification=pfCode;flow=rgCode;commodity=cmdCode;year=yr;partner=ptCode;Trad
eValue=TradeValue;netweight=NetWeight;quantity=TradeQuantity;cur=currency;cf=conversion_factor"/>
Compared to the original value in the Configuration file, the new columns ‘currency’ and ‘conversion_factor’
have been added.
Constant values (e.g., labels) can also be mapped through the SQLConstValueMapping key. In the original
Configuration File, we have:
<add key="SQLConstValueMapping" value="source=UN" />
4.6.5.
Import a Dataset into the Table – Debug Mode
The ComtradeTools command line is used again to transfer data from the UN database and import the
resulting CSV file into ‘tempLoad’ table in Northwind database, as defined in the Configuration File:
ComtradeTools /r:8 /y:2001 /px:H1 /action:DownloadAndImportToSQLDB
/removeTemp:false /DTSDebug:true
With the DownloadAndImportToSQLDB action, the intermediate step that results in the creation of the CSV file
is transparent to the User –its name is still built from the selection parameters. This action is carried out
through SQL Server DTS (Data Transformation Services). The DTSDebug option makes it possible to
create and examine the DTS package; this is useful for troubleshooting.
Refer to SQL Server documentation for details on DTS.
Once the action is completed the User should check the contents of table tempLoad in the database.
4.6.6.
Import Datasets
The IndonesianH1 key has already been appended to the Configuration file:
<add key="IndonesiaH1"value="LastSynch=;SynchOption=TIMESTAMP;Parameters=/r:360/px:H1" />
This key includes the reset of LastSynch (see section 4.5.2)
We can now re-run the direct import with the synchronization option:
ComtradeTools /action:DownloadAndImportToSQLDB /outputDirectory:c:\temp\sdmx\
/projectID:IndonesiaH1 /synchOption:Timestamp
4.6.7.
Use the New ‘Resume’ Parameter
As from 15 Aug 2005, a new parameter in the command line makes it possible to resume the selected
ComtradeTools action up to n times if errors are encountered. This parameter is used in the command
below:
ComtradeTools /action:DownloadAndImportToSQLDB /outputDirectory:c:\temp\sdmx\
/projectID:IndonesiaH1 /synchOption:Timestamp /resume:3
The operation will be resumed up to three times until no error occurs.
INFO³/OECD
Ref: 219546157
UNSD - OECD Joint System
Data Transfer Mechanism - Technical Specifications
Page 18
UN Comtrade Version
FIGURES
Figure 1: statistical data flow for UNSD and OECD in Joint System
Figure 2: example of SDMX cross-sectional schema
Figure 3: transfer mechanism between UNSD and OECD
Figure 4: the import - synchronization process
Figure 5: the three ComtradeTools setup files – unpacked by WinZip
Figure 6: the ComtradeTools files
Figure 7: Add/Remove Programs interface
Figure 8: contents of the destination XML file after the transfer (partial view, may need enlargement)
Figure 9: access to ComtradeTools function denied
Figure 10: command line window
Figure 11: command line execution – error(s) occurred
Figure 12: command line execution – successful transfer
Figure 13: dataset downloaded into a CSV file – first records
Figure 14: the original ComtradeTools Configuration file – notice the comments (HTML format)
Figure 15: SQL Query Analyzer input to create the destination table in a database
INFO³/OECD
3
5
6
7
8
9
10
11
11
12
12
13
13
14
16
Ref: 219546157