WORLD METEOROLOGICAL ORGANIZATION
COMMISSION FOR BASIC SYSTEMS
----------------------------THIRD MEETING OF
INTER-PROGRAMME EXPERT TEAM ON
DATA REPRESENTATION MAINTENANCE AND
MONITORING
IPET-DRMM-III / Doc. 7.3
(13. 7. 2015)
------------------------ITEM 7.3
ENGLISH ONLY
BEIJING, CHINA, 20 - 24 JULY 2015
MIGRATION TO TABLE DRIVEN CODE FORMS (TDCF)
Follow-up of migration to TDCF, including issue of partitioned
TEMP BUFR messages
Submitted by the Secretariat
_______________________________________________________________________
Summary and Purpose of Document
This document summarizes the Status Report of IPET-DRMM on migration
success and remaining issues.
_______________________________________________________________________
ACTION PROPOSED
The meeting will discuss further actions on migration to TDCF, noting the information in;
a.
the Status Report of IPET-DRMM on the migration success and remaining issues,
b.
the letter to the PRs of WMO Members on migration of upper-air reports to BUFR.
REFERENCES:
Abridged final report with resolutions and recommendations (CBS-Ext.(2014),
Asunción, 8-12 September 2014) (WMO-No.1140)
ANNEX:
The letter on migration success and remaining issues
DISCUSSIONS
1.
The Commission for Basic Systems Extraordinary Session (CBS-Ext.(2014)) took place in
Asunción, Paraguay, 8-12 September 2014. It asked the OPAG-ISS to review progress with
migration every six months and to consider whether a further of extension of six months was
required to resolve issues with the consistency and quality of data.
2.
The CBS-Ext.(2014) considered that no changes were necessary to the migration matrix,
while it felt some measures were needed to facilitate and complete the migration, one of which is as
follows.
(f)
Noting the imminent November 2014 deadline, IPET-DRMM should prepare a report to be
sent to Permanent Representatives of Members with WMO to communicate migration successes and
remaining issues;
3.
Ms Jitsuko Hasegawa, a co-chair of IPET-DRMM, has played a role in drafting the Status
Report by email in association with some active contributors. The draft has been endorsed by
the chair of IPET-DRMM and will be ready to be posted to the PRs of WMO Members. The
identified issues stated in the IPET-DRMM Status Report are listed along with some
recommendations in the Annex to this document.
4.
The CBS Management Group (CBS-MG) noticed difficulty in data processing by NWP
centres with regard to upper-air reports in BUFR converted from massages in FM 35 TEMP and
its family (TAC TEMP). In view of its urgent and critical nature, the president of CBS has
approved a letter on migrating upper-air report to BUFR, which was posted on 7 July 2015 to the
Permanent Representatives of WMO Members. It urges continuing transmission of TAC TEMP
parts A, B, C, and D, if it is unable to produce upper-air BUFR reports that contain complete
ascents.
https://www.wmo.int/edistrib_exped/grp_prs/index.php?dir=_en/&file=2015-07-07-PR6855-OBS-WIS-DRMM_en.pdf
5.
The meeting will review the above issues and recommendations, and discuss further
measures to facilitate the migration in short term and mid/long terms.
ANNEX
The Identified issues in encoding and decoding BUFR reports and in managing
migration process along with recommendations stated in the IPET-DRMM Status
Report
CONTENTS
1.
Identified issues in encoding and decoding BUFR reports
1.1
Use of BUFR editions and templates
1.2
Drift information in upper-air data
1.3
Metadata
1.3.1
Observation station position, latitude/longitude, height
1.3.2
Identifier of upper-air data from ships
1.3.3
Instruments types and software versions
1.3.4
BUFR content errors due to conversion from TAC, etc.
1.4
Reformatted BUFR upper-air reports separated into parts
1.5
Use of BUFR reports in NWP systems
2
Identified issues in managing migration process
2.1
Lack of shared understanding of migration goals
2.2
Lack of advanced notification of turning off TAC reporting
2.3
Lack of consolidated process for quality checking and reporting issues
2.4
Training and expertise
2.5
Lack of capacity and infrastructure
3.
Recommendations
3.1
Recommended actions for all WMO Members
3.2
Recommendation on BUFR upper-air reports
3.3
Scheme for managing issues with representation of data
1.
Identified issues in encoding and decoding BUFR reports
1.1
Use of BUFR editions and templates
Some BUFR reports are still using edition 3, rather than edition 4 as they should do. TDCF have
templates, i.e. sequences of descriptors designed for reporting specific observation. The use of
templates is not consistent among BUFR producers, which puts additional work on data users.
New BUFR templates for ships and buoys have not been used widely.
1.2
Drift information in upper-air data
Though drift information (position at each point of ascent) is one of the benefits of native BUFR,
as of January 2015, only about 20 % of upper-air stations provide drift information. Many stations
using radiosonde types with the capability to provide drift information do not report it.
1.3
Metadata
Although having metadata of observation stations and instruments in each message is one of the
strong points of TDCF, there are still issues to be solved.
1.3.1
Observation station position, latitude/longitude, height
Position metadata errors are reported, including missing latitude/longitude, changing positions in
time and different coordinates between registered stations in WMO Publication No.9, Volume A
and BUFR location coordinates. The position information is missing or wrong for up to 30% of
BUFR radiosonde reports. The biggest single cause of error seems to be conversion error from
degrees and minutes to decimals (e.g., 3 30 is converted to 3.30 instead of 3.50). Station height
is also important, especially for surface stations but hasn't been examined yet. In addition, BUFR
is capable of reporting the height of pressure sensor, but in many cases this is filled by the same
value of the height of the station.
1.3.2
Identifier of upper-air data from ships
Identifiers are occasionally missing or blank. Some radiosonde reports from ships have a
numeric WMO identifier and a ship call sign, which is confusing because there is no clean way to
distinguish ship from land reports. Some radiosonde reports from fixed land station has
characters in ship or mobile land station identifier, which is supposed to be missing for fixed land
stations, and cause decoding errors.
1.3.3
Instruments types and software versions
Errors of reported radiosonde types are problematic to NWP systems, where the information can
be used for radiation bias correction, observation error estimates and for decision on use of
upper-tropospheric humidity. Wrong reports of radiosonde type in BUFR is mainly caused by
using the same radiosonde type code figures used in TEMP, because in Common Code table C2 in Manual on Codes (WMO-No.306, Volume I.2 - Part C) for radiosonde types, code figures up
to 99 are used for TEMP and code figures up to 255 are used for BUFR; for example reports
labeled type 30 in TEMP can be 130 in BUFR. There are also cases of missing values and
values that appear to be wrong in TEMP and/or BUFR.
BUFR template for reporting upper-air sounding may have additional descriptors to identify
radiosonde serial number and software version, which can be used to identify instrument and
processing changes to a particular radiosonde type and are potentially useful for radiosonde bias
correction in NWP and climate studies. This information has not been fully used in NWP
systems.
1.3.4
BUFR content errors due to conversion from TAC, etc.
A number of errors are detected in many BUFR reports converted/reformatted from TAC. Many
of them are caused by errors in converting units and parameters, as well as in decoding source
TAC messages, which consist of variety of complicated rules (such as omission of leading digits
to reduce message length). The issue is that TAC decoders used in TAC-TDCF conversion
often do not have skills to treat these specific rules correctly, unlike decoders used in NWP
systems, which have accumulated experience in treating special cases. Below are the lists of
reported errors in reformatted BUFR reports. Generally speaking, surface BUFR reports
(reformatted from SYNOP and SHIP) have fewer and less serious errors than upper-air BUFR
reports (reformatted from TEMP and PILOT).
One of the motivations to introduce TDCF was to graduate from handling TAC’s complex rules.
Due to slow communications speeds at the time the TAC were first put in practice, much effort
was expended on devising coding rules that aimed to minimize the length of messages.
Unfortunately, initial investigation results show that many BUFR reports are still affected by these
rules through the TAC-TDCF reformatting process.
For example, geopotential heights are coded with omitted leading digits in TAC. This practice is
sometimes erroneously perpetuated in converted BUFR TEMP. In TEMP reports, geopotential
heights are reported in whole geopotential meters; the thousands digit is omitted for surfaces up
to, but not including, 500-hPa. For example, 3,249 meters at 700-hPa is coded as 249 in TEMP
and the thousand digit needs to be inferred when decoding the TEMP report to generate a BUFR
message, based on the approximate heights of the standard isobaric surfaces. Wrong guesses
are often made for the height at 700-hPa, because the actual height varies from around 2,100
meters (e.g., in polar region) to 3,400 meters (i.e. the leading digit can be both 2 and 3).
Similarly, geopotential heights are reported in tens of geopotential meters for surfaces at 500hPa and higher (i.e., only the thousands, hundreds, and tens digits of the height are reported;
11,784 meters at 200-hPa level is coded as 178). The tens of thousands digit need to be
inferred, which sometimes brings wrong values in converted/reformatted BUFR messages.
There is also an exceptional rule for coding negative geopotential heights (below sea level),
which is reported by adding 500 to the absolute value of the geopotential in TAC (i.e., –90 meters
is coded as 590). This is often wrongly coded as 590, instead of –90 in converted/reformatted
BUFR messages. These errors are rooted in intrinsic complexity of TAC and illustrate the
difficulty of conversion/reformat.
Reported errors and offsets in reformatted surface BUFR reports include:
 Offsets of temperature, caused by the Celsius (TAC) and Kelvin (BUFR) conversion and
rounding of values in TAC reports.
 Wrong conversion and/or mixture of wind speed between knots and km/hour
 Rounding difference in wind speed


Missing leading digits in pressure
Slight difference in pressure reported in TAC and BUFR
Reported errors and offsets in reformatted upper-air BUFR reports include:
 Confusion of dew point depression and dew point temperature
 Wrong conversion of dew point depression, which has a special treatment in TEMP code
table in a specific depression range.
 Problems with the reported tropopause level
 Surface winds repeated as upper-level significant winds
 Surface data repeated as 1000 hPa data
 Wrong conversion and/or mixture of wind speed between knots and km/hour
 Wrong guesses of omitted leading digits of geopotential heights in TEMP code
 Unrealistic values in temperature and missing vertical coordinate
1.4
Reformatted BUFR upper-air reports separated into parts
The main issue of BUFR upper-air reports for NWP users is the reformatted BUFR which are
received for separated parts A, B, C and D. The practice comes from TAC reports, where levels
are split into standard and significant levels and reported separately up to 100 hPa and above.
The original idea of introducing TDCF was to move towards high vertical-resolution reporting and
also to simplify the reporting structure to include all the information from one ascent to one
report. Unfortunately, however, the reporting structure has been transferred to many BUFR
reports through the TAC-BUFR reformatting process. To make things worse, the same GTS
headers are used, making it all but impossible to identify and merge the constituent parts. This
practice is clearly excluded by B/C 25 regulations1, but many Members seem to be unaware that
these reports are not only uncompliant, but almost unusable.
1.5
Use of BUFR reports in NWP systems
According to the survey conducted by ECMWF in January 2015, surface observation data
provided by BUFR format are used in six global NWP centres, in the proportion of 1–70 % for
SYNOP and in one center in 80 %, upper-air data in four global NWP centres in the proportion of
0.5–2 %, which indicates that NWP centres still have a long way to go to complete the migration.
2
Identified issues in managing migration process
2.1
Lack of shared understanding of migration goals
Not all Members shared a common understanding of the requirements and justification of the
migration to TDCF. In particular, the migration strategy of using TAC-TDCF conversion was
adopted too easily without enough understanding of its deficiencies. The benefits of native
TDCF reports (generated directly from measured data) should have been stressed more
1
A BUFR (or CREX) message shall be sent when the 100 hPa level is reached. Subsequently, a BUFR (or CREX)
message shall be produced when the sounding is completed containing data from the entire sounding. If the
sounding is terminated below 100 hPa level, only the later message shall be produced.
If high resolution data are reported, only one BUFR message shall be sent when the 100 hPa level is reached and
only one BUFR message shall be produced when the sounding is completed, provided that all standard and
significant levels are properly identified in compliance with the relevant B/C 25 Regulations.
forcefully throughout the migration process, especially in the planning phase, so that Members
could better appreciate and take actions towards the goal of migration, which is to improve
observation report quality and availability, not to change data formats. In the case of BUFR
upper-air reports, the importance of native high-resolution BUFR reports should have been
clearly communicated in the early stage, although this is mentioned in B/C 25.
2.2
Lack of advanced notification of turning off TAC reporting
One of the critical issues of managing migration is some Members’ stopping TAC distribution with
no or very-short advanced notification, leaving no time for data users to validate contents of new
BUFR messages. Notification of amendments in WMO publications is supposed to be sent to
the Secretariat at least two months in advance of the effective date of the change (Manual on
GTS), but this seems not enough for data users to validate the contents.
2.3
Lack of consolidated process for quality checking and reporting issues
There is a need for quality checking of new TDCF bulletins and a validation process taking into
account the comparison with TDCF messages and their TAC equivalent, as well as the absence
of a consolidated process for reporting and addressing issues arising from the migration needs to
be resolved. A standard procedure for reporting issues with the content of TDCF reports will help
reduce the number of reports that are unusable.
2.4
Training and expertise
The TDCF represent a substantial paradigm shift over the TAC, whose way is profoundly
ingrained throughout the worldwide meteorological enterprise. A very large amount of training
was invested in TAC over the years, and they are well understood to the point of being second
nature to staff, all the way up to the most senior. Thus, not only is technical training required, but
an actual culture change is needed. Significant efforts have been devoted to TDCF training, but
more is needed.
2.5
Lack of capacity and infrastructure
Aside from training issues, existing infrastructure in low capacity States may not be capable of
handling TDCF. In those cases, an upgrade, eventually leading to completion of the Migration, is
dependent on grants or help from higher capacity partners.
In the case of high capacity operators, the TAC may be deeply hard-wired into the existing
production systems. In such a case, a complete overhaul of the physical and software
infrastructure may be necessary as a precondition for the Migration to take place. Carrying out
such a change in a complex operational system can easily take longer than initially anticipated.
3.
Recommendations
3.1
Recommended actions for all WMO Members
1) To make sure all the data are available in TDCF format, in full compliance with B/C
regulations of Manual on Codes, (WMO-No.306, Volume I.2 – Part C, d. Regulations for
reporting traditional observation data in Table-Driven Code Forms (TDCF): BUFR or CREX).
2) To make sure that all the metadata of observation stations and instruments are properly
maintained and provided in TDCF.
3) To review national migration strategy to include the goal that TDCF reports be generated
directly from measured data where possible, instead of by converting/reformatting TAC
reports.
4) To cease the parallel dissemination of TAC as soon as possible, but only after providing all
centres with advanced warning and ascertaining that the TDCF is of equivalent or better
quality (see “Scheme for managing issues with representation of data”).
5) To review national meteorological training strategy to harmonize current and future
institutional knowledge and practices with TDCF regulations and templates, and to seek or
undertake TDCF training activities as required to facilitate the completion of Migration
activities.
3.2
Recommendation on BUFR upper-air reports
Members are encouraged to pursue ways, depending on the levels of observation and
communication infrastructure capacity and capability, to take advantage of additional reporting
features provided for upper-air data in BUFR, especially native BUFR reports with high vertical
resolution and drift positions. Recognizing that many centres are unable to produce anything
other than reformatted BUFR reports from TEMP and PILOT, NWP centres stated they prefer to
continue using TEMP and PILOT reports rather than using reformatted BUFR reports in parts,
until data producers are ready to provide native BUFR reports containing all data from one
ascent. IPET-DRMM recommends that Members reevaluate BUFR upper-air reports quality to
make sure that they are compliant with the regulations and consider the option to continue TAC
reporting regardless the November 2014 deadline, with a clear migration plan to native BUFR, if
reformatted BUFR in parts is the only possibility at the moment.
3.3
Scheme for managing issues with representation of data
A scheme for managing migration process is being developed, to ensure enough time to validate
new messages and to establish a systematic quality checking and reporting process with
necessary escalations so that identified issues are addressed towards correction and
improvement. The scheme will be informed to Members in due course. This should be
supported by a standard procedure for notifying and managing issues that are detected in the
reports in TDCF.
REFERENCE:
1.
LIST of countries sending BUFR reports corresponding to the separate parts of TAC
TEMP (produced from Volume C1)
TEMP Parts
Other TEMP
Country
A
B
C
D
Other
Grand Total
Antarctic
1
1
1
1
2
6
Bangladesh
1
1
1
1
4
Belarus
2
2
China
9
2
9
2
22
India
1
2
2
2
1
8
Japan
27 27 15 25
94
Mongolia
1
1
1
1
4
Russian federation
4
4
Thailand
1
1
United Kingdom of Great
Britain and Northern Ireland
5
5
5
5
5
25
Algeria
4
4
Hong Kong, china
2
2
Argentina
3
3
Australia
51
51
Cook Islands
4
4
Fiji
4
4
Indonesia
8
8
New Zealand
8
8
Austria
8
8
Belgium
2
2
Croatia
4
4
Cyprus
2
2
Czech Republic
4
4
Denmark and Faroe Islands
19
19
Estonia
2
2
Finland
8
8
France
18
18
Germany
30
30
Greece
6
6
Ireland
2
2
Italy
20
20
Lithuania
2
2
Netherlands
4
4
Norway
18
18
Portugal
5
5
Slovakia
2
2
Spain
14
14
Sweden
Switzerland
French Polynesia
Philippines
Singapore
Israel
Portugal
Romania
Serbia
Slovenia
Turkey
Antarctic - in Region VI
Grand Total
Number reported as parts
Number not reported as parts
Percent reported as parts
2.
52
39
34
37
16
2
8
4
1
1
1
1
1
1
1
1
300
16
2
8
4
1
1
1
1
1
1
1
1
462
162
300
35%
List of countries that are/appear to be sending separate parts (prepared by ECMWF from
the data received on 1-7 June 2015)
II
04
12
40
40/41
41
42/43
44
47
47
47
50-59
60
61
61
65
68
70/72/74
78
78
78
78
Iceland
Poland
Jordan
Saudi Arabia
Pakistan
India
Mongolia
North Korea
South Korea
Japan
China
Algeria
Senegal
St Helena
Burkina Faso
South Africa
USA
Bahamas
Jamaica
Dominican Republic
Puerto Rico
Parts
Parts
Parts
Parts
intermittent BUFR in Parts
Parts, sometimes no BUFR
Parts, identifier problem
Part A only? (occasional)
Parts
Parts
Parts, no BUFR from some stations
Parts
some BUFR, Parts
Parts
Parts
Parts
Parts (+ merged profile sometimes)
Parts
Parts
Parts (+ merged profile)
Parts (+ merged profile sometimes)
78
78
78
78
78
80
81/82/83
84
85
88
91
91
91
91
93
94/95
96/97
96
98
Belize
Curacao + St Maarten
Guadeloupe
Barbados
Trinidad and Tobago
Colombia
Brazil
Ecuador
Chile
Falklands
Hawaii + other islands
Fiji
American Samoa
Cook Islands
New Zealand
Australia
Indonesia
Cocos Islands
Philippines
Parts
Parts + merged profile
Parts
Parts + merged profile
Parts + merged profile
Parts (or merged, some parts/reports missing)
Parts
Parts (sometimes merged)
Parts (sometimes merged), sometimes no BUFR
Parts
Parts sometimes incomplete
Parts
Parts
Parts
Parts
Parts (HiRes soon?)
Parts
Parts
Parts (missing block and index number)