An Introduction to GRIB2 Simon Elliott EUMETSAT simon.elliott@eumetsat.int WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Saturday Afternoon Agenda • Background (i.e. my background!) • Overview (codes, GRID, GRIB, GRIB2) • Code Structure (Sections like others, iterations of sections) • GRIB2 Code Templates (what kinds are there … examples) • Tables (Template tables, code and flag tables, examples) • Identification of parameters (?) • GRIB2 compression (methods, explanations of some) WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Sunday Morning Agenda • Procedure for code form update (addns to tables, addition of tables/templates, new editions) • Changes not requiring code change (adding table/template entries for decoders … down stream code change) • Changes requiring code change (always for data provider, plus new editions) • Case studies of Cloud Mask and Precipitation Data (bespoke Encoder and Decoder, example product) • GRIB2 Interface for NMC (example of NCEP code) • Review/Discussion WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Overview • Intention is to share information efficiently and unambiguously • Data should be accessible by anyone, not encrypted • Format should allow all information to be included (not rounded or skipped) • Format should allow addition of new data types • WMO “codes” developed per data type, TEMP, SYNOP, SATOB, et c. FM 35 TEMP FM 42 AMDAR FM 41 CODAR WORLD METEOROLOGICAL ORGANIZATION FM 13 SHIP RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 FM 88 SATOB FM 87 SARAD FM 86 SATEM FM 85 SAREP GRID, GRIB, GRIB2 • GRID was developed for gridded data (forecast, analysis) – ASCII -> human readable? – ASCII -> big files, slow transfer – Symbolic letters and code tables: F1F2NNN in Section 0 • F1F2 is originating centre as per C-1, • NNN is catalogue number of grid used by centre • GRIB introduced for binary exchange (storage and transmission efficiency). – Edition 0, 1985 – Edition 1, 1990 • GRIB Edition 2 (i.e. GRIB2) because Code Table 2 was full, EPS data, ... WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 GRIB1 structure Section 0 Indicator section: “GRIB” Section 1 Product definition section: unique standard template Section 2 Grid description section: one of the standard templates describing a type of grid Section 3 Bit map section Section 4 Binary data section Section 5 End section: “7777” GRIB uses the concept of Template: « Description of the standardized layout of a set of data items » WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 GRIB usage • • • • On the GTS By the WAFS Centres (« ICAO » products) For archiving of fields on MDD Main limitation is: • One parameter on one level for one grid per GRIB field WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 GRIB1 weaknesses Transmission and archiving of: • Spectral data • Multi-dimension data • Long-range and climate products • Ensemble products (EPS) Also: • No convention for missing data • IEEE not used for floating point data • No support for small time steps • No cross-sections, no time-sections • No Hovmöller Diagrams (ex. Time-longitude) • Limited support for images WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Need for “object oriented” GRIB2 (more modules = modularity) • Modularity: code and parameter tables referred to through templates • Flexibility: new tables and templates can be added • ALSO: – All GRIB1 fields can be described in GRIB2 – More compression schemes (e.g. introduction of JPEG 2000 and PNG) WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 GRIB2 calendar • • • • • Experimental GRIB2 presented at CBS 98 Finalized in Spring 2000 Approved by CBS in Autumn 2000 Operational in November 2001 First products in 2003 WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Generic structure Table driven codes generally have this structure WORLD METEOROLOGICAL ORGANIZATION • Identification: GRIB/BUFR/CREX • Header: Date, time, originator, table versions ... • Optional section: Metadata (potentially XML), private data … • Data description: What sort of data follows • Actual data: here • Closure: “7777” RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 GRIB2 structure • • • • • • • • • Section 0 Indicator section: “GRIB” Section 1 Identification section: data characteristics Section 2 Optional (local) section: anything or nothing Section 3 Grid definition section: geometry of grid used, potential reference to pre-defined grid Section 4 Product definition section: description of data type, e.g. satellite data from spectral bands xx, yy, zz Section 5 Data representation section: packing method used for data, reference value, scale factor et c. Section 6 Bit map section: data present indicators (if used) Section 7 Data section: data values themselves Note: Sections 1 to 7 start with Section 8 End section: “7777” section length and number WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Section 0 Section 0: Indicator section – – – – – Octets 1 - 4: Octets 5 - 6: Octet 7: Octet 8: Octets 9 - 16: “GRIB” in ASCII (i.e. 71 82 73 66) Reserved (normally set to 0) Discipline (CT 0.0, e.g. 3 for space) GRIB edition number (2) Total length in octets Messages of length up to ~18x1018 (or 264 - 1) bytes, i.e. ~18,000,000 Tb WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Section 1 Section 1 Identification section – – – – – Octets 1 - 4: Octet 5: Octets 6 - 9: Octets 10 - 11: Octet 12: time) – Octets 13 - 19: – Octet 20: operational data) – Octet 21: observations) WORLD METEOROLOGICAL ORGANIZATION Section length Section number (1) Originating centre and sub-centre Master and local table version numbers Time significance (CT 1.2, e.g. 3 for observation Date / time Production status of data (CT 1.3, e.g. 0 for Type of data (CT 1.4, e.g. 7 for processed radar RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Section repetition WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Repeatable Section 0 (Indicator) Section 1 (Identifier) Section 2 (Local use) Section 3 (Grid) Section 4 (Product) Section 5 (Data Repn) Section 6 (Bit map) Section 7 (Data) Section 8 (Ending) Repeatable • • • • • • • • • Repeatable GRIB2 allows some groups of sections to be repeated for efficiency (not in GRIB1, one field per message). Non-repeated sections stay in effect Section repetition (example 1) • • • • • • • • • • • • • • • • Section 0 (Indicator) Section 1 (Identifier) Section 3 (Grid) Section 4 (Product) Section 5 (Data Repn) Section 6 (Bit map) Section 7 (Data) Section 4 (Product) Section 5 (Data Repn) Section 6 (Bit map) Section 7 (Data) Section 4 (Product) Section 5 (Data Repn) Section 6 (Bit map) Section 7 (Data) Section 8 (Ending) WORLD METEOROLOGICAL ORGANIZATION 0600Z 1200Z 1800Z Forecast from same centre on same grid for same parameter at same level but from but for different validity times Validity time is in Section 4 (Product Definition) Section 4 to 7 repeated in message RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Section repetition (example 2) • • • • • • • • • • • • • • • • • • Section 0 (Indicator) Section 1 (Identifier) Section 3 (Grid) Section 4 (Product) Section 5 (Data Repn) Section 6 (Bit map) Section 7 (Data) Section 3 (Grid) Section 4 (Product) Section 5 (Data Repn) Section 6 (Bit map) Section 7 (Data) Section 3 (Grid) Section 4 (Product) Section 5 (Data Repn) Section 6 (Bit map) Section 7 (Data) Section 8 (Ending) WORLD METEOROLOGICAL ORGANIZATION GRID A GRID B GRID C Forecast from same centre for same validity time for same parameter at same level but from but on different grids Grid is defined in Section 3 (Grid Definition Section) Section 3 to 7 repeated in message RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Grid definition templates • Grid definition is in Section 3 (GRIB1, Section 2) • Can refer to centre’s own grid or use Grid Definition Template (GDT) to specify details • Many GDTs have been developed, e.g. – – – – 3.0: Latitude/Longitude (or equidistant cylindrical, or Plate Carrée) 3.1: Rotated Latitude/Longitude (or equidistant cylindrical, or Plate Carrée) 3.2: Stretched Latitude/Longitude (or equidistant cylindrical, or Plate Carrée) 3.3: Stretched and Rotated Latitude/Longitude (or equidistant cylindrical, or Plate Carrée) – 3.20: Polar Stereographic – 3.90: Space view perspective, or orthographic WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 A grid definition template example GDT 3.90 - Space view perspective or orthographic Floating point numbers stored in 5 bytes as scale factor (one byte) and scaled value (4 byte IEEE floating point) Octet No. 15 Contents Shape of Earth (CT 3.2: Spherical, oblate sphere, ICAO shape) 16-20 Radius of spherical Earth 21-30 Major and minor radii of oblate spherical Earth 31-38 Number of pixel columns and rows in grid 39-46 Latitude and longitude of sub-satellite point 47 Resolution and component flags (FT 3.3: Reference direction for vector components, grid or east/north) WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 GDT example continued Octet Contents No. 48-55 Apparent diameter of Earth in each direction 56-63 Co-ordinates of sub-satellite point 64 Scanning mode (FT 3.4: location of consecutive scan point and lines, e.g. 1st row in –x dirn, all scan rows in same dirn) 65-68 Orientation of grid (i.e. skeweness relative to longitude meridien) 69-72 Height of camera in equatorial radii units, scaled by 106 73-80 Co-ordinates of origin of image Repeating sections 4 to 7 means this information need not be repeated for multiple fields in one GRIB2 message WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Product definition templates • Product definition is in Section 4 (GRIB1, Section 1) • Many PDTs have been developed, e.g. – 4.0: Analysis or forecast on horizontal level or layer at a point in time – 4.1 to 4.4: Various information pertaining to ensemble forecast systems – 4.5: Probability forecast on horizontal level or layer at a point in time – 4.7: Analysis or forecast error on horizontal level or layer at a point in time – 4.20: Radar products – 4.30: Satellite products • Hybrid sigma levels can be specified in Section 4 WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 A product definition template example PDT 4.30 - Satellite products Octet No. 10 Contents 13 Parameter category (CT 4.1: For space products, i.e. discipline 3, 0 = image, 1 = data are in physical units) Parameter number (CT 4.2: radiance, albedo, brightness temperature, skin temperature, cloud mask, …) Type of generating process (CT 4.3: analysis, forecast, observation, …) Observation generating process identifier (locally defined) 14 Number of contributing spectral bands, NB 11 12 WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 PDT example continued Octet Contents No. 15-16 Satellite series of band 1 (defined by originating centre) 17-18 Satellite number of band 1 (defined by originating centre) 19 Instrument type of band 1 (defined by originating centre) 20-24 Central wave number of band 1 (m-1) Note: Octets 25 to 24 are given for band 1, but are repeated for each contributing band up to NB, as specified in octet 14 WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Code table 4.3: Type of generating process Code figure 0 1 2 3 4 5 6 7 8 192 - 254 255 WORLD METEOROLOGICAL ORGANIZATION Meaning Analysis Initialisation Forecast Bias corrected forecast Ensemble forecast Probability forecast Forecast error Analysis error Observation Reserved for local use Missing RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Data representation templates • Data representation is in Section 5 (GRIB1, within Section 4) • Some DRTs have been developed, e.g. – – – – – – 5.0: Grid point data - simple packing 5.1: Matrix values at grid point - simple packing 5.2: Grid point data - complex packing 5.3: Grid point data - complex packing and spatial differencing 5.50: Spectral data - simple packing 5.51: Spherical harmonics data - complex packing • Section 5 also gives total number of data values to be found in Section 7 WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 A data representation template example DRT 5.0 - Grid point data - simple packing Octet Contents No. 12-15 Reference value, R (IEEE floating point number) 16-17 Binary scale factor, E 18-19 Decimal scale factor, D 20 Number of bits used for each packed value 21 Type of original field values (CT 5.1: Floating point or integer) IEEE floating point numbers in 32 bits: seeeeeee emmmmmmm mmmmmmmm mmmmmmmm WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Simple and complex packing • Simple packing – Y x 10D = R + (X1 + X2) x 2E – Y is original value, D is decimal scale, R is reference, E is binary scale, X1 is 0, and X2 is encoded value – In template R, E, D and bits per value are stored – Field values follow sequentially • Complex packing (intended to decrease message size) – Y x 10D = R + (X1 + X2) x 2E – R, E, D are as for simple packing – X1 is reference for group, X2 is scaled value in group (X1 removed) – Data are split into groups with similar values per group WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Use of D and E D 0 -1 1 0 -1 -2 1 0 -1 -2 0 E 0 -3 4 1 -2 -5 5 2 -1 -4 3 Increment 1.000 1.250 1.600 2.000 2.500 3.125 3.200 4.000 5.000 6.250 8.000 WORLD METEOROLOGICAL ORGANIZATION DD-1 Increment Increment x 10 Here R = 0, X1 = 0, X2 = 1 Increment = Y = original value D -1 -2 0 -1 -2 -3 0 -1 -2 -3 -1 RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 E 0 -3 4 1 -2 -5 5 2 -1 -4 3 Increment 10.000 12.500 16.000 20.000 25.000 31.250 32.000 40.000 50.000 62.500 80.000 R, E and D selection example Temperature on balcony: -15.0 to 35.0, ±0.2 • Set D to give increments in range 1 - 10 – D = 1, i.e scale by 10 – data are now: -150 to 350, ±2 • Set R to minimum – R = -150.0 (IEEE floating point) – data are now: 0 to 500, ±2 • Set E to give required precision – E = 1, i.e. data are in increments of 2 • Set bit width to give required range – Required range = scaled range / increment – Required range = 500 / 2 = 250 so 8 bits R = -150, E = 1, D = 1, bit width = 8 WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Spatial differencing • Saves space for data which vary smoothly (order 1), or data whose variation varies smoothly (order 2), et c. • For order 1, data values, f, define new values, g: – g1 = f1, g2 = f2 - f1, …, gn = fn - fn-1 – 1, 2, 3, 5, 7, 8, 9, 11 becomes 1, 1, 1, 2, 2, 1, 1, 2 • For order 2, g values are replaced by h: – h1 = f1, h2 = f2, h3 = g3 - g2, …, hn = gn - gn-1 – 1, 2, 4, 10, 17, 27, 40, 57 becomes 1, 2, 1, 4, 1, 3, 3, 4 • Minimum value is subtracted to keep values positive Order 1 Complex packing 4 groups WORLD METEOROLOGICAL ORGANIZATION Order 2 Complex packing 3 groups RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Section 6 Section 6: Bit map section – – – – Octets 1 - 4: Octet 5: Octet 6: Octets 7 - xx: Length of section Section number (6) Bit map indicator (CT 6.0) Bit map (if present) Code Table 6.0: Bit map section Code figure 0 1 - 253 254 255 WORLD METEOROLOGICAL ORGANIZATION Meaning Bit map is present and follows in this section Use bit map predefined by originating centre Re-use previously defined bit map Missing (no bit map present) RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Bit map case study With bit map No data No bit map bit map (2000 x 2000 x 1) = 4,000,000 bits 2000 x 2000 x 8 2000 = 32,000,000 bits 2000 Data (8 bits per pixel) data ( x 1000 x 1000 x 8) 25,130,000 bits total 29,130,000 bits Conclusion: bit map may not always save space, but if more than two bits per data point (typically true) it usually will … consider cloud top height data, space corners, et c. WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Updates not requiring code change • Type of changes concerned – Addition of code and or flag table entries • Agreement and implementation – “Fast Track” can be used – Table entries can be approved for pre-operational use within a few months of consideration and successful validation – Full formal CBS approval follows, available around 2 years later • Application updates – – – – Encoder needs new code / flag table available Encoder software needs updating to insert field Decoder needs new code / flag table available Down-stream application (beyond decoder) needs to be updated to process new field – Decoder software can remain unchanged WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Updates requiring code change • Type of changes concerned – Changes to code structure (new edition), or new GDT, PDT or DRT • Agreement and implementation – ET/DRC considers a request and proposes update – Multilateral validation of proposed update – Full formal CBS approval follows, available around 2 to 4 years later • Application updates – – – – Encoder needs new software and /or template available Encoder software needs updating accordingly Decoder needs new software and /or template available Down-stream application (beyond decoder) needs to be updated to process new data • Long lead time necessary … consider PUMA example WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Interfacing to third party software • Use of third party generic software simplifies implementation • NOAA provides GRIB2 software for free download (look at http://www.nws.noaa.gov/tdl/iwt) • ECMWF provides GRIBEX software - will be / is being updated for GRIB2 (look at http://www.ecmwf.int/products/data/software/grib2.html) • Typically software package is set of functions compiled into a library • User’s application sets up required parameters and calls function from library, which are referenced at run time • Specific (not generic) encoders and decoders are quite simple but less flexible WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Some GRIB2 data from EUMETSAT • Cloud mask, CLM, every 15 minutes, ~3km resolution, via EUMETCast or archive request. • Cloud top height, CTH, every 60 minutes, ~9km resolution, via EUMETCast or archive request. • Cloud analysis image, CLAI, every 180 minutes, ~9km resolution, via EUMETCast or archive request. • Clear sky reflectance map, CRM, twice per week, ~3km resolution, via EUMETCast or archive request. • Multi-sensor precipitation estimate, MPE, every 30 minutes, ~5km resolution, via web site. • Fire detection, FIR, every 15 minutes, ~3km resolution, via anonymous FTP (evolving algorithm). WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Case study A, precipitation data WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Case study A, precipitation data • GRIB2 encoded file • Specific decoder program for GRIB2 precipitation data • Decoder output • Dump of GRIB2 file WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Case study B, cloud mask data WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Case study B, cloud mask data • Original data to be encoded • GRIB2 encoded file • Specific en/decoder program for EUMETSAT cloud mask data • Decoder output • Dump of GRIB2 file WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Case study C, cloud analysis image data GRIB2 Cloud analysis image WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Case study C, cloud analysis image data • GRIB2 encoded file • Specific en/decoder program for EUMETSAT cloud mask data • Bit map used WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Case study D, cloud top height data GRIB2 Cloud top height WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Case study D, cloud top height data • GRIB2 encoded file • Specific en/decoder program for EUMETSAT cloud mask data • Bit map used • Two repeats of Sections 4, 5, 6 and 7 WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005 Sources of GRIB2 information The information used in preparing this presentation is based on: Guide to WMO Table Driven Code Form Used for the Representation and Exchange of Regularly Spaced Data in Binary Form: FM 92-XII GRIB Written by Dr Cliff Dey & WMO Manual on Codes, WMO Publication No. 306, Vol. 1, Part B WORLD METEOROLOGICAL ORGANIZATION RA-II/VI Regional Training on Table Driven Codes Muscat, Oman, 10 - 14 December, 2005