Observations
Preprocessing
Carla Cardinali
Training course 2005
slide 1
ECMWF
Observations preprocessing
MakeCMA
BUFR
ECMA
ODB
Screening
CCMA
Minimization
Forecast
MatchUp
ECMA
FeedBack
Training course 2005
slide 2
BUFR
ECMWF
Observations preprocessing
ECMA/ODB
CCMA/ODB
Output BUFRs
Training course 2005
slide 3
ECMWF
Observations preprocessing
 What is ODB?
Developed software at ECMWF to manage large amounts of satellite data.
ODB/SQL language is a subset of ANSI/SQL query language. Compiler
translates ODB/SQL in C-codes files
 How do use it?
Access to ODB database is through Fortran90 modules functions. A ECMA data
base contains table with column entries lat, lon, date, time observation type
which is used to locate observation at a given moment
Obs. Ident ( sat. id, lat, lon, st.alt., date, time)
Obs. V. (wind, temp.,…per pressure) and (radiances per ch., inst. type)
Various flag: active, blacklisted….
Departures (obs-background, obs-analysis)
Training course 2005
slide 4
ECMWF
Observations preprocessing
 Incoming Observation
The observations arrive at ECMWF through GTS and they are stored in a
decoded format in Report Data Base after some rudimentary quality control e.g.
observation format and position, climatological and hydrostatic limit and
temporal consistency. An observation file suitable for assimilation is created
6-hour Preprocessing Observation Array
Format conversions, change of some observed variables (relative humidity
from dry and wet bulb temperature) and assignment of observation error
statistics
Training course 2005
slide 5
ECMWF
Observations Screening
 Select the “best” observation
At the first trajectory run the model counterparts for all the observations are
calculated through the nonlinear observation operators. For the observation
screening, the background errors are interpolated to the locations of the
observed variables. The "extended" observation array (ECMA) contains
observations complemented by the background departures and quality control
information for most of the observations.
This array is stored for later feedback.
After the screening a "compressed"
array is passed to the minimization
(CCMA).
Training course 2005
slide 6
ECMWF
Independent Observations Screening
 The screening logic is to make first those decisions that are not
depending on any other ones
Preliminary checks: completeness of the report
Blacklist
Data Selection: which observation types, variables, vertical ranges will be used
in the assimilation.
Monthly selection: discarding stations that have been reporting excessively
noisy or biased compare to the background field.
Impact studies.
Training course 2005
slide 7
ECMWF
Independent Screening Decision: Conventional
Observations
Background Quality Control
The BgQC is applied to all the variables that are intended to be used in the
assimilation
y  Hx b 2
 3 


  2   b   o
2
2
 1
 


BgQC of wind observations is done simultaneously for both wind
components. For wind direction the error limits of 60, 90 and 120 apply for
flags 1,2 and 3
uo
ub
Training course 2005
slide 8
ECMWF
Independent Screening Decision: Conventional
Observations
Training course 2005
slide 9
ECMWF
When is not working
Training course 2005
slide 10
ECMWF
Independent Screening Decision: Satellite
Observation
Bias Correction
Bias correction coefficients are recomputed from the past 2/4 weeks of
departure statistics. The feedback files are used for monitoring the
performance of the observing and assimilation system: e.g. ATOVS and SSMI
radiances, SCATT wind.
If the removed bias is a model forecast bias, the subsequent assimilation
will enforce it. Usually, only half bias is removed.
Gross check
Measured and background brightness temperatures are present for all required
channels. Bias correction coefficients,
satellite id, and scan position are all
230K
valid before proceeding. For all channels
a cloud detection is performed
290K
Training course 2005
slide 11
ECMWF
Dependent Screening Decision: Satellite Observation
SSMI
HIRS
3
2
1
land +ocean
Channels below
rejected
Training course 2005
slide 12
ocean: LWP=f(ch3/4)
If LWP >  cloud+rain
ECMWF
Dependent Screening Decision: Satellite Observation
AMSU-A
Tb
6
no cloud
5
cloud
rain
4
3
1
2
1
3

ocean: LWP=f(ch1,ch2,ch3)
If LWP >  cloud+rain
land: if (ch1-ch3) >  rain  because of large emissivity over
land no cloud contamination detection
Training course 2005
slide 13
ECMWF
Independent Screening Decision: Satellite
Observation
Background Quality control: ATOVS, SSMI, METEOSAT
Background temperature, specific humidity and ozone profiles are checked to
make sure they are close to or within the range for which the radiative transfer
model is valid. Temperature is within the range 150-350 K, specific humidity is
positive and not supersaturated and the ozone is within climate extremes.
Radiance at the 2 extreme edge positions of the swath are not used in
4D-Var
Rad O  Rad b 2  HBHT  O
Training course 2005
slide 14
ECMWF
Independent Screening Decision: Satellite
Observation
Training course 2005
slide 15
ECMWF
Independent Screening Decision: Satellite
Observation
Training course 2005
slide 16
ECMWF
Dependent Screening Decision: Conventional
Observation
Before performing the dependent screening decisions, the flag information
gathered so far is converted into a report status, namely active, passive,
rejected or blacklisted i.e the RDB datum flag.
Vertical consistency of multilevel reports
Duplicated levels are removed. If some consecutive layers are of suspicious
quality they are rejected and for geopotential obs. also the layer above
Removal of duplicated report
Search globally for duplicated (AIREP) data. Usually, there are aircraft reports
that have the same date/time, roughly the same location and, at least partially,
the same data. Usually, the station ID is slightly different.
Training course 2005
slide 17
ECMWF
Dependent Screening Decision: Conventional
Observation
Redundancy check
Removes redundant SYNOP/PAOB :
co-located reports with the same station ID are searched for and only the
closest active report to the analysis time is retained. For simultaneous
reports, the one with more active data is retained. The same is done for
reports that have equal time difference to the analysis time. Co-located
SYNOP (mass obs) is redundant to TEMP (geop.) that is within 50 hPa
Removes redundant SHIP/DRIBU:
moving platforms within a circle of 1 ° are considered as potentially
redundant: reports closest to the centre of the screening time with most
active date are retained.
Training course 2005
slide 18
ECMWF
Dependent Screening Decision: SYNOP
4D-Var Screening for 4D-Var
The effect of the observation screening
on SYNOP surface pressure observations.
Column height gives the number of
observations available, while the shaded
part displays those actually used in the
assimilation.
3D-Var Screening for 3D/4D-Var
Training course 2005
slide 19
ECMWF
Dependent Screening Decision: Conventional
Observation
Level selection for TEMP/PILOT redundancies:
for one time window rejection in layers around std-levels according the
following priorities
 maximum datum flag
 time difference to analysis time
distance to the std-level
significant level (turning points of the sounding)
temp over pilot
Training course 2005
slide 20
ECMWF
Dependent Screening Decision: Thinning
AIREP thinning
horizontal thinning
from the same platform, a minimum distance between the nearby reports
is enforced:
Box
125
62 km
vertical thinning
is performed around the model vertical levels: one aircraft measurement
per model level.
Training course 2005
slide 21
ECMWF
Dependent Screening Decision: Thinning
+
VERT increased number of AIREP
from 15 to 60 vertical levels
+HOR also increased number
of AIREP in horizontal
Training course 2005
slide 22
ECMWF
Screening Decision: Thinning
TEMP
GTS
Obs
thin
Training course 2005
slide 23
ECMWF
Dependent Screening Decision: Thinning
ATOVS thinning: a repeated scan is performed to get the observation
resolution of 70 km (it depends on sensors/channels)
a sea sounding is preferred to a land one
clear sounding to a cloudy one
closeness of observation time to the centre of the screening
time window
A second thinning takes place that selects one observation every
140km
Training course 2005
slide 24
ECMWF
Dependent Screening Decision: Thinning
The usage of ATOVS reports in the
assimilation on the North Eastern
Atlantic. Filled rings mark reports
contain one or more channels used in
the assimilation, whereas the empty
rings denote rejected reports. Most of
the rejections are due to the horizontal
thinning and much less due to the
quality reasons.Note that both edges of
the swath are rejected.
Training course 2005
slide 25
ECMWF
Dependent Screening Decision: Thinning
SCAT:ERS-2
The process is defined with respect to the particular measurement geometry of
the instrument. The backscatter data are acquired within individual cells related
to a 450 km wide grid with a mesh of 25 km in the across and along track
directions. 19 measurement nodes are thus defined across the scatterometer´s
swath, numbered from 1 to 19 as the incidence angle increases, while 19 rows
are also considered in the along track direction to gather the data in squares of
19 by 19 points. The thinning is then achieved by keeping only every fourth
point within these squares. The data are thus used at a resolution of 100 km
instead of the original 25 km sampling distance.
Sea-ice contamination < 273K
High wind rejection test > 25m/s
Normalized distance to the cone or wind residual. During the
assimilation this quantity is computed and data rejected if a large value is
found.
Training course 2005
slide 26
ECMWF
Dependent Screening Decision: Thinning
1-step
..   
..   
..   
..   
……....
    ..
 ..

    ..
.. 3 .. .. .. 7
2-step
Triplet obs
Training course 2005
slide 27
check the distance
between observations
ECMWF
QuikSCAT
50 km
Quadruplet obs
Hans Hersbach
Lars Isaksen Mark Leidner
Courtesy of
Training course 2005
slide 28
ECMWF
Dependent Screening Decision: Thinning
MLE (u)   ( OBS i   i MOD (u)) 2
i 1
16
High resolution trajectory
to choose only one
Training course 2005
slide 29
ECMWF
Dependent Screening Decision: Thinning
QSCAT 50km ($6dr): data cov$rag$
Ambig. clos$st to ana, from 2001 0518 0412 to 2001 0518 0733
Obs$rv$d winds abov$ 1. m/s
180°
170°W
50°N
50°N
Rej: only 1 beam measur.
40°N
40°N
active
Minima are not well defined
Var-QC
30°N
30°N
White areas: insuf. info for MLE
inversion e.g rain contamination
180°
Training course 2005
170°W
slide 30
ECMWF
Screening Statistics
Obs. Type
Synop
Airep
Satob
Dribu
Temp
Pilot
Satem
Paob
Scat
Training course 2005
Reports
Active
28151
52868
217018
4802
605
805
249279
214
0
24218
29969
40500
1708
602
730
55863
138
0
slide 31
Passive
Rejected
3933
22899
176518
3094
3
75
193416
76
0
Blacklisted
0
0
100699
0
0
7
154867
76
0
ECMWF
After Screening
Compression of CMA-file
After screening only 15% of all observations are active
 10-20% TOVS left
 40%
Conventional left
The observation are resorted among the processors for a more
optimal load balancing of the parallel computer
Training course 2005
slide 32
ECMWF
After Screening
Parallel computing environment
The observation screening should result in exactly the same selection of
observations when different number of processors are used. Independent
decision can be made at different processor fully in parallel. But global view of
observation array is needed when a dependent decision has to be taken which
implies that some communication between the processors is required. The
observation array is too large to be copied in each individual processor then
only a minimum necessary information is globally communicated
Global Array
Location
1
ID
2
Time
Training course 2005
slide 33
ECMWF