Bias correction of satellite data
at ECMWF
Thomas Auligne
Dick Dee, Graeme Kelly, Tony McNally
ITSC XV - October 2006
Slide 1
Motivation for an adaptive system
z Simplify the bias correction process of manual tuning / retuning
z Automatically handle:
-
Instrument problem / contamination
-
New version of RT Model
-
Appearance of new instruments
z Reanalysis issue: remove
inconsistencies due to
changes in the observing system
z Large increase in the number of
satellite data (currently 29 instruments,
~500 channels, ~3000 bias parameters)
+6 000 000 data used daily
Prone to wrongly mapping systematic errors of
the NWP model into radiance bias correction
ITSC XV - October 2006
Slide 2
Variational bias correction
Bias for each satellite/sensor/channel:
b( β , x) = ∑ β i p i
Predictors:
• constant offset
• scan
• air-mass
i
Add the bias parameters βi to the control vector in the variational analysis
Æ joint estimation of bias and model state (Derber and Wu 1998) (Dee 2005)
Jb: background constraint for x
Jo: observation constraint
J(x) = (xb − x)T B x−1 (xb − x) + [y − h(x)] R −1 [y − h(x)]
T
Jb: background constraint for x
Jo: bias-corrected observation constraint
J(x, β) = (x b − x)T B x−1 (xb − x) + [y − b(x, β) − h(x)] R −1 [y − b(x, β) − h(x)]
T
+ (β b − β)T B β−1 (β b − β)
Jβ: background constraint for β
Find optimal bias correction given all available information
ITSC XV - October 2006
Slide 3
NOAA-9 MSU Ch3 disruption (cosmic storm)
NOAA-9
MSU Ch 3
Tropics
200 hPa RS
temperature
Tropics
ITSC XV - October 2006
Slide 4
Performance of the VarBC
reduction of bias wrt RS temperature data
VarBC
NOAA-16 AMSU-A Ch 10 NH
NH
SH
50 hPa RS temperature NH
ITSC XV - October 2006
Slide 5
Control
ERA Interim experimentation
Stratospheric model bias
NOAA-10 HIRS-3
Observation
S. Hemisphere
NOAA-10 MSU-4
Observation
Model bias
wrongly attributed
to observations
NOAA-10 MSU-4
Departures
&
Bias Correction
ITSC XV - October 2006
20 K
1K
Slide 6
Conclusion on VarBC
z Automation = big practical advantage
z Ability to handle sudden instrument shifts and slow
drifts
z New sensors can be integrated easily
(reasonable bias within 1-7 days)
z Consistency within the observing system
(better fit to RS temperatures)
z Ability to (partially) discriminate between observation bias
and systematic NWP model error relies on:
- availability of unbiased data source (anchoring network)
- observational coverage
- parametric form
ITSC XV - October 2006
Slide 7
Parametric form
to represent observation bias
ITSC XV - October 2006
Slide 8
Definitions
It is essential to distinguish…
PARAMETRIC FORM = the predictors
chosen to characterize the bias
(e.g. constant offset, NWP model preds, gamma, …)
ADAPTIVITY = how the bias coefficients are
updated:
VarBC
VarBC
Static
Static
ITSC XV - October 2006
vs
Adaptive
Adaptive
Slide 9
Offline
Offline
Operational parametric form
C
I
T
A
T
S
E
V
I
z Scan correction: 3 order polynomial of Scan Angle
T
P
AD A
z Air-mass regression
E
V
I
T
Linear regression with a limited set of predictors P derived from the NWP model
P
ADA
z γ correction to the RT model: γ = fractional error in layer absorption coefficient
rd
i
Instruments
# of
Predictors
preds
HIRS, AMSU-A, AMSU-B,
4
AIRS
1000-300, 200-50, 10-1, 50-5 hPa
thicknesses
GEOS (GOES, Meteosat)
3
1000-300, 200-50 hPa, TCWV
SSMI
3
Tskin, TCWV, Surface Wind Speed
ITSC XV - October 2006
Slide 10
Estimation of the γ coefficient in VarBC
AMSU-B
AMSU-A
(γ -1)*100
Ch
Ch13
13&&14
14
AIRS
Window
Windowchannels
channels
Analysis cycle
HIRS
ITSC XV - October 2006
Tropospheric
Tropospheric
Water
Vapour
Slide 11
Water
Vapour
Relevant bias predictors
Property 1 = help reduce the
first-guess departures
ΔTb (K)
0
.
.
.. .
.
.
. . . . ..
. . .
Latitude (or scan, …)
Uncorrected departures
ITSC XV - October 2006
STD
.
.
.
.
.
. . . . . . . . .. .
Latitude (or scan, …)
Bias-corrected departures
Slide 12
Relevant bias predictors
Property 1 = help reduce the
first-guess departures
z Compute the variance explained for each potential predictor: not very convenient
z The predictors are normalized (mean=0, std=1). The parameter values from VarBC can be
compared to discard “useless” predictors
z A “compensation” effect can happen b/w predictors that are correlated
Jb: background constraint for x
Weight decay
regularization
Jo: bias-corrected observation constraint
J(x, β) = (x b − x)T B −x1 (xb − x) + [y − b(x, β) − h(x)] R −1 [y − b(x, β) − h(x)]
T
+ (β b − β)T B β−1 (β b − β) + β T (ν .I ) β
Jβ: background constraint for β
ITSC XV - October 2006
Weight decay constraint for β
Slide 13
Relevant bias predictors
Property 1 = help reduce the
first-guess departures
Diagnostic 1 = absolute value of
(normalized) parameters
ITSC XV - October 2006
Slide 14
Relevant bias predictors
Property 1 = help reduce the
first-guess departures
Diagnostic 1 = absolute value of
(normalized) parameters
Property 2 = focus on observation bias
(and not systematic NWP model error)
ITSC XV - October 2006
Slide 15
Relevant bias predictors
Property 1 = help reduce the
first-guess departures
Diagnostic 1 = absolute value of
(normalized) parameters
Property 2 = focus on observation bias
(and not systematic NWP model error)
z VarBC is constrained by all other observation sources (e.g. RS)
z Offline adaptive BC tries to fully correct signal in the departures
z A parametric form only explaining for observation bias only
should be updated identically in both schemes
ITSC XV - October 2006
Slide 16
Relevant bias predictors
Property 1 = help reduce the
first-guess departures
Diagnostic 1 = absolute value of
(normalized) parameters
Property 2 = focus on observation bias
(and not systematic NWP model error)
Diagnostic 2 = (dis)agreement b/w
VarBC and Offline Adaptive BC
ITSC XV - October 2006
Slide 17
AMSU-B
AMSU-A
Stratosphere
Stratosphere
Mesosphere
Mesosphere
HIRS
Water
Water
Vapour
Vapour
.
ITSC XV - October 2006
Gamma value
Gamma difference
Slide 18
(VarBC/Offline)
. LW Temperature
. SW Temperature
. Water Vapour
AIRS
AIRS
Abs(γ-1) in %
Jacobian peak
pressure (hPa)
Δγ VarBC/Offline in %
ITSC XV - October 2006
Slide 19
Conclusion & future work
z VarBC operational at ECMWF since September 12th 2006 and in
ERA-Interim reanalysis
z Works well in many respects. Needs close attention to:
-
NWP model error mapping (e.g. stratosphere)
-
feedback process with Quality Control & Cloud Detection
(e.g. window channels)
z Enables diagnostics to evaluate bias predictor relevance
z These can be used in an objective method to select predictors
ITSC XV - October 2006
Slide 20
END
Thank you…
ITSC XV - October 2006
Slide 21
Introduction of the VarBC in operations:
first step
Feb 2006: implementation of a static bias correction derived from a VarBC
experiment
STD background departures
STD analysis departures
Mean background departures
Mean analysis departures
Static bias corrections
(offset = 0.99)
NOAA-18 AMSU-A Ch5 Tropics
Cy30r1
Coefficients from VarBC
ITSC XV - October 2006
Slide 22
Introduction of the VarBC in operations:
first step
100 hPa RS
temperature
Tropics
500 hPa RS
temperature
Tropics
ITSC XV - October 2006
Slide 23
AIRS operational bias predictors
1000-300 hPa
200-50 hPa
10-1 hPa
50-5 hPa
Jacobian peak
pressure
Diagnostic 1
Abs(Param)
. LW Temperature
. SW Temperature
Diagnostic 2
VarBC/Offline
disagreement
ITSC XV - October 2006
Slide 24
. Water Vapour
Weight decay regularization
AIRS
Bias correction variance
Variance
explained
ITSC XV - October 2006
Slide 25
Relevant bias predictors
ITSC XV - October 2006
Slide 26
ITSC XV - October 2006
Slide 27