VARSY progress meeting
Robin Hogan and Nicola Pounder
(University of Reading)
12 April 2013
Brief summary of progress
No plots today:
• Full error descriptors now implemented for liquid clouds and rain (ice
already done)
• Solar radiance forward model: code included to describe scattering
phase function with Legendre polynomials but still needs to be coupled
to the LIDORT radiative transfer model
Plots today:
• Liquid cloud retrievals using multiple scattering from single field-ofview lidar Calipso
• Overcoming multiple minima in the cost function for liquid cloud
• Possible algorithm speed-up being investigated: Levenberg-Marquardt
minimization rather than quasi-Newton, plus GPU computation of
Jacobian matrix
• Ability to simulate EarthCARE data (including Doppler and HSRL) from
A-Train retrievals, then retrieve from the simulated EarthCARE data
1. New ray of data: define state vector
Use classification to specify variables describing each species at each gate
Ice: extinction coefficient, N0’, lidar extinction-to-backscatter ratio, riming factor
Liquid: extinction coefficient and number concentration
Rain: rain rate, drop diameter and melting ice
Aerosol: extinction coefficient, particle size and lidar ratio
Unified
retrieval
Ingredients developed
Not yet developed
2. Convert state vector to radar-lidar resolution
Often the state vector will contain a low resolution description of the profile
3. Forward model
3a. Radar model
With surface return and
multiple scattering
3b. Lidar model
Including HSRL channels
and multiple scattering
4. Compare to observations
Check for convergence
3c. Radiance model
Solar & IR channels
6. Iteration method
Derive a new state vector
Not converged
Converged
7. Calculate retrieval error
Error covariances & averaging kernel
Proceed to next ray of data
Liquid cloud retrieval
• We have found that the multiple scattering signal from Calipso can be
inverted to get extinction profile for optical depth up to at least 30
• Benefits from a constraint on LWC to be no steeper than adiabatic
• We can validate with CloudSat PIA, or assimilate PIA too
• Example from 1 minute (~400 km) of oceanic stratocumulus:
• Forward
modelled
backscatter
• Observed
backscatter
Assimilate only Calipso backscatter
• LWC
• Effective
radius
• Optical
depth
• CloudSat
PIA
Assimilate also CloudSat PIA
• LWC
• Effective
radius
• Optical
depth
• CloudSat
PIA
Will this work with EarthCARE?
FOV <= 50 m (e.g. EarthCARE)
• Simulated retrieval of
optical depth for idealized
adiabatic clouds, using
spaceborne lidar with
varying field of view (FOV)
• For FOV less than around
50 m, there is simply too
little multiple scattering
signal to retrieve extinction
and optical depth
• Will need to rely more on
radar PIA over ocean and
solar radiances in the day
• Night-time land a problem
Why can the first guess matter?
First guess
Truth
• Consider a cloud with an
optical depth of 50
• If the first guess had an
optical depth of 1 then the
simulated molecular
scattering below the cloud
would look a bit like the
measured multiple
scattering
• Algorithm has difficulty
getting over hump in cost
function because
increasing optical depth
first reduces simulated
backscatter below cloud
top (leading to poorer
agreement with obs)
before multiple scattering
builds up (leading to
better agreement)
Possible solution
• Consider all possible true
optical depths (but only
triangular profiles so that
profiles can be described
uniquely by optical depth)
• Algorithm will converge
provided first guess is
outside the shaded areas
• Should be able to preanalyse the profile (e.g. by
integrating the backscatter
with height) to tell if we are
in the low or high optical
depth regime, then set the
first guess appropriately
Previous plot
considered true
optical depth of 50
Potential optimization
• We need to speed-up the retrieval algorithm
– Can we exploit parallel architectures, e.g. multicore machines or GPUs?
• Trade-off between minimization schemes:
– Quasi-Newton (L-BFGS)
• Uses only the gradient of the cost function, which is fast to calculate
• Many iterations required
– Levenberg-Marquardt (LM; more stable version of Gauss-Newton)
• Uses also the curvature of the cost function which is slow to calculate
• But few iterations required, and a little more robust (in my experience)
• Currently works for ice and rain, not yet for liquid
• Adept’s algorithm for computing the Jacobian matrix (needed by LM)
is potentially parallelizable
– “m” parallel threads, where “m” is number of observations (~100)
– At best, the cost of an LM iteration would be the same as a quasiNewton iteration, so LM would be much faster overall
• I am currently employing a programmer with GPU experience to code
up a parallel Jacobian algorithm using CUDA (for NVIDIA hardware)
Example case
• LevenbergMarquardt
algorithm
run on ice
and rain
region
• CloudSat
and Calipso
observations
and forward
model
Convergence
comparison
• Quasi-Newton needs around five
times more iterations on average
(depending on convergence
criterion)
Levenberg-Marquardt
Quasi-Newton
Convergence comparison cont.
Levenberg-Marquardt
CloudSat
Calipso
Quasi-Newton
Computational cost
Proportional to number of iterations
Levenberg-Marquardt
Multiple
scattering
attering
forward
modelforward model
L-BFGS
Automatic adjoint
Automatic adjoint
Automatic Jacobian
Automatic Jacobian
0
2
4
6
8
10
12
14
Computational cost (arbitrary)
Potentially parallelizable
Total
• Levenberg-Marquardt is already competitive but if Jacobian can be
sped up it would be much faster than qausi-Newton
al after parallelization?
• Further change: perform wide-angle multiple scattering at half the
vertical resolution would gain factor 4 speed-up
0
2
4
6
8
10
12
14
16
18
20
• CloudSat
Unified retrieval of cloud +precip
…then simulate EarthCARE instruments
• EarthCARE
CPR Z
– Higher
sensitivity
• CPR Z error
• CPR Doppler
– Use
Japanese
random
error
• CPR Doppler
error
• Calipso
backscatter
Unified retrieval of cloud +precip
Liquid cloud
…then simulate EarthCARE instruments
• ATLID Mie
channel
– Note
liquid!
• ATLID Mie
error
– Not
rigorous!
• ATLID
Rayleigh
channel
• ATLID
Rayleigh
error
Compare ice retrievals
Extinction
• A-Train
retrieval
Number
concentration
Extinction
Number
concentration
• PseudoEarthCARE
retrieval
• Assimilate
Doppler
and HSRL
(Some difference
due to lidar ratio
not being carried
between retrieval
and simulation)
Compare liquid clouds and rain
Liquid water content
• A-Train
Rain rate
Liquid water content
Rain rate
• EarthCARE
– Poor LWC:
not enough
lidar
multiple
scattering!
Remaining algorithm development
•
•
•
•
•
•
Minimization
– Parallelize Jacobian calculation on GPU and compare speed of LevenbergMarquardt to quasi-Newton
Forward models
– Finish implementation of LIDORT solar radiance model
Ice clouds
– Add “riming” factor
– Add Baran phase functions where appropriate
Liquid clouds
– Test impact of solar radiances on retrievals
– Test size retrieval from two solar wavelengths
Rain
– Test impact of various observations (PIA, radar multiple scattering)
Aerosols
– Implement an aerosol retrieval scheme (contract extension)