Comparing the formulations of CCAM and
VCAM and aspects of their performance
John McGregor
CSIRO Marine and Atmospheric Research
Aspendale, Melbourne
PDEs on the Sphere
Cambridge
28 September 2012
CSIRO Marine and Atmospheric Research
1
Outline
• CCAM formulation
• VCAM formulation
• Some comparisons
CSIRO Marine and Atmospheric Research
2
Alternative cubic grids
Original
Sadourny (1972)
C20 grid
Equi-angular
C20 grid
Conformal-cubic
C20 grid
CSIRO Marine and Atmospheric Research
3
The conformal-cubic
atmospheric model
• CCAM is formulated on the
conformal-cubic grid
• Orthogonal
• Isotropic
Example of quasi-uniform C48 grid with resolution about 200 km
CSIRO Marine and Atmospheric Research
4
CCAM dynamics
•
•
•
•
•
atmospheric GCM with variable resolution (using the Schmidt
transformation)
2-time level semi-Lagrangian, semi-implicit
total-variation-diminishing vertical advection
reversible staggering
- produces good dispersion properties
a posteriori conservation of mass and moisture
CCAM physics
•
•
•
•
•
•
cumulus convection:
- mass-flux scheme, including downdrafts, entrainment,
detrainment
- up to 3 simultaneous plumes permitted
includes advection of liquid and ice cloud-water
- used to derive the interactive cloud distributions (Rotstayn 1997)
stability-dependent boundary layer with non-local vertical mixing
vegetation/canopy scheme (Kowalczyk et al. TR32 1994)
- 6 layers for soil temperatures
- 6 layers for soil moisture (Richard's equation)
enhanced vertical mixing of cloudy air
GFDL parameterization for long and short wave radiation
CSIRO Marine and Atmospheric Research
5
Location of variables in grid cells
All variables are located at
the centres of quadrilateral
grid cells.
However, during
semi-implicit/gravity-wave
calculations, u and v are
transformed reversibly to the
indicated C-grid locations.
Produces same excellent
dispersion properties as
spectral method (see
McGregor, MWR, 2006), but
avoids any problems of
Gibbs’ phenomena.
2-grid waves preserved.
Gives relatively lively winds,
and good wind spectra.
CSIRO Marine and Atmospheric Research
6
Reversible staggering
Where U is the unstaggered velocity component and u is the staggered
value, define (Vandermonde formula)
•
•
•
accurate at the pivot points for up to 4th order polynomials
solved iteratively, or by cyclic tridiagonal solver
excellent dispersion properties for gravity waves, as shown for the
linearized shallow-water equations
CSIRO Marine and Atmospheric Research
7
Dispersion behaviour for linearized shallow-water equations
Typical atmosphere case
Typical ocean case
- large radius deformation
- small radius deformation
N.B. the asymmetry of the R grid response disappears by alternating the reversing direction each time step,
giving the same response as Z (vorticity/divergence) grid
CSIRO Marine and Atmospheric Research
8
Transformation of 2, 3, 4, 6-grid waves
CSIRO Marine and Atmospheric Research
Treatment of pressure-gradient terms
The reversible staggering technique allows a very
consistent, and thus more accurate, calculation of
pressure gradient terms.
For example, in the staggered u equation
the RHS pressure gradient term is first evaluated at the
staggered position, then transformed to the unstaggered
position for calculation of the whole RHS advected value
on the unstaggered grid. That whole term is then
transformed to the staggered grid, fully consistent with
the subsequent implicit evaluation of the LHS on the
staggered grid.
CSIRO Marine and Atmospheric Research
Treatment of ps advection near terrain
Pressure advection equation
Define an associated variable, similar to MSLP
which varies smoothly, even over terrain. It is thus suitable
for evaluation by bi-cubic interpolation, whilst the other term
is found “exactly” by bi-linear interpolation (to avoid any
overshooting effects). Formally, get
CSIRO Marine and Atmospheric Research
Treatment of T advection near terrain
Similarly to surface pressure advection, define an
associated variable
which varies relatively smoothly on sigma surfaces over
terrain. Again the second term can be found “exactly” by
bi-linear interpolation. A suitable function is
Formally, get
This technique effectively avoids the requirement for hybrid
coordinates.
CSIRO Marine and Atmospheric Research
a posteriori conservation
•
•
•
•
a posteriori conservation of mass and moisture
“global” scheme
simultaneously ensures non-negative values
during each time step applies correction to changes
occurring during dynamics (including advection)
• correction is proportional to the “dynamics” increment,
but the sign of the correction depends on the sign of
the increment at each grid point.
The above are all described in the CCAM Tech. Report
CSIRO Marine and Atmospheric Research
MPI implementation
Remapped region 0
Original
Remapping of off-processor neighbour indices to buffer region
Indirect addressing is used extensively in CCAM
- simplifies coding
CSIRO Marine and Atmospheric Research 14
Typical MPI performance


Showing both Face-Centred (FC) and Uniform Decomposition (UD)
for global C192 50 km runs, for 1, 6, 12, 24, 48, 72, 96, 144, 192, 288
CPUs
VCAM a little slower, but is still to be fully optimised
CSIRO Marine and Atmospheric Research 15
An AMIP run 1979-1995
DJF
JJA
Obs
CCAM
Tuning/selecting physics options:
• In CCAM, usually done with 200 km AMIP runs, especially paying
attention to Australian monsoon, Asian monsoon, Amazon region
• No special tuning for stretched runs
CSIRO Marine and Atmospheric Research 16
Variable-resolution conformal-cubic grid
The C-C grid is rotated to locate panel 1 over the region of interest
The Schmidt (1975) transformation is applied
• this is a pole-symmetric dilatation, calculated using spherical polar
coordinates centred on panel 1
• it preserves the orthogonality and isotropy of the grid
• same primitive equations, but with modified values of map factor
Plot shows a C48 grid (Schmidt factor = 0.3) with resolution about 60
km over Australia
CSIRO Marine and Atmospheric Research 17
Schmidt transformation can be used to obtain even finer resolution
Grid configurations used to support Alinghi in America’s Cup, Olympic sailing
C48 8 km grid over New Zealand
C48 1 km grid over New Zealand
CSIRO Marine and Atmospheric Research 18
Preferred CCAM
downscaling methodology
•
•
•
•
Coupled GCMs have coarse resolution, but also
possess Sea Surface Temperature (SST) biases
A common bias is the equatorial “cold tongue”
First run a quasi-uniform 200 km (or modestly
stretched) CCAM run driven by the bias-corrected
SSTs
The 200 km run is then downscaled to 20 km (say) by running CCAM with a stretched grid, but
applying a digital filter every 6 h to preserve large-scale patterns of the 200 km run
Quasi-uniform C48 CCAM grid with resolution about 200 km
Stretched C48 grid with resolution about
20 km over eastern Australia
CSIRO Marine and Atmospheric Research 19
Digital-filter downscaling method
•
•
•
•
Uses a sequence of 1D passes over all panels to efficiently evaluate
broad-scale digitally-filtered host-model fields (Thatcher and
McGregor, MWR, 2009). Very similar results to 2D collocation
method.
These periodically (e.g. 6-hourly or 12-hourly) replace the
corresponding broad-scale CCAM fields
Gaussian filter typically uses a length-scale approximately the width of
finest panel
Suitable for both NWP and regional climate
CSIRO Marine and Atmospheric Research 20
Nonhydrostatic treatment
Being a semi-Lagrangian model, CCAM is able to absorb the extra
phi terms into its Helmholtz equation solver, for “zero” cost
The new dynamical core (VCAM) uses a split-explicit treatment, so
the Miller-White treatment would need its own Helmholtz solver,
so may use Laprise-style nonhydrostatic treatment for VCAM
CSIRO Marine and Atmospheric Research
CCAM simulations of cold bubble, 500 m L35 resolution, on highly stretched global grid
CSIRO Marine and Atmospheric Research
Gnomonic grid showing orientation of the
contravariant wind components
Illustrates the
excellent
suitability of the
gnomonic grid for
reversible
interpolation –
thanks to smooth
changes of
orientation
CSIRO Marine and Atmospheric Research 23
Nonhydrostatic treatment
Being a semi-Lagrangian model, CCAM is able to absorb the extra
phi terms into its Helmholtz equation solver, for “zero” cost
The new dynamical core of VCAM uses a split-explicit treatment, so
the Miller-White treatment would need its own Helmholtz solver,
Probably will use Laprise-style nonhydrostatic treatment for VCAM
CSIRO Marine and Atmospheric Research 24
New dynamical core for VCAM
- Variable Cubic Atmospheric Model
• uses equi-angular gnomonic-cubic grid
- provides extremely uniform resolution
- less issues for resolution-dependent parameterizations
• reversible staggering transforms the contravariant winds to
the edge positions needed for calculating divergence and
gravity-wave terms
• flux-conserving form of equations
– preferable for trace gas studies
– TVD advection can preserve sharp gradients
– forward-backward solver for gravity waves
– avoids need for Helmholtz solver
– linearizing assumptions avoided in gravity-wave terms
CSIRO Marine and Atmospheric Research
CSIRO Marine and Atmospheric Research 26
Horizontal advection
Low-order and highorder fluxes combined
using Superbee limiter
Flow=qyVj+1/2
High order need
covariant vels for LW
term. Linear interp for
edge values of q?
Cartesian components
(U,V,W) of horizontal
wind are advected
vcov
(qx, qy)
q
Ui-1/2
ucov
Flow=qxUi+1/2
Vj-1/2
Transverse components (to be included in low/high order fluxes)
calculated at the centre of the grid cells (loosely following LeVeque)
qx: using dt/2 advection from vcov
qy: using dt/2 advection from ucov
CSIRO Marine and Atmospheric Research 27
Solution procedure
•
•
•
•
Start t loop
Start Nx(Dt/N) forward-backward loop
Stagger (u, v) t+n(Dt/N)
Average ps to (psu, psv) t+n(Dt/N)
Calc (div, sdot, omega) t+n(Dt/N)
Calc (ps, T) t+(n+1)(Dt/N)
Calc phi and staggered pressure gradient terms, then unstagger these
Including Coriolis terms, calc unstaggered (u, v) t+(n+1)(Dt/N)
End Nx(Dt/N) loop
Perform TVD advection (of T, qg, Cartesian_wind_components) using
average ps*u, ps*v, sdot from the N substeps
Calculate physics contributions
End t loop
Main MPI overhead is the reversible staggering at each substep, but this just
needs nearest neighbours in its iterative tridiagonal solver. Also message
passing is needed in the pressure gradient and divergence calcs
CSIRO Marine and Atmospheric Research 28
500 hPa omega (Jan 1979)
Hybrid
coordinates
introduced
non-hybrid
CSIRO Marine and Atmospheric Research
CCAM
VCAM
250 hPa
winds
in 1-year
run
CSIRO Marine and Atmospheric Research 30
DJF
Same physics
JJA
Obs
climate
VCAM
1-year
CCAM
1-year
CSIRO Marine and Atmospheric Research
31
However, can can see some influence of panel edges on
rainfall just south of Australia
CSIRO Marine and Atmospheric Research 32
Eastwards solid
body rotation in
900 time steps
Using superbee
limiter
Problem caused
by spurious
vertical
velocities at
vertices!
CSIRO Marine and Atmospheric Research 33
Spurious vertical
velocities reduced by
factor of 8 by morecareful calculation of
pivot velocities near
panel edges
CSIRO Marine and Atmospheric Research 34
With better
staggered
velocities at
panel edges
CSIRO Marine and Atmospheric Research 35
Comparisons of VCAM and CCAM
•
•
•
•
•
•
VCAM advantages
No Helmholtz equation needed
Includes full gravity-wave terms (no T linearization needed)
Mass and moisture conserving
More modular and code is “simpler”
No semi-Lagrangian resonance issues near steep mountains
Simpler MPI (“computation on demand” not needed)
VCAM disadvantages
• Restricted to Courant number of 1, but OK since grid is very
uniform
• Some overhead from extra reversible staggering during sub
time-steps (needed for Coriolis terms)
• Nonhydrostatic treatment will be more expensive
CSIRO Marine and Atmospheric Research 36
Tentative conclusions
• Reversible interpolation works well for both
CCAM and VCAM
• VCAM seems to perform better than CCAM in
the tropics
- better rain over SPCZ and Indonesia, possibly by
avoiding linearizing ps term in pressure gradients,
and better gravity wave adjustment by not using
semi-implicit
- rainfall presently not as good in midlatitudes
CSIRO Marine and Atmospheric Research 37
Thank you!
CSIRO Marine and Atmospheric Research 38