Paper 16.10
A 10-year roadmap for scalable
modelling capability
Andy Brown and Nigel Wood
© Crown copyright Met Office
Table of Contents
 Roadmap
 Optimisation of current UM
 ENDGame
 GungHo!
© Crown copyright Met Office
Issues
 Weakness of New Dynamics
formulation (accuracy, stability,
conservation)
 Limited scalability
 Need for overhaul of UM code
(approaching 25 years old)
 Completely new model ~20182020 (NGWCP/Gung-Ho)
 Non latitude-longitude grid
 Possible move away from implicit to
explicit scheme (avoid solver)
 Possible changes to advection
scheme (away from semi-Lagrangian)
 What to do in the mean time?
© Crown copyright Met Office
New Dynamics
ENDGAME + Yin-Yang
New Dynamics + V-at-pole
Next generation
ENDGAME
1
2
3
4
5
2012
2016
2020
Recommendations
 Decide whether to progress ENDGame to
operational implementation
 Following that, review strategy for improving
tracer conservation within the UM
 Continue, and strengthen, efforts to
incrementally improve model efficiency
 Strengthen efforts to refresh UM controlling
infrastructure
© Crown copyright Met Office
Recommendations
 Aim for completely new model (science
and code) to be operational by 2018-2020
 An intermediate upgrade to Yin-Yang will
not be pursued
 Engage with UM partners (International
and NERC) over our thinking
© Crown copyright Met Office
Business-as-usual
HPC Optimisation
© Crown copyright Met Office
N512 scalability – Nov 2010
• Figure 1
© Crown copyright Met Office
N512 scalability – Cray XE6
Speedup
9
8
7
6
5
Thanks: Pier
Luigi Vidale,
NCAS.
Ideal
Cray XE6
4
3
2
1
0
0
0
0
20
0
0
40
0
0
60
0
0
80
00
0
10
00
0
12
00
0
14
Cores
• Climate model (atmosphere only)
• Preparation for real science on PRACE XE6
© Crown copyright Met Office
I/O Server savings
4500
Wallclock time / sec
4000
Original
3500
3000
IO Server
2500
2000
1500
1000
500
0
Global (7-day)
Global (2-day)
NAE
UKV
19%
28%
33%
10%
© Crown copyright Met Office
Other Optimisation work
 Continued OpenMP work
 2 threads used routinely
 4 threads quicker than 2 on >= 1024 cores
 Scalability bottlenecks being removed
 River routing parallelised
 Improved algorithms for Latent Heat Nudging in AC
 Some physics restructuring (with science owners)
 Reducing load imbalance impact
 Improving OpenMP implementation
 … and lots more!
© Crown copyright Met Office
ENDGame
© Crown copyright Met Office
Even Newer Dynamics
 Evolution of New Dynamics
 Same equation set & variables (θ-π)
 Same horizontal staggering (Arakawa C-grid)
 Same vertical staggering (Charney-Phillips)
 (Iterative) Semi-implicit semi-Lagrangian
 Physics and DA unchanged
© Crown copyright Met Office
Why bother?
 Improved (iterative) solution procedure:
More implicit, approaching Crank-Nicolson
 Improved robustness and accuracy
 Improved scalability
 “Trivial” change to what is stored at the poles
 Option for improved conservation via SLICE
© Crown copyright Met Office
Since last time
 Coupled to physics packages
 Dynamics development and testing complete
 Two significant formulation changes…
© Crown copyright Met Office
 Semi-implicit profile = model state
at previous time step
 Stability

 Accuracy 
 Efficiency 
© Crown copyright Met Office
 Departure point calculation
Wood et al (2010)
Dr/Dt = u

Thuburn and White (2012)
Dr/Dt = k.u = w &
Dk/Dt = vH
© Crown copyright Met Office

Where are we?
 Global version available in UM since UM7.9
 APP given physics clean bill of health
 Testing phase
 Determine suitability as replacement UM core
 What does that mean…
© Crown copyright Met Office
Aim of initial testing
 How stable is the model?
 Is it more robust than New Dynamics?
 Cost cf. New Dynamics at same Δx and Δt?
 Potential to increase time step?
 Potential to increase grid size?
 Is model more scalable?
© Crown copyright Met Office
Testing strategy
 Held-Suarez (DR)
 Long term stability in presence of forcing
 Aquaplanet tests (APP)
 Moist dynamics, physics
 20 Standard case studies & AMIP run (GMED)
 Full runs with orographic forcing etc
© Crown copyright Met Office
New Dynamics
Held-Suarez: Transient Eddy
Kinetic Energy (Chris Smith)
ENDGame
α=0.55
© Crown copyright Met Office
ENDGame
α=1.0
Explicit
Eulerian
Model
(max 360)
Aquaplanet tests at N48
(Ian Boutle)
 Testing ENDGame with Physics at N48
 50 day simulation, perpetual equinox, 15 minute
timestep
 Results very similar to New Dynamics
 Some evidence that ENDGame has a slightly
more active hydrological cycle
© Crown copyright Met Office
New Dynamics
Relative
Humidity
ENDGame
Difference
© Crown copyright Met Office
Stratospheric winds
(Simon Vosper)
Vertical velocity at 41km N320
New Dynamics
© Crown copyright Met Office
ENDGame
Longer Term
From evolution to revolution?
© Crown copyright Met Office
UK Collaboration of GFD,
numerical and computational
scientists
 5 FTEs from Met Office (Dynamics Research
& HPC Optimisation)
 5 FTEs from NERC (Bath, Exeter, Imperial,
Leeds, Manchester, Reading)
 2 FTEs from STFC
© Crown copyright Met Office
GungHo!
Globally
Uniform
Next
Generation
Highly
Optimized
© Crown copyright Met Office
Working Together Harmoniously
© Crown copyright Met Office
5 Year Project
 “To research, design and develop a new dynamical
core suitable for operational, global and regional,
weather and climate simulation on massively parallel
computers of the size envisaged over the coming 20
years.”
 To address (inter alia):
 What should replace the lat-lon grid?
 How to transport material on that grid?
 Is implicit time scheme viable/desirable on such computers?
 First phase over 2 years, organised around 5
themes:
© Crown copyright Met Office
GungHo Themes
 Quasi-Uniform Grids (icosahedral;
kites/balanced triangles; cubed-sphere; YinYang)
 Advection schemes (conservation, SL, ...)
 Time schemes (explicit vs. implicit)
 Test cases
 Computational science aspects
© Crown copyright Met Office
Highlights from the last 9 months
© Crown copyright Met Office
Quasi-Uniform Grids
 QJ Review paper on all things grid-like
[Staniforth and Thuburn]
 Fundamental research on finite-difference
non-orthogonal C-grid
[Thuburn and Cotter]
 Similar scheme but for finite-elements –
straightforward route to higher order
[Cotter and Shipton]
 Computational modes and grid imprinting
on quasi-uniform grids
[Weller, Thuburn and Cotter]
© Crown copyright Met Office
Yin-Yang ENDGame
(Mohamed Zerroukat)
Lat-long
© Crown copyright Met Office
Yin-Yang
Spectral element analysis
(Tom Melvin)
© Crown copyright Met Office
Spectral element discretization of
Gaussian hill advection
© Crown copyright Met Office
Explicit time schemes
 Split-explicit scheme currently ruled out;
pros outweighed by cons
 Multi-step schemes (e.g. leapfrog)
currently ruled out due to efficiency and
accuracy issues
  Multi-stage schemes (i.e. RungeKutta) under active consideration
© Crown copyright Met Office
Explicit time schemes II
 Vertical grid spacing makes explicit
scheme in vertical too expensive
 Horizontally Explicit Vertically Implicit
 Work started on analysing such
schemes to propose appropriate form
© Crown copyright Met Office
Summary
 Ongoing optimisation of current code
 Algorithmic optimisation of current
dynamical core (ENDGame)
 GungHo! up and running…
 Decision not to pursue medium term
approach
© Crown copyright Met Office
Questions and answers
© Crown copyright Met Office