The Future of Multiscale Computing
Peter Coveney
Centre for Computational Science
University College London
Multiscale Modelling and Computing
Outline
1
Multiscale computing: the past and present
2
Existing multiscale computing challenges
3
Multiscale computing: the future
4
Multiscale approaches for the exascale
5
Summary and Acknowledgements
Introduction
• How has multiscale modelling and computing progressed
up to today?
• What challenges in multiscale computing can we expect to
face in the near future?
• How can we meet these challenges?
Types of coupling: how we categorize it
computationally
Multiscale projects are common
Number of multiscale projects by start year
180
EU
NSF
NIH
DOE
160
140
Groen et al.,
accepted by CiSE,
2014,
120
100
arXiV:1208.6444
80
60
40
20
0
20
20
20
20
20
20
20
20
20
20
20
20
19
19
19
19
11
10
09
08
07
06
05
04
03
02
01
00
99
98
97
96
...but the developments differ between
domains
Number of multiscale projects by start year
70
60
biology
engineering
environment
materials
all other
50
40
30
20
10
0
11
20 0
1
20 9
0
20 8
0
20 7
0
20 6
0
20 5
0
20 4
0
20 3
0
20 2
0
20 1
0
20 0
0
20 9
9
19 8
9
19 7
9
19 6
9
19
Multiscale computing: the requirements
• Each submodel in a multiscale simulation has distinct
requirements:
• Some are compute-intensive, some data-intensive, some
communication-intensive.
• Models may require large amounts of memory, or dedicated
architectures (e.g., GPU).
• Submodels must be quickly started, and coupling
communications need to be done efficiently.
• Advance reservation to allow synchronous starting of cyclic
concurrent applications.
• Specialized coupling libraries allow more flexible coupling
and superior wide area communication performance.
Multiscale computing: coupling tools
• Numerous tools have emerged to facilitate coupling and
multiscale computing.
• All of these tools originated from specific science domains
to address targeted problems.
• Over time, many of these tools have widened their scope.
The coupling tool or (F/f)ramework
landscape
Note: tools written in italics feature a GUI, those in bold text
primarily rely on compiled languages (e.g. C, Fortran), those in
normal text on interpreted/scripting languages.
The Virtual Physiological Human
• 207M Euro initiative in EU-FP7
(2007-2013).
• Aims to:
• Enable collaborative
investigation of the human body
across all relevant scales.
• Introduce multiscale
methodologies into medical and
clinical research.
• The VPH framework is:
• Personalized
• Predictive
• Participatory
• Preventive
Organism
Organ
Tissue
Cell
Organelle
Interaction
Protein
Cell Signals
Transcription
Gene
Molecule
The Virtual Physiological Human - HemeLB
• 3D CT or MR angiography data now commonly acquired
for diagnostic purposes.
• Can we use this to personalize models?
• To be useful, everything must be fast. Need to get data to
clinicians on timescale of surgical planning ≤ 1h
Multiscale simulation of nanomaterials
• We investigate mixtures of two different materials to look
for enhanced performance, e.g.:
• Improved fire retardant properties.
• Similar performance to other composites at much lower
filler volumes.
• Improved barrier properties to gases.
• Main ingredients:
• Montmorillonite clay and polymers, such as
Polyethyleen-Glycol (PEG).
• QM, atomistic and coarse grained simulations.
Nanomaterials: scale separation map
Nanomaterials: our computational approach
today
• Our approach relies both on ensemble simulations and
replica-exchange simulations 1 .
• A Pilot Project for EGI/EUDAT/PRACE, in preparation for
production beyond the petascale. 2
1
2
Suter et al., MRS Online Proceedings Library 1470(1), 2012.
https://wiki.egi.eu/wiki/EEP Pilot2 (grid cert required)
Nanomaterials: Preliminary results
• Visualizations of clay sheets on the quantum scale (left),
atomistic scale (middle), and coarse-grained scale (right).
Multiscale computing: key aspects for the
future
• Techniques for validating models and investigating the
error.
• Especially complicated for multi-physics models.
• Reproducibility and verification.
• In several fields: Multiscale simulations with three or more
submodels
• Multiscale computing on exascale systems
Exascale computing
• The emerging exascale computing architecture will not be
simply 1000x today’s petascale architecture.
• All future exascale computer systems designs will have to
address some of the following challenges:
• processor architecture is still unknown,
• system power is the primary constraint for the exascale
system,
• memory bandwidth and capacity are not keeping pace with
the increase in flops,
• clock frequencies are expected to decrease to conserve
power,
• cost of data movement, both in energy consumed and in
performance is not expected to improve,
• the I/O system at all levels will be much harder to manage,
• and reliability and resiliency will be critical.
Introduction - the monolithic road
• A common approach is to construct one code to solve a
particular large system.
• This code is then scaled up in size when deployed on
emerging peta/exascale infrastructures.
• Only a very limited number of codes run computationally
efficiently using a full top-tier resource.
• In addition, simply increasing size/resolution/number of
steps may not always be the best way to uncover new
science!
Monolithic code scaling
• Many scientific codes are limited by communication
overhead (e.g. spectral methods) and can only scale up to
1000s of compute cores.
• To reach exascale, we need to scale up to 106 cores.
• As shown before, an example of well-scaling approaches
are Lattice-Boltzmann methods
• HemeLB - blood flow (>32,000 nodes)
• HYPO4D - turbulent flow (>250,000 nodes)
• LB3D - complex fluid flow(>250,000 nodes)
Introduction - requirements
• Modelling complex systems on exascale resources comes
with a range of requirements.
• These requirements include:
• not only improved parallelism,
• but also in many cases model coupling,
• and new modes of accessing and using e-infrastructures.
Introduction - parallelism
• Increased parallelism has been key to efficiently scaling up
simulations.
• Several proven ways to improve parallel efficiency involve
optimizing:
• communication (localizing it, reducing the number of
synchronization points),
• domain decomposition (ensure optimal calulcation and
communication load balance),
• new levels of parallelization (hybrid parallelization, GPU,
Xeon Phi).
Hybrid parallelisation
• Accelerators such as GPU and Xeon Phi offer very high
computing power but impose strong restrictions on the
code structure
• The code has to be easily vectorised (no branches, no data
dependencies).
• Coalesced and aligned memory access patterns.
• High compute to memory access ratio.
Introduction - resource access
• Multiscale simulations require new ways for resource
access:
• Advance reservation reduces time to completion for
non-concurrent multi-model work flows.
• Advance reservation eliminates a huge waste of compute
cycles for concurrent multi-model simulations.
• Urgent computing for simulations of immediate importance
(e.g., those used in surgical assistance)
Multiscale Approaches for Exascale
Aim to combine models at various scales
A general multi-scale model for engineering illustrating scale and overlap. 3
3
http://www.srcf.ucam.org/~jae1001/ccp5_2007/multiscale_materials-cam.jpg
Challenge: Multiscale modelling spans
established disciplines
Integration frequently occur across established disciplines
Construction example (Bitumen) 4
4
Source: CCP5
Extreme Scaling
• Monolithic approaches may sometimes be
computationally efficient on the exascale, but
have limited scientific relevance.
• Applications may omit crucial phenomena.
• Larger systems tend to require more
integration steps as well, and therefore take
disproportionally longer to complete.
• Extreme scaling applies the coupling of
multiple petascale submodels.
• Extreme scaling allows for modelling
additional phenomena, while constraining the
number of integration steps required.
Heterogeneous Multi-scale Computing
• A modelling approach for problems where the
constitutive equations of a local state in the
macroscale system of interest are not known
in closed form,
• e.g., as a consequence of the sheer
complexity of the processes at the
microscale.
• Here numerical solvers describe domains of
the macroscopic system.
• Missing macro-scale data and behaviour is
provided using a number of instances of
micro-scale models.
Heterogeneous Multi-scale Computing
• HMC has been applied successfully in several scientific
disciplines, such as
• mechanical systems 5 ,
• and suspension flow 6 .
5
6
Calvo et al., SIAM J. on Sci. Comp. 32(4), 2010
E. Lorenz et al., Multiscale Modeling & Sim. 9(4), 2011
Replica Computing
• Combines a large number of terascale and
petascale simulations (also known as
’replicas’) to produce scientifically important
and statistically robust outcomes.
• The replicas are not part of a larger spatial
structure, but are applied to explore a system
under a broad range of conditions.
Replica Computing
• Three different scenarios:
1. Ensemble simulations
2. Dynamic ensemble simulations
3. Replica-exchange simulations
• For example, Replica-exchange is used in MD simulations
to manage particle exchanges between molecular systems
at different temperatures. 7
7
Suter et al., J. of Mat. Chem. 19(17), 2009
Summary
• Challenges to address and marry fundamental scientific
rigour with ease of use and deployment.
• Traditional applications may be able to efficiently run on the
exascale, e.g. by using neighbour-based algorithms and
hybrid parallelization.
• However, many cutting-edge scientific problems cannot be
solved by simply scaling up existing approaches.
• Novel computing patterns help us to tackle these problems
on the exascale.
• e.g., extreme scaling, heterogeneous multiscale modelling
and replica computing.
Acknowledgements
Thanks go out to:
• James Suter, Jacob Swadling, Derek Groen,
Lara Kabalan (nanomaterials)
• Rupert Nash, Miguel Bernabeu, Timm
Krueger, James Hetherington, Hywel Carver,
Jiri Jaros, Derek Groen (bloodflow)
• Luis Fazendeiro, Bruce Boghosian (HYPO4D)
• Stefan Zasada, Ali Haidar, David Chang and
the MAPPER Consortium (software services)
• Derek Groen and Jiri Jaros (preparing this
talk)