Parallel Linear Solvers for Petascale Reservoir
Simulation using Adaptive and Unstructured Grids
Matthew Jackson, Ann Muggeridge, Matthew Piggott,
Christopher Pain, Gerard Gorman and Jeff Gomes
Applied Modelling and Computational Group (AMCG)
Petroleum Engineering & Rock Mechanics Group (PERM)
Saudi Aramco Technology Quest
10-12/10/2011
Leading team with the
breadth of skills
 Imperial College: Leading university – strong in Engineering.
 AMCG & PERM within ESE: Leading modelling and reservoir physics
groups – largest university research group in the UK and winner of the
Imperial College Research Excellence Award. Developed major models
e.g. the Imperial College Ocean Model as the next gen. Ocean model
for the UK and Internationally.
 The team:
• Prof Matt Jackson: Reservoir characterisation and engineering
• Prof Ann Muggeridge: Reservoir physics, EOR
• Prof Chris Pain: Adaptive meshes, linear solvers, discretisation
• Dr Matt Piggott: Geophysical CFD
• Dr Gerard Gorman: HPC, mesh generation
• Dr Jeff Gomes: Multi-physics CFD modelling
Technology Quest
Development of computational technologies that
enable detailed multi-scale simulations of fluid
flows in giant oil and gas fields.
How we will tackle this Tech Quest
We will develop the following modular technologies that may be
integrated into Saudi Aramco simulator:
 Petascale linear solvers for multi-scale reservoir simulations with
high aspect ratio elements/cells building on open-source code.
 Numerical dynamic core for multiphase flow on unstructured
adaptive meshes that are accurate (exact).
 Demonstration through simulations of fluid flow in giant oil and
gas fields.
CFD Modelling:
Parallel Anisiotropic Mesh Adaptivity and Domain
Decomposition
Adaptive Mesh and Petascale Solvers
Movie
Page 6
CFD Modelling:
Parallel Mesh Adaptivity - Parallel Scaling
Adaptive Mesh and Petascale Solvers
 Parallel adaptive mesh technology – world leading.
 Unique high aspect ratio multi-grid solvers. Front runner in GFD.
 AMCG and Fujitsu are enhancing the adaptive mesh library as part of
their Open Petascale Libraries (OPL) initiative (MPI/OpenMP).
 On-going collaboration with Argonne National Lab on hybridizing
(MPI/OpenMP) PETSc libraries. Maintained by NAG.
 AMCG, Fujitsu, Dainsbury and EPCC: collaboration on HPC and mesh
adaptivity to scale > 100K cores: better able to exploit memory
hierarchy.
Numerical Dynamic Core
 Development of novel family of tetrahedral elements.
 P1DGP2 + overlapping elements have unique abilities to represent key force
balances on unstructured mesh e.g. Darcy’s Law & hydrostatic pressure.
 Exact force balance with only approximation in advection – limited CV-DG or CVFEM and implicit time stepping
 Compositional modelling.
Technology Transfer : Training, Dissemination &
Communication with Saudi-Aramco
 Train PDRAs and PhD students for the future e.g. for Saudi Aramco.
 Intensive established training events and workshops on methods, use of
models, how to develop code & physics.
 Joint workshops, exchange visits and placements.
 Software: Manual, Visualization, Management (regression testing, IRC chat,
bug and issue tracking, code management).
Page 10
Added value and future work
To develop the following modular technologies to be integrated
into Saudi Aramco simulator:
Hiding the details of the hardware that the model is run on e.g.
GPUs.
Predictive modelling (adjoints, EnKF, reduced order modelling,
sensitivity analysis, adaptive observations, data assimilation,
uncertainty).
Cross fertilization with other disciplines.
Summary
 Looking for long term strategic partnership in reservoir
simulators.
 Will develop key components of a next-generation simulator.
 Parallel mesh adaptivity will be a key aspect of a future multiscale simulator.
 Dynamic core that can represent physics accurately on meshes
is essential.
A Few Backup Slides
Conservative Interpolation: Crucial for many applications, with
Adaptivity, Coupled models
‘donor’
mesh
‘target’
mesh
‘super mesh’
and mapping
from donor
mesh
Following Page
construction
15of the supermesh we are able to perform
projection operations to achieve conservative, bounded (etc…)
interpolation schemes. 2D: Farrell et al., 2009
3D: Farrell, Maddison, 2009
Immersed-Body Method (Fluid-Solid
Interaction Modelling)
Mesh Adaptivity and Domain
Decomposition
Lower frame shows the domain
decomposition which has been
optimised to balance the load based
on number of nodes and minimised
edge cut
Applications: Surface-based reservoir
modelling
Applications: Surface-based reservoir
modelling