Lawrence Livermore National Laboratory
Structured Adaptive Mesh Refinement in
a Multiblock Arbitrary–Lagrangian–
Eulerian Radiation–Hydrodynamics Code
Michael E. Wickett
R. W. Anderson, N. S. Elliott, B. T. Gunney,
R. D. Hornung, L. H. Howell, B. S. Pudliner, B. S. Ryujin
International Conference on Numerical Methods For Multi-Material Fluid Flows
September 5 – 9, 2011
Arcachon, France
Lawrence Livermore National Laboratory, P. O. Box 808, Livermore, CA 94551
This work performed under the auspices of the U.S. Department of Energy by
Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344
LLNL-PRES-494894
ALE-AMR in ARES
Outline
 AMR introduction
• Definitions, mesh and algorithm choices
• AMR infrastructure and scalability
• Problem setup and refinement
 AMR algorithmic pieces
•
•
•
•
•
Derefinement tangling
Refinement of multimaterial zones
Multiblock connectivity
Refinement of mesh motion state
Diffusion solver
 Some results
•
•
•
•
Hydrodynamics
Radiation
MHD
Turbulent mix
Lawrence Livermore National Laboratory
LLNL-PRES-494894
2
AMR definitions
Patch (domain) based
refinement
Level 0
Levels are a collection of
patches in the same index
space
Level 1
Hierarchy is the collection
of levels
Levels are completely nested
inside the next coarser (i.e.
properly nested)
Covered coarse zones are still stored and
used for computation, but they are
synchronized with the overlying coarse mesh
Level 2
NO time subcycling
All levels advanced with the
same timestep
Lawrence Livermore National Laboratory
LLNL-PRES-494894
3
Staggered Lagrange hydro gives preference to odd
refinement ratios
 A 1:rd logical correspondence between both cell and nodal quantities is
only possible with odd refinement ratios
 This makes invertible pairs of operators simple to construct
level n+1
1x3
level n
r=2
r=3
anisotropic
refinement
is allowed
3x3
3x1
Lawrence Livermore National Laboratory
LLNL-PRES-494894
4
To preserve quadrilateral zones, coarse solution
drives fine solution at interfaces
Fine node
positions at
coarse-fine
interfaces set by
interpolation from
coarse positions
 Leads to a particularly simple form for each part of the Lagrange step
• Coarse grid solution is advanced
• Fine grid solution is advanced getting incorrect values at coarse-fine interfaces
• Fine grid solution at coarse-fine interfaces is interpolated from coarse
 Has implications for many other parts of the method
Lawrence Livermore National Laboratory
LLNL-PRES-494894
5
AMR capability built with SAMRAI:
Structured Adaptive Mesh Refinement Applications Interface
SAMRAI Provides:

Transparent parallel communication (MPI)

Dynamic gridding support

Common data types (cell, nodes, …)

Inter-patch data transfer operations (copy,
coarsen, refine, time interp, …)

Solver interfaces for SAMR data (hypre,
PETSc, kinsol)

Checkpointing and restart (HDF5)

Visualization support (VisIt)

Multiblock, enhanced/reduced connectivity
User provides:

Numerical routines (serial) for individual patches

Composition of SAMRAI classes to implement desired algorithm.
Lawrence Livermore National Laboratory
LLNL-PRES-494894
6
Unique SAMRAI design characteristics
enable ARES-AMR development
 Arbitrary (structured) mesh coordinates
• ARES staggered-mesh ALE hydro unchanged
• Mesh refinement is variable by coordinate direction (anisotropic refinement)
 Multiblock AMR
• straightforward translation between SAMRAI patch hierarchy and ARES blockstructured mesh
• “enhanced” and “reduced” block connectivity
 SAMRAI PatchData, communication abstractions
• native ARES data operated on directly by AMR infrastructure
 no rewriting of data structure for AMR, no extra copies of data for AMR
• SAMRAI handles interlevel communication (refinement, synchronization)
• ARES handles intralevel communication (neighbor comm, reductions)
 SAMRAI owns neither mesh nor data
• invisible SAMRAI footprint for non-AMR ARES operation
Lawrence Livermore National Laboratory
LLNL-PRES-494894
7
SAMRAI provides advanced AMR scalability
 Distributed mesh-management: processors store only local patch data and
nearest-neighbor patch geometry (No global mesh description)
• Box intersection algorithm uses Recursive Binary Tree search
• Load balance uses only nearest-neighbors on a processor tree
• Clustering (box generation) uses a combined task- and data-parallel algorithm
 Weak scaling results
•
•
•
•
•
problem grown by domain tiling
linear advection of wavy front
regrid every 4 timesteps
run on BG/L (LLNL uBGL)
scaled to 64K processors
 Scaling work continues
• optimization of new algorithms
• removing global data from ARES
Lawrence Livermore National Laboratory
LLNL-PRES-494894
8
We provide easy yet flexible AMR problem setup and
many options for specification of refinement
 Problem is usually specified at coarsest level
• Nodelists/Zonelists: scale automatically from
coarse level definition
• level-specific syntax provided for exceptions
 Direct tagging: can choose minimum/maximum
refinement level over parts of problem
• amr(myzonelist) = minlevel 3
 Refinement criteria: global and local
refine default
criterion den 2nd_diff
threshold_unrefined 0.04
threshold_refined 0.02
normalize local limit 0.1
criterion velocity_magnitude
threshold_unrefined 1e-7
threshold_refined 9e-8
combine and
criterion p 2nd_diff
threshold_unrefined 0.04
threshold_refined 0.02
normalize local limit 0.1
combine or
• tagging on values, 1st differences, or 2nd differences
endrefine
of any problem field
levelop collapse
• logical combinations of criterion
 Levels can be removed, both fine and coarse
Automatic scaling
Level specific
level_num
levelop remove_finest num_levels
levelop remove_coarsest num_levels
Automatic scaling
Level specific
Lawrence Livermore National Laboratory
LLNL-PRES-494894
9
We must take care to avoid tangling on derefinement
It is possible that when regridding the hierarchy, the removal of finer mesh can lead
to tangling of the mesh at the new coarse-fine interfaces.
Removal of
shaded fine mesh
Removal of
unshaded fine mesh
To avoid this, we attempt to detect if such
tangling will occur on regrid and prevent
such zones from derefining.
Lawrence Livermore National Laboratory
LLNL-PRES-494894
10
Multi-material zone refinement is in development
• The capability to refine multi-material zones (“mixed zones”) is currently in
development using a geometric overlay method.
• Because we currently cannot refine mixed zones, we cannot allow them to come
into existence on any other than the finest level.
If a fine zone is allowed
to coarsen, on
subsequent cycles we
cannot prevent it from
being refined
• Our default refinement criteria attempt to avoid this problem.
• We prevent this situation by always tagging mixed zones for refinement.
Lawrence Livermore National Laboratory
LLNL-PRES-494894
11
Multiblock connectivity requires special AMR
boundary conditions
 Traditional AMR algorithms treat coarse-fine boundaries by interpolating
coarse mesh values into “phony” or “ghost” zones or nodes of fine mesh
 Many of ALE-AMR algorithms can use this method, but some cannot
• mesh relaxation: treat hierarchy as “composite” mesh
• multiblock enhanced connectivity: data structures have only 1 phony at corner
1
extra step
added to fill
multiple corner
“phony” zones
2
3
mesh
relaxation
0
4
Lawrence Livermore National Laboratory
LLNL-PRES-494894
12
Refinement of ALE grid motion state
 Grid motion state determines if
and how a give node is
moved/relaxed
 Generally, refinement can be
done by priority: fine nodes take
the value of the highest
“ranking” neighbor state
 “BND” states allow relaxation
along a logical line or surface
Priority:
5 – NULL
4 – Relax
3 – Backup then Relax
2 – Backup
1 – No Relax
Green – iBND
Blue – jBND
Red – No Relax
– NULL
• When refining near a BND
surface, only those nodes along
the surface should get the BND
state upon refinement.
• Between two BND states, we
choose the NULL state
Lawrence Livermore National Laboratory
LLNL-PRES-494894
13
AMR diffusion solver
 Coarse and fine levels are coupled together into a single matrix
• We do not subcycle in time, so all levels participate in every solve
• Covered zones may passively participate in the linear system, but they do not
affect the solution in exposed zones
 Pert discretization is extended to differencing between levels
• The discretization only changes for zones adjacent to coarse-fine boundaries
• Fine interface difference stencils are replaced with finite differences that see both
coarse and fine data
• Coarse interface difference stencils are defined to match the flux of the
corresponding fine zones
• System is conservative, but not quite symmetric
 Sparse matrix built with HYPRE semi-structured interface
• Still experimenting with solver options
• Current default is AMG-preconditioned GMRES
Lawrence Livermore National Laboratory
LLNL-PRES-494894
14
Multiblock Cylindrical Sedov
red - single level
t = 0.07
t = 0.7
black - AMR
blue - exact
density
3 levels
single-level (fine)
0.129
AMR (3-levels)
0.125
single-level (2/3 fine)
0.197
radius
L2 Error Norm
Lawrence Livermore National Laboratory
LLNL-PRES-494894
15
3D Multiblock Sedov
red - single level
t = 0.07
black - AMR
blue - exact
density
t = 0.7
3 levels
single-level (fine)
0.151
AMR (3-levels)
0.147
single-level (2/3 fine)
0.135
radius
L2 Error Norm
Lawrence Livermore National Laboratory
LLNL-PRES-494894
16
2D Multiblock Spherical Piston
red - single level
t = 0.1
black - AMR
blue - exact
density
t = 0.65
radius
3 levels
single-level (fine)
0.948
AMR (3-levels)
0.966
single-level (2/3 fine)
0.979
L2 Error Norm
Lawrence Livermore National Laboratory
LLNL-PRES-494894
17
3D Multiblock Spherical Piston
t = 0.1
red – single level
black – AMR
blue – 1D ref
density
t = 0.65
radius
3 levels
single-level (fine)
0.684
AMR (3-levels)
0.684
single-level (2/3 fine)
0.710
L2 Error Norm
Lawrence Livermore National Laboratory
LLNL-PRES-494894
18
2D Multiblock Balls and Jacks Advection Test
red - single level
black - AMR
Lawrence Livermore National Laboratory
LLNL-PRES-494894
19
2D Multiblock Balls and Jacks Advection Test
AL2 scaled mass ratio
AL1 scaled mass ratio
time
Lawrence Livermore National Laboratory
LLNL-PRES-494894
20
3D Multiblock Balls Advection Test
AL1 scaled mass ratio
AL2 scaled mass ratio
t = 3.5
time
Lawrence Livermore National Laboratory
LLNL-PRES-494894
21
AMR can be used to increase timesteps on a fan mesh
 Fan mesh preserves symmetry better
for spherical problems
 Coarsen mesh toward origin to lessen
impact of CFL limits
Pressure-Driven Capsule
(Post-bounce, hydro only)
• Some problems take 8-10x fewer cycles
 Similar to spatial filtering methods
• Requires interpolation in polar
coordinates
 Has relatively large AMR overhead, but
for this special case, the AMR
infrastructure can be optimized
• No regridding
• Complete synchronization unnecessary
Lawrence Livermore National Laboratory
LLNL-PRES-494894
22
Simulations of shock tube experiments
on the Richtmyer-Meshkov instability
 Sharp Interface w/ single mode perturbation
•
•
•
•
•
•
•
No-Slip Walls
Upstream inflow at post-shock
Under-resolved laminar BL
Base resolution 19 zones/Width
Refined on 2nd differences of density
4 levels with refinement ratio of 3x3
Fully refined resolution 513/Width
 Inclined interface w/ no perturbation
• studied variations in angle, shock strength,
Atwood number, resolution
• resolution 56 to 282 μm
• 3 to 5 AMR levels
• < 30% at finest at late time
Courtesy Jeff Greenough (LLNL)
Shocked
Air
Preshocked Air
Preshocked SF6
Lawrence Livermore National Laboratory
LLNL-PRES-494894
23
AMR radiation diffusion examples
 Graziani-Leblanc crooked pipe
• Refinement on 2nd difference of
radiation temperature
 Planar radiating shock wave
~ 5x speedup
• Semi-analytic solution from
Lowrie and Edwards
~ 8x speedup
Lawrence Livermore National Laboratory
LLNL-PRES-494894
24
2D Cylindrical AMR MHD
 Magnetic field points in theta direction (normal to simulation plane)
 Magnetic diffusion operator discretized with a variation of the Pert scheme
• AMR solver is as described for radiation diffusion
Hollow conducting cylinder with
constant voltage drop across ends
Conductivity is function of radius
Exact time and space dependent
solution by David Miller (LLNL)
Courtesy Rob Rieben (LLNL)
Refinement on 2nd differences of
magnetic field and on material
interfaces
Lawrence Livermore National Laboratory
LLNL-PRES-494894
25
AMR with subgrid turbulence mix models is a recent
addition
 Barenblatt burst problem is an analytic problem with no hydro or sources
• K-L turbulent mix model
 Coupled non-linear diffusion, growth, and dissipation
• AMR diffusion solver is as described for radiation diffusion
turbulent
kinetic
energy
red - analytic
black - AMR
turbulent
length
scale
Lawrence Livermore National Laboratory
LLNL-PRES-494894
26
ALE-AMR Summary
 AMR has become an integral part of the ARES code
• Progress has been made in AMR-izing nearly all components
• user input, control, output is growing complete and (relatively) straightforward
 Solutions on a wide range of problems give good answers compared with
everywhere fine solutions
 Significant performance improvement is seen in many (not all!) problems
• overhead reduction continues to be worked
• SAMRAI scaling and multiblock improvements continue
 Experience is being gained, but much more is needed
• choices refinement criteria, when/where can we avoid refinement?
• number of coarser levels, when to remove levels, etc.?
Lawrence Livermore National Laboratory
LLNL-PRES-494894
27