Modeling and HPC Challenges in MHD:
Magnetically-Confined Fusion Plasmas
US - France Modeling and HPC Workshop
Georgetown University
22 October 2007
Daniel R. Reynolds
drreynolds@ucsd.edu
UC San Diego, Computational and Applied Math Group
October 22, 2007
(1)
Outline
I. Resistive MHD Description and Equations
II. Challenges: Modeling and Computational
III. Current Solution Approach and Results
IV. Future Directions
UC San Diego, Computational and Applied Math Group
October 22, 2007
(2)
Plasma Confinement
• A plasma is a “hot ionized gas” whose motions both generate
electromagnetic fields and are influenced by them.
• No material walls can contain the plasma at millions of degrees Kelvin.
Either the plasma will damage the wall, or the wall will cool the plasma.
tokamak
stellarator
UC San Diego, Computational and Applied Math Group
October 22, 2007
(3)
Resistive Magnetohydrodynamics Primer
The resistive MHD model provides the simplest fluid description of fusion
plasmas. The model couples the viscous, compressible Euler eqns (fluid flow),
∂t ρ + ∇ · (ρv) = 0,
”
“
T
+ ∇p + B × J = ∇ · τ ,
∂t (ρv) + ∇ · ρvv
`
´
∂t e + ∇ · (e + p − 12 B · B)v + ∇ · (E × B) = ∇ · (τ v) + ∇ · (κ∇T ),
with the low-frequency Maxwell equations (electromagnetic fields),
∂t B = −∇ × E
[Faraday’s Law],
J=∇×B
[Ampere’s Law],
E = −v × B + ηJ −
di
ρ
(J × B − ∇pe )
∇·B=0
Here e =
p
γ−1
[Ohm’s Law],
[no magnetic monopoles].
+ ρ v·v
+
2
B·B
,
2
T =
p
,
ρ rgas
`
´
and τ = µ ∇v + (∇v)T − 23 µ(∇ · v)I.
Black terms give ideal MHD; +blue terms give resistive MHD; +green terms
give extended MHD.
UC San Diego, Computational and Applied Math Group
October 22, 2007
(4)
Coupled Resistive MHD Equations
Combining these, we obtain the coupled PDE system,
∂t ρ + ∇ · (ρv) = 0,
”
“
T
+ ∇p + B × (∇ × B) = ∇ · τ ,
∂t (ρv) + ∇ · ρvv
´
`
∂t e + ∇ · (e + p − 21 B · B)v + B × (v × B) = ∇ · (τ v + κ∇T + B × η (∇ × B)) ,
∂t B + ∇ × (B × v) = −∇ × (η (∇ × B)) ,
∇ · B = 0.
Notes:
• At the continuum
“ level
” the last equation is extraneous: if ∇ · B(t0 ) = 0 and
∂
∂
(∇
·
B)
=
∇
·
B = −∇ · (∇ × E) = 0, so ∇ · B(t) = 0, ∀ t ≥ t0 .
∂t
∂t
• Even small errors giving ∇ · B 6= 0 give rise to significant errors in the
coupled solution, providing an unphysical force parallel to the B field
[Brackbill & Barnes, 1980].
UC San Diego, Computational and Applied Math Group
October 22, 2007
(5)
Outline
I. Resistive MHD Description and Equations
II. Challenges: Modeling and Computational
III. Current Solution Approach and Results
IV. Future Directions
UC San Diego, Computational and Applied Math Group
October 22, 2007
(6)
Space/Time Scales Challenge Modeling
Huge range of space, time scales:
• No unified models
• Strong (simplified) orderings
are not possible
Very stiff systems of equations:
• Fast wave: O(10−6 ) s
• Energy transport: O(100 ) s
• Resistive diffusion: O(102 ) s
Large-Scale Simulations Essential:
• Spatial resolution requires 103
to 105 processors
• Need scalable algorithms: time
integration, solvers, . . .
UC San Diego, Computational and Applied Math Group
October 22, 2007
(7)
Fusion Models and Applicability
UC San Diego, Computational and Applied Math Group
October 22, 2007
(8)
Target Applications
Magnetic Reconnection:
[Brin et al., J. Geophys. Res., 2001]
• Replaces hot plasma core with cool plasma
• Interested in investigating macroscopic stability
• Resolution, simulation time scale as S −1/2 , S 1/2
• ITER estimates require S ≈ 109
Pellet Injection Fueling:
[Samtaney et al., Comp. Phys. Comm., 2004]
• Shoot hydrogen pellets into plasma at high velocity
• Interested in location of mass deposition
• Pellet motion O(104 ) slower than fastest waves
• Pellet size O(104 ) smaller than reactor
Both require large-scale, long-time simulations,
highly anisotropic heat conduction.
UC San Diego, Computational and Applied Math Group
October 22, 2007
(9)
Stiffness in Resistive MHD
Resistive MHD stiffness results from fast hyperbolic waves and diffusive effects.
The ideal MHD waves are given by
λe = v
(entropy wave)
λd = v
(magnetic-flux wave)
λa = v ± An
(Alfvén waves)
λf,s = v ± cf,s
(fast/slow magnetosonic)
• The characteristic speeds satisfy cs << An < cf . Typically cf ≈ 106 m/s
for fusion plasmas.
• For magnetic reconnection simulations, explicit methods would require
O(107 ) time steps, even for a coarse spatial mesh.
• Moreover, diffusive (resistive, viscous) effects induce quadratic explicit
time step dependence on the spatial mesh (∆t ∝ (∆x)2 ).
UC San Diego, Computational and Applied Math Group
October 22, 2007
(10)
Outline
I. Resistive MHD Description and Equations
II. Challenges: Modeling and Computational
III. Current Solution Approach and Results
IV. Future Directions
UC San Diego, Computational and Applied Math Group
October 22, 2007
(11)
Computational Fluid Dynamics Approach
• Foliate space-time domain into distinct spatial snapshots,
[
Ω × [t0 , ∞) →
Ω(tn ).
n
• Cut each slab into cells (boxes),
[
Ω(tn ) =
Ωi (tn ),
i
define U(xi , tn ) at cell centers.
• Evolve U(xi ) in time using Gauss’
Theorem (conservation form):
Z
Z
∂t U dx =
∇ · F(U) dx
Ωi
Ωi
Finite-Volume method
Z
F(U) · n ds
=
∂Ωi
UC San Diego, Computational and Applied Math Group
October 22, 2007
(12)
Computational Fluid Dynamics Approach
• Foliate space-time domain into distinct spatial snapshots,
[
Ω × [t0 , ∞) →
Ω(tn ).
n
• Cut each slab into cells (boxes),
[
Ω(tn ) =
Ωi (tn ),
i
define U(xi , tn ) at cell centers.
• Evolve U(xi ) in time using Gauss’
Theorem (conservation form):
Z
Z
∂t U dx =
∇ · F(U) dx
Ωi
Ωi
Spatial domain (Ui,j )
Z
F(U) · n ds
=
∂Ωi
UC San Diego, Computational and Applied Math Group
October 22, 2007
(13)
Computational Fluid Dynamics Approach
• Foliate space-time domain into distinct spatial snapshots,
[
Ω × [t0 , ∞) →
Ω(tn ).
n
• Cut each slab into cells (boxes),
[
Ω(tn ) =
Ωi (tn ),
i
define U(xi , tn ) at cell centers.
• Evolve U(xi ) in time using Gauss’
Theorem (conservation form):
Z
Z
∂t U dx =
∇ · F(U) dx
Ωi
Ωi
Interface reconstruction (Ui+1/2,j )
Z
F(U) · n ds
=
∂Ωi
UC San Diego, Computational and Applied Math Group
October 22, 2007
(14)
Computational Fluid Dynamics Approach
• Foliate space-time domain into distinct spatial snapshots,
[
Ω × [t0 , ∞) →
Ω(tn ).
n
• Cut each slab into cells (boxes),
[
Ω(tn ) =
Ωi (tn ),
i
define U(xi , tn ) at cell centers.
• Evolve U(xi ) in time using Gauss’
Theorem (conservation form):
Z
Z
∂t U dx =
∇ · F(U) dx
Ωi
Ωi
Conservative evolution F(Ui+1/2,j )
Z
F(U) · n ds
=
∂Ωi
UC San Diego, Computational and Applied Math Group
October 22, 2007
(15)
Solution Approach: Time Discretization
We consider time discretization of the system of ordinary differential equations,
Z
1
∂t Ui = fi (U) :=
F(U) · n ds,
|Ωi | ∂Ωi
and evolve the solution using a high-order implicit multistep method (BDF),
q
X
ˆ
˜
g(U ) := U − ∆tβ0 f (U ) −
αi Un−i + ∆tβi f (Un−i ) .
n
n
n
i=1
• The time-evolved state Un = U(x, tn ) is found as the solution to the
system of nonlinear algebraic equations g(U) = 0.
• For a given order of accuracy q, the parameters αi and βi are fixed;
if q ≤ 2, the method is A-stable, otherwise it is A(α)-stable.
• ∆t and q are adaptively chosen at each time step, based on estimates
balancing solution accuracy, solver convergence, and temporal stability.
[Gear & Saad, 1983;
Hindmarsh et al., 2005]
UC San Diego, Computational and Applied Math Group
October 22, 2007
(16)
Inexact Newton-Krylov Solver
Iterate toward g(Un ) = 0 using a sequence of linearized solutions:
1. Given an explicitly-predicted initial guess, U0 ∈ span{Un−i }qi=1
2. For each iterate k, until kg(Uk+1 )k < N tol:
(a) Approximately solve the linear system J(Uk ) δUk = −g(Uk )
(b) Update the approximate solution: Uk+1 = Uk + λ δUk
• k · k is a 2-norm, weighted by relative magnitudes of expected solution
• The Jacobian J(U) provides a linear model of g(U) around U:
J(U) ≡
∂
g(U)
∂U
=
∂
∂U
[U − γf (U)] ,
γ ≡ ∆tn βn,0
• Linear subsystems solved with GMRES (nonsymmetric); requires only
Gâteaux derivatives J(Uk )V, which may be approximated as
J(U)V ≈ [g(U + τ V) − g(U)] /τ,
Inexact Newton: [Dembo et al., 1982;
Dennis & Schnabel, 1983;
Implicit resistive MHD: [D.R., Samtaney & Woodward, 2006;
0 < τ 1.
Saad & Schultz, 1986;
Brown & Saad, 1990]
Keyes, D.R. & Woodward, 2006]
UC San Diego, Computational and Applied Math Group
October 22, 2007
(17)
Implicit MHD Preconditioner Acceleration
GMRES convergence rate depends on λ(J): method minimizes the interpolating
polynomial through approximate spectrum at each step.
Instead of solving the original Newton systems J δU = −g, we may instead solve
(JP −1 )(P δU) = −g, with convergence then depending on λ(JP −1 ).
• P may be any approximation to the linearized system that is efficiently
solved.
• The choice of P does not affect the accuracy of the nonlinear solution,
only the convergence properties.
Since MHD stiffness results from fast hyperbolic and diffusive effects, we set
P −1 = Ph−1 Pd−1 = J(U)−1 + O(∆t2 ).
Often termed operator-splitting, and widely used as a stand-alone solver, we
use it to accelerate convergence of our more stable and accurate approach.
UC San Diego, Computational and Applied Math Group
October 22, 2007
(18)
Ideal MHD Preconditioner (Ph )
Ph solves for the fastest wave effects within the implicit hyperbolic system.
Denoting (·) as the location of action for the linear operator, this has Jacobian
Jh (U) = I + ∆t [JF ∂x (·) + JG ∂y (·) + JH ∂z (·)]
h
i
−1
−1
−1
= I + ∆t JF LF LF ∂x (·) + JG LG LG ∂y (·) + JH LH LH ∂z (·)
h
−1
−1
= I + ∆t JF L−1
∂
(L
(·))
+
J
L
∂
(L
(·))
+
J
L
x
y
F
G
G
H
F
G
H ∂z (LH (·))
i
−1
−1
−1
− JF LF ∂x (LF )(·) − JG LG ∂y (LG )(·) − JH LH ∂z (LH )(·)
We then form a directionally-split, O(∆t2 )-accurate preconditioner as
Ph = PF PG PH Ploc
h
ih
ih
i
−1
−1
−1
= I + ∆tJF LF ∂x (LF (·)) I + ∆tJG LG ∂y (LG (·)) I + ∆tJH LH ∂z (LH (·))
h
i
−1
−1
−1
I + ∆t JF LF ∂x (LF ) + ∆t JG LG ∂y (LG ) + ∆t JH LH ∂z (LH ) .
• PF , PG and PH consist of many tridiagonal sub-systems
• Ploc is spatially-local and dense
UC San Diego, Computational and Applied Math Group
October 22, 2007
(19)
Ph Results (Ideal MHD Tests)
5
4
Krylov iterations
10
0
10
No Prec (dtfac = 10)
FW Prec (dtfac = 10)
No Prec (dtfac = 25)
FW Prec (dtfac = 25)
No Prec (dtfac = 50)
FW Prec (dtfac = 50)
No Prec (dtfac = 100)
FW Prec (dtfac = 100)
No Prec (dtfac = 200)
FW Prec (dtfac = 200)
Scaled Runtimes (s)
10
3
10
!1
10
!2
10
2
10
100
No Prec (dtfac = 10)
FW Prec (dtfac = 10)
No Prec (dtfac = 25)
FW Prec (dtfac = 25)
No Prec (dtfac = 50)
FW Prec (dtfac = 50)
No Prec (dtfac = 100)
FW Prec (dtfac = 100)
No Prec (dtfac = 200)
FW Prec (dtfac = 200)
!3
150
200
250
300
350
mesh size
400
450
500
550
10
100
150
1200
1000
250
300
350
mesh size
400
450
500
550
dtfac = 50
dtfac = 100
dtfac = 500
800
Runtimes (s)
Kelvin Helmholtz test results:
total linear iterations (above),
scaled runtimes (top right).
200
600
400
Linear wave advection weak
scaling results (right).
200
0
1
10
UC San Diego, Computational and Applied Math Group
2
10
Processors
October 22, 2007
3
10
(20)
Pd Preconditioning
Pd solves the remaining diffusive effects within the implicit system,
∂t U − ∇ · Fv = 0.
We set Pd to the Jacobian of this system,
∂
Pd = Jv (U) = I − ∆t ∂U
(∇ · Fv )
2
I
0
6
6
0
I − ∆tDρv
6
=6
6
0
0
4
−∆tLρ
−∆tLρv
0
3
0
0
I − ∆tDB
0
−∆tLB
I − ∆tDe
7
7
7
7
7
5
0
• Due to its lower-triangular structure, the Pd−1 operator may be applied
through three small solves and three linear updates.
• The matrices [I − ∆tDρv ], [I − ∆tDB ] are vector-valued Poisson operators;
the matrix [I − ∆tDe ] is a scalar-valued Poisson operator.
• Each subsystem is solved using optimal multigrid methods (HYPRE).
UC San Diego, Computational and Applied Math Group
October 22, 2007
(21)
Initial Pd Results – Diffusive Example
• Diffusion-dominated resistive MHD pellet-injection test problem (η = 1)
• 10 Fixed steps, Crank-Nicholson Implicit (Ntol= 10−7 )
• 2nd-order spatial Centered-Differences
800
Prec
CFLfrac
Newton
163 Mesh, No Prec
Krylov
163
163
N
1
21
30
Y
1
20
20
163
163
N
10
30
128
Y
10
40
49
163
163
N
100
34
680
Y
100
45
140
323
323
N
1
20
20
Y
1
20
20
323
323
N
10
30
153
Y
10
40
40
323
323
N
100
35
785
Y
100
51
160
700
600
163 Mesh, Prec
323 Mesh, No Prec
323 Mesh, Prec
500
Krylov
Mesh
Total Krylov Iterations Per Implicit Simulation
400
300
200
100
0
1
UC San Diego, Computational and Applied Math Group
10
Δ tCFL Fraction
October 22, 2007
100
(22)
Outline
I. Resistive MHD Description and Equations
II. Challenges: Modeling and Computational
III. Current Solution Approach and Results
IV. Future Directions
UC San Diego, Computational and Applied Math Group
October 22, 2007
(23)
Modeling Extensions
• Increased realism requires additional physics:
– Extended MHD (single-fluid model with 2-fluid effects)
– Full 2-fluid momenta and energies
– Highly-anisotropic, nonlinear heat conduction
• Multiscale, multi-model couplings:
– Particle models for edge plasma turbulence
– Radiation and ionization micro-physics for pellet injection
UC San Diego, Computational and Applied Math Group
October 22, 2007
(24)
Solver Improvements
• Current simulations can only handle relatively large η and µ; physical
systems require much smaller values (hence much higher resolutions).
• Multiphysics “all-in-one” preconditioners for incorporating additional
physics:
– Slow is fine so long as it is effective and scalable
– Target applications will eventually be extremely large so implicit
stability with preconditioner scalability is essential
• Constraint-preservation methods for implicit simulations: explicit CT works for
ideal MHD, but stiffness and diffusive effects require implicit stability
• Constraint-preserving preconditioning as
standard approaches may harm constraints
UC San Diego, Computational and Applied Math Group
October 22, 2007
(25)
Other Applications / Software
In addition to fusion modeling and simulation, I work on problems in
• Domain Decomposition solvers for large unstructured linear systems
• Core-collapse supernovae (high energy radiation hydrodynamics, shocks)
• Cosmological reionization (rad-hydro, gravity and chemical kinetics)
• Nonlinear thermodynamics of shape memory alloys (phase transitions)
Computationally, my work uses, and I contribute to, the following software
• SUNDIALS – nonlinear solver and ODE/DAE integrators
http://www.llnl.gov/CASC/sundials
• HYPRE – parallel iterative linear solvers: MG, AMG, ...
http://www.llnl.gov/CASC/linear solvers
• FeTK – finite element toolkit (2D, 3D, Lagrangian)
http://www.fetk.org
• Enzo – cosmological, multiphysics, finite-volume, AMR package
http://lca.ucsd.edu/portal/software/enzo
UC San Diego, Computational and Applied Math Group
October 22, 2007
(26)
Acknowledgements
• Collaborators
– David Keyes, Columbia University
– Michael Holst, U.C. San Diego
– Randy Bank, U.C. San Diego
– Michael Norman, U.C. San Diego (Astrophysics)
– Ravi Samtaney, Princeton Plasma Physics Laboratory
– Doug Swesty, Stony Brook University (Astrophysics)
– Carol Woodward, Lawrence Livermore National Laboratory
• UCSD Mathematics and Physics Departments
– Computational and Applied Math Group
– Center for Astrophysics and Space Sciences
• U.S. Department of Energy, Scientific Discovery through Advanced
Computing Program.
UC San Diego, Computational and Applied Math Group
October 22, 2007
(27)