The Role of High Performance Computation in Economic Development
Workshop at Rensselaer Polytechnic Institute, October 22 -24, 2008
Simulation of Multiphase Magnetohydrodynamic
Flows for Nuclear Fusion Applications
Roman Samulyak
AMS Department, Stony Brook University
and
Computational Science Center
Brookhaven National Laboratory
Collaborators:
Tianshi Lu (BNL), Paul Parks (General Atomics)
Wurigen Bo, Lingling Wu (Stony Brook)
James Glimm, Xiaolin Li (Stony Brook)
Kirk McDonald (Princeton), Harold Kirk (BNL)
Brookhaven Science Associates
U.S. Department of Energy
1
Talk Outline
• Brief description of wide range of applications
• Numerical algorithms for multiphase hydro and MHD flows
• Large scale computing (BlueGene)
• Application specific mathematical models and numerical algorithms
(for one selected applications - tokamak fueling)
• Numerical simulations: main results and feedback to applications
• Summary and conclusions
Brookhaven Science Associates
U.S. Department of Energy
2
Fusion Energy. ITER project: fuel pellet ablation
• ITER is a joint international research and
development project that aims to
demonstrate the scientific and technical
feasibility of fusion power
• ITER will be constructed in Europe, at
Cadarache in the South of France in ~10
years
Our contribution to ITER science:
Models and simulations of tokamak
fueling through the ablation of frozen
D2 pellets
Collaboration with General Atomics
Brookhaven Science Associates
U.S. Department of Energy
3
Laser driven pellet acceleration
New Ideas in Nuclear Fusion: Magnetized Target Fusion (MTF)
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Brookhaven Science Associates
U.S. Department of Energy
4
Fission Energy. DOE Nuclear Energy Research Initiative (NERI)
• Stony Brook University, together with Rensselaer
Polytechnical Institute, Columbia University, and
Brookhaven National Laboratory, joined a DOE funded
consortium within NERI initiative
• Project title: Deployment of a Suite of High Performance
Computational Tools for Multiscale Multiphysics Simulation
of Generation-IV Reactors
Brookhaven Science Associates
U.S. Department of Energy
5
Numerical Studies within NERI Consortium




Simulation of material relocation in a Core Disruptive Accident
(CDA) in Gen. IV reactors
Numerical investigation of the surface instability and liquid
entrainment in the expansion of an CDA bubble.
Studies of some fundamental problems of multiphase flows
Subgrid models for multiphase flows
Brookhaven Science Associates
U.S. Department of Energy
6
Spallation Neutron Source:
Cavitation induced erosion in mercury target
Left: pressure distribution in the SNS
target prototype. Right: Cavitation
induced pitting of the target flange (Los
Alamos experiments)
• Injection of nondissolvable gas bubbles has been proposed as a
pressure mitigation technique.
• Numerical simulations aim to estimate the efficiency of this approach,
explore different flow regimes, and optimize parameters of the system.
Brookhaven Science Associates
U.S. Department of Energy
7
Mercury Jet Target for Neutrino Factory / Muon Collider
Simulation of the mercury jet target interacting with a proton pulse in a
magnetic field
• Studies of surface instabilities, jet breakup, and cavitation
• MHD forces reduce both jet expansion, instabilities, and cavitation

Jet disruptions
Target schematic
Brookhaven Science Associates
U.S. Department of Energy
8
Numerical Algorithms
Common features of all applications:
• Multiphase / free surface hydro and MHD flows interacting with
external energy sources
• Phase transitions
They require:
• Explicitly resolved multiphase (free surface) flows
• Our choice is Front Tracking
• Numerical algorithms for coupled hyperbolic - elliptic systems
in geometrically complex domains
• Large scale computing
Brookhaven Science Associates
U.S. Department of Energy
9
MHD equations and approximations
Full system of MHD equations
Low magnetic Re approximation

     u 
t
1


   u   u  P  u   J  B 
c
 t

1


   u   e   P  u  J 2


 t

Re M 
1


J      u  B 
c


1
        u  B  ,
c

1
with

(u  B)  n
n  c
B  B ext ( x, t ),   B ext  0
 c2

B
   u  B     
B
t
 4

P  P(  , e),
B  0
Brookhaven Science Associates
U.S. Department of Energy
uL
B
1,
1
2
c
B
10
Multiphase MHD
Solving MHD equations (a coupled hyperbolic – elliptic system) in
geometrically complex, evolving domains subject to interface boundary
conditions (which may include phase transition equations)
Material interfaces:
• Discontinuity of density and
physics properties (electrical
conductivity)
• Governed by the Riemann
problem for MHD equations or
phase transition equations
Brookhaven Science Associates
U.S. Department of Energy
11
Main Ideas of Front Tracking
Front Tracking: A hybrid of Eulerian and Lagrangian methods
Two separate grids to describe the solution:
1. A volume filling rectangular mesh
2. An unstructured codimension-1
Lagrangian mesh to represent interface
Major components:
1. Front propagation and redistribution
2. Wave (smooth region) solution
Advantages of explicit interface tracking:
• No numerical interfacial diffusion
• Real physics models for interface propagation
• Different physics / numerical approximations
in domains separated by interfaces
Brookhaven Science Associates
U.S. Department of Energy
12
The FronTier Code (SciDAC ITAPS Software)
FronTier is a parallel 3D multiphysics code based on front tracking
 Physics models include
Compressible fluid dynamics
 MHD
 Flow in porous media
 Elasto-plastic deformations

Realistic EOS models, phase transition models
 Exact and approximate Riemann solvers
 Adaptive mesh refinement

Turbulent fluid mixing.
Left: 2D
Right: 3D (fragment of
the interface)
Brookhaven Science Associates
U.S. Department of Energy
13
FronTier-MHD numerical scheme
Elliptic step
Hyperbolic step
Fijn
Fijn1/ 2
Fijn1
Point Shift (top) or Embedded Boundary (bottom)
in, j 1/ 2
• Propagate interface
(solve Riemann
problem for contact
or phase transition
equations)
• Untangle interface
• Update interface
states
• Apply hyperbolic
solvers
• Update interior
hydro states
Brookhaven Science Associates
U.S. Department of Energy
in1/1/2,2 j
• Generate finite element grid
• Perform mixed finite element discretization
or
• Perform finite volume discretization
• Solve linear system using fast Poisson solvers
• Calculate
electromagnetic
fields
• Update front and
interior states
Normal propagation of interface points
Contact discontinuity
Brookhaven Science Associates
U.S. Department of Energy
15

Phase boundary problem
Interface jump conditions
u  su
u
2
 P  su
uE  uP  T   sE 
Balance equations
uv  ul
1
,  ,
v  l

P  Pl
P  Pl
M2  v
, uv  sul  s  v
,
v  l
v  l
P  Pl
1
el  ev  v
 l   v    vTv,x   l Tl,x 
2
M
M
Temperature and pressure at the interface
Tl  Tv  Ts

QM  1 1 
Psat (Ts )  Pv
M 
, Psat  P0 exp mol   
2RTs
 R T0 Ts 
Subgrid model for temperature

Brookhaven Science Associates
U.S. Department of Energy


x

T  Ts  T1  Ts erf 
 4t / c 
p 

16
Embedded Boundary Elliptic Solver
Main Ideas
• Based on the finite volume discretization
• Domain boundary is embedded in the
rectangular Cartesian grid, and the
solution is treated as a cell-centered
quantity
• Using finite difference for full cell and
linear interpolation for cut cell flux
calculation
• Advantage: robust, readily parallelizable,
compatible with FronTier grid-based
interface tracking algorithm.
Brookhaven Science Associates
U.S. Department of Energy
17
High Performance Computing
• We are interested in the development of software for
parallel distributed memory supercomputers
• Efficient parallelization
• Scalability to thousands of processors
• State-of-art parallel visualization
• Code portability (for use of local and external (NERSC)
System
(64 cabinets, 64x32x32)
computational resources)
Cabinet
(32 Node boards, 8x8x16)
Node Board
(32 chips, 4x4x2)
16 Compute Cards
Compute Card
(2 chips, 2x1x1)
180/360 TF/s
16 TB DDR
Chip
(2 processors)
Brookhaven Science Associates
U.S. Department of Energy
90/180 GF/s
8 GB DDR
2.8/5.6 GF/s
4 MB
5.6/11.2 GF/s
0.5 GB DDR
18
2.9/5.7 TF/s
256 GB DDR
Application Specific Models
and Simulations
Brookhaven Science Associates
U.S. Department of Energy
19
Pellet Ablation for Tokamak Fueling: Main Models
• Equation of state with atomic processes
• Kinetic model for the interaction of hot electrons with the ablated gas
• Surface ablation model
• Cloud charging and rotation models
• New conductivity model (ionization by electron impact)
Schematic of
processes in the
ablation cloud
Brookhaven Science Associates
U.S. Department of Energy
20
Relation to Other Projects
Macroscale Model: AMR
simulation of ablation flow in
plasma (Ravi Samtaney)
• Focus on plasma flow
• Ablation physics not resolved
(simplified analytical models)
Brookhaven Science Associates
U.S. Department of Energy
Local (“Microscale”) Model: Front
Tracking simulation of pellet ablation
• Focus on detailed ablation physics,
ablation rates etc.
• Far field plasma evolution not
resolved
21
Equation of State with Atomic Processes.
Let's define:
nt  2ng  na  ni   / m
nt is the total number density of nuclei
ng is the number density gas molecules D2
Dissociation (ionization) fractions:
na is the number density atoms D
f d   na  ni  / nt ,
ni is the number density of ions D +
Saha equation for the
dissociation (ionization) fraction
For deuterium:
ed = 4.48 eV
f d2
T  d  ed / T
 Nd
e
1  fd
nt
2
ei = 13.6 eV
N d  1.55 1024 ,  d  0.327
i
fi
T
 i / T

N
e
Brookhaven Science
Associates
i
1 U.S.
f i Department
nt of Energy
fi  ni / nt
22
Ni  3.0 1021 ,  i  3 / 2
EOS with Atomic Processes
1 1
  kT
Incomplete EOS
P    f d  fi 
(known from literature):
2 2
 m
 1  fd
f d  fi  kT 1
ked
kei
E 

 fd
 fi

 1  m 2
m
m
 2( m  1)
High resolution solvers (based on the Riemann problem) require the sound speed
and integrals of Riemann invariant type expressions along isentropes. Therefore
the complete EOS is needed.
T dS  dE  PdV
Using the second law of thermodynamics
we found the complete EOS and showed that the compatibility with the second
law of thermodynamics requires:
d  3 
Brookhaven Science Associates
U.S. Department of Energy
1
3
, i 
 m 1
2
23
Complete EOS with Atomic Processes
Notations:
1 f d 1  f d 
d 
2 2  fd
i 
fi 1  fi 
2  fi
ed
T
e
i   i  i
T
d   d 
1  fd
2
a  f d  fi
m
We will define the sound speed in a form typical for the polytropic gas:
where the effective gamma is
c 2   * pV
the Gruneisen coefficient is
     
(m  a) 1  d d i i 
 
ma 

 * 1 
 d i
1
3
m  a  d  d2  i i2 m  a
 m 1
2
2

m  a  d  d  i i
1
3
m  a  d  d2  i  i2
 m 1
2
ln 1  f d 
S
ln T
ln V 1   d



f d  1  i  fi 
 ln 1  fi 
and the entropy is
R
2(


1)
2
2
2
Brookhaven Science Associates
m
U.S. Department of Energy
24
Numerical Algorithms for EOS
For better numerical efficiency, FronTier operates with three pairs of
independent thermodynamic variables:  , E ,
, P , ,T

 
 

• For the first two pairs of variables, solve numerically nonlinear
algebraic equation, and find T. Using   ,T  , find the remaining state.
• Such an approach is prohibitively slow for the calculation of Riemann
integrals (involves nested nonlinear equations).
• To speedup calculations, we precompute and store values on Riemann
integrals as functions of the density and entropy. Two dimensional table
lookup and bi-linear interpolation are used.
Brookhaven Science Associates
U.S. Department of Energy
25
Influence of Atomic Processes on Temperature and Conductivity
9.675 103
 mho 
 

,
3/ 2
0.059

 0.54Te [eV ]
1/ fi  1
 m  ln Te [eV ]
9 3/ 2
3.6

10
Te
Brookhaven Science Associates
where  
U.S. Department of Energy
ne1/ 2
26
Electron Energy Deposition
z
u ( r , z ) 


n(r , z ')dz '


u ( r , z ) 

n(r , z ')dz '

z
In the cloud:
q n( r , z )
q  
 g (u )  g (u )

g (u )  uK1
On the pellet surface:
q  quK 2
 u /4
Brookhaven Science Associates
U.S. Department of Energy
Te2
 
8 e4 ln (n, Te , fi )
27
 u /2
Pellet charging model
2

 Jhot
  ||
2
    || 2 
 Bzˆ  [  ( u)]      
z
z z
z
 

 J||,sheath( )(z  nˆ )      u  B nˆ  0

||
Brookhaven
z Science Associates 
U.S. Department of Energy
28
Physics Models for Pellet Studies :
Surface Ablation
Features of the pellet ablation:
• The pellet is effectively shielded from incoming electrons by its ablation cloud
• Processes in the ablation cloud define the ablation rate, not details of the phase
transition on the pellet surface
• No need to couple to acoustic waves in the solid/liquid pellet
• The pellet surface is in the super-critical state
• As a result, there is not even well defined phase boundary, vapor pressure etc.
This justifies the use of a simplified model:
• Mass flux is given by the energy balance (incoming electron flux) at constant
temperature
• Pressure on the surface is defined through the connection to interior states by the
Riemann wave curve
• Density is found from the EOS
Brookhaven Science Associates
U.S. Department of Energy
p
p
u 
q
 u
 u  c   c   u  c    
t
n
n 
z
 t
29
Spherically symmetric simulation
Polytropic EOS
Plasma EOS
Normalized ablation gas profiles at 10 microseconds
Poly EOS
Plasma EOS
Sonic radius
0.66 cm
0.45 cm
Temperature
5.51 eV
1.07 eV
Pressure
20.0 bar
26.9 bar
Brookhaven
Associates
Ablation
rate Science112
g/s
106 g/s
U.S. Department of Energy
• Excellent agreement with TF model
and Ishizaki.
• Verified scaling laws of the TF model
G ~ rp4 / 3

7 
for   
30 M    1.8898 

5 
5
Axially Symmetric Hydrodynamic Simulation
Temperature, eV
Pressure, bar
Mach number
Distributions of temperature, pressure, and Mach number of the ablation flow near
the pellet at 20 microseconds.
Clarified the role of directional electron beam heating on the ablation rate.
Brookhaven Science Associates
U.S. Department of Energy
31
Performed first systematic studies of pellet ablation rates in
magnetic fields
3 microseconds
1 microsecond
5 microseconds
Brookhaven Science Associates
U.S. Department of Energy
3 microseconds
329 microseconds
Cloud Rotation
• Revealed new propertied of the ablation flow: Supersonic rotation of the
ablation channel
• Supersonic rotation widens the ablation channel, re-distributes
density, and changes the ablation rate
• Resolution of this phenomenon greatly improves the agreement with
experiments
Isosurfaces of the rotational Mach
number in the pellet ablation flow
Brookhaven Science Associates
U.S. Department of Energy
33
Dependence on Pedestal Properties
Critical observation:
• Formation of the ablation channel and ablation rate
strongly depends on plasma pedestal properties and
pellet velocity
• Simulations suggest that novel pellet acceleration
technique (laser or gyrotron driven) are necessary for ITER
Schematic of the plasma
pedestal. Plasma
pedestal provides critical
influence to the pellet
ablation rate
Brookhaven Science Associates
U.S. Department of Energy
34
Work in Progress
• Current work focuses on the study of striation instabilities
• Striation instabilities, observed in all experiments, are not
well understood
• We believe that the key process causing striation
instabilities is the supersonic channel rotation, observed in
our simulations
Striation instabilities:
Experimental observation
(Courtesy MIT Fusion Group)
Brookhaven Science Associates
U.S. Department of Energy
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
35
Neutrino Factory / Muon Collider target
Simulation tasks:
• Studies of the jet formation in the nozzle
• Entrance of the jet into the magnetic field
• Mercury jet - proton pulse interaction. Jet cavitation
and disruption, stabilizing effect of the magnetic field
• Predictions for the CERN targetry experiment called
MERIT (Fall of 2007)
Brookhaven Science Associates
U.S. Department of Energy
36
Jet entering 15 T solenoid
Brookhaven Science Associates
U.S. Department of Energy
37
Aspect ratio of the jet cross-section
B = 15 T
V0 = 25 m/s
These studies has resulted in the change of MERIT design. The chamber
was re-designed to allow much smaller angle with of the jet with the
magnetic field.
Brookhaven Science Associates
U.S. Department of Energy
38
Experimental data
V = 15 m/s, B = 10T
V = 20 m/s, B = 10T
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
B = 15T
B = 15T
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Simulations predicted slightly smaller value of the jet width in different
view ports.
Brookhaven Science Associates
U.S. Department of Energy
39
Liquid jet cavitation
Brookhaven Science Associates
U.S. Department of Energy
40
Interaction of the mercury jet target with proton pulses
3D FronTier simulations of the
disruption of Hg jet interacting
with a proton pulse.
Brookhaven Science Associates
U.S. Department of Energy
41
Summary of Target Studies
Accomplishments
• Modeling and simulation of the mercury jet target
• Change design parameters of the MERIT experiments
• Successful predictions for MERIT: jet flattening in high gradient
field, jet disruption by proton pulses, stabilizing effect of the
magnetic field
Significance
• Neutrino Factory Collaboration reached the conclusion that
liquid mercury jet targets can successfully work with proton
beams up to 8 MW
Brookhaven Science Associates
U.S. Department of Energy
42