The Center for Astrophysical Thermonuclear Flashes
What’s FLASH?
How does FLASH do it?
Katherine M Riley
Code Group
An Advanced Simulation & Computing (ASC)
Academic Strategic Alliances Program (ASAP) Center
at The University of Chicago
Overview
What is FLASH
Basics of FLASH2 Architecture
Basics behind a problem setup
Walk away with:
 Enough of an idea of these concepts to be
able to look at a sample setup and
understand what is happening.
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
FLASH: The Application
 To simulate matter accreted onto the surfaces of compact stars, nuclear
ignition of the accumulated (and possibly stellar) material, and
subsequent evolution of the star’s interior, surface, and exterior.
 Novae (on white dwarf surfaces)
 Type 1a supernovae (in white dwarf interiors)
 X-ray bursts (on neutron star surfaces)
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
FLASH Results





Compressible reactive flow
Wide range of length of time scales
Many interacting physical processes
Only indirect validation possible for the
astrophysics
Many people in collaboration
Flame-vortex interactions
Compressible turbulence
Shocked cylinder
Nova outbursts on white dwarfs
Intracluster interactions
Cellular detonations
Helium burning on neutron stars
White Dwarf deflagration
Rayleigh-Taylor instability
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
What FLASH Provides
Physics
Infrastructure
 Hydrodynamics
 Setup
 PPM
 AMR: Paramesh
 MHD
 Regular testing
 Relativistic
PPM
 Nuclear Physics
 Gravity
 Cosmology
 Particles
 Parallel I/O

hdf5, pnetcdf,
 Profiling
 Runtime and post-
processing
visualization
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
A Little FLASH History
BAM
 FLASH0
 Paramesh2, Prometheus and EOS/Burn
 FLASH1
 Smoothing out the smash
 First form of module architecture & inheritance
 FLASH2
 Untangle modules from each other (Grid)
 dBase
 Concept of levels of users
 FLASH3
 Stricter interface control & module architecture
 Taming the database
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
FLASH2 Audiences
FLASH
Application
Developer
• Works on just
about everything
• Grid development
• Data access
• Architecture
Application
Programmer
• Develop physics
modules
• Talk to grid
• Module
communication
End User
• Initialize setup
• Boundary
conditions
• Basic data
structure
• (Might add a
kernel)
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
FLASH2 Code Basics
An application code, composed of
units/modules. Particular modules are set up
together to run different physics problems.
Performance, Testing, Usability, Portability
Fortran, C, Python, …
 560,000* lines of code
 75% code, 25% comment
Very portable
Scaling to 1000’s of procs
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
* Internal Release
Basic Computational Unit : Block
 The adaptive grid is
composed of blocks
 All blocks: same
dimensions
 Cover different fraction
of the physical domain.
 Kevin Olson will talk
about this more.
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
First Look at FLASH
‘The tar-ball’
‘source’ directories
 modules or groups of modules
Post-processing tools, docs, setups
Setup
FLASH architecture tool
Selects and sets up these modules
 Collecting variables, runtime parameters, etc
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Structure of FLASH
Modules
(not exact!)
Database
Mesh
dBase_init()
dBaseGetData()
dBasePutData()
dBaseProperty()
mesh_init()
mesh_guardcells()
mesh_updateRef()
mesh_fluxConserve()
IO
MHD
Driver
Visualization
Particles
init()
Checkpoint_write()
Checkpoint_read()
init()
init()
dBaseGetData()
dBasePutData()
dBaseProperty()
init()
render()
init()
advance()
Cosmology
Hydro
Gravity
Source_terms
Materials
init()
init()
tstep()
hydro3d()
init()
tstep()
grav3d()
init()
tstep()
src_terms()
eos3d()
eos1d()
eos()
Explicit
PPM
mhd
Unsplit
constant
burn
point_mass
iso13
Helmholtz
Poisson
...
...
cool
Diffuse
WENO
PPM
Multigrid
Gamma
Multipole
heat
What’s a FLASH Module?
FLASH basic architecture unit: Modules
 Component of the FLASH code providing a
particular functionality
 Different combinations of modules are used for
particular problem setups
 Ex: driver, hydro, mesh, dBase, I/O
Fake inheritance by use of directory structure
Modules communicate
 Driver
 Variable Database
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Abstract FLASH2 Module
FLASH Component
1. Meta-data (Configuration Info)
• Interface with driver and setup
• Variable/parameter registration
• Variable attributes
• Module Requirements
2. Interface Wrapper
• Exchange with variable database
• Prep data for kernels
FLASH Application
driver
Database
mesh
3. Physics Kernel(s)
• Single patch, single proc functions
written in any language
• Can be sub-classed
Collection of Flash2 Modules
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Module Implementations
FLASH2 Modules are directory trees
source/hydro/explicit/split/ppm
Each level might have source
Source relevant for all
directories/implementations below
Preserves interfaces
Allows flexible implementations
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Inheritance Through Directories: Hydro
init
hydro
An ‘empty’ hydro
init, hydro, tstep are defaults
on top of the directory tree.
tstep
Explicit
Hydro/Explicit
Replaces tstep
Introduces ‘shock’
No hydro Implemented yet!
tstep
shock
Split
hydro
implemtation
DeltaForm
init
Hydro/Explicit/Split
hydro implemented
Uses general explicit tstep
Uses general shock
Replaces init
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
The Module Config File
Declare solution variables, fluxes
Declare runtime parameters
Sets defaults
Lists required, exclusive modules
Config files are additive down the
directory tree - no replacements
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Setup Building an Application
Configuration
Tool
(Setup)
Source
Terms
Materials
Gravity
MHD
Database Mesh
Driver
Hydro
Particles
I/O
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Vis
FLASH Setup: Implements Architecture
Python code links together needed
physics and tools for a problem
object
Traverses modules to get
implementations
Determines solution data storage list
Creates list of parameters from modules
Configures Makefiles properly
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Accessing the Mesh
Physics and tools access some scope of mesh
Database
 Unifies method for module to access data out of its
scope
 Mesh data
 Solution
Data, Grid information
 Runtime parameters
Mesh Interface
 Wrappers to the public grid routines
 initialize,
guardcell, fluxconserve, updateRefinment
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
FLASH Audiences
FLASH
Application
Solidify information so far
• Modules and setup
• How do they combine
to make create an
application? That runs?
• Go through a little of
that.
End User
• Initialize setup
• Boundary
conditions
• Basic data
structure
• (Might add a
kernel)
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Simple FLASH Flow
Function
Data
Variable Database
Driver
Init Evolve TimeStep
Evolve
Hydro
SourceTerms
…
UpdateGrid
Hydro
Prepare data for kernels
hydroBlock
Paramesh
dBase
Mesh
updateGrid
ghostCellFill
fluxConserve
SourceTerms
Burn Heat Cool Ioniz…
Burn
burnBlock
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Pulling it All Together
Choose a problem setup
Run setup to configure that problem
 Everything is in a new top-level directory
 ‘object’
Make
Run
 Flash.par for runtime parameters
 Defaults already set from particular modules
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Setups
A basic problem setup
 Config file
 Required
physics modules
 Flash.par
 Default
list runtime parameter configuration
 Init_block
 Initial
conditions for the problem set block by block
 Many other possible files:

Driver, Refinement algorithms, User defined boundary
conditions
Any files in setup take precedence
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Provided Driver
New drivers
Provided:
Second order,
state form, strang
split
 Put in setups
 Welcome
contributions
evolve.F90
flash.F90
Initialize()
Loop over timesteps
evolvePhysics()
timestep()
output()
visualize()
End loop
Finalize()
set time step
hydro
sourceTerms
cosmology
radiation
particles
gravity
set time step
(repeat physics)
Mesh_updateGrid
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
What’s Next
Covered the basics of what FLASH can do
Now - How does it do it?
 Alan Calder
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago