The Center for Astrophysical Thermonuclear Flashes
An Overview of the ASCI/Alliances Program at
Chicago:
Year 3 Site Review
Robert Rosner, Director
October 30, 2000
Chicago, IL
(http://flash.uchicago.edu)
An Accelerated Strategic Computing Initiative (ASCI)
Academic Strategic Alliances Program (ASAP) Center
at The University of Chicago
What is the FLASH Problem?
 To simulate matter accumulation on the surface of
compact stars, nuclear ignition of the accreted (and
possibly underlying stellar) material, and the
subsequent evolution of the star’s interior, surface, and
exterior
• X-ray bursts (on neutron star surfaces)
• Novae (on white dwarf surfaces)
• Type Ia supernovae (in white dwarf interiors)
Credit: NASA/STScI
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Observations of Astrophysical Flashes
X-ray burst: GS 1826-24
Nova: Nova Cygni 1992
Credit: NASA/Rossi XTE
Type Ia SN: SN 1994D
Credit: NASA/STScI
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Credit: NASA/STScI
Schematics of the astrophysical problems
H, He
Type Ia supernova
White dwarf
White dwarf: R ~ REarth
(C, O)
Neutron star: R ~ RChicago
(Neutrons)
White dwarf/neutron star
X-ray burst (neutron star)
Nova (white dwarf)
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Why are these interesting problems - Part 1
 Defense programs
 Paradigm for attacking physics problem without ability to
experiment
 Training of the next generation of computational
scientists
 FLASH is the only Center which includes MHD
 Astrophysics
 Endpoints of accretion
 Nucleosynthesis of elements
 Cosmology
 Physics
 Interface instabilities, mixing
 Fundamentals of combustion
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Why are these interesting problems - Part 2
 Computational Science
 Integration of complex physics
 Exploration of grids, hydro algorithms, ...
 Practical application of OO-based programming
paradigm
 Sociology
 Mixing astrophysicists, physicists, applied
mathematicians, computer scientists …
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
What are the FLASH Center’s Goals?
To build a new-generation simulation code for
computing astrophysical thermonuclear
flashes involving compact stars
 Incorporate advances from CS, applied math,
physics
 modular,
reusable code
 Verify and validate code and its components
To produce great science
 Astrophysics and physics
 Computational sciences [CS, Applied math, Math]
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
The Challenges
 Extreme dynamic ranges of spatial and temporal
phenomena
 AMR, subgrid modeling, front tracking
 Large dynamic range of flow velocities
 Accurate computation of very low and high Mach number flows
 Physical processes can be highly localized and nonuniform
 Load balancing
 Physical processes can be non-local
 Example: Elliptic solvers on AMR grid
 Rapidly evolving computing environment
 Creation of programming and computing environment
 Astrophysical validation is extremely indirect
 Development of laboratory proxy validation experiments
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
What have we accomplished to date?
 We are on track: we have a newly architected code
 Production version: Flash-1.61
 OO-based data structures introduced: Flash-2 (in test)
 We have started on all three astrophysics problems
 We responded to past Site Review suggestions
 Management changes
 Code architecture
 We have obtained exciting science results
(Y. Young 2000)
(F. Timmes 2000)
(A. Calder 2000)
(M. ZIngale 2000)
(N. Vladimirova 2000)
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
What you’ll hear about today
Talks
Flash Code: B. Fryxell, P. Ricker, A. Siegel
Astrophysics: J. Truran
Basic Science/Validation: T. Dupont
Computer Science: E. Lusk
Posters
Astro, CS, Math, Physics
Demos
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
What’s New in Year 3: Part 1
 Re-organization
 Code group and Computational Physics combined
 Code architecture team formed within Code group: Code architect!
 Management team tightened
 T. Dupont (Basic Sci/Validation), B. Fryxell (Code), E. Lusk (CS), R.
Rosner (director), R. Stevens (ex officio), J. Truran (Astro)
 Weekly management team meeting
 Focus on re-engineering Flash
 Re-engineered OO-based architecture: Flash-1.6
 Current optimized production version: Flash-1.61
 Includes improved version of Paramesh
 New features incorporated in Flash-2.0 (now in active test)
 OO-based data structures and data-sharing: Paves the way for
extensibility of Flash
 Python-based user interface, setup, automatic test facility, tools, …
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
What’s New in Year 3: Part 2
 Focus on basic science
 Cellular detonations
 Interface mixing: R-T/R-M, wave breaking
 Flame propagation/modeling
 MHD
 Rad. transfer
 Implicit conduction
 Self-consistent gravity
 Focus on astrophysics
 X-ray bursts
 Novae
 Type Ia supernovae
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
How was this done: Our strategy
 Initial instantiation of Flash Code: Flash 1.0
 Legacy hydro solver (“Prometheus”/F77)
 New adaptive mesh package (“Paramesh”/F90)
 New physics packages (nuclear reaction networks, EOS/F90)
“Classic” modular design
Allowed first-cut astrophysics and validation calculations
Guided refinement of code architecture
“Crude, but useful”
Dick Watson (1999)
 New architecture code
 OO-based code architecture/framework




Optimized for production
New module interface design
Allows continued use as applications code
Gradual transition to/introduction of object-oriented programming
Physics modules
Automatic code verification/test facility
New data sharing scheme
Flash-2.0
Revised user interface
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Flash-1.61
How was this done: Our Center’s structure
Working
Groups
Ad hoc groups
Sub-grid turbulence,
combustion,
front tracking
Astrophysics
J. Truran
Basic Science
& Validation
R. Lusk
Astrophysics modules,
Astrophysics
simulations and
validation
Validation experiments,
Fundamental physics
Architecture
team
A. Siegel
Development,
Maintenance,
& Testing
Physics
Modules
Tool kits, Libraries,
Visualization
Arnett, Brown,Calder,
Dursi, Fryxell, Lamb,
Mignone, Peng,
Ricker, Timmes, Zingale
Alexakis, Constantin,
Curtis, Draganescu,
Kadanoff, Litwin, Liu,
Medved, Oberman,
Ruchayski,
Vladimirova, Y. Young
Caceres, Ricker,
Reilly,
Vladimirova,
K. Young
Calder, Dursi,
Olson, Ricker,
Timmes, Tufo,
Zingale
Calder, Linde,
Mignone,
Olson, Ricker,
Timmes, Tufo,
Weirs, Zingale
Chan, Clark,
Curfman-McInnes,
Foster, Freitag,
Flaherty, Gomez,
Gropp, Hudson,
Loy, Papka
Scott, Shephard,
Smith, Stevens, Tufo
Cattaneo, Constantin, Dupont, Dursi, Fryxell, Malagoli,
Oberman, Rosner, Vladimirova, Weirs, Young
Curfman-McInnes, Flaherty, Freitag, Fryxell, Malagoli,
Olson, Ricker, Scott, Shephard, Siegel, Tufo
Alexakis, Calder, Cattaneo, Constantin, Dupont, Flaherty, Fryxell, Kadanoff, Litwin, Niemeyer,
Oberman, Olson, Rosner, Scott, Shephard, Tufo, Young
Clark, Chan, Freitag, Fryxell, Gomez, Gropp, Lusk, Olson,
Siegel, Smith, Timmes, Tufo, Zingale
Performance/tuning,
scaling
Collaborators
B. Fryxell
T. Dupont
Code Architecture,
grids/AMR
Model problems
(R-T, …)
Computer
Science
FLASH Code
LANL, LLNL, MPI
UCSC,
SUNY, UA, NWU
LANL, LLNL,
SNL, PPPL
GSFC
LANL, LLNL,
SNL
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
PSU,
LANL, LLNL,
SNL
How was this done: Effective interactions
Production Code
Astrophys. modules
Astrophysics Group
Monday 2:00pm Astro meeting
Wednesday 10:00am Flames
Wednesday 11:00am MHD
Flash Code
Group
Production Code
Validation, models
Basic Science/Validation Group
Wednesday 12:30pm Comp. Sci seminar
Monday
3:00pm
Code/CS
meeting
Code
Vis., Tools, Architect.
Computer Science Group
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Responses to Year 2 Site Review: Part 1
 Concerns regarding progress on Flash code
 Decision regarding Flash architecture
 Re-architected Flash exists! See talks by B. Fryxell/P. Ricker/A. Siegel
 “Insertion plan”
 Plan for module insertion in place and being exercised; see talk by A.
Siegel
 Concerns regarding physics modules
 Nuclear reactions
 See talk by J. Truran
 Radiative transport
 See poster by A. Calder
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Responses to Year 2 Site Review: Part 2
 Concerns regarding Center management and staffing
 Management structure tightened
 Code architect
 A. Siegel (Flash-2 on)
 P. Ricker (Basic Flash architecture)
 Code group leader
 B. Fryxell
 Executive director
 Center consensus: position not needed
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Training the next generation ...
Name
Field
Present position
fluid dynamics
algorithms
astro/theory
fluid dynamics
fluid dynamics
astro/theory
(Math) postdoc/SUNY Stony Brook, winter 1999-00
(CS) staff member/ANL, fall 1999
(A&A) postdoc /A&A, fall 2000
(A&A) postdoc at Northwestern/Applied Math, fall 2000
(Physics) postdoc, A. Kerstein/Sandia/Livermore, fall 1998
(A&A) financial company, spring 2000
MHD
fluid dynamics
comp. physics
fluid dynamics
algorithms
astro/theory
parallel tools
MHD
combustion
visualization
comp. Astro
fluid dynamics
student/physics
(A&A) postdoc at RPI/Applied Math, fall 2000
student/physics
student/physics
student/mathematics
student/A&A
Sandia/summer 2000
(CS) assoc. professor at UC San Bernardino, fall 2000
student/A&A
student/mathematics
Sandia/summer 2000
student/CS & staff member/ANL
student/A&A
student/physics
Graduates:
Y. Liu
R. Loy
M. Zingale
Y.N. Young
S. Wunsch
S. Zhan
Current:
A. Alexaksis
J. Biello
A. Caceres
J. Curtis
A. Draganescu
J. Dursi
E. Gomez
A. Mignone
A. Oberman
M. Papka
F. Peng
O. Ruchayskiy
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
The Future of Flash: The larger picture
1. Follow the FLASH Center plan
 Continue Flash code development: push extensibility
 Pursue Flash development for the 3 astrophysics problems
 Pursue Astrophysics, Basic Science/Validation, and Computer
Science activities as planned
2. Up the ante …go beyond our earlier ambitions
 Include new physics features not previously planned on
 Two-fluid hydro/MHD
 N-body
 Attack FLASH-related astrophysics problems not in original
FLASH Center plan

Accretion flows and disks
 Stellar evolution (including star formation)
 Explore new ideas from CS and computational science
 Exploratory/collaborative computing
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
A final comment and summary
 Project oversight by DOE has been productive
 First-rate scientists appointed as project managers
 Project oriented toward helping Centers succeed
 Annual external site review
 Annual DOE-only site visit
 Constant interactions on science
 We have succeeded in getting where we wanted to be,
and in some cases, are further along than anticipated
 New architecture in place, and producing many science results
 All astrophysics problems have been started, with astro. results
 Center structure has evolved to function effectively
 The challenges
 Effective validation and model building is very hard …
 Available computational resources no longer match our needs
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
… and that leads us to
QUESTIONS AND DISCUSSION
The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago