Code B: US3D
During the past twenty years, Candler and his collaborators have been developing implicit
methods for the simulation of hypervelocity nonequilibrium flows. This work started with singleblock grids for two-dimensional and axisymmetric flows on serial vector computers; it then
moved to scalable implicit methods for parallel computers, and to a full three-dimensional multiblock grid capability (the NASA DPLR code [37] was originally developed by the Candler group).
During the past seven years, the Candler group has been developing a next generation implicit
parallel line-relaxation method for use with unstructured grids [13,14]. This new code,
UnStructured 3D (US3D), is now fully functional and is being used for a wide variety of
transonic, supersonic and hypersonic flow simulations.
The main features of the US3D code are:
1. The code uses an unstructured grid data structure. A standard grid partitioning approach
is used and MPI protocols are used to communicate data across partition boundaries.
Grids composed of hexahedra, prisms, pyramids and/or tetrahedra may be used.
2. It has a wide variety of numerical flux functions for compressible flows; these include a
form of flux vector splitting that is widely used in the re-entry flow community and a novel
high-order (up to 6th order on hexahedral grids) low-dissipation kinetic energy consistent
method [33].
3. It has many different time integration methods, including implicit line-relaxation for highReynolds number wall-bounded flows, second-order point-implicit methods that are
useful for many classes of large-eddy simulations, 2nd and 3rd order Diagonally Implicit
Runge-Kutta (DIRK) methods, and standard high-order Runge-Kutta methods. We are
experimenting with implicit-explicit (IMEX) methods, which may have utility for the ICF
problem.
4. US3D has a variety of high-temperature and non-ideal gas thermochemical models and
associated boundary conditions. It includes finite-rate internal energy relaxation and
chemical kinetics models for standard gas mixtures relevant to re-entry flows and
combustion. It is straight-forward to extend the thermochemical models to other gases.
5. The code was designed for scaling to large problems – at least within the compressible
aerodynamics community. We regularly run on 4000 cores with 500 million unstructured
hexahedral elements; we have not had access to larger machines, but we anticipate that
code will continue to scale well beyond this size.
6. Great care has been taken to make the code as efficient as possible (at least within a
conventional grid partitioning and programming approach). We have shown that US3D is
approximately 50% faster than the structured multi-block NASA DPLR code.
7. US3D has been extensively used for DNS and LES of turbulent flows; examples will be
discussed below. It includes a variety of subgrid-scale models and wall-modeled LES
approaches. The high-order methods have been validated for a range of high-speed
turbulent flow applications.
8. The I/O, post-processing, and data-management/analysis tools for statistical analysis
were designed for large-scale simulations.
The Candler group has been performing hybrid RANS-LES simulations since 2004. The work
started with detached eddy simulations (DES) of supersonic base flows with and without bleed
[29, 30], jets in supersonic cross flow [15-17, 21-23, 34, 35], supersonic combustors [19, 20],
the base region of the re-entry flight experiments [3, 26], CEV capsule base flows [2, 4], and the
aerodynamics of spinning projectiles [8].
To get a sense of our present hybrid RANS-LES simulation capability, let us focus on the recent
combustion duct simulations [36]. The flow conditions are representative of flight and
correspond to experiments being conducted in the CUBRC Inc. LENS shock tunnel facility at
about Mach 7. In this work, we use the improved delayed detached-eddy simulation (IDDES)
formulation of Shur et al. [25] with the compressibility-corrected [7] one-equation SpalartAllmaras model [27] as the background RANS model. The simulation was performed in three
steps: first a RANS simulation of the inlet was used to provide mean turbulent boundary layer
data to an unsteady simulation of the isolator. Data were collected and used as inflow to the
injector region simulation, which was performed on a grid composed of 137.5 million elements.
The near-wall spacing wa
were approximately 14 cells per injector diameter;
a 10 ns time step was used resulting in a global CFL of 25 and a local CFL of about 0.01 in the
active region of the flow. This calculation was run on 1200 2.8 GHz Nehalem cores, and
required 13.8 hours per domain flow-through time (corresponding to 90 injector diameters).
Figure 1 shows several slices with injectant mole fraction (hydrogen) in color and a gray-scale
showing density gradient magnitude. This flow field is extremely dynamic with complex shock
interactions and highly unsteady and intermittent plume behavior.
As far as we understand, these simulations represent the state-of-the-art of hybrid RANS-LES
for high-speed air-breathing propulsion systems. We have also performed finite-rate hydrogen
combustion simulations for this flow field and for the NASA Langley SCHOLAR experiment [19]
using a laminar flame model (no chemistry-turbulence interaction). Because of space
constraints we have not included comparisons with experimental data, but this approach has
been shown to reproduce the fuel-air measurements made by Lin et al. [11], as well as LDV and
PIV data [24].
We should also mention that US3D is being used at Sandia National Labs for the simulation of
ablating re-entry vehicles (POC: David Kuntz).
Figure 1. Instantaneous injectant mole fraction for CUBRC Combustion Duct with 5 injectors
operating; gray-scale is density gradient magnitude [36].
[1] Ascher, U.M., S.J. Ruuth and R.J. Spiteri, “Implicit-Explicit Runge-Kutta Methods for Time
Dependent Partial Differential Equations,” Applied Numerical Mathematics, Vol. 25, pp. 151167, 1996.
[2] Barnhardt, M., and G.V. Candler, "Detached Eddy Simulation of Hypersonic Base Flows
During Atmospheric Entry," AIAA 2006-3575, June 2006.
[3] Barnhardt, M., and G.V. Candler, "Detached Eddy Simulation of the Reentry-F Flight
Experiment," AIAA-2008-625, Jan. 2008; accepted in Journal of Spacecraft and Rockets.
[3] Barnhardt, M., G.V. Candler, M. MacLean, "CFD Analysis of CUBRC Base Flow
Experiments," AIAA-2010-1250, Jan. 2010.
[4] Barth, T. and P.O. Frederickson, “Higher Order Solution of the Euler Equations on
Unstructured Grids using Quadratic Reconstruction,” AIAA Paper 1990-0013, January 1990
[5] Brock, J., P.K. Subbareddy, and G.V. Candler, “Numerical Simulation of Hypersonic Base
Flow,” AIAA-2011-4028, June 2011.
[6] Candler, G.V., "Unstructured Grid Approaches for Accurate Aeroheating Simulations," AIAA2007-3959, June 2007.
[7] Catris, S., and Aupoix, B., “Density Corrections for Turbulence Models,” Aerospace Science
and Technology, Vol. 4, No. 1, 2000, pp. 1-11.
[8] Doraiswamy, S., and G.V. Candler, "DES and RANS Calculations of a Spinning Projectile,"
Journal of Spacecraft and Rockets, Vol. 45, No. 5, pp. 935-945, Sept.-Oct. 2008.
[9] Ikeda, T. and P. Durbin, “Mesh Stretch Effects on Convection in Flow Simulations,” Journal
of Computational Physics, Vol. 199, pp. 110-125, 2004.
[10] Kennedy, C.A., and M.H. Carpenter, “Additive Runge-Kutta Schemes for ConvectionDiffusion-Reaction Equations,” Applied Numerical Mathematics, Vol. 44, pp. 139-181, 2003.
[11] Lin, K-C, Ryan, M., Carter, C., Gruber, M., and Raffoul, C., “Raman Scattering
Measurements of Gaseous Ethylene Jets in Mach 2 Supersonic Crossflow,” Journal of
Propulsion and Power, Vol. 26, No. 3, May-June 2010, pp. 503-513.
[12] Mavriplis, D.J., “Unstructured Mesh Discretizations and Solvers for Computational
Aerodynamics,” AIAA Paper 2007-3955, June 2007.
[13] Nompelis, I., T.W. Drayna, and G.V. Candler, “Development of a Hybrid Unstructured
Implicit Solver for the Simulation of Reacting Flows Over Complex Geometries,” AIAA-20042227, June 2004.
[14] Nompelis, I., T.W. Drayna, and G.V. Candler, “A Parallel Unstructured Implicit Solver for
Hypersonic Reacting Flow Simulation,” AIAA-2005-4867, June 2005.
[15] Peterson, D., P. Subbareddy, and G.V. Candler, "DES Investigation of Transverse Injection
into Supersonic Crossflow using a Hybrid Unstructured Solver," AIAA-2006-903, Jan. 2006.
[16] Peterson, D., P. Subbareddy and G.V. Candler, "Simulation of Injection Into a Supersonic
Crossflow using DES with Synthetic Inflow," AIAA-2006-3326, June 2006.
[17] Peterson, D., P. Subbareddy, and G.V. Candler, "Detached Eddy Simulations of Flush Wall
Injection into a Supersonic Freestream," AIAA-2006-4576, July 2006.
[18] Peterson, D., P. Subbareddy and G.V. Candler, "Assessment of Synthetic Inflow
Generation for Simulating Injection Into a Supersonic Crossflow," AIAA-2006-8128, Nov. 2006.
[19] Peterson, D., and G.V. Candler, "Hybrid RANS/LES of a Supersonic Combustor," AIAA2008-6923, Aug. 2008.
[20] Peterson, D., and G.V. Candler, "Supersonic Combustor Simulations Using a Hybrid
RANS/LES Approach," AIAA-2010-0411, Jan. 2010.
[21] Peterson, D., and G.V. Candler, "Hybrid RANS/LES of Normal Injection into a Supersonic
Crossflow," Journal of Propulsion and Power, Vol. 26, No. 3, pp. 533-544, Mar. 2010.
[22] Peterson, D., E. Tylczak, G.V. Candler, "Hybrid Reynolds-Averaged and Large-Eddy
Simulation of Scramjet Fuel Injection," AIAA-2011-2344, April 2011.
[23] Peterson, D., and G.V. Candler, "Numerical Simulations of Mixing for Normal and LowAngled Injection into a Mach 2 Crossflow," AIAA Journal, Vol. 49, No. 12, pp. 2792-2804, Dec.
2011.
[24] Santiago, J. G., and Dutton, J. C., “Velocity Measurements of a Jet Injected into a
Supersonic Crossflow,” Journal of Propulsion and Power, Vol. 13, No. 2, 1997, pp. 264-273.
[25] Shur, M. L., Spalart, P. R., Strelets, M. Kh., and Travin, A. K., ‚ “A Hybrid RANS-LES
Approach with Delayed-DES and Wall-Modeled LES Capabilities,” International Journal of Heat
and Fluid Flow, Vol. 29, No. 6, 2008, pp. 1638-1649.
[26] Sinha, K., M. Barnhardt, and G.V. Candler, "Detached Eddy Simulation of Hypersonic Base
Flows with Application to Fire II Experiments," AIAA 2004-2633, June 2004.
[27] Spalart P.R., and S.R. Allmaras, “A One-Equation Turbulence Model for Aerodynamic
Flows,” AIAA 92-0439, Jan. 1992.
[28] Spalart, P. R., Jou, W-H., Strelets, M., and Allmaras, S. R., “Comments on the Feasibility of
LES for Wings and on a Hybrid RANS/LES Approach,” Advances in DNS/LES, 1 st AFOSR
International Conference on DNS/LES, Greyden, Columbus, OH, Aug. 4-8 1997.
[29] Spalart, P. R., Deck, S., Shur, M. L., Squires, K. D., Strelets, M. Kh., and Travin, A., “A New
Version of Detached-Eddy Simulation, Resistant to Ambiguous Grid Densities,” Theoretical and
Computational Fluid Dynamics, Vol. 20, 2006, pp. 181-195.
[30] Subbareddy, P., K. Sinha and G.V. Candler, "Detached Eddy Simulation of Supersonic
Base Flow with Base Bleed," AIAA Paper 2004-0066, Jan. 2004.
[31] Subbareddy, P., G.V. Candler, "Numerical Investigations of Supersonic Base Flows Using
DES," AIAA 2005-0886, Jan. 2005.
[32] Subbareddy, P., D. Peterson, G.V. Candler and I. Marusic, "A Synthetic Inflow Generation
Method Using the Attached Eddy Hypothesis," AIAA-2006-3672, June 2006.
[33] Subbareddy, P., and G.V. Candler, "A Fully-Discrete, Kinetic Energy Consistent Finite
Volume Scheme for Compressible Flows," Journal of Computational Physics, Vol. 228, pp.
1347-1364, 2009.
[34] Tylczak, E., D. Peterson, and G.V. Candler, "Hybrid RANS/LES Simulation of Transverse
Jet in Supersonic Crossflow with Laser Energy Deposition," AIAA-2010-4856, June 2010.
[35] Tylczak, E., G.V. Candler, D. Peterson, "Simulation of Plasma-Spark-Enhanced Mixing in
Jet in Supersonic Crossflow," AIAA-2011-398, Jan. 2011.
[36] Tylczak, E.B., D.M. Peterson, and G.V. Candler, "Hybrid RANS/LES Simulation of Injection
and Mixing in the CUBRC Combustion Duct," AIAA-2011-3216, June 2011.
[37] Wright, M.J., D. Bose, and G.V. Candler, “A Data-Parallel Line Relaxation Method for the
Navier-Stokes Equations,” AIAA Journal, Vol. 36, No. 9, 1603-1609, Sept. 1998.