Assorted material for PRAC proposal:
Expanding Spatial and Temporal Scales of Physics-Based Seismic Hazard Analysis
Contributed by:
Hercules simulation team
Summary
Broader Impacts
(…) by expanding the spatial and temporal scales of a physics-based approach to seismic hazard analysis,
SCEC researchers will build a robust simulation framework that will provide a more comprehensive
approach to problems spanning from scientific understanding of earthquake processes to daily life
applications in engineering design of infrastructure systems such as lifeline and transportation networks
as well as sensitive projects and facilities like nuclear power plants or dams. We will achieve this by
vertically interconnecting software elements and simulation products (input and output datasets) from the
earthquake source (dynamic simulation of the fault’s rupture), through the ground motion (high frequency
simulation of traveling seismic waves), to the composition of physics-based seismic hazard maps
(CyberShake). Such a simulation framework seeks to transform current approaches based entirely on
statistical models by incorporating a physics-based deterministic background to evaluating the seismic
risk of specific engineering projects or entire regions.
Intellectual Merit
[in reference to forward simulations in the High-F component]
(…) our project plan will improve and advance our simulation capabilities to reproduce the observed
characteristics of the ground motion during earthquakes at frequencies higher than typically done before.
We will do this by incorporating heterogeneities in the underlying models of the source and the crustal
structure as well as by more realistically representing the constitutive material models (attenuation and
plasticity) used in our simulation engines. These are problems that can only be addressed with petascale
computing because they extend the spatial and temporal scales of our models. Therefore, part of our
efforts will be directed at improving and optimizing our simulation codes to maximize the use of
resources while achieving scientific goals of significance to the seismological and earthquake engineering
communities and to the society as a whole.
Introduction
(…)
We propose to advance scientific knowledge and software development applied to understanding
earthquake processes from the mechanics of the rupture at a seismic fault to the realistic representation of
the ground motion at frequencies of engineering interest by expanding the spatial and temporal scales of
earthquake processes simulation. We will advance a physics-based framework that will enable us to
construct robust seismic hazard maps for use in buildings design and resilient urban planning. To this
end, we will vertically integrate software tools and simulation products that require petascale-level
computing and storage at three levels: dynamic rupture simulation, high frequency ground motion
simulation, and physics-based seismic hazard modeling. This involves the continued development of four
core codes: SORD, Hercules, AWP-ODC, and AWP-ODC-GPU; and a software framework for physicsbased seismic hazard mapping: CyberShake. These codes and software framework use finite difference
and finite element methods to solve the underlying dynamic rupture and wave propagation problems, and
concepts such as reciprocity to optimized the repeated forward simulation of multiple scenario
earthquakes needed for seismic hazard analysis, as well as advanced high-performance scientific
computing tools to boost our time to solution.
Dynamic Rupture
[SORD – Not our thing  ]
High-F
Description and Intellectual Merit
[partly taken/modified from the High-F project document]
The SCEC High Frequency (or “High-F”) project seeks to integrate various scientific modeling and
simulation efforts within the Southern California Earthquake Center with the objective of reproducing
earthquake physics and effects at high frequencies (from 5 up to 10 Hz) using deterministic modeling
approaches. The High-F project uses, improves, and develops forward wave-propagation simulation
capabilities and approaches in order to better reproduce the ground response at higher frequencies. In the
activities of the High-F project component of this allocation proposal we will incorporate high-frequency
characteristics in the source representation by vertically integrating the output from the Dynamic Rupture
component of our project with high-frequency simulations by converting the rupture process into a
kinematic model with predefined slip functions on the (shared) fault geometry. This is a key component
of our forward simulations given that we have observed in previous studies that our forward simulations
lack sufficient energy at the higher frequencies.
At the seismic wave propagation level, the High-F component of our project will incorporate more
realistic physics in the models by considering (a) small-scale heterogeneities (stochastic characteristics) in
the material that compose the crustal structure and sedimentary deposits; (b) the elasto-visco-plastic
behavior of geomaterials; (c) the frequency dependent characteristics of the attenuation quality factor Q;
and (d) other site-effects due to scattering introduced by irregular surface topography or the built
environment. These are all particularly challenging to model aspects because of their multi-scale multiphysics nature, and are intrinsic to the ground motion characteristics as observed from records of past
earthquakes—especially at the higher frequencies in which we are interested (5–10 Hz).
The small-scale heterogeneities will be considered through the incorporation of statistical models that
reproduce, on average, the spatial variability and randomness in the material properties (Vs, Vp and
density) as observed from deep exploration logs (at the crustal level) and boreholes (in geotechnical
layers at the upper few hundred meters from the free surface). The elasto-visco-plastic behavior of
geomaterials used in the computational models of the region to represent off-fault and near-surface
plasticity effects will be considered through the incorporation of realistic constitutive models that can
account for the nonlinear stress-strain relationships. Both AWP-ODC and Hercules have implemented
modules that incorporate the Drucker-Prager elasto-plastic constitutive mode, but more realistic models
are likely to be needed, especially for reproducing the behavior of soft-soil sedimentary deposits near the
surface. The frequency dependent characteristics of the attenuation quality factor, Q, will be incorporated
by implementing alternative visco-elastic models. The AWP-ODC team has already implemented an
initial model (reference here) and the Hercules team is currently working on extending their attenuation
model (Bielak et al. 2011) to account for the frequency dependence of Q. Finally, we will consider better
representations of particular site conditions by using the extended modeling capabilities of Hercules to
account for surface topographic effects using special finite elements and to account for the scattering
effects of the built environment by considering coupled soil-structure and site-city interaction models in
densely urbanized regions (Isbiliroglu et al., 2014) using the Domain Reduction Method (Bielak et al.
2003).
The simulation plans in this component of our project will consist of a series of 5–10 Hz simulations of
historic and scenario moderate and large magnitude earthquakes for the region of southern California (see
Table ??). Moderate scenario and historic earthquakes will be used primarily for calibration of the models
and simulation codes. We will perform a sequence of verification exercises directed at ensuring that the
modeling capabilities of each simulation code/team (Hercules and AWP-ODC/AWP-ODC-GPU) yields
equivalent wavefield results with respect to each other. Verification at this level is particularly important
because of the added complexity required in high-frequency simulations. In the past we have successfully
performed verification exercises at much lower frequencies (0.5 Hz), which indicated that our core codes
were robust enough to conduct individual (complementary) simulations (Bielak et al. 2010). This,
however, needs to be replicated for the higher frequency levels of interest proposed here—where the
intrinsic differences in the methodologies implemented in our simulation codes (finite elements, finite
differences) are likely to be exposed in new ways. These differences will need to be resolved so that we
can distribute the simulation workload for the more challenging problem of validation and simulation of
large magnitude events.
Validation of historic events through comparisons of synthetics with data will be done first for the
moderate magnitude events using goodness-of-fit measures that convey physical meaning to both
seismologists and engineers (Anderson 2004; Olsen and Mayhew 2010). We have accumulated extensive
experience in validation of simulations (up to 4 Hz) of moderate events such as the 2008 Chino Hills
earthquakes (Taborda and Bielak 2013, 2014). Through this work we have identified a series of aspects
that affect the accuracy of our simulations at frequencies above 1 and 2 Hz when compared to data. These
aspects include the lack of high-frequency energy in our kinematic source models (which will be resolved
through more realistic simulations of dynamic rupture processes) and the lack of sufficient resolution and
inaccuracy in the representation of material properties in the velocity model (which will be addressed
through inclusion of small-scale heterogeneities and use of more realistic attenuation and constitutive
models). Components (a) through (d) described above are precisely designed to improve the goodnessof-fit of our simulations at higher frequencies.
Computationally speaking, these components are more demanding because they expand the spatial and
temporal resolution of our models—which substantiates our need for high-performance petascale
computing resources. Incorporating small scale heterogeneities, for instance, increases substantially (by
almost an order of magnitude) the number of finite elements or finite difference cells in our models. [In
the case of Hercules] up until now the largest runs made using Blue Waters and other similar systems
have reached maximum frequencies of 4 and 5 Hz with counts of 5 to 15 billion nodes in unstructured
finite-element meshes (will need to draw an equivalency with AWP-ODC here). At this level of
resolution, the physical size of the smallest element is of about 5 m. Future planned simulations that will
include small-scale heterogeneities at even higher frequencies have been calculated to produce meshes
counts between 70 and 100 billion elements. In turn, the incorporation of better physics (as in the case of
nonlinear off-fault and near-surface plasticity) for instance will require the use of smaller simulation timesteps and increases the time to solution per simulation cycle. In addition, better attenuation Q models
increase the memory usage of the codes.
One, or more, historic earthquakes in Southern California will be selected as target events, and the results
of high-frequency ground motion simulations will be compared with observed data from these
earthquakes. The High-F project will help to define a reference framework for the evaluation of
alternative simulation methods, such as the stochastic simulation methods developed within the SCEC
Broadband Platform, and will seek to identify the threshold frequency at which deterministic and
stochastic methods provide a viable tradeoff for hybrid approaches.
Description of Simulation Codes
[Taken from the High-F document]
Recent simulations (Taborda and Bielak, 2012; Isbiliroglu et al., 2012; Cui et al., 2010) have shown that,
codes such as Hercules and AWP-ODC have the capability to simulate earthquakes at maximum
frequencies equal to 4 and 5 Hz, using only a fraction of the resources available in supercomputing
facilities today (Table 1). Estimates based on these results indicate that higher frequency simulations (up
to 10 Hz) are possible in the near future (see Table 2 for performance estimates for Hercules; equi-spaced
grid approaches such as AWP-ODC require additional resources dependent on the scenario parameters).
AWP-ODC and AWP-Graves are already being used by a number of researchers in the SCEC
community. CMU’s Quake Group has started the process of producing and releasing a version of
Hercules as a SCEC community code. Several alternative high-frequency wave propagation codes are
needed for the verification and validation efforts necessary to support the proposed activities—especially
for evaluating the new source, velocity, and attenuation models at high frequencies but smaller physical
scales. We also intend to be able to support code improvement in order to reach the performance
objectives of the High-F project.
(…)
[Taken and modified from previous proposals]
Hercules (Tu et al., 2006; Taborda et al., 2010) is an octree-based parallel finite element (FE) earthquake
simulator developed by the Quake Group at Carnegie Mellon and partners at the University of Memphis
and the University of Southern California, which is part of the SCEC Community Modeling Environment
computational development activities. The standard distribution of Hercules is written in C using MPI
inter-processor communication libraries. Hercules’ finite element forward solver has second-order
accuracy in both time and space. It integrates an efficient unstructured hexahedral mesh generator and an
explicit finite element formulation to solve the linear momentum (partial differential) equation for 3D
wave propagation problems in highly heterogeneous media due to earthquake sources modeled with
kinematic faulting. Alternative distribution of Hercules solve wave propagation problems in nonlinear
elasto-plastic media and can account for surface topography and the presence of the built environment.
A version of Hercules currently in development use C++ and CUDA for use of GPU in hybrid systems.
Hercules has been extensively tested in Blue Waters and other similar systems such as Titan at OLCF and
Kraken at NICS. Figure X shows recent strong and weak scaling curves.
Supporting Material
Scaling curves
Subject to change/update based on one run being done this week on Blue Waters
(to be continued)