High Frequency Ground Motion Simulations for Seismic Hazard Analysis
Lead PI: Jordan
SCEC INCITE OLCF PROJECT PROGRESS REPORT, 2015
Title: High Frequency Ground Motion Simulations for Seismic Hazard Analysis
PI and Co-PI(s): Thomas H. Jordan, Jacobo Bielak, Kim Olsen, Yifeng Cui, Po Chen, Ricardo
Taborda, Philip Maechling
Applying Institution/Organization: University of Southern California
Project Duration: 2 Years (Jan, 2015 - Dec, 2016)
Number of Processor Hours awarded on Titan: 119M core-hours
Executive Summary: Economic exposure to earthquake devastation has increased
significantly over the last few decades due to massive growth of urban areas in seismically
active regions. Physics-based modeling and simulation provide pathways to a more accurate
representation of earthquake systems, from the rupture at the fault to the response of the built
environment. Our goal is to produce earthquake simulations with sufficient bandwidth to be
useful across a wide range of engineering applications. Our specific INCITE project objectives
are twofold: (a) Use DOE supercomputers to produce a comprehensive, physics-based hazard
model for the Los Angeles region valid up to seismic frequencies of 1 Hz, and (b) Extend
realistic earthquake simulations above the 1-Hz frequency barrier by incorporating new aspects
of earthquake physics. Our achievements mark substantial progress towards these goals:
a) A 1-Hz urban seismic hazard model for the Los Angeles region has been completed
(Figure 1). The new model, which comprises more than 300 million synthetic seismograms
sampling the Uniform California Earthquake Rupture Forecast, was computed from a new
high-resolution image of crustal structure derived using full-3D tomography (CVM-S4.26). It
will be registered into the USGS Urban Seismic Hazard Mapping Project, and the results will
be submitted for use in the 2020 update of the Recommended Seismic Provisions of the
National Earthquake Hazards Reduction Program.
b) High-frequency simulations (out to 5 Hz) have been performed on the OLCF Titan
supercomputer using GPU-optimized finite-difference and finite-element codes that include
frequency-dependent attenuation, small-scale near-surface heterogeneities, tomography,
and a nonlinear dissipation in the near-fault and near-surface regions. These simulations set
the stage for the ground motion prediction modeling at frequencies beyond 1 Hz.
Accomplishments led by Dr. Yifeng Cui in the development of GPU-enabled wave-propagation
codes were recognized with NVIDIA’s 2015 Global Impact Award.
Project Accomplishments: SCEC's research team used the OLCF Titan and NCSA Blue
Waters supercomputers to perform CyberShake Study 15.4 (initiated in April, 2015). This
computation doubled the maximum seismic frequency represented in the Los Angeles urban
seismic hazard model, from 0.5 Hz to 1 Hz. Seismic hazard curves were derived from large
ensembles of seismograms at frequencies below this maximum for 336 surface sites distributed
across the Los Angeles region. This new probabilistic model uses refined earthquake rupture
descriptions through revisions to the conditional hypocenter distributions and the conditional slip
distributions. The CS15.4 model provides new seismic hazard information of interest to broad
impact customers of CyberShake, including seismologists, utility companies, and civil engineers
responsible for California building codes. The new model, which samples the complete Uniform
California Earthquake Rupture Forecast, will be registered into the USGS Urban Seismic
Hazard Mapping Project (http://earthquake.usgs.gov/hazards/products/urban/).
The GPU-based anelastic wave propagation AWP-ODC software was used to run CPU-based
post-processing calculations that synthesized over 300 million seismograms. In Study 15.4,
SCEC utilized approximately 200 pilot jobs to run CyberShake tasks on Titan resources. Over
1
High Frequency Ground Motion Simulations for Seismic Hazard Analysis
Lead PI: Jordan
80% of the node-hours burned on Titan were from jobs which ran on 25% or more of the
machine. Approximately 200 TB of SGT data was transferred from Titan to Blue Waters
automatically as part of the workflow. On Titan, the accelerated calculations of the GPU Strain
Green Tensor (SGT) implementation is 6.3 times more efficient than the CPU implementation,
which saved us 2 million node-hours over the course of the study. Our GPU development was
recognized with NVIDIA’s 2015 Global Impact Award. “The full three-dimensional treatment of
seismic-wave propagation has the potential to improve seismic hazard analysis models
considerably, and that is where the accelerating technology is particularly helpful at this
moment,” said Thomas Jordan, director of SCEC. “With GPU computing power we’re gaining
insight as to how the ground will move in high-risk areas, and how we can better plan for the
aftermath of a major event.”
The SCEC finite-element wave propagation solver, Hercules, which integrates an efficient
octree-based hexahedral mesh generator with an explicit FE formulation, has been optimized on
Titan this year achieving near perfect strong and weak scaling. Its GPU capabilities are currently
being used in verification and validation studies for the 2014 Mw 5.1 La Habra earthquake on
Titan, to test the accuracy of the code compared to other codes, and to examine how close the
predicted ground motions are to observations.
We have implemented non-associated Drucker-Prager nonlinear rheology following the return
map algorithm in the scalable AWP-ODC code, and we have used this code to model ground
motions from the M7.8 ShakeOut scenario source description. This work accounts for the limited
strength of crustal rocks; i.e., to simulate the absorption of rupture energy by permanent rock
deformation. Our results suggest that this nonlinear behavior could reduce previous simulationbased predictions of expected ground motion velocity in the Los Angeles basin during a largemagnitude event on the southern San Andreas Fault by 30 to 70 percent. Nonlinear material
response occurs in soft soils near the surface, typically reducing high-frequency (> 1 Hz)
2
High Frequency Ground Motion Simulations for Seismic Hazard Analysis
Lead PI: Jordan
shaking that controls damage to low- and mid-rise buildings. Our simulations show that
nonlinear response in crustal rocks may also reduce the amplitudes of long-period surface
waves that pose a hazard to high-rise buildings, implying less destruction than previously
anticipated. Although more research will be needed to quantify the impact of these findings on
damage and casualty estimates for future large-magnitude earthquakes on the San Andreas
Fault, the study pioneers more accurate earthquake scenarios based on better representations
of the nonlinearity in the Earth's crust.
We have implemented realistic attenuation structure (frequency-dependent Q, or Q(f)) in the
GPU-based AWP-ODC code (Withers et al., 2015). Tests using the 2008 Mw 5.4 Chino Hills
earthquake indicate that Q(f) generally fits the strong motion data better than for constant Q
models for frequencies over 1-Hz, which becomes more and more important as the distance
increases from the fault. We also found that media heterogeneity reduces the within-event
variability to that for observations and is thus important to characterize the ground motion.
Realistic ground-motion simulations require highly accurate crustal structural models. A
significant portion of the awarded computational resources was used to construct full-3D, highresolution crustal seismic velocity models in the Central California region and also in the
statewide California through full-3D seismic waveform tomography (F3DT) (Lee et al. 2014ab).
F3DT represents the latest development in seismic tomography techniques. Its application to
seismic data recorded in Southern California has yielded a new community velocity model for
the region, CVM-S4.26, which has unprecedented resolution of crustal structure. CVM-s4.26 is
the 3D structural model used in the CyberShake 15.4 study.
We have further improved the computational efficiency of our F3DT workflow on ALCF Mira and
are now applying F3DT to Central California and statewide. As of June, 2015, we have carried
out 5 F3DT iterations for Central California and 3 F3DT iterations for the statewide California.
Our improved Central California velocity model provides substantially better fit to over 12,000
seismic waveforms at frequencies up to 0.2 Hz and shows interesting small-scale structures in
the upper to mid crust that agree with local geology and other independent geophysical
evidence. Our latest statewide velocity model significantly improves the fit to over 27,000
waveforms at frequencies up to 0.1 Hz, and it has revealed new structural features in the mid to
3
High Frequency Ground Motion Simulations for Seismic Hazard Analysis
Lead PI: Jordan
lower crust that are consistent with our understanding of the geotectonic development in
California. More F3DT iterations will be carried out for both Central California and statewide.
Gradual improvements in our velocity models have allowed us to incorporate an increasing
volume of observed seismograms into our F3DT workflow, which is allowing us to resolve finer
structural details with higher accuracy.
Impact of Research: The San Andreas fault system is prone to major earthquakes, yet Los
Angeles has not experienced a major quake since its urbanization in the early twentieth century.
Data for the region are available from smaller quakes, but such information doesn’t give
emergency officials and structural engineers the information they need to prepare for a quake of
magnitude 7.5 or bigger. CyberShake Study 15.4 represents a major milestone in physicsbased PSHA for Southern California. The performance of the code and improved workflow
management, combined with the new physics it models (e.g., fault roughness, small-scale
heterogeneities, frequency-dependent attenuation, near-surface nonlinearities), take physicsbased seismic hazard analysis to a new level and pioneer the use of Petascale heterogeneous
computing resources for ground motion simulations used in building engineering design and
evaluation.
The reduction of peak velocities in our models caused by mostly shallow, near-fault nonlinear
effects may have important implications for the scaling of ground motion intensities between
surface-rupturing and buried earthquakes. Our nonlinear simulation results show that
nonlinearity in the fault zone is important even for conservative values of cohesion, suggesting
that current simulations based on a linear behavior of rocks are over-predicting the level of
ground motion in the Los Angles sedimentary basins during future large earthquakes on the
southern San Andreas Fault, and possibly for other large earthquake scenarios. This will have
far-reaching implications on earthquake emergency planning scenarios that are based on
ground motions predictions, such as the damage scenario of the 2008 Great California
ShakeOut. The addition of statistical models of near-surface small-scale heterogeneities has
enabled us to capture the “within-event” variability of earthquakes more accurately, providing
models that can be used to improve physics-based seismic hazard analysis.
Publications:
Isbiliroglu, Y., R. Taborda and J. Bielak (2015) Coupled soil-structure interaction effects of
building clusters during earthquakes. Earthquake Spectra. Vol. 31, No. 1, 463-500, Feb
2015.
Donovan, J. (2015), Forecasting Directivity in Large Earthquakes in Terms of the Conditional
Hypocenter Distribution, PhD Thesis, University of Southern California, 154 pp.
Jordan, T. H. (2015), An effective medium theory for three-dimensional elastic heterogeneities,
Geophys. J. Int., submitted Mar 29, 2015.
Lee E.-J., P. Chen, T. H. Jordan, P. B. Maechling, M. A.M. Denolle and G. C. Beroza (2014a),
Full-3D tomography for crustal structure in Southern California based on the scatteringintegral and the adjoint-wavefield methods, J. Geophys. Res., 119, 6421-6451,
doi:10.1002/2014JB011346.
Lee, E.-J., P. Chen, and T. H. Jordan (2014b), Testing waveform predictions of 3D velocity
models against two recent Los Angeles earthquakes, Seismol. Res. Lett., 85, 12751284, doi:10.1785/0220140093.
Lozos, J., K. B. Olsen, J. Brune, R. Takedatsu, R. Brune, and D.D. Oglesby (2015), Broadband
ground motions from dynamic models of rupture on the northern San Jacinto fault, and
comparison with precariously balanced rocks, Bull. Seismol. Soc. Am., 105. (in press),
4
High Frequency Ground Motion Simulations for Seismic Hazard Analysis
Lead PI: Jordan
doi: 10.1785/0120140328.
Olsen, K. B. and R. Takedatsu (2015), The SDSU Broadband Ground-Motion Generation
Module BBtoolbox Version 1.5, Seism. Res. Letter, 86, 1, 81-88.
Poyraz, E., H. Xu and Y. Cui (2014), Application-specific I/O Optimizations on Petascale
Supercomputers, Proceedings of International Conference on Computational Science,
Elsevier, 29, 910-923, Cairns, June 10-12.
Roten, D., K. B. Olsen, S. M. Day, Y. Cui and D. Faeh (2014), Expected seismic shaking in Los
Angeles reduced by San Andreas fault zone plasticity, Geophysical Research Letters,
41, doi:10.1002/ 2014GL059411.
Roten, D., K. B. Olsen, Y. Cui, and S. M. Day (2015), Quantification of fault zone plasticity
effects with spontaneous rupture simulations, to be submitted to Workshop on Best
Practice in Physics-Based Fault Rupture Models for Seismic Hazard Assessment of
Nuclear Installations, Vienna, Austria, Nov 18-20.
Shaw, J. H., A. Plesch, C. Tape, M. P. Suess. T. H. Jordan, G. Ely, E. Hauksson. J. Tromp, T.
Tanimoto, R. Graves, K. Olsen, C. Nicholson, P. J. Maechling, C. Rivero, P. Lovely, C.
M. Brankman, and J. Munster (2015), Unified Structural Representation of the southern
California crust and upper mantle, Earth Planet. Sci. Lett., 415, 1-15,
doi:10.1016/j.epsl.2015.01.016.
Taborda, R., and J. Bielak (2014), Ground-Motion Simulation and Validation of the 2008 Chino
Hills, California, Earthquake Using Different Velocity Models. Bull. Seismol. Soc. Am.,
104, 1876-1898, doi:10.1785/0120130266
Wang, F., and T. H. Jordan (2014), Comparison of probabilistic seismic hazard models using
averaging-based factorization, Bull. Seismol. Soc. Am., 104, 1230-1257, doi:
10.1785/0120130263.
Withers, K. B., K. B. Olsen, S. M. Day (2015). Memory-efficient simulation of frequency
dependent Q, Bull. Seismol. Soc. Am., in revision.
Zhou, J. (2014). Scalable Parallel Programming for High Performance Seismic Simulation on
Petascale Heterogeneous Supercomputer, PhD Thesis, University of California at San
Diego.
5