INCITE 2015
Project Name: High Frequency Physics-Based Earthquake System Simulations (Year 1 of 2)
PI: Thomas H. Jordan
Co-PI(s): Jacobo Bielak, Carnegie Mellon University,
Po Chen, University of Wyoming,
Yifeng Cui, San Diego Supercomputer Center,
Philip Maechling, Southern California Earthquake Center,
Kim Olsen, San Diego State University,
Ricardo Taborda, University of Memphis
ALCF Project Name: GMSeismicSim
OLCF Project Name: GEO112
Performance Period: January 1 - June 2015 (Q1 and Q2)
Quarterly Update: Q2 - 2015
Report on Project Milestones
 Provide status on each of your project’s simulations milestones as outlined in your
original proposal
Year 1 Milestone Descriptions
M1
Use full 3D tomography and comparative
validations using to improve existing California
velocity models for use in high frequency wave
propagation simulations at 0.2Hz
M2
Run high frequency forward simulations using
alternative material attenuation (Q) and seismic
velocity models (CVMs). Compare the impact of
material properties, topography, and models
including spatial variability (heterogeneities) and
soft-soil deposits (or geotechnical layers) on 4Hz+
simulations by simulating forward events using
alternative models and comparing results among
synthetics and with data.
Run high frequency forward simulations using
alternative approaches to include the effects of
off-fault and near-surface plastic deformation.
Compare the impact of alternative plasticity
M3
Milestone
Achievement Status
Achieved. Used Mira to calculate
five iterations of Central
California Model. The improved
3D model now available to
ground motion modelers through
UCVM.
Started, not completed. AWP-ODC
and Hercules code branches have
tested these physics. Currently
running baseline 4Hz simulations
without these physics with good
agreement at 4Hz among 3 wave
propagation codes with a simple
velocity structure.
Started, not completed. AWP-ODC
and Hercules code branches have
tested these physics. Currently
running baseline 4Hz simulations
M4
M5
M6
M7
M8

models (linear-equivalent, 3D+1D hybrid, full 3D
plastic) on 4Hz+ simulations by simulating
forward events and comparing the results among
synthetics and with empirical relationships and
data.
Calculate a 1.0Hz CyberShake Hazard curve. Use
updated CVMs, source models, and codes to
calculate a higher frequency CyberShake hazard
curve
Year 2 Milestone Descriptions
Use full 3D tomography and comparative
validations using to improve existing California
velocity models for use in high frequency wave
propagation simulations at 0.5Hz
Run high frequency forward simulations using
alternative material attenuation (Q) and seismic
velocity models (CVMs). Compare the impact of
material properties, topography, and models
including spatial variability (heterogeneities) and
soft-soil deposits (or geotechnical layers) on 8Hz+
simulations by simulating forward events using
alternative velocity models and comparing the
results.
Run high frequency forward simulations using
alternative approaches to include the effects of
off-fault and near-surface plastic deformation.
Compare the impact of alternative plasticity
models (linear-equivalent, 3D+1D hybrid, full 3D
plastic) on 8Hz+ simulations by simulating
forward events and comparing the results among
synthetics and with empirical relationships and
data.
Calculate a 1.5Hz CyberShake Hazard curve. Use
updated CVMs, source models, and codes to
calculate a higher frequency CyberShake hazard
curve
without these physics with good
agreement at 4Hz among 3 wave
propagation codes with a simple
velocity structure.
Achieved. Used Titan and Blue
Waters to calculate a CyberShake
1Hz Los Angeles area
probabilistic seismic hazard
model based on 336 site-specific
hazard curves.
Objective
Not started
Not started
Not started
Not started
List major accomplishments thus far in CY2015. Please include scientific and computational
details of simulations undertaken, including images if possible.
Our 2015-2016 INCITE proposal defined the following four objectives:
O1: Improve the resolution of dynamic rupture simulations by an order of magnitude and
investigate the effects of realistic friction laws, geologic heterogeneity, and near-fault stress states
on seismic radiation.
O2: Extend deterministic simulations of strong ground motions to 10 Hz for investigating the
upper frequency limit of deterministic ground-motion prediction.
O3: Compute physics-based Probabilistic Seismic Hazard Attenuation (PSHA) maps and validate
those using seismic and paleo-seismic data.
O4: Improve 3D earth structure models through full 3D tomography using observed seismicity
and ambient noise.
During the first six months of our current allocation, SCEC researchers have worked towards these
objectives in three main ways: (a) Define and improve a new central California 3D velocity model
using full 3D tomography, (b) Produce a comprehensive, physics-based hazard model for the Los
Angeles region valid up to seismic frequencies of 1 Hz, and (c) Extend realistic earthquake
simulations above the 1-Hz frequency barrier by incorporating new aspects of earthquake physics.
Our INCITE project activities in Q1 and Q2 mark substantial progress towards these goals
including the following significant accomplishments:
a) We defined a central California 3D velocity model, that we call Central California Area (CCA),
and used full 3D tomography computational methods to validate and improve the 3D seismic
velocity model using both observed moderate earthquakes and ambient seismic noise
observations.
b) We completed a 1-Hz urban seismic hazard model for the Los Angeles region (Figure 1). The
new model, which comprises more than 300 million synthetic seismograms sampling the
Uniform California Earthquake Rupture Forecast, was computed from a new high-resolution
image of crustal structure derived using full-3D tomography (CVM-S4.26). It will be registered
into the USGS Urban Seismic Hazard Mapping Project, and the results will be submitted for use
in the 2020 update of the Recommended Seismic Provisions of the National Earthquake
Hazards Reduction Program.
c) We performed high-frequency simulations (out to 5 Hz) on the OLCF Titan supercomputer
using GPU-optimized finite-difference and finite-element codes that include frequencydependent attenuation, small-scale near-surface heterogeneities, tomography, and a nonlinear
dissipation in the near-fault and near-surface regions. These simulations set the stage for the
ground motion prediction modeling at frequencies beyond 1 Hz.
Accomplishments led by Dr. Yifeng Cui in the development of GPU-enabled wave-propagation
codes were recognized with NVIDIA’s 2015 Global Impact Award.
Major Project Accomplishments:
a) We defined a central California 3D velocity model, that we call Central California Area
(CCA), and used full 3D tomography computational methods to validate and improve the 3D
seismic velocity model using both observed moderate earthquakes and ambient seismic
noise observations.
SCEC research team used ALCF Mira to produce a 3D seismic velocity model for Central California
that we call Central California Area (CCA). We have used Mira to perform the computational and
data intensive stages of our full 3D tomographic (F3DT) computational method. The purpose of
the F3DT for central California is to improve the crustal 3D seismic velocity model for central
California. An accurate 3D velocity model is essential input needed for accurate deterministic
earthquake wave propagation simulations.
We have further improved the computational efficiency of our F3DT workflow on ALCF Mira and
are now applying F3DT to Central California and statewide. As of 1 July, 2015, we have carried out
5 F3DT iterations for Central California and 3 F3DT iterations for the statewide California. Our
improved Central California velocity model provides substantially better fit to over 12,000 seismic
waveforms at frequencies up to 0.2 Hz and shows interesting small-scale structures in the upper
to mid crust that agree with local geology and other independent geophysical evidence. Our latest
statewide velocity model significantly improves the fit to over 27,000 waveforms at frequencies
up to 0.1 Hz, and it has revealed new structural features in the mid to lower crust that are
consistent with our understanding of the geotectonic development in California. More F3DT
iterations will be carried out for both Central California and statewide. Gradual improvements in
our velocity models have allowed us to incorporate an increasing volume of observed
seismograms into our F3DT workflow, which is allowing us to resolve finer structural details with
higher accuracy.
The map below shows the bounding box of this new velocity model. The bounding corners for the
CCA model are: -120.0000, 33.3999, -122.9500, 36.6000, -118.2962, 39.3548, -115.4454, 36.0403.
The starting model is defined using a 500m grid spacing and we use trilinear interpolation in
between the grid points when constructing meshes. The model covers depths down to 50km.
The purpose of the F3DT for central California is to improve the crustal velocity model in central
California for more accurate ground motion predictions. Our initial model is based on the updated
Community Velocity Model for Southern California, CVM-S4.26 [Lee et al., 2014], and other
existing velocity models for northern California [Xu et al., 2013]. Our study area is located on the
edges of the two models, where the data coverage is poor for the two models. Our advances in this
full 3D tomography work during this performance period include the following:

The 5th Full-3D tomography (F3DT) iteration for Central California crust
During the 2015 Q2, we have completed the 5th iteration for the Central California crustal
F3DT. In this iteration, we used the available ambient noise Green’s functions (ANGFs) among
the broadband and short-period stations to invert the velocity model. We applied two
bandpass filters to the ANGF waveforms to separate the high (0.1~0.18Hz) and low
(0.03~0.1Hz) frequencies sources. In our 5th iteration, more than 12,000 waveform windows
have been picked and more than 59,000 frequency dependent measurements have been made
for the Central California tomographic inversion (Figure 1). To evaluate the waveform
improvement, we measured the difference between observed 𝑢𝑘 (𝑡) and its corresponding
synthetic 𝑢̃𝑘 (𝑡) waveforms within the time window [tk, tk'] by the relative waveform misfit
(𝑅𝑊𝑀) statistic, defined by the integral
𝑡′
𝑅𝑊𝑀𝑘 =
∫𝑡 𝑘[𝑢𝑘 (𝑡) − 𝑢̃𝑘 (𝑡)]2 𝑑𝑡
𝑘
𝑡𝑘′
𝑡′
𝑘
𝑘
√∫ 𝑢𝑘 (𝑡)2 𝑑𝑡 ∫ 𝑘 𝑢̃𝑘 (𝑡)2 𝑑𝑡
𝑡
𝑡
Here, the time [tk, tk'] for the kth waveform window runs from the starting and end time of the
window. After our 5 tomographic inversions, the sum of RWM has reduced more than 28%
when compare to that of the initial model (Figure 2). In addition, the iteratively inversions
have reduced the variance of frequency dependent group delay measurements (dtg) by over
35% relative to the starting model (Figure 3). The perturbations in velocities have begun to
heal the velocity artifacts inherited from starting model (Figure 4). Many features revealed in
the model are consistent with independent geophysical observations in Central California,
including controlled-source tomography, gravity anomalies, and the locations of active faults.

Full-wave centroid moment tensor (CMT) inversion in an updated 3D velocity model for
earthquakes in Central California
To include the earthquake recordings in the next iteration, we have applied a full-wave Central
Moment Tensor (CMT) inversion algorithm to more than 200 earthquakes recorded in Central
California (Figure 5). The procedure relies on the use of receiver-side Green tensors (RGTs),
which comprise the spatial-temporal displacements produced by the three orthogonal unit
impulsive point forces acting at the receiver. We have constructed a RGT database for more
than 180 broadband stations in Central California using the updated Central California F3DT to
reduce the potential errors in velocity structures. In our CMT inversion method, we implement
the Bayesian inference on our measurements. An important advantage of the Bayesian
approach is that, instead of a single best solution, the complete posterior probability density
on the sample space is obtained, which allows formal estimation of the uncertainties
associated with the derived source parameters. More robust earthquake source parameter
estimates are critical for both geologic interpretation of active faults and seismic hazard
analysis.

Validation of Central California F3DT using earthquake recordings not used in previous
inversions
To validate the updated Central California F3DT, we tested the waveform predictions of the
model using the earthquake recordings not used in previous tomographic inversions. More
than 11,500 three-component broadband seismograms with signal-to-noise ratio (SNR) larger
than 4 has been used in this validation (Figure 6). All synthetic and observed seismograms
were band-pass filtered using a Butterworth filter with corners at 0.02 Hz and 0.2 Hz. We
measured 𝑅𝑊𝑀 between an individual observed seismogram 𝑢𝑘 (𝑡) and its corresponding
synthetics 𝑢̃𝑘 (𝑡) of the initial model (CCA00) and the updated model (CCA05). The time
window for the waveforms are from the first arrival to the end of the main surface wave group,
so that 𝑅𝑊𝑀 measures the net waveform difference across all of the main phases on the
seismograms. In those comparisons, the synthetics computed using the updated velocity model
(CCA05) provide better fit to observed seismograms at frequencies below 0.2 Hz than those
computed using the initial model (CCA00) (Figure 6).
Figure 1. Distribution of all frequency dependent measurements for the 5th iteration at different
frequencies for ambient noise Green’s functions.
Figure 2. The histograms of RWMs for ambient noise Green’s functions for the initial model
(CCA00) and the updated model (CCA05).
Figure 3. The histograms of frequency dependent group delay measurements (dtg) for ambient
noise Green’s functions for the initial model (CCA00) and the updated model (CCA05).
Figure 4. Shear wave (S wave) velocity at (top) 2 km, (middle) 10 km, and (bottom) 20 km depths
in (left) the initial model CCA00, (middle) the 5th iteration model CCA05, and (right) the
perturbations. The color bar on the lower right corner of each plot shows the range of the color
scale with red indicating relatively slow S wave velocities and blue indicating relatively fast S
wave velocities. Black solid lines show major faults in our study area.
Figure 5. The CMT solution for earthquakes analyzed in Central California. Yellow triangles
indicate the locations of broadband stations. Red dots indicate the epicenters of those
earthquakes. The box indicates our study area. Major faults in this area are plotted in black solid
lines. The background color shows topography.
Figure 6. The histograms of RWMs for earthquake recordings not used in current tomographic
inversion for the initial model (CCA00) and the updated model (CCA05).
b) We completed a 1-Hz urban seismic hazard model for the Los Angeles region (Figure 1a1b).
SCEC's research team used the OLCF Titan and NCSA Blue Waters supercomputers to perform
CyberShake Study 15.4 (initiated in April, 2015). This computation doubled the maximum seismic
frequency represented in the Los Angeles urban seismic hazard model, from 0.5 Hz to 1 Hz.
Seismic hazard curves were derived from large ensembles of seismograms at frequencies below
this maximum for 336 surface sites distributed across the Los Angeles region. This new
probabilistic model uses refined earthquake rupture descriptions through revisions to the
conditional hypocenter distributions and the conditional slip distributions. This seismic hazard
calculation used the CVM-S4.26 3D velocity model, which was validated and improved using ALCF
Mira, as the best available southern California 3D velocity model. The CS15.4 model provides new
seismic hazard information of interest to broad impact customers of CyberShake, including
seismologists, utility companies, and civil engineers responsible for California building codes. The
new model, which samples the complete Uniform California Earthquake Rupture Forecast, will be
registered into the USGS Urban Seismic Hazard Mapping Project
(http://earthquake.usgs.gov/hazards/products/urban/).
The GPU-based anelastic wave propagation AWP-ODC software was used to run CPU-based postprocessing calculations that synthesized over 300 million seismograms. In Study 15.4, SCEC
utilized approximately 200 pilot jobs to run CyberShake tasks on Titan resources. Over 80% of
the node-hours burned on Titan were from jobs which ran on 25% or more of the machine.
Approximately 200 TB of SGT data was transferred from Titan to Blue Waters automatically as
part of the workflow. On Titan, the accelerated calculations of the GPU Strain Green Tensor (SGT)
implementation is 6.3 times more efficient than the CPU implementation, which saved us 2 million
node-hours over the course of the study.
Our GPU development was recognized with NVIDIA’s 2015 Global Impact Award. “The full threedimensional treatment of seismic-wave propagation has the potential to improve seismic hazard
analysis models considerably, and that is where the accelerating technology is particularly helpful
at this moment,” said Thomas Jordan, director of SCEC. “With GPU computing power we’re gaining
insight as to how the ground will move in high-risk areas, and how we can better plan for the
aftermath of a major event.”
c) We performed high-frequency simulations (out to 5 Hz) on the OLCF Titan
The SCEC finite-element wave propagation solver, Hercules, which integrates an efficient octreebased hexahedral mesh generator with an explicit FE formulation, has been optimized on Titan
this year achieving near perfect strong and weak scaling. Its GPU capabilities are currently being
used in verification and validation studies for the 2014 Mw 5.1 La Habra earthquake on Titan, to
test the accuracy of the code compared to other codes, and to examine how close the predicted
ground motions are to observations.
We have implemented non-associated Drucker-Prager nonlinear rheology following the return
map algorithm in the scalable AWP-ODC code, and we have used this code to model ground
motions from the M7.8 ShakeOut scenario source description. This work accounts for the limited
strength of crustal rocks; i.e., to simulate the absorption of rupture energy by permanent rock
deformation. Our results suggest that this nonlinear behavior could reduce previous simulationbased predictions of expected ground motion velocity in the Los Angeles basin during a largemagnitude event on the southern San Andreas Fault by 30 to 70 percent. Nonlinear material
response occurs in soft soils near the surface, typically reducing high-frequency (> 1 Hz) shaking
that controls damage to low- and mid-rise buildings. Our simulations show that nonlinear
response in crustal rocks may also reduce the amplitudes of long-period surface waves that pose a
hazard to high-rise buildings, implying less destruction than previously anticipated. Although
more research will be needed to quantify the impact of these findings on damage and casualty
estimates for future large-magnitude earthquakes on the San Andreas Fault, the study pioneers
more accurate earthquake scenarios based on better representations of the nonlinearity in the
Earth's crust.
We have implemented realistic attenuation structure (frequency-dependent Q, or Q(f)) in the GPUbased AWP-ODC code (Withers et al., 2015). Tests using the 2008 Mw 5.4 Chino Hills earthquake
indicate that Q(f) generally fits the strong motion data better than for constant Q models for
frequencies over 1-Hz, which becomes more and more important as the distance increases from
the fault. We also found that media heterogeneity reduces the within-event variability to that for
observations and is thus important to characterize the ground motion.
Realistic ground-motion simulations require highly accurate crustal structural models. A
significant portion of the awarded computational resources was used to construct full-3D, highresolution crustal seismic velocity models in the Central California region and also in the
statewide California through full-3D seismic waveform tomography (F3DT) (Lee et al. 2014ab).
F3DT represents the latest development in seismic tomography techniques. Its application to
seismic data recorded in Southern California has yielded a new community velocity model for the
region, CVM-S4.26, which has unprecedented resolution of crustal structure. CVM-s4.26 is the 3D
structural model used in the CyberShake 15.4 study.
Impact of Research: The San Andreas fault system is prone to major earthquakes, yet Los Angeles
has not experienced a major quake since its urbanization in the early twentieth century. Data for
the region are available from smaller quakes, but such information doesn’t give emergency
officials and structural engineers the information they need to prepare for a quake of magnitude
7.5 or bigger. CyberShake Study 15.4 represents a major milestone in physics-based PSHA for
Southern California. The performance of the code and improved workflow management, combined
with the new physics it models (e.g., fault roughness, small-scale heterogeneities, frequencydependent attenuation, near-surface nonlinearities), take physics-based seismic hazard analysis to
a new level and pioneer the use of Petascale heterogeneous computing resources for ground
motion simulations used in building engineering design and evaluation.
The reduction of peak velocities in our models caused by mostly shallow, near-fault nonlinear
effects may have important implications for the scaling of ground motion intensities between
surface-rupturing and buried earthquakes. Our nonlinear simulation results show that
nonlinearity in the fault zone is important even for conservative values of cohesion, suggesting
that current simulations based on a linear behavior of rocks are over-predicting the level of
ground motion in the Los Angles sedimentary basins during future large earthquakes on the
southern San Andreas Fault, and possibly for other large earthquake scenarios. This will have farreaching implications on earthquake emergency planning scenarios that are based on ground
motions predictions, such as the damage scenario of the 2008 Great California ShakeOut. The
addition of statistical models of near-surface small-scale heterogeneities has enabled us to capture
the “within-event” variability of earthquakes more accurately, providing models that can be used
to improve physics-based seismic hazard analysis.
References:
Lee, E.-J., P. Chen, T. H. Jordan, P. B. Maechling, M. A. M. Denolle, and G. C. Beroza (2014), Full-3-D
tomography for crustal structure in Southern California based on the scattering-integral and the
adjoint-wavefield methods, J. Geophys. Res. Solid Earth, 119(8), 6421–6451,
doi:10.1002/2014JB011346.
Xu, Z., P. Chen, and Y. Chen (2013), Sensitivity Kernel for the Weighted Norm of the FrequencyDependent Phase Correlation, Pure Appl. Geophys., 170(3), 353–371, doi:10.1007/s00024-0120507-3.
Project Productivity
Primary
 Publications –
1. Isbiliroglu, Y., R. Taborda and J. Bielak (2015) Coupled soil-structure interaction effects of
building clusters during earthquakes. Earthquake Spectra. Vol. 31, No. 1, 463-500, Feb 2015.
2. Donovan, J. (2015), Forecasting Directivity in Large Earthquakes in Terms of the Conditional
Hypocenter Distribution, PhD Thesis, University of Southern California, 154 pp.
3. Jordan, T. H. (2015), An effective medium theory for three-dimensional elastic heterogeneities,
Geophys. J. Int., submitted Mar 29, 2015.
4. Lozos, J., K. B. Olsen, J. Brune, R. Takedatsu, R. Brune, and D.D. Oglesby (2015), Broadband
ground motions from dynamic models of rupture on the northern San Jacinto fault, and
comparison with precariously balanced rocks, Bull. Seismol. Soc. Am., 105. (in press), doi:
10.1785/0120140328.
5. Olsen, K. B. and R. Takedatsu (2015), The SDSU Broadband Ground-Motion Generation Module
BBtoolbox Version 1.5, Seism. Res. Letter, 86, 1, 81-88.
6. Roten, D., K. B. Olsen, Y. Cui, and S. M. Day (2015), Quantification of fault zone plasticity effects
with spontaneous rupture simulations, to be submitted to Workshop on Best Practice in
Physics-Based Fault Rupture Models for Seismic Hazard Assessment of Nuclear Installations,
Vienna, Austria, Nov 18-20.
7. Shaw, J. H., A. Plesch, C. Tape, M. P. Suess. T. H. Jordan, G. Ely, E. Hauksson. J. Tromp, T.
Tanimoto, R. Graves, K. Olsen, C. Nicholson, P. J. Maechling, C. Rivero, P. Lovely, C. M. Brankman,
and J. Munster (2015), Unified Structural Representation of the southern California crust and
upper mantle, Earth Planet. Sci. Lett., 415, 1-15, doi:10.1016/j.epsl.2015.01.016.
8. Withers, K. B., K. B. Olsen, S. M. Day (2015). Memory-efficient simulation of frequency
dependent Q, Bull. Seismol. Soc. Am., in revision.

Presentations –
1. Seismological Society of America Meeting 2015 - Lee, E., Thomas, H. J., Chen, P.,
Maechling, J. P., Boué, P., Denolle, M., Beroza, G., & Eymold, W. K. (2015) Full-3D
Tomography of Crustal Structure in Central California. Abstract and presentation, 2015 SSA
Annual Meeting.
2. Blue Waters 2015 Symposiu - SCEC presented the results at the annual Blue Waters
symposium, include the CyberShake calculation, an example of a SCEC, NSF Blue Waters,
and INCITE research collaborative effort. Links to the presentation are posted on a SCEC
wiki at: http://scec.usc.edu/scecpedia/Blue_Waters_Symposium_2015
3. NSF Software Infrastructure for Sustained Innovation 2015 PI Meeting: - SCEC
members presented software descriptions and research results related to our Full 3D
tomography (F3DT) and Unified Community Velocity Model (UCVM) work using INCITE
resource at a Feb 2015 NSF Software Infrastructure for Sustained Innovation (SI2) meeting
Jan 2015. The NSF SI2 program currently provides research funding for software
infrastructure including F3DT, UCVM, AWP-ODC, and Hercules used on our SCEC INCITE
research activities. More details are available via the following link to the meeting website:
https://share.renci.org/SI2PI2015/Lists/SI2PI2015Posters/View_01.aspx
Secondary
 Journal Covers, Awards, Honors, Popularizations
In April 2015, SCEC worked with an INCITE and DOE team to create a display kiosk describing
how SCEC researchers were able to use INCITE resources to perform seismic hazard research for
use in the 2015 Science Bowl. Our primary DOE point of contact for that activity was Carolyn
Lauzon, ALCC Program Manager Advanced Scientific Computing Research U.S. Department of
Energy.

Technical Accomplishments – Please list technical accomplishments such as development of
reusable code resulting in a new tool, new algorithm design ideas or programming
methodologies, formal software releases, etc.
We continued to develop our workflow capabilities on Titan to support our CyberShake
computational goals. We used Pegasus-WMS, Condor DAGManager, Globus-based workflows to
organize and automate our CyberShake SGT workflows and data transfer on Titan.

Other, for example: Simulation results used in outreach initiatives/students graduated or
postdocs deployed; Journal Covers; Awards/Honors –

Highlights – the center creates (concise, short, highly visible) bi-weekly center highlights to
submit to DOE—is your project ready, willing, and able to contribute a highlight?
Center Feedback
 Please answer as applicable: Has the support received from the following been beneficial to
your project team? Cite examples if possible
o User Assistance Center
o Scientific Computing Group
o Visualization and Analysis Team
 Any additional feedback from your project team for the ALCF?
We received beneficial support from ALCF user support group during Q1 regarding migration of
existing project data sets on Mira, under a previous allocation (Equake_SS), to project directories
under our current INCITE allocation (GMSeismicSim). The ALCF User support group gave us
appropriate options (delete existing data, migrate existing data to new allocation), and plenty of
advance notice. They also worked around our on-going simulations on Mira, so that the migration
occurred without impacting our new simulations.
We also received helpful support from the OLCF User assistance group. Before we began our
CyberShake production simulations using Titan, the User assistance team participated on a
telecom with our SCEC research group to review our simulation plans. They provided useful
feedback that helped us make better use of Titan during these runs.
Code Description and Characterization
 Name and provide a description of the primary codes used by your project
 What are the typical production run sizes that your team plans to undertake in the coming
year?
During Q2, we performed a large-scale production seismic hazard calculation that we call
CyberShake Study 15.4 using Titan and Blue Waters. In this production workflow, we calculated
1Hz CyberShake probabilistic seismic hazard curves for 336 site locations. The run took 38 days,
considerably less than the 84 days we estimated. The 38 day makespan includes 13 days of
downtime. This CyberShake Study 15.4 was a cooperative and collaborative technical
accomplishment for the INCITE Leadership class computing organization at OLCF and the NSF
Track 1 computing organization at NCSA, and SCEC, working with California seismic hazard
organizations on an important seismic hazard data product.
For this study, we ran over 4000 jobs on both Blue Waters CPU and GPU nodes and on Titan GPU
nodes and used about 642,000 node-hours on Blue Waters and 426,000 node-hours on Titan, a
total of 888,000 node hours. All CyberShake 15.4 required calculations were defined in a scientific
workflow prior to starting, ensuring repeatability of the study. Study jobs were submitted to
standard user queues on Blue Waters and Titan.
For CyberShake 15.4, it averaged 2640 node-hours/site, which is an increase of 5.8x over the 455
node-hours/site required for CyberShake Study 14.2. We expect to have more details
performance metrics for this study in the next few months, after we processes existing workflow
performance logs gather during the run.
This is the first 1Hz CyberShake hazard model calculation ever performed. 1Hz calculations have
been out of reach until recent performance improvements in our wave propagation code,
development of a GPU version of the code, and performance improvements in automation of our
large many-task computing post-processing. Previous CyberShake studies were done at 0.5Hz.
Doubling the frequency to 1.0Hz requires SGT calculations that do 16x as much work, and postprocessing calculations that do 50x. So, an actual increase of only 5.8x to perform a site calculation
actually represents a very impressive efficiency gain for this kind of heterogeneous, end-to-end,
scientific calculation.


What languages and libraries (scientific, I/O, etc.) are used in each code?
If possible and useful, please indicate which of the following algorithmic motifs appear in each
of your major production codes.
Several of our research group members have used both ALCF and OLCF systems in small or
medium scale simulations in order to evaluate some aspects of our research code on INCITE
systems. However, at this time we expect that our F3DT and CyberShake simulations, the two
main computational projects discussed in this progress report, to use most of our computing time
in the next few months. These two computational projects both make use of different versions of
our AWP-ODC finite difference wave propagation software. For both F3DT and CyberShake, our
SCEC research group is working, during Q2, to increase the number of nodes used by our codes
during our production runs. Our codes scale well, and, with what we believe will be some minor
modifications, we are working to increase the nodes used to 20% or more of more of Mira and
Titan during standard production runs.
Code Dense
Name Linear
Algebra
AWPODC
AWPODCGPUSGT
Sparse
Linear
Algebra
Monte
Carlo
FFTs
Particles
Structured
Grids
Yes
Yes
Unstructured
Grids
AMR