Geoinformatics: A Petascale Cyberfacility for Physics-Based Seismic Hazard Analysis
(SCEC PetaSHA-3 Project)
(NSF EAR - 0949443)
Project Final Report
Performance Period 1 September 2010 – 30 August 2013
Principal Investigator:
Thomas H. Jordan – University of Southern California – Earth Sciences
Senior Scientists:
Greg Beroza - Stanford University - Geophysics
Jacobo Bielak – Carnegie Mellon University – Civil Engineering
Po Chen - University of Wyoming – Geophysics
Yifeng Cui - San Diego Supercomputer Center - Computer Science
Steven Day – San Diego State University – Geological Sciences
Ewa Deelman - USC Information Sciences Institute - Computer Science
Robert Graves - U.S.G.S. - Geophysics
Kim Olsen – San Diego State University – Geological Sciences
1. What are the major goals of the project?
The PetaSHA3 proposal defined three major project goals:
G1. Transform probabilistic seismic hazard analysis (PSHA) into a physics-based science by
deploying a cyberfacility that can execute the computational pathways of Fig. 1 using
TeraGrid resources.
G2.Use this cyberfacility to implement physics-based PSHA and validate the results with
data from Southern California.
G3. Integrate CME numerical modeling and predictive simulation capabilities into the
emerging Geoinformatics infrastructure.
The PetaSHA3 proposal also defined 20 specific project objectives. We will enumerate these
objectives in the next section and describe project-related results for each objective.
Figures mentioned in text are provided as a supplemental PDF document.
2. What was accomplished under these goals (you must provide information for at
least one of the 4 categories below)? For this reporting period describe: 1) major
activities; 2) specific objectives; 3) significant results, including major findings,
developments, or conclusions (both positive and negative); and 4) key outcomes or
other achievements. Include a discussion of stated goals not met.
1
In the following sections, we list the 20 objectives (O1 through O20) as defined in
the PetaSHA3 proposal. We provide a brief description of results for each objective, with
references containing additional detailed information for specific results.
O1. Understand the roles of source directivity, rupture complexity, and basin effects on
ground motions; and evaluate how these factors control the CyberShake hazard curves.
PetaSHA3 researchers generalized the formulation of PSHA to accommodate
simulation-based hazard models [Wang, F., Jordan, T H. (2012)], and from this
generalization they developed a ground motion analysis method called Averaging-Based
Factorization, which allows CyberShake hazard models to be decomposed into components
that can be quantitatively compared with each other and with empirical hazard models,
such as the Next Generation Attenuation (NGA) ground motion prediction equations [Wang,
F., Jordan, T H. (2013)]. These comparisons have been used to examine the dependences of
basin effects, directivity effects, and directivity-basin coupling on the structure of the
pseudo-dynamic rupture models and the Community Velocity Models (CVMs) used in the
large-scale simulations, including the CyberShake Study 13.4 run in April 2013 [Wang, F. et
al., 2013].
An important conclusion from this work is that the CyberShake site and path effects
unexplained by the NGA models account 40-50% of total residual variance, suggesting that
improvements to the simulation-based hazard models could reduced the aleatory
variability intrinsic to the current empirical models by as much as 25%, which would have
huge practical implications for Probabilistic Seismic Hazard Analaysis (PSHA) [Wang, F.,
Jordan, T H. (2013)].
O2. Improve PetaSHA simulation capabilities by incorporating new codes that can
model geologic complexities including topography, geologic discontinuities, and source
complexities such as irregular, dipping, and offset faults.
PetaSHA3 researchers made outstanding progress in improving simulation
capabilities:
 Working with USGS and CGS scientists, we developed UCERF3 models with significant
new complexities in fault geometries, and source complexities in rupture propagation
models [Field, Dawson et al. 2012].
 We developed techniques for adding small scale heterogeneities into 3D velocity models
[Olsen, K. B., W. Savran, B. H. Jacobsen (2013)].
 We developed dynamic rupture codes scalable (to thousands of cores), that can
incorporate complex (non-Cartesian) fault geometry, and advanced thermomechanical
models [Shi, Z., S.M. Day, and G. Ely (2012))].
 We developed dynamic rupture simulations that can incorporate multiple physical
models such as frictional breakdown, shear heating, porothermoelastic flow (and the
resulting effective normal-stress fluctuations), as well as multiscale fault roughness
[Roten, D., K. B. Olsen, J. C. Pechmann (2012)].
 We used SORD to perform three-dimensional numerical calculations of dynamic rupture
along non-planar faults to study the effects of fault roughness (self-similar over three
orders of magnitude in scale length) on rupture propagation and resultant ground
motion [Shi, Z., and S. M. Day (2013)]. The present simulations model seismic wave
excitation up to ~10 Hz with rupture lengths of ~100 km, permitting comparisons with
empirical studies of ground- motion intensity measures of engineering interest.
 We continued Hercules development incorporating free-surface topography and the
influence of the built environment in the modeling and simulation scheme in Hercules
2
[Restrepo, D. (2013); Isbiliroglu, Y., R. Taborda, and J. Bielak, J. (2013), Restrepo, D.
(2013)]. This team developed a finite-element based methodology that uses special
integration techniques to account for an arbitrary free-surface boundary in the
simulation domain but preserves the octree- based structure of the code, and thus does
not have a significant effect on performance.
O3. Use dynamic rupture simulations to investigate the effects of realistic friction laws,
geologic heterogeneities, and near-fault stress states on seismic radiation and thereby
improve pseudo-dynamic rupture models of hazardous earthquakes.
PetaSH3 researchers developed rough fault model simulations [Withers, K., K. B.
Olsen, S. Shi, S. M. Day, and R. Takedatsu (2013)]. We used the SORD code [Ely, G., (2013c)]
as a tool for dynamic simulation of geometrically and physically complex ruptures. To do so,
we integrated high-speed frictional weakening (in a rate- and state-dependent formulation)
into the code. This integration was done using a method that time-staggers the state and
velocity variables at the split nodes, producing a stable, accurate and very efficient solution
scheme. We also added the Drucker-Prager formulation of pressure-dependent plastic
yielding into SORD, with added viscoplastic terms to suppress strain localization. The
resulting code was successfully tested using SCEC rupture dynamics benchmarks. We also
implemented and successfully tested a scheme for the generation of SORD meshes for
power-law rough faults.
O4. Use realistic earthquake simulations to evaluate static and dynamic stress transfer
and assess their effects on strain accumulation, rupture nucleation, and stress release.
PetaSHA3 researchers used the earthquake rupture simulator RSQSim of J. Dieterich
and K. Richards-Dinger (2010) to develop a new theoretical approach for analyzing fault
rupture synchronicity [Milner, K.R., Thomas H. Jordan (2013)]. The main object of this
analysis is the complete set of interevent time differences, which can be characterized in
terms of the auto-catalog density function (ACDF) and the cross-catalog density function
(CCDF). The ACDF and CCDF have roles in synchronicity theory similar to those of
autocorrelation and cross-correlation functions in time-series analysis. For RSQSim at the
magnitude threshold M = 7, the ACDF can be well fit by renewal models with fairly small
aperiodicity parameters (α < 0.2) for most major faults (an exception is the San Jacinto
fault). At interseismic (Reid) time scales, we observe pairs of fault segments that are tightly
locked, such as the Cholame and Carrizo sections of the San Andreas Fault (SAF), where the
CCDF and two ACDFs are nearly equal; segments out of phase (Carrizo- SAF/Coachella-SAF
and Coachella-SAF/San Jacinto), where the CCDF variation is an odd function of time; and
segments where events are in phase with integer ratios of recurrence times (2:1
synchronicity of Coachella-SAF/Mojave-SAF and Carrizo-SAF/Mojave-SAF). At near-seismic
(Omori) time scales, we observe various modes of clustering, triggering, and shadowing in
RSQSim catalogs; e.g., events on Mojave-SAF trigger Garlock events, and events on
Coachella-SAF shut down events on San Jacinto. Therefore, despite its geometrical
complexity and multiplicity of time scales, the RSQSim model of the San Andreas fault
system exhibits a variety of synchronous behaviors that increase the predictability of large
ruptures within the system. A key question for earthquake forecasting is whether the real
San Andreas system is equally, or much less, synchronous. If so, the predictability of large
ruptures and sequences of ruptures may be higher than previously thought.
O5. Investigate the upper frequency limit of deterministic ground-motion prediction by
comparing simulations with observed seismograms using goodness-of-fit measures of
engineering relevance.
3
PetaSHA3 researchers used Hercules software, at scale of 10,000s of cores, to
generate synthetics up to 5Hz+ and quantify the spatially varying fit between observation
and simulations using engineering-oriented goodness-of-fit algorithms [Taborda, R. and
Bielak, J. (2012, 2013a)]. We evaluated the amplification effects of the near-surface material
(Vs<500m/s). The CMU group performed a set of simulations for the Mw 5.4 2008 Chino
Hills, California earthquake using the various Southern California Velocity Models (CVM-S,
CVM-H and CVM-H+GTL) [Taborda, R. and Bielak, J. (2013b,c, 2014)]. The simulations were
designed to produce a valid representation of the ground motion up to a maximum
frequency of 4 Hz [Taborda, R. and Bielak, J. (2013d)]. We compared the results of
simulations of the Chino Hills earthquake with seismic records obtained from Southern
California strong motion networks. In total, we compared simulation results against data in
336 stations. The quality of the match between the actual records and the simulated
synthetics was measured in terms of a commonly used engineering-oriented goodness-of-fit
(GOF) criterion. .
PetaSHA3 researchers used AWP-ODC [Olsen, K.B., and J. Mayhew (2010)] up to
2Hz+ and quantified the spatially varying fit between observation and simulations using
engineering-oriented goodness-of-fit algorithms. The SDSU/SDSC group used the 2008
Mw5.4 Chino Hills, CA, earthquake, for most of the analysis in this section. The event lends
itself well to the project, as it is well recorded, and sufficiently small to minimize finite fault
effects at higher frequencies.
O6. Improve the SCEC 3D community velocity models by automated, iterated, full-3D
inversion of large suites of observed waveforms from the Southern California Seismic
Network.
PetaSHA3 researchers ran full-3D tomographic inversions using CVM-S4 as a
starting model, generating a sequence of 26 iterations [Lee, E., Chen, P., Jordan, T. H.,
Maechling, P. J., Denolle,M., and Beroza, G. C. (2013)]. The new Community Velocity Model,
CVM-S4.26, represents a substantial improvement in the seismic velocity structure of
Southern California.
The PetaSHA software group integrated the updated CVM-S4.26 model into the
Unified Community Velocity Model (UCVM) software [Gill, D. and Maechling, P. and Jordan,
T. and Taborda, R. and Callaghan, S. and Small, P. (2013)] for general distribution to the
seismological community. The model is now being prepared for use in CyberShake
simulations.
O7. Use the cross-correlation of the microseismic noise to extract the anelastic Green
functions between pairs of seismometers, including temporary stations, and
incorporate these data into full-3D inversions.
CVM-S4.26 was developed by PetaSHA3 researchers by fitting ambient-noise Green
functions as well as earthquake waveforms [Lee, E., Chen, P., Jordan, T H., Maechling, P J.,
Denolle, M., and Beroza, G C. (2012)]. The ambient-noise data provided considerable
improvement to the upper-crustal structure of Southern California and will thereby provide
improved simulation-based hazard assessments.
O8. Validate CyberShake using hazard-curve parameters robustly estimated from the
NGA strong-motion database and signal-structure metrics developed from strongmotion data by earthquake engineers.
All CyberShake models for Southern California, including the CyberShake Study 13.4
run in April 2013, have been fully evaluated against the NGA models using the new
4
technique of Averaging-Based Factorization, developed in this project [Wang, F. et al., 2013].
See O1 for additional details.
O9.Validate CyberShake using existing seismic data and surveys of precariously
balanced rocks in Southern California.
USC Geoscience graduate student, working with PetaSHA3 PI Thomas Jordan,
developed a new technique for analyzing CyberShake Hazard Models using precariously
balanced rock s [Donovan, J., T. H. Jordan, and J. Brune (2012)]. Precariously balanced rocks
(PBRs) are fragile geologic features that are expected to be overturned by ground
accelerations less than about 0.5 g and represent the only currently available dataset for
constraining maximum ground motions over multiple large earthquake cycles. We applied
PBR constraints to test the 2011 CyberShake hazard model (Graves et al., 2011), which is
based on the Uniform California Earthquake Rupture Forecast, version 2 (UCERF2). We
used a catalog of 17 PBR locations within the southern California CyberShake region and
compute overturning probabilities for the ground motions predicted by the CyberShake
hazard model. We found a discrepancy between high overturning probabilities for some
PBRs and the observation that these rocks are not overturned, particularly in the Mojave
section of the San Andreas Fault. We described the implications of these discrepancies for
future versions of UCERF as well as future CyberShake runs.
O10. Extend the frequency range, aerial density, and geographic extent of the
CyberShake database and provide database access to researchers outside the CME
Collaboration.
One of this project‘s most significant results was the generation of four new
CyberShake hazard models [Callaghan, S., Maechling, P., Juve, G., Vahi, K., Graves, R. W.,
Olsen, K. B., Gill, D., Milner, K., Yu, J. and Jordan, T. H. (2013)], which were calculated using
NSF and XSEDE resources Blue Waters and Stampede. The model were computed using
different HPC codes (AWP-ODC, and RWG), different velocity models (CVM-S4 and CVMH11.9), and the Graves & Pitarka (2010) rupture generator (GP-10).
We applied the Averaging-Based Factorization (ABF) technique of Wang & Jordan
(2013) to compare CyberShake models and assess their consistency with the hazards
predicted by the Next Generation Attenuation (NGA) models [Wang, F. et al. (2013)]. ABF
uses a hierarchical averaging scheme to separate the shaking intensities for large ensembles
of earthquakes into relative (dimensionless) excitation fields representing site, path,
directivity, and source-complexity effects, and it provides quantitative, map-based
comparisons between models with completely different formulations. The CyberShake
directivity effects are generally larger than predicted by the Spudich & Chiou (2008) NGA
directivity factor, but those calculated from the GP-10 sources are smaller than those of GP04, owing to the greater incoherence of the wavefields from the more complex rupture
models. Substituting GP-10 for GP-04 reduces the CyberShake-NGA directivity-effect
discrepancy by a factor of two, from +36% to +18%. The CyberShake basin effects are
generally larger than those from the three NGA models that provide basin-effect factors.
However, the basin excitations calculated from CVM-H are smaller than from CVM-S, and
they show a stronger frequency dependence, primarily because the shear velocities in the
deeper parts of the basins are, on average, lower in CVM-H. Owing to this difference, the
substitution of CVM-H for CVM-S reduces the CyberShake-NGA basin-effect discrepancy.
Among the NGA models, that of Abrahamson & Silva (2008) is the most consistent with the
CyberShake CVM-H calculations, with a basin-effect correlation factor greater than 0.9
across the frequency band 0.1-0.3 Hz. We used these comparisons to draw conclusions
5
regarding strategies for reducing epistemic uncertainties in simulation-based hazard
models [Wang, F. and Jordan, T. H. (2013)].
O11. Develop SHA codes that achieve performance of sustained petaflops on planned
petascale open- science computers and use these codes to better understand
earthquake processes.
PetaSHA3 researchers developed highly parallel, and highly efficient, Cuda-language
wave propagation code called AWP-ODC-GPU [Cui, Y., K.B. Olsen, J. Zhou, P. Small, A.
Chourasia, S. M. Day, P. J. Maechling, T. H. Jordan. (2012)]. This code achieves sustained
Petaflops [Cui, Y., E. Poyraz, K. Olsen, J. Zhou, K. Withers, S. Callaghan, J. Larkin, C. Guest, D.
Choi, A. Chourasia, Z. Shi, S. Day, P. Maechling, and T. H. Jordan (2013)] and was used to run
a 10Hz deterministic ground motion simulation [Withers, K., K. B. Olsen, S. Shi, S. M. Day,
and R. Takedatsu (2013)] using a high frequency earthquake rupture produced by a
dynamic rupture on a rough fault [Shi, Z., S.M. Day, and G. Ely (2012)], in a velocity model
containing small scale heterogeneities [Olsen, K. B., W. Savran, B. H. Jacobsen (2013)].
Co-sponsored with USGS/NEHRP, PetaSHA3 researchers have estimated 0-10Hz
ground motions in the Salt Lake Basin (SLB) during M 7 earthquakes on the Salt Lake City
(SLC) segment of the Wasatch fault (WFSLC) [Roten, D., K. B. Olsen, J. C. Pechmann (2012),
Roten, D., K. B. Olsen, J. C. Pechmann, V. M. Cruz-Atienza, and H. Magistrale (2011)].
O12. Develop computational platforms capable of running long-duration SHA
calculations and increase the CME and solid earth community usage of national HPC
computing resources.
Our PetaSHA3 research group calculated four new CyberShake hazard maps
[Callaghan, S., Maechling, P., Juve, G., Vahi, K., Graves, R. W., Olsen, K. B., Gill, D., Milner, K.,
Yu, J. and Jordan, T. H. (2013)], as described above. This large-scale production research
calculation ran for nearly two months using NSF’s two newest supercomputers, Blue Waters
and Stampede.
O13. Develop and verify open-source SHA codes and incorporate them into vertically
integrated computational platforms that efficiently utilize the current terascale and
future petascale computing resources including capability computing, capacity
computing, large-scale data management, networking, storage, and visualization
resources.
PetaSHA3 researchers ran simulated annealing research calculation in the
development of the UCERF3 [Field, E., Dawson, T.E., et al. (2012))] using XSEDE Stampede.
Four new CyberShake hazard maps [Callaghan, S., Maechling, P., Juve, G., Vahi, K., Graves, R.
W., Olsen, K. B., Gill, D., Milner, K., Yu, J. and Jordan, T. H. (2013)] were calculated using Blue
Waters and Stampede. Newly developed AWP-ODC-GPU software was used for both
forward simulations and on DOE supercomputer Titan [Cui, Y., E. Poyraz, K. Olsen, J. Zhou, K.
Withers, S. Callaghan, J. Larkin, C. Guest, D. Choi, A. Chourasia, Z. Shi, S. Day, P. Maechling,
and T. H. Jordan (2013)] and reciprocity-based CyberShake simulations on Blue Waters
[Cui, Y., E. Poyraz, J. Zhou, S. Callaghan, P. Maechling, T. H. Jordan, L. Shih, and P. Chen
(2013)].
O14. Establish a simulation data management system that enables researchers to
locate and retrieve significant datasets from CME simulations using community-based
data management standards.
The PetaSHA3 group also developed a collaborative web site
[http://scec.usc.edu/SCECpedia] that is used to coordinate distributed collaborative
6
research activities. Web site statistics on this site show that the site received more than
6000 unique visitors during the PetaSHA3 project.
The PetaSHA3 research group developed a prototype CyberShake data access
[https://scec.usc.edu/it/CyberShake_Data_Request] site to distribute results from the
CyberShake 13.4 research calculation.
The PetaSHA3 research group developed the Websims data access site [Olsen, K.B.,
and G. Ely (2009)] as an easy-to-use data management system that can be used to distribute
ground motion simulation results to researchers.
O15. Improve interoperability of CME computational platforms with observational
geoinformatics systems including the EarthScope, ANSS, IRIS, and NEES systems.
PetaSHA3 researchers at SDSC developed selected applications with a selfdescribing machine-independent HDF5 binary file format that supports scalable parallel I/O
performance [Cui, Y., K.B. Olsen, J. Zhou, P. Small, A. Chourasia, S.M. Day, P.J. Maechling, T.H.
Jordan. (2012)]. This code development helps SCEC wave propagation codes produced
simulation results in a well-recognized, self-describing, file format.
PetaSHA3 researchers implemented capabilities within the UCVM software to
export California CVMs in netCDF format, a data format supported by IRIS [Gill, D.,
Maechling, P., Jordan, T., Taborda, R., Callaghan, S. and Small, P. (2013)]. With this software
capability implemented in UCVM, SCEC can now provide updated California CVMs to IRIS in
their preferred CVM format.
O16. Promote use of modern software engineering practices within the geoscience HPC
community including source-code management, acceptance and regression testing, and
documentation on scientific computing projects.
The PetaSHA3 project has improved code scientific and computing performance
including AWP [Christen, M., O. Schenk, and Y. Cui (2012)], Hercules [Taborda, R. and Bielak,
J. (2013a)], and SORD [Ely, G., (2013a)]. We also implemented new capabilities such as GPU
versions of AWP-ODC [Cui, Y., E. Poyraz, K. Olsen, J. Zhou, K. Withers, S. Callaghan, J. Larkin,
C. Guest, D. Choi, A. Chourasia, Z. Shi, S. Day, P. Maechling, and T. H. Jordan (2013)] and
CyberShake[Cui, Y., E. Poyraz, J. Zhou, S. Callaghan, P. Maechling, T. H. Jordan, L. Shih, and P.
Chen (2013)]. PetaSHA3 researchers have also released open-source distributions of the
UCVM [Gill, D., Maechling, P., Jordan, T., Taborda, R., Callaghan, S. and Small, P. (2013)] code
for managing 3D velocity models.
The PetaSHA3 CMU group ported Hercules to a new distributed version control
system (Git) hosted in a more easily accessible repository (GitHub). This has allowed us to
share the code with other SCEC researchers and provides the framework for a sustainable
software development in the future.
O17. Build collaboration between geoscientists and computer scientists that can apply
petascale technology to socially relevant research in earthquake system science.
The PetaSHA3 project was collaborative research effort involving geoscientists and
computer scientists. The PetaSHA3 team developed sustained Petaflops GPU code [Cui, Y., E.
Poyraz, K. Olsen, J. Zhou, K. Withers, S. Callaghan, J. Larkin, C. Guest, D. Choi, A. Chourasia, Z.
Shi, S. Day, P. Maechling, and T. H. Jordan (2013)] as collaboration between SDSC computer
scientists, UCSD graduate students, and SDSU and USC Geophysics researchers. PetaSHA3
researchers also developed a CyberShake workflow system capable of running 100M task
workflows with USC, SDSU, and USGS Geoscientists, and USC ISI computers scientists
[Callaghan, S., Maechling, P J., Milner, K., Graves, R W., Donovan, J., Wang, F., Jordan, T H
(2012)].
7
Our PetaSHA3 project also formed an engineering and computer science
collaboration to improve the algorithm used in Hercules to incorporate anelastic
attenuation and obtained an additional 20–30 percent speed-up in its performance by
eliminating unnecessary computational steps in the case of infinite Q for dilatational
deformation [Bielak, J., H. Karaoglu, and R. Taborda, 2011].
O18. Equip a diverse scientific workforce with the tools to formulate and verify models,
run simulations in a petascale environment, validate the predictions against
observation, and assimilate data into model improvements.
The PetaSHA project is designed to implement this iterative cycle of observation,
model development, evaluation, and improvement. Participants on PetaSHA3 research have
demonstrated their ability to perform the full model development lifecycle during the
PetaSHA3 project, and as a result, have developed outstanding computational science skills
and applied them to practical seismic hazard questions.
The PetaSHA3 team also developed the capability for collocated and concurrent
visualization production pipeline for simulations on current and future high performance
resources. Developed novel visual representations and products for dissemination of
research to a broad range of audience. In our GlyphSea visualization study [Chourasia, Amit,
E. McQuinn, B. Minster and J. Schulze (2011)], we explored a new general approach that
allows much more flexibility: the use of Glyphs. The applicability of the method is very
general and not discipline specific, and the software may be implemented on a wide variety
of platforms, from laptop computers with a single 2D screen, to supercomputers driving a
4D visualization “Cave”. The outcome was extremely successful, and was presented at
numerous scientific meetings, both domain-specific (such as AGU) and IT meetings (IEEE,
SPIE).
O19. Motivate a new generation of system scientists by demonstrating how HPCenabled predictive modeling can transform basic and applied research.
Caltech seismologist Maren Boese used PetaSHA3 CyberShake results to train a
neural network to improve rapid ground motion estimates for use in Earthquake Early
Warning (EEW) application [Boese, M., Graves, R W., Callaghan, S., Maechling, P J. (2012)],
an example of how PetaSHA3 HPC results were used in an unanticipated, non-HPC, broad
impact applications.
PetaSHA3 project-supported graduate student research at UCSD and SDSU
developed a cross-domain vector visualization technique. Earthquake simulations output
vector and tensor fields, such as ground velocity vectors. Vector data is often reduced to
scalar magnitude quantity and plotted either as pseudo-color 2D plots or volume rendering.
Visualization of vector data is also performed by plotting very sparse 2D/3D streamlines,
stream tubes, stream surfaces, line integral convolution (LIC), and particle advection. Even
though these methods are useful, they are insufficient to characterize adequately the
underlying detail, richness and significance of the data. Moreover, particle advection
methods are suitable for flow data, but are not adequate to visualize vector data produced
by earthquake simulations that are similar to vibration where the particles do not move far
from their original location. UCSD graduate Master’s thesis, by Emmett McQuinn, received
an honorable mention award by the National Science Foundation and the AAAS Journal
Science in the 2010 International Science & Engineering Visualization Challenge. The award
was given to A. Chourasi, E. McQinn, J.B, Minster and J. Schulze. The resulting package
SEAGLYPH [Chourasia, Amit, E. McQuinn, B. Minster and J. Schulze (2011)] is easily
obtained, and has been applied to other field, such as Early Universe cosmology.
8
A USC PhD student (Feng Wang) developed a PSHA analysis technique called
Averaging-Based Factorization that make use of PetaSHA3 simulation results produced by
the CyberShake 13.4 simulation results [Wang, F., Jordan, T H. (2012)].
O20. Cross-train diverse groups of undergraduate interns and early-career scientists in
geoscience and computer science and teach them how to solve fundamental problems.
PetaSHA team members from SCEC and CMU completed an initial collaborative
seismology and engineering study on the effect of the built environment on the ground
motion, in which we incorporated the presence of buildings in simulations using simplified
models that capture the principal characteristics of the dynamic response of multiple
structures in regular building clusters and the interactions of the buildings with the ground,
and between the buildings themselves [Isbiliroglu, Y. and Taborda, R. and Bielak, J. (2013)].
We found that interaction effects increase with the number of buildings and density of the
cluster (i.e., smaller separation between buildings).
An interdisciplinary team of PetaSHA3 researchers including J.B. Minster (UCSD),
graduate student Emitt McQuinn (UCSD), and computer scientists A. Chourasia (SDS)
developed a visualization technique that can simultaneously be (1) easily perceived by nonspecialists, but (2) be amenable to quantitative interpretation by domain experts
[Chourasia, Amit, E. McQuinn, B. Minster and J. Schulze (2011)].
3. What opportunities for training and professional development has the project
provided?
Two post-doctoral researchers as well as three geoscience students were employed
on the project within the SDSU and SDSC PetaSHA project teams. The post-doctoral
researchers and students obtained advanced training in high- performance computing,
rupture dynamics and 3D ground motion simulations during the project. They furthermore
presented their work at international conferences and gained important technical
communication skills during the project. Graduate student Emmett McQuinn worked with
UCSD and SDSC for the bulk of this work, under a SCEC Graduate Student ACCESS
scholarship. After securing an MS in Computer Science, Emmett had opportunities to
pursue a PhD at SDSC or USC, or SIO, but preferred to move to Stanford to a more lucrative
career in computer graphics applied to Biomedicine. [His work earned him the cover of
Science, in Feb 2012 http://www.sciencemag.org/content/339/6119.cover-expansion ].
Amit Chourasia (SDSC), Jean-Bernard Minster (SIO), and Jürgen P. Schulze (CalIT2) were
his mentors during the completion of this work.
UCSD PhD students Jun Zhou and Efecan Poyraz were trained through this project
and the project mentored Jun Zhou during development of his Ph.D thesis project.
At Carnegie Mellon University, postdoctoral mentoring was part of the activities to
help prepare Dr. Ricardo Taborda for an academic career. As a postdoctoral fellow in the
Computational Seismology Laboratory at Carnegie Mellon (CMU), Dr. Taborda assisted CoPI Bielak on advising three graduate students, writing journal publications and preparing
poster and oral presentations (see publications). At CMU, Taborda has been particularly
involved in co- advising the Ph.D. thesis work of graduate student Yigit Isbiliroglu, whose
topic of research is a continuation of Taborda’s Ph.D. work on the effects of the built
environment on the ground motion during strong earthquakes and the coupled soilstructure interaction effects on the dynamic response of building clusters. Taborda has also
served as the primary liaison between SCEC/IT group and the Quake Group at CMU, and has
been closely involved with the development of the Unified California Velocity Model
(UCVM). He has used datasets generated using UCVM to conduct his own research on the
validation of the various seismic velocity models available for Southern California (CVM-S,
9
CVM-H). During the spring of 2013, Taborda interviewed at several universities and secured
a position as a new Assistant Professor at the University of Memphis (U of M), starting
August 2013. At the U of M, Taborda has joined the faculty of the Civil Engineering
Department in the School of Engineering and has a joint tenure-track appointment with the
Center for Earthquake Research and Information (CERI). CERI is a University of Memphis
Center of Excellence. We expect that Taborda will continue to collaborate with SCEC and
CMU from CERI. Two other graduate students, Haydar Karaoglu and Doriam Restrepo
participated in the research activities at CMU. Doriam Restrepo has successfully defended
his Ph.D. thesis and will be joining the faculty at the University of EAFIT in Medellin,
Colombia, and Haydar Karaoglu is expected to complete his Ph.D. studies in May 2014.
At USC, three graduate students participated in the project: F. Wang, J. Donovan, and
K. Milner. Wang graduated with a PhD in September, and he is now employed in the position
of Scientist at AIR Worldwide, applying his SCEC-based research to hazard and risk analysis
in the commercial sector. Donovan is expected to receive her PhD and Milner his MS in
2014.
4. How have the results been disseminated to communities of interest?
PetaSHA3 project members have presented and discussed our work in a series of
geoscientific and computational science meeting and workshops during the PetaSHA3
project including the following:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
SCEC Annual Meeting, Sept 9-12, 2013 Palm Springs, CA
IASPEI Joint Assembly, July 25, 2013 Gothenburg, Sweden
SCEC CME Project Meeting, 2 June 2013, Palm Spring CA
SSA April 17-19, 2013 Salt Lake City, Utah
Ground Motion Simulation Validation Technical Activity Group Workshop, April 3, 2013
Organizational Meetings of a SCEC Committee for Utilization of Ground Motion
Simulations, April 3, 2013 Los Angeles CA
CIG-QUEST-IRIS Workshop: Seismic Imaging of Structure and Source. July 14-17, 2013.
University of Alaska Fairbanks
Fall AGU, Dec 3-7, 2012 – San Francisco, CA
Global Earthquake Model Annual Meeting, Dec 9-13, 2012– Pavia, Italy
Lawrence Livermore National Laboratory HPC in Geophysics Meeting, Nov 8-9, 2012,
LLNL, CA
Invited Speaker, HPC China Workshop at SC’12, Nov 13, 2012, International Workshop
on CO-DESIGN, Beijing, Oct 23-25, 2012
SCEC Ground Motion Simulation Validation Workshop Sept 9, 2012, Palm Springs, CA
REAKT Meeting Oct 8-10, 2012 Potsdam Germany
SCEC Annual Meeting, Sept 10-12, 2012 Palm Springs, CA
Building Community Codes for Effective Scientific Research on HPC Platforms Sept 6-7,
2012, Chicago Illinois
SIAM PP’12, Savannah, Feb 15-17, 2012, International Workshop on CO-DESIGN,
Beijing, Oct 25-26, 2011
7th Int’l APEC Cooperation for Earthquake Simulation (APES) Workshop, Otaru, Oct 3-8,
2010, Int’l Conference of Numerical Analysis and Applied Mathematics (ICNAAM’10),
Rhodes, September 19-25, 2010
We have also worked with NSF and DOE scientific communications experts to develop
scientific summaries of our recent accomplishments and impact. Several press articles,
10
including NSF Discoveries articles, included descriptions of SCEC’s use of NSF computer
during year 1. Links to these online press articles include the following:
1. http://www.hpcwire.com/hpcwire/2013-0624/sdsc_geocomputing_lab_named_winner_of_hpc_innovation_excellence_award_by_idc.
html
2. http://www.universityofcalifornia.edu/news/article/29677
3. http://www.nsf.gov/discoveries/disc_summ.jsp?cntn_id=127385&org=NSF
4. http://www.nsf.gov/news/news_summ.jsp?cntn_id=127193
5. http://www.nsf.gov/news/news_summ.jsp?cntn_id=127194&org=NSF&from=news
6. http://www.sciencedaily.com/releases/2013/04/130402144525.htm
7. http://www.foxnews.com/tech/2013/03/27/long-delayed-blue-waterssupercomputer-illinois/
8. http://www.hpcwire.com/hpcwire/2013-03-28/nsf_official_on_new_supers_dataintensive_future.html
9. http://www.hpcwire.com/hpcwire/2013-0327/tacc_unveils_dell_hpc_system_stampede.html
10. http://www.tacc.utexas.edu/news/feature-stories/2013/texas-unleashes-stampede
11. http://www.tacc.utexas.edu/news/feature-stories/2013/texas-unleashesstampede/earthquakes
12. http://www.datacenterknowledge.com/archives/2013/03/28/stampedesupercomputer-goes-live/?goback=.gde-87791-member-227770000
13. https://www.youtube.com/watch?v=4ij3XDLXHvg
14. http://www.sciencecodex.com/uc_san_diego_team_achieves_petafloplevel_earthquake_
simulations_on_gpupowered_supercomputers-109666
15. http://www.hpcwire.com/hpcwire/2013-03-28/nsf_official_on_new_supers_dataintensive_future.html
16. http://www.kurzweilai.net/blue-waters-one-of-the-worlds-most-powerful-computersopens-for-research
17. http://esciencenews.com/articles/2013/04/02/uc.san.diego.team.achieves.petaflop.lev
el.earthquake.simulations.gpu.powered.supercomputers
18. http://www.utexas.edu/know/2013/03/27/texas-unleashes-stampede-for-science/
We have hosted a series of computational workshops at SCEC on PetaSHA3 scientific
issues including UCVM (April 2012), CyberShake (April 2012), and SCEC Data Management
(Feb 2014). Details from these meetings are posted on the public SCECpedia wiki.
PetaSHA3 visualizations from our M8 research received national recognition with
honorable mention in scientific visualizations for the year. This award was widely covered
in the electronic press, such as HPCWire, Futurity.org, and Scientific Computing. These sites
showed images from our visualizations to international audience. M8 visualizations won a
TeraGrid 11 Visualization Award and also received an Office of Advanced Scientific
Computing Research (OASCR) awards announced at SciDAC 2011 conference. Images from
the SCEC M8 simulation are used in the NSF Cyberinfrastructure for the 21st Century
Science and Engineering Advanced Computing Infrastructure Vision and Strategic Plan (NSF
Document nsf12051).
The SCEC PetaSHA3 researchers make use of wiki’s to communicate our research
among ourselves. By default, research communications are open. Many PetaSHA3 research
efforts, including the wave propagation simulation work are described in some detail on the
SCEC wiki with the following home page: http://scec.usc.edu/scecpedia. Google analytics
11
for this site from 1 Sept 2010 through 30 August 2013 show 18,511 visits, from 6,600
unique visitors, with 80,016 pageviews, averaging 4.32 pages/visit.
5. What do you plan to do during the next reporting period to accomplish the goals?
This is our final project report.
6. Products - What has the project produced?
Publication List:
1. Bielak, J., Karaoglu, H., Taborda, R. (2011). Memory-efficient displacement-based
internal friction for wave propagation simulation. Geophysics, 76(6), T131–T145. doi:
10.1190/geo2011-0019.1.
2. Boese, M., Graves, R W., Callaghan, S., Maechling, P J. (2012) Site-specific Ground-Motion
Predictions for Earthquake Early Warning in the LA Basin using CyberShake
Simulations Abstract S53B-2498 Poster presented at 2012 Fall Meeting, AGU, San
Francisco, Calif., 3-7 Dec.
3. Callaghan, S., Maechling, P., Gideon Juve, Karan Vahi, Robert W. Graves, Kim B. Olsen,
David Gill, Kevin Milner, John Yu and Thomas H. Jordan (2013) Running CyberShake
Seismic Hazard Workflows on Distributed HPC Resources, SCEC Annual Meeting 2013,
abstract 195, Sept 8 – 11, 2013, Palm Springs, CA
4. Callaghan, S., Maechling, P J., Milner, K., Graves, R W., Donovan, J., Wang, F., Jordan, T H
(2012) CyberShake: Broadband Physics-Based Probabilistic Seismic Hazard Analysis in
Southern California Abstract S51A-2405 presented at 2012 Fall Meeting, AGU, San
Francisco, Calif., 3-7 Dec.
5. Chavez, M., K. B. Olsen, E. Cabrera, and N. Perea (2011). Observations and Modeling of
Strong Ground Motions for the 9 October 1995 Mw 8 Colima-Jalisco, Mexico,
Earthquake, Bull. Seis. Soc. Am. 101, 1979-2000.
6. Chourasia, A., Zhou, J., Cui, Y., Choi, DJ, Olsen, K. (2012) Role of visualization in porting a
seismic simulation from CPU to GPU architecture (Visualization Showcase), XSEDE’12,
Chicago, July 16-20, 2012.
7. Chourasia, Amit, E. McQuinn, B. Minster and J. Schulze (2011) Glyphsea: An Application
to Visualize Vector Data http://visservices.sdsc.edu/projects/scec/vectorviz/glyphsea/
8. Christen, M., O. Schenk, and Y. Cui (2012) PATUS for Convenient High-Performance
Stencils: Evaluation in Earthquake Simulations, Technical Paper, SC12, Salt Lake City,
Nov 10-16, 2012.
9. Cui, Y., E. Poyraz, K. Olsen, J. Zhou, K. Withers, S. Callaghan, J. Larkin, C. Guest, D. Choi, A.
Chourasia, Z. Shi, S. Day, P. Maechling, and T. H. Jordan (2013), Physics-based seismic
hazard analysis on petascale heterogeneous supercomputers, SC13, Denver, Nov 17-22,
2013 (accepted for publication)
10. Cui, Y., E. Poyraz, J. Zhou, S. Callaghan, P. Maechling, T. H. Jordan, L. Shih, and P. Chen
(2013), Accelerating CyberShake Calculations on the XK7 Platform of Blue Waters,
Extreme Scaling Workshop, Denver, August 15-16, 2013
11. Cui, Y., K.B. Olsen, J. Zhou, P. Small, A. Chourasia, S.M. Day, P.J. Maechling, T.H. Jordan.
(2012). Development and optimizations of a SCEC community anelastic wave
propagation platform for multicore systems and GPU-based accelerators, Seism. Res.
Lett. Seism. Res. Lett. 83:2, 396.
12. Cui, Y., Olsen, K., Jordan, T., Lee, K., Zhou, J., Small, P., Ely, G., Roten, D., Panda, DK,
Chourasia, A., Levesque, J., Day, S. and Maechling, P. (2010) Scalable Earthquake
Simulation on Petascale Supercomputers, Gordon Bell Finalist, Supercomputing’10, 120, New Orleans, Nov, 2010.
12
13. Cui, Y. (2010) Looking forward to Architecture Changes with Seismic Wave Propagation
Using a 3D Finite Difference Code, Int'l Conference of Numerical Analysis and Applied
Mathematics, pp. 1781, edited by T. E. Simos, G. Psihoyios, and Ch. Tsitouras, Rhodes,
Greece, 19-25 September 2010.
14. Day, S.M., D. Roten, and K.B. Olsen (2012). Adjoint analysis of the source and path
sensitivities of basin-guided waves, Geophys. J. Int. , Vol 189, pp. 1103-1124, doi:
10.1111/j.1365-246X.2012.05416.x
15. Day, S.M., K.B. Olsen, and Y. Cui (2011). Large-scale earthquake simulations and the
prediction of strong ground motion (invited talk), SIAM Conference on Mathematical
and Computational Issues in the Geosciences, March 21-24, 2011, Long Beach.
16. Donovan, J., T. H. Jordan, and J. N. Brune (2012). Testing CyberShake using precariously
balanced rocks, 2012 Annual Meeting of the Southern California Earthquake Center,
Palm Springs, Abstract 026, September, 2012.
17. Donovan, J., and T. H. Jordan (2013). Forecasting The Rupture Directivity Of Large
Earthquakes: Constraints From Observations And Earthquake Simulators, Seismological
Society of America Meeting, April 17-19 2013, Salt Lake City, SRL (Vol. 84, No. 2)
18. Ely, G., (2013c) Kernel Optimizations in SORD Earthquake Dynamic Rupture Code, Mira
Community Conference, Argonne Leadership Computing Facility, March 7, 2013
19. Ely, G., (2013b) Threads in SORD Earthquake Dynamic Rupture Code, Mira Community
Conference, Argonne Leadership Computing Facility, March 6, 2013
20. Ely, G., (2013a) The SORD Code for Rupture Dynamics, SIAM Conference on
Computational Science and Engineering, Boston, Feb 25-Mar 1, 2013
21. Ely, G., (2012) Improving Earthquake Ground Motion Estimates with Blue Gene/Q, SIAM
Annual Meeting, July 9-13, 2012
22. Field, E., Dawson, T.E., et al. (2012) Uniform California Earthquake Rupture Forecast,
Version 3 (UCERF3) Framework, Working Group on California Earthquake Probabilities
(WGCEP) Technical Report #8 July 9, 2012
23. Gill, D., Maechling, P., Jordan, T., Taborda, R., Callaghan, S. and Small, P. (2013). SCEC
Unified Community Velocity Model: Mesh Generation and Visualization. In Proc.
CIG/QUEST/IRIS Joint Workshop on Seismic Imaging of Structure and Source, Poster.
Fairbanks, Alaska, July 14–17.
24. Isbiliroglu, Y. and Taborda, R. and Bielak, J. (2013). Coupled Soil-Structure Interaction
Effects of Building Clusters During Earthquakes. Earthquake Spectra (accepted for
publication)
25. Isbiliroglu, Y. D. and Taborda, R. and Bielak, J. (2012). Dynamic Response and GroundMotion Effects of Building Clusters During Large Earthquakes. In Proc. AGU Annu. Meet.
Poster S51A-2404. San Francisco, California, December 3–7.
26. Isbiliroglu, Y. D. and Taborda, R. and Bielak, J. (2012). Dynamic Response and GroundMotion Effects of Building Clusters During Large Magnitude Earthquakes. In Proc. SSA
Annu. Meet. Poster. San Diego, California, April 17–19.
27. Jordan, T. H. (2013) Progress Of The Southern California Earthquake Center Technical
Activity Group On Ground Motion Simulation Validation, Seismological Society of
America Meeting, April 17-19 2013, Salt Lake City, SRL (Vol. 84, No. 2)
28. Jordan, T H., Callaghan, S., Maechling, P J., Juve, G., Deelman, E., Rynge, M., Vahi, K., Silva,
F. (2012) Workflow Management of the SCEC Computational Platforms for PhysicsBased Seismic Hazard Analysis Abstract IN54B-07 presented at 2012 Fall Meeting, AGU,
San Francisco, Calif., 3-7 Dec.
29. Lee, E., Chen, P., Jordan, T. H., Maechling, P. J., Denolle,M., and Beroza, G. C. (2013) Full3d Waveform Tomography For Southern California, Seismological Society of America
Meeting, April 17-19 2013, Salt Lake City, SRL (Vol. 84, No. 2, p. 316)
13
30. Lee, E. & Chen, P. (2013). Automating Seismic Waveform Analysis for Full-3D Waveform
Inversions. Geophys. J. Int. 194 (1): 572-589.
31. Lee, E., Huang, H., Dennis, J. M., Chen, P., & Wang, L. (2013) An optimized parallel LSQR
algorithm for large-scale seismic tomography (submitted to Computers & Geosciences).
32. Lee, E., Chen, P., Jordan, T H., Maechling, P J., Denolle, M., Beroza, G C. (2012) Full-3D
Waveform Tomography for Southern California, Abstract S34B-04 presented at 2012
Fall Meeting, AGU, San Francisco, Calif., 3-7 Dec.
33. Maechling, P. and Gill, D. and Small, P. and Ely, G. and Taborda, R. and Jordan, T. (2013).
SCEC Unified Community Velocity Model: Development Goals and Current Status. In
Proc. CIG/QUEST/IRIS Joint Workshop on Seismic Imaging of Structure and Source.
Fairbanks, Alaska, July 14–17.
34. Maechling, P. (2013) Using Multi-scale Dynamic Rupture Models to Improve Ground
Motion Estimates, ALCF Early Science Program Workshop, 15-16 May 2013
35. Milner, K.R., Thomas H. Jordan (2013), Rupture Synchronicity in Complex Fault Systems,
SCEC Annual Meeting 2013, abstract 262, Sept 9 – 12, 2013, Palm Springs, CA
36. Olsen, K. B., W. Savran, B. H. Jacobsen (2013), Ground motion prediction from lowvelocity sediments including statistical models of inhomogeneities in Southern
California basins, Seismol. Res. Lett., 84:2, 334.
37. Olsen, K.B., B.H. Jacobsen, R. Takedatsu. (2012). Validation of broadband synthetic
seismograms with earthquake engineering-relevant metrics, Seism. Res. Lett.
38. Olsen, K.B., and J. Mayhew (2010). Goodness-of-fit criteria for broadband synthetic
seismograms, with application to the 2008 Mw 5.4 Chino Hills, California, earthquake,
Seism. Res. Lett. 81,5 715-723.
39. Olsen, K.B., and J.E. Mayhew (2010). Goodness-of-fit Criteria for Broadband Synthetic
Seismograms, With Application to the 2008 Mw5.4 Chino Hills, CA, Earthquake, Seism.
Res. Lett. 81 , 715-723.
40. Olsen, K.B., and G. Ely (2009). WebSims: A Web-based System for Storage, Visualization,
and Dissemination of Earthquake Ground Motion Simulations, Seismol. Res. Lett. 80,
1002-1007, doi:10.1785/gssrl.80.6.1002
41. Restrepo, D. (2013). Effects of Topography on 3D Seismic Ground Motion Simulation
with an Application to the Valley of Aburra in Antioquia, Colombia. Ph.D. Thesis,
Carnegie Mellon University, October, Pittsburgh, PA.
42. Restrepo, D. and Taborda, R. and Bielak, J. (2012). Simulation of the 1994 Northridge
earthquake including nonlinear soil behavior. In Proc. SCEC Annu. Meet. Poster GMP015. Palm Springs, California, September 9–12.
43. Roten, D., Olsen, K.B., Day, S.M., Dalguer, L.A. and Fäh, D. (2013). Large-scale 3-D
Simulations of Spontaneous Rupture and Wave Propagation in Complex, Nonlinear
Media, Annual Meeting of the Seismological Society of America 17-19 April, 2013, Salt
Lake City, UT
44. Roten, D., K. B. Olsen, J. C. Pechmann, V. M. Cruz-Atienza, and H. Magistrale (2011). 3D
Simulations of M 7 Earthquakes on the Wasatch Fault, Utah, Part I: Long- Period Ground
Motion, Bull. Seis. Soc. Am. 101, 2045-2063
45. Roten, D., K. B. Olsen, J. C. Pechmann (2012). 3D Simulations of M 7 Earthquakes on the
Wasatch Fault, Utah, Part II: Broadband (0-10Hz) Ground Motions and Nonlinear Soil
Behavior (2012). Bull. Seis. Soc. Am. 102, 2008-2030.
46. Roten, D., and K.B. Olsen (2010). Simulation of Long-Period Ground Motion in the
Imperial Valley Area during the Mw 7.2 El Mayor-Cucapah Earthquake, abstract S51A1920 poster presented at the 2010 Fall Meeting, AGU, CA.
47. Shi, Z., and S. M. Day (2013), Rupture dynamics and ground motion from 3-D roughfault simulations, Journal of Geophysical research, 118, 1–20, doi:10.1002/jgrb.50094.
14
48. Shi, Z., S.M. Day, and G. Ely (2012) Dynamic rupture along the San Gorgonio Pass section
of the San Andreas Fault (2012), Seism. Res. Lett. 83:2, 423.
49. Taborda, R. and Bielak, J. (2013a). Ground-Motion Simulation and Validation of the 2008
Chino Hills, California, Earthquake. Bull. Seismol. Soc. Ame. 103(1):131–156.
50. Taborda, R. and Bielak, J. (2013b). Comparative validation of a set of physics-based
simulations of the 2008 Chino Hills earthquake using different velocity models. In Proc.
CIG/QUEST/IRIS Joint Workshop on Seismic Imaging of Structure and Source, Poster.
Fairbanks, Alaska, July 14–17.
51. Taborda, R. and Bielak, J. (2013). Comparative Validation of a Set of High-Frequency
Physics-Based Simulations Using Two Different Velocity Models. In Abstr. SSA Annu.
Meet. Salt Lake City, Utah, April 17–19.
52. Taborda, R. and Bielak, J. (2013d). Ground-Motion Simulation and Validation of the
2008 Chino Hills, California, earthquake using different velocity models. Bull. Seismol.
Soc. Am., Submitted for publication.
53. Taborda, R. and Bielak, J. (2012). Validation of a 4-Hz physics-based simulation of the
2008 Chino Hills earthquake. In Proc. SSA Annu. Meet. San Diego, California, April 17–
19.
54. Trugman, D. T. and E. M. Dunham (2013), A pseudo-dynamic rupture model generator
for earthquakes on geometrically complex faults, submitted to Bulletin of the
Seismological Society of America on 24 May 2013.
55. Unat, D., Zhou, J., Cui, Y., Cai, X. and Baden, S. (2012) Accelerating an Earthquake
Simulation with a C-to-CUDA Translator, Journal of Computing in Science and
Engineering, Vol. 14, No. 3, 48-58, May/June, CiSESI-2011-09-0094, May, 2012.
56. Wang, F., Jordan, T.H. (2013) Comparison Of Physics-Based Models And Ground Motion
Prediction Equations In Seismic Hazard Analysis For Southern California, Seismological
Society of America Meeting, April 17-19 2013, Salt Lake City, SRL (Vol. 84, No. 2)
57. Wang, F., Jordan, T H. (2012) Using Averaging-Based Factorization to Compare Seismic
Hazard Models Derived from 3D Earthquake Simulations with NGA Ground Motion
Prediction Equations Abstract S51A-2405 Poster presented at 2012 Fall Meeting, AGU,
San Francisco, Calif., 3-7 Dec.
58. Wang, F., and Jordan, T. H. (2013), Comparison of probabilistic seismic hazard models
using averaging-based factorization, Bull. Seismol. Soc. Am., 84 pp., submitted
09/27/13.
59. Wang, F., T. H. Jordan, S. Callaghan, R. Graves, K. Olsen, and P. Maechling, Using
averaging-based factorization to assess Cybershake hazard models, submitted to
American Geophysical Union Annual Meeting, December, 2013.
60. Withers, K., K. B. Olsen, S. Shi, S. M. Day, and R. Takedatsu (2013). Deterministic highfrequency ground motions from simulations of dynamic rupture along rough faults,
Seismol. Res. Lett., 84:2, 335.
61. Withers, K, and Kim B. Olsen (2012). Correlation of peak dynamic and static coulomb
failure stress with seismicity rate change after the M7.2 El Mayor-Cucapah earthquake,
Annual AGU Mts, San Francisco, Dec 2012, poster S43E-2517.
62. Zhou, J., Y. Cui, E. Poyraz, D. Choi, and C. Guest, (2013) Multi-GPU implementation of a
3D finite difference time domain earthquake code on heterogeneous supercomputers,"
Proceedings of International Conference on Computational Science, Vol. 18, 1255-1264,
Elesvier, ICC. 2013, Barcelona, June 5-7, 2013.
63. Zhou, J., Choi, DJ, Cui, Y. (2012) GPU acceleration of a 3D finite difference earthquake
code on XSEDE Keeneland, XSEDE’12, Chicago, July 16-20, 2012.
64. Zhou, J., Didem, U., Choi, D., Guest, C. & Y. Cui (2012) Hands-on Performance Tuning of
3D Finite Difference Earthquake Simulation on GPU Fermi Chipset, Proceedings of
15
International Conference on Computational Science, Vol. 9, 976-985, Elesvier, ICCS
2012, Omaha, Nebraska, June, 2012.
Software and Data Products:
1. Simulation results, as seismograms, ground motion amplitudes, rupture definitions, and
maps from the CyberShake 13.4 Hazard Model generated by the PetaSHA3 are posted
online and available for researcher. These CyberShake data products were used in both
Averaging-based Factorization [Wang et al. 2013], and ShakeAlert Ground Motion
Research [Boese, M., Graves, R W., Callaghan, S., Maechling, P J. (2012)].
2. Open-source software distribution of the Unified California Velocity Model (UCVM)
software released in September 2013, available on SCEC wiki.
3. Hercules is made available upon request via the GitHub code-hosting platform. Versions
of Hercules have been shared with researchers at USC and Argonne National Lab.
4. Data and results from the 2008 Chino Hills earthquake and simulation are made
available upon request via the CMU Quake Group Web site. The full dataset has been
already shared with researchers from USC and UC Irvine.
5. Electronic supplement to Roten et al. (2011) (Supplementary figures of spectral
acceleration and animation of wave propagation) can be found at
http://bssa.geoscienceworld.org/content/101/5/2045/suppl/DC1
6. Electronic supplement to Roten et al. (2012) (Table of coefficients and amplitudedependent correction functions for nonlinear soil effects, and figures showing maps of
SAs at various frequencies, PGA and PGV, with and without correction for nonlinear soil
effects, results of 1D nonlinear simulations, and comparison to ground motion
prediction equations.) can be found at
http://www.seismosoc.org/publications/BSSA_html/bssa_102-5/2011286esupp/index.html
7. M8 and SORD simulation results are posted on the interactive data website called
websims: http://scec.usc.edu/websims/
8. Prototype CyberShake Data Access system to distribute CyberShake results from Study
13.4 http://scec.usc.edu/scecpedia/CyberShake_Data_Request . The prototype is
available and provides access to CyberShake Study 13.4 study results on a SCEC web
site but requires a login: https://scec.usc.edu/it/CyberShake_Data_Request
9. PetaSHA3 ground motion simulation animations have been posted on YouTube as SCEC
Outreach. Examples include M8 [https://www.youtube.com/watch?v=V2Ow0Yuv5co],
UseIT Video [https://www.youtube.com/watch?v=WvKgSCpLtgk] and Yellowstone
simulations [https://www.youtube.com/watch?v=4ij3XDLXHvg].
7. Participants & Other Collaborating Organizations - Who has been involved?
The following individuals have worked on the project. In some cases, such as the USGS
researchers, no funding has been provided to these researchers although they have played
an active role in this research.
1.
2.
3.
4.
5.
6.
7.
Gregory Beroza - beroza@stanford.edu - Professor Geophysics
Jacobo Bielak - jbielak@cmu.edu - Professor Civil Engineering, Hispanic
Scott Callaghan - scottcal@usc.edu - Research Staff Computer Science
Feng Chen - chen@uwyo.edu - PhD Student Geophysics
Po Chen - pchen@uwyo.edu - Professor Geophysics
Dong Ju Choi - dchoi@sdsc.edu - Post-doc geoscience researcher
Amit Chourasia - amit@sdsc.edu - Senior Visualization Scientist SDSC Staff
16
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
Yifeng Cui - yfcui@sdsc.edu - Research Staff Computer Science
Steven Day - sday@mail.sdsu.edu - Professor Geophysics
Ewa Deelman - deelman@isi.edu - Prof. Computer Science (female, non-hispanic)
Jessica Donovan - jrdonova@usc.edu - PhD Student Geophysics
Geoffrey Ely - gely@anl.gov - Post-doctoral Scholar
David Gill - davidgil@usc.edu - Research Staff Computer Science
Robert Graves - rwgraves@usgs.gov - Seismologist
Tran Huynh - tran.huynh@usc.edu - Research Staff Geophysics
Yigit Isbiliroglu - isbiliroglu@cmu.edu - Graduate student, Caucasian
Thomas Jordan - tjordan@usc.edu - Professor Geophysics
Gideon Juve - juve@isi.edu - Research Staff Computer Science
Haydar Karaoglu - hkaraogl@andrew.cmu.edu - PhD Student Civil Engineering, Caucasian
En-Jui Lee - elee8@uwyo.edu - PhD Student Geophysics
Kwangyoon Lee, UCSD GSR, kwl002@cs.ucsd.edu
Philip Maechling - maechlin@usc.edu - Research Staff Computer Science
Nolan Mattox - nmattox@usc.edu - Undergraduate Student Computer Science
John McRaney - mcraney@usc.edu - Research Staff Geophysics
Emmett McQuinn – emcquinn@ucsd.edu - M.S. Grad Student Computer Science
Kevin Milner - kmilner@usc.edu - M.S. Graduate Student Geophysics
Jean-Bernard Minster - jbminster@ucsd.edu - Prof. at Scripps (SIO)
Kim Olsen - kbolsen@mail.sdsu.edu - Professor Geophysics
Efecan Poyraz - efecanpoyraz@gmail.com - PhD Student Computer Science
Dorian Restrepo - drestrep@andrew.cmu.edu - PhD Student Geophysics, Hispanic
Zheqiang Shi - zshi@projects.sdsu.edu - Post-doc, geoscience, male, Asian
Fabio Silva - fsilva@usc.edu - Research Staff Computer Science
Patrick Small - patrices@usc.edu - M.S. Graduate Student Computer Science
Xin Song - xinsong@usc.edu - PhD Student Geophysics
Ricardo Taborda - rtaborda@cmu.edu - Post-doctoral Scholar Civil Engineering, Hispanic
Rumi Takedatsu - aaaum27@hotmail.com - MS student, asian, female,
Daniel Trugman - dtrugman@stanford.edu - PhD Student Geophysics
Feng Wang - fengw@usc.edu - PhD Student Geophysics
Kyle Withers - quantumkylew@aol.com - PhD Student Geophysics male, not latino, white
Karan Vahi - vahi@isi.edu - Research Staff Computer Science
Jun Zhou - zhoujun84@gmail.com - UCSD graduate student, Chinese citizen,
Other collaborators or contacts been involved?
1. Norm Abrahamson – PG&E Civil Engineer
2. Christine Goulet – PEER Civil Engineer
3. Monica Kohler – Caltech Professor Geophysics
4. Maren Bose – Caltech Professor Geophysics
5. Dhabaleswar Panda - OSU CS Professor
6. John Levesque - Cray Inc.
7. Scott Klasky - ORNL
8. Jeffrey Chen - Colorado School of Mines, CS Professor
9. Dr. Daniel Roten - ETHZ, Zurich, Switzerland
10. Dr. Jürgen P. Schulze - California Institute For Telecommunication and Information
Technology Research Scientist
Impact - What is the impact of the project? How has it contributed?
17
The SCEC computational software is in active use by a scientific and private
collaboration. The UCERF3 earthquake rupture forecast [Field, E., Dawson, T.E., et al.
(2012)] contains significant contributions from PetaSHA3 researcher is expect to become
the foundation for the next USGS California seismic hazard map. Commercial electrical
companies, studying seismic hazards of western US nuclear power plants, and California
dams, are using software developed by the PetaSHA3 researchers to study ground motions
using simulation-based ground motion models. On April 3, 2013, members of the California
Ground Motion Simulation Utilization committee met at SCEC. Committee members,
including C.B. Crouse, met with PetaSHA3 researcher to begin their evaluation of SCEC’s
CyberShake PSHA for both urban seismic hazard maps for Los Angeles, and for use in
Building Code development in upcoming review cycles.
The Chino Hills simulations, as well as the rough-fault dynamic rupture and wave
propagation simulations, have demonstrated that accurate deterministic ground motion
estimation for frequencies up to 5-10Hz may be feasible. This result has important
implications for current procedures for such ground motion estimation, which primarily
consists of hybrid low-frequency deterministic-stochastic and high frequency stochastic
methods (e.g., Chavez et al., 2011; Roten et al., 2012). These hybrid methods lack physical
basis, and the PetaSHA3 results show promise for the fully deterministic methods to be able
to replace the hybrid methods in the future.
SCEC’s PetaSHA3 work optimizing our AWP-ODC wave propagation software now
permits SCEC software to run on the Top 500 HPC system. Development of this highly
capable software is an important step towards our goal of real science runs at sustained
petaflops performance. A PetaSHA3 group including SCEC, XSEDE, and ISI researchers
developed a new method for running many short serial jobs using workflows on Kraken has
significant potential for broad use within computational sciences [Callaghan, S., Maechling,
P., Juve, G., Vahi, K., Graves, R. W., Olsen, K. B., Gill, D., Milner, K., Yu, J. and Jordan, T. H.
(2013)]. This new capability will help SCEC and other researchers use workflow technology
on Kraken and other newly developed supercomputers such as Blue Waters.
What is the impact on the development of the principal discipline(s) of the project?
The PetaSHA3 project developed highly scalable, deterministic (Hercules, AWP-ODC,
CyberShake) ground motion simulations. The project is leading the scientific transition from
the current, ground motion prediction equation-based, seismic hazard estimates, to physicsbased, but more computational expensive, deterministic simulation methods.
Our collaborative work with engineering users of ground motion simulation results
distinguishes our PetaSHA3 project from single domain research. SCEC’s PetaSHA3
researcher is bridging the gap between geoscientists that perform ground motion
simulations, and engineering users of ground motion simulations. This collaboration has led
to quantitative evaluation procedures for evaluating new ground motion simulation
methods, and our SEISM software development activities have captured these agreements
and procedures in open-source scientific software.
Through careful analysis, we expect to identify the tradeoffs between the older
(empirical GMPE-based) and new (deterministic simulation-based) PSHA methods,
identifying where deterministic methods provide improved results. This work can inform
the whole seismic hazard analysis field which applications can benefit from the additional
time, and computational expense, of high-frequency deterministic ground motion
simulations.
What is the impact on other disciplines?
18
Civil engineers are interested in using ground motion simulation results to augment
existing observational data for strong ground motions. Despite the rapid increase in the
number of ground motion sensors, close-in (<10km) recordings for large magnitude
earthquakes (M8+) are quite rare. Engineers would like to use simulations to supplement
the observational record. Only through a careful evaluation process of evaluation and
validation, as we performed on the PetaSHA3 project, will simulated ground motions be
accepted for use by engineers.
PetaSHA3 researchers participate on national and international HPC advisory
groups including NICS User Advisory Board, TeraGrid Science Advisory Board, XSEDE
Advisory Board, NEES Cyberinfrastructure Advisory Committee, and the Global Earthquake
Model Scientific Board, and we participate in national cyberinfrastructure development and
planning workshops including the NSF EarthCube activity.
Finally, from the point of view of outreach, it has been our experience that a good
visualization “speaks” to the minds of the viewers, even persons with no technical
background. This has been the case from the very beginning of the SCEC Terashake project
and has continued since. The PetaSHA3 work on representing vector data makes it possible
to use the technique to represent higher order tensor data (e.g. dynamic strain fields), with
seismological, or other, vector-based data sets. Our GlyphSea visualization technique has
made an impact by demonstrating that 4D visualizations of vector and tensor fields is a
superb way for domain scientists to discover features in the enormous volume of data
produced by 3D dynamic simulations. We anticipate that visual pattern recognition will
become an ever more useful analysis tools as the methods become more readily accessible.
Preliminary applications to early universe dynamic cosmological models look promising.
What is the impact on the development of human resources?
The computer-oriented research activities on the PetaSHA3 project are excellent
preparation for many types of work. Graduate students and research staff that have made
significant contributions to SCEC computational research have gone onto computational
science careers including positions with Amazon.com, Microsoft, Intel, Argonne National
Laboratory, and AIR Worldwide.
SCEC Intern programs continue to cross-train students in geoscience and computer
science. SCEC’s current software staff includes 2 developers who first participated in SCEC
as undergraduate interns.
What is the impact on physical resources that form infrastructure?
SCEC computational tools including AWP-ODC, AWP-ODC-GPU, and Hercules, and
CyberShake platform are being considered for use in development of the next generation
building codes. A ground motion utilization committee, chaired by California Building Code
development managers, met at SCEC in March 2013 to review the applicability of SCEC’s
CyberShake platform for use in the next 5-year building code update cycle.
What is the impact on institutional resources that form infrastructure?
None expected.
What is the impact on technology transfer?
As well validated open-source scientific software, the SCEC community codes
including AWP-ODC, AWP-ODC-GPU, Hercules, OpenSHA, and the UCVM Platform have
attracted interest from private researchers and commercial companies for use in seismic
hazard analysis research. This helps to transfer HPC computing into seismic hazard
research.
19
Seismology data centers have years of experience delivering observational data, but
standard practices for delivering simulation results are still being developed. PetaSHA3 data
management tools including our prototype CyberShake hazard model data access site, and
the user-friendly Websims site are likely to change the accessibility of earthquake
simulations and other products generated by earthquake simulations, making them useful
to researchers and the public.
What is the impact on society beyond science and technology?
SCEC’s PetaSHA3 project has potential broad impact through improved public
seismic safety. This group is advance the practice of seismic hazard analysis to integrate,
evaluate, and adopt a much more computational oriented approach than currently used. By
providing engineers with more accurate and more complete information about earthquake
generated strong ground motions, the PetaSHA3 research and software tools have the
potential for large societal impact. In the United States, the U.S.G.S. is responsible for seismic
hazard evaluation and information and our work may improve public USGS seismic hazard
information that has societal impact beyond science and technology.
In the United States, the USGS provides seismic hazard information, including
seismic hazard forecasts, to regulatory agencies and the public. SCEC PetaSHA3 research
includes significant contributions from USGS personnel and our close connection to USGS
seismic hazards programs provides an opportunity for our results to impact national
seismic hazard estimates. We believe that the SCEC PetaSHA3 project has shown how
physics-based computational models, observation-based 3D earth structure models, and
high performance computing can improve seismic hazard forecasts and that software and
computational improvements made by our PetaSHA3 research group is contributing to the
development of the USGS official seismic hazard information in the United States.
20