SI2-SSI: A Sustainable Community Software Framework for Petascale Earthquake
Modeling (SEISM) (OCI-1148493)
Principle Investigator: Thomas H. Jordan
Annual Report Performance Period 1 August 2013 – 31 July 2014
1. What are the major goals of the project?
The principal scientific goal of our SEISM project is to develop and use a new generation of
time-dependent earthquake rupture models for earthquake forecasting that will produce
more accurate hazard maps and to simulate the seismic performance of structural and
geotechnical systems, facilitating a “rupture-to-rafters” modeling approach to earthquake
risk management.
The principal software goal of our SEISM project is to integrate high-level and middle-level
scientific software elements, developed by SCEC, into a sustainable software ecosystem for
physics-based seismic hazard analysis in the United States and elsewhere. Our SEISM
software will support the use of petascale computers by earthquake scientists to generate
and manage the large suites of earthquake simulations needed for physics-based PSHA, as
well as to advance basic research on rupture dynamics, anelastic wave scattering, and Earth
structure. The SEISM software framework we are developing includes high-level SSEs for
developing and managing unified community velocity models, codes for dynamic and
pseudo-dynamic rupture generation, deterministic and stochastic earthquake simulation
engines, and the applications necessary to employ forward simulations in two types of
inverse problems: seismic source imaging and full-3D tomography.
SEISM’s principal organizational goal is to incorporate computational science and
engineering tools and practices into use by seismic hazard analysts, and to engage, train and
expand a diverse STEM workforce of geoscientists, computer scientists, and earthquake
engineers. During its first two years, our SEISM project has had excellent collaborative
activities with public and private research organizations developing and using scientific
codes by an interdisciplinary team that includes geoscientists, computer scientists, and
earthquake engineers.
2. What was accomplished under these goals (you must provide information for at
least one of the 4 categories below)?
For this reporting period, we will report on Significant Results:
During Year 2 of our SEISM award, we have developed improved versions of the Broadband
and Unified Community Velocity Model (UCVM) platforms. We have created public software
releases of these two community software platforms. We have integrated these tools into
use on the CyberShake, High-F, and F3DT research activities, incorporated new physics into
the AWP-ODC and Hercules codes, and performed user-driven validation studies.
Broadband platform. The SCEC Broadband Platform (BBP) has been developed and released
as open-source scientific software that can generate broadband (0-100Hz) ground motions
for earthquakes, integrating complex scientific modules that integrate rupture generation,
deterministic and stochastic seismogram synthesis, non-linear site effects, and visualization
into a software system that supports on-demand computation of seismograms. The BBP has
been developed by a collaboration that involves geoscientists, engineers, and computer
scientists. The BBP operates in two modes: validation simulations and scenario simulations.
1
In validation mode, the BBP runs earthquake rupture and wave propagation modeling
software to calculate seismograms for historical earthquakes for which observed strong
ground motion data are available, computing a number of goodness of fit (GOF)
measurements that quantify how well the synthetics match the observations. It has been
used to evaluate and validate a variety of numerical ground motion modeling techniques
that are now built into the platform. Validation data are currently available for 12 historical
events from the eastern and western United States, eastern North America, and Japan. In
scenario mode, the user specifies a hypothetical earthquake description, a list of station
names and locations, and a 1D velocity model for the region of interest, and the BBP
software then calculates ground motions for the specified stations. Public software releases
of the BBP were made in Sept 2013 and April 2014. The BBP has attracted a substantial user
community, including PEER, PG&E, and the California Department of Water & Power, which
recently let a contract to URS Corporation to manage DWP’s use of the platform in its damsafety studies (Fig. 1).
UCVM platform. During SEISM year 2, we have continued to develop the Unified Community
Velocity Model (UCVM) platform as a common framework for comparing and synthesizing
Earth models and delivering model products to the geoscientists (e.g., the EarthScope
community). UCVM is an integrated software framework designed to provide a standard
interface to multiple, alternative, 3D velocity models. It includes an easy-to-use CVM query
interface; the ability to integrate regional tomographic, basin structure, and geotechnical
models; and automated CVM evaluation. The UCVM software enables users to quickly build
meshes for earthquake simulations through a standardized, high-speed query interface. We
have registered seven different CVMs into the UCVM. UCVM’s interface has been used to
perform high-speed, parallel queries of CVM-S4 and CVM-H11.9 to build simulation meshes
with more than 14 billion mesh points. Its file utilities have been used to export meshes in
both eTree and NetCDF formats. The UCVM open-source software has been ported to many
HPC systems, including USC High-Performance Computing Center, TACC Stampede, NCSA
Blue Waters, and NCCS Titan. Public software releases of the BBP were made in Sept 2013
and April 2014. Each release has included an installation guide, and a detailed user guide.
CyberShake platform. During year 2 of the SEISM project, we have continued development of
the CyberShake Platform. CyberShake is capable of generating the very large suites of
simulations (>108 synthetic seismograms) needed for physics-based probabilistic seismic
hazard analysis (PSHA). By implementing seismic reciprocity, CyberShake reduces the
computational time for PSHA ensembles by more than a factor of 1000. A CyberShake PSHA
Study 13.4, begun in April, 2013 during SEISM project year 1, used the UCERF2 earthquake
rupture forecast and calculated hazard curves for 286 sites in southern California at
frequencies up to 0.5 Hz. This study, performed on NCSA Blue Waters and TACC Stampede
using both CPUs and GPUs, was four times larger than the 2011 CyberShake study, but it
was completed in approximately the same wall-clock time (~61 days).
During SEISM project year 2, we developed a new GPU-based strain Green tensor (SGT)
solver that provides speedup by a factor of 110 in SGT calculations compared to the
previous CPU-based code. Along with porting our workflows to execute entirely on NCSA
Blue Waters, this enhanced code decreased our CyberShake wall clock time by a factor of
four (~14 days). Los Angeles region probabilistic seismic hazard models, produced by the
CyberShake 14.2 study, are shown in Fig. 2. These CyberShake calculations were the first
large-scale scientific workflows run on Blue Waters and our role integration grid-based
workflows onto Blue Waters highlights SCEC’s role in making NSF research computers more
useful to scientific research groups. During SEISM Project year 2, CyberShake study results
were used to calculate ground motions more rapidly and accurately for use in the California
Integrated Seismic Network (CISN) ShakeAlert Earthquake Early Warning system as well as
for calculations in planned updates to model building codes.
2
High-F platform. SCEC SEISM research groups have developed the High Frequency (High-F)
platform to conduct high-frequency deterministic simulations. The High-F project integrates
SEISM-support HPC wave-propagation software, including AWP-ODC, Hercules, and SORD
to into advanced earthquake physics models that include fractal fault complexity, rupture
complexity, and near-surface stochastic variability. Using High-F, we have investigated the
feasibility of pushing deterministic simulations to seismic frequencies as high as 10 Hz, and
we have evaluated the performance of deterministic simulations relative to stochastic
simulations in the 1-10 Hz band. All High-F codes have been ported to Cray XE6/XK7 and
IBM BG/Q platforms. AWP-ODC was tuned on Blue Waters XE6 using Cray’s topologymapping tuning tool Topaware that boosted speed by an additional 35%. Both Hercules and
AWP-ODC solvers have been accelerated on NVIDIA-based GPU architectures including
XSEDE Keeneland, NCSA Blue Waters and OLCF Titan. AWP-ODC achieved 100% parallel
efficiency up to 8,192 Titan nodes and recorded 2.3 PetaFlop/s performance in single
precision on 16,384 Titan GPUs. The AWP-ODC and Hercules performance on XK7 exceeds
XE6 by factors of 3.7 and 2.5, respectively. Other improvements to Hercules include: more
efficient meshing and solving algorithms, which resulted in 1.3x speedup; implementation
of tri-quadratic finite elements, surface topography, an improved attenuation model, the
domain reduction method, and elasto-acoustic and elasto-plastic media; and incorporation
of building models.
SEISM-IO Library. During SEISM year 2, we have continued development of the SCEC SEISMIO library to manage the I/O requirements of seismic applications on petascale machines.
Basic elements of the I/O library are implemented in Fortran, on top of which we built C
interfaces. For straightforward usage of the library, a generalized I/O interface was
designed with high performance, flexibility, runtime dynamic adaptation to new compute
environment, and light-weight user requirements. SEISM-IO is built on top of the available
high-performance I/O libraries: MPI-IO, ADIOS, HDF5 and PnetCDF. The strategies in the
I/O library implementation include designing a data space with variable and group objects
and buffering data so as to reduce the frequency of file system access. The users only need
to communicate with the API through the abstract data space. The primary features
implemented include collective I/O, synchronous or asynchronous I/O calls, and I/O
staging, for inputs and outputs. Both Fortran and C interfaces using MPI-IO and other high
performance libraries have been completed and tested using AWP against validated
modeling results and observed data on NSF XSEDE and DOE INCITE systems. SEISM-IO
demonstrated a comparable I/O rate to manually optimized I/O performance of AWP-ODC
up to 32,768 cores on Blue Waters XE6 nodes.
Simulation Physics. During SEISM project year 2, we have made important advancements
improving the underlying physics of our computational platforms.
Fault roughness. High-frequency simulations require non-planar fault models to capture the
small-scale processes of wave excitation, such as large stress perturbations that accelerate
and decelerate the rupture, releasing bursts of high-frequency seismic waves. We have
modeled the fractal roughness observed on fault surfaces based on the SORD code, along
with off-fault plastic yielding, and rate-and-state fault friction. The SORD solutions have
been linked to the AWP-ODC code to enable propagation of simulated ground motion to
large distances. We used this hybrid approach to propagate 0-10 Hz wavefields up to 100
km from the source for M 7+ strike-slip earthquakes with median distance and period
dependence, absolute level, and intra-event standard deviation remarkably similar to
standard ground motion prediction equations (GMPE) estimates throughout the period
range 0.1-3.0 sec. The largest 0-10 Hz SORD-AWP Titan run comprised 443 billion elements,
with 6.8 TB velocity model and dynamic source inputs, and used 16,640 OLCF Titan nodes.
Near-fault and near-surface plasticity. Because dynamic stresses during rupture can exceed
the yield stress of near-fault rocks and soft-soil deposits, we must model the medium
surrounding faults as a nonlinear (plastic) solid and the soft deposits as nonlinear soil. We
3
have therefore included nonlinear effects into the AWP-ODC and Hercules platforms, and
tested the effects on the 2008 M 7.8 ShakeOut earthquake scenario and other idealized
scenario earthquake simulations. In the case of the ShakeOut, linear ground motion
predictions showed strong long-period ground motions in the Los Angeles basin owing to
the channeling of surface waves through a series of interconnected sedimentary basins. By
simulating the ShakeOut earthquake scenario for a medium governed by Drucker-Prager
plasticity, we have shown that nonlinear material behavior could reduce these predictions
by as much as 70% compared to the linear viscoelastic solutions. These reductions are
primarily due to yielding near the fault. Since fault-zone plasticity remains important even
for conservative values of cohesion, we can infer that current simulations that assume a
linear response of rocks are over-predicting ground motions during future large
earthquakes on the southern San Andreas fault.
Anelasticity. In low-frequency simulations (< 0.5-1 Hz), attenuation structure can be
represented in terms of a spatially variable but frequency-independent attenuation factor
for pure shear. However, as we move simulations to higher frequencies, anelastic
attenuation must be modeled as a frequency-dependent power law. To this end, frequency
dependence has been incorporated into the AWP-ODC code via a power-law formulation
adopting a coarse-grained approach representing the attenuation spectrum as a
superposition of absorption bands.
Topography and near-surface heterogeneities. Ground shaking is significantly modified by
site effects, which are dominated by the surface topography and the properties of the upper
(“geotechnical”) layers. Seismic velocities drop to very low values within these near-surface
layers (< 500 m/s), and the resulting waveguide controls the amplitudes of the seismic
shaking at surface sites. Topographic features are imperceptible to long period waves, and
the near-surface waveguide can be represented by spatially smooth refinements of the
CVMs. The effect of topography and material heterogeneities becomes increasingly
significant at high frequencies. Sonic logs from deep sedimentary basins indicate that the
amplitude of the heterogeneity associated with sedimentary layering is strong (± 6%) at
correlation lengths of ~100 m or less, which is well below deterministic scales. In SEISM
year 2, we developed stochastic models to represent material heterogeneities and
incorporated them into high-frequency simulations using AWP-ODC. We also implemented
a virtual topography modeling approach in Hercules that accounts for free-surface effects
while preserving the octree structure of the finite-element mesh used in the code.
Validation Studies. An important SEISM year 2 project effort has been the validation of 3D
deterministic simulations using historical earthquakes at seismic frequencies > 1 Hz.
Broadband Platform. The BBP operates in two modes: validation of synthetic seismograms
against observed seismograms from historical earthquakes and the simulation of scenario
earthquakes. The Southwestern United States Ground Motion Characterization Project
(SWUS) has used the BBP in validation exercises to evaluate numerical simulation methods.
The SWUS project, which was sponsored by utility operators in California and Arizona,
responded to a request by the US Nuclear Regulatory Commission to update PSHA for
nuclear power plants after the Fukushima Dai-ichi accident. BBP methods were validated
against data and with respect to empirical ground motion prediction equations, and the
results were judged by an independent panel of experts who used the BBP validation tools
to rate the methods and issue directives for ongoing developments. This is an excellent
example of the Broader Impacts of the SEISM project during project year 2.
Ground Motion Simulation Validation Technical Activity Group (GMSV-TAG). Ground motion
simulation validation (GMSV) efforts were advanced through a collaboration between
ground motion modelers and engineering users to test new methodologies for validating
simulations for engineering applications. A Technical Activity Group (TAG) is collaborating
with the SEISM project to develop validation gauntlets comprising elastic and inelastic
4
single- and multi-degree-of-freedom oscillators, geotechnical systems including site
response effects and analysis, nonlinear soil response, and soil failures, including landslides,
and inelastic building models. These validation gauntlets are becoming the standard for
simulation validation.
Evaluation of CVMs. Our validation activities have also focused on the evaluation of the 3D
community velocity models (CVMs) that underlie the earthquake simulations. Examples of
our validation work in SEISM Year 2 are shown in our CVM validation studies using the
M5.4 2008 Chino Hills earthquake (Fig. 3). During project year 2, we performed Hercules
simulations on Kraken and Blue Waters to evaluate simulations up to 4 Hz, and compare
these ground motion simulation results against observations.
SEISM Integration. During SEISM project year 2, we have performed a great deal of
integration among SEISM SSEs. Two examples of this integration are the connection
between 3D tomographic inversion models, the UCVM platform, and the High-F simulation
platform SSEs for ground motion simulations; and the connection between ground-motion
simulations and the response of building infrastructure systems. Recently, Lee et al.
(2014b) produced results for the 2014 La Habra earthquake (M 5.12) earthquake that use
the velocity model CVM-S4.26. This model merges results from a tomographic inversion
done by using the UCVM platform, which was improved to include algorithms that
effectively transitions between coarse grids and arbitrary resolution models. At the same
time, we are regularly using UCVM to transform CVM data into grids and meshes that can be
read by our simulation codes (AWP-ODC and Hercules) as direct input for simulations. We
are currently working on a more detailed simulation of the La Habra event which will also
integrate results from effectively combining tomographic datasets, using UCVM to merge
CVMs and produce simulation input datasets, run simulations with both AWP-ODC and
Hercules GPU versions that use a source model generated with the BBP simulations and
then comparing results using BBP validation tools. We have also used Hercules to simulate
ground motions in combination with the response of buildings in dense urban settings. In
particular, we present results for a suite of simulations of the 1994 Northridge earthquake
(M 6.7) up to seismic frequencies of 5 Hz including the scattering effects due to the presence
of large inventories of building foundations. This was an important advance because our
simulations are ultimately intended for assessing urban seismic hazards and the risks
associated with critical infrastructure systems such as dams and nuclear power plants, as
well as transportation and infrastructure networks.
3. What opportunities for training and professional development has the project
provided?
During SEISM project year 2, our project has provided training and professional
development for undergraduate, graduate, and staff. Our SEISM activities, are large multidisciplinary collaborations, that require close collaboration between geoscientists, civil
engineers, and computer scientists provides particularly valuable training opportunities,
giving project members multiple skills and career flexibility and options.
Our project participants have a broad spectrum of experience ranging from undergraduates,
to master students, to PhD students, to researchers with many years of experience. Students
participating on our SEISM projects are receiving multi-disciplinary training in geosciences,
computer science, and computational science. SCEC projects alternate between developing
new computational capabilities with better scientific methods, and then using those new
computational capabilities with better computer systems providing training in multiple
disciplines as we produce new results.
5
During project year 2, SEISM project members participated in training workshops at both
computer science and geoscience conferences including SC13, Fall AGU 2013, SSA 2014,
International HPC Summer School 2014, and XSEDE 2014 conferences.
4. How have the results been disseminated to communities of interest?
During SEISM year 2, SCEC research Scott Callaghan lead workshops at the International
HPC summer school in Budapest. SCEC SEISM researchers have produced numerous poster
and presentations about our research at the SCEC annual meeting, Fall AGU 2013, SC13, SSA
2014, and other conferences, including the following:
1. 8th International Workshop on Statistical Seismology (Statsei8), 11 Aug - 16 Aug 2013 Beijing, China
2. 2013 SCEC Broadband Platform and Ground Motion Simulations: Recent Progress on
Validation of Methods and Planning the Next Steps, 8 Sept 2013 - Palm Springs, CA
3. 2013 SCEC Ground Motion Simulation Validation TAG Workshop, 8 Sept 2013 - Palm
Spring, CA
4. SCEC Annual Meeting, Sept 9-11, 2013 Palm Springs, CA
5. SCEC BBP Simulation Modelers at SWUS GMC Workshop #2, Oct 22-24, 2013, Palo Alto,
CA
6. NAS Board on Earth Sciences and Resources, Nov 15, Wash. DC.
7. SC13, Nov 17-22, 2013, Denver CO
8. Blue Waters User Workshop December 3-5, 2013 at NCSA
9. Fall AGU, Dec 9-13, 2013 – San Francisco, CA
10. Global Earthquake Model Annual Meeting, Dec 16 2013 – Menlo Park, CA
11. Broadband Platform Meeting, Jan 29-30, 2014 - Berkeley, CA
12. SI2 PI Meeting, Feb 23-26, 2014 Arlington VA
13. SWUS Broadband Platform SSHAC 3 Meeting, March 10, 2014 - Berkeley, CA
14. SSA April 28-May 4, 2014 - Anchorage Alaska
15. Blue Waters Symposium, May 12-14, 2014, NCSA
16. International HPC workshop, June 1- 5, 2014 Budapest Hungary
17. SCEC High-F Meeting, July 7-9, Carnegie Mellon University, Pittsburgh PA
18. National Conference on Earthquake Engineering, July 21-25 2014, Anchorage, Alaska
19. Oak Ridge Leadership Computing Facility Symposium, July 21-23, 2014, Oak Ridge
National Laboratory, TN
SCEC SEISM projects have been featured in numerous press releases, and public
presentations including the following web-based articles:
1. SCEC research discussed in Titan INCITE Highlights:
http://blog.cray.com/?p=6850
2. Interview with Scientists about La Habra M5.1 Earthquake includes SCEC animations:
http://www.cnn.com/2014/03/29/us/california-earthquake/
3. Interview with SCEC Director about SEISM research:
http://losangeles.cbslocal.com/2014/03/30/7-5-quake-on-puente-hills-thrustfault-could-be-disastrous/
6
4. TACC Stampede Year 1 HPCWire Interview Includes SCEC research:
http://www.hpcwire.com/soundbite/taccs-stampede-supercomputer-first-yearreview/
5. USC Dornsife Earthquake System Science Press Release:
http://dornsife.usc.edu/news/stories/1603/strides-in-earthquake-science/
6. Interview with SCEC SEISM researcher Beroza about Ambient Noise Study:
http://m.laweekly.com/los-angeles/blogs/Post?basename=the-big-one-quake-willhit-la-harder-than-we-thought-scientistssay&day=27&id=informer&month=01&year=2014
7. Los Angeles Times article about Beroza Ambient Noise Study:
http://www.latimes.com/science/sciencenow/la-sci-sn-virtual-earthquake-oceanwaves20140124,0,4506840.story?track=rss&utm_source=feedburner&utm_medium=feed&utm_c
ampaign=Feed%3A+latimes%2Fmostviewed+%28L.A.+Times++Most+Viewed+Stories%29#axzz2rO3ClMsh
8. SDSU Article about SEISM researcher Olsen’s Basin Response Research for Vancouver:
http://universe.sdsu.edu/sdsu_newscenter/news.aspx?s=74691
9. USC Article On SCEC Research:
http://news.usc.edu/#!/article/58305/earthquake-science-in-the-era-of-big-data/
10. Cray HPC Coverage of AWP-ODC-GPU:
http://s1018582977.t.en25.com/e/es.aspx?s=1018582977&e=679&elq=316f57a9
63f44872bc9f38127e314424]
11. SCEC Director Media Presentation Twenty Years After Northridge
http://hypocenter.usc.edu/research/CME/SCEC_Media_Briefing_20_Years_Since_No
rthridge_01_09_2014.m4v
12. Article about SCEC Contributions 20 Years after the Northridge Earthquake
http://www.huffingtonpost.com/2014/01/10/northridge-earthquaketechnology_n_4572906.html
13. Cray Science Impact Article about SCEC
http://hypocenter.usc.edu/research/INCITE/XK7-ORNL-Earthquake-0114.pdf
14. SCEC on Titan
https://www.olcf.ornl.gov/2013/12/16/titan-simulates-earthquake-physicsnecessary-for-safer-building-design/
15. SCEC CME IDC Award Announcement
http://www.idc.com/getdoc.jsp?containerId=prUS24451613
16. BBC Earthquake Forecasting Video
http://www.bbc.co.uk/programmes/p01c4qgt
7
5. What do you plan to do during the next reporting period to accomplish the goals?
SEISM project plans for Year 3 will focus on the continued improvements to our SEISM
publicly released software including the Broadband Platform and UCVM. We expect
quarterly software releases for each of these community codes, with the next releases
planned for Sept. 2014. We expect to release of the UCVM Platform with support for
additional, new California velocity models, a USGS Central United States Model, and
improved installation, and documentation. We are pushing the maximum simulated
frequency of our CyberShake calculations, doubling to 1.0Hz, and, with the capabilities of
our GPU codes, pushing towards the 1.5Hz requirements specified by the building code
developers, important potential users of these CyberShake hazard models. We will continue
to extend the Broadband Platform by integrating new ground motion modeling software
and new validation tests. We will continue to extend our CyberShake, AWP and Hercules
software as we develop new scientific capabilities, validation, and use. We will continue to
push our deterministic wave propagation to high frequencies, with multiple groups now
working at 4Hz and above. We also expect to continue development of our GPU versions of
AWP-ODC and Hercules.
6. Products - What has the project produced?
Software products produced by the project include open source software distributions
including the following:
•
Unified Community Velocity Model (UCVM) Community Software is available for
download at http://scec.usc.edu/scecpedia/UCVM
•
SCEC Broadband Platform (BBP) is available for download at
http://scec.usc.edu/scecpedia/Broadband _Platform
•
The AWP-ODC CPU and GPU code is available for download on request.
•
The Hercules software is available for download on request.
7. Participants & Other Collaborating Organizations - Who has been involved?
The following individuals have worked on the project. In some cases, such as the USGS
researchers Graves and Luco, no funding has been provided to these researchers although
they have played an active role in this research.
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
Jack Baker - bakerjw@stanford.edu - Professor Civil Engineering
Gregory Beroza - beroza@stanford.edu - Professor Geophysics
David Gill - davidgil@usc.edu - Research Staff Computer Science
Dorian Restrepo - drestrep@andrew.cmu.edu - PhD Student Geophysics
Daniel Trugman - dtrugman@stanford.edu - PhD Student Geophysics
Eric Dunham - dunham@stanford.edu - Professor Geophysics
Efecan Poyraz - efecanpoyraz@gmail.com - PhD Student Computer Science
En-Jui Lee - elee8@uwyo.edu - PhD Student Geophysics
Feng Chen - chen@uwyo.edu - PhD Student Geophysics
Feng Wang - fengw@usc.edu - PhD Student Geophysics
Fabio Silva - fsilva@usc.edu - Research Staff Computer Science
Geoffrey Ely - gely@anl.gov - Post-doctoral Scholar
Heming Xu - h1xu@sdsc.edu - Research Staff Computer Science
Haydar Karaoglu - hkaraogl@andrew.cmu.edu - PhD Student Civil Engineering
8
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
Jacobo Bielak - jbielak@cmu.edu - Professor Civil Engineering
John Yu - johnyu@usc.edu - Research Staff Computer Science
Jessica Donovan - jrdonova@usc.edu - PhD Student Geophysics
Jonathan Stewart - jstewart@seas.ucla.edu - Professor Civil Engineering
Gideon Juve - juve@isi.edu - Research Staff Computer Science
Karen Young - kayoung@usc.edu - Research Staff Geophysics
Kim Olsen - kbolsen@mail.sdsu.edu - Professor Geophysics
Kevin Milner - kmilner@usc.edu - M.S. Graduate Student Geophysics
Philip Maechling - maechlin@usc.edu - Research Staff Computer Science
John McRaney - mcraney@usc.edu - Research Staff Geophysics
Nico Luco - nluco@usgs.gov - Civil Engineer
Patrick Small - patrices@usc.edu - M.S. Graduate Student Computer Science
Po Chen - pchen@uwyo.edu - Professor Geophysics
Kyle Withers - quantumkylew@aol.com - PhD Student Geophysics
Ricardo Taborda - rtaborda@cmu.edu - Post-doctoral Scholar Civil Engineering
Robert Graves - rwgraves@usgs.gov - Seismologist
Samuel Bydlon- sbydlon@stanford.edu - PhD Student Geophysics
Scott Callaghan - scottcal@usc.edu - Research Staff Computer Science
Steven Day - sday@mail.sdsu.edu - Professor Geophysics
Thomas Jordan - tjordan@usc.edu - Professor Geophysics
Tran Huynh - tran.huynh@usc.edu - Research Staff Geophysics
Karan Vahi - vahi@isi.edu - Research Staff Computer Science
Xin Song - xinsong@usc.edu - PhD Student Geophysics
Yifeng Cui - yfcui@sdsc.edu - Research Staff Computer Science
Farzin Zareian - zareian@uci.edu - Professor Civil Engineering
Iunio Iervolino - iunio.iervolino@unina.it - Professor Civil Engineering
Ossian J. O'Reilly - ooreilly@stanford.edu - PhD student Geophysics
Other collaborators or contacts been involved?
1. Norm Abrahamson – PG&E Civil Engineer
2. Christine Goulet – PEER Civil Engineer
3. Ralph Archuleta – UCSB Professor Geophysics
4. Jeff Bayless – URS Corp Research Staff Geosciences
5. Doug Dreger – UC Berkeley Professor Geophysics
6. Paul Somerville – URS Corp Seismologist
7. John Anderson – U of Nevada at Reno Professor Geophysics
8. Gail Atkinson – University of Western Ontario Professor Geophysics
9. Maren Bose – Caltech Assistant Professor Geophysics
10. Impact - What is the impact of the project? How has it contributed?
The SCEC Broadband Platform is in active use by a scientific and private collaboration
studying seismic hazards of nuclear power plants. The software development work we
performed on the Broadband Platform during SEISM year 2 prepared the code for use in
this demanding application. On Sept 8, 2013, members of the California Ground Motion
Simulation Utilization committee met at a SCEC workshop. Committee members, including
C.B. Crouse, are investigating use of SCEC’s CyberShake PSHA for both urban seismic hazard
maps for Los Angeles, and for use in Building Code development in upcoming review cycles.
9
What is the impact on the development of the principal discipline(s) of the project?
The SEISM project is developing stochastic (Broadband Platform) and deterministic
(Hercules, AWP-ODC, CyberShake) ground motion simulations. The project is leading the
scientific transition from the current, ground motion prediction equation-based, seismic
hazard estimates, to physics-based, but more computational expensive, deterministic
simulation methods.
Our collaborative work with engineering users distinguishes our SEISM project. We are
bridging the gap between geoscientists that perform ground motion simulations, and users
of ground motion simulations. This collaboration has led to quantitative evaluation
procedures for evaluating new ground motion simulation methods, and our SEISM software
development activities have captured these agreements and procedures in open-source
scientific software.
Through careful analysis, we expect to identify the tradeoffs between the methods,
identifying where deterministic methods provide improved results. This work can inform
the whole seismic hazard analysis field which applications can benefit from the additional
time, and computational expense, of high-frequency deterministic ground motion
simulations.
What is the impact on other disciplines?
Civil engineers are interested in using ground motion simulation results to augment existing
observational data for strong ground motions. Despite the rapid increase in the number of
ground motion sensors, close-in (<10km) recordings for large magnitude earthquakes
(M8+) are quite rare. Engineers would like to use simulations to supplement the
observational record. Only through a careful evaluation process of evaluation and
validation, as we are performing on the SEISM project, will simulated ground motions be
accepted for use by engineers.
SCEC’s computational research has also made an important impact on NSF and DOE HPC
resource providers. Our computational research makes use of very large-scale scientific
workflows. During SEISM year 2, we worked with NCSA Blue Waters to make system and
software modifications to the Blue Waters computational environment, so that Blue Waters
can support large-scale workflows. We then used Blue Waters to perform our CyberShake
hazard calculations. These changes to the Blue Waters computational environment
benefitted SCEC research, but they also benefit other researchers using Blue Waters. We are
serving a similar function with DOE HPC resource providers. We have made recent progress
developing workflow solutions for DOE’s Titan system. It does not yet support or
workflows, but the DOE groups are showing their interest and willingness to support this
capability, based on the clear scientific needs of SCEC’s computational research.
What is the impact on the development of human resources?
Computational geoscience has recently emerged as a research discipline related to, but
distinct from, either geoscience or computer science. The SCEC SEISM project is training a
workforce in the multi-disciplinary skills needed to contribute to computational
geoscientific research. Project participants may then be qualified to work across fields
10
expanding their occupational choices. Project participants have accepted jobs in both
geoscientific and computer science positions.
What is the impact on physical resources that form infrastructure?
Our SEISM computational tools are being used to evaluate the seismic safety of public
power plants (Broadband Platform) and are being considered for use in development of the
next generation building codes (CyberShake Platform). California Department of Water
resources are using SEISM Project software in their safety evaluation of California
hydroelectric dams (Fig. 1).
What is the impact on institutional resources that form infrastructure?
None expected.
What is the impact on technology transfer?
As well validated open-source scientific software, both the SEISM UCVM Platform and the
Broadband Platform have attracted interest from private researchers and commercial
companies. SEISM software sustainability plans depend on attracting continued support for
SEISM software through a public/private collaboration supporting ground motion
simulation software development needed by strong motion researchers and building
engineers.
What is the impact on society beyond science and technology?
SEISM project has potential broad impact through improved public safety. By providing
engineers with more accurate and more complete information about earthquake generated
strong ground motions, the SEISM research and SEISM software tools have the potential for
large societal impact. In the United States, the U.S.G.S. is responsible for seismic hazard
evaluation and information and our SEISM research may improve public USGS seismic
hazard information that has societal impact beyond science and technology.
References:
1. Baker, J. W., Luco, N., Abrahamson, N. A., Graves, R. W., Maechling, P. J., & Olsen, K. B.
(2014). Engineering Uses of Physics-based Ground Motion Simulations. Proceedings of
the Tenth US Conference on Earthquake Engineering (in press).
2. Bijelic, N., T. Lin, and G. Deierlein (2014). Seismic response of a tall building to recorded
and simulated ground motions. Proceedings of the Tenth National Conference on
Earthquake Engineering (in press).
3. Böse, M., R. Graves, D. Gill, S. Callaghan and P. Maechling (2014). CyberShake-Derived
Ground-Motion Prediction Models for the Los Angeles Region with Application to
Earthquake Early Warning, Geophys. J. Int, (accepted June 2014).
4. Burks, L. S., and J. W. Baker (2014). Validation of ground motion simulations through
simple proxies for the response of engineered systems, Bull. Seismol. Soc. Am. (in press).
5. Cui, Y., E. Poyraz, K.B. Olsen, J. Zhou, K. Withers, S. Callaghan, J. Larkin, C. Guest, D. Choi,
A. Chourasia, Z. Shi, S.M. Day, J.P. Maechling, and T.H. Jordan (2013a). Physics-based
seismic hazard analysis on petascale heterogeneous supercomputers, in Proceedings of
the International Conference for High Performance Computing, Networking, Storage,
and Analysis, Denver, CO, Nov. 17-21.
11
6. Cui, Y., E. Poyra., S. Callaghan, P. Maechling, P. Chen and T. Jordan (2013b). Accelerating
CyberShake Calculations on XE6/XK7 Platforms of Blue Waters, Extreme Scaling
Workshop 2013, August 15-16, Boulder, CO.
7. Dreger, D., G. C. Beroza, S. M. Day, C. A. Goulet, T. H. Jordan, P. Spudich, and J. P. Stewart
(2013). Evaluation of SCEC Broadband Platform Phase 1 Ground Motion Simulation
Results. Southern California Earthquake Center Report, Los Angeles, California, USA.
8. Dreger, D., G. Beroza, S. Day, C. Goulet, P. Spudich, J. Stewart, and T. Jordan (2014),
Evaluation of SCEC Broadband Platform Phase 1 Pseudo Spectral Acceleration Ground
Motion Simulation Results, submitted to Seismol. Res. Lett. May 2014.
9. Gill, D., P. Small, P. Maechling, T. Jordan, J. Shaw, A. Plesch, P. Chen, E. Lee, R. Taborda, K.
Olsen, and S. Callaghan (2014). UCVM: Open Source Framework for 3D Seismic Velocity
Models, Seismol. Res. Lett., 85(2), 457.
10. Isbiliroglu, Y., R. Taborda, and J. Bielak (2013). Coupled soil-structure interaction effects
of building clusters during earthquakes, Earthquake Spectra, in press, doi
10.1193/102412EQS315M.
11. Lee, E., P. Chen, T. H. Jordan, P. J. Maechling, M. Denolle, and G. C. Beroza (2014a). Full3D Tomography (F3DT) for Crustal Structure in Southern California Based on the
Scattering-Integral (SI) and the Adjoint-Wavefield (AW) Methods, Journal of
Geophysical Research, in review.
12. Lee, E., P. Chen, and T. H. Jordan. (2014b) Testing Waveform Predictions of 3D Velocity
Models Against Two Recent Los Angeles Earthquakes, submitted to Seismol. Res. Lett.
13. Luco, N., T.H. Jordan, and S. Rezaeian (2013). Progress of the Southern California
Earthquake Center Technical Activity Group on Ground Motion Simulation Validation.
Seismol. Res. Lett., 84(2), 336.
14. Maechling, P., F. Silva, S. Callaghan, T. H. Jordan (2014) SCEC Broadband Platform:
System Architecture and Software Implementation, Seismol. Res. Lett.., submitted May
2014.
15. Olsen, K.B., and R. Takedatsu (2014). The SDSU Broadband Ground Motion Generation
Module BBtoolbox Version 1.5, Seism. Res. Lett., submitted May 2014.
16. Poyraz, E., H. Xu and Y. Cui (2014). I/O Optimizations for High Performance Scientific
Applications. ICCS’14, Cairns, June 10-12 (accepted).
17. Restrepo, D. and J. Bielak (2014). Virtual Topography–A fictitious domain approach for
analyzing surface irregularities in large-scale earthquake ground motion simulation,
Int. J. Num. Meth. Eng. (in revision).
18. Roten, D., K.B. Olsen, S.M. Day, Y. Cui, and D. Fah (2014). Expected seismic shaking in Los
Angeles reduced by San Andreas fault zone plasticity, Geophys. Res. Lett., 41(8), 27692777, doi:10.1002/2014GL059411.
19. Savran, W. and K.B. Olsen (2014). Deterministic simulation of the Mw Chino Hills event
with frequency-dependent attenuation, heterogeneous velocity structure, and realistic
source model, Seism. Res. Lett., 85(2), 498.
20. Shi, Z. and S.M. Day (2013a). Validation of dynamic rupture simulations for highfrequency ground motion, Seismol. Res. Lett., 84(2), 334.
21. Shi, Z. and S. M. Day (2013b). Rupture dynamics and ground motion from 3-D roughfault simulations, Journal of Geophysical Research, 118(3), 1–20,
doi:10.1002/jgrb.50094.
22. Taborda, R. and J. Bielak (2013). Ground-motion simulation and validation of the 2008
Chino Hills, California, earthquake, Bull. Seismol. Soc. Am. 103(1), 131–156, doi
10.1785/0120110325.
12
23. Taborda, R. and J. Bielak (2014). Ground-motion simulation and validation of the 2008
Chino Hills, California, earthquake using different velocity models, Bull. Seismol. Soc.
Am., in revision.
24. Wang, F., and T. H. Jordan (2014), Comparison of probabilistic seismic hazard models
using averaging-based factorization, Bull. Seismol. Soc. Am., in press.
25. Withers, K.B., K.B. Olsen, Z. Shi, S.M. Day, and R. Takedatsu (2013a). Deterministic highfrequency ground motions from simulations of dynamic rupture along rough faults,
Seismol. Res. Lett., 84(2), 334.
26. Withers, K.B., K.B. Olsen, and S.M. Day (2013b). Deterministic high-frequency ground
motion using dynamic rupture along rough faults, small-scale media heterogeneities,
and frequency-dependent attenuation, in Proc. SCEC Annu. Meet., Abstract 085, Palm
Springs, CA, September 9–12.
27. Zhou, J., Y. Cui, E. Poyraz, D. Choi, and C. Guest (2013). Multi-GPU implementation of a
3D finite difference time domain earthquake code on heterogeneous supercomputers, in
Proceedings of International Conference on Computational Science, 18, 1255-1264,
Barcelona, Spain, June 5-7.
13