Network for Earthquake Engineering Simulation
– multiple PIs
Dong Ju Choi
Amit Majumdar
Overview of NEES
• The George E. Brown Jr. Network for Earthquake Engineering Simulation (NEES)
is an NSF-funded MRE project for earthquake engineering research
• World-class experimental facilities around the country are linked by a state-ofthe-art network, making it possible for researchers to collaborate remotely on
experiments, computational modeling, data analysis and education
• The NEES Cyberinfrastructure Center (NEESit), based at UCSD, is a servicefocused organization created to deliver technology tools and infrastructure
• Since 2005, NEES is involved in HPC with following goals:
– Simulate large-scale models which require parallel computers with large number of nodes,
enough memory and a parallel file system with reliable parallel I/O
– Conduct extensive parametric studies to facilitate optimal specimen designs prior to testing
– Facilitate hybrid experimentation requiring supercomputer resources
– Enhance and calibrate existing computational models through access to experimental data
using supercomputers and data central facilities
ASTA effort history: past and current
• Collaborated with NEES PIs in writing successful proposal first
time in 2005 - NEES received 200K SUs
• Number of HPC users now grown to about 80; number of
groups 35
• In 2008-2009 ~500K SUs on IA-64, Ranger, Abe, Kraken
300000
250000
200000
150000
100000
50000
0
2006
2007
2008
SUs awarded
SUs used 1st Qr
NEES User Profile
• Mostly Civil, Structural engineers and Geoscientists
• Uses very diverse simulation packages such as ABAQUS, Ansys, LSDyna, and Fluent
• In addition there are codes (e.g OpenSees) based on algorithms
developed directly by the researchers – these are evolving to
improve parallel scaling
• Perform diverse and challenging research in earthquake science
and engineering without having to build new research utilities
from the scratch
User Groups
• PI: Panagiotis Tsopelas (Catholic University of America)
Dr. Tsopelas is simulating bridges using 3D nonlinear-finite element analysis. This analysis
determines the behavior of corrugate web members and their connections under cyclic
loading. A series of girders with different web slenderness ratios and corrugation
parameters will be examined and their effect to the response will be evaluated. Using
this analytical results design recommendations under dynamic cyclic loading will be
developed to gain high strength-to-weight ratios in seismic zones.
• PI: Sri Sritharan (Iowa State University)
Dr. Sritharan is using OpenSees to analyze effects of nonrectangular walls under multidirectional loads.]
• PI: Jean Provost (Princeton University)
Dr. Provost ported DynaFlow to the IBM Power4 cluster environment
• PI: Helmut Krawinkler (Stanford University)
Dr. Krawinkler is investigating sideway collapse of deteriorating structural systems using
Abaqus, OpenSees and Drain-2DX.
• PI: Andrei Reinhorn (SUNY at Buffalo)
Dr. Reinhorn is modeling braced steel frames with innovative bracing schemes (zipper
frames) using Abaqus, LS-Dyna and OpenSees.
User Groups..cont
• PI: Ron Riggs (University of Hawaii at Manoa)
Dr. Riggs is developing a performance-based Tsunami engineering model using Abaqus
and Fluent.
• PI: Wenjian Wang (University of Missouri, Rolla)
Dr. Wang is performing health monitoring and parametric identification of a 4000 ft
long cable-stayed bridge in a seismic zone. A proposed health monitoring model
will be developed after very comprehensive dynamic analysis with potential
damages on the bridge.
• PI: Terence Weigel (University of Louisville)
Dr. Weigel is investigating the non-linear analysis of masonry buildings for seismic
loading using reversed cyclic loading (compression toe crushing with steel
compression bucking, shear cracking and base slicing) using Ansys and OpenSees.
• PI: Shakhzod Takhirov (Univeristy of California, Berkeley)
Dr. Takhirov is modeling seismic isolators using moment resisting beam-to-column
connections including soil-structure interaction using Abaqus.
User Groups ...cont
•
•
•
•
PI: Gregory Fenves (University of California, Berkeley)
Dr. Fenves and his research team are the primary developers of the OpenSees
program. The group also performed the first 2048 processor benchmark run on the
DataStar after correcting the bottleneck in subdomain-channel type communication
in 2006. This was done by introducing numerous efficient parallel linear system
solvers like Mumps , SuperLU Diagonal and Petsc, and using an innovative domain
reduction method by avoiding the original OpenSees subdomains and bypassing the
need for channels. The PI’s main research activity is moved to a newly awarded
independent allocation, but the group is actively contributing to the NEESit
community by aggressively enhancing the OpenSees parallel scalability on the
TeraGrid platforms.
PI: Boris Jeremic (University of California, Davis)
Dr. Jeremic’s research group is also involved in actively developing the OpenSees
parallel version. The group has implemented a version that has improved the
performance of the dynamic computational load balancing which is important for
applications where strong inelastic effects are present.
User Groups..cont
• PI: Michael Gebman (University of California, San Diego)
Dr. Gebman is investigating the transfer of axial force within a steel shell pile foundation
filled with reinforced concrete continuing from his Ph.D. research. The transfer of
axial force was modeled through the surface bond and through a circumferential
mechanism (e.g. shear ring) fixed to the steel shell. Finite element results were
compared to full-scale experimental results to provide a verification of the observed
failure modes and the hysteretic response.
• PI: Kaspar Willam (University of Colorado)
Dr. Willam is investigating the seismic performance of retrofitted non-ductile RC frames
with infill walls.
• PI: Erol Kalan (California Geological Survey)
Dr. Kalan is investigating structural models for many input motions which will provide a
seismic vulnerability assessment and seismic retrofit design of various structures
using different schemes.
• PI: Jaesung Park (AIR Worldwide Corp.)
Dr. Park is investigating the assessment of non-linear building responses to multidirectional strong ground motions.
User Groups...cont
• PI: Zareian Farzin (University of California, Irvine)
The PI is investigating ground motion selection and modification working group
computation. This PI was a NEESit user as a graduate student, and recently joined
University of California Irvine as a faculty after finishing his Ph.D. at Stanford
University and got his own sub-allocation from this NEESit community MRAC
allocation.
• PIs: Charles Roeder and Jong Han Yoo (University of Washington)
Dr. Roeder and his research team are investigating the use of high performance
computing for international hybrid simulations of futuristic braced frame systems to
ultimately improve its seismic performance.
• PI: Stephen Mahin (University of California, Berkeley)
Dr. Mahin is also investigating the use of high performance computing for international
hybrid simulations of braced frame systems and their fracture
• PI: José Restrepo (University of California, San Diego)
Dr. Restrepo is modeling the collapse behavior of existing buildings damaged in past
earthquakes using OpenSees and component properties from experimental tests.
User Groups....cont
• PI: Solomon Yim (Oregon State University)
Dr. Yim is working on performance-based Tsunami engineering which involves probabilitybased structural, coastal, ocean and earthquake engineering analysis and design.
• PI: Roberto Leon (Georgia Institute of Technology)
Dr. Leon is investigating the effects of system factors for composite frames related to the
dynamic behavior and design of the frame structures subjected to seismic input
• PI: Tomasz Haupt (Mississippi State University)
Dr. Haupt and his research team are developing the NEESit SimPortal, an easy to use
interface to edit, view, and submit simulation jobs over the Grid for the Earthquake
Engineering Community. Usage of the current allocation is for portal development and
to test the simulation tools like OpenSees and other analysis packages on related HPC
resources.
• PI: Ahmed Elgamal (University of California, San Diego)
Dr. Elgamal, in addition to the previous PI of the NEESit effort, is also utilizing the NEES
allocation for his own research, which consists of a very large bridge model using
OpenSees soil materials and elements that were actually developed by Dr. Elgamal’s
group.
User Groups...cont
• PI: NEES community account (NEESit Community)
This community account is used for NEESit Grid development and related work, also for
research activity through the NEESit portal by any NEES researchers who were not
awarded with sub-allocations yet, but needs to access the NEES hpc hardware and
software resources.
• PI: Thuy Le (San Jose State University)
The PI is working on experimental determination of performance of drift-sensitive
nonstructural Systems under Seismic Loading
• PI: Yili Huo (University of California, Los Angeles)
The PI studies bridges under liquefaction caused by earthquake
• PI: Mehdi Saiid (University of Nevada Reno)
The PI is working on demonstration of NEES for studying soil-foundation-structure
Interaction
• PI: Pui-shum Ben Shing (University of Colorado)
The PI’s group works on seismic performance assessment and retrofit of non-ductile RC
frames with infill walls
SimPortal
Input GUI
Output GUI
• NEES is community allocation – 80
users
• GUI based Grid Portal for the Open
System for Earthquake Engineering
Simulation on TeraGrid resource
• Once the simulation is complete,
the user may preview the results,
and download them for a detailed
analysis
• Results can be shared through
NEESit repository
• ASTA consulting provided for
resource selection, arrangement
and porting issues
HPC Applications – 3rd party software
• 3rd party software ABAQUS, Ansys, LS-Dyna3D, Fluent run
in both MPI and OpenMP parallel mode on TG machines
• Helped users set up FEM input cases on HPC machines
• Helped with scaling and identifying right processor count for
simulations
• Abaqus: MPI-based parallel – 32-64 procs range depends
on problem type
• Ansys: has parallel solvers CG, JCG, Multigrid – MPI and
SMP
• LS-Dyna has both SMP and MPP parallelism
HPC Applications – research codes
• The Open System for Earthquake Engineering Simulations
(OpenSees) – originally serial FE code (UC Berkeley) for
simulating the seismic response of structural and geotechnical
systems
• Helped parallelize the solver part of OpenSees
• Provided Petsc linear system solver kernels
• Helped with single processor and parallel performance
analysis
• Later researchers implemented SuperLU and MUMPS
solver and scaling is better than Petsc
• Code has options to choose solvers now
OpenSees Performance on DataStar
• The OpenSees parallel performance for a ground motion
simulation using the SDSC DataStar (IBM Power4)
• MUMP solver for a 40X40X40 brick element under 50 loading
steps – explicit time stepping
• hpm shows performance of this code on DataStar is
1.2 Gflops/sec – ~17% of the peak processor performance
18
14
Speed Up
• OpenSees now being run on
Abe, Ranger
16
12
10
8
6
Speed Up
4
Theoretical
2
0
0
500
1000
1500
PEs
2000
2500
NEES Visualization
• http://www.sdsc.edu/us/visservices/projects/nees/
By Amit Chourasia, Steve Cutchin, Mike
Rossmassler,Visualization Services, SDSC
More Information
• OpenSees: http://opensees.berkeley.edu/index.php
• SimPortal: http://simportal.nees.org/sphomepage/
• NeesIt: http://it.nees.org/index.php