3.1
Science and Engineering Highlights 2007
1. Mu-Hyun Baik, Indiana University
2. David Baker, University of Washington
3. Chaitan Baru, SDSC/UCSD; Ramón Arrowsmith , Arizona State
University
4. Jacobo Bielak and David O’Halloran, Carnegie Mellon University
5. Volker Bromm, University of Texas
6. Indrajeet Chaubey, M. Gitau, L. Chiang, and M. Sayeed, Purdue
University
7. Peter Coveney, University College, London
8. Peter Coveney, University College, London
9. Stefano Curtarolo, Duke University
10. Michael Deem, Rice University, David Earl, University of Pittsburgh
11. Alex Demkov, University of Texas
12. Tiziana Di Matteo, Carnegie Mellon University
13. Maria Esteva, University of Texas at Austin
14. E. Farhi and M. Taylor, Institute Max von Laue-Paul Langevin (ILL),
France
15. Wen Jiang, Purdue University, Weimin Wu, Purdue University, Matthew
L. Baker, Joanita Jakana and Wah Chiu, Baylor College of Medicine, Peter
R. Weigele and Jonathan King, MIT
16. Tom Jordan, University of Southern California
17. Masao Kanamitsu, Scripps Institution of Oceanography
18. J. Andrew McCammon, UC San Diego
19. Kumar Mahinthakumar, North Carolina State University, and Jim Uber,
University of Cincinnati
20. P.R. Mukund, Rochester Institute of Technology
21. Aiichiro Nakano, University of Southern California, Yoshio Tanaka,
Hiroshi Takemiya, Satoshi Sekiguchi (AIST, Japan), Shuji Ogata, Takahisa
Kouno (NITECH, Japan), Rajiv K. Kalia, Aiichiro Nakano, Priya Vashishta,
USC
22. 2007 NOAA Hazardous Weather Testbed Spring Experiment
23. Michael Norman, UC San Diego
24. Tom Quinn, University of Washington
25. Randall Q. Snurr, David Dubbeldam, Houston Frost, Krista S. Walton,
Northwestern University
26. Mete Sozen, Nicoletta Adamo-Villani, Purdue University
27. Juri Toomre, University of Colorado at Boulder, and Mark Miesch,
National Center for Atmospheric Research
28. Igor Tsigelny, SDSC, UC San Diego
29. Anne Villamil and Stefan Krasa, University of Illinois at UrbanaChampaign
30. Gregory Voth, University of Utah; Phil Blood, Pittsburgh Supercomputing
Center
31. Gregory Voth, University of Utah
32. Ming Xue, University of Oklahoma
33. Nicoletta Adamo-Villani, Purdue University
34. LEAD WxChallenge
Mu-Hyun Baik, Indiana University
Title: Fighting Alzheimers
Field: Bioscience
As part of the Bandwidth Challenge competition held at the SC07 conference in
Reno, Nevada, a team led by Indiana University enabled Indiana University
faculty member Mu-Hyun Baik to analyze the structure of the Amyloid-β
protein. This protein is thought to be the cause of Alzheimer's disease, and the
analyses performed provided new insights that may help fight Alzheimer’s. The
Amyloid-β protein structural analysis was done with a workflow that involved a
Lustre Wide Area Network file system (based on the IU Data Capacitor), a Lustre
file system set up on the SC07 exhibit floor, and IU’s Big Red supercomputer.
Using Dr. Baik’s structural analysis software, along with a variety of other
workflows, the Indiana University led team demonstrated a bidirectional
transfer rate of 18.2 Gigabits per second (Gbps) out of a possible 20 Gbps. This
shows Lustre-WAN’s tremendous capabilities for supporting data-intensive
research across wide area networks. The TeraGrid-attached Big Red and Data
Capacitor systems made this research possible. In turn, this demonstration
project serves as a proof of concept enabling data-centric research across multiple
institutions within the TeraGrid. TeraGrid collaborators on this project included
PSC and ORNL; international collaborators included Technische Universitaet
Dresden.
NSF Award OCI 0504075 (IU TeraGrid Resource Partner – C. Stewart, PI).
Figure 1. The structure of Amyloid-β protein. Researchers used an experimental
Lustre-based wide area network connecting multiple TeraGrid sites to search for
ways to disrupt the creation of the fibrils that cause Alzheimer’s disease.
David Baker, University of Washington
Title: Decoding Proteins
Field: Bioscience
Proteins are the molecules of life, and the myriad ways they function depend on
the dynamics of their 3-D shapes. The research holds promise for rational design
of drugs for tomorrow’s medicine chest. “Having access to the massive
computing resources of the TeraGrid is revolutionizing the approaches that are
open for protein structure prediction,” says Baker, whose group ran targets for
the most recent CASP7 competition (Critical Assessment of Structure Prediction).
For CASP7, the team used 1.3 M hours on NCSA Condor resource to identify
promising targets (coarse resolution). Then refined 22 promising targets on
730,000 hours of SDSC Blue Gene. SDSC helped improve scaling to run on
40,960 processor BlueGene at IBM’s Watson Research Center, which reduced the
running time for a single prediction to 3 hours, instead of weeks on a typical 1000
processor cluster. BlueGene well suited to compute intensive, small memory
tasks. The prediction for a specific target protein led more than 100 CASP7
entries from other groups.
To let other researchers run Rosetta jobs, Baker’s group also supports the Robetta
Web portal for which NCSA has provided additional computational power,
bringing the throughput time down to days instead of months, and enabling
more researchers to complete more runs using Rosetta. Related links:
http://www.bakerlab.org/
Figure: Predicting Protein Structure Best CASP7 prediction for Target T0380,
showing the native protein structure (blue), which is made public only after
predictions were submitted, the initial starting structure (pink), and the Rosetta
prediction (green). Following preparations on TeraGrid systems at SDSC and
NCSA, the calculation was performed in an unprecedented three hours on 40,960
processors of the IBM Watson Blue Gene system. Image courtesy of Ross Walker,
SDSC, and Srivatsan Raman, University of Washington.
Chaitan Baru, SDSC/UCSD; Ramón Arrowsmith , Arizona State University
Title: GEON Science Gateway
Field: Geology
Geoscientists who study the subterranean world must be ingenious, finding
indirect ways to penetrate the mysteries of that hidden world. A key tool is to
gather the most detailed possible topography data to help understand the
complex 3D structure of the earth, which gives vital clues to both geological
hazards and mineral resources that lie below. The GEON Science Gateway
democratizes access to state-of-the-art geosciences tools and data.
To give geoscientists access to detailed laser scans of the earth’s surface known as
Light Detection And Ranging (LiDAR) -- normally too massive to access -- the
GEON Science Gateway is providing broad community access to this key
topographic data, and much more, in a cyberenvironment that integrates
geoscience tools, applications, high-performance computing resources, and
diverse data collections in a unified portal. Project leaders Chaitan Baru of SDSC
and Ramón Arrowsmith of Arizona State University have coordinated
development of the GEON LiDAR Workflow, which uses TeraGrid data and
computing resources to let hundreds of users from universities and USGS,
industry, and beyond access terabytes of topography data of unprecedented
detail.
In addition to data access, GEON researcher Dogan Seber of SDSC and
colleagues have also developed an easy-to-use portal to the SYNSEIS (SYNthetic
SEISmogram) modeling environment, which lets users compute 2D and 3D
regional seismograms showing earthquake ground motion. GEON is also
helping educate a new generation of cyberinfrastructure-savvy researchers and
serving as a model for other disciplines, with more than five other projects
adopting the GEON infrastructure. Related links: https://portal.geongrid.org
Figure: Perspective View of the San Andreas Fault Through the GEON TeraGrid
Science Gateway, users reach the interface to the GEON LiDAR workflow, which
produced this view looking north. Making this previously unavailable high
resolution topography data accessible to geoscientists is helping them better
understand the earth’s history and detect hidden hazards and resources below.
Image: Chris Crosby, ASU/GEON
Jacobo Bielak and David O’Halloran, Carnegie Mellon University
Title: When the Earth Shakes
Field: Civil Engineering
Civil engineers want to predict how the earth will shake, taking into account
subsurface soil properties and the nature of seismic waves. The CMU Team and
SCEC (Southern California Earthquake Center) create realistic 3-D models of
earthquakes in the Los Angeles basin, using empirical information about the
inhomogeneous basin properties. The changing nature of soil characteristics
demand adaptive meshing (but only once).
It is computationally very demanding to find the ‘high frequency’ (above 1 Hz)
properties because these involve shorter wavelengths and thus finer meshes. But
these are what matter for the building engineers.
The Quake team, a collaboration of computer scientists and computational
scientists and PSC consultants, developed Hercules code for PSC Cray XT3 that
does the meshing, the load balancing, the wave propagation, and the
visualization. The application won the SC06 Analytics Challenge Award.
The model runs on the whole XT3, and sustains over a Teraflop. PSC helped
optimize the code and developed the ability to stream results to remote sites to
enable the researchers to interact with the calculations in real time, changing
what is being visualized.
PSC also developed visualization tools to compare results from Hercules with
those of SCEC (uniform meshes) to validate results.
Volker Bromm, University of Texas
Title: Cosmological Simulation – The End of the Cosmic Dark Ages
Field: Astronomy
In the absence of powerful enough telescopes, Bromm and other astronomers
rely on supercomputers to peer deep into the past. Using the supercomputing
resources of the TeraGrid, Bromm and his team have theoretically modeled
aspects of the first billion years of the 14 billion-year-old universe, computing the
interactions of molecules on the largest and smallest scales over millions of years
and changing atmospheric conditions. Bromm’s conclusions closely match those
of other cosmologists whose computational simulations also predict giant, bright,
unstable stars that played a pivotal role in the enrichment of the universe.
With the planned launch of NASA’s James Webb Space Telescope (JWST) set for
2013, Bromm and his colleagues will learn more about the nature of the early
universe. They may even obtain an observational answer to what the first objects
that coalesced after the Big Bang looked like. However, preliminary studies
suggest the space telescope still won’t have the sensitivity to see isolated early
stars, so Bromm is working on a series of studies to help guide the JWST’s
detection systems in their search for small, galaxy-sized aggregates. If the process
works, Bromm will be able to compare observational data with his
computational models to refine the theory of the first stars.
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Figure: This supercomputer simulation shows how gravity assembles one
hundred million solar masses of dark matter (top row) and gas (middle row) into
the first galaxy. Each column (from left to right) corresponds to a different
snapshot, from 100 million years to half a billion years after the Big Bang. Early
on, the cosmic matter is still quite uniform and dispersed, but gravity inexorably
draws it in, eventually leading to a violent collapse during the birth of the first
galaxy.
Indrajeet Chaubey, M. Gitau, L. Chiang, and M. Sayeed, Purdue University
Title: BMP effectiveness assessment in a pasture dominated watershed
Field: Earth Sciences
The objective is to analyze various Best Management Practices (BMPs) for nonpoint source pollution control and their effects on water quality, considering
different weather scenarios over a 25-year period. This project is part of an ongoing US Department of Agriculture (USDA) funded Conservation Effects
Assessment Project (CEAP) for which the overall goal is to quantify relationships
between water quality, and selection, timing, and spatial distribution of BMPs
within a watershed primarily impacted by land-application of animal manure.
This project is focused on the Lincoln Lake watershed in the Ozark Highlands
eco-region of northwest Arkansas using the Soil and Water Assessment Tool
(SWAT) to analyze BMP effects. The SWAT model is a physically-based
distributed-parameter river basin, or watershed scale hydrological model that
predicts the impact of land management practices on water, sediment and
agricultural chemical yields in complex watersheds with varying soils, land use,
and management conditions over long periods of time. There was a need to
evaluate 172 different BMP scenarios, incorporating 1000 different weather
realizations in each BMP scenario resulting in 172,000 different model runs. On a
LINUX system, X86_64 architecture, SWAT takes 8-10 minutes to complete one
run; thus the time required to complete all runs was 23,000-29,000 CPU hours
(2.6-3.3 years) on a single machine. The TeraGrid Condor clusters were used.
This area has not been explored in depth in the past, due to the complexity and
inherent computationally intensive nature of such techniques.
Figure: Total phosphorus losses as a function of various BMPs implemented in
the watershed and stochasticity of weather. The results of this computational
project will aid watershed planners, state and federal agencies, and agricultural
producers in planning management interventions for watershed water quality
protection. This project will provide information necessary to take BMP
evaluation to the next level, for optimal scenarios for BMP selection and
placement in a watershed. This project will set the stage for research using
various state-of-the-art search algorithms in BMP evaluations.
Peter Coveney, University College, London
Title: Life-Sized Clay
Field: Materials Science
In the late ‘80s, Toyota patented a nanocomposite material made of nylon
interspersed with layers of a common clay called montmorillonite. The result is a
light, strong material that offers improved performance (and recyclability) at
significantly less weight. “The clay confers remarkably enhanced material
properties to the original polymer, like mechanical strength and barrier
properties, such as to prevent diffusion of gases and to act as a flame retardant.
There are many applications for these novel materials,” says theoretical chemist
Peter Coveney of University College, London.
The key is to better understand how clay particles behave at the nanoscale, in
particular to control how they “exfoliate”—scatter from layered arrangements
into individual nanometer-thick sheets that can be dispersed within a matrix of
polymer. Previous simulations have been limited to system sizes in the range of
10,000 atoms—much smaller than the realistic size of the clay particles and too
small to observe many physical properties of the material.
Using TeraGrid systems at TACC, NCSA and PSC in a “federated grid” with
systems at the UK National Grid Service and the EU Deisa grid, Coveney and
colleagues did extensive simulations of clay particles in system sizes that range
up to 10 million atoms. Employing innovative middleware his UK team
developed, called the Application Hosting Environment, the researchers moved
with ease among the three different grids. At large system sizes—up to three
orders of magnitude larger than prior work—the simulations approached the
realistic size of clay “platelets.” The results (reported this year in the Journal of
Physical Chemistry C) revealed thermally-induced undulations in the clay sheets
not before observed, findings that make it possible to calculate elastic properties
difficult to obtain experimentally.
Figure: Visualization of a clay system containing approximately 10 million
atoms after 0.5 nanoseconds of simulation. The system is 0.3 microns wide by 0.2
microns deep by 2.5 nanometers high. The vertical dimension is expanded to
allow visualization of thermal sheet fluctuations. Atoms are colored: Mg (green),
Al (blue), O (red), Si (gray), Na (yellow), H (white). Image courtesy of Peter
Coveney.
Peter Coveney, University College London
Title: Patient Specific Medicine
Field: Biochemistry
Every field of study has its “Holy Grail” — a tantalizing new model with the
potential to revolutionize society. In biomedicine, the Holy Grail is patientspecific medical treatment. With the resources of TeraGrid, Coveney pushes the
limits of biomedical research and tests methods for the fast and accurate
treatment of unique illnesses.
“To be able to tailor medical treatment to a person and their ailments, instead of
giving them some average course of treatment — we’re only going to get to that
level of patient specificity if we use computational science and high performance
computing, of that there can be no doubt,” Coveney said.
Coveney and his colleagues first developed fine-detailed molecular
representations for HIV protease and saquinavir, the first HIV drug to be
discovered that inhibits the replication of HIV by binding to the protease
molecule. They then used molecular dynamics codes that fully exploit the
massively parallel power of TeraGrid’s supercomputers to determine how well
the drug binds to the “wild-type” HIV protease and to three drug-resistant
mutant strains. Coveney’s results mirrored the experimental data and revealed
how the virus protein interacts with the drug, including how mutations alter
critical interactions to lower the strength of drug binding.
Figure: A snapshot of the HIV-1 protease (a key protein that is the target for the
protease inhibitor drugs) from a computational simulation. Mutations from the
"wildtype" can occur within the active site (G48V) and at remote locations along
the protein chain (L90M ). The "asp dyad" is at the centre of the active site, where
polyprotein changes are snipped by the enzyme; this is the region that any drug
must occupy and block.
Stefano Curtarolo, Duke University
Title: Computational Discovery
Field: Materials Research
Curtarolo uses the TeraGrid for his disparate material property research
problems. His explorations of carbon-tube catalysts, super-alloys, quasi-crystal
structures and superconductors netted Curtarolo career awards in 2007 from the
Navy and the National Science Foundation, and led to the prediction of a
superconducting, lithium-boron “sandwich.”
When scientists look beyond the bulky macroscopic characteristics of materials
down to the nano-level — structures as small as one nanometer, or roughly one
thousand times smaller that the width of a human hair — they often find
material characteristics far different than those they expected, with strange and
potentially useful states.
“Going from macro to nano completely changes the physics of the systems,”
Curtarolo explained, “but so far, we’ve only been able to calculate the properties
of very small systems, ten to one-hundred atoms maximum. This is why we need
supercomputers, because by increasing the size of the calculations, we can see
phenomena that happen only on the nanoscale that are worth investigating.”
Curtarolo recently completed a paper using a quantum mechanics simulation of
up to three hundred-atoms, among the largest nanoparticles calculated to date.
Curtarolo’s most significant discovery happened accidentally, when he stumbled
on the superconducting lithium-boron metal-sandwiches while doing theoretical
materials research for another project. The prediction came with the application
of “ab initio” (from first principles) methods that allow high-performance
computing systems to predict and characterize the structure of materials by
simulating the properties of a material based on its atomic components and the
laws of electrodynamics. By parallelizing his code for TeraGrid’s systems,
Curtarolo was able to complete 900,000 hours of computing time on 670 jobs
(across almost 10,000 allocated cores) in nine months.
"Hypothetical" layer of methane adsorbed on a quasi-crystalline surface.
Michael Deem, Rice University, David Earl, University of Pittsburgh
Title: Industry Darlings
Field: Materials Science
Zeolites are silicate minerals with a porous, Swiss-cheese-like structure. For
decades, chemists have relied on zeolites to catalyze chemical reactions on an
industrial scale. They are used to make everything from gasoline and asphalt to
laundry detergent and aquarium filters.
In the past 50 years, the catalog of naturally occurring zeolites – there are about
50 – has been bolstered to about 180 with the addition of synthetic varieties,
Deem and Earl used the TeraGrid systems at TACC, Purdue, Argonne, NCSA,
SDSC to identify potentially new zeolites by searching for hypothetically stable
structures. Their database now contains over 3.5 million structures. By studying
the catalog, scientists might find structures that are more efficient, either in terms
of energy inputs or in waste byproducts.
TACC developed tools like MyCluster, harnessing the distributed,
heterogeneous resources available on the TeraGrid network into a single virtual
environment for the management and execution of their simulation runs.
At Purdue, the application was used within a Condor pool of more than 7,000
processors using standard Linux tools for job management. Over 4M processor
hours were used in 22 months. Performance engineer supported application
revisions and job management scripts to:

Track execution time to detect runaways and terminate gracefully.

Add application self checkpoint to recover from job and system failures

Control of number of jobs in queue to practical maximum w.r.t. system
capabilities. Dynamically adjust cap to hold percentage of jobs executing
to 80-90% of total in queue (number executing varied from <100 to peak of
~2000).
Alex Demkov, University of Texas
Title: Designing the Nano-Future
Field: Physics
It’s practically a given that this year’s PCs will be significantly faster than last
year’s. This trend in computer hardware, dubbed Moore’s Law, depends
primarily on engineers’ ability to miniaturize transistors and double the number
that can be placed on an integrated circuit every two years. However, scientists
recently discovered that at the 32-nanometer scale — the current target for
semiconductor fabrication — the behavior of the materials used to make
transistors changes in distinct and unexpected ways.
A phenomenon called quantum tunneling let electrical charge leak through the
gate dielectric, making the mechanism ineffective. So a search began among
chipmakers and researchers for new processes, new architectures, and new,
unexplored materials to continue shrinking the transistor.
Demkov is at the forefront of the search for new materials in nano-scale
semiconductor production. His research explores the gate stack — the most
important element of a transistor’s speed — and bridges the gap from first
principles to device physics, helping to uncover the rules that govern the
production and operation of working computers and cell-phones.
Prompted by the tunneling leakage problem, the industry converged on an oxide
of a semi-obscure element, hafnium, as the basis for nano-scale semiconductor
technology, stimulating new questions about an old material. To speed the
search for semi-conducting compounds, Demkov relies on the high-performance
computing system of TeraGrid. The immense computational power of TeraGrid
enables Demkov to explore the atomic and crystalline characteristics of novel
compounds to determine their usefulness in semiconductor construction.
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Figure: Transmission electron microscopy image of a 65 nm semi-conductor.
Tiziana Di Matteo, Carnegie Mellon University
Title: Making the Holes Count
Field: Astrophysics
Once thought to be rare, exotic bodies, black holes are now known to be
ubiquitous, often hugely massive features of galaxies. Until recent work by a
team of physicists led by Tiziana Di Matteo of Carnegie Mellon University,
however, computational power had been insufficient to include black holes
within large-scale simulations of the universe.
Using PSC’s Cray XT3, Di Matteo and colleagues from the Max Planck Institute
and the Harvard-Smithsonian Center for Astrophysics simulated the evolution of
the cosmos over the 14 billion years since the Big Bang, with black holes and the
quasars they spawn included. Their findings, published in the Astrophysical
Journal, show that black holes regulate galaxy formation. The researchers tracked
the evolution of fundamental relationships between black holes and the galaxy
where they reside, and they have developed new insight into the formation of
the first quasars in the universe.
“Ours is the first cosmological simulation to incorporate black-hole physics,”
says Di Matteo, a theoretical cosmologist. “It involves more calculations than any
prior similar modeling of the cosmos. The galaxies we see today look the way
they do because of black holes. This result offers the best picture to date of how
the cosmos formed and evolved.”
Figure: Snapshots from the simulation show evolution of structure in a large
volume of the universe. Gas density is shown (increasing with brightness) with
temperature (increasing from blue to red). Yellow circles indicate black holes
(diameter increasing with mass). At about 450 million years after the Big Bang,
left, as the early universe still shows a relatively uniform structure, the first black
hole appears. At about 6 billion years, right, the universe has many black holes
and a pronounced filamentary structure.
Maria Esteva, University of Texas at Austin
Title: Analyzing social relationships within an organization
Field: School of Information
TACC’s Visualization Group worked with Maria Esteva to visualize social
relationships within an organization over a ten-year span on Maverick, a
TeraGrid visualization resource. Using the corpus of electronic documents
produced by all organization employees, Maria computed a cosine similarity
matrix for the corpus and aggregated the document scores for each employee.
With these aggregate scores, she computed a relationship score between each
pair of employees by tracking which to which similar documents each employee
contributed.
The visualization takes the director, the focus employee, and plots the other
employees around it relative to the strength of the employees working
relationship. The visualization demonstrates that the relationship between the
director and the organization managers remains relatively unchanged across the
ten years of the dataset. It further reveals interesting fluctuations in the
relationship between the director and the organization president, and between
the director and other employees in their department.
E. Farhi and M. Taylor, Institute Max von Laue-Paul Langevin (ILL), France
Title: Neutron Scattering
Field: Physics
E. Farhi and M. Taylor have completed virtual neutron scattering experiments
for twelve materials using a TeraGrid cluster at the Oak Ridge National
Laboratory and ILL computing resources. As an example in the figures below,
the simulated time of flight data for copper plate for the 1.1A beam matches the
experimental results very well. The VASP package was used for the timeconsuming ab initio molecular dynamics calculations [G. Kresse, J. Hafner] and
the trajectory to S(q,ω) converter was the nMoldyn package [G. Kneller, K.
Hinsen]. The virtual experiments were completed using the McStas package [E.
Farhi, P. Willendrup, K. Lefmann, E; Knudsen and K. Nielsen] with S(q,ω) input.
This is an initial proof of principal demonstration that advanced
cyberinfrastructure can be used to predict and evaluate experimental data, and in
so doing advance the knowledge acquired from a large science user facility and
tightened the feedback loop between theory and experiment using simulation.
This material is based upon work supported by the National Science Foundation
under the following NSF programs: TeraGrid Partners, Partnerships for
Advanced Computational Infrastructure, Distributed Terascale Facility (DTF)
and Terascale Extensions: Enhancements to the Extensible Terascale Facility
under current TeraGrid project allocation, TG-DMR060000T.
Figure: Time of flight data for Copper plate using 1.1A beam (log scale).
Experimental (left) and simulated (right) results agree well.
Wen Jiang, Purdue University, Weimin Wu, Purdue University
Matthew L. Baker, Joanita Jakana and Wah Chiu, Baylor College of Medicine
Peter R. Weigele and Jonathan King, MIT
Title: High resolution 3-D reconstruction of infectious viruses
Field: Biology
Viral infections are threats to human health, the society and the economy. High
resolution 3-D structure of the virus particles, if available, will provide important
insights to the development of effective prevention and treatment strategies. The
research group of Wen Jiang at Purdue University has used the electron cryomicroscopy and single particle 3-D reconstruction (cryo-EM) technique to
determine the structure of many viruses at intermediate resolutions (6-10 Å).
Now his group, in collaboration with researchers in Baylor College of Medicine
and MIT, has demonstrated the 3-D reconstruction of the infectious bacterial
virus ε15 at 4.5 Å resolution, which allowed tracing of the polypeptide backbone
of its major capsid protein gp7. The structure reveals similar protein architecture
to that of other tailed double-stranded DNA viruses, even in the absence of
detectable sequence similarity. However, the connectivity of the secondary
structure elements (topology) in gp7 is unique. These results have been just
published on Nature (Feb 28, 2008). Large scale computing resource provided by
the Condor pools at Purdue University is indispensible for such an achievement.
For the cryo-EM technique, the purified virus particles are plunge-frozen in
vitreous ice and imaged in high-end cryo-electron
microscopes. Due to radiation damage limitation, low
dose imaging conditions are used which result in very
noisy images. It is essential to collect large number
(104-105) of 2-D images (8002 pixels/image) of the virus
particles to reach high resolution (4.5 Å). The 2-D
images represent the projections of identical 3-D
structure viewed at different angles, requiring
intensive computation to accurately determine their
relative orientations before the 2-D images can be
coherently merged into a single high resolution 3-D
structure. Thus, the availability of large scale
computing resources is critical to the success of the cryo-EM 3-D reconstruction
projects. Wen Jiang group, with the help from RCAC at Purdue, have adapted
the image processing software, EMAN, to successfully run on the Purdue
Condor pools. A large Condor submission could harvest >500 CPUs on average
and >1,400 CPUs at peak for the ε15 project. As the Condor pools are being
expanded at Purdue University and across the TeraGrid, the research projects in
Wen Jiang group will be able to tap into even larger computational resources to
see even higher resolution (3 Å) structural details of the viruses in the future.
Figure: 3-D structure of bacterial virus ε15. Shown are the surface view of the 4.5
Å resolution density map solved by cryo-EM 3-D reconstruction (left) and the
backbone model of the major capsid protein gp7 (right) built from this density
map.
Tom Jordan, University of Southern California
Title: ShakeOut: Unleashing a Virtual Megaquake
Field: Computational Seismology
California is long overdue for a massive earthquake, and earthquake scientists
have turned to simulations, or "virtual earthquakes," that harness the power of
supercomputers to predict the likely location and severity of resulting ground
motion. Researchers from the Southern California Earthquake Center, in
collaboration with computational scientists at the San Diego Supercomputer
Center, have now used NSF-supported TeraGrid resources to achieve the largest
and most realistic simulations ever of a magnitude 7.8 earthquake on the
southern San Andreas Fault – “the big one.” Such simulations are important to
underpin improved seismic hazard estimates, better building codes in high-risk
areas, and safer structural designs, potentially saving lives and property.
Following efforts by SDSC TeraGrid staff to improve model efficiency, the
researchers used systems at SDSC and the Texas Advanced Computing Center to
capture ground motion frequencies as high as 1 Hertz, twice as high as previous
simulations, and producing some 3 terabytes of data which will be archived in
the SCEC Digital Library at SDSC. This higher frequency gives new clues into
how such quakes may shake Southern California and affect tall buildings in the
Los Angeles area, where peak predicted motion reached more than 3.5 meters
per second. Displaying the results in SDSC-produced visualizations helped the
scientists gain new insights into the strong focusing of ground motion in the
direction of the rupture and the “waveguide” effect of the sediment-filled basins
and mountain chains that can channel unexpectedly large amounts of energy
into the Los Angeles area. Related links:
http://www.geology.sdsu.edu/research/
Figure: More than just pretty pictures, visualizations of the ShakeOut magnitude
7.8 earthquake simulations are a vital tool helping scientists understand the "big
picture" of their results and guide the simulations. The figure shows peak
ground velocity, reaching more than 3.5 meters per second in the Los Angeles
area with great potential for structural damage. Warmer colors are higher
ground velocity. Credit: Amit Chourasia, SDSC Visualization Services.
Masao Kanamitsu, Scripps Institution of Oceanography
Title: Regional Spectral Analysis
Field: Climate Research
Climate research—particularly studies aimed at management of water, energy,
forestry, fisheries, or agriculture—requires fine-scale data over long time periods.
But it’s nearly impossible to find data from multiple decades that is consistent,
comparable, and of sufficient resolution. Therefore scientists turn to reanalysis,
the process of integrating data from disparate sources within a numerical model
in order to create a comprehensive dataset.
Using compute resources at SDSC and NCSA, Kanamitsu and postgraduate
researcher Hideki Kanamaru were able to carry out a fine-scale 10 km regional
reanalysis of California covering 57 years, from 1948 to 2005 (CaRD10, for short).
Using a technique called dynamical downscaling with the state-of-the-art
Regional Spectral Model, they found that their reanalysis results compared well
with observations and yielded more accurate wind and temperature data than
other methods on all time scales, whether hourly or across decades. Their results,
to be published in the Journal of Climate in 2007, indicate that dynamical
downscaling could be a reliable way to derive fine-scale regional detail from
coarser analyses.
Their work is supported by the California Energy Commission Public Interest
Energy Research program.
Figure: Near surface wind for a Catalina Eddy event on May 22, 1984. Shades are
surface height (in meters) for each analysis. CaRD10 is the dynamical
downscaling of California. The analysis using surface observation is shown on
the right. Image courtesy of Hideki Kanamaru and Masao Kanamitsu.
J. Andrew McCammon, UC San Diego
Title: Fighting Bird Flu: Simulations Help Identify Promising Vaccine Target
Field: Computational Biology
New drug targets are desperately needed to combat deadly bird flu virus, with
the only drugs now available – Tamiflu and Relenza – in limited supply and
some virus strains already resistant to them. Building on X-ray crystallography
studies, researchers at UC San Diego have used NSF-funded TeraGrid
supercomputers to identify promising new drug targets.
UCSD postdoctoral fellow Rommie Amaro is studying N1 neuraminidase, an
enzyme on the surface of the avian influenza virus, which is needed to spread
the infection. To capture the complex dynamics of proteins, which do not show
up in traditional static crystallography studies, scientists have used molecular
dynamics codes. Zeroing in on likely sites for new drugs, they provide timesaving guidance to those working on the rational design of inhibitors to such
diseases. Using the NAMD molecular dynamics program running on SDSC’s
DataStar and NCSA’s Abe system, the researchers, working in the laboratory of
UCSD professor J. Andrew McCammon, explored two 40 nanosecond (a billionth
of a second) simulations of the N1 neuraminidase, requiring about a day of
compute time for each five to eight nanoseconds of simulations.
In a 2007 paper in the Journal of the American Chemical Society, the team
described surprising new details of interest to drug designers that they observed
in movies of the simulations. They found a key region, or “hot pocket” to be
more flexible, closing and opening much wider than previously supposed and
identified another area that acts in concert. In addition, the group has already
identified several potential inhibitors that might represent new drugs against
avian flu. The research was supported by the National Institutes of Health (NIH),
Accelrys, Inc., and NSF TeraGrid resources. Links: http://mccammon.ucsd.edu/
Figure: The image shows a key enzyme on the surface of avian influenza, N1
neuraminidase, needed to spread infection to new cells. The anti-viral drug
Tamiflu is shown bound in the active sites (yellow). Molecular dynamics
simulations of this enzyme-drug complex are being run on SDSC and NCSA
supercomputers to help develop improved drugs to fight this disease. Image:
Rommie Amaro, UCSD.
Kumar Mahinthakumar, North Carolina State University, and Jim Uber,
University of Cincinnati
Title: Urban Water
Field: Environmental Science
Urban water distribution systems cover hundreds of square miles, but the water
is largely unmonitored after it leaves the treatment plant. This situation creates a
threat to the health and economy of any urban area, through both intentional and
unintentional contamination. Researchers use NCSA's Mercury cluster--and
other TeraGrid systems--to develop new methods of locating the source of
contaminants and testing approaches to limiting their impact. Using real-world
data from the Greater Cincinnati Water Works, they've run numerous
simulations on a "skeletonized" metropolitan distribution system. Current
simulations are already showing officials how to cope with problem situations in
which some sensors within the network malfunction and to improve the fidelity
of their algorithms. Such improvements were presented at the World
Environmental and Water Resources Congress 2007. They have developed better
algorithms for distinguishing different contamination sources which present
similar sensor profiles. Early simulations simulated a few hundreds of sensors;
they have now grown to 11,000 but a city network could have 300,000. Using
data from the Greater Cincinnati Water Works they are showing officials how to
cope with problem situations.
The research team used hundreds of processor simultaneously at NCSA, SDSC
and Argonne. They worked with the Argonne team to build a framework that
figures out how many jobs to send to which site, based on the length of each
system’s queues. SDSC helped them with the grid software, including cross-site
runs using MPICH-G2.
Supported by NSF’s Dynamic Data Driven Applications Systems program.
Figure: Grid-enabled simulationoptimization framework
developed by the North Carolina
State University team, which
allows them to study the efficacy
of the evolutionary computation
algorithms in finding a
contaminant source and its mass
loading history. These images
show the progression of the
evolutionary computation
optimization algorithm in finding
a contaminant source and its mass
loading history. In the final frame,
the true source and its mass
loading history are accurately
captured.
P.R. Mukund, Rochester Institute of Technology
Title: Preserving Ancient Manuscripts
Field: Humanities
TeraGrid storage allocation enables scholars to manage workflows, preserve
ancient manuscripts, and win the SC07 challenge award.
Rochester Institute of Technology used the Indiana University Data Capacitor as
part of their preservation of the Sarvamoola Granthas – 13th century Indian
philosophical writings originally inscribed on palm leaves. Scholars at RIT have
scanned these images, and as part of Indiana University’s Bandwidth Challenge
demonstration transferred digitally enhanced images to the Data Capacitor.
From the Data Capacitor these 700-year-old writings are moved to and stored for
posterity in Indiana University’s archival data storage system. This storage is
made available via an allocation through the NSF-funded TeraGrid. Because
Indiana University’s HPSS installation is geographically distributed, with
duplicate copies kept in Indianapolis and Bloomington by default, these precious
manuscripts will be stored reliably. While this application is perhaps not unique
technologically, it is highly unusual in the sense of an important use of the
TeraGrid by archaeologists – contacts initiated as a result of Indiana University
outreach activities on behalf of the TeraGrid. This activity was performed in
collaboration with PSC and ORNL RPs working on behalf of an allocated
TeraGrid user at the Rochester Institute of Technology.
Figure: 13th-century Indian philosophical writings on palm leaves are digitally
scanned
Aiichiro Nakano, University of Southern California, Yoshio Tanaka, Hiroshi
Takemiya, Satoshi Sekiguchi (AIST, Japan), Shuji Ogata, Takahisa Kouno
(NITECH, Japan), Rajiv K. Kalia, Aiichiro Nakano, Priya Vashishta, USC
Title: Simulation of H-diffusion path in g-alumina
Field: Materials Research
The nudged elastic band (NEB) method is an efficient method for finding the
minimum energy path (MEP) between two given states (i.e., before and after the
reaction). It is often used to evaluate the energy barrier of a reaction process, e.g.,
molecular dissociation and atomic diffusion, through variation of the atomistic
energy among the image-slices. Accurate evaluation of the energy profile
through MEP usually requires an electronic-structure calculation method such as
the density-functional theory (DFT), treating all the involved atoms with such a
compute-intensive method is impractical. We address this challenging and
important problem by combining the NEB method and the hybrid simulation
method that we have developed recently. An appealing aspect of the method is
that parallel computers or a cluster of networked computers can be used
efficiently for finding MEP by assigning each computer for each image-slice. We
have implemented the simulation code based on a hybrid Grid Remote
Procedure Call (GridRPC) + MPI programming that launches MPI processes for
concurrently simulating different time instances, each of which acts as a
GridRPC client which in turn is parallel MD-QM simulation implemented with
hybrid GridRPC + MPI. The hybrid grid application framework enables to
implement flexibility (adaptive resource allocation and task migration), fault
tolerance (automated fault recovery), and efficiency (scalable management of
large computing resources).
The potential of GridRPC has been most recently demonstrated at
Supercomputing’07, where it was used in a 60-hour simulation distributed across
1129 TeraGrid and AIST processors on the Trans-Pacific Grid infrastructure. This
cross-site experiment ran on resources at NCSA, SDSC, Purdue and AIST. The
simulation ran without any need for manual restart, automatically recovering
from system faults. These and other experimental results indicate the strong
potential for
GridRPC
in
implementing
sustainable
grid
applications.
2007 NOAA Hazardous Weather Testbed Spring Experiment
Title: Hazardous Weather Testbed
Field: Climate Research
Better alerts for severe thunderstorms, especially the “supercells” that spawn
tornados could save millions of dollars and countless lives. To that end, TeraGrid
resources made possible an unprecedented forecasting experiment: The 2007
NOAA Hazardous Weather Testbed (HWT) Spring Experiment ran from April
15 to June 8, 2007, and involved more than 60 researchers and forecasters from
government agencies, universities and the private sector.
A major goal of HWT was to test the ability of storm-scale “ensemble”
forecasting, multiple runs of the same model to provide an uncertainty
assessment for the forecast. It was the first time ensemble forecasts were used at
the storm-scale and in real time in a simulated operational environment. The
Center for Analysis and Prediction of Storms at the University of Oklahoma,
Norman, carried out the model forecasts using 66 processors of PSC’s Cray XT3.
Preliminary findings show successful predictions of the overall pattern and
evolution of many of the convective-scale features, sometimes out to the second
day. The ensemble shows good ability in capturing storm-scale uncertainties.
The HWT experiment also used capabilities developed by LEAD (Linked
Environments for Atmospheric Discovery), a TeraGrid Science Gateway, to test
“on-demand” forecasts that were triggered automatically from daily forecasts
indicating regions where storms were likely to develop.
Figure: This plot [c] from the forecast for May 24, 2007, shows probability of
radar reflectivity (proportional to intensity of precipitation) derived from the 10-
member ensemble forecast 21 hours in advance and is related to the “spaghetti”
plot [d] of reflectivity contours for the same time. Courtesy CAPS.
Michael Norman, UC San Diego
Title: Modeling the Early Universe in Unprecedented Detail
Field: Cosmology
Astronomers have a way to see back into the universe’s early history and record
objects that are ever farther away – and therefore older -- as their light travels
billions of years to reach the earth. To help understand these observations, UC
San Diego cosmologist and SDSC Chief Science Officer Michael Norman and
collaborators have conducted the most detailed simulations ever of some 2.5
percent of the universe spanning more than 1.5 billion light years. The results,
published in the Astrophysical Journal, will be used by other researchers
involved in spatial mapping and simulated sky surveys, shedding light on the
underlying physical processes that have produced the structure of the universe.
To keep the model faithful to reality, the researchers need to represent the
extreme variability of matter as it coalesces under gravity, becoming many
orders of magnitude more dense in local areas. The simulations achieved this
unprecedented detail by using the code’s adaptive mesh refinement to zoom in
as matter coalesces under gravity, becoming many orders of magnitude more
dense in local areas. Uniform meshes won’t do, must zoom in on dense regions
to capture the key physical processes- gravitation (including dark matter), shock
heating and radiative cooling of gas. So need an adaptive mesh refinement
scheme (they use 7 levels of mesh refinement). The computation, including an
unprecedented seven levels of subgrids throughout the survey volume, was
achieved thanks to the two large-memory TeraGrid systems, SDSC’s DataStar
and NCSA’s Cobalt.
A collaboration between SDSC’s Harkness and Norman’s helped make major
improvements in the scaling and efficiency of the code (ENZO). Amore, an
NCSA-developed visualization tool, produced high quality visualizations. The
simulations generated some 8 terabytes of data, stored in SDSC’s data archiving
environment and using the Hierarchical Data Format, recently spun-off from
NCSA. Related links: http://cosmos.ucsd.edu/enzo/index.html.
Figure: A simulation of what a radio
telescope will see when surveying a wide
swath of sky 10 degrees square, which
can help telescope builders calibrate their
instruments and understand their
observations. In the most detailed
simulations ever using the ENZO code
on SDSC and NCSA, the effect of all
cosmic structure from two billion years
after the Big Bang to the present is added up, giving the “light cone” a length of
over ten billion light years. Bright white spots are groups and clusters of galaxies;
red background is from filamentary structure. Image: Brian O'Shea, LANL.
Thomas Quinn, University of Washington
Title: Gravitational N-Body Simulations: Planet Formation to Cosmology
Field: Astronomy
This project runs N-body gravitational simulations modeling the formation of the
Oort Cloud, a halo of about a trillion km-sized icy bodies that surrounds the
solar system. It is the most remote part of our solar system, stretching over half
way to the nearest star. For comparison, Neptune, the most distant planet is only
0.01 % of the distance to the nearest star. All of the bodies in the Oort Cloud
were originally distributed between the giant planets of the solar system early in
its history. Through gravitational encounters with the giant planets, however,
these small bodies were steadily scattered to greater and greater distances from
the Sun. As they attained greater and greater distances, their motion started to
be affected by more distant gravitational force, most notably the Milky Way's
tide and the gravity of passing stars. The strength of these external perturbations
determines the exact distribution of bodies throughout the Oort Cloud.
The nice thing about the Oort Cloud is that once it's formed its dynamical
evolution is very slow, so it retains information about the early setting of the
solar system. To constrain the conditions that the solar system first formed in,
we're modeling the formation of the Oort Cloud in several different types of star
formation settings, basically star clusters of different densities. By comparing
our simulation results with real observations of the structure of the Oort Cloud,
we hope to put constraints on the types of environments on the Sun and planets
could've formed in. TeraGrid Condor is a perfect tool for this because our
comets are tiny compared to the giant planets, Sun, and other stars, i.e., their
gravitational fields are dwarfed by the more massive bodies. Furthermore, the
density of comets in the Oort Cloud is extremely low, so they almost never feel
the effects of one another. Because of these two things, we can run 2 different
simulation on 2 different CPUs with the same initial conditions and co-add their
results at the end, and we get the same results as if we ran one larger simulation
on a single CPU. (Supported by NSF planetary science grant, and NSF IGERT.)
Figure: A snapshot from one of the simulations showing the Oort Cloud
surrounding the solar system. The comets are colored by local number density.
The width of the image is ~1 parsec, roughly the distance to the nearest star. The
cloud is stretched in the vertical direction, caused by the tide of the disk of the
Milky Way.
Randall Q. Snurr, David Dubbeldam, Houston Frost, Krista S. Walton,
Northwestern University
Title: Molecular Simulation of Adsorption Sites in Metal-Organic Frameworks
Field: Chemistry
Metal-organic frameworks (MOFs) are an exciting new class of nanoporous
materials with potential applications in energy-related areas such as chemical
separations, gas storage, and catalysis. Understanding their properties at the
molecular level is a key to designing new MOFs for particular applications, and
molecular simulation has therefore become an important tool in MOF research.
Grand canonical Monte Carlo (GCMC) simulations using an atomistic
representation of the MOF and the guest molecules can predict the amount of
molecules adsorbed in the pores for given external conditions in good agreement
with experiment. In addition, this project has shown that predicted positions
and orientations of the adsorption sites match well with x-ray diffraction
experiments from the published literature. At 30 K the molecules are localized
around their crystallographic sites, while at room temperature the molecules are
spread throughout the pore volume, as shown in the figure below. This
molecular-level information provides useful insight into the properties of MOFs
for applications.
This work was supported by the National Science Foundation and the
Department of Energy and used TeraGrid computing resources (TGMCA06N048).
Figure: Adsorption sites for argon in the metal-organic framework IRMOF-1.
Predictions from Monte Carlo simulation are shown as brown patches. Sites
found by x-ray diffraction at 30 K are shown as colored circles. Left: simulated
results at 30 K. Right: simulated results at 298 K.
Reference: “Molecular simulation of adsorption sites of light gases in the metalorganic framework IRMOF-1,” D. Dubbeldam, H. Frost, K.S. Walton, R.Q. Snurr,
Fluid Phase Equilibria, in press.
Mete Sozen, Nicoletta Adamo-Villani, Purdue University (in collaboration with
Turkish engineering group and government)
Title: Design of a Cyber City of the Future
Field: Civil Engineering
Istanbul, populated by 12 million people in brittle concrete and masonry
housing, has been devastated by earthquakes many times in its history, and the
next catastrophe is only a matter of time. Istanbul is at such high risk for a
devastating earthquake that engineers at Purdue University and the Republic of
Turkey have come up with a bold new proposal: build a second city. A second,
satellite city would provide immediate refuge to inhabitants of the old city in the
event of a catastrophic earthquake and soften such an event's effects on the
nation's economy.
Sozen and the team created a 3-D fly-through animation showing what the
proposed new city would look like. The animation of the future Turkish city was
created in two months by using the TeraDRE, a TeraGrid distributed rendering
service. This animation is a 5-minute movie flythrough of the proposed city to be
constructed on the banks of the Black Sea. The city concept was created by
researchers at Purdue University and from Istanbul, Turkey, led by computer
graphics faculty, and produced by a team of researchers and students. This
complex animation presented a significant challenge because of its large frame
size and lighting effects required. Nine virtual cameras were used to provide a
variety of perspectives for the animation. The animation has 9,000 frames each at
720x480 resolution, with more than 30,000 frames rendered including testing and
production, totaling more than 10,000 CPU hours of TeraDRE. Using the
TeraDRE harnessing thousands of computer nodes from the TeraGrid Condor
resource, the researchers were able to render this animation in hours, instead of
weeks and months.
Figure: This animation was produced for The Metropolitan Municipality of
Istanbul, Turkey. It is a 5-minute movie flythrough of a proposed city to be
constructed on the banks of the Black Sea. The city concept was created by
researchers at Purdue University and from Istanbul, Turkey. Technical
information: 720x480 9,000 frame animation, more than 30,000 frames rendered
including testing and production, totaling more than 10,000 CPU hours of
TeraDRE.
Juri Toomre, University of Colorado at Boulder, and Mark Miesch, National
Center for Atmospheric Research
Title: A Stellar Performance
Field: Astrophysics
A new computer model allows scientists to peer deeper into the Sun’s mysteries
to simulate convection patterns in the deep interior of the Sun in unprecedented
detail. The patterns, giant cells, play a critical role in solar variability, and
influence magnetic storms that can reach to Earth. Juri Toomre and Mark Miesch
used TeraGrid to generate simulations of subsurface processes and the Sun’s
unusual rotational pattern. They model the Sun's magnetism and internal
rotation using the Anelastic Spherical Harmonic (ASH) computer code. The
model reveals details about the giant cells that are hidden from current
observational technique.
The team’s simulations indicate that, at low solar latitudes, cooler and denser
plasma sinks along north-south corridors, with corridors moving eastward
relative to hotter plasma that rises. But at higher latitudes, rising and falling
areas of plasma meet and create solar cyclones that last for several days.
Convection near the surface of the Sun occurs as hot plasma rises and cooler,
denser plasma sinks. This also happens deep beneath the surface, where
scientists now suspect that these giant cells, or churning masses of plasma, may
be up to 10 times larger than Earth.
ASH is helping scientists investigate the origins of magnetic activity such as
sunspots and coronal mass ejections. Since the simulations generate vast
amounts of data that cannot be easily transferred, researchers have been
exploring the data remotely using the Visualization and Analysis Platform for
Ocean, Atmosphere, and Solar Researchers (VAPOR), a tool developed under an
NSF grant by NCAR’s Computational and Information Systems Laboratory
(CISL), in partnership with U.C. Davis and Ohio State University. VAPOR allows
researchers to probe three-dimensional volumes in detail, providing rich
visualizations of the complex processes taking place.
Figure: Giant cell convection patterns beneath the surface of the Sun, as revealed
by the new model. Image courtesy of Mark Miesch, NCAR, UCAR.
Igor Tsigelny, SDSC, UC San Diego
Title: Tracking the Causes of Parkinson’s, Alzheimer’s Diseases
Field: Computational biology
A study featured on the cover of the April 2007 Federation of European Biochemical
Societies (FEBS) journal offered – for the first time – a model for the complex
process of aggregation of a protein known as alpha-synuclein, which in turn
leads to harmful ring-like or pore-like structures in human membranes, the kind
of damage found in Parkinson's and Alzheimer's patients. The researchers at
SDSC and UCSD also found that the destructive properties of alpha-synuclein
can be blocked by beta-synuclein – a finding that could lead to treatments for
many debilitating diseases.
Lead author Igor Tsigelny of SDSC and UCSD, said this helped confirm what
researchers suspected. “Using molecular modeling and molecular dynamics
simulations in combination with biochemical and ultrastructural analysis we
showed that alpha-synuclein can lead to the formation of pore-like structures on
membranes.” In contrast, he said, “beta-synuclein appears to block the
propagation of alpha-synucleins into harmful structures.” “This is one of the
first studies to use supercomputers to model how alpha-synuclein complexes
damage cells, and how that could be blocked,” said Eliezer Masliah of UCSD.
The complex calculations were principally performed on the TeraGrid Blue Gene
Data supercomputer at SDSC, as well as another Blue Gene supercomputer at
Argonne National Laboratory. In addition to the NSF supercomputing resources,
the research was supported by the NIH, the DOE, and the SDSC/IBM Institute
for Innovation in Biomedical Simulations and Visualization.
Related Link: http://www.sdsc.edu/News%20Items/PR032207_febs.html
Figure: By simulating how the alpha-synuclein protein forms pore-like structures
such as the pentamer in this image, which lead to neural death, scientists at
SDSC and UCSD gained fresh insights that can help better fight the devastation
of Parkinson’s disease. The complex simulations used SDSC's Blue Gene Data
supercomputer and resources at DOE Argonne National Laboratory. Credit: I.
Tsigelny, Y. Sharikov, M. Miller, and E. Masliah, SDSC/UCSD.
Anne Villamil and Stefan Krasa, University of Illinois at Urbana-Champaign
Title: Secret of Success
Field: Economics
What factors are most responsible for entrepreneurial success? Do personal
characteristics, like optimism or willingness to assume risk, play more of a role
than the institutional environment, including bankruptcy laws and the
availability of credit? University of Illinois economics researchers Anne Villamil
and Stefan Krasa developed a computable model to distinguish among theories
of why people become entrepreneurs. After employing the model at NCSA, they
determined that entrepreneurs need not have personal characteristics that are
significantly different from standard values, but the environment in which they
operate matters greatly to their success.
Their results were recently presented at a Conference on the Resolution of
Financial Distress at the Institute for Advanced Study, the XVI European
Workshop on General Equilibrium, and the SAET International Conference on
Current Trends in Economics.
This research is supported by the National Science Foundation, the Ewing
Marion Kauffman Foundation, and the University’s Academy for
Entrepreneurial Leadership.
Figure: Many entrepreneurs are poorly diversified. Twenty-five percent invest
more than 40 percent of their net worth, and 11 percent invest more than 60
percent. The data show significant differences in firm size. Median firm assets are
about 50 percent of an entrepreneur's net worth. The distribution of firm capital
structures is uniform. Any debt-equity ratio is equally likely.
Gregory Voth, University of Utah; Phil Blood, Pittsburgh Supercomputing
Center
Title: Membrane Remodeling
Field: Bioscience
Modeling of cellular membrane processes is one of the most transformative
challenges of computational bioscience, since it provides atom-level
understanding of important biological processes that can otherwise be observed
only in general features. The research team applied molecular dynamics
simulation to understand how banana-shaped proteins, known as BAR domains,
induce curvature in cellular membranes. They used TeraGrid resources at
multiple sites for what are probably the largest biological simulations ever done,
and for the first time showed how proteins make curves in cell walls.
Known broadly as “remodeling,” protein-induced curvatures in cellular
membranes are involved in many processes, including “endocytosis” — the lifesustaining process by which cells absorb molecules from outside the cell. In work
from 2006, Voth and Blood, showed how BAR domains act as a scaffold, forcing
the membrane to adopt their curvature, and how the orientation of the BAR
domain determines the degree of curvature. In 2007, the researchers used
TeraGrid systems at SDSC, NCSA and UC/Argonne to develop a model of a 50nanometer length of membrane — the longest patch of membrane ever
simulated, totaling 738,000 atoms. They used PSC’s Cray XT3 for the largest-scale
simulations. The researchers refined their simulations to look more deeply and
see more exactly which parts of the BAR domain drive curvature. Contrary to
what had been surmised from experiments, they found that helical structures at
each end of the BAR domain do not by themselves initiate curvature.
“With this kind of problem,” says Voth, “we’re getting at the interplay between
proteins and membranes at a level beyond binding studies and ion-channel
transport, and we’re able to see whole-scale changes in the membrane
morphology driven by these interactions. The next generation of these
simulations will push up to 10-million atoms, which we’ll need to look at
multiple interacting BAR domain proteins and to develop accurate but simpler
coarse-grained models for these systems. In nature, it’s the collective behavior of
many of these proteins interacting with the membrane that creates curvature.
There isn’t a good experimental way to look at this. This is the frontier in this
field. These simulations are indispensable.”
Gregory Voth, University of Utah
Title: Proton transport in the influenza A M2 channel
Field: Chemistry
Voth and his research team in 2007 published a study on the uniqueness of the
influenza A M2 channel and its effect on proton transport in the virus. The M2
channel is a trans-membrane, four-helix channel believed to play a key role in the
viral life cycle by allowing protons to flow though it. This transport of protons
facilitates the viral replication process in a host cell. In terms of basic research,
the M2 channel is of considerable relevance to drug design and virology.
According to Voth, the challenge of understanding proton transport requires
computational simulations benchmarked against experimental results. “I’m a
huge fan of the TeraGrid,” said Voth, who is the second-largest TeraGrid user.
“The TeraGrid cluster systems are phenomenally useful. There isn’t any doubt
that these resources have enabled our research.”
The calculation of proton transport pathways requires a novel computational
methodology combined with extensive simulation over many fast processors to
achieve meaningful statistical convergence. This makes proton transport one of
the most challenging molecular processes to study through computer simulation.
Using TACC’s Dell cluster, Lonestar, and the Big Red HPC system at Indiana
University, the research team solved part of the proton transport mystery. They
also used systems at NCSA to develop a multi-state empirical valence bond
approach (MS-EVB), which allows explicit proton transport to be simulated
using molecular dynamics simulations.Their study explains 1)how the M2
channel operates as a proton conductor in responding to the acidic conditions on
either side of the cell membrane; and 2) how the anti-flu drug amantadine blocks
the channel and causes it to shut down.
Figure: Snapshot of the M2 channel with the inside proton conducting water
wire disrupted by the presence of the anti-flu drug amantadine. The helices of
the M2 channel (blue), the proton-gating His37 residues (mauve), and the
proton-blocking amantadine molecule (orange) are depicted. The lipid bi-layer
membrane is not shown so that the channel can be seen more clearly. Image
courtesy of Hanning Chen.
Paul Woodward, David Porter, University of Minnesota
Title: Simulation of Compressible Turbulent Astrophysical Flows
Field: Advanced Scientific Computing
Paul Woodward and David Porter study turbulence in astrophysical flows. They
used their “PPM” code on the entire XT3 (4,096 XT3 cores plus eight
input/output nodes) to simulate turbulent shear between two fluids on a
computational grid of 576^3 cells, sustaining 9.5 teraflops which is almost half of
peak speed. Their work is an impressive feat of “strong scaling”: getting a large
number of processors to work together efficiently on a moderate-sized problem.
PSC scientists Nathan Stone and Raghurama Reddy developed specialized
software called Portals Direct I/O (PDIO) to route simulation data in real time
from the XT3 to Woodward’s lab at the University of Minnesota where they are
volume-rendered and displayed, allowing Woodward and Porter to interact with
and steer the running XT3 simulation. PSC visualization support using PDIO and
other tools has also enabled significant breakthroughs in converting
oceanographic, arterial blood flow and seismic XT3 simulation data to
“actionable knowledge”. Providing transparent access to distributed file systems
and remote application steering are particularly valuable to TeraGrid
applications.
Carl Wunsch, MIT
Title: Maximizing Knowledge of Remote Oceans
Field: Oceanography
Climate change is one of the major challenges of our times, and the vast size and
heat capacity of the oceans give them a pivotal role in earth’s climate. But the
magnitude and remoteness of the oceans, which cover 70 percent of the earth’s
surface and contain 97 percent of its water, also make them difficult and
expensive to observe.
With support from SDSC’s Scientific Computing Group, climate scientist Carl
Wunsch and colleagues in the Estimating the Circulation and Climate of the
Ocean (ECCO) Consortium are working to better estimate the physical state of
the Southern Ocean – its velocity, temperature, salinity, and other factors.
Running 600 processor jobs on SDSC’s DataStar with their adjoint model, the
scientists create a vast simulation or “virtual ocean,” developing accurate
estimates of the ocean’s state by filling the gaps between relatively scarce ocean
observations.
Along with observational data, to provide valuable new data for climate
scientists and others the ECCO group has been making this simulation data
available from SDSC’s DataCentral, managed by the SRB and offering easy
access through both the SRB client and Web interfaces.
Related Links:
http://puddle.mit.edu/~cwunsch/
http://www.ecco-group.org/posters.html
Caption: To help climate scientists and others, researchers are providing the most
accurate state estimates ever available on the motion, temperature, and other
conditions of the oceans. The map shows the speed of the clockwise Antarctic
Circumpolar current on May 12, 2006, increasing from slow-moving blue water
to dark red indicating speeds above one mile per hour, accurately estimated by
scientists running a powerful model on SDSC’s DataStar system. Credit: M.
Mazloff, MIT/ECCO.
Ming Xue, University of Oklahoma
Title: Ensemble Modeling for Storm Forecasting
Field: Meteorology
Better alerts for thunderstorms, especially supercells that spawn tornados, could
save milions of dollars and many lives.
Unprecedented experiments were conducted every day from April 15- June 8
(tornado season) to test the ability of storm-scale ensemble prediction under real
forecasting conditions for US east of the Rockies. This was the first time for

ensemble forecasting at storm scale (had previously been used for larger
scale models)

real-time in a simulated operational environment
There were successful predictions of the overall pattern and evolution of many of
the convective-scale features, sometimes out to the second day, with a good
ability to capture storm-scale uncertainties.
Ten member ensembles (at 4 km resolution) ran for 6.5 to 9.5 hours each day,
using 66 Cray XT3 processors at PSC. One 600 processor high resolution model (2
km resolution) ran for 9 hours. This entailed 100 times more computing daily
than the most sophisticated National Weather Service operational forecasts to
date. The runs generated and transferred 2.6 TB of data daily to Norman,
Oklahoma
PSC optimized the IO, and modified the reservation and job-processing logic of
its job-scheduling software to implement auto-scheduling of the runs and related
post-processing, (760 jobs/day), demonstrating the ability to use the Cray XT3, a
very large capability resource, on a scheduled, real-time basis.
The experiement used the TeraGrid Gateway LEAD to test on-demand forecasts,
triggering automatically in regions where storms were likely to develop. Those
ran on NCSA Tungsten system at 2km resolution.
Figure: Left- prediction 21 hours ahead of time for May 24, 2007 ; right- observed.
Nicoletta Adamo-Villani, Purdue University
Title: TeraGrid Distributed Rendering used by Computer Graphics Courses
Field: Education and Training
Undergraduate students have access to the TeraGrid in two courses in Computer
Graphics Technology: Motion for Computer Animation, and Advanced Topics in
Computer Animation. Both require students to produce a short film in Maya for
their final project. A number of the students from these courses rendered using
the TeraGrid distributed rendering service TeraDRE. One of the projects is to
create an introductory movie for the game Nano Factor, which is a challenging
new creation toolkit that gives Jr. High-schoolers the opportunity to design and
perform exciting experiments based on real world nano and micro technologies,
promoting inquiry based learning through design, technological literacy, and
attracts young minds to science with fun mini-games and performance rewards.
Students in the Nano Factor project used about 7500 hours of CPU time using the
TeraDRE to render about 30,000 frames at 720x480 resolution.
The Annual Digital Cinema Contest event was hosted by the Purdue Digital
Learning Collaboratory for students to develop and produce video or
animations. Entries are judged on all elements of development and production,
including narrative and technical components and integration of information
resources. One of the judging categories is Animation. TeraDRE was used in
rendering some of the animations for the event in 2007.
Figure: Educational animation about nanotechnology
Figure: Images from student animations in computer graphics class
LEAD-WxChallenge
Title: WxChallenge
Field: Education and Training
In spring 2007, students from colleges and universities across the country faced
off in the WxChallenge hoping to be named the nation’s top weather forecaster.
Competitors were asked to accurately forecast the maximum sustained wind
speeds for select U.S. cities. In an effort to help participating students,
researchers from the Linked Environments for Atmospheric Discovery (LEAD)
Project team at Indiana, Oklahoma, and Millersville universities and the UCAR
Unidata Program Center and NCSA developed the LEAD-WxChallenge Pilot
Project, opening the LEAD gateway to about a dozen university teams.
The LEAD-WxChallenge Pilot Project introduced LEAD technology to up-andcoming scientists outside the LEAD project, allowing them to configure their
own model runs and use the LEAD technology how and where they found it
most valuable. The LEAD gateway provides streamlined entry to LEAD
capabilities, allowing each student to locate a region on the map, establish a
model domain, and launch a forecast workflow on-demand. TeraGrid resources
that support the project include Indiana’s Big Red supercomputer and Data
Capacitor, as well as NCSA’s Tungsten supercomputer on which 160 processors
were reserved 10 hours daily, five days each week, from mid-December to the
end of April. The project team plans to expand the program to reach all
WxChallenge teams in 2008.
Figure: The LEAD Workflow Composer showing event monitoring. The color of
the boxes indicates the status of the computation, with components now
executing shown in green, those completed in dark grey, and those waiting to be
started in gold.