The Anton Project
at the
National Resource
for
Biomedical Supercomputing
HPC User Forum
Houston, TX
April 6, 2011
© 2011 Pittsburgh Supercomputing Center
Contributors
• Joel Stiles, Ph.D, M.D. Director, NRBSC
• Markus Dittrich, Senior Scientific Specialist
NRBSC
• Ralph Roskies, Scientific Director, PSC
• Jim Kasdorf, Director of Special Projects,
PSC
© 2011 Pittsburgh Supercomputing Center
Disclaimer
• Nothing in this presentation represents an
official view or position of the U.S. National
Institutes of Health, nor of D.E. Shaw
Research, nor of the Pittsburgh
Supercomputing Center nor of Carnegie
Mellon University. All material is from public
sources. All errors are my own (JK).
© 2011 Pittsburgh Supercomputing Center
The PSC Biomedical Initiative
• Founded in 1987
• Funded by U.S. National Institutes of Health
– First extramural NIH supercomputing program
• Renewed every five years
• With 2006 grant, renamed:
National Resource
for
Biomedical SuperComputing
© 2011 Pittsburgh Supercomputing Center
National Resource for Biomedical
Supercomputing
• Performs leading-edge research in high
performance computing and the life
sciences
• Fosters exchange between PSC expertise
in computational science and biomedical
researchers nationwide
© 2011 Pittsburgh Supercomputing Center
National Resource for Biomedical
Supercomputing
• Computational biomedical research
• Outreach to the national biomedical
research community
–
–
–
–
User training
Software distribution
Publications
Online courses and workshop webcasts.
© 2011 Pittsburgh Supercomputing Center
NRBSC Research Areas
• Microphysiology
• Volumetric visualization and analysis
• Computational structural biology
© 2011 Pittsburgh Supercomputing Center
Anton
• Anton van Leeuwenhoek
• (October 24, 1632 – August 26, 1723)
• Perfected the microscope
• “Father of microbiology”
© 2011 Pittsburgh Supercomputing Center
Anton
• Massively parallel supercomputer
• Designed and built by D.E. Shaw Research
(DESRES)
• Special-purpose system for Molecular
Dynamics (MD) simulations of proteins and
other biological macromolecules
© 2011 Pittsburgh Supercomputing Center
David E. Shaw
•
•
•
•
B.S. / M.S. UCSD
Ph.D. Stanford, 1980
Columbia CS Department until 1986
Formed D.E. Shaw Group in 1988
– “King Quant” (Fortune, Feb. 5,1996)
– Assets Under Management ~$20B
• Net worth: $2.2B (March 2011)
© 2011 Pittsburgh Supercomputing Center
David E. Shaw
• Founded DESRES, 2002
• Senior Research Fellow, Columbia
University, 2005
• PCAST appointments, 1994 and 2009
• Council of Competitiveness Executive
Committee
• Fellow of AAAS
– American Academy of Arts and Sciences
– American Association for the Advancement of
Science
© 2011 Pittsburgh Supercomputing Center
D.E. Shaw Research
(www.deshawresearch.com)
• Independent research laboratory
• Conducts basic scientific research in computational
biochemistry
• Involved primarily in:
• The design of novel algorithms and machine
architectures for high-speed molecular dynamics
(MD) simulations of proteins and other
biological macromolecules
• The application of these simulations to basic
scientific research in structural biology and
biochemistry, and to the process of computer-aided
drug design
© 2011 Pittsburgh Supercomputing Center
The MD Challenge
Molecular dynamics (MD) simulation allows scientists
to study the microscopic dynamics of biomolecular
systems and thus gain crucial insight into processes
such as:
•Protein folding
•How drug molecules interact with proteins and
nucleic acids
•The mode of operation of many important cellular
proteins such as ion channels or other transport
proteins
Importantly, most relevant biological events take place
on a timescale of milliseconds (10-3 s) or longer
© 2011 Pittsburgh Supercomputing Center
Anton
• Anton is a special-purpose supercomputer
designed to specifically address the need of MD
simulations to access the biologically-relevant
realm of millisecond timescales.
• Anton was designed to dramatically increase
the speed of MD simulations.
• Anton runs MD simulations fully in hardware (as
opposed to running MD software on general
purpose hardware).
© 2011 Pittsburgh Supercomputing Center
Anton
• Compared to the previous state-of-the-art
Anton provides a speedup of ~100x,
rendering millisecond timescales attainable
• Anton uses custom-designed ASICs and
novel simulation algorithms
© 2011 Pittsburgh Supercomputing Center
Anton Processing Node
© 2011 Pittsburgh Supercomputing Center
Anton ASIC
© 2011 Pittsburgh Supercomputing Center
A 512-Node Anton Machine
© 2011 Pittsburgh Supercomputing Center
Anton Benchmarks
DHFR benchmark, simulated biological time)
Computational platform
single CPU codes
Parallel supercomputers
Anton (512 node machine)
© 2011 Pittsburgh Supercomputing Center
Speed (ns/day)
~1
~100
17,400
MD Simulations on Anton
• Standard, unbiased MD runs in NVE, NVT, and NPT
ensembles are possible.
• Simulate proteins, lipids, water, RNA, DNA.
• On a 512-processor Anton, simulation systems with
up to 120,000 atoms can be simulated effectively.
• Supports standard MD forcefields (CHARMM,
AMBER, OPLSAA, ...).
• Input is in Maestro format (similar to what is used in
the DESRES Desmond code).
© 2011 Pittsburgh Supercomputing Center
Anton Benchmarks
Chemical system (PDB ID)
System Size (# atoms)
DHFR (5dfr)
23,558
aSFP (1sfp)
48,423
11.7
FtsZ (1fsz)
98,236
5.7
T7Lig (1a01)
116,650
5.5
Speed (μs/machine day)
All simulations used 2.5-femtosecond time steps with long-range interactions
evaluated at every other time step and a Berendsen thermostat applied every 100 time
steps
© 2011 Pittsburgh Supercomputing Center
17.4
Anton @NRBSC/PSC
• The availability of Anton at the National
Resource for Biomedical Supercomputing
(NRBSC) at PSC was made possible by a
partnership with DESRES.
• DESRES made available an Anton machine
without cost for non-commercial research use
by not-for-profit institutions.
• The Anton at NRBSC is the only one available
outside DESRES.
© 2011 Pittsburgh Supercomputing Center
Anton @NRBSC/PSC
• Operations are funded by a two-year $2.7 million
grant from the National Institute of General
Medical Sciences (NIGMS), part of NIH.
• This is the first time NIH has provided funds to
make available a supercomputing system for
national use.
• Grand Opportunities (GO) grant mechanism
from ARRA funds
• One of only 14 GO awards from NIGMS
© 2011 Pittsburgh Supercomputing Center
© 2011 Pittsburgh Supercomputing Center
Anton Hardware
@NRBSC
• 512-node Anton production machine
– Four cabinets, each with 128 special-purpose
ASICs
– Operational since October 2010
• 64-node Anton test machine
– One cabinet with 64 special-purpose ASICs
– Operational since May 2010
Access to both Anton back-ends is via a Linux frontend running a standard PBS queuing system.
© 2011 Pittsburgh Supercomputing Center
Anton Hardware
@NRBSC
• AntonFS storage cluster
– Dedicated to Anton MD trajectory storage
– Lustre-based file system with ~450 TB of storage
• Kollman analysis cluster
– Dedicated to analysis of MD trajectories
generated on Anton
– Four nodes each with 12 cores (2 * 3.33GHz Intel
Xeon Westmere EP X5680 6-core), 96 GB of RAM
and one NVIDIA Tesla C2050 card, QDR IB
© 2011 Pittsburgh Supercomputing Center
Anton Allocations
• In spring of 2010 NRBSC solicited proposals from
the scientific community; 67 proposals were
received.
• Proposal review and assignment of allocations was
conducted by a committee convened by the National
Research Council at the National Academies of
Science.
• 15 proposals were awarded 100,000 node-hours
each during the first phase of the project (October
2010 through March 2011).
© 2011 Pittsburgh Supercomputing Center
Anton Allocations
• 32 proposals were awarded ~50,000 node hours
each during the second phase of the project (April
2011 through September 2011).
© 2011 Pittsburgh Supercomputing Center
Science Projects
on Anton
The range of science topics proposed for simulation on
Anton is quite diverse:
•Studies of membrane channels and transporters
•Protein folding
•Design and evaluation of novel enzyme catalysts
•DNA sequencing
•Protein dynamics
•Signal transduction
•Force field development
•….
© 2011 Pittsburgh Supercomputing Center
Science Projects
on Anton
Currently (as of March 2011), the investigators of
phase 1 are wrapping up their simulations and are
conducting preliminary analyses of their generated
data.
© 2011 Pittsburgh Supercomputing Center
References
• “New Architectures for a New Biology”, Martin
Deneroff, February 22, 2007
www.nersc.gov/projects/SDSA/meetings/SIAM_CS
E07/SIAM_2-07-desres.pdf
• “Anton, a special-purpose machine for molecular
dynamics simulation”, D.E. Shaw, et al.,
Communications of the ACM, Volume 51, Issue 7,
July 2008
• “Herculean Device for Molecular Mysteries”, John
Markoff, The New York Times, July 8, 2008
© 2011 Pittsburgh Supercomputing Center
References
• “Millisecond-scale molecular dynamics
simulations on Anton”, D.E. Shaw, et al.,
SC’09 Proceedings
• “Exploiting 162-Nanosecond End-to-End
Communication Latency on Anton”, Ron O.
Dror, et al., SC10 Proceedings
© 2011 Pittsburgh Supercomputing Center
References
• “Atomic-Level Characterization of the Structural
Dynamics of Proteins”, D.E. Shaw, et al. Science 15
October 2010: Vol. 330 no. 6002 pp. 341-346
• “Custom-Built Supercomputer Brings Protein
Folding Into View”, Robert F. Service, Science 15
October 2010: 308-309.
© 2011 Pittsburgh Supercomputing Center
References
• “Anton Supercomputer Proves Its Mettle”, Chemical
& Engineering News, October 14, 2010
http://pubs.acs.org/cen/news/88/i42/8842notw1.htm
• “Supercomputer sets protein-folding record”,
Nature, October 14, 2010,
http://www.nature.com/news/2010/101014/full/news
.2010.541.html
© 2011 Pittsburgh Supercomputing Center
Jim Kasdorf
kasdorf@psc.edu
www.psc.edu
www.nrbsc.org
© 2011 Pittsburgh Supercomputing Center