CSE 190
Honors Seminar in High Performance
Computing, Spring 2000
Prof. Sid Karin
skarin@ucsd.edu
x45075
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
•
•
•
•
Definitions
History
SDSC/NPACI
Applications
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Definitions of Supercomputers
• The most powerful machines available.
• Machines that cost about 25M$ in year 2000 $.
• Machines sufficiently powerful to model physical
processes including accurate laws of nature and
realistic geometry, and including large quantities
of observational/experimental data.
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Supercomputer Performance
Metrics
• Benchmarks
• Applications
• Kernels
• Selected Algorithms
• Theoretical Peak Speed
• (Guaranteed not to exceed speed)
• TOP 500 List
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Misleading Performance Specifications in the Supercomputer Field
David H.Bailey
RNR Technical Report RNR-92-005
December 1,1992
http://www.nas.nasa.gov/Pubs/TechReports/RNRreports/dbailey/RN
R-92-005/RNR-92-005.html
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
•
•
•
•
Definitions
History
SDSC/NPACI
Applications
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Applications
•
•
•
•
•
•
•
•
•
•
Cryptography
Nuclear Weapons Design
Weather / Climate
Scientific Simulation
Petroleum Exploration
Aerospace Design
Automotive Design
Pharmaceutical Design
Data Mining
Data Assimilation
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Applications cont’d.
• Processes too complex to instrument
• Automotive crash testing
• Air flow
• Processes too fast to observe
• Molecular interactions
• Processes too small to observe
• Molecular interactions
• Processes too slow to observe
• Astrophysics
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Applications cont’d.
• Performance
• Price
• Performance / Price
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Theory
Numerically
intensive
computing
Dataintensive
computing
(mining)
Experiment
Simulation
Dataintensive
computing
(assimilation)
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Supercomputer Architectures
• Vector
• Parallel Vector, Shared Memory
• Parallel
• Hypercubes
• Meshes
• Clusters
•
•
•
•
SIMD vs. MIMD
Shared vs. Distributed Memory
Cache Coherent Memory vs. Message Passing
Clusters of Shared Memory Parallel Systems
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
The Cray - 1
• A vector computer that worked
• A balanced computing system
• CPU
• Memory
• I/O
• A photogenic computer
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
1976: The
Supercomputing
“Island”
Today:
A Continuum
Performance
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
The Cray X-MP
• Shared Memory
• Parallel Vector
• Followed by Cray Y-MP, C-90, J-90, T90…..
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
The Cray -2
• Parallel Vector Shared Memory
• Very Large Memory (256 MW)
• Actually 256K MW = 262 MW
• One word = 8 Bytes
• Liquid Immersion cooling
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Cray Companies
• Control Data
• Cray Research Inc.
• Cray Computer Company Inc.
• SRC Inc.
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Thinking Machines
• SIMD vs. MIMD
• Evolution from CM-1 to CM-2
• ARPA Involvement
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
1st Teraflops System for US Academia
“Blue Horizon”
•
1 TFLOPs IBM SP
•
•
•
•
•
•
•
•
144 8-processor compute nodes
12 2-processor service nodes
1,176 Power3 processors at 222 MHz
> 640 GB memory (4 GB/node),
10.6GB/s bandwidth, upgrade to > 1 TB
later
6.8 TB switch-attached disk storage
Largest SP with 8-way nodes
High-performance access to HPSS
Trailblazer switch (current ~115MB/s
bandwidth) interconnect with
subsequent upgrade
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
UCSD Currently #10 on Dongarra’s
Top 500 List
• Actual Linpack benchmark
sustained 558 Gflops on 120 nodes
• Projected Linpack benchmark is
650 Gflops on 144 nodes
• Theoretical peak 1.023 Tflops
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
First Tera MTA is at SDSC
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Tera MTA
• Architectural Characteristics
•
•
•
•
Multithreaded architecture
Randomized, flat, shared memory
8 CPUs, 8 GB RAM now going to 16 (later this year)
High bandwidth to memory (word per cycle per CPU)
• Benefits
• Reduced programming effort: single parallel model for
one or many processors
• Good scalability
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Peak speed (flops)
SDSC’s road to terascale computing
10
14
10
13
10
12
10
11
10
World's fastest
supers
SDSC's
vector
supers
10
10
SDSC's
scalable
supers
9
8
10
1980
1985
1990
1995
2000
2005
Year installed
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
ASCI Blue Mountain Site Prep
12,000 sq
ft
120 ft
CSE190
100 ft
SAN DIEGO SUPERCOMPUTER CENTER
ASCI Blue Mountain Site Prep
12,000 sq
ft
120 ft
CSE190
100 ft
SAN DIEGO SUPERCOMPUTER CENTER
ASCI Blue Mountain Facilities
Accomplishments
•
•
•
•
•
•
•
•
12,000 sq. ft. of floor space
1.6 MWatts of power
530 tons of cooling capability
384 cabinets to house the 6144 CPU’s
48 cabinets for metarouters
96 cabinets for disks
9 cabinets for 36 HIPPI switches
about 348 miles of fiber cable
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
ASCI Blue Mountain
SST System Final Configuration
Cray Origin 2000 - 3.072 TeraFLOPS peak
48X128 CPU Origin 2000 (250MHz R10K)
6144 CPUs: 48 X 128 CPU SMPs
1536 GB memory total:
• 32 GB memory per 128 CPU SMP
76 TB Fibre Channel RAID disks
36 x HIPPI-800 switch Cluster Interconnect
To be deployed later this year:
9 x HIPPI-6400 32-way switch Cluster Interconnect
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
ASCI Blue Mountain
Accomplishments
• On-site integration of 48X128 system completed
(including upgrades)
• HiPPI-800 Interconnect completed
• 18GB Fiber Channel Disk completed
• Integrated Visualization (16 IR Pipes)
• Most Site Prep completed
• System integrated into LANL secure computing
environment
• Web based tool for tracking status
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
ASCI Blue Mountain
Accomplishments-cont
• Linpack - achieved 1.608TeraFLOPs
•
•
•
•
accelerated schedule-2 weeks after install
system validation
run on 40x126 configuration
f90/MPI version run of over 6 hours
• sPPM - turbulence modeling code
• validated full system integration
•
•
used all 12 HiPPI boards/SMP and 36 switches
used special “MPI” HiPPI bypass library
• ASCI codes scaling
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Summary
• Installed ASCI Blue Mountain computer
ahead of schedule and achieved Linpack
record two weeks after install. ASCI
application codes are being developed and
used.
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Half
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Rack
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Trench
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Network Design Principles
• Connect any pair of the DSM computers through the crossbar
switches
• Connect directly only computers to switches, optimizing latency
and bandwidth (there are no direct links DSM<==>DSM or
switch<===>switch)
• Support a 3-D toroidal 4x4x3 DSM configuration by establishing
non-blocking simultaneous links across all sets of 6 faces of the
computer grid
• Maintain full interconnect bandwidth for subsets of DSM
computers (48 DSM computers divided into 2, 3, 4, 6, 8,12, 24, or
48 separate, non-interacting groups)
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
18 Separate Networks
18 16x16 Crossbar Switches
1
2
1
3
4
5
2
6
7
8
3
9
10
11
4
12
13
14
15
16
17
5
6 Groups of 8 Computers each
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
6
18
sPPM Hydro on 6144 CPUs
1-HiPPI-800 NIC
Router CPU
Problem Domain:
(4x4x3 DSM layout
48 DSMs/6144 CPUs)
Router CPU on
Neighbor SMP
Problem Subdomain:
8x4x4 process layout
128 CPUs/1 DSM)
12 HiPPI-800 NICs
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
sPPM Scaling on Blue Mountain
Linear
Speedup
%Efficiency
7000
100%
6000
80%
Speedup
5000
4000
60%
3000
40%
2000
20%
1000
0
0
1000
2000
3000
4000
5000
6000
0%
7000
Number of Processors
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
•
•
•
•
Definitions
History
SDSC/NPACI
Applications
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
SDSC
A National Laboratory for Computational
Science and Engineering
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
A Distributed National Laboratory
for Computational Science and
Engineering
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Continuing Evolution
SDSC
Resources
NPACI
Resources
Education
Outreach & Training
Enabling technologies
Applications
Individuals
1985
Technology &
applications
thrusts
Partners
2000
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
NPACI is a Highly Leveraged National
Partnership of Partnerships
46 institutions
20 states
4 countries
5 national labs
Many projects
Vendors and industry
Government agencies
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Mission
Accelerate Scientific Discovery
Through the development
and implementation
of computational
and computer
science techniques
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Vision
Changing How Science is Done
• Collect data from
digital libraries,
laboratories, and
observation
• Analyze the data with
models run on the grid
• Visualize and share
data over the Web
• Publish results in a
digital library
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Goals: Fulfilling the Mission
Embracing the Scientific Community
• Capability Computing
• Provide compute and information resources of exceptional
capability
• Discovery Environments
• Develop and deploy novel, integrated, easy-to-use
computational environments
• Computational Literacy
• Extend the excitement, benefits, and opportunities of
computational science
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
NPACI Objectives
• Deploy teraflops-scale computers
• Create a national metacomputing
infrastructure
• Enable data-intensive computing
• Create persistent intellectual infrastructure
• Conduct outreach to new communities
• Advance computing technology in support
of science
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Partnership Organizing Principle: “Thrusts”
Computational Literacy
EOT
Discovery Environments
Discovery Environments
TECHNOLOGIES
APPLICATIONS
Metasystems
Programming Tools &
Environments
Data-intensive Computing
Interaction Environments
Molecular Science
Neuroscience
Earth Systems Science
Engineering
Capability Computing
RESOURCES
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Technology Thrusts
Metasystems
Programming
Tools and
Environments
DataIntensive
Computing
Interaction
Environments
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Applications Thrusts
Molecular
Science
Neuroscience
Earth Systems
Science
Engineering
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Projects Meld Applications and
Technology
Data-Intensive
Computing
+
Neuroscience
Brain databases
Metasystems and
Parallel Tools
+
Engineering
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
•
•
•
•
Definitions
History
SDSC/NPACI
Applications
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Modeling Galaxy Dynamics and
Evolution
• Project Leaders
• Lars Hernquist, Harvard and
John Dubinski, U Toronto
• Stuart Johnson and Bob Leary,
SDSC SAC Program
• First images from Blue Horizon
Simulation
• 24M particles = 10M stars + 2M
dark matter halo in each galaxy
• Working on 120M particle run
• Run on all 1,152 processors
during acceptance
Animation - Collision between Milky Way and Andromeda
“Due to SAC efforts, our simulations run two to three times faster,
we can ask more precise questions and get better answers” ...Hernquist
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Bioinformatics Infrastructure for
Large-Scale Analyses
• Next-generation tools for
accessing, manipulating,
and analyzing biological
data
• Russ Altman, Stanford University
• Reagan Moore, SDSC
• Analysis of Protein Data
Bank, GenBank and other
databases
• Accelerate key discoveries
for health and medicine
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Protein Folding in a Distributed
Computing Environment
• Simulating protein
movement governing
reactions within cells
•
•
Andrew Grimshaw, U Virginia
Charles Brooks, The Scripps
Research Institute
•
Bernard Pailthorpe, UCSD/SDSC
• Computationally intensive
• Distributed computing
power from Legion
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Telescience for Advanced
Tomography Applications
• Integrates remote
instrumentation, distributed
computing, federated databases,
image archives, and visualization
tools.
• Mark Ellisman, UCSD
• Fran Berman, UCSD
• Carl Kesselman, USC
• 3-D tomographic
reconstruction of
biological specimens
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
MICE: Transparent Supercomputing
• Molecular Interactive
Collaborative
Environment
• Gallery allows
researchers, students
to search for, visualize,
and manipulate
molecular structures
• Integrates key SDSC
technological strengths
•
•
•
Biological databases
Transparent
supercomputing
Web-based Virtual Reality
Modeling Language
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
The
Protein
Data
Bank
• World’s single scientific resource for
depositing and searching protein
structures
• Protein structure data growing
exponentially
• 10,500 structures in PDB today
• 20,000 by the year 2001
1CD3: The PDB’s 10,000th structure.
• Vital to the advancement of biological
sciences
• Working towards a digital continuum
from primary data to final scientific
publication
• Capture of primary data from highenergy synchrotrons (e.g. Stanford
Linear Accelerator Center) requires
50Mbps network bandwidth
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Biological-scale Modeling
• Biodiversity Species Workshop
• Web interface to modeling tools
• Provides geographic, climate,
and other base data
• Species Analyst
• Compiles data from on-line
museum collections
• Scientists can focus on the
scientific questions
• NPACI partners U Kansas,
U New Mexico, and SDSC
Predicted distribution of the mountain trogon.
Data points (pins) are from 14 museum collections.
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Digital Galaxy
• Collaboration with Hayden
Planetarium
• American Museum of
Natural History
• Support from NASA
• Linking SDSC’s mass storage to
Hayden Planetarium requires 155
Mbps
• MPIRE Galaxy Renderer
• Scalable volume visualization
• Linked to database of
astronomical objects
• Produces translucent, filamentlike objects
An artificial nebula, modeled after
a planetary nebula
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
Looking out for San Diego’s
Regional Ecology
• Unique partnership
• 31 federal, state, regional,
and local agencies
• John Helly, et al., SDSC
• Combines technologies
and multi-agency data
• Sensing, analysis, VRML
• Physical, chemical, and
biological data
• Web-based tool for science and public policy
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
AMICO: The Art of Managing Art
• Art Museum Image Consortium
(AMICO)
• 28 art museums working toward
educational use of digital multimedia
• Launch of the AMICO Library
includes more than 50,000
works of art
•
•
•
•
AMICO, CDL, SDSC
XML information mediation
SDSC SRB data management
Links between images, scholarly
research, educational material
CSE190
SAN DIEGO SUPERCOMPUTER CENTER
BNCT & The Treatment Planning Process
Boron Neutron Capture Therapy

MIT Nuclear Engineering Department


Epithermal
Neutrons
(1013/s)


Beam
Characteristics
Beam
Port
Borated
Tumor
MacNCTPlan
Treatment Planning
Software
MCNP Particle
Transport Simulation
(Typically 21x21x25)
Voxel
Energy
Deposition
CT Image
(512x512x250)
Beth Israel Deaconess Medical Center, Boston
MCNP BNCT Simulation Results
Typical MCNP BNCT simulation:
• 1 cm resolution (21x21x25)
• 1 million particles
• 1 hour on 200 MHz PC
ASCI Blue Mountain MCNP simulation:
• 4 mm resolution (64x64x62)
• 100 million particles
• 1/2 hour on 6048 CPUs
ASCI Blue Mountain MCNP simulation:
• 1 mm resolution (256x256x250)
• 100 million particles
• 1-2 hours on 3072 CPUs