4/30/2014
1
University of California, San Diego
• Prism@UCSD – Science DMZ
– PI: P. Papadopulos, co-PI: L. Smarr
– 01/01/2013 to 12/31/2014
• CHERuB – 100G campus gateway
– PI: M. Norman, co-PI: T. Hutton, V. Polichar
– 01/01/2014 to 12/31/2015
UCSD and its environment
General Atomics
Stem Cell Institute
Salk Institute
SDSC
CalIT2
Physics
NCMIR
Skaggs
Medical School
Scripps Institute
of Oceanography
4/30/2014
In addition to the 3 main
UCSD units:
- General Campus
- Medical School
- Scripps I.O.
there are many other
research organizations
on and around campus.
Venter Institute
3
Connecting YOU on UCSD Campus with the World
By Creating a Big Data Freeway System
CHERuB
NSF CC-NIE Has Awarded Prism@UCSD Optical Switch
Phil Papadopoulos, SDSC, Calit2, PI
Prism@UCSD: A Researcher Defined 10 and
40Gbit/s Campus Scale Data Carrier
Project in Brief
• high-bandwidth end-to-end optical connections
• routed by next generation Arista switches (7504)
• connects lab “data producers” with SDSC data-intensive computing &
storage resources
• 10 Terabit/s of aggregate bandwidth, has full bisection similar to inmachine room clusters, but is deployed at a campus scale
• builds upon and upgrades the Quartzite "campus-scale network
laboratory" NSF MRI (awarded 2006)
• adds IPv6 and OpenFlow
• existing optical fiber connection to the SDSC is being expanded to
120Gbps as a high-bandwidth bridge to cloud/parallel storage and NSF
XSEDE resources
PRISM Puts SDSC’s Big Data Gordon Supercomputer
and Data Oasis Storage Into Your Lab
12
PRISM is Connecting CERN’s CMS Experiment
To Our Physics Department
80 Gbps PRISM Connection Has Been Made
UCSD is a Tier-2 LHC Data Center:
CMS Flow into UCSD Physics Dept. Peaks at 2.4 Gbps
Source: Frank Wuerthwein, Physics UCSD
Planning for climate change in California
substantial shifts on top of already high climate variability
SIO Campus Climate Researchers Need to Download
Results from Remote Supercomputer Simulations
to Make Regional Climate Change Forecasts
Dan Cayan
USGS Water Resources Discipline
Scripps Institution of Oceanography, UC San Diego
much support from Mary Tyree, Mike Dettinger, Guido Franco
and other colleagues
Sponsors:
California Energy Commission
NOAA RISA program
California DWR, DOE, NSF
average
average summer
summer
afternoon
afternoon temperature
temperature
GFDL A2 1km downscaled to 1km
Hugo Hidalgo Tapash Das Mike Dettinger
10
Ultra High Resolution Microscopy Images
Created at the National Center for Microscopy Imaging
NIH National Center for Microscopy & Imaging Research
Integrated Infrastructure of Shared Resources
Shared Infrastructure
Scientific
Instruments
Local SOM
Infrastructure
End User
FIONA Workstation
Source: Steve Peltier, Mark Ellisman, NCMIR
PRISM Links Calit2’s VROOM to NCMIR to Explore
Confocal Light Microscope Images of Rat Brains
Protein Data Bank (PDB) Needs
Bandwidth to Connect Resources and Users
• Archive of experimentally
determined 3D structures of
proteins, nucleic acids, complex
assemblies
• One of the largest scientific
resources in life sciences
Virus
Hemoglobin
Source: Phil Bourne and
Andreas Prlić, PDB
PDB Usage Is Growing Over Time
•
•
•
•
More than 300,000 Unique Visitors per Month
Up to 300 Concurrent Users
~10 Structures are Downloaded per Second 7/24/365
Increasingly Popular Web Services Traffic
Source: Phil Bourne and Andreas Prlić, PDB
2010 FTP Traffic
RCSB PDB
PDBe
PDBj
159 million
entry downloads
34 million
entry downloads
16 million
entry downloads
Source: Phil Bourne and Andreas Prlić, PDB
PDB Plans to Establish Global Load Balancing
• Why is it Important?
– Enables PDB to Better Serve Its Users by Providing
Increased Reliability and Quicker Results
• How Will it be Done?
– By More Evenly Allocating PDB Resources at Rutgers and
UCSD
– By Directing Users to the Closest Site
• Need High Bandwidth Between Rutgers & UCSD Facilities
Source: Phil Bourne and Andreas Prlić, PDB
PRISM Will Link Computational Mass Spectrometry
and Genome Sequencing Cores to the Big Data Freeway
Source: proteomics.ucsd.edu
ProteoSAFe: Compute-intensive
discovery MS at the click of a
button
MassIVE: repository and
identification platform for all
MS data in the world
http://cherub.ucsd.edu
SAN DIEGO SUPERCOMPUTER CENTER
at the UNIVERSITY OF CALIFORNIA; SAN DIEGO
CHERuB*: SDSC-ACT partner to bring
100Gbps connectivity to UCSD
Production late 2014
UWisc Madison
- OSG
FNAL Tier-1 LHC
UNL - OSG
LBL - CMMAP
NERSC POLARBEAR, CAIDA
NICS CMMAP
UCSB
UCR
UCSD/SDSC
New 100G path
Austin/TACC
Pink line – New CENIC 100G
Blue lines – Existing/planned ANI 100G
Green lines – Existing PacWave 100G
Maroon lines – XSEDE 10G network
Thin lines – Other existing 10G or lower
SAN DIEGO SUPERCOMPUTER CENTER
the UNIVERSITY OF CALIFORNIA; SAN DIEGO
*Configurable, High-speed,atExtensible
Research Bandwidth
The Plumbing (ask Tom Hutton)
818 W. 7th, Los Angeles, CA
10100 Hopkins Drive, La Jolla, CA
SDSC NAP
Equinix/L3/CENIC POP
DWDM
100G
transponders
existing
CENIC fiber
up to 3 add'l 100G
transponders can be
attached
DWDM
100G
transponders
Nx10G
up to 3 add'l 100G
transponders can be
attached
100G
Existing ESnet
SD router
UCSD/SDSC
Gateway Juniper
MX960 "MX0"
New 2x100G/8x10G
line card + optics
New 40G
line card +
optics
SDSC Juniper
MX960 "Medusa"
PacWave,
CENIC,
Internet2, NLR,
ESnet,
StarLight,
XSEDE & other
R&E networks
100G
Dual Arista 7508
"Oasis"
256x10G
New 100G card/
optics
2x40G
UCSD
DYNES
4x10G
add'l 10G card/optics
Other
SDSC
resources
mult. 40G
connections
UCSD Primary Node
Cisco 6509 "Node B"
DataOasis/
128x10G
Pink/black SDSC Cloud
existing UCSD
infrastructure
GORDON
SAN DIEGO SUPERCOMPUTER CENTER compute
Green/dashed lines cluster
SDSC
new component/
DYNES
equipment in proposal
mult. 10G
connections
UCSD
Production users
PRISM@UCSD
Arista 7504
mult. 40G+
connections
Key:
NEW
10G
UCSD/SDSC
Cisco 6509
100G
to CENIC/
PacWave
switch L2
UCSD
10G
PRISM@UCSD
- many UCSD big
data users
at the UNIVERSITY OF CALIFORNIA; SAN DIEGO
CENIC/ESnet 100G Connection enables
Big Data science collaborations between
NERSC and SDSC
UWisc Madison
- OSG
FNAL Tier-1 LHC
UNL - OSG
LBL - CMMAP
NERSC POLARBEAR, CAIDA
NICS CMMAP
UCSB
UCR
UCSD/SDSC
New 100G path
Austin/TACC
Pink line – New CENIC 100G
Blue lines – Existing/planned ANI 100G
Green lines – Existing PacWave 100G
Maroon lines – XSEDE 10G network
Thin lines – Other existing 10G or lower
SAN DIEGO SUPERCOMPUTER CENTER
at the UNIVERSITY OF CALIFORNIA; SAN DIEGO
A Unique, Powerful, Data-Intensive
Testbed for Scientific Discovery
EDISON HPC SYSTEM
2 PF, 434 TB RAM
GORDON HPD SYSTEM
0.3 PF, 364 TB RAM+SSD
150 GB/s
100 GB/s
ESnet/CENIC
6 PB
DTN
DTN
4.5 PB
100 Gb/s
SAN DIEGO SUPERCOMPUTER CENTER
at the UNIVERSITY OF CALIFORNIA; SAN DIEGO
POLARBEAR Cosmology Telescope
UC Berkeley/NERSC-UCSD/SDSC
• Goal: Measure B-mode
polarization in the CMB from
inflation era
• Data path: Chile (obs)UCB/NERSC (analysis)UCSD/SDSC (analysis)
• Data acquisition rates:
• 22 GB/mo. (current)
• 3 TB/mo. (2014-2016)
• Map making data analysis
NERSC & SDSC
• 100 MC realizations of 100 TB
data = 10 PB
Atacama Desert, Chile
SAN DIEGO SUPERCOMPUTER CENTER
at the UNIVERSITY OF CALIFORNIA; SAN DIEGO
Next Generation Network Measurement
CAIDA (SDSC)-NERSC
• CAIDA operates the UCSD
Network Telescope, which
collects Internet Background
Radiation
• Data paths: global internet,
ESnet
• Data rates: 3-4 TB/mo
• Using NERSC tape archive to
replicate 100 TB historical data
• Other projects: network
measurement tools, Future
Internet Architecture
unassigned
IPv4
addresses
100’s TB archival data
SDSC/NERSC
SAN DIEGO SUPERCOMPUTER CENTER
at the UNIVERSITY OF CALIFORNIA; SAN DIEGO
High Energy Physics LHC/US-CMS
UCSD Tier-2—US-CMS collaboration
• Goals: Higgs boson,
supersymmetry, BSM
• Data Paths: CERNFNAL (Tier 1)-UCSD
(Tier 2) via ESnet and
CENIC/I2
• Peak Bandwidths:
• Current: 10+5 Gbps
• 2015: 40 Gbps when
LHC operates @ 14 Tev
SAN DIEGO SUPERCOMPUTER CENTER
at the UNIVERSITY OF CALIFORNIA; SAN DIEGO
Education & Training:
UCSD Telemedicine Center
SAN DIEGO SUPERCOMPUTER CENTER
at the UNIVERSITY OF CALIFORNIA; SAN DIEGO
CHERuB Implementation Status
• January, 2014
• Project funded, equipment on order
• February, 2014
• Equipment received
• Production network switch upgraded
• March, 2014
• Campus gateway upgraded, connected to regional 100G feed
• Successful border-to-regional test @100Gbps
• Next steps (April/May):
• Connect Prism switch, test @2x40Gbps
• Connect SDSC infrastructure, test @100Gbps
• Connect production switch, test @4x10Gbps
• Production Goal: September 2014
SAN DIEGO SUPERCOMPUTER CENTER
at the UNIVERSITY OF CALIFORNIA; SAN DIEGO
Comet is a ~2000TeraFLOP System Architected
for the “Long Tail of Science”
NSF Track 2 award to SDSC
$12M NSF award to acquire
$3M/yr x 4 yrs to operate
Production early 2015