TeraGrid Resources
Enabling Scientific Discovery
Through Cyberinfrastructure
(CI)
Diane Baxter, Ph.D.
San Diego Supercomputer Center
University of California, San Diego
The National TeraGrid
Grid Infrastructure
Group (UChicago)
UW
PSC
UC/ANL
NCAR
PU
NCSA
Caltech
IU
UNC/RENCI
ORNL
U Tenn.
USC/ISI
SDSC
LSU
Resource Provider
(RP)
Software Integration Partner
TACC
http://www.teragrid.org/
A complex collaboration of over a dozen organizations
working together to provide cyberinfrastructure
that goes beyond what can be provided by individual
institutions,
to improve research productivity and enable
breakthroughs not otherwise possible.
TeraGrid . . . .
• Deep - provides leadership class resources at
11 partner sites
• Wide - is an integrated, persistent
computational resource for broad user
communities
• Open - is an open scientific discovery
infrastructure
• Is the world's largest, most comprehensive
distributed cyberinfrastructure for open
scientific research.
To be more specific, TeraGrid . . .
•
Uses high-performance network connections
(10-30 Tb/sec)
•
Integrates high-performance computers; resources
for data analysis, visualization, and storage; data
collection tools, high-end experimental facilities;
and supporting expertise around the country;
•
Provides more than a petaflop of computing
capability;
•
Offers more than 30 petabytes of online and archival
data storage, as well as systems to manage data
acquisition and access; and
•
Provides researchers access to over 100 disciplinespecific databases.
What’s in it (TeraGrid) for me?
• Instruments that delivers high-end IT resources computation, storage, visualization, and data/service
– A computational facility – over a PetaFLOP in parallel computing
capability
– A data storage and management facility - over 30 PetaBytes of
storage (disk and tape), over 100 scientific data collections
– A high-bandwidth national data network
• Services: help desk and consulting, Advanced Support
for TeraGrid Applications (ASTA), education and training
events and resources
• Access - without financial cost
– Research accounts allocated via peer review
– Startup and Education accounts automatic
TeraGrid Compute Power
UC/ANL
NCAR
NCSA
PSC
PU
IU
ORNL
Tennessee
LONI/LS
U
SDSC
TACC
Computational Resources
(size approximate - not to scale)
Slide Courtesy Tommy Minyard, TACC
TeraGrid Data Storage and Management
•
Persistent storage on disk and tape
•
Allocatable tape-based, geographically distributed
storage systems for backups of critical data :
»
»
»
»
IU (Indiana University)
NCAR (National Center for Atmospheric Research)
NCSA (National Center for Supercomputing Applications)
SDSC (San Diego Supercomputer Center)
•
Command line usage with GridFTP, using the File
Manager tool in the TeraGrid User Portal
•
GPFS-WAN (General Parallel File System Wide Area
Network). ~ 1 petabyte
•
IU Data Capacitor
•
Long term disk storage allocations
(1 Pb spinning disk for short-term data storage)
TeraGrid Architecture
RP 1
RP 2
POPS
User
Portal
Science
Gateways
TeraGrid Infrastructure
Network, Accounting, …
(Network, Authorization,
Accounting, …)
Command
Line
RP 3
Compute
Service
Viz
Service
Data
Service
(Are your eyes glazing over?)
Translation please!
Enter: Science Gateways
• A Science Gateway
– Enables scientific communities of users
with a common scientific goal and
vocabulary
– Has a common interface
– Leverages community investment
• Three common forms:
– Web-based Portals
– Application programs running on users'
machines but accessing services in
TeraGrid
– Coordinated access points enabling
users to move seamlessly between
TeraGrid and other grids.
12
Today, there are approximately 29 gateways
using the TeraGrid
How do Gateways help?
• Make science more productive
–
–
–
–
Researchers use same tools
Complex workflows
Common data formats
Data sharing
• Bring TeraGrid capabilities to
the broad science community
–
–
–
–
Lots of disk space
Lots of compute resources
Powerful analysis capabilities
A community-friendly interface to
information and research tools
But it’s not just ease of use.
What can scientists do that they couldn’t do
previously?
•
•
•
•
•
•
LEAD - access to radar data
NVO – access to sky surveys
OOI – access to sensor data
PolarGrid – access to polar ice sheet data
SIDGrid – analysis tools for social scientists
GridChem – developing multiscale coupling
How would this have been done before gateways?
Gateways can enhance and support
investments in other projects
• Increase access
– To instruments
• Increase capabilities
– To data analysis tools
• Improve workforce development
– For underserved populations, through broad access to
learning resources
• Increase outreach
• Increase public awareness
– Public sees value in investments in large facilities
• Slice bread
Gateways Greatly Expand Access
• Almost anyone can investigate scientific questions using
high end resources
– Not restricted to those in research groups with allocations
– Gateways allow anyone with a web browser to explore
• Fosters new ideas, cross-disciplinary approaches
• Encourages students to experiment
• But Gateways are used in production too
– Significant number of papers resulting from gateways including
GridChem, nanoHUB
– Scientists can focus on challenging science problems rather than
challenging infrastructure problems
How do we develop a new gateway?
Advanced support for Gateway Development
• Same peer review process used to request
resources
– 30,000 CPUs
– + 6 months of help from a TG Gateway Team member
– Reviews based on appropriate use of resources, science
is not reviewed if already funded
•Petascale
•Multisite workflows
•Gateways
•Domain expertise
Support is Very Targeted
• Start with well-defined objectives
– Focus on efficient or novel use of national CI resources
• Minimum .25 FTE for months to a year
– Enough investment to really understand and help solve
complex problems
• Must have commitment from PIs
– Want to make sure work is incorporated into production
codes and gateways
• Good candidates for targeted support include:
– Large, high impact projects
– Ability to influence new communities
– Suggestions from NSF directorates on important projects
• Lessons learned move into training and documentation
When is a gateway be most appropriate?
• Researchers using defined sets of tools in different ways
– Same executables, different input
•GridChem, CHARMM
– Creating multi-scale or complex workflows
– Shared datasets
• Common data formats
– National Virtual Observatory
– Earth System Grid
– Some groups have invested significant efforts already, e.g.:
•caBIG, extensive discussions to develop common terminology and formats
•BIRN, extensive data sharing agreements
• Difficult to access data/advanced workflows
– Sensor/radar input
•LEAD, GEON
Things you can do with the TeraGrid:
Simulate cell membrane processes
Work by Emad Tajkhorshid and James Gumbart,
of University of Illinois Urbana-Champaign.
– Mechanics of Force Propagation in TonBDependent Outer Membrane Transport.
Biophysical Journal 93:496-504 (2007).
– Results of the simulation may be seen at
www.life.uiuc.edu/emad/TonB-BtuB/btub2.5Ans.mpg
• Modeled mechanisms for transport of
molecules through cell membrane.
• Used 400,000 CPU hours [45 processor-years]
on systems at National Center for
Supercomputing Applications, IU, Pittsburgh
Supercomputing Center
Image courtesy of Emad Tajkhorshid, UIUC
Predict storms
• Hurricanes and tornadoes cause massive
loss of life and damage to property
• TeraGrid supported spring 2007 NOAA
and University of Oklahoma Hazardous
Weather Testbed
– Major Goal: assess how well ensemble
forecasting predicts thunderstorms,
including supercells  tornadoes.
– Delivers “better than real time”
prediction
– Used 675,000 CPU hours for the
season
– Used 312 TB on HPSS storage at PSC
Slide courtesy of Dennis Gannon, IU, and LEAD Collaboration
Watch Polar Ice Caps Melt (PolarGrid)
• Cyberinfrastructure Center
for Polar Science (CICPS)
– Experts in polar science,
remote sensing and
cyberinfrastructure
– Indiana, ECSU, CReSIS
• Satellite observations show
disintegration of ice shelves
in West Antarctica and
speed-up of several glaciers
in southern Greenland
– Most existing ice sheet
models, including those used
by IPCC cannot explain the
rapid changes
http://www.polargrid.org/p
olargrid/images/4/42/C005
0-polargrid-big.m4v
Source: Geoffrey Fox
CY2007 Usage by Discipline
Advanced Scientific
Atmospheric
Computing
Sciences
2%
3%
All 19 Others
4%
Earth Sciences
3%
Chemical, Thermal
Systems
5%
Materials Research
6%
3.95B SUs delivered in CY2007
Molecular
Biosciences
31%
Astronomical
Sciences
12%
Physics
17%
24
Chemistry
17%
Do you want to see more Gateway examples?
• Yes
• No
Recent Gateways using TeraGrid
Significantly
• SCEC
• SIDGrid
• CIG
SCEC using gateway to produce hazard map
• PSHA hazard map for
California using newly
released Earthquake
Rupture Forecast
(UCERF2.0) calculated
using SCEC Science
Gateway
• Warm colors indicate
regions with a high
probability of experiencing
strong ground motion in the
next 50 years.
• High resolution map,
significant CPU use
LEAD (portal.leadproject.org/)
• Simple enough an undergraduate can use it! http://wxchallenge.com/
• National Center for Supercomputing Applications (NCSA) and IU teamed up to
support WxChallenge weather forecast competition. 64 teams, 1000 students,
~16,000 CPU hours on Big Red
• XBaya is available from http://www.collab-ogce.org/
NanoHub Harnesses TeraGrid for Education
Nanotechnology education
• Used in dozens of courses
at many universities
• Teaching materials
• Collaboration space
• Research seminars
• Modeling tools
• Access to cutting edge
research software
Social Informatics Data Grid
• Heavy use of “multimodal”
data.
– Subject might be viewing a
video, while a researcher
collects heart rate and eye
movement data.
• Events must be
synchronized for analysis,
large datasets result
• Extensive analysis
capabilities are not
something that each
researcher should have to
create for themselves.
http://www.ci.uchica
go.edu/research/files
/sidgrid.mov
• Social scientists have traditionally worked in isolated labs without the
capability to share data or insights with others.
• SIDGrid enables a number of capabilities.
– Data that is expensive to collect can now be shared with others, increasing
the potential for scientific impact.
– Geographically distant researchers can collaborate on the analysis of the
same data set.
– Complex analysis tools and workflows are now available for all to use, rather
than having each lab duplicate efforts.
– All researchers now have access to the highest quality computational
resources
•SIDGrid uses TeraGrid resources for computationally-intensive tasks such as media
transcoding algorithms for pitch analysis of audio tracks and fMRI image analysis
• SIDGrid is unique among social science data archive projects
– Focused on streaming data which change over time
– Provides the ability to investigate multiple datasets, collected at different
time scales, simultaneously
• Active users of the SIDGrid system include a human neuroscience group
and linguistic research groups from the University of Chicago and the
University of Nottingham, UK
SIDGrid
sidgrid.ci.uchicago.edu
TeraGrid Pathways Activities
• 2 Gateway components
– Adapt gateways for educational use by underrepresented
communities
•GEON – SDSC, Navajo Tech
– Teach participants from underrepresented communities
how to build gateways
•PolarGrid – IU, ECSU
Navajo Technical College and gateways
•Incorporating the use of
gateways in their curricula
•GEON, GISolve areas of initial
interest
Menu
TG Resources and Services
•Computing – over a petaflop of computing
power and growing
•Data
– Data storage facilities & management tools
– Scientific data collections
•Over 30 Science Gateways
•Remote visualization servers and software
•Technical Support
– Central point of contact for support of all systems
– Advanced Support for TeraGrid Applications (ASTA)
•Education and training events and
resources
– K-12 Education
– Pathways
– Campus Champions
35
Human Connection: Your Campus Champion
• The Campus Champions program supports campus representatives as
the local source of knowledge about high-performance computing
opportunities and resources.
Knowledge plus assistance will empower campus researchers,
educators, and students to advance scientific discovery.
• Your campus will benefit by having direct access to the TeraGrid and
input to its staff, resource allocations awarded for their use, and
assistance in using those resources.
• TeraGrid will support the Campus Champion. See
– http://www.teragrid.org/eot/campuschamps.html
– To join the Campus Champions program, contact the TeraGrid Campus
Champions Program Coordinator, at tgcc-help@teragrid.org.
Online Resources
• Online resources at www.teragrid.org
• TeraGrid User Portal for managing allocations and job flow
• Documentation
– Knowledge Base for quick answers to FAQ’s
– HPC University to increase general HPC knowledge
• Calendar of events including upcoming workshops and
training
– Annual conference - TG09
• Arlington, VA
• June 22-26, 2009
TeraGrid: greater than the sum of its parts
• Leadership in cyberinfrastructure development, deployment
and support
• Expertise in building national computing and data resources
• Leveraging extensive resources, expertise, R&D, and EOT
– Leveraging activities at other participant sites
– Learning from each other improves expertise of all TG staff
– Shared training, education, and outreach resources benefit all
• Simplified access to high end resources
–
–
–
–
Single unified allocations process
Single point of contact for problem reporting
Coordinated software environments
Uniform access to heterogeneous resources to solve a single
scientific problem
Would you like to learn more about
getting a TeraGrid allocation ?
Yes
Not today
How does the Allocations process work?
• Startup allocations: for code development, experimentation with
TeraGrid platforms, and application testing. Startup requests may total
up to 200,000 service units (SUs) of computation, up to 5TB on disk
and 25TB on tape of storage.
• Education allocations: for use in classroom instruction or training
activities,
with the same SU and storage limits as Startup allocations.
• Research allocations: requires a detailed justification of resource
usage. Requests are reviewed four times a year by the Resource
Allocations Committee.
– National peer-review process
•allocates computational and data resources
•makes recommendations on allocation of advanced direct support services
•Currently awarding >1B Normalized Units of resources
– Principal investigator (PI) must be a researcher, educator, or postdoctoral
researcher at a US academic or non-profit research institution.
Go to the POPS page https://pops-submit.teragrid.org

Create a POPS Login
Indicate that you are “New” to the Teragrid

Indicate this is a “Start-up” Request

Select Startup or Educational

Fill out PI information
Skip Co-PIs info


Fill out info on your project

Fill out info on your funding
Estimate your computing need (reasonably)





Upload your CV and Submit!

when ready
Acknowledgements
•
This work is made possible by the dedicated efforts of the TeraGrid staff. In particular, slides came from Scott
Lathrop, Craig Stewart, John Towns, Dane Skow, Daphne Siefert-Herron, Vickie Lynch, David Hart (Indiana Dave);
David Hart (California Dave), Fran Berman, Nancy Wilkins-Diehr, Laura McGinnis and probably others.
•
The Grid Infrastructure Group management of the TeraGrid is funded by NSF grant 0503697.
•
The LEAD portal is developed under the leadership of IU Professors Dr. Dennis Gannon and Dr. Beth Plale, and
supported by NSF grant 331480. Marcus Christie and Surresh Marru of the Extreme! Computing Lab contributed the
LEAD graphics
•
The ChemBioGrid Portal is developed under the leadership of IU Professor Dr. Geoffrey C. Fox and Dr. Marlon Pierce
and funded via the Pervasive Technology Labs (supported by the Lilly Endowment, Inc.) and the National Institutes of
Health grant P20 HG003894-01.
•
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author and do
not necessarily reflect the views of the National Science Foundation (NSF), National Institutes of Health (NIH), Lilly
Endowment, Inc., or any other funding agency.
Thank you!
• Questions?