Grid computing using
Open Science Grid (OSG)
Alina Bejan
University of Chicago
Open Science Grid (OSG)
• takes High Throughput Computing to the next
level, to transform data-intensive science
through a cross-domain, self-managed
nationally distributed cyber-infrastructure.
• brings together campuses and communities,
and facilitates the needs of Virtual
Organizations at all scales.
• The OSG Consortium includes
–
–
–
–
universities
national laboratories
scientific collaborations
software developers
working together to meet these goals
What is a grid?
• Grid is a system that:
– coordinates resources that are not
subject to centralized control,
– using standard, open, generalpurpose protocols and interfaces,
– to deliver nontrivial qualities of
service
(based on Ian Foster’s definition in
http://www.gridtoday.com/02/0722/100136.html)
Grids consist of distributed clusters
Grid Site 1:
Fermilab
Grid Client
Grid
Service
Middleware
Application
& User Interface
Grid Client
Middleware
Grid
Protocols
Resource,
Workflow
& Data Catalogs
Grid Site 2:
Sao Paolo
Grid
Service
Middleware
…Grid Site N:
UWisconsin
Grid
Service
Middleware
Grid
Storage
Grid
Storage
Grid
Storage
Compute
Cluster
Compute
Cluster
Compute
Cluster
4
• Do you have a project that takes
too long when running on a
single processor ?
• Do you deal with large amounts
of data from simulations or
experiments ?
Scaling up Science:
Citation Network Analysis in Sociology
1975
1980
1985
1990
1995
2000
2002
Work of James Evans,
University of Chicago,
Department of
Sociology
6
Scaling up the analysis
• Query and analysis of 25+ million
citations
• Work started on desktop
workstations
• Queries grew to month-long
duration
• With data distributed across
U of Chicago TeraPort cluster:
– 50 (faster) CPUs gave 100 X speedup
– Many more methods and hypotheses
can be tested!
• Higher throughput and capacity
7
Mining Seismic data for hazard analysis
(Southern Calif. Earthquake Center).
Seismicity
Paleoseismology
Local site effects
Geologic structure
Faults
Seismic
Hazard
Model
InSAR Image of the
Hector Mine Earthquake
Ґ A satellite
generated
Interferometric
Synthetic Radar
(InSAR) image of
the 1999 Hector
Mine earthquake.
Ґ Shows the
displacement field
in the direction of
radar imaging
Ґ Each fringe (e.g.,
from red to red)
corresponds to a
few centimeters of
displacement.
Stress
transfer
Crustal motion
Crustal deformation
Seismic velocity
structure
Rupture
dynamics
8
Grids work like a CHARMM for
molecular dynamics
• Understanding the
mathematics of
molecular movement
helps researchers
simulate slices of the
atomic world
• But when accurate
nanosecond
simulations pose a
serious challenge,
how can you simulate
full microseconds of
complex molecular
dynamics?
Designing Proteins from
Scratch
• Scientists use
OSG to design
proteins that
adopt specific
3D structures
and more
ambitiously
bind and
regulate target
proteins
important in
cell biology and
pathogenesis
Genetics
• Grid computing
is helping
microbiologists
solve the
mysteries of
mapping new
genomes using
GADU (Genome
Analysis and
Database
Update)
Genome Analysis and Database Update (GADU)
• Runs across TeraGrid and OSG. Uses the Virtual Data System (VDS)
workflow & provenance.
• Pass through public DNA and protein databases for new and
newly updated genomes of different organisms and runs BLAST,
Blocks, Chisel. 1200 users of resulting DB.
• Request: 1000 CPUs for 1-2 weeks. Once a month, every month. On
OSG at the moment >600CPUs and 17,000 jobs a week.
Stormy weather: grid computing
powers fine-scale climate modeling
• Why run
individual models
when you can run
models in
combination?
• When it comes to
climate modeling,
meteorologists
are showing 16
forecasts are
better than one.
Which sciences can benefit ?
•
•
•
•
•
•
•
•
•
particle and nuclear physics
astrophysics
bioinformatics
gravitational-wave science
computer science
mathematics
medical imaging
nanotechnology
potentially any other science …
Grid Resources in the US
OSG
•
Research Participation




Majority from physics : Tevatron, LHC,
STAR, LIGO.
Used by 10 other (small) research groups.
90 members, 30 VOs,
• Research Participation
 Support for Science Gateways
 over 100 scientific data
collections (discipline specific
databases)
Contributors:

5 DOE Labs




TeraGrid

BNL, Fermilab, NERSC, ORNL, SLAC.
65 Universities.
5 partner campus/regional grids.
43,000+ cores
6 Petabytes disk cache
10 Petabytes tape stores
14 internetwork partnership
 Usage




15,000 CPU WallClock days/day
1 Petabyte data distributed/month.
100,000 application jobs/day.
20% cycles through resource sharing,
opportunistic use.
 11 Supercomputing centers
Indiana, LONI, NCAR, NCSA, NICS,
ORNL, PSC, Purdue, SDSC, TACC
and UC/ANL
Accessible resources:




Contributors:
•
Computational resources:
– > 1 Petaflop computing capability
– 30 Petabytes of storage (disk and
tape)
– Dedicated high performance
internet connections (10G)
 750 TFLOPS (161K-cores) in
parallel computing systems
and growing
Open Science Grid
Overview
The Open Science Grid
Consortium brings:
• grid service providers:
– middleware developers
– cluster, network and storage administrators
– local-grid communities
• the grid consumers:
–
–
–
–
global collaborations
single researchers
campus communities
under-served science domains
 into a cooperative infrastructure to share
and sustain a common heterogeneous
distributed facility in the US and beyond.
OSG sites
OSG Snapshot
96 Resources across
production & integration infrastructures
Using production & research networks
Snapshot of Jobs on OSGs
Sustaining through OSG submissions:
3,000-4,000 simultaneous jobs .
~100K jobs/day
~50K CPUhours/day.
Peak test jobs of 15K a day.
30 Virtual Organizations +6 operations
Includes 25% non-physics.
~30,000 CPUs (from 30 to 4000)
~6 PB Tapes
~4 PB Shared Disk
Overlaid by virtual computational environments of
single to large groups of researchers local to
worldwide
To efficiently use a Grid, you must
locate and monitor its resources.
• Check the availability of different
grid sites
• Discover different grid services
• Check the status of “jobs”
• Make better scheduling decisions
with information maintained on the
“health” of sites
Virtual Organization Resource
Selector - VORS
http://vors.grid.iu.edu/
• Custom web interface to a grid scanner
that checks services and resources on:
– Each Compute Element
– Each Storage Element
• Very handy for checking:
– Paths of installed tools on Worker Nodes.
– Location & amount of disk space for
planning a workflow.
– Troubleshooting when an error occurs.
Open Science Grid
VORS entry for OSG_LIGO_PSU
OSG Consortium Mtg March 2007
Quick Start Guide to the OSG
Gratia -- job accounting system
http://gratia-osg.fnal.gov:8880/gratia-reporting/
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
How do you join the OSG?
A software perspective
Joining OSG
• Assumption:
– You have a campus grid
• Question:
– What changes do you need to make
to join OSG?
Your Campus Grid
• assuming that you have a cluster
with a batch system:
– Condor
– Sun Grid Engine
– PBS/Torque
– LSF
Administrative Work
• You need a security contact
– Who will respond to security
concerns
• You need to register your site
• You should have a web page about
your site.
– This will be published
– People can learn about your site.
Big Picture
• Compute Element (CE)
– OSG jobs submitted to CE, which
gives them to batch system
– Also has information services and
lots of support software
• Shared file system
– OSG requires a couple of directories
to be mounted on all worker nodes
• Storage Element (SE)
– How do you manage your storage at
your site
Installing Software
• The OSG Software Stack
– Based on the VDT
• The majority of the software you’ll install
• It is grid independent
– OSG Software Stack:
• VDT + OSG-specific configuration
• Installed via Pacman
What is installed?
• GRAM:
– Allows job submissions
• GridFTP:
– Allows file transfers
• CEMon/GIP:
– Publishes site information
• Some authorization mechanism
– grid-mapfile: file that lists authorized users,
or
– GUMS (grid identity mapping service)
• And a few other things…
OSG Middleware
User Science Codes and Interfaces
Applications
VO Middleware
Biology
Portals,
databases etc
Astrophysics
Data replication etc
HEP
Data and workflow
management etc
OSG Release Cache:
OSG specific configurations, utilities etc.
Infrastructure
Virtual Data Toolkit (VDT)
core technologies + software needed by stakeholders:
many components shared with EGEE
Core grid technology distributions:
Condor, Globus, Myproxy: shared with TeraGrid and others
Existing Operating, Batch systems and Utilities.
Picture of a basic site
Shared file system
• OSG_APP
– For users to store applications
• OSG_DATA
– A place to store data
– Highly recommended, not required
• OSG_GRID
– Software needed on worker nodes
– Not required
– May not exist on non-Linux clusters
• Home directories for users
– Not required, but often very convenient
Storage Element
• Some folks require more sophisticated storage
management
– How do worker nodes access data?
– How do you handle terabytes (petabytes?) of data
• Storage Elements are more complicated
– More planning needed
– Some are complex to install and configure
• Two OSG supported options of SRMs:
– dCache
– Bestman
OSG Education, Training and
Outreach
OpenScienceGrid.org/Education
OpenScienceGrid.org/About/Outreach
eot@OpenScienceGrid.org
OSG EOT Mission
• Organize and deliver training for OSG
– OSG End Users
– Site Administrators
– Support new communities / VOs joining OSG
• Engage young people in (e)Science and CS
– Primary focus: undergraduate and early graduate
students
– Reach high schools through I2U2 (QuarkNet follow-on)
– Promote and train in interdisciplinary collaboration
• Reach out
– To under-represented communities
• Engage and assist minority students and minority serving
institutions by providing resources and opportunities.
– internationally
• Strengthen and assist emerging, underserved regions of
strategic importance to form bonds to US science and Grid
communities
• Focus (for outreach) is on Latin America and Africa
• OISE focus on engagement in Europe and Asia
OSG EOT Program Overview
•
•
•
End User Education
– In-person workshops
– Online training
– EOT VO for student engagement, access and
support
Community Outreach
– International student/faculty exchange via OISE
– Supporting under-represented and under-resourced
communities in US, Latin America and Africa
through workshops, technical assistance and grid
access
– High School Education – I2U2 support http://ed.fnal.gov/uueo/i2u2.html
Site Admin Training
– Training grid administrators in setup and support of
OSG sites using the OSG/VDT software stack
2007-08 Workshop Program
www.opensciencegrid.org/workshops
•
Georgetown University Grid School 2008, April 15-17, DC
•
Tuskegee University Grid School 2008, Feb 6-8 - Tuskegee AL
•
Florida International Grid School 2008, Jan 23-25, at Florida
International University, Miami, Florida
•
Supercomputing ’07 tutorials, Nov 11 & 13, at Reno, Nevada
•
Great Plains Grid School (GPGS’07), Aug 8-10, at the U. of
Nebraska-Lincoln
•
Rio Grande Grid School (RGGS’07), Jun 8-10, at the U. of
Texas at Brownsville, coordinated with UT-Pan American
•
TeraGrid Conference tutorials, Jun 4-8, at the U. of WisconsinMadison
•
South Africa Workshop, Mar 26-30, at the IFIP School on
Software (ISS’07), Gordon's Bay, South Africa
•
Midwest Grid Workshop (MGW’07), Mar 24-25 at the U. of
Illinois at Chicago
•
Argentine Grid Workshop, Mar 12-14 at Santa Fe, Argentina
Grid School Syllabus
•
•
•
•
•
Intro to distributed computing and the Grid
Grid security and basic Grid access
Grid resource and job management
Grid data management
Building, monitoring, maintaining & using
Grids
• Grid applications and frameworks
• Workflow and related issues (scheduling,
provenance)
• Future:
– Porting applications to the Grid
– Web services and the resource framework
– Advanced networking; data mining
Self-paced / online instruction
• opensciencegrid.org/OnlineGridCourse
• Flexible roadmaps for navigating the
material
• Lectures and labs
• Access to online community to provide
support
• Online office hours
I2U2
Interactions In Understanding the Universe
• The Grid for Secondary Science
Education
“educational virtual organization”
• creates an infrastructure to develop
– hands-on laboratory course content and
– an interactive learning experience that
• brings tangible aspects of each experiment into a
“virtual laboratory.”
• These labs use the Grid for education in the
same way that science uses the Grid.
• www.i2u2.org
I2U2
• "e-Labs”
– delivered as Web-based portals accessible in the
classroom and at home
– implemented with of Web-based media capabilities
•
"i-Labs”
– delivered as interactive interfaces typically located
within science museums and similar public venues
– leverage the latest advances in
• display technology and
• human-computer interaction,
– and bring the experiences and appreciation of
scientific investigation and inquiry to the wide
audience of informal education
List of e-Labs
– Cosmic Ray e-Lab
• High school students investigate data from a cosmic ray
detector array. (not necessary to have a detector to
participate.)
• Possible investigations:･Muon Lifetime･Diurnal changes in
flux･Effects of shielding･High-energy showers･Altitude
effects
– CMS Test Beam e-Lab (Beta Version)
• High school students analyze CMS test beam data in an
online graphical ROOT environment.
• Shower Depth･Lateral Shower Size･Beam Purity･Detector
Resolution
– LIGO e-Lab (Beta version)
• High school and middle school students investigate seismic
behavior with data from LIGO ( Laser Interferometer
Gravitational-wave Observatory).
• Earthquake Studies･Frequency Band Studies･Microseismic
Studies･Studies of Human-induced Seismic Activity
– ATLAS e-Lab
– STAR e-Lab
i-Labs
• To engage the general public in science, we
envision using appealing museum exhibits to
attract visitors' attentions and engage them in
a short taste of exploration
• they will use virtual data tools and techniques
to access, process and publish data, report
their results as online posters, have online
discussions about their work with peers, and
then present posters and meet scientists at
museums.
i-Lab Example
• Adler Planetarium
– is developing a cosmic ray i-Lab with
support from QuarkNet and the
Compact Muon Solenoid (CMS)
experiment.
– effort to research an informaleducation model
Cooperation with EGEE
International Schools on Grid Computing
– OSG as co-organizer for ISSGC’07 and
ISSGC’08
– sponsor alumni of US Grid Schools to
attend the International Summer school.
– Joint lectureships and material sharing /
development efforts
– Content sharing
Cooperation with TeraGrid
• Another major national
cyberinfrastructure
• Partnership of 11 organizations
– Mostly supercomputer labs
• Use of TG and OSG resources
• Contribute content
• Joint training
Education VO
• Interested in getting started with OSG ?
• Join OSGEDU VO
– Use OSG resources
– Contribute resources
• Wiki, email lists, follow-up
discussions
– Support, engagement
– Postings of opportunities for students
Students
2004-2008 facts:
• International participation:
– Argentina , Brazil, Canada, Colombia, India, Mexico,
New Zealand, Russia, South Africa, Uruguay
• Women
– Approx. 15%
• Minorities
– Approx 15%
Try to improve these statistics
Participants’ domains
Computer Science
Image processing
Communications
Networking
Physics
Astrophysics
High Energy Nuclear Physics
Optical Networks
Theoretical solid state physics
Atomic Physics
Computational Physics
Chemistry
Computational Chemistry
Molecular Dynamics &
Simulation
Applied Mathematics
Geosciences
Computational Multibody
Dynamics for Distributed
computing
Judicial Administration
Engineering
Materials Science
Quantum theory
…and others …
Acknowledgments
Various OSG members and contributors
(Alain Roy, Mike Wilde, Ruth Pordes,
Gabielle Allen and many others …)
Summary of OSG
• Provides core services, software and a
distributed facility for an increasing set of
research communities.
• Helps VOs access resources on many
different infrastructures.
• Interested in collaborating and contributing
our experience and efforts.
it’s the people…that make the grid a community!
http://www.opensciencegrid.org