2
Abstract
The Grid:
Reality, Technology, and
Applications
The Grid seems to be everywhere, with announcements of
sales from major computer vendors, deployment in a wide
range of application spaces, and many national and
international scale infrastructure deployment. However, in
spite of the popularity of the term, there is often confusion
as to what the Grid is and what problems it solves. Is there
any "there there" or is it all just marketing hype?
In this talk, I will address these questions, describing what
the Grid is, what problems it solves, and what technology
has been developed to build Grid infrastructure and create
Grid applications. I will review the current status of Grid
infrastructure and deployment and give examples of where
Grid technology is being used not only to perform current
tasks better, but to provide fundamentally new types of
capabilities that are not possible otherwise.
Ian Foster
Argonne National Laboratory
University of Chicago
Globus Alliance
Univa Corporation
3
4
Overview
Overview
?
Grid motivation and definition
?
Grid motivation and definition
?
The need for standards
?
The need for standards
?
The power of open source
?
The power of open source
?
Grid in practice
?
Grid in practice
?
Summary
?
Summary
5
Why the Grid?
Origins: Revolution in Science
The Grid
Enable “coordinated resource sharing
& problem solving in dynamic, multiinstitutional virtual organizations.”
?
Pre -Internet
?
(Source: “The Anatomy of the Grid ”)
?
Access to shared resources
? Virtualization, allocation, management
?
?
With predicable behaviors
? Provisioning, quality of service
In dynamic, heterogeneous environments
? Standards - based interfaces and protocols
?
6
Theorize &/or experiment, alone
or in small teams; publish paper
Post-Internet
?
Construct and mine large databases of
observational or simulation data
?
Develop simulations & analyses
?
Access specialized devices remotely
?
Exchange information within
distributed multidisciplinary teams
1
Open Science Grid :
Astronomy Mosaic Construction
Origins in Science
Engage via telepresence in an
experiment at a remote facility
8
Construct custom mosaics on demand from multiple data sources
User specifies projection, coordinates, size, rotation, spatial
sampling
Discover & access a genome analysis
service (running on high-end computer)
Integrate data from multiple sources
in support of global change research
Harness computers across
sites to process data from
a physics experiment
NVO/NASA Montage: A small (1200 node) workflow
Work by Ewa Deelman et al., USC/ISI and Caltech
9
Why the Grid?
New Driver: Revolution in Business
?
?
?
?
Pre -Internet
?
?
?
?
Enterprise computing is highly distributed,
heterogeneous, inter- enterprise (B2B)
?
Business processes increasingly
computing - & data- rich
?
?
Growing complexity & need for
more efficient management
CRM: Workforce
Management (WFM)
A
1
1
2
7
1
2
?
9
Use Globus to discover
& provision resources
9 Servers Each
Strategy B:
Improve
Performance
1
2
1
14
2
IPC
SAP AG R/3 Internet Pricing
& Configurator (IPC)
12
Resource allocation & management
?
Application- driven provisioning
Service -sharing federation and management
Access to programs, hardware, data, services
across departmental boundaries
? Deliver: enhanced performance &/or lower
total cost of ownership
B
1
9
IPC
Delegation of
Dispatcher Request
Response: PricelistServer
Depending on:
- Time
- Discount
- Number of Items
-… 3
?
?
A
1
IPC
Server
2
Common virtualization/provisioning interfaces
1
B
Request:
Price Query
Bridging the Gap Between
Application & Infrastructure
?
B
1
(typically several thousand requests)
Adjust to varying
demand & resources
11 Servers
Strategy A:
Reduce TCO
Web Browsers / Batch Processes
SCM: Advanced Planner
& Optimizer (APO)
11
Cheaper or Faster ...
A
CRM: Internet Pricing Configurator (IPC)
Applications modified to:
?
Outsourcing becomes feasible ?
service providers of various sorts
18 Servers
3 Globus-enabled applns:
?
Central data processing facility
Post-Internet
?
10
eBusiness Use of Grid:
SAP Demonstration @ TechEd
2
?
E.g., large scale data management
?
O n-demand resource allocation
16 Servers Each
2
13
Avoiding Silos & Vendor Lock-In:
Open Standards & Software
Overview
?
Grid motivation and definition
?
The need for standards
?
The power of open source
?
?
?
Grid in practice
Summary
?
Standardized & interoperable mechanisms
for secure & reliable:
?
Authentication, authorization, policy, …
?
Representation & management of state
?
Initiation & management of computation
?
Data access & movement
?
Communication & notification
Good quality open source implementations
to accelerate adoption & development
?
15
?
?
Web services provide
?
Interface definition & service invocation
?
Nice semantics & extensibility
?
First steps towards security, workflow, etc.
?
Considerable commercial adoption
A silver bullet
?
A complete solution
?
Fixed in stone
Web Services
and Stateful Resources
?
?
“State” appears in almost all applications
?
Data in a purchase order
?
Current usage agreement for resources
?
Metrics associated with work load on a server
Web services can model, access and manage
state in many different ways
?
Ad- hoc, per- application approaches
?
WSRF proposes a standard approach
Modeling Stateful Resources with Web Services,
Foster, Frey, Graham, et al. www.globus.org/wsrf , 2004.
16
Applications of the framework
(Compute, network, storage provisioning,
job reservation & submission, data management,
application service QoS, …)
WS-Agreement
(Agreement negotiation)
But are not
?
E.g., Globus Toolkit
Grid Requires that We
Define WS-Based Frameworks
Web Services as a Foundation
14
WS Distributed Management
(Lifecycle, monitoring, …)
WS-Resource Framework & WS-Notification
(Resource identity, lifetime, inspection, subscription, …)
Web services
(WSDL, SOAP, WS-Security, WS-ReliableMessaging, …)
17
18
WSRF & WS-Notification
?
Naming and bindings (basis for virtualization)
?
?
Lifecycle (basis for fault resilient state mgmt)
?
?
?
Every resource can be uniquely referenced, and has one or
more associated services for interacting with it
Resources created by services following factory pattern
Resources destroyed immediately or scheduled
Information model (basis for monitoring, discovery)
?
?
?
Resource properties associated with resources
Operations for querying and setting this info
Asynchronous notification of changes to properties
?
Service groups (basis for registries, collective svcs )
?
Base Fault type
?
Group membership rules & membership management
3
19
Bringing it All Together
WS Distributed Management
?
Scenario: Resource management & scheduling
OASIS Web Services Distributed Management
(WSDM) technical committee
?
?
?
20
Management using Web Services
HP submitted its Web Services Management
Framework (WSMF) to WSDM in July 2003
Notification
Framework that builds on WSRF to provide
(among other things):
?
Service lifecycle (state) models
?
Service monitoring abstractions
?
Service relationships
Grid
Grid “Jobs” and “tasks
” Scheduler
is a
Other
are
kinds
alsoofmodeled
resources
using
are also
“modeled”
WS-Resources
as WS-Resources
andWeb Service
Grid
Service
Scheduler
Resource Properties
Level
Local processor manager
J
J
is “front-ended” with
J
A Web service interface
Blades
R
R
R
WS-Resource used to
A
Service
“model”Level
physical
Agreement
processor resources
is modeled as a
Network
Storage
WS-Resource
WS-Notification
can be
used to “inform ” the Lifetime of SLA
R
scheduler
R
R Properties
when
R R
WS-Resource
Resource
tiedRto
processor
utilization
“project”
processor
status the duration of
changes
(like utilization)
the agreement
B
M
I
I B M
21
22
Overview
?
Grid motivation and definition
?
The need for standards
?
The power of open source
?
Grid in practice
?
Summary
Globus Toolkit Main Components
?
Core Web services
?
Security
?
Execution management
?
Data management
?
Monitoring
?
?
?
?
?
Infrastructure for building new services
Apply uniform policy across distinct systems
Provision, deploy, & manage services
Discover, transfer, & access large data
Discover & monitor dynamic services
Globus Toolkit: Open Source Grid S/W
CAS
OGSA-DAI
[Tech Preview]
WS
Authentication
Authorization
Reliable
File
Transfer
(RFT)
Grid
Resource
Allocation Mgmt
(WS GRAM)
Monitoring
& Discovery
System
(MDS4)
Java WS Core
Pre-WS
Authentication
Authorization
GridFTP
Grid
Resource
Allocation Mgmt
(Pre-WS GRAM)
Monitoring
& Discovery
System
(MDS2)
C Common
Libraries
CLIENT
Non- WS
Your
Your
CC
Client
Client
Your
Your
Java
Java
Client
Client
Your
Your
Python
Python
Client
Client
Interoperable
WS-I-compliant
SOAP messaging
Your
Your
Python
Python
Client
Client
X.509 credentials =
common authentication
Execution
Management
Information
Services
Common
Runtime
SERVER
Java Services in Apache Axis
Plus GT Libraries and Handlers
pyGlobus
WS Core
C WS
Core
Python hosting,
GT Libraries
Pre-W S M D S
Your
C
Service
RLS
Your
Python
Service
Pre-WS GRAM
Your
Your
Java
Java
Service
Service
MyProxy
XIO
Credential
Management
Data
Management
Your
Your
CC
Client
Client
Components
Replica
Location
Service
(RLS)
Security
Your
Your
Java
Java
Client
Client
GridFTP
G
T
4
Web
Services
Components
SimpleCA
G
T
3
GT4 Components
C WS Core
CAS
OGSA-DAI
GTCP
G
T
2
Python WS Core
[contribution]
Index
Trigger
Archiver
G
T
3
Community
Scheduler
Framework
[contribution]
Delegation
Service
GRAM
RFT
Delegation
G
T
4
24
C Services using GT
Libraries and Handlers
4
25
26
?
Control access to shared services across
administrative domains
?
?
Local policy authorities rule
Compute
Center
UK eScience Grid
NEESgrid
TeraGrid
BIRN Biomedical Grid
Earth System Grid
Access Grid
Fusion Grid
Rights ’’
CAS or VOMS
issuing SAML
or X.509 ACs
VO
Users
Rights
Users able to set up dynamic trust domains
?
MPICH-G2
on user’s behalf)
Access
Multi-user collaborations
Federate through mutually trusted services
28
SSL/WS-Security
with Proxy
Services (running
Certificates
Authz Callout
Driven by autonomous management, e.g.,
different policy in different work- groups
?
IBM Grid Toolbox
GT4 Grid Security:
Standards-Based Delegation
Grid Security
?
…
Globus Toolkit
WSDL, SOAP, WS-Security
27
?
EU DataGrid
WS-Addressing, WSRF,
WS-Notification
Butterfly Grid
Custom
GT4
WSRF Web WSRF Web
Services
Services
Virtual Data Toolkit
Custom
Web
Services
Registry
Administration
GT4 Container
User Applications
NSF Middleware Init.
GT-Based Grid Tools & Solutions
Platform Globus
Application Development with GT4
Personal collection of resources working
together based on trust of user
Local Policy
on VO identity
or attribute
authority
MyProxy
VO
Rights ’
29
KCA
30
GT4’s Use of Security Standards
Overview
?
Grid motivation and definition
?
The need for standards
?
The power of open source
?
Grid in practice
?
Summary
5
31
32
A Typical eScience Use of Globus
How it Really Happens
Web
Browser
Compute
Server
Simulation
Tool
Web
Portal
Compute
Server
Registration
Service
Data
Viewer
Tool
Chat
Tool
Credential
Repository
Camera
Telepresence
Monitor
Camera
Database
service
Data
Catalog
Database
service
Database
service
Certificate
authority
Links instruments, data,
computers, people
Users work
with client
applications
Application services
organize VOs & enable
access to other services
Collective services
aggregate &/or
virtualize resources
Resources implement
standard access &
management interfaces
33
34
How it Really Happens
(without Globus)
Web
Browser
Application
Developer
Simulation
Tool
Web
Portal
13
Globus
Toolkit
0
Grid
Community
0
Data
Viewer
Tool
Compute
Server
B
Compute
Server
Chat
Tool
Credential
Repository
Application services
organize VOs & enable
access to other services
Web
Browser
GRAM
Globus
GRAM
Globus Index
Service
CHEF
C
Database
service
D
Database
service
E
Database
service
Application
Developer
2
Off the
Shelf
9
Globus
Toolkit
4
Grid
Community
4
Data
Viewer
Tool
CHEF Chat
Teamlet
MyProxy
Telepresence
Monitor
Compute
Server
Users work
with client
applications
35
Application services
organize VOs & enable
access to other services
Camera
Globus
DAI
Globus
MCS/RLS
Globus
DAI
Globus
Certificate
Authority
Resources implement
standard access &
management interfaces
University of Texas
Compute
Server
Camera
Camera
Data
Catalog
Collective services
aggregate &/or
virtualize resources
Globus
Simulation
Tool
Camera
Telepresence
Monitor
Certificate
authority
Users work
with client
applications
A
Registration
Service
9
Off the
Shelf
How it Really Happens
(with Globus)
DAI
Collective services
aggregate &/or
virtualize resources
Database
service
Database
service
Database
service
Resources implement
standard access &
management interfaces
Global
Community
6
37
38
Early Commercial Applications
Novartis
Grid Services Market Opportunity 2005
Leading adopters (Oct 2003) *
• Financial services: 31%
• Life sciences: 26%
• Manufacturing: 18%
PC Grid links 3,700
desktop systems
Manufacturing
Financial
Services
Energy
Derivatives
Analysis
Seismic
Analysis
Statistical
Analysis
Reservoir
Analysis
Portfolio
Risk
Analysis
?
Mechanical/
Electronic
Design
LS /
Bioinformatics
Process
Simulation
Cancer
Research
Entertainment
Finite
Element
Analysis
Drug
Discovery
Digital
Rendering
Protein
Folding
Failure
Analysis
Other
Massive
Multi-Player
Games
Protein
Sequencing
Streaming
Media
?
Web
Applications
Weather
Analysis
Code
Breaking/
Simulation
?
Academic
?
“Gridified ” Infrastructure
Research & development
applications
Potentially mainstream
business computing
> 5 teraflop/s computing
power
“We have projects we
calculate would take 6
years on a single
supercomputer.
Today, the run time is
12 hours.”
Estimate savings of
$200M over 3 years
Peter Sany,
Novartis CIO
Sources: IDC, 2000 and Bear Stearns- Internet 3.0 - 5/01 Analysis by SAI
*Grids 2004: From Rocket Science To Business Service, The 451 Group
Globus Consortium
(www.globusconsortium.com)
?
39
Acknowledgments
?
“The Globus Consortium is comprised of
global computing leaders who support the
Globus Toolkit for use in enterprise data
centers. Through the Globus Consortium,
vendors of hardware, software and their
customers can work together to accelerate
use of the Globus Toolkit in the enterprise
by hardening features and speaking with
one voice on emerging Grid standards.”
?
?
?
?
?
HP, IBM, Intel, Sun, Nortel, Univa are
founding members
?
Carl Kesselman and Steve Tuecke, my long time Globus co-conspirators
Gregor von Laszewski, Kate Keahey, Jennifer
Schopf, Mike Wilde, other Argonne colleagues
Globus Alliance members at Argonne,
U.Chicago , USC/ISI, Edinburgh, PDC
Miron Livny, U.Wisconsin Condor project, Rick
Stevens, Argonne & U.Chicago
Other partners in Grid technology, application,
& infrastructure projects
DOE, NSF, NASA, IBM for generous support
41
42
Overview
?
Grid motivation and definition
?
The need for standards
?
The power of open source
?
Grid in practice
?
Summary
Summary
?
A significant opportunity
?
Federate resources to increase capability
&/or reduce cost
?
Requires new technology
?
Open standards & software are important
?
We ’re perhaps at an inflection point
?
?
Bridge application- infrastructure gap
To address needs without vendor lock in
?
Major adoption for eScience
?
Growing adoption within industry
7
43
For More Information
?
Globus Alliance
?
?
?
?
www.ggf.org
Background information
?
?
www.globusconsortium.com
Global Grid Forum
?
?
www.globus.org
Globus Consortium
www.mcs.anl.gov/~foster
GlobusWORLD 2005
?
Feb 7-11, Boston
2nd Edition
www.mkp.com/grid2
8