Open Grid Services
Architecture:
An Introduction
Ian Foster
Mathematics and Computer Science Division
Argonne National Laboratory
and
Department of Computer Science
The University of Chicago
http://www.mcs.anl.gov/~foster
OGSA Early Adopters Workshop, Argonne, May 29, 2002
2
Overview
z
Meeting goals and program
z
Grids and the Globus Toolkit
z
Open Grid Services Architecture (OGSA)
z
‹
Motivation & concepts
‹
Core interfaces and examples
OGSA and the Globus Toolkit
‹
Plans and implementation status
3
Overview
z
Meeting goals and program
z
Grids and the Globus Toolkit
z
Open Grid Services Architecture (OGSA)
z
‹
Motivation & concepts
‹
Core interfaces and examples
OGSA and the Globus Toolkit
‹
Plans and implementation status
4
Meeting Goals
z
Communicate to potential early adopters
information about the structure and use of
Globus Toolkit “OGSA Technology Preview”
‹
‹
z
Reference implementation of current GS
Specification => will evolve!
Obtain feedback on technology & directions
Discuss activities underway &/or required
to expand OGSA activities to encompass
higher-level services
‹
Make connections & build teams
5
Meeting Structure
z
z
Day 1: Tutorial
‹
OGSA introduction
‹
Web services introduction
‹
Grid service specification
‹
OGSA Technology Preview Preview
Days 2 and 3: Twin tracks
‹
Programming with Grid services
‹
OGSA service design
Day 1: Tutorial
Auditorium
9:00-10:30:
Introduction to OGSA, Ian Foster
10:30-11:00: Break
11:00-12:30: Web Services, Steve Graham
12:30-2:00:
Lunch (ANL Guest House)
2:00-3:00:
Grid Services, Steve Tuecke
3:30- 4:00:
Break
4:00-5:30:
Implementation Overview
Overview of OGSI Tech Preview Release
& Demo, Thomas Sandholm
OGSI J2EE Design, Scott Sylvester
Days 2 and 3: Write a Grid Service
Room F108
Day 2
8:30-9:00
9:00-10:00
10:00-10:30
10:30-11:00
11:00-12:30
12:30-2:00
2:00-3:00
3:30-4:00
4:00-5:30
Day 3
8:30-9:00
9:00-10:30
10:30-11:00
11:00-12:30
12:30-1:30
Continental Breakfast
Web Services Technologies and WSAD Demo
Use of Web Services Technologies and Tools
Break
Intro to the Development Environment
Lunch (ANL Guest House)
Introduction to Suggested Exercises
Break
Exercises and Ad Hoc Presentations
Continental Breakfast
Exercises and Ad Hoc Presentations
Break
Finish Exercises and Present Results
Lunch (ANL Guest House)
Day 2
Days 2 and 3: Architecture Sessions
Auditorium
8:30-9:00
9:00-10:30
10:30-11:00
11:00-11:45
11:45-12:30
12:30-2:00
2:00-3:30
3:30-4:00
4:00-4:45
4:45-5:30
Day 3
8:30-9:00
9:00-9:45
9:45-10:30
10:30-11:00
11:00-12:30
12:30-1:30
Continental Breakfast
Services Architecture & Brainstorming
Break
Infrastructure Specifications
Security
Lunch (ANL Guest House)
Resource Management and Discovery
Break
Services Definition: databases
Services Definition: data replication
Continental Breakfast
Service Definition: clustering and logging
European DataGrid’s use of OGSA
Break
Wrap-Up
Lunch (ANL Guest House)
Key People
z
Karl Czajkowski, Ian Foster, Carl Kesselman, Frank
Siebenlist, Steve Tuecke, Von Welch, Larry Flon
‹
z
Jeff Frey, Steve Graham, Inderpal Narang, Jeff
Nick, Scott Sylvester, John Rofrano, …
‹
z
OGSA C runtime and services project lead
Gregor von Laszewski
‹
z
OGSI Technology Preview Implementation
Kate Keahey
‹
z
OGSA design team, IBM
Jarek Gawor, Thomas Sandholm, John Bresnahan,
Ravi Madduri, Mike Darcy
‹
z
OGSA design team, Globus Project
Java technologies
Lisa Childers
‹
OGSA project coordinator
9
10
Project Partners
z
Keith Jackson, Mary Thompson (LBNL)
‹
z
Malcolm Atkinson, Norman Paton (UK)
‹
z
Cool Web service stuff
EU DataGrid and EU GridLab projects
‹
z
Database services
Dennis Gannon, Geoffrey Fox, Mary Thomas
‹
z
Security, Python bindings, services
Various higher-level services
Hopefully more by the end of the meeting
11
Support
z
z
IBM
‹
OGSA-enabled Websphere, DB2
‹
Support of Globus Project OGSA R&D
Microsoft
‹
z
Entropia, Platform, United Devices, Avaki
‹
z
OGSA support in .NET
OGSA-compliant products
DOE Office of Science MICS Office
12
Overview
z
Meeting goals and program
z
Grids and the Globus Toolkit
z
Open Grid Services Architecture (OGSA)
z
‹
Motivation & concepts
‹
Core interfaces and examples
OGSA and the Globus Toolkit
‹
Plans and implementation status
13
The Grid Vision
“Resource sharing & coordinated problem
solving in dynamic, multi-institutional
virtual organizations”
‹
‹
On-demand, ubiquitous access to
computing, data, and services
New capabilities constructed dynamically
and transparently from distributed services
“When the network is as fast as the computer's
internal links, the machine disintegrates across
the net into a set of special purpose appliances”
(George Gilder)
14
Underlying Technical Requirements
z
z
z
z
Dynamic formation and management of
virtual organizations
Online negotiation of access to services:
who, what, why, when, how
Configuration of applications and systems
able to deliver multiple qualities of service
Autonomic management of distributed
infrastructures, services, and applications
15
State of the Art (1):
Globus ToolkitTM (since 1996)
z
Small, standards-based set of protocols for
distributed system management
‹
z
Information-centric design
‹
z
Data models; publication, discovery protocols
Open source implementation
‹
z
Authentication, delegation; resource
discovery; reliable invocation; etc.
Large international user community
Successful enabler of higher-level services
and applications
16
State of the Art (2):
Grid Services and Tools
z
z
z
z
Data-oriented: replica location, replica
management, collection management
Security: community authorization
Scheduling: Condor, superschedulers,
brokers
Programming: object models, message
passing, etc.
z
Portal toolkits
z
Etc., etc.
17
The Globus Toolkit in One Slide
z
Grid protocols (GSI, GRAM, …) enable resource
sharing within virtual orgs; toolkit provides reference
implementation ( = Globus Toolkit services)
MDS-2
(Monitor./Discov. Svc.)
Reliable
remote
GSI User
invocation Gatekeeper Reporter
(Grid
(registry +
Authenticate &
(factory)
discovery)
Security create proxy
Create process Register
Infrastruc- credential
ture)
User
process #1
Proxy
User
process #2
Proxy #2
GRAM
(Grid Resource Allocation & Management)
z
Soft state
registration;
enquiry
Other GSIauthenticated
remote service
requests
GIIS: Grid
Information
Index Server
(discovery)
Other service
(e.g. GridFTP)
Protocols (and APIs) enable other tools and services
for membership, discovery, data mgmt, workflow, …
18
Globus Toolkit Structure
Service naming
Soft state
management
Reliable invocation
GRAM
Notification
MDS
GSI
GridFTP
MDS
???
GSI
GSI
Data
Resource
Other Service
or Application
Job
manager
Job
manager
Compute
Resource
Lots of good mechanisms, but (with the exception of GSI) not that easily
incorporated into other systems
19
Globus Toolkit: Evaluation (+)
z
z
Good technical solutions for key problems, e.g.
‹
Authentication and authorization
‹
Resource discovery and monitoring
‹
Reliable remote service invocation
‹
High-performance remote data access
This & good engineering is enabling progress
‹
‹
Good quality reference implementation, multilanguage support, interfaces to many systems,
large user base, industrial support
Growing community code base built on tools
20
Globus Toolkit: Evaluation (-)
z
Protocol deficiencies, e.g.
‹
‹
z
z
Heterogeneous basis: HTTP, LDAP, FTP
No standard means of invocation, notification,
error propagation, authorization, termination, …
Significant missing functionality, e.g.
‹
Databases, sensors, instruments, workflow, …
‹
Virtualization of end systems (hosting envs.)
Little work on total system properties, e.g.
‹
Dependability, end-to-end QoS, …
‹
Reasoning about system properties
21
Grid Services and Tools: Critique
+ A wide variety of wonderful tools have
been constructed
+ Use of common Grid infrastructure has
accelerated development and deployment
- Still relatively little reuse of constituent
components across projects
- Interoperability does not extend to
important issues like error notification
22
Overview
z
Meeting goals and program
z
Grids and the Globus Toolkit
z
Open Grid Services Architecture (OGSA)
z
‹
Motivation & concepts
‹
Core interfaces and examples
OGSA and the Globus Toolkit
‹
Plans and implementation status
Grid Evolution:
Open Grid Services Architecture
z
Four largely orthogonal goals
1) Refactor Globus protocol suite to enable
common base and expose key capabilities
2) Extend for new technical requirements
3) Service orientation to virtualize resources
and unify resources/services/information
4) Embrace key Web services technologies for
standard IDL, leverage commercial efforts
z
Result
‹
Standard interfaces & behaviors for building
distributed systems: the Grid service
23
24
Refactor Globus Protocol Suite
z
z
Extract, generalize, allow modular use of
protocols and mechanisms for
‹
Reliable invocation
‹
Service description & information access
‹
Notification
‹
Policy management
‹
Lifetime management
‹
Service naming
‹
Authentication
Designed in an integrated, uniform fashion
25
Extend into New Areas
z
Extend core protocol suite to address
‹
Manageability
‹
Concurrency control
‹
Others
26
Service Orientation
z
Define all entities by interface & behavior,
so that
‹
‹
Resources and programs are treated in the
same manner & accessed in the same way
Virtualization easy to achieve: e.g. “compute
service” may be computer or network
27
Embrace Standards:
Two Distinct (But Interrelated) Issues
z
Standard means of defining, discovering,
and invoking interfaces
‹
z
Addressed by Web services
Standard means of customizing computer
systems to application requirements
‹
Addressed by hosting environments: J2EE,
.NET, …
28
OGSA System Structure
z
A standard substrate: the Grid service
‹
‹
z
The “Grid Service Specification”
… supports standard service specifications
‹
‹
z
Standard interfaces and behaviors that
address key distributed system issues
Resource management, databases,
workflow, security, diagnostics, etc., etc.
Target of current & planned GGF efforts
… and arbitrary application-specific
services based on these & other definitions
29
OGSA Definition and Delivery
(Very Approximate!!)
Other Systems
T
I
M
E
GGF
OGSI
WG
Grid Service
Specification
GGF
WGs
Other core specs:
• Security
• Res. Mgmt.
• Etc.
GGF
WGs
Other specs:
• Databases
• Etc.
• Etc.
Prototype
Feedback
Globus OGSI
Reference Impln
Globus Toolkit
Version 3
Other OGSAbased software
Open Grid Services Architecture:
Transient Service Instances
z
“Web services” address discovery & invocation
of persistent services
‹
z
z
Interface to persistent state of entire enterprise
In Grids, must also support transient service
instances, created/destroyed dynamically
‹
Interfaces to the states of distributed activities
‹
E.g. workflow, video conf., dist. data analysis
Significant implications for how services are
managed, named, discovered, and used
‹
In fact, much of OGSA (and Grid) is concerned
with the management of service instances
30
31
Open Grid Services Architecture
z
Defines fundamental (WSDL) interfaces and
behaviors that define a Grid Service
‹
‹
z
E.g., serviceType (a group of portTypes)
Delivery via open source Globus Toolkit 3.0
‹
z
A unifying framework for interoperability &
establishment of total system properties
Defines WSDL extensibility elements
‹
z
Required + optional interfaces = WS “profile”
Leverage GT experience, code, community
And also commercial implementations
32
The Grid Service =
Interfaces/Behaviors + Service Data
Service data access
Explicit destruction
Soft-state lifetime
Binding properties:
- Reliable invocation
- Authentication
GridService
(required)
Service
data
element
… other interfaces …
(optional)
Service
data
element
Service
data
element
Implementation
Hosting environment/runtime
(“C”, J2EE, .NET, …)
Standard:
- Notification
- Authorization
- Service creation
- Service registry
- Manageability
- Concurrency
+ applicationspecific interfaces
33
“Web Services”
z
Increasingly popular standards-based
framework for accessing network applications
‹
z
WSDL: Web Services Description Language
‹
z
XML-based RPC protocol; common WSDL target
WS-Inspection
‹
z
Interface Definition Language for Web services
SOAP: Simple Object Access Protocol
‹
z
W3C standardization; Microsoft, IBM, Sun, others
Conventions for locating service descriptions
UDDI: Universal Desc., Discovery, & Integration
‹
Directory for Web services
34
OGSA and Hosting Enviroments
protocol
termination
protocol
termination
protocol
termination
Demarshalling / Decoding /
Routing
container
Grid
service
impl.
Grid
service
impl.
35
Open Grid Services Architecture:
Fundamental Structure
1) WSDL conventions and extensions for
describing and structuring services
‹
Useful independent of “Grid” computing
2) Standard WSDL interfaces & behaviors for
core service activities
‹
portTypes and operations => protocols
36
WSDL Conventions & Extensions
z
portType (standard WSDL)
‹
z
serviceType (extensibility element)
‹
z
Represents actual code
service (standard WSDL)
‹
z
List of port types: enables aggregation
serviceImplementation (extensibility element)
‹
z
Define an interface: a set of related operations
instanceOf extension: map descr.->instance
compatibilityAssertion (extensibility element)
‹
portType, serviceType, serviceImplementation
Standard Interfaces & Behaviors:
Four Interrelated Concepts
z
Naming and bindings
‹
z
Information model
‹
z
z
Every service instance has a unique name,
from which can discover supported bindings
Service data associated with Grid service
instances, operations for accessing this info
Lifecycle
‹
Service instances created by factories
‹
Destroyed explicitly or via soft state
Notification
‹
Interfaces for registering interest and
delivering notifications
37
OGSA Interfaces and Operations
Defined to Date
z
z
z
GridService
‹
FindServiceData
‹
Destroy
‹
SetTerminationTime
Required
z
Factory
‹
z
SubscribeToNotificationTopic
‹
UnsubscribeToNotificationTopic
z
DeliverNotification
z
‹
FindByPrimaryKey
‹
DestroyByPrimaryKey
Registry
NotificationSink
‹
CreateService
PrimaryKey
NotificationSource
‹
38
‹
RegisterService
‹
UnregisterService
HandleMap
Authentication, reliability are binding properties
Manageability, concurrency, etc., to be defined
‹
FindByHandle
39
Service Data
z
A Grid service instance maintains a set of service
data elements
‹
‹
z
FindServiceData operation (GridService interface)
queries this information
‹
z
XML fragments encapsulated in standard <name, type,
TTL-info> containers
Includes basic introspection information, interfacespecific data, and application data
Extensible query language support
See also notification interfaces
‹
Allows notification of service existence and changes in
service data
Grid Service Example:
Database Service
z
z
A DBaccess Grid service will support at
Grid
least two portTypes
‹
GridService
‹
DBaccess
Each has service data
‹
‹
Service
DBaccess
Name, lifetime, etc.
DB info
GridService: basic introspection
information, lifetime, …
DBaccess: database type, query languages
supported, current load, …, …
40
41
Lifetime Management
z
GS instances created by factory or manually;
destroyed explicitly or via soft state
‹
z
z
Negotiation of initial lifetime with a factory (=service
supporting Factory interface)
GridService interface supports
‹
Destroy operation for explicit destruction
‹
SetTerminationTime operation for keepalive
Soft state lifetime management avoids
‹
Explicit client teardown of complex state
‹
Resource “leaks” in hosting environments
42
Factory
z
Factory interface’s CreateService operation
creates a new Grid service instance
‹
z
z
Reliable creation (once-and-only-once)
CreateService operation can be extended to
accept service-specific creation parameters
Returns a Grid Service Handle (GSH)
‹
A globally unique URL
‹
Uniquely identifies the instance for all time
‹
Based on name of a home handleMap service
43
Transient Database Services
“What services
can you create?”
“Create a database
service”
Grid
Service
“What database
services exist?”
DBaccess
Factory
Grid
Service
DBaccess
Instance name, etc.
Name, lifetime, etc.
Factory info
DB info
Grid
Service
Registry
Grid
Service
DBaccess
Instance name, etc.
Name, lifetime, etc.
Registry info
DB info
44
Example:
Data Mining for Bioinformatics
Community
Registry
Mining
Factory
Database
Service
BioDB 1
User
Application
“I want to create
a personal database
containing data on
e.coli metabolism”
Compute Service Provider
.
.
.
Database
Factory
.
.
.
Database
Service
BioDB n
Storage Service Provider
45
Example:
Data Mining for Bioinformatics
“Find me a data Community
mining service, and Registry
somewhere to store
data”
Mining
Factory
Database
Service
BioDB 1
User
Application
Compute Service Provider
.
.
.
Database
Factory
.
.
.
Database
Service
BioDB n
Storage Service Provider
46
Example:
Data Mining for Bioinformatics
GSHs for Mining
and Database
factories
User
Application
Community
Registry
Mining
Factory
Database
Service
BioDB 1
Compute Service Provider
.
.
.
Database
Factory
.
.
.
Database
Service
BioDB n
Storage Service Provider
47
Example:
Data Mining for Bioinformatics
Community
Registry
“Create a data mining
service with initial
lifetime 10”
User
Application
“Create a
database with initial
lifetime 1000”
Mining
Factory
Database
Service
BioDB 1
Compute Service Provider
.
.
.
Database
Factory
.
.
.
Database
Service
BioDB n
Storage Service Provider
48
Example:
Data Mining for Bioinformatics
Community
Registry
“Create a data mining
service with initial
lifetime 10”
User
Application
“Create a
database with initial
lifetime 1000”
Mining
Factory
Database
Service
Miner
BioDB 1
Compute Service Provider
.
.
.
.
.
.
Database
Factory
Database
Service
BioDB n
Database
Storage Service Provider
49
Example:
Data Mining for Bioinformatics
Community
Registry
Mining
Factory
Query
Miner
User
Application
Compute Service Provider
.
.
Query
.
Database
Factory
Database
Service
BioDB 1
.
.
.
Database
Service
BioDB n
Database
Storage Service Provider
50
Example:
Data Mining for Bioinformatics
Community
Registry
Keepalive
Mining
Factory
Query
Miner
Compute Service Provider
.
.
Query
.
User
Application
Keepalive
Database
Factory
Database
Service
BioDB 1
.
.
.
Database
Service
BioDB n
Database
Storage Service Provider
51
Example:
Data Mining for Bioinformatics
Community
Registry
Keepalive
User
Application
Keepalive
Mining
Factory
Database
Service
Miner
BioDB 1
Compute Service Provider
.
.
.
.
.
.
Database
Factory
Results
Database
Service
Results
BioDB n
Database
Storage Service Provider
52
Example:
Data Mining for Bioinformatics
Community
Registry
User
Application
Keepalive
Mining
Factory
Database
Service
Miner
BioDB 1
Compute Service Provider
.
.
.
.
.
.
Database
Factory
Database
Service
BioDB n
Database
Storage Service Provider
53
Example:
Data Mining for Bioinformatics
Community
Registry
Mining
Factory
Database
Service
BioDB 1
Compute Service Provider
.
.
.
User
Application
Keepalive
Database
Factory
.
.
.
Database
Service
BioDB n
Database
Storage Service Provider
54
Notification Interfaces
z
NotificationSource for client subscription
‹
‹
z
z
One or more notification generators
z
Generates notification message of a specific type
z
Typed interest statements: E.g., Filters, topics, …
z
Supports messaging services, 3rd party filter services, …
Soft state subscription to a generator
NotificationSink for asynchronous delivery of
notification messages
A wide variety of uses are possible
‹
E.g. Dynamic discovery/registry services, monitoring,
application error notification, …
55
Notification Example
z
Notifications can be associated with any
(authorized) service data elements
Grid
Service
Grid
Service
Notification
Sink
Name, lifetime, etc.
DB info
DBaccess
Name, lifetime, etc.
Notification
Source
DB info
Subscribers
56
Notification Example
z
Notifications can be associated with any
(authorized) service data elements
Grid
Service
Notification
Sink
“Notify
Name, lifetime, etc.
DB info
me of
new data about
membrane proteins”
Notification
Source
Grid
Service
DBaccess
Name, lifetime, etc.
DB info
Subscribers
57
Notification Example
z
Notifications can be associated with any
(authorized) service data elements
Grid
Service
Name, lifetime, etc.
DB info
Grid
Service
Notification
Sink
Keepalive
Notification
Source
DBaccess
Name, lifetime, etc.
DB info
Subscribers
58
Notification Example
z
Notifications can be associated with any
(authorized) service data elements
Grid
Service
Notification
Sink
New data
Name, lifetime, etc.
DB info
Grid
Service
DBaccess
Name, lifetime, etc.
Notification
Source
DB info
Subscribers
59
Overview
z
Meeting goals and program
z
Grids and the Globus Toolkit
z
Open Grid Services Architecture (OGSA)
z
‹
Motivation & concepts
‹
Core interfaces and examples
OGSA and the Globus Toolkit
‹
Plans and implementation status
60
OGSA and the Globus Toolkit
z
Technically, OGSA enables
‹
‹
‹
z
Refactoring of protocols (GRAM, MDS-2, etc.)—while
preserving all GT concepts/features!
Integration with hosting environments: simplifying
components, distribution, etc.
Greatly expanded standard service set
Pragmatically, we are proceeding as follows
‹
Develop open source OGSA implementation
z
Globus Toolkit 3.0; supports Globus Toolkit 2.0 APIs
‹
Partnerships for service development
‹
Also expect commercial value-adds
GT3: An Open Source OGSACompliant Globus Toolkit
z
GT3 Core
‹
‹
‹
z
Reference impln of evolving
standard
Java first, C soon, C#?
GT3 Base Services
‹
‹
z
Implements Grid service
interfaces & behaviors
Evolution of current Globus
Toolkit capabilities
Backward compatible
Many other Grid services
GT3
Data
Services
Other Grid
Services
GT3 Base Services
GT3 Core
61
62
Evolution
z
This is not happening all at once
‹
z
We have an early prototype of Core
(“Technology Preview” release in May)
‹
Next we will work on Base, others
‹
Full release by end of 2002 (estimate)
‹
Partnerships in place for other services
Backward compatibility
‹
API level seems straightforward
‹
Protocol level: gateways?
‹
We need input on best strategies
Open Grid Services Architecture:
Summary
z
Service orientation to virtualize resources
‹
z
z
Everything is a service
From Web services
‹
z
63
Standard interface definition mechanisms:
multiple protocol bindings, local/remote
transparency
From Grids
‹
Service semantics, reliability and security models
‹
Lifecycle management, discovery, other services
Multiple “hosting environments”
‹
C, J2EE, .NET, …
64
Recap: The Grid Service
Reliable invocation
Authentication
Service data access
Explicit destruction
Soft-state lifetime
GridService
Service
data
element
… other interfaces …
Service
data
element
Service
data
element
Implementation
Hosting environment/runtime
(“C”, J2EE, .NET, …)
Notification
Authorization
Service creation
Service registry
Manageability
Concurrency
Hmm, Isn’t This Just Another
Object Model?
z
Well, yes, in a sense
‹
‹
z
Strong encapsulation
We (can) profit greatly from experiences of
previous object-based systems
But
‹
Focus on encapsulation not inheritance
‹
Does not require OO implementations
‹
‹
Value lies in specific behaviors: lifetime,
notification, authorization, …, …
Document-centric not type-centric
65
Grids and OGSA:
Research Challenges
z
z
Grids pose profound problems, e.g.
‹
Management of virtual organizations
‹
Delivery of multiple qualities of service
‹
Autonomic management of infrastructure
‹
Software and system evolution
OGSA provides foundation for tackling
these problems in a rigorous fashion?
‹
‹
Structured establishment/maintenance of
global properties
Reasoning about total system properties
66
67
Summary: Goals for this Meeting
z
z
z
Jump start OGSA-based development
efforts that will inform and complement
OGSA specification and implementation
Nucleate groups that will work to expand
the set of OGSA specifications
Create an OGSA community