Resource Brokering on Complex
Grids
EUROGRID and GRIP
Presented by John Brooke ESNW
October 3/4
UK/Japan N+N
Contents
Aims of the resource broker
Functionality of the ancestral broker EUROGRID
broker
Interoperability architecture (UNICORE-Globus)
Towards a resource description ontology
Relation to the GGF and OGSA
January 17, 2003
Brokers as VOs
Users
Virtual
Organisation
Brokers
System
Brokers
Compute
Resources
Organization
Firewalls
Federated Brokering
Resource
Client
Client
Broker
Service
Client
Client
Broker
Service
Client
Broker
Service
Broker
Service
Broker
Service
Broker
Service
Resource
Broker
Service
Replication
Broker
Service
VO Layer
Specialist Layer
Resource
Site Layer
Resource
Resource
Resource
Interoperability for Brokering
We want to broker on Grids controlled by either
UNICORE or Globus. In GRIP we developed two
methods
1. Bifurcation, separate “sub-brokers” for a Globus or
a Unicore Grid. This is achieved and is extensible
to a limited extent.
2. Constructing an extendable resource broker
utilising a Grid Resource Ontology to handle
mappings of resource terms.
Ancestral EUROGRID Broker
The API allows two levels of operation:
Resource Checking: Static requirements, capability and
capacity.
QoS Checking: Performance vs cost. Tickets can be issued
as a “guarantee”.
Protocol can be used symmetrically by Broker.
Execution
NJS
2 CheckQoS
1 CheckQoS
User
Broker
NJS
4 CheckQoS_Outcome
3 CheckQoS_Outcome
2 CheckQoS
Execution
NJS
3 CheckQoS_Outcome
1
Bezier
SGI Onyx @ Manchester
Vtk + VizServer
SGI
Ope
n
GL
VizS
erv
er
Firewall
Brokering for Workflows
UNICORE
Gateway and NJS
Manchester
Laptop
SHU Conference Centre
Simulation
Data
UNICORE
Gateway and NJS
QMUL
Dirac
SGI Onyx @ QMUL
LB3D with RealityGrid
Steering API
Steering (XML)
VizServer client
Steering GUI
9 The Mind Electric GLUE web
service hosting environment with
OGSA extensions
9Single sign-on using UK e-Science
digital certificates
9
9
Interoperable Broker – Method 1
1. The Network Job Supervisor (NJS) delegates the
Resource Check to the Broker at the Vsite.
2. The UNICORE brokering track utilises the IDB
exactly as for the ancestral broker.
3. The Globus track uses a translator of the QoS check
object. The translation service is extendable.
4. The results of the translation are used to drive the
LDAP search and the Globus broker then utilises
MDS to perform this.
UNICORE NJS 4.0 gave much greater power and
flexibility in brokering for complex workflows.
Architecture – Method 1
NJS
Delegates resource check
Broker
Unicore Broker
Diagram
Of Broker
Architecture
Globus Broker
Delegates translation
Lookup
resources
IDB
Translator
Uses to drive
LDAP search
Filter
Performs
LDAP search
Basic Translator
MDS(GRIIS/GRIS)
Pros and Cons
•A nice feature of Method 1 is that no alteration needs
to be made to the client side of UNICORE, thus no
alteration for application plugins or “expert” brokers
•Also no alterations need to be made to Globus.
•However the UNICORE description of Grid resources
is very different from the MDS-2 description. MDS-2
does not publish software resource and user
environment, Unicore does not check dynamic
resource, e.g. machine loading.
•The need for resource description translation is thus
highlighted.
Architecture – Method 2
Resource Discovery
Service
NJS
Delegates resource check
Broker
Unicore Broker
Diagram
Of Broker
Architecture
Other
Brokers
Globus Broker
Delegates translation
Lookup
resources
IDB
Hierarchical
Grid Search
Filter
Uses to
Drive MDS
Search
Uses to drive
MDS search
Translator
Filter
Ontology engine
Resource Discovery
Service
Hierarchical
Grid Search
Ontologies
•Defines knowledge domain and allows reasoning on
this domain.
•If we can create a Grid Resource Ontology, creation
of specialist translation classes from basic Grid
translator becomes possible.
•IDB at sites can be created via ontology, it contains
site specific information which the clients job
specification cannot do.
•So brokers take client request formulated in RR
space, at each site use translator to convert to RR
space, offers come back with capability and QoS.
Local Brokering Configurations
Client
Client
Gateway
Gateway
NJS
Broker
IDB
TSI/Host
Broker
GT3
Normal EUROGRID/GRIP Brokering
NJS
NJS
NJS
Host
Host
Host
Site-Wide Brokering
R-GMA
Brokering and OGSA Services
Persistent Virtual
Environments
Metascheduling
Service
Clients
Other Brokers
Banking
Services
Broker
Site Feedback
Policy Manager
Workflow
Manager
Chargeable Schedulable
GridServices
Resource Usage
Monitor
Relevant GGF work
•Grid Protocol Architecture-RG : Core Grid Functions
•Grid Resource Allocation Agreement Protocol-WG
:advanced reservation, co-allocation
•CIM-based schema-WG : successor to LDAP
•GESA-WG looking at economic issues of scheduling
The recently-formed Semantic Grid RG is very
interested in the Grid Resource Ontology idea.
1
Points for Discussion
•What is the relationship between brokering and
scheduling?
•How to deal with legacy (not Grid-aware) schedulers?
•How to relate the ontologies from the application side
(Resource Requestor) to the service provision side
(Resource Provider)?
•How does a broker estimate upper and lower bounds
for turnaround time?
•How far does the broker trust information from the
service provider. Should it monitor running workflows?
1
RR and RP Spaces
request
RR space
RP space
request
B
A
sync
RP space
RR space
C
Request
referral
RP space
D
Figure 1: Request from RR space at A mapped into resource providers at B and C, with C
forwarding a request formulated in RR space to RP space at D. B and C synchronize at end of
workflow before results returned to the initiator A.
January 17, 2003
GRIP First Project Review
1