QuickTime™ and a
GIF decompressor
are needed to see this picture.
A Broker/Scheduler Architecture
for Grid Services
How to Build a Schprokerer
Dr. David Snelling
Fujitsu Laboratories of Europe
Open Issues in Grid Scheduling
October 21-22, 2003
Fujitsu Laboratories of Europe © 2003
What is a Schprokerer?
The Function of Optimizing Workload on the Grid.
It is not Controlling or Monitoring that Workload.
Not the Unicore NJS or GRAM.
Does Include:
Bids, Optimization, SLAs, etc...
What is Workload?
Resource usage over time
“Workload Profile”
Not Necessarily Workflow
See sidebar.
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
Workflow Sidebar
What is Workflow?
A description workload consisting of more than one activity.
Activity = task | job | process | function
A “net” to describe the dependencies between activities.
DAGs, DFGs, Pert Charts, BPDs, ...
Categories of Workflow
Task or Job (partial) sequences
Data/Task/Message Flow
Workflow Description Types
Static vs. Dynamic (changing)
Persistent vs. Dynamic (on demand)
Explicit (DAGs etc.) vs. Implicit (message sets)
Stateful vs. Stateless
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
Workflow Sidebar Continued
Extended Functions
Loops and Conditionals
Error management
Workload Exertion Types
Deployment
Invocation
QuickTi me™ a nd a TIFF (LZW) de com press or are nee ded to s ee this picture.
Entities
Jobs
Processes or Services
Other workflows
People
Tokens
Implicit or explicit on dependency arcs.
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
From BPMN © BPMI
Functions - 1
Resource Publication
Resource Abstraction
Compute # of atoms to processor time on machine X.
Resource Reservation
Issue of Tickets
Generation of Bids for Resource Use
Cost + QoS
User Incarnation
Authorization checking
Negotiation
SLAs, WS-Agreement, ...
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
Functions - 2
Gather Performance Information
Gather Adherence Information
Were SLAs Honored?
Gather Load Information
Locally and Grid wide.
Make Commitments on behalf of other
Schprokerers
Optimize Workload versus Policies
A scheduler for example.
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
Functions - 3
Translate Information Models
Ontology for resource description
Convert “Grid Currency” between VOs
Decide to Migrate Workload
Compute/Data workload and network workload
Not actually migrate the workload
Decide to Preempt Workload
Not actually preempt the workload
Locate Necessary Resources
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
Functions - 4
Aggregate Workflow Task Bids
E.g. Bidding for complete Workflows
Predicting Actual Workload of a WL
description
E.g. Workflow loop, conditionals, etc.
Topology checking
Data not at the end of a thin pipe.
Prediction of Resource Loading
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
OGSA Perspective - I
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
OGSA Perspective - II
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
OGSA Perspective - III
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
Architectural Assumption
Use the WS-Agreement Approach
Distributed Scheduling with Local
Information
From the Grid perspective, we can only “plan”
Not Central scheduling based on
distributed information
The Grid cannot control all the resources therefore
we can’t “schedule”.
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
Class Relationships
Broker
WorkloadSource
Workload
“Demand” side
Initiate Agreements
Consign Work
Describes the work
Dynamic
“Supply” vs
“Demand”
Policy Optimization
Runtime workload
optimization
Scheduler
WorkloadOptimizer
Collective,
Temporal
Optimization
Implement workload
management policy
WorkloadDescription
Describes resource
requirements
Term in agreement
WorkloadManagement
“Supply” side
Provide load info.
WS-Agreement
Manifest the
contract
InfoService
Publish information
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
OntologyService
Translation of
resource
description
CRC Analysis - 1
WorkloadSource: “Demand” resources
Initiate Agreements
Consign workload to ResourceManager
Create workload
Workload: The work
Fully describe the work to be done
Provide WorkloadDescriptions
Subtypes:
Workflow, Job, Task, WS-deployment, WSinvocation
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
CRC Analysis - 2
WorkloadDescriptions: Resource request
Define the (abstract) resource requirements of a workload
Part of an Agreement initiation
WS-Agreement: Contract
Encapsulate agreed terms of usage contract
Define contract between WorkloadSource and
ResourceManager
ResourceManager: “Supplier” of resources
Access control and management of resources
Supply load information etc. to InformationService
Notify GridMonitor of workload deltas etc.
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
CRC Analysis - 3
InformationService: Global Information Source
Maintain and publish information
Gather data from:
ResourceManager
LoadPredictor
GridMonitor
GridSpies
Filter information via ResourceOntologyMapper
ResourceOntologyMapper: Translation
Translate representations of resources
Provide to InformationService only?
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
CRC Analysis - 4
WorkloadOptimizer: Meet policy requirements
Implement workload management policy
Create/Negotiate Agreements with WorkloadSources
Check with AuthorizationService
Subtypes:
Broker: Supply vs. Demand policy agreements
• Check with the GridBank (exchange rates only)
Scheduler: Collective, temporal optimization
DynamicOptimizer: Runtime workload adaptation
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
Functions - 1
Resource Publication - InformationService
Resource Abstraction - Broker
Compute # of atoms to processor time on machine X.
Resource Reservation - ResourceManager
Issue of Tickets - ResourceManager
Generation of Bids for Resource Use - ResourceManager
Cost + QoS
User Incarnation - AuthorizationService
Authorization checking
Negotiation - WorkLoadOptimizer and WorkloadSource
SLAs, WS-Agreement, ...
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
Functions - 2
Gather Performance Information - GridMonitor
Gather Adherence Information - GridSpies
Were SLAs Honored?
Gather Load Information - GridMonitor
Locally and Grid wide.
Make Commitments on behalf of other Schprokerers Broker
Optimize Workload versus Policies - WorkloadOptimizer
A scheduler for example.
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
Functions - 3
Translate Information Models ResourceOntologyMapper
Ontology for resource description
Convert “Grid Currency” between VOs - GridBank
Decide to Migrate Workload DynamicWorkloadOptimizer
Compute/Data workload and network workload
Not actually migrate the workload
Decide to Preempt Workload DynamicWorkloadOptimizer
Not actually preempt the workload
Locate Necessary Resources - InformationService
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003
Functions - 4
Aggregate Workflow Task Bids - Scheduler
E.g. Bidding for complete Workflows
Predicting Actual Workload of a WL
description - Scheduler
E.g. Workflow loop, conditionals, etc.
Topology checking - Scheduler
Data not at the end of a thin pipe.
Prediction of Resource Loading LoadPredictor
QuickTime™ and a
GIF decompressor
are needed to see this picture.
Fujitsu Laboratories of Europe © 2003