London e-Science Centre
Workflow, Planning and Performance
Dr Andrew Stephen McGough
London e-Science Centre
Imperial College London
20th October 2006
Outline
London e-Science Centre
Workflow and Pipelines
The Planner
Using Information
Use of these ideas
2
London e-Science Centre
Workflow
And Pipelines
Workflow Levels
London e-Science Centre
Workflow pipeline
How do
Planned
Executing
Workflow
Design Workflow
Conceptual
you move
VCR
from one
level to the
next?
CE
SE
Information
Data
Process
Control
Process
and reasoning / use
of the information
4
Information
London e-Science Centre
Problem space
knowledge, Application
knowledge
user
Component
developer
Component knowledge
Problem space
knowledge
Domain expert
Resource information
Resource information
Grid Structure
information
Resource Owner
Grid Manager
5
Information
London e-Science Centre
We want to capture what the user wants to
achieve not how they think it should be done
We don’t just want to know what things are we
want to be able to reason about them
 Their meaning
 Their behaviour
 How they work
Current workflow languages don’t support these
ideas – need to add this on-top
6
Example: Linear Solver
London e-Science Centre
If we can describe things at a higher level we have more options
Linear
Equation
Source
Matrix
Vector
Unsymmetric
Matrix
Display
Vector
Vector
LU
C
C
Linear
Equation
Solver
BiCG
Java
C
Java
Java
ScaLAPACK
ScaLAPACK
7
Abstract Workflow Pipeline
London e-Science Centre
Decompose into interacting levels
Specification
Planning
Observation
Execution
8
London e-Science Centre
The Planner
Tasked with deciding where each part of the
workflow should be deployed
Optimisation
London e-Science Centre
A good planner should be able to select
an “optimal” set of resources and
implementations of components
 Execution of workflows on the Grid is an
NP-Hard problem
 Large number of resources available for
use
 Multiple implementations of code that
can be deployed onto resources
 Performance aware scheduling goes a
long way to achieving a good selection
10
Workflow Scheduling with Performance
London e-Science Centre
Component
Repository
Execution
Service
Planner
Performance
Repository
11
Single Workflow Selection
London e-Science Centre
An approach: Constraint Optimisation
Optimise for time
With the following constraints
Cost:
C1
C2
ti tj
5 5£4
5+14
£3
14
R1
R1
7 7£12 7+12
£9
12
Selection:
Deadline:
R2
R2
4 4£6
4+17
£2
17
R3
R3
C3
t t
5+14+49
£2
49
R1
7+12+54
£2
54
R2
4+17+51
£3
51
R3
+ Other Constraints
Temporal ordering:
12
Cross Workflow Selection
London e-Science Centre
An approach: Time interval pre-selection
A resource will run a given service
 Which can be used by multiple workflows
Each service has:
 a cost ci to run
 xi resources running that service
Workflow 1 × 50
Workflow 2 × 75
Workflow 3 × 35
Resources can change the service they
provide zi(tj, tj+1) with associated cost
R1
ki
R2
Hence we need to minimise
cTx + kTz
R3
Can solve analytically for M/G/1 services
R1
R1
R2
R2
R3
R3
13
However:
London e-Science Centre
 Algorithms for determining the “best”
resources and implementations become
slow as the search space increases
 Performance Information is often slow to
obtain
 Current status of resources is often slow to
obtain
 thus out of date by the time you get it
 We need to reduce the search space
before doing performance aware
planning
14
London e-Science Centre
Using the Information
Using this information to better perform the
tasks that the user requires
Using information for Optimisation
London e-Science Centre
We can use Information for:
 Validation of the Workflow
 Make sure that only “correct” workflows are
submitted
 Workflow Optimisation
 By altering the workflow to make it more optimal
 Resource Optimisation
 Pruning the set of resources that are considered
 Resource selection
 Matching the approach implementation to the
resource
16
Workflow Validation
London e-Science Centre
Goal: Ensure that a workflow is correct before it
is accepted into the Grid.
 Reduce the chance of “wasted time”
Requirements: Only look at the workflow and
component information
Achieve the goal through:
 Syntactic Validation
 Verify all required connections are made
 Verify that connections are valid
 Semantic Validation
 Can equate to grammar checking
 To determine if the workflow
 Makes sense
 Will produce what the user intended
17
Workflow Optimisation
London e-Science Centre
Goal : optimise the runtime execution of
workflow before it is executed
Requirements: No performance or
resource information is used
Achieves the goal through:
Re-ordering of components
Substitution of components
Addition of components
Pruning of the workflow
18
Static Workflow Optimisation
London e-Science Centre
Re-ordering: Workflows (often composed from composite
workflows) may contain non-optimal ordering of components
Use re-ordering to improve performance
Matrix
Gen
Matrix
Gen
Matrix
Gen
multiply
Matrix
Gen
multiply
Vector
Gen
Vector
Gen
multiply
multiply
Addition: For a component requiring a specific format of data
as input, a transformer component could be added to
achieve the desired format.
Allows more optimal components to be used together
C1
Output in LP format
LP to MPS
Input required in
MPS format
C2
19
Static Workflow Optimisation
London e-Science Centre
Substitution:
A Jacobi Iteration linear solver replaced by Conjugate Gradient
linear solver according to the output of the Discretizer (FEM)
Based on observing the meta-data associated with previous
A (sparse and diagonally dominant)
components
FEM
Workflow Pruning:
JI linear solver
b
Workflows may contain unused components
Especially when composed from other sub-workflows
Remove redundant components
a
f
b
e
d

g
c
Not needed

h
Not needed
20
Resource Optimisation
London e-Science Centre
Goal : Reduce the number of resources
that are considered for use
Requirements: No workflow or resource
current state information is used
Achieves the goal through:
 Authorisation
 Hardware / Software requirements
 Problem Specific requirements
 Out of bounds
21
London e-Science Centre
Use of these ideas
Where these ideas are being deployed
ICENI: Imperial College e-Science
Network Infrastructure
London e-Science Centre
•
•
•
•
•
•
•
•
•
•
Integrated end2end Grid Middleware Solution
Interoperability between architectures, APIs
Added value layer to other middleware
Usability: Interactive Grid Workflows
Role and policy driven security
Foundation for higher-level Services and
Autonomous Composition
ICENI Open Source licence (extended SISSL)
Collect and provide relevant Grid Meta-Data
Pluggable architecture
Test Architecture for Grid Research
The Iceni, under Queen Boudicca, united the
tribes of South-East England in a revolt
against the occupying Roman forces in AD60. 23
Specification
End-users
Developers and Deployers
High-level Abstract Workflow + QoS
Preferences + Security Constraints
Realisation
Functional Description + Performance
Annotation + Availability
Syntactic and Semantic Validation
Static Workflow Optimisation
Performance Data
Performance Repository
Performance
Profiles
Equivalent Workflow
Candidates
P2P / UDDI
Discovery
Component and
Resource
Availability
Dynamic Optimisation and Scheduling
Concrete Workflow + QoS constraints + Security
constraints
Workflow Orchestration and Re-optimisation
Execution
Service
Service
Service
Virtualised Component Container
Execution Environment
Component
Component
Component
Component Packaging and
Deployment
What is GRIDCC
London e-Science Centre
Grid enabled Remote Instrumentation with Distributed Control and Computation
Adding real time control of instruments to the Grid
What makes this unique:
Instruments may only be
ready at specific times
Reservations
Need to ensure other Grid
services are available with
instruments
Reservations / SLA’s
Real-time visualisation of live results and steering
Through Virtual Control Room (VCR)
Reservations / SLA’s
25
GRIDCC WfMS
London e-Science Centre
WfMS
Performance
Repository
Planner
Observer
Validator
WS Interface
BPEL
+
QoS
Worlflow
Optimiser
Resource Resource
Selector
Pruning
Reserver
Façade
Agreement
Service
BPEL
Engine
F
SE
F
CE
F
IE
26
London e-Science Centre
Any Questions?
Thank you for listening