ICENI Overview
&
Grid Scheduling
Laurie Young
London e-Science Centre
Department of Computing, Imperial College
ICENI
The Iceni, under Queen
Boudicca, united the tribes
of South-East England in a
revolt against the
occupying Roman forces
in AD60.
IC e-Science Networked Infrastructure
Developed by LeSC Grid Middleware Group
Collect and provide relevant Grid meta-data
Use to define and develop higher-level services
Interaction with other frameworks: OGSA, Jxta etc.
2
ICENI (The Big Picture)
Web
Services
Gateway
Public Computational Community
Computational
Resource
JavaCoG
Globus
Storage
Resources
Network
Resources
Identity
Manager
CR
Private
Administrative
Domain
CR
Application
Portal
SR
Resource Browser
Domain Manager
CR
SR
SR
Public Computational Community
SR
Software
Resources
Resource
Broker
Application
Mapper
Policy Manager
CR
Resource Manager
Private
SR
Gateway between private
and public regions
Component
Design Tools
Application
Design Tools
Public
3
ICENI Stack
Portal Interface
Application Construction & Deployment
ICENI Middleware
Grid Fabric
4
Web Portals
• Handheld wireless devices become ubiquitous
– Personal Digital Assistants, Mobile Phones
– Secure access any time, any place, any where
• Use X.509 certificates embedded in a browser to
authenticate user’s identity
• Integration portal infrastructure with ICENI
– EPIC: Use component meta-data to
build portal application
• Goal: Provide secure ‘one stop shop’ for escience
5
EPIC: e-Science Portal at Imperial
College
• Collaborative LeSC industrial project with Sun
Microsystems
• Develop a secure portal infrastructure to:
– Access your own personal environment
– Applications to support day-to-day e-science
– Interaction with other Grid infrastructures
• Allow role based access to resources
– Anonymous: public web pages
– Students: internal pages, email, compute resources
– Staff: restricted pages
6
ICENI Application Model
• Legacy code!
• Component Applications
– Compose applications from many components
– Component does work on data
– Component communicates data
7
Component Motivation
•
•
•
•
•
Logical application model
Collaborative software authoring
Promote component reuse and sharing
Simplify application construction
Enable deployment to diverse Grid resources:
– Communication Selection
– Implementation Selection
8
Layered Abstraction
Meaning
dataflow
abstract data types
may have many
Behaviour
Behaviour
control flow
threads etc.
may each have many
Implementation Implementation
performance,
architectures,
concrete data type
9
Component’s View of the Grid
Context object
Other
Code
SOAP
RMI
You may call
methods
provided by the
middleware
More
Code
My Code
You must
implement
a
provided
interface
10
Visual Component Composition
11
Deployment of Components
End Application
Design
User
Tools
Repository
Code
Code
Access Resource
Information
Application Description
Document
Code
Component Implementation
Design
Annotating
Tools
Tools
Application
Mapper
Run-Time
Representation
APO
Code
Application Proxy
Object
Code
RTR
Grid
Container
12
Component Execution
OGSA, Jxta, etc.
OGSA, Jxta, etc.
APO
Jini
Jini
MPI
Code
Code
Code
RTR
RTR
RTR
SOAP
RMI
Compute Resource
Network
Resource
Hardware
13
Components as Services
Service interface
Context object
SOAP (or other)
protocol
Component
14
ICENI & Jini: P2P
Conceptual Model of peer- to- peer architecture
Service requester
Two- way
Interaction
with service
Lookup
Service
Service
Matches
Register service
Service Locator
Service Provider
Discover services
15
Web Services Architecture
Web Service Model
Service requester
(SOAP Client)
SOAP
message
exchange
Service Provider
(SOAP Server)
UDDI
Lookup
UDDI
Response,
WSDL
location
UDDI
registration
Web Services Crawler?
Service Locator
(UDDI Registry)
16
Synergy
The Synergy
Service Requester
Web Service Client
SOAP
Proxy
SOAP
Java
App
SOAP
P2P
(Jini)
Jini Object
RMI
Jini Lookup
RMI
SOAP
Service Provider
Web Service Proxy
Web Services
Service Locator
Web Service Proxy
17
Grid Service Contracts
Jini
Lookup
Service
DRMAA
Client
DRMAA
Resource
18
Grid Service Contracts
Resource
Browser
DRMAA
Client
User:B
Jini
Lookup
Service
DRMAA
Resource
User: A+B
Duration:1hr
19
Grid Service Contracts
User:A
Duration:10m
DRMAA
Resource
User:A
DRMAA
Client
Jini
Lookup
Service
DRMAA
Client
User:B
DRMAA
Resource
User: A+B
Duration:1hr
20
OGSA & Jini Integration
User:A
Duration:10m
DRMAA
Resource
Gateway
Manager
GSI enabled
Web Service
Hosting
Environment
Jini
Lookup
Service
DRMAA
Resource
User: A+B
Duration:1hr
21
OGSA & Jini Integration
User:A
Duration:10m
DRMAA
Resource
Gateway
Manager
WSDL
Interface
GSI enabled
Web Service
Hosting
Environment
Jini
Lookup
Service
Jini Client
Interface
DRMAA
Resource
User: A+B
Duration:1hr
22
OGSA & Jini Integration
User:A
Duration:10m
DRMAA
Resource
Gateway
Manager
WSDL
Interface
GSI + SOAP
Connection
GSI enabled
Web Service
Hosting
Environment
Jini
Lookup
Service
Jini Client
Interface
DRMAA
Resource
User: A+B
Duration:1hr
23
OGSA & Jini Integration
User:A
Duration:10m
DRMAA
Resource
Gateway
Manager
WSDL
Interface
GSI enabled
Web Service
Hosting
Environment
Jini
Lookup
Service
SOAP->Java
GSI + SOAP
Connection
User Info
Jini Client
Interface
DRMAA
Resource
User: A+B
Duration:1hr
24
Application Mapping (Scheduling)
• Architecture
– How meta-data is collected
– What meta-data is required
• Scheduling Algorithms
– Map components onto resources for “best” results
– Meta-data dependent decisions
25
Scheduling Architecture
ICENI
App Builder (GUI)
Component Repository
Performance Models
Scheduler
Launcher
Resources
26
Multiple Metrics (1)
• “It is the goal of a scheduler to optimise one or
more metrics” (Feitelson & Rudolph)
• Generally one metric is most important
– Application Optimisation
• Execution time
• Execution cost
– Host Optimisation
• Host utilisation
• Host throughput
• Interaction Latency
27
Multiple Metrics (2)
• In a Grid Environment there are three application
optimisation based important metrics
– Start time (τb )
– End time (τe )
– Cost ( ε )
• Relative importance varies on a user by user and
application by application basis
28
Combining Metrics – Benefit Fn
• A Benefit Function maps the metrics we are
interested in to a single Benefit Value metric
• Different benefit functions represent different
optimisation preferences
B = B(τb,τe, ε )
29
Optimisation Preferences
• Cost Optimisation
• Time Optimisation
α
B = if τe < τe max and ε < ε max
ε
β
B = if τe < τe max and ε < ε max
τ
• Cost/Time Optimisation
χ
B = ν η if τe < τe max and ε < ε max
ετ
30
Schedule Benefit
• Each component and communication has a benefit
function
• Each resource and network connection has a
predicted time & cost for each component or
communication that could be deployed
• Fit the tasks onto the resources to get the
maximum Total Predicted Benefit
Bt = ∑ B (τb,τe, ε )
31
Graph Oriented Scheduling (1)
• Applications are described as a graph
– Nodes represent application components
– Edges represent component communication
• Resources are described as a graph
– Nodes represent resources
– Edges represent network connections
32
Graph Oriented Scheduling (2)
Design
Factory
Analyse
Atlas
Scatter
Saturn
Viking
Mesh
Mesh
Mesh
DRACS
DRACS
DRACS
Condor
pool
Gather
33
Graph Oriented Scheduling (3)
Atlas
Design
Condor
pool
Gather
Scatter
Saturn
Analyse
Viking
Factory
34
Summary
• Component framework provides:
– Rich application meta-data
– Decoupled component definition and implementation
• Meta-Data:
– Exploit performance information to map component implementation to
the ‘best’ resources
• Resource Broker:
– Resource selection through user defined policies:
• Minimise cost using computational economics
• Minimise execution time using the application mapper
35
Acknowledgements
• Director: Professor John Darlington
• Technical Director: Dr Steven Newhouse
• Research Staff:
–
–
–
–
–
Anthony Mayer, Nathalie Furmento
Stephen McGough, James Stanton
Yong Xie, William Lee
Marko Krznaric, Murtaza Gulamali
Asif Saleem, Laurie Young, Gary Kong
• Contact:
– http://www.lesc.ic.ac.uk/
– e-mail: lesc@ic.ac.uk
36