Grid meets Economics:
A Market Paradigm for “Resource
Management and Scheduling” for WorldWide Grid Computing
Rajkumar Buyya
Melbourne, Australia
www.buyya.com/ecogrid
WW Grid
2
Need Honest Answers!
WW Grid

I want to have access to your
Grid resources & want to know
how many of you are willing to give me access ?
(following cases)

I am unable to give you access our Australian
machines, but I want to have access to yours!





3
Want to solve academic problems
Want to solve business problems
I am willing to gift you Kangaroos! (bartering)
I am willing to give you access to my machines, if you
want. (sharing, but no measure & no QoS)
I am willing to pay you dollars on usage basis.
(economic incentive, market-based, and QoS)
Overview
A quick glance at today’s Grid computing
 Resource Management challenges for next
generation Grid computing
 A Glance at Approaches to Grid computing.
 Grid Architecture for Computational Economy
 Economy Grid = Globus + GRACE
 Nimrod-G -- Grid Resource Broker
Grid
 Scheduling Experiments
 Case Study: Drug Design
Application on Grid
Economy
 Conclusions
Grid

4
Scheduling
Economics
Scalable HPC: Breaking Administrative
Barriers & new challenges
2100
2100
2100
2100
2100
2100
2100
?
P
E
R
F
O
R
M
A
N
C
E
5
2100
2100
Administrative Barriers
•Individual
•Group
•Department
•Campus
•State
•National
•Globe
•Inter Planet
•Universe
Desktop
SMPs or
SuperComputers
Local
Cluster
Enterprise
Cluster/Grid
Global
Cluster/Grid
Inter Planetary
Grid!
Why Grids? Large Scale Explorations
need them—Killer Applications.

Solving grand challenge applications using
modeling, simulation and analysis
Aerospace
Internet &
Ecommerce
Life Sciences
6
CAD/CAM
Digital Biology
Military Applications
7
What is Grid ?

An infrastructure that logically
couples distributed resources:
Computers – PCs, workstations, clusters,
supercomputers, laptops, notebooks,
mobile devices, PDA, etc;
 Software – e.g., ASPs renting expensive special
data
purpose applications on demand;
archives
 Catalogued data and databases – e.g. transparent
access to human genome database;
 Special devices – e.g., radio telescope –
SETI@Home searching for life in galaxy.
 People/collaborators.
and presents them as an integrated global resource.
It enables the creation of virtual enterprises (VEs)
for resource sharing.



8
Wide
area
Grid Applications-Drivers







9
Distributed HPC (Supercomputing):
 Computational science.
High-throughput computing:
 Large scale simulation/chip design & parameter studies.
Content Sharing (free or paid)
 Sharing digital contents among peers (e.g., Napster)
Remote software access/renting services:
 Application service provides (ASPs).
Data-intensive computing:
 Data mining, particle physics (CERN), Drug Design.
On-demand, realtime computing:
 Medical instrumentation & network-enabled solvers.
Collaborative:
 Collaborative design, data exploration, education.
Building and Using Grids require


Globus
?



10
Services that make our systems Grid Ready!
Security mechanisms that permit resources to
be accessed only by authorized users.
(New) programming tools that make our
applications Grid Ready!.
Tools that can translate the requirements of
an application/user into the requirements of
computers, networks, and storage.
Tools that perform resource discovery,
trading, selection/allocation, scheduling and
distribution of jobs and collects results.
Players in Grid
Computing
11
What users want ?
Users in Grid Economy & Strategy

Grid Consumers



Execute jobs for solving varying problem size and
complexity
Benefit by selecting and aggregating resources wisely
Tradeoff timeframe and cost


Grid Providers



Contribute “idle” resource for executing consumer jobs
Benefit by maximizing resource utilisation
Tradeoff local requirements & market opportunity

12
Strategy: minimise expenses
Strategy: maximise return on investment
Challenges for Next Generation Grid
Technology Development
Sources of Complexity in Resource
Management for World Wide Grid Computing









14

Size (large number of nodes, providers, consumers)
Heterogeneity of resources (PCs, Workstations, clusters, and
supercomputers, instruments, databases, software)
Heterogeneity of fabric management systems (single system image
OS, queuing systems, etc.)
Heterogeneity of fabric management polices
Heterogeneity of application requirements (CPU, I/O, memory,
and/or network intensive)
Heterogeneity in resource demand patterns (peak, off-peak, ...)
Applications need different QoS at different times (time critical
results). The utility of experimental results varies from time to
time.
Geographical distribution of users & located different time zones
Differing goals (producers and consumers have different
objectives and strategies)
Unsecure and Unreliable environment
Traditional approaches to resource
management & scheduling are NOT useful for
Grid ?

They use centralised policy that need



Due to too many heterogenous parameters in the Grid it is
impossible to define/get:




15
system-wide performance matrix and
common fabric management policy that is acceptable to all.
“Economics” paradigm proved to effective institution in managing
decentralization and heterogeneity that is present in human
economies!


complete state-information and
common fabric management policy or decentralised consensus-based
policy.
Fall of USSR & Emergence of US as world superpower! (monopoly?)
So, we propose/advocate the use of computational economics
principles in management of resources and scheduling computations
on world wide Grid.
Think locally and act globally approach to grid computing!
Benefits of Computational
Economies


It provides a nice paradigm for managing self interested and selfregulating entities (resource owners and consumers)
Helps in regulating supply-and-demand of resources.



User-centric / Utility driven
Scalable:







16

Services can be priced in such a way that equilibrium is maintained.
No need of central coordinator (during negotiation)
Resources(sellers) and also Users(buyers) can make their own decisions
and try to maximize utility and profit.
Adaptable,
It helps in offering different QoS (quality of services) to
different applications depending the value users place on them.
It improves the utilisation of resources
It offers incentive for resource owners for being part of the grid!
It offers incentive for resource consumers for being good citizens
There is large body of proven Economic principles and techniques
available, we can easily leverage it.
New challenges of Computational
Economy

Resource Owners







Resource Consumers





17
How do I decide prices ? (economic models?)
How do I specify them ?
How do I enforce them ?
How do I advertise & attract consumers ?
How do I do accounting and handle payments?
…..
How do I decide expenses ?
How do I express QoS requirements ?
How I trade between timeframe & cost ?
….
Any tools, traders & brokers available to automate the
process ?
mix-and-match
Object-oriented
Internet/partial-P2P
Network enabled Solvers
Market/Computational
Economy
18
Many Grid Projects & Initiatives

Australia







Europe











Japan



19
Economy Grid
Nimrod-G
Virtual Lab
Active Sheets
DISCWorld
..new coming up
UNICORE
MOL
Lecce GRB
Poland MC Broker
EU Data Grid
EuroGrid
MetaMPI
Dutch DAS
XW, JaWS
and many more...

USA












Cycle Stealing & .com Initiatives




Distributed.net
SETI@Home, ….
Entropia, UD, Parabon,….
Public Forums

Ninf
DataFarm
and many more...
Globus
Legion
Javelin
AppLeS
NASA IPG
Condor
Harness
NetSolve
AccessGrid
GrADS
and many more...



Global Grid Forum
P2P Working Group
IEEE TFCC
Grid & CCGrid conferences
http://www.gridcomputing.com
Many Testbeds ? & who pays ?,
who regulates demand and supply ?
GUSTO (decommissioned)
WW Grid
World Wide Grid
Legion Testbed
20
NASA IPG
Testbeds so far -- observations

Who contributed resources & why ?



How long ?


Short term: excitement is lost, too much of admin.
Overhead (Globus inst+), no incentive, policy change,…
What we need ? Grid Marketplace!

21
Volunteers: for fun, challenge, fame, charismatic apps,
public good like distributed.net & SETI@Home
projects.
Collaborators: sharing resources while developing new
technologies of common interest – Globus, Legion, Ninf,
Ninf, MC Broker, Lecce GRB,... Unless you know lab.
leaders, it is impossible to get access!
Regulates supply-and-demand, offers incentive for
being players, simple, scalable solution, quasideterministic – proven model in real-world.
Building an Economy Grid
(Next Generation Grid Computing!)
22
To enable the creation of:
Grid Marketplace (competitive)
ASP
Service Oriented Computing
...
And let users focus on their own work (science, engineering, or commerce)!
GRACE: A Reference
Grid Architecture for Computational Economy
Grid Bank
Sign-on
Info ?
Grid Explorer
Application
Job
Control
Agent
Grid Market
Services
Information
Server(s)
Health
Monitor
Grid Node N
Secure
Schedule Advisor
QoS
Grid Node1
Pricing
Algorithms
Trade Server
Trade Manager
…
Deployment Agent
Trading
JobExec
Grid User
Grid Resource Broker
R1
23See PDPTA 2000 paper!
Misc. services
Resource Allocation
Storage
Grid Middleware
Services
Accounting
Resource
Reservation
R2
…
Rm
Grid Service Providers
Economic Models for Trading





Commodity Market Model
Posted Prices Models
Bargaining Model
Tendering (Contract Net) Model
Auction Model




English, first-price sealed-bid, second-price sealedbid (Vickrey), and Dutch (consumer:low,high,rate;
producer:high, low, rate)
Proportional Resource Sharing Model
Shareholder Model
Partnership Model
See SPIE ITCom 2001 paper!: with Heinz Stockinger, CERN!
24
Grid Components
Applications and Portals
Scientific
Collaboration
Engineering
…
Prob. Solving Env.
Development Environments and Tools
Languages
Libraries
Debuggers
Monitoring
Resource Brokers
Web enabled Apps
…
Distributed Resources Coupling Services
Security
Information
Process
Resource Trading
Market Info
Web tools
…
QoS
Grid
Apps.
Grid
Tools
Grid
Middleware
Local Resource Managers
Operating Systems
Queuing Systems
Libraries & App Kernels
…
TCP/IP & UDP
Networked Resources across Organisations
25
Computers
Clusters
Storage Systems
Data Sources
…
Scientific Instruments
Grid
Fabric
Economy Grid = Globus + GRACE
Applications
Science
Engineering
Commerce
…
Portals
High-level Services and Tools
DUROC
MPI-G
Heartbeat
Monitor
Nexus
MDS
Condor
LSF
CC++
GASS
GRD
PBS
DUROC
QBank
eCash
See IPDPS HWC 2001 paper!
26
…
ActiveSheet
Grid Status
Nimrod/G
globusrun
Grid
Apps.
Grid
Tools
Core Services
Globus
Security
Interface
Local
Services
GRACE-TS
GRAM
GARA
GMD
GBank
JVM
TCP
UDP
Linux
Irix
Solaris
Grid
Middleware
Grid
Fabric
GRACE components







27
A resource broker (e.g., Nimrod/G)
Various resource trading protocols for different
economic models
A mediator for negotiating between users and
grid service providers (Grid Market Directory)
A deal template for specifying resource
requirements and services offers
Grid Trading Server
Pricing policy specification
Accounting (e.g., QBank) and payment
management (GridBank, not yet implemented)
Grid Open Trading Protocols
Trade Manager
Trade Server
Get Connected
Call for Bid(DT)
Reply to Bid (DT)
Pricing Rules
Negotiate Deal(DT)
API
….
Confirm Deal(DT, Y/N)
Cancel Deal(DT)
Change Deal(DT)
Get Disconnected
28
DT - Deal Template
- resource requirements (BM)
- resource profile (BS)
- price (any one can set)
- status
- change the above values
- negotiation can continue
- accept/decline
- validity period
Pricing, Accounting, Allocations and Job
Scheduling Flow @ each site/Grid Level
0
Pricing Policy
2
0
1 Trade Server 3
4
DB@Each Site
QBank
5
8
Resource Manager
IBM-LL/PBS/….
6
29
GRID Bank
(digital transactions)
7
Compute Resources
clusters/SGI/SP/...
0. Make Deposits,
Transfers, Refunds,
Queries/Reports
1. Clients negotiates for
access cost.
2. Negotiation is performed
per owner defined policies.
3. If client is happy, TS informs
QB about access deal.
4. Job is Submitted
5. Check with QB for “go ahead”
6. Job Starts
7. Job Completes
8. Inform QB about resource
resource utilization.
Service Items to be Charged


CPU - User and System time
Memory:








30
maximum resident set size - page size
amount of memory used
page faults: with/without physical I/O
Storage: size, r/w/block IO operations
Network: msgs sent/received
Signals received, context switches
Software and Libraries accessed
Data Sources (e.g. Protein Data Bank)
How to decide Price ?














31
Fixed price model (like today’s Internet)
Dynamic/Demand and Supply (like tomorrow’s Internet)
Usage Period
Loyalty of Customers (like Airlines favoring frequent flyers!)
Historical data
Advance Agreement (high discount for corporations)
Usage Timing (peak, off-peak, lunch time)
Calendar based (holiday/vacation period)
Bulk Purchase (register 100 .com domains at once!)
Voting -- trade unions decide pricing structure
Resource capability as benchmarked in the market!
Academic R&D/public-good application users can be offered at
cheaper rate compared to commercial use.
Customer Type – Quality or price sensitive buyers.
Can be Prescribed by Regulating (Govt.) authorities
Payments- Options & Automation


Buy credits in advance / GSPs bill the user later--”pay as you
go”
Pay by Electronic Currency via Grid Bank
 NetCash (anonymity), NetCheque, and Paypal
 NetCheque: - http://www.isi.edu/gost/info/netcash/


NetCash - http://www.isi.edu/gost/info/netcheque/


It supports anonymity and it uses the NetCheque system to
clear payments between currency servers.
Paypal.com– account+email is linked to credit card.


32
Users register with NC accounting servers, can write electronic
cheques and send (e.g email). When deposited, balance is
transferred from sender to receiver account.
Enter the recipient’s email address and the amount you wish to
request.
The recipient gets an email notification and pays you at
www.PayPal.com
Nimrod-G:
The Grid Resource Broker
Soft Deadline and Budget-based
Economy Grid Resource Broker for
Parameter Processing on P2P Grids
Parametric Computing
(What Users think of Nimrod Power)
Parameters
Age
23
23
28
28
19
10
-4000000
Hair
Clean
Beard
Goatee
Clean
Moustache
Clean
Too much
Multiple Runs
Same Program
Multiple Data
34Courtesy: Anand Natrajan, University of Virginia
Magic Engine
Killer Application for the Grid!
See IPDPS 2000 paper!
P-study Applications -Characteristics






35
Code (Single Program: sequential or
threaded)
High Resource Requirements
Long-running Instances
Numerous Instances (Multiple Data)
High Computation-to-Communication Ratio
Embarrassingly/Pleasantly Parallel
Sample P-Sweep Applications
Bioinformatics:
Drug Design / Protein
Modelling
Sensitivity
experiments
on smog formation
Ecological Modelling:
Combinatorial
Control Strategies
Optimization:
for Cattle Tick
Meta-heuristic
Data Mining
parameter estimation
Computer Graphics:
Ray Tracing
High Energy
Physics:
Searching for
Rare Events
Electronic CAD:
Field Programmable
Gate Arrays
VLSI Design:
Finance:
SPICE Simulations
Investment Risk Analysis
Civil Engineering:
Building Design
Automobile:
Crash Simulation
36
Network Simulation
Aerospace:
Wing Design
astrophysics
Thesis


Perform parameter sweep (bag of tasks)
(utilising distributed resources) within “T”
hours or early and cost not exceeding $M.
Three Options/Solutions:



37
Using pure Globus commands
Build your own Distributed App & Scheduler
Use Nimrod-G (Resource Broker)
Executing Remotely
Choose Resource
Transfer Input Files
Set Environment
Start Process
Pass Arguments
Monitor Progress
Read/Write Intermediate Files
Transfer Output Files
38
+Resource Discovery, Trading, Scheduling, Predictions, Rescheduling, ...
Summary View
Job View
Event View
Using Pure Globus commands
Do all yourself! (manually)
Total Cost:$???
39
Build Distributed Application &
Scheduler
Build App case by case basis
Complicated Construction
40 E.g., AppLeS/MPI based
Total Cost:$???
Use Nimrod-G
Aggregate Job Submission
Aggregate View
90
80
70
60
50
40
30
20
10
0
East
West
North
South
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
41
Submit & Play!
Nimrod & Associated Family of Tools
Remote Execution Server
(on demand Nimrod Agent)
P-sweep App. Composition:
Nimrod/Enfusion
Resource Management and
Scheduling:
Nimrod-G Broker
Design Optimisations:
Nimrod-O
90
App. Composition and
80
70
Online Visualization:
Active Sheets 60
50
Grid Simulation in Java: 40
30
GridSim
20
10
Drug Design on Grid:
0
Virtual Lab
East
West
North
South
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
Upcoming?: HEPGrid (+U. Melbourne), GAVE(+Rutherford Appleton Lab)
42
Grid (Un)Aware Virtual Engineering
File Transfer Server
Nimrod/G : A Grid Resource Broker



43
A resource broker for managing and steering task
farming (parametric sweep) applications on
computational Grids based on deadline and
computational economy.
Key Features
 A single window to manage & control experiment
 Resource Discovery
 Resource Trading
 Scheduling & Predications
 Transportation of data & results
 Steering & data management
It allows to study the behaviour of some of the
output variables against a range of different input
scenarios.
A Glance at Nimrod-G Broker
Nimrod/G Client
Nimrod/G Client
Nimrod/G Client
Nimrod/G Engine
Schedule Advisor
Trading Manager
Grid
Store
Grid Dispatcher
Grid Explorer
Grid Middleware
TM
Globus, Legion, Condor, etc.
TS
GE
GIS
Grid Information Server(s)
RM & TS
RM & TS
G
RM & TS
C
L
G
Globus enabled node.
See HPCAsia 2000 paper!
44
L
Legion enabled
node.
RM: Local Resource Manager, TS: Trade Server
G
C L
Condor enabled node.
Nimrod/G Grid Broker Architecture
Legacy Applications
Customised Apps
(Active Sheet)
P-Tools (GUI/Scripting)
(parameter_modeling)
Monitoring and
Steering Portals
Nimrod Clients
XML?
Farming Engine
Meta-Scheduler
XML
Algorithm1
Programmable Entities Management
Resources
IP hourglass ?
Jobs Tasks
Schedule
Advisor
Channels
AlgorithmN
AgentScheduler
Agents
JobServer
Database
(Postgres)
Dispatcher & Actuators
Globus-A
Globus
Computers
45
...
Legion-A
Legion
...
Trading
Manager
P2P-A
Condor
Local Schedulers
PC/WS/Clusters
Grid
Explorer
Nimrod Broker
P2P
...
Storage
Condor/LL/Mosix/
GTS
Networks
Database
...
GMD
...
G-Bank
Instruments
Radio Telescope
Middleware
Fabric
Deadline
A Nimrod/G
Monitor
Cost
66 Arlington
Alexandria
Legion hosts
She na nd o a h
Rive r
64
64
81
Ra p p a ha nno c k Po to m a c
Rive r
Rive r
Ja m e s
Rive r
Roanoke
Richmond
Ap p o m a to x
Rive r
Hampton
Norfolk
Virginia Beach
Portsmouth Chesapeake
Newport News
77
VIRGINIA
85
Globus Hosts
Bezek is in both
Globus and Legion Domains
46
User Requirements: Deadline/Budget
47
Active Sheet: Spreadsheet
Processing on Grid
Nimrod
Proxy
Nimrod/G
48See HPC 2001 paper!
49
Nimrod/G Interactions
Resource
Discovery
Farming
Engine
Grid Info
servers
Scheduler
Grid Trade
Server
Dispatcher
Process
server
I/O
server
Resource
allocation
(local)
Queuing
System
Nimrod
Agent
User
process
File access
“Do this in 30min. for $10?”
50
Root node
Gatekeeper node
Computational node
Adaptive Scheduling Algorithms
Adaptive Scheduling
Algorithms
Time Minimisation
Cost Minimisation
None Minimisation
Execution Time
(not beyond deadline)
Minimise
Limited by deadline
Limited by deadline
Execution Cost
(not beyond budget)
Limited by budget
Minimise
Limited by budget
See HPDC AMS 2001 paper!
Discover Establish
Resources
Rates
Distribute Jobs
51
Compose &
Schedule
Discover
More
Resources
Evaluate &
Reschedule
Meet requirements ? Remaining
Jobs, Deadline, & Budget ?
Cost Model




Without cost ANY shared system
becomes un-managable
Charge users more for remote
facilities than their own
Choose cheaper resources before
more expensive ones
Cost units (G$) may be


52

Dollars
Shares in global facility
Stored in bank
Cost Matrix @ Grid site X

Non-uniform costing
Encourages use of
local resources first
User 1 1
Real accounting
system can control
machine usage
Machine 5

Machine 1

User 5 2
1
3
Resource Cost = Function (cpu, memory, disk, network, software, QoS, current demand, etc.)
53
Simple: price based on peaktime, offpeak, discount when less demand, ..
Deadline and Budget-based
Cost Minimization Scheduling
1.
2.
3.
54
Sort resources by increasing cost.
For each resource in order, assign as
many jobs as possible to the resource,
without exceeding the deadline.
Repeat all steps until all jobs are
processed.
Deadline-based Costminimization Scheduling


M - Resources, N - Jobs, D - deadline
Note: Cost of any Ri is less than any of Ri+1 …. Or Rm



Ct - Time when accessed (Time now)
Ti - Job runtime (average) on Resource i (Ri) [updated periodically]


RL: Resource List need to be maintained in increasing order of cost
Ti is acts as a load profiling parameter.
Ai - number of jobs assigned to Ri , where:

Ai = Min (No.Unassigned Jobs, No. Jobs Ri can complete by remaining
deadline)



ALG: Invoke Job Assignment() periodically until all jobs done.

Job Assignment()/Reassignment():


55
No.UnAssignedJobsi = Diff( N, (A1+…+Ai-1))
JobsRi consume = RemainingTime (D- Ct) DIV Ti
Establish ( RL, Ct , Ti , Ai ) dynamically – Resource Discovery.
For all resources (I = 1 to M) { Assign Ai Jobs to Ri , if required}
Deadline and Budget-based
Time Minimization Scheduling
1.
2.
3.
4.
56
For each resource, calculate the next
completion time for an assigned job, taking into
account previously assigned jobs.
Sort resources by next completion time.
Assign one job to the first resource for which
the cost per job is less than the remaining
budget per job.
Repeat all steps until all jobs are processed.
(This is performed periodically or at each
scheduling-event.)
Deadline and Budget-based
Time+Cost Min. Scheduling
1.
2.
3.
4.
57
Split resources by whether cost per job is less
than budget per job.
For the cheaper resources, assign jobs in
inverse proportion to the job completion time
(e.g. a resource with completion time = 5 gets
twice as many jobs as a resource with
completion time = 10).
For the dearer resources, repeat all steps
(with a recalculated budget per job) until all
jobs are assigned.
[Schedule/Reschedule] Repeat all steps until all
jobs are processed.
Evaluation of Scheduling
Heuristics
A Hypothetical Application on
WW Grid
World Wide Grid
World Wide Grid (WWG)
WW Grid
Australia
North America
ANL: SGI/Sun/SP2
USC-ISI: SGI
UVa: Linux Cluster
UD: Linux cluster
UTK: Linux cluster
Monash Uni.:
Nimrod/G
Linux cluster
Globus+Legion
GRACE_TS
Solaris WS
Globus/Legion
GRACE_TS
Asia/Japan
WW Grid
Internet
Tokyo I-Tech.:
ETL, Tuskuba
Linux cluster
Globus +
GRACE_TS
Chile: Cluster
59
Globus +
GRACE_TS
South America
Europe
ZIB/FUB: T3E/Mosix
Cardiff: Sun E6500
Paderborn: HPCLine
Lecce: Compaq SC
CNR: Cluster
Calabria: Cluster
CERN: Cluster
Pozman: SGI/SP2
Globus +
GRACE_TS
Experiment-1 Setup

Workload:




Deadline: 1 hrs. and budget: 800,000 units
Strategy: minimise cost and meet deadline
Execution Cost with cost optimisation


60
165 jobs, each need 5 minute of cpu time
AU Peaktime:471205 (G$)
AU Offpeak time: 427155 (G$)
Resources Selected & Price/CPU-sec.
61
Resource
Type &
Size
Owner and
Location
Grid
services
Peaktime
Cost (G$)
Offpeak
cost
Linux
cluster
(60 nodes)
Monash,
Australia
Globus/Condor
20
5
IBM SP2
nodes)
ANL, Chicago,
US
Globus/LL
5
10
Sun (8 nodes)
ANL, Chicago,
US
Globus/Fork
5
10
SGI (96 nodes)
ANL, Chicago,
US
Globus/Condor-G
15
15
SGI (10 nodes)
ISI, LA, US
Globus/Fork
10
20
(80
Execution @ AU Peak Time
Linux cluster - Monash (20)
12
Sun - ANL (5)
SP2 - ANL (5)
SGI - ANL (15)
SGI - ISI (10)
10
Jobs
8
6
4
2
Time (minutes)
62
54
52
51
49
47
46
44
43
41
40
38
37
36
34
33
31
30
28
27
25
24
22
21
20
19
17
15
14
12
10
9
8
6
4
3
1
0
0
Execution @ AU Offpeak Time
Linux cluster - Monash (5)
12
Sun - ANL (10)
SP2 - ANL (10)
SGI - ANL (15)
SGI - ISI (20)
10
Jobs
8
6
4
2
Time (minutes)
63
60
57
55
53
50
48
46
43
41
39
37
35
32
31
28
26
23
21
19
17
15
13
10
8
7
4
3
0
0
AU peak: Resources/Cost in Use
40
Resources (No. of CPUs) in Use
35
30
After the calibration phase, note
the difference in pattern of two graphs.
This is when scheduler stopped using
expensive resources.
25
20
15
10
5
500
0
51
47
44
41
38
54
400
Cost of Resources in Use
Time (in min.)
36
33
30
27
24
21
19
15
12
9
6
3
0
450
350
300
250
200
150
100
50
64
Time (in min.)
54
51
47
44
41
38
36
33
30
27
24
21
19
15
12
9
6
3
0
0
65
Time (in min.)
59
53
56
49
43
47
41
38
32
35
29
22
26
20
14
17
11
8
6
3
0
Cost of Resources in Use
Time (in min.)
59
56
53
49
47
43
41
38
35
32
29
26
22
20
17
14
11
8
6
3
0
Resources (No. of CPUs) in Use
AU offpeak: Resources/Cost in Use
30
25
20
15
10
5
350
0
300
250
200
150
100
50
0
Experiment-2 Setup

Workload:




Deadline: 2 hrs. and budget: 396000 units
Strategy: minimise time / cost
Execution Cost with cost optimisation




66
165 jobs, each need 5 minute of CPU time
Optimise Cost: 115200 (G$) (finished in 2hrs.)
Optimise Time: 237000 (G$) (finished in 1 hr.)
In this experiment: Time-optimised scheduling run
costs double that of Cost-optimised.
Users can now trade-off between Time Vs. Cost.
Resources Selected & Price/CPU-sec.
Resource &
Location
Grid services
& Fabric
Cost/CPU No. of Jobs Executed
sec.or
Time_Opt Cost_Op
unit
t.
Linux Cluster-Monash,
Melbourne, Australia
Globus, GTS, Condor
2
64
153
Linux-Prosecco-CNR,
Pisa, Italy
Globus, GTS, Fork
3
7
1
Linux-Barbera-CNR,
Pisa, Italy
Globus, GTS, Fork
4
6
1
Solaris/Ultas2
TITech, Tokyo, Japan
Globus, GTS, Fork
3
9
1
SGI-ISI, LA, US
Globus, GTS, Fork
8
37
5
Sun-ANL, Chicago,US
Globus, GTS, Fork
Total Experiment Cost (G$)
7
42
237000
4
115200
70
119
Time to Complete Exp. (Min.)
67
Scheduling for Time Optimization
Condor-Monash
Linux-Prosecco-CNR
Linux-Barbera-CNR
Solaris/Ultas2-TITech
SGI-ISI
Sun-ANL
12
No. of Tasks in Execution
10
8
6
4
2
68
Time (in Minute)
68
72
60
64
52
56
44
48
36
40
28
32
20
24
16
8
12
4
0
0
Scheduling for Cost Optimization
Condor-Monash
Linux-Prosecco-CNR
Linux-Barbera-CNR
Solaris/Ultas2-TITech
SGI-ISI
Sun-ANL
14
No. of Tasks in Execution
12
10
8
6
4
2
Time (in Minute)
69
10
2
10
8
11
4
96
90
84
78
72
66
60
54
48
42
36
30
24
18
6
12
0
0
Application Case Study
The Virtual Laboratory Project:
"Molecular Modelling for Drug Design"
on Peer-to-Peer Grid
Drug Design: Data Intensive
Computing on Grid



A Virtual Laboratory for
“Molecular Modelling for Drug
Design” on Peer-to-Peer Grid.
It provides tools for
examining millions of chemical
compounds (molecules) in the
Protein Data Bank (PDB) to
identify those having potential
use in drug design.
In collaboration with:

Kim Branson, Structural
Biology, Walter and Eliza Hall
Institute (WEHI)
71http://www.csse.monash.edu.au/~rajkumar/dd@home/
DesignDrug@Home Architecture
A Virtual Lab for “Molecular Modeling for Drug Design” on P2P Grid
Data Replica
Catalogue
Grid Market
Directory
“Give me list PDBs sources
Of type aldrich_300?”
“Screen 2K molecules
in 30min. for $10”
Grid Info.
Service
GTS
Resource
Broker
“mol.5 please?”
GTS
(RB maps suitable
Grid nodes and
Protein DataBank)
PDB2
GTS
GTS
PDB1
72
GTS
(GTS - Grid
Trade Server)
Software Tools







73
Molecular Modelling Tools (DOCK)
Parameter Modelling Tools (Nimrod/enFusion)
Grid Resource Broker (Nimrod-G)
Data Grid Broker
Protein Data Bank (PDB) Management and Intelligent Access
Tools
 PDB databse Lookup/Index Table Generation.
 PDB and associated index-table Replication.
 PDB Replica Catalogue (that helps in Resource Discovery).
 PDB Servers (that serve PDB clients requests).
 PDB Brokering (Replica Selection).
 PDB Clients for fetching Molecule Record (Data Movement).
Grid Middleware (Globus and GrACE)
Grid Fabric Management (Fork/LSF/Condor/Codine/…)
DOCK code*
(Enhanced by WEHI, U of Melbourne)







A program to evaluate the chemical and geometric
complementarities between a small molecule and a
macromolecular binding site.
It explores ways in which two molecules, such as a drug and an
enzyme or protein receptor, might fit together.
Compounds which dock to each other well, like pieces of a
three-dimensional jigsaw puzzle, have the potential to bind.
So, why is it important to able to identify small molecules which
may bind to a target macromolecule?
A compound which binds to a biological macromolecule may
inhibit its function, and thus act as a drug.
Thus disabling the ability of (HIV) virus attaching itself to
molecule/protein!
With system specific code changed, we have been able to
compile it for Sun-Solaris, PC Linux, SGI IRIX, Compaq
Alpha/OSF1
* Original Code: University of California, San Francisco: http://www.cmpharm.ucsf.edu/kuntz/
74
Dock input file
score_ligand
minimize_ligand
multiple_ligands
random_seed
anchor_search
torsion_drive
clash_overlap
conformation_cutoff_factor
torsion_minimize
match_receptor_sites
random_search
. . . . . .
. . . . . .
maximum_cycles
ligand_atom_file
receptor_site_file
score_grid_prefix
vdw_definition_file
chemical_definition_file
chemical_score_file
flex_definition_file
flex_drive_file
ligand_contact_file
ligand_chemical_file
ligand_energy_file
75
yes
yes
no
7
no
yes
0.5
3
yes
no
yes
1
S_1.mol2
ece.sph
ece
parameter/vdw.defn
parameter/chem.defn
parameter/chem_score.tbl
parameter/flex.defn
parameter/flex_drive.tbl
dock_cnt.mol2
dock_chm.mol2
dock_nrg.mol2
Molecule to
be screened
Parameterized Dock input file
76
score_ligand
minimize_ligand
multiple_ligands
random_seed
anchor_search
torsion_drive
clash_overlap
conformation_cutoff_factor
torsion_minimize
match_receptor_sites
random_search
. . . . . .
. . . . . .
maximum_cycles
ligand_atom_file
receptor_site_file
score_grid_prefix
vdw_definition_file
chemical_definition_file
chemical_score_file
flex_definition_file
flex_drive_file
ligand_contact_file
ligand_chemical_file
ligand_energy_file
$score_ligand
$minimize_ligand
$multiple_ligands
$random_seed
$anchor_search
$torsion_drive
$clash_overlap
$conformation_cutoff_factor
$torsion_minimize
$match_receptor_sites
$random_search
Molecule to be
screened
$maximum_cycles
${ligand_number}.mol2
$HOME/dock_inputs/${receptor_site_file}
$HOME/dock_inputs/${score_grid_prefix}
vdw.defn
chem.defn
chem_score.tbl
flex.defn
flex_drive.tbl
dock_cnt.mol2
dock_chm.mol2
dock_nrg.mol2
Dock PlanFile (contd.)
parameter database_name label "database_name" text select oneof "aldrich"
"maybridge" "maybridge_300" "asinex_egc" "asinex_epc" "asinex_pre"
"available_chemicals_directory" "inter_bioscreen_s"
"inter_bioscreen_n" "inter_bioscreen_n_300" "inter_bioscreen_n_500"
"biomolecular_research_institute" "molecular_science"
"molecular_diversity_preservation" "national_cancer_institute"
"IGF_HITS" "aldrich_300" "molecular_science_500" "APP" "ECE" default
"aldrich_300";
parameter score_ligand text default "yes";
parameter minimize_ligand text default "yes";
parameter multiple_ligands text default "no";
parameter random_seed integer default 7;
parameter anchor_search text default "no";
parameter torsion_drive text default "yes";
parameter clash_overlap float default 0.5;
parameter conformation_cutoff_factor integer default 5;
parameter torsion_minimize text default "yes";
parameter match_receptor_sites text default "no";
parameter random_search text default "yes";
. . . . . .
. . . . . .
parameter maximum_cycles integer default 1;
parameter receptor_site_file text default "ece.sph";
parameter score_grid_prefix text default "ece";
parameter ligand_number integer range from 1 to 200 step 1;
Molecules to be
screened
77
Dock PlanFile
task nodestart
copy ./parameter/vdw.defn node:.
copy ./parameter/chem.defn node:.
copy ./parameter/chem_score.tbl node:.
copy ./parameter/flex.defn node:.
copy ./parameter/flex_drive.tbl node:.
copy ./dock_inputs/get_molecule node:.
copy ./dock_inputs/dock_base node:.
endtask
task main
node:substitute dock_base dock_run
node:substitute get_molecule get_molecule_fetch
node:execute sh ./get_molecule_fetch
node:execute $HOME/bin/dock.$OS -i dock_run -o dock_out
copy node:dock_out ./results/dock_out.$jobname
copy node:dock_cnt.mol2 ./results/dock_cnt.mol2.$jobname
copy node:dock_chm.mol2 ./results/dock_chm.mol2.$jobname
copy node:dock_nrg.mol2 ./results/dock_nrg.mol2.$jobname
endtask
78
Nimrod/TurboLinux enFuzion GUI
tools for Parameter Modeling
79
Docking Experiment Preparation

Setup PDB DataGrid





Create Docking GridScore (receptor surface details) for a given
receptor on home node.
Pre-Staging Large Files required for Docking:



80
Index PDB databases
Pre-stage (all) Protein Data Bank (PDB) on replica sites
Start PDB Server
Pre-stage Dock executables and PDB access client on Grid nodes, if
required (e.g., dock.Linux, dock.SunOS, dock.IRIX64, and dock.OSF1
on Linux, Sun, SGI, and Compaq machines respectively). Use globusrcp.
Pre-stage/Cache all data files (~3-13MB each) representing receptor
details on Grid nodes.
This can can be done demand by Nimrod/G for each job, but few input
files are too large and they are required for all jobs). So, prestaging/caching at http-cache or broker level is necessary to avoid the
overhead of copying the same input files again and again!
Protein Data Bank



81
Databases consist of small molecules from
commercially available organic synthesis
libraries, and natural product databases.
There is also the ability to screen virtual
combinatorial databases, in their entirety.
This methodology allows only the required
compounds to be subjected to physical screening
and/or synthesis reducing both time and
expense.
Target Testcase


82
The target for the test
case: electrocardiogram
(ECE) endothelin
converting enzyme. This is
involved in “heart stroke”
and other transient
ischemia.
Is·che·mi·a : A decrease in
the blood supply to a
bodily organ, tissue, or
part caused by
constriction or obstruction
of the blood vessels.
“Screen 2K molecules
in 30min. for $10”
DataGrid Brokering
Nimrod/G
Computational
Grid Broker
1
“Screen
mol.5
please?”
Algorithm1
PDB Broker
AlgorithmN
“advise PDB
source?
“process
& send
results”
7
Data Replica
Catalogue
...
2
5
3
“PDB replicas
please?”
4
“selection &
advise: use
GSP4!”
“Is GSP4
healthy?”
6
“mol.5 please?”
PDB2
PDB
Service
83
GSP1
GSP2
Grid Info.
Service
GSP3
(Grid Service Provider)
GSP4
PDB
Service
GSPm
GSPn
Nimrod/G in Action:
Screening on World-Wide Grid
84
Any Scientific Discovery ? Did your
collaborator invent new drug for xxxx?
Not Yet
Anyway, checkout the
announcement of Nobel-prize
winners for next year
85
?
Conclude with a comparison
with the Electrical Grid………..
Where we are ????
Courtesy: Domenico Laforenza
Alessandro Volta in Paris in 1801 inside French
National Institute shows the battery while in
the presence of Napoleon I
Fresco by N. Cianfanelli (1841)
(Zoological Section "La Specula" of National History Museum of Florence
University)
What ?!?!
Oh, mon
Dieu !
This is a mad man…
88
….and in the future,
I imagine a
worldwide
Power (Electrical)
Grid …...
2001 - 1801 = 200 Years
89
Can we Predict its Future ?
” I think there is a world market for about five computers.”
Thomas J. Watson Sr., IBM Founder, 1943
90
What Enron, World Leader in Power and
Natural Gas Distribution Business, Think of
Economy Grid!...
-------- Original Message -------Subject: Your papers on Economics & Grid allocation
Date: Wed, 14 Mar 2001 12:10:20 -0800
From: Lance_Norskog@enron.net
To: rajkumar@csse.monash.edu.au, davida@csse.monash.edu.au,jon@csse.monash.edu.au
HelloI am researching mass computation infrastructures.
The company I work for, Enron, is a worldwide commodity company. Our
business model is to find mid-sized commodity markets that don't work like
large-scale commodity markets (like wheat, gold, orange juice, etc.) and to
restructure them. The division I work in is working to do this for
long-distance fiber optic bandwidth. Other divisions are pursuing metals,
paper pulp, etc. (We even have "weather derivatives", which is a betting
parlor for industries that depend far too much on hot or cold weather.
Somehow, this is legal!) So, my perspective on mass computation is from the
point of view of how to make it a large-scale market.
91
The papers you have published and are presenting concentrate on the
development of a "spot" market for the commodity of processor time. This
is only part of the economic picture. ( I just found "Calender based" among
your list of bidding parameters, but your analysis seems to be very
oriented to the spot market.)
What Enron, World Leader in Power and
Natural Gas Distribution Business, Think of
Economy Grid!...
Our customers are large corporations. They have a need not to solve some
particular problem now and then, but every day. For example, take a
department store chain that routes one million different items from
warehouses into department stores every day, and wants to do it in a
near-optimal way. They need to run this computation, with different input
vectors, every business day!
If that company is to commit to running its business with the Grid, it needs a
few guarantees around its Grid use:
1) complete reliability and availability
2) a known price for every use, set far in advance
3) an active spot market in case its regular supplier fails
4) enforceable contracts guaranteeing quality of service
This last is a killer: the quantity of processing power you get can't be
squishy. It has to be a measureable unit: "Java MIPS" is computer-driven.
Number 2, a known price for every use, is also missing from your analysis.
A large, active, "liquid" commodity market can supply not merely spot
market purchases, but also future needs at a price fixed today: "6 million
MIPS from midnight to 3 AM every day for six months, starting July 1" will
cost me this much per month. I can lock down my spreadsheet and know that
I won't be driven out of business by a temporary shortage, because my
supply is fully guaranteed.
92
What Enron, World Leader in Power and
Natural Gas Distribution Business, Think of
Economy Grid!...
Large commodity markets have the vast majority of their product change
hands under such long-term agreements, rather than on the spot market.
The strategy is to buy part of your needs in long-term contracts, part in
medium-term contracts, and most of the rest on short-term, just to avoid
getting stuck with 5 years from now with vast amounts of a commodity you
don't want. Every day you might have a missing 3-5% that you need to buy
on the spot market. (I'm in California. We're having so much trouble
because our utilities were banned from buying a heterogeneous basket of
contracts and were required to buy all electricity on the spot market.)
I think my point is that while economics is a fine metaphor for
making operational decisions in scheduling resources, those
numbers will not be visible to end customers. Instead, the customers will buy
blocks of resource for future delivery with pricing based on standard macro-economic
factors like interest rates, falling machine prices, rising electricity
The producers will use your economic-based techniques to
direct their day-to-day operations and to make the interproducer
spot market function.
prices, etc.
Lance Norskog
Sr. Software Engineer
93
Enron Broadband Systems
------------------------------------------------------------------------------------------------------
Conclusions






94
Grid Computing is emerging as a next generation
computing platform.
The use of economics paradigm for management of resources
in Grid Computing is essential to push Grid into mainstream
computing!
Adaptive, scalable, and easy-to-use systems and tools
are essential to make end-users life easier.
It is projected that the impact of World-Wide Grid on 21st
century economy will be similar to the impact made by
electric power grid on the 20th century economy.
To achieve this goal, in my humble opinion, Use Nimrod
Family of Tools (not to mention Nimrod-G Broker) along with
Globus, of course!
Enjoy excitements of World-Wide Grid Computing!
Download Software & Information

Nimrod & Parameteric Computing:


Economy Grid & Nimrod/G:


http://www.buyya.com/dd@home/
Grid Simulation (Java based):


http://www.buyya.com/ecogrid/
Virtual Laboratory/DesignDrug@Home:


http://www.csse.monash.edu.au/~davida/nimrod/
http://www.buyya.com/gridsim/
World Wide Grid testbed:


http://www.buyya.com/ecogrid/wwg/
Looking for new volunteers to grow 


95
Please contact me to barter your & our machines!
Want to build on our work/collaborate:

Talk to me now or email: rajkumar@csse.monash.edu.au
Acknowledgements

Special thanks to the following colleagues for
sharing ideas/works:






Unable to mention all names explicitly here,
however, their efforts are recognized by featuring
their work in this presentation.
 Colleagues from Asia/Japan, Europe: Italy, Germany,
Swiss, Poland, UK, US, & Chille for providing access
to their machines-->WWG testbed.

96
David Abramson, Monash University
Jack Dongarra, University of Tennessee
Wolfgang Gentzsch, Sun Microsystems
Jon Giddy, DSTC @ Monash University
Domenico Laforenza, CNR/CNUCE, Italy
Globus Team!
Thank You… Any ??
97
Further Information

Books:



IEEE Task Force on Cluster Computing


98
www.gridforum.org
IEEE/ACM CCGrid’xy: www.ccgrid.org


http://www.ieeetfcc.org
Global Grid Forum


High Performance Cluster Computing, V1, V2,
R.Buyya (Ed), Prentice Hall, 1999.
The GRID, I. Foster and C. Kesselman (Eds),
Morgan-Kaufmann, 1999.
CCGrid 2002, Berlin: ccgrid2002.zib.de
Grid workshop - www.gridcomputing.org
Further Information

Cluster Computing Info Centre:


Grid Computing Info Centre:


http://computer.org/dsonline/gc
Compute Power Market Project

99
http://www.gridcomputing.com
IEEE DS Online - Grid Computing area:


http://www.buyya.com/cluster/
http://www.ComputePower.com