Grid Tutorial
Cameron Kiddle
Grid Research Centre
University of Calgary
Acknowledgments
Rob Simmonds
Mark Fox
Phil Rizk
Abhishek Gaurav
Andrey Mirtchovski
Nayden Markatchev
Western Grid Summit - 2005
Grid Tutorial
2
Outline
Grid Computing Overview
Basic Grid Services

security, data management, resource
management, information services
Advanced Grid Services

meta schedulers, workflow managers, replica
location service
Future of Grid Computing
Western Grid Summit - 2005
Grid Tutorial
3
Grid Computing
Aimed at providing users with a single
system view of a set of distributed
resources that span administrative
domains
Aids in solving large scale problems
Aids in collaborative research efforts
Western Grid Summit - 2005
Grid Tutorial
4
Current Grid Projects
CERN LCG (LHC Computing Grid)
GriPhyN (Grid Physics Network)
NEES (Network for Earthquake
Engineering Simulation)
BioSimGrid
IVOA (International Virtual Observatory
Alliance)
Western Grid Summit - 2005
Grid Tutorial
5
WestGrid and Grid
Is WestGrid a computational grid?
Provides grid enabled resources



Security services
Data transfer tools
Job submission services
WestGrid resources can be part of
computational grids
Western Grid Summit - 2005
Grid Tutorial
6
Global Grid Forum (GGF)
Leads the global standardization effort for
grid computing
Open Grid Services Architecture (OGSA)
Western Grid Summit - 2005
Grid Tutorial
7
Open Grid Services Architecture
Infrastructure Services
Resource Management Services
Data Services
Context Services
Information Services
Self-Management Services
Security Services
Execution Management Services
Western Grid Summit - 2005
Grid Tutorial
8
Grid Middleware
Globus Toolkit


Globus Alliance
Open source
Legion



University of Virginia
Now commercial product developed by Avaki
Focuses on data grid / enterprise integration
Western Grid Summit - 2005
Grid Tutorial
9
Globus Toolkit
Collection of grid services to aid in
development of grid applications and
environments
GT-2 – pre standards (defacto standard)
GT-3 – Open Grid Services Infrastructure
(OGSI)
GT-4 – Web Services Resource
Framework (WSRF) – beta version
Western Grid Summit - 2005
Grid Tutorial
10
Basic Grid Services
Security Services

GSI (Grid Security Infrastructure)
Data Management

GridFTP
Resource Management

GRAM (Grid Resource Allocation and
Management)
Information Services

MDS (Monitoring and Discovery Service)
Western Grid Summit - 2005
Grid Tutorial
11
Grid Security Infrastructure (GSI)
Based on



Public key cryptography
X.509 certificates
Secure Socket Layer (SSL)
Provides


Mutual authentication
Single sign-on
Western Grid Summit - 2005
Grid Tutorial
12
Certificates
Used to identify and authenticate users
and services
Includes four primary pieces of information




Subject name
Public key
Identity of Certificate Authority (CA)
Digital signature of the Certificate Authority
Provides global name space
Encoded in X.509 certificate format (IETF)
Western Grid Summit - 2005
Grid Tutorial
13
Mutual Authentication
Process of two parties proving to each
other who they say they are
Both parties must trust CAs that signed
each other’s certificates
Each party sends certificate to other


Make sure certificate is valid
Make sure other party really is the party
identified in the certificate
Western Grid Summit - 2005
Grid Tutorial
14
Single Sign-On
Sign-on only once to computational grid
Use a proxy to avoid re-entering password




Consists of new certificate and private key
Certificate signed by owner, not CA
Less secure – limited lifetime
Both proxy certificate and owner’s certificate
sent for mutual authentication
grid-proxy-init
Western Grid Summit - 2005
Grid Tutorial
15
GSI Demo
Western Grid Summit - 2005
Grid Tutorial
16
MyProxy
Developed by NCSA (National Center for
Supercomputing Applications)
Credential repository
Allows proxy credential to be received
from any machine
Can allow trusted servers to renew proxy
credentials
myproxy-init, myproxy-get-delegation
Western Grid Summit - 2005
Grid Tutorial
17
MyProxy Demo
Western Grid Summit - 2005
Grid Tutorial
18
File Transfers
gsiscp


scp version based on GSI
Easy to use, but provides poor performance for large
file transfers
GridFTP





Based on FTP (File Transfer Protocol)
GSI security on control and data channels
Modification of TCP buffer sizes
Parallel transfers
Third party transfers
Western Grid Summit - 2005
Grid Tutorial
19
File Transfer Performance
Western Grid Summit - 2005
Grid Tutorial
20
File Transfer Demo
Western Grid Summit - 2005
Grid Tutorial
21
Grid Resource Allocation and
Management (GRAM)
Provides a single standard interface for
remote job submission and resource
management
Requires users and application developers
to learn only one method to gain access to
a large variety of local management
systems
globus-job-run, globus-job-submit
Western Grid Summit - 2005
Grid Tutorial
22
Job Submission Demo
Western Grid Summit - 2005
Grid Tutorial
23
Monitoring and Discovery Service
(MDS)
Provides information about availability and
status of resources
Uses LDAP (Lightweight Directory Access
Protocol) to access this information
Grid Resource Information Service (GRIS)

Provides resource information
Grid Index Information Service (GIIS)

Provides caching services and search
capabilities
Western Grid Summit - 2005
Grid Tutorial
24
Advanced Grid Services
Meta-schedulers
Workflows
Replica Location Service (RLS)
Abstract Workflows
Western Grid Summit - 2005
Grid Tutorial
25
Meta-Schedulers
Condor-G (Condor)

University of Wisconsin-Madison
Moab Grid Scheduler (Silver)

Cluster Resources Inc.
Western Grid Summit - 2005
Grid Tutorial
26
Condor-G
Can submit and manage jobs at multiple
sites
Uses Globus mechanisms to communicate
with resources at different sites
Can advertise sites that jobs can be
submitted to through Class Ads
Western Grid Summit - 2005
Grid Tutorial
27
Class Ad
Used to represent characteristics and
constraints of resources and jobs
Consists of a list of attributes (name/value
pairs)
Used in matchmaking of jobs to resources
Western Grid Summit - 2005
Grid Tutorial
28
Condor-G Demo
Western Grid Summit - 2005
Grid Tutorial
29
Workflows
C1
E1
A
B
C2
D
E2
C3
Western Grid Summit - 2005
Grid Tutorial
30
DAGMan
Directed Acyclic Graph Manager
Node = program
Edge = dependency
Monitors Condor log files to enforce
ordering of dependencies
Western Grid Summit - 2005
Grid Tutorial
31
DAGMan Input File
Programs in the DAG
Pre/post processing scripts
Dependencies
Number of times to retry
Macro definitions
Western Grid Summit - 2005
Grid Tutorial
32
Sample DAGMan Input File
JOB A A.condor
A
JOB B B.condor
JOB C C.condor
JOB D D.condor
B
C
PARENT A CHILD B C
D
PARENT B C CHILD D
Western Grid Summit - 2005
Grid Tutorial
33
Rescue DAGs
Aids in resubmission of uncompleted
portions of workflows
Produced if DAGMan can not make any
further progress in a DAG due to node
failures
Same as original DAG but nodes that
successfully completed before are marked
as DONE
Western Grid Summit - 2005
Grid Tutorial
34
Life3D Workflow
Animation
Life3D
Simulation
Rendering
Western Grid Summit - 2005
Grid Tutorial
35
Life3D Workflow - II
blackhole (SFU)
2. Life3D Simulation
1.
lattice (UofC)
Data Storage
Grid Research Center
WestGrid
3.
grc15
7.
octarine
5.
4. Rendering
6. Animation
Western Grid Summit - 2005
Grid Tutorial
36
Life3D Workflow - III
grc15
blackhole
Stage
Data
octarine
lattice
Animation Stage
Data
Life3d
Simulation
Stage
Data
Western Grid Summit - 2005
blackhole
Rendering
Grid Tutorial
Stage
Data
37
DAGMan Demo
Western Grid Summit - 2005
Grid Tutorial
38
Life3D Animation
Western Grid Summit - 2005
Grid Tutorial
39
Replica Location Service (RLS)
Co-developed by Globus and DataGrid projects
Provides a mechanism for registering and
locating replicas
Maintains mappings between logical names and
target names


Logical names – unique identifiers for data content
that may have one or more physical replicas
Target name – physical location of a replica or other
logical names representing the data
Western Grid Summit - 2005
Grid Tutorial
40
Chimera Virtual Data System
Part of the GriPhyN project
Distributed with the Virtual Data Toolkit
Keeps track of how data products are derived
Partial
Workflow
Abstract
Workflow
Chimera
(VDL)
Virtual Data Language
Concrete
Workflow
Pegasus
(DAG)
(DAX)
DAG XML
RLS
TC
MDS
(Transformation Catalog)
Western Grid Summit - 2005
Grid Tutorial
41
Chimera
Input - Virtual Data Language (VDL)


Transformation – an executable program
Derivation – an execution of the
transformation along with logical input and
output file names
Output – Abstract Workflow (DAX)

Workflow activities independent of specific
grid resources
Western Grid Summit - 2005
Grid Tutorial
42
Pegasus
Input – Abstract Workflow
Output – Concrete Workflow (DAG)

Workflow activities assigned to specific Gird
resources
Uses RLS to determine data file locations
Uses Transformation Catalog to map logical
transformations to physical locations
Uses MDS to determine available resources
Western Grid Summit - 2005
Grid Tutorial
43
Chimera Example
infile
(blackhole)
transitfile
add10
add20
(lattice)
(nexus)
Western Grid Summit - 2005
Grid Tutorial
outfile
(blackhole)
44
Chimera Demo
Western Grid Summit - 2005
Grid Tutorial
45
Summary
Basic Grid Services
Advanced Grid Services
Security Services (GSI)



Meta-scheduler
grid-proxy-init
myproxy-init
myproxy-get-delegation

Concrete Workflows
Data Management
(GridFTP)


globus-url-copy

DAGMan
Replica Location Service
(RLS)
Resource Management
(GRAM)

Condor-G
globus-job-run
globus-job-submit
Abstract Workflows

Chimera/Pegasus
Information Services

MDS
Western Grid Summit - 2005
Grid Tutorial
46
Future of Grid Computing
Integration of Web Services and Grid Services
Web Services Resource Framework (GT-4)
Standards are more established


Involvement of more developers
Better tools
Grid-enabled applications


Common library interface
“run_job_on_grid”
Wider spread use
Western Grid Summit - 2005
Grid Tutorial
47
Contact Information
Cameron Kiddle
kiddlec@cpsc.ucalgary.ca
http://grid.ucalgary.ca/
http://pages.cpsc.ucalgary.ca/~kiddlec/
Western Grid Summit - 2005
Grid Tutorial
48