Introduction to the current
EDG Testbed Software
Kraków, December 2002
Steve Fisher s.m.fisher@rl.ac.uk – RAL
on behalf of
The European DataGrid Project Team
http://www.edg.org/
The European DataGrid
 Funded

by the European Union
Jan 1, 2001 - Dec 31, 2003
 Develop,
implement and exploit a large-scale data and CPU-oriented
computational GRID.
 Develop
middleware, in collaboration with some of the leading
centres of competence in GRID technology.
 Complement,
and help to coordinate at a European level, several ongoing national GRID projects.
The EDG Testbed Software - 2
The EDG Main Partners

CERN – International (Switzerland/France)

CNRS - France

ESA/ESRIN – International (Italy)

INFN - Italy

NIKHEF – The Netherlands

PPARC - UK
The EDG Testbed Software - 3
EDG Assistant Partners
Industrial Partners
• Datamat (Italy)
• IBM-UK (UK)
• CS-SI (France)
Research and Academic Institutes
• CESNET (Czech Republic)
• Commissariat à l'énergie atomique (CEA) – France
• Computer and Automation Research Institute,
Hungarian Academy of Sciences (MTA SZTAKI)
• Consiglio Nazionale delle Ricerche (Italy)
• Helsinki Institute of Physics – Finland
• Institut de Fisica d'Altes Energies (IFAE) - Spain
• Istituto Trentino di Cultura (IRST) – Italy
• Konrad-Zuse-Zentrum für Informationstechnik Berlin - Germany
• Royal Netherlands Meteorological Institute (KNMI)
• Ruprecht-Karls-Universität Heidelberg - Germany
• Stichting Academisch Rekencentrum Amsterdam (SARA) – Netherlands
• Swedish Research Council - Sweden
The EDG Testbed Software - 4
EDG structure: work packages

WP1: Work Load Management System

WP2: Data Management

WP3: Information and Monitoring

WP4: Fabric Management

WP5: Storage Element

WP6: Testbed and demonstrators

WP7: Network Monitoring

WP8: High Energy Physics

WP9: Earth Observation

WP10: Biology

WP11: Dissemination

WP12: Management
}
Applications
The EDG Testbed Software - 5
Current EDG Testbed
NorduGrid
Testbed1
EDG sites
Reference site:
CERN
Manchester
NIKHEF
RAL
Karlsruhe
CERN
Lyon
Barcelona
Madrid
Lisboa
NorduGrid:
• Bergen
• Copenhagen
• Helsinki
• Lund
• Oslo
• Stockholm
• Uppsala
Italy:
• Bologna
• Cagliari
• Catania
• Milano
• Napoli
• Padova
• Parma
• Pisa
• Roma
• Torino
The EDG Testbed Software - 6
Security: Authentication/Authorization

Authentication

Who you are


users identified by certificates signed by a CA
Authorization

What you are allowed to do

based on membership of Virtual Organizations (VO).
The EDG Testbed Software - 7
Certificate Request
grid-cert-request
user
cert-request
once inVO
every twothree years
The EDG Testbed Software - 8
Requesting a Certificate

grid-cert-request
A certificate request and private key is being created.
[...]
Using configuration from /usr/local/grid/globus/etc/globus-user-ssleay.conf
Generating a 1024 bit RSA private key
[...]
A private key and a certificate request has been generated with the subject:
/O=Grid/O=CERN/OU=cern.ch/CN=Akos Frohner
[...]
Your private key is stored in .../.globus/userkey.pem
Your request is stored in .../.globus/usercert_request.pem
Please e-mail the certificate request to the CERN CA
cat .../.globus/usercert_request.pem | mail cern-globus-ca@cern.ch
Your certificate will be mailed to you within two working days.
The EDG Testbed Software - 9
Certificate Signing
CA
grid-cert-request
user
cert signing
cert-request
certificate
The EDG Testbed Software - 10
Registration/Authorization
User registration in an EDG Virtual Organisation
 convert

your certificate:
openssl pkcs12 –export –in ~/.globus/usercert.pem –inkey
~/.globus/userkey.pem –out user.p12 –name ’Joe Smith’
 import
your certificate in your browser
 sign
the usage guidelines:
https://marianne.in2p3.fr/cgi-bin/datagrid/register/account.pl
 ask
an account from your VO administrator by email
-> You are registered in the VO server and have a user account.
The EDG Testbed Software - 11
Registration
user
certificate
cert.pkcs12
Account Registration
convert
registration
VO
once for the lifetime
of the VO – you may
change the certificate
keys!
Usage guidelines
The EDG Testbed Software - 12
Starting a Session
user
certificate
cert.pkcs12
proxy-cert
grid-proxy-init
every 12/24
hours
The EDG Testbed Software - 13
Usage
You must have a valid certificate from a trusted CA!

“login”: grid-proxy-init
short lifetime certificate: 24 hours
Enter PEM pass phrase:
...........................+++++
....................................+++++

checking the proxy: grid-proxy-info -subject
/O=Grid/O=CERN/OU=cern.ch/CN=Akos Frohner/CN=proxy
-> use

the grid services
“logout”: grid-proxy-destroy
The EDG Testbed Software - 14
Configuration on the Server
CA
cert signing
grid-cert-request
service
cert/crl update
host-request
host-cert
ca-certificate
crl
crl automatically
updated
periodically
The EDG Testbed Software - 15
Authorization Information
service
host-cert
ca-certificates
crls
VO-server
mkgridmap
gridmap
automatically
updated
periodically
The EDG Testbed Software - 16
Using a Service
service
user
host-cert
certificate
ca-certificates
cert.pkcs12
crls
gridmap
proxy-cert
grid-proxy-init
host/proxy certs exchanged
The EDG Testbed Software - 17
EDG Logical Machine Types
1. User Interface (UI)
4. Computing Element (CE)

Gatekeeper
(Front-end Node)

2. Resource Broker (RB)
3. Information Service (IS)
Worker Nodes (WN)
5. Storage Element (SE)
6. Replica Catalog (RC)
The EDG Testbed Software - 18
Information Systems overview
The
aim of the Information and Monitoring Service is to
deliver a flexible infrastructure that provides
information on


the EU DataGrid itself
grid applications
EDG info systems are based upon Globus MDS
(Metacomputing Directory Service or Monitoring and
Discovery Service as it is now called)


Based on OpenLDAP, a hierarchical database
The
information system is currently used mainly by the
middleware.

You can use it to find out what is going on
The EDG Testbed Software - 19
LDAP attributes
A
schema describes the attributes and the types of the attributes
associated with data objects
 Example






- some attributes of SiteInfo:
siteName: RALDEV
sysAdminContact: grid.sysadmin@rl.ac.uk
userSupportContact: grid.support@rl.ac.uk
siteSecurityContact: grid.security@rl.ac.uk
dataGridVersion: 1.2
InstallationDate: 20020704142800Z
The EDG Testbed Software - 20
LDAP hierarchy
 Lightweight
Directory Assess Protocol (LDAP) offers a hierarchical
view of information
 The
objects are arranged in a Directory Information Tree (DIT)
 One
or more attributes represent the Relative Distinguished Name
(RDN)
 An

object is identified by its Distinguished name
This is its RDN with the Distinguished name of its parent
The EDG Testbed Software - 21
RDNs and DNs
site

RDN

DN

Site


SE
SE


supported
protocols

seId=dev02.hepgrid.clrc.ac.uk
Protocols

seProtocol=gridftp

seProtocol=rfio

seProtocol=file
SE


Mds-Vo-name=ral-dev,Mds-Voname=uk,o=Grid
seId=dev02.hepgrid.clrc.ac.uk,MdsVo-name=ral-dev,Mds-Voname=uk,o=Grid
Protocols



seProtocol=gridftp,
seId=dev02.hepgrid.clrc.ac.uk,MdsVo-name=ral-dev,Mds-Voname=uk,o=Grid
seProtocol=rfio,
seId=dev02.hepgrid.clrc.ac.uk,MdsVo-name=ral-dev,Mds-Voname=uk,o=Grid
seProtocol=file,
seId=dev02.hepgrid.clrc.ac.uk,MdsVo-name=ral-dev,Mds-Voname=uk,o=Grid
The EDG Testbed Software - 22
MDS GRISs & GIISs

Information providers are scripts which when invoked by the LDAP server
make available the desired information





Within MDS the EDG information providers are invoked by a local LDAP
server, the Grid Resource Information Server (GRIS)
“Aggregate directories”, Grid Information Index Servers (GIIS), are used
to group resources
The GRISs use soft state registration to register with one or more GIISs
The GIIS can then act as a single point of contact for a number of
resources


Information is cached by the server to improve performance
A GIIS may represent a site, country, virtual organization, etc.
In turn a GIIS may register with another GIIS
The EDG Testbed Software - 23
EDG Information Providers & the
Directory Information Tree
site
computing
element
storage
element
site information
network information
between this and other
sites
status
storage elements that are
close (not necessarily at the
same site)
supported
protocols
file statistics
The EDG Testbed Software - 24
EDG GRIS/GIIS Hierarchy

There is a top level datagrid GIIS to
which all of the country GIISs register
datagrid

countryA
countryB

siteA
siteB
siteC
siteD

information
providers
Each country has a GIIS to which all of
the site GIISs register
information
providers
information
providers
information
providers
Each Site has a Grid Information Index
Server (GIIS) which acts as a single
point of contact for all of the sites
resources. The GRISs register with
their site GIIS
Information providers publish
information to a local LDAP server known
as a Grid Resource Information Server
(GRIS)
The EDG Testbed Software - 25
EDG Information Providers
 The




EDG have produced information providers:
Site information
The Computing Element
The Storage Element
Network Monitoring
 All
of the EDG data objects are dynamic, they have a time stamp and
a time to live (used by the cache mechanism) associated with them
The EDG Testbed Software - 26
Siteinfo
in=siteinfo,Mds-Vo-name=ral-dev,Mds-Vo-name=uk,o=Grid
objectClass: SiteInfo
objectClass: DataGridTop
objectClass: DynamicObject
siteName: RALDEV
sysAdminContact: grid.sysadmin@rl.ac.uk
userSupportContact: grid.support@rl.ac.uk
siteSecurityContact: grid.security@rl.ac.uk
dataGridVersion: 1.2
installationDate: 20020704142800Z
The EDG Testbed Software - 27
Computing Element
ceId=dev01.hepgrid.clrc.ac.uk:2119/jobmanagerpbs-M,hn=dev01.hepgrid.clrc.ac.uk,Mds-Voname=ral-dev,Mds-Vo-name=uk,o=Grid
objectClass: DataGridTop
objectClass: ComputingElement
CEId: dev01.hepgrid.clrc.ac.uk:2119/jobmanagerpbs-M
GlobusResourceContactString:dev01.hepgrid.clrc.ac.
uk:2119/jobmanagerpbs:/O=Grid/O=UKHEP/CN=dev01.hepgrid.clrc.a
c.uk
GRAMVersion: ?
Architecture: intel
OpSys: RH 6.2
MinPhysicalMemory: 258
MinLocalDiskSpace: 2048
TotalCPUs: 1
FreeCPUs: 1
NumSMPs: 0
MinSPUProcessors: 0
MaxSPUProcessors: 0
TotalJobs: 0
RunningJobs: 0
IdleJobs: 0
MaxTotalJobs: 1
MaxRunningJobs: 1
WorstTraversalTime: 108000
EstimatedTraversalTime: 0
Active: TRUE
Priority: 20
MaxCPUTime: 108000
MaxWallClockTime: 432000
AverageSI00: 300
MinSI00: 300
MaxSI00: 300
AuthorizedUser:/O=Grid/O=UKHEP/OU=hepgrid.cl
rc.ac.uk/CN=Tim Eves
AuthorizedUser:/O=Grid/O=UKHEP/OU=hepgrid.cl
rc.ac.uk/CN=Tim Folkes
RunTimeEnvironment: RALDEV
AFSAvailable: FALSE
OutboundIP: TRUE
InboundIP: FALSE
QueueName: M
LRMSType: PBS
LRMSVersion: OpenPBS_2.3
The EDG Testbed Software - 28
Querying the Information & Monitoring
Service
 Queries
can be posed to the current Information and Monitoring Service
using LDAP search commands
 An
LDAP search consists of the following components
$ldapsearch\
-x\
-H ldap://lxshare0225.cern.ch:2135\
-b 'Mds-Vo-name=datagrid,o=grid\
'objectclass=ComputingElment‘\
CEId FreeCPUs \
-s base|one|sub
“simple” authentication
uniform resource identifier
base distinguished name for search
filter
attributes to be returned
scope of the search specifying just
the base object, one-level or the
complete subtree
The EDG Testbed Software - 29
Querying the GRIS/GIIS Hierarchy

Mds-Vo-name=datagrid,o=grid


Mds-Vo-name
=datagrid
Mds-Vo-name=siteB, Mds-Voname=countryA,Mds-Voname=datagrid,o=grid


Mds-Vo-name
=countryB

Mds-Vo-name
=siteB
Mds-Vo-name
=siteC
This will look at all the data from siteB
Mds-Vo-name=siteB,o=grid

Mds-Vo-name
=siteA
This will look at all the data from
countryA
Mds-Vo-name=siteB,Mds-Voname=countryA,o=grid


This will look at all the data from siteB
Mds-Vo-name=countryA,o=grid

Mds-Vo-name
=countryA
This will look at all the data
This will look at all the data from siteB
Mds-Vo-name
=siteD
The EDG Testbed Software - 30
The EDG WMS



The user interacts with GRID via a Workload Management System
The Goal of WMS is the distributed scheduling and resource
management in a GRID environment.
What does it allow GRID users to do?
To submit their jobs
To execute them
To get information about their status
To retrieve their output

The WMS tries to optimize the usage of resources
The EDG Testbed Software - 31
WMS Components

WMS is currently composed of the following parts:
1. User Interface (UI) : access point for the user to the GRID
2. Resource Broker (RB) : the broker of GRID resources, performing the
match-making
3. Job Submission System (JSS) : provides a reliable submission system
4. Information Index (II) : a specialized Globus GIIS (LDAP server) used
by the Resource Broker as a filter to the information service (IS) to
select resources
5. Logging and Bookkeeping services (LB) : store Job Info available for
users to query
The EDG Testbed Software - 32
WMS UI Commands

dg-job-submit
submits a job

dg-job-list-match
lists resources matching a job description

dg-job-cancel
cancels a given job

dg-job-status
displays the status of the job (submitted, waiting, ready, scheduled, running,
outputready, aborted, cleared)

chkpt, done,
dg-job-get-output
returns the job-output to the user

dg-job-get-logging-info
displays logging information about submitted jobs

dg-job-id-info
is a utility for the user to display job info in a formatted style
The EDG Testbed Software - 33
Example of UI Command Options

dg-job-submit –r <res_id> –n <user e-mail address> -c <config
file> -o <output file> <job.jdl>
-r the job is submitted by the RB directly to the computing element identified by <res_id>
-n an e-mail message containing basic information regarding the job (status and
identification) is sent to the specified <e-mail address> when the job enters one of the
following status:
DONE or ABORTED
READY
RUNNING
-c the configuration file <config file> is pointed by the UI instead of the standard
configuration file
-o the generated dg_jobId is written in the <output file>

dg-job-status –i <input file> (or dg_jobId)
-i the bookkeeping information about dg_jobId contained in the <input file> are displayed
The EDG Testbed Software - 34
Job Description Language (JDL)
 Mandatory for every single JDL file:
•
•
Executable (contains the command name)
Other attributes:
•
InputSandbox
•
OutputSandbox
 Mandatory for JDL file dealing with Data Management:
•
ReplicaCatalog (contains the Replica Catalog Identifier)
•
DataAccessProtocol (contains the protocol or the list of protocols which the
application is able to speak with for accessing InputData on a given SE)
If InputData contains at least one PFN and no LFNs, only DataAccessProtocol is
mandatory.
If InputData contains at least one LFN, both ReplicaCatalog and
DataAccessProtocol are mandatory.
The EDG Testbed Software - 35
Example JDL File
Executable = “gridTest”;
InputData = “LF:testbed0-00019”;
ReplicaCatalog = “ldap://sunlab2g.cnaf.infn.it:2010/ \
rc=WP2 INFN Test, dc=infn, dc=it”;
DataAccessProtocol = “gridftp”;
StdError = “stderr.log”;
StdOutput = “stdout.log”;
OutputSandbox = {“stderr.log”, “stdout.log”};
InputSandbox = {“home/joda/test/gridTest”};
Rank = “other.MaxCpuTime”;
Requirements = other.Architecture==“INTEL” && \
other.OpSys==“LINUX” && other.FreeCpus >=4;
The EDG Testbed Software - 36
A Job Submission Example
UI
JDL
Replica
Catalogue
(RC)
Information
Service (IS)
Resource
Broker (RB)
Logging &
Book-keeping
(LB)
Job Submission
Service (JSS)
Storage
Element
(SE)
Compute
Element CE)
The EDG Testbed Software - 37
A Job Submission Example
UI
JDL
Input Sandbox
Replica
Catalogue
(RC)
Job Status
Information
Service (IS)
submitted
Job Submit
Event
Resource
Broker (RB)
Logging &
Book-keeping
(LB)
Job Submission
Service (JSS)
Storage
Element
(SE)
Compute
Element (CE)
The EDG Testbed Software - 38
A Job Submission Example
UI
JDL
Replica
Catalogue
(RC)
Job Status
Information
Service (IS)
submitted
waiting
Resource
Broker (RB)
Logging &
Book-keeping
(LB)
Job Submission
Service (JSS)
Storage
Element
(SE)
Compute
Element (CE)
The EDG Testbed Software - 39
A Job Submission Example
UI
JDL
Replica
Catalogue
(RC)
Job Status
Information
Service (IS)
submitted
waiting
ready
Resource
Broker
(RB)
Logging &
Book-keeping
(LB)
Job Submission
Service (JSS)
Storage
Element
(SE)
Compute
Element (CE)
The EDG Testbed Software - 40
A Job Submission Example
UI
JDL
Replica
Catalogue
(RC)
Information
Service (IS)
Job Status
submitted
waiting
ready
scheduled
Resource
Broker (RB)
Logging &
Book-keeping
(LB)
BrokerInfo
Storage
Element
(SE)
Job Submission
Service
(JSS)
Compute
Element (CE)
The EDG Testbed Software - 41
A Job Submission Example
UI
JDL
Replica
Catalogue
(RC)
Job Status
Information
Service (IS)
submitted
waiting
ready
Input Sandbox
scheduled
Resource
Broker (RB)
Logging &
Book-keeping
(LB)
Job Submission
Service (JSS)
running
Storage
Element
(SE)
Compute
Element (CE)
The EDG Testbed Software - 42
A Job Submission Example
Replica
Catalogue
(RC)
UI
JDL
Job Status
Information
Service (IS)
submitted
waiting
ready
scheduled
Resource
Broker (RB)
Logging &
Book-keeping
(LB)
Job Submission
Service (JSS)
Job Status
running
Storage
Element
(SE)
Compute
Element (CE)
The EDG Testbed Software - 43
A Job Submission Example
Replica
Catalogue
UI
JDL
Job Status
Information
Service
submitted
waiting
ready
scheduled
Resource
Broker
running
Storage
Element
Logging &
Book-keeping
done
Job Submission
Service
Job Status
Compute
Element
The EDG Testbed Software - 44
A Job Submission Example
Replica
Catalogue
UI
JDL
Job Status
submitted
Information
Service
waiting
ready
scheduled
Resource
Broker
running
Storage
Element
Logging &
Book-keeping
Job Submission
Service
Output Sandbox
Job Status
done
outputready
Compute
Element
The EDG Testbed Software - 45
A Job Submission Example
Replica
Catalogue
(RC)
UI
JDL
Job Status
submitted
Information
Service (IS)
waiting
ready
scheduled
Output Sandbox
Resource
Broker (RB)
Logging &
Book-keeping
(LB)
Job Submission
Service (JS)
running
Storage
Element
(SE)
done
outputready
Compute
Element (CE)
cleared
The EDG Testbed Software - 46
EDG Data Management Tools
 Tools





for
Locating data
Copying data
Managing and replicating data
Meta Data management
On EDG Testbed you have




EDG Replica Catalog
globus-url-copy (GridFTP)
EDG Replica Manager
Grid Data Mirroring Package (GDMP)
The EDG Testbed Software - 47
EDG Replica Catalog

Based upon the Globus LDAP Replica Catalog (will be replaced by RLS)

Stores LFN/PFN mappings and additional information (e.g. filesize):



Physical File Name (PFN): host + full path & and file name
Logical File Name (LFN): logical name that may be resolved to PFNs
LFN : PFN = 1 : n

Only files on storage elements may be registered

Each VO has a specific storage dir on an SE

Example PFN: lxshare0222.cern.ch/flatfiles/SE1/iteam/file1.dat
host

storage dir
LFN must be full path of file starting from storage dir
LFN of above PFN: file1.dat
The EDG Testbed Software - 48
EDG Replica Catalog
 API
and command line tools

addLogicalFileName

getLogicalFileName

deleteLogicalFileName

getPhysicalFileName

addPhysicalFileName

deletePhysicalFileName

addLogicalFileAttribute

getLogicalFileAttribute

deleteLogicalFileAttribute
http://cmsdoc.cern.ch/cms/grid/userguide/gdmp-3-0/node85.html
The EDG Testbed Software - 49
globus-url-copy
 Low
level tool for secure copying
globus-url-copy <protocol>://<source file> \
<protocol>://<destination file>
 Main


Protocols:
gsiftp – for secure transfer, only available on SE and CE
file – for accessing files stored on the local file system on e.g. UI, WN
globus-url-copy file://`pwd`/file1.dat \
gsiftp://lxshare0222.cern.ch/ \
flatfiles/SE1/EDGTutorial/file1.dat
The EDG Testbed Software - 50
The EDG Replica Manager

Extends the Globus replica manager

Client side tool

Allows replication (copy) and registering of files in RC

Keeps RC consistent with stored data.
The EDG Testbed Software - 51
The Replica Manager APIs

(un)registerEntry(LogicalFileName lfn,
FileName source)


Replica Catalogue operations only - no file transfer
copyFile(FileName source,
FileName destination,
String protocol)


allows for third-party transfer
transfer between:

two StorageElements or

ComputingElement and Storage Element

Space management policies under development
The EDG Testbed Software - 52
The Replica Manager APIs

copyAndRegisterFile(LogicalFileName lfn,
FileName source,
FileName destination,
String protocol)

third-party transfer but :
files can only be registered in Replica Catalogue if destination PFN contains a
valid SE

replicateFile(LogicalFileName lfn,
FileName source,
FileName destination,
String protocol)

deleteFile(LogicalFileName lfn,
FileName source)
The EDG Testbed Software - 53
 based
on CMS requirements for replicating Objectivity files for
High Level Trigger studies
 production
prototype project for evaluating Grid technologies
(especially Globus)
 http://cern.ch/GDMP
The EDG Testbed Software - 54
Overview of Components
EDG Replica Catalogue
GDMP client
Site1
Site2
Site3
The EDG Testbed Software - 55
Subscription Model

All the sites that
subscribe to a particular
site get notified
whenever there is an
update in its catalog.
Site 1
Site 2
Subscriber
list
Subscriber
list
subscribe
subscribe
Site 3
The EDG Testbed Software - 56
Export / Import Catalogue

Export Catalog



Site 1
Site 2
export
catalog
export
catalog
Import Catalog



information about the new files
produced .
is published
information about the files which
have been published by other sites
but not yet transferred locally
As soon as the file is transferred
locally, it is removed from the
import catalogue.
1)register, publish
new files
1) get info about
new files
import
catalog 3) delete files
Possible to pull the information
about new files into your import
catalogue.
Site 3
2) transfer files
2) transfer files
The EDG Testbed Software - 57
Usage

gdmp_ping


gdmp_host_subscribe


get/put all the files from the import catalogue – update RC
gdmp_remove_local_file


send information of newly created files to subscribed hosts (no real data
transfer) – update RC
gdmp_replicate_get - gdmp_replicate_put


Registers a file in local file catalogue but NOT in Replica Catalogue (RC)
gdmp_publish_catalogue


first thing to be done by a site
gdmp_register_local_file


Ping a GDMP server and get its status
Delete a local file and update RC
gdmp_get_catalogue

Get remote catalogue contents – for error recovery
The EDG Testbed Software - 58
GDMP vs. EDG Replica Manager

GDMP








Replicates sets of files
Replication between SEs

Replica Manager


Replicates single files
Replication between SEs, CEs to SE.
Mass storage interface
File size as logical attribute
Subscription model
Event notification
CRC file size check
Support for Objectivity
The EDG Testbed Software - 59