EU DataGrid and OGSA
Intentions and Plans how to use Grid
Services
Peter Z. Kunszt
European DataGrid Data Management
WP2
P.Kunszt
OGSA 31.5.2002
1
The DataGrid Project
The European DataGrid is a project
funded by the European Union to set up
a computational and data-intensive grid
of resources for the analysis of data
from scientific exploration
P.Kunszt
OGSA 31.5.2002
2
DataGrid Applications
• Provide production quality testbeds, using real-world applications with
real data:
• High Energy Physics
—process the huge amount of data from LHC
experimentations
• Biology and Medical Imaging
—sharing of genomic databases for the benefit of
international cooperation
—processing of medical images for medical
collaborations
• Earth Observations
—access and analysis of atmospheric ozone data
collected by satellites as Envisat-1
P.Kunszt
OGSA 31.5.2002
3
DataGrid Architecture
Local Computing
Grid
Local
Local Application
Application
Local
Local Database
Database
Grid
Grid Application
Application Layer
Layer
Data
Data
Management
Management
Job
Job
Management
Management
Metadata
Metadata
Management
Management
Object
Object to
to
File
File
Mapping
Mapping
Collective
Collective Services
Services
Informatio
Informatio
nn &
&
Monitoring
Monitoring
Replica
Replica
Manager
Manager
Grid
Grid
Scheduler
Scheduler
Underlying
Underlying Grid
Grid Services
Services
SQL
SQL
Database
Database
Services
Services
Computing
Computing
Element
Element
Services
Services
Storage
Storage
Element
Element
Services
Services
Replica
Replica
Catalog
Catalog
Grid
Fabric
Service
Service
Index
Index
Fabric
Fabric services
services
Resource
Resource
Management
Management
P.Kunszt
Authorization
Authorization
Authenticatio
Authenticatio
nn and
and
Accounting
Accounting
Configuration
Configuration
Management
Management
Monitoring
Monitoring
and
and
Fault
Fault
Tolerance
Tolerance
Node
Node
Installation
Installation &
&
Management
Management
OGSA 31.5.2002
Fabric
Fabric Storage
Storage
Management
Management
4
Current Status
First Grid testbed release
based on GT2
– Testbed 1: first release of
middleware, current version
1.1.4, in transition to 1.2.
– Higher-level services
provided by MW
workpackages
– Issues with manageability,
robustness, security.
P.Kunszt
• WP1
– Job resource specification,
description & scheduling
• WP2
– Data access, migration &
replication
• WP3
– Information and Monitoring
Services
• WP4
– Framework for fabric
configuration management &
automatic sw installation
• WP5
– Common interface for Mass
Storage Systems
• WP7
– Network services and monitoring
OGSA 31.5.2002
5
WP2 and OGSA
• Early Adopters are WP2 and WP3.
• Need to interoperate with services of other WPs based
on GT2 – compatibility and interoperability between
GT3 and GT2 necessary.
• WP2 tasks:
– Replication Services
– Metadata Services
– Security of Data Management services
– Optimization
P.Kunszt
OGSA 31.5.2002
6
EDG Replication Services
Reptor
Replica Manager
Optimization
Client
Optor
Transaction
Consistency
File Transfer
Postprocessing
Preprocessing
Subscription
P.Kunszt
Replica Location Giggle
GDMP
Replica Metadata RepMeC
OGSA 31.5.2002
7
Service details
•
•
•
•
P.Kunszt
All Services will be Grid Services according to OGSA GSS, implementing
GridService with at least a SOAP binding.
Giggle
– Set of services nodes, highly configurable.
– Local Replica Catalog services LRC: LFN-SFN mapping
– Replica Location Index services RLI: index on LFNs
– Factory for RLIs
Reptor
– Factory and Registry for subservices
– Entry point for all clients
– Initiate automated replication
RepMeC
– Attributes of logical file names, as well as application-specific metadata
– Metadata for subservices (transaction locks)
– Spitfire RDBMS instance with a specialized schema
– GSI-enabled + local additional authorization (CERT->DB role mapping)
OGSA 31.5.2002
8
Service details
•
•
•
•
•
•
P.Kunszt
Optor
– Replica Selection based on economic modelling
– Automated replication for load balancing
GDMP
– Subscription-based automated replication
Processing
– Hooks for pre- and postprocessing while replicating
Transaction
– Ensure atomic ‘replication’ functionality
– Robustness of service
Consistency
– Optional QoS service, to check consistent state of Replication Services
– Ensure consistent view of files in RLS and SRM
– Ensure consistent Master file
File Transfer
– Expect to use OGSA services by GT3
– GridFTP
– SRM file transfer service hooks
OGSA 31.5.2002
9
Interaction
User Interface
Replica
Replica
Replica
Location
Location
Location
Index
Index
Index
Replica
Metadata Catalogue
Resource Broker
Site
Core API
Site
Replica Manager
Local
Replica
Catalogue
Replica Manager
Optimisation API
Optimiser
Optimiser
Processing API
Pre-/Postprocessing
Pre-/Postprocessing
Computing
Element
P.Kunszt
Storage
Element
OGSA 31.5.2002
Computing
Element
Local
Replica
Catalogue
Storage
Element
10
Security
• Replica Manager has administration rights on replicas: our
definition of ‘file being on the grid’
• Ensures replica consistency: user can’t change replicas directly on
SRM, only read access.
• Write access through controlled Master (single-master
mechanism)
User Request
+ Credential
1.
Replica
Manager
2.
Local Policies
SRM 1
4.
3.
Authorization
Service
4.
5.
Local Policies
SRM 2
4.
P.Kunszt
OGSA 31.5.2002
11
OGSA issues
• Security
– Currently we have to provide our own security
mechanisms. Framework not clear.
• Discovery of services
• Framework to define/query QoS (measurability of
services)
• Manageability : bootstrapping, logging, monitoring
• Robustness : availability, recovery
HELP!
P.Kunszt
OGSA 31.5.2002
12