EU DataGrid Data Management
Services
Peter Kunszt
CERN
Peter.Kunszt@cern.ch
EU DataGrid WP2 Manager
DataGrid is a project funded by the European Union
Virtual Observatory as a Data Grid – WP2 Data Management
Talk Outline

Introdution to EU DataGrid workpackage 2

WP2 Service Design and Interactions
• Replication Services
• Spitfire
• Security

Conclusions and outlook
WP2 Members
Diana Bosio, James Casey, Akos Frohner, Leanne Guy, Peter Kunszt, Erwin
Laure, Levi Lucio, Heinz Stockinger, Kurt Stockinger - CERN
Giuseppe Andronico, Federico DiCarlo, Andrea Domenici, Flavia Donno, Livio
Salconi – INFN
William Bell, David Cameron, Gavin McCance, Paul Millar, Caitriona Nicholson –
PPARC, University of Glasgow
Joni Hahkala, Niklas Karlsson, Ville Nenonen, Mika Silander, Marko Niinimäki –
Helsinki Institute of Physics
Olle Mulmo, Gian Luca Volpato – Swedish Research Council
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 2
EU DataGrid Project Objectives





DataGrid is a project funded by European Union whose objective is
to exploit and build the next generation computing infrastructure
providing intensive computation and analysis of shared large-scale
databases.
Enable data intensive sciences by providing world wide Grid test
beds to large distributed scientific organisations ( “Virtual
Organisations, VO”)
Start ( Kick off ) : Jan 1, 2001
Applications/End Users Communities :
Biology
End : Dec 31, 2003
HEP, Earth Observation,
Specific Project Objetives:
• Middleware for fabric & grid management
• Large scale testbed
• Production quality demonstrations
• To collaborate with and complement other European and US projects
• Contribute to Open Standards and international bodies
( GGF, Industry&Research forum)
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 4
DataGrid Main Partners

CERN – International (Switzerland/France)

CNRS - France

ESA/ESRIN – International (Italy)

INFN - Italy

NIKHEF – The Netherlands

PPARC - UK
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 5
Assistant Partners
Industrial Partners
•Datamat (Italy)
•IBM-UK (UK)
•CS-SI (France)
Research and Academic Institutes
•CESNET (Czech Republic)
•Commissariat à l'énergie atomique (CEA) – France
•Computer and Automation Research Institute,
Hungarian Academy of Sciences (MTA SZTAKI)
•Consiglio Nazionale delle Ricerche (Italy)
•Helsinki Institute of Physics – Finland
•Institut de Fisica d'Altes Energies (IFAE) - Spain
•Istituto Trentino di Cultura (IRST) – Italy
•Konrad-Zuse-Zentrum für Informationstechnik Berlin - Germany
•Royal Netherlands Meteorological Institute (KNMI)
•Ruprecht-Karls-Universität Heidelberg - Germany
•Stichting Academisch Rekencentrum Amsterdam (SARA) – Netherlands
•Swedish Research Council - Sweden
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 6
Project Schedule

Project started on 1/Jan/2001

TestBed 0 (early 2001)
International test bed 0 infrastructure deployed
Globus 1 only - no EDG middleware

Successful Project Review by EU: March 2002

TestBed 1 ( 2002 )

Successful 2nd Project Review by EU: February 2003

TestBed 2 (Now)
Some complete re-writes of components. Builds on TestBed 1 experience.


TestBed 3 (Oktober 2003)
Project stops on 31/Dec/2003, maybe a couple of months extension to
wrap up and document results (no additional funding)
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 7
EDG Highlights

All EU deliverables (40, >2000 pages) submitted
• in time for the review according to the contract technical
annex

First test bed delivered with real production demos

All deliverables (code & documents) available via
www.edg.org
• http://cern.ch/eu-datagrid/Deliverables/default.htm
• requirements, surveys, architecture, design, procedures, testbed
analysis etc.

Project re-orientation last year in August: From R&D
Testbed to ‘production grid’
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 8
Working Areas
The DataGrid project is divided in 12 Work Packages
distributed in four Working Areas
Middleware
Infrastructure
Management
Applications
Testbed

Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 9
Work Packages
WP1: Work Load Management System
WP2: Data Management
WP3: Grid Monitoring / Grid Information Systems
WP4: Fabric Management
WP5: Storage Element
WP6: Testbed and demonstrators
WP7: Network Monitoring
WP8:
High Energy Physics Applications
WP9:
Earth Observation
WP10: Biology
WP11: Dissemination
WP12: Management
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 10
Trying hard to have a real GRID..

Testbed 0 :
• Grid technology was not mature enough
• Configuration and deployment issues
• Stability problems
• Obscure errors

Project reorientation: Stability, Stability, Stability – TB 1
• TB1 revealed a set of design bugs in Globus
• GASS Cache issue – fixed by Condor (rewritten)
• MyProxy issues – could never be used
• MDS did not scale – had to set up fake local info system

Reingeneering of essential components – TB 2
• New resource broker
• R-GMA instead of MDS as info system
• Concrete Support channels (VDT)
• New configuration tool LCFG-ng (from U of Edinburgh!)
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 11
Grid middleware architecture
hourglass
Current Grid architectural functional blocks:
Specific
application layer
Common
application layer
EU
DataGrid
middleware
GLOBUS
2.2
CMS
ATLAS
CMS
LHCb
HEP Application Services (LCG)
Earth
Observation
and
Biomed
High Level Grid Services
Basic Grid Services
OS, Storage & Network services
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 12
EU DataGrid WP2
Data Management Work Package
Responsible for

Transparent data location and secure access

Wide-area replication

Data access optimization

Metadata access
NOT responsible for (but it has to be done)

Data storage (WP5)

Proper Relational Database bindings (Spitfire)

Remote I/O (GFAL)

Security infrastructure (VOMS)
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 13
WP2 Service Paradigms

Choice of technology:
• Java-based servers using Web Services
o
Tomcat, Oracle 9iAS, soon WebSphere
• Interface definitions in WSDL
• Client stubs for many languages (Java, C, C++)
o
Axis, gSOAP
• Persistent service data in Relational Databases
o

MySQL, Oracle, soon DB2
Modularity
• Modular service design for pluggability and extensibility
• No vendor specific lock-ins

Evolvable
• Easy adaptation to evolving standards (OGSA, WSDL 1.2)
• Largely independent of underlying OS, RDBMS – works on Windows
too!
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 14
Replication Services: Basic
Functionality
Each file has a unique Grid ID.
Locations corresponding to the
GUID are kept in the Replica
Location Service.
Users may assign aliases to the
GUIDs. These are kept in the
Replica Metadata Catalog.
Files have replicas stored at
many Grid sites on Storage
Elements.
Replica Metadata
Catalog
Replica Manager
Storage
Element
Storage
Element
Replica Location
Service
The Replica Manager provides
atomicity for file operations, assuring
consistency of SE and catalog
contents.
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 15
Higher Level Replication Services
The Replica Subscription Service issues
Replication commands automatically,
based on a set of subscription rules
Hooks for user-defined pre- and postdefined by the user.
processing for replication operations
are
Theavailable.
Replica Manager may call on the
Replica Optimization service to find the
best replica among many based on
network and SE monitoring.
Replica Manager
Replica Subscription
Service
Replica Metadata
Catalog
Replica Location
Service
Replica Optimization
Service
Storage
Element
Storage
Element
SE
Monitor
Network Monitor
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 16
Interactions with other Grid
components
User Interface or
Worker Node
Resource Broker
Virtual Organization
Membership Service
Replica Subscription
Service
Information Service
Replica Metadata
Catalog
Replica Manager
Replica Location
Service
Replica Optimization
Service
Storage
Element
Storage
Element
Applications and users interface to data
through the Replica Manager either
SE
Network Monitor
Monitor directly or through the Resource
Broker. Management calls should never
go directly to the SE.
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 17
Replication Services Status

Current Status
• All components are deployed right now – except for the RSS
• Initial tests show that expected performance can be met
• Need proper testing in a ‘real user environment’ – EDG2; LCG1

Features for next release
• Currently Worker Nodes need outbound connectivity – Replica
Manager Service needed. Needs proper security delegation
mechanism.
• Logical collections support
• Service-level authorization
• GUI
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 18
Spitfire: Grid-enabling RDBMS

Capabilities:
• Simple Grid enabled front end to any type of local or remote
RDBMS through secure SOAP-RPC
• Sample generic RDBMS methods may easily be customized
with little additional development, providing WSDL interfaces
• Browser integration
• GSI authentication
• Local authorization mechanism

Status: current version 2.1
• Used by EU DataGrid Earth Observation and Biomedical
applications.

Next Step: OGSA-DAI interface
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 19
Spitfire added value : Security

Grid security
• TrustManager deals with GSI proxy certificates
• Support for VOMS certificate extensions
• Secure java, c/c++, perl clients

Local Authorization
• Mapping through Gridmap file supported
• Fine grained authorization hooks : a mapping service is provided
to map VOMS extensions (group, role, capability) to DB roles that
depending on the DB may be row-level authorization mechanisms
(GRANT/DENY).

Installation kit
• Easy installation and configuration of all security options
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 20
Spitfire customization


Spitfire started as a ‘proof of technology’ for Web Services and
java.
Customizable into specific services dealing with persistent data
• All WP2 services are in this sense ‘Spitfire’ services (see later)

Test platform for latest available codebase
• Gained experience with WSDL, JNDI, Tomcat, Axis, gSOAP
• Next things to try : JBOSS (for JMS, JMX)

Experimental add-ons
• Secure browser using JSP (proxy certificates for mozilla, netscape,
ie..)
• Distributed query agent drop-in
• Todo: OGSA-DAI interface as far as possible
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 21
RLS Architecture (evolved!)
A hierarchical RLS topology:
LRCs update RLIs, RLIs may forward information
RLI
RLI
RLI
RLI
RLI
RLI
RLI
LRC
LRC
LRC
LRC
RLI
LRC sending updates to all Tier 1 RLIs
RLIs indexing over the full namespace (all LRCs are indexed) receiving updates directly
RLI receiving updates from other RLIs
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 22
EDG Grid Catalogs (1/2)

Replica Location Service (RLS)
• Local Replica Catalog (LRC)
o
Stores GUID to Physical File Name (PFN) mappings
o
Stores attributes on PFNs
o
Local Replica Catalogs in Grid : One per Storage Element (per VO)
o
Tested to 1.5M entries
• Replica Location Index (RLI)
o
Allow fast lookup of which sites store GUID -> PFN mappings for a given
GUID
o
Replica Location Indices in the Grid :Normally one per Site (per VO),
which indexes all LRCs in the Grid
o
Being deployed as part of EDG 2.1 in July

o
In the process of integration into other components
Tested to 10M entries in an RLI
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 23
EDG Grid Catalogs (2/2)

Replica Metadata Catalog (RMC)
• Stores Logical File Name (LFN) to GUID mappings – user-defined
aliases
• Stores attributes on LFNs and GUIDs
• One logical Replica Metadata Catalog in Grid (per VO)
o
o

Single point of synchronization – current assumption in EDG model
bottleneck ? - move to replicated distributed database
No Application Metadata Catalog provided – see Spitfire
• But Replica Metadata Catalog has support for small level of
application metadata – O(10)

RMC usage not as well understood as Replica Location Service
• Architectural changes likely
• Use cases required
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 24
Typical Location of Services in
LCG-1
CERN
CNAF
Storage Replica
Element Metadata
Catalog
Storage
Element
Local
Replica
Catalog
Replica
Location
Index
Local
Replica
Catalog
Replica
Location
Index
Local
Replica
Catalog
Replica
Location
Index
Local
Replica
Catalog
Replica
Location
Index
Storage
Element
Storage
Element
Storage
Element
IN2P3
RAL
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 25
Catalog Implementation Details

Catalogs implemented in Java as Web Services, and hosted in a
J2EE application server
• Uses Tomcat4 or Oracle 9iAS for application server
• Uses Jakarta Axis for Web Services container
• Java and C++ client APIs currently provided using Jakarta Axis
(Java) and gSoap (C++)

Catalog data stored in a Relational Database
• Runs with either Oracle 9i or MySQL

Catalog APIs exposed as a Web Service using WSDL
• Easy to write a new client if we don’t support your language right
now

Vendor neutral approach taken to allow different deployment
options
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 26
Quality of Service


Quality of Service depends upon both the server software and
architecture used as well as the software components deployed on it
Features required for high Quality of Service
• High Availability
• Manageability
• Monitoring
• Backup and Recovery with defined Service Level Agreements

Approach
• Use vendor solutions for availability and manageability where available
• Use common IT-DB solutions for monitoring and recovery
• Components architected to allow easy deployment in high-availability
environment

A variety of solutions with different characteristics are possible
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 27
Availability
Tradeoffs in different solutions
Clustered
Oracle 9i/9iAS
Clustered
Oracle 9i/Tomcat
Single Instance
MySQL/Tomcat
Single Instance
Oracle 9i/9iAS
Manageability
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 28
System Architecture – High
Availability
Standard n-tier
architecture
• Front end application
layer load-balancer
Oracle 9iAS Web Cache
o
Oracle 9iAS J2EE
container
External LAN
• Cluster of stateless
application servers
Storage Network
o
Internal LAN

• Clustered database
nodes
o
Oracle 9i/RAC
• Shared SAN storage
o
Fibre Channel storage
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 30
Security: Infrastructure for Javabased Web Services

Trust Manager
• Mutual client-server authentication using GSI (ie PKI X509 certificates) for
all WP2 services
• Supports everything transported over SSL

Authorization Manager
• Supports coarse grained authorization:
Mapping user->role->attribute
• Fine grained authorization through policies, role and attribute maps
• Web-based Admin interface for managing the authorization policies and
tables

Status:
• Fully implemented, authentication is enabled on the service level
• Delegation implementation needs to be finished
• Authorization needs more integration, waiting for deployment of VOMS
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 31
Conclusions and outlook

Re-focus on production has been a good but painful choice
• from hype to understanding the implications of wanting to run a production Grid
• reengineering of several components was necessary
• however, the project was not well prepared for this change – the timelines had to be
constantly revised in the last year





The second generation Data Management services have been designed and
implemented based on the Web Service paradigm
Flexible, extensible service framework
Deployment choices : robust, highly available commercial products supported
(eg. Oracle) as well as open-source (MySQL, Tomcat)
First experiences with these services show that their performance meets the
expectations
Real-life usage will show its strengths and weaknesses on the LCG-1 and
EDG2.0 testbeds during the rest of this year.

Proceed with standardization efforts: DAI, RLS

Carry over the experience into the next project : EGEE
Virtual Observatory as a Data Grid– 1 July 2003 – WP2 Data Management – n° 32