WP2+5: Data and Storage
Management
Gavin McCance – University of Glasgow
Jens Jensen - RAL
University of Glasgow
GridPP9 NeSC, Edinburgh
DataGrid is a project funded by the European Union
GridPP is funded by PPARC
GridPP9 – 5 February 2004 – Data Management
EDG Summary Report
WP2: Data Management
Gavin McCance – University of Glasgow
GridPP9 – 5 February 2004 – Data Management – n° 2
The team
 Based
at Glasgow University

William Bell

David Cameron

Tony Doyle

Gavin McCance

Paul Millar

Caitriana Nicholson
 We
formed close collaboration with WP2 of European DataGrid

Became an integral part of the WP2 team

All our products were delivered jointly with EDG
Gavin McCance – University of Glasgow
GridPP9 – 5 February 2004 – Data Management – n° 3
What we set out to do?
 Initial
 Grid



work-plan was in two key areas:
Optimisation
Simulating the Grid from the data-centric viewpoint
Applying the results from the simulations to develop algorithms
for optimisation of replicas
Development of optimisation software for replica management
 Metadata

management
Development of secure metadata services for the experiments
Gavin McCance – University of Glasgow
GridPP9 – 5 February 2004 – Data Management – n° 4
What we actually did…
 Grid




Optimisation
Developed a data-centric Grid simulation [OptorSim] (fully
documented final release available now)
Compared of a variety of replica optimisation techniques and
algorithms (many publications)
Worked on developing and refining these algorithms
Exploitation of some of the early work, we developed the Replica
Optimisation Service (ROS) component of the Replica
Management System

provides file access cost estimation to the user
and the WP1 resource broker.
Gavin McCance – University of Glasgow
GridPP9 – 5 February 2004 – Data Management – n° 5
What we actually did…
 Metadata





Management
Early production of the Spitfire browser service allowing remote
access to RDBMS via a web browser
Early adoption of GSI security into Spitfire to allow secure use of
the service using Grid certificates
Initial adoption of web services into EDG with the creation of the
web services API version of Spitfire, allowing programmatic access
to remote RDMBS
The web services technology built for Spitfire was subsequently
factored out and used in all other WP2 services
The security built into Spitfire was subsequently factored out and
led to the creation of the WP2 Java security modules
Gavin McCance – University of Glasgow
GridPP9 – 5 February 2004 – Data Management – n° 6
What we actually did…
 Replica





Management Services
As focus of EDG changed to “stability, stability, stability” and
production, we changed with it
Helped factor-out the web services and security frameworks from
Spitfire
Moved into the core replica management services

Replica Manager

Replica Location Service

Replica Metadata Catalogue

Replica Optimisation Service
Provided continued support for these products to EDG and LCG
Led the thorough performance testing of our products
for WP2’s final EU deliverable
Gavin McCance – University of Glasgow
GridPP9 – 5 February 2004 – Data Management – n° 7
What we actually did…
 WP2

Integration and Testing
Significant integration expertise developed
Paul Millar is the WP2 integration team member


Glasgow ran the WP2 CVS repository

We created the testing framework for WP2 software
 WP2

Testbed support
Provided support for the WP2 mini-testbed
Machines managed at ScotGrid and at CERN

 WP2

Management
Gavin McCance is WP2 deputy
 Major
contributions to WP2 deliverables
Gavin McCance – University of Glasgow
GridPP9 – 5 February 2004 – Data Management – n° 8
What we actually did…
 Standards

Contribution of GGF Data Area documents
 Publications

Work was fully reported: 15 or so papers published…
Optimisation
Metadata
Replication
GGF
Deployment
6
3
2
3
1
Gavin McCance – University of Glasgow
GridPP9 – 5 February 2004 – Data Management – n° 9
EDG Effort Delivered
 Substantial
UK effort delivered to EDG : ~150% of contracted
Project Months
Integrated Effort
90
80
70
60
50
40
30
20
10
0
Integrated effort PM
Contracted effort PM
1
2
3
4
5
6
7
8
9
10 11 12
Quarter
Gavin McCance – University of Glasgow
GridPP9 – 5 February 2004 – Data Management – n° 10
EDG Summary Report
WP5: Storage Management
(Jens Jensen)
Gavin McCance – University of Glasgow
GridPP9 – 5 February 2004 – Data Management – n° 11
The team
 Based
at RAL

Tim Eaves

Tim Folkes

John Gordon

Jens Jensen

Glen Johnson

Tara Shah

Bonny Strong

Owen Synge

Regina Tam
 UK
WP5 managed the WP5 team at CERN, SARA and IN2P3.
Gavin McCance – University of Glasgow
GridPP9 – 5 February 2004 – Data Management – n° 12
What we did…
 Developed

 SE

Storage Element
uniform control interface to MSS: disk, ADS, CASTOR, HPSS
provides additional functionality for storage systems
e.g. directory hierarchies for mass storage with flat filenames
 Modular
design allows easy deployment in front of different
backing mass storage systems
 SE
provides disk cache in front of mass storage
 SE
can provide secure access (authentication and
authorisation) to the mass storage resource.
Gavin McCance – University of Glasgow
GridPP9 – 5 February 2004 – Data Management – n° 13
What we did…
 Demonstrated
3rd party transfers between different mass
storage systems
 Work
on interfacing with Replica Management service and with
the information services
 LCG

now requires SRM v1 to access storage
EDG SE is architected for easy migration to this
 Significant

work in standards development
Input into the SRM standards process
Gavin McCance – University of Glasgow
GridPP9 – 5 February 2004 – Data Management – n° 14
What we did…

Overall WP5 Management


WP5 Integration and Testing


Significant integration expertise developed

Owen Synge and Regina Tam were EDG integration team members

Regina Tam was on the EDG Software Quality team
EDG Architecture Team


John Gordon and Jens Jensen
Jens Jensen was WP5 member of Architecture team.
UK responsible for all WP5 deliverables
Gavin McCance – University of Glasgow
GridPP9 – 5 February 2004 – Data Management – n° 15
EDG effort delivered
 Substantial
UK effort delivered to EDG : ~150% of contracted
Integrated Effort
160
Project Months
140
120
100
Integrated effort (PM)
80
Contracted effort (PM)
60
40
20
0
1
2
3
4
5
6
7
8
9 10 11 12
Quarter
Gavin McCance – University of Glasgow
GridPP9 – 5 February 2004 – Data Management – n° 16
WP2 + WP5: What we learned
 Software

sometimes learned the hard way…!
 Much

development process…
feedback from our users on the software
Very useful for the design process in the next projects
 The
amount of effort required to make production quality
services out of our software
 Take
all this forward into GridPP2!
Gavin McCance – University of Glasgow
GridPP9 – 5 February 2004 – Data Management – n° 17