LCG
ARDA
Architectural Roadmap towards
Distributed Analysis
4-Feb-2004
GridPP-9 : Edinburgh
Slide 1
What is ARDA ?
•
•
•
•
•
•
LCG
Is it middleware (e.g. “second generation” ) ?
Is it a common application layer above middleware ?
Is it a co-ordination project ?
Is it about distributed analysis or wider HEP grid use ?
Is it “All things to all men” ?
Is it the solution to all our (Grid) problems ?
Possibly ALL the above
…and more !!
• Does any one understand what this all means ?
A call for volunteers !!
4-Feb-2004
GridPP-9 : Edinburgh
Slide 2
Outline
LCG
• History
– HEPCAL I
– GAG and HEPCAL II
• ARDA Research & Technology Assessment Group
–
–
–
–
–
Set up Spring 2003; Reported late Autumn 2003
ToR
Survey
Conclusion
Recommendations
• Workshop (21st/22nd Jan 2004)
• Status
4-Feb-2004
GridPP-9 : Edinburgh
Slide 3
LCG
Some History
4-Feb-2004
GridPP-9 : Edinburgh
Slide 4
What we want from a GRID
LCG
(EDG meeting, NIKHEF, March 2001, © fca 2004)
Specific
application layer
VO common
application layer
ALICE ATLAS
CMS
LHCb
HEP
WP9
WP 10
Earth Obs.
Biology
GRID middleware
Bag of Services (GLOBUS)
OS & Net services
4-Feb-2004
GridPP-9 : Edinburgh
Slide 5
What we have
LCG
(HEPCAL presentation, CHEP 2003, © fca 2004 )
Specific
application layer
Semantic gap
ALICE
ATLAS
WP1
WP2
CMS
WP3
WP4
LHCb
WP5
Middleware
Bag of Services (GLOBUS)
OS & Net services
4-Feb-2004
GridPP-9 : Edinburgh
Slide 6
A proposal
LCG
(HEPCAL presentation, CHEP 2003, © fca 2004 )
If we manage to define
It will be easier for them to arrive at
Specific
application layer
ALICE
ATLAS
VO common
application layer
CMS
LHCb
Common use cases
WP1
WP2
WP3
WP4
WP5
DataGRID
middleware
Middleware
Bag of Services (GLOBUS)
OS & Net services
4-Feb-2004
GridPP-9 : Edinburgh
Slide 7
HEPCAL II
LCG
• (If HEPCAL I was Batch, then HEPCAL II is Analysis)
• Analysis Execution Models
– Support for queries by common layers and at dataset level
– Support for job pipelines
• Users, Groups, Quotas and Permissions
• Interactive .vs. Batch Grid Activity
• System Requirements
–
–
–
–
Provenance & Job Traceability
Log Book and Reports
Persistent Interactive Environment
Analysis Software deployment
• Use cases (as sequence of user operations)
– Production Analysis
– (Sub-)Group level Analysis
– End-user level Anlaysis
4-Feb-2004
GridPP-9 : Edinburgh
Slide 8
LCG
Enter ARDA
4-Feb-2004
GridPP-9 : Edinburgh
Slide 9
LCG
4-Feb-2004
GridPP-9 : Edinburgh
Slide 10
RTAG ToR
LCG
Motivation:
• To agree on requirements as laid out in a first step by recent work
within the GAG and identify commonalities within the current
projects that might allow the LCG (both in the AA and GTA areas)
to provide a focus of effort.
• To provide guidance to the LCG on future Middleware development
directions and interfacing work to match the experiment
requirements
• To build on the richness of the current technical solutions to avoid
duplication of efforts
• To clearly identify the roles and responsibilities of the
components/layers/ services in the experiment DA planning
• To give guidance to the community on the expected division of
work between the experiments, the LCG and the external projects.
4-Feb-2004
GridPP-9 : Edinburgh
Slide 11
RTAG ToR
LCG
Mandate:
• To review the current Distributed Analysis (DA) activities and to capture
their architectures in a consistent way
• To confront these existing projects to the HEPCAL II use cases and the
user's potential work environments in order to explore potential
shortcomings
• To consider the interfaces between Grid, LCG and experiment-specific
services
•
•
– Review the functionality of experiment-specific packages, state of
advancement and role in the experiment
– Identify similar functionalities in the different packages
– Identify functionalities and components that could be integrated in the generic
Grid middleware
To confront the current projects with critical Grid areas
To develop a roadmap specifying wherever possible the architecture, the
components and potential sources of deliverables to guide the medium
term (2 year) work of the LCG and the DA planning in the experiments
4-Feb-2004
GridPP-9 : Edinburgh
Slide 12
LCG
4-Feb-2004
GridPP-9 : Edinburgh
Slide 13
The Report
LCG
• Reviewed existing projects
–
–
–
–
–
–
PROOF and the Grid
AliEn – web services (ALICE) **
Clarens – web services (CMS)
DIAL (ATLAS) – workflow
GANGA (LHCb/ATLAS) – high level job submission
DIRAC (LHCb) – distributed MC production
** selected for further analysis (best meets requirements of
experiment, used in anger by an experiment…)
• The ARDA Blueprint
– Service Descriptions + APIs (access to services)
Information, Authentication, Authorisation, Audit, Accounting, Workload
Management, Job Provenance, File Catalogue, Metadata Catalogue, Data
management, Site Gatekeeper, Storage Element, Computing Element, Job
Monitoring, Package Manager, Grid Monitoring
4-Feb-2004
GridPP-9 : Edinburgh
Slide 14
Example : AliEn
4-Feb-2004
GridPP-9 : Edinburgh
LCG
Slide 15
Example : AliEn
4-Feb-2004
GridPP-9 : Edinburgh
LCG
Slide 16
Example : Clarens
LCG
Web Services
-Secure file access
-SRB
-POOL
-VO mgmt
4-Feb-2004
GridPP-9 : Edinburgh
Slide 17
Expanded AliEn Services
4-Feb-2004
GridPP-9 : Edinburgh
LCG
Slide 18
Set of Services
4-Feb-2004
GridPP-9 : Edinburgh
LCG
Slide 19
The ARDA Prototype
•
LCG
•
“The main goal of an ARDA prototype is to provide a more complete
blueprint and to develop the specifications and interfaces for the ARDA
services and API”
“We recommend that the LCG setup a project to develop a prototype…”
•
6 months for first prototype (driven by need for TDRs in 2005 ?)
•
ARDA Project
–
–
–
–
Careful definition of work areas
Define project constituency
Define project lead -> project team
Work plan, schedule, milestones
• Particularly plan for interfacing to and engaging with LHC experiments and
the LHC related Grid community
4-Feb-2004
GridPP-9 : Edinburgh
Slide 20
RTAG Recommendations
LCG
RTAG report proposed a four-prong approach :
• Re-factoring of AliEn and other services into ARDA, with an initial
release; consolidation of the API working with the experiments
and the LCG-AA; release of a fully functional prototype.
Subsequently implementation of agreed interfaces, testing and
release of the prototype implementation.
• Modelling of an OGSI-based services infrastructure, performance
tests and quality assurance of the prototype implementation
• Interfacing to LCG-AA software like POOL and ROOT
• Interfacing to experiment's frameworks, with specific meta-data
handlers and experiment specific services
• n.b. OGSI  WSRF (or perhaps WS in the first instance)
4-Feb-2004
GridPP-9 : Edinburgh
Slide 21
Message of ARDA Report
LCG
(© Miron Livny)
Deliver end-to-end capabilities (from user to fabric) and
stability (deployable) at the price of services offered
(functionality)
Services provide a natural abstraction and powerful
software engineering constructs
AliEn provides a useful and stable suite of services as it
meets the expectations of the Alice experiment
4-Feb-2004
GridPP-9 : Edinburgh
Slide 22
LCG
The ARDA Workshop
4-Feb-2004
GridPP-9 : Edinburgh
Slide 23
ARDA Workshop - Goals
LCG
CERN : 21st/22nd January 2004 : (Les Robertson)
Define the ARDA project
•
•
The scope of the distributed analysis requirements that should be
addressed
The scope of a generic middleware component,
– the approach to implementation
– target timescales
•
Which HEP-specific components should or could be done in common in the
LCG Applications Area
– e.g. POOL, collections, meta-data, SEAL, ..
•
A process for agreeing on the specification of ARDA services –
middleware projects, experiments
• The framework for an ARDA implementation project, coordinating –
Middleware ↔ LCG AA ↔ experiment analysis s/w ↔ end-users
4-Feb-2004
GridPP-9 : Edinburgh
Slide 24
Workshop Agenda
LCG
•
•
•
•
Requirements (HEPCAL II)
- Federico Carminati
ARDA Architecture
- Lothar Bauerdick
Experiment expectations from ARDA – 4 talks
Generic Middleware
- Frederic Hemmer
- Miron Livny
• Applications Area involvement in ARDA
- Torre Wenaus
• POOL-ARDA collaboration
- Dirk Duellmann
• Discussion…
4-Feb-2004
GridPP-9 : Edinburgh
Slide 25
Experiments Sumamry
LCG
(© Andreas Peters)
4-Feb-2004
GridPP-9 : Edinburgh
Slide 26
Experiments Summary
LCG
(© David Adams)
• ATLAS strategy
– Use grid service model
– Quickly define high-level service interfaces and implement services
and clients
– Deliver end-to-end system to users
– Frequently re-design and re-implement based on user feedback
• ARDA collaboration
– Ideally we would come to consensus within ARDA on a high-level
interface along the lines of AJDL and share end-to-end effort
– In any case, we will work closely with ARDA to define middleware
services
4-Feb-2004
GridPP-9 : Edinburgh
Slide 27
Experiments Summary
LCG
(© Julian Bunn (et al))
CMS
• Distributed Production
– CMS is already using successfully LCG and VDT grid middleware to
build its distributed production system
– A distributed batch-analysis system is being developed on top of the
same LCG and VDT software
– CMS suggests that in the short term (6 months) ARDA extends the
functionality of the LCG middleware to meet the architecture
described in the ARDA document
• Grid Analysis Environment
– CMS asks that the GAE be adopted as the basis for the ARDA work on
a system that supports interactive and batch analysis. Support for
Interactive Analysis is the crucial goal in this context.
4-Feb-2004
GridPP-9 : Edinburgh
Slide 28
Experiments Summary
LCG
(© Andrei Tsaregorodtsev )
LHCb
• Expected from ARDA
– More efficient development process:
• Rapid development cycles;
• Keeping the functional core and adding functionality incrementally;
• Emphasis on intensive testing while the development.
– Concurrent development of components to try out different ideas and to
enhance the quality by competition.
•
Participation
•
The first tests of the distributed analysis will be done during the DC2004
(May) using LHCb developed and LCG tools;
We see the further evolution of the LHCb distributed analysis tools
within the context of the ARDA architecture and the proposed
development process.
•
– Participate to the definition of the services interfaces, testing and feedback;
– Prototyping ARDA components using OGSI compliant implementations;
– Developing the DIRAC WMS into an ARDA compliant service
4-Feb-2004
GridPP-9 : Edinburgh
Slide 29
Experiments
Summary Summary !
LCG
• A cynical view…!!
• We think ARDA’s the way forward
• We will be ARDA compliant
• …but would like ARDA to be based on us as a starting point !!
4-Feb-2004
GridPP-9 : Edinburgh
Slide 30
Middleware
(from report to prototype)
LCG
(© Miron Livny)
• Understand AliEn (including services)
• Identify potential contributions from existing middleware
• Understand requirements (how does analysis differ from
“production”?)
• Develop a plan – what, how, when, who
–
–
–
–
–
Semantic of exposed services
Authentication/protection model
Integration/testing/deployment procedures
Documentation
…
• Execute the plan!
4-Feb-2004
GridPP-9 : Edinburgh
Slide 31
Middleware
(working document)
LCG
(© Miron Livny)
Abstract: This working document is used to break down the high level
services defined by ARDA to actual components and tries to map
these components to existing implementations coming from AliEn,
EDG, and VDT. The structure and initial AliEn input is taken from
Chapter 5 of Draft v0.2 of the ARDA document (unpublished)
• Started after the December meeting as a vehicle to exchange and
record information and ideas among the middleware providers.
– Identification of services
• Service interplay and semantics
– Understand how existing MW could implement these services
• Input from AliEn, EDG, VDT, commercial, …. (others?)
– Specify interfaces to applications
4-Feb-2004
GridPP-9 : Edinburgh
Slide 32
General ARDA-AA Objectives
LCG
(© Torre Wenaus)
• Common software above the middleware layer
– Adapting, extending, interfacing AA software for ARDA
– Participating in ARDA interface definition; ensuring AA requirements
met
– Applying lower level middleware services in specialized higher level
services directed at HEP and analysis
• Early PEB agreement on ARDA: Middleware covers as much as
possible; remaining higher levels covered by AA (if common) or
experiments (if not)
• Integration and validation
– Integrating ARDA middleware services and analysis application level
services into end-to-end distributed analysis prototype
– Assisting integration of distributed analysis prototype or components
thereof into experiment environments
– Validation of the prototype [and feedback to middleware providers]
• Proposal to use the GAG as the principal feedback channel seems a
very good one
4-Feb-2004
GridPP-9 : Edinburgh
Slide 33
Summarizing TW’s Current
Thoughts on WPs (© Torre Wenaus)
LCG
1) Integration and Validation
–
Primarily providing coordination, communication, coherence for
efforts residing in the experiments and projects
•
Some similarity to Physics Validation in the simulation project
•
Though the (majority of the) work will go on in the experiments
and projects, a common focal point is needed if it is to be a
common effort
2) Event data management
–
–
Physics-driven event collections
Joint WP with POOL Collections
3) Framework integration
–
–
‘Thin’ adaptation of middleware services to whatever is required for
integration in experiment analysis frameworks
Joint WP with SEAL
4-Feb-2004
GridPP-9 : Edinburgh
Slide 34
LCG
Status
4-Feb-2004
GridPP-9 : Edinburgh
Slide 35
Experiment Prototype
Applications
ALIEN ADA GAE
Al
Other
Common
project
At
C
ARDA Services
Grid middleware
POOL’, SEAL’,
HEP-specific
DIRAC
L
physicists
with real
requirements
Operational
experience
Al
HEP-common Service
ARDA
SEAL
SEAL’ Services
At
coordination
integration, validation
development specific
services
GAG
C
Consolidated
Requirements
L
POOL’ Services
POOL
Grid Middleware
Services
grid technology
and experience
AliEn
EDG
Nordu
. .. .
VDT
EGEE/
VDT
Experiment Requirements
and Use Cases
Experiment
LCG
Grid MW
ARDA
(© Les Robertson)
ARDA – “A Realisation of
Distributed Analysis”
Alice
ATLAS
CMS
LCG
LHCb
Service specs
LCG
Integration: workers’ forum
LCG AA
projects
GAG
PEB
Requirements
Guidelines
Generic Middleware
Resource providers
4-Feb-2004
GridPP-9 : Edinburgh
Slide 37
Postscript
LCG
• HEPCAL II provides reasonable set of use cases as input
– But must be complemented by use cases from non-HEP (within EGEE)
• Proposed set of services for DA (under HEPCAL II) is reasonable
• Prototypes should leverage existing technologies and experience
within a web services framework
• Prototype(s) needed (urgently) within 6 months
• Applications middleware must interface (POOL, LCG-AA, ROOT,
GAE, …) as should experiments software
• The PEB are still working on the details
4-Feb-2004
GridPP-9 : Edinburgh
Slide 38
LCG
• If you think ARDA is a threat …
– It’s here to stay, so get used to it !!
• If you think ARDA is an opportunity
– Then grasp it (but don’t hang around as timescales are ridiculously
short (what’s new ?)) !!
It’s for you to choose !!
“We should not miss this
opportunity to put it all
together!” (© fca 2004)
4-Feb-2004
GridPP-9 : Edinburgh
Slide 39
LCG
The End
4-Feb-2004
GridPP-9 : Edinburgh
Slide 40