UNICORE in OMII - Europe
Morris Riedel
FZJ, Germany
Forschungszentrum Jülich
Outline
• Production UNICORE
• Road to UNICORE 6
• UNICORE 6 alpha
• UNICORE in OMII-Europe
• UNICORE Open Source & Forum
2/26
EU project: RIO31844-OMII-EUROPE
Production Systems
3/26
EU project: RIO31844-OMII-EUROPE
Production UNICORE
• A vertically integrated Grid middleware system
• Provides seamless, secure, and intuitive access to
distributed resources and data
• Used in production and projects worldwide
• Major Features
–
–
–
–
–
intuitive GUI with single sign-on
X.509 certificates for AA and job/data signing
workflow engine for complex workflows
extensible application support with plug-ins
interactive access with UNICORE-SSH
4/26
EU project: RIO31844-OMII-EUROPE
UNICORE Architecture
Workflow-Engine
Client
Resource Management
SSL
Multi-Site
Job-Monitoring
Jobs
File Transfer
opt. Firewall
opt. Firewall
Gateway User Management
Gateway
Authentication
Application Support
opt. Firewall
NJS
Authorization
NJS
Abstract
NJS
Authorization
UUDB
IDB
UUDB
IDB
Incarnation
IDB
similar to /etc/grid-security/grid-mapfile
TSI
RMS Disc
Vsite
NonTSI
similar toTSI
Globus jobmanager
Abstract
fork
LoadLeveler,
RMS Disc (Open)PBS(Pro),RMS Disc
CCS, LSF, NQE/NQS, ...
Vsite
Usite CONDOR, GT 2.4
Vsite
5/26
Usite
EU project: RIO31844-OMII-EUROPE
UNICORE in DEISA (1)
• Consortium of leading national HPC centers in EU
• Deploy and operate a persistent, production quality,
distributed, heterogeneous HPC environment
IDRIS – CNRS, France
FZJ, Jülich, Germany
RZG, Garching, Germany
CINECA, Bologna, Italy
EPCC, Edinburgh, UK
CSC, Helsinki, Finland
SARA, Amsterdam, NL
HLRS, Stuttgart, Germany
BSC, Barcelona, Spain
LRZ, Munich, Germany
ECMWF, Reading, UK
6/26
EU project: RIO31844-OMII-EUROPE
UNICORE in DEISA (2)
• Fully-meshed UNICORE infrastructure
• Complex multi-site workflows
easily possible
• Heavily used by DECI projects
7/26
EU project: RIO31844-OMII-EUROPE
UNICORE in Industry and Commercial Support
T-Systems Solutions for Research GmbH
HPC / Grid Services
8/26
Alfred Geiger
EU project: RIO31844-OMII-EUROPE
Recent Developments
•
•
•
•
OGSA-based and WSRF-compliant UNICORE 6 alpha
Interactive access Æ UNICORE-SSH (UniGrids)
Improved workflow capabilities (OpenMolGRID)
High-level API for
programming Grids (DEISA)
• DRMAA-based TSI (DGI)
• Comfortable configuration tool (DGI)
9/26
EU project: RIO31844-OMII-EUROPE
Roadmap to UNICORE 6.0
• New infrastructure based on Web services technology
• “User-level” features Æ Production UNICORE
– Atomic: simple tasks, such as „Execute script & fetch outcome“
– Client: workstation GUI
– Workflow: edit, run and monitor graphs of atomic tasks
• “User-level” features Æ Additional features UNICORE 6
– Portal: Web based portal client
– Streaming: client-server streaming support (for visualization or media
applications)
– Tools: manage users and virtual organizations (VOs), manage
software licenses, simplify application deployment
• Alpha in July 2006, Beta in July 2007, Final in End 2007
10/26
EU project: RIO31844-OMII-EUROPE
European Context
SSAs supporting NESSI and the Grid community
Challengers
Challengers
trust, security
GridTrust
GridCoord
Grid services,
business models
Degree
Grid @Asia
GridEcon
AssessGrid
AssessGrid
ArguGrid
Provenance
Provenance
platforms, user
environments
Grid4all
Gredia
Nessi -Grid
SIMDAT
BeinGrid
BeinGrid
industrial
simulations
business
business
experiments
experiments
NextGRID
Akogrimo
service
architecture
mobile
services
QosCosGrid
QosCosGrid
CoreGRID
g-Eclipse
GridComp
UniGrids
HPC4U
Specific support action
virtual laboratories
Integrated project
A-Ware
BREIN
agents &
semantics
XtreemOS
Linux based
Grid
operating
system
Network of excellence
Edutain
@
Edutain@
Grid
Grid
Sorma
data, knowledge,
semantics
KnowArc
K-WF
-WF Grid
Grid
K
Chemomen
Chemomen
tum
tum
InteliGrid
Datamining
Datamining OntoGrid
Grid
Grid
Specific targeted research project
Courtesy: Max Lemke, European Commission
11/26
EU project: RIO31844-OMII-EUROPE
Transforming UNICORE
• Developments
of UNICORE 6
alpha
Client
Portal
Command
Line
WS-RF
WS-RF
WS-RF
JSDL, etc.
WS-RF
SOAP
Gateway
Gateway
WS-RF
Workflow
Engine
„ Message-Layer“: WS-RF
„Data – Layer“: SOAP,…
Application – Layer: HTTPS, …
WS-RF
File
Transfer
WS-RF
User
Management
(AAA)
Network Job
Resource
VISIT WS-RF
WS-RF
WS-RF
Brokering
Management
Service
Supervisor
Secure Socket Layer: SSL
Transport – Layer: TCP, …
Internet – Layer: IPv4, …
Host-To-Network – Layer : PPP,…
12/26
EU project: RIO31844-OMII-EUROPE
Interoperability
• Interoperability between OGSA based Grid systems
• Cross–authentication for UNICORE and Globus
Grid
Client
Grid
Portal
UniGrids Atomic Services
Target
System
Factory
Target
System
Service
Unicore
Job
Management
Service
Globus
Storage
Management
Service
OGSA
Server
File
Transfer
Service
• Enable cross–Grid resource brokering
• Produce ontology for resource models (CIM, GLUE,…)
13/26
EU project: RIO31844-OMII-EUROPE
Interoperability
• Definition
in JanuaryAtomic
2005
UniGrids
Services
Target
Target
Job
Storage
File
• System
Evaluation and
refinement
in
June
2005
System
Management Management
Transfer
Service Interfaces
Service document
Servicedrafted
Service
• Factory
Execution Service
together
by UNICORE and Globus developers
– Feed in standardisation process of GGF via
OGSA Basic Execution Services (BES) Working Group
14/26
EU project: RIO31844-OMII-EUROPE
UNICORE in OMII – Europe
• Maturing and re-engineering of UNICORE 6 alpha
– NA3 Training Æ UNICORE trainings
– SA1 Repository, SA2 Quality Assurance Æ Matured UNICORE 6
• JRA1: Re-engineering of Services
–
–
–
–
–
Database Access
Virtual Organisation Management
Accounting
Job Submission and Job Monitoring
Portal Interface
• JRA3: Infrastructure Integration
– Common Security Models and Interoperation/Interoperability
15/26
EU project: RIO31844-OMII-EUROPE
JRA1 – Database Access
• OGSA – Database Access and Integration Services
– Re-engineering of
OGSA-DAI alpha
implementation Client layer
of EU Project
UniGrids
Presentatio
n layer
Client Application
GridBean
Client Toolkit
UNIGRIDS
WSRF
Client
Toolkit
WSRF Client
Stubs
WSI Client Stubs
Stubs
SOAP
Over
HTTP
SOAP
Over
HTTP
Data Services
Data Services
WSI Data Service
WSRF Data
Service
WSRF Data
Service
OGSA-DAI Core
• OGSA – DAI
services hosted
in UNICORE 6
Business
logic layer
Data layer
Data Service
Resource A
Data Service
Resource B
Relational
Database
XML
Database
Data Service
Resource C
File
16/26
EU project: RIO31844-OMII-EUROPE
JRA1 – Virtual Organisation Management
• Virtual Organisation Membership Service (VOMS)
• GGF OGSA AuthZ
– Use of SAML for
OGSA Authorization
• JRA1 will extend VOMS
to support AuthZ
standard
• Afterwards integrate
VOMS with UNICORE
VO management
– Vsites, Usites, UUDB, X.509
UNICORE
Security Model
17/26
EU project: RIO31844-OMII-EUROPE
JRA1 – Accounting
• GGF Resource Usage Services (RUS) specification
– Base upon GGF Usage Record (UR) specification
• Extending
UNICORE 6
alpha resource
usage records
– Developed in EU
project UniGrids
– Exposed via Job
Management
Service (JMS)
• New RUS Service
– Expose URs
18/26
EU project: RIO31844-OMII-EUROPE
JRA1 – Job Submission and Job Monitoring
• OGSA - Basic Execution Services (BES)
• Unigrids (Uniform Interface to Grid Services)
– Developed WS-RF- based UniGrids atomic services (UAS)
– UAS: basic interfaces for job/file management in UNICORE 6 alpha
Target
System
Factory
Target
System
Service
Job
Management
Service
Storage
Management
Service
File
Transfer
Service
• Execution Services Interface
– Draft by Argonne (Globus) and Fujitsu (UNICORE)
– Consists of UAS input and Globus GRAM demands
– Input to the OGSA - BES GGF working group
• Refactoring of UAS when OGSA-BES is revised
19/26
EU project: RIO31844-OMII-EUROPE
JRA1 – Portal Interface
• Gridsphere Portals
(JSR 168 portlet-compliant)
– Developed within the
EU GridLab project
• Prototypes of GridSphere
Portals accessing
UniGrids Atomic Services
• When OGSA-BES is
revised, standardized
access to UNICORE 6
via GridSphere
20/26
EU project: RIO31844-OMII-EUROPE
JRA3 Task 1: Common Security Infrastructure
• Aligned with already established Grid Middleware
Security Group (GMSG) of EU Project EGEE
– Chair: Olle Mulmo (KTH), GGF area director ‘Security’
– Daniel Mallmann & Morris Riedel as UNICORE developers
• Interoperability in terms of security of the four major
Grid platforms/systems
– Extend the strong security model of UNICORE
– Usage of technologies such as MyProxy, SAML,AuthZ, etc…
• Interoperable security models with UNICORE & others
21/26
EU project: RIO31844-OMII-EUROPE
JRA3 Task 2: Infrastructure Integration
• ‘Towards interoperability and integration of Grid
infrastructures for wider use in science/industry…’
• Complement the development of security aspects
– JRA3 Task 1: Common Security Infrastructure
• Broader interoperability of major Grid platforms
– First interoperation by using existing technologies (GGF GIN)
– When emerging standards become stronger,
standard-based interoperability of the major Grid Systems
• Interoperable job management with UNICORE & others
22/26
EU project: RIO31844-OMII-EUROPE
• http://unicore.sourceforge.net
• Open Source under BSD license
• Supported by FZJ
– integration of own results and
from other projects
– release management
– problem tracking, assistance
– CVS, mailing lists, documentation
• Viable basis for many other projects
23/26
EU project: RIO31844-OMII-EUROPE
• Founded by developers, leading EU HPC centres, and
supporting hardware vendors as a non-profit association
• Foster the distribution and use of UNICORE, organize
workshops, support presentations at conferences,
publish and maintain the specifications, coordinate
further development, certify implementations and
extensions
24/26
EU project: RIO31844-OMII-EUROPE
Forschungszentrum Jülich
Contacts
Morris Riedel
(m.riedel@fz-juelich.de)
Achim Streit
(a.streit@fz-juelich.de)
Daniel Mallmann
(d.mallmann@fz-juelich.de)
Bastian Tweddell
(b.tweddell@fz-juelich.de)
25/26
EU project: RIO31844-OMII-EUROPE
• August 30th – 31st, 2006
•
To be held in conjunction with Euro-Par 2006, Dresden, Germany,
August 29th – September 1st, 2006
•
Unique opportunity for Grid users, developers, administrators,
researchers, and service providers to meet
Call for Papers was published, deadline was: June 18th
Invited Talk by Dave Snelling (FLE)
•
•
•
Join us to get an inside view of UNICORE, share experiences, and
discuss future developments
• http://summit.unicore.org/2006/index.html
26/26
EU project: RIO31844-OMII-EUROPE