http://www.grid-support.ac.uk
http://www.ngs.ac.uk
NGS in the future: emerging
middleware
Mike Mineter
mjm@nesc.ac.uk
http://www.nesc.ac.uk/
http://www.pparc.ac.uk/
http://www.eu-egee.org/
Goal of talk
• The NGS is running a production service
• Different middleware may be deployed in the
future.
• The talk seeks to outline some of the
possibilities
2
NGS middleware evolution
EGEE…
Other software
sources
Prototypes &
specifications
‘Gold’ services
NGS
ETF
Software with proven
capability & realistic
deployment
experience
Operations
UK,
Campus
and other
grids
Feedback & future requirements
Engineering Task Force
Deployment/testing/advice
3
Outline
• Middleware currently being prepared for
deployment
– Resource broker
• Under assessment:
– gLite middleware from EGEE
– OMII
– GT4
4
Resource broker
• (This is NOT the SRB!!!)
• Current NGS middleware : Toolkits inviting
development of higher level services
• On the current NGS we have
– GRAM to submit jobs
– Information service to tell us what queues are busy
• The RB takes the work out of deciding where to
run a job
5
Major components
“User
interface”
Input “sandbox”
Output “sandbox”
DataSets info
Replica
Catalogue
Information
Service
Resource
Broker
Publish
Logging &
Book-keeping
Job Query
Job Submit Event
Author.
&Authen.
Storage
Element
Job Status
Computing
Element
Resource broker
• Job Description Language file: describes
resources needed by a job
• Commands analogous to GT2:
– edg-job-submit <jdl filename>
– edg-job-status <dg-job-id>
– edg-job-get-output <dg-job-id>
7
Example
• edg-job-submit myjob.jdl
– Myjob.jdl
• JobType = “Normal”;
• Executable = "$(CMS)/exe/sum.exe";
• InputSandbox = {"/home/user/WP1testC","/home/file*”,
"/home/user/DATA/*"};
• OutputSandbox = {“sim.err”, “test.out”, “sim.log"};
• Requirements = other. GlueHostOperatingSystemName ==
“linux" &&
• other. GlueHostOperatingSystemRelease == "Red Hat 7.3“
&& other.GlueCEPolicyMaxCPUTime > 10000;
• Rank = other.GlueCEStateFreeCPUs;
8
More about the RB
• Developed by the European DataGrid project, EDG then
“hardened” by LCG, and now one of the sources for the
EGEE middleware
• Uses components of Condor
– matchmaker and Condor-G
• Try the GENIUS portal on GILDA
– GILDA is a dissemination grid running the LCG-2 middleware
– Demo site: https://grid-demo.ct.infn.it/
• And look at
http://lcg.web.cern.ch/LCG/
http://www.hep.ph.ic.ac.uk/escience/projects/demo/index.html
9
Resource broker summary
• The resource broker receives a job description
in JDL
• It chooses a batch queue for job submission,
using the information services
• Its an example of the higher services that can be
deployed for the NGS, built upon the current
toolkits
10
Outline
• Middleware currently being prepared for
deployment
– Resource broker
– (NGS portal – yesterday!)
• Under assessment:
– gLite middleware from EGEE
– OMII
– GT4
11
EGEE – towards e-infrastructure
Enabling Grids for E-sciencE
EGEE is building a large-scale
production grid service to:
• Underpin research,
technology and public service
• Link with and build on
national, regional and
international initiatives
• Foster international
cooperation both in the
creation and the use of the einfrastructure
INFSO-RI-508833
Pan-European Grid
Operations, Support and
training
Collaboration
Network
infrastructure
& Resource
centres
12
In the first 2 years EGEE will
Enabling Grids for E-sciencE
• Establish production quality sustained Grid
services
– 3000 users from at least 5 disciplines
– integrate 50 sites into a common
infrastructure
– offer 5 Petabytes (1015) storage
• Demonstrate a viable general process to
bring other scientific communities on board
• Propose a second phase in mid 2005 to take
over EGEE in early 2006
INFSO-RI-508833
Pilot
New
14
EGEE Organisation
Enabling Grids for E-sciencE
• 70 leading institutions in 27
countries, federated in regional Grids
• ~32 M Euros EU funding for first 2
years starting April 2004
(matching funds from partners)
• Leveraging national and regional grid
activities
• Promoting scientific partnership
outside EU
INFSO-RI-508833
15
Activities Definition
Enabling Grids for E-sciencE
•
Network Activities
–
–
–
–
–
•
NA1: Project Management
NA2: Dissemination and Outreach
NA3: User Training and Induction
NA4: Application Identification and Support
NA5: Policy and International Cooperation
Service Activities
– SA1: Grid Support, Operation and Management
– SA2: Network Resource Provision
•
Joint Research Activities
–
–
–
–
JRA1: Middleware Reengineering + Integration
JRA2: Quality Assurance
JRA3: Security
JRA4: Network Services Development
INFSO-RI-508833
Emphasis in EGEE is on
operating a production
grid and supporting the
end-users
16
gLite: Guiding Principles
Enabling Grids for E-sciencE
•
VDT
EDG
...
AliEn
LCG
...
Service oriented approach
– Allow for multiple interoperable
implementations
•
Lightweight (existing) services
– Easily and quickly deployable
– Use existing services where
possible
 Condor, EDG, Globus, LCG, …
•
Portable
– Being built on Scientific Linux and
Windows
•
•
Security
– Co-existence with LCG-2 and OSG
(US) are essential for the EGEE
Grid services
– Sites and Applications
•
Performance/Scalability &
Resilience/Fault Tolerance
•
Site autonomy
– Reduce dependence on ‘global,
central’ services
– Comparable to deployed
infrastructure
•
INFSO-RI-508833
Co-existence with deployed
infrastructure
Open source license
17
gLite Services for Release 1
Enabling Grids for E-sciencE
JRA3
Grid Access
Service
CERN
API
Access Services
Authorization
Information &
Monitoring
Auditing
Authentication
Security Services
Metadata
Catalog
File & Replica
Catalog
Storage
Element
Data
Management
IT/CZ
Application
Monitoring
Information &
Monitoring Services
Accounting
Job
Provenance
Package
Manager
Site Proxy
Computing
Element
Workload
Management
Data Services
INFSO-RI-508833
UK
Job Management Services
18
Open Middleware
Infrastructure Institute

The slides that follow were selected and (in a
few cases) modified by Mike Mineter (NeSC)
from those presented in January 2005 at an
OMII training day
Steven Newhouse, Peter Henderson
Stephen Crouch & Karen Ng

Goal of this presentation: to raise awareness
of the OMII and its OMII_1 release
MM
19
Open Middleware
Infrastructure Institute
OMII goal: to be the source of open source grid
software





Institute of the University of Southampton
Utilise existing software and standards
Production focused software development
Integrate, test & document ‘a product’
Focus on the user experience


Easy to install & use
Utilise existing software and standards

Provide a solid web service base for others to build on
20
Where does our software come
from?

Open Source Community


Software Repository



Tomcat, Axis, etc.,
Accept software contributions
Software deployed, tested & graded to provide
feedback
Managed Programme


Fill gaps to build a solid enabling infrastructure
Projects to bring research software to production
quality
21
Managed Programme









GridSAM (Job Submission & Monitoring service)
BPEL (Workflow service)
Grimoires (Registry service based on UDDI)
FIRMS (Reliable messaging)
FINS (Notification)
GeodiseLab (Matlab toolbox)
WSRF::Lite integration
OGSA-DAI (Database service)
WSeSS (Using SSH to tunnel requests to resources)
22
OMII_1 release:
A basic File-Compute Grid




Enables a generic computational task
Move input data from the client to the service
provider
Process the data using an application on the
service provider
Retrieve the output data from the service
provider
23
OMII Server Infrastructure
PBAC
ExampleService
TestService
Job
Data
Allocation
Account
Resource Acct
Mgmt
Mgmt
Servlet Servlet
Happy
Axis
WS-Security
AXIS
Static Webpage
TOMCAT
24
Try out the OMII_1 client !



Register at www.omii.ac.uk & login
Goto the downloads page
Download the client distribution



SuSE 9.0
 Client may work on other Linuxs but no exhaustive testing
Windows XP (SP 1 & 2)
Distribution requires JDK 1.4.2_04



Does not work with ‘just’ a JRE
Will not work with JDK 1.4.2_05/06 & JDK 1.5.0
No testing with earlier JDKs.
25
Outline
• Middleware currently being prepared for
deployment
– Resource broker
– (NGS portal – yesterday!)
• Under assessment (by ETF for NGS):
– gLite middleware from EGEE
– OMII
– GT4
26
A Service-Oriented
Infrastructure
Users
• Service-oriented applications
– Wrap applications as
services
– Compose applications
into workflows
• Service-oriented
infrastructure
– Provision physical
resources to support
application workloads
Composition
Workflows
Invocation
Appln
Service
Appln
Service
Provisioning
•Carl Kesselman at Globus Week, NeSC, 4th April
– 8th April 2005
27
Summary
• Middleware currently being prepared for deployment
– Resource broker
– (NGS portal – yesterday!)
• The context for the next generation of middleware:
service orientation, based on WS and the emerging
standards
• Under assessment by the Engineering Task Force:
– gLite middleware from EGEE
– OMII
– GT4
28