PGRADE Portal Extensions
for UK NGS Users
Tamas Kiss, Tamas Kukla, Gabor Terstyanszky
1st P-GRADE Portal User Community Workshop
June 10-11, Zurich, Switzerland
https://grid2-portal.cpc.wmin.ac.uk
The UK National Grid Service (NGS)
•
The National Grid Service (NGS) aims to enable coherent electronic access for UK
researchers to all computational and data based resources and facilities required to
carry out their research, independent of resource or researcher location.
Partner Sites:
Affiliates:
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Cardiff University
University of Glasgow Computing Service
University of Glasgow ScotGrid
Lancaster University
University of Manchester
University of Oxford
Queens University of Belfast
STFC/RAL
University of Westminster
White Rose Grid (Leeds)
University of Birmingham (GridPP)
University of Bristol
Brunel University (GridPP)
Durham University (GridPP)
University of Edinburgh (ECDF)
Imperial College London (GridPP)
Keele University
University of Liverpool (GridPP)
University of Manchester (GridPP)
University Of Oxford (GridPP)
University of Reading
Royal Holloway, University of London (GridPP)
University of Sheffield
University Of Southampton
STFC SCARF
STFC Ceramics and Minerals Consortium (Mott) VO
University of York
https://grid2-portal.cpc.wmin.ac.uk
University of Westminster as an NGS
Partner Site
• Resources and services offered:
• 64 x IBM x3455 2xDual core AMD Opteron 2218
(256 Cores with 4GB memory)
• 11TB of useable storage
• Pre-WS and WS components of the Globus Toolkit
(VDT 1.8.1)
• NGS-P-GRADE portal
• GEMLCA Application Repository
https://grid2-portal.cpc.wmin.ac.uk
NGS P-GRADE GEMLCA Portal
•
•
•
•
•
Operated by the University of Westminster as NGS Partner Site
Interface for NGS GT2 sites
Interface for GT4 Westminster site
Interface for EGEE GILDA sites
Connected to the NGS and the
GILDA Resource Brokers
Westminster
https://grid2-portal.cpc.wmin.ac.uk
Main Features of NGS P-GRADE GEMLCA Portal
• Based on P-GRADE portal 2.4.1(no parameter study support)
• Extensions:
–
–
–
–
Full integration with the GEMLCA Application
Repository
Support for OGSA-DAI: browser portlet and workflow
level integration
Support for SRB: browser portlet and workflow level
integration
Support for executing and embedding non-native
workflows (Triana, Taverna, Kepler)
https://grid2-portal.cpc.wmin.ac.uk
GEMLCA / P-GRADE Portal in a nutshell
• P-GRADE Portal extended with GEMLCA back-end
– Sharing jobs and legacy codes as workflow components
– GEMLCA is a grid service implemented by UoW
• A step towards collaborative e-Science
• Support for Globus 4 grids (besides GT2 and EGEE)
LCG / gLite VOs
Globus 2 VOs
P-GRADE
Portal
GEMLCA
Legacy
code
repository
https://grid2-portal.cpc.wmin.ac.uk
Globus 4 VOs
GEMLCA Legacy Code Repository
Share your applications with authorised colleagues
• Publish your codes in the GEMLCA repository - advanced users only
• Browse the repository and simply select the application to include in
your workflow
– ports are defined automatically
– command line parameters are displayed automatically
https://grid2-portal.cpc.wmin.ac.uk
Define value of
input
parameters or
accept default
The GEMLCA Concept
Compute
Submit Servers
legacy
code job
Send custom input
End-user:
invoke
legacy code
Grid service
Return result
GEMLCA
Grid service
and code
repository Produce
result
LCs
Code owner:
register
legacy code
as a Grid
service
Resource
manager
deploys
GEMLCA
https://grid2-portal.cpc.wmin.ac.uk
LCs
The GEMLCA Service
A GT4 based Grid Service
GT4
Resources
GEMLCA Grid Service
Grid
service
client
Frontend
GEMLCA
Grid
Services:
GLCList
SOAP
XML
Service
Invocation
GLCProcess
GLCAdmin
Core
Management
of internal
GEMLCA
concept and
environment
Back-end
GT4/GRAM4
support
GT2
Resources
GT2/GRAM2
support
LCG/g-Lite
Resources
LCG/gLite
support
https://grid2-portal.cpc.wmin.ac.uk
EGEE
Broker
Extending the data handling capabilities of P-GRADE portal
Grid infrastructure
Portal
server
LOCAL INPUT
FILES
LOCAL
OUTPUT
FILES
GridFTP
servers
User level
storage
LOCAL INPUT
FILES
LOCAL
OUTPUT FILES
Workflow level
Interoperation of local,
GridFTP, SRM and
SRB file catalogues
and databases
exposed by OGSA-DAI
REMOTE
INPUT
FILES
REMOTE
OUTPUT
FILES
Computing
resources
SRB
servers
Data
manipulation
Input to
workflows
Control of remote
input/output
Output from
workflows
EGEE
Storage
elements
https://grid2-portal.cpc.wmin.ac.uk
OGSADAI
services
SRB Extension to P-GRADE portal
SRB: SDSC (San Diego SuperComputing) Storage Resource Broker
•
Access files (and database objects) seamlessly across a
distributed environment
•
•
•
basically a distributed logical file system
single global logical namespace or file hierarchy
Physical location and the way the data is stored is abstracted
from the user
•
•
presents the user with a single file hierarchy for data distributed across
multiple storage systems
provides a way to access data sets and resources based on their logical
names or attributes rather than their names and physical locations.
https://grid2-portal.cpc.wmin.ac.uk
SRB Extension to P-GRADE portal
SRB Browser Portlet
•
•
•
Plug-in structure: generic (SRB, GridFTP, SRM) file browser can be easily implemented
Basic file and directory operations (create, delete, copy etc.)
Metadata handling
https://grid2-portal.cpc.wmin.ac.uk
•
•
SRB Extension to P-GRADE portal
SRB Port Definition
Select Mdas file –
defines Grid
0
1
2
3
Click File
Browser
SRB and GridFTP ports can be
mixed
Executor site and SRB resource can
be in different Grids
https://grid2-portal.cpc.wmin.ac.uk
Use selected Mdas to
connect to resource and
browse tree
OGSA DAI Extension to P-GRADE portal
•
Open Grid Services Architecture Data Access and Integration project is
concerned with constructing middleware to assist with access and integration of
data from separate data sources via the grid.
• An engineered extensible framework
for data access and integration.
• Expose heterogeneous data resources
to a grid through web services.
• Interaction with data resources:
–
–
–
•
Customise for your project using
–
–
–
•
Source: GGF 16, Feb 2006 by Neil Chue Hong
Queries and updates.
Data transformation / compression
Data delivery.
Additional Activities
Client Toolkit APIs
Data Resource handlers
A base for higher-level services
–
federation, mining, visualisation,…
https://grid2-portal.cpc.wmin.ac.uk
OGSA DAI Extension to P-GRADE portal
A set of browser portlets, e.g. Query Manager portlet
Select
OGSADAI
resource
Define
query
statement
Define
target
location
https://grid2-portal.cpc.wmin.ac.uk
Data can be
sliced up
and
compressed
before
delivery
OGSA DAI Extension to P-GRADE portal
OGSA-DAI integration to P-GRADE portal workflows via GEMLCA
Workflow
OGSA-DAI
node
Computational
resources
GEMLCA
repository
...
OGSA-DAI
client
OGSA-DAI
service
submit
...
OGSA-DAI
client
Set custom
parameter
values
Database
The solution is generic as any workflow
engine can be made capable to
communicate with the GEMLCA service
(GT4 based Grid service)
https://grid2-portal.cpc.wmin.ac.uk
OGSA DAI Extension to P-GRADE portal
OGSA-DAI integration to P-GRADE portal workflows
Executes
the given
OGSADAI query
Generates sampler
queries
Selecting
Grid
Selecting legacy
code exposing
OGSA-DAI
functionality
Selecting
executor site
Analysis on
the sample
data
Gathering
results
Setting
OGSA-DAI
service URL
Setting
Database
Resource ID
https://grid2-portal.cpc.wmin.ac.uk
Setting query file:
can be uploaded or
produced by another
job
Heterogeneous workflow nesting
Embedding non-native workflows via GEMLCA
Host workflow system
P-GRADE portal
GEMLCA
client
User selects the required
workflow engine, uploads the
workflow, the input
parameters and input files.
NGS
GEMLCA service
Deployed apps
WF Engine 1
WF Engine 2
WF Engine 3
Backends
GT2
GT4
gLite
EGEE
Executable: WF engine (that will
be submitted)
WF to execute: an input
parameter of the GEMLCA job
https://grid2-portal.cpc.wmin.ac.uk
Heterogeneous workflow nesting
Executed
on EGEE
Embedding non-native workflows via GEMLCA
Executed
on EGEE
Taverna
workflow
Triana workflow
Submitted
to EGEE
Submitted
to NGS
Submitted
to EGEE
Kepler
workflow
P-GRADE
workflow
Executed
on NGS
https://grid2-portal.cpc.wmin.ac.uk
Thank you for your attention!
Any questions?
kisst@wmin.ac.uk
https://grid2-portal.cpc.wmin.ac.uk