GEMLCA and P-GRADE Portal
Experiences with developing GEMLCA
portlets for the P-GRADE portal
Thierry Delaitre (Westminster)
Portals and Porlets 2006 - Edinburgh
www.cpc.wmin.ac.uk/GEMLCA
Presentation outline
• P-GRADE Portal
• Grid Execution Management for Legacy
Code Applications: GEMLCA
• The GEMLCA P-GRADE Portal
• Experiences with developing GEMLCA Portlets
for the P-GRADE portal
• Workflow level Grid interoperability
• Conclusions
www.cpc.wmin.ac.uk/GEMLCA
Motivation to the P-GRADE Portal
• Fast evolution of Grid middleware technologies and
tools:
– GT2, OGSA, GT3 (OGSI), GT4 (WSRF), LCG-2, gLite, …
• Many production Grids are now freely available for eScientists
– EGEE (LCG-2  gLite), UK NGS (GT2),
US Open Science Grid (GT2  GT4), NorduGrid (ARC), …
• The same set of services are available everywhere, but
implemented in different ways
– Computation services, data services, security services, (brokers)
Let’s provide a technology-neutral graphical
interface for the most common Grid middleware
services!
www.cpc.wmin.ac.uk/GEMLCA
P-GRADE Portal in a nutshell
• General purpose, workflow-oriented computational Grid portal.
Supports the development and execution of
workflow-based Grid applications.
• Based on standard portlet framework (Gridsphere)
– Easy to expand with new portlets (e.g. application-specific portlets)
– Easy to tailor to community needs
• Grid services supported by the portal:
Service
Job execution
File storage
EGEE grids
Globus grids
Computing Element
GRAM
Storage Element
GridFTP server
Certificate management
Information system
Brokering
MyProxy
BDII
MDS-2
Workload Management System
---
Job monitoring
Mercury
Workflow & job visualization
PROVE
The P-GRADE Portal hides middleware technologies
and solves Grid interoperability problem at the
workflow level
www.cpc.wmin.ac.uk/GEMLCA
What is a P-GRADE Portal workflow?
• a directed acyclic graph
where
– Nodes represent jobs (batch
programs to be executed on a
computing element)
– Ports represent input/output
files the jobs expect/produce
– Arcs represent file transfer
operations
• semantics of the
workflow:
– A job can be executed if all
of its input files are available
www.cpc.wmin.ac.uk/GEMLCA
Two levels of parallelism by a workflow
• The workflow concept of
the P-GRADE Portal
enables the efficient
parallelization of complex
problems
• Semantics of the
workflow enables two
levels of parallelism:
– Parallel execution
inside a workflow
node
– Parallel execution
among workflow
nodes
Multiple jobs can
run parallel
www.cpc.wmin.ac.uk/GEMLCA
The job can be a
parallel program
The P-GRADE Grid Portal
Provides user-friendly access to the Grid
Log in
to the
portal
Visualise
execution and
download
results
Execute
workflow
Create
application
workflow
Map
execution to
available
resources
www.cpc.wmin.ac.uk/GEMLCA
Grid Enabling Legacy Applications
• Code from the past, maintained because it works
• Often supports business critical functions
• Not Grid enabled
What to do with legacy codes when utilising the Grid?
•
Bin them and implement Grid enabled
applications
•
Reengineer them
•
Port them onto the Grid with
minimum user effort
www.cpc.wmin.ac.uk/GEMLCA
GEMLCA Concept
Code
is
Compute
deployed
Invoke Servers
statically
legacy
code
Custom input
Client 2: to
invoke
result
legacy codeInputReturn
data is
Grid service provided
GEMLCA
Grid service
dynamically
Client 1: to
register
legacy code
as a Grid
service
Access to the code
is controlled by the
GEMLCA Grid
service
www.cpc.wmin.ac.uk/GEMLCA
Legacy
code
Produce
result
Resource
manager deploys:
LC
GEMLCA
The GEMLCA-view of a legacy code
• Any code that correspond to the following model
can be exposed as Grid service by GEMLCA:
Input data
(files, command
line params, env.
vars)
Legacy code
Output data
(files)
...
Perform computation
Communicate over the network
Query databases
Call shared libraries
www.cpc.wmin.ac.uk/GEMLCA
The GEMLCA P-GRADE Portal
• To provide graphical clients to GEMLCA
with a portal-based solution (Web-based
GEMLCA client environment)
• To enable the integration of legacy code
grid services into workflows
www.cpc.wmin.ac.uk/GEMLCA
GEMLCA and the P-GRADE portal
3rd generation Grids
Web browser
Grid Site 1
Desktop 1
GEMLCA
P-GRADE
Portal
LC 1
Legacy applications
Grid Site 2
Web browser
LC 2
Legacy applications
Desktop N
www.cpc.wmin.ac.uk/GEMLCA
Combining legacy and non-legacy components
Code
invocation
Combine legacy codes
with new codes
inside the same workflow!
Code
invocation
Job
submission
Code
invocation
Job
submission
www.cpc.wmin.ac.uk/GEMLCA
The GEMLCA-specific version of the
P-GRADE Portal is different from the
original P-GRADE Portal!
• It contains a web page (portlet) to register legacy
codes as grid services
• It contains a web page (portlet) to monitor
GEMLCA/Grid services
• It contains a GEMLCA-specific workflow editor
– Workflow components can be “legacy code grid
services” (not only batch jobs)
• It contains a GEMLCA-specific workflow
manager subsystem
– It can invoke GEMLCA services (not only submitting
jobs)
www.cpc.wmin.ac.uk/GEMLCA
Legacy code registration page
Mkdir Legacy Code exposed as a Grid Service
Folder : /../.gemlca/legacycodes/mkdir
Content : i) mkdir binary or link ii) config.xml
Legacy Code Interface Description File: config.xml
<?xml version="1.0"?>
<!DOCTYPE GLCEnvironment "gemlcaconfig.dtd">
<GLCEnvironment
id="mkdir" executable="LINUX/mkdir" jobManager="Fork"
maximumJob="11" minimumProcessors="1"
maximumProcessors="1" universe="PVM"
>
<Description>Unix mkdir program</Description>
<GLCParameters>
<Parameter name="-p" friendlyName="Folder to be created"
fixed="No" inputOutput="Input" order="0"
mandatory="No" fileCommandline="Commandline">
<initialValue> </initialValue>
</Parameter>
</GLCParameters>
</GLCEnvironment>
www.cpc.wmin.ac.uk/GEMLCA
GEMLCA/Grid Monitoring Tool
www.cpc.wmin.ac.uk/GEMLCA
GEMLCA/Grid Monitoring Tool
Resource testing for administrators and users
3rd party service providers
Production Grid
(e.g. EGEE, NGS)
MDS4
GMT
(GEMLCA
Monitoring
Toolkit)
Portal
Run GMT
probes and Grid site with
collect data legacy code
resources
User
Map
execution
Grid site with
legacy code
resources
www.cpc.wmin.ac.uk/GEMLCA
GEMLCA Specific Workflow editor
UoW
Sztaki
UoR
www.cpc.wmin.ac.uk/GEMLCA
Experiences with GEMLCA portlets development
for the P-GRADE portal
•
GEMLCA Legacy code registration portlet
•
Grid Monitoring Tool porltet
www.cpc.wmin.ac.uk/GEMLCA
Control flow and UI design
of the GEMLCA LCID portlet
www.cpc.wmin.ac.uk/GEMLCA
Implementing control flow and UI
in GEMLCA LCID portlet
We use GS UI
Beans extensively
doAdmin(RenderFormEvent evt) {
GEMLCAAdminPortletBean AdminBean = getBean(evt);
checkStatus(evt);
if (AdminBean.getSelectedGrid() == null) {
State = "gemlca/gridSelect.jsp";
fillListBox("GridSelect", evt,
toStringArray(AdminBean.getGrids(),
"getName"));
prepareHeader(evt, "GEMLCA Grid selector");
} else if (AdminBean.getSelectedResource() == null) {
State = "gemlca/resourceSelect.jsp";
<%@ taglib uri="/portletUI" prefix="ui" %>
<%@ taglib uri="/portletAPI" prefix="portletAPI" %>
<portletAPI:init/>
<ui:panel>
<ui:form>
<ui:frame>
<ui:tablerow header="true">
<ui:tablecell><ui:text beanId="MainHeader"/>
</ui:tablecell>
</ui:tablerow>
<ui:tablerow>
<ui:tablecell>
<ui:listbox beanId="GridSelect" name="GridSelect"/>
<ui:actionsubmit action="StoreGrid" value="Select Grid"/>
</ui:tablecell>
</ui:tablerow>
<ui:tablerow><ui:tablecell>
<ui:actionsubmit action="Reset" value="Reset"/>
</ui:tablecell></ui:tablerow>
<ui:tablerow header="true">
<ui:tablecell><ui:text beanId="MessageHeader"/></ui:tablecell>
</ui:tablerow>
</ui:frame>
</ui:form>
</ui:panel>
www.cpc.wmin.ac.uk/GEMLCA
Experiences with implementing GEMLCA LCID portlet
Implementation of the portlet
•
Extensively use the GS UI beans.
•
Beans built in Java are rendered by JSP pages
Problems encountered
•
•
•
Several inconsistencies between the tag libraries and the capabilities of the beans
provided by gridsphere.
the tld have had some restrictions which did not allow access to specific attributes
from the JSP code (workaround is to do it in Java).
Same happens from the Java code where some features do not work (workaround is
to do it JSP or hack the JSP generated code and insert it into Java)
•
Example is with the UIBean with disabled mode features
•
•
•
ActionSubmitBean.addParamBean (parameters not generated correctly from Java)
button.setAction("gs_action=" + Action + (Parameter == null ? "" : "&up=" + prefix
+"&"
+ prefix + "_"+ Parameter + "=" + PValue));
Portlet modified to handle Inter-portlet communication from GS1
(FormEvents) to GS2 (Action or Render).
www.cpc.wmin.ac.uk/GEMLCA
Experiences with implementing GMT portlet
•
GMT uses WebMDS servlet as its base
•
GMT portlet was developed in GS1 and upgraded to GS2
•
In GS1, portlets were inherited from HTTPServlet
•
In GS2, portlets are no longer subclass of HTTPServlet, portlets
are subclass of GenericPortlet coming from JSR168. Problems
with obtaining information in the same way as GS1. For example
where the servlet is located. This is solved using the following:
•
response.createRenderURL().toString()
www.cpc.wmin.ac.uk/GEMLCA
Experiences with operational issues
•
Increase JVM memory
•
•
JVM Tuning:
»
-Xms256m (initial heap size)
»
-Xmx1024m (max heap size)
Potential 64 bits issue with 32 bits JNI C code
•
-verbose:jni -Xcheck:jni -XX:+ShowMessageBoxOnError
•
Use Jprofiler for memory profiling or review your code
•
P-Grade portal (based on GridSphere) is being used in production
for EGEE (VOCE, SEE-Grid, HunGrid, Gilda, GIN, NGS)
www.cpc.wmin.ac.uk/GEMLCA
GEMLCA on the UK NGS
The P-GRADE NGS GEMLCA Portal
•
•
Portal Website: http://www.cpc.wmin.ac.uk/ngsportal/
Runs both GT4 GEMLCA and GEMLCA repository
Westminster
www.cpc.wmin.ac.uk/GEMLCA
Workflow Level Grid Interoperability
By GEMLCA and the P-GRADE Portal
P-GRADE
GEMLCA
Portal
GEMLCA
Repository
www.cpc.wmin.ac.uk/GEMLCA
References
• P-GRADE Portal service is available for
–
–
–
–
–
UK National Grid Service (with GEMLCA support)
SEE-GRID infrastructure
HUNGRID VO of EGEE
Central European VO of EGEE
GILDA: Training VO of EGEE
• Under preparation for
– US Open Science Grid, Economy-Grid,
Swiss BioGrid, Bio and Biomed EGEE VOs
• P-GRADE portal can be installed for any
public/private EGEE or Globus Grid/VO
Please contact us!
www.cpc.wmin.ac.uk/GEMLCA
How to get access?
– Take a look at www.lpds.sztaki.hu/pgportal
(manuals, training events, client & server requirements,
installation procedure, etc.)
– Visit or request a training event! (pgportal@sztaki.hu)
• Lectures, demos, hands-on tutorials, application development support
– Get an account for one of its production installations:
•
•
•
•
NGS portal – University of Westminster
www.cpc.wmin.ac.uk/gngsportal
VOCE portal - SZTAKI
SEEGRID portal – SZTAKI
www.lpds.sztaki.hu/pgportal
HUNGrid portal – SZTAKI
– If you are the administrator of a Grid/VO then contact
SZTAKI to get your own P-GRADE Portal!
– If you know the administrator of a P-GRADE Portal you can
ask him/her to give access to your Grid through his/her portal
installation! (Multi-Grid portal)
www.cpc.wmin.ac.uk/GEMLCA
Conclusions
– We have been using GridSphere since the GridLab
project started and migrated all GEMLCA portlets from
GS1 to GS2 since last year.
– Good experience using GridSphere for developing
portlets for the GEMLCA P-Grade portal.
– P-Grade portal is available today as a production service
– What’s next ?
• Provide a multi Grid portal service to major Grids
• Focus on scalability issues
www.cpc.wmin.ac.uk/GEMLCA
Thank you for your attention …
Any
questions?
Email:
gemlca-discuss@cpc.wmin.ac.uk
Websites:
www.cpc.wmin.ac.uk/gemlca
www.lpds.sztaki.hu/pgportal
www.cpc.wmin.ac.uk/GEMLCA