GridSphere’s Grid Portlets
A Grid Portal Development Framework
Jason Novotny
Lawrence Berkeley National Laboratory
Jason Novotny, Michael Russell, Oliver Wehrens
The GridSphere Project
www.gridsphere.org
Michael Russell <michael.russell@aei.mpg.de>
2004
1
The Purpose Of This Talk
The purpose of this talk is to provide a brief overview
of the Grid Portlets web application.
We will describe the basic ideas behind Grid Portlets,
what functionality it currently provides and
development plans
Michael Russell <michael.russell@aei.mpg.de>
2004
2
GridSphere’s Grid Portlets
The GridSphere portlet container is designed to be
web application independent. Indeed, one of the key
advantages of the Portlet API is the reuse of web
applications.
Thus, the GridSphere portlet container does not
contain any support for using Grid technologies.
Instead, GridSphere’s Grid related functionality is
contained in a web application we call Grid Portlets.
Grid Portlets, together with the GridSphere portlet
container, offers a generic Grid portal and can be
used to develop application-specific Grid portal
applications.
Michael Russell <michael.russell@aei.mpg.de>
2004
3
The GridLab Project
Funded by the EU (5+ M€), January 2002 – December 2004
Application and Testbed oriented
Cactus Code, Triana Workflow, all the other applications that want to be Gridenabled
Main goal: to develop a Grid Application Toolkit (GAT) and set of grid
services and tools...:
resource management (GRMS),
data management,
monitoring,
adaptive components,
mobile user support,
security services,
portals,
... and test them on a real testbed with real applications
Michael Russell <michael.russell@aei.mpg.de>
2004
4
GridLab Is An Architecture
Michael Russell <michael.russell@aei.mpg.de>
2004
5
And A Global Effort












PSNC (Poznan) - coordination
AEI (Potsdam)
ZIB (Berlin)
Univ. of Lecce
Cardiff University
Vrije Univ. (Amsterdam)
SZTAKI (Budapest)
Masaryk Univ. (Brno)
NTUA (Athens)
Sun Microsystems
Compaq (HP)
ANL (Chicago, I. Foster)
ISI (LA, C.Kesselman)
UoWisconsin (M. Livny)
Michael Russell <michael.russell@aei.mpg.de>
collaborating with:
Users!
EU Astrophysics Network,
DFN TiKSL/GriKSL
NSF ASC Project
other Grid projects
Globus, Condor,
GrADS,
PROGRESS,
GriPhyn/iVDGL,
Most of the other
European Grid Projects
(GRIDSTART)
GWEN
2004
6
GridLab Testbed Snapshot
Michael Russell <michael.russell@aei.mpg.de>
2004
7
GridLab Goals
Get Computational Scientists using the “Grid” and
Grid services for real, everyday, production work (AEI
Relativists, EU Network, Grav Wave Data Analysis,
Cactus User Community), all the other potential grid
apps
Make it easier for applications to make flexible,
efficient, robust, use of the resources available to
their virtual organizations
Dream up, prototype, and test new application
scenarios which make adaptive, dynamic, wild, and
futuristic uses of resources.
Michael Russell <michael.russell@aei.mpg.de>
2004
8
What Do Our Users Need?
Application oriented environment
Flexible, easy-to-use, simple interfaces
Efficient and effective use of resources
Robustness, fail-safety, adapability
The ability to work in distributed teams
Support for mobile working environments
Michael Russell <michael.russell@aei.mpg.de>
2004
9
What Do Our Users Want?
Larger computational resources
Memory/CPU
Faster throughput
Cleverer scheduling, configurable scheduling, co-scheduling, exploitation of unused cycles
Easier use of resources
Portals, grid application frameworks, information services, mobile devices
Remote interaction with simulations and data
Notification, steering, visualization, data management
Collaborative tools
Notification, visualization, video conferencing, portals
Dynamic applications, New scenarios
Grid application frameworks connecting to services
Michael Russell <michael.russell@aei.mpg.de>
2004
10
Many Application Scenarios!
Dynamic Staging
Dynamic Load Balancing
move to faster/cheaper/bigger machine
inhomogeneous loads
multiple grids
Multiple Universe
create clone to investigate steered parameter
Portal
Automatic Convergence Testing
User/virtual organisation interface to the grid.
from initial data or initiated during simulation
Intelligent Parameter Surveys
Look Ahead
farm out to different machines
spawn off and run coarser resolution to predict
likely future
Make use of
Spawn Independent/Asynchronous Tasks
Running with management tools such as Condor,
Entropia, etc.
Scripting thorns (management, launching new
jobs, etc)
Dynamic use of eg MDS for finding available
resources
send to cheaper machine, main simulation
carries on
Application Profiling
best machine/queue
choose resolution parameters based on
queue
Michael Russell <michael.russell@aei.mpg.de>
2004
11
Where Grid Portals Fit In
GridLab needs an effective way to deliver its tools
and services to its target users.
We want to avoid having to develop new client-side
technologies and methods for deploying them.
Given the ubiquitous Web browser and the evolution
of Grid standards and technologies, a Grid portal
provides an obvious solution to this problem.
Michael Russell <michael.russell@aei.mpg.de>
2004
12
Developing Grid Portals
Grid web application development still remains a tedious
task with little in the way of reusable components, forcing
developers to constantly “re-invent” the wheel.
Often difficult and hard to maintain glue code must be
written connecting the portal to Grid services, due to lack
of/evolving standards.
Lack of real usability has made it difficult to test and
evaluate user interfaces.
A portal is only as good as the underlying deployed
infrastructure…. portal development often involves
debugging underlying middleware
Michael Russell <michael.russell@aei.mpg.de>
2004
13
What is a Grid Portal?
“A portal is a web based
application that commonly
provides personalization, single
sign on, content aggregation from
different sources and hosts the
presentation layer of Information
Systems”(JSR 168)
Grid Portals build upon the familiar
Web portal model, such as Yahoo
or Amazon, to deliver the benefits
of Grid computing to virtual
communities of users, providing a
single access point to Grid
services and resources.
Michael Russell <michael.russell@aei.mpg.de>
2004
14
Early Grid Portal Projects
Grid-Port:
Perl based framework developed by Mary Thomas and Steve
Mock at San-Diego Supercomputing Center (SDSC)
Grid Portal Development Toolkit (GPDK):
Developed by Jason Novotny at Lawrence Berkeley National
Laboratories (LBNL)
Astrophysics Simulation Collaboratory (ASC):
Developed by Michael Russell at University of Chicago
Michael Russell <michael.russell@aei.mpg.de>
2004
15
Grid Portals Work Package
Within the GridLab Project, the Grid Portals Work
Package (WP4) is responsible for developing a Grid
portal for the GridLab Project.
The GridSphere Project was created by WP4 as it
saw a need in the general community for a Grid portal
framework.
WP4 is both developing and deploying GridSphere to
serve target user communities in the GridLab Virtual
Organization.
Michael Russell <michael.russell@aei.mpg.de>
2004
16
GridSphere 2.0
Michael Russell <michael.russell@aei.mpg.de>
2004
17
Personalized Environment
Michael Russell <michael.russell@aei.mpg.de>
2004
18
Single Sign-On Capabilities
Michael Russell <michael.russell@aei.mpg.de>
2004
19
Submit Jobs
Michael Russell <michael.russell@aei.mpg.de>
2004
20
Perform File Transfers
Michael Russell <michael.russell@aei.mpg.de>
2004
21
Manage Resources
Michael Russell <michael.russell@aei.mpg.de>
2004
22
Grid Portal Design Requirements
We require a portal that is generic and not tied to any
particular set of Grid technologies. The GridLab
Testbed, for example, is Globus 2.X based although
integartion with OGSA well under way
Develop a set of portlets and portlet services that
provide a higher level model of Grid concepts.The
Grid portlet services should provide a generic API to
which the Grid portlets are written.
Michael Russell <michael.russell@aei.mpg.de>
2004
23
Grid Portlets API
Our Grid Portlets API attempts to provide a high-level
model of Grid concepts that
Abstracts the developer from particular Grid technologies and
infrastructure.
Is extensible, can build upon its concepts to provide more
complex services, resources and tasks.
Supports multiple implementations of the API.
Configurable at run-time via administrative interfaces
The Grid Portlets API builds from basic concepts to
more advanced ones…
Michael Russell <michael.russell@aei.mpg.de>
2004
24
Basic Concepts
Users
Users should get a personalized a view of the Grid based upon
their access rights to resources and their preferences.
Credentials
Credentials are required to authenticate users to resources and
authorize their rights to utilize them.
Resources
Resources are anything that can be “utilized” (by users, portlet
services and other resources).
Tasks
Many operations are quite naturally modeled as tasks. Workflow
is not provided yet, but is desired in future releases.
Michael Russell <michael.russell@aei.mpg.de>
2004
25
More Advanced Concepts
Jobs
Job submission to LSF, PBS, etc…
Job migration from one resource to another
Files & Related Tasks
Physical file browsing (list, transfer, etc.)
Logical and replica file browsing
Information Resources
LDAP/MDS information providers
Web service based providers
Other types…
Michael Russell <michael.russell@aei.mpg.de>
2004
26
Simple Resource Diagram
Michael Russell <michael.russell@aei.mpg.de>
2004
27
Hardware Resource Model
Hardware resources are
utilized for performing
computations, storing
files, etc….
Hardware resources
contain…
Service resources (such as a
Grid FTP resource)
Software resources (such as
Globus client tools)
Hardware accounts (allowing
people to login and perform
operations in a shell).
Michael Russell <michael.russell@aei.mpg.de>
2004
28
Job Resource Model
Job resources (such as
Globus Gatekeepers) are
service resources that
enable the remote clients
to submit jobs on
computing resources.
Job resources provide
access to job schedulers
(such as PBS).
Job schedulers contain
queues with different
resource capabilities and
usage policies.
Michael Russell <michael.russell@aei.mpg.de>
2004
29
Some Job Resources
In Grid Portlets, we’ve modeled
two types of service resources
as job resources, both of which
have different capabilities for
supporting job submission.
GTX Gatekeepers provide the
ability to submit jobs directly to
job managers running on
hardware resources using the
GRAM protocol.
GridLab Resource Manager
Services (GRMS)s are web
services that provide resource
brokering capabilities and
support job submission with
GRAM as well as Condor.
Michael Russell <michael.russell@aei.mpg.de>
2004
30
Simple Task Diagram
Michael Russell <michael.russell@aei.mpg.de>
2004
31
How Portlets Are Developed
In Grid Portlets, the
presentation layer is
made up of portlets which
contain reusable UI
components.
The business logic layer
is implemented as a
collection of portlet
services which provide
some high-level
capabilities and make use
of external resources.
Michael Russell <michael.russell@aei.mpg.de>
2004
32
Job Submission Portlet
For example, the job
submission portlet uses a
number of UI
components, including file
browser components (i.e.
for specifying locations of
an executable).
These JSP components
utilize a number of portlet
services which provide
access to job resources,
file resources and so on…
Michael Russell <michael.russell@aei.mpg.de>
2004
33
Current Status
Grid Portlets is still in prototype development mode.
Most of what is their works, but API is still evolving.
Documentation is still poor… Mike promises to
improve it
Source code is available via anonymous CVS.
Will move towards a regular development and release
cycle in the coming months.
Mike in Brazil ;-) but plans on continuing to enhance
grid portlets
Michael Russell <michael.russell@aei.mpg.de>
2004
34
Current Functionality
Credential Retrieval & Management
Support for retrieval & storing with MyProxy
Resource Registry & Management
Resources can be described in XML and online
Resource Information Provider
Resource discovery and update with MDS2
Job Submission & Migration
GRAM job submission with Java CoG 1.1
GRMS 1.9 job submission and migration
File Browsing & Transferring
Support for GridFTP with Java CoG 1.1
Michael Russell <michael.russell@aei.mpg.de>
2004
35
What’s Next?
Enhancements to UI component model… These
enhancements may be incorporated into GridSphere
portlet container, but we are still reviewing JSF.
Many enhancements planned to existing portlets,
including better integration among portlets and reuse
of UI components.
Progress already underway to make JSR 168
compliant
Please submit feature requests and additions to
bugtracker and mail list
Michael Russell <michael.russell@aei.mpg.de>
2004
36
Planned Development
Support for logical files & replica management
Support for GridLab Data Management Services
Support for Globus Replica Catalogue
Enhancements to information gathering
Portlets for managing use of information providers
More resource information providers
Support for GridLab Adaptive Information Service
Support for Network Weather Service
Michael Russell <michael.russell@aei.mpg.de>
2004
37
How To Obtain Grid Portlets
Grid Portlets can be obtained from CVS….
cvs -d
:pserver:anonymous@portal.aei.mpg.de:/home/repository
co gridportlets
Required Software:
GridSphere 2.0.2 or higher.
Tomcat 4.1.29 or higher (Tomcat 5.X supported)
OGSA 3.0.2 (Plan to move OGSA 3.2)
MyProxy
Michael Russell <michael.russell@aei.mpg.de>
2004
38
Configuring Grid Portlets
Configuration amounts to specifying what resources
Grid Portlets makes available to users.
The first thing you should do is specify what MyProxy
resource your portal will use for retrieving / storing
Grid credentials.
If want to use our MDS2 information provider, then
you should specify what GIIS resource you want to
use.
Otherwise, you should make sure to specify the
hardware and service resources that make up your
Grid…
Michael Russell <michael.russell@aei.mpg.de>
2004
39
How To Specify Resources
There are 2 ways to add resources to Grid Portlets
1
2
Online with the Resource Registry Portlet
Offline with Resources.xml
Michael Russell <michael.russell@aei.mpg.de>
2004
40
How To Get Involved
Join one of gridsphere mail lists. Ask questions,
provide feedback / comments.
Submit bugs and feature requests to
http://bugs.gridsphere.org.
Because we are a small group, getting involved as a
developer means demonstrating some knowledge /
commitment to the project. Best way is to email ideas
/ patches to the list. If you’re loud enough, we’ll
probably give you commit access and a few tasks!
Michael Russell <michael.russell@aei.mpg.de>
2004
41
Projects Using Grid Portlets
AEI Numerical Relativity Portal
CCT Numerical Relativity Portal
NRC Portal
Unversity of Melborne, Australia
SDSC…
Michael Russell <michael.russell@aei.mpg.de>
2004
42
Conclusion
Grid Portlets contains the Grid functionality in the
GridSphere Project. It provides a framework for
developing Grid portals.
It is still in prototype development mode, but we are
already using it in the GridLab Project and in several
other projects with which we are affiliated.
We will make announcements on our website at
www.gridsphere.org as the status of Grid Portlets
changes.
Feel free to try it out and send us comments.
Michael Russell <michael.russell@aei.mpg.de>
2004
43