Holding slide prior to starting
show
Portals for Workflow
Composition and Service
Management
David W. Walker
School of Computer Science
Cardiff University
http://www.cs.cf.ac.uk/User/David.W.Walker/
System Portals
• We need portals to be able to handle tasks
such as: :
– Problem specification, resource discovery and job
composition
– Service deployment and lifecycle management
– Submission, execution, monitoring, and (maybe)
interaction
• These tasks are quite generic, rather than
application-specific.
• Portal can behave rather like a “Grid Shell”
Web Services
• Everything is a (Web/Grid) service.
• This includes:
– Computation routines
– Access to files and databases
– Components of the Grid infrastructure,
such as workflow enactment engines,
resource monitors, etc.
A Common Approach to
Workflow
• Visual service composition.
• Service interfaces and other metadata
expressed in an XML-based service
description document.
• Services registered with, and
discovered through, a registry.
Visual Service Composition
Portlet
• Applications and high-level services are
composed from interacting services.
• Composition can be done graphically using a
“drag and drop” style of interface to draw a
workflow.
• In a workflow graph the nodes represent
services and the edges represent data flow
or control flow.
• Need special nodes to represent loop and
conditional constructs.
• Need query mechanism to discover services
Workflow Description
• A workflow can be represented with an
XML-based workflow description
language such as WSFL, SWFL,
SCUFL, BPEL4WS, etc.
• A workflow description document can
be passed to a workflow enactment
engine, which provides an execution
environment for the workflow.
Two Hard Problems
• Semantic specification of applications
• Scheduling of workflow nodes on distributed
resources.
– Early binding model: bind to specific
service/platform at composition time (“validation”).
– Intermediate binding model: bind at “compile” time
(when converting from XML to executable form).
– Late binding model: bind dynamically at runtime.
• Later binding allows the use of more up-todate information to make scheduling
decisions.
Mobility
• Should (some) services be mobile
and/or clonable?
• Allows the automatic creation of new
services that are heavily used.
• May be important for autonomic grids.
• Generalises current data migration and
replication concepts.
Communication and
Notification
• Another important issue is how to carry
out the data flow and control flow in a
workflow.
• Need an asynchronous messaging
system to support interoperability and to
handle connection problems.
Portlets
• Portlet technology endorsed by GCE working
group of GGF, e.g., JetSpeed
• Web services have user-facing ports that
provide content to users.
• Portlets can be regarded as interfaces to that
content, specified via an extension to WSDL.
• Portals can be built from collections of
portlets, e.g., as a set of panels for job
submission, job status, visualisation, etc.
• Portlets provide as component model for user
interfaces in the same way that web services
provide a component model to middleware.
Grid-Oriented Portal Work at
Cardiff
• Visual Service Composition Environment (VSCE) for
building applications based on service workflow.
• Service Workflow Language (SWFL) for describing
composite applications.
• Tools for transforming between workflow graph 
SWFL  executable code.
• Jini Service-oriented Grid Architecture (JISGA). An
execution environment taking SWFL as input.
• Java-C Automatic Wrapper (JACAW). Wraps C code
as Java using JNI for deployment as Web service in
JISGA.
• Grid-Service-in-a-Box (GSiB). Supports service
creation, deployment, management, and monitoring.
Motivation and Targets
• Easy-to-use tools are needed to allow
scientists to take full advantage of the
new Grid without them needing detailed
knowledge of the underlying
infrastructure.
• GSiB project is developing a visual
problem-solving environment/portal for
all service users (clients and providers).
Jini Services for the Grid
• Jini system has a similar serviceoriented architecture (SOA) to the Web
service approach.
• JavaSpaces provides communication
middleware.
Service Registry
discovery
publish
Lookup Service
publish
Service Provider
discovery
binding
invoke
Service Requester
GSiB Interfaces and
Functionalities
• GSiB consists of two packages:
– The service provider GUI package
– The service client GUI package
The Service Provider GUI
Package
• Service Deployment/Undeployment
Interface
– Automatic deployment of existing software
routines or libraries as OGSA-compliant Grid
services.
– Visual Service Composition Environment (VSCE)
for building and deploying a composite service.
– Updating/undeployment of existing Grid Services.
Consideration of how to avoid interrupting higherlevel services and running applications.
The Service Provider GUI
Package
• Service Configuration interface for setting up
or changing a service’s security policy and
lifetime.
• Service Monitor Interface for dynamically
monitoring the distribution, performance and
workload of Grid services and resources.
• Other interfaces include Authorization
interface, and Lifecycle Management
Interface.
The Service Client GUI
Package
• Visual Service Composition
Environment (VSCE) for building a
composite Grid application.
– It allows a user to create a representation
of a Grid application by drawing its
workflow.
– Then VSCE transforms the graphic
workflow model into a XML-based
workflow description document.
The Service Client GUI
Package
• Workflow Engine component provides
an execution environment for Grid
applications described in an XML-based
workflow description language.
• Job Tracking and Monitoring Interface
• Service Browser for browsing and
querying services.
Service Deployment
• A web service
– Its public interfaces and bindings are described in
an XML document.
– It is registered in a service registry to allow service
discovery and query.
• Usually additional code or wrapper is needed
to make existing software available as a Web
service – a lot of repeated work.
• Legacy problems exists.
Service Deployment
• JACAW (Java-C Automatic Wrapper)
automatically generates the JNI
wrapping code for a legacy code.
• Axis Java2WSDL utility generates
WSDL document from the Java code.
• UDDI registry server.
• UDDI4J provides a Java API for
interacting with a UDDI registry.
Service Deployment
Legacy code
JACAW
JAVA2WSDL
WSDL Document
JNI Wrapper
UDDI4J
UDDI Registry
Service Composition
• Graphical symbols for constructing a
flow model.
Data input port
Data output port
Normal and
assignment activity
Control input port
Control output port
Data link
Control link
Control activity
An Example
retA = activityA(..);
A
for( i=0; i<100; i++){
retA
activityB(retA, i);
For
NO
}
retC = activityC();
retA
i
B
YES
C
Service Workflow Language
(SWFL)
<activity name="for">
<for setParallel="no">
<input message=“ForInput”/>
<expression><![CDATA[index=0;index<100;index++]]></expression>
</for>
</activity>
<dataLink name=“A_For” source=“A” target=“for”>
<swflMap sourceMessage=“AOutput” targetMessage=“ForInput”>
<part>
<sourcePart name=“retA”/><targetPart name=“retA”/>
</part>
</swflMap>
</dataLink>
To be continued
Service Workflow Language
(SWFL)
<dataLink name=“For_B” source=“for” target=“B”>
<swflMap sourceMessage=“ForOutput” targetMessage=“BInput”>
<part>
<sourcePart name=“retA”/><targetPart name=“retA”/>
<sourcePart name=“i”/><targetPart name=“i”/>
</part>
</swflMap>
</dataLink
<controlLink name=“A_For” source=“A” target=“for” />
<controlLink name=“For_B” source=“for” target=“B” controlPort=“YES”/>
<controlLink name=“For_C” source=“for” target=“C” controlPort=“NO”/>
<controlLink name=“B_For” source=“B” target=“for” />
Service Workflow Language
(SWFL)
• SWFL is XML based for describing
interacting Web services.
• Extends Web Service Flow Language
(WSFL) by
– supporting the application of all the
conditional and loop control constructs of
the Java language to the composition of
Web services.
– allowing more general data mappings.
Another Picture
swfl
swfl
Job
Job
Drawing
Drawing Screen
Screen
Create
Create SWFL
SWFL
Validator
Validator
Visual Service Composition Environment
Workflow
Workflow
Engine
Engine
Execution Environment
SWFL2Java
• It converts the description of a job in
SWFL into executable Java code.
SWFL
Description
Document
Dataflow
Graph
ControlFlow
Graph
FlowModel
Java
Executable
Code
Visual Service Composition
Environment
• Collaborative code development environment using a
visual programming tool for seamlessly integrating
services into higher-level services or complete
applications.
• Applications are created by plugging together services
to form a workflow graph. Service metadata is used to
check if interfaces are compatible.
• Different developers can place their components in a
shared repository, e.g, UDDI or Jini lookup service.
• Output is in the form of SWFL.
JISGA
• JISGA consists of two main parts:
– A WorkflowEngine service
– A JobProcessor service
• Grid application is submitted to a
WorkflowEngine service as SWFL.
• Sequential jobs are handled directly by the
WorkflowEngine service.
• Parallel jobs involve multiple JobProcessor
services.
What the Workflow Engine Is
• Processes service-based applications
described in an XML-based workflow
language.
• Determines the order of execution and
generates the harness code.
• Executes the code which includes
discovering services, invoking services,
receiving results and sending results to
clients.
What the Workflow Engine Does
• Converts a SWFL description of a
composite service-based job into an
executable Java code, and executes it.
SWFL2Java
<?xml…>
<JFlowModel
……………
………….
……….
….
SWFL2Graph Intermediate
FlowModel
object
Graph2Java
Java Executable
Code
SWFL description
To be continued…
Blocking Call
submitjob_block(…)
Harness code
SWFL
Document
Jini Services
Result
Client
Workflow
Engine
Non-blocking Call
submitjob_nonblock(…)
Harness code
SWFL
Document
Jini Services
Client
JobID
Workflow
Engine
JobID
Result
JavaSpaces
Sequential job processing
submitjob_block(String xmlDoc, byte[] inputData, boolean isParallel)
submitjob_nonblock(String xmlDoc, byte[] inputData, boolean isParallel)
A
B
C
E
A
D
TPSOG Algorithm
B
C
D
E
TPSOG: Task Processing Sequential Order Generating
Parallel Jobs
• WorkflowEngine service places parallel
jobs on Parallel Job Queue.
• JobProcessor service has two threads:
– One thread takes jobs from the PJQ are
splits them into sequential sub-jobs that it
places on the Sub-Job Queue.
– Other thread takes sub-jobs from the SJQ
and executes them.
Parallel Job Processing
submitjob_block(String xmlDoc, byte[] inputData, boolean isParallel)
submitjob_nonblock(String xmlDoc, byte[] inputData, boolean isParallel)
I
A
B
C
Job Partition
Algorithm
write
D
A
write
write
JavaSpace
E
take
II
take
B
take
C
F
III
IV
E
To be continued
Parallel Job Processing
SWFL description
JobProcessor
JPT
JobProcessor service
JavaSpaces
SPT
Job Partitioning Thread
Sub-job Processing Thread
PJQ
JobProcessor
JPT
SJQ
SPT
Future Research
• Further development of GSiB for creating,
deploying, managing and monitoring Grid
applications.
• More experiments in integration of Jini
services and Web services in JISGA.
• Demo applications to illustrate use of JISGA
on UK e-Science Grid.
• Integration of better scheduling in JISGA.
Scheduling
• Need to schedule components on
distributed resources to achieve goals
such as
– Minimum execution time, or
– Maximum throughput
• Achieving fair and efficient transparent
access to resources is a difficult
problem in a dynamic environment.
Final Comments
• Service-oriented approach to Grid computing
is generally agreed on
• But there are many options for how such
Grids might be build (GT3, Jini, JXTA, etc)
• Ability to interoperate between different Grids
is important.
• Need to get more experience with actually
using services for e-Science.